Getting Started
Architecture
Concepts
References
Plugins
Kardinal leverages a plugin system that allows developers to encode logic of how dev versions of stateful services (databases, queues, APIs) and external services (Stripe API, Mailchimp, managed database like Amazon RDS or Neon DB) should be handled. Plugins work by offering developers an api to write Python-based scripts that can dynamically alter Kubernetes deployment specs, providing a way to handle stateful and external services in a safer manner.
Kardinal supports a number of plugins already, with more currently in development or planned for future development. If there is a particular plugin you are interested in that is not on this list, please open a github issue with more details.
Plugin | Description | Status |
---|---|---|
Redis Sidecar Plugin | Adds a thin layer over redis that allows for shared reads and isolated writes | ✅ Live |
Neon DB Plugin | Creates a new branch for the database you are using | ✅ Live |
Postgres Seed Plugin | Allows you to spin up a postgres database with seeded data | ✅ Live |
AWS RDS Plugin | Manage the AWS RDS service creation and deletion for dev flows | ✅ Live |
Dummy Plugin | Example Kardinal plugin, useful as a template | ✅ Live |
AWS SQS Plugin | Manage dev flow interactions with AWS SQS | 🚧 In development |
AWS S3 Plugin | Manage dev flow interactions with AWS S3 | 🕑 Planned |
MySQL Plugin | Manage dev flow interactions with MySQL | 🕑 Planned |
Cloud SQL Plugin | Manage dev flow interactions with Google Cloud SQL | 🕑 Planned |
Azure SQL Plugin | Manage dev flow interactions with Azure SQL | 🕑 Planned |
Stripe Plugin | Manage dev flow interactions with Stripe | 🕑 Planned |
How It Works
- Plugin Execution: Kardinal executes the specified plugins when creating or deleting a flow.
- Deployment Spec Modification: Plugins can modify the deployment specification before it's applied to the cluster.
- Config Map Generation: Plugins can generate a config map to store information for later use, particularly during flow deletion.
Designing Plugins
Plugins are Python scripts hosted on GitHub. Each plugin should have two main functions:
- create_flow: Called when creating a new flow.
- delete_flow: Called when deleting a flow.
Basic Plugin Structure
Plugin Guidelines
- You need a
main.py
in the root of your repository with the above structure - Modify the
pod_spec
as needed in thecreate_flow
function. - Use the
config_map
to store information that might be needed during flow deletion. - If your plugin has external dependencies, include a
requirements.txt
file in the root of your repository.
Example Plugin
Here's an example of a simple plugin that replaces text in various parts of the deployment spec:
Plugin Design Best Practices
- Documentation: Clearly document your plugin's purpose, required arguments, and effects on the deployment spec.
- Dependency Management: If your plugin requires external libraries, list them in a
requirements.txt
file in the root of your repository. - Error Handling: Implement proper error handling; raise an error and a non zero exit code if the plugin fails
- Config Map Usage: Use the config map to store any information that might be needed during the delete_flow operation.
Associating Plugins with Kubernetes Services
In the Kardinal system, plugins are associated with specific services in your Kubernetes cluster using annotations. This allows you to specify which plugins should be applied to each service, providing fine-grained control over your deployment modifications.
How to Tag a Service Spec
To use a plugin with a particular service, you need to add special annotations to your Kubernetes service specification. Here's how you can do it:
- Open your Kubernetes service specification YAML file.
- Add an annotation under the
metadata
section of your service. - Use the
kardinal.dev.service/plugins
key to specify the plugins you want to use.
Here's an example of how your service spec might look:
Annotation Structure
The kardinal.dev.service/plugins
annotation uses a YAML-formatted list of plugins. Each plugin in the list has four main parameters:
name
: This is the GitHub repository URL of the plugin.args
: These are the arguments that will be passed to the plugin'screate_flow
function.type
: The type of service a plugin is being used for - currently the two options arestateful
andexternal
(optional)serviceName
: The name to refer to this service as in the Kardinal topology. (required only forexternal
services)
You can specify multiple plugins for a single service by adding more items to the list:
Plugin Execution
When creating a dev flow, Kardinal determines which services to create a dev version of. For all services that require a dev version, Kardinal will:
- Read the
kardinal.dev.service/plugins
annotation. - For each plugin listed:
- Fetch the plugin code from the specified GitHub repository.
- Execute the plugin's
create_flow
function, passing in the service spec, deployment spec, a generated flow UUID, and any arguments specified in theargs
section. - Create a dev version of the service based on the deployment spec returned by the plugin.
When deleting the dev flow and removing dev versions of services, Kardinal will call the delete_flow
function on all plugins to clean up resources.
For example, in the neondb-plugin
, the delete_flow
operation cleans up the Neon database branch created for that dev flow.
Plugin Annotation Best Practices
- Argument Naming: Use clear, descriptive names for your plugin arguments.
- Plugin Order: If using multiple plugins, consider their order as they will be applied sequentially.
By using annotations in your Kubernetes service specs, you can easily associate Kardinal plugins with specific services. This allows for powerful, targeted modifications to your deployments, enhancing the flexibility and manageability of your Kubernetes applications.
Types of Plugins
Currently, there are two ways to use plugins in your Kubernetes application - on a stateful service inside the cluster or on an external service. The usage of plugins for these cases slightly differs.
Stateful Service Plugins
In order for Kardinal to guarantee data isolation and safety, Kardinal needs to know how to create "dev" versions of stateful services in your cluster. The level of data isolation and semantics of "dev" version will be highly dependent on your service, application, and development needs. This is where we can leverage a plugin to encode this information.
For example, say we have a postgres
database in our cluster. When creating dev flows, we'll avoid touching the "baseline" postgres database by using the postgres-seed-plugin
like so:
Anytime a dev version of postgres
is called, the postgres-seed-plugin
is called that will return a deployment spec starting a fresh instance of postgres using the provided seed script.
This deployment spec will then be used to create the dev postgres instance. Notice how the plugin encapsulates the semantics of the "dev" version of postgres in our instance - and can be updated in case our requirements ever change.
External Service Plugins
External services are any service outside a clusters that a service inside depends on. This can be managed databases or queues or APIs like Stripe or Mailchimp that a service inside depends on. (Note external services can also be stateful!) This can be managed databases or queues or APIs like Stripe or Mailchimp. Using plugins to handle dev versions of external services works very similarly.
The difference is that the plugin annotation gets added to the service spec of the service that depends on the external service. Accordingly, when writing the plugin, the plugin will be modifying the deployment spec of the dependent service.
For example, say we have a cartservice
in our app that depends on an external Neon DB. When creating dev flows, we'll avoid touching the "baseline" Neon database by using the neondb-plugin
like so:
Anytime a dev version of cartservice
is called, the neondb-plugin
is called. The plugin will create a dev database branch, forked off of main. It will then return a modified deployment spec starting a dev version of the cartservice that points to the dev database branch.
Notice how here, the plugin annotation goes on the service depending on the external service and this is the deployment spec being modified, in this case, the cartservice
.
Also, notice we add type:external
and serviceName:neon-postgres-db
to tell Kardinal this service is external and how to refer to it.