Plugins

Kardinal leverages a plugin system that allows developers to encode logic of how dev versions of stateful services (databases, queues, APIs) and external services (Stripe API, Mailchimp, managed database like Amazon RDS or Neon DB) should be handled. Plugins work by offering developers an api to write Python-based scripts that can dynamically alter Kubernetes deployment specs, providing a way to handle stateful and external services in a safer manner.

Kardinal supports a number of plugins already, with more currently in development or planned for future development. If there is a particular plugin you are interested in that is not on this list, please open a github issue with more details.

How It Works

1. Plugin Execution: Kardinal executes the specified plugins when creating or deleting a flow.
2. Pod Spec Modification: Plugins can modify the pod specifications, for all the dependants before it's applied to the cluster.
3. Config Map Generation: Plugins can generate a config map to store information for later use, particularly during flow deletion.

Designing Plugins

Plugins are Python scripts hosted on GitHub. Each plugin should have two main functions:

1. create_flow: Called when creating a new flow.
2. delete_flow: Called when deleting a flow.

Basic Plugin Structure

# main.py

def create_flow(service_specs: list, pod_specs: list, flow_uuid, optional_argument):
    # Modify the pod specs from all dependants (all services using same plugin)
    # Generate a config map if needed
    # service_specs - the list of Kubernetes service spec json
    # pod_specs - pod spec list from the dependants
    # flow_uuid - the uuid of the flow string
    # optional_argument - you can have any number of these, passed via annotations
    return {
        "pod_specs": modified_pod_specs,
        "config_map": config_map
    }

def delete_flow(config_map, flow_uuid):
    # Perform any necessary cleanup
    pass

Plugin Guidelines

• You need a main.py in the root of your repository with the above structure
• Modify the pod_specs as needed in the create_flow function.
• Use the config_map to store information that might be needed during flow deletion.
• If your plugin has external dependencies, include a requirements.txt file in the root of your repository.

Example Plugin

Here's an example of a simple plugin that replaces text in various parts of the deployment spec:

REPLACED = "the-text-has-been-replaced"

def create_flow(service_specs: list, pod_specs: list, flow_uuid, text_to_replace):
    modified_pod_specs = []

    for pod_spec in pod_specs:
        modified_pod_spec = copy.deepcopy(pod_spec)
        modified_pod_spec['containers'][0]['name'] = modified_pod_spec['containers'][0]['name'].replace(text_to_replace, REPLACED)

        modified_pod_specs.append(modified_pod_spec)

    config_map = {
        "original_text": text_to_replace
    }

    return {
        "pod_specs": modified_pod_specs,
        "config_map": config_map
    }


def delete_flow(config_map, flow_uuid):
    print(config_map["original_text"])

Plugin Design Best Practices

1. Documentation: Clearly document your plugin's purpose, required arguments, and effects on the pod specs.
2. Dependency Management: If your plugin requires external libraries, list them in a requirements.txt file in the root of your repository.
3. Error Handling: Implement proper error handling; raise an error and a non zero exit code if the plugin fails
4. Config Map Usage: Use the config map to store any information that might be needed during the delete_flow operation.

Defining and associating Plugins with Kubernetes services

In the Kardinal system, plugins are defined using a Kubernetes service with the kardinal.dev.service/plugin-definition annotation It's important to mention this service will be only used for Kardinal to create a plugin and it won't represent a service resource in the cluster.

Annotation Structure

The kardinal.dev.service/plugins annotation uses a YAML-formatted list of plugins. Each plugin in the list has four main parameters:

1. name: This is the GitHub repository URL of the plugin.
2. args: These are the arguments that will be passed to the plugin's create_flow function.
3. type: The type of service a plugin is being used for - currently the two options are stateful and external (optional)
4. serviceName: The name to refer to this service as in the Kardinal topology, it should be unique. (required)

A plugin repository can be used in multiple plugin definitions as long as they use different service names

Here's an example of how your plugins definition might look:

apiVersion: v1
kind: Service
metadata:
  name: my-awesome-plugin
  annotations:
    kardinal.dev.service/plugin-definition: |
      - name: github.com/kurtosis-tech/redis-plugin
        type: stateful
        servicename: my-awesome-plugin
        args:
          text_to_replace: "original-text"

Associating Plugins with Kubernetes Services

In the Kardinal system, plugins are associated with specific services in your Kubernetes cluster using annotations. This allows you to specify which plugins should be applied to each service, providing fine-grained control over your deployment modifications.

How to Tag a Service Spec

To use a plugin with a particular service, you need to add special annotations to your Kubernetes service specification. Here's how you can do it:

1. Open your Kubernetes service specification YAML file.
2. Add an annotation under the metadata section of your service.
3. Use the kardinal.dev.service/plugins key to specify the plugins you want to use.

Here's an example of how your service spec might look:

apiVersion: v1
kind: Service
metadata:
  name: my-awesome-service
  annotations:
    kardinal.dev.service/plugins: "my-awesome-plugin"
spec:
  selector:
    app: my-awesome-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

You can specify multiple plugins for a single service by adding more plugin service names separated by a comma:

annotations:
  kardinal.dev.service/plugins: "my-awesome-plugin,another-plugin"

Multiple services can reference to the same plugin through its service name

Plugin Execution

When creating a dev flow, Kardinal determines which services to create a dev version of. For all services that require a dev version, Kardinal will:

1. Read the kardinal.dev.service/plugins annotation.
2. For each plugin listed:
   • Fetch the plugin code from the specified GitHub repository.
   • Execute the plugin's create_flow function, passing in the service specs, pod specs, a generated flow UUID, and any arguments specified in the args section.
   • Create a dev version of the service based on the pod spec returned by the plugin.

When deleting the dev flow and removing dev versions of services, Kardinal will call the delete_flow function on all plugins to clean up resources. For example, in the neondb-plugin, the delete_flow operation cleans up the Neon database branch created for that dev flow.

Plugin Annotation Best Practices

1. Argument Naming: Use clear, descriptive names for your plugin arguments.
2. Plugin Order: If using multiple plugins, consider their order as they will be applied sequentially.

By using annotations in your Kubernetes service specs, you can easily associate Kardinal plugins with specific services. This allows for powerful, targeted modifications to your deployments, enhancing the flexibility and manageability of your Kubernetes applications.

Types of Plugins

Currently, there are two ways to use plugins in your Kubernetes application - on a stateful service inside the cluster or on an external service. The usage of plugins for these cases slightly differs.

Stateful Service Plugins

In order for Kardinal to guarantee data isolation and safety, Kardinal needs to know how to create "dev" versions of stateful services in your cluster. The level of data isolation and semantics of "dev" version will be highly dependent on your service, application, and development needs. This is where we can leverage a plugin to encode this information.

For example, say we have a postgres database in our cluster. When creating dev flows, we'll avoid touching the "baseline" postgres database by using the postgres-seed-plugin like so:

...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-v1
  labels:
    app: postgres
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
      version: v1
  template:
    metadata:
      labels:
        app: postgres
        version: v1
    spec:
      containers:
        - name: postgres
          image: 'postgres:14'
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: "cart"
            - name: POSTGRES_USER
              value: "postgresuser"
            - name: POSTGRES_PASSWORD
              value: "postgrespass"
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-data
      volumes:
        - name: postgres-data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
    version: v1
  annotations:
    kardinal.dev.service/stateful: "true"
    kardinal.dev.service/plugins: "postgres-seed-plugin"
spec:
  type: ClusterIP
  ports:
    - name: tcp
      port: 5432
      targetPort: 5432
      protocol: TCP
  selector:
    app: postgres
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-seed-plugin
  annotations:
    kardinal.dev.service/plugin-definition: |
      - name: github.com/kurtosis-tech/postgres-seed-plugin
        type: stateful
        servicename: postgres-seed-plugin
        args:
          seed_script: |
            --  create the table
            CREATE TABLE IF NOT EXISTS public.items(
                id bigserial PRIMARY KEY,
                created_at TIMESTAMP WITH TIME ZONE,
                updated_at TIMESTAMP WITH TIME ZONE,
                deleted_at TIMESTAMP WITH TIME ZONE,
                user_id TEXT,
                product_id TEXT,
                quantity INTEGER
            );

            INSERT INTO public.items (id, created_at, updated_at, deleted_at, user_id, product_id, quantity)
            VALUES (1, '2024-08-02 13:02:07.656104 +00:00', '2024-08-02 13:02:07.656104 +00:00', null, '0494c5e0-dde0-48fa-a6d8-f7962f5476bf', '66VCHSJNUP', 1);

            INSERT INTO public.items (id, created_at, updated_at, deleted_at, user_id, product_id, quantity)
            VALUES (2, '2024-08-02 13:02:10.891407 +00:00', '2024-08-02 13:02:10.891407 +00:00', null, '0494c5e0-dde0-48fa-a6d8-f7962f5476bf', '2ZYFJ3GM2N', 1);

            -- Set the sequence to the correct value after inserting records
            SELECT setval('public.items_id_seq', (SELECT MAX(id) FROM public.items));
          db_name: "cart"
          db_user: "postgresuser"
          db_password: "postgrespass"
...

Anytime a dev version of postgres is called, the postgres-seed-plugin is called that will return a pod spec starting a fresh instance of postgres using the provided seed script. This pod spec will then be used to create the dev postgres instance. Notice how the plugin encapsulates the semantics of the "dev" version of postgres in our instance - and can be updated in case our requirements ever change.

External Service Plugins

External services are any service outside a clusters that a service inside depends on. This can be managed databases or queues or APIs like Stripe or Mailchimp that a service inside depends on. (Note external services can also be stateful!) Using plugins to handle dev versions of external services works very similarly.

The difference is that the plugin annotation gets added to the service spec of the service that depends on the external service. Accordingly, when writing the plugin, the plugin will be modifying the pod spec of the dependent service.

For example, say we have a cartservice in our app that depends on an external Neon DB. When creating dev flows, we'll avoid touching the "baseline" Neon database by using the neondb-plugin like so:

...
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cartservice-v1
  labels:
    app: cartservice
    version: v1
spec:
  selector:
    matchLabels:
      app: cartservice
      version: v1
  template:
    metadata:
      labels:
        app: cartservice
        version: v1
    spec:
      terminationGracePeriodSeconds: 5
      containers:
        - name: server
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - all
            privileged: false
            readOnlyRootFilesystem: true
          image: tedim52/newobd-cartservice:0.0.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8090
          readinessProbe:
            httpGet:
              path: /health
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health
              port: 8090
            initialDelaySeconds: 15
            periodSeconds: 20
            timeoutSeconds: 5
            failureThreshold: 3
          env:
            # if POSTGRES is set, uses this to connect
            # otherwise uses environment variables below
            - name: POSTGRES
              value: ""
            - name: PORT
              value: "8090"
            - name: DB_USERNAME
              value: "postgresuser"
            - name: DB_PASSWORD
              value: "postgrespass"
            - name: DB_HOST
              value: "postgres"
            - name: DB_PORT
              value: "5432"
            - name: DB_NAME
              value: "cart"
---
apiVersion: v1
kind: Service
metadata:
  name: cartservice
  labels:
    app: cartservice
    version: v1
  annotations:
    kardinal.dev.service/plugins: "neon-postgres-plugin"
spec:
  type: ClusterIP
  selector:
    app: cartservice
  ports:
    - name: http
      port: 8090
      targetPort: 8090
      protocol: TCP
      appProtocol: HTTP
---
apiVersion: v1
kind: Service
metadata:
  name: neon-postgres-plugin
  annotations:
    kardinal.dev.service/plugin-definition: |
      - name: https://github.com/kurtosis-tech/neondb-plugin.git
              type: external
              servicename: neon-postgres-plugin
              args:
                NEON_API_KEY: ""
                NEON_PROJECT_ID: ""
                NEON_FORK_FROM_BRANCH_ID: ""
...

Anytime a dev version of cartservice is called, the neondb-plugin is called. The plugin will create a dev database branch, forked off of main. It will then return a modified pod spec starting a dev version of the cartservice that points to the dev database branch. Notice how here, the plugin annotation goes on the service depending on the external service and this is the deployment.template spec being modified, in this case, the cartservice. Also, notice we add type:external and serviceName:neon-postgres-db to tell Kardinal this service is external and how to refer to it.