Post-installation configuration

• 1: Setting a Kubernetes cluster
• 2: Configuring a LiveSynchronization
• 3: Recovering from a Bucket
• 4: Resynchronization
• 5: Recovering from a disaster

1 - Setting a Kubernetes cluster

Introduction

Connection to both the source and destination clusters is set using the KubernetesCluster resource. Credentials are stored in Kubernetes secrets from which the KubernetesCluster collection access to connect to the clusters.

Requirements

• The kubeconfig file to access as read-only to the source cluster
• The kubeconfig file to access as cluster-admin to the destination cluster
• The Secret provided by AstroKube to access the Image Registry

Process

1. Prepare

Create Namespace

Create the namespace to configure the recovery process:

kubectl create namespace <namespace_name>

Setup registry credentials

Create the Secret that stores the credentials to the AstroKube image registry:

kubectl -n <namespace_name> create -f pull-secret.yaml

2. Configure the source Cluster

Create secret

Get the kubeconfig file that can be used to access the cluster, and save it as source-kubeconfig.yaml.

Then create the Secret with the following command:

kubectl -n <namespace_name> create secret generic source --from-file=kubeconfig.yaml=source-kubeconfig.yaml

Create resource

Define the ManagedCluster resource with the following YAML, and save it as sourcecluster.yaml:

apiVersion: assets.astronetes.io/v1alpha1
kind: KubernetesCluster
metadata:
  name: source
spec:
  secretName: <secret_name>

Deploy the resource with the following command:

kubectl create -f sourcecluster.yaml

3. Configure the destination Cluster

Create secret

Get the kubeconfig file that can be used to access the cluster, and save it as destination-kubeconfig.yaml.

Then create the Secret with the following command:

kubectl -n <namespace_name> create secret generic destination --from-file=kubeconfig.yaml=destination-kubeconfig.yaml

Create resource

Define the ManagedCluster resource with the following YAML, and save it as destinationcluster.yaml:

apiVersion: assets.astronetes.io/v1alpha1
kind: KubernetesCluster
metadata:
  name: destination
spec:
  secretName: <secret_name>

Deploy the resource with the following command:

kubectl create -f destinationcluster.yaml

2 - Configuring a LiveSynchronization

Introduction

A LiveSynchronization resource indicates a set of Kubernetes resource to replicate or synchronize between the source cluster and the destination cluster.

Requirements

• A KubernetesCluster resource for source cluster.
• A KubernetesCluster resource for destination cluster.

Process

1. Configure the live synchronization

Create the livesynchronization.yaml file according to your requirements. For this example, the goal is to synchronize deployments with the disaster-recovery label set to enabled. It is also desirable that when its replication is completed that no pod is created in the destination cluster and that after a recovery is launched the deployment launches active pods again.

Let’s dissect the following YAML:

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  suspend: false
  plugin: kubernetes-objects-to-kubernetes
  config:
    sourceName: source
    destinationName: destination
    observability:
      enabled: false
    replication:
      resources:
        - group: apps
          version: v1
          resource: deployments
          transformation:
            patch:
              - op: replace
                path: /spec/replicas
                value: 0
          filters:
            namespaceSelector:
              matchLabels:
                disaster-recovery: enabled
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1

spec.config.sourceName and spec.config.destinationName refers to the name and namespace of the KubernetesCluster resources for the corresponding clusters.

The spec.config.replication.resources is a list of the set of resources to deploy. A single LiveSynchronization can cover multiple types or groups of resources, although this example only manages deployments.

The type of the resource is defined at spec.config.replication.resources[*].resource. The filters can be located in spec.config.replication.resources[*].filters. In this case, the RecoveryPlan is matching the content of the disaster-recovery label.

The spec.config.replication.resources[*].transformation and spec.config.replication.resources[*].recoveryProcess establish the actions to take after each resource is synchronized and after they are affected by the recovery process respectively. In this case, while being replicated, each deployment will set their replicas to 0 in the destination cluster and will get back to one after a successful recovery. The resource parameters are always left intact in the source cluster.

2. Suspending and resumen a recovery plan

A keen eye might have noticed the spec.suspend parameter. In this example it is set to true to indicate that the recovery plan is inactive. An inactive or suspended recovery plan will not replicate new or existing resources until it is resumed. Resuming a recovery plan can be done by setting spec.suspend to false and applying the changes in yaml. Alternatively, a patch with kubectl will work as well and will not require the original yaml file:

kubectl patch livesynchronization <livesynchronization_name> -p '{"spec":{"suspend":false}}' --type=merge

3. Deploy the Live Synchronization

The live synchronizarion can be deployed as any other Kubernetes resource:

kubectl -n <namespace_name> apply -f livesynchronization.yaml

Additional steps

For more examples, take a look at our samples.

Modifying synchronized resources.

Depending on the use case and the chosen solution for Resiliency Operator, it is convenient that resources synchronized in the destination cluster differ from the original copy. Taking as example a warm standby scenario, in order to optimize infrastructure resources, certain objects such as Deployments or Cronjobs do not need to be actively running until there is a disaster. The standby destination cluster can run with minimal computing power and autoscale as soon as the recovery process starts, reducing the required overhead expenditure.

While a resource is being synchronized into the destination cluster, its properties can be transformed to adapt them to the organization necessities. Then, if and when a disaster occurs, the resource characteristics can be restored to either its original state or an alternative one with the established recover process.

Filters

FIlters are useful to select only the exact objects to synchronize. They are set in the spec.config.replication.resources[*].filters parameter.

Name selector

The nameSelector filters by the name of the resources of the version and type indicated. The following example selects only the Configmaps that follow the regular expression config.*:

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  plugin: kubernetes-objects-to-kubernetes
  suspend: false
  config:
    sourceName: source
    destinationName: destination
    observability:
      enabled: false
    replication:
      resources:
      - version: v1
        resource: configmaps
        filters:
          nameSelector:
            regex:
              - "config.*"

This selector can also be used negatively with excludeRegex. The following example excludes every configmap that ends in .test:

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  plugin: kubernetes-objects-to-kubernetes
  suspend: false
  config:
    sourceName: source
    destinationName: destination
    observability:
      enabled: false
    replication:
      resources:
      - version: v1
        resource: configmaps
        filters:
          nameSelector:
            excludeRegex:
            - "*.test"

Namespace selector

The namespaceSelector filters resources taking in consideration the namespace they belong to. This selector is useful to synchronize entire applications if they are stored in a namespace. The following example selects every deployment that is placed in a namespace with the label disaster-recovery: enabled:

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  plugin: kubernetes-objects-to-kubernetes
  suspend: false
  config:
    sourceName: source
    destinationName: destination
    observability:
      enabled: false
    replication:
      resources:
        - group: apps
          version: v1
          resource: deployments
          filters:
            selector:
              matchLabels:
                disaster-recovery: enabled

Transformations

Transformations are set in the spec.config.replication.resources[*].transformation parameter and are managed through patches.

Patch modifications alter the underlying object definiton using the same mechanism as kubectl patch. As with jsonpatch, the allowed operations are replace, add and remove. Patches are defined in the spec.config.replication.resources[*].transformation.patch list and admits an arbitary number of modifications.

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  ...
  config:
    ...
    replication:
      resources:
      - ...
        transformation:
          patch:
            - op: replace
              path: /spec/replicas
              value: 0
            - op: remove
              path: /spec/strategy

RecoveryProcess

The RecoveryProcess of a LiveSynchronization is executed in the case of a disaster to recover the original status of the application in the destination cluster. A resource can be either restored from the original definition stored in a bucket or by performing custom patches like with Transformations.

To restore from the original data, read the Recovering from a Bucket section. This option will disregard performed transformations and replace the parameters with those of the source cluster.

Patching when recovering is accessible at spec.config.replication.resources[*].recoveryProcess.fromPatch list and admits an arbitary number of modifications. It will act on the current state of the resource in the destination cluster, meaning it will take into consideration the transformations performed when it was synchronized unlike when recovering from original. As with jsonpatch, the allowed operations are replace, add and remove.

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  ...
  config:
    ...
    replication:
      resources:
        - ...
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1

3 - Recovering from a Bucket

Introduction

A Bucket resource indicates an Object Storage that will be used to restore original objects when recovering from a disaster.

Object Storage stores data in an unstructured format in which each entry represents an object. Unlike other storage solutions, there is not a relationship or hierarchy between the data being stored. Organizations can access their files as easy as with traditional hierarchical or tiered storage. Object Storage benefits include virtually infinite scalability and high availability of data.

Many Cloud Providers include their own flavor of Object Storage and most tools and SDKs can interact with them as their share the same interface. Resiliency Operator officially supports the following Object Storage solutions:

AWS Simple Storage Service (S3) Google Cloud Storage

Resiliency Operator can support multiple buckets in different providers as each one is managed independently.

Contents stored in a bucket

A bucket is assigned to a LiveSynchronization by setting it in a spec.config.bucketName item. It stores every synchronized object in the destination cluster with some internal control annotations added. In the case of a disaster, resources with recoveryProcess.fromOriginal.enabled equal to true will be restored using the bucket configuration.

The path of a stored object is as follows: <bucket_namespace>/<bucket_name>/<object_group-version-resource>/<object_namespace>.<object_name>.

Requirements

• At least an instance of a ObjectStorage service in one of the supported Cloud Providers. This is commonly known as a bucket and will be referred as so in the documentation.
• At least one pair of accessKeyID and secretAccessKey that gives both write and read permissions over all objects of the bucket. Refer to the chosen cloud provider documentation to learn how to create and extract them. It is recommended that each access key pair has only access to a single bucket.

Preparing and setting the bucket

• Cloud Providers
• Google Cloud Storage
• Amazon Web Services S3

apiVersion: v1
kind: Secret
metadata:
  name: bucket-credentials
stringData:
  accessKeyID: <access_key_id>
  secretAccessKey: <secret_access_key>

apiVersion: assets.astronetes.io/v1alpha1
kind: Bucket
metadata:
  name: gcp
  namespace: <namespace>
spec:
  generic:
    endpoint: storage.googleapis.com
    name: <bucket_name>
    useSSL: true
    secretName: bucket-credentials

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  plugin: kubernetes-objects-to-kubernetes
  config:
    sourceName: source
    destinationName: destination
    bucketName: <bucket_object_name>
    observability:
      enabled: false
    replication:
      resources:
        - group: apps
          version: v1
          resource: deployments
          transformation:
            patch:
              - op: replace
                path: /spec/replicas
                value: 0
          filters:
            namespaceSelector:
              matchLabels:
                env: pre
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1
        - group: apps
          version: v1
          resource: deployments
          transformation:
            patch:
              - op: replace
                path: /spec/replicas
                value: 0
          filters:
            namespaceSelector:
              matchLabels:
                env: pre-second
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1
        - group: ""
          version: v1
          resource: services
          filters:
            namespaceSelector:
              matchLabels:
                env: pre
        - group: ""
          version: v1
          resource: services
          filters:
            namespaceSelector:
              matchLabels:
                env: pre-second
        - group: ""
          version: v1
          resource: secrets
          filters:
            namespaceSelector:
              matchLabels:
                env: pre

apiVersion: v1
kind: Secret
metadata:
  name: bucket-credentials
stringData:
  accessKeyID: <access_key_id>
  secretAccessKey: <secret_access_key>

apiVersion: assets.astronetes.io/v1alpha1
kind: Bucket
metadata:
  name: gcp
spec:
  generic:
    endpoint: s3.<bucket-region>.amazonaws.com
    name: <bucket-name>
    useSSL: true
    secretName: bucket-credentials

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: livesynchronization-sample
spec:
  plugin: kubernetes-objects-to-kubernetes
  config:
    sourceName: source
    destinationName: destination
    bucketName: <bucket_object_name>
    observability:
      enabled: false
    replication:
      resources:
        - group: apps
          version: v1
          resource: deployments
          transformation:
            patch:
              - op: replace
                path: /spec/replicas
                value: 0
          filters:
            namespaceSelector:
              matchLabels:
                env: pre
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1
        - group: apps
          version: v1
          resource: deployments
          transformation:
            patch:
              - op: replace
                path: /spec/replicas
                value: 0
          filters:
            namespaceSelector:
              matchLabels:
                env: pre-second
          recoveryProcess:
            fromPatch:
              - op: replace
                path: /spec/replicas
                value: 1
        - group: ""
          version: v1
          resource: services
          filters:
            namespaceSelector:
              matchLabels:
                env: pre
        - group: ""
          version: v1
          resource: services
          filters:
            namespaceSelector:
              matchLabels:
                env: pre-second
        - group: ""
          version: v1
          resource: secrets
          filters:
            namespaceSelector:
              matchLabels:
                env: pre

4 - Resynchronization

Introduction

Due to particular circumstances it might be possible that there are objects that were not synchronized from the source cluster to the destination cluster. To cover this case, Resiliency Operator offers a reconciliation process that adds, deletes or updates objects in the destination cluster if its state differs from the source.

Architecture

Reconciliation is performed at the LiveSynchronization level. Every Live Synchronization is in charge of their covered objects and that they are up to date with the specification. Reconciliation is started by two components, EventsListener and Reconciler. The former is in charge of additive reconciliation and the latter of substractive reconciliation.

Additive reconciliation

Refers to the reconciliation of missing objects that are present in the source cluster but, for any reason, are not present or are not up to date in the destination cluster. The entry point is the EventsListener service which receives events with the current state in the source cluster of all the objects covered by the Recovery Plan with a period of one hour by default.

These resync events are then treated like regular events and follow the syncronization communication flow. If the object does not exist in the destination cluster, the Synchronizer will apply it. In the case of updates, only those with a resourceVersion greater than the existing one for that object will be applied, updating the definition of said object.

Substractive reconciliation

In the case that an object was deleted in the source cluster but it was not in the destination, the Additive Reconciliation will not detect it. The source cluster can send events containing the current state of its existing components, but not of those that ceased to exist in it.

For that, the Reconciler is activated with a period of one hour by default. It compares the state of the objects covered the Recovery Plan in both source and destination clusters. If a change is found, it creates a delete event in the NATS. This event is then processed as an usual delete event throughout the rest of the communication process.

Modifying the periodic interval

By default, the resynchronization process will be launched every hour. It can be changed by modifying the value at spec.config.resyncPeriod in the LiveSynchronization object. The admitted format is %Hh%Mm%Ss e.g. 1h0m0s for intervals of exactly one hour. Modifying this variable updates the schedule for both additive and substractive reconciliations.

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: resync-3h-25m-12s
spec:
  ...
  config:
    replication:
      resyncPeriod: 3h25m12s

5 - Recovering from a disaster

Previous steps

Task CRD

After defining a LiveSynchronization, a Task resource will be created in the destination cluster. The operator processes the spec.config.reaplication.resources[*].recoveryProcess parameter to define the required steps to activate the dormant applications. Taking as an example the following definition:

apiVersion: automation.astronetes.io/v1alpha1
kind: Task
metadata:
  name: set-test-label
spec:
  plugin: kubernetes-objects-transformation
  config:
    resources:
      - identifier:
          group: apps
          version: v1
          resources: deployments
        patch:
          operations:
            - op: replace
              path: '/labels/test'
              value: 'ok'

Every label with key test in a Deployment will be replaced with the value ok. This Task originates from the following LiveSynchronization object:

apiVersion: automation.astronetes.io/v1alpha1
kind: LiveSynchronization
metadata:
  name: set-test-label
spec:
  plugin: kubernetes-objects-to-kubernetes
  suspend: false
  config:
    sourceName: source
    destinationName: destination
    observability:
      enabled: false
    replication:
      resources:
        - group: apps
          version: v1
          resource: deployments
          recoveryProcess:
            fromPatch:
            - op: replace
              path: '/labels/test'
              value: 'ok'

This object should not be tempered with. It is managed by their adjacent LiveSynchronization.

On the day of a disaster

Recovering from a disaster will require the deployment of a TaskRun resource per Task that applies to recover the system and applications. The following example executes the TaskRun resource defined in the previous section:

apiVersion: automation.astronetes.io/v1alpha1
kind: TaskRun
metadata:
  name: restore-apps
spec:
  taskName: set-test-label