1 - Introduction

Synchronization introduction

Synchronization is a critical process that enables the replication of data and configurations across different platform assets. This ensures consistency, integrity and improve the platform resiliency.

Key concepts

Source and destination

Each synchronization has at least two assets:

  • Source: the original location or system from which data and configurations are retrived.
  • Destination: the destination location or system where data and configurations are applied or updated.

Synchronization periodicity

There are three distinct types of synchronization processes designed to meet different operational needs: Synchronization, SynchronizationPlan, and LiveSynchronization.

Synchronization

The Synchronization process is designed to run once, making it ideal for one-time data alignment tasks or initial setup processes. This type of synchronization is useful when a system or component needs to be brought up-to-date with the latest data and configurations from another source without ongoing updates.

The synchronization process follows these rules:

  • Object exists in Source: If a matching object exists in the source asset, it will be synchronized to the destination asset.
  • Object only in Destination: If a matching object exists only in the destination asset, it will be removed from the destination asset.

SynchronizationPlan

The SynchronizationPlan process operates similarly to a cron job, allowing synchronization tasks to be scheduled at regular intervals. This type is ideal for systems that require periodic updates to ensure data and configuration consistency over time without the need for real-time accuracy.

LiveSynchronization

LiveSynchronization provides real-time synchronization, continuously monitoring and updating data and configurations as changes occur. This type of synchronization is essential for environments where immediate consistency and up-to-date information are crucial.

The synchronization process follows these rules:

  • Object Creation/Update in Source: If a matching object is created or updated in the source asset, it will be synchronized to the destination asset.
  • Object Deletion in Source: If a matching object is deleted in the source asset, the corresponding object will be deleted in the destination asset.
  • Object Creation/Update in Destination: If a matching object is created or updated in the destination asset, it will be synchronized from the source asset.
  • Object Deletion in Source: If a matching object is deleted in the source asset, the corresponding object will be deleted in the destination asset.
  • Object Only in Destination: If a matching object exists only in the destination asset, it will be removed from the destination asset.

Resume

PeriodicityDescription
SynchronizationSynchronize data and configurations only once.
SynchronizationPlanSynchronize data and configurations based on a scheduled period.
LiveSynchronizationReal-time synchronization of data and configurations.

Prerequisites

Before initiating the Synchronization process, ensure the following prerequisites are met:

  • Both source and destiation systems have been defined as Asset.
  • There is a network connectivity between the assets and the operator.

2 - Bucket to Kubernetes

Synchronize Bucket files to Kubernetes

2.1 - Introduction

Bucket to Kubernetes introduction

Bucket files can be synchronized in a Kubernetes cluster as Kubernetes objects using the bucket-to-kubernetes plugin.

Synchronization process

Special rules

There are no special rules for this synchronization plugin.

Blacklisted objects

There are some Kubernetes objects that are blacklisted by default and will be ignored by the synchronization process.

This is the list of the blacklisted objects:

  • Namespaces which name starts with kube- or openshift , an the Namespace resiliency-operator.
  • All namespaced objects inside a blacklisted Namespace.
  • ConfigMaps named kube-root-ca.crt or openshift-service-ca.crt.

Objects path

The files from the bucket will be read as Kubernetes objectsfrom files with the following path: <group>.<version>.<kind>/<object_namespace>.<object_name>.

Examples:

  • The Namespace named test will be saved in the file core.v1.Namespace/test.
  • The Deployment named app-1 deployed in the test Namespace will be saved in the file apps.v1.Deployment/test.app-1.

Use cases

Backup restore

Restore your Kubernetes cluster from a Bucket.

Pilot light architecture

Recover from a disaster by running the application saved in the Bucket.

2.2 - Configuration

Bucket to Kubernetes configuration

Introduction

The synchronization process can be configured with some specific parameters using the .spec.config attribute.

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...

Required configuration

Source and destination

The source bucket and the destination cluster can be specified using the .spec.config.sourceName and .spec.config.destinationName properties. Both the KubernetesCluster and the bucket objects should exists in the same Namespace where the synchronization is being created.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    sourceName: bucket-1
    destinationName: cluster-1
    ...

Selectors

The resources that should be synchronized can be configured using the .spec.config.selectors property.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    selectors:
      - target:
          version: v1
          resources:
            - namespaces
        objectSelector:
          labelSelector:
            matchLabels:
              env: pro

Optional configuration

Global selectors

Global selectors are used to set the default value on all selectors defined in .spec.config.selectors. They can be configured with the .spec.config.globalSelector property.

There are two options allowed, that can be configured at the same time:

  • namespaceSelector: to set e dafult namespace selector for namespaced resources
  • objectSelector: to set a default object selector for all resources

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    globalSelector:
      namespaceSelector:
        labelSelector:
          matchLabels:
            env: pro
      objectSelector:
        labelSelector:
          matchLabels:
            env: pro

Path prefix

The Kubernetes objects can be written in a subdirectory of the destination Bucket. The property .spec.config.pathPrefix allows this configuration.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    pathPrefix: prefix-path

Log level

The log level of the Pod deployed to execute the synchronization, can be configured with the .spec.config.logLevel parameter.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    logLevel: warn

Observability

Observability can be enaled using the specific .spec.config.observability parameter. For more information check the Observability page.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    observability:
      enabled: true
      interval: 2m

Limit assigned resources

For each synchronization, a new Pod is deployed inside the same Namespace. The limit and requests resources can be set using the .spec.config.resources field.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 2
        memory: 2Gi

Concurrency

The concurrency parameter can be used to improve the peformance of the synchronization process with .spec.config.concurrency`.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    concurrency: 200

Transformations

Objects from the source bucket can be transformed before being synchronized into the destination cluster.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    transformations:
      resources:
        - version: v1
          resources:
            - namespaces
      operations:
        - jsonpatch:
            operations:
              - op: add
                path: /metadata/labels/test-astrosync
                value: ok

API Reference

NameDescriptionTypeRequired
sourceNameBucket namestringyes
destinationNameKubernetesCluster namestringyes
selectorsThe Kubernetes resources to be syncrhonized[]KubernetesObjectSelectoryes
globalSelectorGlobal selectors to be applied to all selectorsKubernetesGlobalSelectoryes

2.3 - Observability

Obervability

Introduction

The synchronization process exports many metrics in Prometheus format over HTTP.

Exported metrics

The following metrics are available:

MetricDescription
astronetes_synchronization_statusThe status of each synchronization object.
astronetes_synchronization_status_totalThe count of synchronization objects for each state.

Requirements

  • The ServiceMonitor CRD from prometheus (servicemonitors.monitoring.coreos.com) must be enabled in the cluster where the operator is running.

Processes

Enable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to true.

Once enabled the observability, a ServiceMonitor will be created in the same Namespace of the related synchronization.

Disable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to false.

2.4 - API Reference

Configuration details

Config

Configuration for Bucket to Kubernetes synchronization

FieldDescriptionTypeRequired
concurrencyConcurrent processes to be executed to improve performanceintfalse
destinationNameKubernetesCluster name where data will be synchronizedstringfalse
globalSelectorOverrides selectors propertiesKubernetesGlobalSelectorfalse
logLevelLog level to be used by the synchronization Podstringfalse
observabilityObservability configurationObservabilityConfigfalse
optionsSynchronization optionsSynchronizationOptionsfalse
pathPrefixPath prefix to be used to retreive objects in the Bucketstringfalse
resourcesResources to be assigned to the synchronization PodResourceRequirementsfalse
selectorsSelectors to filter the Kubernetes resources to be synchronized[]KubernetesObjectSelectorfalse
sourceNameBucket name from where data will be readstringfalse
transformationsTransform Kubernetes objects before to be written to the destination[]Transformationsfalse
useCachedDataUse cached data instead of get data from Kubernetes clusters on startupboolfalse

KubernetesGlobalSelector

Global selector is used to set the default value on all selectors

FieldDescriptionTypeRequired
objectSelectorRules to filter Kubernetes objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorRules to filter Kubernetes objects by NamespaceSelectorNamespaceSelectorfalse

ObjectSelector

FieldDescriptionTypeRequired
nameSelectorFilter objects by their nameNameSelectorfalse
labelSelectorFilter objects by their labelsLabelSelectorfalse

NameSelector

Select object by their name

FieldDescriptionTypeRequired
includeRegexInclude names that matches at least one regex[]stringfalse
excludeRegexExlcude names that matches at least one regex[]stringfalse

LabelSelector

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects. +structType=atomic

FieldDescriptionTypeRequired
matchLabelsmatchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabelsmap is equivalent to an element of matchExpressions, whose key field is “key”, theoperator is “In”, and the values array contains only “value”. The requirements are ANDed.+optionalmap[string]stringfalse
matchExpressionsmatchExpressions is a list of label selector requirements. The requirements are ANDed.+optional+listType=atomic[]LabelSelectorRequirementfalse

LabelSelectorRequirement

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

FieldDescriptionTypeRequired
keykey is the label key that the selector applies to.stringfalse
operatoroperator represents a key’s relationship to a set of values.Valid operators are In, NotIn, Exists and DoesNotExist.LabelSelectorOperatorfalse
valuesvalues is an array of string values. If the operator is In or NotIn,the values array must be non-empty. If the operator is Exists or DoesNotExist,the values array must be empty. This array is replaced during a strategicmerge patch.+optional+listType=atomic[]stringfalse

LabelSelectorOperator

A label selector operator is the set of operators that can be used in a selector requirement.

FieldDescriptionTypeRequired

NamespaceSelector

FieldDescriptionTypeRequired
nameSelectorFilter Namespaces by their nameNameSelectorfalse
labelSelectorFilter Namespaces by their labelsLabelSelectorfalse

ObservabilityConfig

Configure the synchronization process observability using Prometheus ServiceMonitor

FieldDescriptionTypeRequired
enabledEnable the Observability with a Prometheus ServiceMonitorboolfalse
intervalConfigure the interval in the ServiceMonitor that Prometheus will use to scrape metricsDurationfalse

Duration

Duration is a wrapper around time.Duration which supports correct marshaling to YAML and JSON. In particular, it marshals into strings, which can be used as map keys in json.

FieldDescriptionTypeRequired

SynchronizationOptions

Customize the synchronization process with special options

FieldDescriptionTypeRequired
dryRunSimulate the synchronization process but don’t execute the write operationsboolfalse
forceSyncSynchronize object in the destination even if the object exists in the destination and it doesn’t match the configured selectorsboolfalse
forcePrunePrune object in the destination even if it doesn’t match the configured selectorsboolfalse
showLogIfObjectIsAlreadyInSyncShow a log message if object is already in syncboolfalse
showLogIfObjectHaveBeenAdaptedShow a log message if object have been adapted for the destinationboolfalse

KubernetesObjectSelector

FieldDescriptionTypeRequired
objectSelectorFilter objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorFilter objects by NamespaceSelectorNamespaceSelectorfalse
targetKubernetes resource to be usedGroupVersionResourcesfalse

GroupVersionResources

Select a set of GroupVersionResource

FieldDescriptionTypeRequired
groupKubernetes resource group. Example: appsstringfalse
versionKubernetes resource version. Example: v1stringfalse
resourcesKubernetes resource names. Example: deployments[]stringfalse

Transformations

Transformations is a list of operations to modifiy the Kubernetes objects matching the given selectors

FieldDescriptionTypeRequired
resourcesSelect the objects to be transfomred by their resource type[]GroupVersionResourcesfalse
namespaceSelectorFilter the objects to be transformed by NamespaceSelectorNamespaceSelectorfalse
objectSelectorFilter the objects to be transformed by ObjectSelectorObjectSelectorfalse
operationsOperations to be executed to transform the objects[]TransformationOperationfalse

TransformationOperation

The operation to execute to transform the objects

FieldDescriptionTypeRequired
jsonpatchJSONPatch operationOperationJSONPatchfalse

OperationJSONPatch

The JSONPatch operation

FieldDescriptionTypeRequired
skipIfNotFoundOnDeleteSkip if not found on deleteboolfalse
operationsList of operations to be executed[]JSONPatchOperationfalse

JSONPatchOperation

JSONPAtch operation

FieldDescriptionTypeRequired
opJSONPatch operation: add, copy, move, remove, replace, teststringfalse
pathExecute the operation to the given pathstringfalse
valueOptional value to be used in the operationinterface{}false

3 - Kubernetes to Bucket

Synchronize Kubernetes objects into a Bucket

3.1 - Introduction

Kubernetes to Bucket introduction

Kubernetes objects can be synchronized into a Bucket the kubernetes-to-bucket plugin.

Synchronization process

Special rules

There are no special rules for this synchronization plugin.

Blacklisted objects

There are some Kubernetes objects that are blacklisted by default and will be ignored by the synchronization process.

This is the list of the blacklisted objects:

  • Namespaces which name starts with kube- or openshift , an the Namespace resiliency-operator.
  • All namespaced objects inside a blacklisted Namespace.
  • ConfigMaps named kube-root-ca.crt or openshift-service-ca.crt.

Objects path

Each Kubernetes object will be stored into a file with the following path: <group>.<version>.<kind>/<object_namespace>.<object_name>.

Examples:

  • The Namespace named test will be saved in the file core.v1.Namespace/test.
  • The Deployment named app-1 deployed in the test Namespace will be saved in the file apps.v1.Deployment/test.app-1.

Use cases

Backups

Backup your Kubernetes cluster to a Bucket to recover data when required.

Pilot light architecture

Synchronize applications to a Bucket and recover the applications in another cluster after a disaster.

3.2 - Configuration

Kubernetes to Bucket configuration

Introduction

The synchronization process can be configured with some specific parameters using the .spec.config attribute.

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...

Required configuration

Source and destination

The source cluster and the destination bucket can be specified using the .spec.config.sourceName and .spec.config.destinationName properties. Both the KubernetesCluster and the bucket objects should exists in the same Namespace where the synchronization is being created.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    sourceName: cluster-1
    destinationName: bucket-1
    ...

Optional configuration

Selectors

The resources that should be synchronized can be configured using the .spec.config.selectors property. If not configured, all resources will be included in the synchronization.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    selectors:
      - target:
          version: v1
          resources:
            - namespaces
        objectSelector:
          labelSelector:
            matchLabels:
              disaster-recovery: "true"

Global selectors

Global selectors are used to set the default value on all selectors defined in .spec.config.selectors. They can be configured with the .spec.config.globalSelector property.

There are two options allowed, that can be configured at the same time:

  • namespaceSelector: to set e dafult namespace selector for namespaced resources
  • objectSelector: to set a default object selector for all resources

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    globalSelector:
      namespaceSelector:
        labelSelector:
          matchLabels:
            env: pro
      objectSelector:
        labelSelector:
          matchLabels:
            env: pro

Path prefix

The Kubernetes objects can be written in a subdirectory of the destination Bucket. The property .spec.config.pathPrefix allows this configuration.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    pathPrefix: prefix-path

Log level

The log level of the Pod deployed to execute the synchronization, can be configured with the .spec.config.logLevel parameter.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    logLevel: warn

Observability

Observability can be enaled using the specific .spec.config.observability parameter. For more information check the Observability page.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    observability:
      enabled: true
      interval: 2m

Limit assigned resources

For each synchronization, a new Pod is deployed inside the same Namespace. The limit and requests resources can be set using the .spec.config.resources field.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 2
        memory: 2Gi

Concurrency

The concurrency parameter can be used to improve the peformance of the synchronization process with .spec.config.concurrency`.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    concurrency: 200

Transformations

Kubernetes obejcts from the source cluster can be transformed before being synchronized into the destination bucket.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-bucket
  config:
    ...
    transformations:
      resources:
        - version: v1
          resources:
            - namespaces
      operations:
        - jsonpatch:
            operations:
              - op: add
                path: /metadata/labels/test-astrosync
                value: ok

3.3 - Observability

Obervability

Introduction

The synchronization process exports many metrics in Prometheus format over HTTP.

Exported metrics

The following metrics are available:

MetricDescription
astronetes_synchronization_statusThe status of each synchronization object.
astronetes_synchronization_status_totalThe count of synchronization objects for each state.

Requirements

  • The ServiceMonitor CRD from prometheus (servicemonitors.monitoring.coreos.com) must be enabled in the cluster where the operator is running.

Processes

Enable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to true.

Once enabled the observability, a ServiceMonitor will be created in the same Namespace of the related synchronization.

Disable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to false.

3.4 - API Reference

Configuration details

Config

Configuration for Kubernetes to Bucket synchronization

FieldDescriptionTypeRequired
concurrencyConcurrent processes to be executed to improve performanceintfalse
destinationNameBucket name where data will be synchronizedstringfalse
globalSelectorOverrides selectors propertiesKubernetesGlobalSelectorfalse
logLevelLog level to be used by the synchronization Podstringfalse
observabilityObservability configurationObservabilityConfigfalse
optionsSynchronization optionsSynchronizationOptionsfalse
pathPrefixPath prefix to be used to retreive objects in the Bucketstringfalse
resourcesResources to be assigned to the synchronization PodResourceRequirementsfalse
selectorsSelectors to filter the Kubernetes resources to be synchronized[]KubernetesObjectSelectorfalse
sourceNameKubernetesCluster name from where data will be readstringfalse
transformationsTransform Kubernetes objects before to be written to the destination[]Transformationsfalse
useCachedDataUse cached data instead of get data from assets on startupboolfalse

KubernetesGlobalSelector

Global selector is used to set the default value on all selectors

FieldDescriptionTypeRequired
objectSelectorRules to filter Kubernetes objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorRules to filter Kubernetes objects by NamespaceSelectorNamespaceSelectorfalse

ObjectSelector

FieldDescriptionTypeRequired
nameSelectorFilter objects by their nameNameSelectorfalse
labelSelectorFilter objects by their labelsLabelSelectorfalse

NameSelector

Select object by their name

FieldDescriptionTypeRequired
includeRegexInclude names that matches at least one regex[]stringfalse
excludeRegexExlcude names that matches at least one regex[]stringfalse

LabelSelector

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects. +structType=atomic

FieldDescriptionTypeRequired
matchLabelsmatchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabelsmap is equivalent to an element of matchExpressions, whose key field is “key”, theoperator is “In”, and the values array contains only “value”. The requirements are ANDed.+optionalmap[string]stringfalse
matchExpressionsmatchExpressions is a list of label selector requirements. The requirements are ANDed.+optional+listType=atomic[]LabelSelectorRequirementfalse

LabelSelectorRequirement

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

FieldDescriptionTypeRequired
keykey is the label key that the selector applies to.stringfalse
operatoroperator represents a key’s relationship to a set of values.Valid operators are In, NotIn, Exists and DoesNotExist.LabelSelectorOperatorfalse
valuesvalues is an array of string values. If the operator is In or NotIn,the values array must be non-empty. If the operator is Exists or DoesNotExist,the values array must be empty. This array is replaced during a strategicmerge patch.+optional+listType=atomic[]stringfalse

LabelSelectorOperator

A label selector operator is the set of operators that can be used in a selector requirement.

FieldDescriptionTypeRequired

NamespaceSelector

FieldDescriptionTypeRequired
nameSelectorFilter Namespaces by their nameNameSelectorfalse
labelSelectorFilter Namespaces by their labelsLabelSelectorfalse

ObservabilityConfig

Configure the synchronization process observability using Prometheus ServiceMonitor

FieldDescriptionTypeRequired
enabledEnable the Observability with a Prometheus ServiceMonitorboolfalse
intervalConfigure the interval in the ServiceMonitor that Prometheus will use to scrape metricsDurationfalse

Duration

Duration is a wrapper around time.Duration which supports correct marshaling to YAML and JSON. In particular, it marshals into strings, which can be used as map keys in json.

FieldDescriptionTypeRequired

SynchronizationOptions

Customize the synchronization process with special options

FieldDescriptionTypeRequired
dryRunSimulate the synchronization process but don’t execute the write operationsboolfalse
forceSyncSynchronize object in the destination even if the object exists in the destination and it doesn’t match the configured selectorsboolfalse
forcePrunePrune object in the destination even if it doesn’t match the configured selectorsboolfalse
showLogIfObjectIsAlreadyInSyncShow a log message if object is already in syncboolfalse
showLogIfObjectHaveBeenAdaptedShow a log message if object have been adapted for the destinationboolfalse

KubernetesObjectSelector

FieldDescriptionTypeRequired
objectSelectorFilter objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorFilter objects by NamespaceSelectorNamespaceSelectorfalse
targetKubernetes resource to be usedGroupVersionResourcesfalse

GroupVersionResources

Select a set of GroupVersionResource

FieldDescriptionTypeRequired
groupKubernetes resource group. Example: appsstringfalse
versionKubernetes resource version. Example: v1stringfalse
resourcesKubernetes resource names. Example: deployments[]stringfalse

Transformations

Transformations is a list of operations to modifiy the Kubernetes objects matching the given selectors

FieldDescriptionTypeRequired
resourcesSelect the objects to be transfomred by their resource type[]GroupVersionResourcesfalse
namespaceSelectorFilter the objects to be transformed by NamespaceSelectorNamespaceSelectorfalse
objectSelectorFilter the objects to be transformed by ObjectSelectorObjectSelectorfalse
operationsOperations to be executed to transform the objects[]TransformationOperationfalse

TransformationOperation

The operation to execute to transform the objects

FieldDescriptionTypeRequired
jsonpatchJSONPatch operationOperationJSONPatchfalse

OperationJSONPatch

The JSONPatch operation

FieldDescriptionTypeRequired
skipIfNotFoundOnDeleteSkip if not found on deleteboolfalse
operationsList of operations to be executed[]JSONPatchOperationfalse

JSONPatchOperation

JSONPAtch operation

FieldDescriptionTypeRequired
opJSONPatch operation: add, copy, move, remove, replace, teststringfalse
pathExecute the operation to the given pathstringfalse
valueOptional value to be used in the operationinterface{}false

4 - Kubernetes to Kubernetes

Synchronize Kubernetes objects between two clusters

4.1 - Introduction

Kubernetes to Kubernetes introduction

Kubernetes objects can be synchronized between clusters adopting the kubernetes-to-kubernetes plugin.

Synchronization process

Special rules

Additionally, there are some special rules for source kind of objects:

  • PersistentVolumeClaim: objects will be updated in the destination cluster only if .spec.resources changes.

  • ServiceAccount: If you’re using a Kubernetes cluster version <v1.24, when creating a ServiceAccount, an autogenerated secret will be created. From here, there are two scenarios:

    • Service account with an autogenerated secret: The autogenerated secret will be deleted, and a new one will be generated in the target cluster. The secret is always updated during each synchronization.

    • Service account with a custom secret: The custom secret will remain unchanged.

    To avoid errors, custom secrets must not follow the autogenerated secret naming structure. Secrets are considered autogenerated if they begin with the following prefixes:

    • {serviceAccountName}-token-
    • {serviceAccountName}-dockercfg-

Blacklisted objects

There are some Kubernetes objects that are blacklisted by default and will be ignored by the synchronization process.

This is the list of the blacklisted objects:

  • Namespaces which name starts with kube- or openshift , and the Namespace resiliency-operator.
  • All namespaced objects inside a blacklisted Namespace.
  • ConfigMaps named kube-root-ca.crt or openshift-service-ca.crt.

4.2 - Configuration

Kubernetes to Kubernetes configuration

Introduction

The synchronization process can be configured with some specific parameters using the .spec.config attribute.

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...

Required configuration

Source and destination

The source and the destination clusters can be specified using the .spec.config.sourceName and .spec.config.destinationName properties. Both KubernetesCluster objects should exists in the same Namespace where the synchronization is being created.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    sourceName: cluster-1
    destinationName: cluster-2
    ...

Selectors

The resources that should be synchronized between clusters can be configured using the .spec.config.selectors property.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    selectors:
      - target:
          version: v1
          resources:
            - namespaces
        objectSelector:
          labelSelector:
            matchLabels:
              disaster-recovery: "true"

Optional configuration

Global selectors

Global selectors are used to set the default value on all selectors defined in .spec.config.selectors. They can be configured with the .spec.config.globalSelector property.

There are two options allowed, that can be configured at the same time:

  • namespaceSelector: to set e dafult namespace selector for namespaced resources
  • objectSelector: to set a default object selector for all resources

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: bucket-to-kubernetes
  config:
    ...
    globalSelector:
      namespaceSelector:
        labelSelector:
          matchLabels:
            env: pro
      objectSelector:
        labelSelector:
          matchLabels:
            env: pro

Log level

The log level of the Pod deployed to execute the synchronization, can be configured with the .spec.config.logLevel parameter.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    logLevel: warn

Observability

Observability can be enaled using the specific .spec.config.observability parameter.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    observability:
      enabled: true
      interval: 2m

Limit assigned resources

For each synchronization, a new Pod is deployed inside the same Namespace. The limit and requests resources can be set using the .spec.config.resources field.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 2
        memory: 2Gi

Concurrency

The concurrency parameter can be used to improve the peformance of the synchronization process with .spec.config.concurrency`.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    concurrency: 200

Transformations

Kubernetes obejcts from the source cluster can be transformed before being synchronized into the destination cluster.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: kubernetes-to-kubernetes
  config:
    ...
    transformations:
      - resources:
          - group: ""
            version: v1
            resources:
              - namespaces
        namespaceSelector:
          labelSelector:
            matchLabels:
              sync: "true"
        operations:
          - jsonpatch:
              operations:
                - op: add
                  path: /metadata/labels/test-astrosync
                  value: ok

4.3 - Observability

Obervability

Introduction

The synchronization process exports many metrics in Prometheus format over HTTP.

Exported metrics

The following metrics are available:

MetricDescription
astronetes_synchronization_statusThe status of each synchronization object.
astronetes_synchronization_status_totalThe count of synchronization objects for each state.

Requirements

  • The ServiceMonitor CRD from prometheus (servicemonitors.monitoring.coreos.com) must be enabled in the cluster where the operator is running.

Processes

Enable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to true.

Once enabled the observability, a ServiceMonitor will be created in the same Namespace of the related synchronization.

Disable observability

Update the synchronization configuration, setting the paramente .spec.config.observability.enabled to false.

4.4 - API Reference

Configuration details

Config

Configuration for Kubernetes to Kubernetes synchronization

FieldDescriptionTypeRequired
concurrencyConcurrent processes to be executed to improve performanceintfalse
destinationNameKubernetesCluster name where data will be synchronizedstringfalse
globalSelectorOverrides selectors propertiesKubernetesGlobalSelectorfalse
logLevelLog level to be used by the synchronization Podstringfalse
observabilityObservability configurationObservabilityConfigfalse
optionsSynchronization optionsSynchronizationOptionsfalse
resourcesResources to be assigned to the synchronization PodResourceRequirementsfalse
selectorsSelectors to filter the Kubernetes resources to be synchronized[]KubernetesObjectSelectorfalse
sourceNameKubernetesCluster name from where data will be readstringfalse
transformationsTransform Kubernetes objects before to be written to the destination[]Transformationsfalse
useCachedDataUse cached data instead of get data from Kubernetes clusters on startupboolfalse

KubernetesGlobalSelector

Global selector is used to set the default value on all selectors

FieldDescriptionTypeRequired
objectSelectorRules to filter Kubernetes objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorRules to filter Kubernetes objects by NamespaceSelectorNamespaceSelectorfalse

ObjectSelector

FieldDescriptionTypeRequired
nameSelectorFilter objects by their nameNameSelectorfalse
labelSelectorFilter objects by their labelsLabelSelectorfalse

NameSelector

Select object by their name

FieldDescriptionTypeRequired
includeRegexInclude names that matches at least one regex[]stringfalse
excludeRegexExlcude names that matches at least one regex[]stringfalse

LabelSelector

Select object by their labels

FieldDescriptionTypeRequired
matchLabelsMatch object by the given labelsmap[string]stringfalse
matchExpressionsMatch object by the given expressions[]LabelSelectorRequirementfalse

LabelSelectorRequirement

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

FieldDescriptionTypeRequired
keykey is the label key that the selector applies to.stringfalse
operatoroperator represents a key’s relationship to a set of values.Valid operators are In, NotIn, Exists and DoesNotExist.LabelSelectorOperatorfalse
valuesvalues is an array of string values. If the operator is In or NotIn,the values array must be non-empty. If the operator is Exists or DoesNotExist,the values array must be empty. This array is replaced during a strategicmerge patch.+optional+listType=atomic[]stringfalse

LabelSelectorOperator

A label selector operator is the set of operators that can be used in a selector requirement.

FieldDescriptionTypeRequired

NamespaceSelector

FieldDescriptionTypeRequired
nameSelectorFilter Namespaces by their nameNameSelectorfalse
labelSelectorFilter Namespaces by their labelsLabelSelectorfalse

ObservabilityConfig

Configure the synchronization process observability using Prometheus ServiceMonitor

FieldDescriptionTypeRequired
enabledEnable the Observability with a Prometheus ServiceMonitorboolfalse
intervalConfigure the interval in the ServiceMonitor that Prometheus will use to scrape metricsDurationfalse

Duration

Duration is a wrapper around time.Duration which supports correct marshaling to YAML and JSON. In particular, it marshals into strings, which can be used as map keys in json.

FieldDescriptionTypeRequired

SynchronizationOptions

Customize the synchronization process with special options

FieldDescriptionTypeRequired
dryRunSimulate the synchronization process but don’t execute the write operationsboolfalse
forceSyncSynchronize object in the destination even if the object exists in the destination and it doesn’t match the configured selectorsboolfalse
forcePrunePrune object in the destination even if it doesn’t match the configured selectorsboolfalse
showLogIfObjectIsAlreadyInSyncShow a log message if object is already in syncboolfalse
showLogIfObjectHaveBeenAdaptedShow a log message if object have been adapted for the destinationboolfalse

KubernetesObjectSelector

FieldDescriptionTypeRequired
objectSelectorFilter objects by ObjectSelectorObjectSelectorfalse
namespaceSelectorFilter objects by NamespaceSelectorNamespaceSelectorfalse
targetKubernetes resource to be usedGroupVersionResourcesfalse

GroupVersionResources

Select a set of GroupVersionResource

FieldDescriptionTypeRequired
groupKubernetes resource group. Example: appsstringfalse
versionKubernetes resource version. Example: v1stringfalse
resourcesKubernetes resource names. Example: deployments[]stringfalse

Transformations

Transformations is a list of operations to modifiy the Kubernetes objects matching the given selectors

FieldDescriptionTypeRequired
resourcesSelect the objects to be transfomred by their resource type[]GroupVersionResourcesfalse
namespaceSelectorFilter the objects to be transformed by NamespaceSelectorNamespaceSelectorfalse
objectSelectorFilter the objects to be transformed by ObjectSelectorObjectSelectorfalse
operationsOperations to be executed to transform the objects[]TransformationOperationfalse

TransformationOperation

The operation to execute to transform the objects

FieldDescriptionTypeRequired
jsonpatchJSONPatch operationOperationJSONPatchfalse

OperationJSONPatch

The JSONPatch operation

FieldDescriptionTypeRequired
skipIfNotFoundOnDeleteSkip if not found on deleteboolfalse
operationsList of operations to be executed[]JSONPatchOperationfalse

JSONPatchOperation

JSONPAtch operation

FieldDescriptionTypeRequired
opJSONPatch operation: add, copy, move, remove, replace, teststringfalse
pathExecute the operation to the given pathstringfalse
valueOptional value to be used in the operationinterface{}false

5 - Zookeeper to Zookeeper

Synchronize Zookeeper data between two clusters

5.1 - Introduction

Synchronize Zookeeper data between clusters

You can synchronize Zookeeper data between two clusters using the Zookeeper protocol.

Supported models

One time synchronization

You can synchronize the data just once with the Synchronization Kubernetes object.

Periodic synchronization

You can synchronize periodically the SynchronizationPlan Kubernetes object.

Samples

Synchronize once

Synchronize the data once only in the /test path:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  generateName: synchronize-zookeeper-
spec:
  plugin: zookeeper-to-zookeeper-nodes
  config:
    sourceName: zookeeper-source
    destinationName: zookeeper-destination
    rootPath: /test
    createRoutePath: true

Scheduled synchronization

Synchronize data every hour in the /test path:

apiVersion: automation.astronetes.io/v1alpha1
kind: SynchronizationPlan
metadata:
  name: synchronize-zookeeper
spec:
  schedule: "0 * * * *"
  template:
    spec:
      plugin: zookeeper-to-zookeeper-nodes
      config: 
        sourceName: zookeeper-source
        destinationName: zookeeper-destination
        rootPath: /test

5.2 - Configuration

Plugin parameters and accepted values

Required configuration

Source and destination

The source and the destination clusters can be specified using the .spec.config.sourceName and .spec.config.destinationName properties. Both Database objects should exists in the same Namespace where the synchronization is being created.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: zookeeper-nodes-to-zookeeper
  config:
    sourceName: cluster-1
    destinationName: cluster-2
    ...

Root path

The root path to be used to only synchronize a specific part of the Zookeeper database.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: zookeeper-nodes-to-zookeeper
  config:
    ...
    rootPath: /test

Optional configuration

Create root path

Create the Root Path in the destination database if it doesn’t exist.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: zookeeper-nodes-to-zookeeper
  config:
    ...
    createRootPath: true

Ignore ephemeral

Don’t synchronize ephemeral data in the destination cluster..

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: zookeeper-nodes-to-zookeeper
  config:
    ...
    ignoreEphemeral: true

Exclude paths

Exclude data from being synchronized to the destination cluster filtering on path using regex.

Example:

apiVersion: automation.astronetes.io/v1alpha1
kind: Synchronization
metadata:
  name: example
spec:
  plugin: zookeeper-nodes-to-zookeeper
  config:
    ...
    excludeRegex: ..

5.3 - API Reference

Configuration details

Config

Configuration for Zookeeper to Zookeeper synchronization

NameDescriptionTypeRequired
sourceNameZookeeper instance acting as sourcestringyes
destinationNameZookeeper instance acting as destinationstringyes
rootPathRoot Path of the contents to synchronizestringyes
createRootPathWhether to create the Root Path in the destination databasebooleanno
ignoreEphemeralWhether to ignore ephemeralbooleanno
excludePathRegexpRegular expression for keys to exclude while synchronizingstringno