Recover FalkorDB deployments in another region

​

How the plan works

​

Requirements

• A Kubernetes cluster in the recovery region, with FalkorDB Enterprise installed and its own license. A license is bound to one Kubernetes cluster, so register the recovery cluster in the Enterprise Portal. See Disaster recovery into a replacement cluster.
• Object storage reachable from both regions.
• In both installations, a backup repository on object storage: s3, s3-compatible, gcs-s3comp, minio, azureblob, oss, obs or cos. Either access method works. RDB exports and imports read and write the bucket directly, so they don’t need the csi-s3 driver.
• A recovery installation running the same FalkorDB version as the primary, or a newer one. An export written by a newer FalkorDB can’t be imported into an older one.
• The repository’s credentials in a Secret the Admin Server can read. The chart grants it read access to Secrets in kb-system and default, which covers credentials created by KubeBlocks and by the Admin UI. To keep them elsewhere, add the namespace to adminServer.rdbTransfer.credentialNamespaces. Credentials limited to the repository’s path prefix are enough; the export list then leaves out exports taken by earlier releases, which are stored outside the prefix. Repositories that rely on cloud IAM roles instead of keys aren’t supported for RDB exports.

​

Choose where exports are stored

​

One bucket replicated by your cloud provider

• Turn on delete marker replication, or add a lifecycle rule that expires current objects older than the exports you keep.
• Add a NoncurrentVersionExpiration rule to the replica bucket. Keep noncurrent versions long enough to cover the recovery history you need, so that an export deleted by mistake can still be restored.

​

A repository in each region

• Every export is written twice, which doubles the storage and the traffic between regions.
• Each schedule reads its own snapshot from the deployment. Set the schedules at different times so they don’t add load at the same time.

​

Set up the primary region

1

Create the repository

In the Admin UI, open Backup Repositories and select New Backup Repository. Choose the provider, then enter the bucket, region and credentials. To use a repository in each region, create one repository per region. Note each repository’s path prefix, if you set one.

2

Schedule exports for each deployment

In each deployment, open Data transfer and click Add schedule. Choose the repository, and set how often exports run and how many to keep.

How often exports run is only the lower bound of your RPO. Each export also takes time to run and, with a replicated bucket, to reach the recovery region, and a failed or skipped run leaves the previous export as the newest. Pick an interval with room for these, and monitor the age of the newest complete export in the recovery region.

falkordb-admin rdb schedules create my-graph dr-hourly --namespace prod \
  --schedule "0 * * * *" --retention 24 --repository exports-us-east

With a repository in each region, create one schedule per repository, each with its own name.

3

Keep each deployment's settings outside the region

Recovery creates new deployments, so keep a record of each deployment’s topology, shard count, CPU, memory, storage and FalkorDB version. One way is to store each deployment’s YAML manifest in version control.

<path prefix>/falkordb-rdb/<namespace>/<deployment>/<schedule>/<export name>/

​

Prepare the recovery region

# values.yaml of the recovery installation
adminServer:
  env:
    clusterExternalDomain: dr.db.internal.example.com

aws s3 ls s3://<bucket>/ --recursive | grep 'falkordb-rdb/prod/my-graph/dr-hourly/.*/manifest.json$'

​

Recover after a regional outage

1

Confirm the license

falkordb-admin license status

The license must be active. You can’t create deployments on an invalid license.

2

Make the storage available

With S3 replication, a Google Cloud dual-region or multi-region bucket, or a repository in each region, the exports are already in storage the recovery region can reach. Check that the recovery installation’s repository shows Ready.

3

Create the deployments

Create each deployment from the settings you recorded, with the same name. It needs at least as much memory as the original, and persistent storage even if the original had none, so that a pod restart doesn’t lose the imported data. The topology and shard count can differ, because an import routes each key to the shard that owns it.It gets the recovery domain by default. Don’t give it the primary’s domain: its names would be published before its data is imported.

4

Import the newest complete export

Open the new deployment, select Data transfer, and click Import. Choose Repository file, then the repository. Export lists every export in the bucket, newest first, including the ones the primary installation wrote. Choose the newest one of the deployment’s schedule.

An export marked incomplete is missing a file its manifest lists, for example because replication hadn’t finished. Choose an older one.

Choosing an export fills in its Path. The new deployment is empty, so clear Snapshot before importing, then click Start Import.

With the Admin CLI, list the exports and import one by its path:

falkordb-admin rdb repository-exports my-graph --namespace prod --repository exports-eu-west

falkordb-admin rdb import my-graph --namespace prod \
  --from-repository exports-eu-west \
  --path <path prefix>/falkordb-rdb/prod/my-graph/dr-hourly/<export name> \
  --rollback none

Before writing anything, the import checks every file with the new deployment’s own server. It stops if a file is incomplete or was written by a newer FalkorDB.

5

Check the data

When the import succeeds, compare its key count with the keys listed in manifest.json. Then run a query whose answer you know, for example counting the nodes of a graph.

6

Move clients to the new deployment

The new deployment has its own service names, its own generated password and, with external DNS, names under the recovery domain. Copy them from Connection Details on the deployment’s General tab, and point clients at them.With TLS, the new deployment also has its own certificate, issued for its own names. With the FalkorDB Enterprise issuer it gets a new CA too, so clients that trust the original deployment’s CA can’t connect to it. Give clients the new CA, shown under Connection Details. Issuers you control, such as cert-manager or your own certificates, can sign the new deployment with the CA clients already trust. A certificate you provide yourself must cover the recovery names. Before you send production traffic, connect with hostname verification through a recovery name.The records the lost region published under the primary’s domain stay until ExternalDNS there can clean them up. Remove them by hand if clients that weren’t moved should fail fast instead of timing out.

7

Protect the recovered deployment

Set up backups and an export schedule for the recovered deployment. For the export schedule, use a schedule name that is different from the primary’s (see the warning in Prepare the recovery region).

​

Test the plan

1. In the recovery installation, create a deployment in a scratch namespace. Give it a name that no production deployment uses, and keep the recovery domain, or use a domain set aside for drills.
   External DNS names are built from the deployment name and its external domain, not its namespace. Never give a drill deployment the primary’s domain: with the same name as a production deployment, it publishes records for production’s names, and clients can be sent to the drill copy.
2. Import the newest export into it, as in Recover after a regional outage.
3. Check the data, and note how long the whole run took. That is the database part of your RTO. Your full RTO also includes deciding to fail over, making the storage available and moving clients over, so add those, or extend the drill until a client request succeeds against the recovered deployment.
4. Delete the scratch deployment.

​

Limitations

• Writes made after the newest export that reached the recovery region are lost.
• Exports of a sharded deployment are taken shard by shard. They aren’t a consistent copy across shards at one point in time.
• The recovery region has no standby deployments, so recovery takes as long as creating the deployments plus importing the data. There is no automatic failover.
• Backups and point-in-time restore can’t be used across Kubernetes clusters. See the warning at the top of this page.