Backup strategies for volumes

To have consistent backup it have to be known if there is need to provide extra step(s) before backup happen:

• data don't need extra step(s) to provide consistent backup
• data need extra command(s) executed before backup like dump data from database to separate file - implemented via backup hooks (Velero Docs - Backup Hooks)
• stop pod with data before backup - require external coordination to stop pod and executed backup process and start pod

By default volume data is not backed up even if PersistentVolume object is selected for backup.

To backup data with file system backup there is opt-in and opt-out approach (Velero Docs - File System Backup). Velero supports the use of a global flag to backup data automatically for all volumes. There is option for selective approach by using specific label(s) with value of volume that should be backed up. There is also reverse logic that excludes specific volumes from backup.

Backup

To create backup with Velero is needed to create Backup K8s custom resource (Velero Docs - Backup API Type).

Backup custom resources could be created via Velero CLI command or with K8s custom resource manifest.

Backups created via Schedule K8s custom resource (Velero Docs - Schedule API Type) have advantages over directly created via Backup object.

Most important feature of Schedule object is that could create backups at defined time interval with cron job time syntax:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: cluster-1-obs
  namespace: velero
spec:
  schedule: 0 4 * * *
  template:
    defaultVolumesToFsBackup: true
    includedNamespaces:
      - obs
    storageLocation: cluster-1
    ttl: 720h

Schedule objects allow setting all parameters that are also available in the Backup object.

Velero controller at defined time interval will execute Schedule object and use it as the template to create Backup object.

Names of Backup objects created via Schedule object have format scheduleName-unixTimestamp.

Backup basic commands

Basic backup operation with Velero

• List all backups that have done by Velero

# get all backups via Velero CLI command
velero get backup
# get all backups via kubectl (Velero has been installed in velero namespace)
kubect -n velero get backups

• Describe specific backup to get information about backup

# get basic information about backup
velero describe backup nameOfBackup
# get detailed information about backup
velero describe backup nameOfBackup --details

• Create backup with Velero CLI command

# create ad-hoc backup with given namespace
velero backup create --include-namespaces=nameOfNamspace nameOfBackup
# create ad-hoc backup from defined schedule
velero backup create --from-schedule nameOfSchedule

• Delete backup

Velero backups deleted with Velero CLI command delete K8s Backup objects and all related object storage and volumes backup data.

Velero backups deleted with kubectl command delete only K8s Backup object.

# delete velero backup via Velero CLI
velero delete backup nameOfBackup
# delete velero backup via kubectl
kubectl -n velero delete nameOfBackup

Backup selectors

Backup selectors (Velero Docs - Resource filtering) are useful to limit scope of the backup itself. By default backup operation will backup all objects in all names.

Velero supports a limited scope of backup to include specific types of objects.

The most common way to limit backup scope is to use a namespace filter to limit backup only to specific namespace or namespaces.

Velero also supports K8s labels to limit objects only to specific label or set of or labels.

Backup hooks

Backup hooks (Velero Docs - Backup Hooks) are used when backup process require extra commands to provide consistent backup like freeze command or dump data to separate file.

Backup hooks are executed with K8s exec api with pre and post option.

When a command requires a shell to work properly it needs to be added explicitly to the executed command.

...
hooks:
  resources:
  - name: mysqldump
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: mariadb-backup-helper
    pre:
    - exec:
        container: velero-backup
        command:
          - /bin/bash
          - -c
          - rm -f /bitnami/velero-backup/mysql-backup-all.sql  && mysqldump -h mariadb.mariadb.svc.cluster.local -u root -p$MARIADB_ROOT_PASSWORD -A > /bitnami/velero-backup/mysql-backup-all.sql
        onError: Fail
        timeout: 10m

Preferable way to provide needed command for dump data and extra volume with separate PersistenceVolume is to create separate Deployment.

Separate volume via PersistentVolume to dump data allows to have enough space and avoid sharing space with node storage. Usage of EmptyDir is not recommended due to sharing space with node that could lead to exhaust space of node and lead to space pressure and forcefully moving all node workloads to the other K8s nodes.