Backup of Kubernetes Cluster using Velero[๐Ÿ”—](#backup-of-kubernetes-cluster-using-velero "Permalink to this headline") ===================================================================================================================== What is Velero[๐Ÿ”—](#what-is-velero "Permalink to this headline") --------------------------------------------------------------- [Velero](https://velero.io) is the official open source project from VMware. It can back up all Kubernetes API objects and persistent volumes from the cluster on which it is installed. Backed up objects can be restored on the same cluster, or on a new one. Using a package like Velero is essential for any serious development in the Kubernetes cluster. In essence, you create object store under 3Engines, either using Horizon or Swift module of **3Engines** command and then save cluster state into it. Restoring is the same in reverse โ€“ read from that object store and save it to a Kubernetes cluster. Velero has its own CLI command system so it is possible to automate creation of backups using cron jobs. What We Are Going To Cover[๐Ÿ”—](#what-we-are-going-to-cover "Permalink to this headline") --------------------------------------------------------------------------------------- > * Getting EC2 Client Credentials > * Adjusting โ€œvalues.yamlโ€, the configuration file > * Creating namespace called *velero* for precise access to the Kubernetes cluster > * Installing Velero with a Helm chart > * Installing and deleting backups using Velero > * Example 1 Basics of Restoring an Application > * Example 2 Snapshot of Restoring an Application Prerequisites[๐Ÿ”—](#prerequisites "Permalink to this headline") ------------------------------------------------------------- No. 1 **Hosting** You need a 3Engines Cloud hosting account with Horizon interface . The resources that you require and use will reflect on the state of your account wallet. Check your account statistics at . No. 2 **How to Access Kubernetes cluster post-deployment** We shall also assume that you have one or more Kubernetes clusters ready and accessible via a **kubectl** command: [How To Access Kubernetes Cluster Post Deployment Using Kubectl On 3Engines Cloud 3Engines Magnum](How-To-Access-Kubernetes-Cluster-Post-Deployment-Using-Kubectl-On-3Engines-Cloud-3Engines-Magnum.html.md) The result of that article will be setting up of system variable **KUBECONFIG**, which points to the configuration file for access to the Kubernetes cloud. A typical command will be: ``` export KUBECONFIG=/home/username/Desktop/kubernetes/k8sdir/config ``` In case this is the first time you are using that particular config file, make it more secure by executing the following command as well: ``` chmod 600 /home/username/Desktop/kubernetes/k8sdir/config ``` No. 3 **Handling Helm** To install Velero, we shall use Helm: [Deploying Helm Charts on Magnum Kubernetes Clusters on 3Engines Cloud Cloud](Deploying-Helm-Charts-on-Magnum-Kubernetes-Clusters-on-3Engines-Cloud-Cloud.html.md). No. 4 **An object storage S3 bucket available** To create one, you can access object storage with Horizon interface or CLI. Horizon commands : [How to use Object Storage on 3Engines Cloud](../s3/How-to-use-Object-Storage-on-3Engines-Cloud.html.md). CLI : You can also use command such as ``` 3Engines container ``` to work with object storage. For more information see [How to access object storage using 3Engines CLI on 3Engines Cloud](../3Enginescli/How-to-access-object-storage-using-3Engines-CLI-on-3Engines-Cloud.html.md) Either way, we shall assume that there is a container called โ€œbucketnewโ€: ![bucketnew_created.png](../_images/bucketnew_created.png) Supply your own unique name while working through this article. Before Installing Velero[๐Ÿ”—](#before-installing-velero "Permalink to this headline") ----------------------------------------------------------------------------------- We shall install Velero on Ubuntu 22.04; using other Linux distributions would be similar. Update and upgrade your Ubuntu environment: ``` sudo apt update && sudo apt upgrade ``` It will be necessary to have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled. For more information on supported Kubernetes versions, see Velero [compatibility matrix](https://github.com/vmware-tanzu/velero#velero-compatabilty-matrix). ### Installation step 1 Getting EC2 client credentials[๐Ÿ”—](#installation-step-1-getting-ec2-client-credentials "Permalink to this headline") First fetch EC2 credentials from 3Engines. They are necessary to access private bucket (container). Generate them on your own by executing the following commands: ``` 3Engines ec2 credentials create 3Engines ec2 credentials list ``` Save somewhere the *Access Key* and the *Secret Key*. They will be needed in the next step, in which you set up a Velero configuration file. ### Installation step 2 Adjust the configuration file - โ€œvalues.yamlโ€[๐Ÿ”—](#installation-step-2-adjust-the-configuration-file-values-yaml "Permalink to this headline") Now create or adjust a configuration file for Velero. Use text editor of your choice to create that file. On MacOS or Linux, for example, you can use **nano**, like this: ``` sudo nano values.yaml ``` Use configuration file provided below. Fill in the required fields, which are marked with **##**: **values.yaml** WAW4-1WAW3-1WAW3-2FRA1-2 > ``` > initContainers: > - name: velero-plugin-for-aws > image: velero/velero-plugin-for-aws:v1.4.0 > imagePullPolicy: IfNotPresent > volumeMounts: > - mountPath: /target > name: plugins > > configuration: > provider: aws > backupStorageLocation: > provider: aws > name: ## enter name of backup storage location (could be anything) > bucket: ## enter name of bucket created in 3Engines > default: true > config: > region: default > s3ForcePathStyle: true > s3Url: ## enter URL of object storage (for example "https://s3.waw4-1.3Engines.com") > credentials: > secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. > cloud: | > [default] > aws_access_key_id= > aws_secret_access_key= > ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. > snapshotsEnabled: false > deployRestic: true > restic: > podVolumePath: /var/lib/kubelet/pods > privileged: true > schedules: > mybackup: > disabled: false > schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make. > template: > ttl: "240h" ## choose ttl, after which the backups will be removed. > snapshotVolumes: false > > ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: ## enter name of backup storage location (could be anything) bucket: ## enter name of bucket created in 3Engines default: true config: region: waw3-1 s3ForcePathStyle: true s3Url: ## enter URL of object storage (for example "https://s3.waw3-1.3Engines.com") credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= aws_secret_access_key= ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make. template: ttl: "240h" ## choose ttl, after which the backups will be removed. snapshotVolumes: false ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: ## enter name of backup storage location (could be anything) bucket: ## enter name of bucket created in 3Engines default: true config: region: default s3ForcePathStyle: true s3Url: ## enter URL of object storage (for example "https://s3.waw3-2.3Engines.com") credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= aws_secret_access_key= ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make. template: ttl: "240h" ## choose ttl, after which the backups will be removed. snapshotVolumes: false ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: ## enter name of backup storage location (could be anything) bucket: ## enter name of bucket created in 3Engines default: true config: region: default s3ForcePathStyle: true s3Url: ## enter URL of object storage (for example "https://s3.fra1-2.3Engines.com") credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= aws_secret_access_key= ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 6,18 * * *" ## choose time, when scheduled backups will be make. template: ttl: "240h" ## choose ttl, after which the backups will be removed. snapshotVolumes: false ``` Paste the content to the configuration file **values.yaml** and save. Example of an already configured file: WAW4-1WAW3-1WAW3-2FRA1-2 ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: velerobackupnew bucket: bucketnew default: true config: region: default s3ForcePathStyle: true s3Url: https://s3.waw4-1.3Engines.com credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x aws_secret_access_key= dee1581dac214d3dsa34037e826f9148 ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 * * *" template: ttl: "168h" snapshotVolumes: false ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: velerobackupnew bucket: bucketnew default: true config: region: waw3-1 s3ForcePathStyle: true s3Url: https://s3.waw3-1.3Engines.com credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x aws_secret_access_key= dee1581dac214d3dsa34037e826f9148 ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 * * *" template: ttl: "168h" snapshotVolumes: false ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: velerobackupnew bucket: bucketnew default: true config: region: default s3ForcePathStyle: true s3Url: https://s3.waw3-2.3Engines.com credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x aws_secret_access_key= dee1581dac214d3dsa34037e826f9148 ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 * * *" template: ttl: "168h" snapshotVolumes: false ``` ``` initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:v1.4.0 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: provider: aws backupStorageLocation: provider: aws name: velerobackupnew bucket: bucketnew default: true config: region: default s3ForcePathStyle: true s3Url: https://s3.fra1-2.3Engines.com credentials: secretContents: ## enter access and secret key to ec2 bucket. This configuration will create kubernetes secret. cloud: | [default] aws_access_key_id= c4b4ee62a18f4e0ba23f71629d2038e1x aws_secret_access_key= dee1581dac214d3dsa34037e826f9148 ##existingSecret: ## If you want to use existing secret, created from sealed secret, then use this variable and omit credentials.secretContents. snapshotsEnabled: false deployRestic: true restic: podVolumePath: /var/lib/kubelet/pods privileged: true schedules: mybackup: disabled: false schedule: "0 * * *" template: ttl: "168h" snapshotVolumes: false ``` ### Installation step 3 Creating namespace[๐Ÿ”—](#installation-step-3-creating-namespace "Permalink to this headline") Velero must be installed in an eponymous namespace, *velero*. This is the command to create it: ``` kubectl create namespace velero namespace/velero created ``` ### Installation step 4 Installing Velero with a Helm chart[๐Ÿ”—](#installation-step-4-installing-velero-with-a-helm-chart "Permalink to this headline") Here are the commands to install Velero by means of a Helm chart: ``` helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts ``` The output is: ``` "vmware-tanzu" has been added to your repositories ``` The following command will install velero onto the cluster: ``` helm install vmware-tanzu/velero --namespace velero --version 2.28 -f values.yaml --generate-name ``` The output will look like this: ![installation_of_velero.png](../_images/installation_of_velero.png) To see the version of Velero that is actually installed, use: ``` helm list --namespace velero ``` Note the name used, **velero-1721031498**, and we are going to use it in the rest of the article. In your case, note the correct velero name and swap value of **1721031498** with it. Here is how to check that Velero is up and running: ``` kubectl get deployment/velero-1721031498 -n velero ``` The output will be similar to this: ``` NAME READY UP-TO-DATE AVAILABLE AGE velero-1721031498 1/1 1 1 5m30s ``` Check that the secret has been created: ``` kubectl get secret/velero-1721031498 -n velero ``` The result is: ``` NAME TYPE DATA AGE velero-1721031498 Opaque 1 3d1h ``` ### Installation step 5 Installing Velero CLI[๐Ÿ”—](#installation-step-5-installing-velero-cli "Permalink to this headline") The final step is to install Velero CLI โ€“ Command Line Interface suitable for working from the terminal window on your operating system. Download the client specified for your operating system from: , using **wget**. Here we are downloading version > **velero-v1.9.1-linux-amd64.tar.gz** but it is recommended to download the latest version. In that case, change the name of the **tar.gz** file accordingly. ``` wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.1/velero-v1.9.1-linux-amd64.tar.gz ``` Extract the tarball: ``` tar -xvf velero-v1.9.1-linux-amd64.tar.gz ``` This is the expected result: ``` velero-v1.9.1-linux-amd64/LICENSE velero-v1.9.1-linux-amd64/examples/README.md velero-v1.9.1-linux-amd64/examples/minio velero-v1.9.1-linux-amd64/examples/minio/00-minio-deployment.yaml velero-v1.9.1-linux-amd64/examples/nginx-app velero-v1.9.1-linux-amd64/examples/nginx-app/README.md velero-v1.9.1-linux-amd64/examples/nginx-app/base.yaml velero-v1.9.1-linux-amd64/examples/nginx-app/with-pv.yaml velero-v1.9.1-linux-amd64/velero ``` Move the extracted **velero** binary to somewhere in your $PATH (/usr/local/bin for most users): ``` cd velero-v1.9.1-linux-amd64 # System might force using sudo sudo mv velero /usr/local/bin # check if velero is working velero version ``` ![velero_version_working.png](../_images/velero_version_working.png) After these operations, you should be allowed to use **velero** commands. For help how to use them, execute: ``` velero help ``` Working with Velero[๐Ÿ”—](#working-with-velero "Permalink to this headline") ------------------------------------------------------------------------- So far, we have > * created an object store named โ€œbucketnewโ€ and > * told velero to use it through the **bucket:** parameter in values.yaml file. Velero will create another object store called *backups* under โ€œbucketnewโ€ and then continue creating object stores for particular backups. For example, the following command will add object store called **mybackup2**: > ``` > velero backup create mybackup2 > Backup request "mybackup2" submitted successfully. > > ``` Here is what it will look like in Horizon: ![installed_mybackup2.png](../_images/installed_mybackup2.png) Let us add two other backups. The first should backup all api objects in namespace *velero*: ``` velero backup create mybackup3 --include-namespaces velero ``` The second will backup all api objects in default namespace ``` velero backup create mybackup5 --include-namespaces default Backup request "mybackup4" submitted successfully. ``` This the object store structure after these three backups: ![three_backups_mybackup.png](../_images/three_backups_mybackup.png) You can also use velero CLI command to list the existing backups: ``` velero backup get ``` This is the result in terminal window: ![three_backups_mybackup.png](../_images/three_backups_mybackup.png) Example 1 Basics of Restoring an Application[๐Ÿ”—](#example-1-basics-of-restoring-an-application "Permalink to this headline") --------------------------------------------------------------------------------------------------------------------------- Let us now demonstrate how to restore a Kubernetes application. Let us first clone one example app from GitHub. Execute this: ``` git clone https://github.com/vmware-tanzu/velero.git Cloning into 'velero'... Resolving deltas: 100% (27049/27049), done. cd velero ``` Start the sample nginx app: ``` kubectl apply -f examples/nginx-app/base.yaml kubectl apply -f base.yaml namespace/nginx-example unchanged deployment.apps/nginx-deployment unchanged service/my-nginx unchanged ``` Create a backup: ``` velero backup create nginx-backup --include-namespaces nginx-example Backup request "nginx-backup" submitted successfully. ``` This is what the backup of **nginx-backup** looks like in Horizon: ![nginx-backup.png](../_images/nginx-backup.png) Simulate a disaster: ``` kubectl delete namespaces nginx-example # Wait for the namespace to be deleted namespace "nginx-example" deleted ``` Restore your lost resources: ``` velero restore create --from-backup nginx-backup Restore request "nginx-backup-20220728013338" submitted successfully. Run `velero restore describe nginx-backup-20220728013338` or `velero restore logs nginx-backup-20220728013338` for more details. velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR backup New 0 0 n/a nginx-backup New 0 0 n/a ``` Example 2 Snapshot of restoring an application[๐Ÿ”—](#example-2-snapshot-of-restoring-an-application "Permalink to this headline") ------------------------------------------------------------------------------------------------------------------------------- Start the sample nginx app: ``` kubectl apply -f examples/nginx-app/with-pv.yaml namespace/nginx-example created persistentvolumeclaim/nginx-logs created deployment.apps/nginx-deployment created service/my-nginx created ``` Create a backup with PV snapshotting: ``` velero backup create nginx-backup-vp --include-namespaces nginx-example Backup request "nginx-backup" submitted successfully. Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details. ``` Simulate a disaster: ``` kubectl delete namespaces nginx-example namespace "nginx-example" deleted ``` Important Because the default [reclaim policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming) for dynamically-provisioned PVs is โ€œDeleteโ€, these commands should trigger your cloud provider to delete the disk that backs up the PV. Deletion is asynchronous, so this may take some time. Restore your lost resources: ``` velero restore create --from-backup nginx-backup-vp Restore request "nginx-backup-20220728015234" submitted successfully. Run `velero restore describe nginx-backup-20220728015234` or `velero restore logs nginx-backup-20220728015234` for more details. ``` Delete a Velero backup[๐Ÿ”—](#delete-a-velero-backup "Permalink to this headline") ------------------------------------------------------------------------------- There are two ways to delete a backup made by Velero. Delete backup custom resource only : ``` kubectl delete backup -n ``` will delete the backup custom resource only and will not delete any associated data from object/block storage Delete all data in object/block storage : ``` velero backup delete ``` will delete the backup resource including all data in object/block storage Removing Velero from the cluster[๐Ÿ”—](#removing-velero-from-the-cluster "Permalink to this headline") --------------------------------------------------------------------------------------------------- ### Uninstall Velero[๐Ÿ”—](#uninstall-velero "Permalink to this headline") To uninstall Velero release: ``` helm uninstall velero-1721031498 --namespace velero ``` ### To delete Velero namespace[๐Ÿ”—](#to-delete-velero-namespace "Permalink to this headline") ``` kubectl delete namespace velero ``` What To Do Next[๐Ÿ”—](#what-to-do-next "Permalink to this headline") ----------------------------------------------------------------- Now that Velero is up and running, you can integrate it into your routine. It will be useful in all classical backups scenarios โ€“ for disaster recovery, cluster and namespace migration, testing and development, application rollbacks, compliance and auditing and so on. Apart from these broad use cases, Velero will help with specific Kubernetes cluster tasks for backing up, such as: > * backing up and restoring deployments, service, config maps and secrets, > * selective backups, say, only for specific namespaces or label selectors, > * volume shapshots using cloud provider APIs (AWS, Azure, GCP etc.) > * snapshots of persistent volumes for point-in-time recovery > * saving backup data to AWS S3, Google Cloud Storage, Azure Blob Storage etc. > * integration with **kubectl** command so that Custom Resource Definitions (CRDs) are used to define backup and restore configuration.