Getting Started with BIG-IP Next: Backing Up and Restoring Central Manager

Backing up BIG-IP Next Instances is possible in the Central Manager GUI. Backing up Central Manager, however, requires you to break out those made CLI skilz of yours. And take a backup you shall! You can snapshot your Central Manager virtual machine and restore that as well, but if you want system level backup instead of device level in the event things go south, you need an option currently as high availability, though coming soon to a release near you, is not yet an option for Central Manager.

As there will be no screenshots required, most of this is already covered on Clouddocs how to on this topic, but in this article, I'll walk through the process by executing the steps and sharing the output.

Creating the Central Manager Backup

Login to the Central Manager CLI by SSHing to your fqdn or IP address. If you configured the external storage when you set up Central Manager you can do a full backup, which includes all the analytics from Central Manager and your instances. If you only have local storage, you'll need to do a partial. The command to perform the backup and the restore is /opt/cm-bundle/cm. You use the backup subcommand for a backup operation, and as you can probably guess, the restore subcommand for a restore operation. I don't have the external storage in my lab, so I ran a partial backup.

admin@cm1:~$ /opt/cm-bundle/cm backup 2024-03-09T00:04:15+00:00 Executing /opt/cm-bundle/cm backup Encryption password: Reenter encryption password: 2024-03-09T00:04:21+00:00 info: Backing up Vault... Created vault backup: /tmp/vault-backup.tgz tar: removing leading '/' from member names var/run/vault-init/ var/run/vault-init/linkerd.csr var/run/vault-init/linkerd.crt var/run/vault-init/vault-client-intermediate-ca.csr var/run/vault-init/vault-client-intermediate-ca.crt var/run/vault-init/unsealkeys var/run/vault-init/ca.crt var/run/vault-init/ingress-intermediate-ca.crt var/run/vault-init/unsealkeys.sha256 var/run/vault-init/linkerd-ca.crt var/run/vault-init/ingress-intermediate-ca.csr var/run/vault-init/linkerd-webhook.csr var/run/vault-init/linkerd-webhook.crt 2024-03-09T00:04:22+00:00 info: Vault backup successful! 2024-03-09T00:04:22+00:00 info: Backing up PostgreSQL... 2024-03-09T00:04:23+00:00 info: PostgreSQL backup successful! 2024-03-09T00:04:23+00:00 info: Performing Prometheus backup... 2024-03-09T00:04:55+00:00 info: Creating Prometheus database snapshot... 2024-03-09T00:05:09+00:00 info: Verifying the Prometheus database snapshot... 2024-03-09T00:05:09+00:00 info: Successfully created Prometheus database snapshot 20240309T000505Z-4c5c8cab103961be 2024-03-09T00:05:09+00:00 info: Copying Prometheus snapshot locally... 2024-03-09T00:05:22+00:00 info: Cleanup the Prometheus snapshot in the pod 2024-03-09T00:05:28+00:00 info: Prometheus backup succeeded! 2024-03-09T00:05:28+00:00 info: Performing Elasticsearch backup... 2024-03-09T00:05:28+00:00 info: Creating Elasticsearch snapshot [elasticsearch-snapshot]... 2024-03-09T00:05:28+00:00 info: Elasticsearch backup succeeded! 2024-03-09T00:05:28+00:00 info: Backing up SQLite... 2024-03-09T00:05:29+00:00 info: SQLite backup successful! 2024-03-09T00:05:29+00:00 info: Creating backup bundle backup.20240309-000421.tgz... 2024-03-09T00:08:26+00:00 info: Encrypting backup bundle... 2024-03-09T00:08:40+00:00 info: Backup bundle created at /opt/cm-backup/backup.20240309-000421.tgz.enc

Restoring the Central Manager Backup

Sometime after my backup, suppose AubreyKingF5 logged into to my Central Manager and deleted user jrahm and my backup-test certificate (BAD Aubrey!) Maybe he deleted all the resources. Here's the backup script execution on my Central Manager instance. Note the immediate ask for that backup password. Seriously, vault those passwords, don't lose them!

admin@cm1:~$ /opt/cm-bundle/cm restore /opt/cm-backup/backup.20240309-000421.tgz.enc 2024-03-09T00:12:40+00:00 Executing /opt/cm-bundle/cm restore /opt/cm-backup/backup.20240309-000421.tgz.enc 2024-03-09T00:12:40+00:00 info: Restoring from backup file /opt/cm-backup/backup.20240309-000421.tgz.enc... Enter decryption password: 2024-03-09T00:12:43+00:00 info: Decrypting backup file... 2024-03-09T00:12:46+00:00 info: Checking available disk space... 2024-03-09T00:13:55+00:00 info: Extracting backup to /opt/cm-backup... 2024-03-09T00:14:35+00:00 info: Validating backup contains all required components 2024-03-09T00:14:35+00:00 info: Restoring Vault... var/run/vault-init/ var/run/vault-init/linkerd.csr var/run/vault-init/linkerd.crt var/run/vault-init/vault-client-intermediate-ca.csr var/run/vault-init/vault-client-intermediate-ca.crt var/run/vault-init/unsealkeys var/run/vault-init/ca.crt var/run/vault-init/ingress-intermediate-ca.crt var/run/vault-init/unsealkeys.sha256 var/run/vault-init/linkerd-ca.crt var/run/vault-init/ingress-intermediate-ca.csr var/run/vault-init/linkerd-webhook.csr var/run/vault-init/linkerd-webhook.crt Vault restored using /tmp/vault-backup.tgz 2024-03-09T00:14:47+00:00 info: Vault data has been successfully restored. 2024-03-09T00:14:47+00:00 info: Renewing all certificates. Manually triggered issuance of Certificate default/mbiq-ingress-nginx-root-cert Manually triggered issuance of Certificate default/mbiq-ado-vault-server-cert Manually triggered issuance of Certificate default/mbiq-ado-vault-client-cert Manually triggered issuance of Certificate default/gateway-feature-ingress-cert Manually triggered issuance of Certificate default/central-manager-ui-ingress-cert Manually triggered issuance of Certificate default/mbiq-apm-vault-client-cert Manually triggered issuance of Certificate default/mbiq-certificate-vault-client-cert Manually triggered issuance of Certificate default/mbiq-gateway-vault-client-cert Manually triggered issuance of Certificate default/mbiq-sslo-vault-client-cert Manually triggered issuance of Certificate default/mbiq-system-vault-client-cert Manually triggered issuance of Certificate default/mbiq-ingress-nginx-admission Manually triggered issuance of Certificate default/mbiq-instance-vault-client-cert Manually triggered issuance of Certificate default/mbiq-journeys-vault-client-cert Manually triggered issuance of Certificate default/mbiq-llm-vault-client-cert Manually triggered issuance of Certificate default/mbiq-qkview-vault-client-cert Manually triggered issuance of Certificate default/mbiq-upgrade-manager-vault-client-cert Manually triggered issuance of Certificate default/node-exporter-server-cert 2024-03-09T00:14:50+00:00 info: Waiting for certificates to be renewed. 2024-03-09T00:14:50+00:00 info: Certificate mbiq-ingress-nginx-root-cert renewed. 2024-03-09T00:14:50+00:00 info: Certificate mbiq-ado-vault-server-cert renewed. 2024-03-09T00:14:51+00:00 info: Certificate mbiq-ado-vault-client-cert renewed. 2024-03-09T00:14:56+00:00 info: Certificate gateway-feature-ingress-cert renewed. 2024-03-09T00:15:01+00:00 info: Certificate central-manager-ui-ingress-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-apm-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-certificate-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-gateway-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-sslo-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-system-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-ingress-nginx-admission renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-instance-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-journeys-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-llm-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-qkview-vault-client-cert renewed. 2024-03-09T00:15:09+00:00 info: Certificate mbiq-upgrade-manager-vault-client-cert renewed. 2024-03-09T00:15:09+00:00 info: Certificate node-exporter-server-cert renewed. 2024-03-09T00:15:09+00:00 info: Successfully renewed all certificates. 2024-03-09T00:15:09+00:00 info: Restoring PostgreSQL database... 2024-03-09T00:15:12+00:00 info: Restarting init jobs. W0309 00:16:07.005788 2472134 warnings.go:70] path /(mgmt/shared/.*) cannot be used with pathType Prefix 2024-03-09T00:17:03+00:00 info: Successfully restarted init jobs. 2024-03-09T00:17:05+00:00 info: PostgreSQL database has been successfully restored. 2024-03-09T00:17:05+00:00 info: Restarting mbiq-sslo-feature... 2024-03-09T00:17:09+00:00 info: mbiq-sslo-feature has restarted. 2024-03-09T00:17:09+00:00 info: Restarting mbiq-qkview-feature... 2024-03-09T00:17:13+00:00 info: mbiq-qkview-feature has restarted. 2024-03-09T00:17:13+00:00 info: Restarting mbiq-device-feature... 2024-03-09T00:17:17+00:00 info: mbiq-device-feature has restarted. 2024-03-09T00:17:17+00:00 info: Restarting mbiq-certificate-feature... 2024-03-09T00:17:20+00:00 info: mbiq-certificate-feature has restarted. 2024-03-09T00:17:20+00:00 info: Restarting mbiq-gateway-feature... 2024-03-09T00:17:24+00:00 info: mbiq-gateway-feature has restarted. 2024-03-09T00:17:24+00:00 info: Restarting mbiq-proxy-service... 2024-03-09T00:17:28+00:00 info: mbiq-proxy-service has restarted. 2024-03-09T00:17:28+00:00 info: Restarting mbiq-system-feature... 2024-03-09T00:17:35+00:00 info: mbiq-system-feature has restarted. 2024-03-09T00:17:35+00:00 info: Restarting mbiq-apm-feature... 2024-03-09T00:17:46+00:00 info: mbiq-apm-feature has restarted. 2024-03-09T00:17:46+00:00 info: Restarting mbiq-upgrade-manager-feature... 2024-03-09T00:17:49+00:00 info: mbiq-upgrade-manager-feature has restarted. 2024-03-09T00:17:49+00:00 info: Restoring Prometheus... 2024-03-09T00:17:50+00:00 info: Deleting the current Prometheus data... 2024-03-09T00:17:50+00:00 info: Copying Prometheus data from backup... 2024-03-09T00:18:11+00:00 info: Prometheus data has been successfully restored. It may take a few minutes for Prometheus to be available. 2024-03-09T00:18:11+00:00 warning: Only restoring log indexes of ES 2024-03-09T00:18:11+00:00 info: Restoring Elasticsearch... {"acknowledged":true,"persistent":{"action":{"destructive_requires_name":"false"}},"transient":{}}2024-03-09T00:18:11+00:00 info: Closing all indices... 2024-03-09T00:18:12+00:00 info: Deleting all indices... {"acknowledged":true} {"acknowledged":true,"persistent":{"action":{"destructive_requires_name":"true"}},"transient":{}}2024-03-09T00:18:12+00:00 info: Elasticsearch data has been successfully restored. 2024-03-09T00:18:12+00:00 info: Restoring SQLite database... 2024-03-09T00:18:14+00:00 info: Restarting LLM POD 2024-03-09T00:18:18+00:00 info: SQLite database has been successfully restored. 2024-03-09T00:18:18+00:00 info: Migrating old apps to new schema... 2024-03-09T00:18:18+00:00 info: Waiting for migration job to finish... 2024-03-09T00:18:23+00:00 info: Migration job succeeded 2024-03-09T00:18:23+00:00 info: Post-restore updates started... 2024-03-09T00:18:25+00:00 info: Post-restore updates completed successfully 2024-03-09T00:18:25+00:00 info: Restore completed successfully.

In this quick video, you can see the video evidence of his alleged shenanigans removing those resources, them me restoring the backup and validating the resources indeed were restored.