Mitigating Application Threats with BIG-IP Next WAF
Overview of BIG-IP Next In today's modern world where the digital landscape is continuously evolving and security threats are becoming more sophisticated, the need for a robust and adaptive security solution is essential. BIG-IP Next is a next-generation solution which is setting a new standard for safeguarding your digital assets, protecting your applications, and empowering enterprises with the highest security efficacy.BIG-IP Next is the modernized solution optimized to simplify operations, enhance performance, and strengthen security. As per the official website, BIG-IP Next simplifies day-to-day ADC operations and accelerates application time-to-market through automation so that you can focus more on getting your apps online. BIG-IP Next’s modern, highly scalable software architecture is designed for maximum resiliency to support vast, dynamic application portfolios and their most complex traffic management and security policies, ensuring that applications are always available to end users. BIG-IP Next also provides deep insights into your application health, network performance, traffic patterns, and security threats to improve business decision-making. For a quick overview of BIG-IP Next and how the next-generation attributes can help you with your existing or new deployments, check out the video below. Here are some of the key capabilities that you can checkout and learn how you can mitigate app threats and security complexity with BIG-IP Next WAF: 1. Deploy HTTPS application with WAF Protection The first step in protecting your applications starts with onboarding your application in BIG-IP Next instance and creating a WAF security policy as per application requirements. Finally creating load balancers and applying the above-created WAF policies. Next, users can monitor the application traffic by navigating to their respective security dashboards and take necessary steps as per security insights. For more details, see this video. 2. Create and Manage Security Policies Sometimes creating security policies can be a time-consuming job, and BIG-IP Next has made this user-friendly for creating and managing security policies from a centralized UI. Users can create, delete or update their existing policies in fewer steps and can apply them directly to the applications, thereby decreasing the application delivery time to market. You can check out the video below for more details. 3. Create Security Policies using Templates One more advantage of BIG-IP Next is the support for creating security policies using templates and it’s just a one-click action using 'F5 BIG-IP Next’. Users can make use of default templates and protect their applications with zero effort, for ex. Using the Violation Rating Template. For more information, check below video. 4. Security Policy Migration Going through existing BIG-IP security policies and then creating the same ones in BIG-IP Next solution can be time-consuming. This is made easy so that users can migrate their security policy from 'F5 Advanced WAF' to 'F5 BIG-IP Next WAF' in a simple manner. With fewer steps, you can have your entire WAF security posture up without going through the rough step of creating them from scratch. Please refer to the video below for more insights. 5. Signatures and Threat Campaigns Update Regular update of attack signatures and threat campaigns is a vital step in safeguarding your applications against the latest attacks. This process is super easy using ‘F5 BIG-IP Next’ so that applications can mitigate them without the need for downtime. For step-by-step procedure to update signatures and threat campaigns, please check the video below. You can also check out the demo link below for detailed insights of how BIG-IP Next WAF enables the migration of apps and policies between BIG-IP TMOS and BIG-IP Next. The demo also shows how to deploy new web applications with WAF security policies included within BIG-IP Next Central Manager and finally how to analyze and respond to security incidents within the Next WAF dashboard. Reference links What is BIG-IP Next? | DevCentral Getting Started with BIG-IP Next: Fundamentals | DevCentral https://www.f5.com/products/big-ip-services/big-ip-next
What is BIG-IP Next?
BIG-IP Next LTM and BIG-IP Next WAF hit general availability back in October, and we hit the road for a tour around North America for its arrival party! Those who attended one of our F5 Academy sessions got a deep-dive presentation into BIG-IP Next conceptually, and then a lab session to work through migrating workloads and deploying them. I got to attend four of the events and discuss with so many fantastic community members what's old, what's new, what's borrowed, what's blue...no wait--this is no wedding! But for those of us who've been around the block with BIG-IP for a while, if not married to the tech, we definitely have a relationship with it, for better and worse, right? And that's earned. So any time something new, or in our case "Next" comes around, there's risk and fear involved personally. But don't fret. Seriously. It's going to be different in a lot of ways, but it's going to be great. And there are a crap-ton (thank you Mark Rober!) of improvements that once we all make it through the early stages, we'll embrace and wonder why we were even scared in the first place. So with all that said, will you come on the journey with me? In this first of many articles to come from me this year, I'll cover the high-level basics of what is so next about BIG-IP Next, and in future entries we'll be digging into the tech and learning together. BIG-IP and BIG-IP Next Conceptually - A Comparison BIG-IP has been around since before the turn of the century (which is almost old enough to rent a car here in the United States) and this year marks the 20 year anniversary of TMOS. That the traffic management microkernel (TMM) is still grokking like a boss all these years later is a testament to that early innovation! So whereas TMOS as a system is winding down, it's heart, TMM, will go on (cue sappy Celine Dion ditty in 3, 2, 1...) Let's take a look at what was and what is. With TMOS, the data plane and control plane compete for resources as it's one big system. With BIG-IP, the separation of duties is more explicit and intentionally designed to scale on the control plane. Also, the product modules are no longer either completely integrated in TMM or plugins to TMM, but rather, isolated to their own container structures. The image above might convey the idea that LTM or WAF or any of the other modules are single containers, but that's just shown that way for brevity. Each module is an array of containers. But don't let that scare you. The underlying kubernetes architecture is an abstraction that you may--but certainly are not required to--care about. TMM continues to be its awesome TMM self. The significant change operationally is how you interact with BIG-IP. With TMOS, historically you engage directly with each device, even if you have some other tools like BIG-IQ or third-party administration/automation platforms. With BIG-IP Next, everything is centralized on Central Manager, and the BIG-IP Next instances, whether they are running on rSeries, VELOS, or Virtual Edition, are just destinations for your workloads. In fact, outside of sidecar proxies for troubleshooting, instance logins won't even be supported! Yes, this is a paradigm shift. With BIG-IP Next, you will no longer be configuration-object focused. You will be application-focused. You'll still have the nerd-knobs to tweak and turn, but they'll be done within the context of an application declaration. If you haven't started your automation journey yet, you might not be familiar with AS3. It's been out now for years and works with BIG-IP to deploy applications declaratively. Instead of following a long pre-flight checklist with 87 steps to go from nothing to a working application, you simply define the parameters of your application in a blob of JSON data and click the easy button. For BIG-IP Next, this is the way. Now, in the Central Manager GUI, you might interact with FAST templates that deliver a more traditional view into configuring applications, but the underlying configuration engine is all AS3. For more, I hosted aseries of streams in December to introduce AS3 Foundations, I highly recommend you take the time to digest the basics. Benefits I'm Excited About There are many and you can read about them on the product page on F5.com. But here's my short list: API-first. Period. BIG-IP had APIs with iControl from the era before APIs were even cool, but they were not first-class citizens. The resulting performance at scale requires effort to manage effectively. Not only performance, but feature parity among iControl REST, iControl SOAP, tmsh, and the GUI has been a challenge because of the way development occurred over time. Not so with BIG-IP Next. Everything is API-first, so all tooling is able to consume everything. This is huge! Migration assistance. Central Manager has the JOURNEYS tool on sterroids built-in to the experience. Upload your UCS, evaluate your applications to see what can be migrated without updates, and deploy! It really is that easy. Sure, there's work to be done for applications that aren't fully compatible yet, but it's a great start. You can do this piece (and I recommend that you do) before you even think about deploying a single instance just to learn what work you have ahead of you and what solutions you might need to adapt to be ready. Simplified patch/upgrade process. If you know, you know...patches are upgrades with BIG-IP, and not in place at that. This is drastically improved with BIG-IP Next! Because of the containerized nature of the system, individual containers can be targeted for patching, and depending on the container, may not even require a downtime consideration. Release cycle. A more frequent release cadence might terrify the customers among us that like to space out their upgrades to once every three years or so, but for the rest of us, feature delivery to the tune of weeks instead of twice per year is an exciting development (pun intended!) Features I'm Excited About Versioning for iRules and policies. For those of us who write/manage these things, this is huge! Typically I'd version by including it in the title, and I know some who set release tags in repos. With Central Manager, it's built-in and you can deploy iRules and polices by version and do diffs in place. I'm super excited about this! Did I mention the API? On the API front...it's one API, for all functionality. No digging and scraping through the GUI, tmsh, iControl REST, iControl SOAP, building out a node.js app to deploy a custom API endpoint with iControl LX, if even possible with some of the modules like APM or ASM. Nope, it's all there in one API. Glorious. Centralized dashboards. This one is for the Ops teams! Who among us has spent many a day building custom dashboards to consume stats from BIG-IPs across your org to have a single pane of glass to manage? I for one, and I'm thrilled to see system, application, and security data centralized for analysis and alerting. Log/metric streaming. And finally, logs and metrics! Telemetry Streaming from the F5 Automation Toolchain doesn't come forward in BIG-IP Next, but the ideas behind it do. If you need your data elsewhere from Central Manager, you can set up remote logging with OpenTelemetry (see the link in the resources listed below for a first published example of this.) There are some great features coming with DNS, Access, and all the other modules when they are released as well. I'll cover those when they hit general availability. Let's Go! In the coming weeks, I'll be releasing articles on installation and licensing walk-throughs for Central Manager and the instances, andcontent from our awesome group of authors is already starting to flow as well. Here are a few entries you can feast your eyes on, including an instance Proxmox installation: For the kubernetes crowd, BIG-IP Next CNF Solutions for RedHat Openshift Installing BIG-IP Next Instance on Proxmox Remote Logging with BIG-IP Next and OpenTelemetry Are you ready? Grab a trial licensefrom your MyF5 dashboard and get going! And make sure to join us in the BIG-IP Next Academy group here on DevCentral. The launch team is actively engaged there for next-related questions/issues, so that's the place to be in your early journey! Also...if you want the ultimate jump-start for all things BIG-IP Next, join usatAppWorld 2024 in SanJose next month!
BigIP Next - Health Monitors / Template
Just getting feet wet on BigIP Next. Q: Where are the heath monitor (node/pool) created and reconfiguration. Do they support versioning? Q: In the Classic LTM, there was a iAPP for VMWARE Horizon. I know iApps are not supported. Appear the template have replace them. I found a default list of template, but that appeared a little light in vendor application. Are there more template like the VMWARE published somewhere else.Create F5 BIG-IP Next Instance on Proxmox Virtual Environment
If you are looking to deploy a F5 BIG-IP Next instance on Proxmox Virtual Environment (henceforth referred to as Proxmox for the sake of brevity), perhaps in your home lab, here's how: First, download the BIG-IP Next OVA File from MyF5 Downloads. Copy the OVA file to your Proxmox host. I am using SCP in the example below. local $ scp BIG-IP-Next-20.0.1-2.139.10+0.0.136.ovf root@proxmox:~/ On the Proxmox host, extract the contents in the OVA file: proxmox $ cd ~/ proxmox $ tar -xvf BIG-IP-Next-20.0.1-2.139.10+0.0.136.ova BIG-IP-Next-20.0.1-2.139.10+0.0.136.ovf BIG-IP-Next-20.0.1-2.139.10+0.0.136.mf BIG-IP-Next-20.0.1-2.139.10+0.0.136.cert BIG-IP-Next-20.0.1-2.139.10+0.0.136-disk1.vmdk Then, run the command below to create a virtual machine (VM) from the extracted OVF file. <vm_id> should be an unused ID on Proxmox. # qm importovf <vm_id> BIG-IP-Next-20.0.1-2.139.10+0.0.136.ovf local-lvm proxmox $ qm importovf 112 BIG-IP-Next-20.0.1-2.139.10+0.0.136.ovf local-lvm Logical volume "vm-112-disk-0" created. transferred 0.0 B of 80.0 GiB (0.00%) transferred 819.2 MiB of 80.0 GiB (1.00%) transferred 1.6 GiB of 80.0 GiB (2.00%) <output truncated> transferred 80.0 GiB of 80.0 GiB (100.00%) transferred 80.0 GiB of 80.0 GiB (100.00%) You should now see a new VM created on the Proxmox GUI. Before starting the VM, we need to attach a few hardware components to the VM: a Network Device for the management interface one or more additional Network Devices for the data plane interfaces (e.g. internal and external). Note that the data plane Network Devices must be of VirtIO model Optionally, you could also configure CLI access with the following instructions Finally, start the VM. This will take a few minutes. If CLI access is available, open up the console and run kubectl get pods until you can see all pods are ready. The BIG-IP Next VM is now ready to be onboarded per instructions found here.Getting Started with BIG-IP Next: Configuring Instance High Availability
With BIG-IP classic, there are a lot of design choices to make and steps on both systems to arrive at an HA pair. With BIG-IP Next, this is simplified quite a bit. Once configured, the highly available pair is treated by Central Manager as a single entity. There might be alternative options in the future, but as of version 20.1, HA for instances is active/standby only. In this article, I'll walk you through the steps to configure HA for instances in the Central Manager GUI. Background and Prep Work I set up two HA systems in my preparation for this article. The first had dedicated interfaces for the management interface, the external and internal traffic interfaces, and the HA interface. So when configuring the virtual machine, I made sure each system had four NICs. For the second, I merged all the non-management interfaces on a single NIC and used vlan tagging, so those systems had two NICs. In my lab that looks like this: The IP addressing scheme in my lab is shown below. First the four NIC system: 4-NIC System next-4nic-a next-4nic-b floating mgmt 172.16.2.152/24 172.16.2.153/24 172.16.2.151/24 cntrlplane ha (vlan 245) 10.10.245.1/30 10.10.245.2/30 NA dataplane ha (int 1.3) 10.0.5.1/30 10.0.5.2/30 NA dataplane ext (int 1.1) 10.0.2.152/24 10.0.2.153/24 10.0.2.151/24 dataplane int (int 1.2) 10.0.3.152/24 10.0.3.153/24 10.0.3.151/24 And now the two NIC system: 2-NIC System next-2nic-a next-2nic-b floating mgmt 172.16.2.162/24 172.16.2.163/24 172.16.2.161/24 cntrlplane ha (vlan 245) 10.10.245.5/30 10.10.245.6/30 NA dataplane ha (vlan 50) 10.0.5.5/30 10.0.5.6/30 NA dataplane ext (vlan 30) 10.0.2.162/24 10.0.2.163/24 10.0.2.161/24 dataplane int (vlan 40) 10.0.3.162/24 10.0.2.163/24 10.0.3.161/24 Beyond the self IP addresses for your traffic interfaces, you'll need additional IP addresses for the floating address, the control-plane HA sub-interfaces (which are created for you), and teh data-plane HA interfaces. Before proceeding, make sure you have a plan for network segmentation and addressing similar to above, you've installed two like instances, and that one (and only one) of them is licensed. Configuration This walk through is for the 2-NIC system shown above, but the steps are mostly the same. First, login to Central Manager, and click on Manage Instances. Click on the standalone mode for the system you want to be active initially in your HA pair. For me, that's next-2nic-a. (You can also just click on the system name and then select HA in the menu, but this saves a click.) In the pop-up dialog, select Enable HA. Read the notes below to make sure your systems are ready to be paired. On this screen, a list of available standalone systems will populate. Click the down arrow and select your second system, next-2nic-b in my case. Then click Next. On this next prompt, you'll need to create two vlans, one for the control plane and one for the data plane. The control plane mechanics are taken care of for you and you don't need to plan connectivity other than to select an available vlan that won't conflict with anything else in your system. For the data plane, you need to have a dedicated vlan and/or interface set aside. Click Create VLAN for the control plane. Name and tag your vlan. In my case I used cp-ha as my vlan name and tag 245. Click Done. Now click Create VLAN for the data plane. Because I'm tagging all networks on the 2-NIC system, my own interface is 1.1. So I named my data plan vlan dp-ha, set the tag to 50, selected interface 1.1, and clicked Done. Now that both HA VLANs have been created, click Next. On this screen, you'll name your HA pair system. This will need to be unique from other HA pairs, so plan accordingly. I named mine next-ha-1, but that's generic and unlikely to be helpful in your environment. Then set your HA management IP, this is how Central Manager will connect to the HA pair. You can enable auto-failback if desired, but I left that unchecked. For the HA Nodes Addresses, I referenced my addressing table posted at the top of this article and filled those in as appropriate. When you get those filled out, click Next. Now you'll be presented with a list of your traffic VLANs. On my system I have v102-ext and v103-int for my external and internal networks. First, I clicked v102-ext. On this screen you'll need to add a couple rows so you can populate the active node IP, the standby node IP, and the floating IP. The order doesn't matter, but I ordered them as shown, and again referenced my addressing table. Once populated, click Save. That will return you to this screen, where you'll notice that v102-ext now has a green checkbox where the yellow warning was. Now click into your other traffic VLAN (v103-int in my case) if applicable to your environment or skip this next step. This is a repeat of the external traffic network for the internal traffic network. I referenced my address table one more time and filled the details out as appropriate, then clicked Save. Make sure that you have green checkboxes on the traffic VLANs, then click Next. Review the summary of the HA settings you've configured, and if everything looks right, click Deploy to HA. On the "are you sure?" dialog where you're prompted to confirm your deployment, click Yes, Deploy. You'll then see messaging at the top of the HA configuration page for the instance indicating that HA is being created. Also note that the Mode on this page during creation still indicates standalone. Once the deployment is complete, you'll see the mode has changed to HA and the details for your active and standby nodes are provided. Also present here is the Enable automatic failover option, which is enabled by default. This is for software upgrades. If left enabled, the standby unit will be upgraded first, a failover will be executed, and the the remaining system will be upgraded. If in your HA configuration you specified auto-failback, then after the second system is upgraded there will be another failover executed to complete the process. And finally, as seen in the list of instances, there are three now instead of four, with next-ha-1 taking the place of next-2nic-a and next-2nic-b from where we started. Huzzah! You now have a functioning BIG-IP Next HA pair. After we conclude the "Getting Started" series, we'll start to look at the benefits of automation around all the tasks we've covered so far, including HA. The click-ops capabilities are nice to have, but I think you'll find the ability to automate all this from a script or something like an Ansible playbook will really start to drive home the API-first aspects of Next.Getting Started with BIG-IP Next: Licensing Instances in Central Manager
This article assumes that the license was not applied during the initial instance setup. Download the JSON Web Token from MyF5 I don't have a paid license, so I'm going to use my trial license available at MyF5. Your mileage may vary here. Go to my products & plans, trials, and then in the my trials listing (assuming you've requested/received one) click BIG-IP Next. Click downloads and licenses (note, however, the helpful list of resources down in guides and references). You can just copy your JSON web token, but I chose to download. Install the Token Login to Central Manager and click manage instances. Click on your new unlicensed instance. In the left-hand menu at the bottom, click License. Click activate license. We already downloaded our token, so after reviewing the information, click next. Note that I made sure that my Central Manager has access to the licensing server and the steps covered in this article assume the same. If you've managed classic BIG-IP licenses, copying and pasting dossiers to get licenses should be a well-understood process. On this screen, paste your token into the box, give it a name, and click activate. After a brief interrogation of the licensing server, you should now have a healthy, licensed, BIG-IP Next Instance! Resources How to: Manage BIG-IP Next instance licensesBig-IP Next 20.2.0-2.375.1+0.0.43 iRule count problem
I have very simple iRule to show the problem: when HTTP_REQUEST { set Client_IP [IP::client_addr] if { ($Client_IP starts_with "x.x.x.x") && ([HTTP::uri] equals "/seed") } { table set -subtable TABLE "key1" "value1" 30 table set -subtable TABLE "key2" "value2" 15 table set -subtable TABLE "key3" "value3" 45 HTTP::respond 200 content "Done" TCP::close return } set key_value "key1" set key_value2 "key2" set key_value3 "key3" set count [table keys -subtable TABLE -count] HTTP::respond 200 content " Remaining timeout / defined timeout for ${key_value} => [table lookup -notouch -subtable TABLE ${key_value}] [table timeout -subtable TABLE -remaining ${key_value}]/[table timeout -subtable TABLE ${key_value}] Remaining timeout / defined timeout for ${key_value2} => [table lookup -notouch -subtable TABLE ${key_value2}] [table timeout -subtable TABLE -remaining ${key_value2}]/[table timeout -subtable TABLE ${key_value2}] Remaining timeout / defined timeout for ${key_value3} => [table lookup -notouch -subtable TABLE ${key_value3}] [table timeout -subtable TABLE -remaining ${key_value3}]/[table timeout -subtable TABLE ${key_value3}] Count TABLE ${count}" } It looks like table -keys -subtable <tablename> -count don't work properly: Remaining timeout / defined timeout for key1 => value1 27/30 Remaining timeout / defined timeout for key2 => value2 12/15 Remaining timeout / defined timeout for key3 => value3 42/45 Count TABLE 0 My expected output would be 3 (as it is not timeouted), not 0. Can someone check if I am correct? Or tell me how I can count not expired entries in table.Getting Started with BIG-IP Next: Backing Up and Restoring Central Manager
Backing up BIG-IP Next Instances is possible in the Central Manager GUI. Backing up Central Manager, however, requires you to break out those made CLI skilz of yours. And take a backup you shall! You can snapshot your Central Manager virtual machine and restore that as well, but if you want system level backup instead of device level in the event things go south, you need an option currently as high availability, though coming soon to a release near you, is not yet an option for Central Manager. As there will be no screenshots required, most of this is already covered onClouddocs how to on this topic, but in this article, I'll walk through the process by executing the steps and sharing the output. Creating the Central Manager Backup Login to the Central Manager CLI by SSHing to your fqdn or IP address. If you configured the external storage when you set up Central Manager you can do a full backup, which includes all the analytics from Central Manager and your instances. If you only have local storage, you'll need to do a partial. The command to perform the backup and the restore is /opt/cm-bundle/cm. You use the backup subcommand for a backup operation, and as you can probably guess, the restore subcommand for a restore operation. I don't have the external storage in my lab, so I ran a partial backup. admin@cm1:~$ /opt/cm-bundle/cm backup 2024-03-09T00:04:15+00:00 Executing /opt/cm-bundle/cm backup Encryption password: Reenter encryption password: 2024-03-09T00:04:21+00:00 info: Backing up Vault... Created vault backup: /tmp/vault-backup.tgz tar: removing leading '/' from member names var/run/vault-init/ var/run/vault-init/linkerd.csr var/run/vault-init/linkerd.crt var/run/vault-init/vault-client-intermediate-ca.csr var/run/vault-init/vault-client-intermediate-ca.crt var/run/vault-init/unsealkeys var/run/vault-init/ca.crt var/run/vault-init/ingress-intermediate-ca.crt var/run/vault-init/unsealkeys.sha256 var/run/vault-init/linkerd-ca.crt var/run/vault-init/ingress-intermediate-ca.csr var/run/vault-init/linkerd-webhook.csr var/run/vault-init/linkerd-webhook.crt 2024-03-09T00:04:22+00:00 info: Vault backup successful! 2024-03-09T00:04:22+00:00 info: Backing up PostgreSQL... 2024-03-09T00:04:23+00:00 info: PostgreSQL backup successful! 2024-03-09T00:04:23+00:00 info: Performing Prometheus backup... 2024-03-09T00:04:55+00:00 info: Creating Prometheus database snapshot... 2024-03-09T00:05:09+00:00 info: Verifying the Prometheus database snapshot... 2024-03-09T00:05:09+00:00 info: Successfully created Prometheus database snapshot 20240309T000505Z-4c5c8cab103961be 2024-03-09T00:05:09+00:00 info: Copying Prometheus snapshot locally... 2024-03-09T00:05:22+00:00 info: Cleanup the Prometheus snapshot in the pod 2024-03-09T00:05:28+00:00 info: Prometheus backup succeeded! 2024-03-09T00:05:28+00:00 info: Performing Elasticsearch backup... 2024-03-09T00:05:28+00:00 info: Creating Elasticsearch snapshot [elasticsearch-snapshot]... 2024-03-09T00:05:28+00:00 info: Elasticsearch backup succeeded! 2024-03-09T00:05:28+00:00 info: Backing up SQLite... 2024-03-09T00:05:29+00:00 info: SQLite backup successful! 2024-03-09T00:05:29+00:00 info: Creating backup bundle backup.20240309-000421.tgz... 2024-03-09T00:08:26+00:00 info: Encrypting backup bundle... 2024-03-09T00:08:40+00:00 info: Backup bundle created at /opt/cm-backup/backup.20240309-000421.tgz.enc Restoring the Central Manager Backup Sometime after my backup, suppose AubreyKingF5 logged into to my Central Manager and deleted user jrahm and my backup-test certificate (BAD Aubrey!) Maybe he deleted all the resources. Here's the backup script execution on my Central Manager instance. Note the immediate ask for that backup password. Seriously, vault those passwords, don't lose them! admin@cm1:~$ /opt/cm-bundle/cm restore /opt/cm-backup/backup.20240309-000421.tgz.enc 2024-03-09T00:12:40+00:00 Executing /opt/cm-bundle/cm restore /opt/cm-backup/backup.20240309-000421.tgz.enc 2024-03-09T00:12:40+00:00 info: Restoring from backup file /opt/cm-backup/backup.20240309-000421.tgz.enc... Enter decryption password: 2024-03-09T00:12:43+00:00 info: Decrypting backup file... 2024-03-09T00:12:46+00:00 info: Checking available disk space... 2024-03-09T00:13:55+00:00 info: Extracting backup to /opt/cm-backup... 2024-03-09T00:14:35+00:00 info: Validating backup contains all required components 2024-03-09T00:14:35+00:00 info: Restoring Vault... var/run/vault-init/ var/run/vault-init/linkerd.csr var/run/vault-init/linkerd.crt var/run/vault-init/vault-client-intermediate-ca.csr var/run/vault-init/vault-client-intermediate-ca.crt var/run/vault-init/unsealkeys var/run/vault-init/ca.crt var/run/vault-init/ingress-intermediate-ca.crt var/run/vault-init/unsealkeys.sha256 var/run/vault-init/linkerd-ca.crt var/run/vault-init/ingress-intermediate-ca.csr var/run/vault-init/linkerd-webhook.csr var/run/vault-init/linkerd-webhook.crt Vault restored using /tmp/vault-backup.tgz 2024-03-09T00:14:47+00:00 info: Vault data has been successfully restored. 2024-03-09T00:14:47+00:00 info: Renewing all certificates. Manually triggered issuance of Certificate default/mbiq-ingress-nginx-root-cert Manually triggered issuance of Certificate default/mbiq-ado-vault-server-cert Manually triggered issuance of Certificate default/mbiq-ado-vault-client-cert Manually triggered issuance of Certificate default/gateway-feature-ingress-cert Manually triggered issuance of Certificate default/central-manager-ui-ingress-cert Manually triggered issuance of Certificate default/mbiq-apm-vault-client-cert Manually triggered issuance of Certificate default/mbiq-certificate-vault-client-cert Manually triggered issuance of Certificate default/mbiq-gateway-vault-client-cert Manually triggered issuance of Certificate default/mbiq-sslo-vault-client-cert Manually triggered issuance of Certificate default/mbiq-system-vault-client-cert Manually triggered issuance of Certificate default/mbiq-ingress-nginx-admission Manually triggered issuance of Certificate default/mbiq-instance-vault-client-cert Manually triggered issuance of Certificate default/mbiq-journeys-vault-client-cert Manually triggered issuance of Certificate default/mbiq-llm-vault-client-cert Manually triggered issuance of Certificate default/mbiq-qkview-vault-client-cert Manually triggered issuance of Certificate default/mbiq-upgrade-manager-vault-client-cert Manually triggered issuance of Certificate default/node-exporter-server-cert 2024-03-09T00:14:50+00:00 info: Waiting for certificates to be renewed. 2024-03-09T00:14:50+00:00 info: Certificate mbiq-ingress-nginx-root-cert renewed. 2024-03-09T00:14:50+00:00 info: Certificate mbiq-ado-vault-server-cert renewed. 2024-03-09T00:14:51+00:00 info: Certificate mbiq-ado-vault-client-cert renewed. 2024-03-09T00:14:56+00:00 info: Certificate gateway-feature-ingress-cert renewed. 2024-03-09T00:15:01+00:00 info: Certificate central-manager-ui-ingress-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-apm-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-certificate-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-gateway-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-sslo-vault-client-cert renewed. 2024-03-09T00:15:02+00:00 info: Certificate mbiq-system-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-ingress-nginx-admission renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-instance-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-journeys-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-llm-vault-client-cert renewed. 2024-03-09T00:15:03+00:00 info: Certificate mbiq-qkview-vault-client-cert renewed. 2024-03-09T00:15:09+00:00 info: Certificate mbiq-upgrade-manager-vault-client-cert renewed. 2024-03-09T00:15:09+00:00 info: Certificate node-exporter-server-cert renewed. 2024-03-09T00:15:09+00:00 info: Successfully renewed all certificates. 2024-03-09T00:15:09+00:00 info: Restoring PostgreSQL database... 2024-03-09T00:15:12+00:00 info: Restarting init jobs. W0309 00:16:07.005788 2472134 warnings.go:70] path /(mgmt/shared/.*) cannot be used with pathType Prefix 2024-03-09T00:17:03+00:00 info: Successfully restarted init jobs. 2024-03-09T00:17:05+00:00 info: PostgreSQL database has been successfully restored. 2024-03-09T00:17:05+00:00 info: Restarting mbiq-sslo-feature... 2024-03-09T00:17:09+00:00 info: mbiq-sslo-feature has restarted. 2024-03-09T00:17:09+00:00 info: Restarting mbiq-qkview-feature... 2024-03-09T00:17:13+00:00 info: mbiq-qkview-feature has restarted. 2024-03-09T00:17:13+00:00 info: Restarting mbiq-device-feature... 2024-03-09T00:17:17+00:00 info: mbiq-device-feature has restarted. 2024-03-09T00:17:17+00:00 info: Restarting mbiq-certificate-feature... 2024-03-09T00:17:20+00:00 info: mbiq-certificate-feature has restarted. 2024-03-09T00:17:20+00:00 info: Restarting mbiq-gateway-feature... 2024-03-09T00:17:24+00:00 info: mbiq-gateway-feature has restarted. 2024-03-09T00:17:24+00:00 info: Restarting mbiq-proxy-service... 2024-03-09T00:17:28+00:00 info: mbiq-proxy-service has restarted. 2024-03-09T00:17:28+00:00 info: Restarting mbiq-system-feature... 2024-03-09T00:17:35+00:00 info: mbiq-system-feature has restarted. 2024-03-09T00:17:35+00:00 info: Restarting mbiq-apm-feature... 2024-03-09T00:17:46+00:00 info: mbiq-apm-feature has restarted. 2024-03-09T00:17:46+00:00 info: Restarting mbiq-upgrade-manager-feature... 2024-03-09T00:17:49+00:00 info: mbiq-upgrade-manager-feature has restarted. 2024-03-09T00:17:49+00:00 info: Restoring Prometheus... 2024-03-09T00:17:50+00:00 info: Deleting the current Prometheus data... 2024-03-09T00:17:50+00:00 info: Copying Prometheus data from backup... 2024-03-09T00:18:11+00:00 info: Prometheus data has been successfully restored. It may take a few minutes for Prometheus to be available. 2024-03-09T00:18:11+00:00 warning: Only restoring log indexes of ES 2024-03-09T00:18:11+00:00 info: Restoring Elasticsearch... {"acknowledged":true,"persistent":{"action":{"destructive_requires_name":"false"}},"transient":{}}2024-03-09T00:18:11+00:00 info: Closing all indices... 2024-03-09T00:18:12+00:00 info: Deleting all indices... {"acknowledged":true} {"acknowledged":true,"persistent":{"action":{"destructive_requires_name":"true"}},"transient":{}}2024-03-09T00:18:12+00:00 info: Elasticsearch data has been successfully restored. 2024-03-09T00:18:12+00:00 info: Restoring SQLite database... 2024-03-09T00:18:14+00:00 info: Restarting LLM POD 2024-03-09T00:18:18+00:00 info: SQLite database has been successfully restored. 2024-03-09T00:18:18+00:00 info: Migrating old apps to new schema... 2024-03-09T00:18:18+00:00 info: Waiting for migration job to finish... 2024-03-09T00:18:23+00:00 info: Migration job succeeded 2024-03-09T00:18:23+00:00 info: Post-restore updates started... 2024-03-09T00:18:25+00:00 info: Post-restore updates completed successfully 2024-03-09T00:18:25+00:00 info: Restore completed successfully. In this quick video, you can see the video evidence of his alleged shenanigans removing those resources, them me restoring the backup and validating the resources indeed were restored. And that's a wrap! Get your backups going and your processes documented.
