Knowledge sharing: Velos and rSeries (F5OS) basic troubleshooting, logs and commands

This another part of my Knowledge sharing articles, where I will take a deeper look into Velos and rSeries investigation of issues, logs and command.

1. Velos HA controller and blade issues.

As the Velos system is the one with two controllers in active/standby mode only with Velos it could be needed to check if there is an issue with the controller's HA. As the controller's HA order can be different for the system and the different partitions to check the HA for the system use the /var/log_controller/cc-confd file or for a partition HA issue look at the partition velos log at /var/F5/partition<ID>/log/velos.log . Also you can enable HA debug for the controllers with " system dbvars config debug confd ha-state-machine true ".

Overview of HA:

https://support.f5.com/csp/article/K19204400

Controller HA:

https://support.f5.com/csp/article/K21130014

Partition HA:

https://support.f5.com/csp/article/K58515297

List of Velos/rSeries services:

Overview of F5 VELOS chassis controller services

Overview of F5 VELOS partition services

Overview of F5 rSeries system services

2. Entering into F5OS objects.

The rSeries and Velos tenants are like vCMP quests with VIPRION and sometimes if there are access issues with them it could be needed to open their console. For this the "virtctl" command can be used and as an example " /usr/share/omd/kubevirt/virtctl console <tenant_name>-<tenant_instance_ID> ". Also as velos uses blades and partitions it could be needed to ssh to a blade with " ssh slot<number> " or to enter a partition with " docker exec -it partition<ID>_cli su admin " as sometimes for example to see the GUI logs entering the GUI container for the partition could be needed but F5 support will for this in most cases and maybe this will be the way to enter the BIG-IP NEXT CLI.

Overview of VELOS system architecture:

https://support.f5.com/csp/article/K73364432

Overview of rSeries system architecture:

https://support.f5.com/csp/article/K49918625

rSeries tanant access:

https://support.f5.com/csp/article/K33373310

Velos blade and tenant access:

https://support.f5.com/csp/article/K65442484

Velos partition access:

https://support.f5.com/csp/article/K11206563

3. Usefull commands and logs.

For Velos/rSeries as this is a system with a cluster the "show cluster" command is usefull to see any issues (look fo "cluster is NOT ready."). Also the velos.log for the controller and partitions is a great place to start and debug level can be enabled for it under " SYSTEM SETTINGS Log Settings " as this is also the place for rSeries logging to be set to debug. Also the /var/log/openshift.log is good be checked with velos if there are cluster issues or or ks3.log in rSeries. Also the confd logs are like mcpd logs, so they are really usefull for Velos or rSeries. Other nice commands are docker ps, oc get pod --all-namespaces -o wide, kubectl get pod --all-namespaces -o wide but the support will ask for them in most cases.

Velos cluster status:

https://support.f5.com/csp/article/K27427444

Velos debug:

https://support.f5.com/csp/article/K51486849

Velos openshift example issue:

https://support.f5.com/csp/article/K01030619

Monitoring Velos:

https://clouddocs.f5.com/training/community/velos-training/html/monitoring_velos.html

Monitoring rSeries:

https://clouddocs.f5.com/training/community/rseries-training/html/monitoring_rseries.html

4. Velos and rSeries tcpdumps packet captures, file utility and qkview files.

For Velos qkviews ca be created for controller or partition as they are seperate qkviews. Tcpdumps for client traffic are done a tcpdump utility from the F5OS (su - admin) and a tcpdump in the Linux kernel is just for the managment ip addresses of the appliance , controller (floating or local) , partition or tenant. The file utility allows for file transfers to remote servers or even downloading any log from the Velos/rSeries to your computer as this was not possible before with iSeries or Viprion. Also the file utility starts outbound session to the remote servers so this an extra security as no inbound sessions need to be allowed on the firewall/web proxy and it can be even triggered by API call and I may make a codeshare article for this.

Velos tcpdump utility:

https://support.f5.com/csp/article/K12313135

rSeries tcpdump utility:

https://support.f5.com/csp/article/K80685750

Qkview Velos:

https://support.f5.com/csp/article/K02521182

Qkview Velos CLI location:

https://support.f5.com/csp/article/K79603072

Qkview rSeries:

https://support.f5.com/csp/article/K04756153

SCP:

https://support.f5.com/csp/article/K34776373

For rSeries 2000/4000 tcpdump is different as SR-IOV not FPGA (rSeries Networking (f5.com)) is used to attach interfaces directly to the tenant VM:

Article Detail (f5.com)

5. A final fast check could be to use ''kubectl get pods -o wide--all-namespaces'' (with Velos also ''oc get pods -o wide --all-namespaces'' should also work) to see that all pods are ok and running. Also ''docker ps'' or '' docker ps --format 'table {{.Names}}\t{{.RunningFor}}\t{{.Status}}' '' are usefull to see a container that could be going down and up and this can be correlated with issues seen with "show cluster" command.

6. The new F5OS has much better hardware diagnostics than the old devices, so no more the need to do EUD tests as all system hardware components and their health can be viewed from the GUI or CLI and also this is shown in F5 ihealth!

https://techdocs.f5.com/en-us/velos-1-5-0/velos-systems-administration-configuration/title-system-settings.html

7. For Velos and rSeries always keep the software up to date as for example I will give with the Velos 1.5.1 the cluster rebuild because of the openshift ssl cert being 1 year is much simpler or the F5 rSeries and the Cisco Nexus issues or the corrupt Qkview generation when the GUI not the CLI is used (the velos cluster rebuild with touch /var/omd/CLUSTER_REINSTALL can solve many issues but it will cause some timeout):

http://cdn.f5.com/product/bugtracker/ID1135853.html

https://my.f5.com/manage/s/article/K000092905

https://support.f5.com/csp/article/K79603072

In the future ''docker'' commands could be not available but then just use "crictl" as this replaces the docker init system for kubernetes.

Updated Sep 05, 2024

Version 9.0

application delivery

devops