BIG-IP TMOS Forensic Pointers

Notes on Forensics and Support

Computer forensics is typically conducted by specialized firms that have all of the licensing and documented training needed to conduct said forensics. Typically in the United States, licensing requirements start with the firm being licensed for private investigations, and this also often requires individual forensic analysts to be licensed as private investigators. In some states, they may have additional licensing for computer forensics, and some states attach criminal liability to operating as a forensic analyst for hire without a license. Forensic analysts also need to be able to establish in court their own expertise. This is where documented training and experience comes in as a forensic analyst is only useful testifying in court as an expert witness. Because of these requirements, F5 Support is able to assist in forensic investigations as an advisor to either customers internal forensic analysts or hired outside forensic analysts.  We can provide information about how F5 products work, how data is moved, stored, retrieved and sent on F5 products and other questions the forensic analysts have while conducting their investigation.

 

UCS Files

UCS Files are the configuration archive file for BIG-IP TMOS. They are a gzipped tar archive of configuration files and database dumps. We have previously gone over the contents of the file in an article. Since its a tar archive, it can be unpacked and examined and compared to previous versions using file tree comparison tools.

K4423: Overview of UCS archives

K13132: Backing up and restoring BIG-IP configuration files with a UCS archive

K86995543: Identifying large files in UCS

It is recommended that customers establish a mechanism to periodically capture UCS files and store them in case of an incident, many customers also create before-and after UCS files as part of their change management process.

Passwords and passphrases on the system at rest and in a UCS file are encrypted using the Secure Vault feature. This prevents them from being decrypted without the master key being stored on the hardware. All units in a cluster of BIG-IP systems share the same master key.

K73034260: Overview of the BIG-IP system Secure Vault feature

K000149030: Copying the master key from the source to the target system when restoring a UCS archive

 

QKViews

The "Quick View" or QKView file was introduced in BIG-IP TMOS Version 9.0 as a snapshot of the running BIG-IP for support to analyze. At the time, it was a single zipped file with the output of a myriad of commands in it.  In Version 10.0 it was expanded into what it is today, a gzipped tar archive full of configuration files, log files, database dumps and statistical (RRD) data from the system. The QKView process typically truncates log files and some other files to 5MB during collection, but that size can be set to the maximum allowed by executing the qkview command on the command line with "qkview -s 0", see link below for additional command line options. QKViews are typically viewed in the iHealth tool, which provides a graphical interface for examining logs, command output, and graphs and also runs a number of diagnostic heuristics against the QKView to identify common issues.

K12878: Generating diagnostic data using the qkview utility

K23928121: Overview of qkview command-line options

K000134522: Reviewing journal data files from F5 Qkviews

K60787205: Understand and manage sensitive data in a QKView

The current article overviewing the contents of a QKView is K60787205 referenced above. In addition, we would like to point out the function of some files in the QKView:

• HWINFO: Basic hardware information, like the bas MAC address, cores, memory size and interfaces.
• PLATFORM: Hardware/Software platform type.  Virtual editions are Z100 and vCMP are Z101.
• VERSION(.LTM): Software version information.
• *_module.xml: Files for each QKView module. Most of these correspond to either licensed modules like APM, AFM AWAF (named ASM in the QKView), specific daemons like postgres or QKView collection modules like proc and mcp. The proc_module.xml files contain information about running processes.
• qkview_run.data: Information about the QKView generation process for this QKView as well as error output, if applicable.
• skipped_files: Files that were skipped, typically because they were empty.

The QKView should contain key files in home directories /home/<user> and /root. There are a couple of other directories that may be notable:

• /var/tmp/asm_tables_dump: Contains a partial dump of the MySQL tables AWAF/ASM uses for configuration and logging.
• /var/tmp/qkview-rrd: Contains the RRD data used to generate graphs in the BIG-IP TMOS user interface and also in iHealth when a QKView is loaded there.

 

Daemons

The BIG-IP TMOS system uses a large number of daemons to process traffic, update configurations and log events. Here are the key daemons on a BIG-IP system:

• tmm: Processes traffic and provides all LTM features except some health checks.
• bd: Processes web requests and responses for the Advanced WAF feature.
• avrd: Processes traffic for statistical and DoS detection features.
• bigd: Provides monitoring capability for monitoring pools and other traffic opbjects.
• big3d: Provides a communication channel for the GTM/GSLB feature.
• gtmd: Manages the GTM/GSLB feature.
• iprepd: Downloads IP Reputation updates when that feature is licensed.
• mcpd: Manages the BIG-IP configuration and updates various daemons when configuration changes are made.

For a more complete list see:

K48615077: BIG-IP daemons (15.x - 17.x)

 

Disk Configuration

BIG-IP systems that contain a RAID array utilize the software RAID feature in Linux. All BIG-IP TMOS systems use the Linux LVM system for storage management and utilize a "boot slot" abstraction, for example:

• Volumes that are shared across all boot slots:

/dev/vg-db-vda/dat.maint.1

/dev/vg-db-vda/dat.share

/dev/vg-db-vda/dat.appdata

/dev/vg-db-vda/dat.log

/dev/vg-db-vda/dat.swapvol

 

• Volumes that are specific to boot slot 1:

/dev/vg-db-vda/set.1.root

/dev/vg-db-vda/set.1._usr

/dev/vg-db-vda/set.1._config

/dev/vg-db-vda/set.1._var

 

K14403: Maintaining disk space on the BIG-IP system

The volumes on the BIG-IP TMOS system can be mounted in Linux if they are located in a virtual machine disk image, or if the RMA Disk Retention Option is purchased, by removing the disks from the BIG-IP system.

K92469153: Mounting a BIG-IP VE disk volume for access on a linux VM

It is also possible to use the Maintenance Operating System boot option to get a rudimentary Linux environment and copy disks using tools like dd and netcat, but this is not recommended because it may cause modifications to the disks while being performed.

K14245: Overview of the recovery tasks performed from the MOS (11.x - 17.x)

 

Memory Configuration

The BIG-IP TMOS system uses normal Linux memory allocation for most processes with a couple of exceptions:

• tmm: TMM will allocate all the memory it will ever use on startup and internally manage its memory.
• bd: BD allocates an amount of memory on startup and internally manages that, up to a limit it will allocate more memory as needed, but it will not return that memory when freed in its internal allocator.

Process memory can be copied live from the /proc filesystem, but the standard disclaimers about performing forensics on a running system applies.

 

TMM Memory Configuration

An overview of TMM memory allocation is linked below and the current state of memory allocation can be viewed using the tmsh commands "show sys memory", the output of which is also available when viewing a QKView on iHealth.  We are currently working on determining what additional information related to TMM memory allocation can be publicly documented.

K16419: Overview of BIG-IP memory usage

 

Coring TMM

A core file can be generated for TMM and that file examined, in F5 Support. Core files will only be able to be examined by F5 Engineering Services or F5 Product Development, and we typically do not examine core files from compromised systems.  The process memory can be examined by a forensic analyst using GDB or other tools, but the proprietary symbol files cannot be provided to customers.

K15569543: Producing a TMM diagnostic core file

 

Bash and TMSH History

Two shells are available to users with access to BIG-IP TMOS, the native shell is the TMSH shell, which provides commands to add, view, change and delete configuration, view statistics and log files and other administrative tasks. Since TMSH supports the complete access control scheme of BIG-IP TMOS, it can be made available to users of all access levels. The second shell available is the Bash shell, referred to as the Advanced Shell in the documentation, this shell is only available to users with administrator-level access and runs as a superuser when accessed.

K000135228: Enabling bash or tmsh access on the BIG-IP system

• Bash history is located in the typical .bash_history file in the users directory. Commands are timestamped using the unix epoch.
• TMSH history is located in a file called .tmsh-history-<username> in the users directory. Commands are timestamped with the system time.
• User directories are /root for root and /user/<username> for other users.

 

Log Files

Logs are located in /var/log, this includes many of the typical log files for a Linux system. Some additional logging is done by systemd in /var/log/journal/* information on how to view these log files is available in K000134522. An overview of the different log files is available in K16197, but we would like to review some pointers here:

• /var/log/secure and /var/log/audit will contain information about logins to the system and some information about changes.
• Sometimes changes in the GUI are logged in /var/log/tomcat/catalina.out especially when an error happens.
• Changes made with iControl REST may be logged to /var/log/restjavad*.
• Changes sometimes generate messages in /var/log/ltm*.
• Apache httpd is typically not configured to log accesses, log files are in /var/log/httpd/, errors are still logged.
• Some Linux daemons log to /var/log/daemon.log including BIND/named.

K16197: Reviewing BIG-IP log files

K000134522: Reviewing journal data files from F5 Qkviews

The BIG-IP TMOS system typically rotates log files once a day when cron.daily is executed. By default, log files are retained for 8 days. These settings are configurable depending on how much space is allocated for log files and how much logging is being generated. If the log volume becomes over-full, the log rotation process may be run early to attempt to purge log files to prevent it from becoming completely full. If the log volume becomes completely full, some daemons, in particular httpd may fail to run correctly, so care must be taken with logging to the local filesystem.

K13367: Managing log files on the BIG-IP system (11.x - 17.x)

K29900456: Troubleshooting issues with log rotation and archiving

F5 encourages customers to configure remote logging for the system so that log files are harder to compromise by an attacker and so that if there is a catastrophic failure of the system, some data can be examined to determine what happened.

K13080: Configuring the BIG-IP system to log to a remote syslog server (11.x - 17.x)

 

Malware on BIG-IP TMOS

We have put together an article linked below describing the process the F5 SIRT uses to check for malware in QKViews. This may provide additional insights into a compromised system. There is also a support article linked below that provides guidance on how to deal with a compromised BIG-IP.

How the F5 SIRT Looks for Malware

K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system

 

In Conclusion

This article was put together to assist forensic analysts in examining the BIG-IP TMOS system. We will revise this article as needed when we have more information that can be incorporated into it or articles linked from it. The F5 SIRT team welcomes your feedback here on DevCentral. You can also provide feedback to individual articles in the knowledge base using the feedback tool at the bottom of each article or in the course of support cases with the F5 Support team.

 

You can Contact the F5 SIRT by opening up a service request and/or follow the guidance on the Report a Vulnerability page.