Inspecting a UCS file from a compromised BIG-IP

Introduction

Kyle Fox here again with some information I think may be helpful for customers experiencing a compromised F5 BIG-IP or are looking to examine their UCS files for compromise after a breach or ransomware attack. We ask that customers refer to K11438344: Considerations and guidance when you suspect a security compromise on a BIG-IP system first before using this document.

This article is intended to further extend the guidance in the KB article K4423: Overview of UCS archives, with an eye towards customers who need to check their UCS files for malicious configurations after a breach or ransomware attack. This article is not intended to be complete guidance on how to accomplish this task, but a reference for cybersecurity professionals experienced in examining systems for compromise. If there are missing items or items that need more clarity, please comment below and I will update the article.

As always, the best practice is to keep up-to-date backups of UCS archives in a place where attackers would not be able to modify them, but that is not always possible. Sometimes it is helpful to have a UCS archive, even if it’s not up to date, to compare against the current system archive.

File Type

UCS files are gzipped tar archives of BIG-IP configuration files, so they can be manipulated with any tool that can deal with TAR archives like the Linux "tar" command. See K4423 for more on that.

Contents

In this section, we will break down the contents of the archive file.

You can list the contents using the command: tar -ztf example.ucs

Important Files to Check

First, we are going to look at Linux related files, because these are likely to be modified by an attacker:

Select files from /etc are archived:
etc/adjtime - system time offset from hardware clock.
etc/hosts.deny - Part of an old access control scheme, see hosts.deny(5).
etc/openldap/* - OpenLDAP Configuration files, used as part of LDAP or Active Directory authentication.
etc/motd - Message of the Day file, used to display a banner on login to the system shell.
etc/nsswitch.conf - Name Service Switch configuration, used for external authentication and identity services, like RADIUS, LDAP and Active Directory.
etc/passwd - passwd file, stores a list of users on the system, including Linux internal users.
etc/shadow - Password Shadow file, stores the passwords and additional access control for Linux users.
etc/security/ - Configuration files for Pluggable Authentication Modules.
etc/localtime - Configures system timezone.
etc/ethmap - Part of the mapping of vurtual NICs to interfaces in TMM.
etc/f5pcimap - Part of the mapping of vurtual NICs to interfaces in TMM.
etc/syslog-ng/* - Syslog-NG configuration.
etc/wr_urldbd/* Part of URLDB configuration.
etc/qkview_obfuscate.conf - Part of the configuration for QKView Obfuscation for iHealth.
etc/krb5.conf - Kerberos configuration.

It’s important to note that by default none of the startup scripts are saved in the UCS file, so malware that uses those files to restart itself after a system restart won’t usually persist in a UCS file. In the past, customers have used this functionality to clean a system of persistent malware by installing a new boot slot and booting to it.   Since there is insufficient certainty that this will work, F5 can't recommend it as a solution. It is preferable to wipe out all persistent storage on the BIG-IP and reinstall from scratch.

Next up, we have dotfiles, or user-specific configuration files. In some cases attackers will use these to persist in a system with a number of methods.

Dotfiles from users on the system:
home/
home/f5_remoteuser/
home/f5_remoteuser/.bash_profile
home/f5_remoteuser/.bashrc
home/f5_remoteuser/.bash_logout
home/admin/
home/admin/.bash_profile
home/admin/.ssh/
home/admin/.ssh/README
home/admin/.bashrc
home/admin/.bash_logout
home/root/
home/root/.bash_profile
home/root/.bashrc
home/root/.bash_logout
root/
root/.bash_profile
root/.bash_history
root/.bashrc
root/.tmsh-history-root

I included a full list of those files above from the example UCS file so you get an idea of what dotfiles to watch out for attackers using for persistence. In particular the .bashrc file can be used to execute code on login of a user. This could be used to store a hook that can revive malware that has otherwise been cleaned. the .ssh/* directory includes files for SSH configuration, and attackers can use ssh/authorized_keys to maintain access to the system using an SSH key, in typical customer installations where passwords are used to login to the system, the .ssh/authorized_keys will not exist or be empty.

Remaining Files

The rest of the files are more BIG-IP specific, so we do not expect attackers to have modified them.

Manifest files:
SPEC-Manifest
SPEC-Files

ASM/AWAF configuration save:
var/tmp/ts_db.save_dir_location.cstmp
var/tmp/ts_db.save_dir_#####.cstmp/*
var/ts/*

APM Configuration Files:
var/apm/*

NTP Drift Config:
var/lib/ntp/drift

DHCP Client Data (Current leases, etc):
var/lib/dhclient/*

Files used for IPS/Classification Engine:
var/libdata/*

Files used for the GUI:
var/lib/hsqldb/*

Contents of /config is saved:
config/*

But excluding the filestore contents, which is saved in a different location:
var/tmp/filestore_temp/*

System certificates:
var/tmp/cert_temp/*

GTM config:
var/tmp/gtm_tmp/*

Enterprise Manager related files:
var/tmp/em_db_temp/*

These appear to store the counters for SNMP stats:
var/tmp/storage_temp/*

REST configuration files:
var/config/rest/*

named (BIND) configurations:
var/named/*

Configuration related to cloud based devices (Azure, AWS, etc):
etc/cloud/*

Syntax files used to load configuration on a different version:
var/tmp/tmsh_syntax/_VERSION_/*

Service discovery module:
var/sdm/*

iRules LX:
var/ilx/*

Behavioral DoS configs:
var/Autodosd/*
var/bdosd/*

ASM/AWAF datasyncd configs:
var/datasync/*

Deep Packet Inspection configs:
var/libdata/dpi/*

Related to data classes in iRules:
var/class/ 

Conclusion

I hope that with the above guidance, customers who have had a compromised system and firms engaged in assisting those customers will be better able to handle getting systems cleaned up and back online.