A look on APT operations and using F5 BIG-IP features for mitigation

Introduction

Vulnerabilities are constantly discovered on various platforms and when certain threat actors utilize them and adds their method of operations, these threat actors sometimes become named Advance Persistent Threats - APT for short.

We at the F5 Security Incident Response Team - F5 SIRT, receive questions on mitigating such techniques used

by APTs. Among F5 SIRT's other tasks, we provide guidance on security related questions and where/when possible, share mitigation suggestions using F5 Products.

Advance Persistent Threats

from https://csrc.nist.gov/glossary/term/advanced_persistent_threat

An adversary with sophisticated levels of expertise and significant resources, allowing it through the use of multiple different attack vectors (e.g., cyber, physical, and deception), to generate opportunities to achieve its objectives which are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of continually exfiltrating information and/or to undermine or impede critical aspects of a mission, program, or organization, or place itself in a position to do so in the future; moreover, the advanced persistent threat pursues its objectives repeatedly over an extended period of time, adapting to a defender’s efforts to resist it, and with determination to maintain the level of interaction needed to execute its objectives.

Now we know what an APT is, let’s review an APT group named Sidewinder and their use of the Warhawk backdoor and its delivery.

A research on SideWinder APT and the "WarHawk" backdoor

A walkthrough of SideWinder APT use of a backdoor named "WarHawk".

https://www.zscaler.com/blogs/security-research/warhawk-new-backdoor-arsenal-sidewinder-apt-group-0

SideWinder APT is documented in MITRE | ATT&CK - https://attack.mitre.org/groups/G0121/

Sidewinder is a suspected ... threat actor group that has been active since at least 2012. They have been observed targeting government, military, and business entities throughout Asia...

Here's a quick summary of SideWinder APT use of the Warhawk backdoor based from the walkthrough article:

Victim downloads iso file from - nepra[]org[]pk, file: 32-Advisory-No-32 iso

Victim extracts ISO contents and opens pdf link file and executes included WarHawk backdoor disguised as legit exe files

Malware sends extracted details from the victim to CnC and downloads additional payload

Malware executes commands from CnC – send files to victim machine for next stages – Cobalt Strike, so on and so forth

In the article, certain IP addresses and DNS hostnames were noted.

3[.]239[.]29[.]103 related to fia-gov[.]org & customs-lk[.]org

146[.]190[.]235[.]137

Related hostnames due to being hosted in the same IP as 3[.]239[.]29[.]103

nationalhelpdesk[.]pk

mofa-pk[.]org

sngpl[.]org[.]pk

The malware and its techniques execute in a victim Windows OS system.

A review on the phases of a Cyber Kill Chain

Undestanding the phases of the cyber kill chain will help in the planning for mitigations.

https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html

Phase 1 — Reconnaissance: Adversary identifies and selects a target(s).

Phase 2 — Weaponize: Adversary packages an exploit into a payload designed to execute on the targeted computer/network.

Phase 3 — Deliver: Adversary delivers the payload to the target system(s).

Phase 4 — Exploit: Adversary code is executed on the target system(s).

Phase 5 — Install: Adversary installs remote access software that provides a persistent presence within the targeted environment or system.

Phase 5 — Command and Control: Adversary employs remote access mechanisms to establish a command and control channel with the compromised device.

Phase 6 — Act on Objectives: Adversary pursues intended objectives (e.g., data exfiltration, lateral movement to other targets)

Using F5 BIG-IP features to mitigate certain portions of the cyber kill chain

In the sample APT operations walkthrough, a victim user needs to download and ISO file where the malware is included. Similarly, should the victim user’s machine be exploited, it would connect to the APT controlled IP addresses or hostnames. In the cyber kill chain, this are the “Deliver” and “Command and Control” phases (Phase 3 and 5).

If customer has the BIG-IP in the path of outgoing traffic, such as in forward proxy configurations or BIG-IP as a network gateway or have BIG-IP AFM implemented, there are opportunities to block these IP addresses and hostnames.

Customers can utilize potential BIG-IP mitigation options available to them, provided it matches their network architecture.

Mitigation: Set up the BIG-IP as an explicit http forward proxy and configure an iRule to match malicious hostnames or ip address on the http URI request

The initial idea came from this article https://community.f5.com/t5/technical-articles/configure-the-f5-big-ip-as-an-explicit-forward-web-proxy-using/ta-p/286647 where it shows how to configure the BIG-IP as an explicit http forward proxy.

With the understanding of the Sidewinder APTs use of their controlled IP addresses and hostnames, the BIG-IP in an explicit http forward proxy configuration and position in a customer network can be configured with an iRule to match malicious hostnames or ip addresses in the http request URI of the outgoing traffic.

The operations of the Sidewinder APT required a download of an ISO file for the 1st stage malware delivery and sending messages and beacon traffic to the APT controlled hostname and IP addresses. A BIG-IP in a forward proxy position can inspect outgoing http traffic.

Sample Diagram:

In a lab environment, I have set up a BIG-IP with the following:

BIG-IP Local Traffic Manager (LTM) licensed and provisioned.

BIG-IP DNS licensed and provisioned.

HTTP Profile with Explicit Proxy enabled.

A Standard Virtual Server configured with the HTTP profile with explicit forward proxy set that will be the receiving proxy traffic from a client machine

A Performance Layer 4 Virtual Server that will process tunnelled traffic from the explicit forward proxy Virtual Server

UDP Virtual Server as a DNS listener configured with a DNS profile to handle client machine and act as the DNS resolver of the BIG-IP.

Configure a DNS Zonerunner for a lab/test DNS zone

Configured a DNS Express Zone for a lab/test DNS zone

An iRule watching proxy requests and rejects potentially malicious matched hostname or IP address strings in the URI of the request.

A string type iRule Data group that contains hostnames and ip addresses that were considered malicious.

A client machine that will use the explicit HTTP forward proxy Virtual Server on the BIG-IP.

The BIG-IP DNS configuration used in this setup are optional. However, to simplify the setup, the BIG-IP can function as an Authoritative name server thru DNS Express and maintain a DNS records thru Zonerunner and lessened the number of systems to manage.

Implementing the mitigation only requires the explicit HTTP forward proxy Virtual Server configuration, http proxy irule and iRule data group that contains hostnames and ip addresses that were considered malicious.

Based on the available information, here is the iRule data group configuration with the hostnames and ip addresses.

tmsh output version:

root@(bigip)(cfg-sync Standalone)(Active)(/Common)(tmos)# list ltm data-group internal threatintel
ltm data-group internal threatintel {
    records {
        3[.]239[.]29[.]103 {
            data 3[.]239[.]29[.]103
        }
        146[.]190[.]235[.]137 {
            data 146[.]190[.]235[.]137
        }
        customs[-]lk[.]org {
            data customs[-]lk[.]org
        }
        fia[-]gov[.]org {
            data fia[-]gov[.]org
        }
    }
    type string
}

Here is the proxy iRule – named proxy-block-threatintel for this testing - that matches from the iRule Data Group of malicious IP address and hostname string from the http request URI of the proxied request.

tmsh version:

when HTTP_PROXY_REQUEST {
if { ([class match [string tolower [HTTP::uri]] contains "threatintel"])}{
         reject 
	    log local0. "rejected: source IP: [IP::remote_addr], destination: [IP::local_addr] method [HTTP::method], uri [HTTP::uri]"
	}
}

Here is the explicit http forward proxy VS named vs-explicit-fwd-http-proxy that the client machine will use.

Here is a screenshot from the client machine where it uses the explicit forward http proxy VS IP address and port. It also shows that the client machine can access a web application as the requested site is not matched by the proxy iRule.

Here is a screenshot when the proxied request matches the proxy irule data group IP or hostname string in the request URI.

Here are the corresponding logs from the proxy iRule for matched malicious requests.

Mar 25 09:35:39 bigip info tmm[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method GET, uri http://fia[-]gov[.]org/
Mar 25 09:35:39 bigip info tmm1[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method GET, uri http://fia[-]gov[.]org/
Mar 25 09:35:39 bigip info tmm1[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method GET, uri http://fia[-]gov[.]org/

Here are sample logs when the TLS version of site is accessed – request rejected.

Mar 25 09:38:44 bigip info tmm[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method CONNECT, uri fia[-]gov[.]org:443
Mar 25 09:38:44 bigip info tmm1[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method CONNECT, uri fia[-]gov[.]org:443
Mar 25 09:38:44 bigip info tmm[10568]: Rule /Common/proxy-block-threatintel <HTTP_PROXY_REQUEST>: rejected: source IP: 10.78.189.3, destination: 10.78.189.5 method CONNECT, uri fia[-]gov[.]org:443

Conclusion

F5 BIG-IP have available features that can be used to mitigate potential APT operations and techniques. The sample proxy iRule, data group and explicit http forward proxy virtual server can be considered as an initial configuration should it be needed as a stop gap in environments that may have a BIG-IP device but do not have a full featured forward proxy traffic inspection solution that monitors and actions on outgoing requests to potentially malicious destination host or ip addresses. I would also say that the setup I shared and tested is in a lab environment and with very basic iRule configuration and logic – iRule logic can be improved as needed per your purpose and add/update known malicious hostnames and ip addresses (threat intelligence) in the iRule data group to expand its protections. It is not a SSL Forward proxy setup – which means it can only see the initial proxied request but not the readable decrypted payload. The setup is simple and for its purpose, it works. I hope this article can be helpful in emergency situations and be educational if you are looking into applications of BIG-IP explicit http forward proxy configurations.

Additional Learnings

In the lab setup for this article, I used other BIG-IP configurations such BIG-IP DNS Zonerunner to manage DNS records for the internal DNS zone and DNS Express to act as the authoritative name server the client machine will use.

BIG-IP DNS Zonerunner DNS zone will need to do a zone transfer of DNS records to DNS express to answer queries.

Here is the DNS Zone and records configuration in Zonerunner. I did not modify named Configuration. I only added “also-notify” line in the DNS Zone Options.

Zone configuration in BIG-IP DNS ZoneRunner

In the ltm logs, for any changes to zonerunner zone that is configured with DNS Express, zone transfer logs should be observed.

Mar 27 09:25:22 bigip notice zxfrd[4850]: 0153101c:5: Handling NOTIFY for zone fop.loc.

Mar 27 09:25:24 bigip notice logger[19330]: /bin/sh /usr/lib/csyncd/reloadnamed.sh /var/named/config/namedb/db.external.fop.loc..jnl change  ==> /bin/bigstart start zrd

Mar 27 09:25:27 bigip notice zxfrd[4850]: 0153101c:5: Handling NOTIFY for zone fop.loc.

Mar 27 09:25:27 bigip notice zxfrd[4850]: 0153102c:5: IXFR Transfer of zone fop.loc with SOA Serial 2023032507 from 127.0.0.1 succeeded.

Mar 27 09:25:32 bigip notice zxfrd[4850]: 0153101c:5: Handling NOTIFY for zone fop.loc.

Here are the DNS records configured in Zonerunner

These records can be queried thru the configured DNS lister virtual server on the BIG-IP

Sample query output:

[root@bigip:Active:Standalone] config # dig @10.78.189.4 meh.fop.loc

; <<>> DiG 9.16.33 <<>> @10.78.189.4 meh.fop.loc
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4403
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;meh.fop.loc.                   IN      A

;; ANSWER SECTION:
meh.fop.loc.            30      IN      A       10.78.189.34

;; AUTHORITY SECTION:
fop.loc.                300     IN      NS      ns.fop.loc.

;; ADDITIONAL SECTION:
ns.fop.loc.             300     IN      A       10.78.189.4

;; Query time: 5 msec
;; SERVER: 10.78.189.4#53(10.78.189.4)
;; WHEN: Mon Mar 27 09:27:31 PDT 2023
;; MSG SIZE  rcvd: 89

Notice that the DNS listener is using a DNS profile. The DNS Express feature is enabled for the listener thru the DNS profile.

DNS Express can be enabled on a configured DNS Zone and should match the zone name in Zonerunner. In my lab setup, the zone name is fop.loc and DNS express is enabled on a zone of the same name.

The server defined under DNS Express points to localhost where the BIG-IP zonerunner local bind instance runs.

DNS Express is the feature that actually answers the DNS queries and the records returned are zone transferred to it. Here is the sample output of DNS Express database using the dnsxdump utility where it lists the DNS records it loaded and will send as a response when a query for the configured DNS records are received.

[root@bigip:Active:Standalone] config # dnsxdump
Database serial number 4

DNS-Express DB Dump

-= Arena Allocator =-


-= Region Stats =-
memory: 51 objects (51 small/0 large), 2144 bytes allocated (55 wasted) in 1 chunks, 0 cleanups, 152 in recyclebin 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

-= DB Dump =-
Domain: .
Domain: loc.
Domain: fop.loc.
fop.loc.        300     IN      NS      ns.fop.loc

fop.loc.        300     IN      SOA     ns.fop.loc hostmaster.ns.fop.loc 2023032701 10800 3600 604800 86400

Domain: meh.fop.loc.
meh.fop.loc.    30      IN      A       10.78.189.34

Domain: ns.fop.loc.
ns.fop.loc.     300     IN      A       10.78.189.4

Domain: site1.fop.loc.
site1.fop.loc.  20      IN      A       10.78.189.34

Domain: site2.fop.loc.
site2.fop.loc.  20      IN      A       10.78.189.34

Domain: site3.fop.loc.
site3.fop.loc.  20      IN      A       10.78.189.34


-= DB Stats =-
RR Count: 7
Name Count: 8
RR Count by Type:
 A: 5
 NS: 1
 SOA: 1
[root@bigip:Active:Standalone] config #

That is the DNS part of learning for this lab setup.

Here are articles that I reviewed to setup the DNS portion of this lab.

https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-dns-services-implementations-13-0-0/14.html

https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-dns-services-implementations-13-0-0/1.html#guid-adcc459f-ec0b-4585-b5b6-4619a6b4a491

https://community.f5.com/t5/technical-articles/dns-express-and-zone-transfers/ta-p/281841

While testing the explicit http forward proxy configuration, I encountered unexpected responses from the destination host. I get an ERR_TUNNEL_CONNECTION_FAILED.

I ran a packet capture on the BIG-IP and found the explicit http forward proxy VS returned a HTTP 503 response

Checked the corresponding server side connection towards the backend web application, tcp reset was received with a reset cause of “No server selected”

I looked for instances this reset cause may be received and it was discussed in https://community.f5.com/t5/technical-forum/this-is-a-related-question-when-using-performance-l4-as-a/td-p/305733 which is caused by the enabled Address Translation option on the performance layer 4 wildcard VS. Since there is no pool member on this wildcard VS, the BIG-IP could not find a corresponding pool member address to load balance to, thus, the error/reset cause.

K79443053: Address translation and port translation behavior on a standard virtual server has a nice breakdown of the Address Translation option.

https://my.f5.com/manage/s/article/K79443053

Thus, to resolve this issue in my lab setup, I disabled Address translation on the wildcard VS and traffic flows as expected.

In this sample pcap, the proxied request destination is SSL protected and we can observe the initial HTTP request, but since the setup is not an SSL forward proxy, the actual request to the web application cannot be viewed and what we see is the proxied SSL handshake and data exchange.