Decrypt, Encrypt, Decrypt, Encrypt, Encrypt....
What do you do when you need to decrypt traffic, apply a security service chain policy to it, and then send it to an array of open source security tools and never send an unencrypted packet on the AWS "wire"? And what if these security tools did not support TLS? What if you had to do this transparently?
If we break it down, we need to:
- Decrypt the incoming traffic so we can apply the security service chain.
- Encrypt the traffic and send it to the security system.
- Decrypt the traffic from the security system
- Encrypt the traffic to the tap
- Encrypt the traffic to the backend
The Security Stack
In our deployment, we will use two open source security tools. These were picked by the client that I was working with and I will not cover how to configure them in detail.
Suricata
If you are not familiar with Suricata, it is an open-source IDS/IPS security tool that supports packet captures, alerting among many other functions. In our deployment, we will configure Suricata to be an inline IPS with netfilter.
Arkime
Arkime "is a large-scale, open-source, indexed packet capture and search tool" allowing for graphing, visibility and search of network traffic.
SSL Orchestrator
If you are reading this, you probably know what SSL Orchestrator is. If not, please refer to F5 CloudDocs or search DevCentral to learn about all of the amazing things that SSLO can do. This is where security service logic and traffic steering happens.
The Topology
We are going to use the same Gateway Load Balancer (GWLB) topology that was used in my article series Increase Security in AWS without Rearchitecting your Applications; with that in mind you can use all the items below and use SSLO in a reverse proxy topology.
The change is we are going to use GRE tunnels between Suricata and BIG-IP and a GRETAP tunnel between BIGIP and Arkime. We will then wrap the GRE tunnels in IPSEC to ensure that traffic is encrypted. The backend web servers are using SSL with BIG-IP and the clients are using SSL, which is terminated on SSLO.
Building Overlays in the Overlay of AWS VPC Networking
To bring all of this together, we are going to build tunnels in the VPC. The type of tunnel we are using depends on what we need it to do. For a network tap, we need to copy the same IP packet into an L2 frame that has a different destination mac address. Since we also need to encrypt it, we need to pick a tunnel. I used a GRETAP interface, but VXLAN should work too. To deploy the Suricata instance, we need to move untranslated IP packets AND we need to encrypt them. To accomplish this, I used standard GRE interfaces.
Arkime Network Interfaces and GRE
Here we create the GRETAP interface and assign it the created IP address from SSLO. We will need the mac address for the SSLO config.
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 172.31.100.228 netmask 255.255.255.0 broadcast 172.31.100.255
inet6 fe80::dd:5cff:fe7d:fe9b prefixlen 64 scopeid 0x20<link>
ether 02:dd:5c:7d:fe:9b txqueuelen 1000 (Ethernet)
ens6: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9001
inet 172.31.15.198 netmask 255.255.255.0 broadcast 172.31.15.255
inet6 fe80::99:d5ff:fe61:18b3 prefixlen 64 scopeid 0x20<link>
ether 02:99:d5:61:18:b3 txqueuelen 1000 (Ethernet)
tun0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 8963
inet 198.19.0.10 netmask 255.255.255.0 broadcast 198.19.0.255
inet6 fe80::70a5:76ff:fed2:a3ec prefixlen 64 scopeid 0x20<link>
ether 72:a5:76:d2:a3:ec txqueuelen 1000 (Ethernet)
sudo ip link add tun0 type gretap local 172.31.15.198 remote 172.31.15.10 ttl 255
sudo ifconfig tun0 198.19.0.10
sudo ifconfig tun0 upArkime IPSEC
Now that we have the GRETAP interface, we need to configure one IPSEC process to encrypt the traffic.
ubuntu@ip-172-31-100-228:~$ sudo cat /etc/ipsec.conf
config setup
charondebug="ike 1, knl 1, cfg 0, net 1"
strictcrlpolicy=no
uniqueids=yes
cachecrls=no
conn ipsec-ikev2-vpn
authby=secret
left=%defaultroute
leftid=172.31.15.198
leftsubnet=0.0.0.0/0
right=172.31.15.10
rightsubnet=0.0.0.0/0
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
ubuntu@ip-172-31-100-228:~$ Suricata GRE
For Suricata, we are routing IP packets and a standard GRE tunnel will work.
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 172.31.11.20 netmask 255.255.255.0 broadcast 172.31.11.255
inet6 fe80::80:36ff:fe34:3325 prefixlen 64 scopeid 0x20<link>
ether 02:80:36:34:33:25 txqueuelen 1000 (Ethernet)
ens6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
inet 172.31.12.20 netmask 255.255.255.0 broadcast 172.31.12.255
inet6 fe80::d6:aff:fee3:c25f prefixlen 64 scopeid 0x20<link>
ether 02:d6:0a:e3:c2:5f txqueuelen 1000 (Ethernet)
gre1: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 8977
inet 100.64.0.20 netmask 255.255.255.0 destination 100.64.0.20
inet6 fe80::ac1f:b14 prefixlen 64 scopeid 0x20<link>
gre2: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 8977
inet 100.64.1.20 netmask 255.255.255.0 destination 100.64.1.20
inet6 fe80::ac1f:c14 prefixlen 64 scopeid 0x20<link>
#!/bin/bash
sudo ip tunnel add gre1 mode gre local 172.31.11.20 remote 172.31.11.10
sudo ip tunnel add gre2 mode gre local 172.31.12.20 remote 172.31.12.10
sudo ifconfig gre2 100.64.1.20 netmask 255.255.255.0 up
sudo ifconfig gre1 100.64.0.20 netmask 255.255.255.0 up
sudo ip route add 10.0.0.0/8 via 100.64.1.20
sudo ip route add 0.0.0.0/0 via 100.64.0.10
sudo iptables -I FORWARD -i gre1 -o gre2 -j NFQUEUE
sudo iptables -I FORWARD -i gre2 -o gre1 -j NFQUEUE
sudo suricata -c /etc/suricata/suricata.yaml -q 0 &
SURICATA IPSEC
For the Suricata instance, we need two IPSEC peers to create our network topology.
ubuntu@ip-172-31-11-20:~$ sudo cat /etc/ipsec.conf
config setup
charondebug="ike 1, knl 1, cfg 0, net 1"
strictcrlpolicy=no
uniqueids=yes
cachecrls=no
conn ipsec-ikev2-vpn
authby=secret
left=%defaultroute
leftid=172.31.11.20
leftsubnet=0.0.0.0/0
right=172.31.11.10
rightsubnet=0.0.0.0/0
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
conn ipsec-ikev2-vpn-2
authby=secret
left=%defaultroute
leftid=172.31.12.20
leftsubnet=0.0.0.0/0
right=172.31.12.10
rightsubnet=0.0.0.0/0
ike=aes256-sha2_256-modp1024!
esp=aes256-sha2_256!
keyingtries=0
ikelifetime=1h
lifetime=8h
dpddelay=30
dpdtimeout=120
dpdaction=restart
auto=start
ubuntu@ip-172-31-11-20:~$ sudo ipsec status
Security Associations (4 up, 0 connecting):
ipsec-ikev2-vpn-2[2015]: ESTABLISHED 6 seconds ago, 172.31.12.20[172.31.12.20]...172.31.12.10[172.31.12.10]
ipsec-ikev2-vpn-2{2015}: INSTALLED, TUNNEL, reqid 2, ESP SPIs: c0f72a1f_i 0e5d3a98_o
ipsec-ikev2-vpn-2{2015}: 172.31.12.20/32 === 172.31.12.10/32
ipsec-ikev2-vpn[2014]: ESTABLISHED 108 seconds ago, 172.31.11.20[172.31.11.20]...172.31.11.10[172.31.11.10]
ipsec-ikev2-vpn{2014}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cd5853ac_i 0199a2f6_o
ipsec-ikev2-vpn{2014}: 172.31.11.20/32 === 172.31.11.10/32
ipsec-ikev2-vpn[2013]: ESTABLISHED 7 minutes ago, 172.31.11.20[172.31.11.20]...172.31.11.10[172.31.11.10]
ipsec-ikev2-vpn{2013}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: c78843ac_i 0df07e34_o
ipsec-ikev2-vpn{2013}: 172.31.11.20/32 === 172.31.11.10/32
ipsec-ikev2-vpn-2[2012]: ESTABLISHED 32 minutes ago, 172.31.12.20[172.31.12.20]...172.31.12.10[172.31.12.10]
ipsec-ikev2-vpn-2{2012}: INSTALLED, TUNNEL, reqid 2, ESP SPIs: c0addbb2_i 0d653b4d_o
ipsec-ikev2-vpn-2{2012}: 172.31.12.20/32 === 172.31.12.10/32
ubuntu@ip-172-31-11-20:~$
BIG-IP GRE Tunnels
We need to configure two tunnels for Suricata and one GRETAP for Arkime.
admin@(ip-172-31-100-10)(cfg-sync Standalone)(Active)(/Common)(tmos.net.tunnels)# list
net tunnels geneve aws-geneve {
app-service none
defaults-from geneve
description none
}
net tunnels gre arkime-gre-profile {
app-service none
defaults-from gre
description none
encapsulation transparent-ethernet-bridging
flooding-type none
rx-csum disabled
tx-csum disabled
}
net tunnels tunnel arkime-tap {
if-index 288
local-address 172.31.15.10
profile arkime-gre-profile
remote-address any
}
net tunnels tunnel geneve-tunnel {
if-index 304
local-address 172.31.10.10
profile aws-geneve
remote-address any
}
net tunnels tunnel gre-tunnel-from-suricata {
if-index 320
local-address 172.31.12.10
profile gre
remote-address 172.31.12.20
}
net tunnels tunnel gre-tunnel-to-suricata {
if-index 336
local-address 172.31.11.10
profile gre
remote-address 172.31.11.20
}
BIG-IP IPSEC
We need to configure two IPSEC peers and two traffic selectors for Sericata and one peer and selector for Arkime.
net ipsec ike-peer arkime {
my-id-value 172.31.15.10
peers-id-value 172.31.15.198
phase1-auth-method pre-shared-key
phase1-encrypt-algorithm aes256
phase1-hash-algorithm sha256
preshared-key-encrypted $M$36$sL8Pgqt7panBy0rKOskzfQ==
prf sha256
remote-address 172.31.15.198
traffic-selector { /Common/arkime-ts }
version { v2 }
}
net ipsec ike-peer from-suricata {
my-id-value 172.31.12.10
peers-id-value 172.31.12.20
phase1-auth-method pre-shared-key
phase1-encrypt-algorithm aes256
phase1-hash-algorithm sha256
preshared-key-encrypted $M$NW$XupRAfQhku/WuQx/sRTAaw==
prf sha256
remote-address 172.31.12.20
traffic-selector { /Common/ts-from-suricata }
version { v2 }
}
net ipsec ike-peer to-suricata {
my-id-value 172.31.11.10
peers-id-value 172.31.11.20
phase1-auth-method pre-shared-key
phase1-encrypt-algorithm aes256
phase1-hash-algorithm sha256
preshared-key-encrypted $M$uk$gE48199ic7g0k3OwbrOK0w==
prf sha256
remote-address 172.31.11.20
traffic-selector { /Common/ts-to-suricata }
version { v2 }
}
net ipsec ipsec-policy ipsec-policy-from-suricata {
ike-phase2-auth-algorithm sha256
ike-phase2-encrypt-algorithm aes256
mode tunnel
tunnel-local-address 172.31.12.10
tunnel-remote-address 172.31.12.20
}
net ipsec traffic-selector arkime-ts {
destination-address 172.31.15.0/24
ip-protocol 47
ipsec-policy ipsec-traffic-policy-arkime
order 1
source-address 172.31.15.0/24
}
net ipsec traffic-selector ts-from-suricata {
destination-address 172.31.12.20/32
ipsec-policy ipsec-policy-from-suricata
order 3
source-address 172.31.12.10/32
}
net ipsec traffic-selector ts-to-suricata {
destination-address 172.31.11.20/32
ipsec-policy ipsec-policy-to-suricata
order 2
source-address 172.31.11.10/32
}
Moving Arkime into GRE
By default, your tap will be on a VLAN. You will need to update BIG-IP tunnels to move that mac address into the GRETAP. This mac address is pulled from the GRETAP interface on Linux and is also used in the SSLO wizard (which will show the VLAN interface in the GUI).
admin@(ip-172-31-100-10)(cfg-sync Standalone)(LICENSE EXPIRES IN 2 DAYS:Active)(/Common)(tmos.net.fdb)# list
net fdb tunnel arkime-tap {
records {
72:a5:76:d2:a3:ec {
endpoint 172.31.15.198
}
}
}
Pool Members in SSL Orchestrator
If you have worked with SSL Orchestrator, youknow that you cannot select systems that are on tunnels. This means that we need to configure the security services and then edit those objects after we have created them in SSLO, and to do this we need to disable strictness. Here we add a GRE profile to Arkime and replace the pool members for Suricata.
ltm pool arkime-outside {
members {
172.31.15.198:any {
address 172.31.15.198
session monitor-enabled
state up
}
}
monitor gateway_icmp
profiles {
arkime-gre-profile
}
}
ltm pool suricata-ipsec-pool {
members {
100.64.0.20:any {
address 100.64.0.20
session monitor-enabled
state up
}
}
monitor gateway_icmp
}
ltm virtual ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-t-4 {
app-service /Common/ssloS_suricata_ipsec.app/ssloS_suricata_ipsec
creation-time 2023-06-19:13:44:55
description "In-line service (2023-6-19 13:44:47)"
destination 0.0.0.0%0:any
internal
ip-protocol tcp
last-modified-time 2023-06-19:13:45:35
mask any
pool suricata-ipsec-pool
profiles {
ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-service { }
ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-ss-client {
context serverside
}
ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-tcp-lan {
context clientside
}
ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-tcp-wan {
context serverside
}
}
serverssl-use-sni disabled
source 0.0.0.0/0
translate-address disabled
translate-port disabled
vlans-enabled
vs-index 32
}
The Results
Verify BIG-IP is only sending IPSEC traffic. From the BIG-IP perspective interface 1.2 is traffic TO the Suricata instance, 1.3 is traffic FROM the Suricata instance and 1.6 is traffic to the Arkime instance. Please note you need to look at the traffic from the interface and NOT the VLAN as packet captures at the VLAN will be by after TMM processing.
Please see the attached PDF for more info on BIG-IP config and realized network state.
Suricata Log
If we look in the eve.json file, we can see the http traffic
"timestamp": "2023-12-27T23:06:14.600407+0000",
"flow_id": 1273598159638494,
"event_type": "fileinfo",
"src_ip": "10.1.4.11",
"src_port": 443,
"dest_ip": "x.x.x.x",
"dest_port": 55968,
"proto": "TCP",
"community_id": "1:wA/9qp6gL7lAwPGHPYOVUBrRjVU=",
"http": {
"hostname": "3.x.x.x6",
"url": "/stylesheets/style.css",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15",
"http_content_type": "text/css",
"http_refer": "https://3.x.x.x6/",
"http_method": "GET",
"protocol": "HTTP/1.1",
"status": 200,
"length": 185
},
"app_proto": "http",
"fileinfo": {
"filename": "/stylesheets/style.css",
"sid": [],
"gaps": false,
"state": "CLOSED",
"stored": false,
"size": 176,
"tx_id": 3
}
}
We can also see traffic travesing the GRE interface
ubuntu@ip-172-31-11-20:/var/log/suricata$ sudo tcpdump -ni gre2 port 443
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on gre2, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
01:13:31.173786 IP X.X.X.X.55405 > 10.1.4.11.443: Flags [P.], seq 3043973525:3043974270, ack 3641341613, win 18556, options [nop,nop,TS val 3523761960 ecr 3523749045], length 745
01:13:31.175573 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [.], ack 745, win 25620, options [nop,nop,TS val 3523761962 ecr 3523761960], length 0
01:13:31.182178 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [P.], seq 1:990, ack 745, win 25620, options [nop,nop,TS val 3523761969 ecr 3523761960], length 989
01:13:31.182178 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [P.], seq 990:995, ack 745, win 25620, options [nop,nop,TS val 3523761969 ecr 3523761960], length 5
01:13:31.183529 IP X.X.X.X.55405 > 10.1.4.11.443: Flags [.], ack 990, win 19545,
Arkime
In the graph below, we can see that I have captured 100,000+ flows over the last week
We can also see that I have the ability to look at details of the traffic and examine a packet capture form the system.
Proceed with Careful Consideration
This configuration required making many changes to what SSLO deploys and that you disable strictness. The other area of concern is that IPSEC tunnels end up being pinned to a single TMM; reducing capacity. If you are using this approach because you have decided that you cannot trust the network you do not have physical control of it is worth a discussion on should you deploy this security service next to the environment you do not trust, such as a colo that is cloud-adjacent where you can retain physical control of the network.
Simplification
By adjusting the approach and using an IPS solution that can support SSL; such as F5 Advanced Firewall Manager and leveraging request logging (if the apps are constrained to HTTP) to preclude the use of a network tap much of the complexity would be removed.
BIG-IP Flexibility
I have been working with the BIG-IP platform for many years and I continue to find new capabilities in the product. The flexibility the platform allows customers to solve the problems their organizations face in the environment they are choosing to use them.
Updated Feb 02, 2024
Version 2.0
Heath_Parrott
Employee
EmployeeNo CommentsBe the first to comment