Decrypt, Encrypt, Decrypt, Encrypt, Encrypt....

What do you do when you need to decrypt traffic, apply a security service chain policy to it, and then send it to an array of open source security tools and never send an unencrypted packet on the AWS "wire"? And what if these security tools did not support TLS? What if you had to do this transparently? 

If we break it down, we need to:

• Decrypt the incoming traffic so we can apply the security service chain.
• Encrypt the traffic and send it to the security system.
• Decrypt the traffic from the security system
• Encrypt the traffic to the tap
• Encrypt the traffic to the backend   

The Security Stack

In our deployment, we will use two open source security tools. These were picked by the client that I was working with and I will not cover how to configure them in detail. 

Suricata 

If you are not familiar with Suricata, it is an open-source IDS/IPS security tool that supports packet captures, alerting among many other functions.  In our deployment, we will configure Suricata to be an inline IPS with netfilter.

Arkime 

Arkime "is a large-scale, open-source, indexed packet capture and search tool" allowing for graphing, visibility and search of network traffic. 

SSL Orchestrator

If you are reading this, you probably know what SSL Orchestrator is.  If not, please refer to F5 CloudDocs or search DevCentral to learn about all of the amazing things that SSLO can do.  This is where security service logic and traffic steering happens. 

The Topology

We are going to use the same Gateway Load Balancer (GWLB) topology that was used in my article series Increase Security in AWS without Rearchitecting your Applications; with that in mind you can use all the items below and use SSLO in a reverse proxy topology.

The change is we are going to use GRE tunnels between Suricata and BIG-IP and a GRETAP tunnel between BIGIP and Arkime.  We will then wrap the GRE tunnels in IPSEC to ensure that traffic is encrypted. The backend web servers are using SSL with BIG-IP and the clients are using SSL, which is terminated on SSLO. 

Building Overlays in the Overlay of AWS VPC Networking

To bring all of this together, we are going to build tunnels in the VPC.  The type of tunnel we are using depends on what we need it to do.  For a network tap, we need to copy the same IP packet into an L2 frame that has a different destination mac address.  Since we also need to encrypt it, we need to pick a tunnel.  I used a GRETAP interface, but VXLAN should work too.  To deploy the Suricata instance, we need to move untranslated IP packets AND we need to encrypt them.  To accomplish this, I used standard GRE interfaces. 

Arkime Network Interfaces and GRE

Here we create the GRETAP interface and assign it the created IP address from SSLO. We will need the mac address for the SSLO config. 

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.100.228  netmask 255.255.255.0  broadcast 172.31.100.255
        inet6 fe80::dd:5cff:fe7d:fe9b  prefixlen 64  scopeid 0x20<link>
        ether 02:dd:5c:7d:fe:9b  txqueuelen 1000  (Ethernet)

ens6: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 9001
        inet 172.31.15.198  netmask 255.255.255.0  broadcast 172.31.15.255
        inet6 fe80::99:d5ff:fe61:18b3  prefixlen 64  scopeid 0x20<link>
        ether 02:99:d5:61:18:b3  txqueuelen 1000  (Ethernet)

tun0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8963
        inet 198.19.0.10  netmask 255.255.255.0  broadcast 198.19.0.255
        inet6 fe80::70a5:76ff:fed2:a3ec  prefixlen 64  scopeid 0x20<link>
        ether 72:a5:76:d2:a3:ec  txqueuelen 1000  (Ethernet)

sudo ip link add tun0 type gretap local 172.31.15.198 remote 172.31.15.10 ttl 255
sudo ifconfig tun0 198.19.0.10
sudo ifconfig tun0 up

Arkime IPSEC

Now that we have the GRETAP interface, we need to configure one IPSEC process to encrypt the traffic. 

ubuntu@ip-172-31-100-228:~$ sudo cat /etc/ipsec.conf 
config setup
        charondebug="ike 1, knl 1, cfg 0, net 1"
        strictcrlpolicy=no
        uniqueids=yes
        cachecrls=no
conn ipsec-ikev2-vpn
       authby=secret
       left=%defaultroute
       leftid=172.31.15.198
       leftsubnet=0.0.0.0/0
       right=172.31.15.10
       rightsubnet=0.0.0.0/0
       ike=aes256-sha2_256-modp1024!
       esp=aes256-sha2_256!
       keyingtries=0
       ikelifetime=1h
       lifetime=8h
       dpddelay=30
       dpdtimeout=120
       dpdaction=restart
       auto=start
ubuntu@ip-172-31-100-228:~$ 

Suricata GRE

For Suricata, we are routing IP packets and a standard GRE tunnel will work. 

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.11.20  netmask 255.255.255.0  broadcast 172.31.11.255
        inet6 fe80::80:36ff:fe34:3325  prefixlen 64  scopeid 0x20<link>
        ether 02:80:36:34:33:25  txqueuelen 1000  (Ethernet)

ens6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.31.12.20  netmask 255.255.255.0  broadcast 172.31.12.255
        inet6 fe80::d6:aff:fee3:c25f  prefixlen 64  scopeid 0x20<link>
        ether 02:d6:0a:e3:c2:5f  txqueuelen 1000  (Ethernet)

gre1: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 8977
        inet 100.64.0.20  netmask 255.255.255.0  destination 100.64.0.20
        inet6 fe80::ac1f:b14  prefixlen 64  scopeid 0x20<link>

gre2: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 8977
        inet 100.64.1.20  netmask 255.255.255.0  destination 100.64.1.20
        inet6 fe80::ac1f:c14  prefixlen 64  scopeid 0x20<link>

#!/bin/bash
sudo ip tunnel add gre1 mode gre local 172.31.11.20 remote 172.31.11.10 
sudo ip tunnel add gre2 mode gre local 172.31.12.20 remote 172.31.12.10 
sudo ifconfig gre2 100.64.1.20 netmask 255.255.255.0 up
sudo ifconfig gre1 100.64.0.20 netmask 255.255.255.0 up
sudo ip route add 10.0.0.0/8 via  100.64.1.20
sudo ip route add 0.0.0.0/0 via 100.64.0.10
sudo iptables -I FORWARD -i gre1 -o gre2 -j NFQUEUE
sudo iptables -I FORWARD -i gre2 -o gre1 -j NFQUEUE
sudo suricata -c /etc/suricata/suricata.yaml -q 0 &

SURICATA IPSEC

For the Suricata instance, we need two IPSEC peers to create our network topology. 

ubuntu@ip-172-31-11-20:~$ sudo cat /etc/ipsec.conf 
config setup
        charondebug="ike 1, knl 1, cfg 0, net 1"
        strictcrlpolicy=no
        uniqueids=yes
        cachecrls=no
conn ipsec-ikev2-vpn
       authby=secret
       left=%defaultroute
       leftid=172.31.11.20
       leftsubnet=0.0.0.0/0
       right=172.31.11.10
       rightsubnet=0.0.0.0/0
       ike=aes256-sha2_256-modp1024!
       esp=aes256-sha2_256!
       keyingtries=0
       ikelifetime=1h
       lifetime=8h
       dpddelay=30
       dpdtimeout=120
       dpdaction=restart
       auto=start
conn ipsec-ikev2-vpn-2
       authby=secret
       left=%defaultroute
       leftid=172.31.12.20
       leftsubnet=0.0.0.0/0
       right=172.31.12.10
       rightsubnet=0.0.0.0/0
       ike=aes256-sha2_256-modp1024!
       esp=aes256-sha2_256!
       keyingtries=0
       ikelifetime=1h
       lifetime=8h
       dpddelay=30
       dpdtimeout=120
       dpdaction=restart
       auto=start


ubuntu@ip-172-31-11-20:~$ sudo ipsec status
Security Associations (4 up, 0 connecting):
ipsec-ikev2-vpn-2[2015]: ESTABLISHED 6 seconds ago, 172.31.12.20[172.31.12.20]...172.31.12.10[172.31.12.10]
ipsec-ikev2-vpn-2{2015}:  INSTALLED, TUNNEL, reqid 2, ESP SPIs: c0f72a1f_i 0e5d3a98_o
ipsec-ikev2-vpn-2{2015}:   172.31.12.20/32 === 172.31.12.10/32
ipsec-ikev2-vpn[2014]: ESTABLISHED 108 seconds ago, 172.31.11.20[172.31.11.20]...172.31.11.10[172.31.11.10]
ipsec-ikev2-vpn{2014}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cd5853ac_i 0199a2f6_o
ipsec-ikev2-vpn{2014}:   172.31.11.20/32 === 172.31.11.10/32
ipsec-ikev2-vpn[2013]: ESTABLISHED 7 minutes ago, 172.31.11.20[172.31.11.20]...172.31.11.10[172.31.11.10]
ipsec-ikev2-vpn{2013}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c78843ac_i 0df07e34_o
ipsec-ikev2-vpn{2013}:   172.31.11.20/32 === 172.31.11.10/32
ipsec-ikev2-vpn-2[2012]: ESTABLISHED 32 minutes ago, 172.31.12.20[172.31.12.20]...172.31.12.10[172.31.12.10]
ipsec-ikev2-vpn-2{2012}:  INSTALLED, TUNNEL, reqid 2, ESP SPIs: c0addbb2_i 0d653b4d_o
ipsec-ikev2-vpn-2{2012}:   172.31.12.20/32 === 172.31.12.10/32
ubuntu@ip-172-31-11-20:~$ 

BIG-IP GRE Tunnels

We need to configure two tunnels for Suricata and one GRETAP for Arkime. 

admin@(ip-172-31-100-10)(cfg-sync Standalone)(Active)(/Common)(tmos.net.tunnels)# list
net tunnels geneve aws-geneve {
    app-service none
    defaults-from geneve
    description none
}
net tunnels gre arkime-gre-profile {
    app-service none
    defaults-from gre
    description none
    encapsulation transparent-ethernet-bridging
    flooding-type none
    rx-csum disabled
    tx-csum disabled
}
net tunnels tunnel arkime-tap {
    if-index 288
    local-address 172.31.15.10
    profile arkime-gre-profile
    remote-address any
}
net tunnels tunnel geneve-tunnel {
    if-index 304
    local-address 172.31.10.10
    profile aws-geneve
    remote-address any
}
net tunnels tunnel gre-tunnel-from-suricata {
    if-index 320
    local-address 172.31.12.10
    profile gre
    remote-address 172.31.12.20
}
net tunnels tunnel gre-tunnel-to-suricata {
    if-index 336
    local-address 172.31.11.10
    profile gre
    remote-address 172.31.11.20
}

BIG-IP IPSEC

We need to configure two IPSEC peers and two traffic selectors for Sericata and one peer and selector for Arkime. 

net ipsec ike-peer arkime {
    my-id-value 172.31.15.10
    peers-id-value 172.31.15.198
    phase1-auth-method pre-shared-key
    phase1-encrypt-algorithm aes256
    phase1-hash-algorithm sha256
    preshared-key-encrypted $M$36$sL8Pgqt7panBy0rKOskzfQ==
    prf sha256
    remote-address 172.31.15.198
    traffic-selector { /Common/arkime-ts }
    version { v2 }
}
net ipsec ike-peer from-suricata {
    my-id-value 172.31.12.10
    peers-id-value 172.31.12.20
    phase1-auth-method pre-shared-key
    phase1-encrypt-algorithm aes256
    phase1-hash-algorithm sha256
    preshared-key-encrypted $M$NW$XupRAfQhku/WuQx/sRTAaw==
    prf sha256
    remote-address 172.31.12.20
    traffic-selector { /Common/ts-from-suricata }
    version { v2 }
}
net ipsec ike-peer to-suricata {
    my-id-value 172.31.11.10
    peers-id-value 172.31.11.20
    phase1-auth-method pre-shared-key
    phase1-encrypt-algorithm aes256
    phase1-hash-algorithm sha256
    preshared-key-encrypted $M$uk$gE48199ic7g0k3OwbrOK0w==
    prf sha256
    remote-address 172.31.11.20
    traffic-selector { /Common/ts-to-suricata }
    version { v2 }
}
net ipsec ipsec-policy ipsec-policy-from-suricata {
    ike-phase2-auth-algorithm sha256
    ike-phase2-encrypt-algorithm aes256
    mode tunnel
    tunnel-local-address 172.31.12.10
    tunnel-remote-address 172.31.12.20
}

net ipsec traffic-selector arkime-ts {
    destination-address 172.31.15.0/24
    ip-protocol 47
    ipsec-policy ipsec-traffic-policy-arkime
    order 1
    source-address 172.31.15.0/24
}

net ipsec traffic-selector ts-from-suricata {
    destination-address 172.31.12.20/32
    ipsec-policy ipsec-policy-from-suricata
    order 3
    source-address 172.31.12.10/32
}
net ipsec traffic-selector ts-to-suricata {
    destination-address 172.31.11.20/32
    ipsec-policy ipsec-policy-to-suricata
    order 2
    source-address 172.31.11.10/32
}

Moving Arkime into GRE

By default, your tap will be on a VLAN.  You will need to update BIG-IP tunnels to move that mac address into the GRETAP. This mac address is pulled from the GRETAP interface on Linux and is also used in the SSLO wizard (which will show the VLAN interface in the GUI). 

admin@(ip-172-31-100-10)(cfg-sync Standalone)(LICENSE EXPIRES IN 2 DAYS:Active)(/Common)(tmos.net.fdb)# list
net fdb tunnel arkime-tap {
    records {
        72:a5:76:d2:a3:ec {
            endpoint 172.31.15.198
        }
    }
}

 

Pool Members in SSL Orchestrator

If you have worked with SSL Orchestrator, youknow that you cannot select systems that are on  tunnels. This means that we need to configure the security services and then edit those objects after we have created them in SSLO, and to do this we need to disable strictness. Here we add a GRE profile to Arkime and replace the pool members for Suricata. 

ltm pool arkime-outside {
    members {
        172.31.15.198:any {
            address 172.31.15.198
            session monitor-enabled
            state up
        }
    }
    monitor gateway_icmp
    profiles {
        arkime-gre-profile
    }
}
ltm pool suricata-ipsec-pool {
    members {
        100.64.0.20:any {
            address 100.64.0.20
            session monitor-enabled
            state up
        }
    }
    monitor gateway_icmp
}

ltm virtual ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-t-4 {
    app-service /Common/ssloS_suricata_ipsec.app/ssloS_suricata_ipsec
    creation-time 2023-06-19:13:44:55
    description "In-line service (2023-6-19 13:44:47)"
    destination 0.0.0.0%0:any
    internal
    ip-protocol tcp
    last-modified-time 2023-06-19:13:45:35
    mask any
    pool suricata-ipsec-pool
    profiles {
        ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-service { }
        ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-ss-client {
            context serverside
        }
        ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-tcp-lan {
            context clientside
        }
        ssloS_suricate_ipsec.app/ssloS_suricata_ipsec-tcp-wan {
            context serverside
        }
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    translate-address disabled
    translate-port disabled
    vlans-enabled
    vs-index 32
}

The Results

Verify BIG-IP is only sending IPSEC traffic. From the BIG-IP perspective interface 1.2 is traffic TO the Suricata instance, 1.3 is traffic FROM the Suricata instance and 1.6 is traffic to the Arkime instance.  Please note you need to look at the traffic from the interface and NOT the VLAN as packet captures at the VLAN will be by after TMM processing.

Please see the attached PDF for more info on BIG-IP config and realized network state. 

Suricata Log

If we look in the eve.json file, we can see the http traffic

 "timestamp": "2023-12-27T23:06:14.600407+0000",
  "flow_id": 1273598159638494,
  "event_type": "fileinfo",
  "src_ip": "10.1.4.11",
  "src_port": 443,
  "dest_ip": "x.x.x.x",
  "dest_port": 55968,
  "proto": "TCP",
  "community_id": "1:wA/9qp6gL7lAwPGHPYOVUBrRjVU=",
  "http": {
    "hostname": "3.x.x.x6",
    "url": "/stylesheets/style.css",
    "http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15",
    "http_content_type": "text/css",
    "http_refer": "https://3.x.x.x6/",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 185
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "/stylesheets/style.css",
    "sid": [],
    "gaps": false,
    "state": "CLOSED",
    "stored": false,
    "size": 176,
    "tx_id": 3
  }
}

We can also see traffic travesing the GRE interface

ubuntu@ip-172-31-11-20:/var/log/suricata$ sudo tcpdump -ni gre2 port 443
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on gre2, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
01:13:31.173786 IP X.X.X.X.55405 > 10.1.4.11.443: Flags [P.], seq 3043973525:3043974270, ack 3641341613, win 18556, options [nop,nop,TS val 3523761960 ecr 3523749045], length 745
01:13:31.175573 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [.], ack 745, win 25620, options [nop,nop,TS val 3523761962 ecr 3523761960], length 0
01:13:31.182178 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [P.], seq 1:990, ack 745, win 25620, options [nop,nop,TS val 3523761969 ecr 3523761960], length 989
01:13:31.182178 IP 10.1.4.11.443 > X.X.X.X.55405: Flags [P.], seq 990:995, ack 745, win 25620, options [nop,nop,TS val 3523761969 ecr 3523761960], length 5
01:13:31.183529 IP X.X.X.X.55405 > 10.1.4.11.443: Flags [.], ack 990, win 19545, 

Arkime

In the graph below, we can see that I have captured 100,000+ flows over the last week

 

We can also see that I have the ability to look at details of the traffic and examine a packet capture form the system. 

 

Proceed with Careful Consideration

This configuration required making many changes to what SSLO deploys and that you disable strictness.  The other area of concern is that IPSEC tunnels end up being pinned to a single TMM; reducing capacity.  If you are using this approach because you have decided that you cannot trust the network you do not have physical control of it is worth a discussion on should you deploy this security service next to the environment you do not trust, such as a colo that is cloud-adjacent where you can retain physical control of the network. 

Simplification

By adjusting the approach and using an IPS solution that can support SSL; such as F5 Advanced Firewall Manager and leveraging request logging (if the apps are constrained to HTTP) to preclude the use of a network tap much of the complexity would be removed.

BIG-IP Flexibility 

I have been working with the BIG-IP platform for many years and I continue to find new capabilities in the product. The flexibility the platform allows customers to solve the problems their organizations face in the environment they are choosing to use them.