UDP TCP Packet Duplication
Problem this snippet solves:
This iApp provides full configuration of UDP/TCP packet duplication. It is commonly used to duplicate Syslog, SNMP Traps, Netflow, and Sflow data streams to multiple vendor solutions or customers. It also provides fault tolerance capabilities within each duplicated destination. By pointing Network devices, Appliances, and Servers to a VIP distributing network management traffic modifying distribution of streams can be done in one centralized location. UDP packets retain the original source address when sending to the destination locations.
Notes:
- Prior to 11.5 you must add an IPv6 address to any interface to allow for HSL traffic to be sent to the distribution virtual fdf5::1/64 fdf5::2/64 for an HA pair would do it.* TCP traffic does not maintain original source
- Internal F5 Resources can demo this solution within the UDF environment using the blueprint named "Traffic Duplication Demo"
Contributed by: Ken Bocchino
20200807 - Updated to v2.2
How to use this snippet:
- Ryan77777Altocumulus
I took Ken's excellent work and made it work for my particular use case. Sharing here in case it helps somebody else.
I un-iApp-ified it, added route domain support, fixed the pool problem, and it works great on 13.1 for me.-- Create two VIPs
- -- Create datagroup (nf_destinations.dg) and add IPs you want to send netflow/syslog to with string as the IP and port as the value
- -- Create pool (nf_distribute.pool) that has a member of the distribute VIP
- -- Create UDP profile and assign to both VIPs (collector and distributor) assign immediate timeout and enable datagram lb
- -- Create two iRules, and assign to the VIPs accordingly
nf_collector.irule Acquire UDP Netflow packet from collector and distribute when CLIENT_ACCEPTED { Get source IP and break-out into variables scan [IP::client_addr] %d.%d.%d.%d a b c d Insert placeholder in UDP datagram for our source/dest embed UDP::payload replace 0 0 [binary format ssssa256 255 255 255 255 [string repeat "~" 256]] Set HSL distribute pool side-channel set hsl [HSL::open -proto UDP -pool nf_distribute.pool] Iterate over Netflow Destinations (via established datagroup) set id [class startsearch nf_destinations.dg] while { [class anymore nf_destinations.dg $id] } { set destinationelement [class nextelement nf_destinations.dg $id] set destination [lindex $destinationelement 0] set destinationwithpad "$destination[string repeat "~" [expr 256 - [string length $destination]]]" Embed source/dest and send to side-channel UDP::payload replace 0 264 [binary format ssssa256 $a $b $c $d $destinationwithpad] HSL::send $hsl "[UDP::payload]" Uncomment to help debug the collector log local0. "\[NF_COLLECTOR\] :: $destinationwithpad" } Drop packet... no longer need discard }
nf_distribute.irule Acquire UDP Netflow packet from collector and distribute when CLIENT_ACCEPTED { Get embedded source/dest information from UDP payload, assign to variables binary scan [UDP::payload] ssssa256a* a b c d destinationwithpad data Assign destination (and remove padding) set destination [findstr $destinationwithpad "" 0 "~"] Source NAT packet so it comes from original source -- Add %route_domain after $d if you need route domain support snat "$a.$b.$c.$d" Remove embedded source/dest information from UDP payload, leave original data UDP::payload replace 0 [UDP::payload length] $data Send to embedded node (add %route_domain after $destination if you need route domain support and you do not include in the datagroup) node $destination:9996 Uncomment to help debug the distributor log local0. "\[NF_DISTRIBUTOR\] :: $a.$b.$c.$d \-\-\> $destination 9996" }
Boom. Netflow Replicator without paying 20k for a replication VM.
To be determined if this is resource-prohibitive however...
edit: to clean up bad iRule Formatting. LZ - ChuckR_16063Nimbostratus
We plan to upgrade to 13.1, has anyone gotten this to work on 13.1? Or has anyone come up with another way maybe? Thank you,
- Jacob_Creech_33Nimbostratus
No, I have not tried this on v13 the last version I tried this on was v12.1.1. And would not expect F5 to updated this iApp. I worked extensively with F5 to get this iApp to send and ACT back to the device and at the end of a 2 month, POC was told this is not a supported F5 iApp.
- Sergi0Nimbostratus
I tried v2.2 on tmos v13.1, does not work for me. Does anybody use it with v13?
- DamonL_356592Nimbostratus
We have a situation where we want to duplicate TCP packets, but our receivers can only receive UDP. Would it be possible to protocol convert before duplication?
- tdelamatre_1466Nimbostratus
Does this iApp support multiple "profiles"? For example, suppose we have sources A, B, C defined by loopback subnets and destinations X, Y, Z as NetFlow collectors. Can I send A->XY, B->XYZ and C->YZ or similar combinations all using a single VIP?
- Jacob_Creech_33Nimbostratus
kdt0078 the dummy IPV6 actually is used like a loop back to duplicate the packet. I had a sumulare issue I was able to resolve the issue by making my vlans untagged. Dont know why this fixed my issue but it did. Try it
- Jacob_Creech_33Nimbostratus
@ Ken Bocchino,
Any help would be greatly appreciated. I am not able to get the ACT back to the device.
- Jacob_Creech_33Nimbostratus
I finally got this IApp working.... FYI I had to remove tags from the VLANs if I used tags it would not replicate the traffic once I removed them it started working.
Now I have a new issue I am getting the incoming traffic but the ACT is not making it back to the device. I can see that the ACT is being generated and sent from the server but it is not making it back to the device. Any help would be greatly appreciated.
- Jacob_Creech_33Nimbostratus
I am wanting to use this IApp as well and would like to know if there is some documentation on this to explain the setup process a little more.
Currently, I have UDP traffic coming in on 7 different ports to a server. When these packets are successfully written to a database table the service will send an ACK back to the device. So I would like to use this IApp to keep current traffic going to my production environment as well as duplicate this traffic to a QA server but not allowing the ACK to be sent back from the QA server. Will this be possible with this IApp.