cancel
Showing results for 
Search instead for 
Did you mean: 
Ken_Bocchino_49
Historic F5 Account

Problem this snippet solves:

This iApp provides full configuration of UDP/TCP packet duplication. It is commonly used to duplicate Syslog, SNMP Traps, Netflow, and Sflow data streams to multiple vendor solutions or customers. It also provides fault tolerance capabilities within each duplicated destination. By pointing Network devices, Appliances, and Servers to a VIP distributing network management traffic modifying distribution of streams can be done in one centralized location. UDP packets retain the original source address when sending to the destination locations.

Notes:

  • Prior to 11.5 you must add an IPv6 address to any interface to allow for HSL traffic to be sent to the distribution virtual fdf5::1/64 fdf5::2/64 for an HA pair would do it.* TCP traffic does not maintain original source
  • Internal F5 Resources can demo this solution within the UDF environment using the blueprint named "Traffic Duplication Demo"

Contributed by: Ken Bocchino


20200807 - Updated to v2.2

How to use this snippet:


Comments
Patricia_Gonzal
Nimbostratus
Nimbostratus
I am very interested in this! Has anyone tested?
Ken_Bocchino_49
Historic F5 Account
Yes, this is in may working environments (and just updated to version 2.0) let me know if you have any issues using it.
bigipjr28_13978
Nimbostratus
Nimbostratus
How would you configure syslog for instance that require UDP duplication. What would be the desintation and the primary IPs..I only see the primary IP text box Any help is great thanks
bigipjr28_13978
Nimbostratus
Nimbostratus
How would you configure syslog for instance that require UDP duplication. What would be the desintation and the primary IPs..I only see the primary IP text box Any help is great thanks
Runo_Førrisdah1
Nimbostratus
Nimbostratus
Hi Ken, Thanks for an interesting solution. I've had som issues with this on 11.6 duplication UDP syslogs. It starts off just fine and can work great for X time. Then something causes it to leak packages. It only leaks packages related to the duplication VIPs. Have you seen this kind of behavior?
ep
Nimbostratus
Nimbostratus
Ken, I'm trying to use v2.2 of this iApp to duplicate snmptraps to multiple trap receivers. For some traps, it is working great. For others, though, they aren't getting duplicated. I have a packet capture showing that two nearly identical traps behave differently on the F5. What is the best way to troubleshoot the iApp? Thanks, Brian
ep
Nimbostratus
Nimbostratus

Looks like my issue disappeared. It is working quite well at the moment. Thanks! ep

 

Sanjeev_N_G_183
Nimbostratus
Nimbostratus

Hi Ken,

 

I have installed iApp on 11.4.1 HF8 but i am getting below error when trying to implement.

 

Error parsing template:can't eval proc: "script::run" can't find package iapp 1.1.0 while executing "package require iapp 1.1.0" (procedure "script::run" line 2) invoked from within "script::run" line:1

 

Scott_Crawford_
Nimbostratus
Nimbostratus

Anyone using this with route domains? I'm playing with it (in route domains) and not having luck. Unsure if it's the RD or something else.

 

Sanjeev_N_G_183
Nimbostratus
Nimbostratus

Hi Ken,

 

I have installed Version 2.2 on 11.6.0 HF6, i am not able to get this working. When i grep for log i see below error in log.

 

warning mcpd[5663]: 01071859:4: Warning generated : /Common/Splunk_duplication.app/ir_Splunk_duplication_udp_spray:17: warning: [use curly braces to avoid double substitution][[string length $destination]] warning mcpd[5663]: 01071859:4: Warning generated : /Common/Splunk_duplication.app/ir_Splunk_duplication_distribute:14: warning: [use curly braces to avoid double substitution][![ catch { pool [lindex $nodeandport 0] } ]]

 

Add i do not see any traffic or any activity happening.Please let me know how to solve the issue.

 

Sp33dy_156082
Nimbostratus
Nimbostratus

Hi,

 

I'm also using this duplicator and it works fine. The only thing is that it uses source port 0 for traffic sent to the 2 destinations. According to RFC Firewalls don't allow this traffic with source port 0. Do you guys have the same issue or am i doing something wrong?

 

Please let me know.

 

Thx.

 

Mauz
Nimbostratus
Nimbostratus

Does this IApp works if the clone is in a different subnet from the LTM's subnet

 

Mauz
Nimbostratus
Nimbostratus

Does this IApp works if the clone pool member is in a different subnet from the LTM's subnet?

 

Sp33dy_156082
Nimbostratus
Nimbostratus

I have it working to two different IP's in different subnets. Just make sure you're routing is ok.

 

edolton_204031
Nimbostratus
Nimbostratus

@Sp33dy

 

I see the same thing with the source port being 0. Its an issue for me since they want the backend servers to ACK the traffic. Let me know if you found a solution

 

Sp33dy_156082
Nimbostratus
Nimbostratus

@edolton

 

I fixed this by stripping of the restriction from the IAPP and change the sourceport to preserve on the Virtual Server (both the virtual servers created by the IAPP). Now everything works fine!

 

Regards,

 

Maarten

 

edolton_204031
Nimbostratus
Nimbostratus

Thanks! i made the same change. It doesn't seem to keep the original source but increments a non-zero port 9011 then 9012 then 9013 etc. I'll see if this works for me. Thanks!

 

kdt0078
Nimbostratus
Nimbostratus

Having an issue with this iAPP on 11.6.1. It looks like it creates two virtual servers xxx_distribute and xxx_udp. Looks like the destination address on the xxx_distribute is a dummy ipv6 address and the xxx_udp virtual server is not forwarding traffic.

 

Has this been plug-n-play for those of you who have it working?

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

I am wanting to use this IApp as well and would like to know if there is some documentation on this to explain the setup process a little more.

 

Currently, I have UDP traffic coming in on 7 different ports to a server. When these packets are successfully written to a database table the service will send an ACK back to the device. So I would like to use this IApp to keep current traffic going to my production environment as well as duplicate this traffic to a QA server but not allowing the ACK to be sent back from the QA server. Will this be possible with this IApp.

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

I finally got this IApp working.... FYI I had to remove tags from the VLANs if I used tags it would not replicate the traffic once I removed them it started working.

 

Now I have a new issue I am getting the incoming traffic but the ACT is not making it back to the device. I can see that the ACT is being generated and sent from the server but it is not making it back to the device. Any help would be greatly appreciated.

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

@ Ken Bocchino,

 

Any help would be greatly appreciated. I am not able to get the ACT back to the device.

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

kdt0078 the dummy IPV6 actually is used like a loop back to duplicate the packet. I had a sumulare issue I was able to resolve the issue by making my vlans untagged. Dont know why this fixed my issue but it did. Try it

 

tdelamatre_1466
Nimbostratus
Nimbostratus

Does this iApp support multiple "profiles"? For example, suppose we have sources A, B, C defined by loopback subnets and destinations X, Y, Z as NetFlow collectors. Can I send A->XY, B->XYZ and C->YZ or similar combinations all using a single VIP?

 

DamonL_356592
Nimbostratus
Nimbostratus

We have a situation where we want to duplicate TCP packets, but our receivers can only receive UDP. Would it be possible to protocol convert before duplication?

 

Sergi0
Nimbostratus
Nimbostratus

I tried v2.2 on tmos v13.1, does not work for me. Does anybody use it with v13?

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

No, I have not tried this on v13 the last version I tried this on was v12.1.1. And would not expect F5 to updated this iApp. I worked extensively with F5 to get this iApp to send and ACT back to the device and at the end of a 2 month, POC was told this is not a supported F5 iApp.

 

ChuckR_16063
Nimbostratus
Nimbostratus

We plan to upgrade to 13.1, has anyone gotten this to work on 13.1? Or has anyone come up with another way maybe? Thank you,

 

Ryan77777
Altocumulus
Altocumulus

I took Ken's excellent work and made it work for my particular use case. Sharing here in case it helps somebody else.
I un-iApp-ified it, added route domain support, fixed the pool problem, and it works great on 13.1 for me.

-- Create two VIPs

0691T000006Aqs2QAC.png

  • -- Create datagroup (nf_destinations.dg) and add IPs you want to send netflow/syslog to with string as the IP and port as the value
  • -- Create pool (nf_distribute.pool) that has a member of the distribute VIP
  • -- Create UDP profile and assign to both VIPs (collector and distributor) assign immediate timeout and enable datagram lb
  • -- Create two iRules, and assign to the VIPs accordingly
nf_collector.irule 
 Acquire UDP Netflow packet from collector and distribute
when CLIENT_ACCEPTED {
     Get source IP and break-out into variables
    scan [IP::client_addr] %d.%d.%d.%d a b c d
     Insert placeholder in UDP datagram for our source/dest embed
    UDP::payload replace 0 0 [binary format ssssa256 255 255 255 255 [string repeat "~" 256]]
     Set HSL distribute pool side-channel
    set hsl [HSL::open -proto UDP -pool nf_distribute.pool]
     Iterate over Netflow Destinations (via established datagroup)
    set id [class startsearch nf_destinations.dg]
    while { [class anymore nf_destinations.dg $id] } {
        set destinationelement [class nextelement nf_destinations.dg $id]
        set destination [lindex $destinationelement 0]
        set destinationwithpad "$destination[string repeat "~" [expr 256 - [string length $destination]]]"
         Embed source/dest and send to side-channel
        UDP::payload replace 0 264 [binary format ssssa256 $a $b $c $d $destinationwithpad]
        HSL::send $hsl "[UDP::payload]"
         Uncomment to help debug the collector
         log local0. "\[NF_COLLECTOR\] :: $destinationwithpad"
    }
     Drop packet... no longer need
    discard
}

nf_distribute.irule 
 Acquire UDP Netflow packet from collector and distribute
when CLIENT_ACCEPTED {
     Get embedded source/dest information from UDP payload, assign to variables
    binary scan [UDP::payload] ssssa256a* a b c d destinationwithpad data
     Assign destination (and remove padding)
    set destination [findstr $destinationwithpad "" 0 "~"]
     Source NAT packet so it comes from original source -- Add %route_domain after $d if you need route domain support
    snat "$a.$b.$c.$d"
     Remove embedded source/dest information from UDP payload, leave original data
    UDP::payload replace 0 [UDP::payload length] $data
     Send to embedded node (add %route_domain after $destination if you need route domain support and you do not include in the datagroup)
    node $destination:9996
     Uncomment to help debug the distributor
     log local0. "\[NF_DISTRIBUTOR\] :: $a.$b.$c.$d \-\-\> $destination 9996"
}

Boom. Netflow Replicator without paying 20k for a replication VM.
To be determined if this is resource-prohibitive however...


edit: to clean up bad iRule Formatting. LZ

Paulius_341707
Nimbostratus
Nimbostratus

Does anyone know if this method officially supported by F5?

 

Jacob_Creech_33
Nimbostratus
Nimbostratus

No this iApp is not officially supported.

 

tienbm_356668
Nimbostratus
Nimbostratus

Hi, I tried this for duplicate mysql traffic but I can't login to DBs after do it! Can you help me?

 

Thanks.

 

awilhelm
F5 Employee
F5 Employee

This iApp should not be used. It uses a virtual server as a pool member, which is not supported.

 

A better way to approach the same goal would be to use a single virtual server and distribute to multiple HSL destinations - more control over how packets are distributed within those distributions could be achieved using a Log Publisher and Remote HSL Log Destination rather than a pool.

H__Valbuena
F5 Employee
F5 Employee

If you ever need an iApp based on this template to duplicate traffic to virtual servers instead of to pool or nodes. Below is the procedure I used.

 

 

 

Create an iApp with dummy values for the “Destination Server Questions” section during creation

0691T000008GeeAQAS.png

 

Uncheck the strict updates on the iApp to make manual changes

0691T000008GeeZQAS.png

 

Replace the dummy string records created by the iApp in the data gorup with the virtual server names that you want the traffic to be duplicated to. The label values will not be used 

0691T000008GeeeQAC.png

 

 

Data group string records should look like this

0691T000008GeejQAC.png

 

Change this iRule (name should be ending with “_distribute” ) with the one below

0691T000008GeeGQAS.png 

0691T000008GeeyQAC.png

 

 

when CLIENT_ACCEPTED {

               binary scan [UDP::payload] ssssa256a* a b c d destinationwithpad data

               #log "sending from $a.$b.$c.$d to pool: $destinationwithpad"

               UDP::payload replace 0 [UDP::payload length] $data

               snat "$a.$b.$c.$d"

 

               #get just dest without pad

               set destination [findstr $destinationwithpad "" 0 "~"]

               #log $destination

 

                set nodeandport [split $destination ":"]

               virtual [lindex $nodeandport 0]

}

 

Duplicated traffic would be sent to the virtual server name listed on the data group

 

federicohuman
Nimbostratus
Nimbostratus

Hi

 

I am using the tool, however it creates an IPv6 address with the IP I am providing in: What IP address do you want to use for this virtual server?

 

Is it possible to use it on IPv4?

 

Thanks

steve2
Nimbostratus
Nimbostratus

@awilhelm - Can you please elaborate on this HSL method for UDP packet duplication and distributing?  We are looking at upgrading from 14x to 16x, though I have not tested in the lab yet, I'm suspect of this iAPP continueing to work.  Plus, iAPPs are on the way out being replaced with FAST and AS3, as I understand it. 

awilhelm
F5 Employee
F5 Employee

@steve2 sure. You can use the UDP_DATA event and UDP::payload command in iRules to retrieve the UDP payload, then use HSL (or sideband commands) to send it. HSL commands in iRules do not do any formatting.

Jim_Araujo
Nimbostratus
Nimbostratus

Was the snippet removed? 

JRahm
Community Manager
Community Manager

it looks like it was removed by the author. I have v1 in a zipfile, hit me at j.rahm@f5.com if you want a copy to study/modify. @awilhelm's advice stands, shouldn't use it as is.

Version history
Last update:
‎11-Mar-2015 15:04
Updated by:
Contributors