I work for a large regional public transportation company. We are in the beginning process of rolling out phones with a VoIP over the top application on them. This will eliminate the need for radios (radio transmittion towers are now very expensive). I have determined that, for whatever reason, these phones send a DNS request on TCP port 853 and with that the VIP sends an RST to the requesting phone. That in turn resets the connection, the VoIP app beeps, disconnects for 30 sends, and then reconnects.
I have both the phone and app vendor looking at this but thought I would see what it would take to mask this problem from the F5 side. I do realize that all DNS traffic currently hits this VIP, so at this time I am only exploring my options.
I have 3 DNS servers in a pool, that sit behind a VIP 10.0.12.14. The VIP is UDP port 53. In trouble shooting these Sonim cell phones running an Over the Top VoIP application for some reason try and do DoT to the VIP. The VIP sends a RST, the phone drops the cell connection (really it swaps virtual nics, which in the cell world is miliseconds) but with that the application waits 30 seconds and reconnects. This is all over a private cellular network (an APN) so I can pcap ingress cell traffic and have a phone sitting next to me, and when the phone beeps a warning that it is offline, seconds before this RST is sent from the vip to my phone.
With this information I wondered if I could instruct the VIP to just ignore, vs sending a RST to the device.
After posting the initial question I have learned additional information regarding the phone and qualicom chip set. Once this trouble happens I have to pull the battery to start a fresh VoIP session, otherwise every 3 to 5 min. this disconnect happens. (and the TCP over 853 to the VIP as well) So this really isn't and never intended to be a fix, just maybe used to help roll out the phones.
Okay, so then the VIP itself is sending a RST after the client's initial SYN (never completing the client side 3WHS).
You said the VIP is UDP:53, but the client is trying to connect over TCP:853. That'd explain the RST.
Proxying DNS over TLS Queries to Traditional DNS: https://clouddocs.f5.com/training/community/dns/html/class5/module2/module2.html
O.K. I see where you are going with this. I was looking to stop the RST packets, just dropped. You are looking at it from the perspective of terminating TLS on the F5 and passing upd 53 to the pool.
I'll add, as the RST is not specifically coming from the VIP, since that VIP isn't listening on the correct port, you'd likely need something global to control behavior. There are also a few additional options:
You'll need to tweak the packet filter rules to your environment, but this could effectively be used to discard any traffic coming to the BIG-IP that doesn't match a listening port.