Did you take a tcpdump on the front and back sides simultaneously to verify what is happening for certain at the LTM?
Are you running 9.4.2 HF2 or earlier. If so you need HF3 asap (prefer 9.4.5) to fix several UDP session bugs and a really nasty one where the LTM never releases UDP sessions and eventually you run out of resources.
Try to a "b conn" with grep constantly as the problem arises and see what the session count looks for UDP sessions. Be sure to log all this.
If not any of the above, open a case as previously noted.
joecampos80@yahoo.com