big3d timeouts

F5 support indicated the cause was related to bugs ID1046785 and ID1128369. 

The first bug matched exactly what we were experiencing. After we upgraded our integration environment to 16.1.3.4, which has the bug fix, I confirmed we were no longer seeing unexpected big3d timeouts. Our prod environment wasn't scheduled to be upgraded for a few more weeks, so we applied the workaround of increasing Max Synchronous Monitor Requests from the default of 20 to 200 based on F5 recommendation. This seemed to fix our issue as well. We have since upgraded our prod environment to 16.1.3.4, but we decided to leave MSMR at 200 due to the relatively large number of VS in our environment.

The latter bug ID is not public, but F5 support said that it is related to monitoring several LTMs with the same bigip monitor. This can result in massive bursts of probes being sent to a big3d at the same time, which can result in mcpd becoming full and blocking, causing big3d to drop the query. The recommended fix for this is to apply unique bigip monitors on the LTM clusters with slightly incremented intervals, ex: 30sec, 31sec, etc. We decided not to pursue this since it appears that addressing the first bug fixed our big3d timeout issue.