What is the recommended TCP Profile settings monitor settings for ISO 8583 traffic in F5 Big IP ?

From what you are explaining is that you sending too much traffic to your servers causing the port to become exhausted with connections and because of that they are failing their health monitor which then brings the pool member/s down. The purpose of a stress test is to find out what the maximum capability is of your servers and it seems to me that you are at that threshold now. I recommend scaling back your stress test until the pool members no longer fail their health monitor. Once you get to a point where everything is working as expected with load then you gradually increase that load until it stops working, which will give you the maximum capability of your pool members. The only way around this is to improve the response time of your servers to the health monitor of the F5. Moving to an alternate port really wouldn't help because the entire purpose of a health monitor is to ensure the pool members are capable of receiving a connection. Most likely the reason you don't experience a noticeable issue when bypassing the F5 is because you have no associated health monitor that would bring the server/s down. You should be able to run a tcpdump during your stress test and filter for only the health monitor and you will see how long the pool members take to respond to the query. Additionally, if you aren't doing anything on the F5 other than distributing load you might consider configuring a fastL4 profile rather than anything else.