Forum Discussion
Senthil_Kumar_N
Nimbostratus
NimbostratusMultiple request sent after each 300 seconds
We have a architecture like, cluster enabled weblogic server with F5 load balancer i.e
F5 between our application server and the database server. And we have this setup for more than a year.
Last month onwards, we are facing issue like, when a request is made from application, it send the request to database through F5 and the database takes more time to execute. In the mean time, after 300 seconds, the F5 sending the request again to the database. Like this, its happening on every 300 seconds, and sending request. Since multiple calls made, the database is ended up with duplicate data insertion. Not sure, why all sudden, its happening suddenly. Its been working fine for all these days.
When we call the url has server name and port, i.e bypassing F5, its works fine. Issue is happening only when the request is made via F5.
- nitass
Employee
just curious if it is retransmission, shouldn't sequence number in packet be same? will database server process the packet twice? 
Senthil_Kumar_NI am not sure of what is mean " shouldn't sequence number in packet be same".. sorry, could you please explain little bit?
Yes, each request is treated as new and doing the calculation in database each time. This results in duplication of record.- What_Lies_Bene1
Cirrostratus
Is there a reason the database is taking so long, has this changed? Has the database software etc. been updated/upgraded?
Regardless, it sounds like the best bet at this point, if it's urgent, would be to create a custom TCP Protocol Profile with a higher idle timeout and assign it to the database Virtual Server. - nitassI am not sure of what is mean " shouldn't sequence number in packet be same".. sorry, could you please explain little bit? i think retransmission packet will use same tcp sequence number and just curious if database server still process packet repeatedly.
e.g.[root@ve10:Active] config b virtual bar list virtual bar { snat automap pool foo destination 172.28.19.79:23 ip protocol 6 } [root@ve10:Active] config b pool foo list pool foo { members 200.200.200.101:23 {} } [root@ve10:Active] config b self 200.200.200.10 list self 200.200.200.10 { netmask 255.255.255.0 vlan internal allow default } retransmission packet with same sequence number i.e. 3199438433 [root@ve10:Active] config tcpdump -nni 0.0 -S host 200.200.200.101 and port 23 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on 0.0, link-type EN10MB (Ethernet), capture size 108 bytes 21:27:37.975148 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:27:39.175144 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:27:41.375453 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:27:45.575459 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:27:53.775557 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:28:09.975442 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:28:42.175367 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 21:29:46.175193 IP 200.200.200.10.55822 > 200.200.200.101.23: P 3199438433:3199438434(1) ack 2240270435 win 4622 - Senthil_Kumar_NThere is no change in database like upgrade/update. Usually the procedure calls take more than 8mins to complete execution. Its been working like this for more than a year.
Our network team suggesting not to create ay custome profile.
We wonder that the application was working all this time, but all of sudden it started to behave like this. No servers wer changed/upgraded/updated.
Look like the database consider each request separate and process individually and results in duplicate insertion.
Could you guide on this? - nitassLook like the database consider each request separate and process individually and results in duplicate insertion. do you have tcpdump showing it is retransmission packet indeed?
tcpdump -nni 0.0:nnn -s0 -w /var/tmp/output.pcap host x.x.x.x or host y.y.y.y and port zzzz
x.x.x.x is virtual server ip
y.y.y.y is pool member ip
zzzz is virtual server port/pool member port - Senthil_Kumar_NNo, i dont have now.
Could you tell me how to get the tcpdump. I will try to get it. - nitassCould you tell me how to get the tcpdump. I will try to get it.you can run tcpdump command above on bigip while problem is happening.
- Senthil_Kumar_NSorry, i could not get the tcpdump now.
Could you give me the possible solutions? - What_Lies_Bene1Senthil, I think nitass's approach is the best to take. You need to fully understand where the problem lies as it seems unlikely this is a fault at the TCP/IP layer, it's more likely to be an application issue. As I asked for the database servers, can you also respond regarding the application servers: Has the software/hardware etc. been updated/upgraded? My suspicion is that it's not the F5 retransmission that's the issue, it's the application servers re-submitting the request (so a timeout there, not on the F5).
You could of course increase the idle timeout in the server side TCP Profile Idle Time to a time that's longer than the database execution time but this really is just papering over the cracks.
