Connection table entry removed after idle timer reaches tcp keep alive value

How do i find out from the pcap if the server is sending it first, any specific filter i can apply on the wireshark to see it

you may run tcpdump with FIN and RST filter but i think it is good to capture all and filter in wireshark. anyway, there are some filter example here in case if you are interested.

A tcpdump Tutorial and Primer

in the frames before the LTM VIP sends a FIN, i do see the server sending a RST packet, could this be the cause??

may be. can you find out why serer sends reset?