High Packet Drop and connection failure

 virtual server

[root@ve11a:Active:Changes Pending] config  tmsh list ltm virtual bar
ltm virtual bar {
    destination 172.28.20.111:25
    ip-protocol tcp
    mask 255.255.255.255
    pool foo
    profiles {
        tcp { }
    }
    source 0.0.0.0/0
    source-address-translation {
        type automap
    }
    vs-index 6
}

 client

[root@centos17 ~] telnet 172.28.20.111 25
Trying 172.28.20.111...
Connected to 172.28.20.111 (172.28.20.111).
Escape character is '^]'.
220 ESMTP
quit
221 
Connection closed by foreign host.

 packet trace

[root@ve11a:Active:Changes Pending] config  tcpdump -nni 0.0 host 172.28.20.111 and port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on 0.0, link-type EN10MB (Ethernet), capture size 96 bytes
23:36:36.489354 IP 172.28.20.17.55215 > 172.28.20.111.25: S 2510838242:2510838242(0) win 5840 
23:36:36.489397 IP 172.28.20.111.25 > 172.28.20.17.55215: S 485573986:485573986(0) ack 2510838243 win 4380 
23:36:36.490487 IP 172.28.20.17.55215 > 172.28.20.111.25: . ack 1 win 5840 
23:36:36.874546 IP 172.28.20.111.25 > 172.28.20.17.55215: P 1:24(23) ack 1 win 4380 
23:36:36.876531 IP 172.28.20.17.55215 > 172.28.20.111.25: . ack 24 win 5840 
23:36:37.921334 IP 172.28.20.17.55215 > 172.28.20.111.25: P 1:7(6) ack 24 win 5840 
23:36:37.921360 IP 172.28.20.111.25 > 172.28.20.17.55215: . ack 7 win 4386 
23:36:38.111502 IP 172.28.20.111.25 > 172.28.20.17.55215: P 24:41(17) ack 7 win 4386 
23:36:38.111515 IP 172.28.20.111.25 > 172.28.20.17.55215: F 41:41(0) ack 7 win 4386 
23:36:38.112569 IP 172.28.20.17.55215 > 172.28.20.111.25: . ack 41 win 5840 
23:36:38.112573 IP 172.28.20.17.55215 > 172.28.20.111.25: F 7:7(0) ack 42 win 5840 
23:36:38.112586 IP 172.28.20.111.25 > 172.28.20.17.55215: . ack 8 win 4386 

Frank,

I think we are at a stalemate here. Yes, retransmissions are normal. Resets are common but not a good thing (most of the time). Nitass is right also. You shouldn't get a reset, of course. I just don't want you get hung up on it. It's just a clue (symptom) to uncovering the issue.

Items:

1) These simple tests may not be enough. It is not the same as your production traffic. Just sending a hello and/or just a quit isn't much. Where's the mail to and from? Smtp doesn't really have that many commands but it's the one's your missing in your tests that could be causing the problem. Perhaps there is an authentication issue (if required)? Perhaps a username isn't being recognized. ??? Maybe an unsupported command is coming across. ??

2) I don't know why telnet causes the resets for you. It is probably due to the buffer and the EOF all being pushed down the pipe. I know EOF is just to let the shell know the input is ended but it all gets put into the buffer and it makes a difference in the trace. I]m wondering if the QUIT is getting there before the 220 comes back. It works itself out but it makes the order of the packets look a bit strange. I have attached two wireshark images (taken from LTM tcpudmps). The one called smtp_without_eof.png and is a simple connect and QUIT without using EOF. The second image is called smtpeof.png and is when I used EOF like you did. Notice how it throws off the decodes in wireshark. I get weird results sometimes when replicating http monitors using telnet. There's no reason to put more cycles int this, at this time.

3) Did you compare your telnet session to a legitimate connection using tcpdump/wireshark? Let's focus on getting the right captures at this point. Just take captures from the front and back and then focus on the resets or long delta times for starters. Let's see if the same type of connections/users causing the resets or if it is load. You can also open a case and they will help you parse through the captures.

4) Did you test to both/all of your smtp servers directly or were you hitting the vip everytime? Are you using persistence? I also assume you are running these tests/scripts from the LTM.

When reply back regarding 3, we can examine it further.

Finally, we found happened. It is because the Pool do not have session persistence set. We have changed the persistence profile to source address and the issues is solved. Thx everyone for help.