Backend Server respond with * LibreSSL SSL_read: Connection reset by peer, errno 54

[pullb0x]

Hi
I am new to F5 and LTM and apologize if this is the wrong forum or if I am not providing the appropriate information etc.

I have a custom mobile app I wrote where via custom headers I want to filter via Irule if the traffic is allowed to my backend server. 

I configured a virtual server accepting HTTPS traffic on port 443. I added a SSL profile for my certificates (self signed) to the Client SSL Profile.  I did specify the default serverssl profile for the Server SSL profile.

I created a pool with two nodes ( both pointing to the same server one via Ip the other via FQDN both for HTTPS)

I added the Pool to the virtual server and add my irule to the virtual server.

When I

• working: open in a browser the virtual server IP, I can see that the Irules are parsed (LTM log) and the irules exception is shown in the browser. Due to the nature of the irule it does not forward to the pool but I get the "missing values" response from the irule. 
  
  
• working: I can curl the virtual server and receive the same results as above
  
• no working: If I point my mobile App to the virtual server I can see the irule being parsed "successfully" (below snipped is from the ltm log) but I receive then (after several seconds)  "webpage not available ERR_CONNECTION_RESET"
  Log
  Oct 20 14:43:32 bigip1 info tmm1[16591]: Rule /Common/MobileBot <HTTP_REQUEST>: Valid request
  Oct 20 14:43:32 bigip1 info tmm1[16591]: Rule /Common/MobileBot <HTTP_REQUEST>: Success going to pool now!
• working: If I curl the endpoint directly I get the correct response from the server.
• working: If I open the endpoint in a browser  I am served the correct website.
• not working: If I remove the Irules and point to the virtual server I see the same result as when I use my custom mobile App, after several seconds I get Connection reset by peer, LibreSSL SSL_read: Connection reset by peer.
  
• not working: Curl to the virtual server w/o irules receive the same error as above. 
  

It is not a network/routing issue as I can curl the backend server successfully directly from the F5 terminal.
I have to assume that I have a F5 configuration issue and I would appreciate it if someone could point me in the appropriate direction.

I very much appreciate any help and apologize for the long post.

Thank YOU pullbox

[Samir]

Try to capture the traffic via tcpdump and see if you can see any bloackage 3-Handshake.

if you have time just remove ServerSSL and add serverssl-insecure-compatible. if any issue with server SSL error. Give one try and share result.

[pullb0x]

Samir,

thank you for the feedback.
serverssl-insecure-compatible did not change anything.
Still have to pull the tcpdump.

[Amine_Kadimi]

Try to log the reset cause (https://my.f5.com/manage/s/article/K13223) and see if F5 provides any cause for the reset packet.

Since the iRule logs goes up to the HTTP request, I suspect either a connection problem with the server or a pool member selection failure, the reset logs will tell you

[pullb0x]

Thank you so much for the feedback.
After I enabled it, this is what I am seeing.

-------------------------------------------
TCP/IP Reset Cause
RST Cause: Count
-------------------------------------------
Flow expired (sweeper) 4
No flow found for ACK 1
No local listener 32027
No server selected 1
RST from BIG-IP internal Linux host 187450
TCP RST from remote system 3
TCP retransmit timeout 172
handshake timeout 4

[Amine_Kadimi]

These are only statistical counters, you need to look at /var/log/ltm or tcpdump to find the message related to the reset cause

[pullb0x]

Appreciate all your help. I am a novice in reading pcap files.
Below screenshot of the traffic
ip ending
...209 = F5 
...35 = my endpoint
.6 = mobile device

per the screenshot traffic from .6 (mobile device) to .35 (endpoint) is TCP retransmission.
What do I have to look at to see what is causing this?
Apologize again if this is kind of a stupid question but I am a noob in this. 
Appreciate pointing me in the right direction.
Thank you

[Amine_Kadimi]

This capture does not show any RST packet but instead it shows a weird problem: the TCP connexion is not correctly established between 192.168.83.209 (the self ip doing monitoring?) and 192.168.83.35, you have SYN, SYN-ACK but no ACK which is mandatory for the TCP 3-way handshake, could it be the problem? I can't tell with the few amount of information about your setup. But I can see few things that may be worth troubleshooting: you don't have source address translation enabled in your VS which causes the traffic not getting back from the backend, and it seems that the VS address is the same as your self IP, this later is technically possible but sometimes it may cause problems. 

[pullb0x]

@Amine_Kadimi 

OMG, it appears the source address translation resolved the issue.
Still need to do some more tests to make sure.
THANK YOU so much for your help so far.