RST after client hello

I don't see any captures.

But a common issue is SNI. BIG-IP does not use it by default.

Try to create a specific Server side SSL profile defining the appropriate server name and add it to the monitor.

Of course if it works do not forget to use it also in the Virtual server