Custom Http and Https health check

Your best first start will probably be to watch the monitor traffic between the BIG-IP and server. If you can temporarily turn off SSL at the server and just to HTTP port 80, you won't have to mess with decrypting the SSL. Otherwise you'd need to run ssldump to see the traffic. What you're looking for is to see what the monitor request looks like and what the server's response looks like. You could spend a lot of time troubleshooting a monitor config to find out later that the server wasn't sending what you thought it should.