Hello,
As the only thing that change with your monitor is the client authentication certificate, my advice is to focus on the SSL Layer, both on your side and on the server side, but mainly focus on the certificate authentication configuration of the server. If the new certificate is issued by a new AC, it could be important to check if the server is trusting it.
Regarding your test, all of them have some miss :
1\ Telnet can only be used to test plain text http, not https so the reset is expected if the server is expecting ssl.
2\ The http 400 is probably here because the request in not well formed. Anyhow, you do not replicate the bigip monitor because there is no client certificate in your request,
3\ Same as below, but the payload seems fine. Anyhow, you do not replicate the bigip monitor because there is no client certificate in your request,
For curl, the synthax to used a client cert is :
$ curl --cert client.crt --key client.key --cacert ca.crt
Cheers,
Sam