BTW, this BIG-IP runs version BIG-IP 16.1.4.2 Build 0.0.3 Point Release 2
EmployeeThanks for the clear explainer and data.
This RDP mechanism has to create a cryptographic signature based on the vip's private RSA key and place it into the .RDP file that is transmitted to the client immediately after a user clicks that link. It doesn't support any other types of private keys, and will produce the error you've encountered if the SSL profile on the vip does not have an RSA key.
At a quick glance, BIG-IP itself seems to generically support these types of private keys: RSA, SM2, ECDSA, and DSA.
This stack overflow thread seems to indicate that EC certs must use ECDSA keys and are disallowed from using RSA keys:
https://stackoverflow.com/questions/35155239/can-ecdsa-certificates-have-rsa-signature
So I guess the answer is that this RDP mechanism in BIG-IP does not yet support ECDSA certificates. Please feel free to open a support ticket to request this support, and mention this DC thread so the support person has the background information.
