Without knowing what you have filled in the text fields is difficult to figure out what errors describe this message.
Most likely, you are using some not allowed character.
Anyway, take into account that the self-certificate could be created externaly (or in the F5 bash interface).
And import it into the F5 after that.
Import Cert --> https://support.f5.com/csp/article/K14620#3
Import Key --> https://support.f5.com/csp/article/K14620#4