How to delete SAML External SP Connector

I am having an issue deleting the External SP Connector that I created for my SAML Authentication (Big-IP as IdP). I created one for testing and I am unable to remove it. I have made sure that no SAML IDP service is associated with it, but when I select it the delete button does not become enabled. Is there something I am missing or something else I need to do to delete it?