I am still unable to reproduce this one. Are you getting any other error than the "The host you have specified must be a v9.x BIG-IP"? I'm looking at the connection code and it's pretty basic. When you click the "OK" button on the connection dialog, it attempts to call the System::SystemInfo::get_product_information() method. If that causes an error, it should pop up with a dialog box showing the network error that occurred and the calling app then puts the above error. I don't see a code path where no dialog is present with a network error given one occurs.
Also, are you sure you have "a.b.254.55" in the Hostname field and the port value is 443?
Are you able to connect with any other iControl client?
Just grabbing at straws here. Since the iRule Editor just uses the stock iControl Assembly for .Net, it should be going through the same type of connection your browser is. I don't see how the auth could be different.
If this persists, I can get you a debug build of the iRule Editor that you can run to nail this down.
-Joe