Can I set a serverssl profile in an iRule?

An SSL profile has to be assigned to the VIP before you can use any of the SSL iRule commands. If a profile is assigned at the VIP (even a generic one) you can then enable, disable, switch profiles and trigger renegotiation from within an iRule.

If you can't add a profile to the VIP as you mentioned in your post, then no. You can not add a profile from the iRule.

From the description, it sounds like you are already offloading SSL at the LTM but this one app wants you to use SSL on the back-end connection also. In that case, you may be able to add an empty clientssl profile on the VIP so it will accept connecting to the back-end application.

Another question you should ask your vendor is if they support flagging a connection as "isSecure" by having the LTM inject a header in the request stream. This is most frequently done when the back-end app needs to be aware that the request was already secured when it creates responses. (Two prime examples of this are MS Exchange Web Access and SharePoint 2013). If that is the case, then all you really need to do is add a rule to insert a custom header in the request stream and you can offload the SSL as is.