How do I determine SP origin on a BigIP IdP

Background:

We have a BigIP 14.1 environment, and we've set up an idp to respond to https://idp.domain.com, where only the IdP Entity ID differs, such as https://idp.domain.com/service1, https://idp.domain.com/service2 etc.

We also act as an SP for an external IdP, so the user has two AAA to choose from.

Some of the services are restricted to use only one of these, and we'd like to skip over the form where the user choose AAA to use for these. Together with an F5 consultant, we solved this by using an iRule listening to the event ACCESS_POLICY_AGENT_EVENT. In it, we scan for the Referer header in the session.server.initial_req_hdrs variable.

We then use a switch -glob statement to differentiate between the various origins, setting session.custom.ga_result to 0, 1 or 2 accordingly, meaning choice, local AD or external IdP.

This was the foundation left to us by the consultant, and it has since been refined by us.

Question:

The above solution works for the most part, but as we bind more SPs to the solution, we find that not everybody has the courtesy to provide a Referer header. Their solutions may not even make it possible. It has really been nagging me.

I have tried two things:

1) Receiving the session.bigip_idp_sp_info variable that contains the name of the SP connector. Except it's not set at the point of ACCESS_POLICY_AGENT_EVENT, so it's empty

2) I tried using the ACCESS_SAML_ASSERTION and extract the saml2:Issuer-attribute from the XML, but this event is triggered after ACCESS_POLICY_AGENT_EVENT, so I can't use that either.

I'm not a SAML expert, and I have to no avail asked several of our SPs for advice as I am certain this has to be solved in the SAML scope of things rather than with guesswork within an elaborate iRule.

How do sane people solve this?