I've setup OAuth2 to Azure EntraID following this documentation. It works well but I'm only facing a serious issue.
In the OAuth provider configuration, I've enabled the discovery job to run once per day. This allows the BigIP to fetch any new certificate and/or JWT as provided by the app on EntraID.
The problem is that when the certificate or the JWT change, you have to re-apply the per-session policy in order for the change to take effect.
And on multiple occasion, the access to our critical applications failed because the changes were'nt applied in a timely manner.
Is there a way to automatically apply the changes made by the OAuth discovery job ?
Apply should happen automatically. If it's not happening, there is some kind of problem.
The JWK discovery process is implemented as a "rest worker" that operates inside of TMOS's restjavad processes, along with other control-plane stuff. The name of the worker is "OIDCDiscoverTaskCollectionWorker", so you can use a command like this to view its log files:
grep -i discovertask /var/log/restjavad.0.log
The logs will look something like this:
restjavad.0.log:[I][648][10 Apr 2018 00:20:25 UTC][OIDCDiscoverTaskCollectionWorker] Downloading OpenID metadata for provider /Common/my_google_provider task ID 4c2c547f-788e-453c-bdae-8dec7e6ab15e using OpenID URI https://accounts.google.com/.well-known/openid-configuration and Trusted CA Bundle /Common/ca-bundle.crt
For reference, this is a rough outline of the discovery process:
Interesting... So we know restjavad is at least scheduled and not throwing errors.
By default the logging for that process is sparse and won't log much about success operations. It does log information about Apply at "FINE" logging level. It can be adjusted by editing the file /etc/restjavad.log.conf. The logging configuration line should be added at the end so it’s not overridden by a later setting.
1- SSH to the BIG-IP.
2- Edit /etc/restjavad.log.conf
3- Add the line at the end, for FINE level verbosity:
There are also some other evidence that Apply has been performed:
1- /var/log/apm will have information about updating the "snapshot" version, which is APMd's data-structure of the access policy and its config objects.
2- /var/log/audit will have traces of a command "generation-action increment" and should have some detailed info.
I'm not sure we can get this all resolved with forum posts, but it's good information to understand the issue and symptoms more fully.
So for example this morning at 06:55 the discovery process updated the configuration. (I also run a diff of the config on my f5 every 15 minutes and any change are reported by mail and in a git repository).
In the logs I found :
[I][2402045][23 Aug 2024 06:55:15 UTC][8100/tm/access/oidc/discover OIDCDiscoverTaskCollectionWorker] Downloading OpenID metadata for provider /HES-SO/HES_AGF_AzAD_Provider task ID 3394e5d4-9c7d-4685-808d-738c16e11dc8 using OpenID URI https://login.microsoftonline.com/a372f724-c0b2-4ea0-abfb-0eb8c6f84e40/v2.0/.well-known/openid-configuration and Trusted CA Bundle /Common/ca-bundle.crt
[I][2402048][23 Aug 2024 06:55:17 UTC][8100/tm/access/oidc/discover OIDCDiscoverTaskCollectionWorker] Updating mcp jwt and jwk objects for provider /HES-SO/HES_AGF_AzAD_Provider task ID 3394e5d4-9c7d-4685-808d-738c16e11dc8
[I][2402049][23 Aug 2024 06:55:17 UTC][8100/tm/access/oidc/discover OIDCDiscoverTaskCollectionWorker] Task ID 3394e5d4-9c7d-4685-808d-738c16e11dc8 for provider /HES-SO/HES_AGF_AzAD_Provider has completed one round, we will schedule the next discover after 1440 minutes
We can see that the jwt configuration has been updated.
So, there has been a change in the JWT this week-end on Azure Entra, and I got a lot of logs from restjavad.
I won't post the part where it discovers the new JWTs, delete the old jwt/certs in the config, create the new one, and skip to the SAVE_AND_APPLY phase of OIDCDiscoverTask. (I also stripped the begining of the lines for readability)
All lines start with [F][21133][25 Aug 2024 18:25:27 UTC][8100/tm/access/oidc/discover OIDCDiscoverTaskCollectionWorker]
[18:25:26] install meta data sending mcp request
[18:25:26] intall meta dataa sending mcp request
[18:25:27] OIDC Discover provider name is /HES-SO/HES_AGF_AzAD_Provider and step SAVE_AND_APPLY
[18:25:27] saveAndApply :mcpProviderName /HES-SO/HES_AGF_AzAD_Provider
[18:25:27] applyPolicyForProvider chain >>
[18:25:27] applyPolicyForProvider chain >>
[18:25:27] applyPolicyForProvider total items in chains = 2
[18:25:27] setup doChainQuery queries : chain >>
[18:25:27] do chain query with names [/HES-SO/HES_AGF_AzAD_Provider], remaining step 5
[18:25:27] setup mcp queryAll for name /HES-SO/HES_AGF_AzAD_Provider qi =
[18:25:27] setup doChainQuery queries : chain >>
[18:25:27] do chain query with names [/HES-SO/HES_AGF_AzAD_Provider], remaining step 5
[18:25:27] setup mcp queryAll for name /HES-SO/HES_AGF_AzAD_Provider qi =
[18:25:27] mcpOperation queryInfo=
[18:25:27] results from mcp-multi requests /HES-SO/HES_AGF_M365_JWTP, for queryInfo =
[18:25:27] do chain query with names [/HES-SO/HES_AGF_M365_JWTP], remaining step 4
[18:25:27] setup mcp queryAll for name /HES-SO/HES_AGF_M365_JWTP qi =
[18:25:28] mcpOperation queryInfo=
[18:25:28] results from mcp-multi requests /HES-SO/HES_AGF_OAuth_Srv, for queryInfo =
[18:25:28] do chain query with names [/HES-SO/HES_AGF_OAuth_Srv], remaining step 4
[18:25:28] setup mcp queryAll for name /HES-SO/HES_AGF_OAuth_Srv qi =
[18:25:28] mcpOperation queryInfo=
[18:25:28] results from mcp-multi requests for queryInfo =
[18:25:28] applyPolicyForProvider handling of a chain
[18:25:28] mcpOperation queryInfo=
[18:25:28] results from mcp-multi requests /HES-SO/HES_AGF_M365_act_oauth_client_ag, for queryInfo =
[18:25:28] do chain query with names [/HES-SO/HES_AGF_M365_act_oauth_client_ag], remaining step 3
[18:25:28] setup mcp queryAll for name /HES-SO/HES_AGF_M365_act_oauth_client_ag qi =
[18:25:28] mcpOperation queryInfo=
[18:25:28] results from mcp-multi requests /HES-SO/HES_AGF_M365_act_oauth_client, for queryInfo =
[18:25:28] do chain query with names [/HES-SO/HES_AGF_M365_act_oauth_client], remaining step 2
[18:25:28] setup mcp queryAll for name /HES-SO/HES_AGF_M365_act_oauth_client qi =
[18:25:28] mcpOperation queryInfo=
[18:25:28] policyName = /HES-SO/HES_AGF_M365, resolvedPolicyName = null
[18:25:28] results from mcp-multi requests null, for queryInfo =
[18:25:28] applyPolicyForProvider handling of a chain
[18:25:28] aps is empty.
[18:25:28] OIDC Discover provider name is /HES-SO/HES_AGF_AzAD_Provider and step SLEEP_AND_RUN_AGAIN
[18:25:28] Task ID 3394e5d4-9c7d-4685-808d-738c16e11dc8 for provider /HES-SO/HES_AGF_AzAD_Provider has completed one round, we will schedule the next discover after 1440 minutes
Lines 36-39 looks odd.
and sill no trace of "generation-action increment" in audit log.
Great. From this logging, it seems that the system is trying to fetch and operate on this config data but the expected objects seem to be missing. Specifically it should be chaining together the access policy and access profile. Can you open a support ticket on this one and reference this DC thread? Support will need to check out your config.
The reason this mechanism is complex and a little weird is that APM has different several different access-policy and access-profile "types", including macros and subroutines for both per-session and per-request type policies. These can also be located in folders or partitions. Most of the control-plane processes try to treat these config objects in a generic way, but there are some corners to get stuck in.
Altocumulus
