Forum Discussion

PaulVogt's avatar
PaulVogt
Icon for Altocumulus rankAltocumulus
Feb 21, 2025
Solved

openshift multi cluster CIS HA

I encounter a weird issue configuring a high available CIS 2.19 on Openshift 4.16. The primary cis hangs in a loop, printing:

[WARNING] AutoMonitor value  is not defined or not supported. Defaulting to none

If I switch off the primary and start the secondary, the secondary works as should and creates the objects on the F5 big ip ve. For the routes defined on the secondary cluster.

Attached are the deployment and configmap yamls. 

 

I could not find anything about the AutoMonitor, so I have no idea what this is.

If I configure the primary cluster as a standalone, multi cluster works fine.

 

  • I think I found the reason. The check probe of bigip-ctlr was configured as an Openshift route managed by cis itself. So when starting the primary CIS it had to configure this first on the F5 to get the health check to work. While the secondary did not had this route and was still active, so blocking the creation of this route.

    Creating a static route for the health check, using the default ingress controller on port 80 solved the issue

  • The AutoMonitor warning disappeared after adding the definition to the extended config map

    baseRouteSpec:

        tlsCipher:

           tlsVersion: 1.2

           ciphers: DEFAULT

           cipherGroup: /Common/f5-default

        autoMonitor: none

     

    NOTE: value is case sensitive. Other options are readiness-probe and service-endpoint

     

    still messages are looping in the log. The logs are send to Splunk which will fill up with a log rate like this.

    I guess if have to check how to stop this log loop.

     

    2025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02

    9732025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....

    9742025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...

    9752025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...

    9762025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02

    9772025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02

    9782025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....

    9792025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...

    9802025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...

    9812025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02

    9822025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02

    9832025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....

    9842025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...

    9852025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...

    9862025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02

     

  • I see this log message has disappeared in the master repository. So it will probably be fixed in 2.19.2. For now I run with log level WARNING to stop the log flooding.

  • The logging is not the only issue. 2.19.1 also doe snot create F5 objects in HA cis active-active mode on the primary CIS. The secondary CIs works fine. Switched the primary back to 2.18.1 for now.

  • Strange that you have no "probeInterval:" or "retryInterval: " for the multi cluster. Other than that where you got the " autoMonitor: none" option as from openshift-4-9/next-gen-routes/single-vip/ocp-route/route-tea.yaml at main · mdditt2000/openshift-4-9 · GitHub of Mark_Dittmer I don't see this for next gen routes as annotations are used under the route object ? 

     

    Ratio and Active-Active mode in Multi-Cluster

     

    That is what came to my mind and as I already mentioned Mark's videos and github are great sources of info.

    • PaulVogt's avatar
      PaulVogt
      Icon for Altocumulus rankAltocumulus

      I got the possible values for autoMonitor from the source code, https://github.com/F5Networks/k8s-bigip-ctlr. Setting it to none in the extended configmap made the warning disappear. CIS 2.19.1 keeps looping with [INFO] Starting Namespace Informer for cluster openshift-engineering-02, flooding the log.

      The secondary runs fine. I can try to make the secondary the primary and see what happens then.

       

  • I think I found the reason. The check probe of bigip-ctlr was configured as an Openshift route managed by cis itself. So when starting the primary CIS it had to configure this first on the F5 to get the health check to work. While the secondary did not had this route and was still active, so blocking the creation of this route.

    Creating a static route for the health check, using the default ingress controller on port 80 solved the issue