Forum Discussion
Altocumulusopenshift multi cluster CIS HA
I encounter a weird issue configuring a high available CIS 2.19 on Openshift 4.16. The primary cis hangs in a loop, printing:
[WARNING] AutoMonitor value is not defined or not supported. Defaulting to none
If I switch off the primary and start the secondary, the secondary works as should and creates the objects on the F5 big ip ve. For the routes defined on the secondary cluster.
Attached are the deployment and configmap yamls.
I could not find anything about the AutoMonitor, so I have no idea what this is.
If I configure the primary cluster as a standalone, multi cluster works fine.
I think I found the reason. The check probe of bigip-ctlr was configured as an Openshift route managed by cis itself. So when starting the primary CIS it had to configure this first on the F5 to get the health check to work. While the secondary did not had this route and was still active, so blocking the creation of this route.
Creating a static route for the health check, using the default ingress controller on port 80 solved the issue
- Nikoolayy1
MVP
I admit I am not the biggest specialist on CIS multi cluster with openshift or kubernetes but I could suggest following (323) Mark Dittmer - YouTube (see his videos and example config) and if needed to write an issue under his git examples mdditt2000 (mdditt2000) / Repositories · GitHub . You can also enable a debug to see if there is more info Collect Support data for CIS issues / CIS Troubleshooting — F5 CIS Operations Guide documentation
PaulVogtThe AutoMonitor warning disappeared after adding the definition to the extended config map
baseRouteSpec:
tlsCipher:
tlsVersion: 1.2
ciphers: DEFAULT
cipherGroup: /Common/f5-default
autoMonitor: none
NOTE: value is case sensitive. Other options are readiness-probe and service-endpoint
still messages are looping in the log. The logs are send to Splunk which will fill up with a log rate like this.
I guess if have to check how to stop this log loop.
2025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02
9732025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....
9742025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...
9752025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...
9762025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02
9772025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02
9782025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....
9792025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...
9802025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...
9812025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02
9822025/02/24 11:12:59 [INFO] Starting Namespace Informer for cluster openshift-engineering-02
9832025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bar 78a8d673-506f-4d27-88e8-399b9642a9ba 282171 0 2025-02-12 15:03:34 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bar pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c12 openshift.io/sa.scc.supplemental-groups:1000780000/10000 openshift.io/sa.scc.uid-range:1000780000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift....
9842025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{bigip-ctlr 113b098f-5f26-4f8e-b618-aed4fd95593e 1100409 0 2025-02-14 11:52:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:bigip-ctlr openshift-pipelines.tekton.dev/namespace-reconcile-version:1.17.1 pod-security.kubernetes.io/audit:baseline pod-security.kubernetes.io/audit-version:v1.24 pod-security.kubernetes.io/warn:baseline pod-security.kubernetes.io/warn-version:v1.24] map[openshift.io/sa.scc.mcs:s0:c41,c5 openshift.io/sa.scc.supplemental-groups:1001650000/10000 openshift.io/sa.scc.uid-range:1001650000/10000] [] [] [{kubectl-create Update v1 2025-02-14 11:52:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift.io/sa.scc.supplemental-groups":{},"f:openshift.io/sa.scc.uid-range":{}},"f:labels":{".":{},"f:ingress":{},"f:kubernetes.io/metadata.name":{},"f:openshift-pipelines.tekton.dev/namespace-reconcile-version":{},"f:pod-security.kubernetes.io/audit":{},...
9852025/02/24 11:12:59 [DEBUG] Enqueueing Namespace: &Namespace{ObjectMeta:{cafe 3014dbb4-8579-4868-933b-f4c6c4ea3530 282087 0 2025-02-12 15:03:23 +0000 UTC <nil> <nil> map[ingress:f5nr kubernetes.io/metadata.name:cafe pod-security.kubernetes.io/audit:restricted pod-security.kubernetes.io/audit-version:latest pod-security.kubernetes.io/warn:restricted pod-security.kubernetes.io/warn-version:latest] map[openshift.io/sa.scc.mcs:s0:c28,c7 openshift.io/sa.scc.supplemental-groups:1000770000/10000 openshift.io/sa.scc.uid-range:1000770000/10000] [] [] [{pod-security-admission-label-synchronization-controller Apply v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:pod-security.kubernetes.io/audit":{},"f:pod-security.kubernetes.io/audit-version":{},"f:pod-security.kubernetes.io/warn":{},"f:pod-security.kubernetes.io/warn-version":{}}}}} {cluster-policy-controller Update v1 2025-02-12 15:03:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/sa.scc.mcs":{},"f:openshift...
9862025/02/24 11:12:59 [DEBUG] Clients Created for cluster: openshift-engineering-02
- PaulVogt
I see this log message has disappeared in the master repository. So it will probably be fixed in 2.19.2. For now I run with log level WARNING to stop the log flooding.
- PaulVogt
The logging is not the only issue. 2.19.1 also doe snot create F5 objects in HA cis active-active mode on the primary CIS. The secondary CIs works fine. Switched the primary back to 2.18.1 for now.
- Nikoolayy1
Strange that you have no "probeInterval:" or "retryInterval: " for the multi cluster. Other than that where you got the " autoMonitor: none" option as from openshift-4-9/next-gen-routes/single-vip/ocp-route/route-tea.yaml at main · mdditt2000/openshift-4-9 · GitHub of Mark_Dittmer I don't see this for next gen routes as annotations are used under the route object ?
Ratio and Active-Active mode in Multi-Cluster
That is what came to my mind and as I already mentioned Mark's videos and github are great sources of info.
- PaulVogt
I got the possible values for autoMonitor from the source code, https://github.com/F5Networks/k8s-bigip-ctlr. Setting it to none in the extended configmap made the warning disappear. CIS 2.19.1 keeps looping with [INFO] Starting Namespace Informer for cluster openshift-engineering-02, flooding the log.
The secondary runs fine. I can try to make the secondary the primary and see what happens then.
- PaulVogt
I think I found the reason. The check probe of bigip-ctlr was configured as an Openshift route managed by cis itself. So when starting the primary CIS it had to configure this first on the F5 to get the health check to work. While the secondary did not had this route and was still active, so blocking the creation of this route.
Creating a static route for the health check, using the default ingress controller on port 80 solved the issue
