CIS Controller Logging and Reporting
Use Case: The CIS controller is deployed within an OCP 4.10 Cluster as an the ingress controller & configmap/non-http/non-https multi-tenent load-balancer. The cluster operates as a shared resource cluster hosting 200+ projects. Each project has a single IP that can create an ingress, ingress-fanout, or configmap to support exposing the service(s) for the project. The Functional Issue: The CIS controller processes the requests from the cluster to create and maintain the ingresses for the projects. The F5 partition created for the ingress is managed completly by the CIS controller. When/If CIS unsupported information is pushed (a variable, a wildcard, ...) within the ingress yaml, the cluster supports and ingests the ingress yaml, however the CIS controller stops processing completely. At this point, the CIS controller becomes non-functional. Solution: 1. Increase the logging on the CIS controller to include: Time/Date stamp of when the CIS controller stopped processing; Namespace where the CIS stopped doing work; Service Port; Hostname; IP 2. After the CIS stops working, the logs are still populating making it appear that the CIS is still working. Example: There are 200 VIP's created by the CIS ingress controller. One of the ingress files pushed to the OCP cluster contains a /* for the path. The controller logs only a single error visible only at the start of the log after you kill the CIS controller pod and restart. At this point the CIS continues logging and the appearance that things are processing that are not. Initial Error logged once at that top of the Log: 2022/12/01 16:51:13 [ERROR] - (root): Must validate one and only one schema (oneOf) 2022/12/01 16:51:13 [ERROR] - declaration.Shared: Must validate "then" as "if" was valid 2022/12/01 16:51:13 [ERROR] - declaration.Shared.rules.0.name: Does not match pattern '^[a-zA-Z0-9_\-.:%]+$' 2022/12/01 16:51:13 [ERROR] - declaration.Shared.rules.1.name: Does not match pattern '^[a-zA-Z0-9_\-.:%]+$' 2022/12/01 16:51:13 [ERROR] - declaration.Shared: Must validate all the schemas (allOf) 2022/12/01 16:51:13 [ERROR] - declaration.Shared: Must validate "then" as "if" was valid 2022/12/01 16:51:13 [ERROR] - declaration.Shared.rules.0.name: Does not match pattern '^[a-zA-Z0-9_\-.:%]+$' 2022/12/01 16:51:13 [ERROR] - declaration.Shared.rules.1.name: Does not match pattern '^[a-zA-Z0-9_\-.:%]+$' 2022/12/01 16:51:13 [ERROR] - declaration.Shared: Must validate all the schemas (allOf) 2022/12/01 16:51:13 [ERROR] - declaration.Shared: Must validate all the schemas (allOf) CIS appears then to keep processing but makes no logging that processing has actually stopped, when it stopped, and what caused it to stop. 2022/12/02 17:04:33 [DEBUG] [CORE] Configured rule: {ingress_stephen.com__ingress_stephen_ex1-service stephen.com/ 0 [0xc0137acfc0] [0xc013783b40]} 2022/12/02 17:04:33 [DEBUG] [RESOURCE] Configured policy: {ingress_172-72-72-87_80 OCP-DAL12 [forwarding] true [http] [0xc0137ad020] /Common/first-match} 2022/12/02 17:04:33 [DEBUG] [CORE] Configured rule: {ingress_stephen.com__ingress_stephen_ex1-service stephen.com/ 0 [0xc0137ad0e0] [0xc013783bc0]} 2022/12/02 17:04:33 [DEBUG] [RESOURCE] Configured policy: {ingress_172-72-72-87_443 OCP-DAL12 [forwarding] true [http] [0xc0137ad140] /Common/first-match}