Forum Discussion
Patrick_Lieberg
Nimbostratus
NimbostratusF5 Kubernetes BIG-IP Controller or CIS not connecting to Azure Big-IP deployment
I have started a POC for the BIG-IP Azure deployments, which deployed successfully and I have accessed and set the password. I've deployed the helm chart for CIS, but the pod fails to start. I've tested connectivity to the Azure BIG-IP deployment from a separate pod in the same namespace and it authenticates and returns correct info. I've validated the Azure BIG-IP creds are properly formatted in a secret and that secret is getting mounted in the CIS pod.
Here is the pod log with logging level set to debug:
2021/10/04 21:21:39 [DEBUG] No url in credentials directory, falling back to CLI argument
2021/10/04 21:21:39 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.5.0, BuildInfo: azure-465-1952a80a2165b7fc2d3561795ad09d1eb8615136
2021/10/04 21:21:39 [INFO]TeemServer:product.apis.f5.com
2021/10/04 21:21:39 teemClient:{{CIS-Ecosystem CIS/v2.5.0 df103609-7748-43e4-95a4-6631030e67d0} mmhJU2sCd63BznXAXDh4kxLIyfIMm3Ar product.apis.f5.com}
2021/10/04 21:21:39 [DEBUG] digitalAssetId:950e75d5-7fe0-88bc-eb3c-d654ebb4de47
2021/10/04 21:21:39 [DEBUG] telemetryDatalist:[{"Agent":"as3","ConfigmapsCount":0,"DateOfCISDeploy":"2021-10-04T21:21:39.452535893Z","ExternalDNSCount":0,"IPAMSvcLBCount":0,"IPAMTransportServerCount":0,"IPAMVirtualServerCount":0,"IngressCount":0,"IngressLinkCount":0,"Mode":"cluster","PlatformInfo":"CIS/v2.5.0 K8S/v1.19.11","RoutesCount":0,"RunningInDocker":false,"SDNType":"calico","TransportServerCount":0,"VirtualServerCount":0}]
2021/10/04 21:21:39 [DEBUG] ControllerAsDocker:#{docker}
2021/10/04 21:21:40 Resp Code:204 Status:204 No Content
2021/10/04 21:21:40 [INFO] ConfigWriter started: 0xc000284570
2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) writing section name global
2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) successfully wrote section (global)
2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) writing section name bigip
2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) successfully wrote section (bigip)
2021/10/04 21:21:40 [INFO] Started config driver sub-process at pid: 21
2021/10/04 21:21:40 [DEBUG] [INIT] Invalid trusted-certs-cfgmap option provided.
2021/10/04 21:21:40 [INFO] [INIT] Creating Agent for as3
2021/10/04 21:21:40 [DEBUG] [CORE] Agent Response Worker started and blocked on channel 0xc0004e04e0
2021/10/04 21:21:40 [INFO] [AS3] Initializing AS3 Agent
2021/10/04 21:21:41 [DEBUG] [AS3] No certs appended, using only system certs
2021/10/04 21:21:41 [DEBUG] [AS3] Validating AS3 schema with as3-schema-3.28.0-3-cis.json
2021/10/04 21:21:41 [DEBUG] [AS3] posting GET BIGIP AS3 Version request on https://10.2.0.7:8443/mgmt/shared/appsvcs/info
2021/10/04 21:21:43 [ERROR] [AS3] Response body unmarshal failed: invalid character '<' looking for beginning of value
2021/10/04 21:21:43 [ERROR] [AS3] Internal Error
2021/10/04 21:21:43 [CRITICAL] [INIT] Failed to initialize as3 agent, Internal Error
Turns out I missed a step in adding the AS3 module to my BIG-IP VE poc VM.
Pretty silly really.
Thanks for the suggestions everyone.
- MichaelOLeary
Employee
Hi Patrick,
I've helped a bunch of customers set up CIS and I've come across this log message a few times:
2021/10/04 21:21:43 [ERROR] [AS3] Response body unmarshal failed: invalid character '<' looking for beginning of value
Every time, it's been a problem with the CIS pod authenticating to BIG-IP. Can we double-check your credentials that are stored as a secret in K8s?
Also, if you created the secret with a yaml file, did you remember to base64 encode it? Here's an example of a secret that has the password of "DefaultPass12345!" base64-encoded.
https://github.com/mikeoleary/azure-aks-kic-cis/blob/master/ingress/cis/secret_sa_rbac.yaml#L8
Mike
Hey good call on the auth. Usually a < symbol is part of HTML payload. JSON will have the {} brackets. If you're able to tcpdump or debug the traffic and see entire payload, you will most likely see an unexpected response from the server in HTML which means the initial client call is not correct or in correct format. In my case, I was not working with CIS but was doing JSON auth to a URL...wrong URL ended up providing an error in HTML <> payload. However, the terrraform error only gave the invalid character < message but a full request/response showed bad URL, bad response clearly :) hope that helps some...
Patrick_LiebergTurns out I missed a step in adding the AS3 module to my BIG-IP VE poc VM.
Pretty silly really.
Thanks for the suggestions everyone.
