cancel
Showing results for 
Search instead for 
Did you mean: 

F5 Kubernetes BIG-IP Controller or CIS not connecting to Azure Big-IP deployment

Patrick_Lieberg
Nimbostratus
Nimbostratus

I have started a POC for the BIG-IP Azure deployments, which deployed successfully and I have accessed and set the password. I've deployed the helm chart for CIS, but the pod fails to start. I've tested connectivity to the Azure BIG-IP deployment from a separate pod in the same namespace and it authenticates and returns correct info. I've validated the Azure BIG-IP creds are properly formatted in a secret and that secret is getting mounted in the CIS pod.

 

Here is the pod log with logging level set to debug:

2021/10/04 21:21:39 [DEBUG] No url in credentials directory, falling back to CLI argument 2021/10/04 21:21:39 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.5.0, BuildInfo: azure-465-1952a80a2165b7fc2d3561795ad09d1eb8615136 2021/10/04 21:21:39 [INFO]TeemServer:product.apis.f5.com 2021/10/04 21:21:39 teemClient:{{CIS-Ecosystem CIS/v2.5.0 df103609-7748-43e4-95a4-6631030e67d0} mmhJU2sCd63BznXAXDh4kxLIyfIMm3Ar   product.apis.f5.com} 2021/10/04 21:21:39 [DEBUG] digitalAssetId:950e75d5-7fe0-88bc-eb3c-d654ebb4de47 2021/10/04 21:21:39 [DEBUG] telemetryDatalist:[{"Agent":"as3","ConfigmapsCount":0,"DateOfCISDeploy":"2021-10-04T21:21:39.452535893Z","ExternalDNSCount":0,"IPAMSvcLBCount":0,"IPAMTransportServerCount":0,"IPAMVirtualServerCount":0,"IngressCount":0,"IngressLinkCount":0,"Mode":"cluster","PlatformInfo":"CIS/v2.5.0 K8S/v1.19.11","RoutesCount":0,"RunningInDocker":false,"SDNType":"calico","TransportServerCount":0,"VirtualServerCount":0}] 2021/10/04 21:21:39 [DEBUG] ControllerAsDocker:#{docker} 2021/10/04 21:21:40 Resp Code:204 Status:204 No Content 2021/10/04 21:21:40 [INFO] ConfigWriter started: 0xc000284570 2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) writing section name global 2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) successfully wrote section (global) 2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) writing section name bigip 2021/10/04 21:21:40 [DEBUG] [CCCL] ConfigWriter (0xc000284570) successfully wrote section (bigip) 2021/10/04 21:21:40 [INFO] Started config driver sub-process at pid: 21 2021/10/04 21:21:40 [DEBUG] [INIT] Invalid trusted-certs-cfgmap option provided. 2021/10/04 21:21:40 [INFO] [INIT] Creating Agent for as3 2021/10/04 21:21:40 [DEBUG] [CORE] Agent Response Worker started and blocked on channel  0xc0004e04e0 2021/10/04 21:21:40 [INFO] [AS3] Initializing AS3 Agent 2021/10/04 21:21:41 [DEBUG] [AS3] No certs appended, using only system certs 2021/10/04 21:21:41 [DEBUG] [AS3] Validating AS3 schema with  as3-schema-3.28.0-3-cis.json 2021/10/04 21:21:41 [DEBUG] [AS3] posting GET BIGIP AS3 Version request on https://10.2.0.7:8443/mgmt/shared/appsvcs/info 2021/10/04 21:21:43 [ERROR] [AS3] Response body unmarshal failed: invalid character '<' looking for beginning of value 2021/10/04 21:21:43 [ERROR] [AS3] Internal Error 2021/10/04 21:21:43 [CRITICAL] [INIT] Failed to initialize as3 agent, Internal Error
1 ACCEPTED SOLUTION

Patrick_Lieberg
Nimbostratus
Nimbostratus

Turns out I missed a step in adding the AS3 module to my BIG-IP VE poc VM.

 

Pretty silly really.

 

Thanks for the suggestions everyone.

View solution in original post

3 REPLIES 3

MichaelOLeary
F5 Employee
F5 Employee

Hi Patrick,

I've helped a bunch of customers set up CIS and I've come across this log message a few times:

 

2021/10/04 21:21:43 [ERROR] [AS3] Response body unmarshal failed: invalid character '<' looking for beginning of value

 

Every time, it's been a problem with the CIS pod authenticating to BIG-IP. Can we double-check your credentials that are stored as a secret in K8s?

 

Also, if you created the secret with a yaml file, did you remember to base64 encode it? Here's an example of a secret that has the password of "DefaultPass12345!" base64-encoded.

 

https://github.com/mikeoleary/azure-aks-kic-cis/blob/master/ingress/cis/secret_sa_rbac.yaml#L8

 

Mike

 

Jeff_Giroux
F5 Employee
F5 Employee

Hey good call on the auth. Usually a < symbol is part of HTML payload. JSON will have the {} brackets. If you're able to tcpdump or debug the traffic and see entire payload, you will most likely see an unexpected response from the server in HTML which means the initial client call is not correct or in correct format. In my case, I was not working with CIS but was doing JSON auth to a URL...wrong URL ended up providing an error in HTML <> payload. However, the terrraform error only gave the invalid character < message but a full request/response showed bad URL, bad response clearly 🙂 hope that helps some...

Patrick_Lieberg
Nimbostratus
Nimbostratus

Turns out I missed a step in adding the AS3 module to my BIG-IP VE poc VM.

 

Pretty silly really.

 

Thanks for the suggestions everyone.