f5 cis pod crashes and restarts on checking status of task

[iyumlight]

f5 cis pod crashes and restarts on checking status of task · Issue #2514 · F5Networks/k8s-bigip-ctlr...

Setup Details

CIS Version : Container Ingress Services - Version: 2.9.1, BuildInfo: azure-2810-a8cd93c79f260b6d48f8ed5149df0ad1f785e3d9
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP Build: 1.11.26 lab 7
AS3 Version: BIGIP is serving with AS3 version : 3.36.1-1
Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version:
Pool Mode: Cluster/Nodeport
Additional Setup details: Rancher kubernetes on prem Vmware

Description

Hello , We are implementing F5 CIS +IPAM for load balancing need of our on-prem Rancher kubernetes cluster.

Deployment of F5 CIS and IPAM is fine , But when I create Service type Load balancer , CIS pod starts craching and restarting throwing below error .

2022/07/26 00:47:37 [DEBUG] [AS3] posting request with taskId to https://172.XX.XX.XX/mgmt/shared/appsvcs/task/f22cec12-16f5-4ba6-9f94-eef01d7477bd
--
Tue, Jul 26 2022 8:47:38 am | panic: interface conversion: interface {} is nil, not string
Tue, Jul 26 2022 8:47:38 am |  
Tue, Jul 26 2022 8:47:38 am | goroutine 83 [running]:
Tue, Jul 26 2022 8:47:38 am | github.com/F5Networks/k8s-bigip-ctlr/pkg/controller.(*PostManager).getTenantConfigStatus(0xc0004ceaf0, 0xc000b4c5d0, 0x24)
Tue, Jul 26 2022 8:47:38 am | /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/controller/postManager.go:201 +0xaeb
Tue, Jul 26 2022 8:47:38 am | github.com/F5Networks/k8s-bigip-ctlr/pkg/controller.(*Agent).pollTenantStatus(0xc00061d360)
Tue, Jul 26 2022 8:47:38 am | /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/controller/backend.go:419 +0x405
Tue, Jul 26 2022 8:47:38 am | github.com/F5Networks/k8s-bigip-ctlr/pkg/controller.(*Agent).agentWorker(0xc00061d360)
Tue, Jul 26 2022 8:47:38 am | /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/controller/backend.go:228 +0x65a
Tue, Jul 26 2022 8:47:38 am | created by github.com/F5Networks/k8s-bigip-ctlr/pkg/controller.NewAgent
Tue, Jul 26 2022 8:47:38 am | /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/controller/backend.go:75 +0x33f

When i try to access that URL , I get below error

{"id":"f22cec12-16f5-4ba6-9f94-eef01d7477bd","results":[{"code":500,"declarationFullId":"urn:uuid:B97DFADF-9F0D-4F6C-8D66-E9B52E593694|CIS Declaration","message":"failed to save BIG-IP config (connection to http://admin:XXXXXX@localhost:8100/mgmt/tm/task/sys/config create task save sys config timed out)"}],"declaration":{}}

Steps To Reproduce

1. RKE on vmware latest
2. Install F5 CIS and IPAM latest
3. Create service type LB

Expected Result

It should have created VS in F5 BIG IP

Actual Result

No VS gets created and PODs keep crashing

Diagnostic Information

F5 CIS Values.yaml

USER-SUPPLIED VALUES:
args:
  as3_validation: true
  bigip_partition: rke-np-pt
  bigip_url: 172.XX.XX.XX
  custom-resource-mode: true
  insecure: true
  ipam: true
  log_level: DEBUG
  pool_member_type: nodeport
bigip_login_secret: bigip-login
image:
  pullPolicy: Always
  repo: k8s-bigip-ctlr
  user: f5networks
ingressClass:
  create: false
  ingressClassName: nginx
  isDefaultIngressController: true
livenessProbe: {}
namespace: kube-system
nodeSelector:
  role: worker
rbac:
  create: true
readinessProbe: {}
resources: {}
serviceAccount:
  create: true
  name: k8s-bigip-ctlr
tolerations: []
version: latest

[Mark_Dittmer]

Afternoon. The error message with code 500 below is showing a auth issue. CIS cannot connect to the API. 

"results":[{"code":500,

 Can you please check the following

1) Install AS3 endpoint on AS3 https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/installation.html

2) Needs needs a Admin account to configure BIG-IP. This is added to K8S via secrets. 

Why does CIS need admin credentials? https://clouddocs.f5.com/containers/latest/reference/faq.html

Let me know if this helps.

[iyumlight]

Hi @Mark_Dittmer , 

Thanks for your response.  It turned out to be restarting few services on F5 

bigstart restart restjavad restnoded

While I have you on this thread , I was wondering how do you add tcp health monitors as part of CIS integration ?

Regards,

Prakash.