someguy
Nov 03, 2016Nimbostratus
Why do calls to REST API fail?
We are seeing intermittent failures for calls to the rest api on 11.5.1 HF6. The TCP handshake completes, client sends SSL Client Hello, BigIP sends ACK, and no further packets are seen. Is this related to iControl/REST connection limits?
In the icrd log I see:
Nov 1 12:45:34 mylb notice icrd_child[31618]: 31618,31633, RestRequestSender, INFO,Connection idle too long fd:9 cached.
Nov 1 12:45:42 mylb notice icrd: 8195,13830, RestRequestSender, INFO,Connection idle too long fd:13 cached.
Nov 1 12:50:34 mylb notice icrd_child[31655]: 31655,31670, RestRequestSender, INFO,Connection idle too long fd:10 cached.
Nov 1 12:50:52 mylb notice icrd: 8195,13830, RestRequestSender, INFO,Connection idle too long fd:13 cached.
Nov 1 12:55:33 mylb notice icrd_child[31717]: 31717,31732, RestRequestSender, INFO,Connection idle too long fd:10 cached.
Nov 1 12:56:02 mylb notice icrd: 8195,13830, RestRequestSender, INFO,Connection idle too long fd:13 cached.
Nov 1 12:56:08 mylb notice icrd: 8195,13817, RestServer, INFO,Connection idle too long fd:11
Nov 3 13:05:33 mylb notice icrd_child[31936]: 31936,31951, RestRequestSender, INFO,Connection idle too long fd:10 cached.
Nov 3 13:05:51 mylb notice icrd: 8195,13830, RestRequestSender, INFO,Connection idle too long fd:13 cached.
Nov 3 13:06:08 mylb notice icrd: 8195,13817, RestServer, INFO,Connection idle too long fd:11
The restjavad log doesn't register anything during the failure time, and lsof shows the below.
[myuser@mylb:Active:Changes Pending] ~ lsof -nPu apache | grep -E "(TCP|COMMAND)"
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 12579 apache 3u IPv6 1272168705 TCP *:80 (LISTEN)
httpd 12579 apache 5u IPv6 1272168710 TCP *:443 (LISTEN)
httpd 12579 apache 18u IPv4 1554156729 TCP 127.0.0.1:52743->127.0.0.1:8100 (CLOSE_WAIT)
...several iterations of the above...
httpd 16246 apache 3u IPv6 1272168705 TCP *:80 (LISTEN)
httpd 16246 apache 5u IPv6 1272168710 TCP *:443 (LISTEN)
httpd 16246 apache 16u IPv6 1555163788 TCP 10.1.1.1:443->10.10.10.10:50000 (ESTABLISHED)
httpd 16246 apache 18u IPv4 1554419261 TCP 127.0.0.1:43108->127.0.0.1:8100 (CLOSE_WAIT)