Have you ever run out of file descriptors while updating an AS3 declaration?

[bibilthaysose]

Running AS3 version 3.26.0 on Virtual Edition 15.1.5.  Upon PATCHing the declaration I'm persistently getting nonsensical 422s for objects that can't be found.  Upon checking the restnode log (/var/log/restnoded/restnoded.log), I see errors which indicate an exhaustion of file descriptors which appear to be limited to 4096 for the user restnoded, which the service runs as.  Is there any supported way to work around this?

Here are the aforementioned logs:

Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:37:32 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"}
Wed, 08 Jun 2022 18:37:34 GMT - finest: socket 201 opened
Wed, 08 Jun 2022 18:37:39 GMT - finest: socket 201 closed
Wed, 08 Jun 2022 18:40:21 GMT - finest: socket 202 opened
Wed, 08 Jun 2022 18:40:30 GMT - finest: socket 202 closed
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"}

[Mark_Dittmer]

@bibilthaysose are you seeing the same issue with POST?

[bibilthaysose]

yes, same issue when posting entire declaration.

[LiefZimmerman]

@bibilthaysose - did you get any resolution on this?

[Mark_Dittmer]

please file a Github issue at https://github.com/F5Networks/f5-appsvcs-extension/issues so we can get this priortized. 

[bibilthaysose]

@Mark_Dittmer @LiefZimmerman after upgrading AS3 to 3.36.1 we are no longer running out of FDs, but now we are running out of memory.  I also have a support case open and we're working with an F5 consultant.  I'll post back here with the results of all of that.

[Mark_Dittmer]

Please make sure when you using AS3 you increase RestJavaD memory. Here is the best practices document https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/best-practices.ht... - Recommend 500 to 1000

[bibilthaysose]

Yessir, we've maxed out the restjavad extramb, because our first issue was OOM related errors.  Actually, after upgrading to AS3 3.36.1, we've started to see those again rather than the EMFILE errors due to hitting the fd limits (apparently due to open sockets).