15-Jun-2022 08:13
Running AS3 version 3.26.0 on Virtual Edition 15.1.5. Upon PATCHing the declaration I'm persistently getting nonsensical 422s for objects that can't be found. Upon checking the restnode log (/var/log/restnoded/restnoded.log), I see errors which indicate an exhaustion of file descriptors which appear to be limited to 4096 for the user restnoded, which the service runs as. Is there any supported way to work around this?
Here are the aforementioned logs:
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:37:32 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"}
Wed, 08 Jun 2022 18:37:34 GMT - finest: socket 201 opened
Wed, 08 Jun 2022 18:37:39 GMT - finest: socket 201 closed
Wed, 08 Jun 2022 18:40:21 GMT - finest: socket 202 opened
Wed, 08 Jun 2022 18:40:30 GMT - finest: socket 202 closed
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"}
Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"}
17-Jun-2022 13:42
@bibilthaysose are you seeing the same issue with POST?
20-Jun-2022 05:29
yes, same issue when posting entire declaration.
24-Jun-2022 12:59
@bibilthaysose - did you get any resolution on this?
24-Jun-2022 13:04
please file a Github issue at https://github.com/F5Networks/f5-appsvcs-extension/issues so we can get this priortized.
24-Jun-2022 13:09
@Mark_Dittmer @LiefZimmerman after upgrading AS3 to 3.36.1 we are no longer running out of FDs, but now we are running out of memory. I also have a support case open and we're working with an F5 consultant. I'll post back here with the results of all of that.
24-Jun-2022 15:32
Please make sure when you using AS3 you increase RestJavaD memory. Here is the best practices document https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/best-practices.ht... - Recommend 500 to 1000
27-Jun-2022 06:40
Yessir, we've maxed out the restjavad extramb, because our first issue was OOM related errors. Actually, after upgrading to AS3 3.36.1, we've started to see those again rather than the EMFILE errors due to hitting the fd limits (apparently due to open sockets).