16-Aug-2023 05:25 - edited 16-Aug-2023 06:49
Hi all,
I'm struggling using asynchronus tasks to generate a UCS. I've been through the API reference, and tried exactly what they suggest but something isn't right. I've dug through logs, watched lsof and top while executing the commands to see what happens, and I can't figure out why this isn't working. Searching devcentral hasn't been useful for me either.
I've tried this with 4 different devices, and all exhibit the same behaviour.
I send the POST to start the task, and I get confirmation of the task number and it's state.
curl -k -u apiadmin:<redacted> -H "Content-Type: \
application/json" -X POST -d \
'{
"command": "save",
"name": "myUcs"
}' \
https://192.168.240.15/mgmt/tm/task/sys/ucs
{"_taskId":1692187396506186,"_taskState":"STARTED","_taskTimeInStateMs":0,"_taskResultLink":"https://localhost/mgmt/tm/task/sys/ucs/1692187396506186/result?ver=16.1.3.3","selfLink":"https://localhost/mgmt/tm/task/sys/ucs/1692187396506186?ver=16.1.3.3"}%
_____________
Just to check on the task, I get the status and it shows as STARTED which we would expect. It makes no difference in the end result if I check the task before putting into VALIDATING or not.
❯ curl -k -X GET -u apiadmin:<redacted> https://192.168.240.15/mgmt/tm/task/sys/ucs/1692187396506186
{"selfLink":"https://localhost/mgmt/tm/task/sys/ucs/1692187396506186?ver=16.1.3.3","_taskId":1692187396506186,"_taskState":"STARTED","_taskTimeInStateMs":28000,"_taskResultLink":"https://localhost/mgmt/tm/task/sys/ucs/1692187396506186/result?ver=16.1.3.3"}%
_____________
I PUT the taskstate to VALIDATING, and I get confirmation that the task will start.
curl -k -u apiadmin:<redacted> -H "Content-Type: \
application/json" -X PUT -d \
'{
"_taskState":"VALIDATING"
}' \
https://192.168.240.15/mgmt/tm/task/sys/ucs/1692187396506186
{"code":202,"message":"Task will execute asynchronously.","errorStack":[]}%
_____________
Once that happens, the API seems to hang a bit as I can't make new API calls for 30 seconds to a minute, and during that time both java and gzip are churning away on a file in /var/tmp/imxxxxx/configsync-2.0-1-Linux-3.10.0862.14.4.7..8664.im. In this last case, the directory was im22934, which doesn't correspond to my requested task that I can tell.
im22934:
total 438M
drwxr-xr-x. 2 root root 4.0K Aug 16 07:06 .
drwxrwxrwt. 31 root root 12K Aug 16 07:13 ..
-rw-r--r--. 1 root root 438M Aug 16 07:05 configsync-2.0-1-Linux-3.10.0862.14.4.7..8664.im
From support, these are orphaned attempts at creating a UCS so the process is starting, but dieing and I don't know why : https://my.f5.com/manage/s/article/K49300144
curl -k -X GET -u apiadmin:<redacted> https://192.168.240.15/mgmt/tm/task/sys/ucs/1692187396506186
{"code":404,"message":"Task not found - ID: 1692187396506186 user: apiadmin","errorStack":[],"apiError":1}%
_____________
I've been through logs and not found any errors. I've tried different users, with and without underscores in the name. Nothing seems to want to work for me.
What am I doing wrong?
16-Aug-2023 08:58
Maybe you're running into this bug: https://cdn.f5.com/product/bugtracker/ID797721.html
16-Aug-2023 09:15
Thank you for that, I had come across it before but didn't think it fit since my version isn't listed as an affected version, I'm running 16.1.3.3.
I do see TimeoutExceptions all over the restjavad.x.log, for everything I try to do. The POST to create the task, which is successful, the PUT, and the GET to check the status. So it seems that either the system is able to recover from that status sometimes, or sometimes it's just a meaningless message.
The bug lists icrd being busy as a requisite, but this is on 2 different pairs of lab devices. The ones I tested this morning are idle, like.. 0 connections for the last few days idle, and I am the only person sending any kind of API requests to them.
The bug also has no resolution, do you happen to know of any fix? We're currently generating ucs archives and downloading via central bash scripts that I'm seeing occasional issues with, and I'd like to move to using the API.
16-Aug-2023 09:59
@BrianWeb If this keeps erroring you might use the following article instead which should allow you to automate the UCS creation as well as copying it off using SFTP.
https://community.f5.com/t5/codeshare/f5-iapp-automated-backup/ta-p/288311
16-Aug-2023 10:07
Thank you for that suggestion,
This is not a direction I can go at this time. I need to be able to initiate from and pull to a central location for hundreds of devices. Installing and maintaining an iapp on each device is not sustainable for us right now.
16-Aug-2023 10:56
can you generate the UCS via iControl REST synchronously without the task? If so, I'd suggest opening a case to inquire.
16-Aug-2023 12:13 - edited 16-Aug-2023 12:14
Tried a non task ucs save, and it also failed after sitting a while waiting on a response. Looks like support is my only hope.
❯ curl -k -u apiadmin:<redacted> -H "Content-Type: \
application/json" -X POST -d \
'{
"command": "save",
"name": "myUcs"
}' \
https://192.168.240.15/mgmt/tm/sys/ucs
{"code":400,"message":"remoteSender:192.168.17.87, method:POST ","referer":"192.168.17.87","restOperationId":16040255,"kind":":resterrorresponse"}%
I can see it started, but didn't make it as far as the task request did. It didn't get zipped up
im.16590 # ls -la
total 376
drwxr-xr-x. 2 root root 4096 Aug 16 14:03 .
drwxrwxrwt. 24 root root 12288 Aug 16 14:03 ..
-rw-r--r--. 1 root root 90907 Aug 16 14:03 Files
-rw-r--r--. 1 root root 175954 Aug 16 14:03 SPEC-Files
-rw-r--r--. 1 root root 95809 Aug 16 14:03 SPEC-Manifest
Tried it from tmsh, just in case something was majorly broken :
(cfg-sync In Sync)(Active)(/Common)(tmos)# save sys ucs test_tmsh
Saving active configuration...
/var/local/ucs/test_tmsh.ucs is saved.
# ls -lah /var/local/ucs
total 438M
drwxr-xr-x. 3 root root 4.0K Aug 16 14:08 .
drwxr-xr-x. 9 root root 4.0K May 4 13:31 ..
-r--r--r--. 1 root root 176 Dec 22 2022 cs_backup_rotate.conf
-rw-r--r--. 1 root webusers 438M Aug 16 14:07 test_tmsh.ucs
drwxr-xr-x. 2 root root 4.0K May 4 13:57 tmp
Seeing the same TimeoutException errors from the non-task request as well
[WARNING][750][16 Aug 2023 19:03:53 UTC][8100/mgmt ForwarderPassThroughWorker] URI:http://localhost:8100/mgmt/tm/sys/ucs, Referrer:<redacted>, Method:POST, Exception:java.util.concu
rrent.TimeoutException: remoteSender:<redacted>, method:POST
[WARNING][751][16 Aug 2023 19:03:53 UTC][8100/tm/auth/remote-role PublicWorker] URI:http://localhost:8100/tm/sys/ucs, Referrer:<redacted>, Method:POST, Exception:java.util.concurrent.TimeoutException: remoteSender:<redacted>, method:POST
Thank you for taking a look with me, I appreciate it.
21-Aug-2023 15:37
@BrianWeb - If your post was solved it would be helpful to the community to select *Accept As Solution*.
This helps future readers find answers more quickly and confirms the efforts of those who helped.
Thanks for being part of our community.
Lief
23-Aug-2023 13:24
Hi Lief,
I wouldn't call it solved yet. I'm working with Support to see if we can make this work. currently we're monkeying with api timeouts but at the moment, the only part of the answers here that could be even tangentially considered a solution is to contact support. If that meets the bar to your criteria, I'll mark it as such. I don't believe it does.
24-Aug-2023 08:31
100% understand. No solution.
Thanks for checking back in.