Forum Discussion

Brad_Baker's avatar
Brad_Baker
Icon for Cirrus rankCirrus
Nov 27, 2023

Modifying multiple entries in a datagroup via api?

We have a datagroup with entries like this:

 

domain1.com := virtual /Common/www.domain1.com_vs_443
domain2.com := virtual /Common/www.domain2.com_vs_443
domain3.com := virtual /Common/www.domain3.com_vs_443

 

And so forth. This datagroup gets used by an iRule for re-routing traffic based on SNI. 

I need to be able to make an HTTP call to the F5 to remap these to put up a mainteance page. So in other words I want to modify the above to:

 

domain1.com := virtual /Common/maintenance.domain1.com_vs_443
domain2.com := virtual /Common/maintenance.domain2.com_vs_443
domain3.com := virtual /Common/maintenance.domain3.com_vs_443

 

I figured out that I can make a curl request as such to delete entries from the datagroup:

 

curl -ku "admin:superS3cret" -X PATCH -H 'Content-type: application/json' -d '{ "name":"dummy" }' https://lb1.internal.local/mgmt/tm/ltm/data-group/internal/dummy?options=records%20delete%20%7B%20www.domain1.com,www.domain2.com,www.domain3.com%20%7D |jq .

 

And via this article https://community.f5.com/t5/technical-forum/add-new-key-into-data-group-without-updating-entire-list-using/td-p/272699 I was able to determine we could add a key using this:

 

curl -ku "admin:superS3cret" -X PATCH -H 'Content-type: application/json' -d '{ "name":"dummy" }' 'https://lb1.internal.local/mgmt/tm/ltm/data-group/internal/dummy?options=records%20add%20%7b%20www.domain1.com%20%7b%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain1.com_vs_443%22%20%7d%20%7d' |jq .

 

However then I have to have one HTTP Request per domain. If I need to repoint 20 sites for example I have to make 20 different requests instead of one request with all domains. 

Is there a way to add multiple records at once? I tried something like this:

 

curl -ku "admin:superS3cret" -X PATCH -H 'Content-type: application/json' -d '{ "name":"dummy" }' 'https://lb1.internal.local/mgmt/tm/ltm/data-group/internal/dummy?options%3Drecords%20add%20%7B%20www.domain1.com%20%7B%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain1.com_vs_443%22%20%7D%20domain1.com%20%7B%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain1.com_vs_443%22%20%7D%20www.domain2.com%20%7B%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain2.com_vs_443%22%20%7D%20www.domain3.com%20%7B%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain3.com_vs_443%22%20%7D%20www.domain4.com%20%7B%20data%20%22virtual%20%2FCommon%2Fmaintenance.domain4.com_vs_443%22%20%7D%7D' |jq .

 

Here's the URL HTML decoded (so its easier to read):

 

https://lb1.internal.local/mgmt/tm/ltm/data-group/internal/dummy?options=records add { www.domain1.com { data "virtual /Common/maintenance.domain1.com_vs_443" } domain1.com { data "virtual /Common/maintenance.domain1.com_vs_443" } www.domain2.com { data "virtual /Common/maintenance.domain2.com_vs_443" } www.domain3.com { data "virtual /Common/maintenance.domain3.com_vs_443" } www.domain4.com { data "virtual /Common/maintenance.domain4.com_vs_443" }}

 

However I get this error:

 

{
"code": 400,
"message": "one or more properties must be specified",
"errorStack": [],
"apiError": 26214401
}

 

I haven't been able to figure out if adding multiple records isn't supported or if I have the syntax wrong and frankly the documentation here: https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_ltm_data-group_internal.html and here: https://clouddocs.f5.com/cli/tmsh-reference/v14/modules/ltm/ltm_data-group_internal.html isn't super clear.

Frankly the fact that patch doesn't just modify entries sent seems like a bug to me and using ?options record add seems like a hack.

Alternatively is there some way to run a modify instead of running delete and then add? That would be even more efficient (less HTTP calls) and less opportunity for a call to fail. 

I know I can just get all records, modify them and then post all records back. The trouble with that approach is that then I'm touching records I really don't want to touch and having to do some sort of regex replace. This seems like it could have the potential to accidentally modify records I don't want to modify if my regex isn't very explicit. I would prefer a more targeted approach which only modifies the records that need modifying and that touches nothing else.

Thanks

api
datagroup
devops
TMSH
  • Sebastiansierra's avatar
    Sebastiansierra
    Nov 27, 2023

    Hi Brad_Baker

    You can additionally edit the content for a specific register

    curl -ku "admin:Logi-123" -X PATCH -H 'Content-type: application/json' -d '{ "name":"test_domain" }' https://localhost/mgmt/tm/ltm/data-group/internal/test_domain?options=records%20modify%20%7B%20key3%20%7B%20data%20ke3new%20%7D%20%7D|jq .

    as tmos you can use all of this option to interact with the data-groups

    add      delete      modify     none        replace-all-with

     to avoid mistakes use a tool to encode and decode the URL. 

  • Sebastiansierra's avatar
    Sebastiansierra
    Icon for MVP rankMVP
    Nov 27, 2023

    Hi Brad_Baker,

    The easiest way is to delete all the content of the datagroup and then add the new records, for delete all the records use this curl:

    curl -ku "admin:admin" -X PATCH 'https://192.168.30.10/mgmt/tm/ltm/data-group/internal/test_domain' --header 'Content-Type: application/json' --data '{ "records": []}'

    To add the new records in only one call you can use this curl:

    curl -ku "admin:admin" -X PATCH 'https://192.168.30.1/mgmt/tm/ltm/data-group/internal/test_domain' --header 'Content-Type: application/json' --data '{"records": [{"name": "domain1.com","data": "virtual /Common/maintenance.domain1.com_vs_443"},{"name": "domain2.com","data": "virtual /Common/maintenance.domain2.com_vs_443"},{"name": "domain3.com","data": "virtual /Common/maintenance.domain3.com_vs_443"}]}'

    Hope it works.

     

     

    • Brad_Baker's avatar
      Brad_Baker
      Icon for Cirrus rankCirrus
      Nov 27, 2023

      Hey Sebastiansierra thanks for the response and suggestions.

      Unfortunately, the recommendation above won't accomplish what I am trying to do.

      I have approximately 100-150 entries in my datagroup. When I run the command recommended above: 

      curl -ku "admin:admin" -X PATCH 'https://192.168.30.1/mgmt/tm/ltm/data-group/internal/test_domain' --header 'Content-Type: application/json' --data '{"records": [{"name": "domain1.com","data": "virtual /Common/maintenance.domain1.com_vs_443"},{"name": "domain2.com","data": "virtual /Common/maintenance.domain2.com_vs_443"},{"name": "domain3.com","data": "virtual /Common/maintenance.domain3.com_vs_443"}]}'

      It replaces all 100-150 entries with the 4 entries in the data section.

      I don't want to delete existing entries; I only want to edit specific records. So I'm looking to do a targeted modification to only touch the records that need changing and leave the rest alone.

      Also, I understand I could do a GET to get all records, use a regex to replace just what I want to replace, and then do a PUT/PATCH to send the modified version back.

      But that hinges on getting a regex exactly right. If my regex matches wrong (i.e. too much) I could accidentally edit the wrong record. It just seems a bit risky to do it using regex/pattern matching. Thus wanting a targeted approach that edits specific records.

      Is that possible or does the F5 api just not support that?

  • Sebastiansierra's avatar
    Sebastiansierra
    Icon for MVP rankMVP
    Nov 27, 2023

    Hi Brad_Baker,

    to avoid mistakes you can use this curl call format to have more clear the new records that you want to create:

    curl -ku "admin:admin" -X PATCH 'https://192.168.30.1/mgmt/tm/ltm/data-group/internal/test_domain' \
--header 'Content-Type: application/json' \
--data '{
    "records": [
        {
            "name": "domain1.com",
            "data": "virtual /Common/maintenance.domain1.com_vs_443"
        },
        {
            "name": "domain2.com",
            "data": "virtual /Common/maintenance.domain2.com_vs_443"
        },
        {
            "name": "domain3.com",
            "data": "virtual /Common/maintenance.domain3.com_vs_443"
        }
    ]
}'
  • Sebastiansierra's avatar
    Sebastiansierra
    Icon for MVP rankMVP
    Nov 27, 2023

    Hi Brad_Baker,

    I finally found the solution to modify more than one register, it uses the method option to call tmos, so you  have to adapt the URL to tmos code and it works, in my example, I have added two records, key3, and key4 in the same call:

     curl -ku "admin:Logi-123" -X PATCH -H 'Content-type: application/json' -d '{ "name":"test_domain" }' https://localhost/mgmt/tm/ltm/data-group/internal/test_domain?options=records%20add%20%7b%20key3%20%7b%20data%20value3%20%7d%20key4%20%7b%20data%20value4%20%7d%20%7d |jq .

    I used notepad++ to decode and encode the URL and it looks like:

    curl -ku "admin:admin" -X PATCH -H 'Content-type: application/json' -d '{ "name":"test_domain" }' https://localhost/mgmt/tm/ltm/data-group/internal/test_domain?options=records add { key3 { data value3 } key4 { data value4 } } |jq .

    Hope it works.

    • Sebastiansierra's avatar
      Sebastiansierra
      Icon for MVP rankMVP
      Nov 27, 2023

      Hi Brad_Baker

      You can additionally edit the content for a specific register

      curl -ku "admin:Logi-123" -X PATCH -H 'Content-type: application/json' -d '{ "name":"test_domain" }' https://localhost/mgmt/tm/ltm/data-group/internal/test_domain?options=records%20modify%20%7B%20key3%20%7B%20data%20ke3new%20%7D%20%7D|jq .

      as tmos you can use all of this option to interact with the data-groups

      add      delete      modify     none        replace-all-with

       to avoid mistakes use a tool to encode and decode the URL. 

      • Brad_Baker's avatar
        Brad_Baker
        Icon for Cirrus rankCirrus
        Nov 28, 2023

        Perfect - that's exactly what I was trying to do! I was just using add instead of modify - it seems that was the issue.

        Thank you so much figuring that out! 🙂 Now I don't have to worry about accidentally messing up any other datagroup records during an update.