Technical Articles
F5 SMEs share good practice.
cancel
Showing results for 
Search instead for 
Did you mean: 
Custom Alert Banner
JRahm
Community Manager
Community Manager

Several months ago I wrote up the v10 formatting for internal and external datagroups: iRules Data Group Formatting Rules. In v11, however, there is a change to the format of the internal data group and the data group reference to external class files (the formatting in the external class file itself is unchanged). The formatting rules in v11 for data groups more closely resembles the tmsh commands necessary to build the class at the CLI (these command attributes are masked if you are using the GUI). I’ll follow the same format as the original write-up in showing the various data group types. The format is the same among internal data group types. If there is no value associated with the key, there is a curly bracket pair trailing the key on the same line. If there is an associated value with a key, the curly bracket opens the value, followed by a newline with the keyword data and the value, then another newline with the closing curly bracket. After the records are listed, the type is specified. For external data groups, the file name and the type are specified. If the filename is in /var/class, the path is omitted from the filename reference.

Address Data Groups

Internal Data Group

ltm data-group internal addr_testclass {
    records {
        192.168.1.1/32 { }
        192.168.1.2/32 {
            data "host 2"
        }
        192.168.2.0/24 { }
        192.168.3.0/24 {
            data "network 2"
        }
    }
    type ip
}

External Data Group

ltm data-group external addr_testclass_ext { external-file-name addr_testclass.class type ip }

Integer Data Groups

Internal Data Group

ltm data-group internal int_testclass {
    records {
        1 {
            data "test 1"
        }
        2 {
            data "test 2"
        }
    }
    type integer
}

External Data Group

ltm data-group external int_testclass_ext {
    external-file-name int_testclass
    type integer
}

String Data Groups

Internal Data Group

ltm data-group internal str_testclass {
    records {
        str1 {
            data "value 1"
        }
        str2 {
            data "value 2"
        }
    }
    type string
}

External Data Group

ltm data-group external str_testclass_ext {
    external-file-name str_testclass.class
    type string
}

External Datagroup File Management

Beginning in v11, external datagroups are imported into a local filestore rather than simply existing someplace on the file system (/config/filestore). However, this filestore is not meant to edited manually. Please follow the steps below for creating or modifying external datagroups. Thanks to hoolio for the steps in this external datagroup section.

Create a New External Datagroup from the CLI Non-Interactively

1. Create or copy over to LTM a temporary file containing the external data group contents. If copying, make sure the line terminators are \n only not \r\n.

# cat /var/tmp/string_name_value_external_dg.txt
"name1" := "value1",
"name2" := "value2",
"name3" := "value3",

2. Create the new external data group file

 

tmsh create /sys file data-group string_name_value_external_dg_file separator ":=" source-path file:/var/tmp/string_name_value_external_dg.txt type string

 

3. Create the external data group referencing the file

 

tmsh create /ltm data-group external string_name_value_external_dg external-file-name string_name_value_external_dg_file

 

Modify the External Datagroup File for Existing Datagroup

1. Create a new temporary file containing the updated external data group contents

# cat /var/tmp/string_name_value_external_v2_dg.txt
"name1" := "valueA",
"name2" := "valueB",
"name3" := "valueC",

2. Import the new data group file

 

tmsh create /sys file data-group string_name_value_external_v2_dg_file separator ":=" source-path file:/var/tmp/string_name_value_external_v2_dg.txt type string

 

3. Modify the data group definition to reference the new external data group file

 

tmsh modify /ltm data-group external string_name_value_external_dg external-file-name string_name_value_external_v2_dg_file

 

4. Delete the old data group file if it’s unneeded

 

tmsh delete sys file data-group string_name_value_external_dg_file

 

Handling Line Terminator Discrepencies

Whether imporing external datagroups in the GUI or from the CLI, the system does not accept files with \r\n line terminators, it only accepts \n. If you copy files over from windows, most likely you have the wrong terminator in your file format. To check, you can use the od command.

Datagroup Contents created in vi on LTM
[root@golgotha:Active] data_group_d # od -c /var/tmp/string_name_value_external_dg.txt
0000000   "   n   a   m   e   1   "       :   =       "   v   a   l   u
0000020   e   1   "   ,  \n   "   n   a   m   e   2   "       :   =
0000040   "   v   a   l   u   e   2   "   ,  \n   "   n   a   m   e   3
0000060   "       :   =       "   v   a   l   u   e   3   "   ,  \n
0000077
Datagroup Contents created in Notepad on Windows
[root@golgotha:Active] data_group_d # od -c /var/tmp/notepad_dg.txt
0000000   "   n   a   m   e   1   "       :   =       "   v   a   l   u
0000020   e   1   "   ,  \r  \n   "   n   a   m   e   2   "       :   =
0000040       "   v   a   l   u   e   2   "   ,  \r  \n   "   n   a   m
0000060   e   3   "       :   =       "   v   a   l   u   e   3   "   ,
0000100

If your line terminators are incorrect, you can use the tr command to remove the \r's.

[root@golgotha:Active] tmp # cat /var/tmp/notepad_dg.txt | tr -d '\r' > /var/tmp/notepad_dg_update.txt
[root@golgotha:Active] tmp # od -c /var/tmp/notepad_dg_update.txt
0000000   "   n   a   m   e   1   "       :   =       "   v   a   l   u
0000020   e   1   "   ,  \n   "   n   a   m   e   2   "       :   =
0000040   "   v   a   l   u   e   2   "   ,  \n   "   n   a   m   e   3
0000060   "       :   =       "   v   a   l   u   e   3   "   ,
0000076

 

Comments
hooleylist
Cirrostratus
Cirrostratus
See this post for details on external file data groups in v11:

 

 

http://devcentral.f5.com/tabid/53/aft/1184918/Default.aspx

 

 

In v11 to use an external file, you need to import it first. Here are the steps:

 

 

1. Create a text file containing the data group contents. Here is a sample string data group file:

 

 

"name1" := "value1",

 

"name2" := "value2",

 

"name3" := "value3",

 

 

2. If you're on Windows, use a utility like dos2unix.exe to convert the line endings from CRLF to LF (https://encrypted.google.com/search?q=dos2unix.exe). If you don't do this, you will get an error on import:

 

 

01070627:3: The string data group external file (/config/filestore/.stage_d/55_d/Common_d/data_group_d/:Common:name_value_external_dg_1) has an invalid format, line: 2.

 

 

3. Navigate to System | File Management | Data Group File List | Import

 

 

4. Specify a filename. Example:

 

 

File Name: C:\tmp\name_value_external_dg.dos.txt

 

Name: name_value_external_dg

 

File Contents: string

 

Key / Value Pair Seperator: =:

 

 

5. Navigate to Local Traffic | iRules | Data Group List | Create

 

 

6. Specify a data group name. Example:

 

 

Name: name_value_external_dg

 

Type: External (File)

 

File Name: name_value_external_dg

 

 

Aaron
Puli
Nimbostratus
Nimbostratus
Do you have any sample code for iControl Per to add/delete/list the new string class contents ?

 

thanks
JRahm
Community Manager
Community Manager
what language?
Paul_Davis_7109
Nimbostratus
Nimbostratus
python using pycontrol2
jross_47179
Nimbostratus
Nimbostratus
I would be very interested in the sample code also. We had a working solution in version 10 that we could manage our data group files but it looks like in 11 this has changed and does not work anymore. If there is a code sample could you please post it here.

 

 

Thanks!
Belanger__Yves
Altostratus
Altostratus
I use this in version 10.

 

 

when HTTP_REQUEST {

 

set redirecthost [class search -value $::ListeShwebredir equals [string tolower [HTTP::host]]]

 

if { $redirecthost ne "" } {

 

HTTP::redirect "$redirecthost"

 

log local0. "DEBUG-REDIR [HTTP::host][HTTP::uri] redirection vers : $redirecthost"

 

unset redirecthost

 

return

 

}

 

}

 

 

But, it doesn't work in version 11. We received this error message :

 

 

Nov 21 08:20:59 tmm err tmm[7850]: 01220001:3: TCL error: /Common/Shwebredir-iRule - can't read "::ListeShwebredir": no such variable while executing "class search -value $::ListeShwebredir equals [string tolower [HTTP::host]]"

 

 

Does any body know a solution for it?

 

 

Yves
hooleylist
Cirrostratus
Cirrostratus
Hi Yves,

 

 

Yes, remove the $:: prefix from the data group name:

 

 

https://clouddocs.f5.com/api/irules/class.html

 

 

Note that you should not use a $:: or :: prefix on the datagroup name when using the class command (or in any datagroup reference on 9.4.4 or later).

 

 

In v9.4.4 - 10, using $::datagroup_name will work but demote the virtual server from running on all TMMs. For details, see the CMP compatibility page.

 

 

In v11, using $::datagroup_name will result in a TCL runtime error and a reset being sent to the client!
yoggit
Nimbostratus
Nimbostratus
Hello,

 

 

On 11.3 I'm trying to modify an existing external datagroup as per the example above, but I get this error:-

 

 

[root@L22RRZPEP12:/S1-green-P:Active] tmp tmsh modify /ltm data-group external externaldg external-file-name list2

 

Syntax Error: "external-file-name" may not be specified in the context of the "modify" command. "external-file-name" may be specified using the following commands: create, list

 

[root@L22RRZPEP12:/S1-green-P:Active] tmp

 

 

Any ideas how I can modify the group without deleting and re-creating it?

 

 

Thanks,

 

Jon
yoggit
Nimbostratus
Nimbostratus
To edit an existing data group, the above does not work in 11.3.

 

 

Using the example above, you simple need to re-read the external file. This works:-

 

 

tmsh modify /sys file data-group string_name_value_external_dg_file separator ":=" source-path file:/var/tmp/string_name_value_external_dg_new_file.txt

 

Joe_H
Nimbostratus
Nimbostratus
Hi Jason & Co

 

 

I wonder how the external file format looks like to be used for importing external address data group in v11.3.

 

 

Compete example similar to importing string would be beneficial.

 

 

New to these stuff and hope to get the guidance along.

 

 

Thanks!

 

Joe
Joe_H
Nimbostratus
Nimbostratus
Just answering my own question. 🙂

 

 

In external file, I need to add keyword like network host in front of IP/netmask. i.e

 

 

network 10.0.0.0/8,

 

network 172.16.0.0/16,

 

 

Hope that will help other people.
Hem_66900
Cirrus
Cirrus
Please advise how to cover * wild cards entries using data group. /abc/12*/xyz -> https://coke.com/mno
In case you want to work with network address type external datagroups you may want to use the following structures: Key definition options for host addresses (no quotes used in key definition): host 10.21.21.21 network 10.22.22.22 prefixlen 32 network 10.23.23.23 mask 255.255.255.255 Key definition options for network addresses (no quotes used in key definition): network 10.24.24.0 prefixlen 24 network 10.25.25.0 mask 255.255.255.0 To add a value to a key the following syntax applies (double quotes used for argument definition): host 10.21.21.21 := "host 21", network 10.22.22.22 prefixlen 32 := "network 22", Further notes: - trailing comma in last line is optional - line feed between records is optional (all records can be put in a single line as the comma is used as record separator)
Sorry for missing formatting in my comment. Unfortunately the frontend does not let me change it.
Gullapudi_Venka
Nimbostratus
Nimbostratus
Thank you so much for the detailed explanation of the data group updates. I have a question. I am trying to save a data group file with some similar cookie elements. Using this data group i would like to reject the HTTP REQUEST. I mean if the HTTP::cookie has any one of the cookies in the data group i created. The HTTP request should be rejected. I will be adding new cookies to the data group on regular basis. So, Can you please giving me an idea of , how to approach or what might be the best way to put an iRule satisfying this situation. Here is my iRule when HTTP_REQUEST { if { class match -value [HTTP::cookie $aam_uuid] contains Cookie_Rejects } {reject} } Data group name is Cookie_Rejects and its internal with one string value. Consider i updated the data group with one more cookie value today, if a http request come with the updated cookie value it must be rejected with out changing any iRule. Thanks in advance
The-messenger
Cirrostratus
Cirrostratus

I've read through this thread, get to thinking I understand it but still no success. I've read older threads with script suggestions - I figured this would be a pretty simple process. I'm running 12.1.1. I have an existing Address type data group, I need to add about 250 IP addresses to the group.

 

Can I add/import into my existing Address type data group? Can I use the File Management function to do so?

 

JeffG
Nimbostratus
Nimbostratus

Can you give an example of loading an external datagroup file non-interactively with the type "ip" please? All of the combinations I've tried thus far give me the error, "0107169a:3: The data group external file (/config/filestore/.stage_d/...) has an invalid format, line: 1.". Thank you.

JeffG
Nimbostratus
Nimbostratus

Ok - turns out the format is similar to the V10 rules. The following worked - note that if you give the CIDR you have to use "network" and not "host" even if it's a 32-bit netmask.

network 192.168.1.1/32,
network 192.168.1.2/32,
Version history
Last update:
‎02-Oct-2023 10:59
Updated by:
Contributors