For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

don_23889's avatar
don_23889
Icon for Nimbostratus rankNimbostratus
Dec 28, 2009

grep'ng for 'string' in bigip.conf

grep'ng for 'string' and returning entire block of object definitions from bigip.conf for each instance of 'string', starting with 'virtual' and ending with '}'

 

 

config  less bigip.conf |grep -B5 -A5 string |more 

 

This is cumbersome since you have to guess for -B(x) and -A(x)

 

 

So, as an example, I want to return all object definitions with iRule 'http-xff'. Basically, all lines between [virtual] and [}]

 

 

This example is syntactically incorrect, and only for illustrating what I am trying to achieve....ie...pseudo-code.

 

 

 
  less bigip.conf |grep BEGIN[virtual] END[}] http-xff |more 
  
 virtual vip1.80 { 
    pool vip1.80 
    destination aaa.bbb.ccc.ddd:http 
    ip protocol tcp 
    rules SNAT2VIP 
    profiles 
       http-xff 
       tcp-lan-optimized 
          serverside 
       tcp-wan-optimized 
          clientside 
    persist source_addr_30min 
 } 
  
 virtual vip-2.80 { 
    pool vip-2.80 
    destination mmm.nnn.ooo.ppp:http 
    ip protocol tcp 
    rules SNAT2VIP 
    profiles 
       http-xff 
       tcp-lan-optimized 
          serverside 
       tcp-wan-optimized 
          clientside 
    persist source_addr_30min 
 } 
  
 

 

 

What would really be sweet, would be to grep for a given 'string', and return entire block for 'any' object definition...where the preceding line and following line are are both blank. This would return all object definitions for a given string...virtual, pool, monitor...etc.

30 Replies

  • Mark_Crosland_2's avatar
    Mark_Crosland_2
    Historic F5 Account
    Note that the example demonstrates the data-driven nature of the tmsh scripting API. As new types of components are added to the system (e.g., virtual, pool, monitor, etc...), or, new properties/settings are added to existing components, the script will automatically search new elements. The script does not have to be updated, it just continues to work.
  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    Just wanted to thank everyone for the examples for parsing a bigip.conf (for non-TMSH versions like 9.3.x). I used this post as a start for a script to reorder the bigip.conf objects alphabetically. It's a bit of a hack, but gets the job done.

    The use case is:

    Prepare bigip.conf files from several BIG-IP's in a customers set of environments. The configs are similar between environments, but not identical. It's a royal pain trying to diff two bigip.conf files from separate environments when all the iRules, VIPs, etc, are in different orders.

    Note that I haven't tried loading the resulting bigip.conf--just comparing files so far.

    So thanks guys,

    Aaron

       
     !/usr/bin/perl 
      
      v0.2 - 2010-01-27 
      
      This is a simple script to parse a bigip.conf and print out each instance of an object types in alphabetical order 
      Ex: my_class, whose_class, your_class, my_rule_1, my_rule_2, your_rule_1, my_vip_1, my_vip_2, your_vip_1, your_vip_2,  
      
      Order of the printing of the object types themselves is hardcoded for now using the @ObjectTypeNames array. 
      This could be changed to print in the order the object types are found in the bigip.conf 
      Object types not defined in $ObjectTypeRegex are considered "unknown" and are printed first. 
      
      
     use strict;  Need to declare the ObjectTypeName arrays first 
     use warnings; 
     use Switch; 
      
      Get the filename from the command line arguments. Default to ./bigip.conf 
     @ARGV = 'bigip.conf' unless @ARGV; 
     my ($File) = @ARGV; 
      
      Debug logging? (0=none, 1=debug on result printing, 2=debug on printing and file parsing) 
     my $debug = 0; 
      
      Initialise some variables 
     my ($CurlyBracesCount, $Object) = (0, ""); 
     my $ObjectDefLine = ""; 
      
      Array of object type names.  
      The names need to be in the order that object types are listed in the conf files to ensure the output order is valid. 
     my @ObjectTypeNames = ('unknowns', 'routes', 'classes', 'monitors', 'auths', 'profiles', 'nodes', 'pools', 'snatpools', 'snats', 'rules', 'virtual_addresses', 'virtuals'); 
      
      Regex which matches 
     my $ObjectTypeRegex = qw/(^(?:auth|class|monitor|node|pool|profile|route|rule|snat|snatpool|virtual).*){$/; 
      
      ToDo: Declare array names for each ObjectTypeName and enable 'use strict' 
      
      Open file 
     open (Fh, $File) || die "Cant open $File"; 
      
     print "===========================\nStart...\n\n" if $debug > 1; 
      
      Read in the file 
     while (  ){ 
      
      Append current line 
     $Object .= $_; 
      
      Count the number of open/close curly braces in the current line 
     $CurlyBracesCount += (tr/\{//); 
     $CurlyBracesCount -= (tr/\}//); 
      
      If the curly brace count is 0, we've found a matching close brace 
     if ($CurlyBracesCount == 0) { 
      
      Look for an object type definition 
     if ((split /\n/, $Object)[0] =~ /$ObjectTypeRegex/s) { 
     print "Matched: $1\n" if $debug > 1; 
      Save the match if there was one 
     if ($1 ne "") { 
     $ObjectDefLine = $1; 
     $ObjectDefLine =~ s/\s+$//; 
     print "Saving \$ObjectDefLine: $ObjectDefLine\n" if $debug > 1; 
     } 
     } elsif ($Object =~ /[a-z].*\{$/) { 
      Found a line which starts with an alpha character and ends with an open curly brace 
     print "\nUnknown object definition: $Object\n" if $debug > 1; 
     $ObjectDefLine = "unknown"; 
     } else { 
      Found something else... 
     print "\nDefault. Unknown object definition:\n$Object\n" if $debug > 1; 
     $ObjectDefLine = "unknown"; 
     } 
      
      Save the current object to the corresponding object type array 
     print "Evaluating object type definition: $ObjectDefLine\n" if $debug > 1; 
     switch ((split / /, $ObjectDefLine)[0]) { 
     case "auth" { 
      Add the current class name and definition as a hash to an object type array 
     print "Adding object name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@auths,  
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "class" { 
      Add the current class name and definition as a hash to an object type array 
     print "Adding class name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@classes, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "monitor" { 
     print "Adding monitor: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@monitors, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "node" { 
     print "Adding node: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@nodes, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "pool" { 
     print "Adding pool: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@pools, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "profile" { 
     print "Adding profile: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@profiles, 
     {ObjectName => (split / /, $ObjectDefLine)[1]." ".(split / /, $ObjectDefLine)[2], ObjectDef => $Object} 
     ); 
     } 
     case "route" { 
     print "Adding route: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@routes, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "rule" { 
     print "Adding rule: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@rules, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "snat" { 
     print "Adding snat: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@snats, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "snatpool" { 
     print "Adding snatpool: $ObjectDefLine, name: ".(split / /, $ObjectDefLine)[1]."\n" if $debug > 1; 
     push(@snatpools, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "packet" { 
     print "Adding packet filter: $ObjectDefLine\n, name: ".(split / /, $ObjectDefLine)[1]."" if $debug > 1; 
     push(@packet_filters, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     case "virtual" { 
      ToDo: could parse virtual servers and virtual addresses into separate arrays 
      If so, check for second word of "address" to find virtual addresses 
     if ((split / /, $ObjectDefLine)[1] eq "address") { 
     print "Adding virtual address: $ObjectDefLine\n, name: ".(split / /, $ObjectDefLine)[2]."" if $debug > 1; 
     push(@virtual_addresses, 
     {ObjectName => (split / /, $ObjectDefLine)[2], ObjectDef => $Object} 
     ); 
     } else { 
     print "Adding virtual server: $ObjectDefLine\n, name: ".(split / /, $ObjectDefLine)[1]."" if $debug > 1; 
     push(@virtuals, 
     {ObjectName => (split / /, $ObjectDefLine)[1], ObjectDef => $Object} 
     ); 
     } 
     } 
     else { 
     print "No match: $ObjectDefLine\n\$Object: $Object" if $debug > 1; 
     push(@unknowns, 
     {ObjectName => "unknown", ObjectDef => $Object} 
     ); 
     } 
     } 
     $Object = ""; 
     print "-----------------------------\n" if $debug > 1; 
     } 
     } 
     close (Fh); 
      
      Print the output 
     print "===========================\nResults:\n" if $debug; 
      
      Loop through the Object Types by name 
     foreach $ObjectType (@ObjectTypeNames) { 
      
      Sort the array by the ObjectType hash key name, ObjectName 
     my @SortedObjectType =  sort { $a->{'ObjectName'} cmp $b->{'ObjectName'} } @$ObjectType; 
      
     print "===========================\n" if $debug; 
     print "Object Type: $ObjectType\n\n" if $debug; 
      
      Loop through the array of Object hashes 
     foreach $ObjectHashRef (@SortedObjectType) { 
      
      Loop through the Objects and print the definition 
     foreach $Object ($ObjectHashRef) { 
     print "---------------------------\n" if $debug; 
     print "$Object->{'ObjectName'}:\n" if $debug; 
     print $Object->{'ObjectDef'}; 
     } 
     } 
     } 
       
  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    The output is just each object instance ordered by name. The eventual goal is to use the same ordering of object types as they're listed in the original bigip.conf. For now it's just a way to reorder the bigip.conf to make diff'ing of two different bigip.conf's easier.

     

     

    So if you had a few classes, pools and VIPs defined but listed in mixed alphabetical order:

     

     

    whose_class {...}

     

    your_class {...}

     

    my_class {...}

     

     

    your_rule_1 {...}

     

    my_rule_2 {...}

     

    my_rule_1 {...}

     

     

    your_vip_2 {...}

     

    my_vip_2 {...}

     

    your_vip_1 {...}

     

    my_vip_1 {...}

     

     

    The script would write these out in alphabetical order with the definitions unchanged:

     

     

    my_class {...}

     

    whose_class {...}

     

    your_class {...}

     

     

    my_rule_1 {...}

     

    my_rule_2 {...}

     

    your_rule_1 {...}

     

     

    my_vip_1 {...}

     

    my_vip_2 {...}

     

    your_vip_1 {...}

     

    your_vip_2 {...}

     

     

    At some point, I'd like to write out the object types in the same order as they were parsed from the bigip.conf. But for now, the order is hardcoded in the script. Once I fix that, I'll add it to the codeshare.

     

     

    Aaron
  • Hey hoolio, did you ever post anything to codeshare on this topic? We are sorely in need of a parsing engine that can handle v10 config files. We wrote an engine ourselves that worked for v9 which severely broke when we moved to v10 yesterday.

     

  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    Hi SMP,

     

     

    I haven't had a need to try it yet, but I would imagine that tmsh should provide more functionality for displaying the bigip.conf objects in a systematic way.

     

     

    What specifically are you trying to update for v10 that you had for v9?

     

     

    Aaron
  • The automation we developed works with a local copy of bigip.conf, so tmsh isn't an option.

     

     

    The format of the config file changed (obviously) between 9 and 10. The v10 config file uses nested profile and iRule directives in the definition of the virtual server definition, versus what the v9 did which was to reference these items and define them elsewhere. We basically deconstruct a virtual, and use the data to populate an application a database. We had to write our own parser for v9, and we were hoping someone had already done this for v10. It doesn't look like that that is the case though...
  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    I haven't seen anything in the wild for parsing 10.x bigip.conf's.

     

     

    If you could get live access to a BIG-IP running the config (or maybe a VE running 10.1.0?), I'd think iControl would be the best way to handle a requirement like this as you wouldn't be subject to configuration syntax changes. Else, you could take hwidjaja's method for parsing objects with nested braces like I did for Perl and use that to rework your 9.x parsing tool.

     

     

    Aaron
  • Just FYI, tmsh or icontrol will need to be part of your long range plans as bigpipe will not be around forever.
  • hoolio's avatar
    hoolio
    Icon for Cirrostratus rankCirrostratus
    Posted By hwidjaja on 01/02/2010 09:35 AM

    It seems like this awk one-liner is better than my previous one:

    awk 'BEGIN {RS="\n}";FS=RS} /rules SNAT2VIP/ {print $1"\n}";} ' /config/bigip.conf 

    Hey Humphrey,

     

     

    I ended up using this one liner to extract all of the monitors from the bigip.conf for a 9.x customer working on their upgrades. Just wanted to say thanks again for this gem.

     

     

    In case anyone is looking for an ugly script to update 9.x monitors to make them 10.x compatible, here you go:

     

    b save; awk 'BEGIN {RS="\n}";FS=RS} /monitor .*defaults from/ {print $1"\n}";}' /config/bigip.conf | perl -pe 's/(? /var/tmp/monitors.scf; b merge /var/tmp/monitors.scf; b save

     

    It basically replaces \n's that don't have a preceding \r with \r\n and appends up to two \r\n's to the end of all send strings if they're not there already. The input is the bigip.conf. The output is written to an SCF file which is then imported back into memory and saved to disk with a 'b save'.

     

     

    It's not thoroughly tested (yet), but should save a lot of time manually updating monitor send strings.

     

     

    Aaron