Read dynamic file in LB internal datagroup

[ksrmoor]

We have list of whitelisted IP stored in remote .csv file, file gets updated ever 5 hours with new set of IP.

Currently we read file in perl and update internal data group. In LB using iRule we allow or block IP in making succesful connection

to pool members by reading data group.

During the time of reading and updating data group, connections are getting dropped around 5 to 10 mins

We are replacing content of datagroup and adding each entry, this may not be best approach

but there is no easy way to retrieve existing datagroup values and compare with .csv and do intelligent diff

No of IP has grown from 10 to 1400 IPs and it might go to 5000

Since it is working for a while, facing issue recently probably because of scale, it will be difficult to change entire logic.

but can make minor changes to existing logic/script

#File content is delimited and stored in bcontent

my @bcontent = split("," , $tmp);

#print @bcontent;

my $entry = '';

my $first_cm = 1;

foreach $entry(@bcontent) {

  $entry =~ s/^\s+|\s+$//g;

  if ( $first_cm ) {

    $cmd = "$tmcom modify ltm data-group internal $BLDG records replace-all-with { $entry }";

      system("$cmd");

    }

  } else {

    $cmd = "$tmcom modify ltm data-group internal $BLDG records add { $entry }";

      system("$cmd");

    }

  }

[ksrmoor]

Would welcome any alternative suggestion too

[Niels_van_Sluis]

Another approach that might work for you is by using a table instead of a datagroup. I would think the table itself will always be available by the VS that is looking up table entries, while enties in the table itself can added, deleted and modified. See: https://clouddocs.f5.com/api/irules/table.html