CSV to Address External Datagroup File

JRahm · ‎17-Mar-2023

Code is community submitted, community supported, and recognized as ‘Use At Your Own Risk’.

Short Description

I was today years old when I found out you could upload external data-group files to the GUI.

I couldn't figure out the format, however, until @John_Alam pointed me to the K73862425 knowledge article.

Problem solved by this Code Snippet

This script takes a CSV file of IPs and/or networks keys, with or without values, and transforms it into an acceptable format for upload to the GUI.

How to use this Code Snippet

python dg_format.py

Code Snippet Meta Information

Version: .1
Coding Language: python 3.11

Full Code Snippet

"""
Source File CSV Contents:
10.10.10.10,
10.10.10.11,hi there
10.10.11.0/24,
10.10.12.0/24,/url/path/here
10.10.13.0/24,

Output File Txt Contents:
host 10.10.10.10,
host 10.10.10.11 := hi there,
network 10.10.11.0/24,
network 10.10.12.0/24 := /url/path/here,
network 10.10.13.0/24,
"""

import csv

with open('dg_ipaddrs.csv', mode='r', encoding='utf-8-sig') as f:
    reader = csv.reader(f)
    ipaddrs = list(reader)


f_out = open('dg_formatted.txt', 'w')

for addr in ipaddrs:
    print(addr)
    if "/" in addr[0] or "mask" in addr[0] or "prefexlen" in addr[0]:
        if addr[1] == '':
            f_out.write(f'network {addr[0]},\n')
        else:
            value = addr[1].split('\n')[0]
            f_out.write(f'network {addr[0]} := {value},\n')
    else:
        if addr[1] == '':
            f_out.write(f'host {addr[0]},\n')
        else:
            value = addr[1].split('\n')[0]
            f_out.write(f'host {addr[0]} := {value},\n')

f_out.close()

lnxgeek · ‎19-Mar-2023

Thanks Jason! You always seem to find a smart fix for an annoying problem 🙂

For the minority which doesn't master Python yet (like me 😋) I have concocted a Bash version of it:

#!/bin/bash

# read IP addresses from CSV file into array
mapfile -t ipaddrs < dg_ipaddrs.csv

# loop through array and format each IP address
for addr in "${ipaddrs[@]}"; do
  # remove any leading/trailing whitespace
  addr="$(echo -e "${addr}" | tr -d '[:space:]')"

  # split line into fields on comma delimiter
  IFS=',' read -ra fields <<< "$addr"

  # check if first field contains a network mask or prefix length. The ${fields[0],,} syntax is used to convert the contents of the fields[0] variable to lowercase.
  if [[ "${fields[0]}" == */* ]] || [[ "${fields[0],,}" == *"mask"* ]] || [[ "${fields[0],,}" == *"prefixlen"* ]]; then
    # check if second field is empty
    if [[ -z "${fields[1]}" ]]; then
      echo "network ${fields[0]},"
    else
      value="${fields[1]}"
      echo "network ${fields[0]} := ${value},"
    fi
  else
    # check if second field is empty
    if [[ -z "${fields[1]}" ]]; then
      echo "host ${fields[0]},"
    else
      value="${fields[1]}"
      echo "host ${fields[0]} := ${value},"
    fi
  fi
done > dg_formatted.txt

JRahm · ‎20-Mar-2023

Nice work, @lnxgeek ... love seeing solutions in multiple languages.

Nikoolayy1 · ‎27-Mar-2023

I do not know if there is an article but with icall script and getting the data from url like the one for office 365 and scheduling the icall to run every hour will make the solution dynamic:

https://github.com/f5devcentral/f5-office365-ip-url-automation

edit:

Also external automation system can use the API to upload the cvs file and then an API for the icall script to trigger the python script as a way to update the files:

https://my.f5.com/manage/s/article/K41763344

https://clouddocs.f5.com/api/icontrol-rest/APIRef_tm_sys_icall.html

DevCentral