NGINX App Protect v5 Signature Notifications

When working with NAP (NGINX App Protect) you don't have an easy way of knowing when any of the signatures are updated.

As an old BigIP guy I find that rather strange. Here you have build-in automatic updates and notifications.

Unfortunately there isn't any API's you can probe which would have been the best way of doing it. Hopefully it will come one day.

 

However, "friction" and "hard" will not keep me from finding a solution 😆

 

I have previously made a solution for NAPv4 and I have tried mentally to get me going on a NAPv5 version. The reason for the delay is in the different way NAPv4 and NAPv5 are designed. Where NAPv4 is one module loaded in NGINX, NAPv5 is completely detached from NGINX (well almost, you still need to load a small module to get the traffic from NGINX to NAP) and only works with containers.

NAPv5 has moved the signature "storage" from the actual host it is running on (e.g. an installed package) to the policy. This has the consequence that finding a valid "source of truth", for the latest signature versions, is not as simple as building a new image and see which versions got installed.

There are very good reasons for this design that I will come back to later.

When you fire up NAPv5 you got three containers for the data plane (NGINX, waf-enforcer and waf-config-mgr) and one for the "control plane" (waf-compiler). For this solution the "control plane" is the useful one. It isn't really a control plane but it gives a nice picture of how it is detached from the actual processing of traffic.

When you update your signatures you are actually doing it through the waf-compiler. The waf-compiler is a container hosting the actual signature databases and every time a new verison is released you need to rebuild this container and compile your policies into a new version and reload NGINX.

And this is what I take advantage of when I look for signature updates. It has the upside that you only need the waf-compiler to get the information you need. My solution will take care of the entire process and make sure that you are always running with the latest signatures.

Back to the reason why the split of functions is a very good thing. When you build a new version of the NGINX image and deploy it into production, NAP needs to compile the policies as they load. During the compilation NGINX is not moving any traffic! This becomes a annoying problem even when you have a low number of policies. I have installations where it takes 5 to 10 minutes from deployment of the new image until it starts moving traffic. That is a crazy long time when you are used to working with micro-services and expect everything to flip within seconds. If you have your NAPv4 hooked up to a NGINX Instance Manager (NIM) the problem is somewhat mitigated as NIM will compile the policies before sending them to the gateways. NIM is not a nimble piece of software so it doesn't always fit into the environment.

 

And now here is my hack to the notification problem:

The solution consist of two bash scripts and one html template. The template is used when sending a notification mail. I wanted it to be pretty and that was easiest with html. Strictly speaking you could do with just a simple text based mail.

Save all three in the same directory.

The main script is called "waf_policy_auto_compile.sh"and is the one you put into crontab.

The main script will build a new waf-compiler image and compile a test policy. The outcome of that is information about what versions are the newest. It will then extract versions from an old policy and simply see if any of the versions differ. For this to work you need to have an uncompiled policy (you can just use the default one) and a compiled version of it ready beforehand.

 

When a diff has been identified the notification logic is executed and a second script is called: "compile_waf_policies.sh".

It basically just trawls through the directory of you policies and logging profiles and compiles a new version of them all. It is not necessary to recompile the logging profiles, so this will probably change in the next version.

 

As the compilation completes the main script will nudge NGINX to reload thus implement all the new versions.

You can run "waf_policy_auto_compile.sh" with a verbose flag (-v) and a debug flag (-d). The verbose flag is intended to be used when you run it on a terminal and want the information displayed there. Debug is, well, for debug 😝 

The construction of the scripts are based on my own needs but they should be easy to adjust for any need.

I will be happy for any feedback, so please don't hold back 😄

 

version_report_template.html:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>WAF Policy Version Report</title>
  <style>
    body { font-family: system-ui, sans-serif; }
    .ok { color: #28a745; font-weight: bold; }
    .warn { color: #f0ad4e; font-weight: bold; }
    .section { margin-bottom: 1.2em; }
    .label { font-weight: bold; }
  </style>
</head>
<body>
  <h2>WAF Policy Version Report</h2>
  <div class="section">
    <div class="label">Attack Signatures:</div>
    <div>Current: <span>{{ATTACK_OLD}}</span></div>
    <div>New: <span>{{ATTACK_NEW}}</span></div>
    <div>Status: <span class="{{ATTACK_CLASS}}">{{ATTACK_STATUS}}</span></div>
  </div>
  <div class="section">
    <div class="label">Bot Signatures:</div>
    <div>Current: <span>{{BOT_OLD}}</span></div>
    <div>New: <span>{{BOT_NEW}}</span></div>
    <div>Status: <span class="{{BOT_CLASS}}">{{BOT_STATUS}}</span></div>
  </div>
  <div class="section">
    <div class="label">Threat Campaigns:</div>
    <div>Current: <span>{{THREAT_OLD}}</span></div>
    <div>New: <span>{{THREAT_NEW}}</span></div>
    <div>Status: <span class="{{THREAT_CLASS}}">{{THREAT_STATUS}}</span></div>
  </div>
  <div>Run completed: {{RUN_DATETIME}}</div>
</body>
</html>

compile_waf_policies.sh:

#!/bin/bash
# ==============================================================================
# Script Name: compile_waf_policies.sh
#
# Description:
#   Compiles:
#     1. WAF policy JSON files from the 'policies' directory
#     2. WAF logging JSON files from the 'logging' directory
#   using the 'waf-compiler-latest:custom' Docker image. Output goes to
#   '/opt/napv5/app_protect_etc_config' where NGINX and waf-config-mgr
#   can reach them.
#
# Requirements:
#   - Docker installed and accessible
#   - Docker image 'waf-compiler-latest:custom' present locally
#
# Usage:
#   ./compile_waf_policies.sh
# ==============================================================================

set -euo pipefail
IFS=$'\n\t'
SECONDS=0  # Track total execution time

# ========================
# CONFIGURABLE VARIABLES
# ========================

BASE_DIR="/root/napv5/waf-compiler"
OUTPUT_DIR="/opt/napv5/app_protect_etc_config"

POLICY_INPUT_DIR="$BASE_DIR/policies"
POLICY_OUTPUT_DIR="$OUTPUT_DIR"

LOGGING_INPUT_DIR="$BASE_DIR/logging"
LOGGING_OUTPUT_DIR="$OUTPUT_DIR"

GLOBAL_SETTINGS="$BASE_DIR/global_settings.json"
DOCKER_IMAGE="waf-compiler-latest:custom"

# ========================
# VALIDATION
# ========================

echo "🔧 Validating paths..."

[[ -d "$POLICY_INPUT_DIR" ]] || { echo "❌ Error: Policy input directory '$POLICY_INPUT_DIR' does not exist."; exit 1; }
[[ -f "$GLOBAL_SETTINGS" ]] || { echo "❌ Error: Global settings file '$GLOBAL_SETTINGS' not found."; exit 1; }

mkdir -p "$POLICY_OUTPUT_DIR"
mkdir -p "$LOGGING_OUTPUT_DIR"

# ========================
# POLICY COMPILATION
# ========================

echo "📦 Compiling WAF policies from: $POLICY_INPUT_DIR"
for POLICY_FILE in "$POLICY_INPUT_DIR"/*.json; do
  [[ -f "$POLICY_FILE" ]] || continue

  BASENAME=$(basename "$POLICY_FILE" .json)
  OUTPUT_FILE="$POLICY_OUTPUT_DIR/${BASENAME}.tgz"

  echo "⚙️  [Policy] Compiling $(basename "$POLICY_FILE") -> $(basename "$OUTPUT_FILE")"

  docker run --rm \
    -v "$POLICY_INPUT_DIR":"$POLICY_INPUT_DIR" \
    -v "$POLICY_OUTPUT_DIR":"$POLICY_OUTPUT_DIR" \
    -v "$(dirname "$GLOBAL_SETTINGS")":"$(dirname "$GLOBAL_SETTINGS")" \
    "$DOCKER_IMAGE" \
    -g "$GLOBAL_SETTINGS" \
    -p "$POLICY_FILE" \
    -o "$OUTPUT_FILE"
done

# ========================
# LOGGING COMPILATION
# ========================

echo "📝 Compiling WAF logging configs from: $LOGGING_INPUT_DIR"
if [[ -d "$LOGGING_INPUT_DIR" ]]; then
  for LOG_FILE in "$LOGGING_INPUT_DIR"/*.json; do
    [[ -f "$LOG_FILE" ]] || continue

    BASENAME=$(basename "$LOG_FILE" .json)
    OUTPUT_FILE="$LOGGING_OUTPUT_DIR/${BASENAME}.tgz"

    echo "⚙️  [Logging] Compiling $(basename "$LOG_FILE") -> $(basename "$OUTPUT_FILE")"

    docker run --rm \
      -v "$LOGGING_INPUT_DIR":"$LOGGING_INPUT_DIR" \
      -v "$LOGGING_OUTPUT_DIR":"$LOGGING_OUTPUT_DIR" \
      "$DOCKER_IMAGE" \
      -l "$LOG_FILE" \
      -o "$OUTPUT_FILE"
  done
else
  echo "⚠️  Skipping logging config compilation: directory '$LOGGING_INPUT_DIR' does not exist."
fi

# ========================
# COMPLETION MESSAGE
# ========================

RUNTIME=$SECONDS
printf "\n✅ Compilation complete.\n"
echo "  - Policies output: $POLICY_OUTPUT_DIR"
echo "  - Logging output: $LOGGING_OUTPUT_DIR"
echo
printf "⏱️  Total time taken: %02d minutes %02d seconds\n" $((RUNTIME / 60)) $((RUNTIME % 60))
echo

waf_policy_auto_compile.sh:

#!/bin/bash

###############################################################################
# waf_policy_auto_compile.sh
#
# - Only prints colorized summary output to terminal if -v/--verbose is used
# - Mails a styled HTML report using a template, substituting version numbers/status/colors
# - Debug output (step_log) only to syslog if -d/--debug is used
# - Otherwise: completely silent except for errors
# - All main blocks are modularized in functions
###############################################################################

set -euo pipefail
IFS=$'\n\t'

# ===== CONFIGURABLE VARIABLES =====

WORKROOT="/root/napv5"
WORKDIR="$WORKROOT/waf-compiler"
DOCKERFILE="$WORKDIR/Dockerfile"
BUNDLE_DIR="$WORKDIR/test"
NEW_BUNDLE="$BUNDLE_DIR/test_new.tgz"
OLD_BUNDLE="$BUNDLE_DIR/test_old.tgz"
NEW_META="$BUNDLE_DIR/test_new_meta.json"
COMPILER_IMAGE="waf-compiler-latest:custom"
EMAIL_RECIPIENT="example@example.com"
EMAIL_SUBJECT="WAF Compiler Update Notification"
NGINX_RELOAD_CMD="docker exec nginx-plus nginx -s reload"
HTML_TEMPLATE="$WORKDIR/version_report_template.html"
HTML_REPORT="$WORKDIR/version_report.html"
VERBOSE=0
DEBUG=0

# ===== DEBUG AND ERROR LOGGING =====

exec 2> >(tee -a /tmp/waf_policy_auto_compile_error.log | /usr/bin/logger -t waf_policy_auto_compile_error)

step_log() {
  if [ "$DEBUG" -eq 1 ]; then
    echo "DEBUG: $1" | /usr/bin/logger -t waf_policy_auto_compile
  fi
}

# ===== ARGUMENT PARSING =====

while [[ $# -gt 0 ]]; do
  case "$1" in
    -v|--verbose)
      VERBOSE=1
      shift
      ;;
    -d|--debug)
      DEBUG=1
      echo "Debug log can be found in the syslog..."
      shift
      ;;
    -*)
      echo "Unknown option: $1" >&2
      exit 1
      ;;
    *)
      shift
      ;;
  esac
done

# ----- LOG INITIAL ENVIRONMENT IF DEBUG -----
step_log "waf_policy_auto_compile starting (PID $$)"
step_log "Script PATH: $PATH"
step_log "which docker: $(which docker 2>/dev/null)"
step_log "which jq: $(which jq 2>/dev/null)"

# ===== COLOR DEFINITIONS =====

color_reset="\033[0m"
color_green="\033[1;32m"
color_yellow="\033[1;33m"

# ===== LOGGING FUNCTIONS =====

log() {
  # Only log to terminal if VERBOSE is enabled
  if [ "$VERBOSE" -eq 1 ]; then
    echo "[$(date --iso-8601=seconds)] $*"
  fi
}

# ===== HTML REPORT GENERATOR =====

generate_html_report() {
  local attack_old="$1"
  local attack_new="$2"
  local attack_status="$3"
  local attack_class="$4"
  local bot_old="$5"
  local bot_new="$6"
  local bot_status="$7"
  local bot_class="$8"
  local threat_old="$9"
  local threat_new="${10}"
  local threat_status="${11}"
  local threat_class="${12}"
  local datetime
  datetime=$(date --iso-8601=seconds)
  cp "$HTML_TEMPLATE" "$HTML_REPORT"

  sed -i "s|{{ATTACK_OLD}}|$attack_old|g" "$HTML_REPORT"
  sed -i "s|{{ATTACK_NEW}}|$attack_new|g" "$HTML_REPORT"
  sed -i "s|{{ATTACK_STATUS}}|$attack_status|g" "$HTML_REPORT"
  sed -i "s|{{ATTACK_CLASS}}|$attack_class|g" "$HTML_REPORT"
  sed -i "s|{{BOT_OLD}}|$bot_old|g" "$HTML_REPORT"
  sed -i "s|{{BOT_NEW}}|$bot_new|g" "$HTML_REPORT"
  sed -i "s|{{BOT_STATUS}}|$bot_status|g" "$HTML_REPORT"
  sed -i "s|{{BOT_CLASS}}|$bot_class|g" "$HTML_REPORT"
  sed -i "s|{{THREAT_OLD}}|$threat_old|g" "$HTML_REPORT"
  sed -i "s|{{THREAT_NEW}}|$threat_new|g" "$HTML_REPORT"
  sed -i "s|{{THREAT_STATUS}}|$threat_status|g" "$HTML_REPORT"
  sed -i "s|{{THREAT_CLASS}}|$threat_class|g" "$HTML_REPORT"
  sed -i "s|{{RUN_DATETIME}}|$datetime|g" "$HTML_REPORT"
  
}

# ===== BUILD COMPILER IMAGE =====

build_compiler() {
  step_log "about to build_compiler"
  docker build --no-cache --platform linux/amd64 \
    --secret id=nginx-crt,src="$WORKROOT/nginx-repo.crt" \
    --secret id=nginx-key,src="$WORKROOT/nginx-repo.key" \
    -t "$COMPILER_IMAGE" \
    -f "$DOCKERFILE" "$WORKDIR" > "$WORKDIR/waf_compiler_build.log" 2>&1 || {
      echo "ERROR: docker build failed. Dumping build log:" | /usr/bin/logger -t waf_policy_auto_compile_error
      cat "$WORKDIR/waf_compiler_build.log" | /usr/bin/logger -t waf_policy_auto_compile_error
      exit 1
  }
  step_log "after build_compiler"
}

# ===== COMPILE TEST POLICY =====

compile_test_policy() {
  step_log "about to compile_test_policy"
  docker run --rm -v "$BUNDLE_DIR:/bundle" "$COMPILER_IMAGE" \
    -p /bundle/test.json -o /bundle/test_new.tgz > "$NEW_META"
  step_log "after compile_test_policy"
  if [ -f "$NEW_META" ]; then
    step_log "$(cat "$NEW_META")"
  else
    step_log "NEW_META does not exist"
  fi
}

# ===== CHECK OLD_BUNDLE =====

check_old_bundle() {
  step_log "about to check OLD_BUNDLE"
  if [ -f "$OLD_BUNDLE" ]; then
    step_log "$(ls -l "$OLD_BUNDLE")"
  else
    step_log "OLD_BUNDLE does not exist"
  fi
}

# ===== GET NEW VERSIONS FUNCTION =====

get_new_versions() {
  jq -r '
    {
      "attack": .attack_signatures_package.version,
      "bot": .bot_signatures_package.version,
      "threat": .threat_campaigns_package.version
    }' "$NEW_META"
}

# ===== VERSION EXTRACTION FROM OLD BUNDLE =====

extract_bundle_versions() {
  docker run --rm -v "$BUNDLE_DIR:/bundle" "$COMPILER_IMAGE" \
    -dump -bundle "/bundle/test_old.tgz"
}

extract_versions_from_dump() {
  extract_bundle_versions | awk '
    BEGIN { print "{" }
    /attack-signatures:/     { in_attack=1; next }
    /bot-signatures:/        { in_bot=1; next }
    /threat-campaigns:/      { in_threat=1; next }
    in_attack && /version:/  {
      gsub("version: ", "")
      printf "\"attack\":\"%s\",\n", $1
      in_attack=0
    }
    in_bot && /version:/     {
      gsub("version: ", "")
      printf "\"bot\":\"%s\",\n", $1
      in_bot=0
    }
    in_threat && /version:/  {
      gsub("version: ", "")
      printf "\"threat\":\"%s\"\n", $1
      in_threat=0
    }
    END { print "}" }
  '
}

get_old_versions() {
  extract_versions_from_dump
}

# ===== GET & PRINT VERSIONS =====

get_versions() {
  step_log "about to get_new_versions"
  new_versions=$(get_new_versions)
  step_log "new_versions: $new_versions"
  step_log "after get_new_versions"

  step_log "about to get_old_versions"
  old_versions=$(get_old_versions)
  step_log "old_versions: $old_versions"
  step_log "after get_old_versions"
}

# ===== VERSION COMPARISON =====

compare_versions() {
  step_log "compare_versions start"
  attack_old=$(echo "$old_versions" | jq -r .attack)
  attack_new=$(echo "$new_versions" | jq -r .attack)
  bot_old=$(echo "$old_versions" | jq -r .bot)
  bot_new=$(echo "$new_versions" | jq -r .bot)
  threat_old=$(echo "$old_versions" | jq -r .threat)
  threat_new=$(echo "$new_versions" | jq -r .threat)

  attack_status=$([[ "$attack_old" != "$attack_new" ]] && echo "Updated" || echo "No Change")
  bot_status=$([[ "$bot_old" != "$bot_new" ]] && echo "Updated" || echo "No Change")
  threat_status=$([[ "$threat_old" != "$threat_new" ]] && echo "Updated" || echo "No Change")

  attack_class=$([[ "$attack_status" == "Updated" ]] && echo "warn" || echo "ok")
  bot_class=$([[ "$bot_status" == "Updated" ]] && echo "warn" || echo "ok")
  threat_class=$([[ "$threat_status" == "Updated" ]] && echo "warn" || echo "ok")

  echo "Attack:$attack_status Bot:$bot_status Threat:$threat_status" > "$WORKDIR/status_flags.txt"

  [[ "$attack_status" == "Updated" ]] && attack_status_colored="${color_yellow}*** Updated ***${color_reset}" || attack_status_colored="${color_green}No Change${color_reset}"
  [[ "$bot_status" == "Updated" ]] && bot_status_colored="${color_yellow}*** Updated ***${color_reset}" || bot_status_colored="${color_green}No Change${color_reset}"
  [[ "$threat_status" == "Updated" ]] && threat_status_colored="${color_yellow}*** Updated ***${color_reset}" || threat_status_colored="${color_green}No Change${color_reset}"

  {
    echo -e "Version comparison for container \033[1mNAPv5\033[0m:\n"
    echo -e "Attack Signatures:"
    echo -e "  Current Version: $attack_old"
    echo -e "  New Version: $attack_new"
    echo -e "  Status: $attack_status_colored\n"
    echo -e "Threat Campaigns:"
    echo -e "  Current Version: $threat_old"
    echo -e "  New Version: $threat_new"
    echo -e "  Status: $threat_status_colored\n"
    echo -e "Bot Signatures:"
    echo -e "  Current Version: $bot_old"
    echo -e "  New Version: $bot_new"
    echo -e "  Status: $bot_status_colored"
  } > "$WORKDIR/version_report.ansi"

  sed 's/\x1B\[[0-9;]*[mK]//g' "$WORKDIR/version_report.ansi" > "$WORKDIR/version_report.txt"

  step_log "Calling log_versions_syslog"
  log_versions_syslog "$attack_old" "$attack_new" "$attack_status" "$attack_class" \
    "$bot_old" "$bot_new" "$bot_status" "$bot_class" \
    "$threat_old" "$threat_new" "$threat_status" "$threat_class"
  step_log "compare_versions finished"
}

# ===== SYSLOG VERSION LOGGING and HTML REPORT GEN =====

log_versions_syslog() {
  # Args:
  #   1-attack_old 2-attack_new 3-attack_status 4-attack_class
  #   5-bot_old    6-bot_new    7-bot_status    8-bot_class
  #   9-threat_old 10-threat_new 11-threat_status 12-threat_class

  local msg
  msg="AttackSig (current: $1, latest: $2), BotSig (current: $5, latest: $6), ThreatCamp (current: $9, latest: $10)"
  /usr/bin/logger -t waf_policy_auto_compile "$msg"
  # Also print to terminal if VERBOSE is enabled
  if [ "$VERBOSE" -eq 1 ]; then
    echo "$msg"
  fi

  # Always (re)generate HTML for the mail at this point
  generate_html_report "$@"
}

# ===== RESPONSE ACTIONS =====

compile_all_policies() {
  log "Change detected – compiling all policies..."
  if [ "$VERBOSE" -eq 1 ]; then
    "$WORKDIR/compile_waf_policies.sh"
  else
    "$WORKDIR/compile_waf_policies.sh" > /dev/null 2>&1
  fi
}

reload_nginx() {
  log "Reloading NGINX..."
  eval "$NGINX_RELOAD_CMD"
}

rotate_bundles() {
  log "Archiving new test bundle as old..."
  mv "$NEW_BUNDLE" "$OLD_BUNDLE"
  rm -f "$NEW_META"
}

send_report_email() {
  local html_report="$1"
  mail -s "$EMAIL_SUBJECT" -a "Content-Type: text/html" "$EMAIL_RECIPIENT" < "$html_report"
}

# ===== MAIN LOGIC =====

main() {
  build_compiler
  compile_test_policy
  check_old_bundle
  get_versions
  compare_versions

  if [[ "$VERBOSE" -eq 1 ]]; then
    cat "$WORKDIR/version_report.ansi"
  fi

  if grep -q "Updated" "$WORKDIR/status_flags.txt"; then
    if [[ "$VERBOSE" -eq 1 ]]; then
      echo "Detected updates. Recompiling policies, reloading NGINX, sending report."
    fi
    compile_all_policies
    reload_nginx
    rotate_bundles
    send_report_email "$HTML_REPORT"
  else
    log "No changes detected – nothing to do."
  fi

  log "Done."
}

main "$@"

And should be it.