alertd to trigger script after seeing specific syslog n times

So you need to start with something like

timelapseinlog.pl

A utility script that can carry out a time-based filter on log files with various timestamp formats

#!/usr/bin/perl -ws
# This script parse logfiles for a specific period of time
 
sub usage {
    printf "Usage: %s -s=<start time> [-e=<end time>] <logfile>\n";
    die $_[0] if $_[0];
    exit 0;
}
 
use Date::Parse;
 
usage "No start time submited" unless $s;
my $startim=str2time($s) or die;
 
my $endtim=str2time($e) if $e;
$endtim=time() unless $e;
 
usage "Logfile not submited" unless $ARGV[0];
open my $in, "<" . $ARGV[0] or usage "Can't open '$ARGV[0]' for reading";
$_=<$in>;
exit unless $_; # empty file
# Determining regular expression, depending on log format
my $logre=qr{^(\S{3}\s+\d{1,2}\s+(\d{2}:){2}\d+)};
$logre=qr{^[^\[]*\[(\d+/\S+/(\d+:){3}\d+\s\+\d+)\]} unless /$logre/;
 
while (<$in>) {
    /$logre/ && do {
        my $ltim=str2time($1);
        print if $endtim >= $ltim && $ltim >= $startim;
    };
};

user_alert.conf

this is the custom user alert that looks for logged events

alert syslog_status_trigger_action "your critical message" {
  exec command="/shared/trigger_action.sh";
  exec command="/bin/logger -p local0.info 'action_triggered'"
}
 

Then in your bash script trigger_action.sh:

#!/bin/bash
 
TRIGGER_LEVEL=3
if [ ! -e /tmp/trigger_action_count ]; then
 # filter ltm1 and ltm into a single file so we don't miss events around log-rotation 
  grep 'trigger_action' /var/log/ltm.1 > /tmp/trigger_action_count
  grep 'trigger_action' /var/log/ltm >> /tmp/trigger_action_count
  # set your time period below - I have chosen 10 minutes
  COUNT=$(/shared/timelapseinlog.pl -s=`/bin/date -d'now-10 minutes' +%H:%M` /tmp/trigger_action_count | wc -l)
  rm -f /tmp/trigger_action_count
 
  if [ -p $NP ] && [ "$COUNT" -gt "$TRIGGER_LEVEL" ]; then
      # do something here
  fi
fi

Note that there are some things that will not run in the alertd SELinux context (tcpdump, for example).

You may need to run a script on startup that waits on a named pipe, and then the above trigger script signals using the named pipe.

You must be able to log in to the computer running your Orion server. Click Start > All Programs > SolarWinds Orion > Syslog Viewer. Click File > Settings liteblue.