Custom alerts in 9.4.4 do not send email notifications

Greetings!

I'm banging my head against the wall..

custom alert from /conf/user_alert.conf does not send email notification when pool member is marked down by monitor.

/config/user_alert.conf looks like this:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_251 "Pool member 192.168.253.251:80 monitor status down." {

email toaddress="xxx@xxx.com"

fromaddress="root-bip1@xxx.com"

body="*** Pool member 192.168.253.251 marked DOWN by monitor ***"

}

when I use logger -p local0.info "Pool member 192.168.253.251:80 monitor status down.", i get email notification from the BIG-IP..

when pool member is marked down by monitor and a new log record appears in /var/log/ltm file, no email alert is sent...

for example, /var/log/ltm has two log records:

Mar 31 13:58:28 bip root: Pool member 192.168.253.251:80 monitor status down.

Mar 31 13:58:49 bip mcpd[1772]: 01070638:5: Pool member 192.168.253.251:80 monitor status down.

first one was created by me using logger. Second one was recorded when actual monitor marked pool member down.

I got email message ONLY after using logger.

Please advice!

thanks!

management

monitoring

hoolio
Cirrostratus
Apr 02, 2009
A stab or two in the dark:

Can you add wildcards before and after the string int eh user_alert.conf?

.*Pool member 192.168.253.251:80 monitor status down.*

Else, maybe it's that the monitor log entry has the error code in it. If you add the error code to your logger test string, does it still work?

Aaron
Dmitry_Kuzura
Nimbostratus
Apr 02, 2009
.*Pool member 192.168.253.251:80 monitor status down.*

this one did not work.

Adding an error code in logger command caused it not to sent email alert.

I started deleting error code characters in the logger command one by one starting with the last character to figure out when it will start sending email alerts. Turns out that when error code is 01070638 email alert is sent. To me it means that something disagrees with having ":" in the error code when you want to send emails for custom alerts.

any thoughts?

thank you!
hoolio
Cirrostratus
Apr 02, 2009
I think it's matching the default alert definition first. I'm not sure what you can do about this though. Anyone have ideas?

Aaron
hwidjaja_37598
Altostratus
Apr 03, 2009
It seems like the alertd uses /var/run/bigip_error_maps.dat file when it receives log message that matches an alert code (on this case, it's 01070638). The alertd process then performs the alert actions specified for that alert name in the /etc/alertd/alert.conf file.

Check this out: SOL6420 Click here

The setting would be:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {

email toaddress="xxx@xxx.com"

fromaddress="root-bip1@xxx.com"

body="*** A pool member marked DOWN by monitor ***"

}

I found the Alert ID (BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS) from:

ltm grep 01070638 /var/run/bigip_error_maps.dat

0 LOG_NOTICE 01070638 BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS "Pool member %s:%u monitor status %s."

It could be changed from /etc/alertd/bigip_mcpd_error_maps.h file

I could not find a way to only send the email for a specific pool member. As an alternative, you can use syslog-ng to run log2mail when it sees this message. For more information, go to syslog-ng and log2mail manual:

man log2mail
Dmitry_Kuzura
Nimbostratus
Apr 03, 2009
Thanks!

The idea was, and still is, to send email alerts out only when SPECIFIC pool member changes status..And send those email alerts only to SPECIFIC people who are responcible for those SPECIFIC pools..

Email notifications work great when using just a default alert.conf file..But when using alert.conf, email alerts were sent every time when ANY pool member was changing status..

what's interesting, i have not had this issue when using 9.3.1... 9.4.4 and 9.4.6 do not allow me to use custom alerts..
dennypayne
Employee
Apr 03, 2009
FWIW, I am able to do custom alerts in 9.4.6 that send out SNMP traps.

alert EXPIRING_SSL_CERT "Certificate (.*) in file (.*) will expire on (.*)" { snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.500" }

Not sure why that works and an email version wouldn't...

Denny
hwidjaja_37598
Altostratus
Apr 04, 2009
Well done dpkuzura ...

Try using mail command or create a script to modify the subject or the body before sending it out (e.g. using "sendmail -t")

destination mail_pm_192_168_253_251_80 { program("/bin/mail -s '192_168_253_251_80 is down' user1@domain.com"); };

I still can't find a way to use user_alert.conf, anyone can help?
hwidjaja_37598
Altostratus
Apr 06, 2009
Posted By dennypayne on 04/03/2009 7:52 AM

FWIW, I am able to do custom alerts in 9.4.6 that send out SNMP traps.

alert EXPIRING_SSL_CERT "Certificate (.*) in file (.*) will expire on (.*)" { snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.500" }

Not sure why that works and an email version wouldn't...

Denny

I guess it's due to there is no alert code for EXPIRING_SSL_CERT or the log message. But when alertd receives input from syslog-ng utility that matches an alert code, the alert code is mapped to the alert name (taken from SOL6420).

ltm grep EXPIRING_SSL_CERT /var/run/bigip_error_maps.dat ltm

CMIIW please ...
Dmitry_Kuzura
Nimbostratus
Apr 06, 2009
hwidjaja,

once again, thank you for your advice!

Stuff works!

So, does anyone know if there are any plans to fix this in the next hotfix?

thanks, everyone!!!

dk
dennypayne
Employee
Apr 06, 2009
Posted By hwidjaja on 04/05/2009 7:18 PM

But when alertd receives input from syslog-ng utility that matches an alert code, the alert code is mapped to the alert name (taken from SOL6420).

Ah, I was looking at his BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_251 name and figured that wouldn't match with the 251 added...so maybe it's only matching over some X number of characters in the name?

Denny