email alert notification not working when member came up again

[Abed_AL-R]

Hello guys

i've configured this:

https://support.f5.com/csp/article/K3667

https://support.f5.com/csp/article/K59616664

I receive alerts when node goes down, but not when going up again

What could be the problem?

Oct  3 23:02:57 my.website.com notice mcpd[8459]: 01070640:5: Node /Common/172.17.70.18 address 172.17.70.18 monitor status down. [ /Common/icmp: down ]  [ was up for 24hrs:2mins:5sec ]
Oct  3 23:02:57 my.website.com notice mcpd[8459]: 01070640:5: Node /Common/172.17.70.19 address 172.17.70.19 monitor status down. [ /Common/icmp: down ]  [ was up for 24hrs:2mins:6sec ]
Oct  3 23:02:57 my.website.com notice mcpd[8459]: 01071682:5: SNMP_TRAP: Virtual /Common/dev-myweb has become unavailable
Oct  3 23:02:59 my.website.com notice mcpd[8459]: 01070728:5: Node /Common/172.17.70.18 address 172.17.70.18 monitor status up. [ /Common/icmp: up ]  [ was down for 0hr:0min:2sec ]
Oct  3 23:03:02 my.website.com notice mcpd[8459]: 01070728:5: Node /Common/172.17.70.19 address 172.17.70.19 monitor status up. [ /Common/icmp: up ]  [ was down for 0hr:0min:5sec ]

[Dario_Garrido]

Hello Abed.

Could you share your 'user_alert.conf' file?

Maybe you could try this instead:

alert NODE_UP "(.*) monitor status up" {
   ...
}
 
alert NODE_DOWN "(.*) monitor status down" {
   ...
}

Regards,

Dario.

[Abed_AL-R]

This is my config:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP "Pool member (.*?) monitor status up."{
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.11";
email toaddress="my@email.com"
fromaddress="f5"
body="A pool member went up!"
}
alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS "Node (.*?) monitor status down."{
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.12";
email toaddress="my@email.com"
fromaddress="f5"
body="Please notice the status of the node!"
}
alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS_UP "Node (.*?) monitor status UP."{
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.13";
email toaddress="my@email.com"
fromaddress="f5"
body="Please notice the status of the node!"
}
alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS "Pool member (.*?) monitor status down."{
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10";
email toaddress="my@email.com"
fromaddress="f5"
body="A pool member went down!"
}

How should I adjust it?

[Dario_Garrido]

Hello Abed.

First of all, you are using mixing existing tags with your custom expressions. I don't recommend you to do that.

Sometimes, existing tags have some little mistakes. As you can see below one tag expression has an space between the IP and the address and the other not.

# cat /etc/alertd/bigip_mcpd_error_maps.h | grep MON_STATUS | grep Node
0 LOG_NOTICE    01070640 BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS "Node %s address %s monitor status %s."
0 LOG_NOTICE    01070728 BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS_UP "Node %saddress %s monitor status up."

My recommendation is to use your own definition of tags in the user_alert.conf. This expressions should work in your case.

alert NODE_UP "Node (.*) monitor status up" {
   ...
}
 
alert NODE_DOWN "Node (.*) monitor status down" {
   ...
}
 
alert POOL_UP "Pool (.*) monitor status up" {
   ...
}
 
alert POOL_DOWN "Pool (.*) monitor status down" {
   ...
}

BTW, you have to pay attention carefully to the expression to not miss some character. Take into account dots and remove this '?' in your (.*?) regex.

## POOLS
notice mcpd[2964]: 01070638:5: Pool <Pool_name> member <ServerIP_port> monitor status down [ <MonitorA_name>: down, <MonitorB_name>: down ] [ was up for <#>hrs:<#>mins:<#>sec ]
notice mcpd[2964]: 01070727:5: Pool <Pool_name> member <ServerIP_port> monitor status up. [ <MonitorA_name>: down, <MonitorB_name>: up ] [ was down for <#>hrs:<#>mins:<#>sec ]
 
## NODES
notice mcpd[2964]: 01070640:5: Node <ServerIP> monitor status down.
notice mcpd[2964]: 01070728:5: Node <ServerIP> monitor status up.

REF - https://support.f5.com/csp/article/K12531

Please, don't forget to mark this answer as the best to help me for this contribution.

Regards,

Dario.

[Abed_AL-R]

Thank you.

I will update you if this works.

[Abed_AL-R]

`Hi,

Sorry but still it is not working.

I have the same config in another machine and there its working.

The difference I noticed is that there I see "pool member" down/up, and here I see only "node" down although it is indeed a pool member and the monitor inherited from pool.

So two machines, same alert config, same ltm config, one working and one is not.

I'm not sure why

[Dario_Garrido]

Hello Abed.

Have you restarted the alertd daemon?

tmsh restart /sys service alertd

Regards,

Dario.

[Abed_AL-R]

Yes, of course.

[Dario_Garrido]

If 'user_alert.conf' configuration is exactly the same in both devices, I recommend you to check some other basic stuff like DNS, SNMP or SMTP. I would try to test reachability of both devices and would check configuration as well.

A tcpdump fo checking if those packets are sending out of the box would also be a chance (for SMTP and SNMP Trap).

[Abed_AL-R]

Hi Dario

So this has been resolved with F5 TAC

And this is the final conf:

alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.11";
email toaddress="my@mail.com"
fromaddress="root@f5mail.com"
body="A pool member went up!"
}
alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS{
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.12";
email toaddress="my@mail.com"
fromaddress="root@f5mail.com"
body="Please notice the status of the node!"
}
alert BIGIP_MCPD_MCPDERR_NODE_ADDRESS_MON_STATUS_UP {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.13";
email toaddress="my@mail.com"
fromaddress="root@f5mail.com"
body="Please notice the status of the node!"
}
alert BIGIP_MCPD_MCPDERR_POOL_MEMBER_MON_STATUS {
    snmptrap OID=".1.3.6.1.4.1.3375.2.4.0.10";
email toaddress="my@mail.com"
fromaddress="root@f5mail.com"
body="A pool member went down!"
}

Two things:

• we changed the from address to a real address instead of just typing "root" or "f5"
• we deleted all the (.*) or (.*?) that was going up there

Works like charm