Forum Discussion

Kurt_Knochner_5's avatar
Oct 25, 2011

Event Notification with PERL and SSL/TLS

Hi,

 

 

I'm using the PERL sample for Event Notification (http://devcentral.f5.com/wiki/iControl.EventNotificationListener.ashx). It works fine, as long as I'm using HTTP as the transport method. However I'm having problems when I switch to SSL/TLSv1 (by using 'SSL_version => 'TLSv1', SSL_server => 1' when creating a SOAP::Transport::HTTP::Daemon object). The LB then sends some Events to the Listener, however if the Listener dies or gets stopped, the LB stops sending Events even if I restart the Event Listener. I don't even see packtes with tcpdump, so the LB seems to be "stuck" somehow after the Event Listener died.

 

 

This behaviour is different from what I see with HTTP, where I can restart the Event Listener without any problems, as the LB will immediately send new events as soon as the Event Listener is running.

 

 

BTW: The Enterprise Manager uses SSL as well, so I think this should basically work (output created with Event Subscription Client - http://devcentral.f5.com/wiki/iControl.EventSubscriptionClient.ashx)

 

 

Details for Subscription 'XXXXXXXX-1974-5E1B-7819-XXXXXXXXXX'

 

Name : EM config change subscription: x.x.x.x

 

url : https://x.x.x.x/emupdate/subscription?uid=31

 

credentials (:) type AUTHMODE_BASIC

 

ttl : -1 min_events_per_timeslice : 1

 

max_timeslice : 1

 

enabled_state : STATE_ENABLED EVENTS EVENTTYPE_DELETE EVENTTYPE_CREATE EVENTTYPE_MODIFY

 

 

As you can see, EM uses https://.

 

 

Any idea what could cause this behaviour of the LB (LTM 6900, 10.2.0 HF2) when using SSL/TLSv1?

 

 

Any idea how I can troubleshoot this?

 

 

Thank you!

 

 

Regards

 

Kurt Knochner

 

 

 

 

 

  • Some additional information:

     

     

    TLSv1 seems to cause the trouble. If I create the SOAP::Transport::HTTP::Daemon object WITHOUT SSL_version => 'TLSv1', it seems to be more stable and it's possible to restart the Event Listener without problems. After I used TLSv1, the Event Subsystem on the LB get's somehow "stuck" and I had to delete the Notification Subscription and create a new one, otherwise the LB ceased to send Events. Restarting the httd daemon on the LB did not help.

     

     

    Does the iControl Event Subsystem support TLSv1?

     

     

    Enabling debug for the iControl system (b db iControl.Loglevel debug) did not help either. There are debug messages in the log, however nothing that's related to the SSL/TLS and/or connection problems.

     

     

    BTW: What's the component/process that sends events to the Event Listener? Maybe I can debug that process..

     

     

    Thank you!

     

     

    Regards

     

    Kurt Knochner
  • Hi Kurt, glad you are trying out the event notification interface. You asked a lot of questions so I'll try to start by explaining how eventd works and hopefully that will get you covered.

     

     

    At the core of BIG-IP is our configuration engine mcpd. It contains a notification service for daemons on the device to notify them when the state of it's configuration changes (ie. an object is added/deleted in the GUI).

     

     

     

    I wrote eventd as another one of those listening daemons. Right now, the only way to configure eventd is by building a client with the iControl Management.EventSubscription interface (which you probably figured out). The client then configures "listeners" which contain what type of notifications it wants to hear about.

     

     

     

    The eventd sits in a event loop listening for events. When one occurs, it looks in it's configuration (stored on disk under /config/eventd.xml) for listeners that want to be notified of that specific state change. It then pushes the change into a queue for each listener.

     

     

     

    Listeners can be configured to buffer responses based on various criteria (see the EventSubscription's create method.

     

     

     

    When the threshold for a notification occurs (max messages, timeslice, etc), it will attempt to make a connection to the EventNotification interface on your listener. If the connection attempt fails, it retries a couple of times (also configurable). After the retry limit fails, it will put the listener in a "disabled" state. The only way to bring this back up is to re-enable it with the EventSubscription methods. This is probably what's happening to you.

     

     

     

    As for a SSL connection, I honestly haven't tested that. I'm not sure why it wouldn't work and I'm not sure if it's a client thing (in my daemon) or a server thing (on the perl side). Definitely something I'll have to test out when I can get to it.

     

     

     

    As for the user field, that was originally included as a future placeholder if we ever got the username included down in the configuration daemon for changes. That was on the roadmap at the time but still hasn't been implemented so for now, the username will be empty.

     

     

     

    If you want to debug eventd, you should do the following

     

     

     

    1) bigstart shutdown eventd

     

    2) $ /sbin/eventd -d -f

     

     

     

    That will run eventd in the foreground printing debug statements.

     

     

     

    Hopefully this helps clear some things up.

     

     

     

    -Joe

     

     

     

     

     

  • Hi Joe,

     

     

    thank you for the fast answer!

     

     

    So, eventd is the one that sends the Events to my Perl Listener? If so, I will try to debug that process, as shown in your post.

     

     

    I've done more tests and it seems to be TLSv1. SSLv2/v3 seems to be O.K. Probably it's a problem with both my perl script (IO Library) as it crashes sometimes when TLSv1 is used (mybe due to the data coming from eventd). eventd also seems to lock up after I use TLSv1 on the client. I see no further TCP packets to my listener port after that, unless I remove the subscription and create a new one. I will try to debug eventd and report back. Can you please check how eventd handles TLSv1?

     

     

     

    Regarding the username. Is there any way to get the name of the user who changed the config? Audit logging provides that information, so it must be available somewhere. However, the audit log is written by httpd, which obviously knows the user name. Maybe you could scan /var/log/audit within eventd (similar to alertd) and create Events for any audit log entry ;-))

     

     

    Any idea how I can solve my problem, i.e. implement my own "audit/config change" tracker that also includes the user name?

     

     

    Thank you!

     

     

    Regards

     

    Kurt

     

  • Hi Joe,

     

     

    I've done some eventd debugging now. Output below. Unfortunately, I cannot see any reason for the problem in the debug output. Any idea what's going wrong?

     

     

    [root@xxxxxxxx-C2:Standby] alertd /usr/bin/eventd -d -f

     

     

    RuntimeOptions::queryAdminIP: Admin IP = xx.21.xx.11

     

    Entering Notification Thread...

     

    Received TRANSACTION @ 4ea6c6a2 (1319552674) (r=1/c=0/p=0)!

     

    retrieval start_transaction

     

    ClusterMonitor::handleOperation: clustered.environment = false

     

    ClusterMonitor::handleOperation: s = ST_UK, e = false, p = unknown --> EV_NC

     

    ClusterMonitor::handleOperation: a = ACT_READ, ns = ST_NC

     

    ClusterMonitor::handleRead

     

    Configuration::load()

     

    Creating Consumer

     

    Creating Consumer

     

    ClusterMonitor::handleOperation: cluster.primary = false

     

    ClusterMonitor::handleOperation: s = ST_NC, e = false, p = false --> EV_NC

     

    ClusterMonitor::handleOperation: a = ACT_NONE, ns = ST_NC

     

    ClusterMonitor::handleOperation: cluster.mgmtipaddr = ::

     

    retrieval end_transaction

     

    Received TRANSACTION @ 4ea6c6a8 (6) (r=0/c=0/p=1)!

     

    MCPSerializer::serialize - operation 2908 (modify); object 2122 (db_variable)

     

    MCPSerializer::addData(), db_variable_name : string

     

    MCPSerializer::addData(), db_variable_value : string

     

    MCPSerializer::addData(), db_variable_object_id : ulong

     

    MCPSerializer::addData(), db_variable_transaction_id : ulong

     

    adding event to consumer DF99021F-1974-742B-C185-77067226AA6

     

    Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552650, 1319552680, 30)

     

    adding event to consumer 2AA91888-1974-9995-C03A-461AB90526A

     

    Processing consumer: '2AA91888-1974-9995-C03A-461AB90526A' + objects: 1; time (t0,t1,delta): (1319552636, 1319552680, 44)

     

    timerExpired

     

    timerExpired

     

     

    ==> tcpdump output (running in the background)

     

    LB: 10.11.57.2

     

    Listener (using TLSv1): 10.30.42.98.11011

     

     

     

    16:24:40.113111 IP 10.11.57.2.58881 > 10.30.42.98.11011: S 515597348:515597348(0) win 5840 out slot1/tmm2 lis=

     

    16:24:40.113508 IP 10.30.42.98.11011 > 10.11.57.2.58881: S 3804375753:3804375753(0) ack 515597349 win 5792 in slot1/tmm2 lis=

     

    16:24:40.114328 IP 10.11.57.2.58881 > 10.30.42.98.11011: . ack 1 win 46 out slot1/tmm2 lis=

     

    16:24:40.114550 IP 10.11.57.2.58881 > 10.30.42.98.11011: P 1:74(73) ack 1 win 46 out slot1/tmm2 lis=

     

    16:24:40.114986 IP 10.30.42.98.11011 > 10.11.57.2.58881: . ack 74 win 46 in slot1/tmm2 lis=

     

    16:24:40.115207 IP 10.30.42.98.11011 > 10.11.57.2.58881: F 1:1(0) ack 74 win 46 in slot1/tmm2 lis=

     

    16:24:40.115375 IP 10.30.42.98.11011 > 10.11.57.2.58881: R 2:2(0) ack 74 win 46 in slot1/tmm2 lis=

     

     

    ==> my perl script dies after this !!! I will have to check the TLS options

     

     

    Received TRANSACTION @ 4ea6c6a8 (0) (r=0/c=0/p=1)!

     

    MCPSerializer::serialize - operation 2908 (modify); object 2122 (db_variable)

     

    MCPSerializer::addData(), db_variable_name : string

     

    MCPSerializer::addData(), db_variable_value : string

     

    MCPSerializer::addData(), db_variable_object_id : ulong

     

    MCPSerializer::addData(), db_variable_transaction_id : ulong

     

    adding event to consumer DF99021F-1974-742B-C185-77067226AA6

     

    adding event to consumer 2AA91888-1974-9995-C03A-461AB90526A

     

    timerExpired

     

    Saving Consumer list

     

    Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552680, 1319552680, 0)

     

    timerExpired

     

    startSweepTimer: 4ea6c6a8 -> 4ea6c6b8

     

    timerExpired

     

    Saving Consumer list

     

    Processing consumer: 'DF99021F-1974-742B-C185-77067226AA6' + objects: 1; time (t0,t1,delta): (1319552680, 1319552680, 0)

     

    timerExpired

     

    startSweepTimer: 4ea6c6a8 -> 4ea6c6b8

     

    timerExpired

     

    Saving Consumer list

     

    timerExpired

     

    Processing consumer: '2AA91888-1974-9995-C03A-461AB90526A' + objects: 2; time (t0,t1,delta): (1319552680, 1319552695, 15)

     

    timerExpired

     

     

    ==> my perl died, so TCP Reset.

     

     

    16:24:55.999344 IP 10.11.57.2.60965 > 10.30.42.98.11011: S 700834715:700834715(0) win 5840 out slot1/tmm2 lis=

     

    16:24:55.999798 IP 10.30.42.98.11011 > 10.11.57.2.60965: R 0:0(0) ack 700834716 win 0 in slot1/tmm2 lis=

     

     

    timerExpired

     

    startSweepTimer: 4ea6c6b8 -> 4ea6c6c8

     

    Signal 2 received

     

    Destroying Consumer DF99021F-1974-742B-C185-77067226AA6

     

    Destroying Consumer 2AA91888-1974-9995-C03A-461AB90526A

     

     

    BTW: how can I paste formatted text into the editor window? It denied it with an error: "Your browser does not support pasting rich content". Browser: Firefox.

     

     

    Thank you.

     

     

    Regards

     

    Kurt
  • UPDATE: I've done some further checks. Apparently the perl listener is part of the problem. Sometimes it dies directly after the SSL CLIENT HELLO (coming from the LB), however only if I start the SOAP Server with the option TLSv1, which is strange, as eventd offers TLSv1 anyway !?!? Seems to be a bug in the perl IO library.

     

     

    So, one part of the problem is the perl script.

     

     

    However, in debug mode I was unable to reproduce the second problem ("stuck" eventd). It will now allways retry to connect to the server, maybe due to the FIN/RST of the TCP connection after the script died !??! I don't get it. There's something different in debug mode.

     

     

    Anyway, I will simply not start the SOAP server with the option TLSv1, as eventd is using that anyway and I will restrict access to my listener port by using TCP wrapper or iptables.

     

     

    So: problem understood (partly) and the workaround is acceptable :-)

     

     

    Regards

     

    Kurt