Forum Discussion

Topher_60415's avatar
Topher_60415
Icon for Nimbostratus rankNimbostratus
Jul 23, 2009

F5MonitoringService Keeps Crashing

All servers running Windows Server 2003 R2 x64 with full patches.

 

Installed version of SCOM is 2007 R2 RTM.

 

Installed version of F5 Management Pack is 1.2.0.579-x64.

 

Service is installed and disabled on RMS and is also deployed to a secondary MS in a running capacity.

 

I am monitoring 2 F5 LTM devices

 

 

Behavior - Service gets started with no issues and runs for approximately 2 hours, then crashes and dumps an event to the Event Log. The recovery behavior in the definitions of the service is to automatically restart, but it does not. Creating an automatic recovery response for the monitor in SCOM to start the service also has no effect. Error thrown to the Event Log is as follows:

 

 

Event Type:Error

 

Event Source:F5 Events

 

Event Category:None

 

Event ID:201

 

Date:7/23/2009

 

Time:2:05:18 PM

 

User:N/A

 

Computer:NRUSCA-S3040

 

Description:

 

Unhandled exception in DefaultDomain

 

 

Detailed Exception Information:

 

F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

 

at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

 

at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

 

at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)

 

at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)

 

--- End of inner exception stack trace ---

 

 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

 

 

 

Really needing some help here as the service seems unstable regardless of what I do. As an FYI, this environment was upgraded from SCOM 2007 SP1 and the crashes occurred in SP1 as well so I do not believe this is an R2 related issue (though I could certainly be wrong).

9 Replies

  • We are having this issue as well.

     

     

    Windows 2008 x64 sp2

     

    SCOM 2007 R2 RTM

     

    F5 Management Pack is 1.2.0.579-x64

     

     

    Configured to watch 6 BIG-IP LTMs

     

     

    Log Name: F5 Monitoring Log

     

    Source: F5 Events

     

    Date: 7/28/2009 9:03:00 AM

     

    Event ID: 201

     

    Task Category: None

     

    Level: Error

     

    Keywords: Classic

     

    User: N/A

     

    Computer: crisscom01.us.aegon.com

     

    Description:

     

    Unhandled exception in DefaultDomain

     

     

    Detailed Exception Information:

     

    F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

     

    at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

     

    at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)

     

    --- End of inner exception stack trace ---

     

    Event Xml:

     

     

     

     

    201

     

    2

     

    0

     

    0x80000000000000

     

     

    74

     

    F5 Monitoring Log

     

    crisscom01.us.aegon.com

     

     

     

     

    Unhandled exception in DefaultDomain

     

     

    Detailed Exception Information:

     

    F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)

     

    at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)

     

    at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

     

    at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

     

    at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)

     

    at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)

     

    --- End of inner exception stack trace ---

     

     

     

  • joel_hendrickso's avatar
    joel_hendrickso
    Historic F5 Account
    Hi, The only condition that could cause this error as far as I can tell would be an LTM pool member changing it's port. Since it is not possible to change a pool member's port via the configuration utility (you have to delete the pool member and recreate), I'm not sure how this could be happening.

     

     

    It would really help if I could take a look at your F5 monitoring log. There are instructions here for turning on verbose logging:

     

     

    http://devcentral.f5.com/wiki/default.aspx/MgmtPack/GeneralTroubleshooting.html

     

     

    Please turn on verbose logging, then recreate the condition, then send the log to ManagementPack(at)f5.com or post it here.

     

     

    Thanks, Joel Hendrickson

     

     

  • joel_hendrickso's avatar
    joel_hendrickso
    Historic F5 Account
    We have found the cause of this problem and put in a fix that will be in the next release, scheduled for the week of august 10th (2 weeks from now).

     

     

    If a pool member with port 4444 is discovered, any state changes (such as enable/disable/up/down) to that pool member will cause the service to crash. This has to do with the way that Big-IP translates some ports when it notifies us of state updates (in this case, 4444="KRB524").

     

     

    There is no workaround at this point besides restarting the service.

     

     

  • joel_hendrickso's avatar
    joel_hendrickso
    Historic F5 Account
    My apologies for the problems you are experiencing -- I have been told that it will be up in 1-2 hours so please check back soon. Thanks for your patience,

     

     

    Joel
  • DeanM's avatar
    DeanM
    Icon for Nimbostratus rankNimbostratus
    Hi Joel,

     

    Thanks for posting this new release, there appears to be an issue with the x86 version, I can not open the archive.

     

    Regards,

     

    Dean.
  • joel_hendrickso's avatar
    joel_hendrickso
    Historic F5 Account
    It looks like there was an issue with the zip files being corrupted for some people. We've rezipped and posted the files. Please let me know if they work for you.

     

     

    Thanks!

     

     

    Joel
  • DeanM's avatar
    DeanM
    Icon for Nimbostratus rankNimbostratus
    That's fine now Joel, thanks for your help.

     

    Regards,

     

    Dean.