Forum Discussion
Topher_60415
Nimbostratus
NimbostratusF5MonitoringService Keeps Crashing
All servers running Windows Server 2003 R2 x64 with full patches.
Installed version of SCOM is 2007 R2 RTM.
Installed version of F5 Management Pack is 1.2.0.579-x64.
Service is installed and disabled on RMS and is also deployed to a secondary MS in a running capacity.
I am monitoring 2 F5 LTM devices
Behavior - Service gets started with no issues and runs for approximately 2 hours, then crashes and dumps an event to the Event Log. The recovery behavior in the definitions of the service is to automatically restart, but it does not. Creating an automatic recovery response for the monitor in SCOM to start the service also has no effect. Error thrown to the Event Log is as follows:
Event Type:Error
Event Source:F5 Events
Event Category:None
Event ID:201
Date:7/23/2009
Time:2:05:18 PM
User:N/A
Computer:NRUSCA-S3040
Description:
Unhandled exception in DefaultDomain
Detailed Exception Information:
F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)
at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)
--- End of inner exception stack trace ---
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Really needing some help here as the service seems unstable regardless of what I do. As an FYI, this environment was upgraded from SCOM 2007 SP1 and the crashes occurred in SP1 as well so I do not believe this is an R2 related issue (though I could certainly be wrong).
Ben_Newport_102We are having this issue as well.
Windows 2008 x64 sp2
SCOM 2007 R2 RTM
F5 Management Pack is 1.2.0.579-x64
Configured to watch 6 BIG-IP LTMs
Log Name: F5 Monitoring Log
Source: F5 Events
Date: 7/28/2009 9:03:00 AM
Event ID: 201
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: crisscom01.us.aegon.com
Description:
Unhandled exception in DefaultDomain
Detailed Exception Information:
F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)
at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)
--- End of inner exception stack trace ---
Event Xml:
201
2
0
0x80000000000000
74
F5 Monitoring Log
crisscom01.us.aegon.com
Unhandled exception in DefaultDomain
Detailed Exception Information:
F5Networks.F5Exception: Unhandled exception in DefaultDomain ---> System.InvalidOperationException: Cannot modify key property 'TargetServicePort', that would change the object's identity.
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(MonitoringObject monitoringObj, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.SdkConnector.UpdateObject(Guid monitoringId, EntityData entityChanges, ConfigData configData)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter._UpdateDeviceConfig_Modify(SdkConnector connector, IPAddress deviceIP, ConfigChange configChange, DeviceConfigStats configStats, List`1 entitiesNeedingStateUpdates)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.<>c__DisplayClass22.b__1e()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass6.b__5()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.CoreDoOperation[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrConnectionPool`1.CoreDoOperationWithExceptionMapping[ReturnResult](ManagedOpsMgrConnection`1 connection, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.<>c__DisplayClass9`1.b__8()
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.DoOperationWithRecovery[ReturnResult](GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation[ReturnResult](ManagedConnectionType& connection, Object connectionTag, GenericVoidHandler`1 operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.UseConnectionForOperation(ManagedConnectionType& connection, Object connectionTag, VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.ManagedConnections.LockedConnectionPoolReference`2.UseConnection(VoidVoidDelegate operation)
at F5Networks.ManagementPack.DataAccess.OperationsManager.OpsMgrMonitoringAdapter.UpdateDeviceConfig(ConfigChange configChange, IPAddress deviceIP, Guid deviceId, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.DataAccess.Configuration.ConfigManager`8.UpdateDeviceConfig(Device device, Queue`1 configurationPackets, Boolean saveRuleUpdates, ObjectsModified objectsModified, List`1 entitiesNeedingStateUpdates, List`1 entitiesAdded)
at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateDeviceDataConfiguration(Device device, DataReceivedEventArgs dataArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.ConnectionCache.ListeningDeviceConnectionCache.DoDataOperation(DeviceConnection deviceConnection, DataReceivedEventArgs args)
at F5Networks.ManagementPack.Services.DeviceMonitor._ConnectionDataReceived(ListeningDeviceConnectionCache connectionCache, DataReceivedEventArgs dataReceivedArgs)
at F5Networks.ManagementPack.Services.DeviceMonitor.ParsedResponseProcessor.ProcessData(Queue`1 responses)
at F5Networks.ManagementPack.Services.DeviceMonitor.BatchedParsedResponseProcessor._FlushQueue(Object synchronousCall)
--- End of inner exception stack trace ---- Hi, The only condition that could cause this error as far as I can tell would be an LTM pool member changing it's port. Since it is not possible to change a pool member's port via the configuration utility (you have to delete the pool member and recreate), I'm not sure how this could be happening.
It would really help if I could take a look at your F5 monitoring log. There are instructions here for turning on verbose logging:
http://devcentral.f5.com/wiki/default.aspx/MgmtPack/GeneralTroubleshooting.html
Please turn on verbose logging, then recreate the condition, then send the log to ManagementPack(at)f5.com or post it here.
Thanks, Joel Hendrickson - Ben_Newport_102Those logs were sent in yesterday via email.
- We have found the cause of this problem and put in a fix that will be in the next release, scheduled for the week of august 10th (2 weeks from now).
If a pool member with port 4444 is discovered, any state changes (such as enable/disable/up/down) to that pool member will cause the service to crash. This has to do with the way that Big-IP translates some ports when it notifies us of state updates (in this case, 4444="KRB524").
There is no workaround at this point besides restarting the service. - My apologies for the problems you are experiencing -- I have been told that it will be up in 1-2 hours so please check back soon. Thanks for your patience,
Joel - The release is now available (v 1.3.0.715)
Joel 
DeanMHi Joel,
Thanks for posting this new release, there appears to be an issue with the x86 version, I can not open the archive.
Regards,
Dean.- It looks like there was an issue with the zip files being corrupted for some people. We've rezipped and posted the files. Please let me know if they work for you.
Thanks!
Joel - DeanMThat's fine now Joel, thanks for your help.
Regards,
Dean.
