The F5 Monitoring Service Data Source Singleton Error

Hi Julian,

 

 

I've checked and the F5 Monitering service account is definately a member of an Active Directory OpsMgr Admins group which is in turn a member of the OpsMgr Admin User Role in the SCOM Console so from that perspective the account should have the necessary admin privileges to interact with the health Service on the secondary management server.

 

 

Ive also compared the Log On service account properties associated with the F5 monitoring Service on both the RMS and MS and they appear to be identical. I don't imagine that the password is set incorrectly between them as the service would likely not start on the MS if it were enterred incorrectly at setup time.

 

 

I have just tried restarting the F5 Monitoring service on the MS and got an error indicating "Windows could not stop the F5 Monitoring Service on [MS]. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 0." I clicked OK at the error message and saw that the F5 Monitoring Service was now stuck in a "Stopping" state so I used Task Manager to kill the still running f5mpsvc.exe process (this was consuming quite a lot of memory too). Once that process was stopped I restarted the F5 Monitoring service. I saw the same f5mpsvc.exe re-appear in the Task Manager but then it disappeared. Refreshed the F5 Service view as I suspected it was not actually running. Stopped/restarted the F5 Service and the f5mpsvc.exe is now running int the Task Manager..

 

 

After killing-off the F5 process and restarting the F5 service, however, the F5 Log showed a fresh event (Event ID 201) as follows:

 

 

Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService

 

 

Detailed Exception Information:

 

F5Networks.F5Exception: Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService ---> F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionFailedException: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService ---> System.Runtime.Remoting.RemotingException: Port is Busy: All pipe instances are busy.

 

 

 

Server stack trace:

 

at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)

 

at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

 

 

Exception rethrown at [0]:

 

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

 

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

 

at F5Networks.Remoting.ServerBase.get_HostProcess()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

--- End of inner exception stack trace ---

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.ManagedDataSourceConnection`3.CoreConnect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1._DoConnectWithRecovery()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.Connect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._CreatePoolMember()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._InitializeCache()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.Initialize(UInt32 initialOperationSize)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._CreateDataSourceConnection()

 

at F5Networks.ManagementPack.Services.DeviceMonitor._InitializeDataStore()

 

--- End of inner exception stack trace ---

 

 

There are three such events co-inciding with the number of times I stopped/restarted the service.

 

 

Whereas before the errors were only of warning severity now they zre Critical.

 

 

Previous to these latest errors I checked the F5 Event Log on the MS and noticed that the Event ID 806 last occurred on 3 days ago as though the particular error is no longer being regenerated yet looking at the various views in the F5 Management Pack folder I can now see a health State for the active F5 device which was discovered last though all of its components have a "not monitored" icon. In addition, the view in the F5 Management Pack Monitoirng Services view still show the management server as critical with the same unit monitors as critical.

 

 

I then stopped th f5 Monitoring service, changed the Log On credentials to thatof the SDK service account, applied this and restarted the service. The same Critical Event 201 error was regeneratedin the F5 Log so using the SDK account does not appear to have made a difference only now the severity of the issue seems to be higher.

 

 

Julian, would it be possible for you or a colleague to assist me in troubleshooting this via a Webex or Livemeeting session?

 

 

Kind Regards,

 

 

flipa