The F5 Monitoring Service Data Source Singleton Error

Hi Flipa,

 

 

Usually this type of error in the F5 Management Pack surfaces when F5 Management Pack rule overrides are not targeting the appropriate classes. Do you have any F5 Management Pack overrides defined? If you're not sure, you can check if the F5 Management Pack has any dependent (override) management packs. You can check this in the SCOM Management Console :: Administration :: Management Packs :: F5 Networks :: Properties :: Dependencies :: Management Packs that depend on this Management Pack.

 

 

From your post it almost sounds like you haven't defined any overrides for the F5 MP, yet the error clearly points to an issue with overrides.

 

 

If you have overrides defined, I would recommend removing the override pack where these overrides are defined and start a clean slate creating those overrides, following the suggestions in this F5 DevCentral article:

 

http://devcentral.f5.com/wiki/default.aspx/MgmtPack/PerformanceCollectionAndMonitoring.html

 

 

I would also suggest installing the latest release of the F5 Management Pack (v2.2.0.580) as it would make our troubleshooting easier. About the singleton error you should probably get more details from the Operations Manager event log. If you still get the error while there are no overrides defined and you have the latest version of the F5 MP please zip and email us (to managementpack(at)f5(dot)com) the following logs:

 

 

- setup.log

 

- F5 Monitoring Log event log

 

- Operations Manager event log

 

 

Thank you!

 

Julian

 

Hi Julian,

 

 

You are correct, no overrides or unsealed custom MPs have been defined for the F5 MP as yet.

 

 

Looking at the text of the Operations Manager Conncetor Error that I am seeing I don't think that simply upgrade the MP to fix this.:

 

 

"The PerformanceDataSourceConnector connection to Operations Manager Health Service host localhost was lost: Failed to write to an IPC Port: The pipe is being closed. "

 

 

It seems from this that the issue may be a firewall or a user role access issue.

 

 

To clarify, I have not configured any of the F5 Run As Acccount/Profiles in the SCOM USer Role view since we do not want to use the configuration options of the Management Pack integration at this time and configuring the F5 Big-IP Discovery Operator profile seems unneccessary when there are so few devices to discover that it is easy enough to just get one of the F5 Admins to enter their credentials when performing the discovery.

 

 

So, the apart from the initial credentials supplied by one of the device adminsat the time of the discovery no other account credentials are in play.

 

 

Which leaves network/port access to the IPC Port. I've confirmed that I can telnet to the iControl and IQuery ports from the Management server so if you are able to clarify what the IPC Port is (if not either of the above) as well as its port number I can try that one as well.

 

 

Regards,

 

 

Flipa

Thanks for the update Flipa. The error you're pointing at ("PerformanceDataSourceConnector connection to Operations Manager Health Service host localhost was lost...") is usually a sign that either:

 

 

1. The F5 Monitoring Service cannot connect to the SCOM Health Service, for writing health and performance data, or:

 

2. The F5 Management Pack data sources are not loaded in the SCOM Health Service running on the Management Server where you monitor the F5 Devices from.

 

 

I'm going to assume that you have done the correct deployment of the F5 Management Pack in your distributed SCOM environment: first you have installed the F5 MP on the RMS where you have left the F5 Monitoring Service stopped (and in Manual/Disabled) mode as the F5 MP setup leaves it, and then you deployed the F5 MP on a (secondary) management server, where you plan to discover and monitor the F5 device(s) from. If my assumption is not correct please let me know.

 

 

Coming back to the two possible causes for the error, for 1 make sure that the service account running the F5 Monitoring Service is in the Operations Manager Administrators group.

 

To validate 2, I would suggest running the "Show Failed Rules and Monitors for this Health Service" SCOM task for the Management Server where the F5 devices are being discovered/monitored from. The output of this task will show if the F5 Management Pack data sources are not loaded. Please let me know the result of this task or if you need assistance running it.

 

 

Also, if you send us the logs that I mentioned in my previous post, we could probably give a more accurate diagnostic about the problem. You can zip and email them to managementpack(at)f5(dot)com.

 

 

Thank you.

 

Julian

 

 

 

 

 

Hi julian,

 

 

Thanks for the suggestion,

 

 

You are correct in your assumption as regards the installation procedure and that the F5 Monitoring Service is disabled on the RMS.

 

 

I am struggling to locate the "Show Failed Rules and Monitors for this Health Service" SCOM task in the console. Could you please provide aa more specific location in the Console where I will find this?

 

 

Kind Regards,

 

 

Michael

Michael,

 

 

You can run the task from SCOM Management Console :: Monitoring :: Operations Manager :: Management Server :: Management Server Health State :: select your management server :: Health Service Tasks (in the Action pane) :: Show Failed Rules and Monitors for this Health Service.

 

 

(Make sure you select the Management Server you're monitoring the F5 devices from).

 

 

Julian

Hi Julian,

 

 

Thanks for the pointers.

 

 

I ran the "Show Failed Rules and Monitors for this Health Service" task against the designated Management server but the task resulted in 0 failed workflows being found.

 

 

That leaves checking whether the F5 service account is a member of the Operations manager Admin group in the User roles view which I hope to verify tomorrow.

 

 

As far as the Setup log the relevant portion at the tail of the that log is appended below. I will forward the othe logs as per your instruction separately.

 

 

 

<02-24-2011 11:19:08> Info: HealthService service state is: Running

 

<02-24-2011 11:19:08> Info: Stopping service: HealthService

 

<02-24-2011 11:19:10> Info: HealthService service has been stopped.

 

<02-24-2011 11:19:10> Info: HealthService service state is: Stopped

 

<02-24-2011 11:19:10> Info: Starting service: HealthService

 

<02-24-2011 11:19:19> Info: HealthService service has been started.

 

<02-24-2011 11:19:19> Info: Installer bootstrap executable copied to system directory.

 

<02-24-2011 11:20:57> Info: Licensing of F5 Networks Management Pack has completed successfully.

 

<02-24-2011 11:20:57> Info: Verifying the DataSource and ConditionDetection modules required for the F5 Monitoring Service to run are loaded by Operations Manager monitoring host.

 

<02-24-2011 11:23:06> FatalError: The DataSource and ConditionDetection modules required for the F5 Monitoring Service to run could not be loaded by Operations Manager monitoring host. Check the Operations Manager Event Log for errors and manually start the F5 Monitoring Service once any issues have been resolved.

 

 

Regards,

 

 

Michael

 

 

Michael,

 

 

If the F5 data sources and workflows are loading correctly inside the SCOM Health Service, the only other reason why you would get the error is that the account running the F5 Monitoring Service doesn't have enough permissions to interact with the SCOM Health Service. The F5 Monitoring Service needs to write statistics and health data to the SCOM Health Service. I would try to [temporarily] assign the SCOM SDK service account to the F5 Monitoring Service, if that's possible for the customer, just to try and see if the errors are gone. Definitely the F5 Monitoring Service account should be in the Operations Manager administrators group.

 

 

Let me know.

 

 

Julian

Hi Julian,

 

 

I've checked and the F5 Monitering service account is definately a member of an Active Directory OpsMgr Admins group which is in turn a member of the OpsMgr Admin User Role in the SCOM Console so from that perspective the account should have the necessary admin privileges to interact with the health Service on the secondary management server.

 

 

Ive also compared the Log On service account properties associated with the F5 monitoring Service on both the RMS and MS and they appear to be identical. I don't imagine that the password is set incorrectly between them as the service would likely not start on the MS if it were enterred incorrectly at setup time.

 

 

I have just tried restarting the F5 Monitoring service on the MS and got an error indicating "Windows could not stop the F5 Monitoring Service on [MS]. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 0." I clicked OK at the error message and saw that the F5 Monitoring Service was now stuck in a "Stopping" state so I used Task Manager to kill the still running f5mpsvc.exe process (this was consuming quite a lot of memory too). Once that process was stopped I restarted the F5 Monitoring service. I saw the same f5mpsvc.exe re-appear in the Task Manager but then it disappeared. Refreshed the F5 Service view as I suspected it was not actually running. Stopped/restarted the F5 Service and the f5mpsvc.exe is now running int the Task Manager..

 

 

After killing-off the F5 process and restarting the F5 service, however, the F5 Log showed a fresh event (Event ID 201) as follows:

 

 

Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService

 

 

Detailed Exception Information:

 

F5Networks.F5Exception: Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService ---> F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionFailedException: The PerformanceDataSourceConnector connection to Operations Manager Health Service host HealthService could not be established: Port is Busy: All pipe instances are busy.

 

: HealthService ---> System.Runtime.Remoting.RemotingException: Port is Busy: All pipe instances are busy.

 

 

 

Server stack trace:

 

at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)

 

at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

 

 

Exception rethrown at [0]:

 

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

 

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

 

at F5Networks.Remoting.ServerBase.get_HostProcess()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

--- End of inner exception stack trace ---

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.ManagedDataSourceConnection`3.CoreConnect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1._DoConnectWithRecovery()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.Connect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._CreatePoolMember()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._InitializeCache()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.Initialize(UInt32 initialOperationSize)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._CreateDataSourceConnection()

 

at F5Networks.ManagementPack.Services.DeviceMonitor._InitializeDataStore()

 

--- End of inner exception stack trace ---

 

 

There are three such events co-inciding with the number of times I stopped/restarted the service.

 

 

Whereas before the errors were only of warning severity now they zre Critical.

 

 

Previous to these latest errors I checked the F5 Event Log on the MS and noticed that the Event ID 806 last occurred on 3 days ago as though the particular error is no longer being regenerated yet looking at the various views in the F5 Management Pack folder I can now see a health State for the active F5 device which was discovered last though all of its components have a "not monitored" icon. In addition, the view in the F5 Management Pack Monitoirng Services view still show the management server as critical with the same unit monitors as critical.

 

 

I then stopped th f5 Monitoring service, changed the Log On credentials to thatof the SDK service account, applied this and restarted the service. The same Critical Event 201 error was regeneratedin the F5 Log so using the SDK account does not appear to have made a difference only now the severity of the issue seems to be higher.

 

 

Julian, would it be possible for you or a colleague to assist me in troubleshooting this via a Webex or Livemeeting session?

 

 

Kind Regards,

 

 

flipa

F5 Monitoring service on Management Server now refuses to stay started.

 

 

I've reset the Log On credentials for the F5 Monitoring service to use the designated domain service account and re-entered the password and restarted the service but within 30 seconds the service stops unexpectedly.

 

 

This happens regardless of whether the designated service account is used or the user credentials of a Windows Administrator colleague so I don't belive it is due to an incorrect password being entered when I specify the designated F5 Service Account credentials.

 

 

Julian, can you clarify whether the F5 Monitoring Service needs to be started before the System Center Management service on the Management Server in order to properly bind to the Heathlservice on the Management Server?

 

 

Regards,

 

 

Michael

OK. Have now checked the SQL server to see if that reveals anything.

 

 

Based on a previous thread by tomtux named "Unsuccessful Installation" (http://devcentral.f5.com/Community/GroupDetails/tabid/1082223/asg/54/afv/topic/aft/1171608/aff/59/showtab/groupforums/Default.aspx) I have checked checked the following items as per the reply by Joel Hendrickson:

 

 

1. Verify that the F5 service has the correct connection information for the database:

 

 

a) Open the F5 service configuration file (C:\Program Files\F5 Networks\Management Pack\f5mpsvc.exe.config)

 

b) The value specified for "SQLHostName" should match your SQL Server host name

 

Checked. The correct SQL server hostname is listed in thf5mpsvc.exe.config file.

 

 

c) The value specified for "DatabaseName" should match a database in the above SQL Server instance

 

Checked. The database name in both the f5mpsvc.exe.config file and on the SQL Server databases view match "F5_ManagementPack".

 

 

2. Open the database specified in the config file and verify that the "DynamicData.Entity" table exists:

 

 

a) Open SQL Server Management Studio

 

b) Connect to your SQL Server instance

 

c) Expand the list on the left to Databases => F5_ManagementPack => Tables

 

d) Verify that there is a table named "DynamicData.Entity"

 

Checked. The F5_ManagementPack contains this table.

 

 

If the database and/or table is missing, something probably went wrong in the installation process and we can diagnose further.

 

 

The only thing that I can see that is amiss is the the designated F5 Monitoring Service account that is constantly stopping on the Management Server does not appear listed in either the Security folder immediately below the F%_ManagementPack database or within the Sql Server Security > Logins view which leads me to suspect that the issue is simply one of the service account not having access to the database.

 

 

I have verified that the service account is a member of the local admins group on the SQL Server but I do not see it listed in any of the obvious security/authorization views within SQL Server Management Studio. That said I am not a DBA and I was advised by the DBAs that adding the service account to the Local Admins group gives the service account the necessary system admin access to SQL Server that the F5 MP guide indicates is required. I suspect this is not the case.

 

 

That said, there is a Logins for the BUILTIN/Administrators group of (which as noted above the F5 service account is a member on the SQL Server) which has "public" and "sysadmin" Server Roles but no actual database is ticked in the User Mappings property for this group so I suspect that even if the service account is a member of the BUILTIN/Administrator Security > Login group it is not actually granted any practical pivileges to any databases.

 

 

In order to be granular and not ask my DBAs to do something that would break there preferred security model I would appreciate if you could clarify the correct way to grant the service account the access it needs within SQL Management Studio so that it can write to the F5_ManagementPack database?

 

 

Not being a DBA I would be grateful if you could provide screenshots in a word document -which would be useful to have in the Install Guide too- so that even an SQL dummy such as me would be able to step through creating the necessary access for the account. This would also help me in convincing the DBAs that what I want to do is not going to break there preferred security model if they can see exactly what privileges are being granted.

 

 

Kind Regards,

 

 

Michael