Fatal Error when upgrade or uninstall upgrade from F5 MP-2.1.1.140 to MP-2.1.3.217

Rob,

 

 

I'll investigate the error log and will get back to you with a procedure / workaround to get your F5 MP deployed successfully.

 

 

Julian

Julian,

 

Thanks for the update. We have a distributed installation of SCOM with Clustered RMS server and typyicall put all of our connector installations (like the F5) on a dedicated mgmnt server. I notice that most (if not all) F5 documentation illustrates the F5 install on the RMS or RMS Cluster vs. a Mgmt server. It seems that Microsoft continues to recommend not putting connectors on the RMS unless there is no other option hence our design of trying to keep it on a mgmt server. Is our setup against your design or recommendation and will it be problematic?

 

 

Thanks!

 

Rob

Julian,

 

This issue has been resolved. I broke the RMS cluster and performed a windows (from the add/remove programs) removal of the existing F5 MP and then performed a clean install vs. an upgrade. I've discovered an F5, deployed a few overrides but am still waiting for some perf data to appear.

Hi Rob,

 

 

Thank you for the update. I think the way your clustered RMS environment has been set up would have been supported by the F5 Management Pack deployment, while still conforming to Microsoft's best practices regarding RMS workflows, load, etc. Normally, when deploying the F5 MPk in a clustered RMS environment, the RMS cluster nodes should be the first ones targeted for the F5 MPk deployment, and then the other management servers. By default, the F5 Monitoring Service agent is disabled on the clustered RMS nodes. We basically require the RMS deployment for loading the actual F5 management pack, F5 device configuration and related data-sources, etc. The actual monitoring would happen on the management servers, thus complying to MS best practices.

 

 

Coming back to your initial problem, my feeling is that you initially deployed the F5 MPk on a non-clustered RMS, which eventually has become clustered. So initially, our F5 MPk configuration data would be referencing the FQDN of the RMS host. After promoting the RMS to a cluster node, the 'old' deployment of the F5 MPk would stay cluster-agnostic, as it won't automatically adapt to the clustered configuration. By design. If my theory stays true, attempting an in-place upgrade of the F5 MPk from a non-clustered configuration to an RMS-cluster would probably break, as the upgrade would fail to resolve some of the 'old' FQDN references of the [standalone] RMS to the 'new' RMS clustered group resource's FQDN. And this is unfortunately a user case scenario we haven't thoroughly tested yet with the F5 MPk deployment.

 

 

We will investigate this use case and will try to come up with a resolution, but I don't have an ETA for the fix at this time. If you re-deployed on a non-clustered RMS environment, things should pretty much fall into normal, but if you're missing performance data, the first thing to look at would be to check if there are any errors in the F5 Monitoring Log or the Operations Manager event logs. If you have defined your own overrides, make sure they comply with the guidelines described here: http://devcentral.f5.com/wiki/default.aspx/MgmtPack/PerformanceCollectionAndMonitoring.html

 

 

It is very important to have the appropriate targets set up for the collection rule' configuration override (to F5.Object or derived) and the rule property override (to F5.MonitoringService). See the related article for more information.

 

 

If you have problems setting these overrides up, let us know, and we'll further investigate.

 

Julian

 

 

Julian,

 

Thanks for the update. You mention that "we basically require the RMS deployment for loading the actual F5 MP, F5 device configuration and related data-source,etc". My question is what's the benefit of having the service clusters on the RMS? Is this specific for environments where they dont have a fully distributed Ops Mgr environment but instead of 2 RMS servers that are cluster and the managment server. (meaning they dont have designated mgmt servers that report to the RMS)

 

 

I was able to successfully install the F5 MP on both of our RMS server and I added it as a clusterd service but am now wondering if that was not necessary. I also installed/ran the F5 MP setup on a dedicated mgmt server, discovered an F5 and applied overrides to the MP enabling rules for performance collection of LTM avg/sec connection info. The problem is i see no data or even the list of perf rules in the performance rule. I could be mistaken but i did not think it was necessary to enable the red/green rules to collect this info and I would have expected to at least see the data that is collected from the F5 mp rules that are enabled by default.

 

 

Looking at the F5 Monitoring Log I do see Event 201 & 405 errors

 

 

Event 201 Error detail

 

 

Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942

 

 

Detailed Exception Information:

 

F5Networks.F5Exception: Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942 ---> F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionFailedException: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942 ---> System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

 

 

 

Server stack trace:

 

at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)

 

at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

 

 

Exception rethrown at [0]:

 

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

 

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

 

at F5Networks.Remoting.ServerBase.get_HostProcess()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

--- End of inner exception stack trace ---

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.ManagedDataSourceConnection`3.CoreConnect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1._DoConnectWithRecovery()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.Connect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._CreatePoolMember()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._InitializeCache()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.Initialize(UInt32 initialOperationSize)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._CreateDataSourceConnection() in c:\home\f5cm\sandbox\cc\builds\ballantine\104734\Branches\Trunk\HealthMonitor\Services\DeviceMonitor\DataAccess.cs:line 939

 

at F5Networks.ManagementPack.Services.DeviceMonitor._InitializeDataStore() in c:\home\f5cm\sandbox\cc\builds\ballantine\104734\Branches\Trunk\HealthMonitor\Services\DeviceMonitor\DataAccess.cs:line 662

 

 

Event 405 Error detail

 

 

Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942

 

 

Detailed Exception Information:

 

F5Networks.F5Exception: Unable to connect to data source: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942 ---> F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionFailedException: The PerformanceDataSourceConnector connection to Operations Manager Health Service host OKCMONTST942 could not be established: Failed to connect to an IPC Port: The system cannot find the file specified.

 

: OKCMONTST942 ---> System.Runtime.Remoting.RemotingException: Failed to connect to an IPC Port: The system cannot find the file specified.

 

 

 

Server stack trace:

 

at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)

 

at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)

 

at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

 

 

Exception rethrown at [0]:

 

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

 

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

 

at F5Networks.Remoting.ServerBase.get_HostProcess()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

--- End of inner exception stack trace ---

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.DataSourceConnector`2.Connect()

 

at F5Networks.ManagementPack.DataAccess.OperationsManager.DataSource.ManagedDataSourceConnection`3.CoreConnect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1._DoConnectWithRecovery()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ManagedConnectionBase`1.Connect()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._CreatePoolMember()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2._InitializeCache()

 

at F5Networks.ManagementPack.DataAccess.ManagedConnections.ConnectionPoolBase`2.Initialize(UInt32 initialOperationSize)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._CreateDataSourceConnection() in c:\home\f5cm\sandbox\cc\builds\ballantine\104734\Branches\Trunk\HealthMonitor\Services\DeviceMonitor\DataAccess.cs:line 939

 

at F5Networks.ManagementPack.Services.DeviceMonitor._InitializeDataStore() in c:\home\f5cm\sandbox\cc\builds\ballantine\104734\Branches\Trunk\HealthMonitor\Services\DeviceMonitor\DataAccess.cs:line 662

 

--- End of inner exception stack trace ---

Yes Rob,

 

 

You're right. The only benefit of clustering the F5 Monitoring Service agent on the RMS cluster nodes would be a minimal redundancy support for non-distributed Ops Mgr environments, where there are no other management servers deployed, besides the RMS nodes. And, as you mentioned, in your environment there's no need to cluster the F5 Monitoring Service, since you mostly target the other management servers for monitoring.

 

 

Looking at the error you pointed to, I can see that there are problems with the F5 Monitoring Service trying to connect to the Health Service on host OKCMONTST942. I'd really like to get you unblocked as soon as possible, speed up the troubleshooting time and save you the trouble to gather traces and logs. I think that there might be some mix-up in the F5 MPk configuration data, because of the switch between the clustered and non-clustered RMS environment.

 

 

Let me know if you would be OK for us to set up a remote desktop session (GoToMeeting) with you and try to fix the problems right away. And make it to your best convenience, and I'll accommodate that schedule. Otherwise I can direct you on how to clean-up the deployment of the F5 MP and try to re-deploy again.

 

 

Julian

 

 

Julian,

 

Sounds good. I'm flexible with my time today until 430 EST today and tomorrow. Let me know what time works for you.

 

 

Thanks,

 

Rob

Ok Rob,

 

 

Let's try to do this today, within your availability time frame. I'll send you the GoToMeeting invite through email.

 

 

Thank you!