SCOM F5 Objects Not Consistent with Objects on F5

Jeremy,

 

 

Would you please send your trace logs (see link below if you have not done this yet)? We should be able to identify the problem and get your config back in sync.

 

 

http://devcentral.f5.com/Wiki/default.aspx/MgmtPack/GeneralTroubleshooting.html

 

 

It may also help if I could see your Operations Manager event log (from the Windows Event Viewer).

 

 

Joel Hendrickson

 

 

Here are the SystemInformation and verbose logs. I followed the instructions for the verbose log and copied the log file after a few minutes had gone by.

Thanks for the logs.

 

 

Since the point where verbose logging was turned on I don't see anything wrong, but there are numerous problems evident in the past few days which do indicate that configuration is out of sync. One of the ways that config can get out of sync is if there are problems connecting to Operations Manager and/or the SQL databases while changes are occurring on the BIG-IP. I am seeing problems connecting to OpsMgr and SQL, particularly on 12/4 around 4:45 PM.

 

 

I think the best thing to do is to rediscover your BIG-IP so that config gets in sync. Then, if problems persist, when you send the logs again they will have full verbosity. At this point it is too much of a guessing game.

 

 

To refresh config, click on a BIG-IP in OpsMgr and use the "Rediscover F5 Device Configuration" task. As soon as you notice any problems, send the logs.

 

 

Thanks, Joel

Running the rediscover task produces an error:

 

 

 

The Event Policy for the process started at 4:19:21 PM has detected errors in the output. The 'StdErr' policy expression:

 

\a+

 

 

matched the following output:

 

 

Error: The authentication credentials for device 10.41.52.27 are not cached with account KFDDEV\saadev. Use interactive mode or cache credentials.

 

 

 

 

 

Command executed: "C:\Program Files\F5 Networks\Management Pack\\f5mpcmd.exe" -Discover:NoPrompt 10.41.52.27

 

 

Working Directory: C:\Program Files\F5 Networks\Management Pack\

 

 

 

 

One or more workflows were affected by this.

 

 

 

 

Workflow name: F5.Task_Rediscover

 

 

 

Instance name: kfddcbigipdev.krollfactualdata.com

 

 

 

Instance ID: {CC24EFFE-E2BE-AB1A-5077-8D16082B4FF7}

 

 

 

Management group: DEVSCOM

 

 

Error Code: -2130771918 (Unknown error (0x80ff0032)).

Sorry for the confusion -- this happens when you initially discovered the device under a different account than the account you're currently using to rediscover (KFDDEV\saadev).

 

 

There are several workarounds, but the easiest thing to do is simply use the "Discover F5 Devices" task to discover the same IP address. This is roughly what the "Rediscover" task is doing anyways.

 

 

Thanks for your patience!

 

 

Joel

 

 

Well, I got some odd results when I tried to discover the F5 device again. The F5 wizard said it was successful. But, there is an error in the event viewer that seems to indicate otherwise (event ID 301, source "F5 Events"):

 

 

Failed to discover device at address: 10.41.52.27

 

Network-related failure has occurred: [Category]GZip:[Type]SendFailure;LastError=SSL IO Error 487181392: SYSCALL:

 

 

 

F5Networks.Protocols.iQuery.iQueryException: [Category]GZip:[Type]SendFailure;LastError=SSL IO Error 487181392: SYSCALL:

 

at F5Networks.Protocols.iQuery.RawIQuerySocket.CoreSendRaw(String rawCommand)

 

at F5Networks.Protocols.iQuery.iQuerySocketBase.<>c__DisplayClass5.b__4()

 

at F5Networks.ProgressTracking.Tracer.<>c__DisplayClass1.b__0()

 

at F5Networks.ProgressTracking.Tracer.DoActionWithTryFinally[TReturnResult](GenericVoidHandler`1 activeCode, VoidVoidDelegate preCode, VoidGenericHandler`1 postCode, VoidVoidDelegate postCodeSuccess, VoidGenericHandler`1 postCodeFailure)

 

at F5Networks.ProgressTracking.Tracer.DoActionWithTryFinally(VoidVoidDelegate activeCode, VoidVoidDelegate preCode, VoidGenericHandler`1 postCode, VoidVoidDelegate postCodeSuccess, VoidGenericHandler`1 postCodeFailure)

 

at F5Networks.ProgressTracking.Tracer.DoActionWithTryFinally(VoidVoidDelegate activeCode, VoidVoidDelegate preCode, VoidVoidDelegate postCodeSuccess, VoidGenericHandler`1 postCodeFailure)

 

at F5Networks.ProgressTracking.Tracer.TraceCall(VoidVoidDelegate activeCode, String action, String area, String category, Int32 id)

 

at F5Networks.Protocols.iQuery.iQuerySocketBase.SendRaw(String rawCommand)

 

at F5Networks.Protocols.iQuery.TransactionalIQuerySocket._FlushSendBuffer(Boolean sendData)

 

at F5Networks.Protocols.iQuery.TransactionalIQuerySocket.SendRaw(String rawCommand)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._UpdateStatisticRulesOnDevice(Device device, iQueryConnection connection, Dictionary`2 collectionRules, IList`1 objectsRemoved, Boolean throwOnNetworkError)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._InitializeRulesOnDevice(DeviceDiscoveredEventArgs successContext, Device device)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._CompleteDiscovery(DeviceDiscoveredEventArgs successContext)

 

at F5Networks.ManagementPack.Services.DeviceMonitor._DeviceDiscovered(Object sender, DeviceDiscoveredEventArgs successContext)

 

 

What do you guys think is going on?

We do have a known issue where the BIG-IP can close our connection during the phase of discovery where we're verifying objects in Operations Manager if that phase takes longer than 10 minutes.

 

 

That issue would result in the error you've provided and means that it failed to start gathering statistics after discovery completed. Try running the "Refresh Collection Rules" task to restart statistics gathering. If that fails, restart the service.

 

 

I will continue to follow up on this issue internally and let you know what/when the comprehensive fix will be in place. If you can send the logs, I can tell you for sure whether this is the issue.

 

 

Thanks, Joel

Refreshing the collection rules did not correct the status issues I mentioned earlier. The output of the task is attached.

 

 

Right now, I have shut down an application that the F5 listens to. A pool member's status will go red if the application is shut down. As expected, the F5 shows the pool member as down (see second attachment). But, the pool member in SCOM shows as being up (see third attachment).

 

 

I will use this as my first test case for this issue. Let's focus on getting this working. What do you think might be causing this?

More things in the Event Log:

 

 

I noticed that an informational message (event ID 300) was recorded before the message I mentioned before:

 

 

Discovered F5 device at address: 10.41.52.27: connection established and stored in SQL [lv2devsql01:F5_ManagementPack] and OpsMgr [localhost]

 

 

Also, we have been getting lots of errors in the F5 Monitoring Log with an event ID of 704. They all mention something about failing to find an existing entity. Like below:

 

 

OpsMgr::UpdateDeviceConfig:Modify: Failed to find existing entity, auto-adding entity LtmPoolMember|QSBERRY-DEV/10.41.182.35:17777

 

 

Hopefully, this helps.

If you can please attach or send the logs again I should have a much better picture of what's going on since verbose logging has been enabled since the original discovery.

 

 

I noticed the "failed to find existing entity" issues before (which definately mean config is out of sync) but needed the verbose logs to see why those objects were not already discovered.