SCOM F5 Objects Not Consistent with Objects on F5

Here are the second set of logs.

I'm having trouble with some missing portions of the log. The attached log stops at 12/07/2009 13:02:24, then starts again from 12/08/2009 13:23:44 to 12/08/2009 13:49:20.

 

 

Do you have multiple management servers or have you archived any of the log files? If so please send..

 

 

If not could you please restart the F5 service (which flushes cached output to the log), then resend the log?

 

 

Thanks, Joel

We don't have multiple management servers. I have restarted the F5 monitoring service. I pulled the trace.log file immediately after I reset the service. It is too big to attach here. I will send on an email.

Thanks. A couple observations from the log:

 

 

- There are a couple networking errors while communicating with the big-ip which could be problematic. We are looking into those.

 

- I do not see any of the "Failed to find existing entity" errors since verbose logging was turned on yesterday which would indicate out-of-sync config

 

 

Ideally what I am looking for is the "Failed to find existing entity" error. Do you have any of those errors still showing up in your F5 Event Log?

 

 

If you see another one of those errors in the event log, please restart the service (to flush the logs) and send the trace log again.

 

 

In the meantime, I'll look into the networking error that was in the log.

I don't see any more errors like that in the event log since I performed the re-discover at 5:30 PM on 12-7 (see attached screenshot). The error I have seen since then was the iQuery error mentioned above.

 

 

As for the rest of the information messages seen in the attached screenshot, they are of two varieties: event ID 100 and 300. These are pictured in the other two attached screenshots.

 

 

As far as trace logs go, I restarted the F5 monitoring service and sent the logs I found in the "C:\Program Files\F5 Networks\Management Pack\log" directory. Last couple times, I have sent just the trace.log file found at the location above. Since you haven't been able to find what you need in there, I decided to send everything in the directory. Let me know if you need the setup.log file too.

I should also add that the SCOM object is still out of sync with the F5 object. I have a pool member that shows as down on the F5, but is shown as healthy on SCOM.

Another caveat to throw your way: we use the same F5 LTM device in our DEV and QA environments. Each environment has its own SCOM server. So, two SCOM servers from two different domains are connecting to the same F5 device. Given your question about multiple management servers, I assume this might have some bearing on this problem.

The extra logs (with the long GUID names) helped to fill in the blanks. Sometimes we have to create a separate logging file from the main trace.log if the the file gets locked for some reason. The service can end up competing with itself for the trace log sometimes if it restarts quickly.

 

 

It definitely appears that the "failed to find entity" errors were fixed by the rediscovery. The rediscovery worked despite the error that was displayed which as I mentioned is a known issue when there is a large number of objects on the BIG-IP. The side effect is only that the service initially could not start getting statistics and is self-fixing over time (or you can use the Refresh Collection Rules task to work around).

 

 

As for the pool member that is not showing the correct health, I see the following health updates from the BIG-IP:

 

 

12/09 3:12:23 PM -- Monitor is up (GREEN)

 

12/10 7:24:23 PM -- Monitor is down (RED)

 

12/10 7:24:27 PM -- Monitor is up (GREEN)

 

 

The end result of this is that the pool member should show green. It seems possible that the BIG-IP did not send a notification that the monitor went down again. Does this timeline concur with what you know about the pool member?

 

 

I would recommend selecting the pool that the pool member is in (in OpsMgr) and using the Refresh Health task to sync it up. If that does not cause it to show the same health as on the BIG-IP, I can investigate further.

 

 

 

 

Also, to address the caveat you mentioned regarding DEV/QA environments -- this should not be a factor. It sounds like you have to RMS servers, not one RMS server and one Management Server correct?

That is correct, we have just RMS servers.

 

 

As for the pool member's status, I don't have a "Refresh Health" task (see attached screenshot). Did you mean the refresh button when you talk about the "Refresh Health task"? If so, it won't show the correct status no matter how many times I click it.

 

 

Maybe you meant the Recalculate Health or Reset Health buttons on Health Explorer?

 

 

The state change events you mention seem to be the correct times. But, they don't entirely agree with the events I see in Health Explorer for that pool member (see second attached screenshot).

 

 

The F5 must have sent out the notification that this pool member was down. Our QA SCOM server has the correct status for the pool member (see third screenshot).

 

 

Can you tell me more about these notifications that you speak of? What protocol do they use? Can they only be sent to one recipient, or can they be sent to multiple monitoring applications?