Forum Discussion
Mike_Thompson_9
Nimbostratus
NimbostratusInvalidOperationException - HELP!
I have a SOAP interface which makes calls to the F5 to get data about stats, pools, members, etc. You ask, why write another web service that talks to the F5. Simply because I have simplified a lot of the tasks needed to get the data out of the F5 for different scripts around the company. In other words, they can call my service, pass in a few args and get lots of data back.
To get back to the point, 90%/95% of the time, this service works. However, the other 5%/10% of the time I get the following:
System.InvalidOperationException: Client found response content type of 'text/html; charset=iso-8859-1', but expected 'text/xml'.
Sometimes this happens when calling get_session_status(String[] pool_names) and other times it happens when calling get_monitor_status(String[] pool_names). However, I belive it can happen on any of the functions being called.
It seems to me that the F5 is choking on the request and throwing an exception back to my service in the form of HTML. However, there is no HTML to read in as part of the exception.
This is the full exception:
: GetPoolMemberStatus_Error_InvalidOperationException: System.InvalidOperationException: Client found response content type of 'text/html; charset=iso-8859-1', but expected 'text/xml'.
The request failed with the error message:
--
--.
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at LocalLB.PoolMember.LocalLBPoolMember.get_session_status(String[] pool_names) in c:\Users\thompsmd\AppData\Local\Temp\Temporary ASP.NET Files\f5manager\0ec0b867\38653d1b\App_WebReferences.5.cs:line 588
at WebSvcMgr.GetPoolMbrSession() in C:\My Data\ASP.NET Applications\F5Mgr\App_Code\WebSvcMgr.vb:line 182
at WebSvcMgr.GetPoolMbrDetail() in C:\My Data\ASP.NET Applications\F5Mgr\App_Code\WebSvcMgr.vb:line 162
at F5Services.GetPoolMemberStatus(String[] inServerNames, Int32 inEnvironmentID) in C:\My Data\ASP.NET Applications\F5Mgr\App_Code\F5Services.vb:line 161
As you can see, it is dying on ReadResponse.
Can anyone help here or offer some insight?
Mike_Thompson_9Are there logs from iControl that we could look at on the device that would tell us what the exception was and why it was not handled? I am not on the F5 team at my work place, so excuse my ignorance.
Here is the other strange part. My app is a .NET web app that makes calls to the F5 using the exact same methods and NEVER has this problem. The ONLY time this happens is via the asmx page in the web app that people can call to simplify calls in to the F5. So, it is almost like there is some sort of issue when the call is initiated by another web service in to the F5. Almost like the data is getting corrupted from time to time as it goes from the client to the web service to the F5, but is nearly impossible to track down why.
I was doing a retry from within the asmx page, but that was a nightmare as well. So, I pushed the retry back to the client that initiates the call. If the client sees an error, then he retries and that works fine (but I am not happy with this solution). My next step would be to open a ticket with Support. However, it would be nice if I could find some logs on the device to tell me what happened and why....
Thanks Joe...- Another reason for an exception, as you pointed out, very well could be corrupted client data. I've heard of issues when ASP apps are using keep-alive connections (re-using HTTP connections). If you are using the iControl assembly, that reuses proxy class instances, this could be your problem. A quick and easy way to enable iControl logging on the BIG-IP might be your best bet in diagnosing this. Here's the steps
1. Edit your /etc/syslog-ng/syslog-ng.conf file on your BIG-IP and make sure that the local4 settings are as follows:local4.* /var/log/ltm
filter f_local4 {
facility(local4) and level(debug..emerg);
};
2. Set the database key for iControl tracing$ b db iControl.LogLevel trace
3. Restart the httpd server$ bigstart restart httpd
Now you'll be getting a bunch of data in the /var/log/ltm file including the first 2k of the XML payload along with the data after the server side has extracted it.
***Make sure you turn off the tracing when you are finished with your testing as it will dump a lot of data to the log file.$ b db iControl.LogLevel none
$ bigstart restart httpd
Hope this helps and let me know what you find out...
-Joe - Mike_Thompson_9As soon as I get back to work after the weekend, we will give this a go and I will let you know the results. The problem seems to occur with all the devices in all our environments. So, we will give it a go in Dev and see what happens. Again Joe, your replies are invaluable!
Mike - Mike_Thompson_9Ok, I was finally able to get logging enabled and isolated on the Dev F5 and found some interesting results. First off, I start my test harness application which essentially makes the same call to the ASP.NET web service over and over until it reaches an error.
So, I start the test app, then open the F5 web console and navigate to the ConfigSync menu. Once this is open, I hit the Update buttone once or twice which appears to initiate some SOAP chatter between the two devices. When I do this, my test application stalls and eventually throws a timeout exception:
System.Net.WebException: The operation has timed out
My timeouts are currently set to 45000ms (45 seconds) on both the client and web service, which should be more than sufficient. Also, I am able to duplicate this problem over and over. All that is needed to get this to occur is to click the Update button on the ConfigSync screen while another web service call is being made.
Still trying to get the IOE exception though. However, I do recall seeing some device chatter on or at the same time I received the IOE error in a previous test. Will keep you updated.... - Mike_Thompson_9Apparently these calls can be manually initiated by calling "bigpipe config sync show" and "bigpipe config check" (according to the logs). However, if I manually make either of these calls from the command line while my SOAP calls are being made, I cannot get it to fail. Unless they need to be made at just the right time in the cycle and I cannot duplicate.
I am also assuming these calls get made at timed intervals by the system. If so, these are the ones that appear to affect my SOAP calls. So, a manually initiated config check call does not appear to cause a problem. However, if I am making a SOAP call when the system initiates these calls, then I get an IOE exception on my client.
[Update]
Yes, these management calls do get executed at automatic timed intervals. According to the logs, these calls get made every 30 seconds.
Doest this make sense at all? - That makes sense. The iControlPortal is implemented as a single FIFO queue to avoid two callers from stomping on eachother. If there is a process (in your situation ConfigSync) that is communicating between the boxes for a long lived connection, then if you are trying to make an iControl call to the same target box as your BIG-IP is, you'll have to wait for the first call to return before yours can be processed. I'm surprised that this is taking longer than your intervals. The only other issue I could see is that a load of the configuration forces a restart of httpd which will cause iControl connections currently in the queue to be aborted.
I'm going to have to get my head around a viable solution to this issue...
-Joe - Mike_Thompson_9I went ahead and opened a support case for this issue. More than likely the problem will turn out to be more involved than what we could resolve here on DevCentral. In any case, I will be sure to keep the updates posted here. If you want the case number Joe, I can send it to you.
Mike - Go ahead and throw the case number up here on this thread. That way others can refer to it if they come into this issue.
-Joe - Mike_Thompson_9Case number is C377954. As things progress, I will be sure to update DevCentral as needed...
John_Arnold_106Did you guys ever get an answer to this issue? I'm (we're) building a similiar tool and seeing similiar errors -- very low frequency (0-15 times/hr) on VERY static iControl calls to test-only lab devices. Is this a call contention issue -- one call (to save config) ties up the iControl portal until other iControl calls timeout?
Thanks,
John
