Forum Discussion
Petersim_Simons
Nimbostratus
NimbostratusiControl error
We have a developed tool for enabling and disabling nodes on over 40 BigIPs world wide suporting over 2500 nodes. We have instances when the action failes with below error
SOAPException : 'ITCMCommon::OperationFailed Exception caught on ITCMLocalLB::Node::get_statistics(). primary_error_code : 335 secondary_error_code: 0 error_string : The requested server (node) service was not found.
From the active BigIP we can ping the node, enable and disable the node and all communication looks good. The tool can enable and disable other nodes on the same active BigIP pair.
Question: What specifically causes this iControl error and which troubleshooting steps should be taken to find root cause?
- Hi,
Can you provide the version of BIGIP you're running?
Also, you can enable iControl debugging on the BIGIP to see if the incoming list of nodes you're specifying is correct. Once debug logging is enabled, you should be able to see debug tracing on all incoming method parameters. Please note that iControl debug logging is very verbose, so this should only be done for debugging, and should be turned off when not debugging, or you risk filling up disk space.
To enable iControl debugging:
For 4.x BIG-IP (debug messages will be in /var/log/ITCM.log):
b db set Local.Bigip.CORBA.LogLevel = debug (use "none" to disable)
bigstart restart portal
bigstart startup corbalocallb
bigstart startup corbasystem
bigstart startup corbanetwork
bigstart startup corbamgmt
For 9.x BIG-IP (debug messages will be in /var/log/ltm):
b db icontrol.loglevel debug (use "none" to disable)
bigstart restart httpd
Loc 
Petersim_SimonsWe are running 4.5 PTF 11
Can you outline a theory behind why most of the other nodes works fine but some does not. What would cause some of the nodes to return this error message?
In general I would think the iControl code is accurate since it works for most of the nodes. All our node setup and BigIP configurations are identical. Some nodes does have two different pair of BigIP that they are load balanced with e.g. two different VIP's on two different pairs for two difeerent data flows.
Ideas?- Petersim_SimonsA follow-up question.
It is very difficult for us to allow iControl debugging and at the same time ensure it does not inadvertently casues negative impact on the BigIP pair. There are multiple orgnaizations that owns different support pieces. Don you not have an API that can capture more extensive error codes and forward these to a central deposit? - Petersim_SimonsPosted By Petersim on 5/6/2005 8:22:09 AM
A follow-up question.
It is very difficult for us to allow iControl debugging and at the same time ensure it does not inadvertently cause negative impact on the BigIP pair. There are multiple organizations that own different support pieces. Don you not have an API that can capture more extensive error codes and forward these to a central deposit?quote] - Petersim_SimonsPosted By Petersim on 5/6/2005 8:23:12 AM
Posted By Petersim on 5/6/2005 8:22:09 AM
A follow-up question.
It is very difficult for us to allow iControl debugging and at the same time ensure it does not inadvertently cause negative impact on the BigIP pair. There are multiple organizations that own different support pieces. Don you not have an API that can capture more extensive error codes and forward these to a central deposit? - I can't really put together a theory on why some of your nodes fail without having the right context of what's going on in the system when the errors happen.
You mentioned that some nodes are being managed by 2 different redundant BIGIP pairs. I have some questions:
1- When the error happens, is it for one of those mutually managed nodes?
2- Who manages these 2 redundant pairs of BIGIPs, different people in different departments like you had mentioned previously?
3- Is there a chance that one redundant pair of BIGIPs somehow interferes with the nodes that the other redundant pair is managing?
Loc
