I think an inband monitor will only detect connection failures (or with an HTTP profile no response failures):
Manual Chapter: Health and Performance Monitors
http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm_configuration_guide_10_1/ltm_appendixa_monitor_types.html?sr=138265381192393
With an Inband type of monitor, the BIG-IP system performs passive monitoring as part of client requests. Replacing the need to write an iRule to perform passive monitoring, an Inband monitor works with either a Standard or a Performance (Layer 4) type of virtual server.
When you configure an Inband type of monitor, the monitor, when acting as a client, attempts to connect to a pool member and behaves as follows:
If the pool member does not respond to a connection request after a user-specified number of tries within a user-specified time period, the monitor marks the pool member as down.
After the monitor has marked the pool member as down, and after a user-specified amount of time has passed, the monitor tries again to connect to the pool member (if so configured).
...
Standard with no HTTP profile
If you configure an Inband monitor and associate it with a Standard type of virtual server that does not reference an HTTP type of profile, a passive failure occurs when a connection is established with the pool member, or the pool member is responding to requests (or both), and the maximum retransmit times specified in the TCP profile are exceeded.
After a pool member has been marked down by the Inband monitor for the number of seconds specified in the Retry Time setting (if the value is non-zero), the pool member becomes eligible to receive a request. If the client is able to successfully establish a connection with the pool member, the pool member is marked up when the connection is completed.
How about using Chris' idea of checking the response status in SIP_RESPONSE, calling LB::down and then having the pool action on service down set to reselect?
Else, you could try using a external monitor to send the exact SIP request you'd need to to trigger a failure like the client would receive. This wouldn't reselect on a client request getting a 503, but would take the pool member out of load balancing if the member was giving 503's for all clients.
Here's a template you could try for an external monitor:
http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/TemplateForExternalLtmMonitors.html
Aaron