priority LB - maybe better suited to iRule?

I would suggest you add this:

when HTTP_REQUEST {
   pool main_pool
}

This will set/reset the pool to main_pool on each request. The default pool setting on the virtual server is only the initial pool request. Once the pool has been changed for a given request, all subsequent requests on the same connection will be to the subsequent pool setting. Setting/resetting the pool in the HTTP_REQUEST event will ensure each subsequent request will first be routed to the main_pool, then only after a failure on that request switched to the failover pool.