Rewrite header "WWW-Authenticate: Basic realm="

Well, since the 401 is coming from the VIP itself, you'd need to do add a VIP-targeting layer on top of this VIP to catch these responses.

1. Give your existing application VIP and internal (non-routable) IP address, set for port 80 and remove the client SSL profile (the external VIP will handle the client side SSL).

2. Create another VIP and give it the public address, port 443 and client SSL profile, and the following iRule:

   when CLIENT_ACCEPTED {
       virtual [name of internal virtual server]
   }
   when HTTP_RESPONSE {
       if { [HTTP::status] eq "401" } {
           ... do something here.
       }
   }
   

I can pretty much guarantee that if you simply change the WWW-Authenticate header to something non-standard, the browser will just hang. So you may want to test a few different options, like simply dropping the 401 responses, or issuing an HTTP::retry until the 401 goes away.