Rewrite header "WWW-Authenticate: Basic realm="

You can certainly rewrite an HTTP response header, but I'm curious how this would work if you changed the header value to Xbasic. When a server sends a 401 WWW-Authenticate response, it can send a value of Basic, Negotiate, NTLM, or Digest. These are generally the only "integrated" authentication methods that a browser will understand. In other words, upon receiving an "xbasic" authentication response, a browser is very likely going to simply fail. If, however, you're using the BIG-IP to authenticate to the server in another way, or the BIG-IP is sending the credentials on the user's behalf and you don't want the client to see the 401 response, you could simply drop these responses.

What I'm saying though is that the browser will most likely not handle this non-standard 401 the way you're expecting. As a test, try this on a simple VIP:

when HTTP_REQUEST { 
    if { not ( [HTTP::header exists Authorization] ) } {
        HTTP::respond 401 WWW-Authenticate "Xbasic realm=\"TEST\""
    }
}

What does the browser do? In all of my testing the browser does absolutely nothing. It just stalls.

So if there's no authentication on the backend, and apparently no authentication on the frontend, what are you trying to do?

Well, since the 401 is coming from the VIP itself, you'd need to do add a VIP-targeting layer on top of this VIP to catch these responses.

1. Give your existing application VIP and internal (non-routable) IP address, set for port 80 and remove the client SSL profile (the external VIP will handle the client side SSL).

2. Create another VIP and give it the public address, port 443 and client SSL profile, and the following iRule:

   when CLIENT_ACCEPTED {
       virtual [name of internal virtual server]
   }
   when HTTP_RESPONSE {
       if { [HTTP::status] eq "401" } {
           ... do something here.
       }
   }
   

I can pretty much guarantee that if you simply change the WWW-Authenticate header to something non-standard, the browser will just hang. So you may want to test a few different options, like simply dropping the 401 responses, or issuing an HTTP::retry until the 401 goes away.