Issue with iRule and display of Block page

From my experience with Java applets and trying to do redirects, I don't know that it's possible unless the functionality is built into the applet to accept a 30x level response.

If you are more concerned about the brute-force attack than the redirect, you could just drop or reject the request when it violates the rules so you don't get unnecessary authentication attempts on your auth server.

See Drop, Discard, Reject for more details on those commands (Drop and Discard are basically the same command. Reject is a little bit different).