irule redirect and masking

A redirect to "http://extranet.site.com.au" will cause the browser to show that address. If you want to keep the browser on "https://Mainsiteurl.site.com.au", then your first iRule code was probably more appropriate. That is, instead of physically redirecting the user to another host, you must send the traffic through the Mainsite VIP.

when HTTP_REQUEST {
    if { [string tolower [HTTP::uri]] starts_with "/extranet" } {
        HTTP::uri "/[string range [HTTP::uri] 9 end]"
        HTTP::header replace Host "extranet.site.com.au"
        pool extranet_pool
        SSL::disable serverside
    }
} 

The above assumes that the Extranet service is in a different pool of servers than your regular traffic, and that your regular pool of servers are listening on SSL.