NTLM Authentication - Windows Integrated 401 Challenge

I think it is probably less a BIGIP limitation than it is the design of the auth protocols in question. Kerberos clients need access to the KDC ticketing server (AD domain controller) before it can do anything and NTLM cannot "double-hop" to authenticate to another service because the password is never known.

So a combination of the two is what you are most likely after. NTLM authentication proxying to kerberos delegated service access. Of course the back-end service needs to support the kerberos delegation. If in ISA you had NTLM enabled and published it in a web publishing rule, if it was purely NTLM the ISA server was just a man in the middle and would, to my knowledge, challenge the user.

Not knowing exactly what service you are trying to get to work using the BIGIP, could you provide an example of something that was working in ISA and that is not working on the BIGIP?