AD Query not getting all nested group

What version of BigIP are you running? There was a bug a while back where the memberOf attribute was cut short so if the group happened to be the last one returned then the session variable would not have it set when going to the resource assign logic of the VPE.

If that is not it then you can check to make sure using ldapsearch that the group is returned by AD (this could be an AD issue not an F5 issue).

ldapsearch -xLLL -H 'ldaps://x.x.x.x' -b "dc=fr,dc=del,dc=corp" -s sub -D "cn=Administrator,cn=Users,dc=fr,dc=del,dc=corp" -w mypassword' "(member:1.2.840.113556.1.4.1941:=cn=cal,CN=Users,DC=fr,DC=del,DC=corp)"

Modify the command above and I grep for "dn" to make the list easier to read.

[root@cooper-apm-11-6-0:Active:Standalone] rest  ldapsearch -xLLL -H 'ldaps://x.x.x.x' -b "dc=fr,dc=del,dc=corp" -s sub -D "cn=Administrator,cn=Users,dc=fr,dc=del,dc=corp" -w mypassword' "(member:1.2.840.113556.1.4.1941:=cn=cal,CN=Users,DC=fr,DC=del,DC=corp)" | grep dn
dn: CN=HwSales,CN=Users,DC=fr,DC=del,DC=corp
dn: CN=Onsite,CN=Users,DC=fr,DC=del,DC=corp
dn: CN=Sales,CN=Users,DC=fr,DC=del,DC=corp
dn: CN=VEND1-QA,OU=VPN,DC=fr,DC=del,DC=corp
dn: CN=VEND1-PROD,OU=VPN,DC=fr,DC=del,DC=corp
dn: CN=VEND2-DEV,OU=VPN,DC=fr,DC=del,DC=corp
dn: CN=Vendor A Prod,OU=VPN,DC=fr,DC=del,DC=corp
[root@cooper-apm-11-6-0:Active:Standalone] rest 

In this example "Sales" has member "HwSales" and "HwSales" has member of "cal".

The filter uses the "LDAP_MATCHING_RULE_IN_CHAIN" OID which AD will do the group traversing and return the groups. If this works and you see all groups then we can dig in further into how the APM works. If this doesn't work I would suggest keep looking on the AD side.

-Seth