I have enabled Geolocation TPS-based DoS Detection, but some legal users (Around 10 users) from one country encountered Captcha page. The web site is still not opening to public, so only internal users can access. I wonder why F5 ASM will treat this as attack.
you can start checking the logs what was the root cause of the block request and (captcha challenege)?
because there are geolocaiton enforcmenet in two different locations, one for the DoS profile as you mentioned, and there is another one inside the ASM policy itself.
Regarding the private IPs, you can select (N/A) in the enable list or "allow access" inside the ASM policy as per the below link:
If the captcha was generated because of the DoS profile, you can check what country was matching with these blocked IPs (which were legal internal users) and enabling this country in the DoS profile.
Also, keep your IP geolocation database up to date by following the below article:
