I'm trying to configure the DNS feature of a BIG IP box. I'm using this same box as LTM already.
The DNS feature I'm trying to configure is for external use, but the vservers are configured with the private ip addresses.
What I would like to do is, if someone tries to access application.domain.com, the F5 DNS feature should answers with the public IP of Nat configured at the firewall, that then sends the traffic to the private ip of the vserver on the same BigIP.
You have answered your own question, NAT plays a major role here.
You cannnot reach the private range from the Internet. You'll need to NAT your private IP to your public IP.
Build your DNS setup, make sure its listeners are configured as NS properly for DNS resolutions. Refer the DNS build articles. Add the servers (LTM) with iquery. Because if iquery is setup properly & working, then DNS resolution for external to internal mapping should also work.
You'll have to put the actual public facing address as destination address & in the NAT field define your internal address. This requires co-ordination with your internal team, to know the routes, firewall, range etc.
Just realized, there's also split dns feature, which you can try to research on. Use your existing gtm to handle both external requests & internal requests.
But if you don't need the DNS name to be resolved to internal ip address if the user is internal just read this article as the F5 DNS/GTM virtual servers can be configured with a NAT ip address that will be provided to external users(but then the virtual server auto discovery will not work and you need to configure each virtual server with its real and nat ip addresses):
