Add servers to pool, but don't want them used unless no other servers available

You are correct. To be specific though in order to keep the DR servers from having any traffic before you configure the priority activation group you would want to add the Node first, disable the node, then add the servers to the pool and configure their priority numbers. The higher the priority the higher it recieves traffic. You can set DR to only go active if there are less than the desired number of prod servers. Do this by going into the pool settings and enabling pool priority group activation on the members tab. Set the number to 1 above the number of prod servers you need in service to add the secondary priority group. So if you set the number to 1 then as soon as 0 prod servers are available all the priority 1 servers will become active. It will add all the servers of the next highest priority group.

In answer to will it take the VIP down when reconfiguring it. It will not take the VIP down. It will only impact new connections. The old connections will continue to use the old settings.