it would help if you share more details, you do provide a lot of settings but don't provide the actual values.
also the exact scenario is unclear to me. you have the system setup, you start a ping on a vip, you power the active one down hard? then it takes minutes (how many, again details) for the secondary to start replying?
what do you see in the GUI, does the secondary device become active according to the GUI? have you tried a manual failover? does that go better?
it kinda sounds like a L2 problem, the packets are still send to the wrong switch port because the mac table isnt updated. can you check the MAC tables on your switches? if you clear them, is the VIP reachable directly afterwards?
a lot of check and try, good luck.