Azure load balancer in front of F5 HA acrive/passive pair

I have an Azure LB setup with a public IP address, and both F5s setup in the backend pool. There is a health probe that hits an irule giving back a webpage with a 200 OK. The virtual server is setup using an address list with both the F5s configured as the destination IP. Everything works great on the primary F5. I have the irule logging the health probes and they are showing on the primary.

When I fail over to the secondary, I see the health probes move to the second unit in the log. However, the public IP no longer returns traffic. Traffic is resumed when I fail back to the original device. What am I missing here? I feel like the health probe would see the BigIPs switch active/passive and stop passing traffic to the offlined unit. Or am I getting bit by a SNAT or ARP issue here? Any advice would be great. Thanks.