active/active sites.

we've always been a one datacenter shop but now we'll be running active active from different locations. i do have a scenario that i've been unable to solve without a pretty gross hack.

 

 

here's the old 1-datacenter setup:

 

 

servers running a web app die. load balancer says "hey, no nodes are up" and reroutes to a "downtime" page. pretty simple, right? this however becomes silly when you have two datacenters.. here's what happens with no changes:

 

 

servers running the webapp at datacenter 1 die and downtime page is returned. app at datacenter 2 is running so the app is returned. some users get the downtime page, others get a working page. problem!

 

 

the two f5 pair are able to talk to eachother and eachother's members privately. the only solution i've come up with so far is making two pools for every single app, one for the app servers in each datacenter. here's the rule i'd create on the f5 in datacenter 1:

 

 

if (dc1 pool == up) --> working app

 

if (dc1 pool != up) & (dc2 pool == up) --> reject connection

 

if (dc1 pool != up) & (dc2 pool != up) --> downtime page

 

 

the opposite would exist on datacenter 2. it seems like a semi-reasonable solution but is obviously a hack.

 

 

does anyone have experience with multiple site solutions along with downtime/failover apps?