Forum Discussion

teguh_wilidarm1's avatar
Icon for Nimbostratus rankNimbostratus
Apr 26, 2011

More on GTM + LTM design across 2 DC

Hi, hope friends here can share their thoughts. I searched similar topic on this forum and found this one:



Now I need a little expansion from that previous topic.



Here is what we have:


1. Two GTM machines in active-standby in DC


2. Two GTM machines in active-standby in DRC


3. Two LTM machines in active-standby in DC


4. Two LTM machines in active-standby in DRC


5. Three ISP in DC and in DRC, lets say ISP A, B and C.


6. A redundant application server that can only be active in one site, either DC or DRC. When it is up in DC, the server in DRC should go off, and vice versa.


7. The traffic flow is incoming, the user is accessing this servers




Here is what they need:


1. GTM to offload NS resolution


2. LTM to front end the app server


3. GTM and LTM should be able to accomodate the A record resolution using those three ISPs (round robin)


4. (If possible) In the event when app server in DC active, then the monitor in LTM reports as active to GTM. Both GTM DC and DRC communicates with each other, so every DNS request coming to either DC or DRC, will resolve the wide IP to DC resources. In opposite, when the app server in DC is shut down purposefully, LTM in DC monitors it as off, reports it to GTM in DC. GTM in DC updates the GTM in DRC, so every request will get the wide IP resolution from GTM and LTM in DRC.




Here is what I will do to cover 1,2,3:


1. Set up three VS in LTM. VS for ISP A, ISP B and C. Each VS has the address from each ISP's segment. Pool member for those three VSs is the same, which is the app server.


2. Set up the LTM interfaces to have (sub)interfaces for each ISP as the way to communicate with the ISP router.


3. Set up a wide IP in GTM that has the pool member: VS IP ISP A, VS IP ISP B, VS IP ISP C.




Now to cover requirement number 4:


1. I searched about iQuery (as a way between GTM and LTM to communicate, and also for GTM in DC and in DRC to communicate) but limited documentation on how to implement it to my topology.


2. I am thinking about :


a. modifying the pool member GTM in DC to: VS IP ISP A, B, C, VS IP ISP A (DRC), ISP B (DRC), ISP C (DRC), classify the ISPs in DC in one class, and ISPs in DRC in another class, and make GTM to choose only the DC class if the server is active in DC and DRC class if the server is active in DRC (not sure how to do this) and vice versa in GTM in DRC.


b. Finding a way for GTM to answer NS resolution accordingly. So when server is active in DC, even though a request comes to GTM in DRC, it will tell the requestor that the NS is now on GTM in DC, go ask that person in DC. (so not sure about this)



Hope I could get some answer here



Thanks before






6 Replies

  • Your setup sounds very similar to ours with the exception that you have redundant GTMs. One note.. when setting this up.. auto-link discovery is BAD per F5 professional services and F5 back-line support. Don't turn it on.. ensure it's off.'



    First make sure you've got NTP setup on all servers. Also, make sure your GTMs are all at the same level and are at the same or higher revision of software than the LTMs. ( ideally every device is running the same version )


    Make sure you've done the self-certificates, provisioning and licensing on all of the GTMs as well.



    Installing LTM & GTM certificates on your GTMs



    Your GTMs need to communicate to each other and to your LTMs. This is done through using ssh and port 4353. On every GTM you will need to run "bigip_add [ipaddr]" against a reachable IP on every LTM and GTM it will communicate with. These IP addresses on the target GTM and LTMs must have port 22 and 4353 allowed on the self-ips that will communicate. These IP addresses must be self-IPs that are NON- cluster addresses ( no failover IPs ).



    Example: on GTM00DC run, "bigip_add [gtm00drcl ip]" enter in gtm00drc's root password when prompted. Then repeat for each LTM ( both the active and standby ) in both datacenters.



    Only one self-ip is required for communication.



    Setting up GTM-GTM syncrhronization



    So far.... You've already traded installed the certificates between your GTMs and added all of your LTMs to both GTMs as well. ( see above and the "bigip_add" command ). You've also already created identical "Data Centers" on each of the GTMs.



    Now you need to setup your GTMs in a synchronization group ( they will still be standalones even though they are synchronizing.. yeah, odd.. a redundant pair is only two in the same location).



    On the first GTM ( either one ):



    "System">"Device">"Global Traffic". Select the Synchronization box and create a name for the "Synchronization Group Name" field. ( You'll use this EXACT same name for all GTMs that will be synchronizing DNS entries.



    On the First GTM: ( You've already created identical datacenters and done the certificate swap between all LTMs and GTMs right? ... check to be sure )



    "Server" then click "Create". Type the name of the other GTM, enter in the IP address and select the appropriate "Data Center". Add the "bigip" monitor, and enable "Virtual Server Discovery" but DO NOT enable "Link Discovery". Your second GTM will pull all server definitions from the other GTM and will stay synced should you add a Server or DNS entry to either of the GTMs in the future.... AFTER we do the next step!



    On the second server from the CLI do a "gtm_add [ipaddr]". This will initiate the first pull and will keep you in sync from now on. Adding a DNS record to either side, or a server to either side will be synced to the other GTM(s). This is a bi-directional sync, add information to either side and the GTMs will handle the syncing. Synchronization is based off of file timestamps.




    Then add the LTMs to either GTM definition.


    Adding a "Server" is not creating a "WideIP" nor is it creating a "Virtual IP".. a "Server" by GTM definitions is another load balancer ( LTM for us ) or GTM.



    * From the GUI select "Global Traffic"->"Servers" and click "Create"


    * Give the server a name ( doesn't have to be a resolvable fqhn or anything like that... this is just descriptive. )


    * Select the type of server ( If you select a "Big-IP System (Redundant)" it adds fields for the second system ). For the address do NOT use a relocatable IP. Static Self-IP only.


    * Select the data center and the status. ( enabled )


    * For health monitors, select "bigip" equipment. ( We're not doing anything else currently


    * Under "Resources" set Virtual Server Discovery set to "Enabled".


    * Leave "Link Discovery" set to "DISABLED"!! Rule is: server discovery good.... link discovery bad


    * Then click "Create".



    If all is correct you'll see information start to populate. What? You don't see it populating.. Aha! That's because you'll need to add device certificates between devices for effective communication. To do this, log into your GTM and run "bigip_add [ipaddr]" Where ipaddr is the IP address of the LTM or GTM you are adding to this device.





    I am not certain of your end goal design, but it sounds like you'll want to setup the six VIPs ( one for each ISP in each DC ... three on each LTM pair ). If your LTMs are using the same pool for each VIP then when the pool becomes unavailable all three VIPs in that DC will become unavailable. With the LTMs reporting up to the GTMs, they'll note the VIP as down and not give out it's IP address. Your GTM in each datacenter should be authoritative for whichever domain your DNS entries will be located in. Each GTM should have a 'listener" that is registered as the authoritative IP for that domain.. the way DNS works is that the first one of those IPs to reply to a query will be the one that is used.. so there's no need to have a GTM in one datacenter re-point people to DNS in another datacenter. ( You could do this... but it introduces complexity, points of failure and is simply not the cleanest solution ) ..



    With this setup, your GTMs will see all the VIPs and all the LTMS and will handle failover. If a GTM or even a redundant pair of GTMs disappears, the remaining ones will still have a listener up and available that will be hit by any DNS resolution queries.




  • Superb! Jason, thank you very much!..


    we are in implementation phase right now and your response do help us abundantly, especially with the certificates..


    I will run through your hints one by one, and hopefully it works, with only minor troubleshooting..


    Will update back soon..







  • Hi Jason, what is needed to make sure that the LTM and GTM in DC and DRC can communicate correctly over the Internet?


    The DC and DRC in my topology is geographically separated.



    And btw, are your steps is what is meant by activating iQuery?



    Thanks before




  • To get the nodes to communicate you'll have to ensure the following:


    "These IP addresses on the target GTM and LTMs must have port 22 and 4353 allowed on the self-ips that will communicate."


    So.. your GTMs will need to be able to communicate to a self-IP on each of the LTMs that have "custom" ports allowed of port 22 and 4353.


    Your GTMs will also need to be able to communicate to self-IPs on each other ( the pairs will be doing this via the crossover cable ) using these same ports.


    To set these up you'll need to go to the "Network" section.. then "Self-IPs". Then select the Self-IP that will be used for communication. The communication is secure using ssh, but I highly recommend that all of the communication occur on an internal subnet. For IPV4 use an RFC1918 subnet. For IPV6... you'll have to find someone that knows more about IPV6 than me to get a recommendation.



    iQuery is the API that the bigips use to communicate with each other.



    Hope this helps.



  • Hi Jason,



    We are trying to implement similar design except GTM and LTM are separated from firewall and LTM are configured with private IP address space. GTM. and LTM can. Communicate on iQwuery fine in the same data center once the firewall is open for iquery ports but not from other data enter, ie GTM from DC1 can not communicate to LTM in DC2 and vicevesa. I think it because GTM from remote DC is not able to communicate to LTM over private network.



    I'm thinking of casting VIP on LTM public address and add that VIP as a server in GTM . Will that work.


    Can. You suggest some option to address this issue. Also, NAT on firewall is not an option.