Traffic failover design between 2 data centers

I did some research and it looks like an f5gtm will be needed?

Right.

If so will it be needed at both sites or 1 site? I’m assuming both sites right?

Yes, both, making your DNS redundant.

And will it be a single gtm on each site or a pair of gtms on each site?

Depends on your pocket, one on each site will work but with more nodes comes more resilience.

As far as the secondary data center is concerned only 1 ltm should be enough right?

Yes, in practice DR sites have less capacities than main sites, though some clients (wealthy?) chose to make F5 in DR sites redundant too.

Also is there a template I can refer to for such a design?

F5 provides a DNS reference architecture, but I actually find the KB articles and guides more advanced technically.