Post Upgrade Testing

Hi All...I'm curious as to the process you use to perform post upgrade testing of Prod apps. As like other companies, we have many apps/sites on each pair of f5's and we've struggled to get a process in place for testing apps after upgrades. With between 200-250 apps/sites on each pair, of f5's, it's almost impossible to test all the apps and perform an upgrade during our maint window. So what I typically do is upgrade one of the two f5's, test our top 5 apps/sites immediately after the upgrade. If those 5 apps/sites are good, then we run on the upgraded f5 for one week. During that week, the app owners should be doing thorough deep testing their apps as we burn in the new f5 version. If all is good at the end of the week, i will then upgrade the second f5 of the HA pair, and app owners do a second round of lite smoke testing. FYI...we do have a QA environment but QA apps/sites do not fully match Prod apps/sites, so we can't rely on testing QA apps/sites after upgrading the QA f5.

So to summarize...

1. I upgrade one f5 and test top 5 apps during maint window
2. Run on new version for a week and all other remaining apps are tested during that week
3. If no issues, then after one week, upgrade the second f5 and then another round of lite testing

I'm curious how others handle app/site testing in conjunction with upgrading HA pairs.

Thanks