Splitting traffic to test environment

Hi Sheigh,

We're using splunk here, but I am hoping to use it for both performance and functionality testing.

Basic procedure:

1. Configure TEST with the same memory / cpu as production. (Most services are in our VM environment).
2. Clone production backup to test.
3. Begin test (some differences will be inevitable, but should be ok.)

Performance testing: Track the usual CPU / IO / Memory within both environments to see if they respond the same time. For web servers we create a table of response time, with log response size and average, max, 90th percentile, 95th percentile and 98th percentile across the top.

Functionality testing: Basically track the response codes between production and test. Allow a certain number of 403's above what the production sees. Probably look at each transition (200->403 == file missing on test, 200=>500 == something is not quite working correctly).

Obviously the results will vary greatly depending on what we've changed, but for a lot of changes we can get pretty close to a decent generic functional and performance testing procedure.

Cheers,

Greg