how much data can be collected in irules for one connection?

There is no built-in limit for the amount of data you can collect, but if you collect a lot of data on a lot of connections, you can use up all of your box's memory. The 4M limit mentioned above is the limit of the size of a Tcl variable. So while the payload can be arbitrarily large, you can only operate on it in ~4M chunks. If you're on 10.x, you can supply an offset to TCP::payload and get the chunks that way. If you're not, then you'll have to chunk the data yourself as you collect it (put it into a set of variables and remove it from the TCP::payload and then add it all back in at the end).