Need persistance assistance

The below irule seems to work ok in limited testing, but there are a couple scenerios that could cause it to fail.

Hm. I'm not sure I understand. Are you saying that you think these could cause it to fail, or are you actually seeing failure that you'd like to diagnose?

What if a node fails and the 300-400 clients that were connected to the failed node all reconnect at the same time?

Since there could then be multiple requests at the same time and a few hundred milliseconds before the first connection populates the session table, could this result in load balancing multiple clients with the same "channel identifier" to different nodes?

Is there some sort of serialization or locking in the session table that could/would prevent this?

iRule execution is single threaded, so all commands for a connection in an event will execute without being interrupted. You don't need to worry about a second connection's CLIENT_DATA event firing while you're in the middle of the first connection's event, for example. However, you could have multiple clients with the same "channel identifier" have their CLIENT_DATA events all fire before any of their SERVER_CONNECTED events do, so at the least you should probably change your test in SERVER_CONNECTED to check the persistence table and not the result from CLIENT_DATA that you stored in $PersistTo.

What can I do to make this work without rewriting the application?

Well, if what you posted is the entirety of your iRule, I *think* you just need a straightforward UIE persistence setup. I'm pretty sure that looks like:

when CLIENT_ACCEPTED {
    TCP::collect 44   
}
when CLIENT_DATA {
   set ConStr [TCP::payload 44]
   set SessionID [substr [getfield $ConStr "@" 2] 0 " CHID" ]
   log local0.info $SessionID
   if { $SessionID equals "" } {
       No SessionID, just load balance as usual
      log local0.info "No session ID, load balancing the connection."  
      pool YOUR_DEFAULT_POOL_NAME
   } else {
       All clients with this SessionID go to this server
       timeout is 120 seconds
      persist uie $SessionID 120
   }
}