Hi quangtran,
I'm really not sure at this point about what's wrong, but I always find that some logging within iRules can help you in debugging what's happening.
For instance, you can add a line like this:
log local0. "encrypted data = $enc_data"
...after the CRYPTO::encrypt function, to make sure what has been calculated thus far in your iRule makes sense.
With local0, the logs will be inserted in /var/log/ltm.
If your VS is running on plain HTTP, a simple tcpdump will also help understand what's coming in and what's going out.
/Mike