Change Server SSL Profile within APM Policy based on Users group membership

Yes, I've tried excactly this and it seems to work. However I suspect that if doing this I will cause all connections made from this point to always choose this SSL profile unless I change it back? I can solve this by using a separate "catch" VS for this particular server. It's still a little bit messy solving it this way and I will try to use a data group where key=group and value=new ssl profile...