If don't want to buy another device you may consider SMTP relay services like SendGrid etc. that are cloud based offerings. Symantec even has Email Security.cloud but from F5 point I think that they have given up on this market :
So is that mean, i can create data group contains all blacklist domain then create irule to check the blacklist domains and it will be assigned to email VS ?!
About the irule I don't think there is exactly what you are looking for but probably it could be written but it will take a lot of time to capture the traffic with TCP::collect and to match on the "From: user@domain.com" value in the emails but I admit don't have a use case to try do it. This is why I gave you the example with the iRule with smtp-filter-and-proxy.
For PTR if your device is not the DNS/GTM you may extract the needed value from the email payloadd and with the "RESOLV::lookup" to check it as I don't know if ''NAME::lookup'' supports PTR resolution:
A good note is you can use free feed lists as alternative to the F5 URLDB service for the SWG module like Minemelt if you have AFM you can add custom feed lists and if not you may check the tabul CVS importer iRule that I also use for some things.
