Blocking specific hostnames to send emails through SMTP Realy

That's because it is SMTP traffic and not HTTP traffic. Applying a HTTP profile will just block the traffic.

When I said not easy, you then have to start writing iRules to interpret SMTP traffic. This is not a simple thing to do. See the codeshare for examples https://devcentral.f5.com/s/global-search/%40uri#q=smtp&sort=relevancy&f:@f5_url_filter=[Codeshare]