Irule to check if traffic SMTP is with authentication or not

As there are no SMTP iRule events you will need to use TCP::collect to capture the TCP data (for SMTP over SSL SSL::collect) and you can see https://clouddocs.f5.com/api/irules/TCP__collect.html .

Ah! so.. just have the same IP address and 2 different ports. With F5, a VIP is defined as an ip/port combination. A virtual address is just an IP.  They are different object types.. with a VA responsible for more L2/3 functionality, rather than L4-7 on the VIP. So a VA can have n number of VIPs attached to it.

All of your email heads toward the IP address for mail. The port 25 VIP will have a port 25 pool. The port 587 VIP handles the auth'd traffic... Still same IP address for both.

Regarding resending with auth, I'm certain there's a way to do it in iRules, but I doubt it would be worth it, as the iRule would need to collect client data, then find auth, but then it would likely need to apply the auth for MANY different clients.. you would likely need a way to process traffic per-client. I think your administration would be a nightmare and also that your BIG-IP would suffer a HEAVY load penalty from this iRule.. especially if a hacker figured out what you were doing and dropped a spam bomb on you. I was a mail administrator in a former life.. been there.