ssh/https only on mngm interface

Because my L4 forwarding rule my customers are able to ssh/https using virtual servers addresses.   I configured all to allow only certain IPs, but I'm still being scanned by all source of differ...