How to remove 1 line from a virtual server config with REST

I have a virtual server where i want to add or remove irules with the push of a button. It is easy to add the irule with a POST command but removing this specific irule gets a lot trickier when there are more irules attached to this specific virtual server.

To remove it i now send a PATCH or PUT command to with the following body:

{"rules":[]}

But clearly this removes all the rules from the vip. Is it possible to only remove the rules i just added to the vip?