Query related to name modification and iRules execution order

Can we change the name of a VIP and node which are already UP and working without deleting by "modify" command?

Generally, no. It is possible, however dangerous, to edit the /config/bigip.conf file directly, then do 'tmsh save sys config'. I cannot stress enough how important it is, however, to first make a backup of this file, and to make edits very carefully.

Suppose we have 3 iRules on a VIP, so in what order these iRules will be executed?

There are a few things to consider here. By default, they fire in the order specified in the iRule list attached to the VIP. You can re-order these directly in the UI. You can also more explicitly define order by using the priority command on the event. It's also important to understand that if multiple iRules attached to a VIP include the same events, the BIG-IP will optimize these into a single event, in the order specified.