How to rename a vcmp guest?

The only way I know of is to do what you are suggesting...edit the bigip.conf file. After you do that you'd have to reload the config

tmsh load sys config

and then save the config

tmsh save sys config

. According to F5 "This daemon, named vcmpd, performs most of the work to create and manage guests, as well as to configure the virtual network." If it weren't for that part about the network you might be safe restarting the daemon, but I think I would play it safe and make sure all your guests are standby on that chassis, then shut down the guests, make the changes to the bigip.conf file, load/save your config, restart the service, and power the guests back on.