Renew F5 BIG IP 11.5.4 Device Certificate & Change Certificate Key to 2048 bits

1) To create a new cert with a private key that is 2048 bits follow the "Generating a new self-signed device certificate and private key" procedure in K9114: Creating a new SSL device certificate and key pair. This will overwrite the existing cert and key in /config/httpd/conf/ssl.crt/server.crt and /config/httpd/conf/ssl.key/server.key so if you need to save those for some reason, be sure to make copies before running the command from the procedure. If needed, this article has more detail specific to renewing certs as opposed to creating new: K6353: Updating a self-signed SSL device certificate on a BIG-IP system.

2) Yes. "Renewing the device certificate requires you to reauthenticate if you are using the Configuration utility"

3) Updating the device certificate has no effect on virtual server traffic. Reinitializing iQuery communication involves restarting big3d and gtmd and this should not disrupt existing virtual server connections since these have already resolved domain names and connected; however, iQuery connectivity will be briefly disrupted and the BIG-IP GTM/DNS system cannot respond to any wide IP queries until the gtmd is finished restarting.

Sounds like you already know this, but you need to exchange the new device certs with GTM/DNS sync group peers and re-establish the iQuery connection so it will start using the new certificate. Both K6353 and K9114 have sections about this.

Also note that 11.5.x is no longer supported per K5903: BIG-IP software support policy.