How to encrypt passwords in bigip.conf

Nothing that I am aware of.

You have to understand that most password encryption schemes are designed around never needing the plaintext version of a password, so they use a one way hash to encrypt the password securely.

In the case of a monitor, the plain text password needs to be known so it can be sent in the monitor request. You can use the cisco method, which is not encryption but instead a reversible formula, or you could use a 2-way encryption method and store the secret key for this somewhere else. Both become pretty useless in what you are trying to solve here. If I can gain access to the bigip.conf, you also have access to whatever method was used to encrypt the password.

What I usually do is weigh the risk of having a password in the config vs the risk of shutting off authentication for the monitor page vs the advantage that using an authenticated monitor gives you. In some cases like Exchange CAS servers, using authentication to verify the CAS server can open a mailbox is useful, in other cases where a monitor page just says UP or DOWN, we usually can disable authentication for that particular page.