We have a request to send a specific header and a value to the backend servers based on traffic coming in. Im currently testing this via irules. The header value will be a key stored in irule or datagroup. If needed to, is there a way to encrypt/decrypt that header value locally in big-ip? instead of it being in clear text via irule/datagroup/iFile?
An option I thought to maybe use is APM making an api call via http connector to a key vault(e.g. hashicorp vault) somewhere to extract that string in session variable and then use it via irule.
You can, but assuming that you actually need to look at the value, you will also need to have local decryption keying material. For example, if it were encrypted using AES, you would need to store the symmetric key in a place accessible to the iRule. If somebody has sufficient access to the BIG-IP to see the contents of the data-group, they are very likely to have sufficient permission to see the contents of the iRule, which would in turn contain the key. At that point, the encryption won't hide the contents. If the encrypted information were stored remotely, and an adversary gets access to the BIG-IP in such a way that he or she can connect to the remote system, you're in the same boat. If you store the key in a remote vault, you will presumably need to supply credentials to access that key. Again, if somebody has those credentials and the ability to make a connection from the BIG-IP, then the encryption will fail to keep the value secret.
I'm certainly not arguing against encryption for data-at-rest, but it is of course a trade-off, and it's important to consider under what conditions the data remain secret and when it will not be.