Unable to update device cert

[Patrik_Jonsson]

Hi!

Working on a hobby project to manage F5 certificates using Kubernetes and cert-manager (Ref: https://community.f5.com/t5/technical-forum/kubernetes-cert-manager-letsencrypt-f5/td-p/299218). However, I have been running into problems when configuring httpd to use the new certificate+key.

Pretty much trying to follow https://support.f5.com/csp/article/K12522815

1. Copied new cert
2. Copied the new key
3. Set the certificate + key via a REST call to /mgmt/tm/sys/httpd (like the article suggests)
4.  Next step is to restart httpd using a rest call to /mgmt/tm/sys/service but by then httpd is broken

Try to restart httpd via ssh:

# bigstart restart httpd
Stopping httpd:                                            [  OK  ]

Broadcast message from systemd-journald@bigip.xip.se (Tue 2022-08-02 13:13:51 PDT):

httpd[4453]: [ssl:emerg] [pid 4453] (13)Permission denied: AH02201: Init: Can't open server certificate file /config/httpd/conf/ssl.crt/management.crt

Starting httpd:                                            [FAILED]

The certificate is there:

ls -la /config/httpd/conf/ssl.crt
total 28
drwx------. 2 root root   4096 Aug  2 13:12 .
drwxr-xr-x. 7 root root   4096 Aug  1 14:52 ..
-rw-r-----. 1 root apache 5582 Aug  2 13:12 management.crt
-rw-------. 1 root root   5582 Aug  2 12:28 server.crt
-rw-------. 1 root root   1521 Aug  2 12:27 server.crt.old

So is the key:

ls -la /config/httpd/conf/ssl.key
total 20
drw-------. 2 root root 4096 Aug  2 12:58 .
drwxr-xr-x. 7 root root 4096 Aug  1 14:52 ..
-rw-------. 1 root root 1674 Aug  2 12:58 management.key
-rw-------. 1 root root 1674 Aug  2 12:27 server.key
-rw-------. 1 root root 1675 Aug  2 12:27 server.key.old

They certificate and key matches:

# openssl x509 -noout -modulus -in /config/httpd/conf/ssl.crt/management.crt 
Modulus=B0E9B8F6A9084A134E5575E6374E4BCF30D56D7A2F728921AA4A3A433E0A349EED6455C9AFF2F5D151642D74073DB20BD8250E9DDCE06CA2CA9A3A4ADC55923396AE4F3EA612E4EBC66C2F82DEBEC27F87ED63E051A40FE3B7E33754EA65598467E7B0E81F13E5B508966F3A10B16E6E62A7EB181104C66AA52F013D3BD879A545D3A4C8A473D03B380625823E36D4EA3EAD9039A4BC7F01E9E3C6DF005B593AB5EAD62D59A9CCF7B2AFDC8F744D8BFACDB9B92DB471E52709186540E34893B9525B8DF1A4C4837A98032A648FFCB9D70F2BEDEF388504BAB59191BA893389CC8BD3B0E446143086D1CDAE9D51CB13704C2CF7A61C679293939AAB6FC63F0B87

# openssl rsa -noout -modulus -in /config/httpd/conf/ssl.key/management.key
Modulus=B0E9B8F6A9084A134E5575E6374E4BCF30D56D7A2F728921AA4A3A433E0A349EED6455C9AFF2F5D151642D74073DB20BD8250E9DDCE06CA2CA9A3A4ADC55923396AE4F3EA612E4EBC66C2F82DEBEC27F87ED63E051A40FE3B7E33754EA65598467E7B0E81F13E5B508966F3A10B16E6E62A7EB181104C66AA52F013D3BD879A545D3A4C8A473D03B380625823E36D4EA3EAD9039A4BC7F01E9E3C6DF005B593AB5EAD62D59A9CCF7B2AFDC8F744D8BFACDB9B92DB471E52709186540E34893B9525B8DF1A4C4837A98032A648FFCB9D70F2BEDEF388504BAB59191BA893389CC8BD3B0E446143086D1CDAE9D51CB13704C2CF7A61C679293939AAB6FC63F0B87

 If I run a "bigstart restart" AND "bigstart restart httpd" it suddenly works fine:

# bigstart restart
# bigstart status httpd
httpd (pid 20978) is running...
# bigstart restart httpd
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

 I am guessing additional services needs to be restarted before httpd? What am I missing?

Kind regards,
Patrik

[Dario_Garrido]

Hello mate! 

Looking forward to know more about your project. Are you planning a future article for this? 😊

Regarding your issue, never faced before, but I was wondering if tomcat daemon could be involved.

https://support.f5.com/csp/article/K03531511

[Patrik_Jonsson]

No article planned, but that was a good idea. Might write one if I can get it to work. 👍

[Dario_Garrido]

I've also found this. Funny article (don't use REST 😂)

https://support.f5.com/csp/article/K13292945

[Patrik_Jonsson]

Good ideas, I forgot about tomcat!

I'm afraid neither of the articles worked. Same issue.

Kind regards,
Patrik

[Patrik_Jonsson]

Same thing happens if I update the cert via REST, pause the script and run:

tmsh modify sys httpd ssl-certfile /config/httpd/conf/ssl.crt/management.crt ssl-certkeyfile /config/httpd/conf/ssl.key/management.key

httpd[4571]: [ssl:emerg] [pid 4571] (13)Permission denied: AH02201: Init: Can't open server certificate file /config/httpd/conf/ssl.crt/management.crt

[Patrik_Jonsson]

Came a bit further just now. Looks like SELinux might be making my life harder:

restorecon -RvF  /var/config/rest/downloads/management.crt /config/httpd/conf/ssl.crt/
restorecon -RvF  /var/config/rest/downloads/management.key /config/httpd/conf/ssl.key/

 Then the permission error goes away but restarting the service does not work. Then the article @Dario_Garrido gave came in handy:

bigstart restart httpd
killall -9 httpd
bigstart restart httpd;

Will see if I can solve this further tomorrow. Good input with https://support.f5.com/csp/article/K13292945 @Dario_Garrido . Saved me quite some time!

Kind regards,
Patrik

[Patrik_Jonsson · Accepted solution]

Final solution:

def set_management_cert(self, cert_name, key_name):

    self.run_bash_command(f'restorecon -RvF /config/httpd/conf/ssl.crt/{cert_name}')
    self.run_bash_command(f'restorecon -RvF /config/httpd/conf/ssl.key/{key_name}')

    self.session.put(
        f'https://{self.device}/mgmt/tm/sys/httpd',
        json={
            'sslCertfile': '/config/httpd/conf/ssl.crt/management.crt',
            'sslCertkeyfile': '/config/httpd/conf/ssl.key/management.key'}
    )
    try:
        logger.info('Restarting httpd')
        self.run_bash_command('bigstart restart httpd; killall -9 httpd;bigstart restart httpd;')
    except:
        logger.info('Waiting for management interface to restart')
        time.sleep(3)
        httpd_config = self.get_httpd_config()

        if os.path.basename(httpd_config['sslCertfile']) == cert_name \
                and os.path.basename(httpd_config['sslCertkeyfile']) == key_name:
            print('Certificate has been updated and the httpd interface is responding')
        else:
            raise Exception('Failed to update the certificate')

Thank you for the suggestions and tips @Dario_Garrido . Definitely helped me find the solution!

[Dario_Garrido]

Great solution! Thanks mate 👏

[JRahm]

Nice @Patrik_Jonsson!! Curious what your helper functions look like (get_management_cert, run_bash_command)...

[Patrik_Jonsson]

@PeteWhite Do you have an ace up your sleeve?

[PeteWhite]

good job - i'd have suggested restorecon. Always a good idea to try it first with files, permissions etc