Automate import of SSL Certificate, Key & CRL from BIG-IP to BIG-IQ

[RomanJ]

The functionality to automate the import of SSL cert & key from BIG-IP to BIG-IQ is available in the product starting BIG-IQ 7.0 and above. This script should not be used on BIG-IQ 7.0+ as it has not been tested on those versions.

This script will import all supported SSL Certificate, Key & CRL that exist as unmanaged objects on this BIG-IQ which can be found on the target BIG-IP.

Steps performed by the script:

• Gather certificate and key metadata (including cache-path) from BIG-IPs
• Download certificate and key file data from BIG-IPs
• Upload certificate and key file data to BIG-IQ

Prerequisite: Discover and import LTM services before using this script.The target BIG-IP will be accessed over ssh using the BIG-IP root account.

Installation: The script must be installed in BIG-IQ under /shared/scripts:

# mkdir /shared/scripts# chmod +x /shared/scripts/import-bigip-cert-key-crl.py

Command example:

# ./import-bigip-cert-key-crl.py <big-ip IP address>

​Enter the root user's password if prompted.

Allowed command line options:    -h                show this help message and exit    -l                 LOG_FILE, log to the given file name    --log-level   {debug,info,warning,error,critical}, set logging to the given level (default: info)    -p PORT     BIG-IP  ssh port (default: 22)

Result: Configuration > Certificate Management > Certificates & Keys

Before running the script:

After running the script:

Location of the scripts on GitHub: https://github.com/f5devcentral/f5-big-iq-pm-team

In case you BIG-IQ is running on Hardware:

Step 1: Install packages using pip, targeting a location of your choice

# mkdir py-modules# pip install --target py-modules requests argparse

Step 2: Run using python2.7, adding py-modules to the python path

# PYTHONPATH=py-modules python2.7 import-bigip-cert-key-crl.py <big-ip IP address>

[Simon_Lodge]

Hi Roman,

How does this script handle the import of cert/key pairs secured with passwords?

Thanks,

Simon

[RomanJ]

@Simon Lodge The script does not currently support import of cert/key protected with a password.

We could think about improving the script to handle it so the user would manually enter the password if required. Is this something you would be interested to have?

[Simon_Lodge]

Hi Roman,

Firstly, thanks for the quick response.

Ideally, it would be fantastic if you could get the script to ignore the cert/key pairs protected by password (perhaps by inserting a flag), then report on the pairs it has skipped due to this issue - is this something you would consider looking at?

My particular problem is that I don't control the passwords set on cert/key pairs, the service owners do, and the F5 estate in my org currently holds approx. 15,000 SSL cert/key pairs, so following up with every service owner/group is very time-consuming and ultimately I don't have the resources to do so.

If I had the option to import the non-secured cert/key pairs whilst retaining a record of what's left that would allow me to start managing these whilst defining a standard set of passwords to secure cert/key pairs going forward.

Many thanks,

Simon

[RomanJ]

Actually, the current script already ignore the certs/keys with a passwords and give you a warning with the name of the key. The other object without password should be imported correctly.

2018-09-10 14:40:20,787:WARNING:Associate for Key /var/config/rest/downloads/:Common:myprivkey.key_112988_2 was not successful, final task state: {u'status': u'FAILED', ....

Thanks & Regards,

[Simon_Lodge]

Just tried it and it works great, thanks Roman!

I've tested it on some lab boxes (containing approx. 200 SSL cert/key pairs) and no issues seen so far, I will let you know if I run into any issues when importing larger numbers of cert/key pairs from our production units. This addresses one major issue we've had with using BIGIQ as a centralised config manager - thanks again..

Cheers,

Simon

[acurry583]

Hi,

Is there an option to ignore cert/key pairs with passwords. The passwords are the same, but the encryption makes them appear to be different, so there seems to be no way around this unless we name each cert/key profile different across each BIG-IP which would kind of defeat the process of using BIG-IQ. This is my hold up from importing...

[RomanJ]

Hello, The current script already ignore the certs/keys with a passwords and give you a warning with the name of the key. Cheers, Roman

[acurry583]

I see now! Thanks!

My issue is with actually importing LTM services, not the certificate. The services will not import due to the encrypted output of the passwords being different. The passwords are the same but the encryption of it is different on each BIG-IP. Is there a script to get around that? See example below: The checksum is not allowing import of the certificate, but it is the same certificate, just uploaded to two different BIG-IPs.

[goodsell_116980]

acurry583,

It's strange that you would encounter a diff here. The keys and certs can only be imported from BIG-IP into BIG-IQ when the checksum matches, so I think the checksums must have been the same at the time you ran the import script. Typical workflow would go something like this:

1. Discover & Import LTM from the BIG-IP to BIG-IQ. At this time the BIG-IQ and BIG-IP will have the same checksum for the file, but BIG-IQ will not have the file content.

2. Run the import script. The script will copy the file content from BIG-IP and add it to the storage on BIG-IQ.

Step 2 will only succeed if the file content matches the checksum that originally came from BIG-IP during step 1.

This suggests that somewhere along the line you have modified the certificate. This does not appear to be related to passwords, since certificates don't use passwords (as far as I'm aware, anyway).

The data we can see for the cert looks the same on both sides, so it may be that the meaningful file content is identical, but the actual bytes of the files differ (for example, whitespace could have been added or removed). If you are inclined, you could examine the two versions of the file (from BIG-IP and from BIG-IQ) and see what has changed (though I'm not sure there's an easy way to fetch the file data on BIG-IQ--I could get you details on how to locate it if necessary). However, if you are confident that the file on BIG-IP is correct then I would suggest just accepting the BIG-IP version (which should import the file metadata and leave it unmanaged on BIG-IQ) then re-running the file import script to pull in the file content. Or, alternatively, delete the BIG-IQ object, import LTM, then run the import script.

[fwadmin]

Will this work on code 6.1.0

[RomanJ]

Hello @fwadmin, Yes it should work with BIG-IQ 6.1. Best Regards, Roman

[fwadmin]

Hello,

We keep getting errors on line 27-37 command not found then 40 FILE_IMPORT_DIR: command not found and 42 IMPORT_TASK_PATH: command not found

[goodsell_116980]

fwadmin, please double-check how you are running the script, and ensure that it hasn't been modified from the original. From the info you've given, it sounds like the wrong interpreter is being invoked. The error messages sound like what you would get from a command shell (such as bash) not what you would expect to get from python.

The script should be run on BIG-IQ, from the directory that contains the script, using "./import-bigip-cert-key-crl.py" (followed by arguments).

[RomanJ]

It looks you missed the first line in the script when you copy/past.

!/usr/bin/env python

I just tested the script on a BIG-IQ 6.1 and it worked for me.

[fwadmin]

Adjusted that now getting the below Repeat with additional target BIG-IPs to import more file objects. ./import-bigip-cert-key-crl.py x.x.x.x Traceback (most recent call last): File "./import-bigip-cert-key-crl.py", line 651, in sys.exit(main()) File "./import-bigip-cert-key-crl.py", line 639, in main with SshConnection(arguments.bigip, arguments.port) as conn: File "./import-bigip-cert-key-crl.py", line 308, in init self._start_master_proc() File "./import-bigip-cert-key-crl.py", line 424, in _start_master_proc "ssh connection failed, exit code: %d" % master_rc) StandardError: ssh connection failed, exit code: 255

[RomanJ]

Do you have SSH (port 22) open between BIG-IQ and BIG-IP?

Could you manually ssh your BIG-IP from BIG-IQ to confirm there are no communication issue?

[fwadmin]

That was it thanks for your help! Script Works Great!

[megakoresh]

Helow, we have also issue running the script on BigIQ. When we run the script from bigiq, it throws pickle EOF exception. So we took the lines that are to be evaluated on the BigIP:

import f5.mcp, pickle;
 Include only the few attributes we care about, since not everything
 can be pickled/unpickled.
attrs=["cache_path","name","checksum","passphrase"];
 Function to translate objects into a pickleable form.
m=lambda d: dict((k,v) for (k,v) in d.items() if k in attrs);
 Stub just to test
object_type = 'certificate_key_file_object'
r=f5.mcp.MCPConnection().query_all("' + object_type + '");
print pickle.dumps([m(o) for o in r])

And tried running that on BigIP to see what happens and got

    r=f5.mcp.MCPConnection().query_all("' + object_type + '");
  File "/usr/lib/python2.6/f5/mcp/__init__.py", line 182, in __call__
    raise McpError, "Incompatible Version"
_pymcp.McpError: Incompatible Version

Google is for once completely useless, seems we are the first to have this issue. The mcp submodule does not have any documentation either. Can you help? We have around 500 certs on 20+ loadbalancers and we really need this script to bulk-import them all.

[RomanJ]

@megakoresh,

1. Could you share the exception you are facing (copy/past output)?
2. Could you confirm you can ssh the BIG-IPs from BIG-IQ?
3. Could you confirm you have !/usr/bin/env python in the first line of the file?
4. Which version of BIG-IQ and BIG-IP are you using?

Thanks & Regards, Roman

[goodsell_116980]

megakoresh, there's a problem with your test script that is producing the Incompatible Version error. Change this line:

r=f5.mcp.MCPConnection().query_all("' + object_type + '");

To:

r=f5.mcp.MCPConnection().query_all(object_type);

And see if that produces properly pickled output.

[megakoresh]

@goodsell @Roman we have implemented that fix and yeah now the incompatible version disappeared. Back to the EOF exception:

2019-03-20 09:04:48,797:INFO:['bash', '-c', 'python -c \'import f5.mcp, pickle;attrs=["cache_path","name","checksum","passphrase"];m=lambda d: dict((k,v) for (k,v) in d.items() if k in attrs);r=f5.mcp.MCPConnection().query_all("certificate_file_object");print pickle.dumps([m(o) for o in r])\'']
Traceback (most recent call last):
  File "./import-bigip-cert-key-crl.py", line 655, in 
    sys.exit(main())
  File "./import-bigip-cert-key-crl.py", line 643, in main
    unmanaged_files = find_all_unmanaged_objects(session, conn)
  File "./import-bigip-cert-key-crl.py", line 538, in find_all_unmanaged_objects
    unmanaged_files += find_unmanaged_objects(session, bigip_connection, typ)
  File "./import-bigip-cert-key-crl.py", line 484, in find_unmanaged_objects
    object_type.mcp_type_name)
  File "./import-bigip-cert-key-crl.py", line 440, in get_bigip_file_objects
    return pickle.loads(stdout)
  File "/usr/lib/python2.6/pickle.py", line 1374, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.6/pickle.py", line 880, in load_eof
    raise EOFError
EOFError

We had to wrap the command in bash -c because by default your script does not execute bash and therefore cant find python.

UPD: We have changed default shell for the user we use to login to bash in /etc/passwd and then your module worked when we removed the whole bash -c thing. However we do require it in the end, so do you have any idea of how we could execute it on a user that connects to tmsh by default?

[RomanJ]

@megakoresh You shouldn't need to modify the script or wrap it in bash.

Could you confirm you have !/usr/bin/env python in the first line of the file?

Which version of BIG-IQ and BIG-IP are you using?

Are you running the script in bash mode, not tmsh?

[megakoresh]

@Roman

We have the shebang line. We are running the script from bash mode on bigIQ which is version 6.0.1.1 The problem is that it uses ssh to connect to the bigIP and since we can't use the root user, we use an admin account and that connects to tmsh on BigIP by default, not bash. So it cannot use python - the python command does not work, you have to switch BigIP to bash mode as well. BigIP is version 13.1.1.2-0.0.4

After we switched the admin user's default shell to /bin/bash everything worked as expected without modifications, but we cant login to every loadbalancer and change the admin user's shell just for this script to work.

[RomanJ]

@megakoresh Why don't you create a one off script using root user to give admin user bash shell access to your 20+ Bigips?

Or, it might be faster to manually do this change on your 20+ Bigips than spending the time l trying to make the script work in a way it wasn't designed to work originally 🙂

[megakoresh]

@Roman

Didn't I say that we can't use root on BigIPs? What you are suggesting is for us to create a script that would ssh to all of those loadbalancers and change the default shell to bash for the admin account, and that leaves us with exactly the same problem - because the default shell is not bash and we would have to switch to bash after ssh-ing first to be able to use tools like sed to edit /etc/passwd file.

If there's some other way to quickly switch default ssh shell on BigIPs I would like to know it.

[RomanJ]

@megakoresh You can change the default shell for admin using the following command:

tmsh modify auth user admin shell bash

[Vivisica]

Is anyone running into this issue? I have confirmed that I am running the correct version of phyton. I have made no modifications at all to the script nor to python. I have tested this numerous times and always the same error.

Traceback (most recent call last): File "./import-bigip-cert-key-crl.py", line 27, in import argparse ImportError: No module named argparse

Update: I just checked both of my 7000's and they are both running 2.6 and neither one of the installations out of the box have argparse in the phyton folder.

[goodsell_116980]

Vivisica, to the best of my knowledge argparse should be installed on BIG-IQ, though yours is not the first report I've heard of it not being found. What version of BIG-IQ are you running? I would try the following and look for anything fishy:

 python -c 'import sys; print sys.path; print; import argparse; print argparse.__file__'

If argparse is not being found then I wouldn't expect this to all work, but it could give clues. In my case, I see the following output:

['', '/usr/lib/python26.zip', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']

/usr/lib/python2.6/site-packages/argparse.pyc

This tells me that argparse is installed in site-packages, which is (of course) present in the python system path. You could try checking that argparse is present in your site-packages folder, and that site-packages is in the python path.

[Vivisica]

I'm running 6.1.0 build 0.0.1224. I did do an additional search in /usr/local/lib and 2.7 is there as well as argparse. I just do not know if I should copy it to /usr/lib. I ran the command you provided and this is the output.

python -c 'import sys; print sys.path; print; import argparse; print argparse.__file__'

['', '/usr/lib/python26.zip', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/site-packages']

Traceback (most recent call last): File "", line 1, in ImportError: No module named argparse

[goodsell_116980]

To me that looks like the system has been modified somehow to remove the argparse package, but maybe there's some reason it would not be installed in your case. Please try this command on BIG-IQ:

 rpm -ql python-argparse

This should list the installed files for the package or report that it isn't installed.

[goodsell_116980]

Vivisica, I'm seeing some indications that argparse is installed on Virtual Edition BIG-IQ to satisfy some specific dependencies there. It looks like it might normal for it to be absent from a hardware BIG-IQ. We might need to update the script to use a different argument parsing method.

[Vivisica]

Yep, that is what I was told from the F5 engineer that I am working with. I'm using hardware and it came with 2.6 and argparse wasn't introduced until 2.7. He is checking to see if I can safely upgrade my devices to 2.7 or if the script needs to be rewritten. Thanks for the assistance.

[RomanJ]

@Vivisica

We have updated the script to remove dependency on argparse, not installed on hardware BIG-IQ.

[Vivisica]

Thanks so much Roman.

[Vivisica]

I have these two unless I'm doing something very wrong...I've never used python before this so its a strong possibility. I tried the IP two different ways running from the scripts directory.

[root@bigiqmgmt01:Active:Standalone] scripts ./import-bigip-cert-key-crl.py x.x.x.x Traceback (most recent call last): File "./import-bigip-cert-key-crl.py", line 34, in import requests ImportError: No module named requests [root@bigiqmgmt01:Active:Standalone] scripts ./import-bigip-cert-key-crl.py -bash: syntax error near unexpected token `newline'

[goodsell_116980]

Sorry Vivisica, it looks like argparse is not the only module that is missing on hardware BIG-IQ. Unfortunately we primarily test on virtual devices.

Adapting the script to avoid argparse was relatively easy, but avoiding requests seems significantly more difficult (and I can't be sure that it would work at that point).

I'm not sure what the next steps should be. In principle I think it should be possible to install the missing packages from the BIG-IQ ISO file, but it's not certain that this action would be safe. There may be other ways to add the modules more safely. I'll have to look at options and see if we can find something.

[goodsell_116980]

I have a procedure that might get the script working on hardware, but it's only partially tested so far. This would work for the new version or the previous version using argparse (which is probably a little better because argparse is more advanced and easier to use, making the script code in that version a bit better).

The steps are 1) use the pip command to install the missing packages in a temporary location, then 2) run the script using python2.7 and directing it to the temporary package location. Optionally the package directory can be removed at the end. python2.7 is required here because pip is built to use it, so the packages it installs will use Python 2.7 features and will not work in the default Python 2.6. I'm not sure what BIG-IQ version added Python 2.7, but I've tested this on BIG-IQ 6.1.

Step 1: Install packages using pip, targeting a location of your choice:

 mkdir py-modules
 pip install --target py-modules requests argparse

Step 2: Run using python2.7, and adding py-modules to the python path:

 PYTHONPATH=py-modules python2.7 import-bigip-cert-key-crl.py --help    

This will show the help message if everything is working. From there I'd expect the script to work, just continue to run it with the PYTHONPATH environment variable and using python2.7.

[Vivisica]

Ok, thanks. Seems there are a lot of significant differences between hardware and VE that has made this deployment very difficult as everything is really designed for VE and not hardware. I do have 2.7 on the boxes in /usr/local/lib so that's a plus in my favor. I will continue to tinker with it until I can get it to work.

[Vivisica]

I'm currently running 6.1 .0 build 0.0.1224 on the 7000 hardware.

[RomanJ]

@Vivisica, did the steps described by goodsell worked?

[Vivisica]

I have completed the reinstall and I will follow the last set of instructions you provided and will get back to you.

[Vivisica]

@Roman @Goodsell after a little tweaking...more of me learning on the fly...I was finally able to get the script to work and has successfully completed. Thanks so much for your help and crash course in python.

[RomanJ]

Great news @vivisica. I will update the article to give the instructions for BIG-IQ HW.