Forum Discussion

Ichnafi's avatar
Ichnafi
Icon for Cirrostratus rankCirrostratus
Sep 29, 2023

Ansible - Bricking freshly installed vcmp guests with ansible

 

Hello fellow F5 admins,

currently I try to established a workflow, where new vcmp guests are created and configured with a standard basic config (and even building a HA setup).
The creation part is working, but here begin the problems:

tl;dr

Question: What is the proper way to bootstrap a freshly installed vcmp guests (or appliance), when you are forced to change the default passwords on 1st login, without doing it by hand? 
The only solution I found (link below) will lock me out of the system forever.

Long Version: 
Freshly installed systems enforce a password change for admin user on 1st access.  This password change cannot be accomplished with the standard ansible module "bigip_user". If you try, you will get an error telling you, password has expired and it has to be changed.

I then found an article about the security password policy and how one is supposed to change the password with ansible (https://techdocs.f5.com/en-us/bigip-14-0-0/big-ip-system-secure-password-policy/secure-password-policy-chapter-title.html)

So I gave it a try and the password was changed "a" password, but not the one provided by the playbook variable. Neither GUI nor SSH or REST login will work. I am locked out.

Befor you ask: yes the password in ansible-vault style is correct, because it is used to create the guest on the vcmp hosts. 

Here is my playbook:

 

 

---
- name: Test vCMP-Guest
  hosts: vcmp_guests
  gather_facts: false

  vars:
    f5_api_admin_user: admin
    f5_api_admin_password:  !vault |
      $ANSIBLE_VAULT;1.1;AES256          
        35613438373864653838386266616364666366363332646635303036343266646664656333643932          
        6462363934306365636265313038376436353032303330370a656434643837343165316333393932          
        66616133376433303136366664303563373034353630656531663864323433663166653539303937          
        3937646663613064390a663631623733376339353735633362633139383635386661376137653434          
        6237
     bigip_provider:
      server: "{{ ansible_host }}"
      server_port: 443
      user: "{{ f5_api_admin_user }}"
      password: "{{ f5_api_admin_password }}"
      validate_certs: false
      transport: rest

  tasks:
    - name: Set admin Password
      uri:
        url: "https://{{ ansible_host }}/mgmt/shared/authz/users/admin"
        method: PATCH
        body: '{"oldPassword":"admin","password":"{{ f5_api_admin_password }}"}'
        body_format: json
        validate_certs: false
        force_basic_auth: true
        user: admin
        password: admin
        headers:
          Content-Type: "application/json"
      register: result
      delegate_to: localhost

    - name: Debug
      ansible.builtin.debug:
        var: result

    - name: Try to get system info
      f5networks.f5_modules.bigip_device_info:
        gather_subset:
          - system-info
        provider: "{{ bigip_provider }}"
      register: output
      delegate_to: localhost

    - name: Debug
      ansible.builtin.debug:
        var: output

 

 

The Output of the the password reset task look fine to me:

 

 

TASK [Debug] ********************************************************************************************************************************************************************************
task path: ~/guest-playbook.yml:47
ok: [test-guest] => {
    "result": {
        "cache_control": "no-store, no-cache, must-revalidate",
        "changed": false,
        "connection": "close",
        "content_length": "330",
        "content_security_policy": "default-src 'self'  'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data:  http://127.4.1.1 http://127.4.2.1",
        "content_type": "application/json; charset=UTF-8",
        "cookies": {},
        "cookies_string": "",
        "date": "Fri, 29 Sep 2023 11:48:50 GMT",
        "elapsed": 0,
        "expires": "-1",
        "failed": false,
        "json": {
            "displayName": "Admin User",
            "encryptedPassword": "<removed>",
            "generation": 0,
            "kind": "shared:authz:users:usersworkerstate",
            "lastUpdateMicros": 0,
            "name": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "selfLink": "https://localhost/mgmt/shared/authz/users/********",
            "shell": "/sbin/nologin"
        },
        "msg": "OK (330 bytes)",
        "pragma": "no-cache",
        "redirected": false,
        "server": "Jetty(9.2.22.v20170606)",
        "status": 200,
        "strict_transport_security": "max-age=16070400; includeSubDomains",
        "url": "https://<removed>/mgmt/shared/authz/users/********",
        "x_content_type_options": "nosniff",
        "x_frame_options": "SAMEORIGIN",
        "x_xss_protection": "1; mode=block"

 

 

The next task, will already fail with a "unauthorized" message. From now on, I cannot access the system any more, and believe me, I tried a lot.

One interesting Thing:

When I don't use a ansible-vault encrypted password and instead set the variable directly to the string, login is possible, BUT only to the GUI. I cannot do rest api calls with this password. When I change the admin password again (from within GUI), I can however use rest api again. When I change it back to the original one, api calls will fail. 

There is one difference I noticed in /var/log/audit in the case, when I set the password as clear-text:
User authentication is logged like this and the api request fails: 

 

 

AUDIT - user admin - RAW: httpd(pam_audit): User=admin tty=(unknown)

 

 

After setting a new password within the GUI oder tmsh and running the same api request, audit messaged changed like this and the request is successfull:

 

 

[...] AUDIT - user admin - RAW: rest(pam_audit): user=admin(admin)[...]

 

 

When I now change the password back to the previous one,  api request fails again

 

 

[...]AUDIT - user admin - RAW: httpd(pam_audit): User=admin tty=(unknown)[...]

 

 

What on earth is going on?
How is one supposed to bootstrap a vcmp guest from ground up without manually interaction for setting passwords and stuff? 

Any usefull advice is thoroughly appreciate.

Cheers

Ichnafi

 

  • Got it working!

    I ran into several strange issues.

    1. Password handling seems to be not konsisten. For some reason I had da "\n" at the end of my password in the ansible-vault encrypted string. Why? Don't know. It seems, that normal API and GUI login do not care about this trailing "\n", but password changes do.
    2. After fixing the password and (just for good measugre) adding the Jinja2 filter "trim" to alle my password variables, I ran everything again using the REST endpoint "mgmt/shared/authz/users/admin" with a PATCH leaving the system bricked again.

    3. How did I get it to work

    1. When connecting via SSH as user "root" you are forced to set a new root password.
      This password is also set as an admin password, that still has to be changed. Also worth to mention, that javarestd will automaticly restart.
    2. Connect via SSH as user "admin" with root password and set a new admin password. After the password change you get booted out of you ssh session and the return code is 1, so we have to consider this in ansible.

    To automate the password changes I write tasks using module ansible.builtin.expect.
    This completly renders idempotency useless. Changes in the SSH login flow and/or messages will let the taks fail. 

     

    - name: Change root password
      no_log: true
      ansible.builtin.expect:
    	command: ssh -oStrictHostKeyChecking=no -oCheckHostIP=no root@"{{ ansible_host }}"
    	timeout: 10
    	responses:
    	  '(.*)Password(.*)': default
    	  '(.*)UNIX password:(.*)': default
    	  '(.*)New BIG-IP password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)Retype new BIG-IP password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)config(.*)#': exit
      register: output
      delegate_to: localhost
    
    - name: Debug
      ansible.builtin.debug:
    	var: output
    
    - name: Wait for restjavad to be restarted
      ansible.builtin.wait_for:
    	timeout: 20
      delegate_to: localhost
    
    - name: Change admin password
      no_log: true
      ansible.builtin.expect:
    	command: ssh -oStrictHostKeyChecking=no -oCheckHostIP=no "{{ f5_api_admin_user }}"@"{{ ansible_host }}"
    	timeout: 10
    	responses:
    	  '(.*)Password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)UNIX password:(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)New BIG-IP password(.*)': "{{ f5_api_admin_password | trim }}"
    	  '(.*)Retype new BIG-IP password(.*)': "{{ f5_api_admin_password | trim }}"
      register: output
      failed_when: output.rc not in [0, 1]
      delegate_to: localhost

     

    The hole thing is still really annoying. I don't understand why this has to be resolved like, in times where cloud first, api first, whatever first is key. This should really be done in a different way.

    Cheers

    Ichnafi

  • Got it working!

    I ran into several strange issues.

    1. Password handling seems to be not konsisten. For some reason I had da "\n" at the end of my password in the ansible-vault encrypted string. Why? Don't know. It seems, that normal API and GUI login do not care about this trailing "\n", but password changes do.
    2. After fixing the password and (just for good measugre) adding the Jinja2 filter "trim" to alle my password variables, I ran everything again using the REST endpoint "mgmt/shared/authz/users/admin" with a PATCH leaving the system bricked again.

    3. How did I get it to work

    1. When connecting via SSH as user "root" you are forced to set a new root password.
      This password is also set as an admin password, that still has to be changed. Also worth to mention, that javarestd will automaticly restart.
    2. Connect via SSH as user "admin" with root password and set a new admin password. After the password change you get booted out of you ssh session and the return code is 1, so we have to consider this in ansible.

    To automate the password changes I write tasks using module ansible.builtin.expect.
    This completly renders idempotency useless. Changes in the SSH login flow and/or messages will let the taks fail. 

     

    - name: Change root password
      no_log: true
      ansible.builtin.expect:
    	command: ssh -oStrictHostKeyChecking=no -oCheckHostIP=no root@"{{ ansible_host }}"
    	timeout: 10
    	responses:
    	  '(.*)Password(.*)': default
    	  '(.*)UNIX password:(.*)': default
    	  '(.*)New BIG-IP password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)Retype new BIG-IP password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)config(.*)#': exit
      register: output
      delegate_to: localhost
    
    - name: Debug
      ansible.builtin.debug:
    	var: output
    
    - name: Wait for restjavad to be restarted
      ansible.builtin.wait_for:
    	timeout: 20
      delegate_to: localhost
    
    - name: Change admin password
      no_log: true
      ansible.builtin.expect:
    	command: ssh -oStrictHostKeyChecking=no -oCheckHostIP=no "{{ f5_api_admin_user }}"@"{{ ansible_host }}"
    	timeout: 10
    	responses:
    	  '(.*)Password(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)UNIX password:(.*)': "{{ f5_root_password | trim }}"
    	  '(.*)New BIG-IP password(.*)': "{{ f5_api_admin_password | trim }}"
    	  '(.*)Retype new BIG-IP password(.*)': "{{ f5_api_admin_password | trim }}"
      register: output
      failed_when: output.rc not in [0, 1]
      delegate_to: localhost

     

    The hole thing is still really annoying. I don't understand why this has to be resolved like, in times where cloud first, api first, whatever first is key. This should really be done in a different way.

    Cheers

    Ichnafi

  • f51's avatar
    f51
    Icon for Cirrostratus rankCirrostratus

    You're dealing with a complex issue that is likely due to the secure password policy enforced by F5 BIG-IP systems. Your Ansible Playbook seems to be correctly constructed, and it's indeed peculiar that the password isn't working as expected. One possible explanation could be that F5 uses different authentication modules for GUI and REST API. The GUI uses the httpd module, while the REST API uses the restjavad module. It seems that the password change is not getting recognized by the restjavad module.

    Here are a few suggestions:

    1. Manual Password Change: Although this solution involves manual intervention, you might want to try changing the password manually through the GUI or TMSH first, and then using that password in your Ansible playbook. This would at least allow you to determine if the issue is with Ansible or with F5's password policy.

    2. Use REST API directly: Instead of using Ansible's bigip_user module, try using the uri module to send a PATCH request directly to F5's REST API to change the password. You are already using the uri module in your playbook, but you might want to experiment with different parameters or headers.

    3. Check F5's Password Policy: F5's BIG-IP systems enforce a strong password policy by default. Make sure your new password meets these requirements. If it doesn't, F5 might be rejecting it without providing a clear error message.

    • Ichnafi's avatar
      Ichnafi
      Icon for Cirrostratus rankCirrostratus

      Hi f51,

      thank you for your suggestions.


      f51 wrote:

      [...]One possible explanation could be that F5 uses different authentication modules for GUI and REST API. The GUI uses the httpd module, while the REST API uses the restjavad module. It seems that the password change is not getting recognized by the restjavad module. [...]


      That's what I thought too. When I provided the password in the playbook asl clear-text, I was able to log into GUI. Changing the Admin password there to something different, made REST calls possible. Changing it back to the previous password, let them fail again.
      I then also set a root password, connected via SSH as root, and changed admin password again using command 'tmsh modify auth user admin prompt-for-password'
      Again I set the admin password to something new, and REST calls were successfull.
      Then I changed it back to the original one and REST calls fail again. Again one can observe in the audit logs, that whenn successfull rest(pam-audit) is triggert and back on the ols password its httpd(pam-audit).

      Let me get to your other points:

      1. The chosen password complies to the password policy. This is out of question. It can be set manually without any problems.
      2. The forced password change is accomplished by using the REST API, as it is explained in the policy article. After that done, I tried to do a "regular" REST API call to get a X-F5-Auth-Token with a postman job, that works fine one many our systems (same password as well), and the response was also unauthorized
      3. As said, the password complies and can be set in a manual setup without any complains.

      Is there maybe a way to automize the 1st ssh login with expect or something? I did a writeup on how to run a shell comman on f5 and using expect there in an interactive situation (Ansible-running-bash-commands-with-bigip-command-module), but this only works when one is already on the system. I might need to write a bash script that does the ssh root@<new_vcmp_guest> , accept new fingerprint, enter password and then reenter the new password twice...
      Any bash gurus in the house?

      Cheers

      Ichnafi