Bug (ID 775845) Workaround; REST API httpd restart

Question

So this is less of a question, but a post to help my fellow BIG-IP LTM administrators, since the solution I came up with is quite the hack, but it works for me, so your mileage may vary, and of course -- test in non-production environments.
So some background: I am a F5 administrator and a automation engineer. My main focus is automating much of my work as an administrator to take mundane and repetitive tasks out of my and my colleagues/organizations workflow. So, when it came time to renew the device certificates for my F5 VMs and hosts, combined with the most recently reduction in SSL certificate term length and guidance to renew certs often, I set forth to automate the entire stack of processes that are required to renew device certificates (create key/csr, submit csr to CA and obtain cert, upload cert to F5 and restart the httpd service to read in the new certificates).
I was able to script everything using Python and REST API calls to the F5s and InCommon CA to get the certificates created and put on the F5s. The problem I ran into was the feature to restart the httpd service via a REST API call was broken (aka Bug ID 775845).
I tried using the REST API call:
&nbsp;
/tm/sys/service -X POST -d '{"name":"httpd", "command":"restart"}'
&nbsp;
I also attempted to use the bash command call:
&nbsp;
/mgmt/tm/util/bash -X POST -d "{ "command": "run", "utilCmdArgs": "-c 'service httpd restart'" }
&nbsp;
NONE worked, as documented in the is KB article: https://support.f5.com/csp/article/K13292945
So I needed a workaround, and my solution incorporates a batch script that basically preemptively kills off httpd and then restarts it (as you see in the KB shows as a fix).
First, you need the following bash script (which is actually incorporated into the script below so one can ensure that it always present on the F5 VM or host that needs to have the httpd daemon restarted).&nbsp;
&nbsp;
#/bin/bash
# Pause, restart httpd
# Greg Jewett, 2021-08-26, jewettg@austin.utexas.edu
#
# A known bug (Bug ID 775845) when using the REST API to restart the httpd service.  
# The pause is to allow the REST API call to complete, as script will be launched 
# in background, and should have successful exit code. This script provides an 
# immediate fix to bring environment back up, without manually restarting the 
# httpd daemon on each VM or host.
 
service httpd status | logger -p local0.notice -t RST_HTTPD
logger -p local0.notice -t RST_HTTPD Waiting 2 seconds...
sleep 2s
logger -p local0.notice -t RST_HTTPD Restarting httpd daemon
 
thepids=`pgrep -d " " -f "/usr/sbin/httpd"`
echo "httpd pids are: $thepids"
for aPid in $thepids; do
	echo "Killing PID $aPid"
	kill -9 $aPid
done
service httpd start | logger -p local0.notice -t RST_HTTPD
service httpd status | logger -p local0.notice -t RST_HTTPD
logger -p local0.notice -t RST_HTTPD Done
&nbsp;
NOTE: I am having to attach the rest of my solution via comments, as the platform was allowing me to post a big chuck of text (&gt;10k chars). See below.&nbsp;
&nbsp;

greg_jewett · Answer

The code finds the httpd processes and kills them off; then restarts the service.&nbsp;It also grabs diagnostic and command runtime output data and sends it to the logs that the Python script uses to determine if the restart was successful.
Then the following Python script performs the restart of the httpd using a series of REST API calls that put the script on the VM or host, then executes the bash script. Additional REST API bash commands are executed to fetch parts of the log to verify if the restart was a success or not.
PLEASE NOTE: The Python code below will NOT execute on its own. It requires an outside loop and homegrown/written functions that do not exist natively in Python, but the gist is there; showing how I was able to get this working, working around the bug.
The above batch script is encoded in base64, and if you care, copy the encoded text found in the code below, and put it into a text file.&nbsp;Pipe that text file into "bash64 -d" and it will out the above bash script.&nbsp;&nbsp;I did this to effectively be able to upload the script utilizing the REST API bash command to write the script to a file that could be executed.&nbsp;I could not find any other reliable way to ensure the script was present on the F5.
&nbsp;
# Restart the "httpd" service to read and apply the new certificate bundle.
# -------------------------------------------------------------------------
 
# Create (if it does not exist) the directory path: /opt/scripts
doLog("info", "Creating installation directory (/opt/scripts) on "+str(aHost))
cpURL = "https://"+aHost+"/mgmt/tm/util/bash"
headers = { "content-type" : "application/json", "X-F5-Auth-Token" : str(f5AuthToken)}
theData = { "command": "run",
            "utilCmdArgs": "-c 'mkdir -p /opt/scripts'" }
apiResponse = requests.post(cpURL, headers=headers, verify=False, json=theData)
try:
    apiData = json.loads(apiResponse.text)
except:
    apiData = apiResponse.__dict__
doLog("info", "Directory Create Result: "+str(apiData))
 
# Ensure that the script that will perform the restart and get status info is 
# installed on the F5 host in /opt/scripts/restart_httpd.sh
# The script has been encoded with base64 to prevent the execution of lines and 
# other issues with REST API bash command.
 
script_body = ('Iy9iaW4vYmFzaAojIFBhdXNlLCByZXN0YXJ0IGh0dHBkCiMgR3JlZyBKZXdldHQsIDIwMjEtMDgt'
            'MjYsIGpld2V0dGdAYXVzdGluLnV0ZXhhcy5lZHUKIwojIEEga25vd24gYnVnIChCdWcgSUQgNzc1'
            'ODQ1KSB3aGVuIHVzaW5nIHRoZSBSRVNUIEFQSSB0byByZXN0YXJ0IHRoZSBodHRwZCBzZXJ2aWNl'
            'LiAgCiMgVGhlIHBhdXNlIGlzIHRvIGFsbG93IHRoZSBSRVNUIEFQSSBjYWxsIHRvIGNvbXBsZXRl'
            'LCBhcyBzY3JpcHQgd2lsbCBiZSBsYXVuY2hlZCAKIyBpbiBiYWNrZ3JvdW5kLCBhbmQgc2hvdWxk'
            'IGhhdmUgc3VjY2Vzc2Z1bCBleGl0IGNvZGUuIFRoaXMgc2NyaXB0IHByb3ZpZGVzIGFuIAojIGlt'
            'bWVkaWF0ZSBmaXggdG8gYnJpbmcgZW52aXJvbm1lbnQgYmFjayB1cCwgd2l0aG91dCBtYW51YWxs'
            'eSByZXN0YXJ0aW5nIHRoZSAKIyBodHRwZCBkYWVtb24gb24gZWFjaCBWTSBvciBob3N0LgoKc2Vy'
            'dmljZSBodHRwZCBzdGF0dXMgfCBsb2dnZXIgLXAgbG9jYWwwLm5vdGljZSAtdCBSU1RfSFRUUEQK'
            'bG9nZ2VyIC1wIGxvY2FsMC5ub3RpY2UgLXQgUlNUX0hUVFBEIFdhaXRpbmcgMiBzZWNvbmRzLi4u'
            'CnNsZWVwIDJzCmxvZ2dlciAtcCBsb2NhbDAubm90aWNlIC10IFJTVF9IVFRQRCBSZXN0YXJ0aW5n'
            'IGh0dHBkIGRhZW1vbgoKdGhlcGlkcz1gcGdyZXAgLWQgIiAiIC1mICIvdXNyL3NiaW4vaHR0cGQi'
            'YAplY2hvICJodHRwZCBwaWRzIGFyZTogJHRoZXBpZHMiIHwgbG9nZ2VyIC1wIGxvY2FsMC5ub3Rp'
            'Y2UgLXQgUlNUX0hUVFBECmZvciBhUGlkIGluICR0aGVwaWRzOyBkbwoJZWNobyAiS2lsbGluZyBQ'
            'SUQgJGFQaWQiIHwgbG9nZ2VyIC1wIGxvY2FsMC5ub3RpY2UgLXQgUlNUX0hUVFBECglraWxsIC05'
            'ICRhUGlkCmRvbmUKc2VydmljZSBodHRwZCBzdGFydCB8IGxvZ2dlciAtcCBsb2NhbDAubm90aWNl'
            'IC10IFJTVF9IVFRQRApzZXJ2aWNlIGh0dHBkIHN0YXR1cyB8IGxvZ2dlciAtcCBsb2NhbDAubm90'
            'aWNlIC10IFJTVF9IVFRQRApsb2dnZXIgLXAgbG9jYWwwLm5vdGljZSAtdCBSU1RfSFRUUEQgRG9u'
            'ZQo=')
 
doLog("info", "Installing 'restart_httpd.sh' script on "+str(aHost))
cpURL = "https://"+aHost+"/mgmt/tm/util/bash"
headers = { "content-type" : "application/json", "X-F5-Auth-Token" : str(f5AuthToken)}
theData = { "command": "run",
            "utilCmdArgs": "-c 'echo \'"+script_body+"\' | base64 -d &gt; /opt/scripts/restart_httpd.sh'" }
apiResponse = requests.post(cpURL, headers=headers, verify=False, json=theData)
try:
    apiData = json.loads(apiResponse.text)
except:
    apiData = apiResponse.__dict__
doLog("info", "Install Result: "+str(apiData))
 
# Execute the script that was uploaded to the F5; and restart the httpd daemon
 
doLog("info", "Restarting the httpd service on "+str(aHost))
cpURL = "https://"+aHost+"/mgmt/tm/util/bash"
theData = { "command": "run",
            "utilCmdArgs": "-c '/bin/bash /opt/scripts/restart_httpd.sh &amp;'" }
try:
    apiResponse = requests.post(cpURL, headers=headers, verify=False, json=theData)
except:
    doLog("info", "Expected error exit code from restart on "+str(aHost))
    doLog("info", str(apiResponse.__dict__))
 
# Wait a few seconds to allow the script to perform the tasks.
time.sleep(2)
 
# Get the last 15 lines of the log file and look for a successful restart message
# The excerpt "Starting httpd: [  OK  ]" will indicate the service was successfully
# killed off and restarted.
 
cpURL = "https://"+aHost+"/mgmt/tm/util/bash"
doLog("info", "Verifying httpd service restart on "+str(aHost))
theData = { "command": "run",
            "utilCmdArgs": "-c 'tail -n15 /var/log/ltm | grep -i RST_HTTPD'" }
apiResponse = requests.post(cpURL, headers=headers, verify=False, json=theData)
try:
    apiData = json.loads(apiResponse.text)
except:
    apiData = apiResponse
 
doLog("info", "RESTART Service Result: "+str(apiData.get("commandResult",apiData)))
if "Starting httpd: [  OK  ]" in apiData.get("commandResult",""):
    doLog("info", "Successful restart!")
else:
    doLog("info", "Restart result found issues!  ABORTING!")
    sys.exit(1)
 
&nbsp;
&nbsp;I hopes this helps someone out who wants to automate this same or similar tasks that require the httpd to be restarted.&nbsp;ULTIMATELY I hope that this ENCOURAGES F5 to fix the BUG and makes this solution OBSOLETE! 🙂

Forum Discussion

Bug (ID 775845) Workaround; REST API httpd restart

Recent Discussions

Client SSL Profile set to Require Client Certificate breaks RDP in APM

TS Declarations for DNS

APM file and registry key date check 8 days old

SSL bridging without SSL proxy forward

Should config via cli rather than gui?

Related Content

APM AD authentication case-sensitivity workaround

iRule to restart APM session

Impact of restarting big3d service

configuration utility restarting (upgrade)

GUI restarts when checking Certs on appliance

ABOUT DEVCENTRAL

RESOURCES

SUPPORT

PARTNERS