I would like to insert a timestamp on an iRule that reflects the current time at the irule creation and then somehow to be able to update that irule when is updated. I need this cuz I want to validate sync between many nodes of the same cluster, and I need to be sure that the irules are in sync and also to be able to generate a report with the irules and the corresponding timestamps.
I don't understand. Do you have multiple device groups and wish to sync the same iRule between them?
If you do I'd probably define the iRules in git and roll out the iRules using code and roll them out using a pipeline.
If you are using BigIPReport you could also get a list with MD5s from multiple devices using a simple Python script:
import requests
import hashlib
irules = requests.get('https://bigipreport.xip.se/json/irules.json').json()
for rule in [i for i in irules if i['name'] == 'rulename']:
print(hashlib.md5(rule['definition'].encode('utf-8')).hexdigest())
# Sample output:
# d03f96a58892859e80cbd2be39e04b0c
# d03f96a58892859e80cbd2be39e04b0c
# d03f96a58892859e80cbd2be39e04b0c
# d03f96a58892859e80cbd2be39e04b0c
# d03f96a58892859e80cbd2be39e04b0c
# a2716b3fd773640b7919b91bb7e7d421
Pair this with an alert using a Slack webhook or your monitoring system and it should work fine. I would not recommend this though as it requires manual intervention.
Kind regards, Patrik
Ps. Disclaimer, I am one of the authors of BigIPReport Ds.
Hi Patrick, how are you? first of all thanks for your quick response 😀. To put you in context a little bit. I have an F5 LTM on AWS with an ASG with multiple instances. The cluster synchronizes in an active/active way. And I have an application where developers can generate iRules via REST API against the LTM. Now, what I need is to be able to validate that the irule received by one of the instances is replicated in the rest of the cluster instances. What had occurred to me is precisely, to insert a timestamp when creating/modifying the iRule and that allows me to have a version of it. Where you can also validate that timestamp that is the same throughout all the instances of the cluster. I don't know if maybe F5 LTM already has some mechanism to be able to do this in another way maybe?
My solution using MD5 works in this scenario. Just pull the iRules via REST and run an MD5 against the rule definition and alert/take action if they vary.
But the better way here would be to include this validation in the pipeline. I suppose the deploy would fail if any of the iRules fails to update? Maybe you can alert if it does?
May I ask if you have considered enable auto sync and/or replacing the old REST API with AS3?
Hello Patrik, regarding the use of AS3, unfortunately we cannot contemplate it, because we have done a development entirely in Java that already makes use of the REST API, and it would be quite expensive in terms of code refactor.
The issue is how we can do to validate that pipeline, because we write on one of the active nodes, and then the configuration is replicated (sync) in background, with which at that time of the ACK of the call to add or modify the irule we'll have the Ok if the irule is written in at least one of the nodes correctly, but nothing guarantees us that latter on there could be a problem with the sync and they remain in an inconsistent state.
I like the option of pulling the irules via REST and run the MD5 check against them and take actions based on that, but I will need to install and configure IpReports for that. I was wondering if there is any solution that doesn't involve installing additional components?
You definitely do not need BigIPReport to use the rest API. I just used it as an example to showcase the MD5 idea and the data was available. Here's an article on how to manage iRules using curl.
Just have your application add a single line with a manual timestamp at creation time. All you need is # Modified: date. If a developer updates this is automatically updated to a current date. The you can simply check propagation by the timestamp on the deployed iRules. If you want creation date then add that as well when its new. When they edit an iRule using your application you can strip these values off so the developers have no access to them. In essence they are application controlled.
What I meant is when you go to save an iRule to a BIG-IP your code adds headers to track its deployment. So if my iRule was
when HTTP_REQUEST { my code }
When deployed via API your code would modify it so it becomes
### Deployed: Timestamp Here ### when HTTP_REQUEST { my code }
Then all it has to do is wait for that to appear at the destination to know it's been propagated. Time stamp is just a unique value at the time of deployment. Epoc seconds would be useful unless you want the value human readable.
How does this work? When your API reads the iRule to check for propagation it can verify with the header you added to see if it matches what it should be. When actually reading the iRule to return it downstream it can strip off that header. In essence it's shadow header for tracking propagation, controlled by you. Has zero dependancies on anything else and no effect on the iRule itself because comments are ignored but saved with the iRule.
Using MD5 is another method but without something in the iRule content to make it unique you won't be able to detect propagation if the iRule is redeployed unmodified. When the aim is propagation verification regardless of content each item has to be unique.
( Note: I'm waiting for F5 to step in here and say.. this API mechanisim already exists and you can check it in this way.... @LiefZimmerman )
Using MD5 is another method but without something in the iRule content to make it unique you won't be able to detect propagation if the iRule is redeployed unmodified. When the aim is propagation verification regardless of content each item has to be unique.
Either I don't understand you, or you don't understand me. Irules consists of characters, if even one character changes the whole MD5 sum changes. Fetching the irule definition via API from each of the LTMs is a legitimate way of validating that they're all running the same version of an iRule.
@catoverflow Here's an example in Python3 without BigIPReport:
import requests, hashlib, urllib3, os
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
class F5rest:
def __init__(self, username: str, password: str, device: str, verify_ssl=False):
self.device = device
self.username = username
self.password = password
self.verify_ssl = verify_ssl
self._session = None
@property
def session(self):
if not self._session:
s = requests.Session()
body = {
'username': self.username,
'password': self.password,
'loginProviderName': 'tmos'
}
token_response = s.post(
f'https://{self.device}/mgmt/shared/authn/login',
verify=self.verify_ssl,
auth=(self.username, self.password), json=body) \
.json()
token = token_response['token']['token']
s.headers.update({'X-F5-Auth-Token': token})
s.verify = self.verify_ssl
self._session = s
return self._session
def get_irule(self, name: str):
response = self.session.get(f'https://{self.device}/mgmt/tm/ltm/rule/{name}')
return response.json()
username = os.environ.get('F5_USERNAME')
password = os.environ.get('F5_PASSWORD')
if not (username and password):
raise ValueError('Missing credentials in environment variables F5_USERNAME or F5_PASSWORD')
device_list = ['bigip.xip.se', 'bigip2.xip.se', 'bigip3.xip.se']
hash = None
for device in device_list:
f5 = F5rest(username, password, 'bigip.xip.se')
rule = f5.get_irule('encrypted_time')
rule_hash = hashlib.md5(rule['apiAnonymous'].encode('utf-8')).hexdigest()
if hash is None:
hash = rule_hash
if not hash == rule_hash:
# Post Slack webhook here or raise exception
raise Exception('Hashes does not match')
Now, there are multiple ways to skin the cat. You could also inject headers programmatically using your pipeline that shows the version of the iRule, using the iRule itself. Then read the headers in ie. Splunk/elastic and validate that it has changed by monitoring the traffic logs.
If you just want to know the version by manually logging in to each device and checking the irule with your own eyes then Kevins suggestion to put a version number / deploy time as a comment at the top would work too.
Hard to give an exact answer unless we know how you intend to use this. 🙂
Using MD5 is another method but without something in the iRule content to make it unique you won't be able to detect propagation if the iRule is redeployed unmodified. When the aim is propagation verification regardless of content each item has to be unique.
Either I don't understand you, or you don't understand me. Irules consists of characters, if even one character changes the whole MD5 sum changes. 🙂
As to your first question... its one or the other 🙂 You illustrated my point exactly... if you read my post carefully I said unmodified. This means no character has changed therefore the md5 will not change. Now what happens if they re-deployed the same iRule? You will not be able to tell if it has been propagated as the md5 value never changed.
They have a pipeline which updates iRules so he'd know if the iRule was successfully updated or not on the device which syncs data to the other devices.
He want's to make sure that the same iRule is deployed on all devices. If the md5 is the same, the iRules are the same. I'd say that's a pretty good way to know if the iRules matches across devices or not.
Yes but it does not solve one of the stated goals of the original question - "validate sync between many nodes of the same cluster". What you propose validates they rules are the same but not the sync process is actually working. Why is this not the same thing?
I push A to a device and I want to make sure all of my cluster ends up with A. Your solution works fine. There is clearly evidence sync is working. Now it I push B and I want to make sure all of my cluster ends up with B. Again works fine. There is clearly evidence sync is working as B is different from A. Now I push B again and I want to make sure all my cluster ends up with B. This is where it falls down. Even if the cluster has B already there is no evidence that sync is working because you do not provide a unique discriminator between what your pushing and what is already deployed.
So you're saying that if you push the exact same iRule again there is no way of knowing if the iRule that already is running on all devices is synced again?
So you're saying that if you push the exact same iRule again there is no way of knowing if the iRule that already is running on all devices is synced again?
Bingo. Hence something unique needs to be added, even if you want to use MD5.
Exactly, we want to have a way for us to validate with our control plane that the irules are effectively synchronized and are the same at all times, this, among other things, is because we need to be able to validate that if a request with a specific header enters through an F5, let's say of region A, then if it cannot satisfy the request based on the header, then to be able to evaluate before performing a fallback to the F5 of the other region (let's say B), and be able to control, among other things, that that header exists and that it can resolve it, in which case just there would proceed to perform the fallback and complete the request in the destination of region B, otherwise it would go to the default pool in the source region (in this case A).
Perhaps I am adding one more degree of complexity in the question, which would be the fallback part.
As Patrick suggested MD5 is a great way to determine the rules are the correct. If you want to verify they have been synced thats a great way to do it. It is also efficient in that if they have not changed you will know it.
If you want to verify the sync functionality itself is working as expected then you need something unique for every sync request. Hence the suggestion of adding a header line to your iRule with a millisecond timestamp. Its nothing complex, its just taking the iRule content and adding a string at the top, or bottom of the iRule with a # in front so the F5 ignores it. Then strip it off when you retrieve the iRule. It is probably useful to add a key marker so you know its your header to strip off when you retrieve the iRule. Something like...
### SYNC 1662070557000 ### # Add specific headers required for application # v1.7 iRule code here
As for the sending requests from one BIG-IP to another logging is key. Have the first rule add a tracking id (tid) header and log it with ingress information "tid: virtual source" when handling a request. Have subsequent iRules on different F5's pickup the tid and use it as "tid: log message" when they process the request. Then when you look at your aggregated logs you can track the flow based on the tid.
Pro Tip: If the tid is for tracking TCP connections then the source address and port are a good way to do this as they are required to be unique. They also have the benefit of providing useful information as well.
