Getting Started with iRules: Basic Concepts
Welcome to the third installment of this series, wherein we will, for the first time, actually discuss iRules at length. That may sound odd, but there has been some important foundational work to do before diving too deep into the technology behind iRules themselves. If you’re new to F5, new to programming, new to both, or just looking for a refresher on some very rudimentary concepts and terminology, I recommend checking out the first two articles in this series so that we’re all on the same page as we wade into iRules proper. Don't worry, we'll wait! Back for more? Great! Now that everyone is equally equipped, let’s dig in a bit to the meat of the topic at hand: iRules. Following a similar style as the first two introduction articles in this series, we’ll outline a few topics to cover, and then delve into them. To give a general introduction to iRules as a technology we’ll try to answer the following questions: What is an iRule? How does an iRule work? When would I use an iRule? When would I not use an iRule? What is an iRule? An iRule, in its most simple terminology, is a script that executes against network traffic passing through an F5 device. That’s pretty vague, though, so let’s try and define a bit more about what actually occurs within an iRule. The idea is pretty straightforward; iRules gives you the capability to write simple, network aware pieces of code that will influence your network traffic in a variety of ways. Whether you’re looking to do some form of custom persistence or rate limiting that isn’t currently available within the product’s built-in options, or looking to completely customize the user experience by granularly controlling the flow or even the contents of a given session/packet(s), that’s what iRules was built for. iRules can route, re-route, redirect, inspect, modify, delay, discard or reject, log or … do just about anything else with network traffic passing through a BIG-IP. The idea behind iRules is to make the BIG-IP nearly infinitely flexible. We recognized early on the need for users to be able to configure their systems to interact with network traffic in many ways that either we haven’t thought of, or are simply corner cases and/or in the minority of traffic being dealt with by our users. As such, rather than forcing them to submit requests for us to modify our core architecture every time they wanted to be able to use their F5 devices in a manner that slightly diverged from the collection of check boxes and drop downs available in the standard UI, we offered them iRules, and thereby a way to do what they need, when they need it. At the end of the day iRules is a network aware, customized language with which a user can add business and application logic to their deployment at the network layer. You can see a basic example iRule below, this is what iRules look like, and we will explore the different parts of an iRule in far more depth in coming parts of this series. If you're not fully comfortable with the code yet, don't let that scare you, we'll dig into each part of what you'll need to build iRules as the series continues. For now the idea is to start making iRules look and feel more familiar. # Rename a cookie by inserting a new cookie name with the same value as the original. Then remove the old cookie. when HTTP_REQUEST { # Check if old cookie exists in request if { [HTTP::cookie exists "old-cookie-name"] } { # Insert a new cookie with the new name and old cookie's value HTTP::cookie insert name "new-cookie-name" value [HTTP::cookie value "old-cookie-name"] # Remove the old cookie HTTP::cookie remove "old-cookie-name" } } How does an iRule work? To start at the beginning, as it were, an iRule is first and foremost a configuration object, in F5 terms. This means that it is a part of your general bigip.conf along with your pools, virtual servers, monitors, etc. It is entered into the system either via the GUI or CLI, generally speaking. There is also an iRules Editor available for download on DevCentral that is a windows tool for editing and deploying/testing iRules which can be extremely useful. Unlike most configuration objects, though, an iRule is completely user generated and customizable. An iRule is a script, at its core after all. Regardless of how an iRule gets there, be it UI, CLI or Editor, once an iRule is part of your config, it is then compiled as soon as that configuration is saved. One of the gross misconceptions about iRules is that, as with most interpreted scripting languages such as TCL, and interpreter must be instantiated every time an iRule is executed to parse the code and process it. This is not true at all, because every time you save your configuration all of your iRules are pre-compiled into what is referred to as “byte code”. Byte code is mostly compiled and has the vast majority of the interpreter tasks already performed, so that TMM can directly interpret the remaining object. This makes for far higher performance and as such, increase scalability. Now that the iRule is saved and pre-compiled, it must then be applied to a virtual server before it can affect any traffic. An iRule that is not applied to a virtual is effectively disabled, for all intents and purposes. Once you’ve applied an iRule to a given virtual server, however, it will now technically be applied against all traffic passing through that virtual. Keep in mind though, that this does not necessarily mean that all traffic passing through the virtual in question will be affected. IRules are most often very selective in which traffic they affect, be it to modify, re-route or otherwise. This is done through both logical constructs within the iRules, but also through the use of events within the iRule itself. Events are one of the ways in which iRules have been made to be network aware, as a language. An event, which we’ll dig into in much more detail in the next installment of this series, is a way of executing iRules code at a given point in time within the flow of a networking session. If I only want to execute a section of code once for each new connection to the virtual server to which my iRule is applied, I could easily do so by writing some simple code in the appropriate event. Events are also important because they indicate at which point in the proxy chain (sometimes referred to as a hud chain) an iRule executes. Given that BIG-IP is a bi-directional proxy, it is important for iRules to execute on not only the right side of the proxy, but at the right moment in the network flow. So now you have an iRule added to your configuration, it has been automatically pre-compiled to byte code when the configuration was saved, you have it applied to the appropriate virtual server, and the code within the iRule calls out the desired event in which you want your code to execute; now is when the magic happens, as it were. This is where the massive collection of iRules commands comes into play. From header modification to full on payload replacement to creating a socket connection to an outside system and making a request before processing traffic for your virtual, there are very few limitations to what can be achieved when combining the appropriate series of iRules commands. Those commands are then processed by TMM, which will affect whatever change(s) it needs to the traffic it is processing for the given session, depending on what you’ve designed your iRule to do. The true power of iRules largely comes into play thanks to the massive array of custom commands that we’ve built into the language, allowing you to leverage your BIG-IP to the fullest. When would I use an iRule? The ideal time to use an iRule is when you’re looking to add some form of functionality to your application or app deployment, at the network layer, and that functionality is not already readily available via the built in configuration options in your BIG-IP. Whether it’s looking to perform some kind of custom redirect or logging specific information about users’ sessions or a vast array of other possibilities, iRules can add valuable business logic or even application functionality to your deployment. iRules have a single point of management, your BIG-IP, as opposed to being distributed to every server hosting whichever application you’re trying to modify or affect. This can save valuable management time, and can also be a large benefit in time to deployment. It is often far easier to deploy an iRule or an iRule change than it is to modify your application for a quick fix. As an example, one of the most common uses of iRules when it was first introduced was to redirect all traffic from HTTP (port 80) to HTTPS (port 443) without affecting either the host or the requested URI for the connection. This was (and still is, pictured below) a very simple iRule, but it wasn’t at the time a feature available in the standard configuration options for BIG-IP. when HTTP_REQUEST { HTTP::redirect "https://[HTTP::host][HTTP::uri]" } When would I not use an iRule? The above example of an HTTP to HTTPS redirect iRule actually depicts perfectly when to not use an iRule, because that functionality was so popular that it has since been added as a profile option directly in the BIG-IP configuration. As such, it is more appropriate, and technically higher performance, to use that feature in the profile as opposed to writing an iRule to perform the same task. A general rule of thumb is: Any time you can do something from within the standard config options, profiles, GUI or CLI – do it there first. If you’re looking to perform a task that can’t be accomplished via the “built-in” means of configuration, then it is a perfect time to turn to iRules to expand the possibilities. For examples of using local traffic policies in lieu of iRules, check out Chase’s To iRule or Not to iRule article. This is for a few reasons, not the least of which is performance. iRules are extremely high performance, as a rule, and if written properly, but there is always a slight benefit in performance when you can run functionality directly from built in, core features as opposed to a custom created script, even an iRule. Also, though, it is easier to maintain a feature built into the product through upgrades, rather than re-testing and managing an iRule that could be easily replaced with a few configuration options. This concludes the introductions of core concepts in the series. Hopefully this gives everyone a solid place to start to allow us to begin digging into an array of iRules topics with more confidence and a unified starting place. Starting with the next installment of the series we will begin to delve deeper into individual iRules concepts and functionality, beginning with events and priorities.Getting Started with iRules: Technology & Terms
In the last installment, we took a look at some programming fundamentals, with the hopes to get people on the same page so that we could start building towards some more advanced concepts. To do so, however, we would be remiss if we didn’t look at both sides of the equation. You see when looking to write iRules the programming side of things is only half of the picture. This is one of the things that makes iRules so unique. It is truly a network aware programming language, and as such you need to not only understand the fundamentals of programming, but also how F5 devices work, what options you have within the product, the terminology we use, etc. Let’s face it, if I hand you a bucket full of iRules commands and tell you to go solve a problem you aren’t going to get very far if you don’t know what a pool is, what to call a client IP address, or how to make use of a VIP. Consider the last article a starter’s guide for programming basics to help get the non-programmers up to speed. If that’s the case, this installment of #The101: iRules is intended to be a primer for non networking (or at least, non F5 aware) users to get up to speed. iRules is an amazingly powerful language, but it is only as powerful as the abilities of the person doing the coding, and those abilities start squarely with an understanding of what the device you’re programming is capable of. Using the same format as before, let’s take a look at a glossary of basic terms/technologies that you will see often come up when talking about or working with F5 technology. Here is a glossary of F5 technology for iRules programmers: Virtual Server Pool Pool Member Node Profile Client Side Server Side TMM CMP We’ll take each of these concepts in turn and give a brief overview to illuminate what they mean when speaking in the context of F5 devices. For the more detailed information as it relates to specific versions of TMOS, head over to My F5. Virtual Server (Virtual IP) A Virtual IP, or VIP, also known as a Virtual Server is a key component in any BIG-IP configuration. It’s kind of the starting point most times when people are thinking about building a configuration for a given application. The VIP is the destination (combination of IP and port) to which requests will be sent when bound for whatever application lives behind the BIG-IP. For instance if you have a server hosting your web application living behind an F5 device, it would no longer have a public facing internet address. Instead you would assign that public address to the BIG-IP in the form of a VIP with whichever accompanying port you are expecting traffic on, likely 80, 443 or all (*) in this case. So you would end up with a VIP on the front (or “client side”) of your BIG-IP that directs traffic to the server(s) on the back end. The VIP is integral as it is where all traffic is directed from the outside, where profiles and other configuration options are defined, and much more. It is not uncommon for application configurations to make use of multiple VIPs, especially when they receive traffic on multiple ports, or if they need to make use of multiple profiles for some reason or another (perhaps some requests use a client SSL certificate and others don’t). So it is important to remember that one VIP does not necessarily mean one application. A VIP is a configuration object on the BIG-IP that allows you to tie together a destination IP:port combination and process traffic for that combination. Whether it is to route to a back end server, redirect elsewhere, deny, discard, inspect, or simply log information about said traffic…there is a near limitless number of options with what you can do with the traffic once it enters the F5 device. To get it there, however, you’ll need a VIP. Pool A pool, in the simplest terms I can muster, is a grouping of servers. Like a VIP, a pool is an integral BIG-IP configuration object. This one, however, can be considered effectively one step lower in the configuration stack, as it were. Meaning you must have a VIP in place to allow traffic into your F5 device, generally speaking, and only once it’s there do pools become relevant. A pool is a collection of one or more servers, referred to as members, which we will get into In a moment. The pool is where the type of load balancing desired is chosen, where some options such as rate limiting and the like are applied, and one of the most important of the many layers of monitoring that can be applied to an application’s stack within a BIG-IP. Monitoring the pool level is important because it will allow you a clear representation of which groups of servers are or aren’t available at a given time. Each VIP has the option of selecting a default pool, but it is also possible to direct to another pool should the primary pool be unavailable. In some configurations there isn’t a default pool stated, and instead a pool is chosen based on criteria that is gleaned from the connection once it is in place. Regardless, pools are where the servers hosting the application being served live, and as such they are a crucial part of any deployment. Pool Member A member is one of the servers associated with a given pool. You will hear the term “pool member” often, and this is why, it is a term referring to one of the particular servers associated with the designated pool. Pool members play an important role, obviously, because they are the representation of the actual servers themselves in your configuration. The combination of a VIP, pool and pool members makes up the overall, general structure of a basic application stack within a BIG-IP. There are thousands of permutations and possible options of course, but this is the most basic, generic view, and is important to understand for a starting point. Pool members can have many options toggled on them as well, in addition to the configuration options already inherently in place on any traffic destined to the members of a pool due to the configuration of the pool or VIP upstream from the members in the config hierarchy. So again, generally speaking, traffic will come in destined for a particular VIP. That VIP will then route the traffic to a given pool, based on either the default pol selected for the VIP or some other criteria, perhaps an iRule. The traffic will then arrive at that pool and a load balancing decision will be made within the pool, based on currently available members and the select load balancing algorithm, and traffic will finally be directed to a pool member, which is the final destination (I.E. server) which will process the request and respond accordingly. Node What if you have a VIP, but don’t want to route traffic to the member of one of the pools on your BIG-IP. What if instead you would rather route traffic to a particular server via IP address, regardless of whether it is part of your configuration, lives behind your BIG-IP, or is a member of a pool? This is the concept of a node. A node is any destination IP to which you would like to direct traffic. It is not treated the same as a pool, no load balancing decision is made, no monitoring is done, it is a rather “fire and forget” type of action. It has many uses, but is not generally recommended for the majority of a normal application’s traffic flow. Still, the term will come up in the iRules world, as there is a specific command for sending traffic to a node from within an iRule, so it’s important to understand what one is. Profile A profile is the heart of much of the processing done for each session that is established and flows through the BIG-IP. A profile gets applied to a VIP and dictates what type of traffic is expected, TCP or UDP, it dictates whether or not SSL offloading will be done on the client side, which SSL profile is used if it is, whether or not a particular protocol will be used such as HTTP or SIP, whether or not ONECONNECT will be enabled, and much, much more. If the VIP is the destination for your application traffic think of the profile attached to said VIP as the control center that tags, inspects, interprets and categorizes the traffic once it arrives. A profile is essential for many reasons, not the least of which is that, from an iRules perspective, there are several commands that are only available for use on a VIP that has certain profiles applied. For instance the popular and often used HTTP commands are only available on VIPs that have an HTTP profile applied. This is because the profile applied to the VIP does a large amount of processing, as I’ve said, and the commands rely on some of the data that results from that processing and interrogation. There are many different kinds of profiles when talking about F5 technology. SSL profiles, Auth Profiles, Protocol Profiles and more. The concept is similar for all of them, with different nuances and outcomes. Most often when dealing with iRules you’ll need to be sure you’re using the appropriate Protocol profile (such as HTTP) and applying an SSL profile when required, if you’re trying to inspect and process SSL traffic. To do so you must terminate SSL at the BIG-IP, which is done by applying a ClientSSL profile to the VIP in question. Client Side Unlike the terms in this glossary up this point, client side is not an actual configuration object. It is instead a concept that I feel is important to describe and express to those looking to gain a level of comfort with F5 technology, and iRules in particular. The BIG-IP is a full proxy architecture, which means that there are separate IP stacks for each side of the proxy, client side and server side. This means that as traffic progresses through the BIG-IP , at some point it is transferred from one stack to the other, and vice versa on the responses from the server to the client. This is important because different profiles, configuration objects and iRules commands themselves are only available or function differently within different contexts. It is important to know whether you are trying to or need to affect client side traffic or server side traffic to accomplish what it is you’re looking to do. This is a term you will hear thrown around quite a bit as you delve deeper into working with F5 technology and particularly iRules, and the simplest way to describe it is “anything that occur on the client side of the proxy architecture is in the client side context”. Server Side Server side, as you can likely deduce, is the exact opposite of the above. It is a term used to describe something occurring on the server side of the proxy architecture. This is usually the responses from application servers, but keep in mind that client side and server side depend entirely on which side of the proxy initiates the transaction. If you have a server in a pool reaching out through your BIG-IP to perform an action of its own accord, that server is now the “client” for the purposes of the proxy discussion, and as such it is on the client side of the transaction, despite it being a server itself. TMM Someone could easily write a several thousand-page paper on the TMM in and of itself, so I will not attempt to discuss it in detail. I only want to ensure that when the term is referenced it is not completely foreign. The TMM is the Traffic Management Microkernel. It is the custom kernel that F5 developed specifically to handle traffic processing and routing. It is designed from the ground up to be high performance, reliable, and flexible for our needs. The TMM is what does all of the actual traffic processing on any BIG-IP. Whether it is an iRule being executed, a profile inspecting traffic as it comes through a VIP, or just about anything else that touches the traffic as it traverses an F5 device, it happens within the TMM. The TMM lives separately from the “host OS”, which is likely the only other important thing to know in regards to TMM at this point. The hostOS handles things such as syslog, sshd, httpd, etc. while leaving the TMM free to do only what it is best at – processing traffic in absurdly high volumes. CMP Again, there are papers that already exist that depict CMP in a far more thorough and articulate manner than I could hope to achieve, so I will give the primer version to elucidate the very basics of the concept. CMP is “Clustered Multi-Processing”. This is F5’s proprietary way of dealing with multiple core devices. In essence, in a very rough sense, each core in a device is assigned its own individual TMM (see above) to handle processing of traffic for that core. There is then a custom disaggregator (DAG) built into the system that decides which TMM, and as such, which core to send traffic to for processing. In this way F5 is able to achieve massively linear scalability in multiple CPU, multiple core systems. That barely skims the surface of this technology, but when discussing iRules it is important to know that CMP is a good thing, and breaking or demoting from CMP is, generally speaking, bad. Hopefully this explains why that is, and what CMP is at a basic level. Armed with a basic understanding of these concepts I am hopeful that the world of F5 and iRules will be far easier to understand for those that may not have been exposed to such before. Now that everyone has a solid basis in both programming and F5 concepts we will move forward with discussing iRules in particular. In the next article we will delve into iRules as a technology, how it works, why it exists, how to make use of an iRule and more.Getting Started with iRules: Intro to Programming with Tcl
In this Getting Started with iRules series, We’ll cover topics ranging from this first installment, which includes some programming basics and concepts, up through F5 terminology and concepts, iRules basics and usage, components, etc. This primer is low level, intended for those that might be looking to connect some core concepts of programming and how they interact, as well as a refresher for those that haven't spent much time in the trenches scripting. In this first installment of the series the idea is to discuss programming at more of a general, basic level. Think of it as a sort of “What you need to know” style introduction. For many this will be extremely low level review, so feel free to skip ahead in the series. For those that may be truly new to programming as a whole or feel that you’re rusty enough that you could use a “from the beginning” refresher, the concepts detailed here are important and hopefully will be useful information. They’re certainly concepts that are important to understand if you hope to begin your journey into iRules Fu greatness. So what do you need to know? Let’s get started. A Programmer's Glossary First let’s talk about some very core programming concepts. It’s important that as the series goes on the basic terms and concepts are understood, so think of this as sort of a getting started glossary. The basic things that you need to know at least a bit about to describe “programming” as a concept are: Commands Variables Conditionals Operators We’ll take each of these in turn and give a very brief overview. Commands The concept of commands is an exceedingly simple one, for the most part. Each programming language has an array of many different commands, which can be executed to achieve different desired results. Host name lookups, spawning child processes, accessing the shell to run a command, storing information in memory, directing traffic, or any number of other things are achieved by a command, or a series of commands called from within the code in question. At the core the commands in any code are the part of the code that actually “does something”. The rest is there to manipulate information, create logical support and flow, etc. Without the commands to actually perform an action, there would be no effect when any script was run, despite how pretty the logic and code was. For a basic look at how you'd call some commands programatically in a few different languages, see below: Variables Variables are integral parts of any programming language as they are used to store information in memory. By storing information in a variable you are then able to later recall it later, manipulate that information, pass it to a given command, etc. For instance: If I have code that performs a host lookup to retrieve an IP address, I could execute commands on the result of that and things would work just fine. What happens the next time I want to execute commands against the resulting IP address? My code has to call the operation that did the lookup again. What if, on top of multiple iterations, there were also multiple actions involved? Perhaps I want to compare the IP destination of a given host to value, then based on the result execute a command using that host’s IP. Without storing the result from the host lookup I’m now looking at two lookups every time I execute my snippet of code. By storing the resulting IP in a variable I would be able to perform a single lookup and use the variable for as many iterations and operations as I want without having to perform another lookup every time. Variables are also essential for many things such as conditional testing, string manipulation and more, but more on that later. You can see a basic example of a few different languages and the way they handle simple variables below: Conditionals Conditional statements are a type of control structure in programming that allows you to create a logical flow within your code. For instance if you want to perform a given action every time your script is run, you’d just execute the relevant command. If, however, you only want to run a specific command under particular circumstances, such as only redirecting traffic to a particular server if the primary server is down for instance, you need a way to logically state that. The most basic conditional statement, the if conditional, allows you to do things like dictate that a command or set of code only be executed if a given condition is met, hence the term “conditional”. For instance “look up the IP of this hostname but only if no one has requested it before”. Again, to see what basic conditional constructs look like, we go to a simple chart of a few relatively standard languages: Operators There are two main types of operators that we’ll cover here, logical and comparison. Comparison Operators With commands, variables and conditionals you’ve got a pretty strong skeleton of basic programming functionality, but a very important piece is missing. Comparison operators are an integral part of any language as they are what allow you to evaluate and compare values against one another. What good is a conditional statement saying “if x = 1 do y” without the equals? Standard comparison operators are things we’ve all seen since early math classes. =, >=, <=, != are among the commonly used. There are, of course, others however. Things like regular expression based matching, substring matching and more are all available in most languages. The general concept of a comparison operator is simple though, at its core it is designed to compare one value against another. Whether that is the result of a command being compared against a static value to see if they are equal or two variables being compared to see which one has a greater numeric value, it’s all thanks to comparison operators. Logical Operators The other major type of operator that we’re concerned about in this series is the logical operator. Logical operators are used to build logical expressions which can be useful in conjunction with conditionals to create a slightly more complex logical flow in your code. For instance a standard conditional might say “if x = y … do something”. But what if you want to do that same thing if x is equal to either y or z? Well, you could either write the same code block twice, once for the case of “x=y” and again for “x=z”, but not only would that be wildly inefficient, it would cause all sorts of problems down the road. A simple logical or can simplify this immensely. Simply changing the conditional statement to “if x = y or if x = z … do something” will get the result you’re looking for without the need to repeat your logic and without some of the major pitfalls that making such blatant repetitions can cause if you’re not careful. That should serve as a very rudimentary glossary of terms for just about any programming language out there. Every language has its own nuances and intricacies of course. The language on which iRules is based is Tcl (Tool Command Language). With it Tcl brings its own unique traits in syntax as well as functionality. Tcl Specifics - The What and Why There were many choices when building iRules as to which language would be best suited for the purpose. There are many reasons that Tcl was chosen, but chief among them are performance, customization and ease of use. While it may not be the most robust language when comparing raw, base functionality to others, Tcl serves the purposes of iRules quite well. Performance was a massive factor when considering a language on which to base iRules. Tcl is lightweight, compact and highly performant. In an environment where each “script” may be executed hundreds of thousands of times per second, performance is absolutely paramount. The ability to be compiled ahead of run time is a massive win in the battle for performance. Unlike many interpreted languages Tcl does not have to be interpreted at run time, but rather it can be pre-compiled and run in an extremely efficient manner, which allows for exceptionally high processing rates compared to many similar languages. The efficient and easy to use C API within Tcl is also a major boon as most of the commands built into iRules are custom built. Many of the calls made from iRules make use of this API. Because of this, the customization capabilities of a language had to be taken into account as well when considering languages. Tcl is easily modified and amended to fit our purposes. As the majority of commands that make it into iRules are in fact custom, this is a necessity that Tcl provides well. No language was as network aware as required for our purposes and adding custom commands is the way in which we were able to shape a language to fit the needs of the ADC role, blending network and application functionality, that iRules plays so well. To that end, any language also needed to be easy to use to properly fit the bill for iRules. Tcl is simple to read, quick to learn and easy to troubleshoot even when passing from person to person, at least compared to some of the more complex options. This is especially important given the nature of the code often written in iRules (somewhat simple, quickly produced code when compared to a product release cycle). For the complete story on the why of Tcl you can read more in depth here. For now though, this should hopefully serve as a brief introduction into what benefits Tcl itself brings into the iRules picture. Note - This content is restructured from earlier series written by Colin Walker, Joe Pruitt, Jason Rahm, & Deb Allen.Getting Started with iRules: Logging & Comments
So far in this series we’ve covered some pretty varied topics, from a rudimentary primer on programming generalities to basic iRules components (and why they’re important.) In this article, we’ll cover a truly powerful yet simple function in nearly any programming language. Something that makes development easier, troubleshooting possible and has saved many developers long nights of agony trying to find metaphorical needles in haystacks. Today we’ll talk about logging and cover some basic questions: What is logging? What types of logging are available in iRules? Is there a cost to logging? When should I log and when shouldn't I? What are comments and how do they work? What does commenting my code cost me? When should I comment my iRule? Troubleshooting iRules - Where do I start? What is logging? Logging is simple, really. It’s the act of sending information from whatever script or program you’re writing to a log file. That may seem simple and mundane, and in many ways I suppose it is, but making good use of that functionality is paramount if you want to stay sane and build effective code. Logging is integral to almost any language for a number of reasons. Trapping errors, returning information to the system, relaying troubleshooting and development information, and more. Without log files and the logging functions built into most languages, including iRules, you’d be flying largely blind. Whether you’re on a BIG-IP or just about any other platform the logging concept stays the same. You have a script, you tell it to output information, and you have, generally speaking, two options on where to send it, assuming you’re not sending it off to a socket or another system somewhere. You can send it to one of the available outputs, often times the command line or screen, or you can capture it in a file. To make that whole “capture it in a file” bit easier and more predictable, we have standard log files available to us in most cases. In the case of the BIG-IP iRules log entries will go to /var/log/ltm by default. All you have to do is fire the log command and your information will show up for processing or perusal. A basic log entry contains the data and time of the entry, the facility, severity, log message and more. An example entry looks something like this: Oct 4 00:42:51 tmm err tmm[17084]: 01220001:3: - "Host: domain.com" What types of logging are available in iRules? That explains the “what”, but what about the “how”? In iRules, there are three main ways in which you can log information. Local Logging, Remote Logging, and High Speed Logging Local Local logging is the most basic and common form of logging information. This means that you execute a log command in your iRule and the information is sent to /var/log/ltm, as mentioned above. This is where all standard iRules information will go also, including any iRules error messages that may crop up from the system. Those standard logs are handled by syslog-ng, which means they’re stored in standard syslog format, in case that matters to you. This also means that the logging is not handled by TMM, but rather by the host OS, which will become relevant shortly. Logging locally is great for troubleshooting and for temporary log statements while in the development process. Local log statements do, however, write data back to the host OS and to disk, and that means they aren’t necessarily ideal for high traffic production deployments. It isn’t as great for performance in production deployments, especially high traffic production deployments. Local logging is simple to use. All you need is a log statement that specifies the log command, the log facility, and the message you want to enter into the log file. Log facilities are used to identify the sender of a log message, often times a daemon, and can dictate different logging behaviors in some cases. For our purposes in iRules we’re going to always use a log facility of “local0.”, except in rare, customized cases. The finished log command is quite simple to use, like so: when HTTP_REQUEST { log local0. “Requested hostname: [HTTP::host] from IP: [IP::local_addr]” } For the definitive documentation, more examples and notes check the log command pageon Clouddocs. Remote Logging remote is effectively the exact same idea as logging locally, I.E. still sent from the TMM in standard syslog format, still uses the log command, etc.; with one major difference: the logs aren’t actually stored locally. Instead, with remote logging, you’re able to specify a log server IP that will receive the formatted log statements sent directly from TMM. This is a far preferable option for performance reasons, though beware, a highly trafficked BIG-IP can easily overwhelm many logging servers if sending robust log information on each request. To remotely log using the log command you’d just modify the log command to include a remote IP address, like this: when HTTP_REQUEST { log local0 10.10.10.1 “Requested hostname: [HTTP::host] from IP: [IP::local_addr]” } High Speed Logging As of version BIG-IP version 10.1 there is a third and quite powerful option for logging. High Speed Logging, quite often referred to as HSL, is a way in which you can use TMM to send data off of the BIG-IP at an extremely high rate of speed, in a very efficient manner. These commands were originally designed to be used for logging, as the name indicates, but the reality is that they can be used to send nearly any data you want, as the data sent via HSL is not formatted in any specific way. The general idea behind HSL from a logging sense is the same as remote logging via the log command, except that you have much more control with the HSL commands. Because you are specifically sending data directly to an open HSL connection with the HSL::send command, you can send whatever data you want, in any format you’d like. This has also led people to using the HSL commands for things other than strictly log information. You can read more about the specific HSL commands here but a simple example looks like: when CLIENT_ACCEPTED { set hsl [HSL::open -proto UDP -pool syslog_server_pool] } when HTTP_REQUEST { # Log HTTP request as local7.info; see RFC 3164 Section 4.1.1 # "PRI Part" for more info HSL::send $hsl " [IP::local_addr] [HTTP::uri]\n" } Note that with HSL we had to specifically open a connection to a pool before we could send any log info…this is important. Without the HSL connection opened your send command(s) will fail. This is extremely similar to the sideband connections behavior introduced in a later (11.0) version. That is because sideband connections were developed as an extension of the HSL model once it was realized just how powerful this type of functionality could be. Is there a cost to logging? Yes. Technically speaking there is a cost for any operation you perform in an iRule, it’s all about how often you are performing each action, and the actual overhead of that action. While the log command itself is quite low in overhead, the things that happen after TMM executes its portion of the work become rather costly quickly. iRules, and just about everything else that gets run against traffic on the wire within a BIG-IP, are run from within TMM (Traffic Management Microkernel). These processes are highly optimized and tuned to handle wire-speed operations. An iRule telling TMM to fire the log command and send off data is low drag and happens quickly and easily, once that’s done however, the message is sent over to syslog-ng. Keep in mind that syslog-ng is run by the host OS, meaning it is not handled by TMM. Any time you have to involve the host OS in a wire-speed transaction it can be a rather expensive operation, as the host OS is just not optimized the way that TMM is to handle massive traffic volumes. On top of that, your log entry is going to be stored on disk. If accessing the host OS is bad, depending on disk I/O, possibly multiple times at that, for each request while processing traffic is…a bad idea. It’s because of this that we recommend logging be used for development and testing, but disabled or at least greatly minimized when iRules are put into production. Of course remote logging and HSL logging will have far less impact on the performance of the iRule and the BIG-IP as a whole since the expensive part of the equation, namely actually storing the log entry to disk via syslog, is completely offloaded. If you need real-time logging in a high paced production environment, I strongly recommend using remote logging or HSL where possible. When should I log, and when shouldn’t I? Given the above discussion about performance, we generally recommend logging at development, debugging and test times, but not in production. Can you log in prod? Yes, of course. Do people do it without major issues? Absolutely. But the possible performance implications can be enough of a drawback that it’s not worth it unless truly necessary for your deployment. Why risk the performance of your application when you can easily send the log information to a system that isn’t making thousands of critical business decisions per second? What are comments and how do they work? Commenting is an extremely simple concept. It is the ability to add text to an iRule (or any script, really) without affecting the actual functionality of the script itself. For instance, say I want to leave a note for myself or, even more importantly, another engineer that is likely to encounter my cod, about what a particular section of code does, why I chose a variable name, or anything else. That’s precisely what comments are for. It’s important to keep in mind that someone else will almost inevitably have to deal with your code in some fashion or another eventually. Commenting can be the difference between readable, usable, easy to maintain code and a jumble of strings that make no sense to anyone but the original developer. To begin a comment you just start any line with a hash symbol (#), and anything on that line will be considered part of the comment: #This will log the IP address of the incoming connection when CLIENT_ACCEPTED { log local0. “IP: [IP::client_addr]” } What does commenting my code cost me? Aside from whatever time it takes to enter the comments? Nothing. Comments are a zero cost operation within iRules because they are never actually seen by TMM. When an iRule is loaded from configuration into running memory it undergoes a process we’ve talked about in which it is pre-compiled. During this process the iRule is interpreted by TCL and formatted into a much simpler, lower level set of commands and strings and variables. At this point the comments are completely removed, so there are no comments in line when it comes time to actually execute the code against the traffic passing through a virtual. As such, they are a truly no cost option for your code. When should I comment my iRule? Always. Often. As much as you see fit. No, seriously, commenting has no cost and can be extremely beneficial to yourself, engineers (including F5 engineers attempting to help in many cases), and any future engineer or developer looking at your code. It’s always a good practice to document your code, whether in an external format or directly in line, and comments are an easy, simple way to do just that and be sure that the explanations for how and why you did things a certain way always follow the code wherever it ends up. Comments can also be very useful in development and testing to compare two commands or sections of code, or to temporarily remove a given piece of functionality or logic. You can go overboard of course, but that is really a readability issue more than anything. Use your judgment and don’t write a book inside your code, but generally speaking I have very rarely found code over-commented for my liking. This is doubly true when I’m the one inheriting someone else’s iRules and trying to make sense of them. Troubleshooting iRules – Where do I start? This is a question that gets asked often regarding iRules and the best answer is…right here. There is no formal debugging platform for iRules so logging and solid comments are you best friends. Keeping things clear and easy to understand by properly commenting your code, and using logging to determine what is happening while troubleshooting are invaluable. Generally it is a good practice to determine the depth of code execution when troubleshooting a behavioral problem, by placing log messages inside different logical constructs to see which cases are being matched and executed. From there it’s a matter of checking the output of commands and string formats to narrow down where the misbehavior is occurring. I would also highly recommend the use of tclsh, which as of 11.1 is available on the BIG-IP itself, in both development and troubleshooting of iRules. Tclsh is a tcl shell that allows you to natively execute Tcl commands and immediately observe the outcome. This can be extremely beneficial and a large time savings as opposed to having to re-generate traffic each time you want to make a small iRules change and test the functionality of a given command string to be sure of the output. Of course, tclsh can’t simulate any of the customized F5 additions to Tcl, but the basics are covered.
