Getting Started with iRules LX, Part 4: NPM & Best Practices
So far in this series we've covered basic nomenclature and concepts, and in the last article actually dug into the code that makes it all work. At this point, I'm sure the wheels of possibilities are turning in your minds, cooking up all the nefarious interesting ways to extend your iRules repertoire. The great thing about iRules LX, as we'll discuss in the onset of this article, is that a lot of the heavy lifting has probably already been done for you. The Node.js package manager, or NPM, is a living, breathing, repository of 280,000+ modules you won't have to write yourself should you need them! Sooner or later you will find a deep desire or maybe even a need to install packages from NPM to help fulfill a use case. Installing packages with NPM NPM on BIG-IP works much of the same way you use it on a server. We recommend that you not install modules globally because when you export the workspace to another BIG-IP, a module installed globally won't be included in the workspace package. To install an NPM module, you will need to access the Bash shell of your BIG-IP. First, change directory to the extension directory that you need to install a module in. Note: F5 Development DOES NOT host any packages or provide any support for NPM packages in any way, nor do they provide security verification, code reviews, functionality checks, or installation guarantees for specific packages. They provide ONLY core Node.JS, which currently, is confined only to versions 0.12.15 and 6.9.1. The extension directory will be at /var/ilx/workspaces/<partition_name>/<workspace_name>/extensions/<extension_name>/ . Once there you can run NPM commands to install the modules as shown by this example (with a few ls commands to help make it more clear) - [root@test-ve:Active:Standalone] config # cd /var/ilx/workspaces/Common/DevCentralRocks/extensions/dc_extension/ [root@test-ve:Active:Standalone] dc_extension # ls index.js node_modules package.json [root@test-ve:Active:Standalone] dc_extension # npm install validator --save validator@5.3.0 node_modules/validator [root@test-ve:Active:Standalone] dc_extension # ls node_modules/ f5-nodejs validator The one caveat to installing NPM modules on the BIG-IP is that you can not install native modules. These are modules written in C++ and need to be complied. For obvious security reasons, TMOS does not have a complier. Best Practices Node Processes It would be great if you could spin up an unlimited amount of Node.js processes, but in reality there is a limit to what we want to run on the control plane of our BIG-IP. We recommend that you run no more than 50 active Node processes on your BIG-IP at one time (per appliance or per blade). Therefore you should size the usage of Node.js accordingly. In the settings for an extension of a LX plugin, you will notice there is one called concurrency - There are 2 possible concurrency settings that we will go over. Dedicated Mode This is the default mode for all extensions running in a LX Plugin. In this mode there is one Node.js process per TMM per extension in the plugin. Each process will be "dedicated" to a TMM. To know how many TMMs your BIG-IP has, you can run the following TMSH command - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # show sys tmm-info | grep Sys::TMM Sys::TMM: 0.0 Sys::TMM: 0.1 This shows us we have 2 TMMs. As an example, if this BIG-IP had a LX plugin with 3 extensions, I would have a total of 6 Node.js processes. This mode is best for any type of CPU intensive operations, such as heavy parsing data or doing some type of lookup on every request, an application with massive traffic, etc. Single Mode In this mode, there is one Node.js process per extension in the plugin and all TMMs share this "single" process. For example, one LX plugin with 3 extensions will be 3 Node.js processes. This mode is ideal for light weight processes where you might have a low traffic application, only do a data lookup on the first connection and cache the result, etc. Node.js Process Information The best way to find out information about the Node.js processes on your BIG-IP is with the TMSH command show ilx plugin . Using this command you should be able to choose the best mode for your extension based upon the resource usage. Here is an example of the output - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # show ilx plugin DC_Plugin --------------------------------- ILX::Plugin: DC_Plugin --------------------------------- State enabled Log Publisher local-db-publisher ------------------------------- | Extension: dc_extension ------------------------------- | Status running | CPU Utilization (%) 0 | Memory (bytes) | Total Virtual Size 1.1G | Resident Set Size 7.7K | Connections | Active 0 | Total 0 | RPC Info | Total 0 | Notifies 0 | Timeouts 0 | Errors 0 | Octets In 0 | Octets Out 0 | Average Latency 0 | Max Latency 0 | Restarts 0 | Failures 0 --------------------------------- | Extension Process: dc_extension --------------------------------- | Status running | PID 16139 | TMM 0 | CPU Utilization (%) 0 | Debug Port 1025 | Memory (bytes) | Total Virtual Size 607.1M | Resident Set Size 3.8K | Connections | Active 0 | Total 0 | RPC Info | Total 0 | Notifies 0 | Timeouts 0 | Errors 0 | Octets In 0 | Octets Out 0 | Average Latency 0 | Max Latency 0 From this you can get quite a bit of information, including which TMM the process is assigned to, PID, CPU, memory and connection stats. If you wanted to know the total number of Node.js processes, that same command will show you every process and it could get quite long. You can use this quick one-liner from the bash shell (not TMSH) to count the Node.js processes - [root@test-ve:Active:Standalone] config # tmsh show ilx plugin | grep PID | wc -l 16 File System Read/Writes Since Node.js on BIG-IP is pretty much stock Node, file system read/writes are possible but not recommended. If you would like to know more about this and other properties of Node.js on BIG-IP, please see AskF5 Solution ArticleSOL16221101. Note:NPMs with symlinks will no longer work in 14.1.0+ due to SELinux changes In the next article in this series we will cover troubleshooting and debugging.
HTTP Response from ILXPlugin
Hi DevCentral community! I am currently trying to develop an iRule ILXPlugin, which checks a client HTTP request for certain properties. If some condiditions don't match I would like to send an HTTP response error message from the ILXPlugin to the client (without further forwarding to the backend). If everything is valid I want to forward it to the respective backend pool member. Unfortunately, the ILXTransaction does not provide any real examples (other than the description) on how to use the methods respond and replaceBody to achieve this use case. You can find my current code below, whereas I am trying to send a response with a custom html body and HTTP status code 404: 'use strict'; var f5 = require('f5-nodejs'); var plugin = new f5.ILXPlugin(); var options = new f5.ILXPluginOptions(); options.handleServerData = false; options.handleServerResponse = false; options.handleClientData = false; options.handleClientOpen = true; plugin.on('initialized', function () { console.log('INITIALIZED'); }); plugin.on('connect', function(flow) { var tmmID = flow.tmmId(); var clientSourceAddress = flow.client.remoteAddress; flow.client.on('requestComplete', function(request) { var destinationHost = request.params.headers.host; var requestMethod = request.params.method; // Example if(requestMethod === 'POST') { request.replaceBody(); request.respond(); // Send response with body: 'ERROR RESPONSE!' and Status Code '404' flow.client.end(); } var options = new f5.ILXLbOptions(); options.pool = '/Common/api_pool'; flow.lbSelect(options); flow.client.allow(); request.complete(); }); flow.client.on('error', function(errorText) { console.error('client error event: ' + errorText); }); flow.server.on('error', function(errorText) { console.error('server error event: ' + errorText); }); flow.on('error', function(errorText) { console.error('flow error event: ' + errorText); }); }); plugin.start(options); I would appreciate any suggestions. Thank you in advance.
The PingIntelligence and F5 BIG-IP Solution for Securing APIs
This article describes the PingIntelligence and F5 BIG-IP solution deployment for securing APIs. The integration identifies and automatically blocks cyber attacks on APIs, exposes active APIs, and provides detailed reporting on all API activity. Solution Overview PingIntelligence is deployed in a side-band configuration with F5 BIG-IP. A PingIntelligence policy is installed in F5 BIG-IP and passes API metadata to PingIntelligence for detailed API activity reporting and attack detection with optional client blocking. PingIntelligence software includes support for reporting and attack detection based on usernames captured from JSON Web Token (JWT). Following is a description of the traffic flow through F5 BIG-IP and PingIntelligence API Security Enforcer (ASE): The client sends an incoming request to F5 BIG-IP F5 BIG-IP makes an API call to send the request metadata to ASE ASE checks the request against a registered set of APIs and looks for the origin IP, cookie, OAuth2 token, or API key in PingIntelligence AI engine generated Blacklist. If all checks pass, ASE returns a 200-OK response to the F5 BIG-IP. If not, a different response code is sent to F5 BIG-IP. The request information is also logged by ASE and sent to the AI Engine for processing. F5 BIG-IP receives a 200-OK response from ASE, then it forwards the request to the backend server pool. A request is blocked only when ASE sends a 403 error code. The response from the back-end server poll is received by F5 BIG-IP. F5 BIG-IP makes a second API call to pass the response information to ASE which sends the information to the AI engine for processing. ASE receives the response information and sends a 200-OK to F5 BIG-IP. F5 BIG-IP sends the response received from the backend server to the client. Pre-requisites BIG-IP system must be running TMOS v13.1.0.8 or higher version. Sideband authentication is enabled on PingIntelligence for secure communication with the BIG-IP system. Download the PingIntelligence policy from the download site. Solution Deployment Step-1: Import and Configure PingIntelligence Policy Login to your F5 BIG-IP web UI and navigate to Local Traffic > iRules > LX Workspaces. On the LX Workspaces page, click on the Import button. Enter a Name and choose the PingIntelligence policy that you downloaded from the Ping Identity download site. Then, click on the Import button. This creates LX workspace Open the Workspace by clicking on the name. The policy is pre-loaded with an extension named oi_ext . Edit the ASE configuration by clicking on the ASEConfig.js file. It opens the PingIntelligence policy in the editor: Click on this link to understand various ASE variables. Step-2: Create LX Plugin Navigate to Local Traffic > iRules > LX Plugins. On the New Plugin page, click on the Create button to create a new plugin with the name pi_plugin. Select the workspace that you created earlier from the Workspace drop-down list and click on the Finished button. Step-3: Create a Backend Server Pool and Frontend Virtual Server (Optional) If you already created the virtual server, skip this step Create a Backend Server pool Navigate to Local Traffic > Pools > Pool List and click on the Create button. In the configuration page, configure the fields and add a new node to the pool. When done, click on the Finished button. This creates a backend server pool that is accessed from clients connecting to the frontend virtual server Create a Frontend Virtual Server Navigate to Local Traffic > Virtual Server > Virtual Server List and click on the Create button. Configure the virtual server details. At a minimum, configure the Destination Address, Client SSL Profile and Server SSL Profile When done, click on the Finished button. Under the Resource tab, add the backend server pool to the virtual server and click on the Update button. Step-4: Add PingIntelligence Policy The imported PingIntelligence policy must be tied to a virtual server. Add the PingIntelligence policy to the virtual server. Navigate to Local Traffic > Virtual Servers > Virtual Server List. Select the virtual server to which you want to add the PingIntelligence policy. Click on the Resources tab. In the iRules section, click on the Manage button. Choose the iRule under the pi_plugin that you want to attach to the virtual server. Move the pi_irule to the Enabled window and click on the Finished button. Once the solution is deployed, you can gain insights into user activity, attack information, blocked connections, forensic data, and much more from the PingIntelligence dashboard References Ping Intelligence for API Overview F5 BIG-IP PingIntelligence IntegrationGetting Started with iRules LX, Part 1: Introduction & Conceptual Overview
What is iRules LX? When BIG-IP TMOS 12.1.0 released a couple weeks ago, Danny wrote an intro piece to a new feature called iRules LX. iRules LX is the next generation of iRules that allows you to programmatically extend the functionality of BIG-IP with Node.js. In this article series, you will learn the basics of what iRules LX is, some use cases, how the components fit together, troubleshooting techniques, and some introductory code solutions. What is Node.js ? Node.js provides an environment for running JavaScript independent of a web browser. With Node.js you can write both clients and servers, but it is mostly used for writing servers. There are 2 main componets to Node.js - Google v8 JavaScript Engine Node.js starts with the same open source JavaScript engine used in the Google Chrome web browser. v8 has a JIT complier that optimizes JavaScript code to allow it to run incrediable fast. It also has an optimizing complier that, during run time, further optimizes the code to increase the performance up to 200 times faster. LibUV Asynchronous I/O Library A JavaScript engine by itself would not allow a full set of I/O operations that are needed for a server (you dont want your web browser accessing the file system of computer after all). LubUV gives Node.js the capability to have those I/O operations. Because v8 is a single threaded process, I/O (which have the most latency) would normally be blocking; that is, the CPU would be doing nothing while waiting from a response of the I/O operation. LibUV allows for the v8 to continue running JavaScript while waiting for the response, maximizing the use of CPU cycles and providing great concurrency. Why Node.js? Some might ask "Why was Node.js chosen?" While there is more than one reason, the biggest factor is that you can use JavaScript. JavaScript has been a staple of web development for almost 2 decades and is a common skillset amongst developers. Also, Node.js has a vibrant community that not only contributes to the Node.js core itself, but also a plethora of people writing libraries (called modules). As of May 2016 there are over 280,000 Node.js modules available in the Node Package Manager (NPM) registry, so chances are you can find a library for almost any use case. An NPM client is included with distributions of Node.js. For more information about the specifics of Node.js on BIG-IP, please refer to AskF5 Solution ArticleSOL16221101. iRules LX Conceptual Operation Unlike iRules which has a TCL interpreter running inside the Traffic Management Micro-kernel (TMM), Node.js is running as a stand alone processes outside of TMM. Having Node.js outside of TMM is an advantage in this case because it allow F5 to keep the Node.js version more current than if it was integrated into TMM. In order to get data from TMM to the Node.js processes, a Remote Procedural Call (RPC) is used so that TMM can send data to Node.js and tell it what "procedure" to run. The conceptual model of iRules LX looks something like this - To make this call to Node.js , we still need iRules TCL. We will cover the coding much more extensively in a few days, so dont get too wrapped up in understanding all the new commands just yet. Here is an example of TCL code for doing RPC to Node.js - when HTTP_REQUEST { set rpc_hdl [ILX::init my_plugin my_extension] set result [ILX::call $rpc_hdl my_method $arg1] if { $result eq "yes" } { # Do something } else { # Do something else } } In line 2 you can see with the ILX::init command that we establish an RPC handle to a specific Node.js plugin and extension (more about those tomorrow). Then in line 3 we use the ILX::call command with our handle to call the method my_method (our "remote procedure") in our Node.js process and we send it the data in variable arg1 . The result of the call will be stored in the variable result . With that we can then evaluate the result as we do in lines 5-9. Here we have an example of Node.js code using the iRules LX API we provide - ilx.addMethod('my_method', function(req, res){ var data = req.params()[0]; <…ADDITIONAL_PROCESSING…> res.reply('<some_reply>'); }); You can see the first argument my_method in addMethod is the name of our method that we called from TCL. The second argument is our callback function that will get executed when we call my_method . On line 2 we get the data we sent over from TCL and in lines 3-5 we perform whatever actions we wish with JavaScript. Then with line 6 we send our result back to TCL. Use Cases While TCL programming is still needed for performing many of the tasks, you can offload the most complicated portions to Node.js using any number of the available modules. Some of the things that are great candidates for this are - Sideband connections -Anyone that has used sideband in iRules TCL knows how difficult it is to implement a protocol from scratch, even if it is only part of the protocol. Simply by downloading the appropriate module with NPM, you can easily accomplish many of the following with ease - Database lookups - SQL, LDAP, Memcached, Redis HTTP and API calls - RESTful APIs, CAPTCHA, SSO protocols such as SAML, etc. Data parsing - JSON - Parser built natively into JavaScript XML Binary protocols In the next installment we will be covering iRules LX configuration and workflow.Getting Started with iRules LX, Part 3: Coding & Exception Handling
So far in this series, we have covered the conceptual overview, components, workflows, licensing and provisioning of iRules LX. In this article, we will actually get our hands wet with some iRules LX code! TCL Commands As mentioned in the first article, we still need iRules TCL to make a RPC to Node.js. iRules LX introduces three new iRules TCL commands - ILX::init - Creates a RPC handle to the plugin's extension. We will use the variable from this in our other commands. ILX::call - Does the RPC to the Node.js process to send data and receive the result (2 way communication). ILX::notify - Does an RPC to the Node.js process to send data only (1 way communication, no response). There is one caveat to the ILX::call and ILX::notify commands: you can only send 64KB of data. This includes overhead used to encapsulate the RPC so you will need to allow about 500B-1KB for the overhead. If you need to send more data you will need to create code that will send data over in chunks. TCL Coding Here is a TCL template for how we would use iRules LX. when HTTP_REQUEST { set ilx_handle [ILX::init "my_plgin" "ext1"] if {[catch {ILX::call $ilx_handle "my_method" $user $passwd} result]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" # Send user graceful error message, then exit event return } # If one value is returned, it becomes TCL string if {$result eq 'yes'} { # Some action } # If multiple values are returned, it becomes TCL list set x [ lindex $result 0] set y [ lindex $result 1] } So as not to overwhelm you, we'll break this down to bite size chunks. Taking line 2 from above, we will use ILX::init command to create our RPC handle to extension ext1 of our LX plugin my_plugin and store it in the variable ilx_handle . This is why the names of the extensions and plugin matter. set ilx_handle [ILX::init "my_plgin" "ext1"] Next, we can take the handle and do the RPC - if {[catch {ILX::call $ilx_handle "my_method" $user $passwd} result]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" # Send user graceful error message with whatever code, then exit event return } You can see on line 3, we make the ILX::call to the extension by using our RPC handle, specifying the method my_method (our "remote procedure") and sending it one or more args (in this case user and passwd which we got somewhere else). In this example we have wrapped ILX::call in a catch command because it can throw an exception if there is something wrong with the RPC (Note: catch should actually be used on any command that could throw an exception, b64decode for example). This allows us to gracefully handle errors and respond back to the client in a more controlled manner. If ILX::call is successful, then the return of catch will be a 0 and the result of the command will be stored in the variable result . If ILX::call fails, then the return of catch will be a 1 and the variable result will contain the error message. This will cause the code in the if block to execute which would be our error handling. Assuming everything went well, we could now start working with data in the variable result . If we returned a single value from our RPC, then we could process this data as a string like so - # If one value is returned, it becomes TCL string if {$result eq 'yes'} { # Some action } But if we return multiple values from our RPC, these would be in TCL list format (we will talk more about how to return multiple values in the Node.js coding section). You could use lindex or any suitable TCL list command on the variable result - # If multiple values are returned, it becomes TCL list set x [ lindex $result 0] set y [ lindex $result 1] Node.js Coding On the Node.js side, we would write code in our index.js file of the extension in the workspace. A code template will load when the extension is created to give you a starting point, so you dont have to write it from scratch. To use Node.js in iRules LX, we provide an API for receiveing and sending data from the TCL RPC. Here is an example - var f5 = require('f5-nodejs'); var ilx = new f5.ILXServer(); ilx.addMethod('my_method', function (req, res) { // req.params() function is used to get values from TCL. Returns JS Array var user = req.params()[0]; var passwd = req.params()[1]; <DO SOMETHING HERE> res.reply('<value>'); // Return one value as string or number res.reply(['value1','value2','value3']); // Return multiple values with an Array }); ilx.listen(); Now we will run through this step-by-step. On line 1, you see we import the f5-nodejs module which provides our API to the ILX server. var f5 = require('f5-nodejs'); On line 3 we instantiate a new instance of the ILXServer class and store the object in the variable ilx . var ilx = new f5.ILXServer(); On line 5, we have our addMethod method which stores methods (our remote procedures) in our ILX server. ilx.addMethod('my_method', function (req, res) { // req.params() function is used to get values from TCL. Returns JS Array var user = req.params()[0]; var passwd = req.params()[1]; <DO SOMETHING HERE> res.reply('<value>'); // Return one value as string or number res.reply(['value1','value2','value3']); // Return multiple values with an Array }); This is where we would write our custom Node.js code for our use case. This method takes 2 arguments - Method name - This would be the name that we call from TCL. If you remember from our TCL code above in line 3 we call the method my_method . This matches the name we put here. Callback function - This is the function that will get executed when we make the RPC to this method. This function gets 2 arguments, the req and res objects which follow standard Node.js conventions for the request and response objects. In order to get our data that we sent from TCL, we must call the req.param method. This will return an array and the number of elements in that array will match the number of arguments we sent in ILX::call . In our TCL example on line 3, we sent the variables user and passwd which we got somewhere else. That means that the array from req.params will have 2 elements, which we assign to variables in lines 7-8. Now that we have our data, we can use any valid JavaScript to process it. Once we have a result and want to return something to TCL, we would use the res.reply method. We have both ways of using res.reply shown on lines 12-13, but you would only use one of these depending on how many values you wish to return. On line 12, you would put a string or number as the argument for res.reply if you wanted to return a single value. If we wished to return multiple values, then we would use an array with strings or numbers. These are the only valid data types for res.reply . We mentioned in the TCL result that we could get one value that would be a string or multiple values that would be a TCL list. The argument type you use in res.reply is how you would determine that. Then on line 16 we start our ILX server so that it will be ready to listen to RPC. ilx.listen(); That was quick a overview of the F5 API for Node.js in iRules LX. It is important to note that F5 will only support using Node.js in iRules LX within the provided API. A Real Use Case Now we can take what we just learned and actually do something useful. In our example, we will take POST data from a standard HTML form and convert it to JSON.In TCL we would intercept the data, send it to Node.js to transform it to JSON, then return it to TCL to replace the POST data with the JSON and change the Content-Type header - when HTTP_REQUEST { # Collect POST data if { [HTTP::method] eq "POST" }{ set cl [HTTP::header "Content-Length"] HTTP::collect $cl } } when HTTP_REQUEST_DATA { # Send data to Node.js set handle [ILX::init "json_plugin" "json_ext"] if {[catch {ILX::call $handle "post_transform" [HTTP::payload]} json]} { log local0.error "Client - [IP::client_addr], ILX failure: $json" HTTP::respond 400 content "<html>Some error page to client</html>" return } # Replace Content-Type header and POST payload HTTP::header replace "Content-Type" "application/json" HTTP::payload replace 0 $cl $json } In Node.js, would only need to load the built in module querystring to parse the post data and then JSON.stringify to turn it into JSON. 'use strict' var f5 = require('f5-nodejs'); var qs = require('querystring'); var ilx = new f5.ILXServer(); ilx.addMethod('post_transform', function (req, res) { // Get POST data from TCL and parse query into a JS object var postData = qs.parse(req.params()[0]); // Turn postData into JSON and return to TCL res.reply(JSON.stringify(postData)); }); ilx.listen(); Note: Keep in mind that this is only a basic example. This would not handle a POST that used 100 continue or mutlipart POSTs. Exception Handling iRules TCL is very forgiving when there is an unhanded exception. When you run into an unhandled runtime exception (such as an invalid base64 string you tried to decode), you only reset that connection. However, Node.js (like most other programming languages) will crash if you have an unhandled runtime exception, so you will need to put some guard rails in your code to avoid this. Lets say for example you are doing JSON.parse of some JSON you get from the client. Without proper exception handling any client could crash your Node.js process by sending invalid JSON. In iRules LX if a Node.js process crashes 5 times in 60 seconds, BIG-IP will not attempt to restart it which opens up a DoS attack vector on your application (max restarts is user configurable, but good code eliminates the need to change it). You would have to manually restart it via the Web UI or TMSH. In order to catch errors in JavaScript, you would use the try/catch statement. There is one caveat to this: code inside a try/catch statement is not optimized by the v8 complier and will cause a significant decrease in performance. Therefore, we should keep our code in try/catch to a minimum by putting only the functions that throw exceptions in the statement. Usually, any function that will take user provided input can throw. Note: The subject of code optimization with v8 is quite extensive so we will only talk about this one point. There are many blog articles about v8 optimization written by people much smarter than me. Use your favorite search engine with the keywords v8 optimization to find them. Here is an example of try/catch with JSON.parse - ilx.addMethod('my_function', function (req, res) { try { var myJson = JSON.parse(req.params()[0]) // This function can throw } catch (e) { // Log message and stop processing function console.error('Error with JSON parse:', e.message); console.error('Stack trace:', e.stack); return; } // All my other code is outside try/catch var result = ('myProperty' in myJson) ? true : false; res.reply(result); }); We can also work around the optimization caveat by hoisting a custom function outside try/catch and calling it inside the statement - ilx.addMethod('my_function', function (req, res) { try { var answer = someFunction(req.params()[0]) // Call function from here that is defined on line 16 } catch (e) { // Log message an stop processing function console.error('Error with someFunction:', e.message); console.error('Stack trace:', e.stack); return; } // All my other code is outside try/catch var result = (answer === 'hello') ? true : false; res.reply(result); }); function someFuntion (arg) { // Some code in here that can throw return result } RPC Status Return Value In our examples above, we simply stopped the function call if we had an error but never let TCL know that we encountered a problem. TCL would not know there was a problem until the ILX::call command reached it's timeout value (3 seconds by default). The client connection would be held open until it reached the timeout and then reset. While it is not required, it would be a good idea for TCL to get a return value on the status of the RPC immediately. The specifics of this is pretty open to any method you can think of but we will give an example here. One way we can accomplish this is by the return of multiple values from Node.js. Our first value could be some type of RPC status value (say an RPC error value) and the rest of the value(s) could be our result from the RPC. It is quite common in programming that make an error value would be 0 if everything was okay but would be a positive integer to indicate a specific error code. Here in this example, we will demonstate that concept. The code will verify that the property myProperty is presentin JSON data and put it's value into a header or send a 400 response back to the client if not. THe Node.js code - ilx.addMethod('check_value', function (req, res) { try { var myJson = JSON.parse(req.params()[0]) // This function can throw } catch (e) { res.reply(1); //<---------------------------------- The RPC error value is 1 indicating invalid JSON return; } if ('myProperty' in myJson){ // The myProperty property was present in the JSON data, evaluate its value var result = (myJson.myProperty === 'hello') ? true : false; res.reply([0, result]); //<-------------------------- The RPC error value is 0 indicating success } else { res.reply(2); //<-------------- The RPC error value is 2 indicating myProperty was not present } }); In the code above the first value we return to TCL is our RPC error code. We have defined 3 possible values for this - 0 - RPC success 1 - Invalid JSON 2 - Property "myProperty" not present in JSON One our TCL side we would need to add logic to handle this value - when HTTP_REQUEST { # Collect POST data if { [HTTP::method] eq "POST" }{ set cl [HTTP::header "Content-Length"] HTTP::collect $cl } } when HTTP_REQUEST_DATA { # Send data to Node.js set handle [ILX::init "json_plugin" "json_checker"] if {[catch [ILX::call $handle "check_value" [HTTP::payload]] json]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" HTTP::respond 400 content "<html>Some error page to client</html>" return } # Check the RPC error value if {[lindex $json 0] > 0} { # RPC error value was not 0, there is a problem switch [lindex $json 0] { 1 { set error_msg "Invalid JSON"} 2 { set error_msg "myProperty property not present"} } HTTP::respond 400 content "<html>The following error occured: $error_msg</html>" } else { # If JSON was okay, insert header with myProperty value HTTP::header insert "X-myproperty" [lindex $json 1] } } As you can see on line 19, we check the value of the first element in our TCL list. If it is greater than 0 then we know we have a problem. We move on down further to line 20 to determine the problem and set a variable that will become part of our error message back to the client. Note:Keep in mind that this is only a basic example. This would not handle a POST that used 100 continue or mutlipart POSTs. In the next article in this series, we will cover how to install a module with NPM and some best practices.Getting Started with iRules LX, Part 5: Troubleshooting
When you start writing code, you will eventually run into some issue and need to troubleshoot. We have a few tools available for you. Logging One of the most common tools use for troubleshooting is logging. STDOUT and STDERR for the Node.js processes end up in the file /var/log/ltm . By temporarily inserting functions such as console.log , console.error , etc. you can easily check the state of different objects in your code at run time. There is one caveat to logging in iRules LX; logging by most of most processes on BIG-IP is throttled by default. Basically, if 5 messages from a process come within 1 second, all other messages with be truncated. For Node.js, every line is a separate message so something like a stack trace will get automatically truncated. To remove this limit we can change the DB variable log.sdmd.level to the value debug in TMSH as such - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # modify sys db log.sdmd.level value debug SDMD is the process that manages the Node.js instances. This turns on verbose logging for SDMD and all the Node.js processes so you will want to turn this back to info when you are finished troubleshooting. Once that is done you can start adding logging statements to your code. Here, I added a line to tell me when the process starts (or restarts) - var f5 = require('f5-nodejs'); console.log('Node.js process starting.'); var ilx = new f5.ILXServer(); The logs would then show me this - Jun 7 13:26:08 test-ve info sdmd[12187]: 018e0017:6: pid[21194] plugin[/Common/test_pl1.ext1] Node.js process starting. Node Inspector BIG-IP comes packaged with Node Inspector for advanced debugging. We won't cover how to use Node Inspector because most developers skilled in Node.js already know how to use it and there are already many online tutorials for those that dont. Note: We highly recommend that you do not use Node Inspector in a production environment. Debuggers lock the execution of code at break points pausing the entire application, which does not work well for a realtime network device. Ideally debugging should only be done on a dedicated BIG-IP (such as a lab VE instance) in an isolated development environment. Turn on Debugging The first thing we want to do is to put our extension into single concurrency mode so that we are only working with one Node.js process while debugging. Also, while we are doing that we can put the extension into debugging mode. This can be done from TMSH like so - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # list ilx plugin DC_Plugin ilx plugin DC_Plugin { disk-space 156 extensions { dc_extension { } dc_extension2 { } } from-workspace DevCentralRocks staged-directory /var/ilx/workspaces/Common/DevCentralRocks } root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # modify ilx plugin DC_Plugin extensions { dc_extension { concurrency-mode single command-options add { --debug } }} root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # list ilx plugin DC_Plugin ilx plugin DC_Plugin { disk-space 156 extensions { dc_extension { command-options { --debug } concurrency-mode single } dc_extension2 { } } from-workspace DevCentralRocks staged-directory /var/ilx/workspaces/Common/DevCentralRocks } root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # restart ilx plugin DC_Plugin immediate Increase ILX::call Timeout Also, if you recall from part 3 of this series, the TCL command ILX::call has a 3 second timeout by default that if there is no response to the RPC, the command throws an exception. We don't need to worry about this for the command ILX::notify because it does not expect a response from the RPC. While we have the debugger paused on a break point you are pretty much guaranteed to hit this timeout, so we need to increase it to allow for our whole method to finish execution. To do this, you will need to add arguments to the ILX::call highlighted in red - ILX::call $ilx_handle -timeout big_number_in_msec method arg1 This number should be big enough to walk through the entire method call while debugging. If it will take you 2 minutes, then the value should be 120000. Once you update this make sure to reload the workspace. Launching Node Inspector Once you have put Node.js into debugging mode and increased the timeout, you now you can fire up Node Inspector. We will need to find the debugging port that our Node.js process has been assigned via TMSH - root@(eflores-ve3)(cfg-sync Standalone)(Active)(/Common)(tmos) # show ilx plugin DC_Plugin <----------------snipped -----------------> --------------------------------- | Extension Process: dc_extension --------------------------------- | Status running | PID 25975 | TMM 0 | CPU Utilization (%) 0 | Debug Port 1033 | Memory (bytes) | Total Virtual Size 1.2G | Resident Set Size 5.8K Now that we have the debugging port we can put that and the IP of our management interface into the Node Inspector arguments - [root@test-ve:Active:Standalone] config # /usr/lib/node_modules/.bin/node-inspector --debug-port 1033 --web-host 192.168.0.245 --no-inject Node Inspector v0.8.1 Visit http://192.168.0.245:8080/debug?port=1033 to start debugging. We simply need to take the URL given to us and put it in our web browser. Then we can start debugging away! Conclusion We hope you have found the Getting Started with iRules LX series very informative and look forward to hearing about how iRules LX has helped you solved problems. If you need any help with using iRules LX, please dont hesitate to ask a question on DevCentral.
irule for URI based redirection and to the pool..
Hi, I'm looking for a short irule that helps in two ways. URI redirection Host based pool selection I currently have two irules, I'm looking for one combined irule that serves the purpose. Host to pool selection: when HTTP_REQUEST { switch -glob [ string tolower [HTTP::host]] { "eng.page3.com" { pool page3_https_pool } "eng.devpage3.pega.com" { pool devpage3_https_pool } default { HTTP::respond 404 noserver } } URI based redirection: when HTTP_REQUEST { if {[HTTP::uri] starts_with "/ClientService/BI-Integration/"} { switch -glob [string tolower [HTTP::host]] { "eng.page3.com" {HTTP::redirect "/clientapp[HTTP::uri]"} } } } Can someone help me with a irule that gives me the combined features of both. I don't like to use two separate irules on the same VIP, so checking if there is any better option.
Dynamic DNS registration of APM SSL VPN Client to AWS route53
Problem this snippet solves: This iRules LX code dynamically registers a clients obtained IP from APM when they initiate a VPN tunnel to an AWS route 53 domain. How to use this snippet: Import attached tgz(zip will need to be re-packaged to tgz) to a new iRules LX workspace, create an iRule LX plugin, and attached the editRecord iRule to your virtual server. You can also manually create a new workspace and add the iRules LX iRule and extension with the code below. You will also have to install the aws-sdk module using npm. Eric Flores has a good write-up on how to manage npm, https://devcentral.f5.com/articles/getting-started-with-irules-lx-part-4-npm-best-practices-20426. iRule Code: when CLIENT_ACCEPTED { ACCESS::restrict_irule_events disable } when HTTP_REQUEST { if { [ACCESS::policy result] eq "allow" && [HTTP::uri] starts_with "/myvpn?sess="} { after 5000 { set apmSessionId [ACCESS::session data get session.user.sessionid] set apmProfileName [ACCESS::session data get session.access.profile] set apmPart [ACCESS::session data get session.partition_id] set DNSName "<your zone with trailing dot>" set action "UPSERT" set name "<your hostname>" if {not ($name ends_with ".[string trimright ${DNSName} {.}]")}{ log -noname local1.err "${apmProfileName}:${apmPart}:${apmSessionId}: proper hostname was not provided ($name): 01490000: A Record will not be updated for this session." event disable all return } set TTL 1200 set ip [ACCESS::session data get "session.assigned.clientip"] if {[catch {IP::addr $ip mask 255.255.255.255}]}{ log -noname local1.err "${apmProfileName}:${apmPart}:${apmSessionId}: proper IP addressed was not assigned ($ip): 01490000: A Record will not be updated for this session." event disable all return } set RPC_HANDLE [ILX::init route53] if {[catch {set rpc_response [ILX::call $RPC_HANDLE route53_nodejs $DNSName $action $name $TTL $ip]}]}{ set rpc_response "Check LTM log for execution errors from sdmd. A Record may not have been updated for this session." } log -noname local1. "${apmProfileName}:${apmPart}:${apmSessionId}: Client connected: 01490000: Params sent = $DNSName $action $name $TTL $ip\r\nRPC response = $rpc_response" } } } when ACCESS_SESSION_CLOSED { set apmSessionId [ACCESS::session data get session.user.sessionid] set apmProfileName [ACCESS::session data get session.access.profile] set apmPart [ACCESS::session data get session.partition_id] set DNSName "<your zone with trailing dot>" set action "DELETE" set name "<your hostname>" if {not ($name ends_with ".[string trimright ${DNSName} {.}]")}{ log -noname local1.err "${apmProfileName}:${apmPart}:${apmSessionId}: proper hostname was not provided ($name): 01490000: A Record will not be removed for this session." event disable all return } set TTL 1200 set ip [ACCESS::session data get "session.assigned.clientip"] if {[catch {IP::addr $ip mask 255.255.255.255}]}{ log -noname local1.err "${apmProfileName}:${apmPart}:${apmSessionId}: proper IP addressed was not assigned ($ip): 01490000: A Record will not be removed for this session." event disable all return } set RPC_HANDLE [ILX::init route53] if {[catch {set rpc_response [ILX::call $RPC_HANDLE route53_nodejs $DNSName $action $name $TTL $ip]}]}{ set rpc_response "Check LTM log for execution errors from sdmd. A Record may not have been removed for this session." } log -noname local1. "${apmProfileName}:${apmPart}:${apmSessionId}: Client disconnected: 01490000: Params sent = $DNSName $action $name $TTL $ip\r\nRPC response = $rpc_response" } node.js code: var AWS = require('aws-sdk'); var f5 = require('f5-nodejs'); /* AWS Config */ //AWS.config.update({accessKeyId: "<your access key>", secretAccessKey: "<your secret>"}); /* get the Route53 library */ var route53 = new AWS.Route53(); /* Create a new rpc server for listening to TCL iRule calls. */ var ilx = new f5.ILXServer(); /* Start listening for ILX::call and ILX::notify events. */ ilx.listen(); /* Add a method and expect DNSName, action, name, TTL, and ip parameters and reply with response */ ilx.addMethod('route53_nodejs', function(req, response) { var DNSName = req.params()[0]; var params = { DNSName: DNSName }; var action = req.params()[1]; var name = req.params()[2]; var TTL = req.params()[3]; var ip = req.params()[4]; route53.listHostedZonesByName(params, function(err,data) { if (err) { //console.log(err, err.stack); response.reply(err.toString()); } else if (data.HostedZones[0].Name !== params.DNSName) { response.reply(params.DNSName + " is not a zone defined in Route53"); } else { var zoneId = data.HostedZones[0].Id; var recParams = { "HostedZoneId": zoneId, "ChangeBatch": { "Changes": [ { "Action": action, "ResourceRecordSet": { "Name": name, "Type": "A", "TTL": TTL, "ResourceRecords": [ { "Value": ip } ] } } ] } }; /* edit records using aws sdk */ route53.changeResourceRecordSets(recParams, function(err,data) { if (err) {response.reply(err.toString());} else if (data.ChangeInfo.Status === "PENDING") {response.reply("Record is being updated");} else {response.reply(data);} }); } }); }); This is also on my github, https://github.com/bepsoccer/iRulesLX Code : 75409 Tested this on version: 12.1iRules LX Logger Class
In version 13.1, the ILXLogger class was added to expand the ability to log from iRules LX. The API detailshave been added to to the wikibut in this article, I’ll add context and an exampleto boot. Let’s get to it! The first step is to create a log publisher. For test purposes any destination for the publisher will do, so I’ll just choose local syslog for reach of the three publishers I’ll create here for the three tiers of logging you can do within iRules LX: the global level, the plugin level, and the extension level. In the GUI, you can apply the plugin and extension publishers as shown in the two images below: The log publishers can also be created via tmsh with the sys log-config publisher command. One you have a log publisher defined, you can also use tmsh to apply the publishers to any/all levels of your iRules LX environments. # Global tmsh modify ilx global-settings log-publisher ilx_global # Plugin tmsh modify ilx plugin a_plugin log-publisher ilx_plugin # Extension tmsh modify ilx plugin b_plugin extensions { b_extension { log-publisher ilx_extension } } This works similar to profile parent/child inheritance, with some caveats. Scenario #1 The global publisher is ilx_global The plugin publisher is ilx_plugin The extension publisher is ilx_extension Result: Calling logger.send() will send to the ilx_extension publisher. Scenario #2 The global publisher is ilx_global The plugin publisher is ilx_plugin The extension publisher is none Result: Calling logger.send() will send to the ilx_plugin publisher. Scenario #3 The global publisher is ilx_global The plugin publisher is none The extension publisher is ilx_extension Result: Calling logger.send() will send to the ilx_extension publisher. Scenario #4 The global publisher is ilx_global The plugin publisher is none The extension publisher is none Result: Calling logger.send() will send to the ilx_global publisher. Scenario #5 The global publisher is none The plugin publisher is none The extension publisher is none Result: Calling logger.send() will cause an error as there is no configured publisher. Note: You can alternatively use console.log() to send log data. If a global publisher is configured it will log there and if not it will log to /var/log/ltm by default. Example This is a bare bones example from Q&A posted by F5’s own Satoshi Toyosawa. I created an iRules LX Workspace (my workspace), an iRule (ilxtest), a plugin (ILXLOggerTestRpcPlugin), and an extension (ILXLoggerTestRpcExt). Configuration details for each: The Workspace ilx workspace myworkspace { extensions { ILXLoggerTestRpcExt { files { index.js { } node_modules { } package.json { } } } } node-version 6.9.1 partition none rules { ilxtest { } } staged-directory /var/ilx/workspaces/Common/myworkspace version 14.0.0 } The iRule when HTTP_REQUEST { log local0. "I WANT THE TRUTH!" set RPC_HANDLE [ILX::init ILXLoggerTestRpcPlugin "ILXLoggerTestRpcExt"] ILX::call $RPC_HANDLE func } The Plugin ilx plugin ILXLoggerTestRpcPlugin { extensions { ILXLoggerTestRpcExt { log-publisher ilx_extension } } from-workspace myworkspace log-publisher ilx_plugin node-version 6.9.1 staged-directory /var/ilx/workspaces/Common/myworkspace } The Extension (all defaults except the updated index.js contents shown below) var f5 = require('f5-nodejs'); var ilx = new f5.ILXServer(); var logger = new f5.ILXLogger(); ilx.addMethod('func', function(req, res) { logger.send('YOU CAN\'T HANDLE THE TRUTH!'); res.reply('done'); }); ilx.listen(); Applying this iRules LX package against one of my test virtual servers with an HTTP profile, I anticipate Lt. Daniel Kaffee and Col. Nathan R. Jessup (From A Few Good Men) going at it in the log file: Aug 30 19:33:20 ltm3 info sdmd[25253]: 018e0017:6: pid[35416] plugin[/Common/ILXLoggerTestRpcPlugin.ILXLoggerTestRpcExt] YOU CAN'T HANDLE THE TRUTH! Aug 30 19:33:20 ltm3 info tmm1[29992]: Rule /Common/ILXLoggerTestRpcPlugin/ilxtest : I WANT THE TRUTH! Interesting that the node process dumps the log statement faster than Tcl, though that is not consistent and not guaranteed, as another run shows a different (and more appropriate according to the movie dialog) order: Aug 30 22:47:35 ltm3 info tmm1[29992]: Rule /Common/ILXLoggerTestRpcPlugin/ilxtest : I WANT THE TRUTH! Aug 30 22:47:35 ltm3 info sdmd[25253]: 018e0017:6: pid[35416] plugin[/Common/ILXLoggerTestRpcPlugin.ILXLoggerTestRpcExt] YOU CAN'T HANDLE THE TRUTH! This is by no means a thorough detour into the possibilities of your log publisher options with iRules LX on version 13.1+, but hopefully this gets you started with greater logging flexibility and if you are new to iRules LX, some sample code to play with. Happy coding!
Send OTP via sendgrid(email) API
Problem this snippet solves: This is an example of how to send an OTP generated by the Big-IP via email using the sendgrid api. Preparations Register a free Sendgrid account: https://sendgrid.com/ Generate an API key: https://app.sendgrid.com/settings/api_keys Create an iruleLX workspace or just import it from my github (pay attention to naming conventions): https://github.com/alexnimo/F5/blob/master/iRules_LX/sendgrid/sendgrid_ilx.tgz Configure AD resource Preface Most of the code and examples in this tutorial are just cut and paste from different places with slight modifications. I've decided to try it out after trying the twilio tutorial by Niels van Sluis: https://devcentral.f5.com/codeshare/send-an-one-time-password-otp-via-the-twilio-sms-gateway-1132. Afterward, I wanted to better understand the irule LX architecture and usage, so I watched this great tutorial by Artiom: https://www.youtube.com/watch?v=7yRP2fPCxIs&index=1&list=PLRL802iBI7n-6y8_eNawyeUZ8E0YwWro8 And this is my first attempt to write a tutorial... Configurations In case you're configuring from scratch, you need to install sendgrid plugin, otherwise its already included in the tgz file. To install the plugin: cd /var/ilx/workspaces/Common/ilx_sendgrid/extensions/sendgrid/node_modules npm install --save @sendgrid/mail The index.js includes 2 options to send an email. If you want to use the first option. Simple email, then uncomment before //Use this method to send regular email and comment before //Use this method to send email with dynamic template There are no additional configurations to be done. If you want to send an email with a dynamic template(customized), the second(default) option then first you need to create the template via the sendgrid UI: https://sendgrid.com/dynamic_templates You can make different customizations to your email. I've just used an example template which can be found on their github: https://github.com/sendgrid/email-templates/blob/master/paste-templates/password-reset.html and played with it just to understand how it works. Here is an ugly modified template with the variables that will be presented in the email: https://github.com/alexnimo/F5/blob/master/iRules_LX/sendgrid/otp_template.html More information can be found here: https://sendgrid.com/docs/ui/sending-email/how-to-send-an-email-with-dynamic-transactional-templates/ https://sendgrid.com/docs/ui/sending-email/using-handlebars/ I've generated and used the code from the documentation: https://dynamic-templates.api-docs.io/3.0/mail-send-with-dynamic-transactional-templates/v3-mail-send If the default option is used(dynamic templates) then you must include the template id in the variable template_id Hope I haven't forgot anything.... How to use this snippet: Code variables: var api_key = The api key generated in sendgrid Everything else can be customized. template_id = Template ID created in sendgrid. Code : 'use strict'; // Import the f5-nodejs module. var f5 = require('f5-nodejs'); // Import the sendgrid module. const sgMail = require('@sendgrid/mail'); //Set API key var api_key = "PASTE API KEY HERE" sgMail.setApiKey(api_key); // Create a new rpc server for listening to TCL iRule calls. var ilx = new f5.ILXServer(); //Construct email ilx.addMethod('ilx_sendmail', function(tclArgs, funcResp) { console.log("Method Invoked, Args:", tclArgs.params()); const userEmail = tclArgs.params()[2]; var emailText = 'Dear' + tclArgs.params()[0] + ' Your OTP is: ' + tclArgs.params()[1]; /* //Use this method to send regular email const msg = { to: userEmail, from: 'OTP_Admin@otptest.com', subject: 'Please See your OTP', text: emailText, // html: 'and easy to do anywhere, even with Node.js', }; //Send email sgMail.send(msg, (error, result) => { if (error) { console.log("Error sending email, the issue is:" + error) funcResp.reply(error); } else { return result funcResp.reply("email sent"); } }); */ //Use this method to send email with dynamic template var http = require("https"); var options = { "method": "POST", "hostname": "api.sendgrid.com", "port": null, "path": "/v3/mail/send", "headers": { "authorization": "Bearer " + api_key, "content-type": "application/json" } }; var req = http.request(options, function (res) { var chunks = []; res.on("data", function (chunk) { chunks.push(chunk); }); res.on("end", function () { var body = Buffer.concat(chunks); console.log(body.toString()); }); }); req.write(JSON.stringify({ personalizations: [ { to: [ { email: userEmail , name: tclArgs.params()[0] } ], dynamic_template_data: { user: tclArgs.params()[0], otp: tclArgs.params()[1], partysize: 4, english: true } } ], from: { email: 'OTP_Admin@otptest.com', name: 'OTP Admin' }, reply_to: { email: 'OTP_Admin@otptest.com', name: 'OTP Admin' }, template_id: 'PASTE TEMPLATE ID HERE' })); req.end(); funcResp.reply("email sent"); }); ilx.listen(); Tested this on version: 13.0
