Introducing iRules LX
iRules is a powerful scripting language that allows you to control network traffic in real time that can route, redirect, modify, drop, log or do just about anything else with network traffic passing through a BIG-IP proxy. iRules enables network programmability to consolidate functions across applications and services. iRules LX: The Next Evolution of Network Programmability iRules LX is the next stage of evolution for network programmability that brings Node.js language support to the BIG-IP withversion 12.1. Node.js allows JavaScript developers access to over 250,000 npm packages that makes code easier to write and maintain. Development teams can access and work on code with the new iRules LX Workspace environment and the new plug-in available for the Eclipse IDE and can be used for continuous integration builds. Extend with Node.js Support iRules Language eXtensions (LX) enables node.js capabilitieson the BIG-IP platform. By using iRules LX (Node.js) extensions you can program iRules LX using javascript to control network traffic in real time. There are only three simple Tcl commands that you need to extend iRules to Node.js ILX::init – create Tcl handle, bind extension ILX::call – send/receive data to/from extension method ILX::notify – send data one-way to method, no response There is only one Node.js command to know: ILX.addMethod – create method, for use by above Share and Reuse Code iRules LX makes it easy for JavaScript developers to share and reuse Node.js code, and it makes it easy to update the code. These bits of reusable code are called packages, or sometimes modules. This makes it possible for you to compose complex business logic in node.js without sophisticated developers who know Tcl in-depth. There are over 250,000 npm packages that makes code easier to write and maintain. iRules LX Workspaces LX Workspaces provides a portable development package environment with NPM-like directory structure that acts as the source for creation and modification of plugins and node.js extensions. Workspace LX is developer-friendly, with Git-like staging and publishing that simplifies export and import to keep your files in sync for continuous integration and development. iRules IDE Integration / Eclipse Plugin Available soon, F5 is delivering a free downloadable plugin to enable our customers to develop iRules & iRules LX scripts via their own Eclipse IDE. The pluginprovides modern IDE editing capabilities including syntax highlighting & validation, code completion, and code formatting for both Tcl and Javascript.This plugin will enable authenticated push and pull of iRules code files to and from the BIG-IP platform.This simplifies development workflows by enabling our customers to develop iRules scripts in a familiar development environment. iRules LX Example: Use Node.js to Make an Off Box MySQL Lookup This iRules example showsa connection to MySQL database via Node.js. This example uses Tcl iRulesand JavaScript code to make a MySQL call. The iRules prompts the user for a basic auth username and password, then we lookup the username in a MySQL database table and return the user's groups. MySQL iRule (Tcl) Example: ################################### # mysql_irulelx # 2015-05-17 - Aaron Hooley - First draft example showing iRulesLX querying MySQL database ################################### when RULE_INIT { # Enable logging to /var/log/ltm? # 0=none, 1=error, 2=verbose set static::mysql_debug 2 } when HTTP_REQUEST { if {$static::mysql_debug >= 2}{log local0. "New HTTP request. \[HTTP::username\]=[HTTP::username]"} # If the client does not supply an HTTP basic auth username, prompt for one. # Else send the HTTP username to node.js and respond to client with result if {[HTTP::username] eq ""}{ HTTP::respond 401 content "Username was not included in your request.\nSend an HTTP basic auth username to test" WWW-Authenticate {Basic realm="iRulesLX example server"} if {$static::mysql_debug >= 2}{log local0. "No basic auth username was supplied. Sending 401 to prompt client for username"} } else { # Do some basic validation of the HTTP basic auth username # First fully URI decode it using https://devcentral.f5.com/s/articles?sid=523 set tmp_username [HTTP::username] set username [URI::decode $tmp_username] # Repeat decoding until the decoded version equals the previous value while { $username ne $tmp_username } { set tmp_username $username set username [URI::decode $tmp_username] } # Check that username only contains valid characters using https://devcentral.f5.com/s/articles?sid=726 if {$username ne [set invalid_chars [scan $username {%[-a-zA-Z0-9_]}]]}{ HTTP::respond 401 content "\nA valid username was not included in your request.\nSend an HTTP basic auth username to test\n" WWW-Authenticate {Basic realm="iRulesLX example server"} if {$static::mysql_debug >= 1}{log local0. "Invalid characters in $username. First invalid character was [string range $username [string length $invalid_chars] [string length $invalid_chars]]"} # Exit this iRule event return } # Client supplied a valid username, so initialize the iRulesLX extension set RPC_HANDLE [ILX::init mysql_extension] if {$static::mysql_debug >= 2}{log local0. "\$RPC_HANDLE: $RPC_HANDLE"} # Make the call and save the iRulesLX response # Pass the username and debug level as parameters #set rpc_response [ILX::call $RPC_HANDLE myql_nodejs $username $static::mysql_debug] set rpc_response [ILX::call $RPC_HANDLE myql_nodejs $username] if {$static::mysql_debug >= 2}{log local0. "\$rpc_response: $rpc_response"} # The iRulesLX rule will return -1 if the query succeeded but no matching username was found if {$rpc_response == -1}{ HTTP::respond 401 content "\nYour username was not found in MySQL.\nSend an HTTP basic auth username to test\n" WWW-Authenticate {Basic realm="iRulesLX example server"} if {$static::mysql_debug >= 1}{log local0. "Username was not found in MySQL"} } elseif {$rpc_response eq ""}{ HTTP::respond 401 content "\nDatabase connection failed.\nPlease try again\n" WWW-Authenticate {Basic realm="iRulesLX example server"} if {$static::mysql_debug >= 1}{log local0. "MySQL query failed"} } else { # Send an HTTP 200 response with the groups retrieved from the iRulesLX plugin HTTP::respond 200 content "\nGroup(s) for '$username' are '$rpc_response'\n" if {$static::mysql_debug >= 1}{log local0. "Looked up \$username=$username and matched group(s): $rpc_response"} } } } MySQL iRule LX (Node.js) Example /* ###################################### */ /* index.js counterpart for mysql_irulelx */ /* Log debug to /var/log/ltm? 0=none, 1=errors only, 2=verbose */ var debug = 2; if (debug >= 2) {console.log('Running extension1 index.js');} /* Import the f5-nodejs module. */ var f5 = require('f5-nodejs'); /* Create a new rpc server for listening to TCL iRule calls. */ var ilx = new f5.ILXServer(); /* Start listening for ILX::call and ILX::notify events. */ ilx.listen(); /* Add a method and expect a username parameter and reply with response */ ilx.addMethod('myql_nodejs', function(username, response) { if (debug >= 1) {console.log('my_nodejs' + ' ' + typeof(username.params()) + ' = ' + username.params());} var mysql = require('mysql'); var connection = mysql.createConnection({ host : '10.0.0.110', user : 'bigip', password : 'bigip' }); // Connect to the MySQL server connection.connect(function(err) { if (err) { if (debug >= 1) {console.error('Error connecting to MySQL: ' + err.stack);} return; } if (debug >= 2) {console.log('Connected to MySQL as ID ' + connection.threadId);} }); // Perform the query. Escape the user-input using mysql.escape: https://www.npmjs.com/package/mysql#escaping-query-values connection.query('SELECT * from users_db.users_table where name = ' + mysql.escape(username.params(0)), function(err, rows, fields) { if (err) { // MySQL query failed for some reason, so send a null response back to the Tcl iRule if (debug >= 1) {console.error('Error with query: ' + err.stack);} response.reply(''); return; } else { // Check for no result from MySQL if (rows < 1){ if (debug >= 1) {console.log('No matching records from MySQL');} // Return -1 to the Tcl iRule to show no matching records from MySQL response.reply('-1'); } else { if (debug >= 2) {console.log('First row from MySQL is: ', rows[0]);} //Return the group field from the first row to the Tcl iRule response.reply(rows.pop()); } } }); // Close the MySQL connection connection.end(); }); Getting Started To get started with iRules LX, download a BIG-IP trial and follow the upcoming tutorials available on the DevCentral community.F5 Editor Eclipse Plugin v2
We are pleased to announce availability of our updated F5 Programmability Editor for Eclipse IDE v2. The F5 Programmability Development Environment for Eclipse allows you to use the Eclipse IDE to manage iRules, iRules LX, iControl LX, and iApps LX development. By using Eclipse, you can connect to one or more BIG-IP ® devices to view, modify or create iRules, iRules LX workspaces, iControl LX, or iApps LX applications. The editor functionality includes TCL/iRules and JavaScript language syntax highlighting, code completion, and hover documentation for the iRules API. Editor functionality includes syntax support for TCL/iRules, JavaScript, and JavaScript Object Notation (JSON) format, including syntax highlighting and code completion, as well as API documentation. With this release, F5 Programmability Development Environment for Eclipse includes iControl LX and iApps LX development. The LX development environment supports opening an LX application, adding files or deleting files, and downloading files for offline mode editing. This release also includes support for proxy server use and offline-online mode that lets you edit files when not connected to a BIG-IP system (offline mode) and synchronize with a BIG-IP system when you reconnect (online mode). In addition to supporting iRules and iRules LX script development via your own Eclipse IDE in v1, v2 adds support for the following: iControl LX extension development iApps LX template development Offline Mode BIG-IP proxy server access Datagroups The Eclipse Plugin provides modern IDE editing capabilities including syntax highlighting & validation, code completion, and code formatting for both TCL and Javascript. This enables authenticated push & pull of iRules, iControl LX, and iApps LX code files to & from BIG-IP using our iControl REST interface. This simplifies development workflows by enabling our customers to develop iRules, iControl LX, and iApps LX projects in a familiar development environment. Download with installation instructions and documentation is now available as Open Source on F5 Devcentral GitHub. Note: If you connect to TMM interfaces instead of the management port when using the editor, you will need to allow tcp port 7 in your self-IP configuration since the plugin checks availability on that port before connecting via SSL. Instructional Videos Installation Common Tasks Offline Mode Data Groups iRules LX Note: This tool is provided via DevCentral as a free tool, and is in no way officially supported by F5 or F5 Professional Services. All support, questions, comments or otherwise for this editor should be submitted in the Q&A section of DevCentral and tagged with a customeclipse plugin tag.
Getting Started with iRules LX, Part 1: Introduction & Conceptual Overview
What is iRules LX? When BIG-IP TMOS 12.1.0 released a couple weeks ago, Danny wrote an intro piece to a new feature called iRules LX. iRules LX is the next generation of iRules that allows you to programmatically extend the functionality of BIG-IP with Node.js. In this article series, you will learn the basics of what iRules LX is, some use cases, how the components fit together, troubleshooting techniques, and some introductory code solutions. What is Node.js ? Node.js provides an environment for running JavaScript independent of a web browser. With Node.js you can write both clients and servers, but it is mostly used for writing servers. There are 2 main componets to Node.js - Google v8 JavaScript Engine Node.js starts with the same open source JavaScript engine used in the Google Chrome web browser. v8 has a JIT complier that optimizes JavaScript code to allow it to run incrediable fast. It also has an optimizing complier that, during run time, further optimizes the code to increase the performance up to 200 times faster. LibUV Asynchronous I/O Library A JavaScript engine by itself would not allow a full set of I/O operations that are needed for a server (you dont want your web browser accessing the file system of computer after all). LubUV gives Node.js the capability to have those I/O operations. Because v8 is a single threaded process, I/O (which have the most latency) would normally be blocking; that is, the CPU would be doing nothing while waiting from a response of the I/O operation. LibUV allows for the v8 to continue running JavaScript while waiting for the response, maximizing the use of CPU cycles and providing great concurrency. Why Node.js? Some might ask "Why was Node.js chosen?" While there is more than one reason, the biggest factor is that you can use JavaScript. JavaScript has been a staple of web development for almost 2 decades and is a common skillset amongst developers. Also, Node.js has a vibrant community that not only contributes to the Node.js core itself, but also a plethora of people writing libraries (called modules). As of May 2016 there are over 280,000 Node.js modules available in the Node Package Manager (NPM) registry, so chances are you can find a library for almost any use case. An NPM client is included with distributions of Node.js. For more information about the specifics of Node.js on BIG-IP, please refer to AskF5 Solution ArticleSOL16221101. iRules LX Conceptual Operation Unlike iRules which has a TCL interpreter running inside the Traffic Management Micro-kernel (TMM), Node.js is running as a stand alone processes outside of TMM. Having Node.js outside of TMM is an advantage in this case because it allow F5 to keep the Node.js version more current than if it was integrated into TMM. In order to get data from TMM to the Node.js processes, a Remote Procedural Call (RPC) is used so that TMM can send data to Node.js and tell it what "procedure" to run. The conceptual model of iRules LX looks something like this - To make this call to Node.js , we still need iRules TCL. We will cover the coding much more extensively in a few days, so dont get too wrapped up in understanding all the new commands just yet. Here is an example of TCL code for doing RPC to Node.js - when HTTP_REQUEST { set rpc_hdl [ILX::init my_plugin my_extension] set result [ILX::call $rpc_hdl my_method $arg1] if { $result eq "yes" } { # Do something } else { # Do something else } } In line 2 you can see with the ILX::init command that we establish an RPC handle to a specific Node.js plugin and extension (more about those tomorrow). Then in line 3 we use the ILX::call command with our handle to call the method my_method (our "remote procedure") in our Node.js process and we send it the data in variable arg1 . The result of the call will be stored in the variable result . With that we can then evaluate the result as we do in lines 5-9. Here we have an example of Node.js code using the iRules LX API we provide - ilx.addMethod('my_method', function(req, res){ var data = req.params()[0]; <…ADDITIONAL_PROCESSING…> res.reply('<some_reply>'); }); You can see the first argument my_method in addMethod is the name of our method that we called from TCL. The second argument is our callback function that will get executed when we call my_method . On line 2 we get the data we sent over from TCL and in lines 3-5 we perform whatever actions we wish with JavaScript. Then with line 6 we send our result back to TCL. Use Cases While TCL programming is still needed for performing many of the tasks, you can offload the most complicated portions to Node.js using any number of the available modules. Some of the things that are great candidates for this are - Sideband connections -Anyone that has used sideband in iRules TCL knows how difficult it is to implement a protocol from scratch, even if it is only part of the protocol. Simply by downloading the appropriate module with NPM, you can easily accomplish many of the following with ease - Database lookups - SQL, LDAP, Memcached, Redis HTTP and API calls - RESTful APIs, CAPTCHA, SSO protocols such as SAML, etc. Data parsing - JSON - Parser built natively into JavaScript XML Binary protocols In the next installment we will be covering iRules LX configuration and workflow.
Configuring the BIG-IP as an SSH Jump Server using Smart Card Authentication and WebSSH Client
Based on the feedback I got when talking about this capability on social media, I figured I would write an article and expose everyone to what this solution actually looks like and how to deploy it. First off, I want to dig into the use case itself. While the use case for each organization could and is likely different, for a small group of us at F5 Networks had a requirement to smart card enable network devices. Well of course that statement alone comes with a lot of hesitation and questions from your network shop. One likely being "how the heck do you smart card enable something that doesn't support smart card authentication? Then if you can, how the heck to you configure putty to support smart cards without spending a billion dollars?" This is when you tell them, I'm glad you asked because you can't. However, with F5 being the magical software company it is we can enforce smart card authentication, OCSP validation, generate a one-time password (OTP) and present that to the device to authentication all while using your favorite browser. Sound impossible? I think not, so let's get to it! Before getting to far ahead, I wanted to clarify, this is NOT a solution I developed but rather hoped to educate some folks on. The heavy lifting and development came from F5 all stars like Bill Church and Michael Coleman. So when you get done reading this article and want to thank someone, make sure to send them a note. Prerequisites LTM Licensed and Provisioned APM Licensed and Provisioned iRulesLX Provisioned 8Gb of Memory Alright, now that I have provided you with what my use case is lets go ahead and begin the deployment. From a browser, go ahead and navigate to https://github.com/billchurch/f5-pua to download the offline privileged user access zip file that contains all necessary components and for the most part deploys this solution for you. Yes....once again, thank you Bill Church! Once downloaded, extract the build_pua_offline.sh file from the zip. Copy Shell Executable to BIG-IP If you are using a Windows box like myself, go ahead and either download or launch something like WinSCP so that you can transfer the shell executable to your BIG-IP. I simply transferred the .sh file to my /tmp directory as shown below. Once transferred, go ahead and close WinSCP. Run the build_pua_offline.sh Script Because we haven't deployed our WebSSH solution yet we are going to use putty to SSH to our BIG-IP and run the script. Once authenticated, navigate to the directory you stored the .sh file and run the command bash build_pua_offline.sh. You will first be presented with a set of instructions regarding questions you will be asked during the running of this script. Press any key continue. As the instructions imply, you will be providing several IP addresses for the required virtual servers. Please note, the only IP that can NOT be shared is the IP for the WebSSH proxy. WebSSH IP: 10.1.20.100 Radius Service IP: 10.1.20.101 LDAP Service IP: 10.1.20.101 LDAPS Service IP: 10.1.20.101 Webtop IP: 10.1.20.102 Once you have provided all of the necessary IP addresses you will be presented with an option to create a CA for testing purposes. In this guide, we will select N for this option. After all profiles, virtual servers and policies have been created you will be presented with a question of whether or not to configure the BIG-IP to test Radius by configuring remote user authentication for Radius. We will select N for this option. If the script completes successfully, this will be the last item you are prompted for. In our case we had a successful deployment of the build_pua_offline.sh script so let's take a look at the objects that were created. Virtual Servers LX Workspaces LX Plugins Access Policy Portal Access List Webtop List HTTP Basic Auth Profile While we could review each and every one of these, that is not the intent of this article. Now that the script has been deployed, let's begin by focusing on our defined use case which is smart card auth with a WebSSH client. Configure SSL Client Profile To support client certificate-based authentication, we must also create a Client SSL Profile on the BIG-IP using the steps below. Navigate to Local Traffic > Profiles > SSL > Client > Create Name: WebtopSSLProfile Certificate Key Chain: Place a check mark under the custom field. Click Add to select the appropriate cert/key pair. Client Certificate: Leave it set to ignore as the APM ODCA will perform this function. Trusted Certificate Authorities: Select the CA or CA bundle certificate Advertised Certificate Authorities: Select the CA or CA bundle certificate All other settings can be left at their defaults. Click Finished Create a LDAP Pool Navigate to Local Traffic >> Pools >> Click Create Name: LDAP_Pool Health Monitor: TCP Address: IP of your Directory Server Service Port: 389 Click Add Click Finished Configure BIG-IP LDAP Bypass User When configuring the BIG-IP to use LDAP Authentication as you will see at the end of this article, you will need to include that user account in the ephemeral_LDAP_Bypass Data Group List. Navigate to Local Traffic >> iRules : Data Group List >> Click the ephemeral_LDAP_Bypass list that was created when deploying the PUA Offline Script. String: CN=admin,CN=Users,DC=demo,DC=lab Click Update Configuring a LDAP AAA Resource Navigate to Access >> Authentication >> LDAP and select Create Name: LyonWebtopLDAP Server Connection: Direct Base Port: 389 Admin DN: CN=admin,CN=Users,DC=demo,DC=lab Leave all other settings at their defaults and select Finished Configure APM HTTP Basic SSO Profile Navigate to Access >> Single-Sign-On >> HTTP Basic > Click Create Name: ephemeral_auth_clientcert-ephemeral-basic Username Source: session.ldap.last.attr.sAMAccountName Password Source: session.custom.ephemeral.last.password Click Finished Configure APM Portal Access List for BIG-IP Shell While the script run at the beginning of this article does indeed create a portal access list with resources, we will go ahead and create one in order to show a bit more of the solution and its inner workings. Navigate to Access >> Connectivity / VPN >> Portal Access >> Click Portal Access List Click Create Name: LyonsPortalAccess Link Type: Application URI Application URI: https://IPofWebSSHVS:2222/ssh/host/mgmtIP Caption: BIG-IP Shell Click Create Click Add to create a resource item. Link Type: Paths Destination: IP Address of your WebSSH virtual server Paths: /* Scheme: https Port: 2222 SSO Configuration: ephemeral_auth_clientcert-ephemeral-basic Click Finished Configuring APM Access Policy to Support Smart Card Authentication Navigate to Access >> Profiles / Policies: Access Profiles (Per-Session Policies) From here we are going to use the prebuilt policy as our template by selecting Copy. Provided a Copied Profile Name and select Copy. You will be returned to the previous screen automatically. Select Edit in the same row as the profile you created above. This policy was created for demo purposes only though it also provides a very good starting point for configuring our own policy to support smart card authentication. From the page shown in the screenshot above, select the X above the Logon Page to remove it from our visual policy editor. Leave the defaults and select Delete. Once removed, select the + between USG Warning Banner and Admin Access. Select the Authentication tab and add On-Demand Cert Auth Click Add Item When prompted to select the Auth Mode, select Require from the drop down menu and click Save. Once you have been returned to the visual policy editor, select the + between On-Demand Cert Auth and Admin Access following the Successful branch. From the Assignment tab, select Variable Assign and click Add Item When redirected to the page to configure variables as shown below, select Add new entry. When redirected, select change on line item 1. We will configure the following variables based on F5 solution article K17063 found at https://support.f5.com/csp/article/K17063. In the Custom Variable section, type session.logon.last.username. In the Custom Expression section, type the following. set upn [mcget {session.logon.last.upn}]; # if $upn contains @ symbol, extract the username, otherwise return $upn as-is. if { $upn contains "@" } { # Use string first to find index of the @ symbol, then return everything in-front of the @. return [string range $upn 0 [expr { [string first "@" $upn] - 1 } ] ]; } else { # Assume UPN only contains a username return $upn; } Click Finished Once again click Add new entry and select change. In the Custom Variable section, type session.logon.last.upn. In the Custom Expression section, type the following. Click Finished set x509e_fields [split [mcget {session.ssl.cert.x509extension}] "\n"]; # For each element in the list: foreach field $x509e_fields { # If the element contains UPN: if { $field contains "othername:UPN" } { ## set start of UPN variable set start [expr {[string first " # Return the UPN, by finding the index of opening and closing brackets, then use string range to get everything between. return [string range $field $start [expr { [string first ">" $field $start] - 1 } ] ]; } } # Otherwise return UPN Not Found: return "UPN-NOT-FOUND"; Click Save Navigating back to the visual policy editor, select the + between Variable Assign and Admin Access From the Authentication tab, select LDAP Query and click Add Item. When redirected to the LDAP Query Properties page, configure the following. From the drop down menu select the LDAP AAA Server created in previous steps. SearchDN: CN=Users,DC=demo,DC=lab SearchFilter: userPrincipalName=%{session.logon.last.upn} Click Add new entry and add memberOf Click Add new entry and add sAMAccountName Select the Branch Rules tab Remove the text User Group Membership Type LDAP Query Passed Click change following the branch rule expression Click the X as shown below to remove the existing expression. Click Add Expression From the Agent Sel drop down menu select LDAP Query From the Condition drop down menu select LDAP Query Passed Click Add Expression Click Finished Click Save From the Admin Access Macro click Advanced Resource Assign Click Add/Delete Select the Portal Access tab, remove the check box from the sample_pua_policy-webssh_portal and place a check box in the portal access resource created in the previous steps. Click Update Click Save In the top left hand corner of the VPE, click Apply Access Policy You have now completed the VPE portion of the access policy. Configure the PUA Webtop Virtual Server Navigate to Local Traffic >> Virtual Servers >> click pua_webtop Scroll until you locate SSL profile (Client) and assign the SSL profile created in the previous steps. Scroll until you reach the Access Policy portion of the VS. From the Access Profile drop down select the profile created in the previous step. Click Update. Click the Resources tab. From the Default Pool drop down menu, select the Pool created earlier in this document. Click Update. Configure BIG-IP Authentication Navigate to System >> Users >> Authentication From the Users Authentication page click Change From the User Directory drop down menu select Remote - LDAP Host: IP of LDAP Virtual Server Remote Directory Tree: DC=demo, DC=lab Scope: Sub Bind DN: CN=admin,CN=Users,DC=demo,DC=lab Check the box next to Check Member Attribute in Group Login LDAP Attribute: sAMAccountName Click Finished Configuring Remote Role Groups Navigate to System > Users > Select Remote Role Groups Click Create Group Name: BIGIPAdmins Line Order: 1 Attribute String:memberOF=CN=BIGIPadmins,OU=Groups,DC=demo,DC=lab Note: Use the full DN of the active directory security group you are defining with a preceeding 'memberOF='. Assigned Role: Administrator Partition Access: All Terminal Access: tmsh Validation Testing For my validation testing, I created a DNS record for webtop.demo.lab pointing to my webtop virtual server. From a web browser navigate to webtop.demo.lab. Click OK, Proceed to Application Select your user certificate when prompted and click OK From the Webtop, select the portal access resource you created in previous steps. If authentication is successful, you will be presented with a webSSH session as shown below. While this wraps up an overview of deploying and accessing F5's WebSSH capability with integrated smart card authentication, I would like to continue this into a series which includes other network devices or applications, end point checks, restricting access to the management interface and more. If this benefits at least one of you out there this was well worth it for me. Until next time.Getting Started with iRules LX, Part 2: Configuration & Workflow
The previous article in this series covered the reasons for iRules LX and the conceptual operation. Now we need learn the new configurations items that iRules LX introduces. Since iRules LX has DevOps teams in mind, there is a slightly different workflow than standard iRules. Configuration Items LX Workspaces Our first new configuration item, LX Workspaces, provides a development and staging environment for your Node.js code and the TCL iRules that are used for iRules LX. The code can be authored and saved in the workspace without affecting the running code on the data plane. Workspaces can also be exported as TGZ files and then imported to another BIG-IP. This is especially useful to allow for development and testing on a BIG-IP in a lab/dev environment and then later installed onto a production BIG-IP. Both LX Workspaces and LX Plugins (which we will talk about later) can be found on your BIG-IP in the menu Local Traffic > iRules as shown here - When you select the + option to create a new workspace and give it a name, the workspace will look like this (except that it will be blank with no folders) - You can see in the screen shot that we have a rules folder. This folder will be created once you add an iRule and will always be named "rules". It can contain any number TCL iRules you create for the workspace. You also see the folder named dc_extension. This is an extension. Extensions An extension is folder of Node.js code within an LX Workspace. As we saw with the screenshot above, we had one extension named dc_extension. The extension folder can be given any valid Unix name (except rules), but the name will matter when it is time to start coding (more in tomorrow's article). If we expand the extension folder, we will see this - Each extension will have these files and folders to start off with. index.js - This file is the Node.js code that will eventually run on your BIG-IP. node_modules - This folder will contain node modules you install from NPM. package.json - This is the standard package.json file that is use to store various information and package dependencies about a Node.js program. LX Plugin An LX Plugin is the production code from the LX Workspace that is actively running on the data plane of the BIG-IP.You can see in this screenshot the settings of an LX Plugin - Also, each workspace extension is listed. Each extension has their own Node.js settings and if you click on an extension inside a LX Plugin you will see this - Workflow So now that we know our new configuration objects we need to follow a workflow to get iRules LX up and running. That workflow looks like this - Step 1: Write Code in LX Workspace The first step is to actually write the code you want in a LX Workspace. This would be both your Node.js code and the TCL iRules that would make the RPC calls to Node. Step 2: Load LX Workspace to an LX Plugin To load the workspace into the LX Plugin, you would create the LX Plugin and select the LX Workspace that you want to load into it - When you create the LX Plugin, the extensions will get loaded into a Node.js process(es) and the TCL iRules will be loaded in the TCL iRules menu with the rest of the iRules - You can see on the Partition/Path that this matches the LX Plugin we created. This works just like as if an iApp created the iRule. As such, you can not edit the iRule from here--you can only do it in the LX Workspace itself. Step 3: Load the TCL iRule to a Virtual Server At this point your TCL iRule can now be added to a virtual server - You will find your iRule under the Partition/Path for your LX Plugin. Move it over to the enabled section and you will be running iRules LX. Making Updates to iRules LX If you change code in the LX Workspace, that code will not be active until you reload it to the LX Plugin. An LX Workspace let's you know if the code in the workspace is different than that which is running in the LX Plugin with a status indicator - This indicator is yellow when the code is different. To reload the LX Workspace code into the LX Plugin, just click the Reload From Workspace button next to the indicator. When the code is in sync, the indicator will be green - Licensing and Provisioning iRules LX is licensed to all BIG-IP users in TMOS 12.1 (assuming you have a valid support contract). On new licenses (once that where generated after TMOS 12.1 released) iRules LX will be automatically licensed. If you have an older license and have upgraded to v12.1, you simply need to reactivate your license. This can be done online if your BIG-IP has internet access or manually. To use iRules LX, it must be provisioned. It is not turned on by default, so customers that are not using it do not need to worry about Node.js processes running on their BIG-IP. To provision it, browse over to the Resource Provisioning menu of your BIG-IP, check the box for iRules LX and give it a nominal level of resources - In the next article in this series, we will cover writing code for iRules LX.APM Cookbook: Modify LDAP Attribute Values using iRulesLX
Introduction Access Policy Manager (APM) does not have the ability to modify LDAP attribute values using the native features of the product. In the past I’ve used some creative unsupported solutions to modify LDAP attribute values, but with the release of BIG-IP 12.1 and iRulesLX, you can now modify LDAP attribute values, in a safe and in supported manner. Before we get started I have a pre-configured Active Directory 2012 R2 server which I will be using as my LDAP server with an IP address of 10.1.30.101. My BIG-IP is running TMOS 12.1 and the iRules Language eXtension has been licensed and provisioned. Make sure your BIG-IP has internet access to download the required Node.JS packages. For this solution I’ve opted to use the ldapjs package. The iRulesLX requires the following session variable to be set for the LDAP Modify to execute: Distinguished Name (DN): session.ad.last.attr.dn Attribute Name (e.g. carLicense): session.ldap.modify.attribute Attribute Value: session.ldap.modify.value This guide also assumes you have a basic level of understanding and troubleshooting at a Local Traffic Manager (LTM) level and you BIG-IP Self IP, VLANs, Routes, etc.. are all configured and working as expected. Step 1 – iRule and iRulesLX Configuration 1.1 Create a new iRulesLX workspace Local Traffic >> iRules >> LX Workspaces >> “Create” Supply the following: Name: ldap_modify_workspace Select “Finished" to save. You will now have any empty workspace, ready to cut/paste the TCL iRule and Node.JS code. 1.2 Add the iRule Select “Add iRule” and supply the following: Name: ldap_modify_apm_event Select OK Cut / Paste the following iRule into the workspace editor on the right hand side. Select “Save File” to save. # Author: Brett Smith @f5 when RULE_INIT { # Debug logging control. # 0 = debug logging off, 1 = debug logging on. set static::ldap_debug 0 } when ACCESS_POLICY_AGENT_EVENT { if { [ACCESS::policy agent_id] eq "ldap_modify" } { # Get the APM session data set dn [ACCESS::session data get session.ad.last.attr.dn] set ldap_attribute [ACCESS::session data get session.ldap.modify.attribute] set ldap_value [ACCESS::session data get session.ldap.modify.value] # Basic Error Handling - Don't execute Node.JS if LDAP attribute name or value is null if { (([string trim $ldap_attribute] eq "") or ([string trim $ldap_value] eq "")) } { ACCESS::session data set session.ldap.modify.result 255 } else { # Initialise the iRulesLX extension set rpc_handle [ILX::init ldap_modify_extension] if { $static::ldap_debug == 1 }{ log local0. "rpc_handle: $rpc_handle" } # Pass the LDAP Attribute and Value to Node.JS and save the iRulesLX response set rpc_response [ILX::call $rpc_handle ldap_modify $dn $ldap_attribute $ldap_value] if { $static::ldap_debug == 1 }{ log local0. "rpc_response: $rpc_response" } ACCESS::session data set session.ldap.modify.result $rpc_response } } } 1.3 Add an Extension Select “Add extenstion” and supply the following: Name: ldap_modify_extension Select OK Cut / Paste the following Node.JS and replace the default index.js. Select “Save File” to save. // Author: Brett Smith @f5 // index.js for ldap_modify_apm_events // Debug logging control. // 0 = debug off, 1 = debug level 1, 2 = debug level 2 var debug = 1; // Includes var f5 = require('f5-nodejs'); var ldap = require('ldapjs'); // Create a new rpc server for listening to TCL iRule calls. var ilx = new f5.ILXServer(); // Start listening for ILX::call and ILX::notify events. ilx.listen(); // Unbind LDAP Connection function ldap_unbind(client){ client.unbind(function(err) { if (err) { if (debug >= 1) { console.log('Error Unbinding.'); } } else { if (debug >= 1) { console.log('Unbind Successful.'); } } }); } // LDAP Modify method, requires DN, LDAP Attribute Name and Value ilx.addMethod('ldap_modify', function(ldap_data, response) { // LDAP Server Settings var bind_url = 'ldaps://10.1.30.101:636'; var bind_dn = 'CN=LDAP Admin,CN=Users,DC=f5,DC=demo'; var bind_pw = 'Password123'; // DN, LDAP Attribute Name and Value from iRule var ldap_dn = ldap_data.params()[0]; var ldap_attribute = ldap_data.params()[1]; var ldap_value = ldap_data.params()[2]; if (debug >= 2) { console.log('dn: ' + ldap_dn + ',attr: ' + ldap_attribute + ',val: ' + ldap_value); } var ldap_modification = {}; ldap_modification[ldap_attribute] = ldap_value; var ldap_change = new ldap.Change({ operation: 'replace', modification: ldap_modification }); if (debug >= 1) { console.log('Creating LDAP Client.'); } // Create LDAP Client var ldap_client = ldap.createClient({ url: bind_url, tlsOptions: { 'rejectUnauthorized': false } // Ignore Invalid Certificate - Self Signed etc.. }); // Bind to the LDAP Server ldap_client.bind(bind_dn, bind_pw, function(err) { if (err) { if (debug >= 1) { console.log('Error Binding to: ' + bind_url); } response.reply('1'); // Bind Failed return; } else { if (debug >= 1) { console.log('LDAP Bind Successful.'); } // LDAP Modify ldap_client.modify(ldap_dn, ldap_change, function(err) { if (err) { if (debug >= 1) { console.log('LDAP Modify Failed.'); } ldap_unbind(ldap_client); response.reply('2'); // Modify Failed } else { if (debug >= 1) { console.log('LDAP Modify Successful.'); } ldap_unbind(ldap_client); response.reply('0'); // No Error } }); } }); }); You will need to modify the bind_url, bind_dn, and bind_pw variables to match your LDAP server settings. 1.4 Install the ldapjs package SSH to the BIG-IP as root cd /var/ilx/workspaces/Common/ldap_modify_workspace/extensions/ldap_modify_extension npm install ldapjs -save You should expect the following output from above command: [root@big-ip1:Active:Standalone] ldap_modify_extension # npm install ldapjs -save ldapjs@1.0.0 node_modules/ldapjs ├── assert-plus@0.1.5 ├── dashdash@1.10.1 ├── asn1@0.2.3 ├── ldap-filter@0.2.2 ├── once@1.3.2 (wrappy@1.0.2) ├── vasync@1.6.3 ├── backoff@2.4.1 (precond@0.2.3) ├── verror@1.6.0 (extsprintf@1.2.0) ├── dtrace-provider@0.6.0 (nan@2.4.0) └── bunyan@1.5.1 (safe-json-stringify@1.0.3, mv@2.1.1) 1.5 Create a new iRulesLX plugin Local Traffic >> iRules >> LX Plugin >> “Create” Supply the following: Name: ldap_modify_plugin From Workspace: ldap_modify_workspace Select “Finished" to save. If you look in /var/log/ltm, you will see the extension start a process per TMM for the iRuleLX plugin. big-ip1 info sdmd[6415]: 018e000b:6: Extension /Common/ldap_modify_plugin:ldap_modify_extension started, pid:24396 big-ip1 info sdmd[6415]: 018e000b:6: Extension /Common/ldap_modify_plugin:ldap_modify_extension started, pid:24397 big-ip1 info sdmd[6415]: 018e000b:6: Extension /Common/ldap_modify_plugin:ldap_modify_extension started, pid:24398 big-ip1 info sdmd[6415]: 018e000b:6: Extension /Common/ldap_modify_plugin:ldap_modify_extension started, pid:24399 Step 2 – Create a test Access Policy 2.1 Create an Access Profile and Policy We can now bring it all together using the Visual Policy Editor (VPE). In this test example, I will not be using a password just for simplicity. Access Policy >> Access Profiles >> Access Profile List >> “Create” Supply the following: Name: ldap_modify_ap Profile Type: LTM-APM Profile Scope: Profile Languages: English (en) Use the default settings for all other settings. Select “Finished” to save. 2.2 Edit the Access Policy in the VPE Access Policy >> Access Profiles >> Access Profile List >> “Edit” (ldap_modify_ap) On the fallback branch after the Start object, add a Logon Page object. Change the second field to: Type: text Post Variable Name: attribute Session Variable Name: attribute Read Only: No Add a third field: Type: text Post Variable Name: value Session Variable Name: value Read Only: No In the “Customization” section further down the page, set the “Form Header Text” to what ever you like and change “Logon Page Input Field #2” and “Logon Page Input Field #3” to something meaningful, see my example below for inspiration. Leave the “Branch Rules” as the default. Don’t forget to “Save”. On the fallback branch after the Logon Page object, add an AD Query object. This step verifies the username is correct against Active Directory/LDAP, returns the Distinguished Name (DN) and stores the value in session.ad.last.attr.dn which will be used by the iRulesLX. Supply the following: Server: Select your LDAP or AD Server SearchFilter: sAMAccountName=%{session.logon.last.username} Select Add new entry Required Attributes: dn Under Branch Rules, delete the default and add a new one, by selecting Add Branch Rule. Update the Branch Rule settings: Name: AD Query Passed Expression (Advanced): expr { [mcget {session.ad.last.queryresult}] == 1 } Select “Finished”, then “Save” when your done. On the AD Query Passed branch after the AD Query object, add a Variable Assign object. This step assigns the Attribute Name to session.ldap.modify.attribute and the Attribute Value entered on the Logon Page to session.ldap.modify.value. Supply the following: Name: Assign LDAP Variables Add the Variable assignments by selecting Add new entry >> change. Variable Assign 1: Custom Variable (Unsecure): session.ldap.modify.attribute Session Variable: session.logon.last.attribute Variable Assign 2: Custom Variable (Secure): session.ldap.modify.value Session Variable: session.logon.last.value Select “Finished”, then “Save” when your done.Leave the “Branch Rules” as the default. On the fallback branch after the Assign LDAP Variables object, add a iRule object. Supply the following: Name: LDAP Modify ID: ldap_modify Under Branch Rules, add a new one, by selecting Add Branch Rule. Update the Branch Rule settings: Name: LDAP Modify Successful Expression (Advanced): expr { [mcget {session.ldap.modify.result}] == "0" } Select “Finished”, then “Save” when your done. The finished policy should look similar to this: As this is just a test policy I used to test my Node.JS and to show how the LDAP Modify works, I will not have a pool member attached to the virtual server, I have just left the branch endings as Deny. In a real word scenario, you would not allow a user to change any LDAP Attributes and Values. Apply the test Access Policy (ldap_modify_ap) to a HTTPS virtual server for testing and the iRuleLX under the Resources section. Step 3 - OK, let’s give this a test! To test, just open a browser to the HTTPS virtual server you created, and supply a Username, Attribute and Value to be modified. In my example, I want to change the Value of the carLicense attribute to test456. Prior to me hitting the Logon button, I did a ldapsearch from the command line of the BIG-IP: ldapsearch -x -h 10.1.30.101 -D "cn=LDAP Admin,cn=users,dc=f5,dc=demo" -b "dc=f5,dc=demo" -w 'Password123' '(sAMAccountName=test.user)' | grep carLicense carLicense: abc123 Post submission, I performed the same ldapsearch and the carLicense value has changed. It works! ldapsearch -x -h 10.1.30.101 -D "cn=LDAP Admin,cn=users,dc=f5,dc=demo" -b "dc=f5,dc=demo" -w 'Password123' '(sAMAccountName=test.user)' | grep carLicense carLicense: test456 Below is some basic debug log from the Node.JS: big-ip1 info sdmd[6415]: 018e0017:6: pid[24399] plugin[/Common/ldap_modify_plugin.ldap_modify_extension] Creating LDAP Client. big-ip1 info sdmd[6415]: 018e0017:6: pid[24399] plugin[/Common/ldap_modify_plugin.ldap_modify_extension] LDAP Bind Successful. big-ip1 info sdmd[6415]: 018e0017:6: pid[24399] plugin[/Common/ldap_modify_plugin.ldap_modify_extension] LDAP Modify Successful. big-ip1 info sdmd[6415]: 018e0017:6: pid[24399] plugin[/Common/ldap_modify_plugin.ldap_modify_extension] Unbind Successful. Conclusion You can now modify LDAP attribute values, in safe and in supported manner with iRulesLX. Think of the possibilities! For an added bonus, you can add addtional branch rules to the iRule Event - LDAP Modify, as the Node.JS returns the following error codes: 1 - LDAP Bind Failed 2 - LDAP Modified Failed I would also recommend using Macros. Please note, this my own work and has not been formally tested by F5 Networks.Getting Started with iRules LX, Part 3: Coding & Exception Handling
So far in this series, we have covered the conceptual overview, components, workflows, licensing and provisioning of iRules LX. In this article, we will actually get our hands wet with some iRules LX code! TCL Commands As mentioned in the first article, we still need iRules TCL to make a RPC to Node.js. iRules LX introduces three new iRules TCL commands - ILX::init - Creates a RPC handle to the plugin's extension. We will use the variable from this in our other commands. ILX::call - Does the RPC to the Node.js process to send data and receive the result (2 way communication). ILX::notify - Does an RPC to the Node.js process to send data only (1 way communication, no response). There is one caveat to the ILX::call and ILX::notify commands: you can only send 64KB of data. This includes overhead used to encapsulate the RPC so you will need to allow about 500B-1KB for the overhead. If you need to send more data you will need to create code that will send data over in chunks. TCL Coding Here is a TCL template for how we would use iRules LX. when HTTP_REQUEST { set ilx_handle [ILX::init "my_plgin" "ext1"] if {[catch {ILX::call $ilx_handle "my_method" $user $passwd} result]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" # Send user graceful error message, then exit event return } # If one value is returned, it becomes TCL string if {$result eq 'yes'} { # Some action } # If multiple values are returned, it becomes TCL list set x [ lindex $result 0] set y [ lindex $result 1] } So as not to overwhelm you, we'll break this down to bite size chunks. Taking line 2 from above, we will use ILX::init command to create our RPC handle to extension ext1 of our LX plugin my_plugin and store it in the variable ilx_handle . This is why the names of the extensions and plugin matter. set ilx_handle [ILX::init "my_plgin" "ext1"] Next, we can take the handle and do the RPC - if {[catch {ILX::call $ilx_handle "my_method" $user $passwd} result]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" # Send user graceful error message with whatever code, then exit event return } You can see on line 3, we make the ILX::call to the extension by using our RPC handle, specifying the method my_method (our "remote procedure") and sending it one or more args (in this case user and passwd which we got somewhere else). In this example we have wrapped ILX::call in a catch command because it can throw an exception if there is something wrong with the RPC (Note: catch should actually be used on any command that could throw an exception, b64decode for example). This allows us to gracefully handle errors and respond back to the client in a more controlled manner. If ILX::call is successful, then the return of catch will be a 0 and the result of the command will be stored in the variable result . If ILX::call fails, then the return of catch will be a 1 and the variable result will contain the error message. This will cause the code in the if block to execute which would be our error handling. Assuming everything went well, we could now start working with data in the variable result . If we returned a single value from our RPC, then we could process this data as a string like so - # If one value is returned, it becomes TCL string if {$result eq 'yes'} { # Some action } But if we return multiple values from our RPC, these would be in TCL list format (we will talk more about how to return multiple values in the Node.js coding section). You could use lindex or any suitable TCL list command on the variable result - # If multiple values are returned, it becomes TCL list set x [ lindex $result 0] set y [ lindex $result 1] Node.js Coding On the Node.js side, we would write code in our index.js file of the extension in the workspace. A code template will load when the extension is created to give you a starting point, so you dont have to write it from scratch. To use Node.js in iRules LX, we provide an API for receiveing and sending data from the TCL RPC. Here is an example - var f5 = require('f5-nodejs'); var ilx = new f5.ILXServer(); ilx.addMethod('my_method', function (req, res) { // req.params() function is used to get values from TCL. Returns JS Array var user = req.params()[0]; var passwd = req.params()[1]; <DO SOMETHING HERE> res.reply('<value>'); // Return one value as string or number res.reply(['value1','value2','value3']); // Return multiple values with an Array }); ilx.listen(); Now we will run through this step-by-step. On line 1, you see we import the f5-nodejs module which provides our API to the ILX server. var f5 = require('f5-nodejs'); On line 3 we instantiate a new instance of the ILXServer class and store the object in the variable ilx . var ilx = new f5.ILXServer(); On line 5, we have our addMethod method which stores methods (our remote procedures) in our ILX server. ilx.addMethod('my_method', function (req, res) { // req.params() function is used to get values from TCL. Returns JS Array var user = req.params()[0]; var passwd = req.params()[1]; <DO SOMETHING HERE> res.reply('<value>'); // Return one value as string or number res.reply(['value1','value2','value3']); // Return multiple values with an Array }); This is where we would write our custom Node.js code for our use case. This method takes 2 arguments - Method name - This would be the name that we call from TCL. If you remember from our TCL code above in line 3 we call the method my_method . This matches the name we put here. Callback function - This is the function that will get executed when we make the RPC to this method. This function gets 2 arguments, the req and res objects which follow standard Node.js conventions for the request and response objects. In order to get our data that we sent from TCL, we must call the req.param method. This will return an array and the number of elements in that array will match the number of arguments we sent in ILX::call . In our TCL example on line 3, we sent the variables user and passwd which we got somewhere else. That means that the array from req.params will have 2 elements, which we assign to variables in lines 7-8. Now that we have our data, we can use any valid JavaScript to process it. Once we have a result and want to return something to TCL, we would use the res.reply method. We have both ways of using res.reply shown on lines 12-13, but you would only use one of these depending on how many values you wish to return. On line 12, you would put a string or number as the argument for res.reply if you wanted to return a single value. If we wished to return multiple values, then we would use an array with strings or numbers. These are the only valid data types for res.reply . We mentioned in the TCL result that we could get one value that would be a string or multiple values that would be a TCL list. The argument type you use in res.reply is how you would determine that. Then on line 16 we start our ILX server so that it will be ready to listen to RPC. ilx.listen(); That was quick a overview of the F5 API for Node.js in iRules LX. It is important to note that F5 will only support using Node.js in iRules LX within the provided API. A Real Use Case Now we can take what we just learned and actually do something useful. In our example, we will take POST data from a standard HTML form and convert it to JSON.In TCL we would intercept the data, send it to Node.js to transform it to JSON, then return it to TCL to replace the POST data with the JSON and change the Content-Type header - when HTTP_REQUEST { # Collect POST data if { [HTTP::method] eq "POST" }{ set cl [HTTP::header "Content-Length"] HTTP::collect $cl } } when HTTP_REQUEST_DATA { # Send data to Node.js set handle [ILX::init "json_plugin" "json_ext"] if {[catch {ILX::call $handle "post_transform" [HTTP::payload]} json]} { log local0.error "Client - [IP::client_addr], ILX failure: $json" HTTP::respond 400 content "<html>Some error page to client</html>" return } # Replace Content-Type header and POST payload HTTP::header replace "Content-Type" "application/json" HTTP::payload replace 0 $cl $json } In Node.js, would only need to load the built in module querystring to parse the post data and then JSON.stringify to turn it into JSON. 'use strict' var f5 = require('f5-nodejs'); var qs = require('querystring'); var ilx = new f5.ILXServer(); ilx.addMethod('post_transform', function (req, res) { // Get POST data from TCL and parse query into a JS object var postData = qs.parse(req.params()[0]); // Turn postData into JSON and return to TCL res.reply(JSON.stringify(postData)); }); ilx.listen(); Note: Keep in mind that this is only a basic example. This would not handle a POST that used 100 continue or mutlipart POSTs. Exception Handling iRules TCL is very forgiving when there is an unhanded exception. When you run into an unhandled runtime exception (such as an invalid base64 string you tried to decode), you only reset that connection. However, Node.js (like most other programming languages) will crash if you have an unhandled runtime exception, so you will need to put some guard rails in your code to avoid this. Lets say for example you are doing JSON.parse of some JSON you get from the client. Without proper exception handling any client could crash your Node.js process by sending invalid JSON. In iRules LX if a Node.js process crashes 5 times in 60 seconds, BIG-IP will not attempt to restart it which opens up a DoS attack vector on your application (max restarts is user configurable, but good code eliminates the need to change it). You would have to manually restart it via the Web UI or TMSH. In order to catch errors in JavaScript, you would use the try/catch statement. There is one caveat to this: code inside a try/catch statement is not optimized by the v8 complier and will cause a significant decrease in performance. Therefore, we should keep our code in try/catch to a minimum by putting only the functions that throw exceptions in the statement. Usually, any function that will take user provided input can throw. Note: The subject of code optimization with v8 is quite extensive so we will only talk about this one point. There are many blog articles about v8 optimization written by people much smarter than me. Use your favorite search engine with the keywords v8 optimization to find them. Here is an example of try/catch with JSON.parse - ilx.addMethod('my_function', function (req, res) { try { var myJson = JSON.parse(req.params()[0]) // This function can throw } catch (e) { // Log message and stop processing function console.error('Error with JSON parse:', e.message); console.error('Stack trace:', e.stack); return; } // All my other code is outside try/catch var result = ('myProperty' in myJson) ? true : false; res.reply(result); }); We can also work around the optimization caveat by hoisting a custom function outside try/catch and calling it inside the statement - ilx.addMethod('my_function', function (req, res) { try { var answer = someFunction(req.params()[0]) // Call function from here that is defined on line 16 } catch (e) { // Log message an stop processing function console.error('Error with someFunction:', e.message); console.error('Stack trace:', e.stack); return; } // All my other code is outside try/catch var result = (answer === 'hello') ? true : false; res.reply(result); }); function someFuntion (arg) { // Some code in here that can throw return result } RPC Status Return Value In our examples above, we simply stopped the function call if we had an error but never let TCL know that we encountered a problem. TCL would not know there was a problem until the ILX::call command reached it's timeout value (3 seconds by default). The client connection would be held open until it reached the timeout and then reset. While it is not required, it would be a good idea for TCL to get a return value on the status of the RPC immediately. The specifics of this is pretty open to any method you can think of but we will give an example here. One way we can accomplish this is by the return of multiple values from Node.js. Our first value could be some type of RPC status value (say an RPC error value) and the rest of the value(s) could be our result from the RPC. It is quite common in programming that make an error value would be 0 if everything was okay but would be a positive integer to indicate a specific error code. Here in this example, we will demonstate that concept. The code will verify that the property myProperty is presentin JSON data and put it's value into a header or send a 400 response back to the client if not. THe Node.js code - ilx.addMethod('check_value', function (req, res) { try { var myJson = JSON.parse(req.params()[0]) // This function can throw } catch (e) { res.reply(1); //<---------------------------------- The RPC error value is 1 indicating invalid JSON return; } if ('myProperty' in myJson){ // The myProperty property was present in the JSON data, evaluate its value var result = (myJson.myProperty === 'hello') ? true : false; res.reply([0, result]); //<-------------------------- The RPC error value is 0 indicating success } else { res.reply(2); //<-------------- The RPC error value is 2 indicating myProperty was not present } }); In the code above the first value we return to TCL is our RPC error code. We have defined 3 possible values for this - 0 - RPC success 1 - Invalid JSON 2 - Property "myProperty" not present in JSON One our TCL side we would need to add logic to handle this value - when HTTP_REQUEST { # Collect POST data if { [HTTP::method] eq "POST" }{ set cl [HTTP::header "Content-Length"] HTTP::collect $cl } } when HTTP_REQUEST_DATA { # Send data to Node.js set handle [ILX::init "json_plugin" "json_checker"] if {[catch [ILX::call $handle "check_value" [HTTP::payload]] json]} { log local0.error "Client - [IP::client_addr], ILX failure: $result" HTTP::respond 400 content "<html>Some error page to client</html>" return } # Check the RPC error value if {[lindex $json 0] > 0} { # RPC error value was not 0, there is a problem switch [lindex $json 0] { 1 { set error_msg "Invalid JSON"} 2 { set error_msg "myProperty property not present"} } HTTP::respond 400 content "<html>The following error occured: $error_msg</html>" } else { # If JSON was okay, insert header with myProperty value HTTP::header insert "X-myproperty" [lindex $json 1] } } As you can see on line 19, we check the value of the first element in our TCL list. If it is greater than 0 then we know we have a problem. We move on down further to line 20 to determine the problem and set a variable that will become part of our error message back to the client. Note:Keep in mind that this is only a basic example. This would not handle a POST that used 100 continue or mutlipart POSTs. In the next article in this series, we will cover how to install a module with NPM and some best practices.APM Cookbook: Two-Factor Authentication using YubiKey OTP with iRulesLX.
Introduction It’s been a number of years since I penned my first DC article: Two-Factor Authentication using YubiKey, YubiCloud and APM. A lot has changed over the years, BIG-IP versions and features, new YubiKey models and the YubiCloud Validation API has changed significantly rendering my older article obsolete. This article is a rewrite of the original with a number of improvements, such as: No need for HTTP Auth agent No need to Reverse Proxy the HTTP connection to the YubiCloud API “yub” NPM package used with iRulesLX. This does all the hard work for us, such as signing the message, validating the response and decoding the YubiKey serial. HMAC-SHA1 signed message Signed response validation VPE improvements to protect AD Account Lockouts YubiKey 2-Factor Authentication Process with APM The authentication process can be broken down into a few simple steps which is illustrated below and explained in more detail. Step 1 – The user is presented with a login page. The login page in my example asks for a Username, Password and YubiKey OTP. After entering your username and password, you simply plug in the YubiKey to the USB port and press the button. The YubiKey will generate the unique OTP followed by the enter key. Step 2 & 3 – APM sends the YubiKey OTP to the YubiCloud validation service. If the YubiCloud API returns “Status=OK”, the signature and the nonce is verified, then we know the YubiKey OTP is valid. This is performed by the “yub” NPM package using iRulesLX. Step 4 & 5 – Check to make sure the user has been provisioned a YubiKey and the Serial number assigned to that user matches. I store the 8-digit YubiKey serial number to an Active Directory attribute: “employeeID”. Obviously you can use any attribute field you like or you can modify the policy to query a data group. Step 6 & 7 – The Username and Password is verified by Active Directory/LDAP or what ever is your preference. Step 8 - On success, grant the user access to the resource. An explanation of the validation protocol can be found here: https://developers.yubico.com/yubikey-val/Validation_Protocol_V2.0.html. The “yub” NPM module uses this API and simplifies the validation and signing process. Before we get started I have a pre-configured Active Directory 2012 R2 server which I will be using as my LDAP server with an IP address of 10.1.30.101. My BIG-IP is running TMOS 12.1.2 and the iRules Language eXtension has been licensed and provisioned. Make sure your BIG-IP has internet access to download the required Node.JS packages. This guide also assumes you have a basic level of understanding and troubleshooting at a Local Traffic Manager (LTM) level and your BIG-IP Self IP, VLANs, Routes, etc.. are all configured and working as expected. You have obtained a Client ID and API Key from: https://upgrade.yubico.com/getapikey/ to validate the YubiKey OTP. Step 1 – iRule and iRuleLX Configuration 1.1 Create a new iRulesLX workspace Local Traffic >> iRules >> LX Workspaces >> “Create” Supply the following: Name: yubikey_auth_workspace Select “Finished" to save. You will now have any empty workspace, ready to cut/paste the TCL iRule and Node.JS code. 1.2 Add the iRule Select “Add iRule” and supply the following: Name: yubikey_auth_apm_event_irulelx Select OK Cut / Paste the following iRule into the workspace editor on the right hand side. Select “Save File” to save. # Author: Brett Smith @f5 when RULE_INIT { # Debug logging control. # 0 = debug logging off, 1 = debug logging on. set static::yubikey_debug 0 } when ACCESS_POLICY_AGENT_EVENT { if { [ACCESS::policy agent_id] eq "yubikey_auth" } { # Get the YubiKey OTP from APM session data set yubiotp [ACCESS::session data get session.logon.last.yubiotp] if { $static::yubikey_debug == 1 }{ log local0. "YubiKey OTP: $yubiotp" } # Basic error handling - don't execute Node.JS if session.logon.last.yubiotp is null if { ([string trim $yubiotp] eq "") } { # The YubiKey OTP is not valid ACCESS::session data set session.yubikey.valid 0 if { $static::yubikey_debug == 1 }{ log local0. "YubiKey OTP is not valid!" } } else { # Initialise the iRulesLX extension set rpc_handle [ILX::init yubikey_auth_extension] # Need to change the default RPC timeout from 3 sec to 30 sec to # allow for the HTTPS request to the Yubico API set timeout 30000 # Pass the YubiKey OTP to Node.JS and save the iRulesLX response set rpc_response [ILX::call $rpc_handle -timeout $timeout yubikey_auth $yubiotp] if { $static::yubikey_debug == 1 }{ log local0. "rpc_response: $rpc_response" } # Loop through each key/value pair returned from "yub.verify" foreach {key value} $rpc_response { # Assign the key/value pair to an APM session variable so it # can be referenced in the Access Policy ACCESS::session data set session.yubikey.$key $value if { $static::yubikey_debug == 1 }{ log local0. "$key $value" } } } } } 1.3 Add the Extension Select “Add extenstion” and supply the following: Name: yubikey_auth_extension Select OK Cut / Paste the following Node.JS and replace the default index.js. Select “Save File” to save. Update the “client_id” and “secret_key” variables with your Yubico Client ID and API Key. // Author: Brett Smith @f5 // index.js for yubikey_auth_apm_event_lx // Includes var f5 = require('f5-nodejs'); var yub = require('yub'); // Create a new rpc server for listening to TCL iRule calls. var ilx = new f5.ILXServer(); // Start listening for ILX::call and ILX::notify events. ilx.listen(); // YubiKey Auth ilx.addMethod('yubikey_auth', function(yubiotp, response) { // Get a Yubico Client ID and API Key from here: https://upgrade.yubico.com/getapikey/ var client_id = 'XXXX'; var secret_key = 'XXXXXXXXXXXXXXX'; // Initialise the yub library yub.init(client_id, secret_key); // Attempt to verify the OTP yub.verify(yubiotp.params()[0], function(err,data) { if (err) { console.log('Error: YubiKey OTP Verify Failed!'); response.reply('valid 0'); } else { response.reply(data); } }); }); 1.4 Install the “yub” package SSH to the BIG-IP as root cd /var/ilx/workspaces/Common/yubikey_auth_workspace/extensions/yubikey_auth_extension npm install yub -save You should expect the following output from the above command: [root@big-ip1:Active:Standalone] ldap_modify_extension # npm install yub -save yub@0.11.1 node_modules/yub 1.5 Create a the iRulesLX plugin Local Traffic >> iRules >> LX Plugin >> “Create” Supply the following: Name: yubikey_auth_plugin From Workspace: yubikey_auth_workspace Select “Finished" to save. If you look in /var/log/ltm, you will see the extension start a process per TMM for the iRuleLX plugin. big-ip1 info sdmd[16339]: 018e000b:6: Extension /Common/yubikey_auth_plugin:yubikey_auth_extension started, pid:975 big-ip1 info sdmd[16339]: 018e000b:6: Extension /Common/yubikey_auth_plugin:yubikey_auth_extension started, pid:976 big-ip1 info sdmd[16339]: 018e000b:6: Extension /Common/yubikey_auth_plugin:yubikey_auth_extension started, pid:977 big-ip1 info sdmd[16339]: 018e000b:6: Extension /Common/yubikey_auth_plugin:yubikey_auth_extension started, pid:978 Step 2 – APM Configuration 2.1 Create a new Authentication Server or reuse an existing server 2.1.1 Access Policy >> AAA Servers >> Active Directory >> “Create” Supply the following: Name: f5.demo_ad_aaa (something sensible) Domain Name: f5.demo (Domain Name) Server Connection: Direct or Use Pool depending on your setup. Domain Controller: <FQDN> or (AD server) or leave blank and APM will use DNS. Admin Name and Password Select “Finished" to save. 2.2 Create an Access Profile and Policy 2.2.1 Access Policy >> Access Profiles >> Access Profiles List >> “Create” Supply the following: Name: yubikey_otp_2fa_iruleslx_ap Profile Type: All Profile Scope: Profile Languages: English (en) Use the default settings for all other settings. Select “Finished" to save. 2.2.2 Access Policy >> Access Profiles >> Access Profiles List >> “Edit” On the “fallback” branch after the “Start” object, add a “Logon Page” object. Add a third field: Type: text Post Variable Name: yubiotp Session Variable Name: yubiotp Read Only: No In the “Customization” section further down the page, set the “Form Header Text” to what ever you like and change “Logon Page Input Field #3” to something meaningful, see my example below for inspiration. Leave the “Branch Rules” as the default. Don’t forget to “Save”. 2.2.3 On the “fallback” branch after the “Logon Page” object, add an “iRule Event” object. This step verifies the YubiKey OTP by passing “session.logon.last.yubiotp” from the ”Logon Page” to the iRuleLX created in Step 1. Supply the following Properties: Name: YubiKey Auth ID: yubikey_auth 2.2.3.1 Under “Branch Rules”, add a new one, by selecting “Add Branch Rule”. Update the Branch Rule settings: Name: YubiKey OTP Valid Expression (Advanced): expr { [mcget {session.yubikey.valid}] == "1" } Select “Finished”, then “Save” when your done. 2.2.4 On the “YubiKey OTP Valid” branch after the “YubiKey Auth” object, add an “AD Query” object. This step checks if the user has a YubiKey provisioned in their Active Directory account and the Serial number assigned to that user matches. I’ve added the serial number of the YubiKey to the “employeeID” attribute in Active Directory for each user. I used the “employeeID” attribute for simplicity, but I would recommend creating a custom AD attribute for the YubiKey serial number. Supply the following Properties: Name: YubiKey Serial Match Server: /Common/f5.demo_ad_aaa (select your AD Server) SearchFilter: sAMAccountName=%{session.logon.last.username} Required Attributes: employeeID 2.2.4.1 Under “Branch Rules”, delete the default and add a new one, by selecting “Add Branch Rule”. Update the Branch Rule settings: Name: Not Provisioned Expression (Advanced): expr { [mcget {session.ad.last.attr.employeeID}] == "" } Select “Finished. 2.2.4.2 Add another Branch Rule by selecting “Add Branch Rule”. Update the Branch Rule settings: Name: Match Found Expression (Advanced): expr { [mcget {session.yubikey.serial}] eq [string trim [mcget {session.ad.last.attr.employeeID}] 0] } Select “Finished”, then “Save” when your done. 2.2.5 On the “Match Found” branch after the “YubiKey Serial Match” object, add an “AD Auth” object. This step verifies the username and password is correct against Active Directory. Supply the following Properties: Name: AD Auth AAA Server: /Common/f5.demo_ad_aaa (select your AD Server) Leave the “Branch Rules” as the default. Select “Save” when your done. 2.2.6 On the “Successful” branch after the “AD Auth” object, change the branch end from “ Deny” to “Allow”. This competes the Access Policy. It should resemble something similar to this: Step 3 – Virtual Server Configuration Attach the Access Policy (yubikey_otp_2fa_iruleslx_apldap_modify_ap) to a HTTPS virtual server. Attach the iRuleLX (yubikey_auth_apm_event_irulelx) under the Resources section. Conclusion This is another great example how you can easily add a 2nd factor of authentication to any application using the power of Access Policy Manager (APM). F5 provides a 10 Concurrent User trial version of APM with every BIG-IP licensed with LTM. APM is one of my favourite pieces of technology, it amazes me every day what I can create with this flexible tool. Why not give it a try today.Getting Started with iRules LX, Part 5: Troubleshooting
When you start writing code, you will eventually run into some issue and need to troubleshoot. We have a few tools available for you. Logging One of the most common tools use for troubleshooting is logging. STDOUT and STDERR for the Node.js processes end up in the file /var/log/ltm . By temporarily inserting functions such as console.log , console.error , etc. you can easily check the state of different objects in your code at run time. There is one caveat to logging in iRules LX; logging by most of most processes on BIG-IP is throttled by default. Basically, if 5 messages from a process come within 1 second, all other messages with be truncated. For Node.js, every line is a separate message so something like a stack trace will get automatically truncated. To remove this limit we can change the DB variable log.sdmd.level to the value debug in TMSH as such - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # modify sys db log.sdmd.level value debug SDMD is the process that manages the Node.js instances. This turns on verbose logging for SDMD and all the Node.js processes so you will want to turn this back to info when you are finished troubleshooting. Once that is done you can start adding logging statements to your code. Here, I added a line to tell me when the process starts (or restarts) - var f5 = require('f5-nodejs'); console.log('Node.js process starting.'); var ilx = new f5.ILXServer(); The logs would then show me this - Jun 7 13:26:08 test-ve info sdmd[12187]: 018e0017:6: pid[21194] plugin[/Common/test_pl1.ext1] Node.js process starting. Node Inspector BIG-IP comes packaged with Node Inspector for advanced debugging. We won't cover how to use Node Inspector because most developers skilled in Node.js already know how to use it and there are already many online tutorials for those that dont. Note: We highly recommend that you do not use Node Inspector in a production environment. Debuggers lock the execution of code at break points pausing the entire application, which does not work well for a realtime network device. Ideally debugging should only be done on a dedicated BIG-IP (such as a lab VE instance) in an isolated development environment. Turn on Debugging The first thing we want to do is to put our extension into single concurrency mode so that we are only working with one Node.js process while debugging. Also, while we are doing that we can put the extension into debugging mode. This can be done from TMSH like so - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # list ilx plugin DC_Plugin ilx plugin DC_Plugin { disk-space 156 extensions { dc_extension { } dc_extension2 { } } from-workspace DevCentralRocks staged-directory /var/ilx/workspaces/Common/DevCentralRocks } root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # modify ilx plugin DC_Plugin extensions { dc_extension { concurrency-mode single command-options add { --debug } }} root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # list ilx plugin DC_Plugin ilx plugin DC_Plugin { disk-space 156 extensions { dc_extension { command-options { --debug } concurrency-mode single } dc_extension2 { } } from-workspace DevCentralRocks staged-directory /var/ilx/workspaces/Common/DevCentralRocks } root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # restart ilx plugin DC_Plugin immediate Increase ILX::call Timeout Also, if you recall from part 3 of this series, the TCL command ILX::call has a 3 second timeout by default that if there is no response to the RPC, the command throws an exception. We don't need to worry about this for the command ILX::notify because it does not expect a response from the RPC. While we have the debugger paused on a break point you are pretty much guaranteed to hit this timeout, so we need to increase it to allow for our whole method to finish execution. To do this, you will need to add arguments to the ILX::call highlighted in red - ILX::call $ilx_handle -timeout big_number_in_msec method arg1 This number should be big enough to walk through the entire method call while debugging. If it will take you 2 minutes, then the value should be 120000. Once you update this make sure to reload the workspace. Launching Node Inspector Once you have put Node.js into debugging mode and increased the timeout, you now you can fire up Node Inspector. We will need to find the debugging port that our Node.js process has been assigned via TMSH - root@(eflores-ve3)(cfg-sync Standalone)(Active)(/Common)(tmos) # show ilx plugin DC_Plugin <----------------snipped -----------------> --------------------------------- | Extension Process: dc_extension --------------------------------- | Status running | PID 25975 | TMM 0 | CPU Utilization (%) 0 | Debug Port 1033 | Memory (bytes) | Total Virtual Size 1.2G | Resident Set Size 5.8K Now that we have the debugging port we can put that and the IP of our management interface into the Node Inspector arguments - [root@test-ve:Active:Standalone] config # /usr/lib/node_modules/.bin/node-inspector --debug-port 1033 --web-host 192.168.0.245 --no-inject Node Inspector v0.8.1 Visit http://192.168.0.245:8080/debug?port=1033 to start debugging. We simply need to take the URL given to us and put it in our web browser. Then we can start debugging away! Conclusion We hope you have found the Getting Started with iRules LX series very informative and look forward to hearing about how iRules LX has helped you solved problems. If you need any help with using iRules LX, please dont hesitate to ask a question on DevCentral.Getting Started with iRules LX, Part 4: NPM & Best Practices
So far in this series we've covered basic nomenclature and concepts, and in the last article actually dug into the code that makes it all work. At this point, I'm sure the wheels of possibilities are turning in your minds, cooking up all the nefarious interesting ways to extend your iRules repertoire. The great thing about iRules LX, as we'll discuss in the onset of this article, is that a lot of the heavy lifting has probably already been done for you. The Node.js package manager, or NPM, is a living, breathing, repository of 280,000+ modules you won't have to write yourself should you need them! Sooner or later you will find a deep desire or maybe even a need to install packages from NPM to help fulfill a use case. Installing packages with NPM NPM on BIG-IP works much of the same way you use it on a server. We recommend that you not install modules globally because when you export the workspace to another BIG-IP, a module installed globally won't be included in the workspace package. To install an NPM module, you will need to access the Bash shell of your BIG-IP. First, change directory to the extension directory that you need to install a module in. Note: F5 Development DOES NOT host any packages or provide any support for NPM packages in any way, nor do they provide security verification, code reviews, functionality checks, or installation guarantees for specific packages. They provide ONLY core Node.JS, which currently, is confined only to versions 0.12.15 and 6.9.1. The extension directory will be at /var/ilx/workspaces/<partition_name>/<workspace_name>/extensions/<extension_name>/ . Once there you can run NPM commands to install the modules as shown by this example (with a few ls commands to help make it more clear) - [root@test-ve:Active:Standalone] config # cd /var/ilx/workspaces/Common/DevCentralRocks/extensions/dc_extension/ [root@test-ve:Active:Standalone] dc_extension # ls index.js node_modules package.json [root@test-ve:Active:Standalone] dc_extension # npm install validator --save validator@5.3.0 node_modules/validator [root@test-ve:Active:Standalone] dc_extension # ls node_modules/ f5-nodejs validator The one caveat to installing NPM modules on the BIG-IP is that you can not install native modules. These are modules written in C++ and need to be complied. For obvious security reasons, TMOS does not have a complier. Best Practices Node Processes It would be great if you could spin up an unlimited amount of Node.js processes, but in reality there is a limit to what we want to run on the control plane of our BIG-IP. We recommend that you run no more than 50 active Node processes on your BIG-IP at one time (per appliance or per blade). Therefore you should size the usage of Node.js accordingly. In the settings for an extension of a LX plugin, you will notice there is one called concurrency - There are 2 possible concurrency settings that we will go over. Dedicated Mode This is the default mode for all extensions running in a LX Plugin. In this mode there is one Node.js process per TMM per extension in the plugin. Each process will be "dedicated" to a TMM. To know how many TMMs your BIG-IP has, you can run the following TMSH command - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # show sys tmm-info | grep Sys::TMM Sys::TMM: 0.0 Sys::TMM: 0.1 This shows us we have 2 TMMs. As an example, if this BIG-IP had a LX plugin with 3 extensions, I would have a total of 6 Node.js processes. This mode is best for any type of CPU intensive operations, such as heavy parsing data or doing some type of lookup on every request, an application with massive traffic, etc. Single Mode In this mode, there is one Node.js process per extension in the plugin and all TMMs share this "single" process. For example, one LX plugin with 3 extensions will be 3 Node.js processes. This mode is ideal for light weight processes where you might have a low traffic application, only do a data lookup on the first connection and cache the result, etc. Node.js Process Information The best way to find out information about the Node.js processes on your BIG-IP is with the TMSH command show ilx plugin . Using this command you should be able to choose the best mode for your extension based upon the resource usage. Here is an example of the output - root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos) # show ilx plugin DC_Plugin --------------------------------- ILX::Plugin: DC_Plugin --------------------------------- State enabled Log Publisher local-db-publisher ------------------------------- | Extension: dc_extension ------------------------------- | Status running | CPU Utilization (%) 0 | Memory (bytes) | Total Virtual Size 1.1G | Resident Set Size 7.7K | Connections | Active 0 | Total 0 | RPC Info | Total 0 | Notifies 0 | Timeouts 0 | Errors 0 | Octets In 0 | Octets Out 0 | Average Latency 0 | Max Latency 0 | Restarts 0 | Failures 0 --------------------------------- | Extension Process: dc_extension --------------------------------- | Status running | PID 16139 | TMM 0 | CPU Utilization (%) 0 | Debug Port 1025 | Memory (bytes) | Total Virtual Size 607.1M | Resident Set Size 3.8K | Connections | Active 0 | Total 0 | RPC Info | Total 0 | Notifies 0 | Timeouts 0 | Errors 0 | Octets In 0 | Octets Out 0 | Average Latency 0 | Max Latency 0 From this you can get quite a bit of information, including which TMM the process is assigned to, PID, CPU, memory and connection stats. If you wanted to know the total number of Node.js processes, that same command will show you every process and it could get quite long. You can use this quick one-liner from the bash shell (not TMSH) to count the Node.js processes - [root@test-ve:Active:Standalone] config # tmsh show ilx plugin | grep PID | wc -l 16 File System Read/Writes Since Node.js on BIG-IP is pretty much stock Node, file system read/writes are possible but not recommended. If you would like to know more about this and other properties of Node.js on BIG-IP, please see AskF5 Solution ArticleSOL16221101. Note:NPMs with symlinks will no longer work in 14.1.0+ due to SELinux changes In the next article in this series we will cover troubleshooting and debugging.
