Small URL Generator: Part 1

The Small URL Generator was born partly out of curiosity and partly out of necessity. It started off when I was in our IT department here at F5 when a coworker and I began discussing what it would take to build an in-house application to shorten URLs. Why didn’t we use one of the professionally available services you ask? If we were to make this service available company-wide, there was a possibility that someone could potentially disclose confidential information in an URL, ie. http://sales-app.example.com/johns%20big%20contract%20with%20company%20XYZ.pdf. This may be a fringe case, but it is possible and could be damaging if it wound up in the wrong hands. We came to the conclusion that the best way to provide such a service would be to build and deploy it internally. That’s about as far as the conversation went until a few weeks ago when the topic resurfaced during a meeting. We decided that an iRule using tables would be a great way to facilitate a small URL service.

After a few prototypes and a great lesson in iRule optimization from Colin, this is what we arrived at:

   1: when RULE_INIT {

   2:   set static::small_url_timeout 600

   3:   set static::small_url_lifetime 600

   4: }

5:

   6: when HTTP_REQUEST {

   7:   if { ([HTTP::uri] starts_with "/create?") and ([HTTP::query] ne "") } {

   8:     set url [URI::decode [string tolower [URI::query [HTTP::uri] url]]]

   9:     set url_key [string tolower [scan [string map {/ "" + ""} [b64encode [md5 $url]]] {%6s}]]

10:

  11:     if { ([table lookup -subtable small_url $url_key] eq "") } {

  12:       table add -subtable small_url $url_key $url $static::small_url_timeout $static::small_url_lifetime

  13:       log local0. "Small URL created for $url with key $url_key"

  14:     } else {

  15:       log local0. "Small URL for $url already exists with key $url_key"

  16:     }

17:

  18:     HTTP::respond 200 content "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"><html><head> \

  19:       <title>Small URL Generator</title></head><body><center><h1>Small URL Generator</h1> \

  20:       </center><center>The small URL for <a href=\"$url\">$url</a> is \

  21:       <a href=\"http://[HTTP::host]/$url_key\">http://[HTTP::host]/$url_key</a></center> \

  22:       </body></html>"

  23:   } else {

  24:     set url_key [string map {/ ""} [HTTP::path]]

  25:     set url [table lookup -subtable small_url $url_key]

26:

  27:     if { [string length $url] != 0 } {

  28:       log local0. "Found key $url_key, redirecting to $url"

  29:       HTTP::redirect $url

  30:     } else {

  31:       HTTP::respond 200 content "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"><html><head> \

  32:         <title>Small URL Generator</title> </head><body> <center><h1>Small URL Generator</h1> \

  33:         </center><form action=\"/create\" method=\"get\"><center> \

  34:         <input type=\"text\" name=\"url\">&nbsp;<input type=\"submit\" value=\"make small!\"> \

  35:         </center></form></body></html>"

  36:     }

  37:   }

  38: }

This implementation is based on the bit.ly and TinyURL out-of-band models. It is the simplest and most flexible method of generating short URLs that can be used for both internal and external addresses. In order to deploy this in your environment, all that is needed is an HTTP virtual with this iRule assigned and an FQDN (links.example.com) pointed at the virtual. Then browse over to links.example.com, paste your atrociously long URL, and click “make small!”.

Now onto the guts of this iRule. The URL is shortened to a 6-character pseudo-base 36 encoded key. I say “pseudo” because there is no way to convert it bidirectionally to its original base 64 counterpart. The reason for this is that the “/” and “+” characters are removed from the base 64 encoding and all letters are converted to lower case. This removes 28 of the 64 characters used in base 64 encoding, leaving 36 remaining thus the resulting base 36 key.

   1: set url_key [string tolower [scan [string map {/ "" + ""} [b64encode [md5 $url]]] {%6s}]]

With a 6-character base 36 key, we can store 2,176,782,335 URLs in our subtable. There is a potential for key conflicts in this implementation without any conflict mitigation mechanism, but highly unlikely on the scale in which this was intended to be deployed. The session table where these key value pairs are stored can hold millions of entries. The reason we used a 6 character key was to make the possibility of a key conflict infinitesimally small. With a 5 character key, there are 60.4 million unique keys, and with a 4 character key, there are still 1.67 million unique keys. We will leave it up to you as to how many unique keys are appropriate for your organization, but there is potential to make the URL even shorter if you are willing to accept the key conflict fringe case.

The other configurable options are the lifetime and timeout variables. These dictate how long the short URL key will be stored in the ‘small_url’ subtable. The lifetime variable indicates the total time the URL/key pair will be stored, regardless of the timeout. The timeout is the amount of time that is allowed to pass since the small URL was last accessed before the URL/key pair are removed. In the case of our sample code, the URL/key pair are removed after one day (86,400 seconds) regardless of how many times or how often it was accessed. One day probably isn’t the best lifetime for a short URL in a production environment. Someone will inevitably be out of office, get the link a day late, and the key lookup will fail. More realistic values for a production environment would be a 6 month lifetime (or possibly indefinite) and a 1-2 month timeout.

We have more ideas in the way of Small URLs. Stay tuned for the next piece in this series, which will cover URL obfuscation via short URL. We hope you have as much fun with this iRule was we did writing it.

Published Aug 11, 2010

Version 1.0

BIG-IP

news

tech tip