Mattermost, F5 LTM, and Websockets
I recently worked with a team that wanted to use the F5 Local Traffic Manager (LTM) feature to load balance connections to their new deployment of the Mattermost open source messaging platform within their on-premises datacenter. This application uses both HTTPS and Websockets connections for real-time chat. We ran into a few configuration issues but eventually found the right combination of “nerd knobs” to allow successful ingress traffic. This post is to consolidate these details and hopefully save time to other F5 engineers attempting to do the same. Business Requirements Ingress (client-side) connections TLS 1.1 or higher Support Websockets Only allow customized Mattermost mobile (iOS and Android) applications, provisioned from within the organization and using a custom header, to connect from the public Internet. The same VIP for the mobile traffic should be used by internal desktop or web browsers, which will not include this custom header Virtual Server Configuration The virtual server configuration was fairly straight forward: Protocol Profile (Client) = a mobile optimized TCP profile Protocol Profile (Server) = a LAN optimized TCP profile SSL Profile (Client) = profile with the Option No TLSv1 enabled SSL Profile (Server) = standard serverssl profile or custom one WebSocket Profile = WebSocket (or a custom one with this as the parent) SNAT = custom pool, but AutoMap would work OneConnect = standard oneconnect profile (or a custom one with this as the parent) Default Persistence = cookie Fallback Persistence = source address Pool = Mattermost server pool iRule = custom Mattermost iRule Mattermost iRule To meet some of the business requirements a custom iRule was created to handle some of the conditions outlined. Comments in-line, but this checks to see if the connection was outside of the organization and if so verifies the presence of the custom HTTP header and value. This also checks to see if the connection was requested to upgrade to Websockets, and if it is, change the HTTP filter from full parsing to passthrough mode. when HTTP_REQUEST { if { !(IP:addr [IP::client_addr] equals 192.168.0.0/255.255.255.0]) } { Request from IP outside of organization, check for customer HTTP header if { [HTTP::header x-the-custom-http-header-name] contains "customvalue" } { Custom HTTP header and matching value found if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{ Connection is requesting WebSockets, stop HTTP parsing HTTP::disable } } elseif { [HTTP::cookie exists MMAUTHENTOKEN] && [HTTP::cookie exists MMUSERID] } { Since WebSocket connections do not have HTTP Header, check to see if connection has already authenticated and allow the connection return } else { Connection fails conditions, reject it reject } } else { if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{ Connection is requesting WebSockets, stop HTTP parsing HTTP::disable } } }
Linerate 2.6.1 / Socket.IO - Express
Suppose a farm of Nodejs servers, running Express and Socket.IO, without any reverse proxy such as apache or nginx. Express and Socket.IO are running in HTTPS Is Linerate support Express and Websocket ( socket.io ). I tried to several setup such as VIP ( HTTP:443 ) ---> VS ( HTTP ) ---> RS ( HTTP ) with the right SSL Profile and Hostname or VIP ( TCP:443 ) ----> VS ( TCP ) ----> RS ( TCP ) with the right SSL profile and Hostname Nothing to do, this does not work and claiming for 404 not found. However, with the same setup for regular PHP Backends it was working. Any advices ? Is there any online ressources on how accomplish this setup on linerate ?
iRule for supporting WebSocket in LTM v10.2.4
Hi, We had developed an iRule for support WebSocket in LTM v10.2.4 HF7 (Although 11.4 had native support, but we had no plan to upgrade currently) Sample handshake (http://en.wikipedia.org/wiki/WebSocketWebSocket_protocol_handshake) when HTTP_RESPONSE { if status code is 101 (Switching Protocols) and there exists HTTP header "Upgrade: WebSocket", that means we are switching to WebSocket and thus HTTP filter shall be disabled use case insensitive matching if { [HTTP::status] == 101 && "websocket" eq [string tolower [HTTP::header value "Upgrade"]] } { HTTP::disable } } But it does not work. Adding debug message found that I never saw HTTP::status returns 101 (I had confirmed with tcpdump that the server did reply with status code 101). So I doubt that there are something wrong for handling this status code in LTM v10.2.4 HF7. Currently I had used the following iRule which is less secure as it will disable HTTP filter once client request to switch to WebSocket without confirmation from server. when HTTP_REQUEST { if client request contains a HTTP header "Upgrade: WebSocket", it indicate the client intended to switch to WebSocket, we shall disable HTTP filter here it is not very secure as we shall wait for response from server to confirm with status code 101 (Switching Protocols) with HTTP header "Upgrade: WebSocket" but it seems that LTM 10.2.4 HF7 have problem on that status code, so we have to workaround it. use case insensitive matching if { "websocket" eq [string tolower [HTTP::header value "Upgrade"]] } { HTTP::disable } } 2 questions 1. Is there any problem in the support of status code 101 in LTM v10.2.4 HF7? 2. Any existing iRule for supporting WebSocket?
SSL off-loading and secure WebSocket
Hi, We have a Big-IP load balancer, and we are planning to publish a web application that uses secure WebSockets (WSS). We are a little bit concerned about how the load balancer is going to handle this situation, because the SSL offloading. Is there anything special we have to configure or taken care off? Clients will send an HTTPS request with a WebSocket handshake, that includes the HTTP headers "Upgrade:websocket" and "Connection:Upgrade". Will the load balancer populate those headers to the web server? Will the load balancer understand that those connections are persistent and non-HTTP? Thanks.Spring Framework Spring-Messaging Remote Code Execution (CVE-2018-1270 / CVE-2018-1275)
In the recent days a critical vulnerability in Spring framework was published. The vulnerable component is Spring-Messaging which is the Spring implementation of WebSockets, Spring-Messaging uses the STOMP messaging protocol as the subprotocol for WebSockets. The vulnerability allows attackers to run arbitrary code by sending a crafted STOMP WebSocket message. A Proof of Concept code exploiting this vulnerability is already publicly available. Mitigating the vulnerability with BIG-IP ASM BIG-IP ASM customers with BIG-IP versions greater than 12.1.0 which allows WebSocket inspection are already protected against this vulnerability. The exploitation attempt will be detected by existing Java code injection attack signatures which can be found in signature sets that include the “Server Side Code Injection” attack type or “Java Servlets/JSP” System. Also, operatingsystem command execution signatures will detect the attempt to execute operatingsystem commands as the payload for the exploit. Figure 1:Exploit blocked with attack signature 200003437. Figure 2:Exploit blocked with attack signature 200003073. Figure 3:Exploit blocked with attack signature 200002273.
HTML5 WebSockets Illustrates Need for Programmability in the Network
#HTML5 #SDN The increasing use of HTML5 WebSockets illustrates one of the lesser mentioned value propositions of SDN – and ADN: extensibility. It's likely that IT network and security staff would agree that HTML5 WebSockets has the potential for high levels of disruptions (and arguments) across the data center. Developers want to leverage the ability to define their own protocols while reaping the benefits of the HTTP-as-application-transport paradigm. Doing so, however, introduces security risks and network challenges as never-before-seen protocols start streaming through firewalls, load balancers, caches and other network-hosted intermediaries that IT network and security pros are likely to balk at. Usually because they're the last to know, and by the time they do – it's already too late to raise objections. Aside from the obvious "you folks need to talk more" (because that's always been the answer and as of yet has failed to actually occur) there are other answers. Perhaps not turn-key, perhaps not easy, but there are other answers. One of them points to a rarely discussed benefit of SDN that has long been true for ADN but is often overlooked: extensibility through programmability. In addition, leveraging the SDN controller’s centralized intelligence, IT can alter network behavior in real-time and deploy new applications and network services in a matter of hours or days, rather than the weeks or months needed today. By centralizing network state in the control layer, SDN gives network managers the flexibility to configure, manage, secure, and optimize network resources via dynamic, automated SDN programs. Moreover, they can write these programs themselves and not wait for features to be embedded in vendors’ proprietary and closed software environments in the middle of the network. -- ONF, Software-Defined Networking: The New Norm for Networks The ability to alter behavior of any network component in real-time, to make what has been traditionally static dynamic enough to adapt to changing conditions is the goal of many modern technology innovations including SDN (the network) and cloud computing (applications and services). When developers and vendors can create and deploy new protocols and toss them over the wall into a production environment, operations needs the ability to adapt the network and delivery infrastructure to ensure the continued enforcement of security policies as well as provide support to assure availability and performance expectations are met. Doing so requires extensibility in the network. Ultimately that means programmability. EXTENSIBILITY through PROGRAMMABILITY While most of the networking world is focused on OpenFlow and VXLAN and NVGRE and virtual network gateways, the value of the ability to extend SDN through applications seems to be grossly underestimated. The premise of SDN is that the controller's functionality can be extended through specific applications that provide for handling of new protocols, provide new methods of managing flows, and do other nifty things that likely only network geeks would truly appreciate. The ability to extend packet processing and add new functions or support for new protocols rapidly, through software, is a significant part of the value proposition of SDN. Likewise, it illustrates the value of the same capabilities that currently exist in ADN solutions. ADN, too, enables extensibility through programmability. While varying degrees of control and capabilities exist across the ADN spectrum, at least some provide complete programmatic control over traffic management by offering the ability to "plug-in" applications (of a sort) that provide support for application-specific handling or new (and often proprietary) protocols, like those used to exchange data over WebSockets-transported connections. What both afford is the ability to extend the functionality of the network (SDN) or application traffic management (ADN) without requiring upgrades or new products. This has been a significant source of value for organizations with respect to security, who often turn to the ADN solutions topologically positioned in a strategic point of control within the network to address zero-day or emerging exploits for which there are no quick fixes. When it comes to something like dealing with custom (proprietary) application protocols and the use of WebSockets, for which network infrastructure services naturally has no support, the extensibility of SDN and ADN are a boon to network and security staff looking for ways in which to secure and address operational risk associated with new and heretofore unknown protocols. The Need for (HTML5) Speed SPDY versus HTML5 WebSockets Oops! HTML5 Does It Again Reactive, Proactive, Predictive: SDN Models The Next IT Killer Is… Not SDN SDN is Network Control. ADN is Application Control.Is HTML5 the Answer to Mobile's VDI Challenge?
#HTML5 #VDI #mobile Thin is back in. Remote display and control technology is nothing new. For decades we've had the ability to remotely access and control our desktops using a variety of technologies and protocols: X11, VNC, RDP, and Terminal Services to name a few of the more well-known. Remote display and control has been used for a variety of purposes not the least of which is the IT administrator's need to remotely root around on your desktop to troubleshoot an issue. VDI is not the same as remote display and control, though many neophytes might understand it that way. VDI in general requires some form of client application; a client application that currently does not exist for every possible end-user client computing device. Thus VDI is not universally supported on mobile devices, despite a (apparent) desire by end-users for just such capabilities. The choices for vendors of VDI technology are support an ever-increasing array of devices (and operating system versions) or find a device/operating system agnostic technology with which to deliver the desktop experience on a mobile device. Zero-client technology, as many are wont to describe it as. It's really reinventing remote display and control with technology better suited to unmanaged, widely variable endpoints using near-ubiquitous protocols, like HTML. Hidden among the new features and functions for VMware Workstation 9 is one such "new" wheel: WSX. WSX WSX is a prototype of a new VMware Workstation web interface that enables users to access their Shared virtual machines via a web browser on a tablet, smart phone or PC without installing any additional applications or browser plug-ins. This service renders an HTML5 web page that can connect to your Workstation hosts, enumerate the available Shared virtual machines and allow you to power them on and interact with the desktop. Both the Windows .msi and Linux .bundle installations are available for download along with VMware Workstation 9.0 WSX is currently not supported for production environments. The number of devices and browsers available on the market make it extremely difficult to test this feature thoroughly to ensure it works well everywhere. This feature requires a very modern browser that supports HTML5 with WebSockets. VMware recommends using the Google Chrome 17 browser on PCs and the Apple Safari 5 browser on Mac OS hosts and iPads. Currently there are issues using this feature with Microsoft Internet Explorer 10. WSX may work with other browsers and on Android tablets running Ice Cream Sandwich with the latest version of Google Chrome installed, but more testing is required. http://www.vmware.com/support/ws90/doc/workstation-90-release-notes.html#WSX WSX is a web server, a proxy, enabling access to your virtual desktop (provided its running on vSphere/ESXi 5). Clients communicate via WebSockets (HTTP) and the display is rendered via HTML5 graphic elements. VMware clearly indicates this is a prototype, and lists a number of caveats regarding current limitations, but given that similar technology (Web Sockets and HTML5) is used elsewhere in production VMware solutions it seems clear that this early entry into the zero-client VDI game will continue to evolve and mature into a production-ready solution. But what about … Security? Scale? Access control? Good questions, and ones for which there are currently no good answers. The use of WebSockets as a transport medium for what is essentially proprietary protocols crossing domain boundaries is problematic. While access control can ostensibly be handled by existing solutions providing secure remote access today (WebSockets is, after all, initiated via HTTP which has a variety of means by which authentication and authorization can be enforced) integrity of the data may be an issue as well as the potential for exploitation of the web server providing the proxied WSX service as intermediate devices would not be able to parse let alone recognize anomalies in the actual data exchange. Scalability would be constrained to layer 4 (TCP) due to the lack of meaningful (and open) data at layer 7 (application) upon which intermediaries could manage flows and sessions. Persistence to an existing WSX server would necessarily revert to decades old techniques such as IP/Port combinations that have long since proven to be problematic, particularly as such schemes tend to result in unpredictably uneven load distribution. There are a variety of issues raised by the use of WebSockets across networking domains in general, and as a means to enable remote display and control these issues are no different. The advantages, of course, lie in the ability to define a protocol specific to the application (in this case VDI "light") and use a single, long-lived asynchronous control layer that has significant benefits in terms of performance. If the trend toward leveraging HTML5 and in particular WebSockets continues, then WebSockets will likely turn out to be one of the more disruptive technologies on the networking market since … well, since HTTP. The Need for (HTML5) Speed HTML5 WebSockets: High-Speed Infrastructure Integration Bus? SPDY versus HTML5 WebSockets The HTTP 2.0 War has Just Begun Oops! HTML5 Does It Again Y U No Support SPDY Yet? Curing the Cloud Performance ArrhythmiaHTML5 WebSockets: High-Speed Infrastructure Integration Bus?
#HTML5 #SPDY Web 2.0 style integration has become the norm with REST and SOA-based APIs – WebSockets may provide the speed and scale necessary to make it the perfect answer to infrastructure integration challenges One of the concerns that arose early in the days of discussing Infrastructure 2.0, aka Dynamic Infrastructure, was the performance of integration methods. While collaboration across and between infrastructure components as a means to combat volatility and rapidity of change in the infrastructure made sense, what was harder to envision was a way in which such integration could be realized without compromising on performance. Traditional data center infrastructure components were already adopting web-based integration methods – SOA, REST, etc… – but the reliance on HTTP was troubling. The synchronous nature of HTTP posed one challenge, which cascaded into more pedantic issues of idempotency, consistency, and the like. While HTML5 WebSockets may not be wholly appropriate for public-facing web applications for a variety of reasons, it may be not only appropriate but advantageous for use in intra-data center integration. By addressing issues of synchronicity (WebSockets is asynchronous by design) as well as performance through the elimination of HTTP headers and provision for transferring data in binary formats, HTML5 WebSockets combines the simplicity of a REST or SOA-based API with higher performance. This is particularly evident as cloud and virtualization-based architectures continue to integrate more thoroughly to streamline provisioning and management processes and automate operations. As diverse systems attempt to coordinate to enable a comprehensive application architecture deployment – complete with required infrastructure services such as acceleration and optimization as well as load balancing – the need for better performing methods of integration become necessary. But problems abound with traditional web-based integration, especially where bi-directional communication is required. With traditional HTTP, such communication required either polling or two separate communication channels, requiring a significant investment in integration efforts on both sides of the integration. The problem is that web-based integration is primarily one-sided; it’s used to integrate remote content into a site, and therefore the one-sided nature of such a pull of information makes HTTP appropriate for such an integration. With infrastructure integration – particularly when it’s focused on automation and orchestration of on-demand provisioning of services – is not so much pulling information as it is invoking operational tasks. Infrastructure integration is about instructing other components to execute a task, while information integration is about, well, aggregating information. The difference in goals means that traditional web-based integration methods, while preferable for its cross-platform agnosticism and ease of use, is not so desirable when the goal of the integration is part of a larger, orchestrated process. Infrastructure integration needs a message bus. HTML5 WebSockets offers the same agnosticism and ease of use as HTTP, but with the characteristics more often associated with a message bus – a more publish and subscribe-based model than the polling one associated with HTTP and Web 2.0. Thus, infrastructure integration will likely turn to HTML5 WebSockets to realize the high-speed intra-data center collaboration required without sacrificing the simplicity of a web-like integration paradigm. Could SPDY provide the same benefits? Yes and no. SPDY is more geared toward accelerating and optimizing communications occurring across the web, particularly those originating over mobile networks. It does provide performance-enhancing benefits, but it does so without stripping HTTP to a bare-metal set of frames wherein data is wholly contained within frames that have no identifiable markers. That means more work for end-points – which will unduly task resource-constrained mobile devices – that may ultimately result in a net-loss of performance. SPDY maintains a balance between taking advantage of existing, optimized parsers with a reduction in content that addresses latency in transfer times. It’s simply not enough for intra-data center integration needs, but more than enough to provide a performance boost for mobile clients. As noted in SPDY versus HTML5 WebSockets, SPDY is far more appropriate for public-facing web applications than is HTML5 WebSockets for reasons above and beyond performance. But HTML5 WebSockets is far more appropriate as an intra-data center message bus for infrastructure integration. Round peg. Square hole. You know the idiom, and in this case, the shoe fits. WebSockets - Basic Overview and Getting Started - HTML5 Tutorial SPDY versus HTML5 WebSockets Y U No Support SPDY Yet? The HTTP 2.0 War has Just Begun Oops! HTML5 Does It Again SPDY Momentum Fueled by Juggernauts Google SPDY Protocol Would Require Mass Change in Infrastructure Cloud is an Exercise in Infrastructure Integration Infrastructure Integration: Metadata versus APISPDY versus HTML5 WebSockets
#HTML5 #fasterapp #webperf #SPDY So much alike, yet so vastly a different impact on the data center … A recent post on the HTTP 2.0 War beginning garnered a very relevant question regarding WebSockets and where it fits in (what might shape up to be) an epic battle. The answer to the question, “Why not consider WebSockets here?” could be easily answered with two words: HTTP headers. It could also be answered with two other words: infrastructure impact. But I’m guessing Nagesh (and others) would like a bit more detail on that, so here comes the (computer) science. Different Solutions Have Different Impacts Due to a simple (and yet profound) difference between the two implementations, WebSockets is less likely to make an impact on the web (and yet more likely to make an impact inside data centers, but more on that another time). Nagesh is correct in that in almost all the important aspects, WebSockets and SPDY are identical (if not in implementation, in effect). Both are asynchronous, which eliminates the overhead of “polling” generally used to simulate “real time” updates a la Web 2.0 applications. Both use only a single TCP connection. This also reduces overhead on servers (and infrastructure) which can translate into better performance for the end-user. Both can make use of compression (although only via extensions in the case of WebSockets) to reduce size of data transferred resulting, one hopes, in better performance, particularly over more constrained mobile networks. Both protocols operate “outside” HTTP and use an upgrade mechanism to initiate. While WebSockets uses the HTTP connection header to request an upgrade, SPDY uses the Next Protocol Negotiation (proposed enhancement to the TLS specification). This mechanism engenders better backwards-compatibility across the web, allowing sites to support both next-generation web applications as well as traditional HTTP. Both specifications are designed, as pointed out, to solve the same problems. And both do, in theory and in practice. The difference lies in the HTTP headers – or lack thereof in the case of WebSockets. Once established, WebSocket data frames can be sent back and forth between the client and the server in full-duplex mode. Both text and binary frames can be sent full-duplex, in either direction at the same time. The data is minimally framed with just two bytes. In the case of text frames, each frame starts with a 0x00 byte, ends with a 0xFF byte, and contains UTF-8 data in between. WebSocket text frames use a terminator, while binary frames use a length prefix. -- HTML5 Web Sockets: A Quantum Leap in Scalability for the Web WebSockets does not use HTTP headers, SPDY does. This seemingly simple difference has an inversely proportional impact on supporting infrastructure. The Impact on Infrastructure The impact on infrastructure is why WebSockets may be more trouble than its worth – at least when it comes to public-facing web applications. While both specifications will require gateway translation services until (if) they are fully adopted, WebSockets has a much harsher impact on the intervening infrastructure than does SPDY. WebSockets effectively blinds infrastructure. IDS, IPS, ADC, firewalls, anti-virus scanners – any service which relies upon HTTP headers to determine specific content type or location (URI) of the object being requested – is unable to inspect or validate requests due to its lack of HTTP headers. Now, SPDY doesn’t make it easy – HTTP request headers are compressed – but it doesn’t make it nearly as hard, because gzip is pretty well understood and even intermediate infrastructure can deflate and recompress with relative ease (and without needing special data, such as is the case with SSL/TLS and certificates). Let me stop for a moment and shamelessly quote myself from a blog on this very subject, “Oops! HTML5 Does it Again”: One of the things WebSockets does to dramatically improve performance is eliminate all those pesky HTTP headers. You know, things like CONTENT-TYPE. You know, the header that tells the endpoint what kind of content is being transferred, such as text/html and video/avi. One of the things anti-virus and malware scanning solutions are very good at is detecting anomalies in specific types of content. The problem is that without a MIME type, the ability to correctly identify a given object gets a bit iffy. Bits and bytes are bytes and bytes, and while you could certainly infer the type based on format “tells” within the actual data, how would you really know? Sure, the HTTP headers could by lying, but generally speaking the application serving the object doesn’t lie about the type of data and it is a rare vulnerability that attempts to manipulate that value. After all, you want a malicious payload delivered via a specific medium, because that’s the cornerstone upon which many exploits are based – execution of a specific operation against a specific manipulated payload. That means you really need the endpoint to believe the content is of the type it thinks it is. But couldn’t you just use the URL? Nope – there is no URL associated with objects via a WebSocket. There is also no standard application information that next-generation firewalls can use to differentiate the content; developers are free to innovate and create their own formats and micro-formats, and undoubtedly will. And trying to prevent its use is nigh-unto impossible because of the way in which the upgrade handshake is performed – it’s all over HTTP, and stays HTTP. One minute the session is talking understandable HTTP, the next they’re whispering in Lakota, a traditionally oral-only language which neatly illustrates the overarching point of this post thus far: there’s no way to confidently know what is being passed over a WebSocket unless you “speak” the language used, which you may or may not have access to. The result of all this confusion is that security software designed to scan for specific signatures or anomalies within specific types of content can’t. They can’t extract the object flowing through a WebSocket because there’s no indication of where it begins or ends, or even what it is. The loss of HTTP headers that indicate not only type but length is problematic for any software – or hardware for that matter – that uses the information contained within to extract and process the data. SPDY, however, does not eliminate these Very-Important-to-Infrastructure-Services HTTP headers, it merely compresses them. Which makes SPDY a much more compelling option than WebSockets. SPDY can be enabled for an entire data center via the use of a single component: a SPDY gateway. WebSockets ostensibly requires the upgrade or replacement of many more infrastructure services and introduces risks that may be unacceptable to many organizations. And thus my answer to the question "Why not consider WebSockets here” is simply that the end-result (better performance) of implementing the two may be the same, WebSockets is unlikely to gain widespread acceptance as the protocol du jour for public facing web applications due to the operational burden it imposes on the rest of the infrastructure. That doesn’t mean it won’t gain widespread acceptance inside the enterprise. But that’s a topic for another day… HTML5 Web Sockets: A Quantum Leap in Scalability for the Web Oops! HTML5 Does it Again The HTTP 2.0 War has Just Begun Fire and Ice, Silk and Chrome, SPDY and HTTP Grokking the Goodness of MapReduce and SPDY Google SPDY Protocol Would Require Mass Change in Infrastructure
