Protecting gRPC based APIs with NGINX App Protect

gRPC support on NGINX

Developed back in 2015, gRPC keeps attracting more and more adopters due to the use of HTTP/2.0 as efficient transport, tight integration with interface description language (IDL), bidirectional streaming, flow control, bandwidth effective binary payload, and a lot more other benefits. About two years ago NGINX started to support gRPC (link) as a gateway. However, the market quickly realized that (like any other gateway) it is subject to cyber-attacks and requires strong defense. As a response for such challenges, App Protect WAF for NGINX just released a compelling set of security features to defend gRPC based services.

gRPC Security

It is a fact that App Protect for NGINX provides much more advanced security and performance than any ModSecurity based WAFs (most of the WAF market). Hence, even before explicit gRPC support, App Protect armory in conjunction with NGINX itself could protect web services from a wide variety of threats like:

Injection attacks
Sensitive data leakage
OS Command execution
Buffer Overflow
Threat Campaigns
Authentication attacks
Denial-of-service
and more (link)

With gRPC support, App Protect provides an even deeper level of security. The newly added gRPC content profile allows to parse binary payload, make sure there is no malicious data, and ensures its structure conforms to interface definition (protocol buffers) (link).

gRPC Profile

Similar to JSON and XML profiles gRPC profile attaches to a subset of URLs and serves to define and enforce a payload structure. gRPC profile extracts application URLs and request/response structures from Interface Definition Language (IDL) file. IDL file is a mandatory part of every gRPC based application. Following policy listing shows an example of referencing the IDL file from a gRPC profile.

  {
    "policy": {
        "name": "online-boutique-policy",
        "grpc-profiles": [
            {
                "name": "hipstershop-grpc-profile",
                "defenseAttributes": {
                    "maximumDataLength": 100,
                    "allowUnknownFields": true
                },
                "idlFiles": [
                    {
                        "idlFile": {
                            "$ref": "file:///hipstershop/demo.proto"
                        },
                        "isPrimary": true
                    }
                ]
                ...omitted...
            }
        ],
        ...omitted...
    }
}

gRPC profile references the IDL file to extract all required data to instantiate a positive security model. This means that all URLs and payload formats from it will be considered as valid and pass.

To catch anomalies in the gRPC traffic, App Protect introduces three kinds of violations. Requests that don't match IDL trigger "VIOL_GRPC_MALFORMED" or "VIOL_GRPC_METHOD". Requests with unknown or longer than allowed fields cause "VIOL_GRPC_FORMAT" violation.

In addition to the above checkups, App Protect looks up signatures or disallowed meta characters in gRPC data. Because of this, it protects applications from a wide variety of attacks that worked for plain HTTP traffic. The following listing gives an example of signatures and meta characters enforcement in gRPS profile (docs).

"grpc-profiles": [
            {
                "name": "online-boutique-profile",
                "attackSignaturesCheck": true,
                "signatureOverrides": [
                    {
                        "signatureId": 200001213,
                        "enabled": false
                    },
                    {
                        "signatureId": 200089779,
                        "enabled": false
                    }
                ],
                "metacharCheck": true,
                ...omitted...
            }
        ],

Example Sandbox

Here is an example of how to configure NGINX as a gRPC gateway and defend it with the App Protect. As a demo application, I use "Online Boutique" (link). This application consists of multiple micro-services that talk to each other in gRPC. The picture below represents the structure of entire application.

Picture 1.

I slightly modified the application deployment such that the frontend doesn't talk to micro-services directly but through NGINX gateway that proxies all calls.

Picture 2.

Before I jump to App Protect configuration, here is how NGINX config looks like for proxying gRPC services.

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    sendfile on;
    keepalive_timeout 65;

    include /etc/nginx/conf.d/upstreams.conf;
    
    server {
        server_name boutique.online;

        listen 443 http2 ssl;
        ssl_certificate /etc/nginx/ssl/nginx.crt;
        ssl_certificate_key /etc/nginx/ssl/nginx.key;
        ssl_protocols TLSv1.2 TLSv1.3;

        include conf.d/errors.grpc_conf;
        default_type application/grpc;

        app_protect_enable on;
        app_protect_policy_file "/etc/app_protect/conf/policies/online-boutique-policy.json";
        app_protect_security_log_enable on;
        app_protect_security_log "/opt/app_protect/share/defaults/log_grpc_all.json" stderr;

        include conf.d/locations.conf;
    }
}

The listing above is self-explaining. NGINX virtual server "boutique.online" listens on port 443 for HTTP2 requests. Requests are routed to one of a micro-service from "upstreams.conf" based on the map defined in "locations.conf". Below are examples of these config files.

$ cat conf.d/locations.conf
upstream adservice {
    server 10.101.11.63:9555;
}
location /hipstershop.CartService/ {
    grpc_pass grpc://cartservice;
}
...omitted...

$ cat conf.d/uptsreams.conf
location /hipstershop.AdService/ {
    grpc_pass grpc://adservice;
}
upstream cartservice {
    server 10.99.75.80:7070;
}
...omitted...

For instance, any call with URL starting from "/hipstershop.AdService" is routed to "adservice" upstream and so on. For more details on NGINX configuration for gRPC refer to this blog article or official documentation.

App Protect is enabled on the virtual server. Hence, all requests are subject to inspection. Let's take a closer look at the security policy applied to the virtual server above.

{
    "policy": {
        "name": "online-boutique-policy",
        "grpc-profiles": [
            {
                "name": "online-boutique-profile",
                "idlFiles": [
                    {
                        "idlFile": {
                            "$ref": "https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/master/pb/demo.proto"
                        },
                        "isPrimary": true
                    }
                ]
                "associateUrls": true,
                "defenseAttributes": {
                    "maximumDataLength": 10000,
                    "allowUnknownFields": false
                },
                "attackSignaturesCheck": true,
                "signatureOverrides": [
                    {
                        "signatureId": 200001213,
                        "enabled": true
                    },
                    {
                        "signatureId": 200089779,
                        "enabled": true
                    }
                ],
                "metacharCheck": true,
                
            }
        ],
        "urls": [
            {
                "name": "*",
                "type": "wildcard",
                "method": "*",
                "$action": "delete"
            }
        ]
    }
}

The policy has one gRPC profile called "online-boutique-profile". Profile references IDL file for the demo application (similar to open API file reference) as a source of the application structure. "associateUrls: true" directive instructs App Protect to extract all possible URLs from IDL file and enforce parent profile on them. Notice URL section removes wildcard URL "*" from the policy to only allow URLs that are in IDL and therefore establish a positive security model.

"defenseAttributes" directive enforces payload length and tolerance to unknown parameters. "attackSignaturesCheck" and "metaCharactersCheck" directives look for a malicious pattern in the entire request. Now let's see what does this policy block and pass.

Experiments

First of all, let's make sure that valid traffic passes. As an example, I construct a valid call to "Ads" micro-service based on IDL content below.

syntax = "proto3";
	
package hipstershop;

service AdService {
	    rpc GetAds(AdRequest) returns (AdResponse) {}
	}
	
	message AdRequest {
	    repeated string context_keys = 1;
	}

Based on the definition above a valid call to the service should go to "/hipstershop.AdService/GetAds" URL and contain "context_keys" identifiers in a payload. I use the "grpcurl" tool to construct and send the call that passes.

$ grpcurl -proto ../microservices-demo/pb/demo.proto -d '{"context_keys": "example"}' boutique.online:8443 hipstershop.AdService/GetAds 

{
 "ads": [
  {
   "redirectUrl": "/product/2ZYFJ3GM2N",
   "text": "Film camera for sale. 50% off."
  },
  {
   "redirectUrl": "/product/0PUK6V6EV0",
   "text": "Vintage record player for sale. 30% off."
  }
 ]
}

As you may note, the URL constructs out of package name, service name, and method name from IDL. Therefore it is expected that all calls which don't comply IDL definition will be blocked.

A call to invalid service.

$ curl -X POST -k --http2 -H "Content-Type: application/grpc" -H "TE: trailers" https://boutique.online:8443/hipstershop.DoesNotExist/GetAds

<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<br><br>Your support ID is: 16472380185462165521<br><br><a href='javascript:history.back();'>[Go Back]</a></body></html>

A call to an unknown service. Notice that the previous call got "html" response page when this one got a special "grpc" response. This happened because only valid URLs considered as type gRPC others are "html" by default.

A call to an invalid method.

$ curl -v -X POST -k --http2 -H "Content-Type: application/grpc" -H "TE: trailers" https://boutique.online:8443/hipstershop.AdService/DoesNotExist

< HTTP/2 200 
< content-type: application/grpc; charset=utf-8
< cache-control: no-cache
< grpc-message: Operation does not comply with the service requirements. Please contact you administrator with the following number: 16472380185462166031
< grpc-status: 3
< pragma: no-cache
< content-length: 0

A call with junk in the payload.

curl -v -X POST -k --http2 -H "Content-Type: application/grpc" -H "TE: trailers" https://boutique.online:8443/hipstershop.AdService/GetAds -d@trash_payload.bin
< HTTP/2 200 
< content-type: application/grpc; charset=utf-8
< cache-control: no-cache
< grpc-message: Operation does not comply with the service requirements. Please contact you administrator with the following number: 13966876727165538516
< grpc-status: 3
< pragma: no-cache
< content-length: 0

All gRPC wise invalid calls are blocked.

In the same way attack signatures are caught in gRPC payload ("alert() (Parameter)" signature).

$ grpcurl -proto ../microservices-demo/pb/demo.proto -d '{"context_keys": "example"}' boutique.online:8443 hipstershop.AdService/GetAds 

{
 "ads": [
  {
   "redirectUrl": "/product/2ZYFJ3GM2N",
   "text": "Film camera for sale. 50% off."
  },
  {
   "redirectUrl": "/product/0PUK6V6EV0",
   "text": "Vintage record player for sale. 30% off."
  }
 ]
}

Conclusion

With gRPC support App Protect provides even deeper controls for gRPC traffic along with all existing security inventory available for plain HTTP traffic. Keep in mind that this release only supports Unary gRPC traffic and doesn't support the server reflection feature. Refer to the official documentation for detailed information on gRPC support (link).

Published Mar 29, 2021

Version 1.0