Scalable AI Deployment: Harnessing OpenVINO and NGINX Plus for Efficient Inference

Introduction

In the realm of artificial intelligence (AI) and machine learning (ML), the need for scalable and efficient AI inference solutions is paramount. As organizations deploy increasingly complex AI models to solve real-world problems, ensuring that these models can handle high volumes of inference requests becomes critical.

NGINX Plus serves as a powerful ally in managing incoming traffic efficiently. As a high-performance web server and reverse proxy server, NGINX Plus is adept at load balancing and routing incoming HTTP and TCP traffic across multiple instances of AI model serving environments.

The OpenVINO Model Server, powered by Intel's OpenVINO toolkit, is a versatile inference server supporting various deep learning frameworks and hardware acceleration technologies. It allows developers to deploy and serve AI models efficiently, optimizing performance and resource utilization. When combined with NGINX Plus capabilities, developers can create resilient and scalable AI inference solutions capable of handling high loads and ensuring high availability. Health checks allow NGINX Plus to continuously monitor the health of the upstream OVMS instances. If an OVMS instance becomes unhealthy or unresponsive, NGINX Plus can automatically route traffic away from it, ensuring that inference requests are processed only by healthy OVMS instances. Health checks provide real-time insights into the health status of OVMS instances. Administrators can monitor key metrics such as response time, error rate, and availability, allowing them to identify and address issues proactively before they impact service performance.

In this article, we'll delve into the symbiotic relationship between the OpenVINO Model Server, and NGINX Plus to construct a robust and scalable AI inference solution. We'll explore setting up the environment, configuring the model server, harnessing NGINX Plus for load balancing, and conducting testing. By the end, readers will gain insights into how to leverage Docker, the OpenVINO Model Server, and NGINX Plus to build scalable AI inference systems tailored to their specific needs.

Flow explanation:

Now, let's walk through the flow of a typical inference request. When a user submits an image of a zebra for inference, the request first hits the NGINX load balancer.
The load balancer then forwards the request to one of the available OpenVINO Model Server containers, distributing the workload evenly across multiple containers.
The selected container processes the image using the optimized deep-learning model and returns the inference results to the user. In this case, the object is named zebra.

OpenVINO™ Model Server is a scalable, high-performance solution for serving machine learning models optimized for Intel® architectures. The server provides an inference service via gRPC, REST API, or C API -- making it easy to deploy new algorithms and AI experiments.

You can visit https://hub.docker.com/u/openvino for reference.

Setting up:

We'll begin by deploying model servers within containers. For this use case, I'm deploying the model server on a virtual machine (VM). Let's outline the steps to accomplish this:

Get the docker image for OpenVINO ONNX run time

docker pull openvino/onnxruntime_ep_ubuntu20

You can also visit https://docs.openvino.ai/nightly/ovms_docs_deploying_server.html for OpenVINO model server deployment in a container environment.
Begin by creating a docker-compose file following the structure below:

https://raw.githubusercontent.com/f5businessdevelopment/F5openVino/main/docker-compose.yml

version: '3'
services:
  resnet1:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9001
    volumes:
      - ./models:/models
    ports:
      - "9001:9001"

  resnet2:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9002
    volumes:
      - ./models:/models
    ports:
      - "9002:9002"

  # Add more services for additional containers
  resnet3:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9003
    volumes:
      - ./models:/models
    ports:
      - "9003:9003"

  resnet4:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9004
    volumes:
      - ./models:/models
    ports:
      - "9004:9004"

  resnet5:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9005
    volumes:
      - ./models:/models
    ports:
      - "9005:9005"

  resnet6:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9006
    volumes:
      - ./models:/models
    ports:
      - "9006:9006"

  resnet7:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9007
    volumes:
      - ./models:/models
    ports:
      - "9007:9007"

  resnet8:
    image: openvino/model_server:latest
    command: >
      --model_name=resnet
      --model_path=/models/resnet50
      --layout=NHWC:NCHW
      --port=9008
    volumes:
      - ./models:/models
    ports:
      - "9008:9008"

Make sure you have Docker and Docker Compose installed on your system.
Place your model files in the `./models/resnet50` directory on your local machine.
Save the provided Docker Compose configuration to a file named `docker-compose.yml`.
Run the following command in the directory containing the `docker-compose.yml` file to start the services:

docker-compose up -d

You can now access the OpenVINO Model Server instances using the specified ports (e.g., `http://localhost:9001` for `resnet1` and `http://localhost:9002` for `resnet2`).

- Ensure that the model files are correctly placed in the `./models/resnet50` directory before starting the services.

Set up an NGINX Plus proxy server.

You can refer to https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-plus/ for NGINX Plus installation also

You have the option to configure VMs with NGINX Plus on AWS by either:

Utilizing the link provided below, which guides you through setting up NGINX Plus on AWS via the AWS Marketplace:
NGINX Plus on AWS Marketplace

Following the instructions available on GitHub at the provided repository link. This repository facilitates spinning up VMs using Terraform on AWS and deploying VMs with NGINX Plus under the GitHub repository - F5 OpenVINO

The NGINX Plus proxy server functions as a proxy for upstream model servers. Within the upstream block, backend servers (model_servers) are defined along with their respective IP addresses and ports. In the server block, NGINX listens on port 80 to handle incoming HTTP/2 requests targeting the specified server name or IP address. Requests directed to the root location (/) are then forwarded to the upstream model servers utilizing the gRPC protocol. The proxy_set_header directives are employed to maintain client information integrity while passing requests to the backend servers. Ensure to adjust the IP addresses, ports, and server names according to your specific setup.

Here is an example configuration that is also available at GitHub https://github.com/f5businessdevelopment/F5openVino

upstream model_servers { server 172.17.0.1:9001; server 172.17.0.1:9002; server 172.17.0.1:9003; server 172.17.0.1:9004; server 172.17.0.1:9005; server 172.17.0.1:9006; server 172.17.0.1:9007; server 172.17.0.1:9008; zone model_servers 64k; } server { listen 80 http2; server_name 10.0.0.19; # Replace with your domain or public IP location / { grpc_pass grpc://model_servers; health_check type=grpc grpc_status=12; # 12=unimplemented proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } }

If you are using gRPC with SSL please refer to the detailed configuration at NGINX Plus SSL Configuration

Here is the explanation:

upstream model_servers { server 172.17.0.1:9001; # Docker bridge network IP and port for your container server 172.17.0.1:9002; # Docker bridge network IP and port for your container .... .... }

This section defines an upstream block named model_servers, which represents a group of backend servers. In this case, there are two backend servers defined, each with its IP address and port. These servers are typically the endpoints that NGINX will proxy requests to.

server { listen 80 http2; server_name 10.1.1.7; # Replace with your domain or public IP

This part starts with the main server block. It specifies that NGINX should listen for incoming connections on port 80 using the HTTP/2 protocol (http2), and it binds the server to the IP address 10.1.1.7. Replace this IP address with your domain name or public IP address.

location / { grpc_pass grpc://model_servers; health_check type=grpc grpc_status=12; # 12=unimplemented proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }

Within the location/block, NGINX defines how to handle requests to the root location. In this case, it's using gRPC (grpc_pass grpc://model_servers;) to pass the requests to the upstream servers defined in the model_servers block. The proxy_set_header directives are used to set headers that preserve client information when passing requests to the backend servers. These headers include Host, X-Real-IP, and X-Forwarded-For. Health checks with type=grpc enable granular monitoring of individual gRPC services and endpoints. You can verify the health of specific gRPC methods or functionalities, ensuring each service component is functioning correctly.

In summary, this NGINX configuration sets up a reverse proxy server that listens for HTTP/2 requests on port 80 and forwards them to backend servers (model_servers) using the gRPC protocol. It's commonly used for load balancing or routing requests to multiple backend servers.

Inference Testing:

This is how you can conduct testing. On the client side, we utilize a script named predict.py. Below is the script for reference

# Import necessary libraries import numpy as np from classes import imagenet_classes from ovmsclient import make_grpc_client # Create a gRPC client to communicate with the server # Replace "10.1.1.7:80" with the IP address and port of your server client = make_grpc_client("10.1.1.7:80") # Open the image file "zebra.jpeg" in binary read mode with open("zebra.jpeg", "rb") as f: img = f.read() # Send the image data to the server for prediction using the "resnet" model output = client.predict({"0": img}, "resnet") # Extract the index of the predicted class with the highest probability result_index = np.argmax(output[0]) # Print the predicted class label using the imagenet_classes dictionary print(imagenet_classes[result_index])

This script imports necessary libraries, establishes a connection to the server at the specified IP address and port, reads an image file named "zebra.jpeg," sends the image data to the server for prediction using the "resnet" model, retrieves the predicted class index with the highest probability, and prints the corresponding class label.

Results:

Execute the following command from the client machine. Here, we are transmitting this image of Zebra to the model server.

python3 predict.py zebra.jpg #run the Inference traffic zebra. The prediction output is 'zebra'.

Let's now examine the NGINX Plus logs

cat /var/log/nginx/access.log 10.1.1.7 - - [13/Apr/2024:00:18:52 +0000] "POST /tensorflow.serving.PredictionService/Predict HTTP/2.0" 200 4033 "-" "grpc-python/1.62.1 grpc-c/39.0.0 (linux; chttp2)"

This log entry shows that a POST request was made to the NGINX server at the specified timestamp, and the server responded with a success status code (200). The request was made using gRPC, as indicated by the user agent string.

Conclusion :

Using NGINX Plus, organizations can achieve a scalable and efficient AI inference solution. NGINX Plus can address disruptions caused by connection timeouts/errors, sudden spikes in request rates, or changes in network topology. OpenVINO Model Server optimizes model performance and inference speed, utilizing Intel hardware acceleration for enhanced efficiency. NGINX Plus acts as a high-performance load balancer, distributing incoming requests across multiple model server instances for improved scalability and reliability. Together, this enables seamless scaling of AI inference workloads, ensuring optimal performance and resource utilization.

You can look at this video for reference: https://youtu.be/Sd99woO9FmQ