# How I did it - "Securing Nvidia Triton Inference Server with NGINX Plus Ingress Controller” **URL:** \ **Category:** F5 Technical Articles\ **Tags:** nginx-plus, devops, security, application-delivery, nginx-app-protect\ **Created:** [April 25, 2024, 12:00pm UTC](https://community.f5.com/t/how-i-did-it-securing-nvidia-triton-inference-server-with-nginx-plus-ingress-controller/74369 "2024-04-25T12:00:00Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Greg\_Coward](https://d1p9zq3aats0t8.cloudfront.net/user_avatar/community.f5.com/greg_coward/32/5566_2.png) [@Greg\_Coward](https://community.f5.com/u/Greg_Coward)\ **Post date:** [April 25, 2024, 12:00pm UTC](https://community.f5.com/t/how-i-did-it-securing-nvidia-triton-inference-server-with-nginx-plus-ingress-controller/74369/1 "2024-04-25T12:00:00Z")
Welcome to another edition of “How I Did It” where we explore a variety of F5 and partner technologies and dive into the details of their implementation; well at least how I do it 😀. In this installment, we step into the world of AI and Machine learning (ML) and take a look at how F5’s [NGINX Plus Ingress Controller](https://www.nginx.com/products/nginx-ingress-controller/) can provide secure and scalable external access to Nvidia’s Triton Inference Servers hosted on Kubernetes. [NVIDIA Triton Inference Server](https://www.nvidia.com/en-us/ai-data-science/products/triton-inference-server/) is a powerful tool for deploying machine learning models in production environments, specifically designed to run on Kubernetes. It plays a crucial role in AI and machine learning by serving as a platform for deploying trained models into production environments. It allows these models to process incoming data and generate predictions or insights in real-time.  The platform supports various deep learning frameworks such as TensorFlow, PyTorch, and ONNX Runtime. Triton manages model deployment, scaling, and versioning, providing a seamless experience for developers and operators. [NGINX Plus Ingress Controller](https://www.nginx.com/products/nginx-ingress-controller/) is architected to provide and manage external connectivity to applications running on Kubernetes. It enhances the standard Kubernetes Ingress capabilities by providing advanced features such as SSL/TLS termination, traffic throttling, and advanced routing based on request attributes. NGINX Plus can also integrate with external authentication services and provide enhanced security features like [NGINX App Protect](https://www.nginx.com/products/nginx-app-protect/).  The controller is designed to improve the performance, reliability, and security of applications deployed on Kubernetes, making it a popular choice for managing ingress traffic in production environments. ## Deployment Overview The self-guided demonstration provides a working example of how NGINX Plus Ingress Controller can provide secure external access -as well as load balancing- to a Kubernetes-hosted [NVIDIA Triton Inference Server cluster](https://www.nvidia.com/en-us/ai-data-science/products/triton-inference-server/) (see below). ![image_329187.png](https://d20hrnpixdzcsd.cloudfront.net/original/2X/7/77f6a7023ee839976fe5d6f9f08085cd81cd27f1.png) Demonstration resources and step-by-step guidance are available on [GitHub](https://github.com/f5devcentral/Triton-Server-NGINX-Plus-Ingress-Controller).  The repository is based on forks from both the NVIDIA [Triton Inference Server repo](https://github.com/triton-inference-server/server) and [NGINX Plus Ingress Controller](https://github.com/nginxinc/kubernetes-ingress). The repo includes a [Helm chart](https://helm.sh/) along with instructions for installing a scalable NVIDIA Triton Inference Server and NGINX+ Ingress Controller in an on-premises or cloud-based Kubernetes cluster. ![image_329187.png](https://d20hrnpixdzcsd.cloudfront.net/original/2X/0/05789900a6cb85001cde60f20cddd60815410469.png) In addition to the Triton Inference server and ingress controller, the repo deploys [Promethues](https://prometheus.io/) and [Grafana](https://grafana.com/grafana/), (along with the NGINX Plus dashboard) to aid in autoscaling and provide visibility into the Triton server service as well as monitoring of ingress/egress traffic (see below). ![image_329187.png](https://d20hrnpixdzcsd.cloudfront.net/original/3X/3/f/3ff901e3c06437f7ef7fcdbb8aeab85108024942.png) ## Check it Out Want to get a feel for it before putting hands to keys?  The video below provides a step-by-step walkthrough of configuring and deploying the demonstration environment. ## Try it Out Liked what you saw? If that’s the case, (as I hope it was) try it out for yourself. The F5Devcentral [repo](https://github.com/f5devcentral/Triton-Server-NGINX-Plus-Ingress-Controller) includes nearly everything you need (with the exception of an NGINX Plus license and certificates) to deploy a fully functioning environment; all you need is a Kubernetes platform to deploy to host the deployment.  With respect to licensing, if needed, F5 provides a fully functional [30-day trial license](https://www.nginx.com/free-trial-connectivity-stack-kubernetes/). ## Additional Links [Triton-server & Nginx Plus Ingress Controller Demo Repo](https://github.com/f5devcentral/Triton-Server-NGINX-Plus-Ingress-Controller/tree/main) [Nvidia Triton Inference Server Overview](https://www.nvidia.com/en-us/ai-data-science/products/triton-inference-server/) [Nvidia Triton Inference Server Repos](https://github.com/triton-inference-server) [NGINX Plus Ingress Controller Overview](https://www.nginx.com/products/nginx-ingress-controller/) [Nginx Ingresss Controller on Kubernetes Repo](https://github.com/nginxinc/kubernetes-ingress) [NGINX App Protect](https://www.nginx.com/products/nginx-app-protect/)