How to rewrite a path to a backend service dropping the prefix and passing the remaining path?

[michaelgardner]

Hello,
I am not sure whether my posting is appropriate in this area, so please delete it if there is a violation of posting rules...

This must be a common task, but I cannot figure out how to do the following fanout rewrite in our nginx ingress:
http://abcccc.com/httpbin/anything ->   /anything (the httpbin backend service)

When I create the following ingress with a path of '/' and send the query, I receive a proper response.

curl -I -k http://abczzz.com/anything

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mikie-ingress
  namespace: mikie
spec:
  ingressClassName: nginx
  rules:
    - host: abczzz.com
      http:
        paths:
          - path: / 
            pathType: Prefix
            backend:
              service:
                name: httpbin-service
                port:
                  number: 8999

What I really need is to be able to redirect to different services off of this single host, so I changed the ingress to the following, but the query always fails with a 404. Basically, I want the /httpbin to disappear and pass the path onto the backend service, httpbin.

curl -I -k http://abczzz.com/httpbin/anything

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mikie-ingress
  namespace: mikie
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  rules:
    - host: abczzz.com
      http:
        paths:
          - path: /httpbin(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: httpbin-service
                port:
                  number: 8999

Thank you for your time and interest,

Mike

[xuwen]

when HTTP_REQUEST {
 set uri [HTTP::uri]
 if { $uri contains "anything" } {
     HTTP::uri "/[join [lrange [split $uri "/"] 2 end] "/"]"
 }
}

[JRahm]

That seems to match the syntax on the ingress example. A quick search on stack seems to also support the same syntax. Are you on at least ingress controller 2.0 and kubernetes 1.22? the referenced API version doesn't seem to be compatible earlier.

[michaelgardner]

Hello xuwen & JRahm,

Thank you fo your quick responses and help.  After a break, I adjusted the path, as below, and was able to successfully route to httpbin and preserve the path after the prefix.  Hopefully, this will help others... 🙂

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mikie-ingress
  namespace: mikie
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  rules:
    - host: abczzz.com
      http:
        paths:
          - backend:
              service:
                name: httpbin-service
                port:
                  number: 8999
            path: /httpbin/(.*)
            pathType: Prefix

[michaelgardner]

In order to compare my issues, I deployed nginx-ingress, which we will be using for our deployment, and the community edition, to two separate Kubernetes clusters.  When I reported success a few minutes ago, it was with the community edition.  However, with nginx-ingress and the same manifests used for the community edition, the rewrites are not succeeding... 404s...  Arggg... 

I've attached my deployment scripts, which I was documenting for our team.  Could you please take a look at them to see whether you spot any reasons that they wouldn't work, as expected?  I am going to dig back in when I get back into the office in the morning and will report what I find.  I've left the httpbin backend services running, in case you want to play with anything.  No issues with your access, if desired.

Thank you for your help and support,

Mike

[JRahm]

I'll try to track down some help for you, @michaelgardner. In the meantime, can you clarify if you are using the kuburnetes flavor of nginx ingress, or the nginx ingress for kubernetes? Clear as mud, I know...this might help: https://www.nginx.com/blog/guide-to-choosing-ingress-controller-part-4-nginx-ingress-controller-opti....

Examples of rewrites from the nginx ingress BY nginx: https://github.com/nginxinc/kubernetes-ingress/tree/v2.2.2/examples/rewrites

[JRahm]

ok, maybe you're using the kubernetes version of the nginx ingress instead of the nginx version of the nginx ingress. Give a read-through on that article I linked and take a look at the examples. Not telling you which version to use, but the expertise around here will be more geared toward the nginx version.

[michaelgardner]

Hi, Actually, I have both deployed to separate Kubernetes clusters at Digital Ocean...  One cluster has the community version of nginx (aboh-cluster) and the other cluster (abitofhelp-cluster) has nginx.com. In both cases, I've deployed the ingress.  For the community version, I used DigitalOceans "MarketPlace" installer.  For the commercial version, I installed using the helm chart without any alterations.

The use case that I've shown is such a common one, that having this discussion in one location with the same set of manifests configuring the deployment/service and ingress for the commercial and community editions will be very helpful for others.  Although nginx is very popular and stable, which is a good thing, the downside is that there is a lot of incorrect and out-of-date information on the Net.

  I am going to repeat my tests today and may start over with fresh deployments of everything, but use the manifests that I have shared since they worked in my previous testing on the community version.  In the commercial installation, there were no customizations made.  In the community installation, I'd have to dig into DigitalOcean's scripting or contact them for details.  I'd expect that they are plain-jane since they could have such a large variety of users of the service.

I agree that things get a bit cloudy, sometimes...  commercial... community...  ingress... VMs... standalone... Arggggg!  Thankfully, nginx is a stable product with a lot of interested supporters.  

Thank you for any help that you can provide.  If any of this is falling outside of the boundaries of this forum, please let me know and we can drop the investigation here.  

[michaelgardner]

So, I started from scratch at Digital Ocean... I created two Kubernetes clusters using their default configuration:
  * aboh-cluster is the community version of nginx ingnress
 * abitofhelp-cluster is the commercial version of nginix ingress

Nginx community: helm 4.1.3, ingress-nginx 1.21, nginx 1.19.10
Nginx commercial: helm 2.2.2, ingress 1.22

Unfortunately, the outcome is the same as in my previous test.  The community version works with the deployment/service and ingress manifests.  The commercial version with 404 errors using the same manifests and curl commands.  

I have captured the following in the attached compressed file:

* Descriptions of the services and ingresses;
* Manifests for the deployments, services, and ingresses;
* Logs capturing the curl command and the result 

I am not sure where to go from here...  We want to use the commercial version with support, but cannot seem to get it to work.  

Thanks for any hints or help, 

Mike

[BrianEhlert]

Hey Michael
The first thing that I will mention is that annotations are unique to each implementation.
The annotations specific to the community ingress controller are not implemented by other projects.  This is probably the first hurdle you are hitting.

This block here: 

 nginx.ingress.kubernetes.io

Defines that your annotation is specific to the community project.

Rewrites for the NGINX Inc ingress controller project have an example here for the Ingress object: https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/rewrites

And an example here using the project specific CRDs: https://github.com/nginxinc/kubernetes-ingress/tree/main/examples/custom-resources/rewrites

[JSON-Williams_NGIИX]

@Rob_Michel As @BrianEhlert mentioned, you need to change your annotation and Ingress resource slightly to match the commercial version of NGINX Ingress controller.

You can try something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx.org/rewrites: "serviceName=httpbin-service rewrite=/"
spec:
  rules:
  - host: cat.abitofhelp.io
    http:
      paths:
      - path: /httpbin/
        pathType: Prefix
        backend:
          service:
            name: httpbin-service
            port:
              number: 8999

Make sure you adjust accordingly to match your environment.

[michaelgardner]

Hi Guys,

Thank you very much for your detailed responses!  I am traveling today but will make the changes when I get to the office and will report back.   If things go well, I will put together a simple demo that implements current best practices for gateway security and routes gRPC and HTTP->HTTPS traffic.  I will use it for a presentation to Management and will post it here and in a few other locations to help others kick-start a demo.

We are currently using Emissary Ingress and are tired of wrangling poor documentation, bugs, and lack of quality community/support.  My hope is to get the green light to migrate to nginx.com.

I will report back and want to thank you for your help!

Mike

[michaelgardner]

Since haproxy is based upon nginx, I set up the same environment and test in a new cluster at Digital Ocean.  Unfortunately, the tests failed as with the commercial version.  Attached is an updated compressed file including the haproxy work.

[michaelgardner]

Hi guys, Just checking in... Any news?

[JRahm]

you see @BrianEhlert and @JSON-Williams_NGIИX 's posts below?

[Rebecca_Moloney]

Just popping in with more NGINX Ingress related content in case you're interested: 

F5 Hybrid Security Architectures (Part 3 - F5 XC API Protection and NGINX Ingress Controller)

How to deploy NGINX App Protect WAF on the NGINX Ingress Controller using argoCD

NGINX ingress proxy for Consul Service Mesh

Using NGINX Plus Ingress Controller Image from AWS Marketplace with EKS and Private Offers

Deploying NGINX Ingress Controller with OpenShift on AWS Managed Service: ROSA

Mastering API Architectures Ebook from NGINX

Better together - F5 Container Ingress Services and NGINX Plus Ingress Controller Integration