Nginx as Reverse Proxy

Question

Hi everyone,We are using Nginx as a reverse proxy which is in front of 2 clustered application and sitting behind a Load Balancer. We are using ip_hash as Load balancing algorithm for session persistance as mentioned in this link&nbsp;but we see that all the requests goes to only one 1 application server due to which load is not getting balanced. In access.log file we see that all the incoming requests consists of only LB ip and not the Real client IP address due to which Nginx assumes all the requests are coming from a single user, hence transferring request to only 1 application server (another sits idle all the time). How can we get the real client IP address so that the ip_hash algorithm can recognize the real client ip instead of LB ip.&nbsp;Alot thanks in advance.

p_kueppers · Answer

You should choose least connection by doing this:upstream tomcat{
         least_conn;
         server 192.168.x.y:8080;
         server 192.168.x.y+1:8080;

}With this nginx will send all requests evenly to both servers and will choose the server with less connections for new ones.&nbsp;But, since nginx cant work with persistance cookies (only nginx plus) it will choose the next connection by the requested ip address and this will always be the LoadBalancer IP. So nginx assumes this is allways the same user and redirects this to always the first choosen backend server.&nbsp;But you can trick this by setting the followingreal_ip_header X-Forwarded-For;
set_real_ip_from IPv4_LB_IP;in the server context (works only if nginx comes with realip module)&nbsp;Try and restart and let me know if this was correct 🙂&nbsp;

nikoolayy1 · Answer

You seem to have a NAT device (router/firewall) or another proxy before the NGINX.
&nbsp;
If it is another proxy as this is easily solvalble with F5 with and universal persistance using the XFF header (https://community.f5.com/t5/technical-forum/enable-source-ip-persistence-based-on-x-forwarded-ip-info/td-p/98748) but with nginix you may need to see if you can set the real client ip address to be based on XFF and then see if ip_hash will work as it should or try other methods like "hash" or "consistent_hash" based on the value of the XFF header.
&nbsp;
https://www.loadbalancer.org/blog/nginx-and-x-forwarded-for-header/
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky
https://www.nginx.com/resources/wiki/modules/consistent_hash/
&nbsp;
&nbsp;
Other than that try maybe least_connections load balancing with session cookie:
&nbsp;
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
&nbsp;
https://www.nginx.com/products/nginx/load-balancing/#session-persistence
&nbsp;
The is another persistance if you do not decrypt the SSL traffic that ssl session persistance by SSL session ID F5 supports this and I do not think Nginx does.
&nbsp;

shubhankarsoni · Answer

Hi, Thanks for your reply.Since it is Nginx Open source hence unable to use "consistent_hash" directive (it gives error " unknown directive "consistent_hash"&nbsp;). Also we checked at the LB end and the client IP address are getting passed from LB via X-Forwarded-For header. We tried configuring the same in nginx config file but still the issue persist.Can you kindly let us know if there are any configuration issues or are we missing something?Here is my configuration, upstream tomcat{
         ip_hash;
         server 192.168.x.y:8080;
         server 192.168.x.y+1:8080;
 
 }
 
 log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$status $body_bytes_sent "$http_referer"'
                       '"$http_user_agent" "$http_x_forwarded_for"'
                         '--"$proxy_add_x_forwarded_for"--';

server {
     listen 192.168.a.b:80;
     server_name example.com www.example.com;
     access_log  /var/log/nginx/access.log  main;
 
     client_max_body_size 5120M;
     add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
     add_header Referrer-Policy "strict-origin";
     add_header X-XSS-Protection "1; mode=block";

location / {
         proxy_pass http://tomcat;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Real-IP $http_x_forwarded_for;
 
     }

##where 192.168.x.y , 192.168.x.y+1 ##are the application servers. 
##And 
##192.168.a.b:80 is the Nginx server ##running on port 80&nbsp;The Sample of output log comes like below10.*.*.* - - [20/Jun/2022:19:43:17 +0530] "GET /x/x/x/x HTTP/1.1" 302 154 "https://x.x.x.x/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36" "117.x.x.x%2"

10.*.*.* - - [20/Jun/2022:19:43:17 +0530] "GET /x/x/x/x HTTP/1.1" 302 154 "https://x.x.x.x/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36" "49.x.x.x%2"

where 10.*.*.* is the IP of the LB and 117.x.x.x, 49.x.x.x are the client IPs.

10.*.*.* remains same in all the subsequent access log.Alot thanks!!

shubhankarsoni · Answer

Hi Nikoolayy1, Thanks for your suggestion!We are using Nginx Open Source so, we are not able to use "consistent_hash" algorithm, though we tried but got the error like "unknown directive consistent_hash". Also, regarding LB we don't have much idea which is sitting above our Nginx proxy servers. But, the LB is forwarding the Client IP address using X-Forwarded-For header. We tried configuring in the Nginx but still the issue persists. Here is our website config file&nbsp;upstream tomcat{
         ip_hash;
         server 192.168.x.y:8080;
         server 192.168.x.y+1:8080;

}

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$status $body_bytes_sent "$http_referer"'
                       '"$http_user_agent" "$http_x_forwarded_for"'
                         '--"$proxy_add_x_forwarded_for"--';

server {
     listen 192.168.a.b:80;
     server_name example.com www.example.com;
     access_log  /var/log/nginx/access.log  main;

client_max_body_size 5120M;
     add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
     add_header Referrer-Policy "strict-origin";
     add_header X-XSS-Protection "1; mode=block";

location / {
         proxy_pass http://tomcat;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Real-IP $http_x_forwarded_for;

}

##where 192.168.x.y , 192.168.x.y+1 are the application servers.
##And
##192.168.a.b:80 is the Nginx server running on port 80 The ouptut in access log file we receive is like below10.*.*.* - - [20/Jun/2022:19:43:17 +0530] "GET /x/x/x/x HTTP/1.1" 302 154 "https://x.x.x.x/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36" "117.x.x.x%2"

10.*.*.* - - [20/Jun/2022:19:43:17 +0530] "GET /x/x/x/x HTTP/1.1" 302 154 "https://x.x.x.x/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36" "49.x.x.x%2"

## where 10.*.*.* is the LB IP and 117.x.x.x,49.x.x.x are the Client IP addresses.

##The access log format is  
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                       '$status $body_bytes_sent "$http_referer"'
                       '"$http_user_agent" "$http_x_forwarded_for"'
                         '--"$proxy_add_x_forwarded_for"--';We require the session persistance/sticky session hence we have chose the ip_hash algorithm.Kindly advise if we are missing something or is there anything else needs to be added.Much Thanks!!

Forum Discussion

Nginx as Reverse Proxy

Recent Discussions

Upgrade F5 BIGIP

DNS HM Probe issue

Is XFF a must for ASM WAF DoS

Switch ssl profile based on weak cipher detection via IRULE

F5 Switches in 'Changes Pending' Status

Related Content

NGINX ingress proxy for Consul Service Mesh

F5 NGINX Plus R33 Release Now Available

F5 SSL Offload behind Nginx Reverse Proxy

F5 Reverse Proxy setup

JWT authorization with NGINX Ingress Controller

ABOUT DEVCENTRAL

RESOURCES

SUPPORT

PARTNERS