Forum Discussion
Chris_Stamm_183
Nimbostratus
Aug 10, 2005RPC load balancing among multiple tiers using iRules, pools and forwarding
Scenario:
We have an app that uses...
TCP port 7496 for one piece.
TCP port 3372 for another piece.
TCP port 135 for DTC that will renegotiate a high port in this case we set the RPC range for AppServerA to 40000 to 40999, AppServerB to 41000 to 41999, AppServerC to 42000 to 42999, AppServerD to 43000 to 43999, AppServerE to 44000 to 44999. We do this so that we can load balance RPC by saying if the port negotiated is between 42000 and 42999 then we know we send that to AppServerD.
Our application is tiered where we have a web layer that is load balanced, an application layer that is load balanced by the above configuration and finally a database layer that does not load balance but does require the same ports as above. The environment looks something like this. BigIP (BigIPWeb) connected on external interface to internet and internal interface to web layer. Bigip (BigIPApp)connected on external interface to web layer and internal interface to app layer. Firewall connected on external interface to app layer and internal interface to database layer.
Incoming web requests are good but here is what I need to do for the app to communicate properly (I am already doing this on 4.5 but 9.x is quite a bit different)...
Web request comes in through BigIPWeb and gets LB'd to WebServerA. WebServerA needs to make request to the application layer VIP on port 7496 initially, then initiates another request to application layer VIP for port 135 (has to be the same VIP for all application layer traffic - not necessarily same node) which will renegotiate depending on the app server that gets the request on 135 - in this instance lets say AppServerA gets the 135 port so the negotiated port must lay between 40000 and 40999. After all that is done one last communication happens which is all from a lower tier to a higher tier i.e. sql to app or app to web.
So to sumarize we need to LB port 7496 to a five nodes, 135 to five nodes, 40000-40999 to AppServerA, 41000 to 41999 to AppServerB, 42000 to 42999 to AppServerC, 43000 to 43999 to AppServerD, 44000 to 4$999 to AppServerE and finally upward traffic on 3372.
I intend on doing this with two iRules which are here...
This is the inbound to app VIP.
APP_POOL will have servers AppServerA, AppServerB, AppServerC, AppServerD, AppServerE.
AppServerA will have AppServerA in pool with port set to 0 (to include all)
AppServerB will have AppServerB in pool with port set to 0 (to include all)
and so on up to AppServerE
VIP is set to this
when CLIENT_ACCEPTED {
if { [TCP::server_port] == 7495 }
{
pool APP_POOL
}
elseif { [TCP::server_port] == 135 }
{
pool APP_POOL
}
elseif { [TCP::server_port] > 39999 and [TCP::server_port] < 41000 }
{
pool AppServerA
}
elseif { [TCP::server_port] > 40999 and [TCP::server_port] < 42000 }
{
pool AppServerB
}
elseif { [TCP::server_port] > 41999 and [TCP::server_port] < 43000 }
{
pool AppServerC
}
elseif { [TCP::server_port] > 42999 and [TCP::server_port] < 44000 }
{
pool AppServerD
}
elseif { [TCP::server_port] > 43999 and [TCP::server_port] < 45000 }
{
pool AppServerE
}
elseif { [TCP::server_port] == 3372 }
{
forward
}
else {
discard
}
}NetworkBasedVIP set to this for access to APP network for DTC forwards and callbacks...
when CLIENT_ACCEPTED {
if { [TCP::server_port] == 7495 }
{
forward
}
elseif { [TCP::server_port] == 135 }
{
forward
}
elseif { [TCP::server_port] > 39999 and [TCP::server_port] < 41000 }
{
forward
}
elseif { [TCP::server_port] > 40999 and [TCP::server_port] < 42000 }
{
forward
}
elseif { [TCP::server_port] > 41999 and [TCP::server_port] < 43000 }
{
forward
}
elseif { [TCP::server_port] > 42999 and [TCP::server_port] < 44000 }
{
forward
}
elseif { [TCP::server_port] > 43999 and [TCP::server_port] < 45000 }
{
forward
}
elseif { [TCP::server_port] == 3372 }
{
forward
}
else {
discard
}
}NetworkBasedVIP set to this for access to DATABASE network for DTC forwards and callbacks...
when CLIENT_ACCEPTED {
if { [TCP::server_port] == 7495 }
{
forward
}
elseif { [TCP::server_port] == 135 }
{
forward
}
elseif { [TCP::server_port] > 39999 and [TCP::server_port] < 41000 }
{
forward
}
elseif { [TCP::server_port] > 40999 and [TCP::server_port] < 42000 }
{
forward
}
elseif { [TCP::server_port] > 41999 and [TCP::server_port] < 43000 }
{
forward
}
elseif { [TCP::server_port] > 42999 and [TCP::server_port] < 44000 }
{
forward
}
elseif { [TCP::server_port] > 43999 and [TCP::server_port] < 45000 }
{
forward
}
elseif { [TCP::server_port] == 3372 }
{
forward
}
else {
discard
}
}13 Replies
- unRuleY_95363Historic F5 AccountI'm not sure if you are asking a question here?
I will make a couple of notes though.
First, TCP::server_port will likely return 0 in the CLIENT_ACCEPTED event. This is because the backend connection has yet to be made and we made the following changes from 4.x:
client = clientside remote
server = serverside remote
remote = clientside source or serverside destination
local = clientside destination or serverside source
So, TCP::server_port refers to the serverside TCP::remote_port or the serverside's destination port. At the time CLIENT_ACCEPTED is evaluated, the proxy hasn't connected the serverside, so this information is not yet available. You probably want to use TCP::local_port instead.
Second, your last two rules look identical and you could probably simply use the same rule for both virtuals.
Third, you could probably use a condensed if expression or a switch statement in the second rule since all the cases use a forward.
As a single if:when CLIENT_ACCEPTED { if { [TCP::local_port] == 135 or \ [TCP::local_port] == 3372 or \ [TCP::local_port] == 7495 or \ ( [TCP::local_port] >= 40000 and [TCP::local_port] < 45000 ) } { forward } else { discard } }
As a switch:when CLIENT_ACCEPTED { switch -glob [TCP::local_port] { 135 - 3372 - 7495 - 4[01234]??? { forward } default { discard } } } - Chris_Stamm_183
Nimbostratus
Ya, my post kept running on and on and I forgot to pose the question. Mainly my question was is this the proper way and is this the most efficient way. Both of which seem to be no. That makes sense on the client/server/remote/local so I will make that change and test it out.
If you have a device on the internal side is there a way to send directly to that machine through a vip. So...
If port equals 40000-40999 send to machine1
If port equals 41000-41999 send to machine2
If port equals 42000-42999 send to machine3
etc.
The only way I know how to do it is to create a pool with a single machine in it but I doubt that is the correct way to do it. So right now I have pool L6APP01 with a single device, pool L6APP02 with a single device, pool L6APP03 with a single device and pool L6APP04 with a single device.
when CLIENT_ACCEPTED { switch -glob [TCP::local_port] {
135 -
3372 -
749[6789] { pool APP_POOL }
40??? { pool L6APP01 }
41??? { pool L6APP02 }
42??? { pool L6APP03 }
43??? { pool L6APP04 }
default { discard }
}
}
Is there a better way?
I did find this but I dont see anywhere in the iRule that specifies the specific node - maybe it's a typo?
Selecting a specific server
As an alternative to the pool command, you can also write an iRule that directs traffic to a specific server within a pool. To do this, you use the node command. Figure 2 shows an example of this command.
Figure 2: Example of the node command within an iRule
iRule my_iRule1 {
when HTTP_REQUEST
if { [IP::remote_addr] ends_with ".gif" } {
node my_pool
}
} - unRuleY_95363Historic F5 AccountYes. I hate to say it, but our docs are very lacking... Depending on which version of the doc you have, it may be way off. We recently made a pass through and tried to correct all the examples, so either you don't have that updated copy or we missed that.
There are actually two methods that operate similarly in this regard. The first and more preferred method is that you can select a specific pool member from within a pool. You do this with some extra arguments to the pool command:
orpool memberpool member :
The <> denote arguments you need to supply.
The advantage to using a pool member is that the member status is checked and adhered to and statistics are updated appropriately. So, if the member is down or the member does not exist, the LB_FAILED event will be fired to allow some corrective action.
The other method, which you found, is directly selecting a node. This basically is similar to rewriting the destination and then forwarding. There are no checks or statistics done on this method, so if the node doesn't exist or is down, the connection will just eventually time out.
The syntax for the node method is:
ornodenode :
(Since you edited out your question about switch -glob, I'm assuming you figured it out). - unRuleY_95363Historic F5 AccountOh, I should add that if your pool members are wildcard port (:0), then you can leave off the port or specify it as 0.
- Chris_Stamm_183
Nimbostratus
For some reason all ports are being allowed through on both circumstances. AKA I can telnet to the APPVIP (169.25.5.23) address that is listening on the external VLAN on tcp port 3389 (remote desktop since I know it is listening on all the devices on the internal VLAN) and a machine responds. I would think it should discard (drop the packet,deny, send a RST). Here is the information I filled out for the VIP and the code for the iRule. As always, your help is much appreciated.
------------------------------------------------------------------------
Name APPVIP
Destination Type: host
Address: 169.25.5.23
Service Port0
State Enabled
Configuration: Advanced
Type Standard
Protocol: TCP
Protocol Profile (Client) tcp
Protocol Profile (Server) (Use Client Profile)
OneConnect Profile None
HTTP Profile None
FTP ProfileNone
SSL Profile (Client) None
SSL Profile (Server) None
Authentication Profiles
Stream Profile None
VLAN Traffic Enabled Oninternal
Connection Limit 0
Address Translation Disabled
Port TranslationDisabled
SNAT Pool None
Clone Pool (Client) None
Clone Pool (Server) None
Last Hop Pool None
THIS iRULE IS ASSOCIATED WITH THE ABOVE VIPwhen CLIENT_ACCEPTED { switch -glob [TCP::local_port] { 135 - 3372 - 749[6789] { pool APP_POOL } 40??? { pool L6APP01 } 41??? { pool L6APP02 } 42??? { pool L6APP03 } 43??? { pool L6APP04 } default { discard } } }
------------------------------------------------------------------------
Name TIER1_PNET_PORTS
Destination Type: Network
Address: 169.25.5.0
Mask: 255.255.255.224
Service Port0
State Enabled
Configuration: Advanced
Type Standard
Protocol: TCP
Protocol Profile (Client) tcp
Protocol Profile (Server) (Use Client Profile)
OneConnect Profile None
HTTP Profile None
FTP ProfileNone
SSL Profile (Client) None
SSL Profile (Server) None
Authentication Profiles
Stream Profile None
VLAN Traffic Enabled Oninternal
Connection Limit 0
Address Translation Disabled
Port TranslationDisabled
SNAT Pool None
Clone Pool (Client) None
Clone Pool (Server) None
Last Hop Pool None
THIS iRULE IS ASSOCIATED WITH THE ABOVE VIPwhen CLIENT_ACCEPTED { switch -glob [TCP::local_port] { 135 - 3372 - 749[6789] - 4[01234]??? { forward } default { discard } } } - unRuleY_95363Historic F5 AccountWhy don't you try adding a log statement before the switch:
when CLIENT_ACCEPTED { log local0. "Client [IP::remote_addr]:[TCP::remote_port] -> [IP::local_addr]:[TCP::local_port]" }
This will help debug what's happening. Log statement output to /var/log/ltm. - Chris_Stamm_183
Nimbostratus
I think what is happening is that since the VIP is configured with an ip address and port set to 0 that the BigIP completes the three-way handshake to any incoming request and if it matches one of my ports it does the load balance or forward depending on my rule. My reason for this thought is that if I telnet from a device on the external VLAN to the VIP (also on external VLAN) on port 3389 the connect happens but if I actually run Terminal Services client that uses 3389 it doesnt actually send it to a device on the internal VLAN. I also did some random ports that I know no internal VLAN machines would be listening on and all of them completed the three-way handshake with the BigIP.
It is a little misleading since I would normally expect any connection not on the specific ports to be dropped.
Do you know of a way to do a drop, deny, reject or something of that nature so that the BigIP will not answer TCP connections aside from the ones in the iRule?
---Perhaps it is the CLIENT_ACCEPTED that is the source of the answer-any-TCP-request. Is there a CLIENT_REQUEST or something that looks at the port number before connection is made and then truely drops/discards so that a TCP connection cannot be made on any port? - Chris_Stamm_183
Nimbostratus
I thought about that but wasn't sure how they tie into the VIP's, into the ability to manage device via browser and SSH. - drteeth_127330Historic F5 AccountBIG-IP can send a reset to an established connection using the reject command. BIG-IP can also silently remove a connection so that subsequent traffic will likely be reset (see the drop command). However, there is no way to not answer a TCP connection aside from a packet filter or deleting the vip. There is a very good reason for this. Relatively speaking, rule processing is fairly expensive. Processing a rule on evey SYN would make BIG-IP extremely susceptible to a SYN flood DoS attack. That would be bad.
- Chris_Stamm_183
Nimbostratus
The rule is still checked after the TCP connection to validate against the iRule. You are still checking the rule either way I would think and if so you would be checking against the rule prior to SYN, SYNACK, ACK and therefore reject before the 3way handshake thus doing less work. I agree about the SYN flood but on the inverse wouldnt you also be equally vulnerable to an established TCP connection or do you ignore any commands issued after the open TCP connection?
Just so I am clear and to avoid looking argumentative, I'm not trying to argue just understand.
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects