Forum Discussion

smp_86112's avatar
smp_86112
Icon for Cirrostratus rankCirrostratus
Oct 21, 2011

Reverse Monitor and Pool Member Recovery

I have constructed a monitor to check a simple page and look for the string "up" in the response. Here's the sanitized definition:


ltm monitor http reverse {
    defaults-from http
    destination *:*
    interval 5
    recv "up"
    reverse enabled
    send "GET /monitor.html HTTP/1.1\r\nHost: app.mydomain.com\r\n\r\n"
    time-until-up 0
    timeout 6
}

In normal mode (i.e. "Reverse" is not enabled), I can change the text and the monitor goes up/down as I expect. Then when I enable the "Reverse" parameter while the page contains "up", the member gets marked down as I expect. However when I then change the HTML to "down" in Reverse mode, the member does not recover.

The doc doesn't really specifically describe what happens when a Reverse monitor recovers, but it seems logical to me that a monitor in Reverse mode should recover when the test fails. Am I not understanding now Reverse mode works? The only think I could find is SOL10443, but I am running 10.2.0.

  • mine seems to be ok. by the way, is timeout 6 correct?

    [root@iris:Active] config  b version|grep -iA 1 version
    BIG-IP Version 10.2.0 1707.0
    Final Edition
    
    [root@iris:Active] config  b pool foo list
    pool foo {
       monitor all mymon
       members 10.10.70.110:http {}
    }
    
    [root@iris:Active] config  b monitor mymon list
    monitor mymon {
       defaults from http
       reverse
       recv "up"
       send "GET /foo.html HTTP/1.1\r\nHost:\r\nConnection: Close\r\n\r\n"
    }
    
    [root@iris:Active] config  curl http://10.10.70.110/foo.html
    
    
    WELCOME!!!!
    This's 10.10.70.110.
    up
    
    
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   inactive,down
    
    [root@iris:Active] config  curl http://10.10.70.110/foo.html
    
    
    WELCOME!!!!
    This's 10.10.70.110.
    down
    
    
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   active,up
    
    
  • I chose 6 seconds just because I was testing and I didn't want to wait.

     

     

    Thanks for posting your config. I copied your test exactly, and it behaved the way I expected. Then I started tweaking it a bit, and found that the Connection: close header is the key. When I remove it from the monitor, the member does not recover when the monitor is in Reverse mode.
  • I did some more testing by adding random characters to the html page (and validating that the receive string was not contained). I found that without the Connection: close header, a response containing less than 1705 bytes (including the headers) would not allow the member to recover. If I add one single caracter more to the page (a total of 1706 or more bytes), the member gets marked up/down as expected.

     

    Can someone else verify?

     

  • mine seems working fine.

    [root@iris:Active] config  b monitor mymon list
    monitor mymon {
       defaults from http
       reverse
       recv "up"
       send "GET /foo.html HTTP/1.1\r\nHost:\r\n\r\n"
    }
    
    [root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  1418  100  1418    0     0   328k      0 --:--:-- --:--:-- --:--:--     0
    up
    
    [root@iris:Active] config  curl -I http://10.10.70.110/foo.html
    HTTP/1.1 200 OK
    Date: Tue, 25 Oct 2011 05:19:31 GMT
    Server: Apache/2.0.59 (rPath)
    Last-Modified: Tue, 25 Oct 2011 05:18:59 GMT
    ETag: "66a9-58a-ad3cf6c0"
    Accept-Ranges: bytes
    Content-Length: 1418
    Vary: Accept-Encoding
    Set-Cookie: testcookie=helloworld
    Content-Type: text/html; charset=UTF-8
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   inactive,down
    
    [root@iris:Active] config  curl -I http://10.10.70.110/foo.html
    HTTP/1.1 200 OK
    Date: Tue, 25 Oct 2011 05:21:00 GMT
    Server: Apache/2.0.59 (rPath)
    Last-Modified: Tue, 25 Oct 2011 05:20:45 GMT
    ETag: "66ae-58c-b38e6540"
    Accept-Ranges: bytes
    Content-Length: 1420
    Vary: Accept-Encoding
    Set-Cookie: testcookie=helloworld
    Content-Type: text/html; charset=UTF-8
    
    [root@iris:Active] config  curl -I http://10.10.70.110/foo.html|grep -i up
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0  1420    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   active,up
    
    
  • Thanks again for posting your test. I calculate the length of your headers to be 290 bytes. So even when I add in the 1418 bytes of content, the total length is 1708 bytes. I would be curious to see the behavior if you removed 100 or so bytes from the content, just to be sure you are well under my theoretical threshold. But I understand if you don't want to do any more - thanks for responses. You were able to help me isolate this further than I would have alone.
  • just a quick question. how come 290 bytes? is it a number of char? i count it 265 bytes. is there anything i missed?

     

     

    by the way, don't worry i can test it. just wanna make sure we are at the same number. ;-)

     

     

  • You have to think about this in terms of what's transmitted on the wire, so characters count, spaces count, and so do CRLF characters. I count 291 bytes in the header below, if you include a standard blank line which seperates the headers from the content. I pasted into a text file and ran the wc command.

    
    HTTP/1.1 200 OK
    Date: Tue, 25 Oct 2011 05:21:00 GMT
    Server: Apache/2.0.59 (rPath)
    Last-Modified: Tue, 25 Oct 2011 05:20:45 GMT
    ETag: "66ae-58c-b38e6540"
    Accept-Ranges: bytes
    Content-Length: 1420
    Vary: Accept-Encoding
    Set-Cookie: testcookie=helloworld
    Content-Type: text/html; charset=UTF-8
    
    
  • this is when it is not working. if i add one more char in the response, it starts working.

    [root@iris:Active] config  b monitor mymon list
    monitor mymon {
       defaults from http
       reverse
       recv "up"
       send "GET /foo.html HTTP/1.1\r\nHost:\r\n\r\n"
    }
    
    [root@iris:Active] config  curl -I http://10.10.70.110/foo.html
    HTTP/1.1 200 OK
    Date: Tue, 25 Oct 2011 06:42:24 GMT
    Server: Apache/2.0.59 (rPath)
    Last-Modified: Tue, 25 Oct 2011 06:41:35 GMT
    ETag: "66a9-57a-d4a385c0"
    Accept-Ranges: bytes
    Content-Length: 1402
    Vary: Accept-Encoding
    Set-Cookie: testcookie=helloworld
    Content-Type: text/html; charset=UTF-8
    
    [root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  1402  100  1402    0     0   292k      0 --:--:-- --:--:-- --:--:--  250k
    up
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   inactive,down
    
    [root@iris:Active] config  curl -I http://10.10.70.110/foo.html
    HTTP/1.1 200 OK
    Date: Tue, 25 Oct 2011 06:43:03 GMT
    Server: Apache/2.0.59 (rPath)
    Last-Modified: Tue, 25 Oct 2011 06:42:57 GMT
    ETag: "66ae-57c-d986be40"
    Accept-Ranges: bytes
    Content-Length: 1404
    Vary: Accept-Encoding
    Set-Cookie: testcookie=helloworld
    Content-Type: text/html; charset=UTF-8
    
    [root@iris:Active] config  curl -i http://10.10.70.110/foo.html|grep -i up
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  1404  100  1404    0     0   173k      0 --:--:-- --:--:-- --:--:--     0
    
    [root@iris:Active] config  b pool foo|grep -i pool\ member
    +-> POOL MEMBER foo/10.10.70.110:http   inactive,down
    
    
  • oop sorry. i forgot to say thanks for the byte's explanation. ;-)
  • No problem, thank you for all your independent test work. So it seems your threshold is 1695 bytes (1404 content + 291 headers), and mine is 1705 bytes. But in my mind, that's close enough to convince me this is a bug. It's not exactly the situation described in SOL10443, but it's evident there was some type of problem with that part of the TMM code.