Forum Discussion

cpt_ri_F5's avatar
cpt_ri_F5
Icon for Cirrostratus rankCirrostratus
Jun 13, 2023

Download file from AWS S3 bucket / http profile

Hello,
I'm trying to download a file from AWS S3 bucket (backend) through a standard VS (HTTP/iRule) but it doesn't work. It downloads an empty/corrupt file.
 
"curl path/file" works from F5.
 
I tried to deactivate the http profile, but still ko!
when HTTP_REQUEST {
    if { [class match [string tolower [HTTP::uri]] contains uri-data] } {
         HTTP::disable
         pool MYPOOL
}
}
when HTTP_RESPONSE {
         HTTP::disable
}
 
Someone has experienced this issue?
 
Thank you for your help
 
  • Hello,

    Resolved, add only : HTTP::header replace Host "hostname"

    FYI : url is : hostname.bucket.xxxxxxaws......

  • Hi cpt_ri_F5 , 
    Could you please disable any AWAF/ASM Policies if exists , Remove attached iRule and convert Virtual server type to performance layer 4 , If the issue is resolved so , Keep disabling ASM policy and attach http profile again , and test one more time. 
    If it works with http profile , so you have issue with AWAF policy and need to check yout policy and AWAF system Variables. 

    If the Download didn't work so you the issue is with http profile , and you need really to disable it , in this Case I recommend to add this : 

     

     

    when HTTP_REQUEST {
        if { [class match [string tolower [HTTP::uri]] contains uri-data] } {
             HTTP::disable
             log local0. "Condition True in Request" 
             return
    }
    }
    when HTTP_RESPONSE {
             HTTP::disable
             log local0. "Response Action achieved" 
    }

     

     

    > I addedd [Log local ] as a guide to make sure that your requests match with Conditions of if statement , and to make sure that http profile really disabled. 
    > I added Return to break the irule , and prefer to attach the Pool "MYPOOL" under virtual server resources. 

    The Last thing if this available with you, Take a Packet capture for both sides ( Client and server side ) with identifing the Download file request to see why it gives you an empty/corrupted file.

    Follow the above steps , I hope this helps you

    • whisperer's avatar
      whisperer
      Icon for MVP rankMVP

      Just like Mohamed_Ahmed_Kansoh said, you really need to look closer in what variables you can eliminate, and the actual data from TCP dump. Open in wireshark, and check to see that the HTTP request leaves the F5 (sounds like it does since you get an AWS error messsage). Look at the host header, make sure it is correct. Also, look at the URI and make sure it is correct on the server side of the connection. Also, make sure you are using the same protocol -- HTTP and HTTP or HTTPS and HTTPS, if you are doing a conversion it may not work if AWS is not listening on 80 or 443.

      Just a few more nuggets to dig and check 😉 Definitely an interesting issue!

  • Hello,

    Resolved, add only : HTTP::header replace Host "hostname"

    FYI : url is : hostname.bucket.xxxxxxaws......

  • So, 2 questions here:
    - Does datagroup 'uri-data' actually exist? Do you have the URI path there for the file you are looking to download?
    - The HTTP::disable in the HTTP_RESPONSE will trigger on EVERY request, regardless if there is a match in the HTTP_REQUEST. You need to specify a variable in the HTTP_REQUEST and check it in the HTTP_RESPONSE. I presume that when that particular URI is matched, you only then want to disable the HTTP profile for that session -- both client side (http_request) and server side (http_response.

    • cpt_ri_F5's avatar
      cpt_ri_F5
      Icon for Cirrostratus rankCirrostratus

      Hello,

      - Yes, the 'uri-data' datagroup exists (also tested with fixed uri in code)

      - Same problem, HTTP_RESPONSE with matched URI !

      I presume that when that particular URI is matched, you only then want to disable the HTTP profile for that session -- both client side (http_request) and server side (http_response.

      => I don't know how it works, I found this solution in the two links shared, I'm interested in any other solutions.

      My question, is there a special configuration when trying to view or download a file (http://VS_F5/path_file) with a Standard VS ?

      I would also like to confirm: I access in http and the backend servers are in https, correct : no client ssl, with default serverssl ?

      Thank you

      • whisperer's avatar
        whisperer
        Icon for MVP rankMVP

        Ok, so you have a virtual server configured. What do you have configured as Pool Members? Sounds like you have a standard VS and want to server side proxy to AWS S3 buckets?

        Silly question, but do you have SNAT enabled on VS? Could be routing issue.

  • Hello Mohamed_Ahmed_Kansoh, whisperer 

    Thank you for yor answers,

    - Not ASM/WAF

    - L4 (with/without http) : download empty/corrupted file.

    - L4 (with http + Mohamed_Ahmed_Kansoh's iRule) : download empty/corrupted file.

    - Standard (with http + Mohamed_Ahmed_Kansoh's iRule + no serverssl ) : 

    <HTTP_REQUEST>: Condition True in Request

    download empty/corrupted file.

    - Standard (with http + Mohamed_Ahmed_Kansoh's iRule + serverssl ) : 

    S3 XML Error : 

    <Code>NoSuchBucket</Code>
    <Message>The specified bucket does not exist</Message>
    <HTTP_REQUEST>: Condition True in Request
    <HTTP_RESPONSE>: Response Action achieved

     

    I am in http (client) and https (server), MYPOOL S3 := 443

    I keep looking at the pcap....

    Thank you.

    • Hi cpt_ri_F5 , 

      Can you bypass bigip and try to download it directly from server ? To make sure that bigip is point of interrupt your download. 

      Pcap may give us some insights , you may find TCP-Window-FULL and TCP-Zero-Window. 
      you may need to increase bigip buffer of window scale from bigip. 

      But test first from server itself directly without bigip in path.