Technical Forum
Ask questions. Discover Answers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Download file from AWS S3 bucket / http profile

cpt_ri_F5
Cirrus
Cirrus
Hello,
I'm trying to download a file from AWS S3 bucket (backend) through a standard VS (HTTP/iRule) but it doesn't work. It downloads an empty/corrupt file.
 
"curl path/file" works from F5.
 
I tried to deactivate the http profile, but still ko!
when HTTP_REQUEST {
    if { [class match [string tolower [HTTP::uri]] contains uri-data] } {
         HTTP::disable
         pool MYPOOL
}
}
when HTTP_RESPONSE {
         HTTP::disable
}
 
Someone has experienced this issue?
 
Thank you for your help
 
1 ACCEPTED SOLUTION

cpt_ri_F5
Cirrus
Cirrus

Hello,

Resolved, add only : HTTP::header replace Host "hostname"

FYI : url is : hostname.bucket.xxxxxxaws......

View solution in original post

13 REPLIES 13

whisperer
Cumulonimbus
Cumulonimbus

So, 2 questions here:
- Does datagroup 'uri-data' actually exist? Do you have the URI path there for the file you are looking to download?
- The HTTP::disable in the HTTP_RESPONSE will trigger on EVERY request, regardless if there is a match in the HTTP_REQUEST. You need to specify a variable in the HTTP_REQUEST and check it in the HTTP_RESPONSE. I presume that when that particular URI is matched, you only then want to disable the HTTP profile for that session -- both client side (http_request) and server side (http_response.

Hello,

- Yes, the 'uri-data' datagroup exists (also tested with fixed uri in code)

- Same problem, HTTP_RESPONSE with matched URI !

I presume that when that particular URI is matched, you only then want to disable the HTTP profile for that session -- both client side (http_request) and server side (http_response.

=> I don't know how it works, I found this solution in the two links shared, I'm interested in any other solutions.

My question, is there a special configuration when trying to view or download a file (http://VS_F5/path_file) with a Standard VS ?

I would also like to confirm: I access in http and the backend servers are in https, correct : no client ssl, with default serverssl ?

Thank you

Ok, so you have a virtual server configured. What do you have configured as Pool Members? Sounds like you have a standard VS and want to server side proxy to AWS S3 buckets?

Silly question, but do you have SNAT enabled on VS? Could be routing issue.

yes, a standard VS, pool fqdn (up)

wWy proxy? I just want to retrieve the file in http, curl or wget

Not snat, automap (route ok)

Thanks

Your response is confusing. "Not snat" - so this is not enabled? You are also using a pool of FQDNs. Do you have DNS enabled on the F5 and is this being properly resolved to the IP addresses that host your S3 bucket? Have you tried to perform a TCP dump on both client side and server side of the connection? Have you tried to tail -f /var/log/ltm to see if there are any errors flagged?

Sorry, snat with automap, fqdn proprly resolved (pool up), no error /var/log/ltm, tcpdump client, serveur ok ( Syn, Syn Ack, Ack, ...GET, Continuation ...)

Now it shows me: "the specified bucket does not exist",

But, the file exists on S3! + "curl backend/path" is ok from F5

Thanks

Hi @cpt_ri_F5 , 
Could you please disable any AWAF/ASM Policies if exists , Remove attached iRule and convert Virtual server type to performance layer 4 , If the issue is resolved so , Keep disabling ASM policy and attach http profile again , and test one more time. 
If it works with http profile , so you have issue with AWAF policy and need to check yout policy and AWAF system Variables. 

If the Download didn't work so you the issue is with http profile , and you need really to disable it , in this Case I recommend to add this : 

 

 

when HTTP_REQUEST {
    if { [class match [string tolower [HTTP::uri]] contains uri-data] } {
         HTTP::disable
         log local0. "Condition True in Request" 
         return
}
}
when HTTP_RESPONSE {
         HTTP::disable
         log local0. "Response Action achieved" 
}

 

 

> I addedd [Log local ] as a guide to make sure that your requests match with Conditions of if statement , and to make sure that http profile really disabled. 
> I added Return to break the irule , and prefer to attach the Pool "MYPOOL" under virtual server resources. 

The Last thing if this available with you, Take a Packet capture for both sides ( Client and server side ) with identifing the Download file request to see why it gives you an empty/corrupted file.

Follow the above steps , I hope this helps you

_______________________
Regards
Mohamed Kansoh

Just like @Mohamed_Ahmed_Kansoh said, you really need to look closer in what variables you can eliminate, and the actual data from TCP dump. Open in wireshark, and check to see that the HTTP request leaves the F5 (sounds like it does since you get an AWS error messsage). Look at the host header, make sure it is correct. Also, look at the URI and make sure it is correct on the server side of the connection. Also, make sure you are using the same protocol -- HTTP and HTTP or HTTPS and HTTPS, if you are doing a conversion it may not work if AWS is not listening on 80 or 443.

Just a few more nuggets to dig and check 😉 Definitely an interesting issue!

cpt_ri_F5
Cirrus
Cirrus

Hello Mohamed_Ahmed_Kansoh, whisperer 

Thank you for yor answers,

- Not ASM/WAF

- L4 (with/without http) : download empty/corrupted file.

- L4 (with http + Mohamed_Ahmed_Kansoh's iRule) : download empty/corrupted file.

- Standard (with http + Mohamed_Ahmed_Kansoh's iRule + no serverssl ) : 

<HTTP_REQUEST>: Condition True in Request

download empty/corrupted file.

- Standard (with http + Mohamed_Ahmed_Kansoh's iRule + serverssl ) : 

S3 XML Error : 

<Code>NoSuchBucket</Code>
<Message>The specified bucket does not exist</Message>
<HTTP_REQUEST>: Condition True in Request
<HTTP_RESPONSE>: Response Action achieved

 

I am in http (client) and https (server), MYPOOL S3 := 443

I keep looking at the pcap....

Thank you.

Hi @cpt_ri_F5 , 

Can you bypass bigip and try to download it directly from server ? To make sure that bigip is point of interrupt your download. 

Pcap may give us some insights , you may find TCP-Window-FULL and TCP-Zero-Window. 
you may need to increase bigip buffer of window scale from bigip. 

But test first from server itself directly without bigip in path.

_______________________
Regards
Mohamed Kansoh

cpt_ri_F5
Cirrus
Cirrus

+

from F5 :

curl http//fqdn/path_file
=> ok

curl https//fqdn/path_file
=> ok

curl http://IP_node/path_file
=> same S3 XML Error

curl https://IP_node/path_file
=> SSL: certificate subject name 's3.xx-xxxx-x.amazonaws.com' does not match target host name IP_node

 

cpt_ri_F5
Cirrus
Cirrus

Hello,

Resolved, add only : HTTP::header replace Host "hostname"

FYI : url is : hostname.bucket.xxxxxxaws......

Excellent. Checking headers for the win! 🙂