Squid forward caching proxy server conflicting with Load Balancer; images, JS, CSS not rendering in application

Have an interesting one here that I hope others can help unravel.

 

A user tells me that the website application, which sits behind an F5 LB, is not rendering properly:

 

E.g.: is missing images, stylesheets, javascript files, and the like.

 

And it's not just this user but a colleague at his workplace has the same issue and seemingly others in the company also can reproduce this issue.

 

I will say that this client (as in the company) is the only one who has reported such an issue. No other companies who use the application are reporting pages not rendering content properly. He had tried testing with a work laptop, work phone, personal phone, over the company network, cellular network, and home network and using multiple browsers. It was consistent across multiple browsers. I asked him to clear cache and cookies and that did not help.

 

Here are the results of his testing:

 

• Work laptop on home network: Pass.
• Work laptop on company network : Fail.
• Work laptop at their customer's location (possibly connected to customer's network): Fail
• Work phone on company network: Fail.
• Work phone on cellular network (Verizon)*: Fail.
• Personal phone on cellular network (AT&T)*: Pass.
• Work colleague of user laptop connected to said colleague's phone configured as hotspot (Sprint) (not sure if devices are work or personal)**:

**This was conducted while on company premises.

 

It didn't seem to matter what browser was employed. I didn't get a report that it worked in one browser but not another, for instance.

 

To make a long story short, I asked him to send me a fiddler log and the logs showed something that I cannot reproduce on my end. The Fiddler log shows the page loaded with HTTP 200 but the content on the page (i.e. JavaScript files, stylesheets, images) show HTTP 304.

 

In the response headers, under Transport, for all requests, I see Connection: close and Via: 1.1 {unique ID} (squid/3.5.23) (The unique ID is some kind of specific value. It might be sensitive information so I decided to not include it in this post).

 

For , the response header Cache shows:

 

• X-Cache: MISS from {unique ID}
• X-Cache-Lookup: HIT from {unique ID}:{Port number}

For , the response header Cache shows:

 

• X-Cache: HIT from {unique ID}
• X-Cache-Lookup: HIT from {unique ID}:{Port number}

I don't recall seeing anything like this before. It looks to be Squid, a caching and forward proxy server, that is sitting in front of the client and making requests to the LB. Since this company is the only one who has reported this issue and I cannot reproduce it on my end, it's probably safe to say that either this company is running Squid, their ISP is running Squid, or even both. I pressed the user to inquire with the company's IT if they are running any proxies and the answer was no. It's certainly possible the company's IT could be mistaken.

 

Today, the user says that he came into his office and everything is working now. He tried Firefox, IE, wireless network, cellular network and does not understand why it's working.

 

The likely possibilities I can think of as to what and why is:

 

• Squid cache was flushed, which means this problem may return in the future.
• Squid was not configured properly by company's IT/ISP and now it is, thus resolving the issue.
• Squid was taken offline and the client is connecting directly to the LB now.

What I am very concerned about is what happens if the company reports the same issue or maybe another company who is running Squid or some other forward caching/proxy server reports the same issue? I really don't know if this is something where I have to tell the user that this is not our problem, this is your IT infrastructure and/or your local ISP's problem.

 

In other words, whether the Squid server is configured properly or not, is this something where the LB needs to be configured such that it works around the problem? Does that make sense?

 

If there is a configuration change that I need to enact on the LB, what are these changes and what are step-by-step instructions?

 

I'm sorry for the long-winded explanation but I'm trying to be detailed and thorough with this.

 

Thank you very much.