Forum Discussion
brad9iner_11512
Altostratus
AltostratusError processing HTTP request header
I'm running into an issue on my apache tomcat 8 servers when receiving traffic routed through the F5. We have enabled TLS on the application servers and have setup an SSL Server profile to send the traffic through over HTTPS. The pool is configured to send the traffic to the servers on ports 8443 and 8444. However we are getting the following error from the servers.
2019-03-20 05:26:32,938+0000 INFO org.apache.coyote.http11.Http11Processor service - Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:462)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
I am not certain what to look at on the F5 for how it is formatting/encrypting the traffic before it sends it onto the servers. Any advice on the direction I should take this would be very much appreciated.
brad9iner_11512I forgot to mention these application servers are on Oracle Linux 7.6.
DennisJann_1272Nimbostratus
It looks like this error occurs when you send an HTTPS request to a Tomcat server running an HTTP listener.
I got a similar error message from a sample Tomcat Docker image (
) when I sent it an HTTPS request (docker run -it --rm -p 8888:8080 tomcat:alpine
).curl -Ivk https://localhost:8888/21-Mar-2019 00:36:25.559 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:428) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)You could confirm this by removing the serverssl profile from the VIP in question.
You could also try connecting to ports 8443 and 8444 on your Tomcat hosts to verify that they are in fact configured to accept HTTPS connections.
If neither of these resolves the issue, you could try sharing a sanitized version of your VIP configuration, along with confirmation that the Tomcat servers are accepting HTTPS connections.
- brad9iner_11512
Here is the config we have setup on our tomcat servers:
``Which is the same execpt 8444 for our other JVM. We have 2 nodes in this cluster with the same configs. I don't have direct access to the F5 our VIP is configured on. But I had my resource do some testing. When they disabled 8444 in the pool in question for both nodes we are able to get successful responses. But when they disable 8443 and leave 8444 enabled then that is when we start getting "empty reply from server" messages as well as the stack trace in the description.
- brad9iner_11512
new update on this, looking further into the F5 with my resource that has access we do have an iRule applied
when SERVER_CONNECTED { if {[LB::server port] != 8443 && [LB::server port] != 443} { SSL::disable } }What would I need to edit on that to make it also look at ports 8444 and 8445? I can't quite find the "or" equivalent for this kind of iRule.
- DennisJann_1272
Given your existing iRule, you would enclose the "or" conditions in ()'s like this:
when SERVER_CONNECTED { if { ([LB::server port] != 8443 or [LB::server port] != 8444 or [LB::server port] != 8445) && [LB::server port] != 443 } { SSL::disable } }I don't know your specific requirement, but I've generally used SSL::disable in the HTTP_REQUEST event like this:
when HTTP_REQUEST { if { [HTTP::path] starts_with "/plaintext-path" } { SSL::disable pool http_pool } else { pool https_pool } }Hope this helps.
- DennisJann
It looks like this error occurs when you send an HTTPS request to a Tomcat server running an HTTP listener.
I got a similar error message from a sample Tomcat Docker image (
) when I sent it an HTTPS request (docker run -it --rm -p 8888:8080 tomcat:alpine
).curl -Ivk https://localhost:8888/21-Mar-2019 00:36:25.559 INFO [http-nio-8080-exec-8] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level. java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:428) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:684) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)You could confirm this by removing the serverssl profile from the VIP in question.
You could also try connecting to ports 8443 and 8444 on your Tomcat hosts to verify that they are in fact configured to accept HTTPS connections.
If neither of these resolves the issue, you could try sharing a sanitized version of your VIP configuration, along with confirmation that the Tomcat servers are accepting HTTPS connections.
- brad9iner_11512
Here is the config we have setup on our tomcat servers:
``Which is the same execpt 8444 for our other JVM. We have 2 nodes in this cluster with the same configs. I don't have direct access to the F5 our VIP is configured on. But I had my resource do some testing. When they disabled 8444 in the pool in question for both nodes we are able to get successful responses. But when they disable 8443 and leave 8444 enabled then that is when we start getting "empty reply from server" messages as well as the stack trace in the description.
- brad9iner_11512
new update on this, looking further into the F5 with my resource that has access we do have an iRule applied
when SERVER_CONNECTED { if {[LB::server port] != 8443 && [LB::server port] != 443} { SSL::disable } }What would I need to edit on that to make it also look at ports 8444 and 8445? I can't quite find the "or" equivalent for this kind of iRule.
- DennisJann
Given your existing iRule, you would enclose the "or" conditions in ()'s like this:
when SERVER_CONNECTED { if { ([LB::server port] != 8443 or [LB::server port] != 8444 or [LB::server port] != 8445) && [LB::server port] != 443 } { SSL::disable } }I don't know your specific requirement, but I've generally used SSL::disable in the HTTP_REQUEST event like this:
when HTTP_REQUEST { if { [HTTP::path] starts_with "/plaintext-path" } { SSL::disable pool http_pool } else { pool https_pool } }Hope this helps.