GRPC through F5 Virtual Server [RST_STREAM with error code: INTERNAL_ERROR]

[Reqbaln]

Hello everyone. 

We have a GRPC service running in a K8s cluster and it's reachable through Nginx ingress from inside the cluster.  

We need to access the GRPC service from outside the cluster through F5 Virtual server and we've configured it as described in this guide https://my.f5.com/manage/s/article/K08041451

So the traffic route should be: External Client (GRPC) -> F5 Virtual server (GRPC) -> Nginx ingress running in a k8s cluster (GRPC) -> GRPC Server. However, this rote doesn't work using the VIP as we are getting this error:

Error Message   Json Failed to list services: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

Please note that this traffic route is working as expected: Internal Client (GRPC)-> Nginx ingress running in a k8s cluster (GRPC) -> GRPC Server. 

What could be the issue here? 

Thanks!

[JRahm]

hi @Reqbaln ,

  I don't have experience with this setup, but have you taken a tcpdump on the BIG-IP to capture the transaction? You can enable some flags in tcpdump to allow for the tls keys to be included in the capture so it's easily decrypted in wireshark. I wrote a python script to actually take the capture, download it, and decrypt it for you for these situations.

On the solution info, do you have a confirmed working monitor with the nginx ingress as a pool member, and do you get successful feedback when running this from the BIG-IP command line?

nghttp -ynv https://<server_IP_address>:<port number>

[Reqbaln]

@JRahm 

Thank you for response 

Here what I got when I run nghttp 

nghttp -ynv https://thanos-query-k8s-web-pp.com
[ 0.012] Connected
The negotiated protocol: h2
[ 0.079] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
(niv=3)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):10]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):32768]
[SETTINGS_MAX_HEADER_LIST_SIZE(0x06):32768]
[ 0.079] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
(niv=2)
[SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[ 0.079] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
; ACK
(niv=0)
[ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
(dep_stream_id=0, weight=201, exclusive=0)
[ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
(dep_stream_id=0, weight=101, exclusive=0)
[ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
(dep_stream_id=0, weight=1, exclusive=0)
[ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
(dep_stream_id=7, weight=1, exclusive=0)
[ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
(dep_stream_id=3, weight=1, exclusive=0)
[ 0.079] send HEADERS frame <length=52, flags=0x25, stream_id=13>
; END_STREAM | END_HEADERS | PRIORITY
(padlen=0, dep_stream_id=11, weight=16, exclusive=0)
; Open new stream
:method: GET
:path: /
:scheme: https
:authority: thanos-query-k8s-web-pp.com
accept: */*
accept-encoding: gzip, deflate
user-agent: nghttp2/1.40.0
[ 0.090] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
; ACK
(niv=0)
[ 4.092] recv RST_STREAM frame <length=4, flags=0x00, stream_id=13>
(error_code=INTERNAL_ERROR(0x02))
[ 4.092] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
(last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])
Some requests were not processed. total=1, processed=0

For the tcpdump, unfortunately I don't have access to the virtual server right now. will check it once I have it. 

Thanks!