Forum Discussion

Reqbaln's avatar
Reqbaln
Icon for Nimbostratus rankNimbostratus
May 24, 2023

GRPC through F5 Virtual Server [RST_STREAM with error code: INTERNAL_ERROR]

Hello everyone. 

We have a GRPC service running in a K8s cluster and it's reachable through Nginx ingress from inside the cluster.  

We need to access the GRPC service from outside the cluster through F5 Virtual server and we've configured it as described in this guide https://my.f5.com/manage/s/article/K08041451

So the traffic route should be: External Client (GRPC) -> F5 Virtual server (GRPC) -> Nginx ingress running in a k8s cluster (GRPC) -> GRPC Server. However, this rote doesn't work using the VIP as we are getting this error:

Error Message   Json Failed to list services: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

Please note that this traffic route is working as expected: Internal Client (GRPC)-> Nginx ingress running in a k8s cluster (GRPC) -> GRPC Server. 

 

What could be the issue here? 

 

Thanks!

 

security

3 Replies

Sort By
  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    May 25, 2023

    hi Reqbaln ,

      I don't have experience with this setup, but have you taken a tcpdump on the BIG-IP to capture the transaction? You can enable some flags in tcpdump to allow for the tls keys to be included in the capture so it's easily decrypted in wireshark. I wrote a python script to actually take the capture, download it, and decrypt it for you for these situations.

    On the solution info, do you have a confirmed working monitor with the nginx ingress as a pool member, and do you get successful feedback when running this from the BIG-IP command line?

    nghttp -ynv https://<server_IP_address>:<port number>
  • Reqbaln's avatar
    Reqbaln
    Icon for Nimbostratus rankNimbostratus
    May 28, 2023

    JRahm 

    Thank you for response 

    Here what I got when I run nghttp 

    nghttp -ynv https://thanos-query-k8s-web-pp.com
    [ 0.012] Connected
    The negotiated protocol: h2
    [ 0.079] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
    (niv=3)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):10]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):32768]
    [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):32768]
    [ 0.079] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
    (niv=2)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [ 0.079] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
    ; ACK
    (niv=0)
    [ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
    (dep_stream_id=0, weight=201, exclusive=0)
    [ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
    (dep_stream_id=0, weight=101, exclusive=0)
    [ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
    (dep_stream_id=0, weight=1, exclusive=0)
    [ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
    (dep_stream_id=7, weight=1, exclusive=0)
    [ 0.079] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
    (dep_stream_id=3, weight=1, exclusive=0)
    [ 0.079] send HEADERS frame <length=52, flags=0x25, stream_id=13>
    ; END_STREAM | END_HEADERS | PRIORITY
    (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
    ; Open new stream
    :method: GET
    :path: /
    :scheme: https
    :authority: thanos-query-k8s-web-pp.com
    accept: */*
    accept-encoding: gzip, deflate
    user-agent: nghttp2/1.40.0
    [ 0.090] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
    ; ACK
    (niv=0)
    [ 4.092] recv RST_STREAM frame <length=4, flags=0x00, stream_id=13>
    (error_code=INTERNAL_ERROR(0x02))
    [ 4.092] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
    (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])
    Some requests were not processed. total=1, processed=0

     

    For the tcpdump, unfortunately I don't have access to the virtual server right now. will check it once I have it. 

    Thanks!

  • LiefZimmerman's avatar
    LiefZimmerman
    Icon for Admin rankAdmin
    Jun 20, 2023

    Reqbaln - If your post was solved it would be helpful to the community to select *Accept As Solution*.
    This helps future readers find answers more quickly and confirms the efforts of those who helped.
    Thanks for being part of our community.
    Lief