Help with tcp/http payload irule

Glad to assist, and thanks for that, totally my privilege to invest in the community!

Have you looked at the POLICY::rules command? If you use the matched attribute with contains and your conflicting policy, you might be able to bypass that error and proceed with the more elegant HTTP solution.

So maybe something like this would work?

when HTTP_REQUEST {
    if { [POLICY::rules matched] contains "your conflicting executed policy rule" } {
        return
    } else {
        # ... your logic here in HTTP instead of TCP
    }
}

I've never used it personally but that's the way I'd try to make this work before attempting it at the TCP layer for two reasons: 1) TCP inspection is more expensive, especially since you have access to the HTTP since it's applied and 2) Division of logic into multiple layers complicates your workflow.

Regarding the rules as written, I would just focus on the match instead of trying to handle both the match and unmatch condition. So on your front-end 80 vip, I'd just remove the custom header if exists and eliminate the last two lines that aren't brackets:

## WAF FRONT-END VIRTUAL SERVER 80
when HTTP_REQUEST {
  ## AVOID PREVIOUS CUSTOM_HEADER HEADER
  if {[HTTP::header exists CUSTOM_HEADER]}{
    HTTP::header remove "CUSTOM_HEADER"
  }
}

and then on your backend something like:

when RULE_INIT {
  set static::debug_secure_conn 1
}
when CLIENT_ACCEPTED {
  TCP::collect
}
when CLIENT_DATA {
  set payload [TCP::payload]
  if { $static::debug_secure_conn } {
    log local0. "Payload: $payload"
  }
  if { $payload contains "SECURE_CONN" } {
    set SECURE_CONNBSEG 1
    if { $static::debug_secure_conn } {
      log local0. "SECURE_CONN found - $SECURE_CONN"
    }
  } 
  TCP::release
}
when HTTP_REQUEST_RELEASE {
  STREAM::disable
}
when HTTP_RESPONSE {
  if { [info exists SECURE_CONNBSEG] } {
    if { $static::debug_secure_conn } {
      log local0. "Value SECURE = $SECURE_CONN"
    }
    STREAM::expression {@http://domain.com@https://domain.com@}
    STREAM::enable
  }
}

Not at all tested, but food for thought.