Can the F5 SSL Orchestrator(SSLO) send traffic to a not directly attached Layer 3 device?

The security device IP has to match the to-service self-IP subnet. That's why you're getting this error.

The basic premise of an "inline" device is that SSLO sends traffic to/through it, and that device sends it back. For L3 devices, SSLO routes to the device, and the device must route back on a spearate subnet. It's not generally advisable to send that (decrypted) traffic across a network, so SSLO will by default create a private network enclave for each security service.

The GUI error is to prevent sending unencrypted traffic out onto a network. But if you abssolutely need to do this, just provide a dummy IP here, disable strictness on the service after deploying, and then modify the associated pool member(s). But again, keep in mind that SSLO expects the traffic sent to an L3/HTTP device to be routed back to SSLO on a separate subnet.