ssl handshake failure issue from loabalancer to the backend application server

Do you mean one pool has servers which use TLS, but the other pool or pools don't?

In this case, you should attach a Server SSL profile to the Virtual Server, but disable it using the iRule that selects your pool, when required using the command: SSL::disable serverside

I don't know how your iRule decides on pools, but for example:

when HTTP_REQUEST {
    if { [HTTP::uri] starts_with "/nonhttpspool" }{
        SSL::disable serverside
        pool nonhttps_pool
    } else {
      pool https_pool
    }
}

Remember that without a OneConnect profile, only the first request of the connection can select the pool.