Does F5 VE trail version 11.3 support device service cluster(DSC) feature?

The release notes don't mention that you can't cluster VEs, although I didn't actually think you could (perhaps that was when it was at v10).

Anyway, your scenario suggests a configuration/setup issue as it's obviously trying to add the peer device.

I would re-check your steps in setting up the DSC.

Hope this helps,

N