What is the REST API to check / uncheck checkbox client SSL profile custom cipher cipher string ?

The right-hand side checkbox is only for GUI to protect the configuration items from accidental overwriting, hence it does not present in tmsh or iControl REST API. The selection in the middle is also for GUI. When "Cipher Group" is selected, it shows a list of available (preconfigured) groups (and the text field disappears). When "Cipher String" is selected, you can specify the ciphers of your choice.

If you want to create a client-ssl profile with a specific cipher string, use the "ciphers" property: For example, run the following curl command to create a profile named "clientssl-sat" with the cipher string "AES" (others are all default inherited from clientssl).

curl -sku <user:pass> https://<IP>/mgmt/tm/ltm/profile/client-ssl \
 -X POST  -H "Content-type: application/json" \
 -d '{"name":"clientssl-sat", "ciphers":"AES"}'

To create a client-ssl from a cipher group, specify it to the "cipherGroup" property. Because you cannot specify both "ciphers" (by default, "DEFAULT") and "cipherGroup" at the same time, you have to specify "none" to the "ciphers" property: e.g.,

curl -sku <user:pass> https://<IP>/mgmt/tm/ltm/profile/client-ssl \
  -X POST -H "Content-type: application/json" \
  -d '{"name":"clientssl-sat", "cipherGroup":"f5-secure", "ciphers":"none"}'

See also K10866411: Creating a custom cipher group using the Configuration utility