Difference between LTM Profiles serverssl & custom_server_ssl

"custom_server_ssl" is not something that is shipped with the BIG-IP i believe. it might have been created by someone else on the BIG-IP you are working on. i would assume it has custom setting for a specific application, so you use it when you need it for that application.

for all the settings you probably best use the help function on the BIG-IP or read the documentation: https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-implementations-11-2-1/15.html