need to upgrade f5 ltm from 11.4.1HF5 to 11.5.1 HF7

there isn't a true document except from a small section in the release notes

http://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-5-1.html

section Installation overview, which points to the general active-standby / active-active upgrade documentation which in my opinion still seems to focus on 10 to 11 upgrades and not the 11 to 11 upgrades.

https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-upgrade-active-standby-11-3-0.html

beyond that you should read the release notes and check for any possible issues and tackle those before the upgrade.

do you have HA ?

for major release you should have valid support, for hotfix a valid support is not required

Do I need to reactivate the license before upgradation.

it depends on service check date and version you are going to upgrade to.

sol7727: License activation may be required prior to software upgrade for the BIG-IP or Enterprise Manager system

Yes, It is active/standby mode.

Yes, you will most likely have to reactivate.

I would like to know whether 11.5.1 HF7 is supported in LTM 3600 series as it has 2 CPU and 3.8 G memory.

Total Phys Memory 3.8G 3.8G 3.8G OS Used Memory 3.8G 3.8G 3.8G TMM Alloc Memory 2.9G 2.9G 2.9G TMM Used Memory 1.0G 1.0G 1.0G

Host: 0 Available 3.8G Used 3.8G

I would like to know whether 11.5.1 HF7 is supported in LTM 3600 series as it has 2 CPU and 3.8 G memory.

sol9476: The F5 hardware/software compatibility matrix

As there were changes to the default SSL cipher set between the specified TMOS versions I would closely monitor this aspect after running the upgrade. Especially if you are using your BIG-IP in a "closed environment" some clients may still rely on SSLv3 (now these ciphers are now disabled by default).