When is the floating IP or anything used?

Floating self IP addresses are used for lots of things on a BIG-IP system, including as a possible default gateway for your pool members, as YossiV indicated. They are also used for source address translation addresses (SNAT) when Auto Map is specified.

Why do you need floating self IP addresses? In an HA configuration, a floating self IP is shared between device group members, and will always point to the device that is active for its associated traffic group. Static (non-floating) self IP addresses are dedicated to a particular device and never point to anything but that device. For example, non-floating self IP addresses are used for monitor tests, as each device does its own monitoring, whether it's active or standby. Non-floating self IPs are also used for HA communication between devices in an HA device group. As a result, the MAC address of a non-floating self IP never changes. But you wouldn't want the default gateway on back-end servers (nodes) pointing to a non-floating self IP on a BIG-IP system as then traffic might get directed to a standby device. Same for SNATs. To support traffic processing through a BIG-IP HA configuration, the MAC address of a floating self IP address changes during failover to a MAC address on the active device. (This is why we say the IP address is "shared" between the device group members.)

While you can initially set up an HA configuration without floating self IP addresses, you cannot successfully process traffic without floating self IP addresses. Defining an HA (sync-failover) device group simply involves identifying the non-floating self IP addresses that will be used by each device for HA communication. Eventually, you must define the floating self IP addresses that will be associated with traffic processing.