BigIP 11.5 VE lab edition

To get the LTM to control browser traffic, would I put a virtual server IP into the browser's Proxy Settings?

Not at all. A bare minimum configuration of a virtual server is a destination IP and port, and a pool of resources (your web servers). Client traffic connects to the VIP address and is proxied to a pool member. From there you can add additional "filters" that can act on the payload in different ways. For example, to use an iRule to examine and control HTTP data through the proxy, you'd apply an HTTP profile. To optimize that HTTP traffic, you might apply a web acceleration profile. And to terminate and optionally re-encrypt the data (ie. HTTPS), you'd apply client and server SSL profiles respectively.

The basic VIP configuration I just described is generally for REVERSE proxy environments (unlimited set of external clients accessing a limited set of internal resources). As of 11.4 and above, there are now some really cool FORWARD proxy capabilities (a limited set of internal clients accessing an unlimited set of remote resources). For this you have several options, in which one includes explicit browser proxy settings.