kerberos to web app

to my understanding, a seperate account is used to run Kerberos from the F5 to the web server and that has to be in the host/apmsso.mydomain.com format, correct??

Correct. Mostly. For Kerberos SSO you need an account in the AD that can perform delegation. To be able to do delegation that account must have a servicePrincipalName value (without it the delegation tab is missing). Now, how APM accesses that account is another story. You can simply use the short NETBIOS name in the SSO profile config, but that can lead to ambiguity when doing cross-domain authentication. The recommendation then is to use the SPN value of that account (host/apmsso.mydomain.com) as the user logon name as well (host/apmsso.mydomain.com), and then also use that as the name in the Kerberos SSO profile - with REALM added (host/apmsso.mydomain.com@MYDOMAIN.COM). That account then delegates to all of the HTTP/ SPNs of the web servers.

It sounds like the users and servers are all in the same domain (mydomain.com). If that's true then the secondary domain doesn't affect the SSO as all three parties are in the same domain (users, servers, delegation account). Users from mydomain.com access an APM VIP that has a keytab from workdomain.com. The client goes out and does its own cross-domain auth to get a ticket for this VIP and then presents that to APM for validation. After that the username from the ticket and the domain (mydomain.com) is presented to the SSO for server side auth. Is that about right?

At this point I'd highly recommend those packet captures from the DC. You want to see the Kerberos and DNS traffic.