Possible to read TIFF header and reject if format is incorrect?

Certainly. iRules are a complete language, so essentially anything can be done.

Notifying the server might be problematic though.

Probably you'd be looking at something similar to:

1. Get HTTP request (HTTP_REQUEST).
2. Decide if it's the POST you're looking for (HTTP::method == 'POST', HTTP::header probably Content-Disposition filename matches whatever.tiff).
3. If so, begin collecting the data (HTTP::collect xxx).
4. Examine the body file header bytes for your required information (HTTP_REQUEST_DATA).
5. If the information is correct, release the connection to the server (HTTP_RELEASE), upload proceeds as normal.
6. If the information is not correct, (HTTP::respond) with some kind of user-facing error page

Here, the server would get no notification of the invalid upload but you could log it for future auditing.