Apparently, Microsoft IE patch KB974455 enabled "Extended Protection" for Integrated Windows Authentication. Normally, with SPNEGO/Kerberos authentication, the client machine acquires a Kerberos/Active Directory ticket for the server and presents this ticket during the HTTP authentication negotiation. As of at least Java 1.6, the Java JGSS-API library is capable of interpreting the SPNEGO/Kerberos negotiation and authenticating the ticket.
With the Extended Protection (see also Extended Protection for Authentication), IE adds a channel binding to the SPNEGO negotiation; what data the channel binding is based on is currently unknown to me, other than that the SSL session identifier seems to be part of it. The Java JGSS-API library attempts to validate the channel binding and canno without the data the binding is based on. It then throws the channel binding mismatch exception.
The issue has resulted in some internet traffic, including Sun Bug ID 6851973.
According to comments associated with 6851973, RFC 4121, says,
If the caller to GSS_Accept_sec_context [RFC2743] passes in
GSS_C_NO_CHANNEL_BINDINGS [RFC2744] as the channel bindings, then the
acceptor MAY ignore any channel bindings supplied by the initiator,
returning success even if the initiator did pass in channel bindings.
and "all major krb5 implementors implement this MAY ". JGSS appears to be requiring the acceptor to provide the channel binding if the initiator is presenting it. Further, the fix is available in Java 7, build 64 and will be back ported to Java 5 and 6, although the Java 6u18 does not appear to have it as reported in 6851973.
A work-around as seen in Extended Protection for Authentication is to set the
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLSASuppressExtendedProtection
registry setting to 0x02. This disables Extended Protection.