The newly added document.compatMode does not update when the doctype changes, or is removed, in cases when the compatMode has changed. This can be observed by going to http://software.hixie.ch/utilities/js/live-dom-viewer/ and removing the doctype from the example text. It should change from CSS1Compat to BackCompat but doesn't.
Fixed by patch for bug 12751 in r30431.