As far as I can tell, there is no exception thrown when CSS3 Namespaces syntax is used in the string passed to querySelectorAll, and at the same time the second argument (the NSResolver) is completely ignored. The latter means the selector has no way to work correctly. The former means that the page can't detect this. I would have expected either correct use of the NSResolver or a SYNTAX_ERR or NAMESPACE_ERR on the attempt to use namespace selectors if they're not supported, so that pages can fallback as needed. As currently implemented this makes it easy for pages to start relying on the incorrect behavior, causing problems for other UAs.
Created attachment 22134 [details] Testcase showing the problem
As of r35041, we now throw a NOT_SUPPORTED_ERR when passing in an NSResolver as the second argument as per spec. We still have to deal with the issue of CSS3 Namespaces syntax in the selector string. I will have to read the spec more closely to figure out what to do in that case.
As a data point, IE8 beta 1 in standards mode (doesn't support this in quirks) completely ignores the second argument, and throws SYNTAX_ERR on uses of namespaces syntax. The SYNTAX_ERR is certainly what I'd expect if your CSS parser doesn't do namespaces. NAMESPACE_ERR is what I'd expect per current draft text if it does and there is no NSResolver (as in all cases now that you're throwing NOT_SUPPORTED_ERR). In any case, it's not as important what exception is thrown as that an exception is thrown if there is no NSResolver and the string includes namespaces syntax.
Right now I am trying to determine what the behavior should be in the face of a selector such as "*|div" which uses namespace syntax but should not require a NSResolver.
I think if your parser parses it, it should work without an NSResolver (since there is nothing to resolve in this case).
Fixed in r35056.