Bug 37989 – Safari (still) doesn't always send basic credentials preemptively

• Component: Page Loading
• Status: RESOLVED FIXED

I have observed this problem for several years and with all versions of webkit based browsers. When a client visits a web site that is using Apache Basic Authentication, the server will respond with a 401 HTTP status code requesting a userID/password.  However, at times, the browser will ignore the 401 status code and repeatedly requests content without providing the userID/password on the request.  There could be several hundred requests that are generated without properly providing this information even though the server responds to each one with a 401.

Problem here is that this behavior will trigger server side security monitors that consider this to be an failed attempt to login to a secured site. 

Thanks.

[Mark Rowe (bdash)]

In my testing HTTP basic authentication has functioned exactly as I would expect.  I tested using <http://www.he.net/~jdoe/info/htaccess/example/restricted.html> with the username “john” and password of “orange”.  Safari correctly prompts for credentials and the page is displayed when they are entered.  If basic authentication was broken in general it would certainly have been noticed already, which indicates that any problem you’re seeing is somehow specific to your environment.  You’ll need to provide more information for us to be able to investigate this issue.  Please provide a URL that can be used to reproduce the problem that you describe.

Mark,

I absolutely agree that if there was something obvious wrong with basic authentication, it would have been noticed.  The problem is that it is not obvious.  Let me elaborate...

First, let me say that I do not use or have not downloaded a webkit based browser to do extensive testing myself. Perhaps I will do that.  But I do know as a fact that the problem appears in various browsers all of which are webkit based.  For example, on my Google G1 phone, I have the same problem.

The reason the problem is not apparent is two fold. One reason is that the browser does not always send a request without the credentials.  Most of the time it does but something at some point triggers it to stop sending the uID/PW with some of the requests.  The other reason why it has not been reported more often is that perhaps all sites that use basic authentication for login do not have any additional monitoring for failed logins.  What I am trying to say is the following.  Let's assume that out of 100 requests from the client, 80 of them did not include the credentials on the request.  The result of this would be 80 server responses with a 401 HTTP result code.  The only place you can see this in is in the server logs.   The client does not see any problems or prompts.  It just happens behind the scenes.  Unless you monitor the server logs for 401 errors, you won't see this.  I have seen a few similar complaints while using google to search the web. 

The reason this is affecting my site is because I have tools that actively monitor the Apache logs for 401 errors.  This is how I found out there was a problem.  This happens on multiple servers on using various applications.  For example we have java based applications behind a basic authentication Apache/Tomcat server.  I also see this on a Horde/IMP webmail server that is PHP based.  The client connects, does the initial apache basic authentication and after a short time, lots of request begin coming in from the client without the client credentials.

If you would let me know how or what type of additional information I can provide you, I will be happy to do so. 

Thanks  


(In reply to comment #1)
> In my testing HTTP basic authentication has functioned exactly as I would
> expect.  I tested using
> <http://www.he.net/~jdoe/info/htaccess/example/restricted.html> with the
> username “john” and password of “orange”.  Safari correctly prompts for
> credentials and the page is displayed when they are entered.  If basic
> authentication was broken in general it would certainly have been noticed
> already, which indicates that any problem you’re seeing is somehow specific to
> your environment.  You’ll need to provide more information for us to be able to
> investigate this issue.  Please provide a URL that can be used to reproduce the
> problem that you describe.

[Alexey Proskuryakov]

There is almost nothing in common between how Safari and Android-based phones handle authentication. Both underlying networking libraries and glue code in WebKit are completely independent and different. Different versions of WebKit on Mac have also had drastically different code, even in recent versions.

> If you would let me know how or what type of additional information I can
> provide you, I will be happy to do so. 

A complete request with Referer headers etc. would help. If you can catch this in action, tcpdump output is obviously best.

If you could investigate which WebKit-based browsers do that, and which exact versions, that would also help.

(In reply to comment #3)
> There is almost nothing in common between how Safari and Android-based phones
> handle authentication. Both underlying networking libraries and glue code in
> WebKit are completely independent and different. Different versions of WebKit
> on Mac have also had drastically different code, even in recent versions.
> 

Yes, I realize that but the point I was trying to make was that I see this behavior only from browsers based on Webkit.  That is the only common thing between them that I can see from the Apache logs.


> > If you would let me know how or what type of additional information I can
> > provide you, I will be happy to do so. 
> 
> A complete request with Referer headers etc. would help. If you can catch this
> in action, tcpdump output is obviously best.
> 
> If you could investigate which WebKit-based browsers do that, and which exact
> versions, that would also help.

I will begin to do what you asked.  I will see if something triggers this behavior and what that is.  I can collect a wireshark trace.

Ok, I am able to reproduce the problem (sometimes) using Safari 4.0.5 (531.22.7).  I connect to a local server with basic authentication, login, and then see requests generated to the server without user credentials, then a 401 is sent and then later the same object is requested but this time with user credentials.

I attempted to do a tcp trace but quickly realized that the protocol is https and therefore, I cannot see the communication.  I will attempt to setup a server with http protocol and see if I can trace then.  

Attached is a sanitized file with a small segment of the server log.  You can clearly see requests come in without the user name.

Created attachment 54185 [details]
Apache log

[Alexey Proskuryakov]

I don't see anything wrong in the attached log. Basic credentials can be only sent preemptively for resources in the same (or deeper) directories as resources that were fetched with authentication before, see RFC 2617:

   A client SHOULD assume that all paths at or deeper than the depth of
   the last symbolic element in the path field of the Request-URI also
   are within the protection space specified by the Basic realm value of
   the current challenge. A client MAY preemptively send the
   corresponding Authorization header with requests for resources in
   that space without receipt of another challenge from the server.

In the log, there are no prior requests for /themes/graphics/nav or /themes/graphics, so we must send a request without credentials and get a 401 challenge first.

(In reply to comment #7)
> I don't see anything wrong in the attached log. Basic credentials can be only
> sent preemptively for resources in the same (or deeper) directories as
> resources that were fetched with authentication before, see RFC 2617:
> 
>    A client SHOULD assume that all paths at or deeper than the depth of
>    the last symbolic element in the path field of the Request-URI also
>    are within the protection space specified by the Basic realm value of
>    the current challenge. A client MAY preemptively send the
>    corresponding Authorization header with requests for resources in
>    that space without receipt of another challenge from the server.
> 

I agree with the RFC and since these objects are under different directories, I understand that the RFC states that the browser MAY preemptively send credentials.  In fact, other browsers (firefox, IE, etc.) do. 


> In the log, there are no prior requests for /themes/graphics/nav or
> /themes/graphics, so we must send a request without credentials and get a 401
> challenge first.

My apologies.  I attached a small section of the log and therefore, it was not clear that some objects had already been retrieved from the same directories.  I have attached a new log that shows the entire transaction.  Please note for example objects such as '/js/prototype.js' which is requested repeatedly.   

Also, this is a small example.  I recently had to change have the server send an expiration header for image objects so that they are not repeatedly ask for.  Otherwise, we would see repeated requests with and without credentials for image type objects as well.  Please see new attached file.  Thanks.

Created attachment 54188 [details]
More complete log file

[Alexey Proskuryakov]

Yes, the more complete log does look wrong. Some more questions:
- Which OS version are you running Safari with?
- Does this happen with a nightly build from <http://nightly.webkit.org>?

Also, a tcpdump/wireshark trace would still be helpful. I understand it can be hard to sanitize, so just a few full requests/responses ripped out from it would help.

(In reply to comment #10)
> Yes, the more complete log does look wrong. Some more questions:
> - Which OS version are you running Safari with?

I actually have no Safari installations here locally.  To do the test for you, I downloaded Safari 4.0.5 (531.22.7) and running it on Win XP.  Like I said, from my server logs, I see this problem with a variety of OS and browser versions.  For example I see it on Mac OS X and on my own Android based G1 phone.  All the browsers I see this on are webkit based. That's why I assumed it may have to do with webkit.



> - Does this happen with a nightly build from <http://nightly.webkit.org>?

I am sure it does.  This problem has been on my back for over a year and a half.  I have also been able to see similar complaints on the web with no real solution.  Here are a few examples:


http://www.gossamer-threads.com/lists/zope/users/223295
http://discussions.apple.com/thread.jspa?threadID=2136821&tstart=60
http://www.experts-exchange.com/Software/Internet_Email/Web_Browsers/Safari/Q_23153036.html


> 
> Also, a tcpdump/wireshark trace would still be helpful. I understand it can be
> hard to sanitize, so just a few full requests/responses ripped out from it
> would help.

I have such a trace.  I setup another server with the same application and running on http port rather than https.  I can send you the complete trace.  Rather than post it here, can I e-mail it to you directly?

[Alexey Proskuryakov]

> Rather than post it here, can I e-mail it to you directly?

Yes, please.

[Alexey Proskuryakov]

I do see a problem in Safari with the packet trace I got. We only remember default credentials for a directory when getting credentials from the user, not when getting them from credential storage. I have a fix for this.

This is purely a Safari 4 issue, which couldn't possibly affect any other version of Safari or any other browser. Please file new bugs if you can observe this problem with any other WebKit-based browser, or if a similar will still be present in Safari.

[Alexey Proskuryakov]

Created attachment 55769 [details]
proposed fix

[Alexey Proskuryakov]

Committed <http://trac.webkit.org/changeset/59185>.

Please feel free to verify this fix with the next nightly build from <http://nightly.webkit.org>.

[WebKit Review Bot]

http://trac.webkit.org/changeset/59185 might have broken GTK Linux 32-bit Release
The following changes are on the blame list:
http://trac.webkit.org/changeset/59184
http://trac.webkit.org/changeset/59185

[Alexey Proskuryakov]

Skipped the test on Gtk, filed bug 38956.

[Alexey Proskuryakov]

<rdar://problem/7972450>

[Eric Seidel (no email)]

Appears to need an expectations update on Qt:
http://build.webkit.org/results/Qt%20Linux%20Release/r59198%20(11612)/http/tests/xmlhttprequest/null-auth-diffs.txt

[Alexey Proskuryakov]

> Appears to need an expectations update on Qt:

That's not a good change, and I don't understand it. But there were no code changes that could possibly affect Qt, so it might be some kind of fallback from a newly added test.

[Eric Seidel (no email)]

I should note that http://trac.webkit.org/changeset/59184 is also on the blame list for the Qt test change, however that commit looks unrelated to this failure.

[Eric Seidel (no email)]

The regression to Qt is still in:
http://build.webkit.org/results/Qt%20Linux%20Release/r59212%20(11625)/http/tests/xmlhttprequest/null-auth-pretty-diff.html

Should we skip this test for now?  Update the results?

Ap said over IRC that these new results were bad, but he didn't know why they were happening.

Either way we need to get back to a green state, ideally by fixing whatever regression this may be. :)

[Alexey Proskuryakov]

Qt issues are being investigated in bug 32961.