Bug 37989 - Safari (still) doesn't always send basic credentials preemptively
Summary: Safari (still) doesn't always send basic credentials preemptively
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Page Loading (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2010-04-22 06:55 PDT by bugs
Modified: 2010-05-12 08:28 PDT (History)
6 users (show)

See Also:


Attachments
Apache log (5.20 KB, text/plain)
2010-04-23 13:03 PDT, bugs
no flags Details
More complete log file (71.12 KB, text/plain)
2010-04-23 13:42 PDT, bugs
no flags Details
proposed fix (13.45 KB, patch)
2010-05-11 15:52 PDT, Alexey Proskuryakov
beidson: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description bugs 2010-04-22 06:55:19 PDT
I have observed this problem for several years and with all versions of webkit based browsers. When a client visits a web site that is using Apache Basic Authentication, the server will respond with a 401 HTTP status code requesting a userID/password.  However, at times, the browser will ignore the 401 status code and repeatedly requests content without providing the userID/password on the request.  There could be several hundred requests that are generated without properly providing this information even though the server responds to each one with a 401.

Problem here is that this behavior will trigger server side security monitors that consider this to be an failed attempt to login to a secured site. 

Thanks.
Comment 1 Mark Rowe (bdash) 2010-04-22 14:41:41 PDT
In my testing HTTP basic authentication has functioned exactly as I would expect.  I tested using <http://www.he.net/~jdoe/info/htaccess/example/restricted.html> with the username “john” and password of “orange”.  Safari correctly prompts for credentials and the page is displayed when they are entered.  If basic authentication was broken in general it would certainly have been noticed already, which indicates that any problem you’re seeing is somehow specific to your environment.  You’ll need to provide more information for us to be able to investigate this issue.  Please provide a URL that can be used to reproduce the problem that you describe.
Comment 2 bugs 2010-04-22 14:55:56 PDT
Mark,

I absolutely agree that if there was something obvious wrong with basic authentication, it would have been noticed.  The problem is that it is not obvious.  Let me elaborate...

First, let me say that I do not use or have not downloaded a webkit based browser to do extensive testing myself. Perhaps I will do that.  But I do know as a fact that the problem appears in various browsers all of which are webkit based.  For example, on my Google G1 phone, I have the same problem.

The reason the problem is not apparent is two fold. One reason is that the browser does not always send a request without the credentials.  Most of the time it does but something at some point triggers it to stop sending the uID/PW with some of the requests.  The other reason why it has not been reported more often is that perhaps all sites that use basic authentication for login do not have any additional monitoring for failed logins.  What I am trying to say is the following.  Let's assume that out of 100 requests from the client, 80 of them did not include the credentials on the request.  The result of this would be 80 server responses with a 401 HTTP result code.  The only place you can see this in is in the server logs.   The client does not see any problems or prompts.  It just happens behind the scenes.  Unless you monitor the server logs for 401 errors, you won't see this.  I have seen a few similar complaints while using google to search the web. 

The reason this is affecting my site is because I have tools that actively monitor the Apache logs for 401 errors.  This is how I found out there was a problem.  This happens on multiple servers on using various applications.  For example we have java based applications behind a basic authentication Apache/Tomcat server.  I also see this on a Horde/IMP webmail server that is PHP based.  The client connects, does the initial apache basic authentication and after a short time, lots of request begin coming in from the client without the client credentials.

If you would let me know how or what type of additional information I can provide you, I will be happy to do so. 

Thanks  


(In reply to comment #1)
> In my testing HTTP basic authentication has functioned exactly as I would
> expect.  I tested using
> <http://www.he.net/~jdoe/info/htaccess/example/restricted.html> with the
> username “john” and password of “orange”.  Safari correctly prompts for
> credentials and the page is displayed when they are entered.  If basic
> authentication was broken in general it would certainly have been noticed
> already, which indicates that any problem you’re seeing is somehow specific to
> your environment.  You’ll need to provide more information for us to be able to
> investigate this issue.  Please provide a URL that can be used to reproduce the
> problem that you describe.
Comment 3 Alexey Proskuryakov 2010-04-22 16:22:57 PDT
There is almost nothing in common between how Safari and Android-based phones handle authentication. Both underlying networking libraries and glue code in WebKit are completely independent and different. Different versions of WebKit on Mac have also had drastically different code, even in recent versions.

> If you would let me know how or what type of additional information I can
> provide you, I will be happy to do so. 

A complete request with Referer headers etc. would help. If you can catch this in action, tcpdump output is obviously best.

If you could investigate which WebKit-based browsers do that, and which exact versions, that would also help.
Comment 4 bugs 2010-04-23 06:33:22 PDT
(In reply to comment #3)
> There is almost nothing in common between how Safari and Android-based phones
> handle authentication. Both underlying networking libraries and glue code in
> WebKit are completely independent and different. Different versions of WebKit
> on Mac have also had drastically different code, even in recent versions.
> 

Yes, I realize that but the point I was trying to make was that I see this behavior only from browsers based on Webkit.  That is the only common thing between them that I can see from the Apache logs.


> > If you would let me know how or what type of additional information I can
> > provide you, I will be happy to do so. 
> 
> A complete request with Referer headers etc. would help. If you can catch this
> in action, tcpdump output is obviously best.
> 
> If you could investigate which WebKit-based browsers do that, and which exact
> versions, that would also help.

I will begin to do what you asked.  I will see if something triggers this behavior and what that is.  I can collect a wireshark trace.
Comment 5 bugs 2010-04-23 13:02:54 PDT
Ok, I am able to reproduce the problem (sometimes) using Safari 4.0.5 (531.22.7).  I connect to a local server with basic authentication, login, and then see requests generated to the server without user credentials, then a 401 is sent and then later the same object is requested but this time with user credentials.

I attempted to do a tcp trace but quickly realized that the protocol is https and therefore, I cannot see the communication.  I will attempt to setup a server with http protocol and see if I can trace then.  

Attached is a sanitized file with a small segment of the server log.  You can clearly see requests come in without the user name.
Comment 6 bugs 2010-04-23 13:03:53 PDT
Created attachment 54185 [details]
Apache log
Comment 7 Alexey Proskuryakov 2010-04-23 13:17:07 PDT
I don't see anything wrong in the attached log. Basic credentials can be only sent preemptively for resources in the same (or deeper) directories as resources that were fetched with authentication before, see RFC 2617:

   A client SHOULD assume that all paths at or deeper than the depth of
   the last symbolic element in the path field of the Request-URI also
   are within the protection space specified by the Basic realm value of
   the current challenge. A client MAY preemptively send the
   corresponding Authorization header with requests for resources in
   that space without receipt of another challenge from the server.

In the log, there are no prior requests for /themes/graphics/nav or /themes/graphics, so we must send a request without credentials and get a 401 challenge first.
Comment 8 bugs 2010-04-23 13:41:56 PDT
(In reply to comment #7)
> I don't see anything wrong in the attached log. Basic credentials can be only
> sent preemptively for resources in the same (or deeper) directories as
> resources that were fetched with authentication before, see RFC 2617:
> 
>    A client SHOULD assume that all paths at or deeper than the depth of
>    the last symbolic element in the path field of the Request-URI also
>    are within the protection space specified by the Basic realm value of
>    the current challenge. A client MAY preemptively send the
>    corresponding Authorization header with requests for resources in
>    that space without receipt of another challenge from the server.
> 

I agree with the RFC and since these objects are under different directories, I understand that the RFC states that the browser MAY preemptively send credentials.  In fact, other browsers (firefox, IE, etc.) do. 


> In the log, there are no prior requests for /themes/graphics/nav or
> /themes/graphics, so we must send a request without credentials and get a 401
> challenge first.

My apologies.  I attached a small section of the log and therefore, it was not clear that some objects had already been retrieved from the same directories.  I have attached a new log that shows the entire transaction.  Please note for example objects such as '/js/prototype.js' which is requested repeatedly.   

Also, this is a small example.  I recently had to change have the server send an expiration header for image objects so that they are not repeatedly ask for.  Otherwise, we would see repeated requests with and without credentials for image type objects as well.  Please see new attached file.  Thanks.
Comment 9 bugs 2010-04-23 13:42:27 PDT
Created attachment 54188 [details]
More complete log file
Comment 10 Alexey Proskuryakov 2010-04-23 14:35:05 PDT
Yes, the more complete log does look wrong. Some more questions:
- Which OS version are you running Safari with?
- Does this happen with a nightly build from <http://nightly.webkit.org>?

Also, a tcpdump/wireshark trace would still be helpful. I understand it can be hard to sanitize, so just a few full requests/responses ripped out from it would help.
Comment 11 bugs 2010-04-23 15:09:31 PDT
(In reply to comment #10)
> Yes, the more complete log does look wrong. Some more questions:
> - Which OS version are you running Safari with?

I actually have no Safari installations here locally.  To do the test for you, I downloaded Safari 4.0.5 (531.22.7) and running it on Win XP.  Like I said, from my server logs, I see this problem with a variety of OS and browser versions.  For example I see it on Mac OS X and on my own Android based G1 phone.  All the browsers I see this on are webkit based. That's why I assumed it may have to do with webkit.



> - Does this happen with a nightly build from <http://nightly.webkit.org>?

I am sure it does.  This problem has been on my back for over a year and a half.  I have also been able to see similar complaints on the web with no real solution.  Here are a few examples:


http://www.gossamer-threads.com/lists/zope/users/223295
http://discussions.apple.com/thread.jspa?threadID=2136821&tstart=60
http://www.experts-exchange.com/Software/Internet_Email/Web_Browsers/Safari/Q_23153036.html


> 
> Also, a tcpdump/wireshark trace would still be helpful. I understand it can be
> hard to sanitize, so just a few full requests/responses ripped out from it
> would help.

I have such a trace.  I setup another server with the same application and running on http port rather than https.  I can send you the complete trace.  Rather than post it here, can I e-mail it to you directly?
Comment 12 Alexey Proskuryakov 2010-04-23 15:19:33 PDT
> Rather than post it here, can I e-mail it to you directly?

Yes, please.
Comment 13 Alexey Proskuryakov 2010-05-11 15:47:02 PDT
I do see a problem in Safari with the packet trace I got. We only remember default credentials for a directory when getting credentials from the user, not when getting them from credential storage. I have a fix for this.

This is purely a Safari 4 issue, which couldn't possibly affect any other version of Safari or any other browser. Please file new bugs if you can observe this problem with any other WebKit-based browser, or if a similar will still be present in Safari.
Comment 14 Alexey Proskuryakov 2010-05-11 15:52:12 PDT
Created attachment 55769 [details]
proposed fix
Comment 15 Alexey Proskuryakov 2010-05-11 16:42:58 PDT
Committed <http://trac.webkit.org/changeset/59185>.

Please feel free to verify this fix with the next nightly build from <http://nightly.webkit.org>.
Comment 16 WebKit Review Bot 2010-05-11 17:12:25 PDT
http://trac.webkit.org/changeset/59185 might have broken GTK Linux 32-bit Release
The following changes are on the blame list:
http://trac.webkit.org/changeset/59184
http://trac.webkit.org/changeset/59185
Comment 17 Alexey Proskuryakov 2010-05-11 17:27:49 PDT
Skipped the test on Gtk, filed bug 38956.
Comment 18 Alexey Proskuryakov 2010-05-11 19:41:38 PDT
<rdar://problem/7972450>
Comment 19 Eric Seidel (no email) 2010-05-11 20:17:23 PDT
Appears to need an expectations update on Qt:
http://build.webkit.org/results/Qt%20Linux%20Release/r59198%20(11612)/http/tests/xmlhttprequest/null-auth-diffs.txt
Comment 20 Alexey Proskuryakov 2010-05-11 20:20:47 PDT
> Appears to need an expectations update on Qt:

That's not a good change, and I don't understand it. But there were no code changes that could possibly affect Qt, so it might be some kind of fallback from a newly added test.
Comment 21 Eric Seidel (no email) 2010-05-11 20:28:16 PDT
I should note that http://trac.webkit.org/changeset/59184 is also on the blame list for the Qt test change, however that commit looks unrelated to this failure.
Comment 22 Eric Seidel (no email) 2010-05-12 01:00:01 PDT
The regression to Qt is still in:
http://build.webkit.org/results/Qt%20Linux%20Release/r59212%20(11625)/http/tests/xmlhttprequest/null-auth-pretty-diff.html

Should we skip this test for now?  Update the results?

Ap said over IRC that these new results were bad, but he didn't know why they were happening.

Either way we need to get back to a green state, ideally by fixing whatever regression this may be. :)
Comment 23 Alexey Proskuryakov 2010-05-12 08:28:29 PDT
Qt issues are being investigated in bug 32961.