Bug 168774 - Add a test verifying cache deduplication is not sensitive to SHA1 collision attack
Summary: Add a test verifying cache deduplication is not sensitive to SHA1 collision a...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Page Loading (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-23 07:48 PST by Antti Koivisto
Modified: 2017-02-27 07:29 PST (History)
17 users (show)

See Also:


Attachments
patch (2.19 MB, patch)
2017-02-23 08:14 PST, Antti Koivisto
sam: review+
buildbot: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ews104 for mac-elcapitan-wk2 (1.08 MB, application/zip)
2017-02-23 09:26 PST, Build Bot
no flags Details
Archive of layout-test-results from ews123 for ios-simulator-wk2 (1017.77 KB, application/zip)
2017-02-23 09:31 PST, Build Bot
no flags Details
patch (2.19 MB, patch)
2017-02-23 09:42 PST, Antti Koivisto
buildbot: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ews107 for mac-elcapitan-wk2 (1.36 MB, application/zip)
2017-02-23 10:56 PST, Build Bot
no flags Details
Archive of layout-test-results from ews124 for ios-simulator-wk2 (1011.75 KB, application/zip)
2017-02-23 11:01 PST, Build Bot
no flags Details
try to deal with inconsistent pdf rendering (2.19 MB, patch)
2017-02-23 11:03 PST, Antti Koivisto
buildbot: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ews105 for mac-elcapitan-wk2 (1.12 MB, application/zip)
2017-02-23 11:56 PST, Build Bot
no flags Details
Archive of layout-test-results from ews124 for ios-simulator-wk2 (1.08 MB, application/zip)
2017-02-23 12:01 PST, Build Bot
no flags Details
try to deal with inconsistent pdf rendering (2.19 MB, patch)
2017-02-23 12:52 PST, Antti Koivisto
buildbot: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ews105 for mac-elcapitan-wk2 (1.36 MB, application/zip)
2017-02-23 13:42 PST, Build Bot
no flags Details
Archive of layout-test-results from ews121 for ios-simulator-wk2 (1.02 MB, application/zip)
2017-02-23 14:11 PST, Build Bot
no flags Details
try iframe instead of img to get consistent rendering (2.19 MB, patch)
2017-02-24 03:10 PST, Antti Koivisto
commit-queue: commit-queue-
Details | Formatted Diff | Diff
now with php script to generate the colliding files in memory (1.09 MB, patch)
2017-02-25 01:46 PST, Antti Koivisto
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Antti Koivisto 2017-02-23 07:48:56 PST
We use SHA1 for deduplicating disk cache resources. Since a real world SHA1 collision was demonstrated recently (http://shattered.io/) we can add a test that shows it can't be used for cache poisoning.
Comment 1 Antti Koivisto 2017-02-23 08:14:07 PST
Created attachment 302513 [details]
patch
Comment 2 Sam Weinig 2017-02-23 08:24:24 PST
Comment on attachment 302513 [details]
patch

Nice!
Comment 3 Build Bot 2017-02-23 09:25:58 PST
Comment on attachment 302513 [details]
patch

Attachment 302513 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/3179194

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 4 Build Bot 2017-02-23 09:26:02 PST
Created attachment 302519 [details]
Archive of layout-test-results from ews104 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6
Comment 5 Build Bot 2017-02-23 09:31:54 PST
Comment on attachment 302513 [details]
patch

Attachment 302513 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3179187

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 6 Build Bot 2017-02-23 09:31:59 PST
Created attachment 302520 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6
Comment 7 Antti Koivisto 2017-02-23 09:42:39 PST
Created attachment 302522 [details]
patch
Comment 8 Build Bot 2017-02-23 10:56:19 PST
Comment on attachment 302522 [details]
patch

Attachment 302522 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/3179522

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 9 Build Bot 2017-02-23 10:56:23 PST
Created attachment 302537 [details]
Archive of layout-test-results from ews107 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6
Comment 10 Build Bot 2017-02-23 11:01:33 PST
Comment on attachment 302522 [details]
patch

Attachment 302522 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3179521

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 11 Build Bot 2017-02-23 11:01:38 PST
Created attachment 302539 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6
Comment 12 Antti Koivisto 2017-02-23 11:03:19 PST
Created attachment 302540 [details]
try to deal with inconsistent pdf rendering
Comment 13 Build Bot 2017-02-23 11:56:49 PST
Comment on attachment 302540 [details]
try to deal with inconsistent pdf rendering

Attachment 302540 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/3179877

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 14 Build Bot 2017-02-23 11:56:56 PST
Created attachment 302550 [details]
Archive of layout-test-results from ews105 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6
Comment 15 Build Bot 2017-02-23 12:01:34 PST
Comment on attachment 302540 [details]
try to deal with inconsistent pdf rendering

Attachment 302540 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3179874

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 16 Build Bot 2017-02-23 12:01:39 PST
Created attachment 302552 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6
Comment 17 Antti Koivisto 2017-02-23 12:52:30 PST
Created attachment 302559 [details]
try to deal with inconsistent pdf rendering
Comment 18 Build Bot 2017-02-23 13:42:44 PST
Comment on attachment 302559 [details]
try to deal with inconsistent pdf rendering

Attachment 302559 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/3180793

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 19 Build Bot 2017-02-23 13:42:49 PST
Created attachment 302569 [details]
Archive of layout-test-results from ews105 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6
Comment 20 Build Bot 2017-02-23 14:11:23 PST
Comment on attachment 302559 [details]
try to deal with inconsistent pdf rendering

Attachment 302559 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3181004

New failing tests:
http/tests/cache/disk-cache/shattered-deduplication.html
Comment 21 Build Bot 2017-02-23 14:11:27 PST
Created attachment 302572 [details]
Archive of layout-test-results from ews121 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6
Comment 22 Antti Koivisto 2017-02-24 03:10:50 PST
Created attachment 302662 [details]
try iframe instead of img to get consistent rendering
Comment 23 WebKit Commit Bot 2017-02-24 04:47:44 PST
Comment on attachment 302662 [details]
try iframe instead of img to get consistent rendering

Rejecting attachment 302662 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'land-attachment', '--force-clean', '--non-interactive', '--parent-command=commit-queue', 302662, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
ttered-1-nocollision.pdf
	A	LayoutTests/http/tests/cache/disk-cache/resources/shattered-1.pdf
	A	LayoutTests/http/tests/cache/disk-cache/resources/shattered-2-nocollision.pdf
Checksum mismatch: LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf
expected: 5bd9d8cabc46041579a311230539b8d1
    got: ee4aa52b139d925f8d8884402b0a750c


Failed to run "['git', 'svn', 'dcommit', '--rmdir']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit
Updating OpenSource
Current branch master is up to date.

Full output: http://webkit-queues.webkit.org/results/3184769
Comment 24 Antti Koivisto 2017-02-24 05:16:18 PST
Uh, looks like landing this doesn't actually work due to git infrastructure. Oh well.
Comment 25 Antti Koivisto 2017-02-24 05:20:25 PST
Reverted the partial commit in https://trac.webkit.org/r212951
Comment 26 Antti Koivisto 2017-02-24 05:20:58 PST
I mean https://trac.webkit.org/changeset/212952
Comment 27 Carlos Alberto Lopez Perez 2017-02-24 06:07:06 PST
It seems that the git-svn mirror stopped updating at r212950, and the bots all are red, the svn client prints an error that looks like:

0svn: E200014: Checksum mismatch for [...] shattered-2.pdf'
Comment 28 Michael Catanzaro 2017-02-24 06:11:40 PST
Oh wow, incredible.

Is it fixable, or are we just totally hosed? Are we going to need to delete all the SVN history since this commit from the server in order to avoid the hash collision?
Comment 29 Carlos Alberto Lopez Perez 2017-02-24 06:24:18 PST
(In reply to comment #28)
> Oh wow, incredible.
> 
> Is it fixable, or are we just totally hosed? Are we going to need to delete
> all the SVN history since this commit from the server in order to avoid the
> hash collision?

It seems its not a problem with GIT, but with SVN ?

This simple SVN checkout on a new empty directory will fail:

$ svn co https://svn.webkit.org/repository/webkit/trunk/LayoutTests/http/tests/cache/

output: http://sprunge.us/XffD
Comment 30 Michael Catanzaro 2017-02-24 08:17:42 PST
(In reply to comment #28)
> Oh wow, incredible.
> 
> Is it fixable, or are we just totally hosed? Are we going to need to delete
> all the SVN history since this commit from the server in order to avoid the
> hash collision?

For the record: the commits have been deleted, but the SVN is still hosed.

Mailing list thread: https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html
Comment 31 Antti Koivisto 2017-02-24 09:28:50 PST
Checkouts should be working again.
Comment 32 Csaba Osztrogonác_OOO_until_21st_Aug 2017-02-24 09:29:37 PST
(In reply to comment #30)
> (In reply to comment #28)
> > Oh wow, incredible.
> > 
> > Is it fixable, or are we just totally hosed? Are we going to need to delete
> > all the SVN history since this commit from the server in order to avoid the
> > hash collision?
> 
> For the record: the commits have been deleted, but the SVN is still hosed.
> 
> Mailing list thread:
> https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html

It broke our SVN mirror too, it can't be sync anymore after r212950:

Transmitting file data .....svnsync: E200014: Checksum mismatch for resulting fulltext
(/trunk/LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf):
   expected:  5bd9d8cabc46041579a311230539b8d1
     actual:  ee4aa52b139d925f8d8884402b0a750c
Comment 33 Carlos Alberto Lopez Perez 2017-02-24 09:36:35 PST
(In reply to comment #32)
> (In reply to comment #30)
> > (In reply to comment #28)
> > > Oh wow, incredible.
> > > 
> > > Is it fixable, or are we just totally hosed? Are we going to need to delete
> > > all the SVN history since this commit from the server in order to avoid the
> > > hash collision?
> > 
> > For the record: the commits have been deleted, but the SVN is still hosed.
> > 
> > Mailing list thread:
> > https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html
> 
> It broke our SVN mirror too, it can't be sync anymore after r212950:
> 
> Transmitting file data .....svnsync: E200014: Checksum mismatch for
> resulting fulltext
> (/trunk/LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf):
>    expected:  5bd9d8cabc46041579a311230539b8d1
>      actual:  ee4aa52b139d925f8d8884402b0a750c

Right.

And this is the proper way to fix this.

The current repository should be replaced with a mirror until r212950
Comment 34 Antti Koivisto 2017-02-24 12:23:46 PST
Things have recovered. More info here https://lists.webkit.org/pipermail/webkit-dev/2017-February/028800.html
Comment 35 Carlos Alberto Lopez Perez 2017-02-24 12:37:56 PST
This wontfix means that we won't have sha1 collision tests because of a SVN limitation???

I would like to switch to git, but I know that such thing won't happen easily.

So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64?

Something like

<embed src="data:application/pdf;base64,base64encodedpdf"> .... 

?
Comment 36 Konstantin Tokarev 2017-02-24 12:49:31 PST
>So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64?

AFAIU point of test was to have 2 different cached resources with equal SHA1. Embedding into HTML will break collision.
Comment 37 Antti Koivisto 2017-02-24 12:52:08 PST
Yeah, we can keep the bug open waiting for the day we have a version control that can handle this.
Comment 38 Carlos Alberto Lopez Perez 2017-02-24 13:02:00 PST
(In reply to comment #36)
> >So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64?
> 
> AFAIU point of test was to have 2 different cached resources with equal
> SHA1. Embedding into HTML will break collision.

Another idea is to download the pdf files from somewhere before starting the test?
Comment 39 Michael Catanzaro 2017-02-24 13:15:04 PST
(In reply to comment #38)
> Another idea is to download the pdf files from somewhere before starting the
> test?

Do we currently have any tests that depend on network access (besides to the local Apache test server)? I don't think so. Let's not start that now.
Comment 40 Andy Estes 2017-02-24 13:22:16 PST
(In reply to comment #37)
> Yeah, we can keep the bug open waiting for the day we have a version control
> that can handle this.

It's possible to disable the Subversion feature that caused this collision by setting CONFIG_OPTION_ENABLE_REP_SHARING = false, at the expense of using extra space on the server.
Comment 41 Carlos Alberto Lopez Perez 2017-02-24 13:30:27 PST
(In reply to comment #39)
> (In reply to comment #38)
> > Another idea is to download the pdf files from somewhere before starting the
> > test?
> 
> Do we currently have any tests that depend on network access (besides to the
> local Apache test server)? I don't think so. Let's not start that now.

We have tooling that automatically downloads tarballs and decompress them on a specific directory before starting the layout tests.

We use it to auto-install several python libraries locally before running the tests

Check, for example: 
Tools/Scripts/webkitpy/thirdparty/autoinstalled/
LayoutTests/imported/w3c/web-platform-tests/tools/

All the stuff on that directories got downloaded from internet and decompressed there.

So, maybe an idea is creating a git repository named sha1-collisions, upload it to github and add it to the list of things that should be auto-installed before running the tests.

Then the tests can rely on all this files beeing on the filesystem when they start.
Comment 42 Carlos Alberto Lopez Perez 2017-02-24 13:44:18 PST
I got this suggestion via e-mail:


    You should try zipping the PDFs. They should end up with different hashes in that case. The test harness can unzip them before the test...
Comment 43 Ryosuke Niwa 2017-02-24 13:54:02 PST
We can probably generate files (e.g. by unzipping) in /tmp/ and symlink to a valid path in a php script during the test.
Comment 44 Michael Catanzaro 2017-02-24 14:00:55 PST
Yeah, that's a good idea. Thanks, Internet publicity!
Comment 45 Alexey Proskuryakov 2017-02-24 19:58:09 PST
We now have rep sharing disabled, so technically, re-landing the test would not cause the problem. But it would break mirrors that haven't made this change, so let's not do that. 

As the tests can incorporate PHP or any CGI script, it would indeed be straightforward to do some processing at run time. It can be done entirely in memory, no need to lay down temporary files. Can be any kind of archive or even as simple as flipping one bit.
Comment 46 Antti Koivisto 2017-02-25 01:46:05 PST
Created attachment 302755 [details]
now with php script to generate the colliding files in memory

This patch contains non-colliding versions of the files only:

> shasum shattered-nocollision-*
5439274cf677fe3b7c51264f88a5ecee97319ee9  shattered-nocollision-1.pdf
7fdd163dc21064b7f26e1199fc560ee6e0307498  shattered-nocollision-2.pdf

make-sha1-collision.php turns them into colliding ones with simple string replacement.
Comment 47 Michael Catanzaro 2017-02-26 11:04:21 PST
The awaits mean the test will take four seconds to complete...? :/
Comment 48 Michael Catanzaro 2017-02-26 11:04:53 PST
(In reply to comment #47)
> The awaits mean the test will take four seconds to complete...? :/

Can they be run simultaneously instead?
Comment 49 Ryosuke Niwa 2017-02-26 17:11:15 PST
Comment on attachment 302755 [details]
now with php script to generate the colliding files in memory

Is this patch up for review?
Comment 50 Antti Koivisto 2017-02-26 23:29:52 PST
(In reply to comment #49)
> Is this patch up for review?

Was waiting for the buildbots to come back.
Comment 51 Antti Koivisto 2017-02-26 23:30:12 PST
ews that is
Comment 52 Antti Koivisto 2017-02-26 23:34:46 PST
(In reply to comment #47)
> The awaits mean the test will take four seconds to complete...? :/

For the test to work the blobs need to land to disk. That's when the depduplication kicks in. There are some write delays that need to be compensated for.
Comment 53 Alex Christensen 2017-02-27 06:57:57 PST
Comment on attachment 302755 [details]
now with php script to generate the colliding files in memory

View in context: https://bugs.webkit.org/attachment.cgi?id=302755&action=review

> LayoutTests/http/tests/cache/disk-cache/resources/make-sha1-collision.php:7
> +$collidingContent = str_replace("SVN is the best!", "SHA-1 is dead!!!", $content);

lol
Comment 54 WebKit Commit Bot 2017-02-27 07:29:02 PST
Comment on attachment 302755 [details]
now with php script to generate the colliding files in memory

Clearing flags on attachment: 302755

Committed r213064: <http://trac.webkit.org/changeset/213064>
Comment 55 WebKit Commit Bot 2017-02-27 07:29:13 PST
All reviewed patches have been landed.  Closing bug.