We use SHA1 for deduplicating disk cache resources. Since a real world SHA1 collision was demonstrated recently (http://shattered.io/) we can add a test that shows it can't be used for cache poisoning.
Created attachment 302513 [details] patch
Comment on attachment 302513 [details] patch Nice!
Comment on attachment 302513 [details] patch Attachment 302513 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3179194 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302519 [details] Archive of layout-test-results from ews104 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews104 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Comment on attachment 302513 [details] patch Attachment 302513 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/3179187 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302520 [details] Archive of layout-test-results from ews123 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews123 Port: ios-simulator-wk2 Platform: Mac OS X 10.11.6
Created attachment 302522 [details] patch
Comment on attachment 302522 [details] patch Attachment 302522 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3179522 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302537 [details] Archive of layout-test-results from ews107 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews107 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Comment on attachment 302522 [details] patch Attachment 302522 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/3179521 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302539 [details] Archive of layout-test-results from ews124 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews124 Port: ios-simulator-wk2 Platform: Mac OS X 10.11.6
Created attachment 302540 [details] try to deal with inconsistent pdf rendering
Comment on attachment 302540 [details] try to deal with inconsistent pdf rendering Attachment 302540 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3179877 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302550 [details] Archive of layout-test-results from ews105 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews105 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Comment on attachment 302540 [details] try to deal with inconsistent pdf rendering Attachment 302540 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/3179874 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302552 [details] Archive of layout-test-results from ews124 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews124 Port: ios-simulator-wk2 Platform: Mac OS X 10.11.6
Created attachment 302559 [details] try to deal with inconsistent pdf rendering
Comment on attachment 302559 [details] try to deal with inconsistent pdf rendering Attachment 302559 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3180793 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302569 [details] Archive of layout-test-results from ews105 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews105 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Comment on attachment 302559 [details] try to deal with inconsistent pdf rendering Attachment 302559 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/3181004 New failing tests: http/tests/cache/disk-cache/shattered-deduplication.html
Created attachment 302572 [details] Archive of layout-test-results from ews121 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews121 Port: ios-simulator-wk2 Platform: Mac OS X 10.11.6
Created attachment 302662 [details] try iframe instead of img to get consistent rendering
Comment on attachment 302662 [details] try iframe instead of img to get consistent rendering Rejecting attachment 302662 [details] from commit-queue. Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'land-attachment', '--force-clean', '--non-interactive', '--parent-command=commit-queue', 302662, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit Last 500 characters of output: ttered-1-nocollision.pdf A LayoutTests/http/tests/cache/disk-cache/resources/shattered-1.pdf A LayoutTests/http/tests/cache/disk-cache/resources/shattered-2-nocollision.pdf Checksum mismatch: LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf expected: 5bd9d8cabc46041579a311230539b8d1 got: ee4aa52b139d925f8d8884402b0a750c Failed to run "['git', 'svn', 'dcommit', '--rmdir']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit Updating OpenSource Current branch master is up to date. Full output: http://webkit-queues.webkit.org/results/3184769
Uh, looks like landing this doesn't actually work due to git infrastructure. Oh well.
Reverted the partial commit in https://trac.webkit.org/r212951
I mean https://trac.webkit.org/changeset/212952
It seems that the git-svn mirror stopped updating at r212950, and the bots all are red, the svn client prints an error that looks like: 0svn: E200014: Checksum mismatch for [...] shattered-2.pdf'
Oh wow, incredible. Is it fixable, or are we just totally hosed? Are we going to need to delete all the SVN history since this commit from the server in order to avoid the hash collision?
(In reply to comment #28) > Oh wow, incredible. > > Is it fixable, or are we just totally hosed? Are we going to need to delete > all the SVN history since this commit from the server in order to avoid the > hash collision? It seems its not a problem with GIT, but with SVN ? This simple SVN checkout on a new empty directory will fail: $ svn co https://svn.webkit.org/repository/webkit/trunk/LayoutTests/http/tests/cache/ output: http://sprunge.us/XffD
(In reply to comment #28) > Oh wow, incredible. > > Is it fixable, or are we just totally hosed? Are we going to need to delete > all the SVN history since this commit from the server in order to avoid the > hash collision? For the record: the commits have been deleted, but the SVN is still hosed. Mailing list thread: https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html
Checkouts should be working again.
(In reply to comment #30) > (In reply to comment #28) > > Oh wow, incredible. > > > > Is it fixable, or are we just totally hosed? Are we going to need to delete > > all the SVN history since this commit from the server in order to avoid the > > hash collision? > > For the record: the commits have been deleted, but the SVN is still hosed. > > Mailing list thread: > https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html It broke our SVN mirror too, it can't be sync anymore after r212950: Transmitting file data .....svnsync: E200014: Checksum mismatch for resulting fulltext (/trunk/LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf): expected: 5bd9d8cabc46041579a311230539b8d1 actual: ee4aa52b139d925f8d8884402b0a750c
(In reply to comment #32) > (In reply to comment #30) > > (In reply to comment #28) > > > Oh wow, incredible. > > > > > > Is it fixable, or are we just totally hosed? Are we going to need to delete > > > all the SVN history since this commit from the server in order to avoid the > > > hash collision? > > > > For the record: the commits have been deleted, but the SVN is still hosed. > > > > Mailing list thread: > > https://lists.webkit.org/pipermail/webkit-dev/2017-February/028792.html > > It broke our SVN mirror too, it can't be sync anymore after r212950: > > Transmitting file data .....svnsync: E200014: Checksum mismatch for > resulting fulltext > (/trunk/LayoutTests/http/tests/cache/disk-cache/resources/shattered-2.pdf): > expected: 5bd9d8cabc46041579a311230539b8d1 > actual: ee4aa52b139d925f8d8884402b0a750c Right. And this is the proper way to fix this. The current repository should be replaced with a mirror until r212950
Things have recovered. More info here https://lists.webkit.org/pipermail/webkit-dev/2017-February/028800.html
This wontfix means that we won't have sha1 collision tests because of a SVN limitation??? I would like to switch to git, but I know that such thing won't happen easily. So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64? Something like <embed src="data:application/pdf;base64,base64encodedpdf"> .... ?
>So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64? AFAIU point of test was to have 2 different cached resources with equal SHA1. Embedding into HTML will break collision.
Yeah, we can keep the bug open waiting for the day we have a version control that can handle this.
(In reply to comment #36) > >So maybe an idea is to instead of committing this pdf files with the sha1 collision as files, to embed them on the html of the test with base64? > > AFAIU point of test was to have 2 different cached resources with equal > SHA1. Embedding into HTML will break collision. Another idea is to download the pdf files from somewhere before starting the test?
(In reply to comment #38) > Another idea is to download the pdf files from somewhere before starting the > test? Do we currently have any tests that depend on network access (besides to the local Apache test server)? I don't think so. Let's not start that now.
(In reply to comment #37) > Yeah, we can keep the bug open waiting for the day we have a version control > that can handle this. It's possible to disable the Subversion feature that caused this collision by setting CONFIG_OPTION_ENABLE_REP_SHARING = false, at the expense of using extra space on the server.
(In reply to comment #39) > (In reply to comment #38) > > Another idea is to download the pdf files from somewhere before starting the > > test? > > Do we currently have any tests that depend on network access (besides to the > local Apache test server)? I don't think so. Let's not start that now. We have tooling that automatically downloads tarballs and decompress them on a specific directory before starting the layout tests. We use it to auto-install several python libraries locally before running the tests Check, for example: Tools/Scripts/webkitpy/thirdparty/autoinstalled/ LayoutTests/imported/w3c/web-platform-tests/tools/ All the stuff on that directories got downloaded from internet and decompressed there. So, maybe an idea is creating a git repository named sha1-collisions, upload it to github and add it to the list of things that should be auto-installed before running the tests. Then the tests can rely on all this files beeing on the filesystem when they start.
I got this suggestion via e-mail: You should try zipping the PDFs. They should end up with different hashes in that case. The test harness can unzip them before the test...
We can probably generate files (e.g. by unzipping) in /tmp/ and symlink to a valid path in a php script during the test.
Yeah, that's a good idea. Thanks, Internet publicity!
We now have rep sharing disabled, so technically, re-landing the test would not cause the problem. But it would break mirrors that haven't made this change, so let's not do that. As the tests can incorporate PHP or any CGI script, it would indeed be straightforward to do some processing at run time. It can be done entirely in memory, no need to lay down temporary files. Can be any kind of archive or even as simple as flipping one bit.
Created attachment 302755 [details] now with php script to generate the colliding files in memory This patch contains non-colliding versions of the files only: > shasum shattered-nocollision-* 5439274cf677fe3b7c51264f88a5ecee97319ee9 shattered-nocollision-1.pdf 7fdd163dc21064b7f26e1199fc560ee6e0307498 shattered-nocollision-2.pdf make-sha1-collision.php turns them into colliding ones with simple string replacement.
The awaits mean the test will take four seconds to complete...? :/
(In reply to comment #47) > The awaits mean the test will take four seconds to complete...? :/ Can they be run simultaneously instead?
Comment on attachment 302755 [details] now with php script to generate the colliding files in memory Is this patch up for review?
(In reply to comment #49) > Is this patch up for review? Was waiting for the buildbots to come back.
ews that is
(In reply to comment #47) > The awaits mean the test will take four seconds to complete...? :/ For the test to work the blobs need to land to disk. That's when the depduplication kicks in. There are some write delays that need to be compensated for.
Comment on attachment 302755 [details] now with php script to generate the colliding files in memory View in context: https://bugs.webkit.org/attachment.cgi?id=302755&action=review > LayoutTests/http/tests/cache/disk-cache/resources/make-sha1-collision.php:7 > +$collidingContent = str_replace("SVN is the best!", "SHA-1 is dead!!!", $content); lol
Comment on attachment 302755 [details] now with php script to generate the colliding files in memory Clearing flags on attachment: 302755 Committed r213064: <http://trac.webkit.org/changeset/213064>
All reviewed patches have been landed. Closing bug.
i hate this bug https://www.facebook.com/CreamPemutihWajahAlamiAman/