207986 – [GTK][Tests] Layout test result upload step fails with too many files exception (needs Zip64)

RESOLVED WORKSFORME 207986

[GTK][Tests] Layout test result upload step fails with too many files exception (needs Zip64)

https://bugs.webkit.org/show_bug.cgi?id=207986

Summary [GTK][Tests] Layout test result upload step fails with too many files excepti...

Lauro Moura

Reported 2020-02-19 19:56:18 PST

Sample trace: 18:14:23.533 85898 Finished writing JSON file for the test results server. 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit Release (Tests) 18:14:27.317 85898 JSON uploaded. 18:14:27.361 85898 18:14:27.361 85898 Preparing to upload test archive ... LargeZipFile raised: Files count would require ZIP64 extensions Traceback (most recent call last): File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 91, in main run_details = run(port, options, args, stderr) File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 485, in run run_details = manager.run(args) File "/home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 325, in run shutil.make_archive(archive, 'zip', self._results_directory) File "/usr/lib/python2.7/shutil.py", line 595, in make_archive filename = func(base_name, base_dir, **kwargs) File "/usr/lib/python2.7/shutil.py", line 497, in _make_zipfile zf.write(path, path) File "/usr/lib/python2.7/zipfile.py", line 1171, in write self._writecheck(zinfo) File "/usr/lib/python2.7/zipfile.py", line 1137, in _writecheck " would require ZIP64 extensions") LargeZipFile: Files count would require ZIP64 extensions program finished with exit code 254 elapsedTime=3965.470351 Python 3.4 enabled by allowZip64 by default (in 2014), but shutil.make_archive does not expose this option.

Attachments
Tentative patch, replacing shutil with raw ZipFile (2.78 KB, patch) 2020-02-19 20:05 PST, Lauro Moura	no flags	Details Formatted Diff Diff
View All Add attachment proposed patch, testcase, etc.

Lauro Moura

Comment 1 2020-02-19 20:05:49 PST

Created attachment 391248 [details] Tentative patch, replacing shutil with raw ZipFile

Lauro Moura

Comment 2 2020-02-19 20:10:16 PST

Maybe instead of the patch we could run `run-webkit-tests` with python 3.4+ in the GTK bots?

Carlos Alberto Lopez Perez

Comment 3 2020-02-20 03:19:39 PST

(In reply to Lauro Moura from comment #0) > Sample trace: > > 18:14:23.533 85898 Finished writing JSON file for the test results server. > 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit > Release (Tests) > 18:14:27.317 85898 JSON uploaded. > 18:14:27.361 85898 > 18:14:27.361 85898 Preparing to upload test archive ... > > LargeZipFile raised: Files count would require ZIP64 extensions Which caused this? This was not happening before.

Carlos Alberto Lopez Perez

Comment 4 2020-02-20 03:27:46 PST

(In reply to Carlos Alberto Lopez Perez from comment #3) > (In reply to Lauro Moura from comment #0) > > Sample trace: > > > > 18:14:23.533 85898 Finished writing JSON file for the test results server. > > 18:14:23.533 85898 Uploading JSON files for builder: GTK Linux 64-bit > > Release (Tests) > > 18:14:27.317 85898 JSON uploaded. > > 18:14:27.361 85898 > > 18:14:27.361 85898 Preparing to upload test archive ... > > > > LargeZipFile raised: Files count would require ZIP64 extensions > > Which caused this? This was not happening before. Also.. why it only happens on GTK but not on WPE? I'm seeing a different kind of exception on the GTK bot now: 00:35:59.580 100943 Preparing to upload test archive ... 00:36:05.478 100943 Uploading archive to https://results.webkit.org ... 00:36:05.531 100943 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 00:36:13.093 100943 https://results.webkit.org:443 "POST /api/upload/archive HTTP/1.1" 403 146 00:36:13.094 100943 Upload to https://results.webkit.org failed: 00:36:13.095 100943 <html> 00:36:13.095 100943 <head><title>403 Forbidden</title></head> 00:36:13.095 100943 <body> 00:36:13.095 100943 <center><h1>403 Forbidden</h1></center> 00:36:13.095 100943 <hr><center>Apple</center> 00:36:13.095 100943 </body> 00:36:13.095 100943 </html> 00:36:13.095 100943 This error is not fatal, continuing 00:36:13.128 100943 Testing completed, Exit status: 50 Something smells weird here... this have to be investigated to know what its causing this

Lauro Moura

Comment 5 2020-02-20 04:09:24 PST

(In reply to Carlos Alberto Lopez Perez from comment #4) > Also.. why it only happens on GTK but not on WPE? WPE buid 17060: 41170 tests ran as expected, 120 didn't GTK failed build 12712: 53188 tests ran as expected, 118 didn't Maybe just a matter of time before we hit this with WPE too? > > I'm seeing a different kind of exception on the GTK bot now: > > 00:35:59.580 100943 Preparing to upload test archive ... > 00:36:05.478 100943 Uploading archive to https://results.webkit.org ... > 00:36:05.531 100943 Starting new HTTPS connection (1): results.webkit.org:443 > /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/ > webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: > InsecureRequestWarning: Unverified HTTPS request is being made. Adding > certificate verification is strongly advised. See: > https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > InsecureRequestWarning, > 00:36:13.093 100943 https://results.webkit.org:443 "POST /api/upload/archive > HTTP/1.1" 403 146 > 00:36:13.094 100943 Upload to https://results.webkit.org failed: > 00:36:13.095 100943 <html> > 00:36:13.095 100943 <head><title>403 Forbidden</title></head> > 00:36:13.095 100943 <body> > 00:36:13.095 100943 <center><h1>403 Forbidden</h1></center> > 00:36:13.095 100943 <hr><center>Apple</center> > 00:36:13.095 100943 </body> > 00:36:13.095 100943 </html> > 00:36:13.095 100943 This error is not fatal, continuing > 00:36:13.128 100943 Testing completed, Exit status: 50 > > > Something smells weird here... this have to be investigated to know what its > causing this Mojave WK2 with the same 403 issue: https://build.webkit.org/builders/Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10510/steps/layout-test/logs/stdio

Carlos Alberto Lopez Perez

Comment 6 2020-02-20 04:22:44 PST

(In reply to Lauro Moura from comment #5) > (In reply to Carlos Alberto Lopez Perez from comment #4) > > Also.. why it only happens on GTK but not on WPE? > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > Maybe just a matter of time before we hit this with WPE too? > > LargeZipFile: Files count would require ZIP64 extensions Which limit its being hit that caused this to be raised? on the GTK Release test bot I see layout-test-results.zip at /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb.

Lauro Moura

Comment 7 2020-02-20 04:38:18 PST

(In reply to Carlos Alberto Lopez Perez from comment #6) > (In reply to Lauro Moura from comment #5) > > (In reply to Carlos Alberto Lopez Perez from comment #4) > > > Also.. why it only happens on GTK but not on WPE? > > > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > > > Maybe just a matter of time before we hit this with WPE too? > > > > > > > LargeZipFile: Files count would require ZIP64 extensions > > Which limit its being hit that caused this to be raised? > > on the GTK Release test bot I see layout-test-results.zip at > /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb. Number of files in the archive. Regular zip files have a limit of 64k entries. https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64

Carlos Alberto Lopez Perez

Comment 8 2020-02-20 04:45:14 PST

(In reply to Lauro Moura from comment #7) > (In reply to Carlos Alberto Lopez Perez from comment #6) > > (In reply to Lauro Moura from comment #5) > > > (In reply to Carlos Alberto Lopez Perez from comment #4) > > > > Also.. why it only happens on GTK but not on WPE? > > > > > > WPE buid 17060: 41170 tests ran as expected, 120 didn't > > > GTK failed build 12712: 53188 tests ran as expected, 118 didn't > > > > > > Maybe just a matter of time before we hit this with WPE too? > > > > > > > > > > > > LargeZipFile: Files count would require ZIP64 extensions > > > > Which limit its being hit that caused this to be raised? > > > > on the GTK Release test bot I see layout-test-results.zip at > > /home/buildbot/webkitgtk/gtk-linux-64-release/build is only 41Mb. > > Number of files in the archive. Regular zip files have a limit of 64k > entries. > > https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64 I still don't see how we can be hitting that. For the last run on the GTK bot, 256917 I see the zip generated only contains 15.8K files inside. You can find the zips generated here: https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/ cd $(mktemp -d) wget 'https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/r256917%20%2812705%29.zip' unzip 'r256917 (12705).zip' $ find|wc -l 15800

Carlos Alberto Lopez Perez

Comment 9 2020-02-20 05:40:10 PST

I triggered a clean build on the GTK bot and it seems to be working fine noew: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/12718

Carlos Alberto Lopez Perez

Comment 10 2020-02-20 05:44:03 PST

(In reply to Carlos Alberto Lopez Perez from comment #9) > I triggered a clean build on the GTK bot and it seems to be working fine > noew: > > https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/12718 Well, at least the zip creation seems working... There its still the failure with 05:37:28.676 93202 Preparing to upload test archive ... 05:37:41.563 93202 Uploading archive to https://results.webkit.org ... 05:37:41.638 93202 Starting new HTTPS connection (1): results.webkit.org:443 /home/slave/webkitgtk/gtk-linux-64-release-tests/build/Tools/Scripts/webkitpy/thirdparty/autoinstalled/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, 05:37:47.138 93202 https://results.webkit.org:443 "POST /api/upload/archive HTTP/1.1" 403 146 05:37:47.140 93202 Upload to https://results.webkit.org failed: 05:37:47.141 93202 <html> 05:37:47.142 93202 <head><title>403 Forbidden</title></head> 05:37:47.142 93202 <body> 05:37:47.142 93202 <center><h1>403 Forbidden</h1></center> 05:37:47.142 93202 <hr><center>Apple</center> 05:37:47.142 93202 </body> 05:37:47.142 93202 </html> 05:37:47.142 93202 This error is not fatal, continuing 05:37:47.186 93202 Testing completed, Exit status: 36

Lauro Moura

Comment 11 2020-02-20 06:40:12 PST

(In reply to Carlos Alberto Lopez Perez from comment #10) > Well, at least the zip creation seems working... > > There its still the failure with > snip. > 05:37:47.142 93202 <head><title>403 Forbidden</title></head> This seems to be happening to Mojave since Feb 13th at least: https://build.webkit.org/builders/Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10311/steps/layout-test/logs/stdio And in GTK release, build 12520 from Jan 30th shows 403 error but no detailed html error message. Thee detailed message started with r255500 (gtk release build 12535 is the first one with it) (https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/12535). So, it seems there is indeed something odd with uploading results (but kinda expected, as 403 is not a fatal error). But not related to this ZIP64 issue. Should we close this with invalid or wontfix? In case the exception appears again we could reopen it.

Jonathan Bedard

Comment 12 2020-02-20 07:39:01 PST

(In reply to Lauro Moura from comment #11) > (In reply to Carlos Alberto Lopez Perez from comment #10) > > Well, at least the zip creation seems working... > > > > There its still the failure with > > > > snip. > > > 05:37:47.142 93202 <head><title>403 Forbidden</title></head> > > This seems to be happening to Mojave since Feb 13th at least: > > https://build.webkit.org/builders/ > Apple%20Mojave%20Release%20WK2%20%28Tests%29/builds/10311/steps/layout-test/ > logs/stdio > > And in GTK release, build 12520 from Jan 30th shows 403 error but no > detailed html error message. > > Thee detailed message started with r255500 (gtk release build 12535 is the > first one with it) > (https://build.webkit.org/builders/GTK%20Linux%2064- > bit%20Release%20%28Tests%29/builds/12535). > > So, it seems there is indeed something odd with uploading results (but kinda > expected, as 403 is not a fatal error). But not related to this ZIP64 issue. > > Should we close this with invalid or wontfix? In case the exception appears > again we could reopen it. That's a failure server side. At the moment, we're deliberately ignoring them because there is still work to be done with the reporting service to make upload archives useful, and we're still not 100% sure we want to go with this approach. Basically, we have a load-balancer on https://results.webkit.org that is rejecting upload archives because they're too large. We will, in the relatively near future, be fixing this, so I think this change is worth our time.

Jonathan Bedard

Comment 13 2020-02-20 07:44:58 PST

Comment on attachment 391248 [details] Tentative patch, replacing shutil with raw ZipFile View in context: https://bugs.webkit.org/attachment.cgi?id=391248&action=review Heads up, EWS does not exercise this code-path. So a green EWS doesn't guarantee this its actually working. > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:180 > + def _make_archive(self, archive, results_directory): Since we aren't actually using self, can we make this an @staticmethod or an @classmethod?

Lauro Moura

Comment 14 2020-03-02 06:13:33 PST

(In reply to Jonathan Bedard from comment #13) > Comment on attachment 391248 [details] > Tentative patch, replacing shutil with raw ZipFile > > View in context: > https://bugs.webkit.org/attachment.cgi?id=391248&action=review > > Heads up, EWS does not exercise this code-path. So a green EWS doesn't > guarantee this its actually working. > > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:180 > > + def _make_archive(self, archive, results_directory): > > Since we aren't actually using self, can we make this an @staticmethod or an > @classmethod? Thanks for the review Jonathan. I'll update the patch if this happens again.

Note You need to log in before you can comment on or make changes to this bug.

Status RESOLVED

Resolution WORKSFORME

Priority P2

Severity Critical

Classification Unclassified

Version Other

Hardware Unspecified

OS Unspecified

Product WebKit

Component Tools / Tests

Assignee

Lauro Moura

Reported

2020-02-19 19:56 PST

Modified

2020-06-04 08:07 PDT History

CC List

6 users Show

URL

Keywords

Depends on

Blocks