Bug 304213 – [WPE] Debug bot is unable to upload build products to Amazon S3 due to hitting the 5GB hard limit for PUT requests

• Status: NEW
• Component: WPE WebKit

[Carlos Alberto Lopez Perez]

WPE Debug bot is unable to upload the built products to S3 anymore: https://build.webkit.org/#/builders/66/builds/40810

Upon further manual debugging of the issue inside the machine where this bot runs I can see this:

$ curl -v --trace-ascii /tmp/curl-trace.log -X PUT -H 'Content-Length: 5371794756' -T WebKitBuild/debug.zip 'https://s3.us-west-2.amazonaws.com/archives.webkit.org/wpe-x86_64-debug/304470%40main.zip?AWSAccessKeyId=[redacted]'
Warning: --trace-ascii overrides an earlier trace/verbose option
Note: Unnecessary use of -X or --request, PUT is already inferred.
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>5371794756</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>7WZ8258MSB7KFATF</RequestId><HostId>cRaNq1clrmow4cjwOev+8jOnRDAFPuuDPICRAwH0cjk54ySrcmtww72pggTk42sUozpNI0Ek9fXpGVISrUGC30jkX4dws5ANMrLoccaxC+w=</HostId></Error>

This is related to https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

Depending on the size of the data that you're uploading, Amazon S3 offers the following options:

* Upload an object in a single operation by using the AWS SDKs, REST API, or AWS CLI – With a single PUT operation, you can upload a single object up to 5 GB in size.

* Upload a single object by using the Amazon S3 console – With the Amazon S3 console, you can upload a single object up to 160 GB in size.

* Upload an object in parts by using the AWS SDKs, REST API, or AWS CLI – Using the multipart upload API operation, you can upload a single large object, up to 50 TB in size.


We are hitting the 5GB limit for single PUT operations


The fix to allow uploads of files bigger than 5GB to S3 is to use the multi-part upload API, but that is not trivial as it doesn't work with one single pre-signed URL

[Carlos Alberto Lopez Perez]

This issue has been temporally workaround-ed in WPE by linking jsc and other jsc-related binaries (testb3, testair, TestJavaScriptCore, etc) against libWPEWebKit. See: 304666@main