Bug 152919

Summary: Custom protocol loading through AVFoundation does not support byte-range requests.
Product: WebKit Reporter: Jer Noble <jer.noble>
Component: New BugsAssignee: Jer Noble <jer.noble>
Status: RESOLVED FIXED    
Severity: Normal CC: achristensen, buildbot, ossy, peavo, rniwa
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch
none
Archive of layout-test-results from ews103 for mac-yosemite
none
Archive of layout-test-results from ews112 for mac-yosemite
none
Archive of layout-test-results from ews106 for mac-yosemite-wk2
none
Patch
none
Patch
none
Patch
none
Patch
achristensen: review+, buildbot: commit-queue-
Archive of layout-test-results from ews101 for mac-yosemite
none
Archive of layout-test-results from ews104 for mac-yosemite-wk2
none
Archive of layout-test-results from ews112 for mac-yosemite
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
none
Patch for landing
buildbot: commit-queue-
Archive of layout-test-results from ews113 for mac-yosemite none

Jer Noble
Reported 2016-01-08 12:54:19 PST
Custom protocol loading through AVFoundation does not support byte-range requests.
Attachments
Patch (23.93 KB, patch)
2016-01-11 10:25 PST, Jer Noble
no flags
Patch (31.52 KB, patch)
2016-01-11 15:55 PST, Jer Noble
no flags
Archive of layout-test-results from ews103 for mac-yosemite (944.70 KB, application/zip)
2016-01-11 16:56 PST, Build Bot
no flags
Archive of layout-test-results from ews112 for mac-yosemite (861.02 KB, application/zip)
2016-01-11 16:59 PST, Build Bot
no flags
Archive of layout-test-results from ews106 for mac-yosemite-wk2 (893.84 KB, application/zip)
2016-01-11 17:00 PST, Build Bot
no flags
Patch (29.65 KB, patch)
2016-01-12 15:38 PST, Jer Noble
no flags
Patch (30.02 KB, patch)
2016-01-12 15:42 PST, Jer Noble
no flags
Patch (43.13 KB, patch)
2016-01-12 15:54 PST, Jer Noble
no flags
Patch (46.72 KB, patch)
2016-01-13 11:22 PST, Jer Noble
achristensen: review+
buildbot: commit-queue-
Archive of layout-test-results from ews101 for mac-yosemite (736.55 KB, application/zip)
2016-01-13 12:23 PST, Build Bot
no flags
Archive of layout-test-results from ews104 for mac-yosemite-wk2 (755.14 KB, application/zip)
2016-01-13 12:26 PST, Build Bot
no flags
Archive of layout-test-results from ews112 for mac-yosemite (808.87 KB, application/zip)
2016-01-13 12:32 PST, Build Bot
no flags
Patch for landing (47.02 KB, patch)
2016-01-13 13:19 PST, Jer Noble
no flags
Patch for landing (47.05 KB, patch)
2016-01-14 12:21 PST, Jer Noble
no flags
Patch for landing (47.06 KB, patch)
2016-01-14 14:28 PST, Jer Noble
no flags
Patch for landing (33.63 KB, patch)
2016-01-14 16:40 PST, Jer Noble
no flags
Patch for landing (47.39 KB, patch)
2016-01-14 16:41 PST, Jer Noble
no flags
Patch for landing (47.39 KB, patch)
2016-01-19 12:38 PST, Jer Noble
no flags
Patch for landing (47.42 KB, patch)
2016-01-19 13:22 PST, Jer Noble
no flags
Patch for landing (33.62 KB, patch)
2016-01-27 14:35 PST, Jer Noble
no flags
Patch for landing (47.37 KB, patch)
2016-01-27 14:37 PST, Jer Noble
buildbot: commit-queue-
Archive of layout-test-results from ews113 for mac-yosemite (782.42 KB, application/zip)
2016-01-27 15:47 PST, Build Bot
no flags
Jer Noble
Comment 1 2016-01-11 10:25:42 PST
Jer Noble
Comment 2 2016-01-11 15:55:51 PST
Alex Christensen
Comment 3 2016-01-11 16:32:05 PST
Comment on attachment 268721 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268721&action=review > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:68 > URL requestURL = [[m_avRequest.get() request] URL]; You should store the result of [m_avRequest.get() request] to avoid having to call it twice. > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:162 > + ParsedContentRange contentRange = resource->response().contentRange(); resource should be a reference, or it should be null-checked before using it for the first time, which in this case was before setting data. > Source/WebCore/platform/network/BlobResourceHandle.h:104 > long long m_rangeSuffixLength; > + long long m_totalSize { 0 }; I think you should add the { 0 } for all of these, and remove the initializers in the constructors. As it is now, it just looks inconsistent. > Source/WebCore/platform/network/ResourceResponseBase.cpp:456 > + m_haveParsedContentRangeHeader = true; I think this needs to be set back to false in ResourceResponseBase::updateHeaderParsedState. > Source/WebCore/xml/XMLHttpRequest.cpp:433 > + case HTTPHeaderName::ContentRange: Why is this being added as a forbidden request header? https://fetch.spec.whatwg.org/#forbidden-header-name says it's ok, and it will probably be ignored by the server, right?
Build Bot
Comment 4 2016-01-11 16:56:20 PST
Comment on attachment 268721 [details] Patch Attachment 268721 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/680339 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 5 2016-01-11 16:56:22 PST
Created attachment 268727 [details] Archive of layout-test-results from ews103 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews103 Port: mac-yosemite Platform: Mac OS X 10.10.5
Build Bot
Comment 6 2016-01-11 16:59:16 PST
Comment on attachment 268721 [details] Patch Attachment 268721 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/680331 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 7 2016-01-11 16:59:18 PST
Created attachment 268728 [details] Archive of layout-test-results from ews112 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews112 Port: mac-yosemite Platform: Mac OS X 10.10.5
Build Bot
Comment 8 2016-01-11 17:00:41 PST
Comment on attachment 268721 [details] Patch Attachment 268721 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/680349 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 9 2016-01-11 17:00:43 PST
Created attachment 268729 [details] Archive of layout-test-results from ews106 for mac-yosemite-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews106 Port: mac-yosemite-wk2 Platform: Mac OS X 10.10.5
Alex Christensen
Comment 10 2016-01-11 18:13:44 PST
Comment on attachment 268721 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268721&action=review Make the current tests pass. > Source/WebCore/platform/network/ParsedContentRange.cpp:92 > + instanceLength = headerValue.substring(instanceLengthSeparatorToken + 1).toUInt64(); What if there is no valid number or '*' here? I think toUInt64 would return 0, but it should fail if you have something like "bytes asdf-1/2" or "bytes 0-asdf/*" or "bytes 0-1/asdf". What should happen if there's whitespace, like "byte 0 - 1 / *"? I think right now we would succeed parsing most of that, but the * would not succeed if there is whitespace before it. Please add tests with strange and invalid headers. > Source/WebCore/platform/network/ParsedContentRange.cpp:94 > + return areContentRangeValuesValid(firstBytePosition, lastBytePosition, instanceLength); I think this would be nicer if it only set any of the output variables if it succeeded. Then it would be impossible to get a firstBytePosition without a lastBytePosition, for example.
Jer Noble
Comment 11 2016-01-12 11:18:19 PST
Comment on attachment 268721 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268721&action=review >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:68 >> URL requestURL = [[m_avRequest.get() request] URL]; > > You should store the result of [m_avRequest.get() request] to avoid having to call it twice. Not only that, but since requestURL is only used in a LOG() statement, this will throw a "unused variable" warning in release builds. Will fix. >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:162 >> + ParsedContentRange contentRange = resource->response().contentRange(); > > resource should be a reference, or it should be null-checked before using it for the first time, which in this case was before setting data. I think an ASSERT is fine here, as getting a null value here would indicate that the caller of CashedResourceClient was behaving badly. >> Source/WebCore/platform/network/BlobResourceHandle.h:104 >> + long long m_totalSize { 0 }; > > I think you should add the { 0 } for all of these, and remove the initializers in the constructors. As it is now, it just looks inconsistent. Sure. >> Source/WebCore/platform/network/ParsedContentRange.cpp:92 >> + instanceLength = headerValue.substring(instanceLengthSeparatorToken + 1).toUInt64(); > > What if there is no valid number or '*' here? I think toUInt64 would return 0, but it should fail if you have something like "bytes asdf-1/2" or "bytes 0-asdf/*" or "bytes 0-1/asdf". What should happen if there's whitespace, like "byte 0 - 1 / *"? I think right now we would succeed parsing most of that, but the * would not succeed if there is whitespace before it. > > Please add tests with strange and invalid headers. With this patch, both "0-asdf/*" and "asdf-1/*" will fail, but "0-1/asdf" will not. I'll fix it so that the latter one will fail. I'll make the "*" check whitespace agnostic as well. >> Source/WebCore/platform/network/ParsedContentRange.cpp:94 >> + return areContentRangeValuesValid(firstBytePosition, lastBytePosition, instanceLength); > > I think this would be nicer if it only set any of the output variables if it succeeded. Then it would be impossible to get a firstBytePosition without a lastBytePosition, for example. I considered this, but thought that, for diagnostic purposes, it might be nice to see the values which caused the range to become invalid. >> Source/WebCore/platform/network/ResourceResponseBase.cpp:456 >> + m_haveParsedContentRangeHeader = true; > > I think this needs to be set back to false in ResourceResponseBase::updateHeaderParsedState. Ok. >> Source/WebCore/xml/XMLHttpRequest.cpp:433 >> + case HTTPHeaderName::ContentRange: > > Why is this being added as a forbidden request header? https://fetch.spec.whatwg.org/#forbidden-header-name says it's ok, and it will probably be ignored by the server, right? It will be ignored by the server. I'm guessing that, since Content-Length is a forbidden header, the lack of Content-Range in the spec is an oversight. But until they update the spec to add it, i'll remove it here.
Jer Noble
Comment 12 2016-01-12 15:38:39 PST
Jer Noble
Comment 13 2016-01-12 15:42:22 PST
Created attachment 268822 [details] Patch Forgot the change to clear m_haveParsedContentRangeHeader in updateHeaderParsedState().
Jer Noble
Comment 14 2016-01-12 15:54:33 PST
Alex Christensen
Comment 15 2016-01-12 16:12:05 PST
Comment on attachment 268824 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268824&action=review > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:79 > + LOG_ERROR("Failed to start load for media at url %s", URL([nsRequest URL]).string().ascii().data()); Since we have an NSURL here, may as well do [[nsurl absoluteString] UTF8String] > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:118 > + ParsedContentRange contentRange = resource->response().contentRange(); There's no need to copy the whole data structure. Use ParsedContentRange& Also, I don't think resource is null-checked. > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:152 > void WebCoreAVFResourceLoader::fulfillRequestWithResource(CachedResource* resource) > { > + ASSERT(resource); Maybe for a followup patch, but if resource is always non-null it should be a CachedResource&. Same with WebCoreAVFResourceLoader::responseReceived > Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:163 > + ParsedContentRange contentRange = resource->response().contentRange(); As the patch is right now, I think responseOffset should be left as 0 if resource is null. > Source/WebCore/platform/network/BlobResourceHandle.h:105 > + long long m_totalSize { 0 }; > long long m_totalRemainingSize; You didn't make all of these { 0 }; :( > Source/WebCore/platform/network/ResourceResponseBase.h:43 > +class ParsedContentRange; You shouldn't need to forward declare this if you include ParsedContentRange.h. > Tools/ChangeLog:18 > +2016-01-11 Jer Noble <jer.noble@apple.com> Too many changelog entries. > Tools/TestWebKitAPI/Tests/WebCore/ParsedContentRange.cpp:35 > +TEST(WebCore, ParsedContentRangeFromString) Add a test if -0 is the start of the range. > Tools/TestWebKitAPI/Tests/WebCore/ParsedContentRange.cpp:39 > + ASSERT_TRUE(ParsedContentRange("bytes 0 - 1 / * ").isValid()); This is probably looser than the RFC standard.
Jer Noble
Comment 16 2016-01-13 11:14:25 PST
Comment on attachment 268824 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268824&action=review >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:79 >> + LOG_ERROR("Failed to start load for media at url %s", URL([nsRequest URL]).string().ascii().data()); > > Since we have an NSURL here, may as well do [[nsurl absoluteString] UTF8String] Ok. >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:118 >> + ParsedContentRange contentRange = resource->response().contentRange(); > > There's no need to copy the whole data structure. Use ParsedContentRange& > Also, I don't think resource is null-checked. Ok. Will add an ASSERT(resource) (see below). >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:152 >> + ASSERT(resource); > > Maybe for a followup patch, but if resource is always non-null it should be a CachedResource&. Same with WebCoreAVFResourceLoader::responseReceived Unfortunately, this would require re-writing every client of CachedResourceClient. (Which maybe we should do, but would absolutely need to be in a follow-up patch). >> Source/WebCore/platform/graphics/avfoundation/objc/WebCoreAVFResourceLoader.mm:163 >> + ParsedContentRange contentRange = resource->response().contentRange(); > > As the patch is right now, I think responseOffset should be left as 0 if resource is null. We asserted above that resource will never be null. >> Source/WebCore/platform/network/BlobResourceHandle.h:105 >> long long m_totalRemainingSize; > > You didn't make all of these { 0 }; > :( Whoops! >> Source/WebCore/platform/network/ResourceResponseBase.h:43 >> +class ParsedContentRange; > > You shouldn't need to forward declare this if you include ParsedContentRange.h. Whoops! >> Tools/ChangeLog:18 >> +2016-01-11 Jer Noble <jer.noble@apple.com> > > Too many changelog entries. Whoops! >> Tools/TestWebKitAPI/Tests/WebCore/ParsedContentRange.cpp:35 >> +TEST(WebCore, ParsedContentRangeFromString) > > Add a test if -0 is the start of the range. Added. Fails correctly. >> Tools/TestWebKitAPI/Tests/WebCore/ParsedContentRange.cpp:39 >> + ASSERT_TRUE(ParsedContentRange("bytes 0 - 1 / * ").isValid()); > > This is probably looser than the RFC standard. Will tighten.
Jer Noble
Comment 17 2016-01-13 11:22:26 PST
Alex Christensen
Comment 18 2016-01-13 11:53:49 PST
Comment on attachment 268883 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=268883&action=review r=me > Tools/TestWebKitAPI/Tests/WebCore/ParsedContentRange.cpp:79 > + ASSERT_FALSE(ParsedContentRange(0, 2, 1).isValid()); Add some negative values here and (0, 0, 0).
Build Bot
Comment 19 2016-01-13 12:23:42 PST
Comment on attachment 268883 [details] Patch Attachment 268883 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/687383 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 20 2016-01-13 12:23:45 PST
Created attachment 268888 [details] Archive of layout-test-results from ews101 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews101 Port: mac-yosemite Platform: Mac OS X 10.10.5
Build Bot
Comment 21 2016-01-13 12:25:57 PST
Comment on attachment 268883 [details] Patch Attachment 268883 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/687384 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 22 2016-01-13 12:26:00 PST
Created attachment 268889 [details] Archive of layout-test-results from ews104 for mac-yosemite-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews104 Port: mac-yosemite-wk2 Platform: Mac OS X 10.10.5
Build Bot
Comment 23 2016-01-13 12:32:54 PST
Comment on attachment 268883 [details] Patch Attachment 268883 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/687391 New failing tests: http/tests/xmlhttprequest/blob-request-byte-range.html
Build Bot
Comment 24 2016-01-13 12:32:56 PST
Created attachment 268892 [details] Archive of layout-test-results from ews112 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews112 Port: mac-yosemite Platform: Mac OS X 10.10.5
Jer Noble
Comment 25 2016-01-13 13:06:28 PST
(In reply to comment #24) > Created attachment 268892 [details] > Archive of layout-test-results from ews112 for mac-yosemite > > The attached test failures were seen while running run-webkit-tests on the > mac-debug-ews. > Bot: ews112 Port: mac-yosemite Platform: Mac OS X 10.10.5 Sigh. The expected.txt is missing an extra line-feed. Will fix.
Jer Noble
Comment 26 2016-01-13 13:19:08 PST
Created attachment 268898 [details] Patch for landing
Jer Noble
Comment 27 2016-01-14 12:21:59 PST
Created attachment 268986 [details] Patch for landing
Jer Noble
Comment 28 2016-01-14 14:28:49 PST
Created attachment 269009 [details] Patch for landing
Jer Noble
Comment 29 2016-01-14 16:40:13 PST
Created attachment 269019 [details] Patch for landing
Jer Noble
Comment 30 2016-01-14 16:41:46 PST
Created attachment 269020 [details] Patch for landing
Jer Noble
Comment 31 2016-01-19 12:38:59 PST
Created attachment 269281 [details] Patch for landing
Jer Noble
Comment 32 2016-01-19 13:22:21 PST
Created attachment 269285 [details] Patch for landing
Jer Noble
Comment 33 2016-01-27 14:35:59 PST
Created attachment 270042 [details] Patch for landing
Jer Noble
Comment 34 2016-01-27 14:37:12 PST
Created attachment 270043 [details] Patch for landing
Jer Noble
Comment 35 2016-01-27 14:37:57 PST
I think the Windows EWS bot was broken at the time I added the last Patch for landing; adding another one to see if it passes this time.
Build Bot
Comment 36 2016-01-27 15:47:28 PST
Comment on attachment 270043 [details] Patch for landing Attachment 270043 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/747293 New failing tests: imported/w3c/web-platform-tests/streams-api/readable-streams/garbage-collection-1.html
Build Bot
Comment 37 2016-01-27 15:47:32 PST
Created attachment 270056 [details] Archive of layout-test-results from ews113 for mac-yosemite The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews113 Port: mac-yosemite Platform: Mac OS X 10.10.5
Jer Noble
Comment 38 2016-01-28 11:17:29 PST
Csaba Osztrogonác
Comment 39 2016-01-28 14:06:40 PST
(In reply to comment #38) > Committed r195764: <http://trac.webkit.org/changeset/195764> It broke the Windows build.
Jer Noble
Comment 40 2016-01-28 14:29:08 PST
Note You need to log in before you can comment on or make changes to this bug.