Bug 131069 - Enable fullscreen captions selection
Summary: Enable fullscreen captions selection
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKit2 (show other bugs)
Version: 528+ (Nightly build)
Hardware: iPhone / iPad Unspecified
: P2 Normal
Assignee: Jeremy Jones
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-04-01 15:07 PDT by Jeremy Jones
Modified: 2014-05-02 10:31 PDT (History)
9 users (show)

See Also:


Attachments
Patch (23.93 KB, patch)
2014-04-01 15:19 PDT, Jeremy Jones
no flags Details | Formatted Diff | Diff
Patch (23.64 KB, patch)
2014-04-04 14:50 PDT, Jeremy Jones
thorton: review+
Details | Formatted Diff | Diff
Patch (23.62 KB, patch)
2014-04-08 22:40 PDT, Jeremy Jones
eric.carlson: review+
commit-queue: commit-queue-
Details | Formatted Diff | Diff
Patch (23.62 KB, patch)
2014-04-09 14:50 PDT, Jeremy Jones
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Jones 2014-04-01 15:07:05 PDT
Enable captions button in fullscreen.
Comment 1 Jeremy Jones 2014-04-01 15:19:47 PDT
Created attachment 228327 [details]
Patch
Comment 2 Eric Carlson 2014-04-02 12:19:03 PDT
Comment on attachment 228327 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=228327&action=review

> Source/WebCore/platform/ios/WebVideoFullscreenInterface.h:52
> +    

Nit: extra blank line.

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:290
> +    

Can't you return here if option is nil?

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:299
> +        self.delegate->selectAudioMediaOption(index != NSNotFound ? index : UINT64_MAX);

index can't be NSNotFound.

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:323
> +    [_currentLegibleMediaSelectionOption release];
> +    _currentLegibleMediaSelectionOption = [option retain];
> +    
> +    ASSERT(self.delegate);
> +    
> +    NSInteger index = NSNotFound;
> +    
> +    if (option && self.legibleMediaSelectionOptions)
> +        index = [self.legibleMediaSelectionOptions indexOfObject:option];
> +    
> +    if (index != NSNotFound)
> +        self.delegate->selectLegibleMediaOption(index != NSNotFound ? index : UINT64_MAX);

Ditto the comments about setCurrentAudioMediaSelectionOption.

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:488
> +    NSMutableArray* webOptions = [NSMutableArray array];

Nit: you can optimize this slightly by allocating the number of elements you will need.

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:509
> +    NSMutableArray* webOptions = [NSMutableArray array];
> +    for (auto& name : options) {
> +        RetainPtr<WebAVMediaSelectionOption> webOption = adoptNS([[WebAVMediaSelectionOption alloc] init]);
> +        RetainPtr<NSString> nameString = adoptNS([[NSString alloc] initWithUTF8String:name.utf8().data()]);
> +        [webOption setLocalizedDisplayName:nameString.get()];
> +        [webOptions addObject:webOption.get()];
> +    }

Instead of duplicating this code here and in setAudioMediaSelectionOptions, why don't you put it into a shared function?

> Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.h:70
> +    

Nit: extra blank line.

> Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:89
> +    

Nit: extra spaces (hint: enable trimming "whitespace-only lines" in Xcode editing prefs).

> Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:208
> +void WebVideoFullscreenModelMediaElement::selectAudioMediaOption(uint64_t)
> +{
> +}

This needs a FIXME.

> Source/WebKit2/UIProcess/ios/WebVideoFullscreenManagerProxy.h:71
> +    virtual void selectAudioMediaOption(uint64_t index) override;
> +    virtual void selectLegibleMediaOption(uint64_t index) override;

Nit: "index" doesn't add anything.
Comment 3 Jeremy Jones 2014-04-04 12:12:10 PDT
(In reply to comment #2)
> (From update of attachment 228327 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=228327&action=review
> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterface.h:52
> > +    
> 
> Nit: extra blank line.

Done.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:290
> > +    
> 
> Can't you return here if option is nil?

No, because we still need to inform the delegate of the change.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:299
> > +        self.delegate->selectAudioMediaOption(index != NSNotFound ? index : UINT64_MAX);
> 
> index can't be NSNotFound.

Fixed.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:323
> > +    [_currentLegibleMediaSelectionOption release];
> > +    _currentLegibleMediaSelectionOption = [option retain];
> > +    
> > +    ASSERT(self.delegate);
> > +    
> > +    NSInteger index = NSNotFound;
> > +    
> > +    if (option && self.legibleMediaSelectionOptions)
> > +        index = [self.legibleMediaSelectionOptions indexOfObject:option];
> > +    
> > +    if (index != NSNotFound)
> > +        self.delegate->selectLegibleMediaOption(index != NSNotFound ? index : UINT64_MAX);
> 
> Ditto the comments about setCurrentAudioMediaSelectionOption.

Ditto.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:488
> > +    NSMutableArray* webOptions = [NSMutableArray array];
> 
> Nit: you can optimize this slightly by allocating the number of elements you will need.

Done.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:509
> > +    NSMutableArray* webOptions = [NSMutableArray array];
> > +    for (auto& name : options) {
> > +        RetainPtr<WebAVMediaSelectionOption> webOption = adoptNS([[WebAVMediaSelectionOption alloc] init]);
> > +        RetainPtr<NSString> nameString = adoptNS([[NSString alloc] initWithUTF8String:name.utf8().data()]);
> > +        [webOption setLocalizedDisplayName:nameString.get()];
> > +        [webOptions addObject:webOption.get()];
> > +    }
> 
> Instead of duplicating this code here and in setAudioMediaSelectionOptions, why don't you put it into a shared function?

Done.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.h:70
> > +    
> 
> Nit: extra blank line.

Deleted.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:89
> > +    
> 
> Nit: extra spaces (hint: enable trimming "whitespace-only lines" in Xcode editing prefs).

Done and done.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:208
> > +void WebVideoFullscreenModelMediaElement::selectAudioMediaOption(uint64_t)
> > +{
> > +}
> 
> This needs a FIXME.

Added // FIXME: 131236 Implement audio track selection

> 
> > Source/WebKit2/UIProcess/ios/WebVideoFullscreenManagerProxy.h:71
> > +    virtual void selectAudioMediaOption(uint64_t index) override;
> > +    virtual void selectLegibleMediaOption(uint64_t index) override;
> 
> Nit: "index" doesn't add anything.

Deleted.
Comment 4 Jeremy Jones 2014-04-04 14:50:45 PDT
Created attachment 228618 [details]
Patch
Comment 5 Eric Carlson 2014-04-04 15:41:31 PDT
Comment on attachment 228618 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=228618&action=review

The WebCore changes look OK to me modulo my comments, which can be addressed in a follow up. I am not a WK2 reviewer so I won't mark this r+.

> Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:93
> +        legibleOptions.append(textTrack->label());

This is wrong, not all tracks have a label but every menu item needs to have a label. We calculate the display name in CaptionPreferences::displayNameForTrack. Probably the right thing to do here is to add a displayName method to PlatformTextTrack and have TextTrack::platformTextTrack call CaptionPreferences::displayNameForTrack.

I think this can be done in a follow-up bug.

> Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:216
> +        platformTrack = m_mediaElement->platformTextTracks()[static_cast<size_t>(index)];

I should have noticed this before, but why pass a uint64_t when we need a size_t?
Comment 6 Jeremy Jones 2014-04-07 12:37:54 PDT
(In reply to comment #5)
> (From update of attachment 228618 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=228618&action=review
> 
> The WebCore changes look OK to me modulo my comments, which can be addressed in a follow up. I am not a WK2 reviewer so I won't mark this r+.
> 
> > Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:93
> > +        legibleOptions.append(textTrack->label());
> 
> This is wrong, not all tracks have a label but every menu item needs to have a label. We calculate the display name in CaptionPreferences::displayNameForTrack. Probably the right thing to do here is to add a displayName method to PlatformTextTrack and have TextTrack::platformTextTrack call CaptionPreferences::displayNameForTrack.
> 
> I think this can be done in a follow-up bug.

Filed:
https://bugs.webkit.org/show_bug.cgi?id=131311
Use displayNameForTrack instead of textTrack->label() for captions.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenModelMediaElement.mm:216
> > +        platformTrack = m_mediaElement->platformTextTracks()[static_cast<size_t>(index)];
> 
> I should have noticed this before, but why pass a uint64_t when we need a size_t?

I tried size_t first. It is unprecedented and has no encoder/decoder. My hypothesis is that it is not used because web process and UI process aren't required to be the same architecture. In theory, there could be a 32-bit UI process talking to a 64-bit web process. They would disagree on how this value is serialized. So it makes sense to require more explicit data sizes. Just a hypothesis.
Comment 7 Tim Horton 2014-04-07 13:25:02 PDT
Comment on attachment 228618 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=228618&action=review

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:106
> +@property (retain) NSArray *audioMediaSelectionOptions; // readonly

what’s up with the // readonly?

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:250
> +    return ([self hasAudioMediaSelectionOptions] || [self hasLegibleMediaSelectionOptions]);

no need for parens

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:260
> +    return ([[self audioMediaSelectionOptions] count] > 0);

nor here nor in a few other places.

> Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:484
> +static NSMutableArray* mediaSelectionOptions(const Vector<String>& options)

all of the ObjC stars are on the wrong side!
Comment 8 Jeremy Jones 2014-04-08 22:40:08 PDT
Created attachment 228937 [details]
Patch
Comment 9 Jeremy Jones 2014-04-08 22:40:42 PDT
(In reply to comment #7)
> (From update of attachment 228618 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=228618&action=review
> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:106
> > +@property (retain) NSArray *audioMediaSelectionOptions; // readonly
> 
> what’s up with the // readonly?

Deleted

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:250
> > +    return ([self hasAudioMediaSelectionOptions] || [self hasLegibleMediaSelectionOptions]);
> 
> no need for parens

Deleted.

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:260
> > +    return ([[self audioMediaSelectionOptions] count] > 0);
> 
> nor here nor in a few other places.

Deleted. 

> 
> > Source/WebCore/platform/ios/WebVideoFullscreenInterfaceAVKit.mm:484
> > +static NSMutableArray* mediaSelectionOptions(const Vector<String>& options)
> 
> all of the ObjC stars are on the wrong side!

Moved.
Comment 10 WebKit Commit Bot 2014-04-09 10:57:39 PDT
Comment on attachment 228937 [details]
Patch

Rejecting attachment 228937 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.appspot.com', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 228937, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
VideoFullscreenManagerProxy.messages.in
patching file Source/WebKit2/UIProcess/ios/WebVideoFullscreenManagerProxy.mm
patching file Source/WebKit2/WebProcess/ios/WebVideoFullscreenManager.h
patching file Source/WebKit2/WebProcess/ios/WebVideoFullscreenManager.messages.in
patching file Source/WebKit2/WebProcess/ios/WebVideoFullscreenManager.mm

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Eric Carlson']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: http://webkit-queues.appspot.com/results/5561260254953472
Comment 11 Jeremy Jones 2014-04-09 14:50:52 PDT
Created attachment 228986 [details]
Patch
Comment 12 WebKit Commit Bot 2014-04-09 15:33:56 PDT
Comment on attachment 228986 [details]
Patch

Clearing flags on attachment: 228986

Committed r167044: <http://trac.webkit.org/changeset/167044>