Bug 234105 – REGRESSION (r286259): [ BigSur+ wk1 ] 2X imported/w3c/web-platform-tests/speech-api/SpeechSynthesis (layout-tests) are constant text failures

• Status: RESOLVED
• Resolution: FIXED
• Component: Accessibility

[Robert Jenner]

The following two tests are constant text failures on Big Sur and later in WK 1:

imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-speak-events.html
imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-pause-resume.tentative.html

HISTORY:
https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fspeech-api%2FSpeechSynthesis-pause-resume.tentative.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fspeech-api%2FSpeechSynthesis-speak-events.html


DIF URLS:
https://build.webkit.org/results/Apple-Monterey-Release-WK1-Tests/r286796%20(580)/imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-pause-resume.tentative-pretty-diff.html
https://build.webkit.org/results/Apple-Monterey-Release-WK1-Tests/r286796%20(580)/imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-speak-events-pretty-diff.html

TEXT DIFFS:
FOR "imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-pause-resume.tentative.html":
 
-PASS speechSynthesis.pause() and resume() state and events
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT speechSynthesis.pause() and resume() state and events Test timed out
+



FOR "imported/w3c/web-platform-tests/speech-api/SpeechSynthesis-speak-events.html":
-FAIL speechSynthesis.speak() fires start and end events with empty utterance assert_true: Not expecting event, but got end event expected true got false
+FAIL speechSynthesis.speak() fires start and end events with empty utterance assert_true: Not expecting event, but got start event expected true got false
 FAIL speechSynthesis.speak() fires start and end events assert_true: Not expecting event, but got start event expected true got false

[Radar WebKit Bug Importer]

<rdar://problem/86292065>

[Robert Jenner]

I was able to reproduce the failure at Monterey Production ToT using the following test:

run-webkit-tests <test name goes here> -1 



I was able to also discover the regression point. The history shows that these tests weren't running on wk1 prior to r286259. When I ran the same test at r286258, I found that the tests were passing. They started failing after the introduction of r286259:

https://trac.webkit.org/changeset/286259/webkit

[chris fleizach]

(In reply to Robert Jenner from comment #2)
> I was able to reproduce the failure at Monterey Production ToT using the
> following test:
> 
> run-webkit-tests <test name goes here> -1 
> 
> 
> 
> I was able to also discover the regression point. The history shows that
> these tests weren't running on wk1 prior to r286259. When I ran the same
> test at r286258, I found that the tests were passing. They started failing
> after the introduction of r286259:
> 
> https://trac.webkit.org/changeset/286259/webkit

I re-enabled them with this change, but I'm testing on 12.0. I think we should skip on Big Sur and just run on Monterey if this is the case

[Robert Jenner]

(In reply to chris fleizach from comment #3)
> (In reply to Robert Jenner from comment #2)
> > I was able to reproduce the failure at Monterey Production ToT using the
> > following test:
> > 
> > run-webkit-tests <test name goes here> -1 
> > 
> > 
> > 
> > I was able to also discover the regression point. The history shows that
> > these tests weren't running on wk1 prior to r286259. When I ran the same
> > test at r286258, I found that the tests were passing. They started failing
> > after the introduction of r286259:
> > 
> > https://trac.webkit.org/changeset/286259/webkit
> 
> I re-enabled them with this change, but I'm testing on 12.0. I think we
> should skip on Big Sur and just run on Monterey if this is the case



Okay, that sounds good for BigSur. I will get them skipped for wk1 there. 

What should be done for it failing on wk1 Monterey?

[chris fleizach]

(In reply to Robert Jenner from comment #4)
> (In reply to chris fleizach from comment #3)
> > (In reply to Robert Jenner from comment #2)
> > > I was able to reproduce the failure at Monterey Production ToT using the
> > > following test:
> > > 
> > > run-webkit-tests <test name goes here> -1 
> > > 
> > > 
> > > 
> > > I was able to also discover the regression point. The history shows that
> > > these tests weren't running on wk1 prior to r286259. When I ran the same
> > > test at r286258, I found that the tests were passing. They started failing
> > > after the introduction of r286259:
> > > 
> > > https://trac.webkit.org/changeset/286259/webkit
> > 
> > I re-enabled them with this change, but I'm testing on 12.0. I think we
> > should skip on Big Sur and just run on Monterey if this is the case
> 
> 
> 
> Okay, that sounds good for BigSur. I will get them skipped for wk1 there. 
> 
> What should be done for it failing on wk1 Monterey?

Can you skip all WK1. I will look into the Monterey case
https://bugs.webkit.org/show_bug.cgi?id=234125

I feel like that should pass without issue

[Robert Jenner]

(In reply to chris fleizach from comment #5)
> (In reply to Robert Jenner from comment #4)
> > (In reply to chris fleizach from comment #3)
> > > (In reply to Robert Jenner from comment #2)
> > > > I was able to reproduce the failure at Monterey Production ToT using the
> > > > following test:
> > > > 
> > > > run-webkit-tests <test name goes here> -1 
> > > > 
> > > > 
> > > > 
> > > > I was able to also discover the regression point. The history shows that
> > > > these tests weren't running on wk1 prior to r286259. When I ran the same
> > > > test at r286258, I found that the tests were passing. They started failing
> > > > after the introduction of r286259:
> > > > 
> > > > https://trac.webkit.org/changeset/286259/webkit
> > > 
> > > I re-enabled them with this change, but I'm testing on 12.0. I think we
> > > should skip on Big Sur and just run on Monterey if this is the case
> > 
> > 
> > 
> > Okay, that sounds good for BigSur. I will get them skipped for wk1 there. 
> > 
> > What should be done for it failing on wk1 Monterey?
> 
> Can you skip all WK1. I will look into the Monterey case
> https://bugs.webkit.org/show_bug.cgi?id=234125
> 
> I feel like that should pass without issue


Disabled for all Mac wk1 here:

https://trac.webkit.org/changeset/286980/webkit

[chris fleizach]

(In reply to Robert Jenner from comment #6)
> (In reply to chris fleizach from comment #5)
> > (In reply to Robert Jenner from comment #4)
> > > (In reply to chris fleizach from comment #3)
> > > > (In reply to Robert Jenner from comment #2)
> > > > > I was able to reproduce the failure at Monterey Production ToT using the
> > > > > following test:
> > > > > 
> > > > > run-webkit-tests <test name goes here> -1 
> > > > > 
> > > > > 
> > > > > 
> > > > > I was able to also discover the regression point. The history shows that
> > > > > these tests weren't running on wk1 prior to r286259. When I ran the same
> > > > > test at r286258, I found that the tests were passing. They started failing
> > > > > after the introduction of r286259:
> > > > > 
> > > > > https://trac.webkit.org/changeset/286259/webkit
> > > > 
> > > > I re-enabled them with this change, but I'm testing on 12.0. I think we
> > > > should skip on Big Sur and just run on Monterey if this is the case
> > > 
> > > 
> > > 
> > > Okay, that sounds good for BigSur. I will get them skipped for wk1 there. 
> > > 
> > > What should be done for it failing on wk1 Monterey?
> > 
> > Can you skip all WK1. I will look into the Monterey case
> > https://bugs.webkit.org/show_bug.cgi?id=234125
> > 
> > I feel like that should pass without issue
> 
> 
> Disabled for all Mac wk1 here:
> 
> https://trac.webkit.org/changeset/286980/webkit
Thanks for your help