Bug 194350 – [ Mac WK2 ] Layout Test http/tests/security/contentSecurityPolicy/block-all-mixed-content/insecure-image-in-iframe-with-enforced-and-report-policies.html is a flaky failure

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Shawn Roberts]

The following layout test is flaky on Mac WK@

http/tests/security/contentSecurityPolicy/block-all-mixed-content/insecure-image-in-iframe-with-enforced-and-report-policies.html

Probable cause:

Appears to have started being flaky at 231450

Prior builds fail 100% of the time

Reproducible with:

run-webkit-tests t231450 http/tests/security/contentSecurityPolicy/block-all-mixed-content/insecure-image-in-iframe-with-enforced-and-report-policies.html --iterations 500 -f

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2Fsecurity%2FcontentSecurityPolicy%2Fblock-all-mixed-content%2Finsecure-image-in-iframe-with-enforced-and-report-policies.html

Diff:

--- /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/http/tests/security/contentSecurityPolicy/block-all-mixed-content/insecure-image-in-iframe-with-enforced-and-report-policies-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/http/tests/security/contentSecurityPolicy/block-all-mixed-content/insecure-image-in-iframe-with-enforced-and-report-policies-actual.txt
@@ -2,6 +2,8 @@
 frame "<!--frame1-->" - didStartProvisionalLoadForFrame
 CONSOLE MESSAGE: The Content Security Policy 'block-all-mixed-content' was delivered in report-only mode, but does not specify a 'report-uri'; the policy will have no effect. Please either add a 'report-uri' directive, or deliver the policy via the 'Content-Security-Policy' header.
 frame "<!--frame1-->" - didCommitLoadForFrame
+CONSOLE MESSAGE: Blocked mixed content http://127.0.0.1:8000/security/resources/compass.jpg because 'block-all-mixed-content' appears in the Content Security Policy.
+CONSOLE MESSAGE: [Report Only] Blocked mixed content http://127.0.0.1:8000/security/resources/compass.jpg because 'block-all-mixed-content' appears in the Content Security Policy.
 CONSOLE MESSAGE: Blocked mixed content http://127.0.0.1:8000/security/resources/compass.jpg because 'block-all-mixed-content' appears in the Content Security Policy.
 CONSOLE MESSAGE: [Report Only] Blocked mixed content http://127.0.0.1:8000/security/resources/compass.jpg because 'block-all-mixed-content' appears in the Content Security Policy.
 frame "<!--frame1-->" - didFinishDocumentLoadForFrame

[Radar WebKit Bug Importer]

<rdar://problem/47937466>

[Shawn Roberts]

Created attachment 362448 [details]
Update TestExpectations

[WebKit Commit Bot]

Comment on attachment 362448 [details]
Update TestExpectations

Rejecting attachment 362448 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-01', 'apply-attachment', '--no-update', '--non-interactive', 362448, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Logging in as [e-mail]...
Fetching: https://bugs.webkit.org/attachment.cgi?id=362448&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=194350&ctype=xml&excludefield=attachmentdata
Processing 1 patch from 1 bug.
Processing patch 362448 from bug 194350.
Fetching: https://bugs.webkit.org/attachment.cgi?id=362448
Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Parsed 2 diffs from patch file(s).
patching file LayoutTests/ChangeLog
Hunk #1 succeeded at 1 with fuzz 3.
patching file LayoutTests/platform/mac-wk2/TestExpectations
Hunk #1 FAILED at 911.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/mac-wk2/TestExpectations.rej

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: https://webkit-queues.webkit.org/results/11211741

[Shawn Roberts]

Created attachment 362540 [details]
Update TestExpectations

[Truitt Savell]

Comment on attachment 362540 [details]
Update TestExpectations

Clearing flags on attachment: 362540

Committed r241836: <https://trac.webkit.org/changeset/241836>

[Shawn Roberts]

Additional repro steps: 

I searched through Trac and the last change of the file I could find was 209577, so I started my bisecting from the oldest build I could spade r228049

241026 --iterations 500 -f --exit-after-n-failures=5 : 174 tests ran as expected, 5 didn't (321 didn't run) failed unexpectedly (text diff)
239655 Most tests between 231450 - 241026 had 5 failures within 200 iterations
239644 
233844 
232404 
231684 
231501 
231453 
231451 
231450 178 tests ran as expected, 5 didn't (317 didn't run)

231449 --exit-after-n-faiulres=25 . All tests below failed 100% of the time
231445 
231437 
231404   
231324 
230947 
228049

[Chris Dumez]

Double load of compass.jpg, I suspect preloading.

[Chris Dumez]

(In reply to Chris Dumez from comment #7)
> Double load of compass.jpg, I suspect preloading.

As I suspected, if I disable the HTMLPreloadScanner, then the test is no longer flaky.
I do not believe this is a regression from r231450, likely just a timing change.

[Chris Dumez]

Created attachment 365625 [details]
Patch

[Chris Dumez]

Created attachment 365626 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 365626 [details]
Patch

Clearing flags on attachment: 365626

Committed r243353: <https://trac.webkit.org/changeset/243353>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.