Bug 191290 – Fix steps_unittest.py failures related to recent Win10/Win7 changes

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Ryan Haddad]

➜  build.webkit.org-config python steps_unittest.py
  import sets as _sets
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................FFFFF............FF.................F...................................
======================================================================
FAIL: test_builder Apple Win 10 Debug (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 598, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple Win 10 Debug (Build)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile', 'compile-webkit', 'archive-built-product', 'upload', 'transfer-to-s3', 'trigger']

======================================================================
FAIL: test_builder Apple Win 10 Debug (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 598, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple Win 10 Debug (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile', 'download-built-product', 'extract-built-product', 'jscore-test', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'builtins-generator-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

======================================================================
FAIL: test_builder Apple Win 10 Release (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 598, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple Win 10 Release (Build)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile', 'compile-webkit', 'archive-built-product', 'upload', 'transfer-to-s3', 'trigger']

======================================================================
FAIL: test_builder Apple Win 10 Release (Tests) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 598, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple Win 10 Release (Tests)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile', 'download-built-product', 'extract-built-product', 'jscore-test', 'layout-test', 'run-api-tests', 'webkitpy-test', 'webkitperl-test', 'bindings-generation-tests', 'builtins-generator-tests', 'archive-test-results', 'upload', 'MasterShellCommand']

======================================================================
FAIL: test_builder Apple Win 7 Release (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 598, in doTest
    self.assertTrue(builder['name'] in expected_build_steps, "Missing expected result for builder: %s\n Actual result is %s" % (builder['name'], buildSteps))
AssertionError: Missing expected result for builder: Apple Win 7 Release (Build)
 Actual result is ['configure build', 'svn', 'kill old processes', 'delete WebKitBuild directory', 'delete stale build files', 'compile', 'compile-webkit', 'archive-built-product', 'upload', 'transfer-to-s3', 'trigger']

======================================================================
FAIL: test_builder GTK Linux 64-bit Release Debian Stable (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 599, in doTest
    self.assertListEqual(expected_build_steps[builder['name']], buildSteps)
AssertionError: Lists differ: ['configure build', 'svn', 'ki... != ['configure build', 'svn', 'ki...

First differing element 5:
compile-webkit
jhbuild

Second list contains 1 additional elements.
First extra element 6:
compile-webkit

  ['configure build',
   'svn',
   'kill old processes',
   'delete WebKitBuild directory',
   'delete stale build files',
+  'jhbuild',
   'compile-webkit']

======================================================================
FAIL: test_builder GTK Linux 64-bit Release Ubuntu LTS (Build) (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 599, in doTest
    self.assertListEqual(expected_build_steps[builder['name']], buildSteps)
AssertionError: Lists differ: ['configure build', 'svn', 'ki... != ['configure build', 'svn', 'ki...

First differing element 5:
compile-webkit
jhbuild

Second list contains 1 additional elements.
First extra element 6:
compile-webkit

  ['configure build',
   'svn',
   'kill old processes',
   'delete WebKitBuild directory',
   'delete stale build files',
+  'jhbuild',
   'compile-webkit']

======================================================================
FAIL: test_unnecessary_expected_results (__main__.BuildStepsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "steps_unittest.py", line 609, in test_unnecessary_expected_results
    self.assertTrue(builder in builders, "Builder %s doesn't exist, but has unnecessary expected results" % builder)
AssertionError: Builder Apple Win Release (Build) doesn't exist, but has unnecessary expected results

----------------------------------------------------------------------
Ran 902 tests in 0.028s

FAILED (failures=8)

[Ryan Haddad]

Created attachment 353936 [details]
Patch

[Alexey Proskuryakov]

Comment on attachment 353936 [details]
Patch

Why is there a change to GTK steps? Did that change earlier?

[Ryan Haddad]

(In reply to Alexey Proskuryakov from comment #2)
> Comment on attachment 353936 [details]
> Patch
> 
> Why is there a change to GTK steps? Did that change earlier?
It appears to be due to:
[GTK] Disable MSE build support on Debian and Ubuntu LTS bots
​https://bugs.webkit.org/show_bug.cgi?id=190581

[Michael Catanzaro]

The GTK/WPE change is wrong and unexpected. We were just trying to disable a build flag, not introduce a new build step. Are we sure this change was related to r237113? Anyway, I'm impressed that the test caught this problem.

CC Adrian. Adrian, the problem is the Ubuntu LTS bots have unexpectedly started running a JHBuild environment. They're supposed to be using system dependencies only, not JHBuild.

[Ryan Haddad]

(In reply to Michael Catanzaro from comment #4)
> The GTK/WPE change is wrong and unexpected. We were just trying to disable a
> build flag, not introduce a new build step. Are we sure this change was
> related to r237113? Anyway, I'm impressed that the test caught this problem.

I came to this conclusion using git bisect. Checking build history, the step appearance lines up with when I restarted buildbot on October 15th (https://bugs.webkit.org/show_bug.cgi?id=190581#c7).

I'll just land the Win10/Win7 portion of my change.

[Michael Catanzaro]

Reported bug #191318 for the GTK problem.

[Ryan Haddad]

I landed the Win10/Win7 test fixes in https://trac.webkit.org/changeset/237991/webkit

[Radar WebKit Bug Importer]

<rdar://problem/45917298>