Bug 259748
| Summary: | test-webkitpy -i fails; webkitpy.common.config.committers_unittest.CommittersTest integration tests are super fragile | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Sam Sneddon [:gsnedders] <gsnedders> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | jbedard, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | WebKit Nightly Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Sam Sneddon [:gsnedders]
Once again the webkitpy.common.config.committers_unittest.CommittersTest.integration_test_* tests bite us:
```
[344/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:
Traceback (most recent call last):
File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
self._assert_fuzz_match('Ada', 'Ada Chan', 0)
File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
First differing element 0:
['Ada Chan', 'Ada Rose Cannon']
['Ada Chan']
- (['Ada Chan', 'Ada Rose Cannon'], 0)
+ (['Ada Chan'], 0)
[384/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
Traceback (most recent call last):
File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
First differing element 0:
['Bri Harris', 'David Harrison', 'Harrison Weinerman']
['David Harrison']
- (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
+ (['David Harrison'], 4)
```
It also seems we don't run the integration tests in either EWS or post-commit?
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Radar WebKit Bug Importer
<rdar://problem/113292283>
Sam Sneddon [:gsnedders]
test-webkitpy -i webkitpy.common.config.committers_unittest.CommittersTest now gives us three failures:
[9/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:
Traceback (most recent call last):
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
self._assert_fuzz_match('Ada', 'Ada Chan', 0)
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
First differing element 0:
['Ada Chan', 'Ada Rose Cannon']
['Ada Chan']
- (['Ada Chan', 'Ada Rose Cannon'], 0)
+ (['Ada Chan'], 0)
[16/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_eric_seidel failed:
Traceback (most recent call last):
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 241, in integration_test_contributors_eric_seidel
self._assert_fuzz_match('Eric S', 'Eric Seidel', 0)
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
AssertionError: Tuples differ: (['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[74 chars] 2.0) != (['Eric Seidel'], 0)
First differing element 0:
['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[68 chars] Li']
['Eric Seidel']
- (['Eric Carlson',
- 'Eric Hutchison',
- 'Eric Penner',
- 'Eric Roman',
- 'Eric Seidel',
? ^^
+ (['Eric Seidel'], 0)
? ^^ + +++
- 'Eric Staggs',
- 'Eric Uhrhane',
- 'Erica Li'],
- 2.0)
[48/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
Traceback (most recent call last):
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
First differing element 0:
['Bri Harris', 'David Harrison', 'Harrison Weinerman']
['David Harrison']
- (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
+ (['David Harrison'], 4)
Ran 50 tests in 1.246s
FAILED (failures=3, errors=0)
That this has gotten worse shows quite how fragile all of this is, because relying on the live committers.json is inherently bad.
Sam Sneddon [:gsnedders]
Honestly I don't see any sensible fix here _aside_ from making these into unit tests. We don't want to be testing with the full live data, really.