Bug 259748 – test-webkitpy -i fails; webkitpy.common.config.committers_unittest.CommittersTest integration tests are super fragile

• Status: NEW
• Component: Tools / Tests

[Sam Sneddon [:gsnedders]]

Once again the webkitpy.common.config.committers_unittest.CommittersTest.integration_test_* tests bite us:

```
[344/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:                                         
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
      self._assert_fuzz_match('Ada', 'Ada Chan', 0)
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
  
  First differing element 0:
  ['Ada Chan', 'Ada Rose Cannon']
  ['Ada Chan']
  
  - (['Ada Chan', 'Ada Rose Cannon'], 0)
  + (['Ada Chan'], 0)
  
[384/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
      self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
  
  First differing element 0:
  ['Bri Harris', 'David Harrison', 'Harrison Weinerman']
  ['David Harrison']
  
  - (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
  + (['David Harrison'], 4)
```

It also seems we don't run the integration tests in either EWS or post-commit?

[Radar WebKit Bug Importer]

<rdar://problem/113292283>

[Sam Sneddon [:gsnedders]]

test-webkitpy -i webkitpy.common.config.committers_unittest.CommittersTest now gives us three failures:

[9/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
      self._assert_fuzz_match('Ada', 'Ada Chan', 0)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
  
  First differing element 0:
  ['Ada Chan', 'Ada Rose Cannon']
  ['Ada Chan']
  
  - (['Ada Chan', 'Ada Rose Cannon'], 0)
  + (['Ada Chan'], 0)
  
[16/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_eric_seidel failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 241, in integration_test_contributors_eric_seidel
      self._assert_fuzz_match('Eric S', 'Eric Seidel', 0)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[74 chars] 2.0) != (['Eric Seidel'], 0)
  
  First differing element 0:
  ['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[68 chars] Li']
  ['Eric Seidel']
  
  - (['Eric Carlson',
  -   'Eric Hutchison',
  -   'Eric Penner',
  -   'Eric Roman',
  -   'Eric Seidel',
  ? ^^
  
  + (['Eric Seidel'], 0)
  ? ^^             + +++
  
  -   'Eric Staggs',
  -   'Eric Uhrhane',
  -   'Erica Li'],
  -  2.0)
  
[48/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
      self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
  
  First differing element 0:
  ['Bri Harris', 'David Harrison', 'Harrison Weinerman']
  ['David Harrison']
  
  - (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
  + (['David Harrison'], 4)
  
Ran 50 tests in 1.246s                                                                                        
FAILED (failures=3, errors=0)

That this has gotten worse shows quite how fragile all of this is, because relying on the live committers.json is inherently bad.

[Sam Sneddon [:gsnedders]]

Honestly I don't see any sensible fix here _aside_ from making these into unit tests. We don't want to be testing with the full live data, really.