Bug 259748 - test-webkitpy -i fails; webkitpy.common.config.committers_unittest.CommittersTest integration tests are super fragile
Summary: test-webkitpy -i fails; webkitpy.common.config.committers_unittest.Committers...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-08-02 14:49 PDT by Sam Sneddon [:gsnedders]
Modified: 2024-10-21 14:21 PDT (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sam Sneddon [:gsnedders] 2023-08-02 14:49:56 PDT 
Once again the webkitpy.common.config.committers_unittest.CommittersTest.integration_test_* tests bite us:

```
[344/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:                                         
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
      self._assert_fuzz_match('Ada', 'Ada Chan', 0)
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
  
  First differing element 0:
  ['Ada Chan', 'Ada Rose Cannon']
  ['Ada Chan']
  
  - (['Ada Chan', 'Ada Rose Cannon'], 0)
  + (['Ada Chan'], 0)
  
[384/2617] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
      self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
    File "/Volumes/gsnedders/projects/Safari/OpenSource/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
  
  First differing element 0:
  ['Bri Harris', 'David Harrison', 'Harrison Weinerman']
  ['David Harrison']
  
  - (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
  + (['David Harrison'], 4)
```

It also seems we don't run the integration tests in either EWS or post-commit?
Comment 1 Radar WebKit Bug Importer 2023-08-02 14:50:10 PDT 
<rdar://problem/113292283>
Comment 2 Sam Sneddon [:gsnedders] 2024-10-21 13:41:04 PDT 
test-webkitpy -i webkitpy.common.config.committers_unittest.CommittersTest now gives us three failures:

[9/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_ada_chan failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 140, in integration_test_contributors_ada_chan
      self._assert_fuzz_match('Ada', 'Ada Chan', 0)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Ada Chan', 'Ada Rose Cannon'], 0) != (['Ada Chan'], 0)
  
  First differing element 0:
  ['Ada Chan', 'Ada Rose Cannon']
  ['Ada Chan']
  
  - (['Ada Chan', 'Ada Rose Cannon'], 0)
  + (['Ada Chan'], 0)
  
[16/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_eric_seidel failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 241, in integration_test_contributors_eric_seidel
      self._assert_fuzz_match('Eric S', 'Eric Seidel', 0)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[74 chars] 2.0) != (['Eric Seidel'], 0)
  
  First differing element 0:
  ['Eric Carlson', 'Eric Hutchison', 'Eric Penner'[68 chars] Li']
  ['Eric Seidel']
  
  - (['Eric Carlson',
  -   'Eric Hutchison',
  -   'Eric Penner',
  -   'Eric Roman',
  -   'Eric Seidel',
  ? ^^
  
  + (['Eric Seidel'], 0)
  ? ^^             + +++
  
  -   'Eric Staggs',
  -   'Eric Uhrhane',
  -   'Erica Li'],
  -  2.0)
  
[48/50] webkitpy.common.config.committers_unittest.CommittersTest.integration_test_contributors_david_harrison failed:
  Traceback (most recent call last):
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 213, in integration_test_contributors_david_harrison
      self._assert_fuzz_match('Dr. Harrison', 'David Harrison', 4)
    File "/Volumes/gsnedders/projects/Safari/OS2/Tools/Scripts/webkitpy/common/config/committers_unittest.py", line 105, in _assert_fuzz_match
      self.assertEqual(([contributor.full_name for contributor in contributors], distance), (expected_names, expected_distance))
  AssertionError: Tuples differ: (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4) != (['David Harrison'], 4)
  
  First differing element 0:
  ['Bri Harris', 'David Harrison', 'Harrison Weinerman']
  ['David Harrison']
  
  - (['Bri Harris', 'David Harrison', 'Harrison Weinerman'], 4)
  + (['David Harrison'], 4)
  
Ran 50 tests in 1.246s                                                                                        
FAILED (failures=3, errors=0)

That this has gotten worse shows quite how fragile all of this is, because relying on the live committers.json is inherently bad.
Comment 3 Sam Sneddon [:gsnedders] 2024-10-21 14:21:15 PDT 
Honestly I don't see any sensible fix here _aside_ from making these into unit tests. We don't want to be testing with the full live data, really.