Bug 95243 – webkit-patch rebaseline-expectations hangs

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Julien Chaffraix]

Here is a trace of webkit-patch --verbose rebaseline-expectations run inside the LayoutTests/ subdirectory when it hang:

webkitpy.common.system.autoinstall: [DEBUG] URL for mechanize already downloaded.  Skipping...
webkitpy.common.system.autoinstall: [DEBUG]     "http://pypi.python.org/packages/source/m/mechanize/mechanize-0.2.5.tar.gz"
webkitpy.common.system.executive: [DEBUG] "svn info" took 0.07s
webkitpy.common.system.executive: [DEBUG] "svn info" took 0.06s
webkitpy.common.system.executive: [DEBUG] "svn info" took 0.06s
webkitpy.common.system.executive: [DEBUG] "svn info" took 0.07s
webkitpy.common.system.executive: [DEBUG] "perl Tools/Scripts/webkit-build-directory --chromium-linux" took 0.08s
webkitpy.common.find_files: [DEBUG] Test gathering took 0.017516 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025551 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000436 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000176 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.010846 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025721 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000427 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000175 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.010764 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025546 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000514 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000176 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.010848 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025513 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000683 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000278 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.010931 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025781 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000428 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000177 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.010792 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.025706 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000423 seconds
webkitpy.common.find_files: [DEBUG] Test gathering took 0.000177 seconds
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/efl/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Skipped does not exist: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/gtk/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/mac/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/mac-lion/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/mac/Skipped
webkitpy.common.system.executive: [DEBUG] "qmake -v" took 0.02s
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/qt-4.8/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/qt-linux/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/qt/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Using Skipped file: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/win/Skipped
webkitpy.layout_tests.port.base: [DEBUG] Skipped does not exist: /usr/local/google/Projects/Gardening/WebKit/LayoutTests/platform/win-7sp0/Skipped

ps reports the following:

139751   32266  3.1  0.7 169804 89668 pts/6    Sl+  14:01   0:04 python /home/jchaffraix/Projects//Gardening/WebKit/Tools/Scripts/webkit-patch -v rebaseline-expectations

I don't know what the triggers is as it happens randomly after several rebaselinings. Once it started, it is happening consistently and the only I found around that is blowing off the entire repository :(

[Julien Chaffraix]

Let's forget about SVN, I got it to happen on a GIT checkout too. Same trace, different repository.

[Julien Chaffraix]

After more debugging, here is the reason why we are hanging up:

* AbstractParallelRebaselineCommand's _rebaseline is called with an empty test_list ({})
* Because we have no test, we call Executive's run_in_parallel without any command to run.
* We then call join() we wait forever as there is nothing to wait for.

Now, what I don't understand is how we get no test to rebaseline when there are entries that match.

[Ojan Vafai]

rebasline-expectations just delegates to rebaseline-json. Can you print out the rebaseline-json command that's used?

[Dirk Pranke]

perhaps we're not able to retrieve the results from the bots? I've actually seen this happen a few times in the past hour or two as well.

As an aside, we also don't check the return values from the commands and don't notice if they fail :(.

[Dirk Pranke]

Fixing this is simple but it would be nice if we could see the rebaseline-json command first to confirm that there isn't also another bug cropping up.

[Julien Chaffraix]

(In reply to comment #5)
> Fixing this is simple but it would be nice if we could see the rebaseline-json command first to confirm that there isn't also another bug cropping up.

Any pointer as to how to dump this command? I tried to understand how rebaseline-expectations goes through rebaseline-json but couldn't so it's hard to know where to do the dump.

[Dirk Pranke]

Sorry, Ojan's comment was wrong and threw me off. rebaseline-expectations delegates to rebaseline-test-internal, not rebaseline-json.

It would be interesting to see if we're making it to the call in _rebaseline (on line 302), and we could wrap those lines in some _log.debug() calls. That said, it sounds like maybe your hang is different than mine (and the trace looks different than I would've expected as well), which is why it would be interesting to dig a little further ...

[Julien Chaffraix]

> It would be interesting to see if we're making it to the call in _rebaseline (on line 302), and we could wrap those lines in some _log.debug() calls.

Mmh, I posted a kinda stacktrace in comment #2 which confirms that _rebaseline (line 300) is indeed called. The issue is that |test_list| is empty when we do and that leads webkit-patch to hang on line 302:

command_results = self._tool.executive.run_in_parallel(commands)

>  That said, it sounds like maybe your hang is different than mine (and the trace looks different than I would've expected as well), which is why it would be interesting to dig a little further ...

Let me digg further as it is preventing me from doing any meaningful rebaselining.

[Dirk Pranke]

(In reply to comment #8)
> > It would be interesting to see if we're making it to the call in _rebaseline (on line 302), and we could wrap those lines in some _log.debug() calls.
> 
> Mmh, I posted a kinda stacktrace in comment #2 which confirms that _rebaseline (line 300) is indeed called. The issue is that |test_list| is empty when we do and that leads webkit-patch to hang on line 302:
> 
> command_results = self._tool.executive.run_in_parallel(commands)
> 

Right, I saw that, and it's definitely a bug. I'll post a patch for it shortly. That said, the trace in the bug description looks different than what I would expect ...


> >  That said, it sounds like maybe your hang is different than mine (and the trace looks different than I would've expected as well), which is why it would be interesting to dig a little further ...
> 
> Let me digg further as it is preventing me from doing any meaningful rebaselining.

I assume you've worked around this for now? Definitely let me know if you're blocked on something.

[Julien Chaffraix]

> I assume you've worked around this for now? Definitely let me know if you're blocked on something.

I haven't found a real work-around apart from forcefully blowing off the source tree which is slow and annoying.

Debugging further it looks like TestExpectationModel doesn't contain any REBASELINE test (self._modifier_to_tests[MODIFIERS.REBASELINE] is empty) when this happens even if Chromium's TestExpectation was changed.

[Dirk Pranke]

Created attachment 161328 [details]
Patch

[Tony Chang]

Comment on attachment 161328 [details]
Patch

OK

[Dirk Pranke]

Committed r127057: <http://trac.webkit.org/changeset/127057>