Bug 104184 – nrwt: record the order and run times of all of the tests in a single file

• Component: New Bugs
• Status: RESOLVED FIXED

[Dirk Pranke]

nrwt: record the order and run times of all of the tests in a single file

[Dirk Pranke]

Created attachment 177858 [details]
Patch

[Ojan Vafai]

Comment on attachment 177858 [details]
Patch

Are python dicts ordered and will json.dumps preserve the ordering?

[Dirk Pranke]

No, and no. That's why the results tuples include the worker and the order in each worker. I actually wrote a separate script (not included in this patch) that verified that you could reconstruct the tests_run*.txt files from the json and they matched.

I believe you can recreate the total ordering of tests across the whole run to within a millisecond with this data; in the unit test data all of the tests complete in less than a millisecond, so the values are all zeros and I'm not doing more fancy testing to verify that we can-do cross-shard ordering here (that, and the unit test only runs with one worker).

[WebKit Review Bot]

Comment on attachment 177858 [details]
Patch

Rejecting attachment 177858 [details] from commit-queue.

New failing tests:
inspector-protocol/debugger-terminate-dedicated-worker-while-paused.html
Full output: http://queues.webkit.org/results/15231765

[Dirk Pranke]

Committed r137356: <http://trac.webkit.org/changeset/137356>

[Csaba Osztrogonác]

(In reply to comment #3)
> No, and no. That's why the results tuples include the worker and the order in each worker. I actually wrote a separate script (not included in this patch) that verified that you could reconstruct the tests_run*.txt files from the json and they matched.

Can we find this script somewhere? tests_run*.txt files were 
really useful to debug flakinesses related to the order.

[Balazs Kelemen]

(In reply to comment #6)
> (In reply to comment #3)
> > No, and no. That's why the results tuples include the worker and the order in each worker. I actually wrote a separate script (not included in this patch) that verified that you could reconstruct the tests_run*.txt files from the json and they matched.
> 
> Can we find this script somewhere? tests_run*.txt files were 
> really useful to debug flakinesses related to the order.

+1
IMHO it was better to generate them by default.

[Dirk Pranke]

I have no idea what happened to that script but I'd be happy to write another one. I will post it in a separate bug.

[Dirk Pranke]

script posted in bug 116571. Feel free to play with it / review it / land it at your convenience.

[Balazs Kelemen]

(In reply to comment #9)
> script posted in bug 116571. Feel free to play with it / review it / land it at your convenience.

Thanks!. I vote for merging the code into rwt and generate the files by default. Ossy? Others?

[Dirk Pranke]

(In reply to comment #10)
> (In reply to comment #9)
> > script posted in bug 116571. Feel free to play with it / review it / land it at your convenience.
> 
> Thanks!. I vote for merging the code into rwt and generate the files by default. Ossy? Others?

I don't particularly want to merge this code back into rwt (since this patch took it out in favor of the new format). It takes up space in the results files, complicates the core run-webkit-tests code, and is unused 99% of the time.

That said, I won't stop you if you want to do the work :).