DRT/WTR should clear the memory cache between tests

• Component: Tools / Tests
• Status: NEW ---

[Simon Fraser (smfr)]

DRT doesn't seem to clear the memory cache between tests. I was trying to make a test that has window.internals.settings.setImageSubsamplingEnabled(true); and a ref test that does not, but the CachedImage was created by the test then just re-used by the reference, giving me a bogus pass.

[Simon Fraser (smfr)]

Antti says:
I think that is a too big of a hammer. You can use internals.clearMemoryCache() if your test relies on empty memory cache.

[Chris Dumez]

I am personally in favor of this change. I will help reduce flakiness. I had to add internals.clearMemoryCache() calls at the beginning of 24 tests recently to make them not flaky, I don't think this is very scalable. We already disable PageCache during testing, I think it makes sense to make the MemoryCache opt-in as well.

[Chris Dumez]

(In reply to comment #2)
> I am personally in favor of this change. I will help reduce flakiness. I had
> to add internals.clearMemoryCache() calls at the beginning of 24 tests
> recently to make them not flaky, I don't think this is very scalable. We
> already disable PageCache during testing, I think it makes sense to make the
> MemoryCache opt-in as well.

I would also add that I wasted engineering time investigating the source of the flakiness until I was able to determine the reason was the memory cache. This WILL happen again and more people WILL waste time on this.

[Simon Fraser (smfr)]

Reopening.

[Alexey Proskuryakov]

One concern if that *may* make the tests substantially slower. We'll need to measure performance if we do this.

[Antti Koivisto]

internals.clearMemoryCache is not to meant to be flakiness fix but used when a test actually relies on having a clear cache.

Doing it everywhere would slow down tests (no idea if enough to matter) and reduce implicit test coverage, moving testing further from real browser behavior in general.