Bug 42959 – Security meta: investigate cross_fuzz crashes

• Component: Security
• Status: RESOLVED FIXED

[Michal Zalewski]

Hi folks,

This fuzzer tends to crash WebKit nightly within several seconds (r63957):

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_fixed.html

The bulk of these crashes are NULL pointers, but I allowed it to run for about 2 hours while collecting crash data, and there seems to be a body of more elusive reads and writes to invalid memory addresses (with a prevalence of about 1 per 100 crashes) that seem exploitable, but will be impossible to track down without these NULL ptrs cleaned up.

ap@ proved to be instrumental in tracking down crashes for a related fuzzer, ref_fuzz (https://bugs.webkit.org/show_bug.cgi?id=26824), so cc:ing him here.

[Michal Zalewski]

Just FYI, to help prioritize: my current plan is to publish this fuzzer in approx 60 days (or earlier, if all vendors address the problems sooner).

Currently, looks like all the major browsers are affected.

[Chris Evans]

@Alexey, how much beer do I need to bribe you to clean this up like you did Michal's ref_fuzz?

[Berend-Jan Wever]

I've collected some information about crashes and I am opening up bugs to track each crash when I have enough details.

[Sam Weinig]

(In reply to comment #1)
> Just FYI, to help prioritize: my current plan is to publish this fuzzer in approx 60 days (or earlier, if all vendors address the problems sooner).
> 
> Currently, looks like all the major browsers are affected.

Can you bring this up on webkit-security.  That is a much better place for us to discuss timelines.

[Michal Zalewski]

For the record, the most current & reliable version is:

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_final_20100728.html

There is substantial logging code in the fuzzer, but disabled for performance reasons; it can probably be diverted to some low-overhead channel to recover additional data.

Sadly, the dynamic DOM-crawling behavior of the fuzzer largely rules out the possibility to generate repros in the form of batches of static JS. I am not sure I can simplify this much further.

[Berend-Jan Wever]

Created attachment 62812 [details]
Server to use to log what the fuzzer does.

I have implemented a server in Python that can accept XHR requests to log information to a file. With minimal changes, the existing fuzzer can use this to output to a file:

- function LOG(message) {

+ var log_id = new Date().valueOf().toString(16).toUpperCase();
+ var log_counter = 0;
+ 
+ function LOG(message) {
+   var xhr = new XMLHttpRequest();
+   xhr.open('POST', '/logs/' + log_id + '.log?' + log_counter++, false);
+   xhr.setRequestHeader("Content-Type", "text/plain; charset=UTF-8");
+   xhr.send(message);

Note that this server was not designed with security in mind: it does allow full R/W access to your file system, so please do not expose it to the intertubes.

How to use this:
1) Put a copy of the fuzzer in a local folder.
2) Copy the Python server in the same folder.
3) Update the LOG function in the fuzzer as described above.
4) Optionally modify the settings at the start of the Python server code.
5) Start the Python server.
6) Point your favorite browser to the server.
The fuzzer will run as usual, but send the log messages to the server along with a unique ID which is based on the start time of the fuzzer. The server will save the messages in a log file based on the unique ID, so you get a new log each time you run the fuzzer.

[Michal Zalewski]

Keep in mind that there is something like 50,000-100,000 LOG() calls per second if you uncomment all the instances, and probably should be even more to better track references.

Replacing this with a mechanism that is some three orders of magnitude slower (XHR) is probably not going to be conductive of getting results in a sane timeframe (at least for the more elusive arbitrary memory access bugs).

[Berend-Jan Wever]

@mz: I was working under the assumption that any way to log data is better than nothing. We should be able to get some info if we run it for, say, 60 days :)

Either way, I got my FuzzFramework port of ref_fuzz updated to work in a similar manner and be more effective at finding bugs. Today it started finding many of the same bugs and producing reduced repros. I'll start filing bugs now :).

[Berend-Jan Wever]

Here's a list of crashes I found during an earlier run in Chromium. This list is probably not exhaustive; I've found crashes not included in this list with my own port of the fuzzer. Note that I have already found a number of different crashes that are caused by almost the same repro, which seems to imply that they are caused by the same bug in different locations. However, the list is quite long, so I expect we have a lot of bugs to fix.

--- Totals ---------------------------------------------------------------------
Total runs: 1632 in 47588 seconds.
Application crashed 1548 out of 1632 runs
Crashes found:
    31 * WebCore::v8ValueToWebCoreString ReadAV@NULL (cb89e8eb9c0d203f405c1d6513d14b97)
        Attempt to read from NULL pointer in WebCore::v8ValueToWebCoreString
     1 * WebCore::CSSStyleSelector::SelectorChecker::checkOneSelector ReadAV@NULL (ee1436017020027e7d0af466472ee8a3)
        Attempt to read from NULL pointer (+0x10) in WebCore::CSSStyleSelector::SelectorChecker::checkOneSelector
     1 * WTF::HashTable<...> ReadAV@NULL (0ad97e2f1c02a050003c01b3c6661a98)
        Attempt to read from NULL pointer (+0x10) in WTF::HashTable<...>
     2 * v8::internal::ProfileTree::TraverseDepthFirst<...> ReadAV@NULL (e6dd0db2316d0681041db7df20ab48ef)
        Attempt to read from NULL pointer (+0x2C) in v8::internal::ProfileTree::TraverseDepthFirst<...>
     4 * [unknown] in v8::internal::Heap::UpdateNewSpaceReferenceInExternalStringTableEntry ExecAV@NULL (e04c151252e686bfdc36fe357a9556c4)
        Security: Attempt to execute non-executable NULL pointer (+0x1F) in [unknown] in v8::internal::Heap::UpdateNewSpaceReferenceInExternalStringTableEntry
    14 * WebCore::v8StringToWebCoreString<...> ReadAV@NULL (82c75721458c2d1fce8189ca52cd8701)
        Attempt to read from NULL pointer in WebCore::v8StringToWebCoreString<...>
    30 * [unknown] in v8::internal::SymbolTableCleaner::VisitPointers ExecAV@NULL (c875397782229150cc8e13416ac91a39)
        Security: Attempt to execute non-executable NULL pointer (+0x1F) in [unknown] in v8::internal::SymbolTableCleaner::VisitPointers
    27 * tcmalloc::ThreadCache::FreeList::PopRange ReadAV@NULL (4f2e4263d2cc9083a01e819d73b99a15)
        Attempt to read from NULL pointer (+0x1FA0) in tcmalloc::ThreadCache::FreeList::PopRange
     1 * v8::internal::ExternalTwoByteString::ExternalTwoByteStringGet ReadAV@NULL (7e0dc27afae1303f90f31a7b5be2731c)
        Attempt to read from NULL pointer (+0x8) in v8::internal::ExternalTwoByteString::ExternalTwoByteStringGet
    27 * WebCore::v8StringToWebCoreString<...> ReadAV@NULL (f8422e8b997988240159f083e2806467)
        Attempt to read from NULL pointer in WebCore::v8StringToWebCoreString<...>
     2 * [unknown] in IPC::ChannelProxy::Context::TryFilters ExecAV@NULL (230f5af34b088778e449a311ae7b95b2)
        Security: Attempt to execute non-executable NULL pointer in [unknown] in IPC::ChannelProxy::Context::TryFilters
  1084 * WebCore::CSSStyleSelector::styleForElement ReadAV@NULL (dc7b32067c1b2c657a6337dd1beb1998)
        Attempt to read from NULL pointer (+0x24) in WebCore::CSSStyleSelector::styleForElement
    55 * v8::Value::IsUndefined ReadAV@NULL (46cf324ff4d90096390ce66a8724d8a2)
        Attempt to read from NULL pointer in v8::Value::IsUndefined
     3 * kZerob ExecAV@Arbitrary (4b1acb8b140d549bc5c7757b28cf9713)
        Security: Attempt to execute non-executable arbitrary memory @ 0x64A68528 in kZerob
     2 * [unknown] in WebCore::V8Storage::namedPropertyEnumerator ExecAV@NULL (de39145667a622f56fc4cf25af85441c)
        Security: Attempt to execute non-executable NULL pointer (+0x1F) in [unknown] in WebCore::V8Storage::namedPropertyEnumerator
    11 * v8::internal::ExternalTwoByteString::ExternalTwoByteStringGet ReadAV@NULL (63208b4dfc31329f25be59e71b94da52)
        Attempt to read from NULL pointer (+0x8) in v8::internal::ExternalTwoByteString::ExternalTwoByteStringGet
    40 * tcmalloc::CentralFreeList::RemoveRange ReadAV@NULL (861a4f18fc3c9a208ad77c4a73079962)
        Attempt to read from NULL pointer (+0x4830) in tcmalloc::CentralFreeList::RemoveRange
     2 * std::list<...>::_Tidy ReadAV@NULL (7db290478ddc450ea5eda549e811f84b)
        Attempt to read from NULL pointer (+0x239E) in std::list<...>::_Tidy
     1 * tcmalloc::ThreadCache::FreeList::PopRange ReadAV@NULL (e7f247aaf6305bd98a58b66163f32ecf)
        Attempt to read from NULL pointer in tcmalloc::ThreadCache::FreeList::PopRange
    86 * tcmalloc::ThreadCache::Allocate ReadAV@NULL (916b1b80b0657b01ca62e934c8bfe435)
        Attempt to read from NULL pointer (+0x6060) in tcmalloc::ThreadCache::Allocate
     1 * WTF::HashMap<...>::find ReadAV@NULL (57234c212b70515421f7f0a865c70d63)
        Attempt to read from NULL pointer (+0x40) in WTF::HashMap<...>::find
     1 * WebCore::ResourceHandle::setDefersLoading ReadAV@NULL (964235000062ac871e13770a6f2465db)
        Attempt to read from NULL pointer (+0x5E4) in WebCore::ResourceHandle::setDefersLoading
     1 * WebCore::Frame::settings ReadAV@Arbitrary (f1f252723a8fba1d1ab2d5638719d85c)
        Security: Attempt to read from arbitrary memory @ 0x1B056606 in WebCore::Frame::settings
    48 * WebCore::v8StringToWebCoreString<...> ReadAV@NULL (13d73eadafae4a0e3f3be0f8b67da3b5)
        Attempt to read from NULL pointer in WebCore::v8StringToWebCoreString<...>
     1 * WebCore::ResourceLoader::releaseResources ReadAV@NULL (b41759dd7a44d2220455d8b6133966f7)
        Attempt to read from NULL pointer in WebCore::ResourceLoader::releaseResources
    13 * v8::internal::SymbolTableCleaner::VisitPointers ReadAV@NULL (b18a7578fdd9e438e324ca402e05de54)
        Attempt to read from NULL pointer (+0x4) in v8::internal::SymbolTableCleaner::VisitPointers
     1 * WebCore::AutoTableLayout::recalcColumn ReadAV@NULL (47c4a759037b83d2f9f51ff84aeecef3)
        Attempt to read from NULL pointer (+0xAF) in WebCore::AutoTableLayout::recalcColumn
     3 * [unknown] in WebCore::CSSStyleSelector::applyProperty ExecAV@NULL (976755a6a4943014432368bf55b269fc)
        Security: Attempt to execute non-executable NULL pointer in [unknown] in WebCore::CSSStyleSelector::applyProperty
     3 * tcmalloc::CentralFreeList::FetchFromSpansSafe ReadAV@NULL (25a95c62d87e4e6097c3c1c93a7fcefa)
        Attempt to read from NULL pointer (+0x1D2C) in tcmalloc::CentralFreeList::FetchFromSpansSafe
     1 * v8::internal::ExternalTwoByteString::ExternalTwoByteStringReadBlockIntoBuffer ReadAV@NULL (52da055f45c2cf4cf2cfed5e67a916b4)
        Attempt to read from NULL pointer (+0x8) in v8::internal::ExternalTwoByteString::ExternalTwoByteStringReadBlockIntoBuffer
     1 * webkit_glue::SerializeNPIdentifier ReadAV@NULL (d8225a2aa13e714081e3b7b3879c10f2)
        Attempt to read from NULL pointer in webkit_glue::SerializeNPIdentifier
     4 * v8::internal::Heap::UpdateNewSpaceReferenceInExternalStringTableEntry ReadAV@NULL (d91d2eb1264f5476ad2c929c9091e3e0)
        Attempt to read from NULL pointer (+0x4) in v8::internal::Heap::UpdateNewSpaceReferenceInExternalStringTableEntry
     9 * std::basic_string<...> ReadAV@NULL (b1664ffeee133fe441240d0f384f63cd)
        Attempt to read from NULL pointer in std::basic_string<...>
     1 * WebCore::Chrome::windowRect ReadAV@NULL (83c16086e53c44a806a015b12dbab5a5)
        Attempt to read from NULL pointer (+0x1760) in WebCore::Chrome::windowRect
     1 * [unknown] in WebCore::V8Proxy::runScript ExecAV@Arbitrary (103362849b00f5068909c9bbc1d8faf5)
        Security: Attempt to execute non-executable arbitrary memory @ 0x00EB45A0 in [unknown] in WebCore::V8Proxy::runScript
     1 * `anonymous namespace'::PureCall DebugBreak (318e1cd3ad6ea781d52d6600f83d944d)
        Debugger breakpoint in `anonymous namespace'::PureCall
     1 * WebCore::RenderLayer::paintList ReadAV@NULL (14683c4f4c3756ddb633951de3999428)
        Attempt to read from NULL pointer in WebCore::RenderLayer::paintList
     1 * [unknown] in WebCore::CSSRule::parentStyleSheet ExecAV@Arbitrary (85dce828ec3b4dcd1e8309a8628d368f)
        Security: Attempt to execute non-executable arbitrary memory @ 0x37353A38 in [unknown] in WebCore::CSSRule::parentStyleSheet
     8 * WebCore::CSSStyleSelector::matchRules ReadAV@NULL (ddadcc564002a587b76843cf206b4d62)
        Attempt to read from NULL pointer (+0x8) in WebCore::CSSStyleSelector::matchRules
    14 * v8::internal::ProfilerEventsProcessor::AddCurrentStack ReadAV@NULL (480a56e37d10e09542742ea90abe880c)
        Attempt to read from NULL pointer (+0x50) in v8::internal::ProfilerEventsProcessor::AddCurrentStack
     8 * kZerob ExecAV@Arbitrary (a9ede760a659c2e0e07f2ceaa3e0f871)
        Security: Attempt to execute non-executable arbitrary memory @ 0x63138528 in kZerob
     2 * tcmalloc::ThreadCache::ReleaseToCentralCache ReadAV@NULL (c17ede9be4ef0c35752557d3b9fd0595)
        Attempt to read from NULL pointer in tcmalloc::ThreadCache::ReleaseToCentralCache

[Michal Zalewski]

New major version. The two most significant changes include randomizing DOM crawl order, and limiting object crawl fanout.

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_randomized_20100729.html

It seems to trigger a more varied set of faults, and is less prone to hitting the NULL ptrs.

[Alexey Proskuryakov]

<rdar://problem/8260939>

[Adam Barth]

https://bugs.webkit.org/show_bug.cgi?id=43677
https://bugs.webkit.org/show_bug.cgi?id=43680

Not marking as blockers because I don't want to link to this bug from non-security bugs.

[Adam Barth]

I can't get much more useful out of this fuzzer because I keep hitting Bug 43676.  Hopefully dumi can fix that one soon.

[Adam Barth]

dumi posted a patch for Bug 43676.  He says he ran the fuzzer for a while after that without hitting any other problems.  I'll give it another spin at some point.

[Adam Barth]

I played with this a bit more and filed a bunch of bugs (some with patches).  It found at least one real security bug.  I suspect there's more to find here.  On my last run, it ran for a while and then locked up in some SVG morphology code that I didn't understand.

In any case, I think it would be more productive if other folks started looking at this fuzzer too.

[Michal Zalewski]

OK - not to be annoying, but is there any specific plan to tackle these issues any further at this point?

[Adam Barth]

(In reply to comment #16)
> OK - not to be annoying, but is there any specific plan to tackle these issues any further at this point?

Do you mean w.r.t. the open bugs?  The open bugs appear to be a null pointer crash, an assert, a request for a test case, and one nasty crash with zero information about how to reproduce.  That seems pretty non-actionable / low-impact to me.

If you mean w.r.t. finding new bugs with the fuzzer, I think there's more work to be done there, however.

[Michal Zalewski]

Yes, the latter (unless there are no more exploitable crashes on trunk - in which case, I can verify and close this bug).

[Adam Barth]

I think the fuzzer needs more running.  As I said in Comment #15, I'd prefer if someone else would give it a spin, but if no one steps up to the plate, it will rise to the top of my list again at some point.

[Justin Schuh]

We got a bit delayed on our end, but we'll have it running and outputting reproductions in the next day or so. And we're just going to leave it chugging until it stops finding things.

[Michal Zalewski]

Note that logging *can* be enabled in the original fuzzer very easily, to collect a trace of all executed JS commands.

There is some (commented out) code along these lines in place already, which also provides debug information about the DOM crawl status; but the surest and most robust way to do this is just to replace all calls to eval() with calls a logging version.

The only downside of this is that:

1) It is infuriatingly slow (but that's better than not making progress at all),

2) The repros will be huge and not necessarily easy to minimize.

[Michal Zalewski]

Here's an untested, but probably close-to-usable version of the fuzzer that passes all the executed JS commands to LOG() - which in turn can be piped to a suitable channel, and with minimal effort, used to recover (probably still very unhelpful) repro cases. The default <textarea> output is painfully slow. Perhaps console.log() is better.

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_randomized_20100729_logging.html

Note that cross_fuzz_randomized_20100729.html should remain the canonical test case for confirming there are no outstanding issues.

[Alexey Proskuryakov]

The important part is being able to reproduce whatever crash was encountered during a given run. Logging all operations is one way to approach this. Another way is to log pseudo-random number generator seed, and then replay a sequence of subtests - that's what we've been getting from iExploder, for example. 

There might be other ways that are more appropriate for cross_fuzz. We could even add temporary hooks in local builds for that.

[Michal Zalewski]

Yes, sure; plugging a seed-enabled PRNG into the fuzzer is 2 minutes of work, but in the previous conversations, I had the impression that people are not happy with this alone.

[Michal Zalewski]

Here's a version that does just that:

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_randomized_20100729_seed.html

It accepts seed in location.hash, say:

http://lcamtuf.coredump.cx/cross_fuzz/cross_fuzz_randomized_20100729_seed.html#1234

If no seed is found, it picks one based on system time, and updates location.hash accordingly.

[Alexey Proskuryakov]

Alas, this doesn't seem to help much for some reason - even with a seed, I'm getting wildly different crashes each time.

Logging versions seem to be quite slow indeed, even when using console.log. I've modified JSC to add all eval arguments to a static Vector<UChar*> internally, that seem to work better.

[Michal Zalewski]

The seeded version works reasonably well for me in Chrome, but there is probably some jitter related to document loading times, etc. Do you mean you can never repro anything with the same seed, or that results vary only from time to time?

[Alexey Proskuryakov]

In my experience, it seemed like the seed didn't help achieve reproducible results. But it usually takes a minute or two to crash, so there is a lot of time for loading time to affect results.

[Michal Zalewski]

There hasn't been any activity on this bug for a while, so just wanted to check status. Looks like almost all the linked bugs are fixed, but I think there are still some security-relevant crashes in nightly builds?

[Michal Zalewski]

Just FYI, I have made modifications to cross_fuzz that greatly improve the ability to reproduce crashes from a known seed. It's not nearly as good as having clean repros, but it allows you to run cross_fuzz continuously, pick the seed that crashed the soonest, and work from there reproducing at will.

To make it work, you need to do the following:

1) wget -r -np the entire http://lcamtuf.coredump.cx/cross_fuzz/ directory to file:///

2) Set home page for the test profile to about:blank, disable
safesearch, auto-updates, and any other mechanisms that may be introducing additional unpredictable latency. Also disable pop-up blocking.

3) Start the browser in incognito mode, allowing JS file:/// access. For Chrome, this is:

chrome --incognito --allow-file-access-from-files
file:///somewhere/cross_fuzz_randomized_20110105_seed.html#1234

...where #1234 is your desired seed (pick a 32-bit random integer across fuzzing runs).

In Chrome, this yields a very good repro rate.

[Alexey Proskuryakov]

We've got a lot of mileage from this, and this bug doesn't appear to be tracking anything practical at this time. Folks of course continue running fuzzers, but there has been no orchestrated effort to fix cross_fuzz crashes in more than two years.

Resolved/Fixed.

[Alex Mayer]

Hi!
I added cross _fuzz preservation test pages. Store them in localStorage. When you restart, the last test page in a browser, on which there was a crash, crash does not happen again. Why is that?

[Alex Mayer]

Hi!
I added cross _fuzz preservation test pages. Store them in localStorage. When you restart, the last test page in a browser, on which there was a crash, crash does not happen again. Why is that?(In reply to comment #30)
> Just FYI, I have made modifications to cross_fuzz that greatly improve the ability to reproduce crashes from a known seed. It's not nearly as good as having clean repros, but it allows you to run cross_fuzz continuously, pick the seed that crashed the soonest, and work from there reproducing at will.
> 
> To make it work, you need to do the following:
> 
> 1) wget -r -np the entire http://lcamtuf.coredump.cx/cross_fuzz/ directory to file:///
> 
> 2) Set home page for the test profile to about:blank, disable
> safesearch, auto-updates, and any other mechanisms that may be introducing additional unpredictable latency. Also disable pop-up blocking.
> 
> 3) Start the browser in incognito mode, allowing JS file:/// access. For Chrome, this is:
> 
> chrome --incognito --allow-file-access-from-files
> file:///somewhere/cross_fuzz_randomized_20110105_seed.html#1234
> 
> ...where #1234 is your desired seed (pick a 32-bit random integer across fuzzing runs).
> 
> In Chrome, this yields a very good repro rate.