Turns out that we consult robots.txt before talking to EWS. We shouldn't - it's a waste of time, and prevents us from easily blocking search indexers.
Created attachment 221667 [details] proposed patch
Committed <http://trac.webkit.org/r162357>.
Fixed webkitpy regression tests (hopefully) in <http://trac.webkit.org/r162371>.