Bug 71726 - https://bugs.webkit.org/robots.txt should allow indexing of bugs
Summary: https://bugs.webkit.org/robots.txt should allow indexing of bugs
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-07 13:19 PST by Eric Seidel (no email)
Modified: 2015-12-07 10:08 PST (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Seidel (no email) 2011-11-07 13:19:42 PST 
https://bugs.webkit.org/robots.txt should allow indexing of bugs

We can give it an insanely high crawl-delay if we're worried about server load.  I just want to be able to search for bugs from 3 years ago in Google, since bugzilla's built in stemming is very primitive.

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive