Bug 95688 - Allow robots to index bugs on Bugzilla
Summary: Allow robots to index bugs on Bugzilla
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Peter Beverloo
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-09-03 07:14 PDT by Peter Beverloo
Modified: 2012-09-24 09:39 PDT (History)
6 users (show)

See Also:

Attachments
Patch (1.21 KB, patch)
2012-09-03 07:16 PDT, Peter Beverloo 		no flags Details | Formatted Diff | Diff
Patch (1.22 KB, patch)
2012-09-06 06:19 PDT, Peter Beverloo 		no flags Details | Formatted Diff | Diff
Show Obsolete (1) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Beverloo 2012-09-03 07:14:57 PDT 
Allow robots to index bugs on Bugzilla
Comment 1 Peter Beverloo 2012-09-03 07:16:05 PDT 
Created attachment 161923 [details]
Patch
Comment 2 Peter Beverloo 2012-09-03 07:21:36 PDT 
Hi Bill, Lucas,

With performance of the WebKit sites having been improved significantly lately, I was hoping we could enable indexing of show_bug.cgi by search engines. Bugzilla's search function is not always ideal, and this will make bugs much more discoverable. Almost all bugs are referenced by commits, review requests and the webkit-unassigned list, so indexing should go quickly.

One thing we preemptively could do against performance worries, is setting a crawl delay in the robots.txt file itself to limit the number of requests that will be made.
Comment 3 William Siegrist 2012-09-04 13:43:38 PDT 
I'm fine with allowing more crawling but do please add the crawl delay. For trac, we set a delay of 20.
Comment 4 Peter Beverloo 2012-09-06 06:19:57 PDT 
Created attachment 162490 [details]
Patch
Comment 5 Peter Beverloo 2012-09-06 06:20:13 PDT 
Comment on attachment 162490 [details]
Patch

Done.
Comment 6 Peter Beverloo 2012-09-24 04:24:05 PDT 
Bill, could you please take another look? The scroll delay is now equal to trac's.
Comment 7 William Siegrist 2012-09-24 08:12:20 PDT 
Patch looks okay to me.
Comment 8 Eric Seidel (no email) 2012-09-24 08:23:46 PDT 
Comment on attachment 162490 [details]
Patch

If bill is ok with the patch, then it's LGTM.
Comment 9 WebKit Review Bot 2012-09-24 08:33:24 PDT 
Comment on attachment 162490 [details]
Patch

Clearing flags on attachment: 162490

Committed r129369: <http://trac.webkit.org/changeset/129369>
Comment 10 WebKit Review Bot 2012-09-24 08:33:27 PDT 
All reviewed patches have been landed.  Closing bug.
Comment 11 Peter Beverloo 2012-09-24 08:46:22 PDT 
Thanks Bill and Eric!
Comment 12 Eric Seidel (no email) 2012-09-24 09:39:27 PDT 
Maybe we should also fix-up robots.txt to explicitly allow webkit-patch so webkit-patch can stop cheating the system:
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py#L286

:)

(I also doubt it matters much.)