Bug 95688

Summary: Allow robots to index bugs on Bugzilla
Product: WebKit Reporter: Peter Beverloo <peter>
Component: New BugsAssignee: Peter Beverloo <peter>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, eric, lforschler, mrowe, webkit.review.bot, wsiegrist
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Peter Beverloo 2012-09-03 07:14:57 PDT 
Allow robots to index bugs on Bugzilla
Comment 1 Peter Beverloo 2012-09-03 07:16:05 PDT 
Created attachment 161923 [details]
Patch
Comment 2 Peter Beverloo 2012-09-03 07:21:36 PDT 
Hi Bill, Lucas,

With performance of the WebKit sites having been improved significantly lately, I was hoping we could enable indexing of show_bug.cgi by search engines. Bugzilla's search function is not always ideal, and this will make bugs much more discoverable. Almost all bugs are referenced by commits, review requests and the webkit-unassigned list, so indexing should go quickly.

One thing we preemptively could do against performance worries, is setting a crawl delay in the robots.txt file itself to limit the number of requests that will be made.
Comment 3 William Siegrist 2012-09-04 13:43:38 PDT 
I'm fine with allowing more crawling but do please add the crawl delay. For trac, we set a delay of 20.
Comment 4 Peter Beverloo 2012-09-06 06:19:57 PDT 
Created attachment 162490 [details]
Patch
Comment 5 Peter Beverloo 2012-09-06 06:20:13 PDT 
Comment on attachment 162490 [details]
Patch

Done.
Comment 6 Peter Beverloo 2012-09-24 04:24:05 PDT 
Bill, could you please take another look? The scroll delay is now equal to trac's.
Comment 7 William Siegrist 2012-09-24 08:12:20 PDT 
Patch looks okay to me.
Comment 8 Eric Seidel (no email) 2012-09-24 08:23:46 PDT 
Comment on attachment 162490 [details]
Patch

If bill is ok with the patch, then it's LGTM.
Comment 9 WebKit Review Bot 2012-09-24 08:33:24 PDT 
Comment on attachment 162490 [details]
Patch

Clearing flags on attachment: 162490

Committed r129369: <http://trac.webkit.org/changeset/129369>
Comment 10 WebKit Review Bot 2012-09-24 08:33:27 PDT 
All reviewed patches have been landed.  Closing bug.
Comment 11 Peter Beverloo 2012-09-24 08:46:22 PDT 
Thanks Bill and Eric!
Comment 12 Eric Seidel (no email) 2012-09-24 09:39:27 PDT 
Maybe we should also fix-up robots.txt to explicitly allow webkit-patch so webkit-patch can stop cheating the system:
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py#L286

:)

(I also doubt it matters much.)