RESOLVED FIXED 95688
Allow robots to index bugs on Bugzilla 
https://bugs.webkit.org/show_bug.cgi?id=95688
Summary Allow robots to index bugs on Bugzilla
Peter Beverloo
Reported 2012-09-03 07:14:57 PDT
Allow robots to index bugs on Bugzilla
Attachments
Patch (1.21 KB, patch)
2012-09-03 07:16 PDT, Peter Beverloo 		no flags
DetailsFormatted DiffDiff
Patch (1.22 KB, patch)
2012-09-06 06:19 PDT, Peter Beverloo 		no flags
DetailsFormatted DiffDiff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.
Peter Beverloo
Comment 1 2012-09-03 07:16:05 PDT
Created attachment 161923 [details] Patch
Peter Beverloo
Comment 2 2012-09-03 07:21:36 PDT
Hi Bill, Lucas, With performance of the WebKit sites having been improved significantly lately, I was hoping we could enable indexing of show_bug.cgi by search engines. Bugzilla's search function is not always ideal, and this will make bugs much more discoverable. Almost all bugs are referenced by commits, review requests and the webkit-unassigned list, so indexing should go quickly. One thing we preemptively could do against performance worries, is setting a crawl delay in the robots.txt file itself to limit the number of requests that will be made.
William Siegrist
Comment 3 2012-09-04 13:43:38 PDT
I'm fine with allowing more crawling but do please add the crawl delay. For trac, we set a delay of 20.
Peter Beverloo
Comment 4 2012-09-06 06:19:57 PDT
Created attachment 162490 [details] Patch
Peter Beverloo
Comment 5 2012-09-06 06:20:13 PDT
Comment on attachment 162490 [details] Patch Done.
Peter Beverloo
Comment 6 2012-09-24 04:24:05 PDT
Bill, could you please take another look? The scroll delay is now equal to trac's.
William Siegrist
Comment 7 2012-09-24 08:12:20 PDT
Patch looks okay to me.
Eric Seidel (no email)
Comment 8 2012-09-24 08:23:46 PDT
Comment on attachment 162490 [details] Patch If bill is ok with the patch, then it's LGTM.
WebKit Review Bot
Comment 9 2012-09-24 08:33:24 PDT
Comment on attachment 162490 [details] Patch Clearing flags on attachment: 162490 Committed r129369: <http://trac.webkit.org/changeset/129369>
WebKit Review Bot
Comment 10 2012-09-24 08:33:27 PDT
All reviewed patches have been landed. Closing bug.
Peter Beverloo
Comment 11 2012-09-24 08:46:22 PDT
Thanks Bill and Eric!
Eric Seidel (no email)
Comment 12 2012-09-24 09:39:27 PDT
Maybe we should also fix-up robots.txt to explicitly allow webkit-patch so webkit-patch can stop cheating the system: http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py#L286 :) (I also doubt it matters much.)
Note You need to log in before you can comment on or make changes to this bug.