95688 2012-09-03 07:14:57 -0700 Allow robots to index bugs on Bugzilla 2012-09-24 09:39:27 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 peter peter abarth eric lforschler mrowe webkit.review.bot wsiegrist oldest_to_newest 710630 0 peter 2012-09-03 07:14:57 -0700 Allow robots to index bugs on Bugzilla 710632 1 161923 peter 2012-09-03 07:16:05 -0700 Created attachment 161923 Patch 710635 2 peter 2012-09-03 07:21:36 -0700 Hi Bill, Lucas, With performance of the WebKit sites having been improved significantly lately, I was hoping we could enable indexing of show_bug.cgi by search engines. Bugzilla's search function is not always ideal, and this will make bugs much more discoverable. Almost all bugs are referenced by commits, review requests and the webkit-unassigned list, so indexing should go quickly. One thing we preemptively could do against performance worries, is setting a crawl delay in the robots.txt file itself to limit the number of requests that will be made. 711415 3 wsiegrist 2012-09-04 13:43:38 -0700 I'm fine with allowing more crawling but do please add the crawl delay. For trac, we set a delay of 20. 713224 4 162490 peter 2012-09-06 06:19:57 -0700 Created attachment 162490 Patch 713226 5 162490 peter 2012-09-06 06:20:13 -0700 Comment on attachment 162490 Patch Done. 726539 6 peter 2012-09-24 04:24:05 -0700 Bill, could you please take another look? The scroll delay is now equal to trac's. 726674 7 wsiegrist 2012-09-24 08:12:20 -0700 Patch looks okay to me. 726681 8 162490 eric 2012-09-24 08:23:46 -0700 Comment on attachment 162490 Patch If bill is ok with the patch, then it's LGTM. 726693 9 162490 webkit.review.bot 2012-09-24 08:33:24 -0700 Comment on attachment 162490 Patch Clearing flags on attachment: 162490 Committed r129369: <http://trac.webkit.org/changeset/129369> 726694 10 webkit.review.bot 2012-09-24 08:33:27 -0700 All reviewed patches have been landed. Closing bug. 726709 11 peter 2012-09-24 08:46:22 -0700 Thanks Bill and Eric! 726758 12 eric 2012-09-24 09:39:27 -0700 Maybe we should also fix-up robots.txt to explicitly allow webkit-patch so webkit-patch can stop cheating the system: http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py#L286 :) (I also doubt it matters much.) 161923 2012-09-03 07:16:05 -0700 2012-09-06 06:19:54 -0700 Patch bug-95688-20120903151549.patch text/plain 1235 peter U3VidmVyc2lvbiBSZXZpc2lvbjogMTI3NDE1CmRpZmYgLS1naXQgYS9XZWJzaXRlcy9idWdzLndl YmtpdC5vcmcvQ2hhbmdlTG9nIGIvV2Vic2l0ZXMvYnVncy53ZWJraXQub3JnL0NoYW5nZUxvZwpp bmRleCBhZjYxMWJhOGZkN2JiNWNmZTUzZTllNzY2NDllNzMxNDNjYTcyMTdlLi45MGU1MWY1ZWRh MDA2MDA4Nzk0ZTk4NzViN2Q0NWIzMTA4OWE0ZTY3IDEwMDY0NAotLS0gYS9XZWJzaXRlcy9idWdz LndlYmtpdC5vcmcvQ2hhbmdlTG9nCisrKyBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxNiBAQAorMjAxMi0wOS0wMyAgUGV0ZXIgQmV2ZXJsb28gIDxwZXRl ckBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgQWxsb3cgcm9ib3RzIHRvIGluZGV4IGJ1Z3Mgb24g QnVnemlsbGEKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lk PTk1Njg4CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAg VGhpcyBhbGxvd3Mgcm9ib3RzIHRvIGFjY2VzcyBzaG93X2J1Zy5jZ2ksIHRodXMgZ3JhbnRpbmcg dGhlbSBhY2Nlc3MgdG8KKyAgICAgICAgaW5kZXggY29udGVudHMgYW5kIGRpc2N1c3Npb24gb24g V2ViS2l0IGJ1Z3MuIEFsbCBvdGhlciBwYWdlcywgZXhjZXB0CisgICAgICAgIGZvciB0aGUgc2l0 ZSBpbmRleCBpdHNlbGYsIHN0aWxsIGhhdmUgaW5kZXhpbmcgZGlzYWJsZWQuCisKKyAgICAgICAg KiByb2JvdHMudHh0OgorCiAyMDEyLTA3LTEzICBPamFuIFZhZmFpICA8b2phbkBjaHJvbWl1bS5v cmc+CiAKICAgICAgICAgUHJldHR5UGF0Y2gucmIgY29tcGxhaW5zIGFib3V0IG1pc3NpbmcgY2hl Y2tzdW0gZm9yIG5ldyBwaXhlbCByZXN1bHRzCmRpZmYgLS1naXQgYS9XZWJzaXRlcy9idWdzLndl YmtpdC5vcmcvcm9ib3RzLnR4dCBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9yb2JvdHMudHh0 CmluZGV4IDBmODIzY2IyNDYxMTZkM2JkYTYwZjg2Njc2YzVmM2I4MTk0ZWMwOWMuLjAyMzk4Nzg3 YjBmMDBlNzRjMDJiYzVkMjY4ZjhlZjM3NmJiNjQ5ODQgMTAwNjQ0Ci0tLSBhL1dlYnNpdGVzL2J1 Z3Mud2Via2l0Lm9yZy9yb2JvdHMudHh0CisrKyBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9y b2JvdHMudHh0CkBAIC0xLDMgKzEsNCBAQAogVXNlci1hZ2VudDogKgogQWxsb3c6IC9pbmRleC5j Z2kKK0FsbG93OiAvc2hvd19idWcuY2dpCiBEaXNhbGxvdzogLwo= 162490 2012-09-06 06:19:57 -0700 2012-09-24 08:33:24 -0700 Patch bug-95688-20120906141938.patch text/plain 1252 peter U3VidmVyc2lvbiBSZXZpc2lvbjogMTI3NzMzCmRpZmYgLS1naXQgYS9XZWJzaXRlcy9idWdzLndl YmtpdC5vcmcvQ2hhbmdlTG9nIGIvV2Vic2l0ZXMvYnVncy53ZWJraXQub3JnL0NoYW5nZUxvZwpp bmRleCBhZjYxMWJhOGZkN2JiNWNmZTUzZTllNzY2NDllNzMxNDNjYTcyMTdlLi4wZDk1MDQ5MjZh M2NjODhjMDJhMmJlMmFkMmNmNGRiODBkMDY3MjhjIDEwMDY0NAotLS0gYS9XZWJzaXRlcy9idWdz LndlYmtpdC5vcmcvQ2hhbmdlTG9nCisrKyBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9DaGFu Z2VMb2cKQEAgLTEsMyArMSwxNiBAQAorMjAxMi0wOS0wNiAgUGV0ZXIgQmV2ZXJsb28gIDxwZXRl ckBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgQWxsb3cgcm9ib3RzIHRvIGluZGV4IGJ1Z3Mgb24g QnVnemlsbGEKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lk PTk1Njg4CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAg VGhpcyBhbGxvd3Mgcm9ib3RzIHRvIGFjY2VzcyBzaG93X2J1Zy5jZ2ksIHRodXMgZ3JhbnRpbmcg dGhlbSBhY2Nlc3MgdG8KKyAgICAgICAgaW5kZXggY29udGVudHMgYW5kIGRpc2N1c3Npb24gb24g V2ViS2l0IGJ1Z3MuIEFsbCBvdGhlciBwYWdlcywgZXhjZXB0CisgICAgICAgIGZvciB0aGUgc2l0 ZSBpbmRleCBpdHNlbGYsIHN0aWxsIGhhdmUgaW5kZXhpbmcgZGlzYWJsZWQuCisKKyAgICAgICAg KiByb2JvdHMudHh0OgorCiAyMDEyLTA3LTEzICBPamFuIFZhZmFpICA8b2phbkBjaHJvbWl1bS5v cmc+CiAKICAgICAgICAgUHJldHR5UGF0Y2gucmIgY29tcGxhaW5zIGFib3V0IG1pc3NpbmcgY2hl Y2tzdW0gZm9yIG5ldyBwaXhlbCByZXN1bHRzCmRpZmYgLS1naXQgYS9XZWJzaXRlcy9idWdzLndl YmtpdC5vcmcvcm9ib3RzLnR4dCBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9yb2JvdHMudHh0 CmluZGV4IDBmODIzY2IyNDYxMTZkM2JkYTYwZjg2Njc2YzVmM2I4MTk0ZWMwOWMuLjcyZWI3ZmU4 OTY2OTU4MjdmMzNlMGVhZDk1YWJjNDJmYjQ2YzAxODYgMTAwNjQ0Ci0tLSBhL1dlYnNpdGVzL2J1 Z3Mud2Via2l0Lm9yZy9yb2JvdHMudHh0CisrKyBiL1dlYnNpdGVzL2J1Z3Mud2Via2l0Lm9yZy9y b2JvdHMudHh0CkBAIC0xLDMgKzEsNSBAQAogVXNlci1hZ2VudDogKgogQWxsb3c6IC9pbmRleC5j Z2kKK0FsbG93OiAvc2hvd19idWcuY2dpCiBEaXNhbGxvdzogLworQ3Jhd2wtZGVsYXk6IDIwCg==