Replace Google App Engine with AppScale

[Aakash Jain]

Google App Engine is deprecating the data model we’re currently using(Master-slave datastore : https://cloud.google.com/appengine/docs/deprecations/ms_datastore ).

Currently two of the apps webkit-queues and webkit-test-results(flakiness dashboard) use this data model. We need to switch to AppScale(open source solution) to host these apps.

[Aakash Jain]

Created attachment 257590 [details]
Patch for migrating from google appengine to appscale

[Daniel Bates]

Comment on attachment 257590 [details]
Patch for migrating from google appengine to appscale

View in context: https://bugs.webkit.org/attachment.cgi?id=257590&action=review

> LayoutTests/fast/harness/results.html:636
> +    return 'http://webkit-test-results.webkit.org:8081/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=' + encodeURIComponent(testList);

This is OK for now. Once we transition from appspot.com we should look to remove the need to explicitly specify a port number when accessing webkit-test-results.webkit.org. We will need to apply this change throughout this patch.

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/BubbleQueueServer.js:41
> +    this.baseURL = "http://webkit-queues.webkit.org:8080/";

Notice the queues.webkit.org currently HTTP redirects to <http://webkit-queues.appspot.com>. Can we make use of this existing CNAME instead of defining a new CNAME, webkit-queues?

This is OK for now. Once we transition from appspot.com we should look to remove the need to explicitly specify a port number when accessing webkit-queues.webkit.org. We will need to apply this change throughout this patch.

> Tools/QueueStatusServer/app.yaml:4
> +threadsafe: false

I'm assuming you verified that this setting is acceptable.

> Tools/TestResultServer/app.yaml:4
> +threadsafe: false

Ditto.

> Tools/TestResultServer/static-dashboards/loader.js:34
> -var TEST_RESULTS_SERVER = 'https://webkit-test-results.appspot.com/';
> +var TEST_RESULTS_SERVER = 'http://webkit-test-results.webkit.org:8081/';

Could this cause mixed content warnings?

> Websites/bugs.webkit.org/code-review.js:583
> -      statusBubble.src  = 'https://webkit-queues.appspot.com/status-bubble/' + attachment_id;
> +      statusBubble.src  = 'http://webkit-queues.webkit.org:8080/status-bubble/' + attachment_id;

r-; This will lead to a mixed content warning for each status bubble (*) because we are fetching content from a HTTP URL instead of an HTTPS URL.

(*) We redirect all HTTP traffic on bugs.webkit.org to use HTTPS.

> Websites/bugs.webkit.org/template/en/default/attachment/edit.html.tmpl:267
> -          <iframe src="https://webkit-queues.appspot.com/status-bubble/[% attachment.id %]"
> +          <iframe src="http://webkit-queues.webkit.org:8080/status-bubble/[% attachment.id %]"

This will cause a mixed content warning by a similar argument as for the status bubbles (above).

> Websites/bugs.webkit.org/template/en/default/attachment/list.html.tmpl:160
> -            <iframe src="https://webkit-queues.appspot.com/status-bubble/[% attachment.id %]"
> +            <iframe src="http://webkit-queues.webkit.org:8080/status-bubble/[% attachment.id %]"

Ditto.

> Websites/bugs.webkit.org/template/en/default/attachment/reviewform.html.tmpl:97
> -        <iframe src="https://webkit-queues.appspot.com/status-bubble/[% attachment.id %]"
> +        <iframe src="http://webkit-queues.webkit.org:8080/status-bubble/[% attachment.id %]"

Ditto.

[Aakash Jain]

Created attachment 257669 [details]
Updated Patch, updated https port numbers.

Need to add appropriate SSL certificates to the server before committing.

[Daniel Bates]

Comment on attachment 257669 [details]
Updated Patch, updated https port numbers.

View in context: https://bugs.webkit.org/attachment.cgi?id=257669&action=review

> Tools/QueueStatusServer/app.yaml:4
> +runtime: python27
> +threadsafe: false

It would be good to understand the need to explicitly use the Python 2.7 runtime.

Elaborating on the reasoning for specifying the YAML property threadsafe, we must explicitly define this property when using the Python 2.7 runtime per <https://cloud.google.com/appengine/docs/python/python25/migrate27#appyaml>. We define threadsafe to be false because the status server code was written to be run in AppEngine's CGI environment (by <http://trac.webkit.org/browser/trunk/Tools/QueueStatusServer/main.py?rev=174622#L90> and <https://cloud.google.com/appengine/docs/python/tools/webapp/utilmodule#run_wsgi_app>). And AppEngine's CGI programming model handles requests serially per section "Concurrent Requests and WSGI" of <https://cloud.google.com/appengine/docs/python/python25/migrate27>.

[Aakash Jain]

Created attachment 257772 [details]
Updated port number corresponding to http(8080/8081) in master.cfg and urls.py as they are using http.

SSL certificates are already installed on the server, so we should be good to go.

[Aakash Jain]

Created attachment 257816 [details]
Using default port numbers (80/443), server updated.

Reconfigured the server to listen on default ports (80,443) for both the apps. No need to hard-code the port numbers in the code now.

[Daniel Bates]

Comment on attachment 257816 [details]
Using default port numbers (80/443), server updated.

View in context: https://bugs.webkit.org/attachment.cgi?id=257816&action=review

> Tools/QueueStatusServer/app.yaml:4
> +runtime: python27
> +threadsafe: false

We should add a remark to the appropriate ChangeLog line to describe how we came to these settings. See my remark in comment #4.

> Tools/TestResultServer/app.yaml:4
> +runtime: python27
> +threadsafe: false

Ditto.

[Aakash Jain]

Created attachment 257847 [details]
Updated ChangeLog

[Alexey Proskuryakov]

Committed <http://trac.webkit.org/r187605>.