Bug 8843 – add a way to build Objective-C test cases in HTML

• Component: Tools / Tests
• Status: RESOLVED FIXED

[James G. Speth]

I've been working on a way to write test cases for areas of the Objective-C API that would be easier and more flexible than patching DumpRenderTree.  What I've come up with is a WebKit plugin that allows you to embed python (and PyObjC) scripts into the HTML of the tests, so it gets executed when the test page is loaded.  This provides an extremely flexible way to build tests which run in the browser, but can access the resources necessary to test the obj-c API.

The plugin itself is written in Python using PyObjC.  To build, run "python setup.py py2app" in the source directory.  You have to have Python (2.3.5 is ok) and an up-to-date install of PyObjC.  It will create a "dist" directory and put the plugin (pythonscript.plugin) there.  To install, copy the plugin to ~/Library/Internet Plug-Ins

Use: the plugin should be embedded in the test html file using a line like:

  <embed name="pythonscript" width="1" height="1" type="application/x-pythonscript"></embed>

where the dimensions can be larger for testing things like embedded views.  Elsewhere in the file, you put your python code inside a script tag, like this:

  <script type="application/x-pythonscript">
      class MyWindowDelegate (NSObject):
          def windowWillClose_(self, window):
              NSLog('%s is closing' % `window`)
      delegate = MyWindowDelegate.alloc().init()
  </script>

Indentation of the entire block is relative to the first line of code in the block, so the above example would be valid.  Code inside a script tag like this will be executed in the global namespace (the document namespace), like you were typing it at the interpreter prompt.  So it's a good place for imports and class definitions.

You can also attach python scripts to the event triggers on an element, like this:

  <div __init__="x = 0" __mouseover__="self.setValue_forKey_('test%d' % x, 'innerHTML')" __mouseout__="x += 1">test</div>

The script stored in the __init__ attribute will be executed once, just after the page loads.  The other ones are executed when their events occur.  (The plugin translates "__mouseover__" to "onmouseover" and wraps the value in a javascript call to itself.)  You can use "\n" in the scripts for multiple lines.  Code attached to an element like this executes in its own local namespace associated with the element.  Its globals are set to the document's namespace.  (So, this div's scripts could use the "delegate" variable from the first example, but the delegate wouldn't be able to use the div's "x" variable.)

Requirements: Python 2.3.5 or higher, PyObjC (latest from svn)

Security: this is a huge security problem!  don't use this plugin when browsing or viewing anything other than known and trusted test files.  it can execute arbitrary code with your user's priveledges.

Examples: there are a few examples, which I'll try to document well.

This is very alpha, so be warned, though I don't think it will destroy anything (unless your test file tells it to).  I'd really like to hear feedback about this.  Is it potentially useful?, what needs improving, etc.

[James G. Speth]

Created attachment 8233 [details]
plugin source and examples

[Timothy Hatcher]

THis looks really promising, however requiring PyObjC to be installed makes it hard for us to consider...

[James G. Speth]

Created attachment 8342 [details]
patch to add pythonscript source and pre-built binary to a new WebKitTools subdirectory

[Timothy Hatcher]

Speth mentioned to me that you only need PyObjC installed to build the plugin, any one with Tiger can run the binary version. So his patch includes a binary.

[Darin Adler]

Comment on attachment 8342 [details]
patch to add pythonscript source and pre-built binary to a new WebKitTools subdirectory

Nice work. We really need something like this.

Is the prebuilt binary Universal?

Is a plug-in really the best way to do this? How can we set things up so that the plug-in is not installed during normal browsing, but is there for testing purposes? Maybe we can configure the plug-in to refuse to run in domains other than local files?

descendentNodes has the word descendant misspelled.

I'd like to know more about how this works. If we're going to have a layout test that requires this I want to understand what everyone has to do to use it, how they would re-build and check in a new binary if they changed the test harness etc.

Specifically, I think we need instructions about how to build the plug-in (what to install, how to build, how to check in the newly-built plug-in).

[Darin Adler]

Comment on attachment 8342 [details]
patch to add pythonscript source and pre-built binary to a new WebKitTools subdirectory

I'm going to mark this review+ even though there are some outstanding questions I have. Lets see what it takes to land this.

[Eric Seidel (no email)]

DumpRenderTree will load any plugins it finds in the same directory as its executable when it is run.  TestNetscapePlugin works like this.  ggaren added that functionality.  So there is no need to install the plugin in ~/Library/Internet Plug-Ins

[Geoffrey Garen]

(In reply to comment #7)
> DumpRenderTree will load any plugins it finds in the same directory as its
> executable when it is run.

In fact, we could just make the prebuilt binary a part of the DRT project, and create a DRT build phase that copied it to the XCode SYMROOT directory. That would obviate the need for doing anything special to get the Objc tests to run.

[James G. Speth]

> In fact, we could just make the prebuilt binary a part of the DRT project, and
> create a DRT build phase that copied it to the XCode SYMROOT directory. That
> would obviate the need for doing anything special to get the Objc tests to run.

I was just working on a patch to do just that.  I'll post it soon.  It adds the plugin to the DRT project, and adds a copy files phase to the All target that copies the pre-built binary.  There is a separate target for building new versions of the plugin, which is not one of All's dependencies.  I'm also throwing in a script to get, patch, and build a universal PyObjC for Tiger's python (purely optional for normal use).

[James G. Speth]

Created attachment 8380 [details]
patch adding pythonscript plugin to DumpRenderTree project

this patch has an empty subdirectory called "prebuilt" which should contain the prebuilt binary, which is zipped and in the next attachment.

[James G. Speth]

Created attachment 8381 [details]
prebuilt universal binary of plugin - put in "prebuilt" directory

can someone with an intel machine just try this out and make sure it works.  i have no way to test it.

[James G. Speth]

more improvements are on their way, don't land it just yet.  i've reworked the script a bit to make building tests much simpler.  it now uses the python "doctest" style tests, which are super easy (they look like you copied and pasted from the interpreter session).  the plugin now let's you mix code and doctest strings freely in the html.  also, the test runner produces nice clean output in text mode under DumpRenderTree now.  patch is on its way.

[James G. Speth]

Created attachment 8415 [details]
ok, this is it.  patch for review.

the way that tests were written in the earlier version was clumsy.  this version incorporates the python doctest idioms for writing tests in a natural, low-overhead manner. also, this version has output that looks decent when dumped in text mode by DumpRenderTree.  so go ahead and land this, and i'll post a new binary to obsolete the one that's here.

[James G. Speth]

Created attachment 8416 [details]
prebuilt universal binary of plugin

so, just ignore #8381 and use this one, unzip it in the "prebuilt" directory that the patch creates.

[James G. Speth]

Created attachment 9002 [details]
patch with a different solution to this problem

this patch provides a simpler way of solving this problem, and i believe should be used instead of my earlier patches.  this simply adds a method to the objcPlugin in DumpRenderTree that, when invoked, allows javascript tests to have unrestricted access to the objective-c API.

i'll put up a patch next with some actual test cases that show this in action.

[Darin Adler]

Comment on attachment 9002 [details]
patch with a different solution to this problem

Looks good. Lets give this a try.

[Darin Adler]

I wonder what kinds of tests can be done with this and what kinds won't work with it. Are there some types that we can't manipulate in JavaScript?

[James G. Speth]

Created attachment 9004 [details]
example tests

i think these do a good job of illustrating how this tool would be used.

[James G. Speth]

(In reply to comment #17)
> I wonder what kinds of tests can be done with this and what kinds won't work
> with it. Are there some types that we can't manipulate in JavaScript?

I've used it to make a page that is its own UI and frame load delegate. :)  There are limitations though, like various types not bridged across, but I think as we use this to build tests, it might also push the development of the bridge and make it even cooler.

[David Kilzer (:ddkilzer)]

Committed revision r15020.

[David Kilzer (:ddkilzer)]

Reopening for test case review.

[James G. Speth]

those tests probably don't really both belong in plugins, and should contain a description of what they're testing, and should probably be renamed.  the "simple" one is really testing the plugin, so should be in plugins, but the other one is testing bug 8331 and should probably go in fast/dom.  does that seem right?

[David Kilzer (:ddkilzer)]

(In reply to comment #21)
> Reopening for test case review. 

Example tests (Attachment 9004 [details]) committed as r15022.

[David Kilzer (:ddkilzer)]

(In reply to comment #22)
> those tests probably don't really both belong in plugins, and should contain a
> description of what they're testing, and should probably be renamed.  the
> "simple" one is really testing the plugin, so should be in plugins, but the
> other one is testing bug 8331 and should probably go in fast/dom.  does that
> seem right? 

Please file another bug with the fix.  Thanks!