28207 2009-08-11 18:33:24 -0700 Another deadlock in the DB code 2009-08-18 14:58:50 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) All All RESOLVED FIXED P2 Normal --- 0 dumi webkit-unassigned andersca ap aroben beidson dglazkov eric michaeln oldest_to_newest 139203 0 dumi 2009-08-11 18:33:24 -0700 Bug 27966 describes how 2 transactions run on 2 different handles to the same DB can deadlock the DB thread. Here's an example of how a transaction combined with a openDatabase() call can deadlock the DB thread and block the main thread at the same time: 1: var db1 = openDatabase("SomeDB", "1.0", "Blah", 10000); 2: db1.transaction(function(tx) { 3: tx.executeSql("SomeWriteStatement", [args], 4: function(result) { var db2 = openDatabase("SomeDB", "1.0", "Blah", 10000); }, 5: function(tx, error) { }); 6: }, 7: function(error) { }, 8: function() { } 9: ); Description: 1. The user opens a handle to a database (line #1). 2. The user starts running a transaction on that handle (line #2). The transaction must have a "write" statement, that will make SQLite acquire an exclusive lock on the DB file (line #3). 3. Eventually, we get to running the statement callback (line #4). 4. openDatabase() is called on the main thread (line #4). 5. The main thread creates a new SQLite connection to the same DB file, schedules a task on the DB thread (to make sure the version is correct), and blocks waiting for this task to complete (openDatabase() is a sync call according to the spec). 6. In order to check the version, the new task needs to acquire a "read" lock on the DB file first and check if the __WebKitDatabaseInfoTable__ table exists. 7. This "read" lock cannot be acquired because the transaction has an exclusive lock on the DB file. At the same time, the transaction cannot release the exclusive lock until the statement callback is executed. So the DB thread is deadlocked, and the main thread is blocked too, because it waits for the openDatabase() call to finish. Safari's debug build from trunk deadlocks for 30 seconds (SQLite's busy_timeout) and then crashes. PROPOSED FIX: It's OK for the first call to openDatabase(SomeDB, ...) to execute SQL statements, because we know there are no in-flight transactions on other DB handles to the same database. However, the 2nd, 3rd, ... calls to openDatabase(SomeDB, ...) cannot run any SQL statements without risking a deadlock. So in addition to caching the DB version (this is what WebCore does now), we need to also cache a flag that tells us if the __WebKitDatabaseInfoTable__ exists in the database. 140030 1 michaeln 2009-08-13 14:14:04 -0700 Good bug. Agreed that the root cause of this problem is the synchronous window.openDatabase method that must read the 'version' attribute value, and that caching that value and using the cached value at open time should help. I don't follow how caching whether the __WebKitDatabaseInfoTable__ exists solves the problem. (i see what your driving at now, because it gets created prior to checking for the cached version, we could just move the table creation till a little further along in all cases). Seems like this could be fixed within performOpenAndVerify(). If a cached version value exists, don't bother touching the dbfile in anyway beyond opening the sqlite3 handle, simply trust the cached value. Otherwise its the first opening of this DB in this session. (Create the special table if needed and) read the version value from the DB and populate the cache with that value. Provided this happens behind the guidMutex, i think there won't be a deadlock. 140938 2 35004 dumi 2009-08-17 17:53:07 -0700 Created attachment 35004 patch + layout test 141110 3 35004 dglazkov 2009-08-18 10:01:30 -0700 Comment on attachment 35004 patch + layout test r=me. Good work! 141218 4 35004 eric 2009-08-18 14:11:51 -0700 Comment on attachment 35004 patch + layout test Rejecting patch 35004 from commit-queue. This patch will require manual commit. Failed to run "['git', 'svn', 'rebase']" exit_code: 1 cwd: None 141236 5 35004 eric 2009-08-18 14:25:56 -0700 Comment on attachment 35004 patch + layout test Sorry, we hit https://bugs.webkit.org/show_bug.cgi?id=28436 141263 6 35004 eric 2009-08-18 14:58:42 -0700 Comment on attachment 35004 patch + layout test Clearing flags on attachment: 35004 Committed r47458: <http://trac.webkit.org/changeset/47458> 141264 7 eric 2009-08-18 14:58:50 -0700 All reviewed patches have been landed. Closing bug. 35004 2009-08-17 17:53:07 -0700 2009-08-18 14:58:41 -0700 patch + layout test patch text/plain 7109 dumi SW5kZXg6IFdlYkNvcmUvQ2hhbmdlTG9nCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFdlYkNvcmUvQ2hhbmdlTG9n CShyZXZpc2lvbiA0NzQwMCkKKysrIFdlYkNvcmUvQ2hhbmdlTG9nCSh3b3JraW5nIGNvcHkpCkBA IC0xLDMgKzEsMTkgQEAKKzIwMDktMDgtMTcgIER1bWl0cnUgRGFuaWxpdWMgIDxkdW1pQGNocm9t aXVtLm9yZz4KKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAg ICBvcGVuRGF0YWJhc2UoKSBzaG91bGQgbm90IGV4ZWN1dGUgYW55IFNRTCBzdGF0ZW1lbnQgb24g dGhlCisgICAgICAgIGRhdGFiYXNlIGl0J3MgdHJ5aW5nIHRvIG9wZW4sIHVubGVzcyB0aGVyZSBh cmUgbm8gb3RoZXIgb3BlbgorICAgICAgICBoYW5kbGVzIHRvIHRoZSBzYW1lIGRhdGFiYXNlLiBP dGhlcndpc2UsIHdlIG1pZ2h0IHJ1biBpbnRvIGEKKyAgICAgICAgZGVhZGxvY2suCisKKyAgICAg ICAgVGVzdDogc3RvcmFnZS9vcGVuLWRhdGFiYXNlLXdoaWxlLXRyYW5zYWN0aW9uLWluLXByb2dy ZXNzLmh0bWwKKworICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/ aWQ9MjgyMDcKKworICAgICAgICAqIHN0b3JhZ2UvRGF0YWJhc2UuY3BwOgorICAgICAgICAoV2Vi Q29yZTo6RGF0YWJhc2U6OnBlcmZvcm1PcGVuQW5kVmVyaWZ5KToKKwogMjAwOS0wOC0xNyAgU2hp bmljaGlybyBIYW1hamkgIDxoYW1hamlAY2hyb21pdW0ub3JnPgogCiAgICAgICAgIFJldmlld2Vk IGJ5IERhcmluIEFkbGVyLgpJbmRleDogV2ViQ29yZS9zdG9yYWdlL0RhdGFiYXNlLmNwcAo9PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09Ci0tLSBXZWJDb3JlL3N0b3JhZ2UvRGF0YWJhc2UuY3BwCShyZXZpc2lvbiA0NzM5NSkK KysrIFdlYkNvcmUvc3RvcmFnZS9EYXRhYmFzZS5jcHAJKHdvcmtpbmcgY29weSkKQEAgLTQ0Miwx NCArNDQyLDYgQEAgYm9vbCBEYXRhYmFzZTo6cGVyZm9ybU9wZW5BbmRWZXJpZnkoRXhjZQogICAg IG1fc3FsaXRlRGF0YWJhc2Uuc2V0QXV0aG9yaXplcihtX2RhdGFiYXNlQXV0aG9yaXplcik7CiAg ICAgbV9zcWxpdGVEYXRhYmFzZS5zZXRCdXN5VGltZW91dChtYXhTcWxpdGVCdXN5V2FpdFRpbWUp OwogCi0gICAgaWYgKCFtX3NxbGl0ZURhdGFiYXNlLnRhYmxlRXhpc3RzKGRhdGFiYXNlSW5mb1Rh YmxlTmFtZSgpKSkgewotICAgICAgICBpZiAoIW1fc3FsaXRlRGF0YWJhc2UuZXhlY3V0ZUNvbW1h bmQoIkNSRUFURSBUQUJMRSAiICsgZGF0YWJhc2VJbmZvVGFibGVOYW1lKCkgKyAiIChrZXkgVEVY VCBOT1QgTlVMTCBPTiBDT05GTElDVCBGQUlMIFVOSVFVRSBPTiBDT05GTElDVCBSRVBMQUNFLHZh bHVlIFRFWFQgTk9UIE5VTEwgT04gQ09ORkxJQ1QgRkFJTCk7IikpIHsKLSAgICAgICAgICAgIExP R19FUlJPUigiVW5hYmxlIHRvIGNyZWF0ZSB0YWJsZSAlcyBpbiBkYXRhYmFzZSAlcyIsIGRhdGFi YXNlSW5mb1RhYmxlTmFtZSgpLmFzY2lpKCkuZGF0YSgpLCBkYXRhYmFzZURlYnVnTmFtZSgpLmFz Y2lpKCkuZGF0YSgpKTsKLSAgICAgICAgICAgIGUgPSBJTlZBTElEX1NUQVRFX0VSUjsKLSAgICAg ICAgICAgIHJldHVybiBmYWxzZTsKLSAgICAgICAgfQotICAgIH0KLQogICAgIFN0cmluZyBjdXJy ZW50VmVyc2lvbjsKICAgICB7CiAgICAgICAgIE11dGV4TG9ja2VyIGxvY2tlcihndWlkTXV0ZXgo KSk7CkBAIC00NjcsNiArNDU5LDE1IEBAIGJvb2wgRGF0YWJhc2U6OnBlcmZvcm1PcGVuQW5kVmVy aWZ5KEV4Y2UKICAgICAgICAgICAgIExPRyhTdG9yYWdlQVBJLCAiQ3VycmVudCBjYWNoZWQgdmVy c2lvbiBmb3IgZ3VpZCAlaSBpcyAlcyIsIG1fZ3VpZCwgY3VycmVudFZlcnNpb24uYXNjaWkoKS5k YXRhKCkpOwogICAgICAgICB9IGVsc2UgewogICAgICAgICAgICAgTE9HKFN0b3JhZ2VBUEksICJO byBjYWNoZWQgdmVyc2lvbiBmb3IgZ3VpZCAlaSIsIG1fZ3VpZCk7CisKKyAgICAgICAgICAgIGlm ICghbV9zcWxpdGVEYXRhYmFzZS50YWJsZUV4aXN0cyhkYXRhYmFzZUluZm9UYWJsZU5hbWUoKSkp IHsKKyAgICAgICAgICAgICAgICBpZiAoIW1fc3FsaXRlRGF0YWJhc2UuZXhlY3V0ZUNvbW1hbmQo IkNSRUFURSBUQUJMRSAiICsgZGF0YWJhc2VJbmZvVGFibGVOYW1lKCkgKyAiIChrZXkgVEVYVCBO T1QgTlVMTCBPTiBDT05GTElDVCBGQUlMIFVOSVFVRSBPTiBDT05GTElDVCBSRVBMQUNFLHZhbHVl IFRFWFQgTk9UIE5VTEwgT04gQ09ORkxJQ1QgRkFJTCk7IikpIHsKKyAgICAgICAgICAgICAgICAg ICAgTE9HX0VSUk9SKCJVbmFibGUgdG8gY3JlYXRlIHRhYmxlICVzIGluIGRhdGFiYXNlICVzIiwg ZGF0YWJhc2VJbmZvVGFibGVOYW1lKCkuYXNjaWkoKS5kYXRhKCksIGRhdGFiYXNlRGVidWdOYW1l KCkuYXNjaWkoKS5kYXRhKCkpOworICAgICAgICAgICAgICAgICAgICBlID0gSU5WQUxJRF9TVEFU RV9FUlI7CisgICAgICAgICAgICAgICAgICAgIHJldHVybiBmYWxzZTsKKyAgICAgICAgICAgICAg ICB9CisgICAgICAgICAgICB9CisKICAgICAgICAgICAgIGlmICghZ2V0VmVyc2lvbkZyb21EYXRh YmFzZShjdXJyZW50VmVyc2lvbikpIHsKICAgICAgICAgICAgICAgICBMT0dfRVJST1IoIkZhaWxl ZCB0byBnZXQgY3VycmVudCB2ZXJzaW9uIGZyb20gZGF0YWJhc2UgJXMiLCBkYXRhYmFzZURlYnVn TmFtZSgpLmFzY2lpKCkuZGF0YSgpKTsKICAgICAgICAgICAgICAgICBlID0gSU5WQUxJRF9TVEFU RV9FUlI7CkluZGV4OiBMYXlvdXRUZXN0cy9DaGFuZ2VMb2cKPT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gTGF5b3V0 VGVzdHMvQ2hhbmdlTG9nCShyZXZpc2lvbiA0NzQwMCkKKysrIExheW91dFRlc3RzL0NoYW5nZUxv Zwkod29ya2luZyBjb3B5KQpAQCAtMSwzICsxLDE2IEBACisyMDA5LTA4LTE3ICBEdW1pdHJ1IERh bmlsaXVjICA8ZHVtaUBjaHJvbWl1bS5vcmc+CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZ IChPT1BTISkuCisKKyAgICAgICAgQWRkaW5nIGEgdGVzdCB0byB2ZXJpZnkgdGhhdCBvcGVuRGF0 YWJhc2UoKSBkb2VzIG5vdCBleGVjdXRlCisgICAgICAgIFNRTCBzdGF0ZW1lbnRzIG9uIHRoZSBk YXRhYmFzZSBpdCdzIHRyeWluZyB0byBvcGVuLCB1bmxlc3Mgd2UncmUKKyAgICAgICAgb3Blbmlu ZyB0aGUgZmlyc3QgaGFuZGxlIHRvIHRoYXQgZGF0YWJhc2UuCisKKyAgICAgICAgaHR0cHM6Ly9i dWdzLndlYmtpdC5vcmcvc2hvd19idWcuY2dpP2lkPTI4MjA3CisKKyAgICAgICAgKiBzdG9yYWdl L29wZW4tZGF0YWJhc2Utd2hpbGUtdHJhbnNhY3Rpb24taW4tcHJvZ3Jlc3MtZXhwZWN0ZWQudHh0 OiBBZGRlZC4KKyAgICAgICAgKiBzdG9yYWdlL29wZW4tZGF0YWJhc2Utd2hpbGUtdHJhbnNhY3Rp b24taW4tcHJvZ3Jlc3MuaHRtbDogQWRkZWQuCisKIDIwMDktMDgtMTcgIFNoaW5pY2hpcm8gSGFt YWppICA8aGFtYWppQGNocm9taXVtLm9yZz4KIAogICAgICAgICBSZXZpZXdlZCBieSBEYXJpbiBB ZGxlci4KSW5kZXg6IExheW91dFRlc3RzL3N0b3JhZ2Uvb3Blbi1kYXRhYmFzZS13aGlsZS10cmFu c2FjdGlvbi1pbi1wcm9ncmVzcy1leHBlY3RlZC50eHQKPT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gTGF5b3V0VGVz dHMvc3RvcmFnZS9vcGVuLWRhdGFiYXNlLXdoaWxlLXRyYW5zYWN0aW9uLWluLXByb2dyZXNzLWV4 cGVjdGVkLnR4dAkocmV2aXNpb24gMCkKKysrIExheW91dFRlc3RzL3N0b3JhZ2Uvb3Blbi1kYXRh YmFzZS13aGlsZS10cmFuc2FjdGlvbi1pbi1wcm9ncmVzcy1leHBlY3RlZC50eHQJKHJldmlzaW9u IDApCkBAIC0wLDAgKzEsMyBAQAorVGhpcyBpcyBhIHRlc3QgdG8gc2VlIGlmIG9wZW5pbmcgYSBk YXRhYmFzZSB3aGlsZSBhIHRyYW5zYWN0aW9uIGlzIHJ1bm5pbmcgb24gYSBkaWZmZXJlbnQgaGFu ZGxlIHRvIHRoZSBzYW1lIGRhdGFiYXNlIHJlc3VsdHMgaW4gYSBkZWFkbG9jay4KK29wZW5EYXRh YmFzZSgpIHN1Y2NlZWRlZC4KKwpJbmRleDogTGF5b3V0VGVzdHMvc3RvcmFnZS9vcGVuLWRhdGFi YXNlLXdoaWxlLXRyYW5zYWN0aW9uLWluLXByb2dyZXNzLmh0bWwKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gTGF5 b3V0VGVzdHMvc3RvcmFnZS9vcGVuLWRhdGFiYXNlLXdoaWxlLXRyYW5zYWN0aW9uLWluLXByb2dy ZXNzLmh0bWwJKHJldmlzaW9uIDApCisrKyBMYXlvdXRUZXN0cy9zdG9yYWdlL29wZW4tZGF0YWJh c2Utd2hpbGUtdHJhbnNhY3Rpb24taW4tcHJvZ3Jlc3MuaHRtbAkocmV2aXNpb24gMCkKQEAgLTAs MCArMSw3MSBAQAorPGh0bWw+Cis8aGVhZD4KKzxzY3JpcHQ+CisKK2Z1bmN0aW9uIGxvZyhtZXNz YWdlKQoreworICAgIGRvY3VtZW50LmJvZHkuaW5uZXJIVE1MICs9IG1lc3NhZ2UgKyAiPGJyPiI7 Cit9CisKK3ZhciBjb21wbGV0ZSA9IDA7CisKK2Z1bmN0aW9uIGNoZWNrQ29tcGxldGlvbigpCit7 CisgICAgLy8gVGhlIHRlc3Qgc2hvdWxkIGVuZCBhZnRlciB0d28gdHJhbnNhY3Rpb25zCisgICAg aWYgKCsrY29tcGxldGUgPT0gMSAmJiB3aW5kb3cubGF5b3V0VGVzdENvbnRyb2xsZXIpCisgICAg ICAgIGxheW91dFRlc3RDb250cm9sbGVyLm5vdGlmeURvbmUoKTsKK30KKworLy8gT3BlbnMgdGhl IGRhdGFiYXNlIHVzZWQgaW4gdGhpcyB0ZXN0IGNhc2UKK2Z1bmN0aW9uIG9wZW5UZXN0RGF0YWJh c2UoKQoreworICAgIHJldHVybiBvcGVuRGF0YWJhc2UoIk9wZW5EYXRhYmFzZVdoaWxlVHJhbnNh Y3Rpb25JblByb2dyZXNzVGVzdCIsCisgICAgICAgICAgICAgICAgICAgICAgICAiMS4wIiwKKyAg ICAgICAgICAgICAgICAgICAgICAgICJUZXN0IHRvIG1ha2Ugc3VyZSB0aGF0IGNhbGxpbmcgb3Bl bkRhdGFiYXNlKCkgd2hpbGUgYSB0cmFuc2FjdGlvbiBpcyBpbiBwcm9ncmVzcyBvbiBhIGRpZmZl cmVudCBoYW5kbGUgdG8gdGhlIHNhbWUgZGF0YWJhc2UgZG9lcyBub3QgcmVzdWx0IGluIGEgZGVh ZGxvY2suIiwKKyAgICAgICAgICAgICAgICAgICAgICAgIDIxMDAwMDApOyAvLyAyTUIgKyBlcHNp bG9uCit9CisKKy8vIFNlZSBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9 MjgyMDcKKy8vIEluIG9yZGVyIHRvIHRyaWdnZXIgdGhpcyBidWcsIHRoZSB0cmFuc2FjdGlvbiBt dXN0IGFjcXVpcmUgYW4gZXhjbHVzaXZlCisvLyBsb2NrIG9uIHRoZSBEQiBmaWxlIGJlZm9yZSB0 cnlpbmcgdG8gb2J0YWluIGEgc2Vjb25kIGhhbmRsZSB0byB0aGUgc2FtZSBEQi4KKy8vIFRoZSBv bmx5IHdheSB0byBmb3JjZSBTUUxpdGUgdG8gb2J0YWluIGFuIGV4Y2x1c2l2ZSBsb2NrIGlzIHRv IGNoYW5nZSBtb3JlCisvLyB0aGFuIGNhY2hlX3NpemUgKiBwYWdlX3NpemUgYnl0ZXMgaW4gdGhl IGRhdGFiYXNlLiBUaGUgZGVmYXVsdCB2YWx1ZSBmb3IKKy8vIGNhY2hlX3NpemUgaXMgMjAwMCBw YWdlcywgYW5kIHRoZSBkZWZhdWx0IHBhZ2Vfc2l6ZSBpcyAxMDI0IGJ5dGVzLiBTbyB0aGUKKy8v IHNpemUgb2YgdGhlIGJsb2IgbXVzdCBiZSBhdCBsZWFzdCAyTUIuCitmdW5jdGlvbiBydW5UZXN0 KCkKK3sKKyAgICBpZiAod2luZG93LmxheW91dFRlc3RDb250cm9sbGVyKSB7CisgICAgICAgIGxh eW91dFRlc3RDb250cm9sbGVyLmR1bXBBc1RleHQoKTsKKyAgICAgICAgbGF5b3V0VGVzdENvbnRy b2xsZXIud2FpdFVudGlsRG9uZSgpOworICAgIH0KKworICAgIHRyeSB7CisgICAgICAgIHZhciBk YjEgPSBvcGVuVGVzdERhdGFiYXNlKCk7CisgICAgICAgIGRiMS50cmFuc2FjdGlvbihmdW5jdGlv bih0eCkgeworICAgICAgICAgICAgLy8gQ3JlYXRlIHRoZSBUZXN0IHRhYmxlIGlmIGl0IGRvZXMg bm90IGV4aXN0CisgICAgICAgICAgICB0eC5leGVjdXRlU3FsKCJDUkVBVEUgVEFCTEUgSUYgTk9U IEVYSVNUUyBUZXN0IChGb28gQkxPQik7Iik7CisgICAgICAgICAgICB0eC5leGVjdXRlU3FsKCJJ TlNFUlQgSU5UTyBUZXN0IFZBTFVFUyAoUkFORE9NQkxPQigyMDk3MTUyKSk7IiwgW10sCisgICAg ICAgICAgICAgICAgICAgICAgICAgIGZ1bmN0aW9uKHJlc3VsdCkgeworICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgdmFyIGRiMiA9IG9wZW5UZXN0RGF0YWJhc2UoKTsKKyAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIGxvZygib3BlbkRhdGFiYXNlKCkgc3VjY2VlZGVkLiIpOworICAg ICAgICAgICAgICAgICAgICAgICAgICB9LAorICAgICAgICAgICAgICAgICAgICAgICAgICBmdW5j dGlvbih0eCwgZXJyb3IpIHsKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxvZygiRXhl Y3V0aW5nIHN0YXRlbWVudCBmYWlsZWQ6ICIgKyBlcnJvci5tZXNzYWdlKTsKKyAgICAgICAgICAg ICAgICAgICAgICAgICAgfSk7CisKKyAgICAgICAgICAgIC8vIENsZWFuIHVwIHRoZSBEQiB0byBh bGxvdyBmb3IgcmVwZWF0ZWQgcnVucyBvZiB0aGlzIHRlc3QKKyAgICAgICAgICAgIC8vIHdpdGhv dXQgbmVlZGluZyB0byBpbmNyZWFzZSB0aGUgZGVmYXVsdCBhbGxvd2VkIHF1b3RhICg1TUIpCisg ICAgICAgICAgICB0eC5leGVjdXRlU3FsKCJERUxFVEUgRlJPTSBUZXN0OyIpOworICAgICAgICB9 LCBmdW5jdGlvbihlcnJvcikgeworICAgICAgICAgICAgbG9nKCJUcmFuc2FjdGlvbiBmYWlsZWQ6 ICIgKyBlcnJvci5tZXNzYWdlKTsKKyAgICAgICAgfSwgZnVuY3Rpb24oKSB7CisgICAgICAgICAg ICBjaGVja0NvbXBsZXRpb24oKTsKKyAgICAgICAgfSk7CisgICAgfSBjYXRjaChlcnIpIHt9Cit9 Cis8L3NjcmlwdD4KKzxoZWFkPgorPGJvZHkgb25sb2FkPSJydW5UZXN0KCk7Ij4KK1RoaXMgaXMg YSB0ZXN0IHRvIHNlZSBpZiBvcGVuaW5nIGEgZGF0YWJhc2Ugd2hpbGUgYSB0cmFuc2FjdGlvbiBp cyBydW5uaW5nIG9uIGEgZGlmZmVyZW50IGhhbmRsZSB0byB0aGUgc2FtZSBkYXRhYmFzZSByZXN1 bHRzIGluIGEgZGVhZGxvY2suPGJyPgorPC9ib2R5PgorPC9odG1sPgo=