Bug 138821 - Make sure range based iteration of Vector<> still receives bounds checking
Summary: Make sure range based iteration of Vector<> still receives bounds checking
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Web Template Framework (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Oliver Hunt
URL:
Keywords:
Depends on: 139658
Blocks:
  Show dependency treegraph
 
Reported: 2014-11-17 21:04 PST by Oliver Hunt
Modified: 2015-09-10 09:17 PDT (History)
11 users (show)

See Also:


Attachments
Patch (65.21 KB, patch)
2014-11-17 21:18 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (65.50 KB, patch)
2014-11-19 08:52 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (65.59 KB, patch)
2014-11-19 10:51 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (66.41 KB, patch)
2014-11-19 16:27 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
trying to fix efl, gtk, and msvc (67.42 KB, patch)
2014-11-20 09:56 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
more efl fixes (68.41 KB, patch)
2014-11-20 10:32 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (68.25 KB, patch)
2014-11-20 10:43 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
sigh, another attempt to fix GIFDecoder (68.23 KB, patch)
2014-11-20 15:59 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (69.32 KB, patch)
2014-11-21 15:16 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (69.69 KB, patch)
2014-11-21 15:38 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
sigh, webkit-patch was waiting for me to say yes before uploading... (69.79 KB, patch)
2014-11-29 11:08 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (63.34 KB, patch)
2014-12-03 11:13 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (63.32 KB, patch)
2014-12-03 11:58 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (63.19 KB, patch)
2014-12-13 13:55 PST, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (59.04 KB, patch)
2015-08-02 15:18 PDT, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (61.96 KB, patch)
2015-09-02 14:43 PDT, Oliver Hunt
no flags Details | Formatted Diff | Diff
Patch (61.85 KB, patch)
2015-09-02 16:21 PDT, Oliver Hunt
darin: review-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Oliver Hunt 2014-11-17 21:04:32 PST
Make sure range based iteration of Vector<> still receives bounds checking
Comment 1 Oliver Hunt 2014-11-17 21:18:42 PST
Created attachment 241762 [details]
Patch
Comment 2 WebKit Commit Bot 2014-11-17 21:20:58 PST
Attachment 241762 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 27 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 3 Oliver Hunt 2014-11-19 08:52:50 PST
Created attachment 241862 [details]
Patch
Comment 4 Oliver Hunt 2014-11-19 10:51:23 PST
Created attachment 241869 [details]
Patch
Comment 5 WebKit Commit Bot 2014-11-19 10:54:03 PST
Attachment 241869 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 27 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 6 Chris Dumez 2014-11-19 14:46:42 PST
Comment on attachment 241869 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=241869&action=review

> Source/WTF/wtf/IndexedIterator.h:32
> +    if (!(check))\

Doesn't this impact performance?
Comment 7 Oliver Hunt 2014-11-19 14:49:14 PST
Comment on attachment 241869 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=241869&action=review

>> Source/WTF/wtf/IndexedIterator.h:32
>> +    if (!(check))\
> 
> Doesn't this impact performance?

This has already passed all the performance tests (JSC, PLT, Speeddoohicky, ...)
Comment 8 Oliver Hunt 2014-11-19 16:27:28 PST
Created attachment 241906 [details]
Patch
Comment 9 WebKit Commit Bot 2014-11-19 16:28:51 PST
Attachment 241906 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 28 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 10 Oliver Hunt 2014-11-20 09:56:02 PST
Created attachment 241954 [details]
trying to fix efl, gtk, and msvc
Comment 11 WebKit Commit Bot 2014-11-20 09:57:22 PST
Attachment 241954 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 29 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 12 Oliver Hunt 2014-11-20 10:32:17 PST
Created attachment 241958 [details]
more efl fixes
Comment 13 WebKit Commit Bot 2014-11-20 10:35:45 PST
Attachment 241958 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 29 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 14 Oliver Hunt 2014-11-20 10:43:58 PST
Created attachment 241959 [details]
Patch
Comment 15 WebKit Commit Bot 2014-11-20 10:45:14 PST
Attachment 241959 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 29 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 16 Oliver Hunt 2014-11-20 11:55:52 PST
Performance is good, etc, etc
Comment 17 Oliver Hunt 2014-11-20 15:59:38 PST
Created attachment 241999 [details]
sigh, another attempt to fix GIFDecoder
Comment 18 WebKit Commit Bot 2014-11-20 16:02:02 PST
Attachment 241999 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 29 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 19 Mark Lam 2014-11-21 13:39:00 PST
Comment on attachment 241999 [details]
sigh, another attempt to fix GIFDecoder

View in context: https://bugs.webkit.org/attachment.cgi?id=241999&action=review

> Source/JavaScriptCore/profiler/ProfileNode.cpp:157
> +

Please remove.

> Source/WTF/WTF.xcodeproj/project.pbxproj:101
> +		A7DC2F041A09A22D0072F4E3 /* IndexedIterator.h in Headers */ = {isa = PBXBuildFile; fileRef = A7DC2F031A099DE30072F4E3 /* IndexedIterator.h */; };

You’ll need to add the new header to WTF.vcxproj and WTF.vcxproj.filters as well.

> Source/WTF/wtf/IndexedIterator.h:33
> +    if (!(check))\
> +        OverflowHandler::overflowed();\

nit: indentation is funny here.  Can you indent these 2 lines 1 more level so that it is not at the same level as the while(0) below?

> Source/WTF/wtf/IndexedIterator.h:111
> +        if (increment < 0) {
> +            INDEXED_ITERATOR_OVERFLOW(increment != std::numeric_limits<int32_t>::max());

if increment < 0, then it can never == std::numeric_limits<int32_t>::max().  Hence, this check will always pass.  The goal is so that when we do the negation below, we don’t overflow, right?  So, instead, I think you actually meant to check:
    INDEXED_ITERATOR_OVERFLOW(increment > std::numeric_limits<int32_t>::min());

> Source/WTF/wtf/IndexedIterator.h:123
> +            INDEXED_ITERATOR_OVERFLOW(decrement != std::numeric_limits<int32_t>::max());

Ditto.  I think you actually meant to check:
    INDEXED_ITERATOR_OVERFLOW(decrement > std::numeric_limits<int32_t>::min());

> Source/WTF/wtf/IndexedIterator.h:150
> +            INDEXED_ITERATOR_OVERFLOW(increment != std::numeric_limits<int32_t>::max());

Ditto.  Should be:
    INDEXED_ITERATOR_OVERFLOW(increment > std::numeric_limits<int32_t>::min());

> Source/WTF/wtf/IndexedIterator.h:162
> +            INDEXED_ITERATOR_OVERFLOW(decrement != std::numeric_limits<int32_t>::max());

Ditto.  Should be:
    INDEXED_ITERATOR_OVERFLOW(decrement > std::numeric_limits<int32_t>::min());

> Source/WTF/wtf/IndexedIterator.h:209
> +        return result += static_cast<size_t>(increment);

You’re casting a signed long long increment into an unsigned size_t.  Is this correct and safe?  I’m thinking of a 32-bit platform where size_t is smaller than long long.

> Source/WTF/wtf/IndexedIterator.h:215
> +        return result += static_cast<size_t>(decrement);

Ditto.  Signed long long to potentially smaller size_t conversion.

> Source/WTF/wtf/IndexedIterator.h:235
> +        return result += static_cast<size_t>(increment);

Here, we’re casting a signed long into an unsigned size_t.  Is this correct and safe?

> Source/WTF/wtf/IndexedIterator.h:241
> +        return result += static_cast<size_t>(decrement);

Ditto.  Signed long to unsigned size_t conversion.
Comment 20 Oliver Hunt 2014-11-21 14:26:37 PST
Comment on attachment 241999 [details]
sigh, another attempt to fix GIFDecoder

View in context: https://bugs.webkit.org/attachment.cgi?id=241999&action=review

>> Source/WTF/wtf/IndexedIterator.h:33
>> +        OverflowHandler::overflowed();\
> 
> nit: indentation is funny here.  Can you indent these 2 lines 1 more level so that it is not at the same level as the while(0) below?

fighting Xcode auto indent, fixed

>> Source/WTF/wtf/IndexedIterator.h:111
>> +            INDEXED_ITERATOR_OVERFLOW(increment != std::numeric_limits<int32_t>::max());
> 
> if increment < 0, then it can never == std::numeric_limits<int32_t>::max().  Hence, this check will always pass.  The goal is so that when we do the negation below, we don’t overflow, right?  So, instead, I think you actually meant to check:
>     INDEXED_ITERATOR_OVERFLOW(increment > std::numeric_limits<int32_t>::min());

Aieeeee, good catch.
Comment 21 Oliver Hunt 2014-11-21 15:16:21 PST
Created attachment 242077 [details]
Patch
Comment 22 WebKit Commit Bot 2014-11-21 15:19:22 PST
Attachment 242077 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 23 Oliver Hunt 2014-11-21 15:38:23 PST
Created attachment 242087 [details]
Patch
Comment 24 WebKit Commit Bot 2014-11-21 15:40:44 PST
Attachment 242087 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 25 Mark Lam 2014-11-21 16:04:14 PST
Comment on attachment 242087 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=242087&action=review

> Source/WTF/wtf/IndexedIterator.h:225
> +        return result += static_cast<unsigned long long>(decrement);

This should be:
    return result -= static_cast<unsigned long long>(decrement);

> Source/WTF/wtf/IndexedIterator.h:247
> +            return result + static_cast<size_t>(-increment);

This should be:
    return result - static_cast<size_t>(-increment);

> Source/WTF/wtf/IndexedIterator.h:259
> +        return result += static_cast<size_t>(decrement);

This should be:
    return result -= static_cast<size_t>(decrement);
Comment 26 Oliver Hunt 2014-11-29 11:08:56 PST
Created attachment 242275 [details]
sigh, webkit-patch was waiting for me to say yes before uploading...
Comment 27 WebKit Commit Bot 2014-11-29 11:12:12 PST
Attachment 242275 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 28 Mark Lam 2014-12-01 09:46:58 PST
Comment on attachment 242275 [details]
sigh, webkit-patch was waiting for me to say yes before uploading...

r=me
Comment 29 Oliver Hunt 2014-12-01 09:50:09 PST
Committed r176592: <http://trac.webkit.org/changeset/176592>
Comment 31 Oliver Hunt 2014-12-01 11:32:32 PST
(In reply to comment #30)
> (In reply to comment #29)
> > Committed r176592: <http://trac.webkit.org/changeset/176592>
> 
> It broke Apple's 32 bit build:
> https://build.webkit.org/builders/Apple%20MountainLion%20Release%20%2832-
> bit%20Build%29/builds/17559

goddammit. Looking into it.
Comment 33 Carlos Alberto Lopez Perez 2014-12-01 16:47:43 PST
(In reply to comment #32)
> The fix https://trac.webkit.org/changeset/176603 broke non Apple builds:
> - EFL ARM Thumb2:
> https://build.webkit.org/builders/
> EFL%20Linux%20ARMv7%20Thumb2%20Release%20%28Build%29/builds/8754
> - EFL ARM:
> https://build.webkit.org/builders/
> EFL%20Linux%20ARMv7%20Traditional%20Release%20%28Build%29/builds/8607
> - GTK x86:
> https://build.webkit.org/builders/GTK%20Linux%2032-bit%20Release/builds/48526

It broke also GTK ARM bot: https://build.webkit.org/builders/GTK%20Linux%2032-bit%20Release/builds/48526

I think the issue is incorrectly assuming that !WIN==MAC|IOS.

What about this fix? (I didn't tested it)

--- a/Source/WTF/wtf/IndexedIterator.h
+++ b/Source/WTF/wtf/IndexedIterator.h
@@ -225,7 +225,7 @@ template <typename VectorType, typename T, typename OverflowHandler = CrashOnOve
         return result -= static_cast<unsigned long long>(decrement);
     }
 
-#if __SIZEOF_SIZE_T__ != __SIZEOF_INT__ || !PLATFORM(WIN)
+#if __SIZEOF_SIZE_T__ != __SIZEOF_INT__ || (PLATFORM(COCOA) && !PLATFORM(WIN))
     IndexedIterator operator+(unsigned increment) const
     {
         IndexedIterator result(*this);
Comment 34 Carlos Alberto Lopez Perez 2014-12-02 08:29:53 PST
Fixed by: https://trac.webkit.org/changeset/176616
Comment 36 Oliver Hunt 2014-12-02 11:13:21 PST
ok, this is weird as stephanie had perf checked it before - i'll see if i can get her to re-test the final commit.  I didn't think that there had been any changes that would impact perf, but maybe i was wrong?
Comment 37 Mark Lam 2014-12-02 20:15:40 PST
Rolled out r176592, r176603, r176616, and r176705 in 176709: <http://trac.webkit.org/r176709>.
Comment 38 Oliver Hunt 2014-12-03 11:13:19 PST
Created attachment 242506 [details]
Patch
Comment 39 WebKit Commit Bot 2014-12-03 11:16:24 PST
Attachment 242506 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 40 Oliver Hunt 2014-12-03 11:58:36 PST
Created attachment 242511 [details]
Patch
Comment 41 WebKit Commit Bot 2014-12-03 12:01:52 PST
Attachment 242511 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 42 Mark Lam 2014-12-03 12:51:17 PST
Comment on attachment 242511 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=242511&action=review

It looks like the change you made is to allow the iterator value to be out of range.  However, when the iterator is actually used to access the element via get(), you to the overflow check then.  I understand that this is probably done for performance reasons.

Based on how the iterators work, it looks like you're allowing overflow and underflow when applying signed increments / decrements, as long as the end result is a legal index into the vector.  The result is that bad code that adds / subtracts a ridiculously large / negative delta can go undetected as long as the resultant index is within range.  I assume that this is intentional, and that we're willing to live with this limitation?

r=me with the blank line removed, assuming the above is the intended behavior you want.

> Source/WTF/wtf/IndexedIterator.h:150
> +

Extra line here which can be removed.
Comment 43 Oliver Hunt 2014-12-03 12:54:32 PST
Comment on attachment 242511 [details]
Patch

Yeah, basically we're now relying on the Vector's own bounds checking logic for more or less everything. This means we should remain memory safe even if you misuse the iterators. Still waiting on final perf numbers.
Comment 44 Anders Carlsson 2014-12-03 12:55:23 PST
If this is still a 1.5-1.7% performance regression I don't see how it can be landed.
Comment 45 Oliver Hunt 2014-12-03 13:01:08 PST
(In reply to comment #44)
> If this is still a 1.5-1.7% performance regression I don't see how it can be
> landed.

It should not be a regression anymore - it's dropped all the expensive checks. If it's not possible to fix this we need to stop use of the range syntax as every use of it is a security regression vs. the old indexed iteration.
Comment 46 Mark Lam 2014-12-08 09:37:27 PST
(In reply to comment #45)
> (In reply to comment #44)
> > If this is still a 1.5-1.7% performance regression I don't see how it can be
> > landed.
> 
> It should not be a regression anymore - it's dropped all the expensive
> checks. If it's not possible to fix this we need to stop use of the range
> syntax as every use of it is a security regression vs. the old indexed
> iteration.

Is this ready to land yet?  Did the perf test run pan out?
Comment 47 Stephanie Lewis 2014-12-11 17:33:44 PST
Patch passed all perf. tests and might even be a little faster.  Due to the unsigned vs. size_t discussion in Vector the patch no longer applies cleanly.
Comment 48 Oliver Hunt 2014-12-13 13:55:15 PST
Created attachment 243259 [details]
Patch
Comment 49 WebKit Commit Bot 2014-12-13 13:57:46 PST
Attachment 243259 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 30 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 50 Mark Lam 2014-12-14 07:24:09 PST
Comment on attachment 243259 [details]
Patch

r=me
Comment 51 WebKit Commit Bot 2014-12-15 10:04:16 PST
Comment on attachment 243259 [details]
Patch

Clearing flags on attachment: 243259

Committed r177284: <http://trac.webkit.org/changeset/177284>
Comment 52 WebKit Commit Bot 2014-12-15 10:04:24 PST
All reviewed patches have been landed.  Closing bug.
Comment 53 Geoffrey Garen 2014-12-15 11:27:02 PST
Does this mean that any time you use .data() you go fast and any time you use .begin() you go slow?

How do you propose that authors go about choosing one or the other?

This strikes me as a dangerous and confusing booby trap to leave in the code.
Comment 54 Oliver Hunt 2014-12-15 11:39:32 PST
(In reply to comment #53)
> Does this mean that any time you use .data() you go fast and any time you
> use .begin() you go slow?
> 
> How do you propose that authors go about choosing one or the other?
> 
> This strikes me as a dangerous and confusing booby trap to leave in the code.

Seriously, there's no performance regression from this. The correct thing is to use begin()/end() if you want to iterate, and data() if you need to pass a pointer to some non-c++ API.

Stephanie's perf measuring says that if anything this improves performance.
Comment 55 Geoffrey Garen 2014-12-15 11:41:02 PST
> Seriously, there's no performance regression from this.

In that case, what changed between the first landed version of this patch and the second, which accounts for fixing the 1.7% performance regression?
Comment 56 Oliver Hunt 2014-12-15 11:45:39 PST
(In reply to comment #55)
> > Seriously, there's no performance regression from this.
> 
> In that case, what changed between the first landed version of this patch
> and the second, which accounts for fixing the 1.7% performance regression?

I removed a lot of the intermediate overflow checks, so it's not as brutal as it used to be. Most of the original performance regression occurred due to complications added when trying to convince all the various changes needed to appease MSVC
Comment 57 WebKit Commit Bot 2014-12-15 15:19:34 PST
Re-opened since this is blocked by bug 139658
Comment 58 Andreas Kling 2014-12-15 15:20:00 PST
I think this is about to get rolled out for breaking debug builds, but FWIW the single data point we got on perf.webkit.org's mac-mavericks builder saw a 0.85% regression on Speedometer and 1.96% on html5-full-render.html
Comment 59 Oliver Hunt 2015-08-02 15:18:44 PDT
Created attachment 258044 [details]
Patch
Comment 60 WebKit Commit Bot 2015-08-02 15:21:45 PDT
Attachment 258044 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 26 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 61 Oliver Hunt 2015-08-02 15:37:31 PDT
Throwing this at the bots because i'm too lazy to build myself :D
Comment 62 Oliver Hunt 2015-09-02 14:43:55 PDT
Created attachment 260442 [details]
Patch
Comment 63 Oliver Hunt 2015-09-02 14:44:41 PDT
Comment on attachment 260442 [details]
Patch

Not complete yet (aside from anything else i'm going to make it debug only at first)
Comment 64 Oliver Hunt 2015-09-02 16:21:29 PDT
Created attachment 260450 [details]
Patch
Comment 65 WebKit Commit Bot 2015-09-02 16:23:31 PDT
Attachment 260450 [details] did not pass style-queue:


ERROR: Source/WTF/wtf/IndexedIterator.h:40:  difference_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:41:  value_type is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
ERROR: Source/WTF/wtf/IndexedIterator.h:44:  iterator_category is incorrectly named. Don't use underscores in your identifier names.  [readability/naming/underscores] [4]
Total errors found: 3 in 28 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 66 Darin Adler 2015-09-10 09:17:39 PDT
Comment on attachment 260450 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=260450&action=review

Currently compiles only on iOS and Mac, so needs more work.

Performance testing results?

Should be using get() more and getPtr() much less (see below).

Vector uses pointers and iterators interchangeably in its implementation. I think this patch causes it to use the iterators too much instead of pointers.

We are now range checking approach too much. We have functions like Vector::remove that already have range checking with ASSERT_WITH_SECURITY_IMPLICATION, but then we are letting them be written in terms of these new checked iterators so they are checking twice. That’s sloppy. We can switch to the iterators and do lots of range checking and remove ASSERT_WITH_SECURITY_IMPLICATION, or we can decide to use pointers inside Vector’s own implementation. We need a principle about how to implement Vector itself that is separate from how code outside Vector uses iterators.

I like the idea of landing the changes that make this refactoring possible separately before changing Vector itself. Most changes that are outside Vector could be landed that way. (That’s one argument for using getPtr, since it would work both before and after, but I’d still prefer to keep that to a minimum so maybe hold off on the changes that actually require using get/getPtr explicitly.)

> Source/JavaScriptCore/bytecode/UnlinkedCodeBlock.cpp:96
>      for (FunctionExpressionVector::iterator ptr = thisObject->m_functionDecls.begin(), end = thisObject->m_functionDecls.end(); ptr != end; ++ptr)
> -        visitor.append(ptr);
> +        visitor.append(WTF::getPtr(ptr));
>      for (FunctionExpressionVector::iterator ptr = thisObject->m_functionExprs.begin(), end = thisObject->m_functionExprs.end(); ptr != end; ++ptr)
> -        visitor.append(ptr);
> +        visitor.append(WTF::getPtr(ptr));

I think it would be nicer to rewrite these two loops to use modern C++ iteration, since we have to touch them.

    for (auto& expression : thisObject->m_functionDecls)
        visitor.append(&expression);
    for (auto& expression : thisObject->m_functionExprs)
        visitor.append(&expression);

> Source/WTF/wtf/IndexedIterator.h:2
> + * Copyright (C) 2014 Apple Inc. All rights reserved.

2014?

> Source/WTF/wtf/IndexedIterator.h:34
> +#define INDEXED_ITERATOR_OVERFLOW(check) do {\
> +    if (!(check))\
> +        OverflowHandler::overflowed();\
> +} while (0)

Seems unhelpful to use a macro for this; we only use the macro in one place.

> Source/WTF/wtf/IndexedIterator.h:36
> +template <typename VectorType, typename T, typename OverflowHandler = CrashOnOverflow> struct IndexedIterator {

This should be a class template, not a struct template. We use struct when we have public data members as the primary interface and in that case we do not use the "m_" prefix on the data members; in cases like this where we don’t, then we use class, not struct.

> Source/WTF/wtf/IndexedIterator.h:37
> +    typedef struct IndexedIterator<const VectorType, const T, OverflowHandler> const_iterator;

I don’t think const_iterator is a good name for this type. That’s a name used on collections to give the type for their const iterator. The IndexedIterator class is an iterator, not a collection. Using the same name is a bit clever but I think subtly confusing.

> Source/WTF/wtf/IndexedIterator.h:50
> +    IndexedIterator()
> +        : m_vector(nullptr)
> +        , m_position(0)
> +    {
> +    }

Newly written code should initialize data members where they are defined instead of listing them all in places like this. If we did that, then this would be more like this:

    IndexedIterator() -> default;

> Source/WTF/wtf/IndexedIterator.h:56
> +    IndexedIterator(std::nullptr_t)
> +        : m_vector(nullptr)
> +        , m_position(0)
> +    {
> +    }

And this would just be a function with an empty body, no need to specify m_vector and m_position.

> Source/WTF/wtf/IndexedIterator.h:58
> +    IndexedIterator(VectorType& vector, ValueType* position)

Since ValueType* must never be null, I suggest we let it be a reference even if it feels a little “pointer-y”.

> Source/WTF/wtf/IndexedIterator.h:68
> +    IndexedIterator(const IndexedIterator& other)
> +        : m_vector(other.m_vector)
> +        , m_position(other.m_position)
> +    {
> +    }

Should not declare this. This is what the compiler will generate as long as we don’t write it explicitly.

> Source/WTF/wtf/IndexedIterator.h:82
> +        m_position++;

Our idiom in WebKit code is to use pre-increment in places like this.

> Source/WTF/wtf/IndexedIterator.h:89
> +        m_position++;

Ditto.

> Source/WTF/wtf/IndexedIterator.h:95
> +        m_position--;

Ditto.

> Source/WTF/wtf/IndexedIterator.h:102
> +        m_position--;

Ditto.

> Source/WTF/wtf/IndexedIterator.h:108
> +    // This conversion operator allows implicit conversion to bool but not to other integer types.
> +    typedef size_t (IndexedIterator::*UnspecifiedBoolType);
> +    operator UnspecifiedBoolType() const { return m_vector ? &IndexedIterator::m_position : nullptr; }

This is not allowed or needed any more in more modern C++. Instead just write this:

    explicit operator bool const() { return m_vector; }

> Source/WTF/wtf/IndexedIterator.h:122
> +#if __SIZEOF_SIZE_T__ != __SIZEOF_INT__ || PLATFORM(COCOA)
> +#define _UNSIGNED_If_DIFFERENT(macro) macro(unsigned)
> +#else
> +#define _UNSIGNED_If_DIFFERENT(macro)
> +#endif
> +
> +#define FOR_EACH_INTEGER_TYPE(macro) \
> +    macro(size_t) \
> +    macro(int32_t) \
> +    macro(long) \
> +    macro(long long) \
> +    macro(unsigned long long) \
> +    _UNSIGNED_If_DIFFERENT(macro)

This is not the right approach. I suggest overloading for int, long, long long, unsigned, unsigned long, and unsigned long long. Don’t overload for size_t or int32_t. Then you won’t need this complex #if.

But also, you should ask Anders if there’s a cleaner way to achieve this.

> Source/WTF/wtf/IndexedIterator.h:169
> +    ptrdiff_t operator-(const const_iterator& right) const
> +    {
> +        return m_position - right.m_position;
> +    }
> +
> +    ptrdiff_t operator-(const ValueType* right) const
> +    {
> +        return get() - right;
> +    }

These should not be member functions; should just be non-member functions; friends if necessary.

> Source/WTF/wtf/IndexedIterator.h:175
> +    void operator=(const IndexedIterator& other)
> +    {
> +        m_vector = other.m_vector;
> +        m_position = other.m_position;
> +    }

Please omit this and let the compiler generate it. It will generate exactly this if you don’t write it out explicitly.

> Source/WTF/wtf/IndexedIterator.h:235
> +    bool operator==(const const_iterator& other) const
> +    {
> +        return unsafeGet() == other.unsafeGet();
> +    }
> +
> +    bool operator!=(const const_iterator& other) const
> +    {
> +        return unsafeGet() != other.unsafeGet();
> +    }
> +
> +    bool operator==(const ValueType* other) const
> +    {
> +        return unsafeGet() == other;
> +    }
> +
> +    bool operator!=(const ValueType* other) const
> +    {
> +        return unsafeGet() != other;
> +    }
> +
> +    bool operator<(const ValueType* other) const
> +    {
> +        return unsafeGet() < other;
> +    }
> +
> +    bool operator<=(const ValueType* other) const
> +    {
> +        return unsafeGet() <= other;
> +    }
> +
> +    bool operator<(const const_iterator& other) const
> +    {
> +        return unsafeGet() < other.unsafeGet();
> +    }
> +
> +    bool operator<=(const const_iterator& other) const
> +    {
> +        return unsafeGet() <= other.unsafeGet();
> +    }
> +
> +    bool operator>(const IndexedIterator& other) const
> +    {
> +        return unsafeGet() > other.unsafeGet();
> +    }
> +
> +    bool operator>=(const const_iterator& other) const
> +    {
> +        return unsafeGet() >= other.unsafeGet();
> +    }
> +
> +    bool operator>(const ValueType* other) const
> +    {
> +        return unsafeGet() > other;
> +    }
> +
> +    bool operator>=(const ValueType* other) const
> +    {
> +        return unsafeGet() >= other;
> +    }

These should not be member functions; should just be non-member functions; friends if necessary. Unclear why these need to do unsafe_get.

I believe it’s possibly inefficient to only define versions that compare with the const_iterator class. We might be creating and destroying temporaries unnecessarily unless the compiler does a great job optimizing.

> Source/WTF/wtf/IndexedIterator.h:243
> +    operator const_iterator() const
> +    {
> +        const_iterator result;
> +        result.m_vector = m_vector;
> +        result.m_position = m_position;
> +        return result;
> +    }

Surprised this compiles when const_iterator is the same type as this class’s type.

> Source/WTF/wtf/IndexedIterator.h:246
> +    VectorType* m_vector;
> +    size_t m_position;

VectorType* m_vector { nullptr };
    size_t m_position { 0 };

> Source/WTF/wtf/IndexedIterator.h:253
> +    ValueType* unsafeGet() const
> +    {
> +        if (!m_vector)
> +            return nullptr;
> +        return get();
> +    }

We don’t normally put functions after the data members. Please let the data members be last.

> Source/WTF/wtf/IndexedIterator.h:257
> +inline typename IndexedIterator<T, ValueType, OverflowHandler>::ValueType* getPtr(IndexedIterator<T, ValueType, OverflowHandler> p)

This should take const IndexedIterator&, not copy an IndexedIterator.

> Source/WTF/wtf/Vector.h:615
> +            TypeOperations::initialize(getPtr(begin()), getPtr(end()));

The getPtr function is only intended to be used in code that’s so generic that we can’t know whether it’s a class or a raw pointer. This code doesn’t qualify. Here we should write begin().get(). Or alternatively have private functions that return pointers for use inside the class? I don’t like the idea of doing the range checking unnecessarily inside the class so it seems bad to be calling get() here instead of unsafeGet().

> Source/WTF/wtf/Vector.h:624
> +            TypeOperations::uninitializedFill(getPtr(begin()), getPtr(end()), val);

Ditto.

> Source/WTF/wtf/Vector.h:693
> +    iterator begin() { return IteratorSelector::makeIterator(*this, data()); }

Why isn’t this just:

    return { *this, data() };

I don’t understand the need for IteratorSelector.

> Source/WTF/wtf/Vector.h:694
> +    iterator end() { return begin() + size(); }

Is this efficient enough? Would it be better to write:

    return { *this, data() + size(); }

> Source/WTF/wtf/Vector.h:695
> +    const_iterator begin() const { return IteratorSelector::makeConstIterator(*this, data()); }

Ditto.

> Source/WTF/wtf/Vector.h:696
> +    const_iterator end() const { return begin() + size(); }

Ditto.

> Source/WTF/wtf/Vector.h:737
> +    void append(const_iterator, size_t);

Not sure we want this. If we do want it, I think we’d want a template version that supports iterators from vectors with other element types too, just like all the other append functions.

> Source/WebCore/platform/graphics/cairo/GradientCairo.cpp:65
> -    Vector<ColorStop>::iterator stopIterator = m_stops.begin();
> +    auto stopIterator = m_stops.begin();
>      while (stopIterator != m_stops.end()) {

Better to use a modern for loop here.

> Source/WebCore/svg/SVGFontElement.cpp:246
> +        size_t it = kerningVector->size();

I don’t think “it” is a good name for an index.

> Source/WebCore/svg/SVGFontElement.cpp:248
> +            auto& value = kerningVector->at(it);

We normally prefer the [] syntax to the at() function.

> Source/WebCore/svg/SVGFontElement.cpp:257
> +            size_t it = kerningVector->size();

Ditto.

> Source/WebCore/svg/SVGFontElement.cpp:259
> +                auto& value = kerningVector->at(it);

Ditto.

> Source/WebCore/svg/SVGFontElement.cpp:266
> +            size_t it = kerningMap.kerningUnicodeRangeMap.size();

Ditto.