Specifically, RenderBlock::xPositionForFloatIncludingMargin and RenderBlock::yPositionForFloatIncludingMargin do not ignore the margin if the float has a shape outside. It would be good to look for other cases as well when fixing this bug, but it might also be hard to find tests for this.
When doing this, add a comment to these methods to explain that when floats have shape-outside, the FloatingObject has different size/coordinates than the RenderBox. (it is a bit confusing)
I can't come up with any situations in which this actually is an issue, so I'm closing it. Things seem to work properly, and I think if I come up with an actual use case where this is a problem, I'll file a bug for that. This is almost too low level to be able to come up with a proper test case.