According to spec (https://drafts.csswg.org/cssom-view/#dom-mouseevent-offsetx), offsetX/offsetY should return the values relative to the element's padding edge.
Firefox does this correctly, WebKit (and also Chromium, currently), return the value relative to the nearest block ancestor.
Corresponding Chromium bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1141782
Relevant WPT issue: https://github.com/web-platform-tests/wpt/issues/15089
This is now fixed on Blink. Safari is the outlier.
This is why we can't have nice things.
We will always need some kind of library to abstract away these browser differences.