This bug is extremly curious. After setting setCustomValidity('error') on an element with a constrain (i.e.: required). The checkValidity method return false, if the field is not empty and the setCustomValidity was set to an empty string. The validityState returns true for the valid-flag. You will find a testcase @ http://afarkas.github.com/webshim/tests/safari.html. (Hope this makes it much clearer). This bug should have a high priority and should block bug 28649.
I confirmed that Safari 5.0.2 had this problem. However Safari nightly with today's WebKit and Google Chrome 8.0.552.18 doesn't have the problem. I'm not sure what revision fixed the problem.