Currently, CodeBlockHash once converts the entire source code string to the UTF-8 string and pass it to SHA1 to calculate the hash value. However, it's memory consuming because this process allocates a large buffer to store the entire UTF-8 string. Instead of the current process (alloc, convert, and use), we should implement the streaming decoder decoding UTF-16 into UTF-8 and stream the bytes into SHA1 iteratively.
I’m also not exactly sure why we SHA1 the UTF-8 representation. Seems like we could do it on the UTF-16 representation, although that’s not necessarily better. For non-Latin-1 strings we already store the UTF-16 representation. Also not sure why we chose SHA1 rather than the efficient hash function we use for our hash tables. I guess a 32-bit hash is just too small.
(In reply to comment #1) > I’m also not exactly sure why we SHA1 the UTF-8 representation. Seems like > we could do it on the UTF-16 representation, although that’s not necessarily > better. For non-Latin-1 strings we already store the UTF-16 representation. > > Also not sure why we chose SHA1 rather than the efficient hash function we > use for our hash tables. I guess a 32-bit hash is just too small. Hm, but seeing the m_hash which may be used as hash value for CodeBlockHash, it's unsigned. So the current hash value is 32-bit long. CodeBlockHash is introduced in the following bug. https://bugs.webkit.org/show_bug.cgi?id=103623 Seeing this issue, CodeBlockHash is introduced for debugging purpose (not for HashTable); To add a roughly unique name to CodeBlock to print it rather than using the pointer value of CodeBlock. For this purpse, the important thing is strong distribution. Even if it costs long time to calcuate that, we need to get a well-distributed hash value. I think this is why SHA1 is chosen here rather than StringHasher.
I’d like to understand better how much better SHA1 is than the fast hash in StringHasher and strongly distributing the bits of the data. I’m not surprised it’s better, but I’d be surprised if it’s a lot better.