Instead, we could rely on NetworkProcess::m_serverToContextConnections. That might be an opportunity to better separate the organization of the connections from their users so that refactoring like done in https://bugs.webkit.org/show_bug.cgi?id=195649 would be simpler.
This is only needed for configurations where we have more than one NetworkProcess in a given process and service workers running for both.