On specific connections, the NetworkProcess can consume a whole CPU core for longer periods. This is not reasonable. The problem was tracked down to an inefficiency in GnuTLS. As such, it should be reported to that project as well. This bug will serve as a tracker for that effort. A deeper analysis will follow in this bug, and will also be used for the GnuTLS bug report.
When establishing HTTPS connections to specific hostnames, a lot of time can be spent under gnutls_x509_crt_check_hostname2() -- in some cases seconds. This can severely affect the NetworkProcess performance or in the case of resource-limited systems even the performance of the whole system. This is reproducible with GnuTLS 3.5.7. The 'images.washingtonpost.com' hostname will be used throughout this analysis. On a reasonably powerful Intel CPU, over 2 seconds can be spent under gnutls_x509_crt_check_hostname2() when establishing a secure connection to that server. The problem in gnutls_x509_crt_check_hostname2() are repetitive calls to gnutls_x509_crt_get_subject_alt_name(). In case of the certificate provided by images.washingtonpost.com, there are 567 subject alternative names available, and the 519th alternative name is the one that matches the hostname that's being checked. The bottleneck under gnutls_x509_crt_get_subject_alt_name() stems from the fact that for every call, the alternative names are reimported from the certificate through gnutls_x509_ext_import_subject_alt_names(). This function alone takes roughly 4ms on every invocation, so at 519 invocations it does in total consume about 2 seconds. This is also easily reproducible using the gnutls-cli tool.`gnutls-cli images.washingtonpost.com` takes a few seconds to determine that the certificate is trusted, while `gnutls-cli washingtonpost.com` does that instantly. gnutls_x509_crt_check_hostname2(): https://gitlab.com/gnutls/gnutls/blob/gnutls_3_5_x/lib/x509/hostname-verify.c#L181 gnutls_x509_crt_get_subject_alt_name(): https://gitlab.com/gnutls/gnutls/blob/gnutls_3_5_x/lib/x509/x509.c#L1726 gnutls_x509_ext_import_subject_alt_names(): https://gitlab.com/gnutls/gnutls/blob/gnutls_3_5_x/lib/x509/x509_ext.c#L215
Created attachment 299712 [details] gnutls -V images.washingtonpost.com Output of `gnutls -V images.washingtonpost.com`. Lists all 567 subject alternative names, with the '*.washingtonpost.com' hostname being the 519th in the list and the one against which images.washingtonpost.com successfully matches after 518 previous invocations of gnutls_x509_crt_get_subject_alt_name().
Fixed in the GnuTLS project: https://gitlab.com/gnutls/gnutls/issues/165