validate_hostname() returns -1 on UTF-8 conversion failure, which converts to true in its bool return type

[maskit]

validate_hostname() in src/tscore/X509HostnameValidator.cc has return type bool, but on ASN1_STRING_to_UTF8() failure it does:

astrlen = ASN1_STRING_to_UTF8(&astr, str);

if (astrlen < 0) {
  return -1;
}

-1 implicitly converts to true when returned from a bool function, so a UTF-8 conversion failure on the certificate CN is reported as a successful hostname match instead of a failure.

This traces back to the original addition of the function in 1649abc (2015) and is present on current master, independent of any other in-flight work. It was noticed during review of #13476 (OpenSSL 4 build-compatibility fixes), which touches nearby lines for unrelated reasons but does not change this logic.

Suggested fix

return false; instead of return -1;