NO_PROXY implementation is incomplete

[benjsc]

The function shouldProxy was
// Copied from Rob Wu's great proxy-from-env library: https://github.com/Rob--W/proxy-from-env/blob/96d01f8fcfdccfb776735751132930bbf79c4a3a/index.js#L62

Sadly the implementation is incomplete and misses many use cases.
There is a great page at:

https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/

Which gives a good discussion about NO_PROXY and the issues that have arisen from it. Rob Wu's implementation suffers many problems the article discusses.

For example given:

export HTTP_PROXY=something
export HTTPS_PROXY=something
export NO_PROXY=subdomain.domain.com

A gitlab server at gitlab.subdomain.domain.com would not bypass the proxy as the block:

    if (!/^[.*]/.test(parsedProxyHostname)) {
      // No wildcards, so stop proxying if there is an exact match.
      return hostname !== parsedProxyHostname;
    }

Would do a comparison:

if(!/^[.*]/.test(subdomain.domain.com)){
   return gitlab.subdomain.domain.com !== subdomain.domain.com
}

resulting in a true response and hence the gitlab server being pushed through to the proxy.
Often proxies do not allow access to internal services hence this results in errors occurring in the semantic-release/gitlab plugin:

[12:06:30 AM] [semantic-release] › ℹ  Start step "verifyConditions" of plugin "@semantic-release/gitlab"
[12:06:30 AM] [semantic-release] [@semantic-release/gitlab] › ℹ  Verify GitLab authentication (https://gitlab.subdomain.domain.com/api/v4)
[12:06:30 AM] [semantic-release] › ✘  Failed step "verifyConditions" of plugin "@semantic-release/gitlab"
[12:06:30 AM] [semantic-release] › ✘  An error occurred while running semantic-release: RequestError: Bad response: 403

This bug report is to request a fix to the shouldProxy method to correctly handle NO_PROXY values which list [sub]domains without wild cards.

It seems the proxy-from-env still has the bug, but with the core code not being maintained for 5 years, and even nodejs commenting that it was incomplete when they implemented NO_PROXY support, the recommendation would be to either update the existing code base or convert the codebase to use an alternative library.

[fgreinacher]

Thanks for reaching out!

I think a potential way forward, also considering #756, would be to give undici a shot, it does provide https://github.com/nodejs/undici/blob/7f635e51f6170f4b59abedc7cb45e6bcda7f056d/docs/docs/api/EnvHttpProxyAgent.md#L4 for handling proxy variables.

WDYT @JonasSchubert @travi?

[benjsc]

So looking at undici, it looks like it uses the same 'shouldProxy' method that proxy-from-env did.
https://github.com/nodejs/undici/blob/7f635e51f6170f4b59abedc7cb45e6bcda7f056d/lib/dispatcher/env-http-proxy-agent.js#L113

Still has the bug

[travi]

With progress like nodejs/node#57165, I'd personally prefer to leverage the native implementation and direct requests like this one directly to the node project. I don't think it makes sense for our packages to have custom additional logic that would be better solved elsewhere.

[fgreinacher]

With progress like nodejs/node#57165, I'd personally prefer to leverage the native implementation and direct requests like this one directly to the node project. I don't think it makes sense for our packages to have custom additional logic that would be better solved elsewhere.

Yes, I agree, we should not solve this locally. @benjsc Could you bring this topic upstream to undici?

[benjsc]

Following up on this, it looks like the upstream udici fixed this bug in their code base in Feb via nodejs/undici#4676 Hence switching to use undici would remove the need to handle proxy code directly in this repo.