Codecov Report

Merging #1847 into master will not change coverage by %.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master     #1847   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           22        22           
  Lines         2013      2013           
=========================================
  Hits          2013      2013           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0654046...56eed22. Read the comment docs.

@jalopezsilva I tried to explain the current state of HTTPS proxies support. Can you please take a look?

So what I'm seeing is we're looking at what may potentially be released as a minor release (the only feature is cadata which is very minor) if we exclude the new features related to HTTPS proxies from the release.

I'm wondering if we should make a 1.25.9 release with everything on master except the HTTPS proxies and then allowing the HTTPS proxy features to be released all at once in 1.26. We can do this by creating a 1.25-series branch and fast-forwarding everything except the HTTPS proxy commit and making the release from that branch.

What are our thoughts here?

That sounds good to me!

I can handle that construction. :)

That sounds good to me as well!

If you're doing a minor release, it might be worth adding the warning that I added when people attempt to contact an HTTPS proxy through their configuration. You'll have to cherry-pick the change and don't mention the _allow_https_proxy_to_see_traffic flag.

Right now, people can have a configuration like the following:

{
"http_proxy":"http://proxy:8080",
"https_proxy":"https://proxy:8080",
}

We currently ignore the HTTPS on the URL for the HTTPS proxy definition. When we close #1806, we'll start using it and potentially attempting to connect using TLS to a non-TLS port.

The benefit of adding the warning now, is that we'll give users a chance to clean their configuration before we add the HTTPS proxy support. If we don't do it, we might end up trying to do a TLS connection to a proxy that doesn't support TLS and the error might be even more confusing for users.

I noticed this misconfiguration happening a lot when internally patching our libraries.

@jalopezsilva Sorry for taking so much time to reply here, I've been trying things out over the past few days.

On the 1.25-series branch I have committed every commit to master except for 8c7a43b which the commit that changes HTTPS proxies. I plan on making the 1.25.9 tag on that branch.

What I see on 1.25-series:

• Proxy HTTP + Dest HTTP -> Forwarding (Dest)
• Proxy HTTP + Dest HTTPS -> CONNECT (Dest) + TLS Handshake (Dest)
• Proxy HTTPS + Dest HTTP -> CONNECT (Proxy) + TLS Handshake (Proxy) (not correct)
• Proxy HTTPS + Dest HTTPS -> CONNECT (Dest) + TLS Handshake (Dest) (not correct)

With the change on master I'm seeing:

• Proxy HTTP + Dest HTTP -> Forwarding (Dest)
• Proxy HTTP + Dest HTTPS -> CONNECT (Dest) + TLS Handshake (Dest)
• Proxy HTTPS + Dest HTTP -> TLS Handshake (Proxy) + Forwarding (Dest)
• Proxy HTTPS + Dest HTTPS -> TLS Handshake (Proxy) + Forwarding (Dest) (not correct)

Ideally we want for the last scenario to be:
Proxy HTTPS + Dest HTTPS -> TLS Handshake (Proxy) + CONNECT (Dest) + TLS Handshake (Dest)

I'm wondering if we need to have a warning at all for v1.25.9 if HTTPS proxies in urllib3 both don't work? I'm just unsure about adding a warning to a minor that will be going away with a proper fix in the next major. Warnings are a tricky subject when the library is so widespread.

We currently ignore the HTTPS on the URL for the HTTPS proxy definition.

This comment has me confused, I don't see urllib3 ignoring HTTPS proxies, just not working at all? Is that what you meant or did you mean "we" meaning your configuration/scripts?

Hey @sethmlarson, sorry I should have been clearer. The issue arises with a misconfiguration by users on their proxies. If you apply the following local patch, you'll understand what I'm talking about: https://pastebin.com/raw/GZDcwPE7

The patch expands the test_proxy_poolmanager.py with a quick test case to validate the behavior with an HTTPS proxy and a HTTPS destination. My concern arises when users have incorrectly classified their proxy as HTTPS when it's really HTTP.

With the 1.25-series the behavior I see in the test is:

• [WORKS AS INTENTED]Proxy HTTP + Dest HTTP -> Forwarding (Dest)
• [WORKS AS INTENTED]Proxy HTTP + Dest HTTPS -> CONNECT (Dest) + TLS Handshake (Dest)
• [FAILS AS INTENTED]Proxy HTTPS + Dest HTTP -> CONNECT (Proxy) + TLS Handshake (Proxy) (not correct)
• [DOESN'T FAIL] Proxy HTTPS + Dest HTTPS -> CONNECT (Dest) + TLS Handshake (Dest) (not correct)

The last case with a HTTPS proxy with an HTTPS destination should fail but doesn't. That's because the proxy scheme is mostly ignored. It's possible for users to have a 'HTTPS://' on their configuration when they really mean 'HTTP://'.

With the 1.25-series this misconfiguration still works. Once we enable TLS in TLS, it's going to start failing as we'll attempt to perform a TLS connection to a non-TLS proxy.

The question is how prevalent is this misconfiguration? I found around 82 cases in our monorepo across teams. I don't have a repro handy but I suspect this misconfiguration also affects requests: https://github.com/psf/requests/blob/master/requests/adapters.py#L193.

I cleaned up our configurations internally so I'm not worried about us. I do suspect this misconfiguration is out on the wild though.. I'm worried that if we roll TLS in TLS in 1.26.X we'll get a bunch of users saying something broke when it was really their misconfiguration.

@joelverhagen Okay now I perfectly understand, thank you so much for the super-detailed write up (again). Sorry for all the confusion :)

I'll create a PR against 1.25-series and have you take a look then we can release it.

I've created the PR: #1851

Closing in favor of #1852, thanks all.

@pquentin We'll need to update the changelog on master as well, people get confused about releases "missing" without it. :)

Warning looks great. I've subscribed myself to 1850 to see how many people are affected. Hopefully not too many.

Thanks everyone!

@sethmlarson Thanks for updating the master branch, nice catch. 👍