X Tutup
The Wayback Machine - https://web.archive.org/web/20210506192335/https://github.com/reactor/reactor-netty/issues/887
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy settings via system properties not working with Spring WebClient #887

Open
VsKrishna87 opened this issue Nov 4, 2019 · 23 comments
Open

Comments

@VsKrishna87
Copy link

@VsKrishna87 VsKrishna87 commented Nov 4, 2019

My following WebClient is working fine with internet connection but not through our proxy connection.

WebClient webClient = WebClient.builder()
        .baseUrl("https://targetsite.com")
        .build();

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

Event though, the same client is working through proxy, if I set it as mentioned below,

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

ReactorClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);

WebClient webClient = WebClient.builder()
        .clientConnector(connector)
        .baseUrl("https://targetsite.com")
        .build();

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

But if I set the same Proxy details either using System.setProperty("http.proxyHost","ourproxy.com"); System.setProperty("http.proxyPort","8080"); or JVM run-time arguments -Dhttp.proxyHost=ourproxy.com -Dhttp.proxyPort=8080

WebClient webClient = WebClient.builder()
        .baseUrl("https://targetsite.com")
        .build();

System.setProperty("http.proxyHost", "ourproxy.com");
System.setProperty("http.proxyPort", "8080");

webClient.post()
    .uri("/service/serviceName")
    .body(BodyInserters.fromObject(reqData))
    .retrieve()
    .bodyToMono(WebServiceResponse.class)

The calls are getting failed with UnknownHostException like,

[04/11/2019 12:32:43.031 IST] DEBUG [reactor-http-epoll-3] [PooledConnectionProvider:254] - Creating new client pool [http] for targetsite.com:443
[04/11/2019 12:32:43.033 IST] DEBUG [reactor-http-epoll-3] [PooledConnectionProvider:254] - [id: 0xe4a0dc15] Created new pooled channel, now 0 active connections and 1 inactive connections
[04/11/2019 12:32:43.045 IST] DEBUG [reactor-http-epoll-3] [SslProvider:254] - [id: 0xe4a0dc15] SSL enabled using engine SSLEngineImpl and SNI targetsite.com:443
[04/11/2019 12:32:43.046 IST] DEBUG [reactor-http-epoll-3] [BootstrapHandlers:254] - [id: 0xe4a0dc15] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (BootstrapHandlers$BootstrapInitializerHandler#0 = reactor.netty.channel.BootstrapHandlers$BootstrapInitializerHandler), (SimpleChannelPool$1#0 = io.netty.channel.pool.SimpleChannelPool$1), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpClientCodec), (reactor.left.decompressor = io.netty.handler.codec.http.HttpContentDecompressor), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
[04/11/2019 12:32:43.165 IST] ERROR [reactor-http-epoll-2] [AbstractErrorWebExceptionHandler:117] - [13ebf1eb] 500 Server Error for HTTP POST "/service/serviceName"
java.net.UnknownHostException: targetsite.com: Name or service not known

Please help, why my code is not functioning even I set the proxy details via JVM run-time arguments or system properties.

Actually I want to avoid code level proxy setting. So please guide me to correct my code or approach, so that I can use JVM run-time argument option.

@violetagg
Copy link
Member

@violetagg violetagg commented Nov 4, 2019

Only this is supported (system properties are not supported)

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

You can easily add/remove proxy configuration using the system properties in your code.

@nhomble
Copy link
Contributor

@nhomble nhomble commented Jun 22, 2020

@violetagg just curious, do we ever see the system properties to be pulled in by reactor-netty or do you expect frameworks like spring's webclientbuilder to expose this?

@rstoyanchev
Copy link
Contributor

@rstoyanchev rstoyanchev commented Jun 23, 2020

@nhomble I would say no. It's not customary to check System properties in the Spring Framework. Typically external properties from different sources (environment, properties file, etc) are injected into classes that can then configure things accordingly.

@nhomble
Copy link
Contributor

@nhomble nhomble commented Jun 23, 2020

I totally agree @rstoyanchev the properties wouldn't come directly from system properties. I was thinking reactor-netty could consider using the system properties directly but integration with spring should rely on configuration properties.

I have this working in my project with the config prop approach, but I wanted to make sure I am not duplicating any enhancements either reactor or spring would consider taking in.

@violetagg
Copy link
Member

@violetagg violetagg commented Jul 1, 2020

@nhomble I prefer to have only an explicit configuration using the API instead of system properties to avoid mixture of priorities .... or should it be a choice between those two configurations or a merge of those two?
2. Also let say that we have more than one client in one JVM, then should these system properties apply to all clients or only to some of them ...

@nhomble
Copy link
Contributor

@nhomble nhomble commented Jul 2, 2020

Great points @violetagg. For (1) I think the client builder could support both styles and to avoid the merging/ambiguity problem the builder could either take the config as-is inTcpConfiguration with .proxy(Consumer<..>) or explicitly .proxyWithSystemProperties() for lack of a better name.

For (2), I think we can continue to be flexible by not assuming system properties are a source of configuration.

My current application just provides a proxy consumer that loads the env vars manually which I figure would belong in the library. I also imagine this would help teams like mine migrate easier 😄

@violetagg
Copy link
Member

@violetagg violetagg commented Jul 2, 2020

@nhomble

explicitly .proxyWithSystemProperties() for lack of a better name.

I like the idea and will reopen the issue. So if somebody wants can provide a PR.

@nhomble
Copy link
Contributor

@nhomble nhomble commented Jul 2, 2020

@violetagg thanks - do you have any thoughts around reconciling the http and https settings? The proxy enum only has http. Similar to the socket enums, should we introduce a new enum for https? Then I think it's clean to call the systemProperties method on the builder for that type.

@violetagg
Copy link
Member

@violetagg violetagg commented Jul 2, 2020

@nhomble Netty provides only one type of HttpProxyHandler and it uses CONNECT method.
https://netty.io/4.1/api/io/netty/handler/proxy/package-summary.html

@kmariappan2016
Copy link

@kmariappan2016 kmariappan2016 commented Aug 12, 2020

Only this is supported (system properties are not supported)

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));

You can easily add/remove proxy configuration using the system properties in your code.

I am using spring webflux latest and the reactor netty as well.But its not using Proxy

@violetagg
Copy link
Member

@violetagg violetagg commented Aug 12, 2020

@kmariappan2016 You mean that proxy functionality is not working?

@kmariappan2016
Copy link

@kmariappan2016 kmariappan2016 commented Aug 13, 2020

@kmariappan2016 You mean that proxy functionality is not working?

Yes. I have configured the proxy like the following and still its not taking the proxy.
HttpClient httpClient = HttpClient.create()
.tcpConfiguration(tcpClient -> tcpClient.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
.doOnConnected(conn -> conn.addHandlerLast(new ReadTimeoutHandler(2000, TimeUnit.MILLISECONDS)))
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP).host("ourproxy").port(9000)));

	ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient);
	return WebClient.builder().clientConnector(connector).filter(oauth2FilterFunction).build();

Its not redirecting via proxy server.

I am using the following library
implementation 'org.springframework.security:spring-security-oauth2-client:5.3.4.RELEASE'
// https://mvnrepository.com/artifact/org.springframework/spring-webflux
implementation 'org.springframework:spring-webflux:5.2.8.RELEASE'
// https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty
implementation 'io.projectreactor.netty:reactor-netty:0.9.11.RELEASE'

I am not sure if I am missing anything
Please advice me.
Thanks

@violetagg
Copy link
Member

@violetagg violetagg commented Aug 13, 2020

@kmariappan2016 Create this as a separate issue and let's investigate there. Ok?

@kmariappan2016
Copy link

@kmariappan2016 kmariappan2016 commented Aug 13, 2020

@kmariappan2016 Create this as a separate issue and let's investigate there. Ok?

Create a new issue,seems that any version above spring-security-oauth2-client:5.3.0.RELEASE is bypassing proxy server settings.

@violetagg violetagg added this to the 1.0.0 Backlog milestone Oct 19, 2020
@nagyzsolthun
Copy link

@nagyzsolthun nagyzsolthun commented Nov 19, 2020

tcpConfiguration is deprecated. What is the recommended approach for adding proxy config?

https://projectreactor.io/docs/netty/release/api/index.html?deprecated-list.html states:
Apply a TcpClient mapping function to update TCP configuration and return an enriched HttpClient to use.

Can you please provider an example? Similar to

HttpClient httpClient = HttpClient.create()
            .tcpConfiguration(tcpClient -> tcpClient
                .proxy(proxy -> proxy
                    .type(ProxyProvider.Proxy.HTTP)
                    .host("ourproxy.com")
                    .port(8080)));
@violetagg
Copy link
Member

@violetagg violetagg commented Nov 19, 2020

@nagyzsolthun Everything that you was able to configure via tcpConfiguration you can configure now (since Reactor Netty version 1.0.0) directly on HttpClient level.

HttpClient httpClient =
    HttpClient.create()
              .proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP)
                                   .host("ourproxy.com")
                                   .port(8080));
@preethirajaram
Copy link

@preethirajaram preethirajaram commented Dec 15, 2020

@nagyzsolthun Everything that you was able to configure via tcpConfiguration you can configure now (since Reactor Netty version 1.0.0) directly on HttpClient level.

HttpClient httpClient =
    HttpClient.create()
              .proxy(proxy -> proxy.type(ProxyProvider.Proxy.HTTP)
                                   .host("ourproxy.com")
                                   .port(8080));

Hi,
I tried the above code with below combination,

  1. reactor-netty version 1.0.0 and spring-security-test

httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.valueOf("HTTP"))
.host("abc.com")
.port(8080)

Am getting the below error:

Description:

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

The injection point has the following annotations:
- @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean of type 'javax.servlet.http.HttpServletRequest' in your configuration.

Please suggest.

@violetagg
Copy link
Member

@violetagg violetagg commented Dec 15, 2020

Hi,
I tried the above code with below combination,

  1. reactor-netty version 1.0.0 and spring-security-test

httpClient = HttpClient.create()
.proxy(proxy -> proxy.type(ProxyProvider.Proxy.valueOf("HTTP"))
.host("abc.com")
.port(8080)

Am getting the below error:

Description:

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

The injection point has the following annotations:

  • @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean of type 'javax.servlet.http.HttpServletRequest' in your configuration.

Please suggest.

@preethirajaram

Field context in com.db.ct.cdh.util.security.WebSecurityConfiguration$X509Endpoints required a bean of type 'javax.servlet.http.HttpServletRequest' that could not be found.

This is not related to Reactor Netty

@violetagg violetagg changed the title Proxy setting not working with Spring WebClient Proxy settings via system properties not working with Spring WebClient Feb 23, 2021
@agarwaljayant7
Copy link

@agarwaljayant7 agarwaljayant7 commented Apr 4, 2021

in my case connection seems to be not getting established between my local system/WebClient to the remote proxy server.
"[id: 0x62f7d272, L:/51.135.65.49:52907 ! R:abc-proxy.uk.xyz.abcintra.net/59.11.19.11:3128] The connection observed an error". It seems the connection b/w my local WebClient and proxy server is not getting established. Note; i've not pasted here the exact Local and Remote addresses and have mentioned the sample one. Thanks!

underlying Code snippet :

final ReactorClientHttpConnector connector = new ReactorClientHttpConnector( HttpClient.create() .tcpConfiguration( tcpClient -> tcpClient.proxy( proxy -> proxy .type(ProxyProvider.Proxy.HTTP) .host("abc-proxy.uk.xyz.abcintra.net") .port(3128) ) ) );

  String tokenResponse = WebClient.builder()
  .clientConnector(connector)
  .build()
  .post()
  .uri("https://dev.api.abc.com/oauth2/token.oauth2").headers(httpHeaders -> {
    httpHeaders.add("Authorization", authHeader);    // authHeader was generated to something like "Basic jWdb2njwUiOD3jcIfNs4dDjk3sle9oRj3sjer=" 
    httpHeaders.add("Content_Type", "application/x-www-form-urlencoded");
}).body(BodyInserters.fromValue("grant_type=client_credentials")).retrieve().bodyToMono(String.class).block();
@violetagg
Copy link
Member

@violetagg violetagg commented Apr 5, 2021

@agarwaljayant7 For issues please open a new issue, for questions please use Gitter

@agarwaljayant7
Copy link

@agarwaljayant7 agarwaljayant7 commented Apr 5, 2021

ok, tx!

@fabiohenriquebayma
Copy link

@fabiohenriquebayma fabiohenriquebayma commented Apr 20, 2021

Hi, everyone!

Thank you for all information here!

I'm working with RestTemplate right now and works fine connected in a VPN which is System Proxied. So I don't need to configure anything about the existence of the proxy to make the Spring Boot working. I was using OAuth2RestTemplate but after upgrading to Spring Boot 2.4.4 it is marked as deprecated. But see, I had to write my own CustomPasswordOAuth2AuthorizedClientProvider facing all pain to handle with private methods final classes to simple get a 'Id_token' value that I can get with two lines of code with OAuth2RestTemplate.

So, Why they deprecated a top dependency when actually makes sense to deprecated the main feature to not cut someone else legs? Why not just keep the OAuth2RestTemplate for the ones that are not importing Webflux yet cause they architecture are not async yet? I miss the concept of @Legacy instead of @deprecated. And in my case, I don't want to be forced to pass my password to a http proxy because WebClient needs...

So, I faced issues with WebClient and Java 8 if the system has a system proxy and using Java 11 I could workaround configuring it before the application startup. But unfortunately, I had to remove 11 because some reactor ran in the test with Java 11 instead of 8 which is the version of my target source and source compatibility configuration and IntellJ configuration.

Now, I rolled back to RestTemplate until we have this problem resolved.

This code worked with Java 11
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
X Tutup