https_proxy support #169
Comments
|
that's correct, out of the box the SDK does not support HTTP proxy. which README are you referring to? Would be good to correct that. It's possible to replace Hyper with with Reqwest in the SDK to gain HTTP proxy support, although it requires a bit of code. |
|
I was referring to the main project readme - but it looks like you already fixed it :) I'm behind a corporate proxy. What's the best way forward for me to continue using aws-sdk ? Any direction appreciated. |
jmklix
added
feature-request
|
I'm not spun up on the AWS SDK yet, but if it allows replacing the transport, we (@MaterializeInc) recently wrote some code to teach hyper to respect It's not open source right now, but let me see what I can do about that. It's based on the |
The http_proxy, https_proxy, and no_proxy support we built in the
http-util crate will be useful to at least one other person [0], and
probably many more. This commit prepares the crate for release to
crates.io.
Specifically:
* The crate is renamed to "mz-http-proxy", for consistency with
"mz-avro".
* The license headers are updated to indicate the crate is Apache 2.0
licensed.
* The adapters are placed behind feature flags, so that you don't need
to bring in reqwest if you just want hyper, or vice versa.
* The documentation is improved, with a smidge of Materialize branding
to boot.
I also switched `bin/doc` back to using the nightly toolchain in CI,
with support for optionally doing so locally too, to enable the
`doc_cfg` attributes that indicate which features must be enabled for
which items.
[0]: awslabs/aws-sdk-rust#169 (comment)
The http_proxy, https_proxy, and no_proxy support we built in the
http-util crate will be useful to at least one other person [0], and
probably many more. This commit prepares the crate for release to
crates.io.
Specifically:
* The crate is renamed to "mz-http-proxy", for consistency with
"mz-avro".
* The license headers are updated to indicate the crate is Apache 2.0
licensed.
* The adapters are placed behind feature flags, so that you don't need
to bring in reqwest if you just want hyper, or vice versa.
* The documentation is improved, with a smidge of Materialize branding
to boot.
I also switched `bin/doc` back to using the nightly toolchain in CI,
with support for optionally doing so locally too, to enable the
`doc_cfg` attributes that indicate which features must be enabled for
which items.
[0]: awslabs/aws-sdk-rust#169 (comment)
|
Goal: Create a smithy_client that I can use with sts, but with a hyper_proxy::Connector. I'm trying to get access to the underlying hyper client, but so far without success. So far I have this: //! Handles everything to do with reading the https_proxy environment variable
use hyper::client::HttpConnector;
use hyper::Client;
use hyper_proxy::{Intercept, Proxy, ProxyConnector};
use smithy_client::Builder;
use smithy_http::operation::BuildError;
use std::env;
/// Returns a proxy if the https_proxy environment varablie is set
pub fn proxy() -> anyhow::Result<Option<ProxyConnector<HttpConnector>>> {
// TODO: Convert errors to thiserror
let https_proxy = env::var("https_proxy")?;
if https_proxy.is_empty() {
Ok(None)
} else {
let proxy = {
let proxy_uri = https_proxy.parse()?;
let proxy = Proxy::new(Intercept::All, proxy_uri);
let connector = HttpConnector::new();
let proxy_connector = ProxyConnector::from_proxy(connector, proxy)?;
proxy_connector
};
Ok(Some(proxy))
}
}
/// If the https_proxy environment variable is set, returns a DynConnector you can use with other AWS services
/// Otherwise, returns nothing, so you can just use the aws-sdk-rust library normally
pub fn connector() -> anyhow::Result<Option<smithy_client::Client>> {
Ok(proxy()?
// Turn the proxy into a hyper::Client
.map(|proxy| Client::builder().build(proxy))
// Turn the hyper client into a DynConnector (it used to be like that, but then I changed it. Neither work so far).
.map(|client| Builder::hyper(client).build()))
}
// .. and later...
let proxy_conn = connector().unwrap().unwrap();
let client = Client::from_conf_conn(config, proxy_conn);
log::info!("Getting token");
let token = client
// It dies here..
.get_session_token()Error message: I'm working on this in my little spare. I'll probably get there eventually, but any direction from devs would help. |
|
I suspect the error is that the response body type hasn't been converted. You can use use hyper::client::HttpConnector;
use hyper::Client;
use hyper_proxy::{Intercept, Proxy, ProxyConnector};
use smithy_client::bounds::SmithyConnector;
//use smithy_http::body::SdkBody;
//use tower::{BoxError, Service};
fn main() {
let proxy = {
let proxy_uri = "http://my-proxy:8080".parse().unwrap();
let proxy = Proxy::new(Intercept::All, proxy_uri);
//proxy.set_authorization(Authorization::basic("John Doe", "Agent1234"));
let connector = HttpConnector::new();
let proxy_connector = ProxyConnector::from_proxy(connector, proxy).unwrap();
proxy_connector
};
let hyper_client = Client::builder().build(proxy);
// need to ensure the `hyper` feature of smithy-client is enabled
let hyper_client = smithy_client::hyper_ext::Adapter::from(hyper_client);
let hyper_client = check_bounds(hyper_client);
let conf = aws_sdk_sts::Config::builder().build();
let sts = aws_sdk_sts::Client::from_conf_conn(conf, hyper_client);
sts.get_session_token().send();
} |
|
That got me going. Thank you :) @rcoh I'd like to leave this open though. Feature request:
This would be consistent with the other (python for example) sdks (I think). |
|
The above example didn't work for me, had to change it a bit: hyper-tls = "0.5.0"
hyper-proxy = "0.9.1"
hyper = "0.14.16"
aws-smithy-client = { version = "0.34.1", features = ["hyper"] }let proxy = {
let proxy_uri = "http://my-proxy:8080".parse().unwrap();
let proxy = Proxy::new(Intercept::All, proxy_uri);
//proxy.set_authorization(Authorization::basic("John Doe", "Agent1234"));
let connector = HttpConnector::new();
let proxy_connector = ProxyConnector::from_proxy(connector, proxy).unwrap();
proxy_connector
};
// need to ensure the `hyper` feature of smithy-client is enabled
let hyper_client = aws_smithy_client::hyper_ext::Adapter::builder()
.build(proxy);
let client = aws_sdk_s3::client::Client::from_conf_conn(config, hyper_client);The difference is that Also the Otherwise thanks guys, was helpful :) |
|
Another example I eventually got working - using socks proxy: aws-smithy-client = { version = "0.38.0", features = ["hyper"] }
hyper = "0.14.16"
hyper-proxy = "0.9.1"
hyper-socks2 = "0.6.0"use aws_sdk_cloudsearchdomain::{Client, Config, Endpoint};
use http::Uri;
use hyper_proxy::ProxyConnector;
use hyper_socks2::SocksConnector;
...
let proxy = {
let mut connector = hyper::client::HttpConnector::new();
connector.enforce_http(false);
let socks_connector = SocksConnector {
proxy_addr: Uri::from_static("socks5://localhost:1080"),
auth: None,
connector,
};
ProxyConnector::new(socks_connector).unwrap()
};
let hyper_client =
aws_smithy_client::hyper_ext::Adapter::builder().build(proxy);
let config = aws_config::load_from_env().await;
let cloud_search_domain_config = Config::builder()
.credentials_provider(config.credentials_provider().unwrap().clone())
.endpoint_resolver(Endpoint::immutable("http://REDACTED.cloudsearch.amazonaws.com".parse().unwrap()))
.region(config.region().unwrap().clone())
.build();
let client = Client::from_conf_conn(cloud_search_domain_config, hyper_client); |
|
Several notes here to make above http proxy solutions work:
I'd like to see an option for the sdk to auto-determine and auto-configure proxy based on env. variables when the user desires to do so. |
|
Helpful comments here... in my case I was looking to point at a local proxy i.e. OWASP ZAP (mainly for learning and troubleshooting). This required a TLS connector to be configured on the proxy connector with the addition of the self-signed cert from the "Dynamic SSL Certificates" in ZAP > Tools > Options. Here is what I ended up going with (following from the above examples but with some variable renaming): let proxy = {
// TLS cert config
let mut f = File::open("/tmp/owasp_zap_root_ca.cer").unwrap();
let mut buffer = vec![];
f.read_to_end(&mut buffer).unwrap();
let cert = Certificate::from_pem(buffer.as_slice()).unwrap();
let connector_tls = TlsConnector::builder()
.add_root_certificate(cert)
// .danger_accept_invalid_certs(true) // less safe alternative to add_root_certificate(...)
.build()
.unwrap();
// Proxy config
let proxy_uri = "http://127.0.0.1:8080".parse().unwrap();
let proxy = Proxy::new(Intercept::All, proxy_uri);
// proxy.set_authorization(Authorization::basic("user", "pass"));
let mut connector_http = HttpConnector::new();
connector_http.enforce_http(false);
let mut proxy_connector = ProxyConnector::from_proxy(connector_http, proxy).unwrap();
proxy_connector.set_tls(Some(connector_tls));
proxy_connector
};Hopefully this will help with others in dealing with errors like the following when dealing with a self-signed cert in the chain: |
|
My code stopped working after upgrading aws-smithy-client from 0.46.0 to 0.47.0. let conn = aws_smithy_client::hyper_ext::Adapter::builder().build(env_proxy.clone().unwrap());
aws_sdk_kinesis::Client::from_conf_conn(
config_kinesis,
conn,
) |
We recently upgraded
|
Deleting Output of |
|
But it did reveal the issue:
It looks like your project is pulling in runtime crates from multiple versions. Those two crates should have the same version number. Could you post your |
[package]
name = "kinesis-client"
version = "0.1.0"
edition = "2021"
readme = "README.md"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
futures-util = "0.3.21"
log = "0.4"
serde = { version = "1.0.142", features = ["derive"] }
flate2 = "1.0.24"
aws-config = { version = "0.46.0", default-features = false }
aws-types = "0.46.0"
aws-sdk-kinesis = { version = "0.16.0", default-features = false }
aws-sdk-dynamodb = { version = "0.16.0", default-features = false }
dynomutex = { git = "ssh://git@private-git/dynomutex-rs.git", rev = "v0.1.7" }
rayon = "1.5.3"
tokio = { version = "1", features = ["full"] }
derive_builder = "0.11.2"
md5 = "0.7.0"
serde_dynamo = { version = "4.0.4", features = ["aws-sdk-dynamodb+0_16"] }
prost = "0.11.0"
async-trait = "0.1.57"
eyre = "0.6.8"
http = "0.2.8"
serde_with = "2.0.0"
aws-arn = "0.3.1"
metrics = "0.20.1"
# To make proxy work if there is no direct connection to AWS
hyper-proxy = "0.9.1"
hyper = "0.14.19"
aws-smithy-client = { version = "0.47.0", features = ["client-hyper"] }
headers = "0.3.7"
url = "2.2.2"
[dev-dependencies]
criterion = "0.3"
testcontainers = { version = "0.14.0" }
tracing = "0.1"
tracing-subscriber = "0.3"
[build-dependencies]
# Creates .proto files for Kinesis deaggregation
# prost-build = { version = "0.11.1" }
# protobuf-src = "1.0.5+3.19.3"
[features]
native-tls = ["aws-sdk-kinesis/native-tls", "aws-sdk-dynamodb/native-tls", "aws-smithy-client/native-tls", "aws-config/native-tls"]
default = ["native-tls"]Does that mean I have to wait until |
|
I completely forgot that we just implemented a thing to decouple our release versions from one another. Your issue may be caused by that. Let me confer with the rest of the team on what's happening here and I'll get back to you. In the mean time, try setting |
|
I checked with the rest of the team and downgrading |
|
with the following crate versions: Had to do the following for proxy support: |
|
In case you are having trouble with IMDS after upgrading to newer versions of to in @tshcpt 's code snippet. This makes sure the SDK queries IMDS for the instance making the request, not for the proxy. |
|
For reference, this is how I created proxied connections with the latest sdk. async fn main() {
/// ....
let shared_config = config_loader(credentials_provider, region, &profile.profile, proxy)
.load()
.await;
aws_sdk_iam::Client::new(&shared_config);
fetch_iam_roles_with_tags(&iam_client)
.await
.expect("Failed to fetch IAM roles")
}
/// Returns `ProxyConnector<HttpConnector>` if env. variable 'https_proxy' is set
pub fn determine_proxy() -> Option<ProxyConnector<HttpConnector>> {
let proxy_url: Url = env::var("https_proxy").ok()?.parse().ok()?;
let proxy_uri: Uri = env::var("https_proxy").ok()?.parse().ok()?;
let mut proxy = Proxy::new(Intercept::All, proxy_uri);
if let Some(password) = proxy_url.password() {
proxy.set_authorization(Authorization::basic(proxy_url.username(), password));
}
let connector = HttpConnector::new();
Some(ProxyConnector::from_proxy(connector, proxy).unwrap())
}
fn config_loader(
credentials_provider: AssumeRoleProvider,
region: Region,
profile: &str,
proxy: &Option<ProxyConnector<HttpConnector>>,
) -> aws_config::ConfigLoader {
let shared_config = aws_config::from_env()
.credentials_provider(credentials_provider)
.region(RegionProviderChain::first_try(region))
.profile_name(profile);
may_add_proxy(proxy, shared_config)
}
fn may_add_proxy(
proxy: &Option<ProxyConnector<HttpConnector>>,
shared_config: aws_config::ConfigLoader,
) -> aws_config::ConfigLoader {
match &proxy {
Some(proxy) => {
println!("Adding proxy");
let http_client = HyperClientBuilder::new().build(proxy.clone());
shared_config
.http_client(http_client)
.app_name(AppName::new("orbitwiz").expect("valid app name"))
}
None => shared_config,
}
} |
|
Since I just stumbled over that Problem and the snippet from @lcmgh seems to mostly work, just a question, where does the |
Dependency
|
|
I have no working solution for the latest v1 versions of the SDK. https://github.com/tafia/hyper-proxy is also not maintained anymore so it does not support hyper v1. |
|
yeah, I have hit the same issues and temporarily have given up on upgrading... |
|
this gives some hope seanmonstar/reqwest#2107 but no timeline. would be nice if aws could sponsor the development of such things.. |
|
Somebody aware of any workarounds? |
|
I found there is use hyper_proxy2::{Intercept, Proxy, ProxyConnector};
use url::Url;
use headers::Authorization;
use hyper::{Uri};
use hyper_util::client::legacy::connect::HttpConnector;
/// Returns `ProxyConnector<HttpConnector>` if env. variable 'https_proxy' is set
pub fn determine_proxy() -> Option<ProxyConnector<HttpConnector>> {
let proxy_url: Url = std::env::var("https_proxy").ok()?.parse().ok()?;
let proxy_uri: Uri = std::env::var("https_proxy").ok()?.parse().ok()?;
let mut proxy = Proxy::new(Intercept::All, proxy_uri);
if let Some(password) = proxy_url.password() {
proxy.set_authorization(Authorization::basic(proxy_url.username(), password));
}
let connector = HttpConnector::new();
Some(ProxyConnector::from_proxy(connector, proxy).unwrap())
}However I cannot get it working with aws_types::sdk_config::Builder
pub fn http_client(self, http_client: impl HttpClient + 'static) -> Selfwhen using let proxy = determine_proxy().unwrap();
let proxy = determine_proxy().unwrap();
let client = aws_smithy_runtime::client::http::hyper_014::HyperClientBuilder::new().build(proxy);
let aws_config = aws_config::load_from_env()
.await
.into_builder()
.http_client(client)
.region(Region::new("eu-central-1"))
.build();59 | let client = aws_smithy_runtime::client::http::hyper_014::HyperClientBuilder::new().build(proxy);
| ----- ^^^^^ the trait `tower_service::Service<http::uri::Uri>` is not implemented for `ProxyConnector<HttpConnector>`Is there no smithy HyperClientBuilder for v1? |
The SDK doesn't support hyper 1.x yet. Adding this support is being tracked in #977 |
|
Here is a solution that works with AWS SDK v1. hyper 0.14 can still be used in there (I assumed one has to use hyper v1). |
|
A Hyper 1.0 based client now exists: https://docs.rs/aws-smithy-experimental/latest/aws_smithy_experimental/hyper_1_0/struct.HyperClientBuilder.html |
|
I ended up providing custom |
Bug Report
Though the readme says that you can set the environment variables: http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY and the SDK will use them, it completely ignores them.
Version
Platform
AWS Services
Description
When providing https_proxy et. al environment variables, I expect it to use them, but it tries to connect directly instead:
instead, it tries to connect directly.
I used this code (Just trying to get an sts session_token):
When running with (the http_proxy et. al are already in the env):
I get:
The text was updated successfully, but these errors were encountered: