Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible issue when using Cvxportfolio's internet access through a VPN #129

Open
fengwz17 opened this issue Jan 27, 2024 · 4 comments
Open

Possible issue when using Cvxportfolio's internet access through a VPN #129

fengwz17 opened this issue Jan 27, 2024 · 4 comments

Comments

@fengwz17
Copy link

fengwz17 commented Jan 27, 2024 •
edited
Loading

  • OS: Ubuntu 22.04.1 LTS
  • Cvxportfolio version: 1.1.1
  • Python version: 3.10.12
  • Cvxpy version: 1.4.2
  • Pandas version: 2.2.0

I followed the instructions in Readme and run such commands:

pip install -U cvxportfolio
python3 -m cvxportfolio.tests

Then I got the following traceback:

.
======================================================================
ERROR: test_signature (cvxportfolio.tests.test_data.TestMarketData)
Test partial-universe signature of MarketData.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/tests/test_data.py", line 532, in test_signature
    md = DownloadedMarketData(['WM2NS'],
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 1303, in __init__
    self._get_market_data(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 1328, in _get_market_data
    database_accesses[stock] = self.datasource(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 103, in __init__
    self.update(grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 186, in update
    updated = self._download(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 567, in _download
    return self._internal_download(symbol)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 547, in _internal_download
    return pd.read_csv(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1024, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 618, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1618, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1878, in _make_engine
    self.handles = get_handle(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 728, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 384, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 289, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 1448, in connect
    super().connect()
  File "/usr/lib/python3.10/http/client.py", line 952, in connect
    self._tunnel()
  File "/usr/lib/python3.10/http/client.py", line 920, in _tunnel
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.10/http/client.py", line 256, in __init__
    self.fp = sock.makefile("rb")
AttributeError: '_socket.socket' object has no attribute 'makefile'

======================================================================
ERROR: test_user_provided_market_data (cvxportfolio.tests.test_data.TestMarketData)
Test UserProvidedMarketData.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/tests/test_data.py", line 440, in test_user_provided_market_data
    _ = UserProvidedMarketData(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 1234, in __init__
    self._add_cash_column(cash_key, grace_period=grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 951, in _add_cash_column
    data = Fred(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 103, in __init__
    self.update(grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 186, in update
    updated = self._download(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 574, in _download
    new = self._internal_download(symbol)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 547, in _internal_download
    return pd.read_csv(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1024, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 618, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1618, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1878, in _make_engine
    self.handles = get_handle(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 728, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 384, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
  File "/home/fengwz/.local/lib/python3.10/site-packages/pandas/io/common.py", line 289, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.10/http/client.py", line 1448, in connect
    super().connect()
  File "/usr/lib/python3.10/http/client.py", line 952, in connect
    self._tunnel()
  File "/usr/lib/python3.10/http/client.py", line 920, in _tunnel
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.10/http/client.py", line 256, in __init__
    self.fp = sock.makefile("rb")
AttributeError: '_socket.socket' object has no attribute 'makefile'

======================================================================
ERROR: test_yahoo_finance_cleaning (cvxportfolio.tests.test_data.TestMarketData)
Test our logic to clean Yahoo Finance data.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/tests/test_data.py", line 632, in test_yahoo_finance_cleaning
    data = YahooFinance("ENI.MI", base_location=self.datadir).data
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 103, in __init__
    self.update(grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 186, in update
    updated = self._download(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 486, in _download
    updated = self._get_data_yahoo(symbol, **kwargs)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 412, in _get_data_yahoo
    res = requests.get(
  File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 697, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 971, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self._tunnel()
  File "/usr/lib/python3.10/http/client.py", line 920, in _tunnel
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.10/http/client.py", line 256, in __init__
    self.fp = sock.makefile("rb")
AttributeError: '_socket.socket' object has no attribute 'makefile'

======================================================================
ERROR: test_yahoo_finance_errors (cvxportfolio.tests.test_data.TestMarketData)
Test errors with Yahoo Finance.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/tests/test_data.py", line 626, in test_yahoo_finance_errors
    YahooFinance("DOESNTEXIST", base_location=self.datadir)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 103, in __init__
    self.update(grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 186, in update
    updated = self._download(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 486, in _download
    updated = self._get_data_yahoo(symbol, **kwargs)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 412, in _get_data_yahoo
    res = requests.get(
  File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 697, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 971, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self._tunnel()
  File "/usr/lib/python3.10/http/client.py", line 920, in _tunnel
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.10/http/client.py", line 256, in __init__
    self.fp = sock.makefile("rb")
AttributeError: '_socket.socket' object has no attribute 'makefile'

======================================================================
ERROR: setUpClass (cvxportfolio.tests.test_simulator.TestSimulator)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/tests/test_simulator.py", line 43, in setUpClass
    cls.market_data_1 = cvx.DownloadedMarketData(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 1303, in __init__
    self._get_market_data(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 1328, in _get_market_data
    database_accesses[stock] = self.datasource(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 103, in __init__
    self.update(grace_period)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 186, in update
    updated = self._download(
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 486, in _download
    updated = self._get_data_yahoo(symbol, **kwargs)
  File "/home/fengwz/.local/lib/python3.10/site-packages/cvxportfolio/data.py", line 412, in _get_data_yahoo
    res = requests.get(
  File "/usr/lib/python3/dist-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 697, in urlopen
    self._prepare_proxy(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 971, in _prepare_proxy
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self._tunnel()
  File "/usr/lib/python3.10/http/client.py", line 920, in _tunnel
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.10/http/client.py", line 256, in __init__
    self.fp = sock.makefile("rb")
AttributeError: '_socket.socket' object has no attribute 'makefile'

----------------------------------------------------------------------
Ran 107 tests in 29.056s

FAILED (errors=5, skipped=2)

From the traceback info, it seems like there are some problems in pands.read_csv() or something, but if I test it as following:

$ python3 
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
<stdin>:1: DeprecationWarning: 
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
        
>>> pd.read_csv("test.csv")
Empty DataFrame
Columns: [1]
Index: []

It appears that pandas.read_csv() itself no problems.. Then I am not sure if there is something else that causing such issues or not, thanks for the help.

BTW, I am in Beijing, China. Not sure if it is caused by network issues..
@enzbus
Copy link
Collaborator

enzbus commented Jan 27, 2024 •
edited
Loading

Hello, it appears there are issues with low-level internet access Python modules. Is the machine you ran this on connected to the internet? You can use Cvxportfolio without internet access (look at the documentation of UserProvidedMarketData) but the tests assume you have a working connection. And no, I don't think there's any missing package there.

Or maybe you have some proxy or firewall configuration that prevents that machine to access the two http addresses (Yahoo Finance and Fred) those tests were targeting ?

@enzbus
Copy link
Collaborator

enzbus commented Jan 27, 2024

Re: Chinese firewall. That probably explains it, the pd.read_csv in question targets a http resource (you can see it in the source of Fred). If you have access to a VPN you may retry...

@fengwz17
Copy link
Author

Re: Chinese firewall. That probably explains it, the pd.read_csv in question targets a http resource (you can see it in the source of Fred). If you have access to a VPN you may retry...

Thanks for the help! I use VPN and seems I am able to access Yahoo Finance and Fred website, also successfully ping these main page links but still face the same issues. I agree it should be some issues with low-level internet access Python modules or some complicate/strange internet proxy configuration problems...

I tests some local examples successfully, and I have decided to give up trying the online tests. Thanks again for your quick and kindly reply :)

@enzbus
Copy link
Collaborator

enzbus commented Jan 27, 2024

Thank you! I'm not sure how to help, for reference one of the two lines on which it fails is something like this:

import pandas as pd
pd.read_csv("https://alfred.stlouisfed.org/graph/alfredgraph.csv?id=CES0500000003&vintage_date=2023-07-06")

If you are able to open that same address in your browser then there's probably something wrong with how the VPN is set up, and beyond my area of expertise...

@enzbus enzbus changed the title _socket.socket has no attribute makefile Possible issue when using Cvxportfolio's internet access through a VPN Jan 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fengwz17 @enzbus