Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duckduckgo issue #404

Open
gleitz opened this issue Jul 9, 2021 · 3 comments
Open

Duckduckgo issue #404

gleitz opened this issue Jul 9, 2021 · 3 comments
Assignees
Labels

Comments

@gleitz
Copy link
Owner

gleitz commented Jul 9, 2021

What happened:

Duckduckgo seems to be blocking requests

What you expected to happen:

Request should succeed

Output with --explain

» python -m howdoi make a pizza --engine duckduckgo --explain
INFO: Version: 2.0.16
INFO: Fetching answers for query: make a pizza
INFO: Searching duckduckgo with URL: https://duckduckgo.com/html?q=site:stackoverflow.com%20make%20a%20pizza&t=hj&ia=web
INFO: Received HTTPError
ERROR: Unable to find an answer because the search engine temporarily blocked the request. Please wait a few minutes or select a different search engine.
Traceback (most recent call last):
  File "/Users/gleitz/.homebrew/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/gleitz/.homebrew/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/gleitz/projects/howdoi/howdoi/__main__.py", line 3, in <module>
    command_line_runner()
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 788, in command_line_runner
    utf8_result = howdoi(args).encode('utf-8', 'ignore')
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 615, in howdoi
    res = _get_answers(args)
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 418, in _get_answers
    question_links = _get_links_with_cache(args['query'])
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 397, in _get_links_with_cache
    links = _get_links(query)
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 282, in _get_links
    raise BlockError('Temporary block by search engine')
howdoi.howdoi.BlockError: Temporary block by search engine

When I open the page referenced, I occasionally get the error

If this error persists, please let us know: [email protected]
@gleitz gleitz added the bug label Jul 9, 2021
ShezaMunir added a commit to ShezaMunir/howdoi that referenced this issue Jul 12, 2021
Comment out DDG test for nose test for PR to pass. Enable this when issue gleitz#404 is resolved
@ShezaMunir
Copy link
Contributor

Reminder: once this issue is resolved, enable DDG test case in test_howdoi.py

@SaurusXI
Copy link
Contributor

@gleitz perhaps using a headless browser like puppeteer or selenium might fix this problem? Not sure how it would affect performance, but might be worth looking into.

@alirazacodes
Copy link

Hi @gleitz, I would like to contribute to resolving this issue;

  • I have done some research and I think if we use some data scrapping python libraries like scrappy, beautifulsoap etc
  • I can also use request-html along with selenium for better data rendering
  • Another method that also came along the way of my research was by using API like SerpApi

Please do assign me this issue, I have already started working on it, having some issues while running howdoi in my local machine due to some python packages being broken while installation which will be resolved by today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants