Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Preliminary investigation: updating to python 3 #1407

Closed

Commits on Sep 15, 2020

  1. feat(BTE):Init cl-binary-transformers-and-extractors

    Add new docker image to CL
    Add simple test to begin replacing weird binaries
    flooie committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    7896fe9 View commit details
    Browse the repository at this point in the history
  2. refactor(make_png_thumbnail..): Use docker BTE

    Use our new docker container to generate the thumbnail
    and return the contents
    flooie committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    36f3b8f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f952859 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2020

  1. refactor(scrapers/tasks): Remove subprocess methods

    Refactor subprocess into our new BTE container
    flooie committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    40053f2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8db9b46 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c8775b9 View commit details
    Browse the repository at this point in the history
  4. refactor(get_page_count): Add extension param

    Not sure its necessary or used in any way
    The method only extracts page count for PDFs
    prior to this refactor.
    flooie committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    ee42731 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3894abc View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    1983f5d View commit details
    Browse the repository at this point in the history
  7. refactor(docker): Update compose files

    Add volumes to new containers to task server
    Remove unneeded installs to task server
    flooie committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    3576775 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2020

  1. refactor(scraper.tasks):Update calls for docker changes

    Update tests
    flooie committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    f6cfa5e View commit details
    Browse the repository at this point in the history
  2. refactor(tests.yml):Update tests.yml

    Currently tests.yml is not running
    this is an attempt to get it working in
    a new syntax
    flooie committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    cbfa9d4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    dec2f1b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fd4781f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    716e6a8 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    56f0cb7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4cefe43 View commit details
    Browse the repository at this point in the history
  8. tests(cl/docker-compose): Test bte_beta containers

    Built and pushed beta containers for Django
    and the celery task-server.
    
    Task server is imported from django image and
    neither has the 'weird binaries'.
    
    The new images are 36% of the old ones
    flooie committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    aca26b5 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    19a3aed View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    06d9be7 View commit details
    Browse the repository at this point in the history
  11. refactor(docker-compose): Add env variables

    Add JUDGE_PICS_PATH
    add SEAL_ROOKERY_PATH
    flooie committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    581b51b View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    87eaee6 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2020

  1. Configuration menu
    Copy the full SHA
    2de8519 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2773d96 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    effa9ee View commit details
    Browse the repository at this point in the history
  4. style(black): Black last push

    flooie committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    e594219 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a1dc92e View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    ae2a90b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    53faffc View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2020

  1. fix(settings): Add Binary Transformers & Extractors

    Add BTE URL to settings and point to it in functions
    flooie committed Sep 19, 2020
    Configuration menu
    Copy the full SHA
    9446c20 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6dd8f2c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    128ac81 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2020

  1. refactor(get_page_count): Update get_page_count params

    Remove extension from get_page_count
    flooie committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    ba4d61e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9f71d61 View commit details
    Browse the repository at this point in the history
  3. refactor(generate_thumbnail): Change error handling

    Remove error catching from generate thumnail method.
    Move exceptions to the calling code.
    flooie committed Sep 21, 2020
    Configuration menu
    Copy the full SHA
    e9c4125 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    91b7937 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2020

  1. Configuration menu
    Copy the full SHA
    442675b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f167de3 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2020

  1. fix(transformer_extractor_utils): Add filename discovery to post corr…

    …ect file type to bte
    flooie committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    ffaf8f1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a71a001 View commit details
    Browse the repository at this point in the history
  3. fix(tests.py): Update document extraction test

    Check to validate successful response from BTE
    flooie committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    0e87056 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    50831fe View commit details
    Browse the repository at this point in the history
  5. fix(scrapers/tests): Fix test_txt_extraction_with_bad_data

    Update test to reflect BTE changes
    flooie committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    32ddcd8 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. Configuration menu
    Copy the full SHA
    911948f View commit details
    Browse the repository at this point in the history
  2. 2to3: convert to py3 unicode handling.

    Remove `u` prefix from unicode strings - all strings are unicode.
    
    Change `unicode(...)` to `str(...)`.
    
    Change `isinstance(x, basestring)` to `isinstance(x, str)`.
    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    5b5f904 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4e407d5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4bb8e13 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4bab940 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d87fdc5 View commit details
    Browse the repository at this point in the history
  7. 2to3: Convert iterators to lists where necessary.

    Some functions, particularly those on dicts like `d.items()` and
    `d.values()` return iterators, not lists. In some circumstances (i.e.
    when you're not immediately iterating through the results) you need to
    construct the list with `list(...)` to work with py3.
    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    463cad5 View commit details
    Browse the repository at this point in the history
  8. Move to py3 imports.

    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    1185caa View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    760cc1e View commit details
    Browse the repository at this point in the history
  10. Library replacements: StringIO, httplib, HTMLParser, urlparse

    httplib => http.client
    HTMLParser => html.parser
    cStringIO, StringIO => io.StringIO
    urlparse => urllib.parse
    urllib2 => urllib.error
    izip => builtin zip
    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    be5bc1e View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    dde5826 View commit details
    Browse the repository at this point in the history
  12. Starting number with '0' is invalid.

    I didn't think this was a Python 2 vs. 3 thing but hey.
    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    dbeba70 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    64a1962 View commit details
    Browse the repository at this point in the history
  14. Convert dunders to py3: __bool__, __str__.

    __nonzero__ => __bool__
    __unicode__ => __str__
    drewsilcock committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    4e4deec View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    e0ea84c View commit details
    Browse the repository at this point in the history
  16. feat(convert_audio): Update convert audio BTE call

    Update method to call for audio conversion.
    Includes a serializer for passing the necessary information to BTE.
    flooie committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    197450b View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2020

  1. Configuration menu
    Copy the full SHA
    6865a0e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6ff9832 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c65ba73 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7cecd27 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    971f36e View commit details
    Browse the repository at this point in the history
  6. Format with Black.

    drewsilcock committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    78d7734 View commit details
    Browse the repository at this point in the history
  7. Rename submodule ca9-briefs to ca9_briefs.

    Using dashes in submodule names isn't valid Python (2 or 3, I think).
    drewsilcock committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    4529517 View commit details
    Browse the repository at this point in the history
  8. refactor(req.txt): Remove unused imports

    Seal-rookery, and eyed3 are not needed anymore
    flooie committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    f2151d9 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    edfe701 View commit details
    Browse the repository at this point in the history
  10. refactor(scrapers/tests.py): Update tests

    add test_BTE_audio_conversion for audio conversion and metadata addition
    flooie committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    2a20c2d View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    51f26da View commit details
    Browse the repository at this point in the history
  12. refactor(req.txt): Remove Pillow and PyPdf2

    No longer used in CL
    flooie committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    930267c View commit details
    Browse the repository at this point in the history
  13. Merge branch 'pr/1407' into python38-merger

    # Conflicts:
    #	cl/scrapers/tasks.py
    #	cl/search/management/commands/cl_count_recap_pages.py
    flooie committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    efc0d2b View commit details
    Browse the repository at this point in the history
  14. feat(CL):Add python3.8 support

    Building on @drewsilcock I merged his branch
    with the dockerize weird binaries modified juriscraper slightly and voila.
    
    I havent tested it out but it will load the homepage and search solr.
    flooie committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    d082862 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2020

  1. Tweaks to get tests to partially run

    Include changes to migration files help text encoding
    flooie committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    e3207da View commit details
    Browse the repository at this point in the history
  2. Tweaks to get tests to partially run

    Include changes to migration files help text encoding
    flooie committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    d970773 View commit details
    Browse the repository at this point in the history
  3. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    ae9989f View commit details
    Browse the repository at this point in the history
  4. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    515d6c9 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 5, 2020
    Configuration menu
    Copy the full SHA
    a558eb9 View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. Configuration menu
    Copy the full SHA
    df07471 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b2e61bb View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5d18148 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    94c42a5 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    a135f03 View commit details
    Browse the repository at this point in the history
  6. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    5afe86d View commit details
    Browse the repository at this point in the history
  7. Merge branch 'python38-merger' of github.com:freelawproject/courtlist…

    …ener into python38-merger
    flooie committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    ec66de4 View commit details
    Browse the repository at this point in the history
  8. refactor(text_filters): Integer division py2-> py3

    use // instead of / for integer division
    flooie committed Oct 6, 2020
    Configuration menu
    Copy the full SHA
    95942b5 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    c152ad9 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    b293dad View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. refactor(search_utils): Switch sunburnt for scorched

    In cleanup main query and get related clusters with cache
    replaced sunburnt with scorched
    flooie committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    813752f View commit details
    Browse the repository at this point in the history
  2. feat(req.txt): Upgrade req txt file

    Upgrade scorched, ipython, cryptography and comment out psycopg2
    flooie committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    76119c4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    804b246 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f57245d View commit details
    Browse the repository at this point in the history
  5. refactor(cl): Replace magic with magic

    Remove old magic code with python magic, add to req.
    flooie committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    32e025f View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5a2f288 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    76b7d43 View commit details
    Browse the repository at this point in the history
  8. Change dependency specification from pip to poetry.

    Where pip needs `==a.b.*,>=a.b.c`, poetry can just do `^a.b.c`. Strictly
    speaking this isn't exactly the same as `^a.b.c` will satisfy any
    `a.d.e` where d > b and e > c. This makes the versioning more reliable,
    because the pip version could install major package updates without you
    realising.
    drewsilcock committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    36dd7f9 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    01e79a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2020

  1. Configuration menu
    Copy the full SHA
    ff9ccb8 View commit details
    Browse the repository at this point in the history