WIP: Preliminary investigation: updating to python 3

.github/workflows/black.yml

@@ -2,9 +2,11 @@ name: black
 
 on:
   push:
-    branches: [master, ]
+    branches:
+      - '**'
   pull_request:
-    branches: [master]
+    branches:
+      - 'master'
 
 jobs:
   black:

.github/workflows/codeql-analysis.yml

@@ -2,10 +2,12 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [master, ]
+    branches:
+      - '**'
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [master]
+    branches:
+      - 'master'
   schedule:
     - cron: '0 2 * * 0'
 

.github/workflows/sentry-release.yml

@@ -2,7 +2,8 @@ name: sentry-release
 
 on:
   push:
-    branches: [master, ]
+    branches:
+      - '**'
 
 jobs:
   createSentryRelease:

.github/workflows/tests.yml

@@ -2,9 +2,11 @@ name: docker build and test
 
 on:
   push:
-    branches: [master, ]
+    branches:
+      - '**'
   pull_request:
-    branches: [master]
+    branches:
+      - 'master'
 
 jobs:
   build:

.gitignore

@@ -1,3 +1,6 @@
+.venv
+*.egg-info
+
 # no compiled files!
 *.class
 *.pyc

UpdatingToPython3-8.md

@@ -0,0 +1,85 @@
+# Updating to Python 3.8
+
+Preliminary caveat: I haven't tested running the codebase after these changes, this is an introductory investigation. Currently the `reporters-db` dependency is the blocker to actually running the `manage.py`.
+
+## Dependencies
+
+Dependencies that needed updating:
+
+- `pydot` version `1.1.0` is not compatible with Python 3. Update to version `20.9.0`.
+- `pandas` version `0.18.1` is not compatible with Python 3(.8 ?). Update to version `1.1.2`. This also requires updating `python-dateutil` from `2.5.0` to `>=2.7.3` - updated to `2.8.1`.
+- `pycopg2` version `2.7.7` needs updating. Update to `2.8.6`. Note that this requires `export CONFIGURE_OPTS="CPPFLAGS=-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib"` on macOS.
+- `wsgiref` is now built into Python 3, so isn't necessary to include as a dependency.
+-` enum34` isn't necessary anymore.
+- it's not possible to install `seal-rookery` from PyPI due to missing `sdist` distributions. As workaround, install directly from GitHub.
+- `reporters-db` can't be installed because it relies on `python-dateutil==2.5.0`.
+- `mock` is no longer needed as this is part of the standard library in `unittest.mock` as of py3.3.
+
+## Code changes
+
+Updating of the actual code:
+
+- Run 2to3 and hand-modify the output to remove false positives and other automatic errors. It picked up a false positive around importing `from celery import Celery` when there was a local file called `celery.py`, and there were a bunch of false positives where 2to3 converts `for x, y in dict.values()` to `for x, y in list(dict.values())` which is unnecessary and inefficient. Other than that the results were pretty good. The main consideration going forwards is that somewhere there's a comparison between a `b"some bytes string"` and a `"some unicode string"` which can't be found at runtime but will never be equal. More details of 2to3 changes below.
+- In `10-public.py`, the `maxBytes` setting needs to be changed to an int, not a string.
+- `django.utils.encoding.smart_unicode` was renamed to `django.utils.encoding.smart_text` but it's actually just an alias to `django.utils.encoding.force_str`, so may as well just change it to that.
+- Any instances of `import mock` or `from mock import ...` need to be trivially changed to `from unittest import mock` and `from unittest.mock import ...`.
+
+### 2to3
+
+This summaries the automatic conversions that 2to3 can apply:
+
+- Remove `u"blah"` from strings to just be `"blah"` - in Python 3 these are equivalent.
+- Change `isinstance(x, basestring)` to `isinstance(x, str)` as `basestring = (str, unicode)` doesn't exist in py3.
+- Change `unicode(...)` to `str(...)`.
+- Remove `from __future__ import unicode_literals`.
+- Change `raw_input()` to `input()` and `input()` to `eval(input())` (note: no-where in this repo is the `eval()` bit required, so best to leave it off for obvious security reasons).
+- Change `print "blah"` to `print("blah")` and `print "foo",` to `print("foo", end=" ")`.
+- Remove `from __future__ import absolute_import`.
+- Convert py2 method of importing local files `from local_file import myfunction` to py3 version: `from .local_file import myfunction`. You need to be careful that 2to3 doesn't get false positives where name of local file == local of repository, which happened once here.
+- Replace `xrange()` with `range()`.
+- Replace `execfile()` with `exec(compile(open("file.py").read(), "file.py", "exec"))`.
+- Some functions like `dict.values()`, `dict.items()`, etc. return iterators instead of lists. Sometimes (but not always), you need to convert these to lists yourself to avoid strange bugs. For instance, `for x, y in d.items()` isn't necessary because you're immediately consuming the iterator, but if you do `d = {"a": 1}; x = d.items(); l1 = len(x); l2 = len(x)` then `l1 == 1` and `l2 == 0` because the first `len(x)` consumes the iterator meaning there's nothing left in it. Figuring out which one is immediately consumed and which isn't just needs to be done by hand.
+- Change stdlib library names:
+  - `httplib` => `http.client`
+  - `HTMLParser` => `html.parser`
+  - `StringIO => io.StringIO`
+  - etc.
+- Change `long` to `int`, as py2 `long`s are the same as py3 `int`s.
+- Starting number with `0`, e.g. `07` is not valid in py3. Either use `0o7` for an octal or `7` for a decimal.
+- Change dunder methods:  `__nonzero__` to `__bool__` and `__unicode__` to `__str__`.
+- `reduce` needs to be imported from `functools`.
+
+There's a bunch of other things too but they don't appear in the courtlistener codebase.
+
+## Poetry
+
+To install dependencies with Poetry, run:
+
+```bash
+poetry install
+
+# If you don't want dev dependencies, you have to explicitly say so.
+poetry install --no-dev
+
+# If you want the extras, you need to say so.
+poetry install -E flp
+```
+
+Then you can run the `manage.py` command using either:
+
+```bash
+poetry run manage.py
+# OR
+cl-manage
+```
+
+If you invoke `poetry` without being in a virtual environment, poetry will create one for you automatically and use that. If you're already activated, it'll use the venv that you've created. I prefer to create my own because it gives you more control of where the venv is:
+
+```bash
+python -m venv .venv
+source ./.venv/bin/activate
+poetry install
+
+# cl-manage has now been installed into the virtualenv
+which cl-manage
+```

cl/__init__.py

@@ -1,5 +1,3 @@
-from __future__ import absolute_import
-
 # This will make sure the app is always imported when
 # Django starts so that shared_task will use this app.
 from .celery import app as celery_app

cl/alerts/forms.py

@@ -21,8 +21,8 @@ def clean_rate(self):
             # Somebody is trying to hack past the JS/HTML block on the front
             # end. Don't let them create the alert until they've donated.
             raise ValidationError(
-                u"You must donate more than $10 per year to create Real Time "
-                u"alerts."
+                "You must donate more than $10 per year to create Real Time "
+                "alerts."
             )
         else:
             return rate

cl/alerts/management/commands/handle_old_docket_alerts.py

@@ -110,9 +110,7 @@ class Command(VerboseCommand):
             │
    T+187-∞ ─┴─ Item terminated more than 180 days ago and
                alert is disabled.
-""".decode(
-        "utf-8"
-    )
+"""
 
     def create_parser(self, *args, **kwargs):
         parser = super(Command, self).create_parser(*args, **kwargs)

cl/alerts/migrations/0001_initial.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import models, migrations
 

cl/alerts/migrations/0002_alert_user.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import models, migrations
 from django.conf import settings

cl/alerts/migrations/0003_auto_20150807_1826.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import models, migrations
 import datetime

cl/alerts/migrations/0004_auto_20150910_1450.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import models, migrations
 

cl/alerts/migrations/0005_remove_alert_always_send_email.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 

cl/alerts/migrations/0006_add_alert_secret_key.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 

cl/alerts/migrations/0007_populate_alert_secret_key.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import migrations
 from django.utils.crypto import get_random_string

cl/alerts/migrations/0008_add_docket_alerts.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.conf import settings
 from django.db import migrations, models

cl/alerts/migrations/0009_add_date_last_hit.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 

cl/alerts/migrations/0010_noop_update_help_text.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Generated by Django 1.11.29 on 2020-04-25 00:02
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 

cl/alerts/migrations/0011_add_recap_docket_search_type_noop.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 # Generated by Django 1.11.29 on 2020-07-22 23:27
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 

cl/alerts/models.py

@@ -55,8 +55,8 @@ class Alert(models.Model):
         max_length=40,
     )
 
-    def __unicode__(self):
-        return u"Alert %s: %s" % (self.pk, self.name)
+    def __str__(self):
+        return "Alert %s: %s" % (self.pk, self.name)
 
     class Meta:
         ordering = ["rate", "query"]
@@ -99,8 +99,8 @@ class DocketAlert(models.Model):
     class Meta:
         unique_together = ("docket", "user")
 
-    def __unicode__(self):
-        return u"DocketAlert %s: %s" % (self.pk, self.docket_id)
+    def __str__(self):
+        return "DocketAlert %s: %s" % (self.pk, self.docket_id)
 
     def save(self, *args, **kwargs):
         """Ensure we get a token when we save the first time."""

cl/alerts/tests.py

@@ -40,7 +40,7 @@ def test_create_alert(self):
             reverse("show_results"), self.alert_params, follow=True
         )
         self.assertEqual(r.redirect_chain[0][1], 302)
-        self.assertIn("successfully", r.content)
+        self.assertIn("successfully", r.content.decode())
         self.client.logout()
 
     def test_fail_gracefully(self):
@@ -54,7 +54,7 @@ def test_fail_gracefully(self):
         )
         r = self.client.post("/", bad_alert_params, follow=True)
         self.assertEqual(r.status_code, 200)
-        self.assertIn("error creating your alert", r.content)
+        self.assertIn("error creating your alert", r.content.decode())
         self.client.logout()
 
     def test_new_alert_gets_secret_key(self):

cl/api/tests.py

@@ -1,5 +1,5 @@
 # coding=utf-8
-from __future__ import print_function
+
 import json
 import shutil
 from datetime import timedelta, date
@@ -913,15 +913,15 @@ class BulkDataTest(TestCase):
 
     def setUp(self):
         docket = Docket(
-            case_name=u"foo",
+            case_name="foo",
             court=Court.objects.get(pk="test"),
             source=Docket.DEFAULT,
         )
         docket.save()
         # Must be more than a year old for all tests to be runnable.
         last_month = now().date() - timedelta(days=400)
         self.doc_cluster = OpinionCluster(
-            case_name=u"foo", docket=docket, date_filed=last_month
+            case_name="foo", docket=docket, date_filed=last_month
         )
         self.doc_cluster.save(index=False)
         opinion = Opinion(cluster=self.doc_cluster, type="Lead Opinion")

cl/api/views.py

@@ -1,5 +1,6 @@
 import logging
 
+import scorched
 from django.conf import settings
 from django.http import JsonResponse
 from django.shortcuts import render, get_object_or_404
@@ -45,8 +46,10 @@ def annotate_courts_with_counts(courts, court_count_tuples):
 
 def make_court_variable():
     courts = Court.objects.exclude(jurisdiction=Court.TESTING_COURT)
-    conn = sunburnt.SolrInterface(settings.SOLR_OPINION_URL, mode="r")
-    response = conn.raw_query(**build_court_count_query()).execute()
+    # conn = sunburnt.SolrInterface(settings.SOLR_OPINION_URL, mode="r")
+    # response = conn.raw_query(**build_court_count_query()).execute()
+    si = scorched.SolrInterface(settings.SOLR_OPINION_URL, mode="r")
+    response = si.query(**build_court_count_query()).execute()
     court_count_tuples = response.facet_counts.facet_fields["court_exact"]
     courts = annotate_courts_with_counts(courts, court_count_tuples)
     return courts

cl/audio/feeds.py

@@ -16,13 +16,13 @@ class JurisdictionPodcast(JurisdictionFeed):
     )
     subtitle = description
     summary = description
-    iTunes_name = u"Free Law Project"
-    iTunes_email = u"[email protected]"
+    iTunes_name = "Free Law Project"
+    iTunes_email = "[email protected]"
     iTunes_image_url = (
-        u"https://www.courtlistener.com/static/png/producer-2000x2000.png"
+        "https://www.courtlistener.com/static/png/producer-2000x2000.png"
     )
-    iTunes_explicit = u"no"
-    item_enclosure_mime_type = u"audio/mpeg"
+    iTunes_explicit = "no"
+    item_enclosure_mime_type = "audio/mpeg"
 
     def title(self, obj):
         return "Oral Arguments for the %s" % obj.full_name
@@ -44,15 +44,15 @@ def items(self, obj):
 
     def feed_extra_kwargs(self, obj):
         extra_args = {
-            "iTunes_name": u"Free Law Project",
-            "iTunes_email": u"[email protected]",
-            "iTunes_explicit": u"no",
+            "iTunes_name": "Free Law Project",
+            "iTunes_email": "[email protected]",
+            "iTunes_explicit": "no",
         }
         if hasattr(obj, "pk"):
-            path = u"/static/png/producer-%s-2000x2000.png" % obj.pk
+            path = "/static/png/producer-%s-2000x2000.png" % obj.pk
         else:
             # Not a jurisdiction API -- A search API.
-            path = u"/static/png/producer-2000x2000.png"
+            path = "/static/png/producer-2000x2000.png"
         extra_args["iTunes_image_url"] = (
             "https://www.courtlistener.com%s" % path
         )
@@ -63,7 +63,7 @@ def item_extra_kwargs(self, item):
         return {
             "author": get_item(item)["court"],
             "duration": str(item["duration"]),
-            "explicit": u"no",
+            "explicit": "no",
         }
 
     def item_enclosure_url(self, item):

cl/audio/migrations/0001_initial.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from __future__ import unicode_literals
+
 
 from django.db import migrations, models
 import cl.lib.model_helpers