This repo provides starter code and documentation for new Councilmatic instances.
The councilmatic family is a set of web apps for keeping tabs on city representatives and their legislative activity.
Councilmatic started as a Code for America project by Mjumbe Poe, who designed the earliest version of a Councilmatic site – Councilmatic Philadelphia. DataMade then implemented Councilmatic in New York City (in partnership with the Participatory Politics Foundation), Chicago (in partnership with the Sunlight foundation), and Los Angeles (in partnership with the Los Angeles County Metropolitan Transportation Authority).
To simplify redeployment of Councilmatic instances, DataMade identified two necessities: (1) Open Civic Data, a data standard for describing people, organizations, events, and bills, and (2) the abstraciton of core functionality into a separate app. And so, DataMade built django-councilmatic
– a django app with base functionality, common across all cities, which implements the OCD data standard.
With this template, you can create a Councilmatic site for your own city, built using the django-councilmatic
app. Read on!
- Python 3.4+
- PostgreSQL 9.4+
git clone https://github.com/datamade/councilmatic-starter-template.git yourcity_councilmatic
We recommend using virtualenv and virtualenvwrapper for working in a virtualized development environment. Read how to set up virtualenv.
Once you have virtualenvwrapper set up, run the following commands in your terminal:
cd yourcity_councilmatic
mkvirtualenv councilmatic
pip install -r requirements.txt
Afterwards, whenever you want to use this virtual environment, run:
workon yourcity_councilmatic
Before you can run the website, you need to create a database and install PostGIS.
createdb yourcity_councilmatic
psql -d yourcity_councilmatic -c "CREATE EXTENSION postgis"
Then, run migrations. (Be sure that you are "working on" the correct virtual environment.)
python manage.py migrate --no-initial-data
Finally, create an admin user. Set a username and password when prompted.
python manage.py createsuperuser
django-councilmatic
leverages, and in some instances, lightly extends the Open Civic Data standard, implemented in Django as python-opencivicdata
. The next step toward running a new Councilmatic instance is to locate data about your city and use it to populate these OCD models.
How you do that is up to you. At DataMade, we maintain a series of external municipal scrapers that retrieve data from Legistar-backed sites.
If your city runs a Legistar-backed site, see scrapers-us-municipal
for several examples of scrapers that will populate a given database with legislative data in the format required by Councilmatic. These scrapers leverage python-legistar-scraper
to scrape Legistar and the pupa
framework to shape and import data.
If your city does not run a Legistar-backed site, we still recommend using pupa
as a legislative scraping framework. You can add your scrapers to your Councilmatic repository, or compose them elsewhere.
Next, install and create a settings file for pupa
–
pip install pupa
touch pupa_settings.py
– and add the following settings:
# Leave these blank
OCD_CITY_COUNCIL_NAME = ''
CITY_COUNCIL_NAME = ''
STATIC_PATH = ''
INSTALLED_APPS = (
'django.contrib.contenttypes',
'opencivicdata.core.apps.BaseConfig',
'opencivicdata.legislative.apps.BaseConfig',
'pupa',
'councilmatic_core'
)
# Change this if you called your database something different
DATABASE_URL = 'postgres:///yourcity_councilmatic'
Finally, initialize pupa
.
pupa dbinit us
pupa init YOUR_CITY_SCRAPER
Once you've filled out the prompts, you will see a new directory called YOUR_CITY_SCRAPER
. Inside, you'll find the scaffolding for the necessary scrapers to populate the database.
Define the scrape
methods in each of the resulting scraper classes, then run pupa update YOUR_CITY
to import data. If you aren't sure where to start, the City Scrapers project includes many examples of scraping all sorts of government websites.
N.b., if you wish to make any changes to the models, see the django-councilmatic
README for information on approaches.
django-councilmatic
comes with management commands to add headshots and geography shapes to Person
objects and Post
objects, respectively.
If you've imported people with headshot URLs, run update_headshots
to download the images and store them where Django knows where to find them.
python manage.py update_headshots
Likewise, if you've imported Post
objects and you'd like to include geographic boundaries that pertain to them, create a GeoJSON file containing shapes for each division represented in your instance, then run the import_shapes
command. See the GeoJSON file in chi-councilmatic
for an example of the expected format.
python manage.py import_shapes /path/to/your/shapes.geojson
Now that you have the data, set up your instance.
Inside the git repository that you cloned above, you should see a folder called city
. Rename this folder to something that makes sense for your project, e.g., "chicago."
mv city yourcity
Now, update the main settings file - councilmatic/settings.py
. First, in INSTALLED_APPS
, add the name of the folder that you just renamed:
INSTALLED_APPS = (
...
~'city',~
'yourcity',
)
Then, change the TIME_ZONE. You can use this list to find the correct formatting.
TIME_ZONE = 'America/Chicago'
Look for councilmatic/settings_jurisdiction.py
. This settings file tells your Councilmatic instance how to populate different parts of the UI and get fresh data from the OCD database. The following table explains why and how to adjust these values.
Name | Description | Example value | Optional? |
---|---|---|---|
OCD_JURISDICTION_IDS |
An array of the OCD IDs of jurisdictions covered by your Councilmatic instance. Query the |
['ocd-jurisdiction/country:us/state:il/place:chicago/government'] | No |
OCD_CITY_COUNCIL_NAME |
This identifies the legislative body you'll be tracking. It should correspond to an Organization in your database. Query the opencivicdata_organization table (or opencivicdata.legislative.Organization model in the ORM) for options.
|
Chicago City Council | No |
CITY_COUNCIL_NAME | Display name for the legislative body you'll be tracking. | Chicago City Council | No |
LEGISLATIVE_SESSIONS | A list of years that tells Councilmiatic when a new body is elected and which of these sessions you have data for in your OCD database. | ['2007', '2011', '2015', '2019'] | No |
CITY_NAME | Complete, pretty name of the jurisdiction | New York City | No |
CITY_NAME_SHORT | Shortened version of the jurisdiction name | NYC | No |
CITY_VOCAB | A dictionary telling Councilmatic how to refer to different entities in the jurisdiction | (See individual values below) | No |
CITY_VOCAB['MUNICIPAL_DISTRICT'] | The name of the city sections, which could be a "ward", "sector", "district", etc. | Ward | No |
CITY_VOCAB['SOURCE'] | The name of the entity responsible for the source data that your Councilmatic will be based upon. | Chicago City Clerk | No |
CITY_VOCAB['COUNCIL_MEMBER'] | The name of a member of the legislative body that your Councilmatic tracks. | Alderman | No |
CITY_VOCAB['COUNCIL_MEMBERS'] | Plural form of the name of a member of the legislative body your Councilmatic tracks. | Aldermen | No |
CITY_VOCAB['EVENTS'] | The name of the meetings that members of the legislative body attend. | Meetings | No |
APP_NAME | The name of the folder where your jurisdiction specific code lives. | chicago | No |
SITE_META |
Dictionary containing values stored in meta tags.
|
(See individual values below) | Yes |
SITE_META['site_name'] |
Used for the og:name , og:title , and twitter:title meta tags
|
Chicago Councilmatic | Yes |
SITE_META['site_desc'] |
Used for the description , og:description , and twitter:description meta tags
|
City Council, demystified. Keep tabs on Chicago legislation, aldermen, & meetings. | Yes |
SITE_META['site_author'] |
Used for the author meta tag
|
DataMade | Yes |
SITE_META['site_url'] | Used to build absolute URLs for the meta tags. | https://chicago.councilmatic.org | Yes |
SITE_META['twitter_site'] |
Used for twitter:site meta tag.
|
@councilmatic | Yes |
SITE_META['twitter_creator'] |
Used for twitter:creator meta tag.
|
@DataMadeCo | Yes |
Look for councilmatic/settings_deployment.py.example
. Make a copy of it so that you can customize it for your city:
cp councilmatic/settings_deployment.py.example councilmatic/settings_deployment.py
This file is important! It's where you keep the parts of your Councilmatic that should not end up in version control (e.g., passwords, database connection strings, etc., which will vary based on where the app is running and which you probably won't want other people to know). Most of these variables are generic Django settings, which you can look up in the Django docs; but some of these variables require customization. This table shows only the variables that you need to customize.
Name | Description | Optional? |
---|---|---|
HAYSTACK_CONNECTIONS | Dictionary of connections used by Haystack. More on that here. | No |
FLUSH_KEY |
Used by the flush view in django-councilmatic to clear Django's cache.
|
No |
DISQUS_SHORTNAME | Will cause Disqus comment threads to be attached to individual bill pages. | Yes |
ANALYTICS_TRACKING_CODE | Google Analytics property ID | Yes |
GOOGLE_API_KEY | API key for rendering maps on council members and person detail pages. Learn about creating your own Google API Key. | No |
HEADSHOT_PATH | Absolute path to where the data loader will save images of members of the legislative body that are often available through Legistar. | Yes |
EXTRA_APPS | List of additional django apps that you want to use with your custom councilmatic | Yes |
Get an image (suggested 310x310), and transform it in device-specific favicon images using this site. Then, move the images into <city_app>/static/images/icons
.
Now, you are ready to run your unique version of Councilmatic!
python manage.py runserver
Navigate to http://localhost:8000/.
On a Councilmatic site, users can search bills according to given query parameters. To power our searches, we use Django Haystack to connect with Solr, an open source tool, written in Java.
Because it's a bit of a heavy dependency, we recommend running a containerized instance of Solr. See the Solr image in DockerHub for more information and startup instructions. Don't forget to update your Haystack config in councilmatic/settings_deployment.py
, if you map a port other than the Solr default (8983) to your container!
If you prefer to run Solr locally, see the Solr documentation for startup instructions.
Haystack provides several management commands that make it easy to change and add data to your search index.
To make your data searchable:
python manage.py rebuild_index
If, during the course of development, you need to make changes to the fields that are indexed in yourcity/search_indexes.py
, you need to regenerate the Solr schema:
python manage.py build_solr_schema > solr_scripts/schema.xml
Remove and recreate your Solr container to capture changes in the Solr schema.
If something is not behaving intuitively, it is a bug, and should be reported. Report it here.
We welcome your ideas and feedback! Here's how to make a contribution:
- Fork the project.
- Make your feature addition or bug fix.
- Commit - please be descriptive, but concise.
- Send a pull request. Bonus points for well-titled topic branches!
Copyright (c) 2019 Participatory Politics Foundation and DataMade. Released under the MIT License.