Datasette improvement ideas

[aesharpe]

Some ideas to improve Datasette:

• Add database descriptions to the homepage so that folks know the difference between pudl and ferc1, for instance, and don't just default to ferc1 if that's what they are looking for.

• Add table descriptions to the database homepages so that it's easier to see what each table is about (without having to click on them). This is especially nice for some of the tables with weird names. We can definitely do this for PUDL tables, and maybe XBRL tables what with the xbrl metadata.

• Change the database name ferc1 to ferc1_dbf to match ferc1_xbrl. Might even consider adding the word "raw" or the years associated with each of these databases so it's clear what's inside. At the very least, we should update the description of the ferc1_xbrl database to say that it's 2021 + data.

[bendnorman]

These are all great ideas! I think the ferc1, ferc1_xbrl and pudl distinctions aren't super clear to new users.

I agree it might be easier for new users to have the database and table descriptions viewable on the browsing pages. I've created a discussion in the datasette repo to see if this is possible.

[aesharpe]

Do you mean are clear or aren't clear?

[bendnorman]

Oops! aren't

[cmgosnell]

these are great ideas! we should (at the very least!) 100% add _dbf to the ferc1 db!!

[zaneselvans]

Totally agree about DB descriptions, table descriptions, and _dbf for ferc1 for symmetry. I think we could even write up or pull basic table descriptions for all of the FERC 1 tables if we wanted to in a day.

Datasette has a lot of templating functionality that relies on Jinja for customizing layouts and what information shows up where, and creating customized pages. The main datasette.io website is actually a Datasette instance. If we wanted to compile appropriate metadata / tags for the various databases and tables, we could probably create an index page or pages that organize data differently than per source DB, but that could be a lot of work and it might be better to put that kind of effort into a different kind of public facing platform.

[TrentonBush]

In general I think organizing data by source, rather than by topic, is a design centered on us rather than on users. It's a necessary and useful level of organization, but maybe not ideal as our primary output.

[bendnorman]

How would you organize the data by topic? I have talked to some users who asks a question like "If I wanted information about thing X, what tables would I look for?".

[TrentonBush]

I guess what I'm really suggesting is to link Datasette to output tables rather than the warehouse. Then we have a handful of wide tables about all the generators, all the utilities, etc instead of 12 normalized tables about each

[bendnorman]

Agreed. I like the idea of promoting the output tables rather than the data warehouse on datasette.

[zaneselvans]

IIRC we can choose which tables are presented in Datasette, so all of them can be in the underlying DB, and only the ones we want to highlight will show up in the index of tables.

[zaneselvans]

We might want to look at the large collection of Datasette plugins and see if any of them make sense to enable. E.g. full text search or SpatiaLite for the utility service territories table and Census DP1 data which is geospatial.

[katie-lamb]

Late to the party here but I think that in addition to a ferc1_dbf and ferc1_xbrl distinction. We need to update the descriptions to explain the DBF to XBRL change and when this happened. And also that these are raw tables and if you want clean tables then first check PUDL. The many avenues to access FERC data has gotten confusing without us making many changes to the Datasette and docs instructions.

[katie-lamb]

Oh, also I think we should update the section in the docs about how to download that data and mention the option to use the AWS CLI or download the DB from Datasette. A sentence about downloading data could also be included on Datasette since the DB download link is all the way at the bottom of the page.

[zaneselvans]

Now that the data can be downloaded directly from AWS either over HTTPS or with their CLI we might want to deprecate downloading whole DBs from Datasette, and prominently link to the AWS URLs from the front page.

[bendnorman]

Linking to AWS seems like a great idea so we can cut our egress costs even more. We could also provide a download link for the CEMS data. I realized yesterday it's not super clear how to get access to the CEMS data without installing pudl or using our 2i2c instance.

[katie-lamb]

Agreed on both linking directly to AWS in Datasette (and having that link be in a more prominent place) and that it's not clear how to access CEMS. Is CEMS in AWS? We could link to Zenodo for that one.

[zaneselvans]

Yes, the CEMS Parquet datasets are produced by the nightly builds and available from AWS / GCP. They can be queried remotely through Pandas, but that's probably only workable with a v. fast net connection -- or if/when we split the metadata out from the data files. Or you can download them in bulk.