Writing descriptions
Which layer — column, table or dataset — each thing you learned belongs to
There are three layers to write down what you know about a dataset: the column, the table, and the dataset.
Deciding case by case means everything ends up in one of them. Queria’s own e-Stat dataset had caveats covering 6 schemas and 30 tables piled into the coarsest layer there is. That is sorting by length, not by meaning.
The rule: write it where it is true
Pick the smallest thing your sentence is true of. That decides it.
| The sentence is true of | Where it goes |
|---|---|
| One column | that column’s description |
| One table | that table’s description |
| More than one table | the dataset’s description |
| What people call this dataset | keywords |
| Something you can run as SQL | a cookbook manuscript |
Once it is written small, do not repeat it upward. What stays at the top is only what you cannot say without comparing several things. Repeat a column note in the dataset’s description and one of the two copies will go stale.
What is worth writing
Five things:
- Grain — what one row represents
- Definition, source, how it was derived
- How it differs from something similar
- What breaks an aggregation or a join
- What is outside the coverage
4 and 5 are the ones people get wrong by not reading, and they are often the bulk of it. Push them down to the smallest layer where they hold.
- “This column mixes totals with their breakdown” → that column’s
description - “This table’s grain differs from the boundary data” → that table’s
description - “These two schemas join at different grains” → the dataset’s
description
keywords
keywords feeds the search index. Put in the words that classify the dataset and the other names people call it by, so someone who does not know the official name can still find it.
keywords: [statistics, population, society, e-Stat, SSDS, 政府統計]Words specific to one table belong in that table’s own keywords, not the dataset’s.
SQL belongs in the cookbook
Do not put runnable SQL in a description. It goes in your repository’s docs/*.md cookbook manuscripts. A description is what keeps someone from getting it wrong; a manuscript is what they can run.
Before you stop
- Could this sentence be said about something smaller than where you put it?
- Is the same thing written in two places?
- Are the other names people use in
keywords? - Did any SQL end up in a description?