Data quality and cleansing policy

The preprocessing Queria applies to source data and its approach to quality

Queria doesn’t just line up published open data as-is — it reshapes the data into an analysis-friendly form before serving it. This page explains that preprocessing policy.

Why preprocessing is needed

When open data is first published, character encodings, column names, and units vary from source to source. The same “municipality code” can differ in digit count or type, and columns that should be numeric can have annotation text mixed in. This kind of variance becomes a major obstacle the moment you try to connect datasets together.

Queria does this groundwork properly, once, so that users never have to write the same preprocessing again.

The preprocessing Queria applies

• Format unification: align character encodings and file formats, and serve as Parquet
• Column name and type normalization: tidy column naming and convert numeric, date, and other types appropriately
• Value cleansing: strip Japanese annotations and units mixed into numeric columns, and absorb missing or inconsistent values
• Key preparation: align code systems so datasets can be connected with JOIN
• Metadata: attach descriptions of each table’s and column’s meaning, units, and aggregation method

All of this transformation logic is public and can be inspected via the source code link on each table’s detail page.

Our stance toward source data

Preprocessing exists solely to “shape data into usable form” — we never rewrite the source data’s actual values. To keep the cleansing transparent, all transformation logic is public. For the accuracy of the original data, refer to the primary sources listed at each table’s data source.

Next steps

• Attribution and licensing: the conditions for using the data
• How it works: how queries run on your side
• Concept: the idea of taking on the groundwork