Publish data

How to get your own data into Queria's catalog, and how to write the declarations

How to get your own data into Queria’s catalog. It takes two things: the data itself, and declarations saying what it is.

What you need to produce

Queria serves DuckLake catalogs. Getting your data into DuckLake tables is the build’s job — dropping a file somewhere does not publish it.

The tool is your choice; anything that writes DuckLake works. So is the input format, as long as DuckDB can read it (CSV, Parquet, JSON, Excel, whatever you fetched from an API).

The loop

uvx queria login                  # authenticate
uvx queria create my-dataset      # register the dataset (private by default)
# write dataset.yml and *.table.yml
uvx queria init                   # first time only: create the catalog you build into
uvx queria sync -- <build command>  # pull, build, publish

The first time you run uvx queria login, you choose a handle in the browser. Your datasets belong to that handle and are published at queria.io/datasets/<handle>/<dataset name>.

Creating a dataset (queria create) and publishing — queria push, and queria sync which includes it — need a token with “Allow modifying datasets” turned on. The queria login approval page turns it on by default, so there is nothing to set up from your own machine. To publish from CI, create a token with the option turned on at the API tokens page and pass it with the QUERIA_TOKEN environment variable. See Authentication for details.

If you belong to an organization account, --owner says where to create the dataset. Without it, the dataset belongs to your own handle. To keep publishing as the organization, use an organization service token, which an admin creates at queria.io/accounts/<org handle>/settings/tokens — it keeps working after the person who created it leaves.

uvx queria create my-dataset --owner my-org

When the same dataset name exists in more than one of your accounts, queria edit and queria delete stop rather than guess which one you meant. Give the name as <handle>/<dataset name> instead.

queria sync runs “fetch the published catalog → run your build → publish the result” as one command. Anything after -- works.

uvx queria sync -- python main.py
uvx queria sync -- dbt build

Before publishing, check that your declarations and your data agree.

uvx queria validate           # check the declarations against the data
uvx queria validate --strict  # fail on warnings too (for CI)

The declarations

There are two kinds. Directory names are up to you — what a file declares comes from its contents (schema and name).

# dataset.yml
spec_version: "0.1"
name: calendar
title: Japanese calendar
description: A date spine from 1955 to 2027 with holidays, weekdays and Japanese eras.
language: ja
licenses: [CC-BY-4.0]
contributors:
  - title: Cabinet Office
    roles: [rightsHolder]

# models/main/mart_calendar.table.yml
schema: main
name: mart_calendar
title: Japanese calendar
description: A date spine with holidays, weekdays, Japanese eras and fiscal years.
published: true
fields:
  - name: date
    title: Date

Do not write column types. Types, column order and nullability are read from your data, so they cannot drift from it.

Tables that share a column layout can go in one file, with YAML anchors carrying the shared part.

_fields: &fields
  - { name: area, title: Area code }
  - { name: value, title: Value, semantic: { role: measure, agg: sum } }
tables:
  - { schema: ssds, name: a_population, title: Population, fields: *fields }
  - { schema: ssds, name: b_land,       title: Land,       fields: *fields }

Declare key columns

Declaring with semantic what identifies a table’s rows lets search find the table through its key columns, and through column conditions that name the key. Give the same key only to columns that can be matched against each other, across datasets too.

fields:
  - name: municipality_code
    title: Local government code
    semantic: { role: entity, name: city }

These are the keys name takes. Columns that can be matched after a conversion, such as 5-digit and 6-digit (with check digit) local government codes, get the same key.

A key outside this list makes queria compile warn. Publishing still goes ahead, but the table does not show up under that key in search.

What happens when you make it public

A dataset is either public or private. While it is private, only you (and members of your organization) can see it. To switch:

uvx queria create my-dataset --public              # decide at creation
uvx queria edit my-dataset --visibility public     # change it later

Once public, it appears in these places.

Your handle is shown everywhere. Cards in the list and the dataset page both render it as <handle> / <dataset>, so readers always know who published the data.

Search engines may crawl these pages. Making the dataset private again stops it from being listed on Queria, but removal from search engine indexes takes time.

A change to the visibility setting takes up to 60 seconds to reach data delivery. For those 60 seconds after you switch back to private, anyone who already knew the delivery URL may still be able to read the data.

What trips people up

Licenses are never guessed. An id that is not in the registry stops validate. You may omit them, but then you reserve every right and what others may do with the data is left to Queria’s terms of use — validate says so as a warning.

If your *.table.yml files live under dbt’s model-paths, add *.table.yml to .dbtignore. dbt reads every .yml there as one of its own property files and fails on these. validate tells you when this is missing.

Small tables may produce no parquet at all. DuckLake inlines values into the catalog below a row threshold, so a dataset can end up with no data files. Set DATA_INLINING_ROW_LIMIT to 0 in your build.

A build longer than fifteen minutes has to re-issue its credentials. The credentials a build writes with expire after fifteen minutes, and DuckDB does not re-issue them when they do. A longer build stops being able to write partway through and fails with 403 SignatureDoesNotMatch. Call queria’s Secret at the seams between writes.

from queria import Secret

secret = Secret(conn)
secret.install()

for batch in batches:
    secret.refresh_if_due()   # does nothing until the interval has passed
    write(batch)

Call refresh_if_due() outside a transaction. A single statement that runs longer than fifteen minutes cannot be rescued this way – split the statement instead.

An account can hold at most five write credentials at a time. queria sync uses several in one build, so running builds for the same account side by side – in CI and on your machine, say – can make push fail with HTTP 429 (Too many credential requests). The build itself did not fail. The error tells you when the next credential expires; run the build again after that time.

Deciding what to write

There is a convention for which layer each sentence belongs to. See Writing descriptions.

Letting an agent do it

Both the publishing steps and the writing conventions ship as agent skills. See Queria skill for installation.

• queria-publish-dataset — register, build and publish
• queria-describe-dataset — what to write where