DataStore API

The CKAN DataStore offers an API for reading, searching and filtering data without the need to download the entire file first. The DataStore is an ad-hoc database which means that it is a collection of tables with unknown relationships. This allows you to search within one DataStore resource (a table in the database) as well as query across DataStore resources.

Data can be written incrementally to the DataStore through the API. New data can be inserted while existing data can be updated or deleted. You can even add new columns or data dictionaries to an existing table even if the DataStore resource already exists.

A DataStore resource cannot be created on its own. It is always required to have an associated CKAN resource. If data is stored in the DataStore, it will automatically be reviewed by the DataExplorer extension.

Making a DataStore API request

Making a DataStore API request is the same as making an Action API request: post a JSON dictionary via an HTTP POST request and the API returns its response via a JSON dictionary. For further information please see the latest API guide.

DataStore supports querying with multiple API endpoints. They are similar but support different features. The following list gives an overview of the different methods:

API Example

ckanext.datastore.logic.action.datastore_create(context, data_dict)

The datastore_create action allows you to post JSON data to be stored against a resource. This endpoint also supports altering tables, aliases and indexes, bulk insertion and data dictionaries. This endpoint can be called multiple times to initially insert more data, add fields, change the aliases or indexes as well as the primary keys and metadata.

Fields

Fields define the column names and the type of data in a column. A field is defined as follows:

Example Definition

[
    {
        "id": "code_number", # the column name (required)
        "type": "numeric" # the data type for the column
    },
    {
        "id": "description" # the column name (required)
        "type": "text", # the data type for the column
        "info": {
            "label": "Description", # human-readable label for column
            "notes": "A brief usage description for this code", # markdown description of column
            "type_override": numeric # datatype for datapusher to use when importing data
            "example": "Used for temporary service interruptions"
            ...: # additional user fields
        }
    }
]

Records

A record is the data to be inserted in a DataStore resource and is defined as follows:

Example Definition

[
    {
        "code_number": 10, # column_1_id: value_1,
        "description": "Submitted successfully" # column description
    },
    {
        "code_number": 42, # column_2_id: value_2,
        "description": "In progress" # column description
    }
]