diff --git a/.github/styles/config/vocabularies/docs/accept.txt b/.github/styles/config/vocabularies/docs/accept.txt index 5b949f3..493d81b 100644 --- a/.github/styles/config/vocabularies/docs/accept.txt +++ b/.github/styles/config/vocabularies/docs/accept.txt @@ -25,6 +25,7 @@ Opendata APIs (?i)datapoint (?i)datapoints +(?i)queryable dataclass subtask subtasks diff --git a/AGENTS.md b/AGENTS.md index 17fb9f0..2ad6369 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,6 +54,13 @@ Common page flow patterns in this repo: 3. Step-by-step sections for procedures. 4. `Next steps` links/cards at the end. +When documenting behavior shared by Tilebox clients and the CLI: + +1. State the behavior once as a Tilebox capability. Treat support across Python, Go, and the CLI as the default instead of enumerating each interface in prose. +2. Let code examples show interface-specific syntax. Use `CodeGroup` when every alternative is a single code snippet. Use `Tabs` only when alternatives contain other elements such as text, tables, or multiple code blocks. +3. Name a specific client only when documenting a real difference, limitation, version requirement, or unavailable feature. Put syntax-only guidance inside the relevant example tab when possible. +4. Keep introductions to examples brief. State the feature facts and let the code demonstrate the mechanics instead of describing each snippet line by line. + For command examples in user guides, optimize for reader copy/paste rather than fully scripted automation. Prefer direct `tilebox` commands and simple shell setup commands such as `cd`, `mkdir`, and `export`. Avoid Bash command substitution, uppercase helper variables such as `RELEASE_ID` or `JOB_ID`, `jq` pipelines, heredocs, and `--json` unless the page is explicitly about structured output or automation. When a later command needs a value returned by an earlier command, show the earlier command output and use a placeholder such as `` in follow-up commands. ## Terminology, Capitalization, And Naming diff --git a/api-reference/go/datasets/Create.mdx b/api-reference/go/datasets/Create.mdx index b4f5df8..ecbc33e 100644 --- a/api-reference/go/datasets/Create.mdx +++ b/api-reference/go/datasets/Create.mdx @@ -17,6 +17,9 @@ func (datasetClient) Create( Create a dataset with the given code name, display name, schema kind, and custom fields. +Fields created with `field.String`, `field.Bool`, `field.Int32`, `field.Int64`, `field.Uint64`, or `field.Float64` can be +marked queryable. Queryable fields cannot be repeated, and a dataset can contain at most two queryable string fields. + ## Parameters @@ -44,6 +47,12 @@ Create a dataset with the given code name, display name, schema kind, and custom Set the dataset's markdown description. +## Field options + + + Make the field available for server-side custom field filters. Choose queryable fields before ingesting datapoints. + + ## Dataset kinds @@ -64,10 +73,13 @@ dataset, err := client.Datasets.Create(ctx, "my_catalog", "My catalog", []datasets.Field{ - field.String("source").Description("Source system"), - field.Float64("cloud_cover"), + field.String("source").Description("Source system").Queryable(), + field.Float64("cloud_cover").Queryable(), }, datasets.WithSummary("Scenes prepared for analysis"), ) ``` + +See [Queryable fields](/datasets/concepts/datasets#queryable-fields) for schema constraints and +[Filter by custom fields](/datasets/query/filter-by-fields) for query syntax. diff --git a/api-reference/go/datasets/CreateOrUpdate.mdx b/api-reference/go/datasets/CreateOrUpdate.mdx index d2075ae..cd17d44 100644 --- a/api-reference/go/datasets/CreateOrUpdate.mdx +++ b/api-reference/go/datasets/CreateOrUpdate.mdx @@ -18,6 +18,7 @@ func (datasetClient) CreateOrUpdate( Create a dataset or update an existing dataset if a dataset with the given `codeName` already exists. If the dataset already exists, Tilebox applies the same schema update rules as a direct update. New fields can be added to non-empty datasets. Breaking schema changes are only allowed for empty datasets. +New queryable fields and changes to an existing field's queryable annotation are also only allowed for empty datasets. ## Parameters @@ -66,6 +67,9 @@ If the dataset already exists, Tilebox applies the same schema update rules as a A boolean field + + A 32-bit signed integer field + A 64-bit signed integer field @@ -99,6 +103,26 @@ If the dataset already exists, Tilebox applies the same schema update rules as a Set the example value of the field for documentation purposes + + Optional. Set the RFC 6901 path to this field in the source JSON, such as `/properties/eo:cloud_cover`. This is useful + when transforming datapoints to JSON because Tilebox can reconstruct nested source objects from flattened dataset + fields. + + + Make a non-repeated `String`, `Bool`, `Int32`, `Int64`, `Uint64`, or `Float64` field available for server-side custom + field filters. A dataset can contain at most two queryable string fields. + + + Optional. Set a JSON Schema reference URI or URI fragment for the field. Use this when the field follows a well-known + schema, such as a STAC extension. Tilebox emits the reference as `$ref` when advertising the field in STAC queryables. + + + Set semantic display roles for the field. The only currently supported role is `field.RolePrimaryTitle`. + + +Queryable string values can contain at most 1,024 Unicode code points. See +[Queryable fields](/datasets/concepts/datasets#queryable-fields) for schema constraints and +[Filter by custom fields](/datasets/query/filter-by-fields) for query syntax. ## Returns @@ -111,9 +135,10 @@ dataset, err := client.Datasets.CreateOrUpdate(ctx, "my_catalog", "My catalog", []datasets.Field{ - field.String("field1"), - field.Int64("field2").Repeated(), - field.Geometry("field3").Description("Field 3").ExampleValue("Value 3"), + field.String("platform").Queryable(), + field.Float64("cloud_cover").Queryable(), + field.Int64("shape").Repeated(), + field.Geometry("footprint").Description("Source product footprint"), }, datasets.WithSummary("Scenes prepared for analysis"), ) diff --git a/api-reference/go/datasets/Datapoints.Query.mdx b/api-reference/go/datasets/Datapoints.Query.mdx index 397a89a..7e0bd77 100644 --- a/api-reference/go/datasets/Datapoints.Query.mdx +++ b/api-reference/go/datasets/Datapoints.Query.mdx @@ -45,6 +45,10 @@ The output sequence can be transformed into a typed `proto.Message` using [Colle Restrict the query to specific dataset collections by collection ID. + + Filter by fields marked queryable in the dataset schema. Multiple expressions are combined with each other and with + temporal, spatial, and collection filters using logical AND. + Skip the data when querying datapoints. If set, only the required and auto-generated fields will be returned. @@ -82,3 +86,5 @@ datapoints, err := datasets.CollectAs[*v1.Sentinel1Sar]( ) ``` + +See [Filter by custom fields](/datasets/query/filter-by-fields) for comparisons, boolean expressions, and null checks. diff --git a/api-reference/go/datasets/Datapoints.QueryInto.mdx b/api-reference/go/datasets/Datapoints.QueryInto.mdx index 7ec85b7..7ae3ecc 100644 --- a/api-reference/go/datasets/Datapoints.QueryInto.mdx +++ b/api-reference/go/datasets/Datapoints.QueryInto.mdx @@ -48,6 +48,10 @@ QueryInto is a convenience function for [Query](/api-reference/go/datasets/Datap Restrict the query to specific dataset collections by collection ID. + + Filter by fields marked queryable in the dataset schema. Multiple expressions are combined with each other and with + temporal, spatial, and collection filters using logical AND. + Skip the data when querying datapoints. If set, only the required and auto-generated fields will be returned. @@ -84,3 +88,5 @@ err := client.Datapoints.QueryInto(ctx, ) ``` + +See [Filter by custom fields](/datasets/query/filter-by-fields) for comparisons, boolean expressions, and null checks. diff --git a/api-reference/go/datasets/Datapoints.QueryPage.mdx b/api-reference/go/datasets/Datapoints.QueryPage.mdx index f6af90f..df88651 100644 --- a/api-reference/go/datasets/Datapoints.QueryPage.mdx +++ b/api-reference/go/datasets/Datapoints.QueryPage.mdx @@ -42,6 +42,10 @@ Use `QueryPage` when you need manual pagination. Use [`Datapoints.Query`](/api-r Restrict the query to specific dataset collections by collection ID. + + Filter by fields marked queryable in the dataset schema. Multiple expressions are combined with each other and with + temporal, spatial, and collection filters using logical AND. + Skip datapoint data and return only required and generated fields. @@ -62,6 +66,7 @@ page, err := client.Datapoints.QueryPage(ctx, dataset.ID, datasets.WithTemporalExtent(queryInterval), datasets.WithCollectionIDs(collection.ID), + datasets.WithFilters(query.Field("quality").GreaterThanOrEqual(80)), datasets.WithLimit(100), ) if err != nil { @@ -73,6 +78,7 @@ if page.NextCursor != nil { dataset.ID, datasets.WithTemporalExtent(queryInterval), datasets.WithCollectionIDs(collection.ID), + datasets.WithFilters(query.Field("quality").GreaterThanOrEqual(80)), datasets.WithCursor(page.NextCursor), datasets.WithLimit(100), ) @@ -81,3 +87,5 @@ if page.NextCursor != nil { } ``` + +See [Filter by custom fields](/datasets/query/filter-by-fields) for comparisons, boolean expressions, and null checks. diff --git a/api-reference/go/datasets/Update.mdx b/api-reference/go/datasets/Update.mdx index d6cdfb8..953a5c7 100644 --- a/api-reference/go/datasets/Update.mdx +++ b/api-reference/go/datasets/Update.mdx @@ -18,6 +18,9 @@ func (datasetClient) Update( Update an existing dataset by ID with the given code name, display name, schema kind, custom fields, and metadata. +New non-queryable fields can be added to a non-empty dataset. Changing whether an existing field is queryable or adding +a new queryable field is only supported while the dataset is empty. + ## Parameters @@ -48,6 +51,13 @@ Update an existing dataset by ID with the given code name, display name, schema Set the dataset's markdown description. +## Field options + + + Make a non-repeated `String`, `Bool`, `Int32`, `Int64`, `Uint64`, or `Float64` field available for server-side custom + field filters. A dataset can contain at most two queryable string fields. + + ## Returns The updated dataset object. @@ -60,11 +70,14 @@ dataset, err := client.Datasets.Update(ctx, "my_catalog", "My catalog", []datasets.Field{ - field.String("source").Description("Source system"), - field.Float64("cloud_cover"), + field.String("source").Description("Source system").Queryable(), + field.Float64("cloud_cover").Queryable(), field.Timestamp("processed_at"), }, datasets.WithDescription("Catalog of scenes prepared for analysis."), ) ``` + +See [Queryable fields](/datasets/concepts/datasets#queryable-fields) for schema constraints and +[Filter by custom fields](/datasets/query/filter-by-fields) for query syntax. diff --git a/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx b/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx index 44f91ad..632afe4 100644 --- a/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx +++ b/api-reference/python/tilebox.datasets/Client.create_or_update_dataset.mdx @@ -16,6 +16,9 @@ def Client.create_or_update_dataset( Create a dataset, or update the existing dataset with the same code name. +New non-queryable fields can be added to a non-empty dataset. Changing whether an existing field is queryable or adding +a new queryable field is only supported while the dataset is empty. + ## Parameters @@ -89,6 +92,26 @@ Note that the type can also be a list of one of the types, indicating that the f Set the example value of the field for documentation purposes + + Optional. Set the RFC 6901 path to this field in the source JSON, such as `/properties/eo:cloud_cover`. This is useful + when transforming datapoints to JSON because Tilebox can reconstruct nested source objects from flattened dataset + fields. + + + Make the field available for server-side custom field filters. Queryable fields must be non-repeated `str`, `bool`, + `int`, `np.uint64`, or `float` fields. A dataset can contain at most two queryable string fields. + + + Optional. Set a JSON Schema reference URI or URI fragment for the field. Use this when the field follows a well-known + schema, such as a STAC extension. Tilebox emits the reference as `$ref` when advertising the field in STAC queryables. + + + Set semantic display roles for the field. The only currently supported role is `primary_title`. + + +Queryable string values can contain at most 1,024 Unicode code points. See +[Queryable fields](/datasets/concepts/datasets#queryable-fields) for schema constraints and +[Filter by custom fields](/datasets/query/filter-by-fields) for query syntax. ## Returns @@ -107,18 +130,24 @@ dataset = client.create_or_update_dataset( code_name="my_catalog", fields=[ { - "name": "field1", + "name": "platform", "type": str, + "queryable": True, + }, + { + "name": "cloud_cover", + "type": float, + "queryable": True, }, { - "name": "field2", + "name": "shape", "type": list[int], }, { - "name": "field3", + "name": "footprint", "type": Geometry, - "description": "Field 3", - "example_value": "Value 3", + "description": "Source product footprint", + "example_value": "POLYGON ((11 46, 12 46, 12 47, 11 47, 11 46))", }, ], name="My personal catalog", diff --git a/api-reference/python/tilebox.datasets/Collection.query.mdx b/api-reference/python/tilebox.datasets/Collection.query.mdx index a0ec842..15ad00b 100644 --- a/api-reference/python/tilebox.datasets/Collection.query.mdx +++ b/api-reference/python/tilebox.datasets/Collection.query.mdx @@ -7,6 +7,7 @@ icon: layer-group def Collection.query( *, temporal_extent: TimeIntervalLike, + filter: Expression | None = None, spatial_extent: SpatialFilterLike | None = None, skip_data: bool = False, show_progress: bool | Callable[[float], None] = False, @@ -33,6 +34,11 @@ If no data exists for the requested time or interval, an empty `xarray.Dataset` + + Optional expression over fields marked queryable in the dataset schema. Build expressions with `field()` and combine + them with `&`, `|`, and `~`. See [Filter by custom fields](/datasets/query/filter-by-fields). + + Optional spatial filter. Use this for spatial queries in spatio-temporal datasets. @@ -52,6 +58,7 @@ An [`xarray.Dataset`](/sdks/python/xarray) containing the requested data points. ```python Python from datetime import datetime +from tilebox.datasets import field from tilebox.datasets.query import TimeInterval # querying a specific time @@ -72,6 +79,7 @@ data = collection.query( data = collection.query( temporal_extent=interval, spatial_extent=geometry, + filter=(field("cloud_cover") < 10) & (field("platform") == "sentinel-2c"), ) # querying a time interval with TimeInterval diff --git a/api-reference/python/tilebox.datasets/Dataset.query.mdx b/api-reference/python/tilebox.datasets/Dataset.query.mdx index 47ab957..f097776 100644 --- a/api-reference/python/tilebox.datasets/Dataset.query.mdx +++ b/api-reference/python/tilebox.datasets/Dataset.query.mdx @@ -8,6 +8,7 @@ def Dataset.query( *, collections: list[str] | list[UUID] | list[Collection] | list[CollectionInfo] | list[CollectionClient] | dict[str, CollectionClient] | None = None, temporal_extent: TimeIntervalLike, + filter: Expression | None = None, spatial_extent: SpatialFilterLike | None = None, skip_data: bool = False, show_progress: bool | Callable[[float], None] = False, @@ -37,6 +38,11 @@ If no data matches the filters, an empty `xarray.Dataset` is returned. The time or time interval to query. This can be a single time scalar, a tuple of two time scalars, or a `TimeInterval` object. + + Optional expression over fields marked queryable in the dataset schema. Build expressions with `field()` and combine + them with `&`, `|`, and `~`. See [Filter by custom fields](/datasets/query/filter-by-fields). + + Optional spatial filter. Use this for spatial queries in spatio-temporal datasets. @@ -69,11 +75,19 @@ An [`xarray.Dataset`](/sdks/python/xarray) containing matching datapoints. ```python Python +from tilebox.datasets import field + # query all collections in the dataset data = dataset.query( temporal_extent=("2025-04-01", "2025-05-01"), ) +# query using custom fields marked queryable in the dataset schema +data = dataset.query( + temporal_extent=("2026-07-20", "2026-07-28"), + filter=(field("cloud_cover") < 1) & (field("platform") == "sentinel-2c"), +) + # query selected collections by name data = dataset.query( collections=["S2A_S2MSI2A", "S2B_S2MSI2A"], diff --git a/changelog.mdx b/changelog.mdx index 58ca5c0..0bfd863 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -5,6 +5,38 @@ icon: rss mode: center --- + + ## Queryable custom dataset fields + + Custom dataset schemas can now mark selected fields as queryable. Tilebox evaluates these field expressions on the + server together with temporal, spatial, and collection filters, so clients only receive matching datapoints. + + Filters support comparisons, boolean logic, and null checks. + + ```python + from tilebox.datasets import Client, field + + dataset = Client().dataset("open_data.aws_earth.sentinel2") + data = dataset.query( + collections=["L2A"], + temporal_extent=("2026-07-20", "2026-07-28"), + filter=( + (field("cloud_cover") < 1) + & (field("platform") == "sentinel-2c") + ), + ) + ``` + + Queryable fields support strings, booleans, signed and unsigned integers, and floating-point values. Dataset authors + select queryable fields before ingesting datapoints. + + + + Define queryable fields and filter matching datapoints. + + + + Deploying a workflow release to a cluster and rolling the cluster back to an earlier release in the Tilebox Console diff --git a/datasets/concepts/datasets.mdx b/datasets/concepts/datasets.mdx index dd840dd..5629646 100644 --- a/datasets/concepts/datasets.mdx +++ b/datasets/concepts/datasets.mdx @@ -47,11 +47,12 @@ The required fields of the dataset type, as well as the custom fields specific t **dataset schema**. Once a **dataset schema** is defined, existing fields cannot be removed or edited as soon as data has been ingested into it. -You can always add new fields to a dataset, since all fields are always optional. +You can add new non-queryable fields to a non-empty dataset because all custom fields are optional. You cannot add a new +queryable field or change whether an existing field is queryable after ingesting data. - The only exception to this rule are empty datasets. If you empty all collections in a dataset, you can freely - edit the data schema, since no conflicts with existing data points can occur. + Empty datasets are the exception. If all collections are empty, you can freely edit the data schema and its + queryable field annotations because no existing datapoints can conflict with the change. ## Field types @@ -63,6 +64,7 @@ When defining the data schema, you can specify each field's type. The following | Type | Description | Example value | | --- | --- | --- | | string | A string of characters of arbitrary length. | `Some string` | +| int32 | A 32-bit signed integer. | `123` | | int64 | A 64-bit signed integer. | `123` | | uint64 | A 64-bit unsigned integer. | `123` | | float64 | A 64-bit floating-point number. | `123.45` | @@ -92,6 +94,26 @@ When defining the data schema, you can specify each field's type. The following Every type is also available as an array, allowing to ingest multiple values of the underlying type for each data point. The size of the array is flexible, and can be different for each data point. +## Queryable fields + +Custom fields can be marked as queryable when you define the dataset schema. Tilebox projects these fields into query +storage so you can [filter datapoints by their values](/datasets/query/filter-by-fields) together with temporal, spatial, +and collection filters. + +| Constraint | Limit | +| --- | --- | +| Supported types | `string`, `bool`, `int32`, `int64`, `uint64`, and `float64` | +| Repeated fields | Not supported | +| Queryable string fields | At most two per dataset | +| Queryable string values | At most 1,024 Unicode code points | +| Field location | Top-level custom fields only | +| Field names | Snake case, starting with a letter, and at most 100 characters | + + + Select queryable fields before ingesting datapoints. For a non-empty dataset, you cannot change whether an existing + field is queryable or add a new queryable field. + + ## Listing datasets You can use [your client instance](/datasets/introduction#creating-a-datasets-client) to access the datasets available to you. To list all available datasets, use the `datasets` method of the client. @@ -151,10 +173,12 @@ Once you have your dataset object, you can use it to [list the available collect ## Creating / Updating a dataset -You can create a dataset using one of the available [client SDKs](/sdks/introduction), or use the [Tilebox Console](/console) when you prefer a visual schema editor. +Create a dataset by defining its kind, custom fields, and field annotations. The [Tilebox Console](/console) provides a +visual schema editor. - -```python Python + + +```python from datetime import timedelta from tilebox.datasets import Client from tilebox.datasets.data.datasets import DatasetKind @@ -170,12 +194,14 @@ dataset = client.create_or_update_dataset( "type": str, "description": "The name of the Landsat-8 granule.", "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1", + "queryable": True, }, { "name": "cloud_cover", "type": float, "description": "Cloud cover percentage", "example_value": "91", + "queryable": True, }, { "name": "proj_shape", @@ -187,22 +213,62 @@ dataset = client.create_or_update_dataset( name="Personal Landsat-8 catalog", ) ``` -```go Go + + +```go dataset, err := client.Datasets.Create(ctx, datasets.KindSpatiotemporal, "my_landsat8_oli_tirs", "Personal Landsat-8 catalog", []datasets.Field{ - field.String("granule_name").Description("The name of the Earth View Granule").ExampleValue("20220830_185202_SN18_10N_552149_5297327"), - field.Float64("cloud_cover").Description("Cloud cover percentage").ExampleValue("91"), + field.String("granule_name").Description("The name of the Earth View Granule").ExampleValue("20220830_185202_SN18_10N_552149_5297327").Queryable(), + field.Float64("cloud_cover").Description("Cloud cover percentage").ExampleValue("91").Queryable(), field.Int64("proj_shape").Repeated().Description("Raster shape").ExampleValue("[6971, 7801]"), } ) ``` - + + +```json schema.json +{ + "kind": "spatiotemporal", + "fields": [ + { + "name": "granule_name", + "type": "string", + "description": "The name of the Landsat-8 granule.", + "example_value": "LE07_L1TP_174061_20010913_20200917_02_T1", + "queryable": true + }, + { + "name": "cloud_cover", + "type": "float64", + "description": "Cloud cover percentage", + "example_value": "91", + "queryable": true + }, + { + "name": "proj_shape", + "type": "int64", + "repeated": true, + "description": "Raster shape", + "example_value": "[6971, 7801]" + } + ] +} +``` + +```bash +tilebox dataset create \ + --name "Personal Landsat-8 catalog" \ + --code-name my_landsat8_oli_tirs \ + --schema-file schema.json +``` + + This code will create a new catalog with 7 fields in total. -4 of those fields are auto-generated by choosing the spatio-temporal dataset type, and 3 (`granule_name`, `cloud_cover`, `proj_shape`) are custom fields that are defined. +4 of those fields are auto-generated by choosing the spatio-temporal dataset type, and 3 (`granule_name`, `cloud_cover`, `proj_shape`) are custom fields that are defined. `granule_name` and `cloud_cover` can be used in custom field filters; the repeated `proj_shape` field cannot be queryable. If a dataset with the same `code_name` already exists, it will be updated instead. diff --git a/datasets/query/filter-by-fields.mdx b/datasets/query/filter-by-fields.mdx new file mode 100644 index 0000000..d03e630 --- /dev/null +++ b/datasets/query/filter-by-fields.mdx @@ -0,0 +1,177 @@ +--- +title: Filter by custom fields +description: Filter datapoints server-side with queryable custom dataset fields. +icon: filter-list +--- + +Queryable fields let you filter datapoints by custom metadata before Tilebox returns the query result. You can combine +custom field expressions with temporal, spatial, and collection filters in the same query. + +## Find queryable fields + +The dataset schema identifies which fields are queryable. Inspect the schema to find fields available for filtering. + +```bash +tilebox dataset get open_data.aws_earth.sentinel2 +``` + +The `open_data.aws_earth.sentinel2` dataset exposes these queryable fields: + +| Field | Type | +| --- | --- | +| `stac_id` | `string` | +| `platform` | `string` | +| `cloud_cover` | `float64` | +| `nodata_pixel_percentage` | `float64` | + +## Filter datapoints + +This query returns Sentinel-2C Level-2A datapoints with less than one percent cloud cover between July 20 and July 28, +2026. + + +```python Python +from tilebox.datasets import Client, field + +client = Client() +dataset = client.dataset("open_data.aws_earth.sentinel2") + +data = dataset.query( + collections=["L2A"], + temporal_extent=("2026-07-20", "2026-07-28"), + filter=(field("cloud_cover") < 1) & (field("platform") == "sentinel-2c"), +) +``` +```go Go +startDate := time.Date(2026, 7, 20, 0, 0, 0, 0, time.UTC) +endDate := time.Date(2026, 7, 28, 0, 0, 0, 0, time.UTC) + +dataset, err := client.Datasets.Get(ctx, "open_data.aws_earth.sentinel2") +if err != nil { + return err +} +collection, err := client.Collections.Get(ctx, dataset.ID, "L2A") +if err != nil { + return err +} + +page, err := client.Datapoints.QueryPage(ctx, + dataset.ID, + datasets.WithCollections(collection), + datasets.WithTemporalExtent(query.NewTimeInterval(startDate, endDate)), + datasets.WithFilters( + query.Field("cloud_cover").LessThan(1.0), + query.Field("platform").Equal("sentinel-2c"), + ), +) +if err != nil { + return err +} +fmt.Println(len(page.Datapoints)) +``` +```bash CLI +tilebox dataset query open_data.aws_earth.sentinel2 \ + --collections L2A \ + --after 2026-07-20 \ + --before 2026-07-28 \ + --filter "cloud_cover < 1 AND platform = 'sentinel-2c'" +``` + + +Tilebox combines custom field expressions with temporal, spatial, and collection filters using `AND`. + +## Operators + +Custom field filters support comparisons, boolean composition, and null checks. + + + +| Operation | Syntax | +| --- | --- | +| Equal | `field("x") == value` | +| Not equal | `field("x") != value` | +| Less than | `field("x") < value` | +| Less than or equal | `field("x") <= value` | +| Greater than | `field("x") > value` | +| Greater than or equal | `field("x") >= value` | +| AND | `left & right` | +| OR | `left \| right` | +| NOT | `~expression` | +| Is null | `field("x").is_null()` | +| Is not null | `field("x").is_not_null()` | + +Use `&`, `|`, and `~` instead of `and`, `or`, and `not`, and place each comparison in parentheses. + + +| Operation | Syntax | +| --- | --- | +| Equal | `query.Field("x").Equal(value)` | +| Not equal | `query.Field("x").NotEqual(value)` | +| Less than | `query.Field("x").LessThan(value)` | +| Less than or equal | `query.Field("x").LessThanOrEqual(value)` | +| Greater than | `query.Field("x").GreaterThan(value)` | +| Greater than or equal | `query.Field("x").GreaterThanOrEqual(value)` | +| AND | `query.And(left, right)` | +| OR | `query.Or(left, right)` | +| NOT | `query.Not(expression)` | +| Is null | `query.Field("x").IsNull()` | +| Is not null | `query.Field("x").IsNotNull()` | + + +The CLI accepts a subset of the [CQL2 Text](https://docs.ogc.org/is/21-065r2/21-065r2.html) syntax. + +| Operation | Syntax | +| --- | --- | +| Equal | `x = value` | +| Not equal | `x <> value` | +| Less than | `x < value` | +| Less than or equal | `x <= value` | +| Greater than | `x > value` | +| Greater than or equal | `x >= value` | +| AND | `left AND right` | +| OR | `left OR right` | +| NOT | `NOT expression` | +| Is null | `x IS NULL` | +| Is not null | `x IS NOT NULL` | + +String, boolean, and numeric literals and parentheses are supported. Repeating `--filter` combines expressions with `AND`. + + + +Fields of type `bool` support only equality and inequality. + +## Missing and null values + +Filter expressions use three-valued logic: `true`, `false`, and `null` (unknown). A datapoint matches only when the +complete expression is `true`. Comparisons against a missing or explicitly null field return `null` and do not match. + +This behavior also applies to inequality. For example, `quality != 1` requires `quality` to be present. Combine the +comparison with an explicit null check when missing values should also match. + + +```python Python +filter = (field("quality") != 1) | field("quality").is_null() +``` +```go Go +filter := query.Or( + query.Field("quality").NotEqual(1), + query.Field("quality").IsNull(), +) +``` + + +## Query safeguards + +A single query can contain filters with at most 64 top-level expressions or operands, 256 expression nodes in total, +and eight levels of nesting. These safeguards keep query evaluation bounded. + +## Next steps + + + + Learn how dataset kinds, custom fields, and schema updates work. + + + Create a custom catalog, ingest metadata, and query its custom fields. + + diff --git a/datasets/query/querying-data.mdx b/datasets/query/querying-data.mdx index 424dcd0..496d38d 100644 --- a/datasets/query/querying-data.mdx +++ b/datasets/query/querying-data.mdx @@ -1,12 +1,13 @@ --- title: Querying data sidebarTitle: Querying data -description: Access and filter data stored in your datasets using time-based and spatial queries, with built-in support for pagination and progress tracking. +description: Access and filter data stored in your datasets by time, location, and custom fields, with built-in pagination and progress tracking. icon: magnifying-glass --- Tilebox offers a powerful and flexible querying API to access and filter data from your datasets. When querying, you can -[filter by time](/datasets/query/filter-by-time) and for [Spatio-temporal datasets](/datasets/types/spatiotemporal) optionally also [filter by a location in the form of a geometry](/datasets/query/filter-by-location). +[filter by time](/datasets/query/filter-by-time), [filter by custom fields](/datasets/query/filter-by-fields), and for +[Spatio-temporal datasets](/datasets/types/spatiotemporal), optionally [filter by a location in the form of a geometry](/datasets/query/filter-by-location). ## Running a query @@ -112,8 +113,8 @@ if err != nil { ``` -To learn more about how to specify filters to narrow down the query results, check out the following sections about filtering by -time, by geometry or by datapoint ID. +To learn more about how to narrow down query results, see the following pages about filtering by time, geometry, custom +fields, or datapoint ID. Learn how to filter your query results by a certain geographical extent. + + Filter datapoints by queryable fields in the dataset schema. + Learn the required fields and query behavior. + + Combine queryable field expressions with temporal and spatial filters. + Load CSV, Parquet, GeoParquet, and NetCDF data before ingestion. diff --git a/index.mdx b/index.mdx index c7a1592..58fa7a3 100644 --- a/index.mdx +++ b/index.mdx @@ -287,7 +287,7 @@ func (t *ComputeVisibleChange) Execute(ctx context.Context) error {
- + Tilebox dataset explorer Tilebox dataset explorer diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 0000000..6ab6825 --- /dev/null +++ b/package-lock.json @@ -0,0 +1,6 @@ +{ + "name": "docs", + "lockfileVersion": 3, + "requires": true, + "packages": {} +}