diff --git a/api-reference/python/tilebox.workflows/Task.mdx b/api-reference/python/tilebox.workflows/Task.mdx index 0e0df6e..85ea0cd 100644 --- a/api-reference/python/tilebox.workflows/Task.mdx +++ b/api-reference/python/tilebox.workflows/Task.mdx @@ -37,8 +37,7 @@ class MyTask(Task): data: dict[str, int] ``` -Optional task [input parameters](/workflows/concepts/tasks#input-parameters), defined as class attributes. Supported types -are `str`, `int`, `float`, `bool`, as well as `lists` and `dicts` thereof. +Optional task [input parameters](/workflows/concepts/tasks#input-parameters), defined as annotated class attributes. See [Python task inputs](/sdks/python/task-inputs) for the supported types and their package requirements. ```python Python diff --git a/assets/changelog/2026-08-18-job-view.webp b/assets/changelog/2026-08-18-job-view.webp new file mode 100644 index 0000000..01c8fad Binary files /dev/null and b/assets/changelog/2026-08-18-job-view.webp differ diff --git a/changelog.mdx b/changelog.mdx index 7c9cdbd..80f8c04 100644 --- a/changelog.mdx +++ b/changelog.mdx @@ -5,6 +5,26 @@ icon: rss mode: center --- + + ## Understand and debug workflow jobs faster + + + Updated job details page + + + The redesigned job details page makes it easier to understand what a workflow is doing, where it spends time, and why it failed. Job progress, execution statistics, tasks, traces, and logs now form one connected view, so you can move from the state of the whole job to the work of an individual task without losing context. + + Tasks appear in their workflow hierarchy instead of a flat list. Expand the branches you care about, follow state and timing through nested work, and navigate large jobs without loading the entire task graph at once. + + Select any task to see its input, timing, retries, compute location, execution trace, and logs together. A failed or slow task is no longer an isolated telemetry record: you can see where it sits in the workflow, inspect what it received, and trace exactly what happened during its execution. + + + + Learn how Tilebox connects job tasks, execution traces, logs, and runner context. + + + + ## Sentinel-2 imagery, ready to query and read diff --git a/docs.json b/docs.json index c418e7f..2cde75d 100644 --- a/docs.json +++ b/docs.json @@ -201,6 +201,7 @@ "group": "Python", "pages": [ "sdks/python/install", + "sdks/python/task-inputs", "sdks/python/sample-notebooks", "sdks/python/xarray", "sdks/python/async" diff --git a/sdks/python/task-inputs.mdx b/sdks/python/task-inputs.mdx new file mode 100644 index 0000000..5769f6d --- /dev/null +++ b/sdks/python/task-inputs.mdx @@ -0,0 +1,168 @@ +--- +title: Python task inputs +sidebarTitle: Task inputs +description: Supported Python types for Tilebox workflow task inputs, including geospatial and raster types. +icon: list-check +--- + +Python tasks are data classes. Annotate each task field with one of the supported types below, and Tilebox reconstructs that type before the task runs. + +This page applies to tasks executed by Python runners. For tasks submitted and executed across different languages, use an input schema supported by both SDKs. See [Multi-language workflows](/guides/workflows/multi-language). + +The examples focus on task input declarations and omit the `execute` method. + +## Python standard library + +These types require no extra packages: + +- **Values:** [`str`](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str), [`int`](https://docs.python.org/3/library/functions.html#int), [`float`](https://docs.python.org/3/library/functions.html#float), [`bool`](https://docs.python.org/3/library/functions.html#bool), [`bytes`](https://docs.python.org/3/library/stdtypes.html#bytes), and [`bytearray`](https://docs.python.org/3/library/stdtypes.html#bytearray) +- **Collections:** [`list`](https://docs.python.org/3/library/stdtypes.html#list), [`tuple`](https://docs.python.org/3/library/stdtypes.html#tuple), [`dict`](https://docs.python.org/3/library/stdtypes.html#dict), [`set`](https://docs.python.org/3/library/stdtypes.html#set), and [`frozenset`](https://docs.python.org/3/library/stdtypes.html#frozenset) +- **Type annotations:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional), [`Union`](https://docs.python.org/3/library/typing.html#typing.Union), and [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal) +- **Structured values:** [data classes](https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass) and [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) +- **Dates and times:** [`datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime), [`date`](https://docs.python.org/3/library/datetime.html#datetime.date), [`time`](https://docs.python.org/3/library/datetime.html#datetime.time), [`timedelta`](https://docs.python.org/3/library/datetime.html#datetime.timedelta), and [`ZoneInfo`](https://docs.python.org/3/library/zoneinfo.html#zoneinfo.ZoneInfo) +- **Other values:** [`UUID`](https://docs.python.org/3/library/uuid.html#uuid.UUID), [`Decimal`](https://docs.python.org/3/library/decimal.html#decimal.Decimal), and [`PurePath`](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath) subclasses such as [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) + +**Example usage** + +```python +from datetime import datetime +from pathlib import Path +from uuid import UUID + +from tilebox.workflows import Task + +class BuildSentinel2Mosaic(Task): + scene_ids: list[UUID] + bands: tuple[str, ...] + acquired_after: datetime + output_path: Path +``` + +## Protocol buffers + +`protobuf` provides generated message classes for strongly typed schemas and is installed with `tilebox-workflows`. + +Tilebox supports [`Message`](https://googleapis.dev/python/protobuf/latest/google/protobuf/message.html#google.protobuf.message.Message) and its generated subclasses. + +**Example usage** + +```python +from google.protobuf.timestamp_pb2 import Timestamp +from tilebox.workflows import Task + +class ProcessSceneAcquisition(Task): + scene_id: str + acquired_at: Timestamp +``` + +## Tilebox Datasets + +`tilebox-datasets` provides value types for dataset and job queries and is installed with `tilebox-workflows`. + +| Type | Use in a workflow | +| --- | --- | +| [`TimeInterval`](/datasets/query/filter-by-time#manual-endpoint-inclusivity), [`IDInterval`](/api-reference/python/tilebox.workflows/JobClient.query) | Pass dataset or job query ranges to a task | +| [`SpatialFilter`](/datasets/query/filter-by-location) | Pass a dataset spatial query to a task; its geometry requires Shapely | + +**Example usage** + +```python +from tilebox.datasets.data.data_access import SpatialFilter +from tilebox.datasets.query import TimeInterval +from tilebox.workflows import Task + +class QuerySentinel2Scenes(Task): + collections: list[str] + temporal_extent: TimeInterval + spatial_extent: SpatialFilter +``` + +## Shapely + +`shapely` provides geometry types for vector features, footprints, and areas of interest. + +- **Single geometries:** [`Geometry`](https://shapely.readthedocs.io/en/stable/reference/shapely.Geometry.html), [`Point`](https://shapely.readthedocs.io/en/stable/reference/shapely.Point.html), [`LineString`](https://shapely.readthedocs.io/en/stable/reference/shapely.LineString.html), [`LinearRing`](https://shapely.readthedocs.io/en/stable/reference/shapely.LinearRing.html), and [`Polygon`](https://shapely.readthedocs.io/en/stable/reference/shapely.Polygon.html) +- **Geometry collections:** [`MultiPoint`](https://shapely.readthedocs.io/en/stable/reference/shapely.MultiPoint.html), [`MultiLineString`](https://shapely.readthedocs.io/en/stable/reference/shapely.MultiLineString.html), [`MultiPolygon`](https://shapely.readthedocs.io/en/stable/reference/shapely.MultiPolygon.html), and [`GeometryCollection`](https://shapely.readthedocs.io/en/stable/reference/shapely.GeometryCollection.html) + +**Example usage** + +```python +from shapely import MultiPolygon +from tilebox.workflows import Task + +class ComputeSentinel2CloudStatistics(Task): + area_of_interest: MultiPolygon + preceding_hours: int +``` + +## Coordinate systems and raster transforms + +`affine` provides two-dimensional affine transformation matrices. `pyproj` provides coordinate reference systems and coordinate transformations. + +| Type | Use in a workflow | +| --- | --- | +| [`Affine`](https://affine.readthedocs.io/en/latest/index.html#affine.Affine) | Preserve the pixel-to-world transform for raster processing | +| [`pyproj.CRS`](https://pyproj4.github.io/pyproj/stable/api/crs/crs.html#pyproj.crs.CRS) | Pass a coordinate reference system without reducing it to a string | + +**Example usage** + +```python +from affine import Affine +from pyproj import CRS +from tilebox.workflows import Task + +class ReprojectRasterTile(Task): + source_crs: CRS + target_crs: CRS + source_transform: Affine +``` + +## ODC Geo + +`odc-geo` provides projection-aware geometry and raster grid types. + +| Type | Use in a workflow | +| --- | --- | +| [`CRS`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.crs.CRS.html) | Preserve an ODC coordinate reference system | +| [`Geometry`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geom.Geometry.html), [`BoundingBox`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geom.BoundingBox.html) | Pass projection-aware geometries and bounds | +| [`XY`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.XY.html), [`Resolution`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.Resolution.html) | Describe grid coordinates and spatial resolution | +| [`Index2d`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.Index2d.html), [`Shape2d`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.Shape2d.html) | Describe a grid index or shape | +| [`GeoBox`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geobox.GeoBox.html) | Preserve an aligned, georeferenced raster grid | +| [`GeoboxTiles`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.geobox.GeoboxTiles.html), [`AnchorEnum`](https://odc-geo.readthedocs.io/en/latest/_api/odc.geo.AnchorEnum.html) | Partition and align a `GeoBox` for tiled processing | + +**Example usage** + +```python +from odc.geo import GeoBox +from tilebox.workflows import Task + +class ReprojectSentinel2Product(Task): + product_location: str + source_grid: GeoBox + target_grid: GeoBox +``` + +## Raster windows + +`rasterio` provides raster data access and processing. `async-geotiff` provides asynchronous GeoTIFF and Cloud Optimized GeoTIFF reads. Install either package separately when your workflow uses its window type. + +| Type | Use in a workflow | +| --- | --- | +| [`rasterio.windows.Window`](https://rasterio.readthedocs.io/en/stable/api/rasterio.windows.html#rasterio.windows.Window) | Pass a rectangular pixel region to a task that uses `rasterio` | +| [`async_geotiff.Window`](https://developmentseed.org/async-geotiff/latest/api/window/) | Pass a rectangular pixel region to an async GeoTIFF task | + +**Example usage** + +```python +from rasterio.windows import Window +from tilebox.workflows import Task + +class ComputeHyperspectralChunkStatistics(Task): + product_path: str + window: Window + output_key: str +``` + +## Keep task inputs compact + +Task inputs are part of the workflow graph and are not intended for large arrays, file contents, pandas DataFrames, clients, or open files. Store large data in object storage or the [job cache](/workflows/run-and-inspect/caches), then pass a compact reference such as an ID, object prefix, cache key, time interval, geometry, or raster window. diff --git a/workflows/concepts/tasks.mdx b/workflows/concepts/tasks.mdx index f35298c..6d967f9 100644 --- a/workflows/concepts/tasks.mdx +++ b/workflows/concepts/tasks.mdx @@ -63,67 +63,68 @@ For Go, the key components are: - The code samples on this page do not illustrate how to execute the task. That will be covered in the - [next section on runners](/workflows/concepts/runners). The reason for that is that executing tasks is a separate concern from implementing tasks. + Defining and executing tasks are separate concerns. This page covers task definitions; see [Runners](/workflows/concepts/runners) for how Tilebox assigns tasks for execution. ## Input Parameters -Tasks often require input parameters to operate. These inputs can range from simple values to complex data structures. By inheriting from the `Task` class, the task is treated as a Python `dataclass`, allowing input parameters to be defined as class attributes. +Task inputs are the small values that define one task execution. Declare them as fields on the task and provide concrete values when you create it. Tilebox serializes the fields so a [runner](/workflows/concepts/runners) on another machine can reconstruct the task before executing it. - - Tasks must be **serializable to JSON or to protobuf** because they may be distributed across a cluster of [runners](/workflows/concepts/runners). - - - - Keep task inputs small. Task inputs are part of the workflow graph and are not intended for large manifests, file lists, arrays, or binary payloads. Store large data in object storage or the [job cache](/workflows/run-and-inspect/caches), then pass a compact reference such as a cache key, object prefix, scene ID, shard index, or time range. - +Supported inputs include standard values, collections, structured types, protobuf messages, and integrated library types such as [Shapely geometries](/sdks/python/task-inputs#shapely). See the supported task inputs for [Python](/sdks/python/task-inputs) or [Go](/api-reference/go/workflows/Task). - In Go, task parameters must be exported fields of the task struct (starting with an uppercase letter), otherwise they will not be serialized to JSON. + Use task inputs for values known when the task is submitted, such as IDs, time intervals, areas of interest, and output keys. Put large data, generated results, and values shared between tasks in object storage or the [job cache](/workflows/run-and-inspect/caches), then pass only a compact reference. -Supported types for input parameters include: - -- Basic types such as `str`, `int`, `float`, `bool` -- Lists and dictionaries of basic types -- Nested data classes that are also JSON-serializable or protobuf-serializable - ```python Python - class ParametrizableTask(Task): - message: str - number: int - data: dict[str, str] + from datetime import datetime + + from shapely import Polygon + from tilebox.workflows import ExecutionContext, Task - def execute(self, context: ExecutionContext): - print(self.message * self.number) + class ProcessSentinel2Scene(Task): + scene_id: str + acquired_at: datetime + area_of_interest: Polygon - task = ParametrizableTask("Hello", 3, {"key": "value"}) + def execute(self, context: ExecutionContext) -> None: + context.logger.info("Processing Sentinel-2 scene", scene_id=self.scene_id) + + task = ProcessSentinel2Scene( + scene_id="S2A_20260818_32TPT", + acquired_at=datetime.fromisoformat("2026-08-18T10:30:00+00:00"), + area_of_interest=Polygon([ + (16.2, 48.1), + (16.5, 48.1), + (16.5, 48.3), + (16.2, 48.3), + ]), + ) ``` ```go Go - type ParametrizableTask struct { - Message string - Number int - Data map[string]string + type ProcessSentinel2Scene struct { + SceneID string + AcquiredAt time.Time + Bounds [4]float64 } - func (t *ParametrizableTask) Execute(context.Context) error { - slog.Info(strings.Repeat(t.Message, t.Number)) + func (t *ProcessSentinel2Scene) Execute(context.Context) error { + slog.Info("Processing Sentinel-2 scene", "scene_id", t.SceneID) return nil } - task := &ParametrizableTask{ - message: "Hello", - number: 3, - data: map[string]string{"key": "value"}, + task := &ProcessSentinel2Scene{ + SceneID: "S2A_20260818_32TPT", + AcquiredAt: time.Date(2026, time.August, 18, 10, 30, 0, 0, time.UTC), + Bounds: [4]float64{16.2, 48.1, 16.5, 48.3}, } ``` ## Task Composition and subtasks -Until now, tasks have performed only a single operation. But tasks can be more powerful. **Tasks can submit other tasks as subtasks.** This allows for a modular workflow design, breaking down complex operations into simpler, manageable parts. Additionally, the execution of subtasks is automatically parallelized whenever possible. +**A task can submit other tasks as subtasks.** This breaks complex operations into smaller units that Tilebox can execute in parallel when their dependencies allow it. ```python Python @@ -186,7 +187,7 @@ Parent task do not have access to results of subtasks, instead, tasks can use [s ### Larger subtasks example -A practical workflow example showcasing task composition might help illustrate the capabilities of tasks. Below is an example of a set of tasks forming a workflow capable of downloading a set number of random dog images from the internet. The [Dog API](https://thedogapi.com/) can be used to get the image URLs, and then download them. Implementing this using Task Composition could look like this: +This task composition example downloads random dog images from the internet. `DownloadRandomDogImages` fetches image URLs from the [Dog API](https://thedogapi.com/) and submits one `DownloadImage` task for each URL: ```python Python @@ -372,7 +373,7 @@ Every task goes through a set of states during its lifetime. - If the task fails, it transitions to `FAILED`, unless it's an [optional task](#optional-tasks), or nested within an [optional task](#nested-optional-tasks), in which case it transitions to `FAILED_OPTIONAL`. - As soon as all subtasks of a task are `COMPUTED` (or `FAILED_OPTIONAL`), the task is considered `COMPLETED`, allowing dependent tasks to be executed. -The table below summarizes the different task states and their meanings. +Each task state has the following meaning: | Task State | Description | |------------|-------------| @@ -400,7 +401,7 @@ The table below summarizes the different task states and their meanings. ## Map-Reduce Pattern Often times the input to a task is a list, with elements that should then be **mapped** to individual subtasks, whose results are later aggregated in a **reduce** step. This pattern is commonly known as [MapReduce](https://en.wikipedia.org/wiki/MapReduce) and a common pattern in workflows. In Tilebox, the reduce step is typically defined as a separate task that depends on all the map tasks. -For example, the workflow below applies this pattern to a list of numbers to calculate the sum of all squares of the numbers. The `Square` task takes a single number and squares it, and the `Sum` task reduces the list of squared numbers to a single sum. +This MapReduce workflow calculates the sum of the squares of a list of numbers. The `Square` task maps each number to its square, and the `Sum` task reduces those results to one value. ```python Python @@ -478,7 +479,7 @@ Computed sum of squares result=336448 ## Recursive subtasks Tasks can not only submit other tasks as subtasks, but also instances of themselves. This allows for a recursive breakdown of a task into smaller chunks. Such recursive decomposition algorithms are referred to as [divide and conquer algorithms](https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm). -For example, the `RecursiveTask` below is a valid task that submits smaller instances of itself as subtasks. +`RecursiveTask` demonstrates this pattern by submitting smaller instances of itself as subtasks. When implementing a recursive task, it's important to define a base case that stops the recursion. Otherwise, the task will keep submitting subtasks indefinitely, resulting in an infinite loop. @@ -514,9 +515,9 @@ For example, the `RecursiveTask` below is a valid task that submits smaller inst ### Recursive subtask example -An example for this is the [random dog images workflow](#larger-subtasks-example) mentioned earlier. In the previous implementation, downloading images was already parallelized. But the initial orchestration of the individual download tasks was not parallelized, because `DownloadRandomDogImages` was responsible for fetching all random dog image URLs and only submitted the individual download tasks once all URLs were retrieved. For a large number of images this setup can bottleneck the entire workflow. +The non-recursive [random dog images workflow](#larger-subtasks-example) waits for `DownloadRandomDogImages` to retrieve every URL before submitting any download tasks. For large batches, this delays the first downloads and can bottleneck orchestration. -To improve this, recursive subtask submission decomposes a `DownloadRandomDogImages` task with a high number of images into two smaller `DownloadRandomDogImages` tasks, each fetching half. This process can be repeated until a specified threshold is met, at which point the Dog API can be queried directly for image URLs. That way, image downloads start as soon as the first URLs are retrieved, without initial waiting. +A recursive version decomposes a `DownloadRandomDogImages` task with a high number of images into two smaller `DownloadRandomDogImages` tasks, each fetching half. This repeats until a specified threshold is met, at which point the Dog API is queried directly for image URLs. Image downloads can then start as soon as the first URLs are retrieved. An implementation of this recursive submission may look like this: @@ -585,7 +586,7 @@ func (t *DownloadRandomDogImages) Execute(ctx context.Context) error { ``` -With this implementation, downloading a large number of images (for example, 9) results in the following tasks being executed: +Downloading nine images with the recursive implementation produces this task graph: A failed task may be picked up by any available runner and not necessarily the same one that it failed on. @@ -1125,7 +1124,7 @@ func (t *FinalTask) Execute(context.Context) error { ``` -In this example, `FlakyTask` is submitted as an optional subtask. If it fails, `FinalTask` still executes because it depends on an optional task. The job completes successfully. A visualization of such a job is shown below. +In this example, `FlakyTask` is submitted as an optional subtask. If it fails, `FinalTask` still executes because it depends on an optional task. The resulting job completes successfully: Optional Subtasks Workflow @@ -1278,9 +1277,9 @@ If instead `Step1B` was also marked as optional, `Step1C` and `Step2` would stil A task identifier is a unique string used by the Tilebox Workflow Orchestrator to identify the task. It's used by [runners](/workflows/concepts/runners) to map submitted tasks to a task class and execute them. It also serves as the default name in execution visualizations. -If unspecified, the identifier of a task defaults to the class name. For instance, the identifier of the `PrintHeadlines` task in the previous example is `"PrintHeadlines"`. This is good for prototyping, but not recommended for production, as changing the class name also changes the identifier, which can lead to issues during refactoring. It also prevents different tasks from sharing the same class name. +If unspecified, the identifier of a task defaults to the class name. For instance, the identifier of `PrintHeadlines` in the [task dependencies example](#dependencies-example) is `"PrintHeadlines"`. This default is useful for prototyping but not recommended for production: changing the class name also changes the identifier, and different tasks cannot share the same class name. -To address this, Tilebox Workflows offers a way to explicitly specify the identifier of a task. This is done by overriding the `identifier` method of the `Task` class. This method should return a unique string identifying the task. This decouples the task's identifier from its class name, allowing you to change the identifier without renaming the class. It also allows tasks with the same class name to have different identifiers. The `identifier` method can also specify a version number for the task—see the section on [semantic versioning](#semantic-versioning) below for more details. +To address this, Tilebox Workflows offers a way to explicitly specify the identifier of a task. This is done by overriding the `identifier` method of the `Task` class. This method should return a unique string identifying the task. This decouples the task's identifier from the class name, allowing you to change the identifier without renaming the class. It also allows tasks with the same class name to have different identifiers. The `identifier` method can also specify a version number; see [Semantic Versioning](#semantic-versioning). ```python Python @@ -1329,7 +1328,7 @@ func (t *MyTask2) Execute(context.Context) error { ## Semantic Versioning -As seen in the previous section, the `identifier` method can return a tuple of two strings, where the first string is the identifier and the second string is the version number. This allows for semantic versioning of tasks. +The `identifier` method can return both a stable identifier and a version number, allowing Tilebox to distinguish compatible task implementations. Versioning is important for managing changes to a task's execution method. It allows for new features, bug fixes, and changes while ensuring existing workflows operate as expected. Additionally, it enables multiple versions of a task to coexist, enabling gradual rollout of changes without interrupting production deployments.