Storage & Pipelines¶

from sqlspec.storage import (
    AsyncStoragePipeline,
    PartitionStrategyConfig,
    ResolvedStorageTarget,
    StorageBridgeJob,
    StorageCapabilities,
    StorageDestination,
    StorageFormat,
    StorageRegistry,
    StorageTelemetry,
    SyncStoragePipeline,
    create_storage_bridge_job,
    resolve_storage_path,
    storage_registry,
)

Multi-backend object storage registry (local, fsspec, obstore) and Apache Arrow / Parquet / JSONL streaming storage pipelines. See Arrow & Storage for configuration and usage examples.

Storage Registry¶

StorageRegistry ¶

Global storage registry for named backend configurations.

Allows registering named storage backends that can be accessed from anywhere in your application. Backends are automatically selected based on URI scheme unless explicitly overridden.

register_alias ¶

register_alias(alias, uri, *, backend=None, base_path='', **kwargs)

Register a named alias for a storage configuration.

Parameters:

Name	Type	Description	Default
`alias`	`str`	Unique alias name	required
`uri`	`str`	Storage URI	required
`backend`	`str \| None`	Force specific backend ("local", "fsspec", "obstore") instead of auto-detection	`None`
`base_path`	`str`	Base path to prepend to all operations	`''`
`**kwargs`	`Any`	Backend-specific configuration options	`{}`

get ¶

get(uri_or_alias, *, backend=None, **kwargs)

Get backend instance using URI-first routing with automatic backend selection.

Parameters:

Name	Type	Description	Default
`uri_or_alias`	`str \| Path`	URI to resolve directly OR named alias	required
`backend`	`str \| None`	Force specific backend ("local", "fsspec", "obstore") instead of auto-selection	`None`
`**kwargs`	`Any`	Additional backend-specific configuration options	`{}`

Returns:

Type	Description
`ObjectStoreProtocol`	Backend instance with automatic backend selection

Raises:

Type	Description
`ImproperConfigurationError`	If alias not found or invalid input

is_alias_registered ¶

is_alias_registered(alias)

Check if a named alias is registered.

list_aliases ¶

list_aliases()

List all registered aliases.

clear_cache ¶

clear_cache(uri_or_alias=None)

Clear resolved backend cache.

clear ¶

clear()

Clear all aliases and instances.

clear_instances ¶

clear_instances()

Clear only cached instances, keeping aliases.

clear_aliases ¶

clear_aliases()

Clear only aliases, keeping cached instances.

resolve_storage_path ¶

resolve_storage_path(
    path, base_path="", protocol=FILE_PROTOCOL, strip_file_scheme=True
)

Resolve path relative to base_path with protocol-specific handling.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path`	Path to resolve.	required
`base_path`	`str`	Base path to prepend if path is relative.	`''`
`protocol`	`str`	Storage protocol.	`FILE_PROTOCOL`
`strip_file_scheme`	`bool`	Whether to strip `file://` prefixes.	`True`

Returns:

Type	Description
`str`	Resolved path string suitable for the storage backend.

Raises:

Type	Description
`StoragePathTraversalError`	If the path contains a parent reference or escapes `base_path`.

Storage Pipelines¶

SyncStoragePipeline ¶

Bases: _StoragePipelineBase

Pipeline coordinating storage registry operations and telemetry.

write_rows ¶

write_rows(rows, destination, *, format_hint=None, storage_options=None)

Write dictionary rows to storage using cached serializers.

write_arrow ¶

write_arrow(
    table,
    destination,
    *,
    format_hint=None,
    storage_options=None,
    compression=None,
)

Write an Arrow table to storage using zero-copy buffers.

read_arrow ¶

read_arrow(source, *, file_format, storage_options=None)

Read an artifact from storage and decode it into an Arrow table.

stream_read ¶

stream_read(source, *, chunk_size=None, storage_options=None)

Stream bytes from an artifact.

AsyncStoragePipeline ¶

Bases: _StoragePipelineBase

Async variant of the storage pipeline leveraging async-capable backends when available.

write_rows ¶

write_rows(rows, destination, *, format_hint=None, storage_options=None)

write_arrow ¶

write_arrow(
    table,
    destination,
    *,
    format_hint=None,
    storage_options=None,
    compression=None,
)

read_arrow_async ¶

read_arrow_async(source, *, file_format, storage_options=None)

stream_read_async ¶

stream_read_async(source, *, chunk_size=None, storage_options=None)

Stream bytes from an artifact asynchronously.

Bridge Jobs, Capabilities & Telemetry¶

StorageBridgeJob ¶

Bases: NamedTuple

Handle representing a storage bridge operation.

job_id ¶

job_id

status ¶

status

telemetry ¶

telemetry

StorageCapabilities ¶

Bases: TypedDict

Runtime-evaluated driver storage capabilities.

Attributes:

Name	Type	Description
`arrow_export_enabled`	`bool`	Whether the driver supports exporting query results as Apache Arrow tables or batches.
`arrow_import_enabled`	`bool`	Whether the driver supports ingesting Apache Arrow tables or record batches.
`parquet_export_enabled`	`bool`	Whether direct or staged Parquet export is available.
`parquet_import_enabled`	`bool`	Whether direct or staged Parquet ingestion is available.
`partition_strategies`	`list[str]`	Supported partition fan-out strategy names (such as `fixed`, `rows_per_chunk`, or `manifest`).
`default_storage_profile`	`NotRequired[str \| None]`	Named storage registry profile used when callers omit an explicit backend URI.

PartitionStrategyConfig ¶

Bases: TypedDict

Configuration for partition fan-out strategies.

Attributes:

Name	Type	Description
`kind`	`str`	Partitioning strategy identifier (`fixed`, `rows_per_chunk`, or `manifest`).
`partitions`	`int`	Target number of output partitions to produce.
`rows_per_chunk`	`int`	Maximum row count written per output file or batch chunk.
`manifest_path`	`str`	Optional destination URI or relative path for writing a partition manifest.

ResolvedStorageTarget ¶

Bases: NamedTuple

A storage destination resolved to an address and its backend protocol.

uri ¶

uri

protocol ¶

protocol

StorageTelemetry ¶

Bases: TypedDict

Telemetry payload for storage bridge operations.

Attributes:

Name	Type	Description
`destination`	`str`	Resolved target or source storage URI for the bridge transfer.
`bytes_processed`	`int`	Total uncompressed or encoded payload bytes read or written.
`rows_processed`	`int`	Total number of records transferred across the storage bridge.
`partitions_created`	`int`	Number of partitioned output objects written during export.
`duration_s`	`float`	Elapsed wall-clock transfer time in seconds.
`format`	`str`	Serialized wire/file format (such as `parquet`, `arrow`, `csv`, or `jsonl`).
`extra`	`dict[str, object]`	Adapter-specific diagnostic counters or native COPY options.
`backend`	`str`	Storage backend implementation name (`local`, `obstore`, or `fsspec`).
`correlation_id`	`str`	Active request correlation identifier attached to the storage job.
`config`	`str`	Adapter configuration class name executing the storage transfer.
`bind_key`	`str`	Registered SQLSpec configuration bind key, when configured.

create_storage_bridge_job ¶

create_storage_bridge_job(status, telemetry)

Create a storage bridge job handle with a unique identifier.

get_storage_bridge_diagnostics ¶

get_storage_bridge_diagnostics()

Return aggregated storage bridge + serializer cache metrics.

get_storage_bridge_metrics ¶

get_storage_bridge_metrics()

Return aggregated storage bridge metrics.

reset_storage_bridge_metrics ¶

reset_storage_bridge_metrics()

Reset aggregated storage bridge metrics.