Python API Reference¶

Write an output message through click.echo.

click.echo strips color from a pipe, follows the CliRunner color setting, and wraps a Windows console. The library writer does not.

Print a PluginWarning as an output warning. Pass other warnings on.

A plugin warning names the plugin and the problem. The source path and line that Python adds point into Portolan, which does not help the user.

Determine if JSON output should be used.

Checks both the global --format option and per-command --json flags. Global --format=json takes precedence, but per-command flags also work for backward compatibility.

Parameters:

Returns:

Output a JSON envelope to stdout.

Parameters:

Load .env from catalog and warn if config.yaml has sensitive settings.

This should be called after the actual catalog path is resolved (not in the global CLI callback) to ensure the correct .env is loaded when --catalog/--portolan-dir overrides the default.

Parameters:

Find and validate catalog root, or exit with git-style error.

Git-style behavior: walks up from cwd to find .portolan/config.yaml. Commands that need to operate on the entire catalog use this to find the root regardless of which subdirectory they're run from.

Parameters:

Returns:

Raises:

Resolve remote destination with precedence: CLI > env var > config.

Parameters:

Returns:

Resolve the AWS profile: CLI, then env var, then config.

A Portolan setting wins, because a user writes it for this tool. None means that nothing here names a profile. The sync layer then reads AWS_PROFILE, and falls back to the default profile, as boto3 and the AWS CLI do.

Parameters:

Returns:

Resolve AWS region with precedence: CLI > env var > config.

Parameters:

Returns:

Portolan - Publish and manage cloud-native geospatial data catalogs.

Initialize a new Portolan catalog.

Creates a catalog.json at the root level and a .portolan directory with management files (config.yaml, metadata.yaml). Also creates versions.json at the root.

Derives the catalog ID from the directory name. Pass --id to set it yourself. The ID is the catalog's public STAC identity, so a directory named for a step in your workflow makes a poor one.

PATH is the directory where the catalog should be created (default: current directory).

A license is required. Every collection inherits it from the catalog's metadata.yaml, and 'portolan add' refuses to write a collection without one. Pass an SPDX identifier, or 'other' with --license-url when the license has no identifier. Without --auto or --json you are prompted for it.

Use --auto to skip all prompts and use default values. Use --id, --title, and --description to set catalog metadata directly.

A logo is optional. Pass --logo with a local image to copy it into _assets/ and publish it as a rel="icon" link, or add one later with 'portolan logo'.

\b Examples: portolan init # Prompts for the license portolan init --auto --license CC-BY-4.0 # Skip prompts portolan init --title "My Catalog" --license MIT portolan init --id phl-housing --license CC-BY-4.0 portolan init --license other --license-url https://x.org/terms portolan init --backend iceberg --license CC0-1.0 portolan init --auto --license CC-BY-4.0 --logo brand.png

List all files in the catalog with tracking status.

Git-style behavior: automatically finds the catalog root by walking up from the current directory. Works from any subdirectory within a catalog. Use --catalog to override and specify an explicit path.

Shows all files organized by collection in a hierarchical tree view. Each file shows its tracking status, format type, and file size.

\b Status indicators: + = tracked (in versions.json, unchanged) + = untracked (on disk, not in versions.json) ~ = modified (in versions.json, checksum changed) ! = deleted (in versions.json, missing from disk)

\b Example output: censo-2010/ data/ (3 tracked, 2 untracked) + census-data.parquet (GeoParquet, 4.5MB) + metadata.parquet (GeoParquet, 1.2MB) + README.md (2KB) + style.json (1KB)

\b Examples: portolan list # List all files with status portolan list --collection demographics # Filter by collection portolan list --tracked-only # Show only tracked files portolan list --untracked-only # Show only untracked files portolan list --json # JSON output

Show local vs remote version state for collections.

Git-style status showing version sync state, modified files, and untracked files for each collection in the catalog.

\b Status information: Local version Current version in local versions.json Remote version Current version on remote (unless --offline) Sync state in_sync, ahead, behind, or unknown Modified Files changed since last version Untracked Files on disk not in versions.json Deleted Files in versions.json but missing from disk

\b Examples: portolan status # Status for all collections portolan status -c demographics # Status for one collection portolan status --offline # Skip remote check portolan status --json # JSON output for agents

Show information about a file, collection, or catalog.

TARGET can be: - A file path (e.g., demographics/census.parquet) - shows file metadata - A collection directory (e.g., demographics/) - shows collection metadata - Omitted - shows catalog-level metadata

\b Examples: portolan info demographics/census.parquet # File info portolan info demographics/ # Collection info portolan info # Catalog info portolan info demographics/census.parquet --json # JSON output

Validate a Portolan catalog against the Portolan spec.

Validation runs on rashid, which reports PTL-* rule ids citing the spec requirements they enforce. By default it checks metadata, STAC 1.1.0 structure, and the asset bytes themselves. The metadata and structural passes are offline. The data pass reads asset bytes, including remote https assets over HTTP range requests. Use --data-scope local to read only the assets inside the catalog tree. Use --no-data to skip the pass. With --fix, applies the mechanical repairs and re-checks.

The --live pass probes the published host for HTTP Range support and CORS headers. It also requests every link target under the base URL, and reports a linked file the host does not serve. It skips an upstream host only when it knows the base URL, from --url or publish.public_url. Without a base URL it probes every absolute href as declared, which can report CORS failures against a host you do not control.

PATH is the directory to check (default: current directory).

Use --metadata or --geo-assets to limit scope: - --metadata: Only validate the catalog (its own metadata, structure, data) - --geo-assets: Only check source files on disk for cloud-native status - Neither: Both (default)

Examples:

portolan check # Validate all (metadata + geo-assets)

portolan check --metadata # Validate the catalog only

portolan check --geo-assets # Check source files only

portolan check --data-scope local # Read only assets inside the catalog tree

portolan check --no-data # Skip reading asset bytes (faster)

portolan check --live # Also probe the published host

portolan check --fix # Fix what can be fixed, then re-check

portolan check --fix --dry-run # Preview all fixes

Scan a directory for geospatial files and potential issues.

Discovers files by extension, validates shapefile completeness, and reports issues that may cause problems during import.

PATH is the directory to scan (default: current directory).

\b Fix Mode: Use --fix to auto-rename files with: - Invalid characters (spaces, parentheses, non-ASCII) - Windows reserved names (CON, PRN, AUX, etc.) - Long paths (> 200 characters)

Use --dry-run to preview changes without applying.

Examples:

portolan scan # Scan current directory

portolan scan --json # JSON output in current directory

portolan scan /data/geospatial

portolan scan /large/tree --max-depth=2

portolan scan /data --no-recursive

portolan scan /data --fix --dry-run

portolan scan /data --fix

Track files in the catalog.

Accepts multiple paths like git add. Each path is processed independently with automatic collection inference based on directory structure.

Works like git: run from anywhere inside a catalog and it auto-detects the catalog root. Use --portolan-dir to override.

\b Item ID derivation: By default, the item ID is derived from the parent directory name. For example, adding 'census/2020/data.parquet' creates an item named '2020'. Use --item-id to override this automatic derivation. All other files in the item directory are tracked as companion assets.

\b Datetime handling: --datetime applies to ALL items added in this command. For items with different acquisition dates, run separate add commands:

    portolan add census/2020/ --datetime 2020-04-01
    portolan add census/2023/ --datetime 2023-04-01

If --datetime is omitted, items have a null temporal extent and carry
the sentinel start/end range. Run 'portolan check' to find items
needing dates.

\b Examples: portolan add demographics/census.parquet portolan add file1.geojson file2.geojson # Add multiple files portolan add imagery/ # Add all files in directory portolan add . # Add all files in catalog portolan add data.geojson --item-id my-id # Override item ID (single file only) portolan add sat.tif --datetime 2024-06-15 # Explicit acquisition date

Smart behavior: - Unchanged files are silently skipped (use --verbose to see them) - Changed files are re-extracted with new metadata - Sidecar files (.dbf, .shx, .prj for shapefiles) are auto-detected - All files in the item directory are tracked, not just geo files

\b Large file partitioning: GeoParquet files exceeding 2GB are automatically partitioned into spatial chunks using KD-tree partitioning. In interactive mode, you'll be prompted before partitioning. Configure via:

    partitioning.enabled: true/false (default: true)
    partitioning.prompt: true/false (default: true)
    partitioning.threshold_gb: size in GB (default: 2.0)

Register remote data as a collection WITHOUT downloading or converting it.

The external counterpart to 'portolan add'. Some valuable data sources are already published cloud-natively at a remote location and should be referenced in place rather than copied. This creates a collection.json whose collection-level 'data' asset href is the remote URL (kept as-is, carrying the 'external' role) plus a rel:via provenance link.

The remote asset is never fetched, and 'portolan check' will not try to convert it (the metadata scanner skips scheme-qualified hrefs).

A license is required. Pass an SPDX identifier, or 'other' with --license-url when the license has no identifier. Referencing someone else's data in place does not make its license unknowable, and a collection without one is an error under 'portolan check'.

\b Examples: # Overture Maps places — planet-scale GeoParquet on Overture's S3 portolan add-external \ "s3://overturemaps-us-west-2/release/2024-09-18.0/theme=places/type=place/*" \ --collection overture-places \ --title "Overture Maps — Places" \ --license ODbL-1.0 \ --via "https://docs.overturemaps.org/guides/places/"

portolan add-external "https://example.org/data/buildings.parquet" \\
    --license other --license-url "https://example.org/terms"

Remove files from tracking.

By default, removes the file from disk AND untracks it from the catalog. Requires --force for destructive operations (deleting files).

Works like git: run from anywhere inside a catalog and it auto-detects the catalog root. Use --portolan-dir to override.

A file that is already gone from disk is still untracked, so you can drop a tracked asset that you deleted by hand. The removal is recorded as a new version.

\b Safety flags: - --keep: Untrack file but preserve it on disk (safe, no --force needed) - --force: Required for destructive rm (when not using --keep) - --dry-run: Preview what would be removed without actually removing

\b Examples: portolan rm --keep imagery/old_data.tif # Safe: untrack only portolan rm --dry-run vectors/ # Preview what would be removed portolan rm -f demographics/census.parquet # Force delete and untrack portolan rm -f vectors/ # Force remove entire directory

Push local catalog changes to cloud object storage.

Git-style behavior: automatically finds the catalog root by walking up from the current directory. Works from any subdirectory within a catalog. Use --catalog to override and specify an explicit path.

Syncs collection(s) to a remote destination (S3, GCS, Azure). Uses optimistic locking to detect concurrent modifications.

DESTINATION is the object store URL (e.g., s3://mybucket/my-catalog). If not provided, uses 'remote' from PORTOLAN_REMOTE env var or .env file.

If --collection is specified, pushes that collection only. If --collection is omitted, pushes all collections in the catalog.

\b Examples: # Push a single collection portolan push s3://mybucket/catalog --collection demographics portolan push gs://mybucket/catalog -c imagery --dry-run

# Push all collections
portolan push s3://mybucket/catalog
portolan push --dry-run  # Uses configured remote

Pull updates from a remote catalog.

Git-style behavior: automatically finds the catalog root by walking up from the current directory. Works from any subdirectory within a catalog. Use --catalog to override and specify an explicit path.

Fetches changes from a remote catalog and downloads updated files. Similar to git pull, this checks for uncommitted local changes before overwriting.

REMOTE_URL is the remote catalog URL (e.g., s3://bucket/catalog).

If --collection is specified, pulls that collection only. If --collection is omitted, pulls all collections in the catalog.

\b Examples: # Pull a single collection portolan pull s3://mybucket/my-catalog --collection demographics portolan pull s3://mybucket/catalog -c imagery --dry-run

# Pull all collections
portolan pull s3://mybucket/catalog
portolan pull s3://mybucket/catalog --workers 4

Sync local catalog with remote storage (pull + push).

Orchestrates a full sync workflow: Pull -> Init -> Scan -> Check -> Push. This is the recommended way to keep a local catalog in sync with remote.

DESTINATION is the object store URL (e.g., s3://mybucket/my-catalog).

\b Examples: portolan sync s3://mybucket/catalog --collection demographics portolan sync s3://mybucket/catalog -c imagery --dry-run portolan sync s3://mybucket/catalog -c data --fix --force portolan sync s3://mybucket/catalog -c data --profile prod portolan sync --collection demographics # Uses configured remote

Clone a remote catalog to a local directory.

This is essentially "pull to an empty directory" with guardrails. Creates the target directory and pulls collections from remote storage.

REMOTE_URL is the object store URL (e.g., s3://mybucket/my-catalog).

LOCAL_PATH is optional - if not provided, it will be inferred from the catalog name in the URL (git clone style).

--collection is optional - if not provided, all collections in the remote catalog will be cloned.

\b Examples: # Infer directory from URL, clone all collections portolan clone s3://mybucket/my-catalog

# Clone to current directory (must be empty)
portolan clone s3://mybucket/my-catalog.

# Clone specific collection
portolan clone s3://mybucket/catalog -c demographics

# Clone all collections to specific directory
portolan clone s3://mybucket/catalog ./local-copy

# Clone specific collection with profile
portolan clone s3://mybucket/catalog ./data -c imagery --profile prod

Manage catalog configuration.

Configuration is stored in .portolan/config.yaml and follows this precedence:

\b 1. CLI argument (highest) 2. Environment variable (PORTOLAN_) or .env file 3. Collection-level config 4. Catalog-level config 5. Built-in default (lowest)

Note: Sensitive settings (remote, profile, region) must use env vars or .env.

\b Examples: portolan config set backend iceberg portolan config get remote portolan config list

Set a configuration value.

KEY is the setting name (e.g., backend, statistics.enabled). VALUE is the value to set.

Note: Sensitive settings (remote, profile, region) cannot be stored in config.yaml. Use environment variables or .env files instead.

\b Examples: portolan config set backend iceberg portolan config set statistics.enabled true portolan config set pmtiles.enabled false --collection demographics

Get a configuration value.

Shows the resolved value and its source (env, catalog, collection, or not set).

KEY is the setting name (e.g., remote, aws_profile).

\b Examples: portolan config get remote portolan config get aws_profile --collection restricted

List all configuration settings.

Shows all settings with their values and sources.

\b Examples: portolan config list portolan config list --collection demographics

Remove a configuration value.

Removes the setting from the config file. Does not affect environment variables.

KEY is the setting name to remove.

\b Examples: portolan config unset remote portolan config unset aws_profile --collection restricted

Remove all Portolan metadata while preserving data files.

Removes catalog.json, collection.json, item.json (STAC metadata), versions.json, and the .portolan/ directory. Preserves all data files (.parquet, .tif, .gpkg, .geojson, etc.).

Use --dry-run to preview what would be removed without deleting anything.

\b Examples: portolan clean # Remove all metadata portolan clean --dry-run # Preview what would be removed

Manage catalog metadata for README generation.

metadata.yaml files supplement STAC with human-enrichable fields like titles, descriptions, contact info, and citations. These files can exist at any level in the catalog hierarchy (catalog, subcatalog, collection).

\b Examples: portolan metadata init # Create template at catalog root portolan metadata init demographics # Create template for collection portolan metadata validate # Validate metadata.yaml

Generate a metadata.yaml template.

Creates .portolan/metadata.yaml files at all STAC levels (catalogs, subcatalogs, collections) by default. Skips items (item.json directories) and preserves existing files unless --force is used.

If PATH is provided, starts from that directory. Otherwise, starts at the catalog root.

\b Examples: portolan metadata init # All levels in catalog portolan metadata init climate # All levels under climate/ portolan metadata init --force # Overwrite existing portolan metadata init --no-recursive # Only at catalog root portolan metadata init demographics --no-recursive # Only for collection

Validate metadata.yaml against schema.

Validates all .portolan/metadata.yaml files in the catalog tree by default. Checks for: - Required fields: contact (name + email), license - Format validation: email, SPDX license identifier, DOI

Uses hierarchical resolution: child metadata.yaml files inherit from parent levels and override specific fields.

\b Examples: portolan metadata validate # Validate all levels portolan metadata validate climate # Validate under climate/ portolan metadata validate --no-recursive # Only at catalog root portolan metadata validate demographics --no-recursive # Only for collection

Generate README.md from STAC metadata and metadata.yaml.

Generates READMEs for the catalog and all collections by default. The README is a pure output - always generated from STAC (machine-extracted metadata) plus .portolan/metadata.yaml (human enrichment). Never hand-edit the README; edit metadata.yaml instead and regenerate.

Use --check in CI to verify READMEs are up-to-date:

\b Examples: portolan readme # Generate for catalog and all collections portolan readme climate # Generate under climate/ portolan readme --check # CI mode: exit 1 if any stale portolan readme --no-recursive # Only at catalog root portolan readme demographics --no-recursive # Only for collection portolan readme --stdout --no-recursive # Print single README

Publish SOURCE as the catalog logo.

Copies the image to _assets/ beside the root catalog.json and writes a rel="icon" link pointing at it. A registry lists many catalogs side by side, and the logo is what makes yours recognizable.

SOURCE must be a local file in one of the seven permitted formats: APNG, AVIF, GIF, JPEG, PNG, SVG, or WebP. A URL is rejected — download the image first. SVG conforms but STAC Browser will not render it.

Re-run to replace the logo: the previous link and image are removed, so the catalog always carries exactly one.

\b Examples: portolan logo brand.png # Title from the catalog portolan logo brand.png --title "Acme GIS" # Explicit title

Extract data from external sources into Portolan catalogs.

Convert data from ArcGIS services, APIs, or other sources into well-structured Portolan catalogs with STAC metadata.

\b Examples: portolan extract arcgis https://services.arcgis.com/.../FeatureServer ./output portolan extract arcgis URL --layers "Census" --dry-run portolan extract arcgis URL --filter "sdn_" --resume

Partition a large GeoParquet file for better query performance.

Splits a GeoParquet file into spatially-organized partitions using geoparquet-io. Per OGC best practices, files over 2GB should be partitioned.

\b Output structure (Hive-style): output_dir/ ├── kdtree_cell=001/ │ └── data.parquet ├── kdtree_cell=002/ │ └── data.parquet └── ...

\b Examples: # Preview partition strategy portolan partition buildings.parquet --preview

# Partition with default settings (kdtree, 120k rows/partition)
portolan partition buildings.parquet output/

# Custom target rows
portolan partition buildings.parquet output/ --target-rows 50000

Version management commands.

Works with any versioning backend (file, iceberg). Backend is auto-detected from catalog configuration.

\b Subcommands: current Show current version of a collection list List all versions of a collection rollback Rollback to a previous version (iceberg only) prune Remove old versions (iceberg only)

Show the current version of a collection.

Works with any versioning backend (auto-detected from config).

\b Examples: portolan version current boundaries portolan version current boundaries --json

List all versions of a collection.

Works with any versioning backend (auto-detected from config).

\b Examples: portolan version list boundaries portolan version list boundaries --json

Create a new version from current file state.

Detects modified files by comparing checksums, computes new checksums, and creates a new version entry in versions.json.

NEW_VERSION must be an explicit semver string (e.g., "1.2.0").

\b Examples: portolan version bump demographics 1.4.0 -m "Updated source data" portolan version bump demographics 2.0.0 --breaking -m "Schema change" portolan version bump demographics 1.4.0 -y # Skip confirmation

Generate items.parquet for efficient STAC queries.

Creates a GeoParquet file containing all items in a collection, enabling fast spatial/temporal queries without N HTTP requests.

This is optional but recommended for collections with >100 items. The parquet file is added as a link in collection.json.

If --collection is omitted, generates for ALL collections in the catalog.

\b Examples: portolan stac-geoparquet # Generate for ALL collections portolan stac-geoparquet -c landsat # Generate for landsat collection portolan stac-geoparquet -c imagery --dry-run # Preview without creating portolan stac-geoparquet --json # JSON output for all collections

List and view AI skills for Portolan workflows.

Skills have moved to: https://github.com/portolan-sdi/portolan-skills

List available skills.

View a skill.