CLI reference

dtst

dtst - dataset toolkit for datasets creation and curation.

Usage:

dtst [OPTIONS] COMMAND [ARGS]...

Options:

dtst analyze

Compute image metrics and write JSON sidecars.

Analyzes images in the source folders and writes per-image sidecar JSON files containing the requested metrics. Sidecars are merged incrementally — running with different metrics accumulates results.

CPU metrics: phash, blur. IQA metrics (GPU-accelerated): any metric from IQA-PyTorch (e.g. musiq, clipiqa, topiq_nr, dbcnn, hyperiqa, niqe, brisque).

Examples: dtst analyze --from raw --metrics phash,blur -d ./my-dataset dtst analyze config.yaml --metrics phash dtst analyze --from raw --metrics musiq,clipiqa -d ./my-dataset dtst analyze --from raw --metrics phash,blur,musiq --force dtst analyze --from raw --clear -d ./my-dataset

Usage:

dtst analyze [OPTIONS] [CONFIG]

Options:

dtst annotate

Write source and license metadata into image sidecars.

Annotates all images in the given folders with provenance metadata (source, license, origin). Useful for manually imported images that were not fetched through the pipeline. Sidecars are merged incrementally — existing fields are preserved unless --overwrite is used.

At least one of --source, --license, or --origin is required.

Examples: dtst annotate --from extra --source "unsplash" --license "cc0" -d ./my-dataset dtst annotate config.yaml dtst annotate --from raw,extra --source "personal" --license "all-rights-reserved" dtst annotate --from extra --source "flickr" --overwrite -d ./my-dataset dtst annotate --from extra --source "personal" --dry-run

Usage:

dtst annotate [OPTIONS] [CONFIG]

Options:

dtst augment

Augment a dataset by applying image transformations.

Reads images from one or more source folders and writes transformed copies to a destination folder. By default the original images are also copied to the output; use --no-copy to write only the transformed versions.

At least one transform flag (--flipX, --flipY, --flipXY) is required. Multiple flags can be combined in a single run to produce several variants of each image.

Transformed files are named with a suffix indicating the transform: photo.jpg becomes photo_flipX.jpg, photo_flipY.jpg, photo_flipXY.jpg.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst augment -d ./project --from faces --to augmented --flipX
dtst augment -d ./project --from faces --to augmented --flipX --flipY --flipXY
dtst augment -d ./project --from faces --to augmented --flipX --no-copy
dtst augment config.yaml --dry-run

Usage:

dtst augment [OPTIONS] [CONFIG]

Options:

dtst cluster

Cluster images by visual similarity.

Groups images into clusters based on embedding similarity using HDBSCAN. Each cluster is written to a numbered subdirectory (000 = largest, 001 = second largest, etc.) within the output folder. Images that do not belong to any cluster are placed in a noise/ subdirectory.

Supports two embedding models: arcface for face identity clustering (requires face images) and clip for general visual similarity clustering (works with any images).

--min-cluster-size sets the smallest group HDBSCAN will consider a real cluster (default: 5). Raise it to suppress small or spurious clusters; lower it to capture smaller groups.

--min-samples controls how conservative the density estimate is (default: 2). It decides how many close neighbors a point needs before it can join a cluster. Lower values (1-2) let borderline images in; higher values push more images into the noise folder. Keeping this low while adjusting --min-cluster-size is usually the best starting point.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst cluster config.yaml
dtst cluster -d ./project --from faces --to clusters
dtst cluster -d ./project --model clip --from raw --to clusters
dtst cluster -d ./project --top 3 --min-cluster-size 10
dtst cluster -d ./project --min-samples 1 --min-cluster-size 8
dtst cluster config.yaml --model arcface --dry-run

Usage:

dtst cluster [OPTIONS] [CONFIG]

Options:

dtst dedup

Deduplicate images by perceptual hash similarity.

Groups images by phash hamming distance and keeps the best image from each duplicate group. By default, original (non-upscaled) images are preferred; use --prefer-upscaled to reverse this. Within each preference tier, the winner is chosen by resolution (width x height), then file size, then blur sharpness. Losers are moved to a duplicated/ subdirectory within the source folder (configurable with --to).

Requires phash sidecar data from dtst analyze --metrics phash. Blur scores (from dtst analyze --metrics blur) are used as a tiebreaker when available.

Examples: dtst dedup -d ./project --from faces dtst dedup -d ./project --from faces --threshold 4 dtst dedup -d ./project --from faces --to my-dupes dtst dedup config.yaml --dry-run dtst dedup -d ./project --from faces --clear

Usage:

dtst dedup [OPTIONS] [CONFIG]

Options:

dtst detect

Detect objects in images using OWL-ViT 2.

Uses open-vocabulary object detection to find specific objects in images and writes the results into per-image sidecar JSON files under a "classes" key. Each class gets all detections (score + bounding box) sorted by confidence, or null if not found.

Each run replaces the entire "classes" key in the sidecar.

Examples: dtst detect -d ./project --from raw --classes "microphone,chair,table" dtst detect config.yaml dtst detect -d ./project --from raw --classes "microphone" --threshold 0.4 dtst detect -d ./project --from raw --classes "microphone" --dry-run dtst detect -d ./project --from raw --clear

Usage:

dtst detect [OPTIONS] [CONFIG]

Options:

dtst extract-classes

Extract image crops from class detection bounding boxes.

Reads class detections from sidecar JSON files (produced by dtst detect) and crops the corresponding regions from each image. Supports expanding the bounding box with a margin ratio and squaring the crop.

Examples:

dtst extract-classes config.yaml
dtst extract-classes config.yaml --classes flower --square --margin 0.1
dtst extract-classes -d ./dahlias --from images --to flowers --classes flower
dtst extract-classes config.yaml --min-score 0.5 --skip-partial

Usage:

dtst extract-classes [OPTIONS] [CONFIG]

Options:

dtst extract-faces

Extract aligned face crops from images.

Detects faces in each image using MediaPipe (default) or dlib, then produces an aligned and cropped face image for each detection. The alignment normalises eye and mouth positions for consistent face crops.

Reads images from one or more source folders within the working directory and writes face crops to a destination folder. Multiple source folders can be specified as a comma-separated list with --from.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst extract-faces config.yaml
dtst extract-faces config.yaml --engine dlib --max-size 512
dtst extract-faces -d ./crowd --from raw --to faces
dtst extract-faces -d ./crowd --from raw,extra --to faces
dtst extract-faces config.yaml --max-faces 3 --no-padding

Usage:

dtst extract-faces [OPTIONS] [CONFIG]

Options:

dtst extract-frames

Extract keyframes from video files using ffmpeg.

Reads video files from one or more source folders and extracts keyframes (I-frames) to a destination folder. Each video produces a set of numbered images named as {video_stem}_{frame_number}.{format}.

Only I-frames are decoded, which avoids interpolated or blurry frames and produces the sharpest possible output. The --keyframes option sets the minimum interval between extracted frames: with the default of 10, at most one keyframe every 10 seconds is kept. Lower values produce more frames, higher values produce fewer.

Supported video formats: .mp4, .mkv, .avi, .mov, .webm, .flv, .wmv, .m4v.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst extract-frames -d ./project --from videos --to frames
dtst extract-frames -d ./project --from videos --to frames --keyframes 5
dtst extract-frames -d ./project --from videos --to frames --keyframes 30 --format png
dtst extract-frames config.yaml
dtst extract-frames config.yaml --keyframes 20 --dry-run

Usage:

dtst extract-frames [OPTIONS] [CONFIG]

Options:

dtst fetch

Download images and videos from a URL list.

Reads a URL list from the working directory specified by --input. Two formats are supported:

.jsonl JSON Lines with a "url" field per line (search output). Supports --min-size and --license filtering. .txt Plain text with one URL per line. Lines starting with # are treated as comments.

URLs are routed automatically: known video hosting domains (YouTube, Vimeo, etc.) are downloaded with yt-dlp, all other URLs are downloaded directly with HTTP requests.

Image files are named by the MD5 hash of the URL. Video files are named by yt-dlp using the video ID and original extension. Existing files are skipped unless --force is set.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst fetch config.yaml
dtst fetch -d ./chanterelle --to raw --input results.jsonl
dtst fetch -d ./project --to videos --input urls.txt
dtst fetch config.yaml --workers 16 --timeout 60
dtst fetch config.yaml --force
dtst fetch -d ./chanterelle --to raw --input results.jsonl --no-wait --license cc

Usage:

dtst fetch [OPTIONS] [CONFIG]

Options:

dtst format

Convert and normalize image formats, channels, and metadata.

Reads images from source folders and writes converted copies to a destination folder. Can change format (jpg/png/webp), enforce channel mode (rgb/grayscale), and strip EXIF metadata.

When --format is omitted the source format is preserved, but other transformations (--channels, --strip-metadata) still apply.

Examples: dtst format -d ./project --from raw --to formatted -f jpg -q 90 dtst format -d ./project --from raw --to clean --strip-metadata --channels rgb dtst format -d ./project --from raw --to gray --channels grayscale dtst format config.yaml --dry-run

Usage:

dtst format [OPTIONS] [CONFIG]

Options:

dtst frame

Resize images to a target width and/or height.

Reads images from one or more source folders and writes resized copies to a destination folder. Uses Lanczos resampling for high-quality downscaling.

When both --width and --height are given, the --mode option controls how aspect ratio differences are handled:

stretch Resize to exact dimensions, distorting if needed. crop Scale to cover the target area, then trim excess (default). pad Scale to fit within the target area, then fill the rest.

When only one dimension is given, the other is computed proportionally and --mode is ignored.

Examples:

dtst frame -d ./project --from faces --to resized -W 512 -H 512
dtst frame -d ./project --from faces --to resized -W 512 -H 512 --mode pad --fill blur
dtst frame -d ./project --from faces --to resized -W 512 -H 512 --mode crop --gravity top
dtst frame -d ./project --from faces --to resized --width 512
dtst frame config.yaml --dry-run

Usage:

dtst frame [OPTIONS] [CONFIG]

Options:

dtst rename

Sequentially rename images in-place with a prefix and zero-padded number.

Renames all images in the given folders to {prefix}{number}.{ext}, where the number is zero-padded to the specified number of digits. Sidecar JSON files are moved along with their images. Operates in-place — there is no --to option.

Examples: dtst rename --from raw --prefix "img_" -d ./my-dataset dtst rename --from raw --prefix "photo_" --digits 5 -d ./my-dataset dtst rename config.yaml --dry-run dtst rename --from faces --prefix "face_" -n 4

Usage:

dtst rename [OPTIONS] [CONFIG]

Options:

dtst review

Launch a web UI for manual image review.

Opens a local web server with an image grid. Click images to select or deselect them, then apply to move filtered images into a subfolder. Use the view toggle to switch between source and filtered images to restore previously filtered images.

Press Ctrl+C to stop the server.

Examples: dtst review config.yaml dtst review -d ./project --from faces dtst review -d ./project --from faces --to rejected --port 9000 dtst review config.yaml --no-open

Usage:

dtst review [OPTIONS] [CONFIG]

Options:

dtst run

Run a named workflow defined in a config file.

Executes a sequence of dtst commands and shell commands as defined in the workflows section of the config file. Each command step inherits its defaults from the corresponding config section unless inherit: false is set.

Examples: dtst run pipeline config.yaml dtst run pipeline config.yaml --dry-run dtst run pipeline config.yaml -d ./my_dataset

Usage:

dtst run [OPTIONS] WORKFLOW CONFIG

Options:

dtst search

Search for images across multiple engines.

Reads an optional YAML config file and generates image URLs from Flickr, Serper (Google Images), Brave, Wikimedia Commons, and iNaturalist. Text-based engines use an expanded query matrix of search terms and suffixes. iNaturalist uses taxon IDs instead. Results are deduplicated and written to a JSONL file in the working directory (default: results.jsonl) so multiple runs accumulate new results.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Query matrix: By default, the command runs two kinds of queries for each term: (1) the term alone, e.g. "chanterelle"; (2) the term with each suffix, e.g. "chanterelle mushroom", "chanterelle forest". Use --suffix-only to run only the second kind.

Examples:

dtst search config.yaml
dtst search config.yaml --dry-run
dtst search config.yaml --max-pages 3 --engines flickr,wikimedia
dtst search --terms "chanterelle" --suffixes "mushroom,forest" --engines brave -d ./chanterelle
dtst search --taxon-ids 47169,54743 -d ./fungi

Usage:

dtst search [OPTIONS] [CONFIG]

Options:

dtst select

Select images from source folders into a destination folder.

Copies (or moves with --move) images from one or more source folders into a destination folder. When filter criteria are provided, only images that pass all criteria are selected. Without criteria, all images are selected.

Files that already exist in the destination (by name) are skipped.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples: dtst select -d ./project --from raw --to backup dtst select -d ./project --from raw,extra --to combined dtst select -d ./project --from faces --to curated --min-side 256 dtst select -d ./project --from faces --to curated --min-metric blur 5 dtst select -d ./project --from faces --to curated --min-metric blur 5 --min-metric musiq 60 dtst select -d ./project --from faces --to curated --max-metric brisque 40 dtst select -d ./project --from raw --to clean --max-detect microphone 0.5 dtst select -d ./project --from raw --to licensed --source serper,flickr dtst select config.yaml --dry-run

Usage:

dtst select [OPTIONS] [CONFIG]

Options:

dtst upscale

Upscale images using AI super-resolution models.

Reads images from one or more source folders and writes upscaled copies to a destination folder. Uses spandrel to load PyTorch super-resolution models (Real-ESRGAN, SwinIR, HAT, etc.).

By default uses a 4x Real-ESRGAN model. Use --scale to choose between 2x and 4x upscaling, or --model to provide a custom .pth weights file (scale is auto-detected from the model).

Use --denoise to control how much denoising is applied (4x only). 0.0 preserves the most texture, 1.0 applies full denoising. This activates a lighter general-purpose model with controllable denoise strength via weight interpolation.

Large images are processed in tiles to avoid GPU memory issues. Adjust --tile-size to control memory usage (smaller = less VRAM).

Examples: dtst upscale -d ./project --from faces --to upscaled dtst upscale -d ./project --from faces --to upscaled --scale 2 dtst upscale -d ./project --from faces --to upscaled --denoise 0.5 dtst upscale -d ./project --from faces --to upscaled --model ./custom.pth dtst upscale config.yaml --dry-run

Usage:

dtst upscale [OPTIONS] [CONFIG]

Options:

dtst validate

Validate that all images in a folder are consistent.

Checks that every image shares the same dimensions and channel mode. Optionally checks that images are square. Warns if any PNG files use compression (which slows down loading).

Examples: dtst validate --from faces -d ./my-dataset dtst validate --from faces --square -d ./my-dataset dtst validate config.yaml

Usage:

dtst validate [OPTIONS] [CONFIG]

Options: