dtst

dtst - dataset toolkit for datasets creation and curation.

Usage:

dtst [OPTIONS] COMMAND [ARGS]...

Options:

dtst analyze

Compute image metadata and write JSON sidecars.

Analyzes images in the source folders and writes per-image sidecar JSON files containing the requested metadata (perceptual hash, blur score, or both). Sidecars are merged incrementally — running with --phash then --blur accumulates both.

At least one analyzer flag (--phash, --blur) is required unless using --clear.

Examples:

dtst analyze --from raw --phash --blur -d ./my-dataset dtst analyze config.yaml --phash dtst analyze --from raw,extra --blur --force dtst analyze --from raw --phash --dry-run -d ./my-dataset dtst analyze --from raw --clear -d ./my-dataset

Usage:

dtst analyze [OPTIONS] [CONFIG]

Options:

dtst annotate

Write source and license metadata into image sidecars.

Annotates all images in the given folders with provenance metadata (source, license, origin). Useful for manually imported images that were not fetched through the pipeline. Sidecars are merged incrementally — existing fields are preserved unless --overwrite is used.

At least one of --source, --license, or --origin is required.

Examples: dtst annotate --from extra --source "unsplash" --license "cc0" -d ./my-dataset dtst annotate config.yaml dtst annotate --from raw,extra --source "personal" --license "all-rights-reserved" dtst annotate --from extra --source "flickr" --overwrite -d ./my-dataset dtst annotate --from extra --source "personal" --dry-run

Usage:

dtst annotate [OPTIONS] [CONFIG]

Options:

dtst augment

Augment a dataset by applying image transformations.

Reads images from one or more source folders and writes transformed copies to a destination folder. By default the original images are also copied to the output; use --no-copy to write only the transformed versions.

At least one transform flag (--flipX, --flipY, --flipXY) is required. Multiple flags can be combined in a single run to produce several variants of each image.

Transformed files are named with a suffix indicating the transform: photo.jpg becomes photo_flipX.jpg, photo_flipY.jpg, photo_flipXY.jpg.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst augment -d ./project --from faces --to augmented --flipX
dtst augment -d ./project --from faces --to augmented --flipX --flipY --flipXY
dtst augment -d ./project --from faces --to augmented --flipX --no-copy
dtst augment config.yaml --dry-run

Usage:

dtst augment [OPTIONS] [CONFIG]

Options:

dtst cluster

Cluster images by visual similarity.

Groups images into clusters based on embedding similarity using HDBSCAN. Each cluster is written to a numbered subdirectory (000 = largest, 001 = second largest, etc.) within the output folder. Images that do not belong to any cluster are placed in a noise/ subdirectory.

Supports two embedding models: arcface for face identity clustering (requires face images) and clip for general visual similarity clustering (works with any images).

--min-cluster-size sets the smallest group HDBSCAN will consider a real cluster (default: 5). Raise it to suppress small or spurious clusters; lower it to capture smaller groups.

--min-samples controls how conservative the density estimate is (default: 2). It decides how many close neighbors a point needs before it can join a cluster. Lower values (1-2) let borderline images in; higher values push more images into the noise folder. Keeping this low while adjusting --min-cluster-size is usually the best starting point.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst cluster config.yaml
dtst cluster -d ./project --from faces --to clusters
dtst cluster -d ./project --model clip --from raw --to clusters
dtst cluster -d ./project --top 3 --min-cluster-size 10
dtst cluster -d ./project --min-samples 1 --min-cluster-size 8
dtst cluster config.yaml --model arcface --dry-run

Usage:

dtst cluster [OPTIONS] [CONFIG]

Options:

dtst dedup

Deduplicate images by perceptual hash similarity.

Groups images by phash hamming distance and keeps the best image from each duplicate group. The winner is chosen by resolution (width x height), then file size, then blur sharpness. Losers are moved to a duplicated/ subdirectory within the source folder (configurable with --to).

Requires phash sidecar data from dtst analyze --phash. Blur scores (from dtst analyze --blur) are used as a tiebreaker when available.

Examples: dtst dedup -d ./project --from faces dtst dedup -d ./project --from faces --threshold 4 dtst dedup -d ./project --from faces --to my-dupes dtst dedup config.yaml --dry-run dtst dedup -d ./project --from faces --clear

Usage:

dtst dedup [OPTIONS] [CONFIG]

Options:

dtst detect

Detect objects in images using OWL-ViT 2.

Uses open-vocabulary object detection to find specific objects in images and writes the results into per-image sidecar JSON files under a "classes" key. Each class gets all detections (score + bounding box) sorted by confidence, or null if not found.

Each run replaces the entire "classes" key in the sidecar.

Examples: dtst detect -d ./project --from raw --classes "microphone,chair,table" dtst detect config.yaml dtst detect -d ./project --from raw --classes "microphone" --threshold 0.4 dtst detect -d ./project --from raw --classes "microphone" --dry-run dtst detect -d ./project --from raw --clear

Usage:

dtst detect [OPTIONS] [CONFIG]

Options:

dtst extract-faces

Extract aligned face crops from images.

Detects faces in each image using MediaPipe (default) or dlib, then produces an aligned and cropped face image for each detection. The alignment normalises eye and mouth positions for consistent face crops.

Reads images from one or more source folders within the working directory and writes face crops to a destination folder. Multiple source folders can be specified as a comma-separated list with --from.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst extract-faces config.yaml
dtst extract-faces config.yaml --engine dlib --max-size 512
dtst extract-faces -d ./crowd --from raw --to faces
dtst extract-faces -d ./crowd --from raw,extra --to faces
dtst extract-faces config.yaml --max-faces 3 --no-padding

Usage:

dtst extract-faces [OPTIONS] [CONFIG]

Options:

dtst extract-frames

Extract keyframes from video files using ffmpeg.

Reads video files from one or more source folders and extracts keyframes (I-frames) to a destination folder. Each video produces a set of numbered images named as {video_stem}_{frame_number}.{format}.

Only I-frames are decoded, which avoids interpolated or blurry frames and produces the sharpest possible output. The --keyframes option sets the minimum interval between extracted frames: with the default of 10, at most one keyframe every 10 seconds is kept. Lower values produce more frames, higher values produce fewer.

Supported video formats: .mp4, .mkv, .avi, .mov, .webm, .flv, .wmv, .m4v.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst extract-frames -d ./project --from videos --to frames
dtst extract-frames -d ./project --from videos --to frames --keyframes 5
dtst extract-frames -d ./project --from videos --to frames --keyframes 30 --format png
dtst extract-frames config.yaml
dtst extract-frames config.yaml --keyframes 20 --dry-run

Usage:

dtst extract-frames [OPTIONS] [CONFIG]

Options:

dtst fetch

Download images and videos from a URL list.

Reads a URL list from the working directory specified by --input. Two formats are supported:

.jsonl JSON Lines with a "url" field per line (search output). Supports --min-size and --license filtering. .txt Plain text with one URL per line. Lines starting with # are treated as comments.

URLs are routed automatically: known video hosting domains (YouTube, Vimeo, etc.) are downloaded with yt-dlp, all other URLs are downloaded directly with HTTP requests.

Image files are named by the MD5 hash of the URL. Video files are named by yt-dlp using the video ID and original extension. Existing files are skipped unless --force is set.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples:

dtst fetch config.yaml
dtst fetch -d ./chanterelle --to raw --input results.jsonl
dtst fetch -d ./project --to videos --input urls.txt
dtst fetch config.yaml --workers 16 --timeout 60
dtst fetch config.yaml --force
dtst fetch -d ./chanterelle --to raw --input results.jsonl --no-wait --license cc

Usage:

dtst fetch [OPTIONS] [CONFIG]

Options:

dtst frame

Resize images to a target width and/or height.

Reads images from one or more source folders and writes resized copies to a destination folder. Uses Lanczos resampling for high-quality downscaling.

When both --width and --height are given, the --mode option controls how aspect ratio differences are handled:

stretch Resize to exact dimensions, distorting if needed. crop Scale to cover the target area, then trim excess (default). pad Scale to fit within the target area, then fill the rest.

When only one dimension is given, the other is computed proportionally and --mode is ignored.

Examples:

dtst frame -d ./project --from faces --to resized -W 512 -H 512
dtst frame -d ./project --from faces --to resized -W 512 -H 512 --mode pad --fill blur
dtst frame -d ./project --from faces --to resized -W 512 -H 512 --mode crop --gravity top
dtst frame -d ./project --from faces --to resized --width 512
dtst frame config.yaml --dry-run

Usage:

dtst frame [OPTIONS] [CONFIG]

Options:

dtst review

Launch a web UI for manual image review.

Opens a local web server with an image grid. Click images to select or deselect them, then apply to move filtered images into a subfolder. Use the view toggle to switch between source and filtered images to restore previously filtered images.

Press Ctrl+C to stop the server.

Examples: dtst review config.yaml dtst review -d ./project --from faces dtst review -d ./project --from faces --to rejected --port 9000 dtst review config.yaml --no-open

Usage:

dtst review [OPTIONS] [CONFIG]

Options:

dtst run

Run a named workflow defined in a config file.

Executes a sequence of dtst commands and shell commands as defined in the workflows section of the config file. Each command step inherits its defaults from the corresponding config section unless inherit: false is set.

Examples: dtst run pipeline config.yaml dtst run pipeline config.yaml --dry-run dtst run pipeline config.yaml -d ./my_dataset

Usage:

dtst run [OPTIONS] WORKFLOW CONFIG

Options:

dtst search

Search for images across multiple engines.

Reads an optional YAML config file and generates image URLs from Flickr, Serper (Google Images), Brave and Wikimedia Commons using an expanded query matrix of search terms and suffixes. Results are deduplicated and written to a JSONL file in the working directory (default: results.jsonl) so multiple runs accumulate new results.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Query matrix: By default, the command runs two kinds of queries for each term: (1) the term alone, e.g. "chanterelle"; (2) the term with each suffix, e.g. "chanterelle mushroom", "chanterelle forest". Use --suffix-only to run only the second kind.

Examples:

dtst search config.yaml
dtst search config.yaml --dry-run
dtst search config.yaml --max-pages 3 --engines flickr,wikimedia
dtst search --terms "chanterelle" --suffixes "mushroom,forest" --engines brave -d ./chanterelle

Usage:

dtst search [OPTIONS] [CONFIG]

Options:

dtst select

Select images from source folders into a destination folder.

Copies (or moves with --move) images from one or more source folders into a destination folder. When filter criteria are provided, only images that pass all criteria are selected. Without criteria, all images are selected.

Files that already exist in the destination (by name) are skipped.

Can be invoked with just a config file, just CLI options, or both. When both are provided, CLI options override config file values.

Examples: dtst select -d ./project --from raw --to backup dtst select -d ./project --from raw,extra --to combined dtst select -d ./project --from faces --to curated --min-side 256 dtst select -d ./project --from faces --to curated --max-side 2048 dtst select -d ./project --from faces --to curated --min-width 512 --max-height 1024 dtst select -d ./project --from faces --to curated --move --min-blur 50 dtst select -d ./project --from raw --to clean --max-detect microphone 0.5 dtst select config.yaml --dry-run

Usage:

dtst select [OPTIONS] [CONFIG]

Options:

dtst upscale

Upscale images using AI super-resolution models.

Reads images from one or more source folders and writes upscaled copies to a destination folder. Uses spandrel to load PyTorch super-resolution models (Real-ESRGAN, SwinIR, HAT, etc.).

By default uses a 4x Real-ESRGAN model. Use --scale to choose between 2x and 4x upscaling, or --model to provide a custom .pth weights file (scale is auto-detected from the model).

Use --denoise to control how much denoising is applied (4x only). 0.0 preserves the most texture, 1.0 applies full denoising. This activates a lighter general-purpose model with controllable denoise strength via weight interpolation.

Large images are processed in tiles to avoid GPU memory issues. Adjust --tile-size to control memory usage (smaller = less VRAM).

Examples: dtst upscale -d ./project --from faces --to upscaled dtst upscale -d ./project --from faces --to upscaled --scale 2 dtst upscale -d ./project --from faces --to upscaled --denoise 0.5 dtst upscale -d ./project --from faces --to upscaled --model ./custom.pth dtst upscale config.yaml --dry-run

Usage:

dtst upscale [OPTIONS] [CONFIG]

Options: