Paul Hudson

Appearance

Book a call

Python ​

import skippr is the engine. Config describes the pipelines. Session runs one. You get Arrow. dbt and Soda stay warehouse tools.

bash
pip install skippr

The wheel and the CLI are the same engine. Connector plugins download on first use from install.skippr.io.

Session ​

python
import skippr

cfg = skippr.Config.discover()
s = skippr.Session(cfg.get_pipeline("bikehire"))
s.doctor()
s.discover()
s.sync(once=True)
s.df()
bash
skipprd doctor
skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

Session takes a pipeline ref, never a name string. Config.discover() loads the discovered skippr.yml; Config.load(path) loads a specific file. Another pipeline is another Session. There is no process-wide pipeline name.

Build a config ​

Every plugin is a typed class: DataSourceS3, DataSinkPostgres, SchemaSinkGlue. Required fields are required keyword arguments, and the wheel ships type stubs, so editors and coding agents see every field. Registration returns a typed ref; Pipeline takes refs.

python
import skippr
from skippr import Config, DataSourceS3, LocalStorage, Pipeline

cfg = Config().workspace("bikehire").storage(LocalStorage())
src = cfg.data_source("sample", DataSourceS3(s3_bucket="bucket", s3_prefix="bike-hire"))
bikehire = cfg.pipeline("bikehire", Pipeline(data_source=src))

cfg.save("skippr.yml")
skippr.Session(bikehire).discover()
bash
skipprd --workspace bikehire --storage-mode local connect data-source s3 \
  --pipeline bikehire \
  --name sample \
  --s3-bucket bucket \
  --s3-prefix bike-hire
skipprd discover --pipeline bikehire --log
yaml
skippr:
  workspace: bikehire
  skipprd_el_storage_mode: local

pipelines:
  bikehire:
    data_source: data_sources.sample

data_sources:
  sample:
    S3:
      s3_bucket: bucket
      s3_prefix: bike-hire

A Config lives in memory until you call save. Re-registering a name, save, and skipprd connect share one merge rule: each field you set replaces that field (a nested field such as object_store is replaced whole), and everything else is kept. For a Config loaded from a file, cfg.to_yaml() before save loads to the same config as that file after it. Secret fields take EnvRef("POSTGRES_PASSWORD"), which persists as ${POSTGRES_PASSWORD}; plaintext cannot reach the file. Full source, sink, Glue, and secrets examples: Connect.

df and query ​

This pipeline's views, the same ones skipprd query shows for it. Live WAL, unioned with Iceberg when the pipeline sink is SkipprLake, AthenaIceberg, or Duckdb. No Iceberg sink → WAL only.

python
s.df()                          # every namespace for this pipeline
s.df("rides")                   # SELECT * FROM bikehire.rides
s.query("SELECT count(*) FROM bikehire.rides")
s.df("rides").to_pandas()
s.df().schema                   # Arrow schema
bash
skipprd df --pipeline bikehire
skipprd query --sql "SELECT count(*) FROM bikehire.rides"

df() and query() return pyarrow.Table. Pandas is .to_pandas().

doctor() checks config, env refs, source, sink if present, and WAL. Run it before a long sync.

Optional sink ​

Leave data_sink out and df() still works. The WAL is the dataset. Add Snowflake or Postgres when you want a warehouse: register the sink, re-register the pipeline with data_sink=, and open a new Session. skipprd does not compact or reclaim that WAL until a sink exists.

dbt and Soda ​

Session loads bronze. It does not run dbt.

  1. s.sync(once=True) lands bronze (WAL, then the sink if you set one).
  2. dbt run builds silver and gold in the warehouse.
  3. dbt test and soda scan check that warehouse.

See Model with dbt and Test with dbt and Soda.

Reference: elt.skippr.io.

© 2026 Paul Hudson