Getting started

Public bike-hire JSON on S3. No warehouse. Write a pipeline, then a dataframe.

Install

pip

pip install skippr

Homebrew

brew tap skipprd/tap
brew install skipprd

The Python wheel and the CLI are the same engine. Connector plugins download on first use from install.skippr.io. If install fails, see elt.skippr.io/getting-started/install.

Write skippr.yml

Python, CLI, and a hand-written file write the same config. No data_sink. Ingest stops at the write-ahead log. That log is the dataset.

Python

from skippr import Config, DataSourceS3, LocalStorage, Pipeline

cfg = Config().workspace("quickstart").storage(LocalStorage())
src = cfg.data_source(
    "sample",
    DataSourceS3(s3_bucket="skippr-public-sample-data", s3_prefix="bike-hire"),
)
cfg.pipeline("bikehire", Pipeline(data_source=src))
cfg.save("skippr.yml")

CLI

skipprd --workspace quickstart --storage-mode local connect data-source s3 \
  --pipeline bikehire \
  --name sample \
  --s3-bucket skippr-public-sample-data \
  --s3-prefix bike-hire

skippr.yml

skippr:
  workspace: quickstart
  skipprd_el_storage_mode: local

pipelines:
  bikehire:
    data_source: data_sources.sample

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

The sample bucket is public. Your AWS keys only need s3:GetObject on that prefix.

export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

Sinks, Glue, and secrets: Connect.

Discover, sync, dataframe

Python

import skippr

s = skippr.Session(skippr.Config.discover().get_pipeline("bikehire"))
s.doctor()
s.discover()
s.sync(once=True)
s.df().to_pandas()

CLI

skipprd doctor
skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

A Session is bound to one pipeline when you create it. discover infers nested types. sync(once=True) reads one pass and commits it to the WAL. df() is SELECT * on those query views — live WAL, no destination yet.

df() and query() return a PyArrow table. Pandas is .to_pandas().

Next

• Connect — sources, sinks, and schema sinks
• Python — Session, query(), config with no YAML file
• Snowflake and Postgres — add a data_sink
• Model with dbt — silver and gold in the warehouse
• Test with dbt and Soda — prove the load