Getting started

Public bike-hire JSON on S3. No warehouse. Three calls, then a dataframe.

Install

pip

pip install skipprd

Homebrew

brew tap skipprd/tap
brew install skipprd

The Python wheel and the CLI are the same engine. Connector plugins download on first use from install.skippr.io. If install fails, see elt.skippr.io/getting-started/install.

skippr.yml

No data_sink. Ingest stops at the write-ahead log. That log is the dataset.

skippr:
  workspace: quickstart
  skipprd_el_storage_mode: local

pipelines:
  bikehire:
    data_source: data_sources.sample

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

The sample bucket is public. Your AWS keys only need s3:GetObject on that prefix.

export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

Discover, sync, dataframe

Python

import skipprd

s = skipprd.Session(config="skippr.yml", pipeline="bikehire")
s.discover()
s.sync(once=True)
s.df().to_pandas()

CLI

skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

discover infers nested types. sync(once=True) reads one pass and commits it to the WAL. df() is SELECT * on those query views — live WAL, no destination yet.

df() and query() return a PyArrow table. Pandas is .to_pandas().

Next

• Python — Session, query(), config with no YAML file
• Snowflake and Postgres — add a data_sink
• Model with dbt — silver and gold in the warehouse
• Test with dbt and Soda — prove the load