Postgres

Same Session as getting started. Add a sink. Skipprd creates the schema and tables, then inserts. Use this when you want a warehouse on your machine.

Credentials

export POSTGRES_PASSWORD="secret"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

${POSTGRES_PASSWORD} in YAML reads the environment. Do not put a literal password in git. Flags: Postgres quickstart.

Write skippr.yml

Python

from skippr import Config, DataSinkPostgres, DataSourceS3, EnvRef, Pipeline, S3Storage

cfg = Config().workspace("quickstart").storage(S3Storage("your-state-bucket"))
src = cfg.data_source(
    "sample",
    DataSourceS3(s3_bucket="skippr-public-sample-data", s3_prefix="bike-hire"),
)
warehouse = cfg.data_sink(
    "warehouse",
    DataSinkPostgres(
        host="localhost",
        port=5432,
        user="skippr",
        password=EnvRef("POSTGRES_PASSWORD"),
        database="analytics",
        schema="public",
    ),
)
cfg.pipeline("bikehire", Pipeline(data_source=src, data_sink=warehouse))
cfg.save("skippr.yml")

CLI

skipprd --workspace quickstart --storage-mode s3 --skippr-s3-bucket your-state-bucket \
  connect data-source s3 \
  --pipeline bikehire \
  --name sample \
  --s3-bucket skippr-public-sample-data \
  --s3-prefix bike-hire

skipprd connect data-sink postgres \
  --pipeline bikehire \
  --name warehouse \
  --host localhost \
  --port 5432 \
  --user skippr \
  --password '${POSTGRES_PASSWORD}' \
  --database analytics \
  --schema public

skippr.yml

skippr:
  workspace: quickstart
  skippr_s3_bucket: your-state-bucket
  skipprd_el_storage_mode: s3

pipelines:
  bikehire:
    data_source: data_sources.sample
    data_sink: data_sinks.warehouse

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

data_sinks:
  warehouse:
    Postgres:
      host: localhost
      port: 5432
      user: skippr
      password: ${POSTGRES_PASSWORD}
      database: analytics
      schema: public

Sync

Python

import skippr

s = skippr.Session(skippr.Config.discover().get_pipeline("bikehire"))
s.doctor()
s.discover()
s.sync(once=True)
s.df().to_pandas()

CLI

skipprd doctor
skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

df() is the engine view. To prove the destination table:

select count(*) from public.bikehire;

Table names are the pipeline namespace, lowercased, dots turned into underscores.

dbt source

dbt reads the warehouse table, not the WAL.

version: 2

sources:
  - name: skipprd
    database: analytics
    schema: public
    tables:
      - name: bikehire

Then model it and test it.