Postgres

Same Session as getting started. Add a sink. Skipprd creates the schema and tables, then inserts. Use this when you want a warehouse on your machine.

Credentials

export POSTGRES_PASSWORD="secret"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

${POSTGRES_PASSWORD} in YAML reads the environment. Do not put a literal password in git. Flags: Postgres quickstart.

skippr.yml

skippr:
  workspace: quickstart
  skippr_s3_bucket: your-state-bucket

pipelines:
  bikehire:
    data_source: data_sources.sample
    data_sink: data_sinks.warehouse

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

data_sinks:
  warehouse:
    Postgres:
      host: localhost
      port: 5432
      user: skippr
      password: ${POSTGRES_PASSWORD}
      database: analytics
      schema: public

Sync

Python

import skipprd

s = skipprd.Session(config="skippr.yml", pipeline="bikehire")
s.discover()
s.sync(once=True)
s.df().to_pandas()

CLI

skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

df() is the engine view. To prove the destination table:

select count(*) from public.bikehire;

Table names are the pipeline namespace, lowercased, dots turned into underscores.

dbt source

dbt reads the warehouse table, not the WAL.

version: 2

sources:
  - name: skipprd
    database: analytics
    schema: public
    tables:
      - name: bikehire

Then model it and test it.