Snowflake

Same Session as getting started. Add a sink. Skipprd stages Parquet and runs COPY INTO. Schema DDL runs before the data.

Credentials

export SNOWFLAKE_ACCOUNT="myorg-myaccount"
export SNOWFLAKE_USER="skippr_loader"
export SNOWFLAKE_PRIVATE_KEY_PATH="/path/to/rsa_key.p8"
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_DEFAULT_REGION="us-east-1"

Key-pair auth. Flags: Snowflake quickstart.

skippr.yml

skippr:
  workspace: quickstart
  skippr_s3_bucket: your-state-bucket

pipelines:
  bikehire:
    data_source: data_sources.sample
    data_sink: data_sinks.warehouse

data_sources:
  sample:
    S3:
      s3_bucket: skippr-public-sample-data
      s3_prefix: bike-hire

data_sinks:
  warehouse:
    Snowflake:
      account: "myorg-myaccount"
      user: "skippr_loader"
      private_key_path: "${SNOWFLAKE_PRIVATE_KEY_PATH}"
      warehouse: "COMPUTE_WH"
      database: "RAW_DATA"
      schema: "PUBLIC"
      role: "LOADER_ROLE"
      stage: "@SKIPPR_STAGE"

Sync

Python

import skipprd

s = skipprd.Session(config="skippr.yml", pipeline="bikehire")
s.discover()
s.sync(once=True)
s.df().to_pandas()

CLI

skipprd discover --pipeline bikehire --log
skipprd sync --pipeline bikehire --once --log
skipprd df --pipeline bikehire

df() is the engine view (WAL, unioned with the datalake when one exists). To prove the destination table:

select count(*) from RAW_DATA.PUBLIC.bikehire;

Table names are the pipeline namespace, lowercased, dots turned into underscores.

dbt source

dbt reads the warehouse table, not the WAL.

version: 2

sources:
  - name: skipprd
    database: RAW_DATA
    schema: PUBLIC
    tables:
      - name: bikehire

Then model it and test it.