skipprd
ELT engine written in Rust. Ingests source data, with schema inference and backward-compatible schema evolution on the fly, in-stream.
Also supports directly querying the in-stream Write-Ahead Log via Apache Arrow and DataFusion, which is very powerful and turns an ELT tool into a low-latency, strongly consistent time-series data lake.
My current work involves formalising the P specification of the S3 WAL and adding a clustered local disk backend to the WAL.
RustPythonArrowParquetDataFusionS3
Use it
skipprd lands bronze: a write-ahead log first, then an optional warehouse. Python Session is the same engine as the CLI. Model in dbt. Test with dbt and Soda. Connector flags and the full reference live at skippr.io.
- Getting started — pip install, then a dataframe
- Connect — write
skippr.ymlfrom Python, CLI, or YAML - Python —
Session,df(),query() - Snowflake — add a sink, then SQL
- Postgres — the same, locally
- Model with dbt — bronze in skipprd, silver and gold in dbt
- Test with dbt and Soda — prove the load
skipprd-wal-p
P specification of skipprd's ingest write-ahead log.
PS3DiskLeases