Open source · MIT · .NET Standard 2.0

The open-source ETL toolkit for .NET.

EtlKit is a free, MIT-licensed ETL library for .NET. Move data between databases, files and message queues with a few lines of C# — readable, testable, and versioned in git with the rest of your code.

$ dotnet add package EtlKit

ImportOrders.cs

var source = new CsvSource<Order>("orders.csv");
var clean  = new RowTransformation<Order>(o => { o.Email = o.Email.Trim(); return o; });
var dest   = new DbDestination<Order>(sqlConnection, "Orders");

source.LinkTo(clean).LinkTo(dest);
await source.ExecuteAsync(); await dest.Completion;

Read the docs View source on GitHub

6 relational databases — SQL Server, PostgreSQL, MySQL, SQLite, ClickHouse, Access
3 streaming sources — Kafka, MongoDB change streams, Postgres table tail
4 file formats — CSV, JSON, XML, Excel; from disk or over HTTP
XML pipelines declared as configuration, loaded at runtime
MIT all 15 NuGet packages, no paid tier

Code your ETL, not boilerplate.

Every pipeline is three steps: create the components, link them with LinkTo(), then execute the source and wait for the destination. Rows stream through TPL Dataflow buffers, so only the rows in flight are held in memory.

Sources

DbSource
CsvSource
JsonSource
ExcelSource
KafkaJsonSource
MongoChangeStreamSource
PostgresXminTailSource

Transformations

RowTransformation
LookupTransformation
Aggregation
Multicast
MergeJoin
RestTransformation
AIBatchTransformation

Destinations

DbDestination
DbMerge
CsvDestination
JsonDestination
KafkaTransformation
RabbitMqTransformation
CheckpointWriter

LinkErrorTo() routes rows that fail in any component to an error table or log, instead of stopping the load.

Build ETL in .NET

Write pipelines in your favourite .NET language, with your IDE, debugger and unit tests. Source, transformation and destination components compose into a flow.

Runs everywhere

Written in .NET Standard 2.0 and tested on Linux, macOS and Windows with current .NET. Embed it in a web app, a worker service or a console job.

Made for big data

Built on Microsoft's TPL Dataflow with per-component buffers and batched bulk writes — SqlBulkCopy on SQL Server, binary COPY on PostgreSQL.

Manage change

Track ETL logic in git, review it, and run it through your existing CI/CD. Control-flow tasks replace ADO.NET ceremony for DDL and SQL with one line each.

What teams build with it.

EtlKit fits wherever data has to move on a schedule or as it arrives, and the people who own that job already write .NET.

Reporting loads

Nightly or hourly loads from OLTP databases into a reporting store — SQL Server or PostgreSQL into ClickHouse, with lookups and aggregations on the way.

Table synchronisation

Keep a copy of a table in another database in step with its source: DbMerge inserts, updates and deletes only what changed.

Partner file imports

CSV, Excel and JSON files from partners or SFTP drops into your database, with malformed rows captured in an error table for follow-up.

Event ingestion

Kafka topics, MongoDB change streams or new rows in a PostgreSQL table, written to a database as they arrive and resumed from a checkpoint after a restart.

Moving off SSIS

Re-implement SSIS packages as C# that builds, tests and deploys with the rest of your services — and runs on Linux.

Enrichment

Call a REST API per row, or send batches to an OpenAI-compatible model to classify or extract fields, inside the same pipeline.

Connect to what you already use.

Source and destination components for common databases, queues and file types — and a CustomSource / CustomDestination for anything else. Most live in the core EtlKit package; the rest are opt-in add-ons.

Connector	Source	Destination	Streaming	Resumable	Bulk write	Package
SQL Server	✓	✓	—	—	SqlBulkCopy	EtlKit
PostgreSQL	✓	✓	—	—	Binary COPY	EtlKit
PostgreSQL table tail	✓	—	✓ polling	✓ checkpoint	—	EtlKit.PostgresStreaming
MySQL	✓	✓	—	—	Multi-row INSERT	EtlKit
SQLite	✓	✓	—	—	Batched in a transaction	EtlKit
ClickHouse	✓	✓	—	—	INSERT … FORMAT CSV	EtlKit.ClickHouse
MS Access (ODBC)	✓	✓	—	—	Multi-row INSERT	EtlKit
MongoDB change streams	✓	—	✓	✓ resume token	—	EtlKit.MongoDB
Kafka	✓	✓ producer	✓	Consumer group offsets	Batched producer	EtlKit.Kafka
RabbitMQ	—	✓ publish	—	—	—	EtlKit.RabbitMq
CSV · JSON · XML	✓	✓	—	—	Streamed, file or HTTP	EtlKit
Excel	✓	—	—	—	—	EtlKit
REST / HTTP	✓	✓	—	—	Per-row calls	EtlKit.Rest
OpenAI-compatible LLM	Batch transformation: prompt template in, structured rows out					EtlKit.AI
.NET collections	✓	✓	—	—	—	EtlKit

Resumable sources save their position through ICheckpointStore and continue from it after a restart (at-least-once). The PostgreSQL tail polls by transaction ID: it sees inserts and updates, but not deletes — it is not logical-replication CDC.

Hand-written code vs EtlKit.

What a typical data task takes when you write it from scratch on ADO.NET and client libraries, and what it takes with EtlKit.

Task	By hand	With EtlKit
Bulk load	`SqlBulkCopy` with a `DataTable` on SQL Server, binary `COPY` on Postgres, batched INSERTs elsewhere — one code path per database.	`DbDestination<T>` picks the fastest write path for the connection and batches rows for you.
Upsert / sync a table	A staging table plus `MERGE` or `ON CONFLICT` in each SQL dialect, and a separate pass for deletions.	`DbMerge<T>` with `[IdColumn]` and `[CompareColumn]`: full, no-deletions or delta mode, same code on five databases.
CSV → database	A reader loop, type conversion, batching and try/catch around every bad line.	`CsvSource<T>` → `DbDestination<T>`; bad rows go to `LinkErrorTo()`.
Kafka → database	A consumer loop, JSON deserialisation, batching writes and shutdown handling.	`KafkaJsonSource<T>` → `DbDestination<T>` with a cancellation token.
Tail a Postgres table	A polling query, a stored cursor, and care not to skip rows from transactions still in flight.	`PostgresXminTailSource<T>` + `CheckpointWriter` with a `DbCheckpointStore`.
MongoDB changes → SQL	A `Watch()` loop and your own storage for resume tokens.	`MongoChangeStreamSource<T>` resumes from the token in its checkpoint store.

Code, or configuration.

With EtlKit.Serialization the same components can be declared in XML and loaded at runtime: element names are class names, child elements are properties. Change a pipeline without recompiling the host.

orders-flow.xml

<EtlDataFlowStep>
  <Pipeline>
    <CsvSource><Uri>orders.csv</Uri></CsvSource>
    <ExpressionRowFiltration>
      <FilterExpression>Amount &gt; 0</FilterExpression>
    </ExpressionRowFiltration>
    <LinkTo>
      <DbDestination>
        <ConnectionManager type="PostgresConnectionManager">
          <ConnectionString>Host=db;Database=sales</ConnectionString>
        </ConnectionManager>
        <TableName>orders</TableName>
      </DbDestination>
    </LinkTo>
  </Pipeline>
</EtlDataFlowStep>

Is EtlKit the right tool?

EtlKit is a library for developers. If that is not what you need, one of these will serve you better.

If you need	Pick
Pipelines written in C#, reviewed and tested like the rest of your code, running inside your own app or job on any OS	EtlKit — MIT, no server to run
A visual designer for a SQL Server-centric team, scheduled by SQL Server Agent	SSIS — ships with SQL Server licences, Windows-first
A vendor-supported .NET ETL library with commercial support and the newer ETLBox feature set	ETLBox 2.x+ — closed-source, commercial licence. EtlKit continues its open 1.x line
Hundreds of ready-made SaaS connectors and a UI for loading data into a warehouse	Airbyte — an ELT platform you deploy and operate
A long-running data routing server with a visual flow editor and provenance tracking	Apache NiFi — JVM-based
One query, one table, run once	Plain ADO.NET or Dapper — no library needed