KEFCore: Entity Framework Core provider for Apache Kafka™

KEFCore is the Entity Framework Core provider for Apache Kafka™. Based on KNet client-side features it allows to use Apache Kafka™ as a distributed database and more: KNet client-side features are also compatible with any broker that implements the Kafka wire protocol — see Backend compatibility below.

Libraries and Tools

Package Version Downloads
Core Core nuget Core downloads
Templates Templates nuget Templates downloads
Json Serialization Json Serialization nuget Json Serialization downloads
Avro Serialization Avro Serialization nuget Avro Serialization downloads
Protobuf Serialization Protobuf Serialization nuget Protobuf Serialization downloads
Note

This project is built on KNet, which follows the target framework support of JCOBridge, which in turn follows the Microsoft .NET support lifecycle. Public releases do not take on the maintenance of runtimes that Microsoft no longer supports: when Microsoft ends support for a .NET version, that target is dropped starting with the next JCOBridge release.

Note

The supported Entity Framework Core versions follow the Microsoft support lifecycle of EF Core: when Microsoft ends support for an EF Core version, the corresponding target is dropped from the public packages starting with the next release.

Need a build for a legacy or out-of-support target, or expert help with .NET and JVM™ integration? See Professional Support Options.

Project disclaimer

KEFCore is a project curated by MASES Group and supported by the open-source community. Its primary scope is to support other MASES Group projects — both open-source and commercial — though it is freely available for any use. Dedicated community and commercial subscription plans are available. The repository and releases may contain bugs. The release cycle depends on critical issues discovered and/or enhancement requests from this or other dependent projects.

Looking for Entity Framework Core and Apache Kafka™ expertise? MASES Group can help you design, build, deploy, and manage Entity Framework Core and Apache Kafka™ applications. Find out more.


Scope of the project

KEFCore provides an Entity Framework Core provider for Apache Kafka™, enabling .NET applications to use Kafka topics as a data store through the standard EF Core programming model — DbContext, LINQ queries, and strongly-typed entities — with no Kafka-specific consumer or producer code.

The EF Core introduction page opens with this example:

public class BloggingContext : DbContext
{
    public DbSet<Blog> Blogs { get; set; }
    public DbSet<Post> Posts { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseSqlServer(
            @"Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;ConnectRetryCount=0");
    }
}

With KEFCore, replacing the SQL Server backend with an Apache Kafka™ cluster requires changing a single line:

optionsBuilder.UseKEFCore("my-application", "localhost:9092");

From that point on, standard EF Core code works unchanged against Kafka topics:

// Query
var blogs = await db.Blogs
    .Where(b => b.Rating > 3)
    .OrderBy(b => b.Url)
    .ToListAsync();

// Write
db.Blogs.Add(new Blog { Url = "http://sample.com" });
await db.SaveChangesAsync();

KEFCore is developed following the guidelines in the official EntityFrameworkCore repository and the Writing a provider documentation published by Microsoft.

Currently the project tries to support, at our best, the official supported Apache Kafka™ binary distribution:

KEFCore State KNet Apache Kafka™ .NET JVM™
4.* Active 3.* 4.* .NET 8+ Java 17+
3.* Active 2.9.* 3.9.* .NET 8+ Java 11+
2.6.*+ Deprecated 3.2.x 4.2.x .NET 8+ Java 17+
2.5.* Deprecated 2.9.x 3.9.x .NET 8+ Java 11+

Backend compatibility

Entity Framework Core provider for Apache Kafka™ uses the official Apache Kafka™ Java client packages directly through KNet client-side features. This architecture has a direct impact on backend compatibility.

Client-side features — Producer, Consumer, Admin Client, Kafka Streams, KNet Streams SDK, KNet Connect SDK, KNetPS scriptable cmdlets — communicate with the broker exclusively through the Kafka wire protocol and are therefore compatible with any broker that implements it, not only Apache Kafka™ itself.

Examples of compatible brokers: Redpanda, Amazon MSK, Confluent Platform / Cloud, Aiven for Apache Kafka™, IBM Event Streams, WarpStream, AutoMQ, and others.

See Supported Backends for the full compatibility matrix covering all KNet feature areas.


Community and Contribution

If you find Entity Framework Core provider for Apache Kafka™ useful:

  • Leave a ⭐ on the repository
  • Open issues to report bugs 🐛 or request features
  • Submit Pull Requests to improve the project

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to coc_reporting@masesgroup.com.


Summary


Performance

Project Framework Test Cache Scenario Backend Median iteration time (ms)
Benchmark net10.0 IterationTotal non-cached load KNetStreams.Buffered 120.270
Benchmark net10.0 IterationTotal non-cached load KNetStreams.Buffered.Prefetch 145.009
Benchmark net10.0 IterationTotal non-cached load KNetStreams.Raw 87.297
Benchmark net10.0 IterationTotal non-cached load KNetStreams.Raw.Prefetch 108.264
Benchmark net10.0 IterationTotal non-cached load KafkaStreams.Buffered 116.372
Benchmark net10.0 IterationTotal non-cached load KafkaStreams.Raw 85.217
Benchmark net10.0 IterationTotal cached load KNetStreams.Buffered 23.100
Benchmark net10.0 IterationTotal cached load KNetStreams.Buffered.Prefetch 23.059
Benchmark net10.0 IterationTotal cached load KNetStreams.Raw 22.579
Benchmark net10.0 IterationTotal cached load KNetStreams.Raw.Prefetch 23.390
Benchmark net10.0 IterationTotal cached load KafkaStreams.Buffered 22.534
Benchmark net10.0 IterationTotal cached load KafkaStreams.Raw 22.809
Benchmark net8.0 IterationTotal non-cached load KNetStreams.Buffered 145.958
Benchmark net8.0 IterationTotal non-cached load KNetStreams.Buffered.Prefetch 170.984
Benchmark net8.0 IterationTotal non-cached load KNetStreams.Raw 112.681
Benchmark net8.0 IterationTotal non-cached load KNetStreams.Raw.Prefetch 135.667
Benchmark net8.0 IterationTotal non-cached load KafkaStreams.Buffered 144.298
Benchmark net8.0 IterationTotal non-cached load KafkaStreams.Raw 113.449
Benchmark net8.0 IterationTotal cached load KNetStreams.Buffered 16.845
Benchmark net8.0 IterationTotal cached load KNetStreams.Buffered.Prefetch 16.790
Benchmark net8.0 IterationTotal cached load KNetStreams.Raw 16.699
Benchmark net8.0 IterationTotal cached load KNetStreams.Raw.Prefetch 18.338
Benchmark net8.0 IterationTotal cached load KafkaStreams.Buffered 17.288
Benchmark net8.0 IterationTotal cached load KafkaStreams.Raw 17.619
Benchmark net9.0 IterationTotal non-cached load KNetStreams.Buffered 132.363
Benchmark net9.0 IterationTotal non-cached load KNetStreams.Buffered.Prefetch 162.036
Benchmark net9.0 IterationTotal non-cached load KNetStreams.Raw 99.295
Benchmark net9.0 IterationTotal non-cached load KNetStreams.Raw.Prefetch 117.832
Benchmark net9.0 IterationTotal non-cached load KafkaStreams.Buffered 131.121
Benchmark net9.0 IterationTotal non-cached load KafkaStreams.Raw 97.030
Benchmark net9.0 IterationTotal cached load KNetStreams.Buffered 16.677
Benchmark net9.0 IterationTotal cached load KNetStreams.Buffered.Prefetch 16.510
Benchmark net9.0 IterationTotal cached load KNetStreams.Raw 16.385
Benchmark net9.0 IterationTotal cached load KNetStreams.Raw.Prefetch 16.862
Benchmark net9.0 IterationTotal cached load KafkaStreams.Buffered 16.584
Benchmark net9.0 IterationTotal cached load KafkaStreams.Raw 17.079
Complex net10.0 IterationTotal non-cached load KNetStreams.Raw 216.019
Complex net10.0 IterationTotal non-cached reload KNetStreams.Raw 198.640
Complex net10.0 IterationTotal cached load KNetStreams.Raw 256.200
Complex net10.0 IterationTotal cached reload KNetStreams.Raw 236.928
Complex net10.0 ScalarOnlyProjection non-cached load KNetStreams.Raw 0.244
Complex net10.0 ScalarOnlyProjection non-cached reload KNetStreams.Raw 0.327
Complex net10.0 ScalarOnlyProjection cached load KNetStreams.Raw 0.264
Complex net10.0 ScalarOnlyProjection cached reload KNetStreams.Raw 0.362
Complex net8.0 IterationTotal non-cached load KNetStreams.Raw 276.670
Complex net8.0 IterationTotal non-cached reload KNetStreams.Raw 264.949
Complex net8.0 IterationTotal cached load KNetStreams.Raw 271.114
Complex net8.0 IterationTotal cached reload KNetStreams.Raw 255.504
Complex net8.0 ScalarOnlyProjection non-cached load KNetStreams.Raw 0.283
Complex net8.0 ScalarOnlyProjection non-cached reload KNetStreams.Raw 0.358
Complex net8.0 ScalarOnlyProjection cached load KNetStreams.Raw 0.286
Complex net8.0 ScalarOnlyProjection cached reload KNetStreams.Raw 0.362
Complex net9.0 IterationTotal non-cached load KNetStreams.Raw 252.904
Complex net9.0 IterationTotal non-cached reload KNetStreams.Raw 236.328
Complex net9.0 IterationTotal cached load KNetStreams.Raw 252.893
Complex net9.0 IterationTotal cached reload KNetStreams.Raw 240.984
Complex net9.0 ScalarOnlyProjection non-cached load KNetStreams.Raw 0.268
Complex net9.0 ScalarOnlyProjection non-cached reload KNetStreams.Raw 0.331
Complex net9.0 ScalarOnlyProjection cached load KNetStreams.Raw 0.268
Complex net9.0 ScalarOnlyProjection cached reload KNetStreams.Raw 0.353

Full breakdown: performance benchmarks.


Runtime engine

KEFCore uses KNet, and indeed JCOBridge with its features, to obtain many benefits:

  • Cyber-security:
    • JVM™ and CLR, or CoreCLR, runs in the same process, but are insulated from each other;
    • JCOBridge does not make any code injection into JVM™;
    • JCOBridge does not use any other communication mechanism than JNI;
    • .NET (CLR) inherently inherits the cyber-security levels of running JVM™ and Apache Kafka™;
  • Direct access the JVM™ from any .NET application:
    • Any Java/Scala class behind Apache Kafka™ can be directly managed: Consumer, Producer, Administration, Streams, Server-side, and so on;
    • No need to learn new APIs: we try to expose the same APIs in C# style;
    • No extra validation cycle on protocol and functionality: bug fix, improvements, new features are immediately available;
    • Documentation is shared;
Note

JCOBridge 2.6.* can be used for free without any obligations. A commercial license must be purchased — or the software uninstalled — if you derive direct or indirect income from its usage.

JCOBridge resources

Have a look at the following JCOBridge resources:

JCOBridge 2.5.* series 2.6.* series
KEFCore > 1.0.* series > 2.6.1 series
Release notes Link Link
Community Edition Conditions Conditions
Commercial Edition Information Information

Latest release: JCOBridge nuget

KAFKA is a registered trademark of The Apache Software Foundation. KEFCore has no affiliation with and is not endorsed by The Apache Software Foundation. Microsoft is a registered trademark of Microsoft Corporation. EntityFramework is a registered trademark of Microsoft Corporation.