KEFCore: Entity Framework Core provider for Apache Kafka™
KEFCore is the Entity Framework Core provider for Apache Kafka™. Based on KNet client-side features it allows to use Apache Kafka™ as a distributed database and more: KNet client-side features are also compatible with any broker that implements the Kafka wire protocol — see Backend compatibility below.
Libraries and Tools
| Core | Templates | Json Serialization | Avro Serialization | Protobuf Serialization |
|---|---|---|---|---|
  |
  |
  |
  |
  |
Project disclaimer
KEFCore is a project curated by MASES Group and supported by the open-source community. Its primary scope is to support other MASES Group projects — both open-source and commercial — though it is freely available for any use. Dedicated community and commercial subscription plans are available. The repository and releases may contain bugs. The release cycle depends on critical issues discovered and/or enhancement requests from this or other dependent projects.
Looking for Entity Framework Core and Apache Kafka™ expertise? MASES Group can help you design, build, deploy, and manage Entity Framework Core and Apache Kafka™ applications. Find out more.
Scope of the project
KEFCore provides an Entity Framework Core provider for Apache Kafka™, enabling .NET applications to use Kafka topics as a data store through the standard EF Core programming model — DbContext, LINQ queries, and strongly-typed entities — with no Kafka-specific consumer or producer code.
The EF Core introduction page opens with this example:
public class BloggingContext : DbContext
{
public DbSet<Blog> Blogs { get; set; }
public DbSet<Post> Posts { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer(
@"Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;ConnectRetryCount=0");
}
}
With KEFCore, replacing the SQL Server backend with an Apache Kafka™ cluster requires changing a single line:
optionsBuilder.UseKEFCore("my-application", "localhost:9092");
From that point on, standard EF Core code works unchanged against Kafka topics:
// Query
var blogs = await db.Blogs
.Where(b => b.Rating > 3)
.OrderBy(b => b.Url)
.ToListAsync();
// Write
db.Blogs.Add(new Blog { Url = "http://sample.com" });
await db.SaveChangesAsync();
KEFCore is developed following the guidelines in the official EntityFrameworkCore repository and the Writing a provider documentation published by Microsoft.
Currently the project tries to support, at our best, the official supported Apache Kafka™ binary distribution:
| KEFCore | State | KNet | Apache Kafka™ | .NET | JVM™ | |
|---|---|---|---|---|---|---|
| 4.* | Active | 3.* | 4.* |  |
 |
|
| 3.* | Active | 2.9.* | 3.9.* | |
 |
|
| 2.6.+ | Deprecated | 3.2.x | 4.2.x | |
|
|
| 2.5. | Deprecated | 2.9.x | 3.9.x | |
|
Backend compatibility
Entity Framework Core provider for Apache Kafka™ uses the official Apache Kafka™ Java client packages directly through KNet client-side features. This architecture has a direct impact on backend compatibility.
Client-side features — Producer, Consumer, Admin Client, Kafka Streams, KNet Streams SDK, KNet Connect SDK, KNetPS scriptable cmdlets — communicate with the broker exclusively through the Kafka wire protocol and are therefore compatible with any broker that implements it, not only Apache Kafka™ itself.
Examples of compatible brokers: Redpanda, Amazon MSK, Confluent Platform / Cloud, Aiven for Apache Kafka™, IBM Event Streams, WarpStream, AutoMQ, and others.
See Supported Backends for the full compatibility matrix covering all KNet feature areas.
Community and Contribution
If you find Entity Framework Core provider for Apache Kafka™ useful:
- Leave a ⭐ on the repository
- Open issues to report bugs 🐛 or request features
- Submit Pull Requests to improve the project
This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to coc_reporting@masesgroup.com.
Summary
- Getting started
- How it works
- Supported backends
- Usage
- Use cases
- Templates usage
- Options
- Conventions
- Serialization
- Schema migration
- Performance tips
- Troubleshooting
- External application
- Roadmap
- Current state
- KEFCoreDbContext
Performance
| Project | Framework | Test | Cache | Scenario | Backend | Median iteration time (ms) |
|---|---|---|---|---|---|---|
| Benchmark | net10.0 | IterationTotal | non-cached | load | KNetStreams.Buffered | 122.767 |
| Benchmark | net10.0 | IterationTotal | non-cached | load | KNetStreams.Buffered.Prefetch | 148.676 |
| Benchmark | net10.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 92.566 |
| Benchmark | net10.0 | IterationTotal | non-cached | load | KNetStreams.Raw.Prefetch | 117.701 |
| Benchmark | net10.0 | IterationTotal | non-cached | load | KafkaStreams.Buffered | 120.718 |
| Benchmark | net10.0 | IterationTotal | non-cached | load | KafkaStreams.Raw | 93.495 |
| Benchmark | net10.0 | IterationTotal | cached | load | KNetStreams.Buffered | 23.636 |
| Benchmark | net10.0 | IterationTotal | cached | load | KNetStreams.Buffered.Prefetch | 22.545 |
| Benchmark | net10.0 | IterationTotal | cached | load | KNetStreams.Raw | 23.033 |
| Benchmark | net10.0 | IterationTotal | cached | load | KNetStreams.Raw.Prefetch | 23.678 |
| Benchmark | net10.0 | IterationTotal | cached | load | KafkaStreams.Buffered | 22.298 |
| Benchmark | net10.0 | IterationTotal | cached | load | KafkaStreams.Raw | 23.641 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KNetStreams.Buffered | 142.610 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KNetStreams.Buffered.Prefetch | 166.332 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 99.779 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KNetStreams.Raw.Prefetch | 121.325 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KafkaStreams.Buffered | 138.196 |
| Benchmark | net8.0 | IterationTotal | non-cached | load | KafkaStreams.Raw | 101.304 |
| Benchmark | net8.0 | IterationTotal | cached | load | KNetStreams.Buffered | 17.758 |
| Benchmark | net8.0 | IterationTotal | cached | load | KNetStreams.Buffered.Prefetch | 18.899 |
| Benchmark | net8.0 | IterationTotal | cached | load | KNetStreams.Raw | 17.993 |
| Benchmark | net8.0 | IterationTotal | cached | load | KNetStreams.Raw.Prefetch | 17.764 |
| Benchmark | net8.0 | IterationTotal | cached | load | KafkaStreams.Buffered | 17.187 |
| Benchmark | net8.0 | IterationTotal | cached | load | KafkaStreams.Raw | 18.547 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KNetStreams.Buffered | 129.290 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KNetStreams.Buffered.Prefetch | 156.428 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 99.231 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KNetStreams.Raw.Prefetch | 120.338 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KafkaStreams.Buffered | 126.899 |
| Benchmark | net9.0 | IterationTotal | non-cached | load | KafkaStreams.Raw | 99.415 |
| Benchmark | net9.0 | IterationTotal | cached | load | KNetStreams.Buffered | 16.953 |
| Benchmark | net9.0 | IterationTotal | cached | load | KNetStreams.Buffered.Prefetch | 17.425 |
| Benchmark | net9.0 | IterationTotal | cached | load | KNetStreams.Raw | 17.167 |
| Benchmark | net9.0 | IterationTotal | cached | load | KNetStreams.Raw.Prefetch | 16.833 |
| Benchmark | net9.0 | IterationTotal | cached | load | KafkaStreams.Buffered | 16.402 |
| Benchmark | net9.0 | IterationTotal | cached | load | KafkaStreams.Raw | 17.234 |
| Complex | net10.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 232.204 |
| Complex | net10.0 | IterationTotal | non-cached | reload | KNetStreams.Raw | 212.550 |
| Complex | net10.0 | IterationTotal | cached | load | KNetStreams.Raw | 241.411 |
| Complex | net10.0 | IterationTotal | cached | reload | KNetStreams.Raw | 228.475 |
| Complex | net10.0 | ScalarOnlyProjection | non-cached | load | KNetStreams.Raw | 0.251 |
| Complex | net10.0 | ScalarOnlyProjection | non-cached | reload | KNetStreams.Raw | 0.325 |
| Complex | net10.0 | ScalarOnlyProjection | cached | load | KNetStreams.Raw | 0.253 |
| Complex | net10.0 | ScalarOnlyProjection | cached | reload | KNetStreams.Raw | 0.338 |
| Complex | net8.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 257.521 |
| Complex | net8.0 | IterationTotal | non-cached | reload | KNetStreams.Raw | 244.239 |
| Complex | net8.0 | IterationTotal | cached | load | KNetStreams.Raw | 269.003 |
| Complex | net8.0 | IterationTotal | cached | reload | KNetStreams.Raw | 253.845 |
| Complex | net8.0 | ScalarOnlyProjection | non-cached | load | KNetStreams.Raw | 0.292 |
| Complex | net8.0 | ScalarOnlyProjection | non-cached | reload | KNetStreams.Raw | 0.367 |
| Complex | net8.0 | ScalarOnlyProjection | cached | load | KNetStreams.Raw | 0.300 |
| Complex | net8.0 | ScalarOnlyProjection | cached | reload | KNetStreams.Raw | 0.372 |
| Complex | net9.0 | IterationTotal | non-cached | load | KNetStreams.Raw | 249.132 |
| Complex | net9.0 | IterationTotal | non-cached | reload | KNetStreams.Raw | 236.983 |
| Complex | net9.0 | IterationTotal | cached | load | KNetStreams.Raw | 253.225 |
| Complex | net9.0 | IterationTotal | cached | reload | KNetStreams.Raw | 238.613 |
| Complex | net9.0 | ScalarOnlyProjection | non-cached | load | KNetStreams.Raw | 0.262 |
| Complex | net9.0 | ScalarOnlyProjection | non-cached | reload | KNetStreams.Raw | 0.342 |
| Complex | net9.0 | ScalarOnlyProjection | cached | load | KNetStreams.Raw | 0.273 |
| Complex | net9.0 | ScalarOnlyProjection | cached | reload | KNetStreams.Raw | 0.355 |
Full breakdown: performance benchmarks.
Runtime engine
KEFCore uses KNet, and indeed JCOBridge with its features, to obtain many benefits:
- Cyber-security:
- JVM™ and CLR, or CoreCLR, runs in the same process, but are insulated from each other;
- JCOBridge does not make any code injection into JVM™;
- JCOBridge does not use any other communication mechanism than JNI;
- .NET (CLR) inherently inherits the cyber-security levels of running JVM™ and Apache Kafka™;
- Direct access the JVM™ from any .NET application:
- Any Java/Scala class behind Apache Kafka™ can be directly managed: Consumer, Producer, Administration, Streams, Server-side, and so on;
- No need to learn new APIs: we try to expose the same APIs in C# style;
- No extra validation cycle on protocol and functionality: bug fix, improvements, new features are immediately available;
- Documentation is shared;
Note
JCOBridge 2.6.* can be used for free without any obligations. A commercial license must be purchased — or the software uninstalled — if you derive direct or indirect income from its usage.
JCOBridge resources
Have a look at the following JCOBridge resources:
| JCOBridge | 2.5.* series | 2.6.* series |
|---|---|---|
| KEFCore | > 1.0.* series | > 2.6.1 series |
| Release notes | Link | Link |
| Community Edition | Conditions | Conditions |
| Commercial Edition | Information | Information |
Latest release:
KAFKA is a registered trademark of The Apache Software Foundation. KEFCore has no affiliation with and is not endorsed by The Apache Software Foundation. Microsoft is a registered trademark of Microsoft Corporation. EntityFramework is a registered trademark of Microsoft Corporation.