KEFCore: Entity Framework Core provider for Apache Kafka™

KEFCore is the Entity Framework Core provider for Apache Kafka™. Based on KNet client-side features it allows to use Apache Kafka™ as a distributed database and more: KNet client-side features are also compatible with any broker that implements the Kafka wire protocol — see Backend compatibility below.

Libraries and Tools

Core Templates Json Serialization Avro Serialization Protobuf Serialization
Core downloads Templates downloads Serialization downloads Serialization Avro downloads Serialization Protobuf downloads

Project disclaimer

KEFCore is a project curated by MASES Group and supported by the open-source community. Its primary scope is to support other MASES Group projects — both open-source and commercial — though it is freely available for any use. Dedicated community and commercial subscription plans are available. The repository and releases may contain bugs. The release cycle depends on critical issues discovered and/or enhancement requests from this or other dependent projects.

Looking for Entity Framework Core and Apache Kafka™ expertise? MASES Group can help you design, build, deploy, and manage Entity Framework Core and Apache Kafka™ applications. Find out more.

Scope of the project

KEFCore provides an Entity Framework Core provider for Apache Kafka™, enabling .NET applications to use Kafka topics as a data store through the standard EF Core programming model — DbContext, LINQ queries, and strongly-typed entities — with no Kafka-specific consumer or producer code.

The EF Core introduction page opens with this example:

public class BloggingContext : DbContext
{
    public DbSet<Blog> Blogs { get; set; }
    public DbSet<Post> Posts { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseSqlServer(
            @"Server=(localdb)\mssqllocaldb;Database=Blogging;Trusted_Connection=True;ConnectRetryCount=0");
    }
}

With KEFCore, replacing the SQL Server backend with an Apache Kafka™ cluster requires changing a single line:

optionsBuilder.UseKEFCore("my-application", "localhost:9092");

From that point on, standard EF Core code works unchanged against Kafka topics:

// Query
var blogs = await db.Blogs
    .Where(b => b.Rating > 3)
    .OrderBy(b => b.Url)
    .ToListAsync();

// Write
db.Blogs.Add(new Blog { Url = "http://sample.com" });
await db.SaveChangesAsync();

KEFCore is developed following the guidelines in the official EntityFrameworkCore repository and the Writing a provider documentation published by Microsoft.

Currently the project tries to support, at our best, the official supported Apache Kafka™ binary distribution:

KEFCore State KNet Apache Kafka™ .NET JVM™
2.6.+ Active 3.2.x 4.2.x .NET 8+ Java 17+
2.5. Deprecated 2.9.x 3.9.x .NET 8+ Java 11+

Backend compatibility

Entity Framework Core provider for Apache Kafka™ uses the official Apache Kafka™ Java client packages directly through KNet client-side features. This architecture has a direct impact on backend compatibility.

Client-side features — Producer, Consumer, Admin Client, Kafka Streams, KNet Streams SDK, KNet Connect SDK, KNetPS scriptable cmdlets — communicate with the broker exclusively through the Kafka wire protocol and are therefore compatible with any broker that implements it, not only Apache Kafka™ itself.

Examples of compatible brokers: Redpanda, Amazon MSK, Confluent Platform / Cloud, Aiven for Apache Kafka™, IBM Event Streams, WarpStream, AutoMQ, and others.

See Supported Backends for the full compatibility matrix covering all KNet feature areas.

Community and Contribution

If you find Entity Framework Core provider for Apache Kafka™ useful:

  • Leave a ⭐ on the repository
  • Open issues to report bugs 🐛 or request features
  • Submit Pull Requests to improve the project

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to coc_reporting@masesgroup.com.

Summary

Runtime engine

KEFCore uses KNet, and indeed JCOBridge with its features, to obtain many benefits:

  • Cyber-security:
    • JVM™ and CLR, or CoreCLR, runs in the same process, but are insulated from each other;
    • JCOBridge does not make any code injection into JVM™;
    • JCOBridge does not use any other communication mechanism than JNI;
    • .NET (CLR) inherently inherits the cyber-security levels of running JVM™ and Apache Kafka™;
  • Direct access the JVM™ from any .NET application:
    • Any Java/Scala class behind Apache Kafka™ can be directly managed: Consumer, Producer, Administration, Streams, Server-side, and so on;
    • No need to learn new APIs: we try to expose the same APIs in C# style;
    • No extra validation cycle on protocol and functionality: bug fix, improvements, new features are immediately available;
    • Documentation is shared;
Note

JCOBridge 2.6.* can be used for free without any obligations. A commercial license must be purchased — or the software uninstalled — if you derive direct or indirect income from its usage.

JCOBridge resources

Have a look at the following JCOBridge resources:

JCOBridge 2.5.* series 2.6.* series
KEFCore > 1.0.* series > 2.6.1 series
Release notes Link Link
Community Edition Conditions Conditions
Commercial Edition Information Information

Latest release: JCOBridge nuget

KAFKA is a registered trademark of The Apache Software Foundation. KEFCore has no affiliation with and is not endorsed by The Apache Software Foundation. Microsoft is a registered trademark of Microsoft Corporation. EntityFramework is a registered trademark of Microsoft Corporation.

Edit this page