Skip to content

Apache Kafka

Reading Records from Kafka

Quine has full support for reading records from Apache Kafka topics. The means by which Quine interprets records into graph data is configurable via REST API and Recipes.


In this example we will ingest messages from a Kafka topic and store them as nodes in the graph.


For this example we will run Kafka locally. Because Kafka depends on ZooKeeper, we will start that too. Download Kafka, and extract the files to your local filesystem. Start each of ZooKeeper and Kafka in separate terminal sessions by running each of the following commands from the directory where you extracted Kafka.

bin/ config/
bin/ config/

With Kafka up and running, messages can be manually sent to the topic using the following command:

bin/ --bootstrap-server localhost:9092 --topic test-topic
>{"Message": "Hello, world."}

While is running, messages are generated by inputting text followed by a new line. To end the program and stop generating messages, use Control-D.

Using a Recipe

The following is a simple Recipe that ingests each message from the Kafka topic as a node in the graph:

version: 1
title: Kafka Ingest
summary: Ingest Kafka topic messages as graph nodes
description: Ingests each message in the Kafka topic "test-topic" as a graph node
  - type: KafkaIngest
      - test-topic
    bootstrapServers: localhost:9092
      type: CypherJson
      query: |-
        MATCH (n)
        WHERE id(n) = idFrom($that)
        SET n = $that
standingQueries: [ ]
nodeAppearances: [ ]
quickQueries: [ ]
sampleQueries: [ ]

To run this Recipe, run Quine as follows:

❯ java -jar quine-1.5.4.jar -r kafka-ingest.yaml
Graph is ready
Running Recipe Kafka Ingest
Running Ingest Stream INGEST-1
Quine app web server available at http://localhost:8080

 | => INGEST-1 status is running and ingested 0

Quine has downloaded the Recipe and begun execution. As shown above, use to send a JSON record to the stream. Quine should immediately report that it has ingested the record.

| => INGEST-1 status is running and ingested 1

Results should already be available in the web UI at https://<hostname>:8080.