Supported Connectors and Persistors¶
Persistors¶
The persistor is where Quine saves the graph. RocksDB is the default and needs no configuration, so most deployments only revisit this choice when one of the following applies:
- RocksDB has no build for your architecture. It ships as a native binary used through JNI, and startup fails on an unsupported platform. MapDB is the portable fallback, at the cost of a 2 GB practical limit per memory-mapped file and off-heap memory use.
- You need replication and failover. Cassandra provides them, and
reaches Cassandra-compatible services too. Amazon Keyspaces is the same
trade with AWS operating it, at the cost of accepting only the
ONE,LOCAL_ONE, andLOCAL_QUORUMread consistency levels. - Nothing needs to survive the run. In-memory simulates a persistor without touching disk, and Empty makes every write a no-op, leaving only the node cache. Both are for tests and demos.
Set the choice with
quine.store.type.
Persistors covers each in depth, and backup is
delegated to whichever store you pick.
| Persistor | Configuration value | Description |
|---|---|---|
| RocksDB | rocks-db |
An embedded log-structured merge tree on the local filesystem. The default, and the fastest choice for a single host. |
| MapDB | map-db |
An embedded Java store on the local filesystem. The fallback where RocksDB has no native build for the host architecture. |
| Apache Cassandra | cassandra |
A distributed database giving high throughput, replication, and failover. |
| ScyllaDB | cassandra |
A Cassandra-compatible database, connected through the Cassandra persistor. |
| Astra DB | cassandra |
DataStax's serverless Cassandra-compatible service. Needs an application token and a secure connect bundle. |
| Amazon Keyspaces | keyspaces |
AWS's managed Cassandra-compatible service, for a distributed store without operating Cassandra yourself. |
| In-memory | in-memory |
Holds everything in memory and writes nothing to disk. Useful for tests and short experiments; all data is lost on shutdown. |
| Empty | empty |
Discards every write and returns nothing on read. No history, and nothing survives a restart. |
ScyllaDB and Astra DB are Cassandra-compatible, so they share the cassandra
configuration value and differ only in how you point it at them.
Ingest sources¶
An ingest stream pairs one source with a Cypher query run on each record. The source decides only where the bytes
come from, so choose it by where your data already lives: a broker such as Kafka
or Kinesis for a continuous feed, a file or S3 object for data already at rest,
or Standard input and Number iterator to try something without setting up any
infrastructure.
Set it in the source block of
Create Ingest Stream: POST /api/v2/graph/quine/ingests, or see
Ingest Streams for how the rest of a stream
is built.
| Name | Configuration value | Formats | Compression (ZLIB, GZIP, BASE64) | Description |
|---|---|---|---|---|
| File Ingest | File |
AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An active stream of data being ingested from a file on this Quine host. |
| Kafka Ingest Stream | Kafka |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kafka. |
| Kinesis Data Stream | Kinesis |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis. |
| Kinesis Data Stream Using Kcl lib | KinesisKCL |
Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis |
| Number Iterator Ingest | NumberIterator |
— | An infinite ingest stream which requires no data source and just produces new sequential numbers every time the stream is (re)started. The numbers are Java Longs` and will wrap at their max value. |
|
| Reactive Stream Ingest | ReactiveStream |
Avro, Drop, Json, Protobuf, Raw | A stream of data being ingested from a reactive stream. | |
| S3 Ingest | S3 |
AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An ingest stream from a file in S3, newline delimited. This ingest source is experimental and is subject to change without warning. In particular, there are known issues with durability when the stream is inactive for at least 1 minute. |
| Server Sent Events Stream | ServerSentEvent |
Avro, Drop, Json, Protobuf, Raw | ✓ | A server-issued event stream, as might be handled by the EventSource JavaScript API. Only consumes the data portion of an event. |
| Simple Queue Service Queue | SQS |
Avro, Drop, Json, Protobuf, Raw | ✓ | An active stream of data being ingested from AWS SQS. |
| Standard Input Ingest | StdInput |
AvroContainer, CSV, Json, JsonL, Line, Parquet | An active stream of data being ingested from standard input to this Quine process. | |
| WebSocket File Upload | WebSocketFileUpload |
AvroContainer, CSV, Json, JsonL, Line, Parquet | Streamed file upload via WebSocket protocol. | |
| Websockets Ingest Stream (Simple Startup) | WebsocketClient |
Avro, Drop, Json, Protobuf, Raw | A websocket stream started after a sequence of text messages. |
Standing query destinations¶
A standing query can have one or more destinations, routed in parallel. Run Cypher Query is the only one that acts back on the graph, enriching a match or filtering out ones that don't qualify. The rest send results down stream, so pick by the system that should receive them: a broker such as Kafka or Kinesis to feed another pipeline, or an HTTP endpoint or Slack to notify a service directly. Drop and Log JSON to Console are what you use while building, before pointing anything at a real system.
Destinations are set in the
outputs of
Create Standing Query: POST /api/v2/graph/quine/standingQueries,
and Standing Queries covers how
results are produced in the first place.
| Name | Configuration value | Formats | Description |
|---|---|---|---|
| Broadcast to Reactive Stream | ReactiveStream |
JSON, Protobuf | Broadcasts data to a created Reactive Stream. Other thatDot products can subscribe to Reactive Streams. ⚠️ Warning: Reactive Stream outputs do not function correctly when running in a cluster. |
| Drop | Drop |
— | Effectively no destination at all, this does nothing but forget the data sent to it. |
| Log JSON to Console | StandardOut |
JSON | Prints each result as a single-line JSON object to stdout on the application server. |
| POST to HTTP[S] Webhook | HttpEndpoint |
JSON | Makes an HTTP[S] POST for each result. For the format of the result, see "Standing Query Result Output". |
| Publish to Kafka Topic | Kafka |
JSON, Protobuf | Publishes provided data to the specified Apache Kafka topic. |
| Publish to Kinesis Data Stream | Kinesis |
JSON, Protobuf | Publishes provided data to the specified Amazon Kinesis stream. |
| Publish to Slack Webhook | Slack |
Slack message | Sends a message to Slack via a configured webhook URL. See https://api.slack.com/messaging/webhooks. |
| Publish to SNS Topic | SNS |
JSON, Protobuf | Publishes an AWS SNS record to the provided topic. ⚠️ Double check your credentials and topic ARN! If writing to SNS fails, the write will be retried indefinitely. If the error is unfixable (e.g., the topic or credentials cannot be found), the outputs will never be emitted and the Standing Query this output is attached to may stop running. |
| Run Cypher Query | CypherQuery |
— | Runs the query, where the given parameter is used to reference the data that is passed in. Runs at most parallelism queries simultaneously. |
| Write JSON to File | File |
JSON | Writes each result as a single-line JSON record. For the format of the result, see "Standing Query Result Output". |