Skip to content

Streams

The Streams page is where you manage ingest streams and standing queries for your Quine data pipeline. It is accessible from the left sidebar.

API v2 Required

The Streams page is available only when the v2 API is enabled. It does not appear in the sidebar when running in API v1 mode.

Overview

The Streams page is organized into two panels:

  • Ingest Streams: Create, monitor, pause, resume, and delete ingest streams that feed data into the streaming graph.
  • Standing Queries: Create, monitor, and delete standing queries that watch the graph for patterns and route results to outputs.

Each panel has a list view and a creation form. Both panels auto-refresh to keep stats current.

Streams page showing ingest streams in various states and a standing query

Ingest Streams Panel

The Ingest Streams panel displays all configured ingest streams in a table with the following columns:

Column Description
Name The unique name assigned to the ingest stream
Type The source type (e.g., Kafka, Kinesis, FileIngest, StdInput)
Status Current state: Running, Paused, Restored, Completed, or Failed
Ingested Total number of records processed
Rate (1m) One-minute rolling ingest rate (records per second)
Uptime How long the stream has been active
Actions Controls to pause, resume, or delete the stream

Ingest Stream Status

Each ingest stream displays a color-coded status badge:

  • Running (green): The stream is actively consuming records.
  • Paused / Restored (yellow): The stream is temporarily halted and can be resumed.
  • Completed (gray): The source was fully consumed (e.g., a finite file or a number iterator with a limit).
  • Terminated (gray): The stream was stopped by a user or API call.
  • Failed (red): The stream encountered an error. The entire row highlights in red with the error message displayed. Only the delete action is available.

Managing Ingest Streams

  • Pause: Temporarily halt an ingest stream. The stream can be resumed later.
  • Resume: Continue a paused or restored stream.
  • Delete: Remove the stream entirely.

Creating an Ingest Stream

Click + New Ingest to switch to the creation form. The form uses a two-step workflow:

Step 1: Source Type. Select the type of system that will feed data into your stream. Available source types include Kafka, Kinesis, SQS, files, standard input, S3, server-sent events, WebSocket, and more.

Step 1: Choose a source type from the available ingest sources

Step 2: Configure. Fill in the source-specific settings (such as topic names, bootstrap servers, or file paths), the ingest query, and any optional settings like record decoders or error handling. The form fields adapt to the source type selected in Step 1.

Step 2: Configure the Kafka ingest stream with topics, bootstrap servers, format, and query

Provide a unique Name for the stream and click Create Ingest Stream to submit.

Click Back at any step to return to the previous step or cancel the creation.

See Ingest Streams for configuration options, source types, and error handling.

Standing Queries Panel

The Standing Queries panel displays all configured standing queries in a table with the following columns:

Column Description
Name The unique name assigned to the standing query
Pattern The Cypher pattern query (truncated in the table, full text on hover)
Mode The matching mode for the query
Outputs Number of output destinations configured
Rate (1m) One-minute rolling match rate
Actions Control to delete the standing query

Creating a Standing Query

Click + New Standing Query to switch to the creation form. The form collects:

  • Name: A unique identifier for the standing query.
  • Pattern: The Cypher MATCH / RETURN query that defines what graph structure to watch for.
  • Mode: The matching mode (e.g., Distinct ID).
  • Outputs: One or more output destinations with optional result enrichment queries.
  • Additional options: Such as includeCancellations and inputBufferSize.

Standing query creation form with name, pattern, mode, and outputs fields

Click Create Standing Query to submit.

See Standing Queries for standing query patterns, outputs, and query enrichment.

Viewing Configuration

Click the chevron on any row in either panel to expand it and view the full JSON configuration for that resource.

Expanded ingest stream row showing the full JSON configuration

For standing queries, the expanded view also includes:

  • Configuration: The full JSON configuration, nested under a collapsible header.
  • Outputs: A sub-table listing each output destination by name and type. Each output can be individually removed.

Click + Add Output from the expanded view to add an output to an existing standing query.

Expanded standing query showing the outputs table and Add Output button

Next Steps

  • Ingest Streams: Detailed reference for all ingest source types, formats, error handling, and the API.
  • Standing Queries: How standing queries work, pattern constraints, output destinations, and result enrichment.
  • Exploration UI: Explore the streaming graph interactively with Cypher queries on an interactive canvas.