---
# Common Pitfalls
URL: https://quine.io/getting-started/common-pitfalls/

# Common Pitfalls

Quine is a streaming graph — not a database. Many patterns that work in traditional databases or graph databases like Neo4j will silently produce wrong results, lose data, or cause severe performance problems in Quine. Read this page before writing your first ingest query.

For Cypher-specific differences from Neo4j, see [Quine Cypher vs. Neo4j Cypher](../learn/cypher/quine-cypher-differences.md).

## Ingest Queries

### Every query must anchor nodes by ID

Any query that finds nodes by property instead of by ID triggers a full scan of every node in the graph. This includes `MERGE` with property matchers.

```cypher
-- WRONG: scans all nodes
MATCH (user) WHERE user.email = "alice@example.com" ...

-- CORRECT: direct lookup
MATCH (user) WHERE id(user) = idFrom("user", "alice@example.com") ...
```

[Details → Troubleshooting Ingest](../learn/troubleshooting/ingest.md#2-poorly-optimized-ingest-queries)

### Mismatched idFrom() arguments silently orphan data

If one ingest stream uses `idFrom("order", orderId)` and another uses `idFrom(orderId)` without the prefix, they address different nodes. Relationships between them will never form, and there is no error message.

[Details → Troubleshooting Ingest](../learn/troubleshooting/ingest.md#1-unstable-or-mismatched-id-selection)

### Ingest queries run in parallel — don't depend on other records

Each record is processed independently and concurrently. A query that assumes another record has already been ingested (e.g., looking up a customer node by property to attach an order) will silently skip records when the dependency hasn't been processed yet.

**Solution**: Make each ingest query self-contained. Every record should create all the nodes and edges it references, using `idFrom()` to address them. Order of arrival should not matter.

[Details → Troubleshooting Ingest](../learn/troubleshooting/ingest.md#2-ingest-race-conditions)

### Ingest queries should be idempotent

At-least-once delivery means records may be processed more than once after a restart. Design queries so that processing the same record twice produces the same graph state.

[Details → Delivery Guarantees](../core-concepts/delivery-guarantees.md#idempotent-ingest-queries)

## Standing Queries

### Distinct ID mode fires once per root node, not per match

In the default Distinct ID mode (`DISTINCT_ID`, v1: `DistinctId`), once a pattern matches for a given root node, additional matches from that same root node do not emit new results. If you need a result for every match, use Multiple Values mode (`MULTIPLE_VALUES`, v1: `MultipleValues`).

[Details → Standing Query Modes](../learn/standing-queries/standing-queries.md#distinct-id-pattern-queries)

### Results are best-effort and can be dropped

Standing query outputs are not durably queued. Results can be lost if the output queue overflows, an output destination fails, or the process restarts. Treat standing query outputs as notifications, not authoritative records.

[Details → Delivery Guarantees](../core-concepts/delivery-guarantees.md#standing-query-outputs-best-effort)

### New standing queries fire on existing data

When you register a standing query, it evaluates against all data already in the graph — not just new data arriving after registration. This can produce a burst of results from historical data.

[Details → Troubleshooting Queries](../learn/troubleshooting/queries.md)

## Exploration and Ad-Hoc Queries

### Sample queries and query bar queries need ID anchoring too

The "no indexes" rule applies to all queries, not just ingest. Sample queries in recipes and queries typed into the Exploration UI query bar are ad-hoc queries — they scan the full graph unless anchored by ID.

```cypher
-- WRONG: scans all nodes (labels are not indexed)
MATCH (p:Person) RETURN p

-- WRONG: scans all nodes looking for property match
MATCH (n) WHERE n.type = "order" RETURN n LIMIT 10

-- CORRECT: look up a specific node by computed ID
MATCH (n) WHERE id(n) = idFrom("customer", "CUST-123") RETURN n

-- CORRECT: when no specific node is known, sample from recently accessed nodes
CALL recentNodes(10)

-- CORRECT: sample recent nodes filtered by label
CALL recentNodes(1000) YIELD node AS nId
MATCH (n)
WHERE id(n) = nId AND labels(n) = ["Person"]
RETURN n
```

Always use `idFrom()` when the identity of the target node is known. When no specific node is known, use `recentNodes()` or `recentNodeIds()` to sample from recently accessed nodes. Never rely on label or property scans — Quine has no indexes, so `MATCH (n:Label)` is always a full graph scan.

[Details → Quine Indexing](../core-concepts/id-provider.md)

### Quick queries are already node-anchored

Quick queries (right-click context menu on a node) receive the clicked node bound to the variable `n`, so they are already anchored to a specific starting point. The Cypher in a quick query should expand outward from `n` — not scan for unrelated nodes.

```cypher
-- Good quick query: expand from the clicked node
MATCH (n)-[:PURCHASED]->(order) RETURN order

-- Bad quick query: ignores the starting node, scans the graph
MATCH (order:Order) RETURN order LIMIT 10
```

## Performance

### Counting nodes is expensive

`MATCH (n) RETURN count(*)` scans the entire graph. Unlike a database, Quine does not maintain a running node count.

[Details → Streaming Graph vs. Database](../core-concepts/streaming-graph-vs-database.md#counting-is-hard)

### Labels do not improve query performance

In Neo4j, `MATCH (p:Person)` uses a label index. In Quine, labels are stored as properties with no index — filtering by label still requires scanning all nodes.

[Details → Cypher Differences](../learn/cypher/quine-cypher-differences.md#labels-are-not-indexed)

### Supernodes degrade traversal performance

Nodes with thousands of edges (supernodes) cause performance problems when queries traverse outward from them. Supernodes that are only traversed *to* (as endpoints) are fine. Consider using properties instead of edges for high-cardinality relationships, or partitioning supernodes by time period.

[Details → Diagnosing Bottlenecks](../learn/troubleshooting/diagnosing-bottlenecks.md)

### Default parallelism may not be optimal

Ingest parallelism defaults to 16. The right value depends on your data, queries, and infrastructure. Experimenting with this value can significantly improve throughput. When running multiple ingests on the same host, divide the optimal single-ingest parallelism across them.

[Details → Troubleshooting Ingest](../learn/troubleshooting/ingest.md#1-parallelism-configuration)

### Standing query backpressure slows ingest

If standing queries can't keep up with ingest, Quine pauses ingest to prevent result loss. Monitor the `shared.valve.ingest` metric — a non-zero value means ingest is being throttled.

[Details → Diagnosing Bottlenecks](../learn/troubleshooting/diagnosing-bottlenecks.md)

## Operations

### Graceful shutdown is required to prevent data loss

Use the `POST /api/v2/system:shutdown` endpoint to shut down cleanly. A hard kill (SIGKILL, container eviction) can lose data that hasn't been persisted yet.

[Details → Operational Considerations](../core-concepts/operational-considerations.md)

### JVM heap should not exceed 16GB

Garbage collection pauses grow significantly above 16GB heap. 12GB is a good starting point. Additional physical memory beyond the heap is needed for off-heap overhead — budget 25–33% extra.

[Details → Configuration](../reference/config/configuration.md)

### Recipes use temporary storage by default

When Quine launches a recipe, it creates a temporary data store in the system temp directory. Each subsequent launch replaces the previous data. Use `--force-config` with a persistent data path to retain data between runs.

[Details → Recipes Tutorial](recipes-tutorial.md)


---
# Quine Cypher vs. Neo4j Cypher
URL: https://quine.io/learn/cypher/quine-cypher-differences/

# Quine Cypher vs. Neo4j Cypher

Quine implements a dialect of [OpenCypher v9](https://s3.amazonaws.com/artifacts.opencypher.org/openCypher9.pdf). If you have experience writing Cypher for Neo4j, most of your knowledge transfers directly. However, Quine is a streaming graph, not a database, and several behaviors differ in important ways. This page covers every major difference so that queries written for Quine work correctly the first time.

## Fundamental Differences

### All Nodes Always Exist

In Neo4j, `CREATE (n:Person {name: "Alice"})` adds a new node to the database. In Quine, nodes are never created — they always exist. You address a node by its ID and start using it. This means:

- `CREATE` on a node doesn't fail or produce a duplicate — it simply addresses the node at the generated ID.
- There is no need to check whether a node exists before writing to it.
- Two separate data streams can reference the same node simultaneously without coordination.

The `MATCH ... WHERE id(n) = idFrom(...)` pattern is the standard way to address nodes:

```cypher
-- Neo4j style (don't use in Quine)
MERGE (customer:Customer {customerId: "CUST-123"})
SET customer.name = "Alice"

-- Quine style
MATCH (customer)
WHERE id(customer) = idFrom("customer", "CUST-123")
SET customer:Customer, customer.name = "Alice"
```

### No Indexes — Use idFrom() Instead

Neo4j uses indexes to look up nodes by property value. Quine has no indexes. Instead, `idFrom()` deterministically computes a node ID from input values.

**Any query that searches for nodes by property without an `id(n) = idFrom(...)` constraint will scan every node in the graph**, which is extremely slow at scale.

```cypher
-- Neo4j style: property lookup via index (don't use in Quine)
MATCH (user:User {email: "alice@example.com"})
RETURN user

-- Quine style: direct ID lookup via idFrom()
MATCH (user)
WHERE id(user) = idFrom("user", "alice@example.com")
RETURN user
```

Always include at least one `id(n) = idFrom(...)` constraint in ad-hoc and ingest queries. The only exception is [standing queries](../../learn/standing-queries/standing-queries.md), which use incremental matching and do not require `idFrom()`.

### idFrom() Basics

`idFrom()` accepts any number of arguments and deterministically produces a node ID:

```cypher
idFrom("user", "alice@example.com")        -- single key
idFrom("sensor-reading", sensorId, timestamp) -- composite key
```

**Always prefix with a type string** to prevent collisions between different node types:

```cypher
-- Good: different types won't collide even if IDs overlap
id(customer) = idFrom("customer", "123")
id(order) = idFrom("order", "123")

-- Bad: these resolve to the same node if the IDs match
id(customer) = idFrom("123")
id(order) = idFrom("123")
```

**Stay consistent** across all ingest streams. If one stream uses `idFrom("customer", id)` and another uses `idFrom("cust", id)`, they will create separate nodes for the same logical entity.

For details, see [Quine Indexing](../../core-concepts/id-provider.md).

## Behavioral Differences

### MATCH Does Not See Same-Query Updates

In Neo4j, a `SET` operation updates the node immediately and subsequent clauses in the same query can see the change. In Quine, nodes found in a `MATCH` do not reflect updates made later in the same query.

```cypher
-- In Neo4j, this works as expected.
-- In Quine, the WHERE clause sees the ORIGINAL value of n.status,
-- not the value set by the preceding SET.
MATCH (n) WHERE id(n) = idFrom("order", "123")
SET n.status = "shipped"
WITH n
MATCH (n) WHERE n.status = "shipped"  -- may not match in Quine
RETURN n
```

The same node can even be aliased under two different variable names, and those aliases may show different property values if there were intervening writes.

**Workaround**: Split the operation into separate queries, or use standing queries that react to the updated state.

### CREATE Is Idempotent for Edges

In Neo4j, running `CREATE (a)-[:KNOWS]->(b)` twice creates two separate `:KNOWS` edges. In Quine, an edge is uniquely identified by its direction, label, and endpoints. Running the same `CREATE` twice has no effect the second time — the edge already exists.

```cypher
-- Running this twice in Neo4j creates 2 edges.
-- Running this twice in Quine creates 1 edge.
MATCH (a), (b)
WHERE id(a) = idFrom("person", "Alice") AND id(b) = idFrom("person", "Bob")
CREATE (a)-[:KNOWS]->(b)
```

This means there can never be multiple edges with the same label and direction between the same two nodes.

### Edges Have No Properties

Neo4j supports properties on relationships. Quine does not. If you need to attach data to a relationship, model it as an intermediate node:

```cypher
-- Neo4j style (won't work in Quine)
CREATE (a)-[:PURCHASED {amount: 99.99, date: "2024-01-15"}]->(product)

-- Quine style: use an intermediate node
MATCH (customer), (order), (product)
WHERE id(customer) = idFrom("customer", $that.customerId)
  AND id(order) = idFrom("order", $that.orderId)
  AND id(product) = idFrom("product", $that.productId)
SET order.amount = $that.amount, order.date = $that.date
CREATE (customer)-[:PLACED]->(order)-[:CONTAINS]->(product)
```

### Edges Have No IDs

In Neo4j, `RETURN id(e)` returns an edge's internal ID. In Quine, edges do not have IDs. `MATCH (n)-[e]->(m) RETURN id(e)` does not return a useful value. Look up edges from one of the endpoint nodes instead.

### Labels Are Not Indexed

In Neo4j, `MATCH (p:Person)` uses a label index to efficiently find all Person nodes. In Quine, labels are just properties — there is no label index. A query like `MATCH (p:Person) RETURN p` scans every node in the graph to check for the `:Person` label.

Labels are still useful for organization and for standing query pattern matching, but they do not make ad-hoc queries faster.

### Counting All Nodes Is Expensive

In Neo4j, `MATCH (n) RETURN count(*)` is a fast metadata lookup. In Quine, this query scans the entire graph and should be avoided.

## Unsupported Features

| Feature | Neo4j | Quine | Alternative |
|:--------|:------|:--------------|:------------|
| Edge properties | Supported | Not supported | Use intermediate nodes |
| Multiple edges (same label/direction/endpoints) | Supported | Not supported | N/A — edges are idempotent |
| `shortestPath` in MATCH/MERGE patterns | Supported | Not supported | Use as expression: `RETURN shortestPath((a)-[*]->(b))` |
| `allShortestPaths` | Supported | Not supported | — |
| Variable-length patterns in standing queries | Supported | Not supported | Use in ad-hoc queries only |
| `DETACH DELETE` on a path | Supported | Not supported | Delete nodes individually |
| `percentileCont`, `percentileDisc`, `stDev`, `stDevP` | Supported | Not supported | — |
| `sum`/`avg` on durations | Supported | Not supported | — |
| Query hints | Supported | Silently ignored | — |
| Cypher commands (system management) | Supported | Not supported | Use REST API |

### shortestPath Usage

`shortestPath` works in Quine, but only as an expression — not inside a `MATCH` pattern. Bind the start and end nodes first:

```cypher
-- Neo4j style (won't work in Quine)
MATCH p = shortestPath((a:Person)-[*]->(b:Person))
WHERE a.name = "Alice" AND b.name = "Bob"
RETURN p

-- Quine style
MATCH (a), (b)
WHERE id(a) = idFrom("person", "Alice") AND id(b) = idFrom("person", "Bob")
RETURN shortestPath((a)-[*]->(b))
```

The default maximum path length is 10 hops. Override with range syntax: `shortestPath((a)-[*..20]->(b))`.

## Standing Query Constraints

Standing queries use a restricted subset of Cypher. These constraints do not apply to ad-hoc or ingest queries.

### Distinct ID Mode (Default)

- `MATCH` patterns must be tree-shaped or linear — no cycles
- Edges cannot be aliased to variables (`-[:LABEL]->` is fine, `-[e:LABEL]->` is not)
- Edges must be directed, have exactly one label, and cannot be variable-length
- `WHERE` clauses only support: literal comparisons, `IS NULL`, `IS NOT NULL`, regex, and `id(n) = idFrom(...)`
- Must return exactly one value: `RETURN DISTINCT id(n)` or `RETURN DISTINCT strId(n)`

### Multiple Values Mode

Relaxes Distinct ID constraints:

- Can return multiple values including property values (e.g., `RETURN n.name, id(m)`)
- `WHERE` clause supports broader expressions on node properties
- Does not support `DISTINCT`
- Still cannot use variable-length patterns, sub-queries, or procedures in `WHERE`

## Quine-Specific Features

These features exist in Quine but not in Neo4j.

### Atomic Property Updates

In a streaming system, concurrent operations can cause race conditions. Quine provides atomic procedures to safely update properties:

```cypher
-- Atomically increment a counter (avoids read-modify-write races)
CALL int.add(node, "count", 1) YIELD result

-- Atomically add to a set
CALL set.insert(node, "tags", "important") YIELD result

-- Atomically merge sets
CALL set.union(node, "categories", ["A", "B"]) YIELD result
```

These lock the node for the duration of the operation. See [Atomic Property Updates](advanced-cypher.md#atomic-property-updates).

### Relationship Patterns as Predicates

A bare relationship pattern can be used as a boolean condition in ad-hoc queries, including with `AND`/`OR`/`NOT` and inside `CASE WHEN`, without wrapping it in `exists(...)`:

```cypher
MATCH (p) WHERE NOT (p)-[:has_manager]->() RETURN p
```

Standing query patterns do not support pattern expressions in boolean positions. See [Relationship Patterns as Predicates](advanced-cypher.md#relationship-patterns-as-predicates).

### Composite Property Types

Quine can store `MAP` and `LIST OF ANY` as node properties — not just scalar types:

```cypher
SET n.metadata = {source: "kafka", topic: "events"}
SET n.tags = ["urgent", "reviewed"]
```

Use `castOrThrow.map()` or `castOrNull.map()` to hint the type to the query compiler when using composite values in subsequent operations. See [Casting Property Types](advanced-cypher.md#casting-property-types).

### Temporal Functions

Quine includes functions for working with time that are not in standard OpenCypher:

- `datetime()`, `localdatetime()`, `date()`, `time()`, `localtime()`, `duration()`
- Temporal arithmetic and comparison
- `reify.time()` procedure for materializing time as graph nodes

See [Temporal Functions](temporal-functions.md).

### Exploring Without Known IDs

When you don't know specific node IDs, use `recentNodes` or `recentNodeIds` to sample recently accessed nodes. This applies to all ad-hoc queries: the Exploration UI query bar, sample queries in recipes, and any query run outside of ingest or standing queries.

```cypher
-- Best: direct lookup when you know the node identity
MATCH (n) WHERE id(n) = idFrom("customer", "CUST-123") RETURN n

-- When no specific node is known: sample recently accessed nodes
CALL recentNodes(20)

-- Use recent nodes to anchor a larger query
CALL recentNodeIds(1000) YIELD nodeId
MATCH (n)-[:KNOWS]->(m)
WHERE id(n) = nodeId
RETURN n.name, m.name
LIMIT 10

-- Sample recent nodes filtered by label
CALL recentNodes(1000) YIELD node AS nId
MATCH (n)
WHERE id(n) = nId AND labels(n) = ["Order"]
RETURN n
```

**Recipe sample queries** (the drop-down in the Exploration UI) and **query bar queries** are ad-hoc queries. They must use `idFrom()` when the target node identity is known, or `recentNodes()`/`recentNodeIds()` when it is not — never a bare label or property scan.

**Recipe quick queries** (right-click context menu on a node) receive the clicked node bound to the variable `n`, so they are already anchored. Their Cypher should expand outward from `n`, not scan for unrelated nodes.

## Common Mistakes

### Searching by Property Instead of ID

```cypher
-- WRONG: scans all nodes (slow at any scale)
MATCH (user:User)
WHERE user.email = "alice@example.com"
RETURN user

-- CORRECT: direct lookup by computed ID
MATCH (user)
WHERE id(user) = idFrom("user", "alice@example.com")
RETURN user
```

### Using MERGE Like Neo4j

```cypher
-- WRONG: MERGE does a property-based search, causing a full scan
MERGE (customer:Customer {customerId: "CUST-123"})
SET customer.name = "Alice"

-- CORRECT: address by ID, set properties and labels separately
MATCH (customer)
WHERE id(customer) = idFrom("customer", "CUST-123")
SET customer:Customer, customer.name = "Alice", customer.customerId = "CUST-123"
```

### Putting Properties on Edges

```cypher
-- WRONG: edge properties are not supported
CREATE (a)-[:SENT {amount: 100, currency: "USD"}]->(b)

-- CORRECT: use an intermediate node to hold the data
MATCH (sender), (tx), (receiver)
WHERE id(sender) = idFrom("account", $that.from)
  AND id(tx) = idFrom("transaction", $that.txId)
  AND id(receiver) = idFrom("account", $that.to)
SET tx.amount = $that.amount, tx.currency = $that.currency, tx:Transaction
CREATE (sender)-[:SENT]->(tx)-[:RECEIVED_BY]->(receiver)
```

### Inconsistent idFrom Arguments

```cypher
-- WRONG: these create two separate nodes for the same customer
-- Stream 1:
MATCH (c) WHERE id(c) = idFrom("customer", $that.customer_id) ...
-- Stream 2:
MATCH (c) WHERE id(c) = idFrom("cust", $that.customerId) ...

-- CORRECT: use identical type prefix and field across all streams
-- Stream 1:
MATCH (c) WHERE id(c) = idFrom("customer", $that.customer_id) ...
-- Stream 2:
MATCH (c) WHERE id(c) = idFrom("customer", $that.customer_id) ...
```

### Writing Sample Queries with Label Scans

```cypher
-- WRONG: scans all nodes (labels are not indexed)
MATCH (p:Person) RETURN p LIMIT 10

-- WRONG: scans all nodes looking for property value
MATCH (n) WHERE n.type = "order" RETURN n LIMIT 10

-- CORRECT: look up a known node by ID
MATCH (n) WHERE id(n) = idFrom("customer", "CUST-123") RETURN n

-- CORRECT: when no specific node is known, sample recently accessed nodes
CALL recentNodes(10)

-- CORRECT: sample recent nodes, then filter by label
CALL recentNodes(1000) YIELD node AS nId
MATCH (n)
WHERE id(n) = nId AND labels(n) = ["Person"]
RETURN n
```

### Using shortestPath in a MATCH Pattern

```cypher
-- WRONG: shortestPath cannot be used in MATCH patterns
MATCH p = shortestPath((a)-[*]->(b))
WHERE id(a) = idFrom("node", "start")
RETURN p

-- CORRECT: bind endpoints first, use shortestPath as expression
MATCH (a), (b)
WHERE id(a) = idFrom("node", "start") AND id(b) = idFrom("node", "end")
RETURN shortestPath((a)-[*]->(b))
```


---
# Quick Start
URL: https://quine.io/getting-started/quick-start/

# Quick Start

## What is Quine

Quine combines the real-time event stream processing capabilities of systems like Flink and ksqlDB with a graph-structured data model as found in graph databases like Neo4j and TigerGraph. Quine is a key participant in a streaming event data pipeline that consumes data, builds it into a graph structure, runs computation on that graph to answer questions, find patterns, run algoritms, or otherwise compute results and stream them out.

![Quine Streaming Graph Pipeline](images/what_is_quine_linear_diagram.png)

All together, Quine can:

* Ingest high-volume streaming event data from stream processing systems like Kafka, Kinesis, files, APIs, or databases/data warehouses
* Convert and merge it into durable, versioned, connected data (a graph)
* Monitor that connected data for complex structures or values
* Trigger arbitrary computation every time your pattern matches the data
* Emit high-value events in real-time to stream processing systems like Kafka, Kinesis, APIs or databases/data warehouses

This collection of capabilities represents a robust system for stateful event-driven arbitrary computation in a platform scalable to any size of data or desired throughput.

## Before you begin

Concepts that you should already be familiar with:

* [Event driven architectures](https://en.wikipedia.org/wiki/Event-driven_architecture)
* [Graph database concepts](https://en.wikipedia.org/wiki/Graph_database)
* [Cypher graph query language](https://en.wikipedia.org/wiki/Cypher_%28query_language%29)
* Interacting with [REST API](https://en.wikipedia.org/wiki/Representational_state_transfer) endpoints

If you are unsure of any of these, please consider familiarizing yourself with them in depth before you continue. Start with [Installing Quine](installing-quine-tutorial.md) for a more thorough introduction to Quine and streaming graph concepts.

## Install Quine

Start Quine using a distribution or locally compiled source code (described below).

### From a Docker container

* With Docker installed, run Quine from Docker Hub.
* `docker run -p 8080:8080 thatdot/quine`

### From an executable

Quine requires a Java 11 or newer JRE.

* [Download](https://quine.io/download/)  the executable `jar` file.
* From a working directory: `java -jar quine-2.1.1.jar`

### From source code

* Clone the [source code](https://github.com/thatdot/quine)  from GitHub.
* Ensure that you have a recent Java Development Kit (11 or newer) and `sbt` installed.
* From the main directory of the repository on your machine: `sbt quine/run`

## Connect to Quine

There are two main structures that you need to configure in Quine; the [**ingest stream**](/reference/rest-api/?av=v2#/operations/create-ingest) forms the event stream into the graph, and [**standing queries**](/reference/rest-api/?av=v2#/operations/create-standing-query) which match and take action on nodes in the graph. All data operations in the API are scoped to a named graph — the default graph is named `quine`. Follow the links in the tutorial to the API documentation to learn more about the schema for each object.

### Rapid API testing with provided `.rest` file

We’ve created a short [**.rest**](https://that.re/quine-quickstart-rest) file that can be used with VSCode’s [REST Client plugin](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) or [IntelliJ’s HTTP Client](https://www.jetbrains.com/help/idea/http-client-in-product-code-editor.html) to rapidly try out the following REST API calls. Use it as a point-and-click alternative to cURL.

<iframe src="https://www.youtube.com/embed/7Oi1eY8w-mQ?si=07IXbnXEBTq-vKgx" title="Quine REST API Quickstart" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

### Connect an Event Stream

For example, let's ingest the live stream of new pages created on Wikipedia; [mediawiki.page-create](https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_page_create).

Create a "server sent events" ingest stream to connect Quine to the `page-create` event stream using the [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) API endpoint.

Issue the following `curl` command in a terminal running on the machine were you started Quine.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests" \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "wikipedia-page-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/page-create"
  },
  "query": "CREATE ($that)"
}'
```

**Congratulations!** You are ingesting raw events into Quine and manifesting nodes in the graph.

!!! Warning
    Now that you have some data in Quine, you might be tempted to do some common simple operations most devs do with a new database. But Quine isn't (exactly) a database! Sure, it stores data and let's you query it, but "simple for a database" and "simple for Quine" aren't always the same. For example: `MATCH (n) RETURN count(n)`  That's a perfectly normal operation that almost everyone would do to begin testing out a new database because it's a simple sanity check expected to run quickly. But in contrast to a database, counting all nodes in Quine is actually quite an expensive operation! For more on why counting is hard and [how Quine differs from traditional databases](../core-concepts/streaming-graph-vs-database.md), see that page, and especially [the section on counting](../core-concepts/streaming-graph-vs-database.md#counting-is-hard).

### View the Data

Let's look at a node to see what it contains by submitting a Cypher request via the [Cypher Query Return Nodes: `POST /api/v2/graph/quine/cypher:queryNodes`](/reference/rest-api/?av=v2#/operations/query-cypher-nodes) API endpoint.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/cypher:queryNodes" \
     -H 'Content-Type: text/plain' \
     -d "CALL recentNodes(1)"
```

This query calls the `recentNodes` Cypher procedure to retrieve the most recent one (1) node. You will get a response back that looks something like this:

```json
{
  "columns": [
    "node"
  ],
  "results": [
    [
      {
        "id": "7a9a936f-ae1a-49c5-ba99-0ec6401bfd7d",
        "labels": [],
        "properties": {
          "database": "enwikisource",
          "rev_slots": {
            "main": {
              "rev_slot_content_model": "proofread-page",
              "rev_slot_origin_rev_id": 12558576,
              "rev_slot_sha1": "lqrhvc49cgzegvvfqnzg3c6bpxs55up",
              "rev_slot_size": 1699
            }
          },
          "rev_id": 12558576,
          "rev_timestamp": "2022-08-23T18:34:25Z",
          "rev_len": 1699,
          "rev_minor_edit": false,
          "parsedcomment": "Proofread",
          "page_title": "Page:The_Works_of_H_G_Wells_Volume_6.pdf/423",
          "rev_content_format": "text/x-wiki",
          "page_id": 4034745,
          "page_is_redirect": false,
          "meta": {
            "domain": "en.wikisource.org",
            "dt": "2022-08-23T18:34:25Z",
            "id": "3c8c9150-19aa-4f33-be5d-bf3ef8d8a994",
            "offset": 241649600,
            "partition": 0,
            "request_id": "0ba80d5c-3c97-4971-b0d2-360c5d20e0f6",
            "stream": "mediawiki.page-create",
            "topic": "eqiad.mediawiki.page-create",
            "uri": "https://en.wikisource.org/wiki/Page:The_Works_of_H_G_Wells_Volume_6.pdf/423"
          },
          "page_namespace": 104,
          "rev_sha1": "lqrhvc49cgzegvvfqnzg3c6bpxs55up",
          "comment": "/* Proofread */",
          "rev_content_model": "proofread-page",
          "$schema": "/mediawiki/revision/create/1.1.0",
          "performer": {
            "user_edit_count": 10176,
            "user_groups": [
              "autopatrolled",
              "*",
              "user",
              "autoconfirmed"
            ],
            "user_id": 141433,
            "user_is_bot": false,
            "user_registration_dt": "2009-07-12T12:33:52Z",
            "user_text": "MER-C"
          }
        }
      }
    ]
  ]
}
```

The API call is the functional equivalent to issuing the `CALL recentNodes(1)` query in the Exploration UI:
![image](https://user-images.githubusercontent.com/99685020/186532706-15ea5919-9391-4d0a-87a1-488d3c551cd2.png)

!!! Note
    Your API response will contain a different set of parameters than above because you are ingesting a stream of live events from Wikipedia.

### Connect Data in Streams

This ingest stream is performing the most basic of data ingest functionality; it manifests a disconnected node directly from each event emitted from the Wikipedia event stream.

### Create a Standing Query

A Standing Query matches some graph structure incrementally while new event data is ingested. Creating a standing query is done with a single call to the [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query) API endpoint.

Right now, Quine is the only component in our data pipeline. Let's configure a standing query that watches for new nodes to enter the graph and print the node contents to the console.

!!! Note
    A standing query can emit the event data, re-form the event into new events, or trigger actions to inform elements downstream in your data pipeline (e.g., a Kafka topic).

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/standingQueries" \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "wikipedia-new-page-node",
  "pattern": {
    "query": "MATCH (n) RETURN DISTINCT id(n)",
    "type": "Cypher"
  },
  "outputs": [
    {
      "name": "print-output",
      "destinations": [
        { "type": "StandardOut" }
      ]
    }
  ]
}'
```

You will see new node events similar to the one below appear in the same console window where you launched Quine immediately after running the `curl` command. These events contain the `id` of each new node created in the graph.

```shell
2022-08-23 14:06:55,174 Standing query `print-output` match: {"meta":{"isPositiveMatch":true,"resultId":"dab367a3-b272-7dba-c12e-a65bc9f5e0b8"},"data":{"id(n)":"911d88e0-413a-42bd-a0f8-dd15bbf6aff6"}}
```

## Ending the Stream

This quick-start is a foundation that you can build on top of to ingest and interpret your own streams of data. But for now, we can [Pause Ingest Stream: `POST /api/v2/graph/quine/ingests/{ingestName}:pause`](/reference/rest-api/?av=v2#/operations/pause-ingest) the ingest stream and [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) Quine before moving on.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests/wikipedia-page-create:pause"
```

`curl` will return a confirmation that the ingest stream is paused and metrics about what had been ingested to that point.

```json
{
  "name": "wikipedia-page-create",
  "status": "PAUSED",
  "settings": {
    "source": {
      "type": "ServerSentEvent",
      "url": "https://stream.wikimedia.org/v2/stream/page-create"
    },
    "query": "CREATE ($that)"
  },
  "stats": {
    "ingestedCount": 3096,
    "rates": {
      "count": 3096,
      "oneMinute": 1.1004373606561983,
      "fiveMinute": 1.1045410320126854,
      "fifteenMinute": 1.123947504968256,
      "overall": 1.12992666124191
    },
    "byteRates": {
      "count": 4471549,
      "oneMinute": 1529.8568026569903,
      "fiveMinute": 1553.9585381108302,
      "fifteenMinute": 1614.01351325884,
      "overall": 1631.9520432126692
    },
    "startTime": "2022-08-23T18:30:58.571823Z",
    "totalRuntime": 2739271
  }
}
```

You can stop Quine by either typing `CTRL-c` into the terminal window or perform a *graceful shut down* by issuing a POST to the [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) endpoint.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/system:shutdown"
```

## Next Steps

Learn how Quine uses recipes to store a graph configuration and UI enhancements in the [recipes](recipes-tutorial.md) getting started guide.


---
# Metrics Quick Start
URL: https://quine.io/learn/metrics/quick-start/

# Metrics Quick Start

Quine collects metrics about ingest rates, graph operations, persistence, and system health. This guide shows you how to view these metrics immediately without setting up external monitoring tools.

For production monitoring with dashboards and alerting, see:

## View Metrics via REST API

The simplest way to view metrics is through the built-in REST API endpoint:

This returns a JSON object containing all current metrics:

```json
{
  "counters": {
    "quine.shard.shard-0.sleep-counters.slept-success": { "count": 1523 },
    "quine.shard.shard-0.sleep-counters.woken": { "count": 1847 }
  },
  "gauges": {
    "shared.valve.ingest.my-ingest.metric": { "value": 0 }
  },
  "meters": {
    "quine.ingest.my-ingest.count": {
      "count": 50000,
      "mean_rate": 2534.21,
      "m1_rate": 2100.50,
      "m5_rate": 1890.33,
      "m15_rate": 1756.12
    }
  },
  "timers": {
    "persistor.persist-event": {
      "count": 12500,
      "mean_rate": 625.0,
      "duration_units": "milliseconds",
      "mean": 1.23,
      "p50": 0.95,
      "p99": 4.21
    }
  }
}
```

### Key Metrics to Monitor

| Metric                                                | Type    | What It Tells You                                 |
|:------------------------------------------------------|:--------|:--------------------------------------------------|
| `quine.ingest.{ingest-name}.count` | Meter   | Records ingested and ingest rate                  |
| `quine.ingest.{ingest-name}.bytes` | Meter   | Data volume ingested                              |
| `shared.valve.ingest.{ingest-name}.metric`            | Gauge   | Backpressure level (0 = healthy)                  |
| `persistor.*.persist-event`                           | Timer   | Persistence latency                               |
| `quine.standing-queries.results.*` | Meter   | Standing query output rate                        |
| `quine.standing-queries.dropped.*` | Counter | Dropped results due to backpressure (should be 0) |

### Polling Metrics

To monitor metrics over time, poll the endpoint periodically:

## View Metrics via JMX

Quine exports all metrics via JMX by default. You can browse them using JConsole, VisualVM, or any JMX client.

### Connect with JConsole

1. Start JConsole (included with the JDK):

    ```bash
    jconsole
    ```

2. Select the Quine process from the list of local Java applications, or connect to a remote host

3. Navigate to the **MBeans** tab

4. Expand **metrics** to browse all available metrics organized by category

### Remote JMX Access

To enable remote JMX connections, start Quine with these JVM options:

Then connect from a remote machine:

```bash
jconsole YOUR_HOST_IP:9010
```

For production environments, enable authentication and SSL. See [Oracle's JMX documentation](https://docs.oracle.com/en/java/javase/17/management/monitoring-and-management-using-jmx-technology.html) for secure configuration options.

## Metrics Output Formats

Quine supports multiple metrics reporters. By default, only JMX is enabled. Configure additional reporters in your configuration file:

## Next Steps


---
# Data Modeling and Query Design
URL: https://quine.io/core-concepts/data-modeling/

# Data Modeling and Query Design

Effective Quine implementations require understanding the relationship between graph structure, ingest queries, and standing queries. Unlike traditional databases where schema design precedes query writing, Quine works best when the design process starts from the patterns to be detected.

The core challenge is translating a question, such as "detect when the same IP address logs into multiple user accounts", into a graph structure and queries that answer it efficiently.

## The Design Relationship

### Working Backward from the Goal

In traditional databases, the workflow is:

1. Understand the data schema
2. Design tables/collections to store it
3. Write queries to answer questions

In Quine, the most effective workflow inverts this:

1. **Define what to find**: What pattern or question matters?
2. **Design the graph structure**: What nodes, edges, and properties enable finding that pattern?
3. **Write the ingest query**: How do incoming records create that structure?

This inversion exists because the ingest queries and standing queries are tightly coupled to the shape of the graph. If the ingest query changes, the graph changes. If the standing query needs to change, then it is likely the graph will need to change, which means the ingest query will need to change.

### The Three-Part System

Every Quine implementation involves three interconnected components:

![Three-part system: Standing Query defines Graph Shape, which is created by Ingest Query](core-concepts-images/three-part-system.png)

- **Standing Query**: Defines the pattern to detect. Its requirements determine the necessary graph structure.
- **Graph Shape**: The nodes, edges, and properties that make up the data model. Must contain the structure that the standing query pattern requires.
- **Ingest Query**: Transforms incoming records into the graph structure. Creates nodes, sets properties, and establishes edges.

The standing query's requirements shape the graph design, which in turn shapes the ingest query. This relationship is the key to effective Quine implementations.

### Why This Order Matters

Standing queries evaluate incrementally as data arrives. Each node maintains awareness of which standing query patterns it participates in, so when new data creates or modifies a node, only the relevant patterns are checked. This incremental evaluation makes standing queries inherently efficient.

However, the graph must contain the structure that the standing query pattern describes. If a pattern requires `(user)-[:PURCHASED]->(order)`, the ingest queries must create both the nodes and the edge from ingested records. The standing query cannot match structure that does not exist.

The primary performance consideration is in **ingest queries**, not standing queries. Ingest queries that search for nodes by property (rather than selecting by ID) cause expensive all-node scans. For more on this distinction, see [IDs Over Indices](streaming-graph-vs-database.md#ids-over-indices).

## Discovery: Questions to Ask First

Before designing a graph structure, gather key information about the use case:

**Data Analysis Questions:**

- What questions about the data need to be answered?
- Is this real-time streaming data or batch data?
- What is the format? (JSON, CSV, Protobuf, etc.)

**Data Relationship Questions:**

- What connections between data points are relevant to the use case?
- What entities in the data have natural identifiers?
- Will multiple data sources reference the same entity?

**Output Questions:**

- Where should results go? (Kafka, webhook, database, back to the graph?)
- How many consumers will read the results?
- What latency is acceptable for results?

These questions help clarify whether nodes, edges, properties, or some combination are needed, and guide standing query design.

## JSON to Graph Translation

Most data entering Quine arrives as JSON records from streams like [Kafka](../learn/ingest-sources/kafka.md) or [Kinesis](../learn/ingest-sources/kinesis.md). The core design decisions involve mapping JSON structure to graph elements.

### Core Decisions

When examining incoming data, consider these questions:

| JSON Element | Graph Element | Decision Criteria |
|:-------------|:--------------|:------------------|
| Objects with identity | **Node** | Will you reference this object from multiple places? Does it have a natural key? |
| Scalar values | **Property** | Is this an attribute of a node rather than a node itself? |
| Relationships | **Edge** | Do you need to traverse from one node to another in a standing query? |
| Enumerable values | **Property, not Edge** | For high-cardinality categorical data (status, type, region), use properties to avoid supernodes |
| Natural keys | **Node ID** | What combination of fields uniquely identifies this node? |

!!! tip "Edge vs Property Decision"

    Use **edges** when you need to traverse relationships in standing queries (e.g., "find users connected to this IP").

    Use **properties** for categorical or enumerable values that you'll filter on but not traverse (e.g., status="active", region="us-east"). Creating edges for every possible status value creates supernodes and degrades performance.

**Example**: Consider a purchase event:

```json
{
  "orderId": "ORD-123",
  "customerId": "CUST-456",
  "product": "Widget",
  "category": "Electronics",
  "amount": 99.99,
  "timestamp": "2024-01-15T10:30:00Z"
}
```

Possible mappings:

- **Order node**: Identified by `orderId`, with properties `amount`, `timestamp`
- **Customer node**: Identified by `customerId`
- **Category node**: Identified by `category` (if you need to find purchases by category)
- **Product as property**: If you only need the product name, not to traverse to it
- **Edges**: `PURCHASED` from customer to order, `IN_CATEGORY` from order to category

The right mapping depends on standing query requirements. Finding "customers who purchased from multiple categories" requires category nodes and edges. Finding "orders over $100" only requires category as a property.

### The "All Nodes Exist" Philosophy

A fundamental concept in Quine is that nodes are never created. Instead, they are selected by ID and behave as if they already exist.

```cypher
// This doesn't "create" a customer; it selects the customer node
// by ID and sets properties on it
MATCH (customer)
WHERE id(customer) = idFrom("customer", $that.customerId)
SET customer.name = $that.customerName
```

This design enables powerful capabilities:

- **Multiple streams can reference the same node**: An order stream and a customer profile stream can both update the same customer node without coordination.
- **Order independence**: It doesn't matter which stream's data arrives first. The customer node accumulates data from all sources.
- **No existence checks**: There is no need to check if a node exists before referencing it.

For more details, see [All Nodes Exist](streaming-graph-vs-database.md#all-nodes-exist).

### ID Selection Strategies

The `idFrom` function deterministically generates a node ID from input values. The ID strategy is one of the most important design decisions.

**Prefix by type** to prevent ID collisions between different node types:

```cypher
// Good: Prefixed IDs won't collide even if customer and order
// happen to have the same numeric ID
id(customer) = idFrom("customer", $that.customerId)
id(order) = idFrom("order", $that.orderId)

// Risky: If customerId and orderId could both be "123",
// they'd resolve to the same node
id(customer) = idFrom($that.customerId)
id(order) = idFrom($that.orderId)
```

**Use natural keys** when the data has them:

```cypher
// Email as natural key for user
id(user) = idFrom("user", $that.email)

// Composite key for time-series data
id(reading) = idFrom("sensor-reading", $that.sensorId, $that.timestamp)
```

**Stay consistent** across all ingests. If one ingest uses `idFrom("customer", id)` and another uses `idFrom("cust", id)`, they will create separate nodes for the same customer.

For detailed ID provider options, see [ID Provider](id-provider.md).

## Designing Standing Queries First

Since standing queries drive the design, the first step is clearly defining what to find.

### Start with the Question

Express the goal as a graph pattern:

- What nodes are involved?
- What edges connect them?
- What property conditions must be true?
- What should happen when the pattern is found?

**Example questions and their patterns:**

| Question | Pattern |
|:---------|:--------|
| "Find users who login from multiple countries" | `(user)-[:LOGGED_IN_FROM]->(country)` with count > 1 |
| "Detect when a device connects to a known-bad IP" | `(device)-[:CONNECTED_TO]->(ip:BadIP)` |
| "Track orders that ship to a different address than billing" | `(order)-[:SHIPS_TO]->(addr1), (order)-[:BILLS_TO]->(addr2)` where addr1 != addr2 |

### Choose the Right Mode

Quine supports two standing query modes:

**Distinct ID** (default): Returns once per unique user node match. More efficient, lower resource usage.

```cypher
// Emits once when a user first has any friend
MATCH (user:Person)-[:FRIEND]->(friend:Person)
RETURN DISTINCT id(user)
```

**Multiple Values**: Can return multiple results per match, including property values.

```cypher
// Emits for each friend relationship, returning friend details
MATCH (user:Person)-[:FRIEND]->(friend:Person)
RETURN id(user) AS userId, friend.name AS friendName
```

Distinct ID is appropriate when only the existence of a pattern matters. Multiple Values is appropriate when details about each match are needed.

For syntax details and constraints, see [Standing Query Modes](../learn/standing-queries/standing-queries.md#distinct-id-pattern-queries).

### Common Standing Query Patterns

**Pattern Detection**: Find a specific subgraph structure.

```cypher
// Match each login event connected to an IP
MATCH (ip:IP)<-[:FROM]-(login:Login)
RETURN DISTINCT id(ip) AS ipId
```

**Aggregation via Graph Update**: Instead of emitting downstream, use an output query to update the graph. Building on the standing query above:

```cypher
// Output query: atomically increment a login counter on the IP node
MATCH (ip) WHERE id(ip) = $that.data.ipId
CALL int.add(ip, "loginCount", 1) YIELD result
RETURN result
```

**Chained Queries**: A second standing query matches on values computed by the first. Building on the aggregation above, alert when an IP exceeds a login threshold:

```cypher
// Second standing query: Alert on IPs with excessive logins
MATCH (ip:IP)
WHERE ip.loginCount = 5
RETURN DISTINCT id(ip)
```

### The Output Decision: Emit vs. Update

When a standing query matches, the output determines what happens:

**Emit downstream** when external systems need the results:

- Send to Kafka for downstream processing
- POST to a webhook for alerting
- Write to a file for batch analysis

**Update the graph** when you need computed values for further analysis:

- Maintain counters or aggregations
- Mark nodes with computed flags
- Create derived relationships

Both approaches can be combined by chaining outputs. For all output options, see [Standing Query Outputs](../learn/standing-queries/standing-queries.md#result-outputs).

## Designing the Graph Structure

With the standing query pattern defined, the next step is designing the graph structure that enables it.

### Map Pattern to Required Structure

For each element in the standing query pattern:

| Pattern Element | Required Structure |
|:----------------|:-------------------|
| Node with label | Ingest must SET the label |
| Node property condition | Ingest must SET that property |
| Edge between nodes | Ingest must CREATE the edge |
| Node identity | Consistent `idFrom` strategy |

**Example**: For the pattern `(user:User)-[:PURCHASED]->(order:Order {status: "completed"})`

The ingest must: SET `:User` and `:Order` labels, SET `order.status`, CREATE the `:PURCHASED` edge, and use consistent `idFrom` strategies (e.g., `idFrom("user", $that.userId)` and `idFrom("order", $that.orderId)`).

!!! note
    While graph structure typically comes from ingest queries, standing query outputs can also modify the graph by adding labels, properties, or edges that subsequent standing queries match on.

### Design Considerations

#### Supernodes

A supernode is a node with an excessive number of edges. Supernodes cause performance problems only when queries traverse edges from the supernode to other nodes. If no queries traverse those edges, or if standing query patterns terminate on the supernode rather than start from it, the supernode won't degrade performance.

**Problem pattern:**

```cypher
// Every order connects to a single "store" node
MATCH (order), (store)
WHERE id(order) = idFrom("order", $that.orderId) AND id(store) = idFrom("store", "main-store")
SET order:Order, store:Store
CREATE (order)-[:FROM_STORE]->(store)

// This query traverses FROM the store supernode
MATCH (store:Store)<-[:FROM_STORE]-(order:Order)
RETURN order
```

**When supernodes are safe:**

```cypher
// This query terminates ON the store node—edges aren't traversed
MATCH (order:Order)-[:FROM_STORE]->(store:Store)
RETURN DISTINCT id(store)
```

**Solutions when supernodes cause problems:**

- Use properties instead of edges (e.g. in the example above, set `order.storeName = "main-store"` instead of creating an edge to a store node)
- Partition supernodes (e.g., by time period: `store-2024-01`, `store-2024-02`)
- Reconsider if the relationship is actually needed for the queries

For monitoring supernodes, see [Diagnosing Bottlenecks](../learn/troubleshooting/diagnosing-bottlenecks.md#step-4-check-for-supernodes).

#### All-Node Scans

Matching on properties instead of IDs forces Quine to scan all nodes, which is extremely slow at scale.

```cypher
// Bad: Scans all nodes looking for matching email
MATCH (user)
WHERE user.email = $that.userEmail

// Good: Direct ID lookup
MATCH (user)
WHERE id(user) = idFrom("user", $that.userEmail)
```

The warning "Cypher query may contain full node scan" indicates this problem. Always anchor queries with `id(n) = idFrom(...)`.

See [Selecting Nodes vs Searching for Nodes](index.md#selecting-nodes-vs-searching-for-nodes) for more detail.

#### Idempotency and ID Consistency

Quine is an eventually consistent system. Standing queries will match patterns once all relevant data arrives, regardless of arrival order. However, two issues require attention:

- **Non-idempotent operations**: If ingested data or ingest queries aren't idempotent, duplicate processing can cause incorrect results. Design ingest queries so that processing the same record twice produces the same graph state.

- **ID mismatches**: Inconsistent `idFrom` arguments across ingests create duplicate nodes for the same logical entity. Document ID strategies to ensure all ingests use identical arguments.

For detailed examples and troubleshooting, see [Troubleshooting Ingest](../learn/troubleshooting/ingest.md).

## Designing Ingest Queries

With the graph structure defined, the next step is writing ingest queries to create it from incoming data.

### The Ingest Query Pattern

A typical ingest query follows this structure:

```cypher
// 1. Receive the incoming record as $that
WITH $that AS data

// 2. Select all nodes by ID (they "exist" already)
MATCH (node1), (node2), (node3)
WHERE id(node1) = idFrom("type1", data.field1)
  AND id(node2) = idFrom("type2", data.field2)
  AND id(node3) = idFrom("type3", data.field3)

// 3. Set properties and labels
SET node1.property = data.value,
    node1:Label1
SET node2 = data.nestedObject,  // Copies all fields from nested object
    node2:Label2

// 4. Create relationships
CREATE (node1)-[:RELATIONSHIP]->(node2),
       (node2)-[:ANOTHER_REL]->(node3)
```

### Key Principles

**Always anchor by ID**: Every node in the `MATCH` *should* have an `id(n) = idFrom(...)` condition.

**Create all structure from each record**: Each record should create all the nodes and edges it references. Other records should not be assumed to create missing pieces.

**Use labels for organization**: Labels like `:User`, `:Order`, `:Event` make queries clearer and enable label-based standing query patterns.

**Handle optional fields**: Use `coalesce` or conditional logic for fields that may be missing.

```cypher
SET node.nickname = coalesce(data.nickname, data.name)
```

For ingest configuration details, see [Ingest Streams](../learn/ingest-sources/index.md).

### Multiple Data Streams

When multiple streams reference the same node, Quine's "all nodes exist" philosophy enables powerful patterns.

**Example: Customer data from two sources**

Imagine two separate event streams about customers:

- Stream 1: Address changes from a CRM system
- Stream 2: Sales funnel status from a marketing platform

```cypher
// Stream 1: Address updates
WITH $that AS addressData
MATCH (customer)
WHERE id(customer) = idFrom("customer", addressData.customer_id)
SET customer.address1 = addressData.address1,
    customer.city = addressData.city,
    customer.state = addressData.state

// Stream 2: Sales funnel updates
WITH $that AS salesData
MATCH (customer)
WHERE id(customer) = idFrom("customer", salesData.customer_id)
SET customer.salesStatus = salesData.status,
    customer.highValueProspect = salesData.deal_size > 100000
```

Both streams reference the same customer node using `idFrom("customer", customer_id)`. The node accumulates properties from both sources regardless of arrival order.

**Key principles:**

- **Ensure ID consistency**: Both streams must use identical `idFrom` arguments for the same logical entity
- **Design for any arrival order**: The first event for a customer might come from either stream, and the design should work either way
- **Each stream owns its properties**: Avoid having multiple streams SET the same property, which can cause overwrites

**Heterogeneous streams**: Unlike some graph databases, Quine is designed to process heterogeneous streams where the same ingest creates both nodes and edges together. Separate ingests for nodes and edges are not required.

See the [Entity Resolution Recipe](../recipes/entity-resolution.md) for an example of building subgraphs from property data with consistent ID strategies.

## Worked Example: Wikipedia Revision Events

This section applies the design process to a real scenario: detecting non-bot edits to English Wikipedia pages.

### Step 1: Define the Goal

**Question**: "Find human-generated edits (not bots) to English Wikipedia pages"

**Pattern needed**: A revision (not by a bot, to the English Wikipedia) with its responsible user

```cypher
MATCH (user:user)-[:RESPONSIBLE_FOR]->(revNode:revision {bot: false, database: 'enwiki'})
```

### Step 2: Design the Graph Structure

Examining the [Wikipedia revision-create event schema](https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_revision_create), we identify entities:

![Wikipedia page-create graph model](../getting-started/images/page-create-graph.png)

| Entity | Node Type | ID Strategy | Key Properties |
|:-------|:----------|:------------|:---------------|
| Revision | `:revision` | `idFrom("revision", rev_id)` | bot flag, database, content |
| Page | `:page` | `idFrom("page", page_id)` | title, namespace |
| Database | `:db` | `idFrom("db", database)` | database name |
| User | `:user` | `idFrom("user", user_id)` | name, bot flag, edit count |
| Parent Revision | `:revision` | `idFrom("revision", rev_parent_id)` | Links revision history |

**Edges needed**:

- `(revision)-[:TO]->(page)`: revision targets a page
- `(page)-[:IN]->(db)`: page belongs to a database
- `(user)-[:RESPONSIBLE_FOR]->(revision)`: user made the revision
- `(parentRevision)-[:NEXT]->(revision)`: revision history chain

### Step 3: Write the Ingest Query

```cypher
MATCH (revNode), (pageNode), (dbNode), (userNode), (parentNode)
WHERE id(revNode) = idFrom('revision', $that.rev_id)
  AND id(pageNode) = idFrom('page', $that.page_id)
  AND id(dbNode) = idFrom('db', $that.database)
  AND id(userNode) = idFrom('user', $that.performer.user_id)
  AND id(parentNode) = idFrom('revision', $that.rev_parent_id)

SET revNode = $that,
    revNode.bot = $that.performer.user_is_bot,
    revNode:revision

SET parentNode.rev_id = $that.rev_parent_id

SET pageNode.id = $that.page_id,
    pageNode.namespace = $that.page_namespace,
    pageNode.title = $that.page_title,
    pageNode.comment = $that.comment,
    pageNode.is_redirect = $that.page_is_redirect,
    pageNode:page

SET dbNode.database = $that.database,
    dbNode:db

SET userNode = $that.performer,
    userNode.name = $that.performer.user_text,
    userNode:user

CREATE (revNode)-[:TO]->(pageNode),
       (pageNode)-[:IN]->(dbNode),
       (userNode)-[:RESPONSIBLE_FOR]->(revNode),
       (parentNode)-[:NEXT]->(revNode)
```

This creates the full graph structure from each revision event.

### Step 4: Write the Standing Query

**Pattern query** — matches the structure we need:

```cypher
MATCH (userNode:user)-[:RESPONSIBLE_FOR]->(revNode:revision {bot: false, database: 'enwiki'})
RETURN DISTINCT id(revNode) as id
```

**Output query** — enriches the match with full details:

```cypher
MATCH (revNode)<-[:RESPONSIBLE_FOR]-(userNode:user)
WHERE id(revNode) = $that.data.id
RETURN properties(revNode), userNode.name AS userName
```

The standing query monitors the streaming graph, emitting results the instant a non-bot English Wikipedia edit arrives. See the [Wikipedia Recipe](../recipes/wikipedia.md) for a complete runnable example.

For the complete tutorial with API calls, see [Ingest Streams Tutorial](../getting-started/ingest-streams-tutorial.md) and [Standing Queries Tutorial](../getting-started/standing-queries-tutorial.md).

## Advanced Patterns

### Standing Queries That Update the Graph

Instead of emitting results downstream, a standing query can write computed values back to the graph.

**Use case**: Count how many revisions each user has made.

```cypher
// Standing query pattern
MATCH (user:user)-[:RESPONSIBLE_FOR]->(rev:revision)
RETURN DISTINCT id(user) as userId

// Output: Update the user node with a count
MATCH (user)
WHERE id(user) = $that.data.userId
CALL int.add(user, "revisionCount", 1) YIELD result
RETURN result
```

!!! note

    The `int.add` function provides atomic increments, ensuring accurate counts even when multiple matches occur simultaneously. See [Atomic Property Updates](../learn/cypher/advanced-cypher.md#atomic-property-updates) for all available atomic procedures.

See the [Ethereum Recipe](../recipes/ethereum.md) for a complete example of tag propagation using standing queries that update the graph.

### Chaining Standing Queries

Complex analysis can be broken into stages:

**Stage 1**: Detect a threshold crossing and mark it in the graph

Use Multiple Values mode to enable the WHERE clause filter:

```cypher
// Multiple Values pattern: check threshold
MATCH (user:user)-[:RESPONSIBLE_FOR]->(rev:revision)
WHERE user.revisionCount > 100
RETURN id(user) as userId

// Output: Mark high-activity users
MATCH (user) WHERE id(user) = $that.data.userId
SET user.highActivity = true
```

**Stage 2**: Use the computed value in another standing query

```cypher
// Distinct ID pattern: Match on the flag set by Stage 1
MATCH (user:user {highActivity: true})-[:RESPONSIBLE_FOR]->(rev:revision {database: 'enwiki'})
RETURN DISTINCT id(rev)
```

This staged approach:

- Keeps individual queries simple and focused
- Allows intermediate results to be reused by multiple downstream queries
- Models complex workflows as a pipeline

See the [Entity Resolution Recipe](../recipes/entity-resolution.md) for a complete example that uses chained standing queries to resolve entities across overlapping address data.

### Recursive Patterns

A standing query's output can modify the graph in ways that trigger the same standing query again. This enables recursive algorithms like graph traversal or propagation.

```cypher
// Pattern: Find tainted nodes connected to untainted nodes
MATCH (source {tainted: true})-[:SENT_TO]->(recipient)
WHERE recipient.tainted IS NULL
RETURN DISTINCT id(recipient) as recipientId

// Output: Propagate taint to the recipient
MATCH (n) WHERE id(n) = $that.data.recipientId
SET n.tainted = true
// This node now matches as "source" in the pattern, propagating to its recipients
```

!!! warning

    Recursive patterns are powerful but require careful design to avoid infinite loops. Ensure your output query eventually stops creating conditions that match the pattern.

The [Ethereum Recipe](../recipes/ethereum.md) demonstrates this pattern for tracking "tainted" transactions, where the taint level propagates along transaction paths. For a more advanced example, the [Conway's Game of Life Recipe](../recipes/conways-gol.md) uses coordinated recursive standing queries to implement a cellular automaton, demonstrating that standing queries are Turing complete.

### Performance Considerations

**Split complex queries**: A single complex standing query can often be decomposed into simpler queries that chain together, improving maintainability and sometimes performance.

**Compute early**: Perform complex computations during ingest rather than in standing query outputs. Ingest queries should prepare data so standing queries only need to monitor for patterns. When computation must happen after a match, use chained standing queries rather than complex output queries.

**Balance ingest parallelism**: When running multiple ingest streams on the same host, the total parallelism across all ingests should be balanced. If optimal parallelism for a single ingest is 120, running 4 ingests on the same host should use parallelism ~30 each.

**Monitor backpressure**: If standing queries cannot keep up with ingest rate, the system backpressures to prevent data loss. Monitor the [`shared.valve.ingest`](../learn/troubleshooting/diagnosing-bottlenecks.md#standing-query-backpressure-valve) metric.

**Prefer Distinct ID mode**: Distinct ID standing queries use less memory to track pattern state compared to Multiple Values. Use Multiple Values only when multiple results per match are needed or when Distinct ID's restrictions prevent expressing the query.

For detailed performance tuning, see [Diagnosing Bottlenecks](../learn/troubleshooting/diagnosing-bottlenecks.md).


---
# Quine Indexing
URL: https://quine.io/core-concepts/id-provider/

# Quine Indexing

How do you write a query against an infinite amount of data?

A streaming system like Quine continuously receives data from an upstream source. In a sense, the new data must manifest in the correct location in the graph and index into the previous data.

## ID Providers

Each node in Quine's graph is defined by its ID—referred to internally as `QuineId`. The ID itself is fundamentally an uninterpreted sequence of bytes, but using each ID is mediated by a class of objects referred to as `IdProviders`. ID providers make working with IDs more convenient and allow for multiple types to be used or migration from one type to another.

An ID Provider is chosen at startup for an instance of the graph. the default ID provider creates and expects to find IDs that can be read as UUIDs. This means that every node in the graph is defined by a UUID.

Different ID Providers can implement different strategies for allocating new IDs. For instance, alternate UUID providers can be configured to generate UUIDs which conform to specification of UUID versions 3, 4, or 5.

## Supported ID Providers

- `uuid` - This ID provider will generate RFC compliant UUIDs, but will allow for reading a looser interpretation of UUIDs which allows for any use of the 128 bits available in a UUID. This is the default ID provider.
- `uuid-3` - Generate and read only Version-3 compliant UUIDs.
- `uuid-4` - Generate and read only Version-4 compliant UUIDs. When returning random UUIDs, and using `idFrom` (described below), deterministic UUIDs with Version-4 identifying bytes will be used.
- `uuid-5` - Generate and read only Version-5 compliant UUIDs.
- `long` - Generate random integer IDs in the range: [-(2^53-1), 2^53-1] -- these may be safely used as IEEE double-precision floating-point values without loss of precision. This id scheme is not appropriate for large-scale datasets because of the high likelihood of a collision.
- `byte-array` - generate unstructured byte arrays as IDs.

## idFrom(…)

Quine has a unique challenge: how to maintain state for a potentially infinite stream of data. One key strategy Quine uses is to deterministically generate known IDs from data. The `idFrom` function does exactly that. `idFrom` is a function we've added to Cypher which will take any number of arguments and deterministically produce a ID from that data. This is similar to a consistent-hashing strategy, except that the ID produced from this function is always an ID that conforms to the type chosen for the ID provider.

For example, if data ingested from a stream needs to set a new telephone number for a customer, the Cypher ingest query would use `idFrom` to locate the relevant node in the graph and update its property like this:

```cypher
MATCH (customer) 
WHERE id(customer) = idFrom('customer', $that.customer-id)
SET customer.phone = $that.new-phone-number
```

In this example, `idFrom('customer', $that.customer-id)` is used to generate a specific ID for a single node in the graph, determined by the constant string `'customer'` and the value of `customer-id` being read from the newly streamed record. The ID returned from this function call will be a single fixed ID in the space of whichever ID provider was chosen (`long`, `uuid`, etc.). That ID is used in the `WHERE` clause to select a specific node from the graph, allowing Quine to perform the corresponding update efficiently.

For guidance on ID strategies and graph design patterns, see [Data Modeling and Query Design](data-modeling.md).

## Using IDs in a Query

Quine fetches data when provided an anchor like a node ID. 
When issuing queries to explore an existing data set, all of the nodes necessary for your query may not be populated. So the first goal of an exploration is to efficiently find starting points for the data you want to explore.

Since the ID of a node is not usually known statically, we recommend using the `idFrom` function to have Quine compute the IDs of nodes based on data values.

```cypher
// Get a particular person's name, and the names of their paternal grandparents
MATCH
  (person :Person)-[:HAS_FATHER]->(dad :Person),
  (grandpa :Person)<-[:HAS_FATHER]-(dad)-[:HAS_MOTHER]->(grandma :Person)
WHERE id(person) = idFrom('person', person.name)
RETURN
  person.name AS person,
  grandpa.name AS paternalGrandfather,
  grandma.name AS maternalGrandmother
```
Unless you are running in a debug environment that has little data, we recommend avoiding queries that involve scanning the entire graph. The simplest way to avoid a scan is to make sure at least some part of a `MATCH (n) …` pattern has a constraint of the form `id(n) = ...` in the `WHERE` clause. This allows the query compiler to optimize the execution plan to begin with a hop to a node with a known ID instead of needing to consider every node ever seen as a potential starting point.

!!! Note
    Examples of queries that scan the entire graph are `MATCH (n) RETURN n LIMIT 20` or `MATCH (n) RETURN count(*)`. These queries are tempting to use for exploration on traditional databases, but are not efficient to use in streaming analysis.

## Finding Recently Accessed Nodes

Another common query pattern is to pull out some small number of nodes from the graph (for instance to verify that data is being written in the desired graph structure). For those cases, the `recentNode` and `recentNodesIds` procedures are fast and efficient ways to get back a sample of recently modified/queried nodes or their IDs. These procedures both take an argument indicating the desired number of elements to sample.

```cypher
// Get 20 sample nodes:
CALL recentNodes(20)
// or with YIELD if you need to use each result in larger queries:
CALL recentNodes(20) YIELD node RETURN node
```

This approach can be used to anchor larger queries too, like the paternal grandparents query from above. This time, we don't need to know the ID of any of the nodes in the pattern -- we just need to constrain at least one of them using the `recentNodesIds`function.

```cypher
// Sample the recent part of the graph looking for people and paternal grandparents
CALL recentNodesIds(1000) YIELD nodeId AS personId
MATCH
  (person :Person)-[:HAS_FATHER]->(dad :Person),
  (grandpa :Person)<-[:HAS_FATHER]-(dad)-[:HAS_MOTHER]->(grandma :Person)
WHERE id(person) = idFrom('person', person.name)
RETURN
  person.name AS person,
  grandpa.name AS paternalGrandfather,
  grandma.name AS maternalGrandmother
LIMIT 20
```

This query will inspect the set of 1000 nodes that have most recently streamed in, and it will run the query on those IDs and return results if the larger structure in the `MATCH` clause matches the data found at the IDs returned from `recentNodesIds`.

The same approach of sampling recently touched nodes can be used to quickly compute some aggregate statistics over recent data. For instance, here is a query for quickly sampling the distributions of labels in recently created or accessed data:

```cypher
// Count the number of each type of node label for 1000 recently accessed nodes
CALL recentNodes(1000) YIELD node
RETURN labels(node), count(*)
```


---
# Streaming Graph vs. Graph Database
URL: https://quine.io/core-concepts/streaming-graph-vs-database/

# Streaming Graph vs. Graph Database

## Similarities

Quine is a streaming graph—and that's a brand new thing! But it's a good place to begin understand streaming graphs from the perspective of a graph database. They have a lot in common; they both store data and represent it as a native graph. They both get to that data through [query languages](./supported-query-languages.md). They can both be [plugged into other tools](./streaming-systems.md) to build a pipeline for data processing workflows.

## Differences

### Counting Is Hard

When learning a new database, the first thing most engineers would do is load in a couple rows of data and then query the system to count how many items were loaded. This is a common sanity check to make sure a really simple operation behaves as expected, then you build you understanding from there, knowing that it's working as you expected.

If you try to count nodes in Quine, you might be disappointed because the simple operation of counting can actually be quite expensive. Databases usually keep a running count of how many rows they manage. Every time a new row is added, that count is incremented by one. Remove a row and 1 is subtracted. For the sake of high-volume streaming performance, Quine works differently both with how nodes are created (see the next section) and how a node contributes to the count of "all nodes". 

### All Nodes Exist

A normal database starts empty and gets filled with data over time. When you start Quine for the first time, every node that __could__ exist __does__ exist! What, what? Yep. As a design principle, Quine never "creates" a node. Instead you simply address a node and start using it. This turns out to be a critical design principle for building a graph from streaming data. If data from two separate streams each refer to the same node:

- In a normal database you'd first have to check if the node exists (a read query), create it it doesn't exist (a write query) and then continue with your operation (another read, write, both, or more). That needs to be done for every single item arriving per stream! That turns one operation into three, and it interleaves reading and writing which harms lots of optimization techniques. These are real-world performance destroyers which most benchmarks just skip over.
- In Quine, you never create a node. The node you want is already there and waiting for you to use it. So if two streams reference the same node, they can both simply start using that node and avoid all the extra bookkeeping and error handling that normal databases would have to do. 

This is a design principle; it does not require allocating all possible space. "All nodes exist"… by convention. 

### IDs Over Indices

In order to find starting points for queries, databases usually maintain indices (a.k.a. indexes). An index is a duplicate copy of some of the data which is organized in a different way to make finding specific values faster/easier. It comes at the cost of having to maintain that duplicate copy (the index) every time another value changes. That's why indices degrade performance. A smart database admin/user will carefully limit which values are indexed. The more that values change and the more values there are, the greater the performance cost.

Quine is built to operate on infinite streams of data. Literally. You can't realistically build an index over an infinite set of data. So how would Quine find starting points for queries? In a graph, every node could be a starting point, and the edge connections that make up the graph are the perfect way to navigate through the data. But how would you get started? 

Quine is built so that addressing a node by its ID is an extremely fast and efficient operation. If you can find a node by identifying its ID, then every other part of the query can execute efficiently. So Quine includes a special query function called `idFrom(…)` to determine node IDs based on values—exactly the kind of values that would get indexed. But instead of paying the maintenance cost of indexing those values, the `idFrom(…)` function efficiently turns a set of values into a consistent node ID. It's similar to a consistent hash, but it produces a consistent node ID — even if you're using custom ID types or any of the other types of IDs that Quine ships with.

The key takeaway for understanding this difference is: when running ad hoc or ingest queries in Quine, you should always include at least one call to `idFrom(…)` to efficiently find a starting point for your query. Without a call to `idFrom(…)` Quine will have to scan all nodes to find starting points for your query—which will be very slow!

!!! tip
    If you're writing a [Standing Query](../learn/standing-queries/standing-queries.md), you do not need to use `idFrom(…)` because Standing Queries are an efficient way of scanning all nodes…

### Standing Queries

The most unusual and most powerful capability in Quine is the ability to execute a graph query as a [Standing Query](../learn/standing-queries/standing-queries.md). Standing Queries are just like normal ad hoc queries, except instead of running once and terminating when all results are found, a Standing Query continues to run forever or until its cancelled. A Standing Query efficiently monitors the entire graph, continually watching each change to determine if it produces a new result to the query. Standing Queries mean that you can subscribe to a stream of future results. Whenever the data changes in a way that produces a new result, the Standing Query output handler is triggered and a new result is produced.

### Save History, Not State

Databases save the state of each item to disk. That's their job. To lean in to the world of infinite streaming and continuous updates, Quine takes a different approach. Quine doesn't save the state of each node in the graph it manages; Quine saves the history of changes to each node. It's a technique known as [event sourcing](https://martinfowler.com/eaaDev/EventSourcing.html), and its used widely in stream processing tools, but not so often with databases. Some databases produce a [change data capture (CDC)](https://en.wikipedia.org/wiki/Change_data_capture) stream as a secondary feature compared to their primary state storage job. Quine essentially inverts this picture. The log of changes is the primary data stored, and saving the rolled up materialized state of those changes is just an optimization.

!!! note
    This is customizable in Quine. It's possible to disable saving journals (the append-only log of changes) and instead to save only the materialized state.

### Read / Write Performance

Databases are usually optimized for either read-heavy workloads __OR__ write-heavy workloads. Quine is designed to solve for both. How is that possible? Well, because Quine stores data as an append-only log, it's very fast to save new updates. Those updates have to be computed somewhere before they're written, so that is done in the graph, in memory. The graph-shaped representation of data is only held in memory, and each node is its own independent process. When new data comes in, each node computes what changes are needed and saves those deltas to disk asynchronously and with a timestamp. When a query arrives to read data, that query only needs to work with the in-memory state of the node which ends up behaving like a cache. So read operations (which require the complex structure of the graph) don't usually need to touch the disk. The only time a read causes a read from disk is when a node is expired out of memory and needs to be loaded back in to serve a query. This causes the log to be read (sped up by a snapshot if one is available) and the node to become cached in memory and available for other operations without touching disk until it is expired from the cache. This approach lets Quine use both the streaming activity and the graph structure itself to make smart choices about which nodes are worth keeping in memory for longer.

### The Persistor Layer

Underneath the in-memory graph there is a layer of Quine known as the [persistor](../learn/persistors/index.md). This is where data is __actually__ stored to disk. And this is where we finally get to part of the Quine that is exactly like a database—because it __IS__ a database. Which database? Well, you can choose. Quine supports many types of databases for the persistence layer and we add new ones all the time. You can even add your own!

As a general rule of thumb, the persistence layer should be at least a key-value store. Key-value stores are usually the fastest option at the expense of the ability to do complex queries. Well Quine handles the complex query in memory as mentioned above, so we don't need the database to do anything complex; just append values quickly and occasionally return a range of values in a chunk. Quine inherits much of the functionality of the underlying chosen database, and latency of these operations is usually the most significant factor. So scalable key-value stores (or column stores used simply) are usually the best choice. Quine has integrations with for [Cassandra](https://cassandra.apache.org/doc/latest/) and [ScyllaDB](https://docs.scylladb.com/stable/) for high-volume networked use cases, but also ships with [MapDB](https://mapdb.org/) for in-process data storage (including in-memory only as an option) and [RocksDB](https://rocksdb.org/) as the default persistor for persisting to disk on the same machine running Quine.


---
# Architecture
URL: https://quine.io/core-concepts/architecture/

# Architecture

![Quine Architecture](./core-concepts-images/abstractQuine.png)

## Data Ingest: Event-Driven Data

Data enters Quine primarily through streaming data sources like Kafka, Kineses, or even POSIX named pipes. These data sources are effectively infinite, and Quine works with them as if they will never end. Other types of data sources as supported as well, like ordinary files in CSV, JSON, or other formats. Quine calls this connection an **ingest stream**.

Each ingest stream connection performs four primary steps:

1. **Consume a stream of bytes** - e.g. Open local file on disk, or connect to a Kafka topic.
2. **Delimit into a sequence of finite byte arrays** - e.g. Use newlines to separate individual lines from a file. Kafka provides delimiting records by its design.
3. **Parse byte array into an object** - e.g. Parse as a string into JSON, or use a provided [protobuf](https://developers.google.com/protocol-buffers) schema to deserialize each object.
4. **Ingest query constructs the graph** - e.g. Provide the parsed object as `$that` to a user-defined Cypher query which creates any graph structure desired.

When a new ingest stream is configured, Quine will connect to the source and follow the steps described above to use the incoming data stream to build the internal graph.

Not all ingest configurations require distinct choices for each step. For instance, a file ingest can define its source as a CSV (Comma Separated Values) file, and the line-delimiting and field parsing are done automatically.

Quine ingest streams are backpressured. When Quine is busy with intensive tasks, or possibly waiting for the durable storage to finish processing, Quine will slow down the ingest stream so that it does not overwhelm other components.

Backpressured ingest streams ensure maximum throughput while preserving stability of the overall system.

## Asynchronous Graph: Efficient Incremental Computation

The centerpiece of Quine is the graph used to represent the internal data structure and perform computation on it.

Quine's unique design combines the graph data model of a property graph with a graph computational model implemented with actors. Each node is backed by actors as needed, allowing the node to send and receive messages, and to perform arbitrary computation to handle them.

Actors representing nodes are managed automatically by Quine. They are loaded on-demand and reclaimed when no longer needed. When live, an actor represents the state of a node at a particular moment in time. That moment might be the thoroughgoing present moment representing the continually changing graph, or it could be a historical moment with the node participating in resolving a historical query.

Historical queries allow Quine to very easily answer queries about what the data *used to be* at any moment in the past. For the developer, this is simple a matter of including a timestamp in their query. Under the hood, node actors in Quine implement an event-sourcing strategy where changes to any node are saved as small deltas to the durable storage layer. When needed, these deltas are replayed in order to restore a node to its state at any moment in history.

## Durable Persistent Storage: All Data, No Time Windows

Quine saves data to disk using one or more "[persistor](https://english.stackexchange.com/a/206980/120983)." A [persistor](../learn/persistors/index.md) is the interface from the live graph to the underlying durable data storage.

As nodes compute which changes need to be saved, those deltas are delivered to the persistor to save to disk. When a node is needed to participate in a query or other computation, its log of changes is loaded from disk and delivered to the node to replay itself up to the relevant moment.

The fundamental data structure provided by the persistor is essentially a key-value store. Quine supports many types of data storage, but they divide broadly into two categories: local vs. remote persistors.

* Local persistors save data on the same machine on which the Quine graph is running.
* Remote persistors save data on an external system across a network.

Quine uses a local instance of RocksDB as the default persistor. Production deployments typically use a remote persistor like Cassandra to achieve high availability, data redundancy, and horizontal scalability.

## Standing Query Output: Data-Driven Events

Standing queries persist at all times in the graph and propagate efficiently and automatically. They match any pattern you can describe in a Cypher query. When a standing query completes a new result, it collects any desired data and streams out to the next step immediately.

Standing query outputs are meant to be flexible to fit into existing workflows. A few examples of standing query outputs include:

* Print a message with event data to the console or stdout
* Publish records to a queue or streaming system like Kafka
* Make calls back in to the graph to fetch additional data
* Make calls back in to the graph to update other values or execute any query (very powerful!)

Some examples of how you might use standing queries in your workflow:

* Stream results out to the next system in a data pipeline
* Produce a dataset of cleaned-up data to feed machine learning pipelines
* Maintain indices
* Produce alerts for misconfigured systems
* Detect patterns of suspicious activity


---
# Standing queries
URL: https://quine.io/learn/standing-queries/standing-queries/

# Standing Queries

!!! info "API v2 is now the default"
    The v2 API is the default for all standing query operations. API v1 remains available but is planned for deprecation and will be removed in a future release. See [Migrating from API v1](../../reference/upgrade/migrating-from-api-v1.md) for migration guidance.

A standing query is a feature unique in Quine to incrementally `MATCH` a subgraph as new data enters the graph. Then, when a full subgraph matches, the result is further processed to modify the graph, produce output as a data source, or take additional action.

## Standing Query Structure

Standing queries have two parts: a **pattern** query and **outputs** destinations. The pattern query defines the structure of what we're looking for, and output destinations specify actions for each result produced.

Consider the pattern query as a coarse filter that is specific enough to `MATCH` a model event but not so specific that it would match a unique event. A query in an output destination can process the result further with a more expressive Cypher query to then update the graph, act as an event source, generate metrics, and much more.

Cypher within a pattern query must only contain a `MATCH` and a `RETURN`, with an optional `WHERE`, while Cypher within output destinations is unconstrained, allowing for more expressive queries.

![Standing Query Model](standingQuery.png)

As described in the [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query) API documentation, a standing query is created via POST to the `/api/v2/graph/quine/standingQueries` endpoint. Standing queries can also be created from the [Streams](../../getting-started/streams.md) page in the UI.

```json title="POST /api/v2/graph/quine/standingQueries"
{
  "name": "STANDING-1",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "enrich-and-print",
      "resultEnrichment": {
        "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN n.line",
        "parallelism": 16
      },
      "destinations": [
        { "type": "StandardOut" }
      ]
    }
  ]
}
```

Or in YAML if you are writing a recipe.

``` yaml
standingQueries:
  - name: STANDING-1
    pattern:
      type: Cypher
      query: MATCH (n) RETURN DISTINCT id(n) AS id
    outputs:
      - name: enrich-and-print
        resultEnrichment:
          query: >-
            MATCH (n)
            WHERE id(n) = $that.data.id
            RETURN n.line
          parallelism: 16
        destinations:
          - type: StandardOut
```

This structure ensures that the set of positive matches, minus the set of negative matches (also referred to as **cancellations**) produced by a standing query are like the results produced if the same Cypher query had been issued in a batch fashion after all data has been written into the graph.

### Propagation to Existing Data

A standing query naturally matches data that changes after it is registered. **Propagation** additionally evaluates registered standing queries against data already in the graph. The create endpoint accepts a `propagateTo` query parameter controlling this at creation time:

| `propagateTo` | Behavior |
|:--------------|:---------|
| `NONE` | No propagation. The query matches only data changed after registration. |
| `EXCLUDE_SLEEPING` (default) | Propagates to nodes currently in the in-memory cache. Relatively inexpensive. |
| `INCLUDE_SLEEPING` | Propagates to all nodes, waking sleeping nodes from the persistor. Significantly more expensive, since it requires disk and network IO to iterate through and wake sleeping nodes in the background. The `wakeUpParallelism` parameter (default 4) controls how many nodes are woken at a time; higher values iterate faster but backpressure ingest more. |

Propagation applies **all** currently-registered standing queries, not only the newly-created one, and runs in the background after the create request returns. When creating several standing queries at once, create them all with `propagateTo=NONE` and then call [Propagate Standing Queries: `POST /api/v2/graph/quine/standingQueries:propagate`](/reference/rest-api/?av=v2#/operations/propagate-standing-queries) once; propagating for multiple standing queries costs essentially the same as propagating for one. The standalone endpoint takes `includeSleeping` (default `false`) and `wakeUpParallelism` parameters.

## Pattern Match Query

The pattern query in a standing query is a declarative graph pattern expressed using a subset of the Cypher query language containing a `MATCH` and a `RETURN`, with an optional `WHERE`.

### Distinct ID Pattern Queries

!!! note "v2 API wire format"
    In v2 API calls and recipes, use `DISTINCT_ID` and `MULTIPLE_VALUES` (SCREAMING_SNAKE_CASE). The PascalCase forms `DistinctId` and `MultipleValues` are the v1 wire format. See [Migrating from API v1](../../reference/upgrade/migrating-from-api-v1.md#enum-wire-format) for details.

Quine has two modes available for writing pattern queries, Distinct ID and Multiple Values. The mode is set to Distinct ID by default unless you explicitly set the mode to Multiple Values within your pattern query. Quine can process standing queries that contain either mode within the same runtime or recipe.

The following constraints apply to Cypher contained in the pattern `query` string when `mode` is set to the default `DISTINCT_ID`:

  * Each node identified by the `MATCH` shall have the following:
  * Node variable name
  * Label (optional but not more than one)
  * Optional map of literal property values to match
  * Nodes in the `MATCH` must form a [connected graph](https://en.wikipedia.org/wiki/Connectivity_(graph_theory)).
  * Nodes in the `MATCH` must **not** contain any cycles. In other words, the pattern must be either linear or tree-shaped.
  * Only node variables can be bound in the query `MATCH`. Edges **cannot** be aliased to a variable, and path expressions cannot be used (so `-[:HAS_FATHER]->` is fine, but `-[e:HAS_FATHER]->` is not).
  * Edges in the `MATCH` must be directed, have exactly one edge label, and **cannot be variable-length**.
  * Constraints inside the `WHERE` clause must be `AND`-ed together and of one of the following forms:

    * `nodeName.property = 1` - the property has the literal value on the right
    * `nodeName.property <> 1` - the property must exist but be different than the literal value on the right
    * `nodeName.property IS NOT NULL` - the property must exist
    * `nodeName.property IS NULL` - the property must not exist
    * `nodeName.property =~ "regex"` - the property must be a string matching the regex
    * `id(nodeName) = 12` - the ID of the node must be exactly the literal value on the right
    * `id(nodeName) = idFrom('values', 'to', 'hash')` - the ID of the node must match exactly the `idFrom()` computed from the literal values on the right

  * Exactly **one** value must be returned, and it must be either the `DISTINCT` `id` or `strId` of a node bound in the `MATCH`.
  
For example, `RETURN DISTINCT strId(n)` or `RETURN DISTINCT id(n) as nId` are OK, but not `RETURN n.name` or `RETURN id(n) AS nId`. The node whose id is returned is the root node - the location in the graph from which the pattern starts being incrementally matched.

### Multiple Values Pattern Queries (Beta)

??? warning "Beta Feature"

    This feature is in the beta phase of development.

    Pattern queries with Multiple Values mode could:

    * Require syntax changes when the feature releases as GA
    * Use more RAM
    * Consume more disk space
    * Differ in performance from Distinct ID pattern query queries

Multiple Values mode pattern query queries relax some of the constraints imposed by Distinct ID. In particular, the `WHERE` and `RETURN` portions of the query allow Cypher expressions to be much more expressive.

The syntax and structure of this mode is designed to supersede the Distinct ID mode. Thus, any Distinct ID standing query pattern is a valid Multiple Values standing query, though not the other way around.

The Multiple Values mode retains the `MATCH` - `WHERE` - `RETURN` shape from Distinct ID mode with the addition of the constraints below.

* Any number of results (not just one) can be returned in the `RETURN`, including results that aren't node IDs
* Constraints in the `WHERE` clause are reduced
* `DISTINCT` is required for Distinct ID standing queries, but the Multiple Values mode does not support `DISTINCT` return values.
* Use of variables must represent a node
* Variable usage is ok when dereferencing node properties. For example `RETURN n.name` is ok but `RETURN n` is not.
* `WHERE` and `RETURN` support `id(n)` and `strId(n)` but other functions are not supported.

The `MATCH` portion of standing queries using the Multiple Values mode removes the syntactic requirements for running in Distinct ID mode with two exceptions:

* Multiple IDs and property values from matched nodes can be returned by `RETURN`. For example, `RETURN n.age + * strId(n) + " " + m.name` is fine, but `RETURN properties(n)` is not.

* Constraints in the `WHERE` clause must be defined in the IDs and properties of matched nodes and can not include sub-queries or procedures.
* Can not `MATCH` variable length patterns
* `MATCH` does not support pattern expressions

Since there isn't exactly one ID being returned, the root of the standing query pattern (the place in the pattern from which incremental matching starts) is instead set to be the first node in the `MATCH` pattern. This makes it possible to make any node in the pattern the "root".

### Pattern Match Results

Both modes for the pattern query return a `StandingQueryResult` JSON object with `meta` and `data` sub-objects.

The `meta` JSON sub-object consists of the following:

* `isPositiveMatch`: whether the result is a new match. When this value is false, it signifies that a previously matched result no longer matches

* `resultId`: a UUID generated for each result. This is useful if you wish to track a result in some external system since the `resultId` of the result with `isPositiveMatch = false` will match the `resultId` of the original result (when `isPositiveMatch = true`).

The `data` JSON sub-object consists of the following:

* On a positive match, the `data` JSON object contains results returned by the pattern query.
* This objects keys are the names of the values returned (ex: `RETURN DISTINCT strId(n)` would have key `"strId(n)"` and `RETURN DISTINCT id(n) AS theId` would have key `"theId"`).
* Each query data returned is analogous to a row returned from a regular Cypher query - the key names match what would normally be Cypher column names.

When `DISTINCT_ID` `mode` is set, a result is emitted when a complete pattern matches or stops matching, but additional results won't be emitted if there are interim new complete pattern matches.

???+ abstract "Example of single-result per root semantics"

    Consider the following query for watching friends.

    ```cypher
    // Find people with friends
    MATCH (n:Person)-[:friend]->(m:Person)
    RETURN DISTINCT strId(n)
    ```

    If we start by creating disconnected "Peter", "John", and "James" nodes, there will be no matches.

    ```cypher
    CREATE (:Person { name: "Peter" }),
          (:Person { name: "John" }),
          (:Person { name: "James" })
    ```

    Then, if we add a "friend" edge from "Peter" to "John", "Peter" will trigger a new standing query match.

    ```cypher
    MATCH (peter:Person { name: "Peter" }), (john:Person { name: "John" })
    CREATE (peter)-[:friend]->(john)
    ```

    However, adding a second "friend" edge from "Peter" to "James", "Peter" will not trigger a new match since he is already matching (that is, the "Peter" node is not distinct).

    ```cypher
    MATCH (peter:Person { name: "Peter" }), (james:Person { name: "James" })
    CREATE (peter)-[:friend]->(james)
    ```

    **Note**, unlike Distinct ID mode queries, Multiple Values mode pattern query results can be emitted from each root node. This means that the "Find people with friends" example, if run in the Multiple Values mode, would produce two results (one for each friend) unlike the single result produced in the Distinct ID mode.

Sample `StandingQueryResult`:

```json
{
    "meta": {
        "resultId": "b3c35fa4-2515-442c-8a6a-35a3cb0caf6b",
        "isPositiveMatch": true
    },
    "data": {
        "strId(n)": "a0f93a88-ecc8-4bd5-b9ba-faa6e9c5f95d"
    }
}
```

## Result Outputs

Once a full pattern match occurs, a `StandingQueryResult` is produced. A standing query can have any number of output destinations to route `StandingQueryResults`. The output destinations are processed in parallel.

### Output Workflows

Quine provides workflow-based output processing with composable multi-stage pipelines. Results flow through these optional stages in order:

1. **[Filtering](#filtering)** — Route only positive matches, filtering out cancellations
2. **[Transformation](#transformation)** — Reshape results with `InlineData` before output
3. **[Enrichment](#enrichment)** — Augment results with additional data via Cypher queries
4. **[Destinations](#output-destinations)** — Route to one or more output destinations (Kafka, webhooks, etc.)

Each stage is optional except destinations—you must configure at least one output destination.

Results can be routed to any of the following destinations, set with the `type` field of an output.

| Name | Configuration value | Formats | Description |
|:---|:---|:---|:---|
| [Broadcast to Reactive Stream](#publish-to-reactive-stream) | `ReactiveStream` | JSON, Protobuf | Broadcasts data to a created Reactive Stream. Other thatDot products can subscribe to Reactive Streams. Reactive Stream outputs are only supported in standalone (single-host) deployments. |
| [Drop](#drop) | `Drop` | — | Effectively no destination at all, this does nothing but forget the data sent to it. |
| [Log JSON to Console](#log-json-to-standard-out) | `StandardOut` | JSON | Prints each result as a single-line JSON object to stdout on the application server. |
| [POST to HTTP[S] Webhook](#post-to-webhook) | `HttpEndpoint` | JSON | Makes an HTTP[S] POST for each result. For the format of the result, see "Standing Query Result Output". |
| [Publish to Kafka Topic](#publish-to-kafka-topic) | `Kafka` | JSON, Protobuf | Publishes provided data to the specified Apache Kafka topic. |
| [Publish to Kinesis Data Stream](#publish-to-kinesis-stream) | `Kinesis` | JSON, Protobuf | Publishes provided data to the specified Amazon Kinesis stream. |
| [Publish to Slack Webhook](#publish-to-slack) | `Slack` | Slack message | Sends a message to Slack via a configured webhook URL. See [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks). |
| [Publish to SNS Topic](#publish-to-sns-topic) | `SNS` | JSON, Protobuf | Publishes an AWS SNS record to the provided topic. To guarantee delivery, writes that fail are retried indefinitely, so confirm the credentials and topic ARN before starting this output. An unfixable error (e.g., an invalid topic ARN or missing credentials) will retry forever without emitting results, which may stop the Standing Query this output is attached to. |
| [Run Cypher Query](#cypher-query) | `CypherQuery` | — | Runs the `query`, where the given `parameter` is used to reference the data that is passed in. Runs at most `parallelism` queries simultaneously. |
| [Write JSON to File](#log-json-to-a-file) | `File` | JSON | Writes each result as a single-line JSON record. For the format of the result, see "Standing Query Result Output". |

#### Filtering

Filter results before processing with predicates. Use `OnlyPositiveMatch` to route only positive matches (when the pattern first matches), filtering out cancellations:

```json
{
  "filter": {
    "type": "OnlyPositiveMatch"
  }
}
```

Standing query results include an `isPositiveMatch` metadata flag:

* `true` - Pattern newly matched (positive match)
* `false` - Pattern no longer matches (cancellation)

#### Transformation

Transform result structure before enrichment. Use `InlineData` to unwrap the result data from its metadata envelope:

```json
{
  "preEnrichmentTransformation": {
    "type": "InlineData"
  }
}
```

**Before transformation:**
```json
{
  "meta": { "isPositiveMatch": true },
  "data": { "id": "abc123", "severity": "high" }
}
```

**After transformation:**
```json
{ "id": "abc123", "severity": "high" }
```

#### Enrichment

Execute Cypher queries to add context to results before routing:

```json
{
  "resultEnrichment": {
    "query": "MATCH (n)-[:RELATED_TO]->(m) WHERE id(n) = $that.id RETURN m.name AS related",
    "parallelism": 16,
    "allowAllNodeScan": false,
    "shouldRetry": true
  }
}
```

| Setting            | Description                                         | Default |
|:-------------------|:----------------------------------------------------|:--------|
| `query`            | Cypher query with `$that` parameter for result data | —       |
| `parallelism`      | Concurrent query executions                         | 32      |
| `allowAllNodeScan` | Permit queries that scan all nodes                  | `false` |
| `shouldRetry`      | Retry failed queries on recoverable errors          | `true`  |

When referencing node IDs from standing query results in enrichment queries, use the `quineId()` function to convert the ID value:

```cypher
MATCH (n) WHERE id(n) = quineId($that.data.id) RETURN n.details
```

!!! warning "Non-Deterministic Ordering"
    When `parallelism` is greater than 1, enrichment queries execute concurrently. This means results may arrive at destinations in a different order than they were produced by the standing query. If ordering matters for your use case, set `parallelism: 1`.

!!! note "Idempotency"
    If your enrichment query is not idempotent and `shouldRetry` is `true`, effects may occur multiple times on transient failures.

#### Output Structure

Control how result data is wrapped for destinations.

**Bare** - Send raw result data without metadata:

```json
{
  "structure": { "type": "Bare" }
}
```

**WithMetadata** - Wrap data with metadata including match status:

```json
{
  "structure": { "type": "WithMetadata" }
}
```

#### Output Formats

Destinations that support formats can serialize as JSON or Protobuf.

**JSON (Default):**
```json
{
  "format": { "type": "JSON" }
}
```

**Protobuf:**
```json
{
  "format": {
    "type": "Protobuf",
    "schemaUrl": "http://schema-registry:8081/schemas/ids/1",
    "typeName": "com.example.ResultMessage"
  }
}
```

### Output Destinations

#### Cypher Query

The Cypher query destination is particularly powerful, making it possible to post-process pattern query results to collect more information from the graph or to filter out matches that don't meet some requirements.

The result object is passed to the Cypher query via the parameter `$that`, for use in the `query` Cypher.

Be aware that non-trivial or long-running operations with results will consume system resources and cause the system to backpressure and slow down other processing (like data ingest).

#### Drop

Drop the current result output and end processing the destination.

#### POST to Webhook

Makes an HTTP[S] POST for each result. The data in the request payload can be customized in a Cypher query preceding this step.

#### Publish to Slack

Sends a message to Slack via a configured Slack App webhook URL. See [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks).

| Setting                 | Description                                     | Default |
|:------------------------|:------------------------------------------------|:--------|
| `hookUrl`               | Slack webhook URL                               | —       |
| `onlyPositiveMatchData` | Only send positive matches (skip cancellations) | `false` |
| `intervalSeconds`       | Minimum seconds between messages                | 20      |

Slack limits the rate of messages which can be posted (1 message per second). Quine batches results that arrive faster than the configured `intervalSeconds` and publishes them as a single aggregated message when the interval allows.

#### Log JSON to Standard Out

Prints each result as a single-line JSON object to standard output on the Quine server.

This output type can be configured with `Complete` to print a line for every result, backpressuring and slowing down the stream as needed to print every result.

Or it can be configured with `FastSampling` to log results in a best effort, by dropping some results to avoid slowing down the stream.

Note that neither option changes the behavior of other outputs registered on the same standing query.

#### Log JSON to a File

Write each result as a single-line JSON object to a file on the local filesystem.

#### Publish to Kafka Topic

Publishes a record for each result to the provided Apache Kafka topic. Records can be serialized as JSON or Protocol Buffers before being published to Kafka.

| Setting                  | Description                                        | Default |
|:-------------------------|:---------------------------------------------------|:--------|
| `topic`                  | Kafka topic name                                   | —       |
| `bootstrapServers`       | Kafka bootstrap servers                            | —       |
| `format`                 | Output format (JSON or Protobuf)                   | JSON    |
| `kafkaProperties`        | Additional Kafka producer configuration            | None    |
| `sslKeystorePassword`    | Password for SSL keystore (secret)                 | None    |
| `sslTruststorePassword`  | Password for SSL truststore (secret)               | None    |
| `sslKeyPassword`         | Password for private key in keystore (secret)      | None    |
| `saslJaasConfig`         | SASL authentication configuration (secret)         | None    |

!!! note "Credential Redaction"
    For security, `sslKeystorePassword`, `sslTruststorePassword`, `sslKeyPassword`, and `saslJaasConfig` credentials are automatically redacted in API responses, displaying as `Secret(****)`. See [Kafka Ingest](../ingest-sources/kafka.md#secure-kafka-configuration) for details on SASL authentication types.

#### Publish to Kinesis Stream

Publishes a record for each result to the provided Kinesis stream. Records can be serialized as JSON or Protocol Buffers before being published to Kinesis.

| Setting                      | Description                      | Default |
|:-----------------------------|:---------------------------------|:--------|
| `streamName`                 | Kinesis stream name              | —       |
| `credentials`                | AWS credentials (optional)       | None    |
| `region`                     | AWS region (optional)            | None    |
| `format`                     | Output format (JSON or Protobuf) | JSON    |
| `kinesisParallelism`         | Concurrent publish operations    | None    |
| `kinesisMaxBatchSize`        | Maximum records per batch        | None    |
| `kinesisMaxRecordsPerSecond` | Rate limit (records/second)      | None    |
| `kinesisMaxBytesPerSecond`   | Rate limit (bytes/second)        | None    |

!!! note "Credential Redaction"
    For security, `accessKeyId` and `secretAccessKey` values are automatically redacted in API responses, displaying as `Secret(****)`. See [Kinesis Ingest](../ingest-sources/kinesis.md#aws-credentials) for details.

#### Publish to SNS Topic

Publishes an AWS SNS record to the provided topic containing JSON for each result.

| Setting       | Description                      | Default |
|:--------------|:---------------------------------|:--------|
| `topic`       | SNS topic ARN                    | —       |
| `credentials` | AWS credentials (optional)       | None    |
| `region`      | AWS region (optional)            | None    |
| `format`      | Output format (JSON or Protobuf) | JSON    |

!!! note "Credential Redaction"
    For security, `accessKeyId` and `secretAccessKey` values are automatically redacted in API responses, displaying as `Secret(****)`. See [SQS/SNS](../ingest-sources/sqs---sns.md) for details.

!!! warning "Credential Validation"
    Ensure your credentials and topic ARN are correct. If writing to SNS fails, the write will be retried indefinitely. If the error is not fixable (e.g., the topic or credentials cannot be found), the outputs will never be emitted and the output could stop running.

#### Publish to Reactive Stream

Broadcasts results to a TCP-based reactive stream endpoint. Clients can connect to receive a continuous stream of results.

| Setting   | Description                        | Default     |
|:----------|:-----------------------------------|:------------|
| `address` | Address to bind the stream server  | `localhost` |
| `port`    | Port to bind the stream server     | —           |
| `format`  | Output format (JSON or Protobuf)   | JSON        |

!!! warning "Cluster Limitation"
    Reactive Stream outputs do not function correctly when running in a cluster. Use Kafka or Kinesis for clustered deployments.

## Inspecting and Debugging Standing Queries

For detailed guidance on debugging standing queries, see the [Troubleshooting Queries](../troubleshooting/index.md) guide, which covers:

* [EXPLAIN](../troubleshooting/query-execution-plans.md) to understand query execution plans
* [standing.wiretap()](../troubleshooting/queries.md#standingwiretap) and the [Results API Endpoint](../troubleshooting/queries.md#results-api-endpoint) for streaming live results
* [Common failure patterns](../troubleshooting/queries.md#standing-query-not-matching) specific to standing queries

Since standing queries use a subset of Cypher syntax, you can run the match pattern as a regular query to understand what data would match. When doing so, constrain the starting points if there is already a large amount of data in the system (see [Using IDs in a Query](../../core-concepts/id-provider.md))


---
# Ingest Streams
URL: https://quine.io/learn/ingest-sources/

# Ingest Streams

!!! info "API v2 is now the default"
    The v2 API is the default for all ingest operations. API v1 remains available but is planned for deprecation and will be removed in a future release. See [Migrating from API v1](../../reference/upgrade/migrating-from-api-v1.md) for migration guidance.

## Overview

An **ingest stream** connects a potentially infinite stream of incoming events to Quine and prepares the data for the streaming graph. Within the ingest stream, an ingest query, written in Cypher, updates the streaming graph nodes and edges as data is received.

Working with data in Quine is a two-step process:

1. Load a stream of data into the graph with an **ingest stream**: *event-driven data*
2. Monitor the graph for results and act with a **standing query**: *data-driven events*

All data sources require an "ingest stream". The ingest stream is the first opportunity we have to affect the data. In its most basic form, an ingest stream maps JSON to Quine nodes directly. The following query creates a new Quine node and applies all of the properties from the incoming JSON payload. It then adds an "Event" label to help with organization.

```cypher
MATCH (n) WHERE id(n) = idFrom($that) SET n.line = $that
```

For guidance on designing graph structures and ingest queries, see [Data Modeling and Query Design](../../core-concepts/data-modeling.md).

!!! Hint
    Select nodes, don't search for nodes. If your ingest query involves traversing nodes not by edges but e.g. a `WHERE` condition filtering on nodes matching a specific property value, the execution of this involves scanning all nodes. If we detect that the query provided may entail this behavior, we will log a warning:
    ```
    Cypher query may contain full node scan; for improved performance, re-write without full node scan."
    ```

    Quine adds an `idFrom` function to Cypher that takes any number of arguments and deterministically produces a node ID from that data.

    This is similar to a consistent-hashing approach where a collection of values are hashed together to produce a unique result that can be used for an ID.

    Quine supports many different kinds of IDs (numbers, UUIDs, strings, tuples of values, and more…), `idFrom` produces consistent results appropriate for the dataset regardless of the which ID type is used.

Quine parses JSON data into a graph structure according to the following assumptions:

* Each JSON object is treated as a node in the graph.
* Nested objects are treated as separate nodes. In this case, the JSON field name is treated as the name of the outgoing edge.
* The field id is the default field defining the ID of the node. The name of this field is customizable and can be set with the string config setting at quine.json.id-field.
* The ID computed for each object must be a string that can be parsed by the IdProviderType set in the configuration. IDs fields in JSON which do not have the proper type will result in an error returned from the API.
* Objects without any ID will be assigned a random ID. Duplicate identical objects with no ID field may result in multiple separate nodes created—depending on the structure of the data provided.

## Ingesting Event-Driven Data

Data enters Quine from streaming data sources like Kafka, Kinesis, or even POSIX named pipes. These data sources are effectively infinite — Quine works with them as if they will never end. Other types of data sources as supported as well, like ordinary files in CSV, JSON, or other formats.

Each ingest stream performs four primary operations:

1. **Consume a stream of bytes** - e.g. Open local file on disk, or connect to a Kafka topic.
2. **Delimit into a sequence of finite byte arrays** - e.g. Use newlines to separate individual lines from a file. Kafka provides delimiting records by its design.
3. **Parse byte array into an object** - e.g. Parse as a string into JSON, or use a provided [protobuf](https://developers.google.com/protocol-buffers) schema to deserialize each object. Bytes will first be decoded if the ingest has specified **recordDecoders**.
4. **Ingest query constructs the graph** - e.g. Provide the parsed object as `$that` to a user-defined Cypher query which creates any graph structure desired.

When a new ingest stream is configured, Quine will connect to the source and follow the steps described above to use the incoming data stream to update the internal graph. Not all ingest configurations require distinct choices for each step. For instance, a file ingest can define its source as a CSV (Comma Separated Values) file, and the line-delimiting and field parsing are done automatically.

Each ingest stream is backpressured. When Quine is busy with intensive tasks downstream, or possibly waiting for the durable storage to finish processing, Quine will slow down the ingest stream so that it does not overwhelm other components. Backpressured ingest streams ensure maximum throughput while preserving stability of the overall system.

## Data Sources

An ingest stream can receive data from the following data sources. Use the
configuration value in the `type` field of the ingest `source` block.

| Name | Configuration value | Formats | Compression (ZLIB, GZIP, BASE64) | Description |
|:---|:---|:---|:---|:---|
| [File Ingest](files-and-named-pipes/) | `File` | AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An active stream of data being ingested from a file on this Quine host. |
| [Kafka Ingest Stream](kafka/) | `Kafka` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kafka. |
| [Kinesis Data Stream](kinesis/) | `Kinesis` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis. |
| [Kinesis Data Stream Using Kcl lib](kinesis/) | `KinesisKCL` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis |
| Number Iterator Ingest | `NumberIterator` | — |  | An infinite ingest stream which requires no data source and just produces new sequential numbers every time the stream is (re)started. The numbers are Java `Long`s` and will wrap at their max value. |
| [Reactive Stream Ingest](reactive-streams/) | `ReactiveStream` | Avro, Drop, Json, Protobuf, Raw |  | A stream of data being ingested from a reactive stream. |
| S3 Ingest | `S3` | AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An ingest stream from a file in S3, newline delimited. This ingest source is experimental and its behavior is subject to change. It's best suited to continuously active streams; durability is not guaranteed once a stream has been inactive for 1 minute or more. |
| Server Sent Events Stream | `ServerSentEvent` | Avro, Drop, Json, Protobuf, Raw | ✓ | A server-issued event stream, as might be handled by the EventSource JavaScript API. Only consumes the `data` portion of an event. |
| [Simple Queue Service Queue](sqs---sns/) | `SQS` | Avro, Drop, Json, Protobuf, Raw | ✓ | An active stream of data being ingested from AWS SQS. |
| [Standard Input Ingest](stdin/) | `StdInput` | AvroContainer, CSV, Json, JsonL, Line, Parquet |  | An active stream of data being ingested from standard input to this Quine process. |
| WebSocket File Upload | `WebSocketFileUpload` | AvroContainer, CSV, Json, JsonL, Line, Parquet |  | Streamed file upload via WebSocket protocol. |
| Websockets Ingest Stream (Simple Startup) | `WebsocketClient` | Avro, Drop, Json, Protobuf, Raw |  | A websocket stream started after a sequence of text messages. |

!!! tip "Need help?"

    See [Troubleshooting Ingest](../troubleshooting/ingest.md) for help with missing data, slow ingests, and other common issues.
  
## Ingest Stream Structure

An ingest stream requires a `name`, `source`, and `query`. The `source` specifies where data comes from and how to parse it, while the `query` is a Cypher statement that processes each record into the graph.

Quine supports multiple types of ingest sources. Each source type has specific configuration options described in the API documentation.

For example, creating an ingest stream via [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) to read data from standard input and store each line as a node:

```json
{
    "name": "standardIn",
    "source": {
        "type": "StdInput",
        "format": "Line",
        "characterEncoding": "UTF-8"
    },
    "query": "MATCH (n) WHERE id(n) = idFrom($that) SET n.line = $that"
}
```

Quine reads from standard input, passing each line into the Cypher query as the parameter `$that`. A unique node ID is generated using `idFrom($that)`. Then, each line is stored as a `line` property associated with a new node in the streaming graph.

When creating an ingest stream via the API, you must provide a unique `name` that identifies the stream. For example, the above ingest stream is named `standardIn` to make it easier to reference in your application.

Alternatively, when creating an ingest stream via a recipe, Quine automatically assigns a name to each stream using the format `INGEST-#` where the first ingest stream defined in the recipe is `INGEST-1` and subsequent ingest streams are named in order with `#` counting up.

Here is the same ingest stream defined in a [Recipe](../../learn/recipe-ref-manual.md):

```yaml
ingestStreams:
  - type: StdInput
    format: Line
    characterEncoding: UTF-8
    query: |-
        MATCH (n)
        WHERE id(n) = idFrom($that)
        SET n.line = $that
```

### Record Decoding

An ingest may specify a list of decoders to support decompression. Decoders are applied in the order they are specified and are applied per-record.

* Decoding is applied in the order specified in the **recordDecoders** array.
* **recordDecoders** are supported for **File**, **S3**, **Kafka**, **Kinesis**, **KinesisKCL**, **ServerSentEvents**, and **SQS** sources.
* Decoding types currently supported are **Base64**, **Gzip**, and **Zlib**.
* The **recordDecoders** member is optional.

The following ingest stream specifies that each record is Gzipped, then Base64 encoded:

```json
{
  "name": "kinesis-ingest",
  "source": {
    "type": "Kinesis",
    "format": "Json",
    "streamName": "my-stream",
    "recordDecoders": ["Base64", "Gzip"]
  },
  "query": "CREATE ($that)"
}
```

## Error Handling

Quine provides granular control over error handling at both the record and stream level.

### Record-Level Errors

Configure `onRecordError` to control behavior when individual records fail:

```json
{
  "onRecordError": {
    "retrySettings": {
      "minBackoff": 2000,
      "maxBackoff": 20,
      "randomFactor": 0.2,
      "maxRetries": 6
    },
    "logRecord": true,
    "deadLetterQueue": { ... }
  }
}
```

| Setting           | Description                                    | Default |
|:------------------|:-----------------------------------------------|:--------|
| `retrySettings`   | Retry failed records with configurable backoff | None    |
| `logRecord`       | Log failed records to application logs         | `true`  |
| `deadLetterQueue` | Route failures to a dead letter queue          | None    |

#### Retry Settings

| Setting        | Type   | Default | Description                               |
|:---------------|:-------|:--------|:------------------------------------------|
| `minBackoff`   | int    | 2000    | Minimum backoff between retries (ms)      |
| `maxBackoff`   | int    | 20      | Maximum backoff between retries (seconds) |
| `randomFactor` | double | 0.2     | Jitter factor for backoff (0.0 - 1.0)     |
| `maxRetries`   | int    | 6       | Maximum number of retry attempts          |

### Stream-Level Errors

Configure `onStreamError` to control behavior when the entire stream encounters errors:

```json
{
  "onStreamError": {
    "type": "RetryStreamError",
    "maxRetries": 5
  }
}
```

| Type               | Description                                |
|:-------------------|:-------------------------------------------|
| `LogStreamError`   | Log the error and stop the stream          |
| `RetryStreamError` | Retry stream connection with a retry limit |

### Dead Letter Queue

When records fail to process, route them to a dead letter queue for later analysis or reprocessing. Supported destinations include:

* **Kafka** - publish to a Kafka topic
* **Kinesis** - publish to a Kinesis stream
* **SNS** - publish to an SNS topic
* **HTTP** - POST to a webhook endpoint
* **File** - write to a local JSON file
* **Stdout** - write to standard output
* **ReactiveStream** - broadcast to a reactive stream endpoint

Configure dead letter queue settings in your ingest stream definition:

```json
{
  "name": "my-ingest",
  "source": {
    "type": "Kafka",
    "format": "Json",
    "topics": ["events"],
    "bootstrapServers": "kafka:9092"
  },
  "query": "CREATE ($that)",
  "onRecordError": {
    "deadLetterQueueSettings": {
      "destinations": [
        {
          "type": "Kafka",
          "topic": "failed-records",
          "bootstrapServers": "kafka:9092",
          "outputFormat": {
            "type": "JSON",
            "withInfoEnvelope": true
          }
        }
      ]
    }
  }
}
```

#### Output Formats

| Format    | Configuration                                                 | Description                                 |
|:----------|:--------------------------------------------------------------|:--------------------------------------------|
| JSON      | `{"type": "JSON"}`                                            | Raw JSON output                             |
| JSON+Info | `{"type": "JSON", "withInfoEnvelope": true}`                  | JSON with error details and original record |
| Protobuf  | `{"type": "Protobuf", "schemaUrl": "...", "typeName": "..."}` | Binary Protobuf serialization               |

When `withInfoEnvelope` is `true`, failed records are wrapped with metadata:

```json
{
  "error": "Cypher execution failed: property 'id' is required",
  "timestamp": "2024-01-15T10:30:00Z",
  "ingestName": "my-ingest",
  "originalRecord": { "name": "incomplete" }
}
```

!!! note "Kafka DLQ Security"
    Kafka dead letter queue destinations support the same [secure configuration parameters](kafka.md#secure-kafka-configuration) as other Kafka integrations (ingest sources and [standing query outputs](../standing-queries/standing-queries.md#publish-to-kafka-topic)), including `sslKeystorePassword`, `sslTruststorePassword`, `sslKeyPassword`, and `saslJaasConfig`. These values are automatically redacted in API responses.

## Record Formats

The `format` field on a streaming source tells Quine how to decode each record before binding it to `$that` in your Cypher query. The format determines the Cypher type of `$that`.

For streaming sources (Kafka, Kinesis/KinesisKCL, SQS, ServerSentEvents, WebSocket, ReactiveStream, S3), the available formats are:

| Format     | `$that` is...            | Description                                                                                  |
|:-----------|:-------------------------|:--------------------------------------------------------------------------------------------|
| `Json`     | a `Map` (or list/scalar) | Each record is parsed as a JSON value. Cypher accesses fields with `$that.field`.            |
| `Raw`      | a `Bytes` value          | Each record is passed through unparsed. Property access on `$that` fails with a type error.  |
| `Protobuf` | a `Map`                  | Each record is a Protobuf-encoded message. Requires `schemaUrl` and `typeName`.              |
| `Avro`     | a `Map`                  | Each record is Avro-encoded. Requires `schemaUrl`. See [Avro Format](#avro-format) below.    |
| `Drop`     | (no record produced)     | Records are discarded without invoking the Cypher query. Useful for testing connectivity.    |

!!! warning "`Raw` does not parse JSON"
    `Raw` passes the unparsed bytes through, even when the record is JSON. Property access (`$that.field`) on a `Bytes` value fails with a type error. For JSON payloads, use `Json`.

### Choosing between `Json` and `Raw`

For JSON payloads, use `Json`. Cypher accesses fields on `$that` directly:

```json
"source": {
  "type": "Kinesis",
  "format": "Json",
  "streamName": "events"
}
```

```cypher
MATCH (n) WHERE id(n) = idFrom($that.eventId) SET n = $that
```

Use `Raw` for non-JSON binary payloads, or when you need the original bytes before parsing. With `Raw`, `$that` is the raw bytes of the record, so decode and parse them in Cypher with `text.utf8Decode` (`Bytes` → `String`) and `parseJson` (`String` → Cypher value):

```json
"source": {
  "type": "Kinesis",
  "format": "Raw",
  "streamName": "events"
}
```

```cypher
WITH parseJson(text.utf8Decode($that)) AS json
MATCH (n) WHERE id(n) = idFrom(json.eventId)
SET n = json
```

## Avro Format

Quine supports native Apache Avro for efficient binary serialization.

```json
{
  "name": "avro-ingest",
  "source": {
    "type": "Kafka",
    "format": {
      "type": "Avro",
      "schemaUrl": "http://schema-registry:8081/schemas/ids/1"
    },
    "topics": ["avro-events"],
    "bootstrapServers": "kafka:9092"
  },
  "query": "CREATE ($that)"
}
```

The `schemaUrl` can point to:

* A Schema Registry URL
* An HTTP/HTTPS URL serving the Avro schema JSON
* A local file path containing the schema

## JavaScript Transformation

Pre-process records with JavaScript before executing your Cypher query. This enables data normalization, filtering, and enrichment without modifying the source.

```json
{
  "name": "transformed-ingest",
  "source": {
    "type": "Kafka",
    "format": "Json",
    "topics": ["raw-events"],
    "bootstrapServers": "kafka:9092"
  },
  "query": "CREATE (n:Event $that)",
  "transformation": {
    "type": "JavaScript",
    "function": "function transform(record) { record.processedAt = Date.now(); return record; }"
  }
}
```

### Transformation Use Cases

* **Field normalization** - standardize field names across sources
* **Data enrichment** - add computed fields
* **Filtering** - return `null` to skip records
* **Format conversion** - reshape nested structures

The JavaScript function receives the parsed record and must return the transformed record (or `null` to skip).

## WebSocket File Upload

Stream files directly to Quine via WebSocket with real-time progress feedback.

### Configuration

Create a WebSocket file upload ingest that accepts file data:

```json
{
  "name": "ws-upload",
  "source": {
    "type": "WebSocketFileUpload",
    "format": "Json"
  },
  "query": "CREATE ($that)"
}
```

Supported file formats:

| Format  | Description                           |
|:--------|:--------------------------------------|
| `Line`  | Line-delimited text                   |
| `JsonL` | Newline-delimited JSON (JSON Lines)   |
| `Json`  | Single JSON array or object           |
| `Csv`   | Comma-separated values with headers   |

### Progress Feedback

The server sends JSON messages during upload to provide feedback:

| Message Type | Description                              |
|:-------------|:-----------------------------------------|
| `Ack`        | Acknowledges receipt of data chunk       |
| `Progress`   | Reports number of records processed      |
| `Error`      | Reports parsing or processing errors     |

```json
{ "type": "Ack" }
{ "type": "Progress", "count": 100 }
{ "type": "Error", "message": "Parse error at line 50", "index": 50, "record": "..." }
```

The server buffers up to 8 WebSocket messages for backpressure handling.

## Inspecting Ingest Streams via the API

Quine exposes API endpoints to monitor and manage ingest streams while in operation. You can also manage ingest streams from the [Streams](../../getting-started/streams.md) page in the UI.

| Operation                | Endpoint                                                                                                                    |
|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------|
| List all ingest streams  | [List Ingest Streams: `GET /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/list-ingests)                   |
| Create ingest stream     | [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest)                |
| Get ingest stream status | [Ingest Stream Status: `GET /api/v2/graph/quine/ingests/{ingestName}`](/reference/rest-api/?av=v2#/operations/get-ingest-status)      |
| Pause ingest stream      | [Pause Ingest Stream: `POST /api/v2/graph/quine/ingests/{ingestName}:pause`](/reference/rest-api/?av=v2#/operations/pause-ingest)     |
| Resume ingest stream     | [Resume Ingest Stream: `POST /api/v2/graph/quine/ingests/{ingestName}:resume`](/reference/rest-api/?av=v2#/operations/resume-ingest) |
| Delete ingest stream     | [Delete Ingest Stream: `DELETE /api/v2/graph/quine/ingests/{ingestName}`](/reference/rest-api/?av=v2#/operations/delete-ingest)  |

For complete API documentation, see the [REST API Reference](/reference/rest-api/?av=v2).

!!! tip "Quine Enterprise"

    For even higher ingest rates and high availability, see Enterprise clustering. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).


############################################################
## SECTION: General
############################################################


---
# Home
URL: https://quine.io/

Quine is a streaming graph interpreter. It's a server-side program which consumes data, builds a graph structure, and runs live computation on that graph to answer questions or compute results, and then stream them out.

The main idea is two-sided: event-driven data <==> data-driven events.


---
# Download
URL: https://quine.io/download/

Select the distribution that suits you to evaluate Quine from the list below, then [get started](getting-started/installing-quine-tutorial.md) building a streaming graph solution with Quine.

| Distribution | Requires                                | Launch Quine                                                        |
| :----------- | :-------------------------------------- | :------------------------------------------------------------------ |
| Docker File  | Docker or Docker Desktop                | `docker run -p 8080:8080 thatdot/quine`{ class="force-select-all" } |
| Executable   | Java JRE 11 or newer                    | `java -jar quine-2.1.1.jar`{ class="force-select-all" }                 |
| Source Code  | Java 11 or new JDK, sbt, node, and yarn | `sbt quine/run`{ class="force-select-all" }                         |

- **Pre-compiled JAR file**

    ---

    No need to set up a full Scala development environment to run Quine if you already have a Java Runtime Environment (JRE) set up.

    [:fontawesome-brands-java:   Download **JAR** file](https://github.com/thatdot/quine/releases/download/v2.1.1/quine-2.1.1.jar)

- **Docker Container**

    ---

    If you prefer to keep your initial evaluation in a container, we have one already set up for you and ready to go!

    [:fontawesome-brands-docker:  Pull **Docker** container](https://hub.docker.com/r/thatdot/quine/tags)

- **GitHub Repository**

    ---

    Quine is written in Scala and released as open source. Build Quine from source in your own environment.

    [:fontawesome-brands-github:   Clone from **GitHub**](https://github.com/thatdot/quine)

---

## [:octicons-tag-24: Release 2.1.1](https://github.com/thatdot/quine/releases/tag/v2.1.1)

### Enhancements:

- Background queries and scheduled jobs run long and recurring adhoc queries beyond the limits of a synchronous request:
    - Background queries survive request and page timeouts, report status throughout, and cancel on demand.
    - Results stream to any supported destination (Kafka, Kinesis, SNS, HTTP, Cypher, files), or to none for effect-only runs.
    - Scheduled jobs fire on a fixed interval or wall-clock time (hourly, daily, weekly, monthly) in any IANA time zone, survive restarts, and re-fire interrupted runs.
- Graph feeds now have an API endpoint and are supported in recipes.
- Result table columns are drag-resizable; long values, arrays, and objects stay on one line with the full value on hover.
- The dashboard's data-flow diagram drops its scope bar for full page width.

### Security:

- Multiple transient dependencies were updated to address CVEs.

### Bugfixes:

- A standing query whose pattern spans a relationship now emits matches as each relationship resolves, instead of waiting for all of them.
- `toLower` and `toUpper` are locale-independent, returning the same result on any host.
- A saved Exploration session will now successfully load what it can even if part of the loaded state is invalidated by a software update.
- A failed query inspection reports the error on its card with a reconnect action.
- A misspelled recipe field fails with a clear error naming it.
- File ingest distinguishes a policy denial from an unresolvable path, so a missing file or unmounted volume reports as such.
- Dashboard tooltips can be hovered and scrolled, and long ingest or standing query names wrap instead of overflowing.
- Exploration UI refinements: "Standing Query" spelled out in full, closed-hand cursor on panel drag, the topmost standing query name fits the data-flow diagram, and the graph feed editor shows only guidance for the selected tap point.

### Updates:

- AWS SDK to 2.54.7
- Amazon Kinesis client to 3.5.1
- Cypher editor (Monaco) to 0.56.0
- Interactive API documentation viewer (Stoplight Elements) to 9.0.25
- Logback logging library to 1.6.3
- API framework (Tapir) to 1.13.30
- TLS configuration library to 10.0.6
- Apache Commons Codec to 1.22.1
- LZ4 compression library to 1.11.1

---

## [:octicons-tag-24: Release 2.1.0](https://github.com/thatdot/quine/releases/tag/v2.1.0)

### New Features:

- A refreshed visual experience makes it easier to understand what's happening in your graph and faster to write and debug queries in a more polished and consistent interface:
    - Status Dashboard:
        - The home dashboard was redesigned around a live diagram of how data is actually flowing through your system, from ingests into the write pipeline, on to standing queries, round-trip to the persistor, and out through your outputs.
        - Color coding shows at a glance whether things are flowing normally, paused, failed, or backpressured; including where the backpressured component is so you can optimize it.
        - Hover tooltips explain exact throughput and counts at each stage, so you can spot and diagnose a slowdown without digging through logs
    - Cypher Editor: A rebuilt Cypher editor adds autocomplete, inline documentation, and inline diagnostics, so you can write queries faster and catch mistakes before you run them
    - Exploration UI:
        - Standing Query Inspections: You can now tap directly into a standing query's output pipeline at multiple points to sample its live results without writing a query
        - Graph Feeds: You can now set up a feed from a standing query that automatically updates the Exploration UI graph with live results; toggleable widgets for enabling them appear on the Exploration UI
        - Text query results and standing query inspections now appear in their own floating cards that expand into a resizable viewer or minimize into a compact stack, keeping the graph view clear
        - A new node properties viewer/editor lets you inspect and edit a node's properties directly, without writing a query
        - You can now bookmark queries you run often as sample queries making them available with a single click right from the query bar
        - Node and edge counts now show as a brief indicator over the canvas after a query completes, instead of taking up permanent toolbar space
        - A new Exploration UI Settings enables editing sample queries, defining quick queries for one-click actions on nodes, customizing how different node types appear in the Exploration UI, and managing graph feeds
- Though not built into Quine, there is a new "Quine Recipe Analyzer" to visualize and make recommendations for user recipes or ingest and standing query sets. Available at: [https://www.thatdot.com/quine/recipe-analyzer.html](https://www.thatdot.com/quine/recipe-analyzer.html)

### Enhancements:

- The Interactive API docs now remember your place; expanded sections, scroll position, and in-progress requests are preserved when you navigate away and back, instead of resetting every time
- The "create standing query" API can now trigger propagation to existing graph data on disk in the same request instead of requiring a separate follow-up call; new standing queries now automatically propagate to in-memory data by default.
- Administrators can enable verbose actor-system diagnostic logging via a JVM system property, without turning on debug logging for all third-party libraries
- Cypher queries using a bare relationship pattern as a boolean condition (for example combined with AND/OR/NOT, or inside CASE WHEN) are now supported
- The v2 API now rejects requests containing unrecognized or misspelled JSON fields with a clear error naming the field, instead of silently ignoring them and falling back to defaults.
- Startup configuration errors for unrecognized config keys now clearly name the offending key and its location instead of showing an unhelpful generic error
- The web UI now automatically loads the latest version after a new release is deployed, without requiring a hard refresh to clear stale cached files
- Retry count, backoff timing, and retry window are now configurable per ingest via the API, so you can tune resilience to fit each data source

### Security:

- Updated the system configuration API endpoint to return only an explicit, pre-approved set of operational fields instead of the entire raw configuration, so credentials and other secrets embedded in the configuration can never be exposed
- Updated the AWS SDK, Amazon Kinesis client, Hadoop, and Jackson dependencies to address several reported vulnerabilities

### Bugfixes:

- CSV ingestion using the v2 ingest pipeline occasionally failed after the first row, but now ingests every row in a file
- Historical (time-travel) Cypher queries using SKIP or LIMIT, including from the Exploration UI, now return results correctly instead of failing with a server error
- Relationship patterns in Cypher used as a predicate or projection inside a list comprehension now evaluate properly
- Ingest stream and standing query actions (delete, pause, resume) on the Streams management page now show an error message on failure instead of failing silently
- Ingest streams now recover automatically from transient broker hiccups, with throughput metrics staying live and accurate through a restart
- Exporting a graph view to SVG now correctly displays each node's icon, including emoji and other symbols, instead of falling back to plain circles
- Recreating an ingest with the same name now starts with a clean slate, so status counts no longer carry over from the deleted one.

### Updates:

- Cassandra database driver to 4.19.3
- Pekko actor and streaming libraries to 1.6.0
- Logback logging library to 1.5.38
- Minimatch to 5.1.9

---

## [:octicons-tag-24: Release 2.0.2](https://github.com/thatdot/quine/releases/tag/v2.0.2)

### Enhancements:

- New WebSocket wiretapping endpoints for real-time monitoring of standing query results at three pipeline stages: raw, pre-enrichment, and post-enrichment  
- New support for Parquet and Avro file formats for ingest streams  
- API resource names for ingests, standing queries, and outputs now follow Google AIP-122 naming conventions, ensuring consistent and unambiguous routing of API verbs. Note: colons (`:`) are now reserved for API verb routing and are no longer permitted in resource names

### Bugfixes:

- Standing query output creation now returns specific error details for invalid configurations instead of generic server errors  
- Standing queries with no outputs no longer back pressure and stop ingests

### Security:

- Updated Docker base image to address CVE-2026-5419 and CVE-2026-42014  
- Updated uuid to 14.0.0 (CVE-2026-41907)  
- Updated minimatch to 5.1.8 (CVE-2026-27903, CVE-2026-27904)  
- Updated js-cookie to 3.0.8 (CVE-2026-46625)

### Updates:

- uuid to 14.0.0  
- minimatch to 5.1.8  
- js-cookie to 3.0.8  
- parquet4s-core to 2.23.0  
- hadoop-client-api to 3.4.3  
- hadoop-client-runtime to 3.4.3

---

## [:octicons-tag-24: Release 2.0.1](https://github.com/thatdot/quine/releases/tag/v2.0.1)

### Enhancements:

- Improved Streams page with expandable configuration details and improved metrics layout
- Improved interactive API documentation with reordered sidebar sections and clearer endpoint descriptions
- Reorganized sidebar navigation and page layout for easier access to key features

### Security:

- Updated Netty to 4.1.133.Final (CVE-2026-41417)
- Updated ayza to 10.0.5, resolving bundled Bouncy Castle vulnerability (CVE-2026-5598)

### Bugfixes:

- Fixed V2 recipes failing to load when using features not present in V1 schema
- Fixed standing query validation errors returning generic 500 errors instead of descriptive 400 errors
- Fixed API documentation not displaying discriminated union types correctly
- Fixed dashboard showing animated data flow indicators on idle or stopped streams
- Fixed sidebar collapse button not accessible after expanding on narrow viewports
- Fixed server connectivity errors on the dashboard displaying as decode failures

### Updates:

- AWS SDK to 2.42.41
- Tapir to 1.13.19
- Amazon Kinesis Client to 3.4.3
- commons-codec to 1.22.0
- caffeine to 3.2.4
- msgpack-core to 0.9.12
- jcl-over-slf4j to 2.0.18

---

## [:octicons-tag-24: Release 2.0.0](https://github.com/thatdot/quine/releases/tag/v2.0.0)

### Enhancements:

- **NEW**: Redesigned web interface with modernized layout, sidebar navigation, and consistent theming across all pages

    - Added a Dashboard landing page with system overview diagram, host metrics, ingest stream status, and standing query status
    - Add a Streams management page for easily creating and managing ingest streams and standing queries through the UI
    - Added JSON-LD graph export option to the Exploration UI download menu
    - Exploration UI nodes are now pinned in place by default when dragged, and Shift+click-hold unpins them
    - Default node appearance now renders the "name" property for all nodes regardless of label; customizable in the same way with UI Styling endpoints

- **NEW**: Official release of API V2 which is now the default in the UI; API V1 is still available for backwards compatibility but will be deprecated in the future

    - Follows the Google API style guides with established patterns and strategies for modern REST APIs
    - Enhanced ingest and output capabilities with persistent configuration, expanded source and destination support, and improved error handling bringing parity across all products. Includes javascript transformation of data in all ingests and outputs, and a customizable dead letter queue for unreadable ingested events
    - Standing Query outputs enhanced with workflow steps, providing clearer output structures; can be chained together
    - Cypher query parameter support for executing queries via API
    - Comprehensive developer documentation with examples is built in and enhanced
    - Recipes now support a schema version 2, enabling configuration enhancements supported by API V2

- Docker base image updated to eclipse-temurin 21.0.10\_7-jre-noble

- Added Kafka output destination connectivity validation on configuration, returning an early error when bootstrap servers are unreachable

- All-node scan queries now filter out empty nodes with no current edges, properties, or labels at the queried historical moment

### Security Fixes:

- AWS credentials are automatically redacted in API responses and logs
- Kafka configuration sensitive values (SSL passwords, SASL credentials) are redacted in API responses and logs
- Disallow unsafe-eval from Content Security Policy
- Updated Bouncy Castle to 1.84 (CVE-2026-5598)
- Switched lz4 compression library to maintained fork (CVE-2025-66566)
- Updated Netty to patched version (CVE-2025-67735)
- Updated msgpack (CVE-2026-21452)
- Fixed multiple JavaScript dependency vulnerabilities

### Bugfixes:

- Fixed Kafka output property values being incorrectly serialized
- Fixed reverse proxy path prefix support for UI routing and V2 OpenAPI endpoints
- Fixed checkpoint forward/backward navigation on the Exploration UI restoring incorrect state
- Fixed SVG export cutting off node labels at the viewbox edge
- Fixed standing query output failures causing the standing query to be removed in some cases
- Fixed standing query error message to clarify when edge labels are required
- Fixed GraalJS 25+ compatibility with Multi-Release JAR manifest

### Updates:

- Apache Pekko to 1.5.0
- Pekko HTTP to 1.3.0
- Pekko Connectors (CSV, Kinesis, S3, SNS, SQS) to 1.3.0
- RocksDB JNI to 10.10.1.1
- Tapir to 1.13.15
- Scala to 2.13.18
- Cats Effect to 3.7.0
- Circe JSON libraries to 0.14.15
- GraalVM JS engine to 25.0.2
- protobuf-java to 4.34.1
- Kafka clients to 3.9.2
- Amazon Kinesis Client to 3.4.2
- Netty to 4.1.130.Final
- Cassandra Java driver to 4.19.2
- AWS SDK to 2.42.24
- logback-classic to 1.5.32
- ANTLR4 to 4.13.2
- Dropwizard Metrics to 4.2.38
- PureConfig to 0.17.10
- commons-codec to 1.21.0
- commons-text to 1.15.0
- jnr-posix to 3.1.22
- schema-registry-serde to 1.1.27
- Docker base image to eclipse-temurin 21.0.10\_7-jre-noble

---

## [:octicons-tag-24: Release 1.10.0](https://github.com/thatdot/quine/releases/tag/v1.10.0)

### Enhancements:
- Added Conway's Game of Life [recipe](https://quine.io/recipes/conways-gol/)
- Improved graph query performance by limiting edges returned during traversals
- Increased default max SSE line and event sizes to 5MB to prevent infinite retry loops on large messages

### Security Fixes:
- File ingest operations now require an `allowedDirectories` configuration; files loaded by recipe or from the working directory are allowed by default
- Reduced in-memory lifecycle of passwords by using character arrays that are blanked after use
- Enhanced UUID generation to use FIPS 140-2 compliant cryptographically secure random values
- Enforced strict-transport-security HTTP header across all endpoints
- Added additional HTTP security headers including tightened CSP
- ClickHouse persistence now requires credentials to be set via environment variables instead of configuration files

### Bugfixes:
- Fixed empty Kafka message handling to prevent exceptions from tombstone messages
- Fixed keyspace name for AWS Keyspaces to be case-sensitive - Thanks to contribution by [fredsensibill](https://github.com/fredsensibill)
- Removed application base URL configuration in favor of runtime inference
- Improved error response formatting by using correlation codes that link user-facing errors to detailed server logs for easier troubleshooting
- Replaced documented IP addresses with RFC 5737 TEST-NET addresses

### Updates:
- Updated amazon-kinesis-client to 3.1.3
- Updated logback-classic to 1.5.20
- Updated caffeine to 3.2.3
- Updated pekko-actor, pekko-cluster, and related components to 1.2.1
- Updated avro to 1.12.1
- Updated commons-csv to 1.14.1
- Updated commons-codec to 1.19.0
- Updated commons-io to 2.20.0
- Updated netty-handler to 4.1.127.Final
- Updated scalajs-dom to 2.8.1
- Updated msgpack-core to 0.9.10
- Updated java-driver-core and java-driver-query-builder to 4.19.1
- Updated metrics-core, metrics-jmx, and metrics-json to 4.2.37
- Updated amqp-client to 5.26.0
- Updated AWS SDK components (aws-core, aws-query-protocol, kinesis, kinesis-video, and others) to 2.31.78
- Updated scalacheck to 1.19.0
- Updated scala-logging to 3.9.6
- Updated pprint to 0.9.4
- Updated kind-projector to 0.13.4
- Updated org.eclipse.jgit to 7.3.0.202506031305-r
- Updated sbt and related components to 1.11.7
- Updated sbt-scalafmt to 2.5.6
- Updated sbt-jmh to 0.4.8
- Updated sbt-sbom to 0.5.0
- Updated sbt-scalajs and scalajs-compiler to 1.20.1

---

## [:octicons-tag-24: Release 1.9.3](https://github.com/thatdot/quine/releases/tag/v1.9.3)

### Bugfixes
- Initialize browser history on UI page load, correcting behavior of browser navigation when navigating between tabs, and using the back/forward browser buttons

---

## [:octicons-tag-24: Release 1.9.2](https://github.com/thatdot/quine/releases/tag/v1.9.2)

This patch release includes foundational changes in preparation for an upcoming major release; current functionality remains unaffected.

### Enhancements

- Show Java runtime version in admin build-info endpoint
- Handle yaml format errors in API endpoint requests

### Bugfixes

- Use reverse-proxy compatible path definitions for resources and introduce optional server advertise path parameter. Thank you for the contribution [@kollolsb](https://github.com/kollolsb)!
- Removing quotes from keyspace names in keyspaceExistsQuery for AWS Keyspaces. Thank you for the contribution [@fredsensibill](https://github.com/fredsensibill)!
- Remove use of `SELECT COUNT(*)`, not supported in Keyspaces. Thank you for the contribution [@LeifW](https://github.com/LeifW)!
- Fix GraalVM dependency

### Updates

- Remove now-redundant `reactor-netty-*` dependency overrides
- Update aws-sdk, ... to 2.31.54
- Update pekko-http-circe to 3.0.1
- Update pureconfig to 0.17.9
- Update embedded-cassandra to 5.0.2
- Update cats-effect to 3.6.1
- Update kafka-clients to 3.9.1
- Update tapir-core, tapir-json-circe, ... to 1.11.33
- Update pekko-management, ... to 1.1.1
- Update protobuf-java to 3.25.8
- Update amazon-kinesis-client to 3.0.3
- Update pekko-http, pekko-http-spray-json, ... to 1.2.0
- Update graalvm.js to 24.2.1
- Update bootstrap to 5.3.6
- Update @stoplight/elements to 9.0.1
- Update plotly.js to 2.25.2

---

## [:octicons-tag-24: Release 1.9.1](https://github.com/thatdot/quine/releases/tag/v1.9.1)

### Enhancements

- Add support for KCL-based Kinesis ingestion, offering stronger delivery guarantees through checkpointing via DynamoDB, advanced configuration options, and optional enhanced fan-out support.
- Introduce OAuth2 optionally for Cassandra persistor

### Bugfixes

- Fix AWS SQS ingest with missing dependency
- Fix Exploration UI header logo when rendered on Safari on latest macOS update

### Updates

- Update circe-core, circe-generic, ... to 0.14.12
- Update logback-classic to 1.5.18
- Update marketplacemetering, s3, sso, ssooidc, ... to 2.30.38
- Update shapeless to 2.3.13
- Update flatbuffers-java to 25.2.10
- Update tapir-core, tapir-json-circe, ... to 1.11.20
- Update circe-yaml from 0.11.7 to 0.11.9

---

## [:octicons-tag-24: Release 1.9.0](https://github.com/thatdot/quine/releases/tag/v1.9.0)

### Enhancements:

- Enable thatDot products to stream data directly to each other using reactive streams
- Support optionally omitting metadata from Standing Query output by setting the new `structure` property `Bare`
- Simplified logging of awake nodes

### Bugfixes:

- Add netty-handler override for CVE-2025-24970

### Updates:

- Update Pekko dependencies to 1.1.3
- Update pekko-connectors-*, ... to 1.1.0
- Update Tapir dependencies to 1.11.16
- Update sttp:tapir/sttp:apispec to 1.10.15 / 0.11.2
- Update rocksdbjni 9.7.3
- Update logback-classic to 1.5.17
- Update jnr-posix to 3.1.20
- Update kafka-clients to 3.9.0
- Update project reactor netty version
- Update cats-effect to 3.5.7
- Update marketplacemetering, s3, sso, ssooidc, ... to 2.30.31
- Update pureconfig to 0.17.8
- Update schema-registry-serde to 1.1.22
- Update openapi-circe-yaml to 0.11.7
- Update protobuf-java to 3.25.6
- Update scala-library to 2.13.16
- Update cats-core to 2.13.0
- Update caffeine to 3.2.0
- Update msgpack-core to 0.9.9
- Update software.amazon.glue:schema-registry-serde to 1.1.23
- Update rsocket-core, rsocket-transport-netty to 1.1.5
- Update embedded-cassandra to 5.0.1
- Update java-driver-query-builder to 4.19.0
- Update jcl-over-slf4j to 2.0.17
- Update rocksdbjni to 9.7.4
- Update commons-text to 1.13.0
- Update commons-codec to 1.18.0
- Update commons-csv to 1.13.0

---

## [:octicons-tag-24: Release 1.8.2](https://github.com/thatdot/quine/releases/tag/v1.8.2)

### Bugfixes

- Fix MultipleValuesStandingQuery State for LocalPropertyState, LabelsState, and AllPropertiesState to ensure subscribers are updated on node wake
- Improve Cassandra snapshot read to avoid reading extra parts from a singleton snapshot

---

## [:octicons-tag-24: Release 1.8.1](https://github.com/thatdot/quine/releases/tag/v1.8.1)

### Updates with special concerns

MultipleValues standing queries will not migrate from Quine 1.7.3.
If you have any MultipleValues standing queries and you are using RocksDB, your database files will not carry forward.
If you are using MultipleValues standing queries and Cassandra, perform the following steps to remove all MultipleValues data while leaving your graph intact:

1. Before shutting down Quine 1.7.3, use the API to delete all MultipleValues standing queries
2. After shutting down Quine 1.7.3, but before launching Quine 1.8.1, use cqlsh to `TRUNCATE standing_query_states`
3. After launching Quine 1.8.1, use the API to recreate all MultipleValues standing queries

### Enhancements

- Disable log redaction by default
- Stop treating queries as PII
- Add CSP and X-Frame-Options headers for clickjacking hardening
- Node/edge query endpoints now silently permit (and skip over) `null` return values to improve ergonomics of OPTIONAL MATCH queries
- Simplify configuration for TLS termination (reported by [`@min-mwei`](https://github.com/min-mwei))

### Bugfixes

- Refactored MultipleValues Standing Query registration behavior to improve result consistency and reduce initial match latency
- Fix an NullPointerException in the recipe interpreter
- LocalProperty standing queries no longer drop results after a node is re-awakened from persistence
- Changed the ID generation algorithm for MultipleValues standing query internals to prevent standing querise from overwriting each other
- Log a warning if MultipleValues standing queries overwrite each other
- Some user errors that were returned as HTTP 500-class responses are now correctly 400-class responses

### Updates

- Replaced Guava caching library in favor of Caffeine
- sslcontext-kickstart, ... to 8.3.7
- pekko-http to 1.1.0, pekko-http-circe to 3.0.0
- endpoints4s to 1.12.1 (+ related)
- clickhouse-http-client to 0.6.5
- logback-classic to 1.5.8
- commons-io to 2.17.0
- pekko to 1.1.1
- scalacheck to 1.18.1
- protobuf-java to 3.25.5
- sbt-scalajs, scalajs-compiler, ... to 1.17.0
- embedded-cassandra to 5.0.0
- scala-library to 2.13.15
- commons-csv to 1.12.0
- org.eclipse.jgit to 7.0.0.202409031743-r
- pekko-connectors-kafka to 1.1.0

---

## [:octicons-tag-24: Release 1.7.3](https://github.com/thatdot/quine/releases/tag/v1.7.3)

### Enhancements

- Add support for modern `SHOW PROCEDURES` syntax via rewrite rule
- Add support for sum and avg aggregators with `duration`-typed values
- Improve the warning offered when trying to resolve an illegal QuineId and add suggestions for how to fix the query
- Normalize logging and error messages which timeout for ExactlyOnceAsks
- Add timers for ingest query runtime, ingest deserialization runtime and SQ result queue time
- Add timers for node sleeps and node wakes
- Add histogram for property sizes
- Standardize logging framework to slf4j/logback

### Bugfixes

- Fix "Promise already completed" warning that could occur when sending [ask] messages
- Update unit subqueries to correctly return one row rather than the number of rows from the inner query
- Fix unit subquery scope for side-effect-only subqueries to no longer leak to the higher scope
- Filter ethereum transaction ingest to only use valid transactions
- Fix rare race condition in node messaging
- Account for node lifecycle in node size histograms
- Fix a typo in query API that suppressed user-facing log references
- Fix false-positive "Query cannot end with CALL combined with YIELD" errors
- Fix some scenarios where ingest status displays the wrong value
- Fix NullPointerException that could occur when logging some messages
- Improve resiliency of Standing Query wake-up protocol with slow persistors

### Updates

- AWS SDK to 2.26.31
- cassandra to 4.18.1
- circe-yaml to 0.16.0
- circe to 3.9.8
- commons-compress to 1.27.1
- dropwizard metrics to 4.2.27
- flatbuffers-java to 24.3.25
- guava to 33.3.0-jre
- kafka-clients to 3.8.0
- logback-classic to 1.5.7
- rocksdbjni to 9.0.0
- schema-registry-serde to 1.1.20
- snappy-java to 1.1.10.6

---

## [:octicons-tag-24: Release 1.7.2](https://github.com/thatdot/quine/releases/tag/v1.7.2)

### Enhancements

- Added log sanitization. This feature suppresses exception logging and hides values that may have been sourced from ingested records or query definitions. This behavior is enabled by default, and may be customized with the `quine.log-config` configuration section.
- Added `text.regexReplaceAll` Cypher function for regex-based string substitution
- Added function `text.regexGroups` that will, given a regular expression and a string, return all matching capture groups from the string.
- Added support for `<duration>.<unit>` and `<duration>["<unit">]` syntax in Cypher queries
- Added support for constructing duration Cypher values with approximate units, e.g., days
- Updated logo colors in exploration UI
- The REST API documentation UI now makes requests based on relative rather than absolute paths, allowing it to work in more deployment environments
- Added additional logging to alert on issues caused by poor network connection
- Clarified the language in some error messages
- Simplify error formatting in ingest and standing query output streams
- Added additional error handling for certain uses of invalid QuineIds and illegal regular expressions
- Reduced log noise at `DEBUG` log level
- Persistor errors that occur during all-node scans will now be reported consistently with all other persistor errors
- Improved error handling for missing file errors in the recipe interpreter and file ingests

### Bugfixes

- Fixed compilation of `WITH DISTINCT` Cypher query clauses to now properly interpret the returned distinct set in all cases **(contributed by <a target="_blank" href="https://github.com/harpocrates">@harpocrates</a>)**
- The example standing query in the API no longer uses the removed `exists(n.property)` syntax
- The error message returned by `standing.wiretap` when a standing query is not found now refers to the name argument, rather than the procedure name
- MultipleValues standing queries with emitCancellations = false will no longer emit cancellations
- Multiple Values Standing Queries no longer read node labels as a property

### Updates

- skunk-circe, skunk-core to 0.6.4
- s3, sts to 2.25.70
- scala-java-time to 2.6.0
- pureconfig to 0.17.7
- tapir-json-circe, ... to 1.10.10
- pekko to 1.0.3
- pekko-http-circe to 2.6.0
- protobuf-java to 3.25.4
- circe to 0.14.9
- circe-yaml to 0.15.3
- circe-generic-extras to 0.14.4
- Dropwizard metrics to 4.2.26
- scaffeine to 5.3.0
- commons-codec to 1.17.1
- Removed scala-java8-compat

---

## [:octicons-tag-24: Release 1.7.0](https://github.com/thatdot/quine/releases/tag/v1.7.0)

### Updates with special concerns

MultipleValues standing queries will not migrate from Quine 1.6.4.

If you have any MultipleValues standing queries and you are using RocksDB, your database files will not carry forward.

If you are using MultipleValues standing queries and Cassandra, perform the following steps to remove all MultipleValues data while leaving your graph intact:

1. Before shutting down Quine 1.6.4, use the API to delete all MultipleValues standing queries
2. After shutting down Quine 1.6.4, but before launching Quine 1.7.0, use cqlsh to `TRUNCATE standing_query_states`
3. After launching Quine 1.7.0, use the API to recreate all MultipleValues standing queries

Ad-hoc queries now reflect SETs and REMOVEs made during the query

If your ad-hoc (i.e., ingest, Cypher standing query output, or /query/cypher/ POSTed) queries rely on the original "capture state on MATCH" behavior, add a variable with WITH capturing the state you need.

For example, `MATCH (n) SET n.x = 2 RETURN n.x AS oldX` becomes `MATCH (n) WITH n, n.x AS oldX SET n.x = 2 RETURN oldX`

### Enhancements

- MultipleValues standing queries have been rewritten from the ground up for massive performance gains
- Failing to deserialize an ingested record now logs a warning + INFO message, and resumes the stream, rather than moving the stream to an error state
- Updated the behavior of the SET and REMOVE clause to reflect their expected changes on the in-memory copies of query-related values
- Protobuf message types can now be set by short name or full name
- Added `toProtobuf` Cypher serialization procedure
- Protobuf procedures now yield `null` values on failing to serialize/deserialize
- Enhanced telemetry with basic feature usage indicators to support prioritizing most-used features
- Simplified and tuned ingest registration logic

### Bugfixes

- Cassandra persistor now respects the `should-create-tables` configuration option when using only the default namespace
- Cypher `log` procedure now accepts log levels in a case-insensitive manner
- ANSI control codes are no longer logged except when running a recipe
- Cypher `node.debug` error no longer reports an error on nodes tracking MultipleValues standing queries

### Updates

- Remove aws-java-sdk-sts
- scalajs to 1.16.0
- kafka-clients to 3.6.2
- amazon-kinesis-client to 2.5.8
- marketplacemetering, s3, sts to 2.25.42
- classgraph to run tests on scala 2.13 and push to main
- scala-parser-combinators to 2.4.0
- circe to 3.9.6
- sbt-paradox to 0.10.7
- commons-io to 2.16.1
- commons-compress to 1.26.1
- commons-text to 1.12.0
- circe to 0.14.7
- logback-classic to 1.5.6
- tapir to 1.10.8

---

## [:octicons-tag-24: Release 1.6.4](https://github.com/thatdot/quine/releases/tag/v1.6.4)

### Enhancements

 - Add round() and radians() cypher functions
 - Improve error handling in the exploration UI
 - Add `parseProtobuf` cypher procedure. This procedure takes 3 arguments: bytes to decode, schema URL, and type name; and yields a single value named `value`.

### Bugfixes

 - Unregistered routes will now return a "not found" page instead of an error about a missing query parameter
 - The cypher interpreter will no longer report "undefined variable" when returning variables bound to pattern expressions (e.g., `MATCH p=(a) RETURN p`)
 - SET n.prop = NULL now correctly removes the property
 - Multiple SET clauses in sequence now correctly updates the node's properties instead of replacing them

### Updates

 - cats-effect to 3.5.4
 - amazon-kinesis-client to 2.5.7
 - guava to 33.1.0-jre
 - commons-io to 2.16.0
 - webjars-locator to 0.52
 - pekko-http-circe to 2.4.0
 - logback-classic to 1.5.3
 - jedis to 5.1.2
 - scala-library to 2.13.13
 - Cassandra java-driver to 4.18.0.
 - commons-codec to 1.17.0
 - tapir to 1.10.6

---

## [:octicons-tag-24: Release 1.6.2](https://github.com/thatdot/quine/releases/tag/v1.6.2)

### Enhancements

- Lowered node wake latency by parallelising journal rehydration

### Bugfixes

- Webserver bind port or address can now be set without setting both

### Updates

- Update antlr to 4.13.1
- Update sbt-scalajs, scalajs-compiler, ... to 1.13.2
- Update slinky-core, slinky-web to 0.7.4
- Update jnr-posix to 3.1.19
- Update aws-java-sdk-sts to 1.12.670
- Update mapdb to 3.1.0
- Update rocksdbjni to 8.11.3
- Update pekko-http-circe to 2.3.4
- Update pekko-http, pekko-http-xml to 1.0.1
- Update marketplacemetering, s3, sts to 2.20.162

---

## [:octicons-tag-24: Release 1.6.1](https://github.com/thatdot/quine/releases/tag/v1.6.1)

### Changes that may require attention:

* Standardize the names of the NumberIteratorIngest fields `startAtOffset` and `maximumPerSecond`. If you were previously using the names `startAt` or `throttlePerSecond`, you will need to update your recipes and API calls

### Enhancements

* Reintroduce the classic single-line query bar on the Explore UI with SHIFT-ENTER to run a text query
* Add support for arbitrary kafka properties on WriteToKafka Standing Query output via `kafkaProperties` field
* Allow Multiple Values Standing Queries to event on arbitrarily-keyed property changes by using a `RETURN properties(n)` rather than having to list all properties manually

### Bugfixes:

* Reintroduce error messages encountered during Cypher query compilation which were lost as part of the Scala 2.13 migration in 1.6.0
* Improve favicon support on all platforms

### Updates:

* pekko, pekko-stream, pekko-connectors to 1.0.2
* Fix CVE-2024-25710: add explicit apache commons compress dep 1.26.0
* scalajs-dom to 2.8.0
* commons-codec to 1.16.1
* pureconfig to 0.17.6
* bootstrap to 5.3.3
* sbt, sbt-dependency-tree to 1.9.9
* logback-classic to 1.5.1
* protobuf-java to 3.25.3
* jedis to 5.1.1
* everit-json-schema to 1.14.4
* sttp to 3.9.2
* webjars-locator to 0.50
* pekko-http-circe to 2.1.1
* circe-optics to 0.15.0
* dropwizard metrics to 4.2.25
* guava to 33.0.0-jre
* skunk-circe, skunk-core to 0.6.3
* thatDot-opencypher to 9.1.1
* commons-text to 1.11.0
* msgpack-core to 0.9.8
* cats-effect to 3.5.3
* rocksdbjni to 8.5.4

---

## [:octicons-tag-24: Release 1.6.0](https://github.com/thatdot/quine/releases/tag/v1.6.0)

### Enhancements

* Migrate from Lightbend Akka to Apache Pekko Framework
* Dropped Apache Pulsar ingest support due to Akka dependency
* Migrate from Scala 2.12 to Scala 2.13
* Forbid specifying `JdniLoginModule` as a `sasl.jaas.config` to avoid CVE-2023-25194
* Limit InfluxDB metrics logging to error
* Added open source telemetry, for more info visit quine.io/telemetry
* Support multiline queries in the Exploration UI

### Bugfixes:

* Fixed "Run again as text query" button in the results panel in the Exploration UI

### Updates:

* `scala-library` 2.13 to 2.13.12
* `msgpack-core` to 0.9.6
* `dropwizard metrics-core`, `metrics-jmx`, `metrics-jvm` to 4.2.20
* `commons-compress` to 1.25.0 to address CVE-2021-35516, CVE-2021-35517, CVE-2021-36090
* `mapdb` to 3.0.10
* `kafka-clients` to 2.7.2
* `avro` to 1.11.3 to address CVE-2023-39410
* `aws-sdk` to 2.20.159
* `guava` to 32.1.3-jre
* `circe-config` to 0.10.1
* `circe-yaml-v12` to 0.15.1
* `commons-io` to 2.15.1
* `logback-classic` to 1.4.13

---

## [:octicons-tag-24: Release 1.5.7](https://github.com/thatdot/quine/releases/tag/v1.5.7)

### Enhancements:

* SSL is now supported when interacting with Quine in an untrusted environment by using the `SSL_KEYSTORE_PATH` and `SSL_KEYSTORE_PASSWORD` environment variables
* Add `create.setProperty` procedure to allow setting a property with a dynamic key, dual to accessing a property with square-bracket syntax
* Additional performance metrics can be added with `enableDebugMetrics` setting to help diagnose internal messaging and node load/unload throughput

### Bugfixes:

* Multiple Value Standing Query (MVSQ) will now properly trigger a change when a property is set/unset to `null`

### Updates:

* stoplight elements to v7.12.0
* snakeyaml-engine to 2.7
* logback-classic to 1.4.11
* sts to 2.20.139
* cats-core to 2.10.0
* circe-core, circe-parser, ... to 0.14.6
* msgpack-core to 0.9.5
* scalajs-dom to 2.6.0
* guava to 32.1.2-jre

---

## [:octicons-tag-24: Release 1.5.6](https://github.com/thatdot/quine/releases/tag/v1.5.6)

### Enhancements

- Added support for STS assumed-role authentication to experimental AWS Keyspaces persistor

### Bugfixes

- Removed "total memory" gauge from metrics dashboard when maximum total memory can't be calculated based on the running JVM
- Fixed a "stream stopped before async invocation was processed" error message that could occur when resuming ingests on startup

### Updates

- webjars-locator to 0.47
- sbt-jmh to 0.4.5
- guava to 32.0.1-jre
- dropwizard metrics to 4.2.19
- logback-classic to 1.4.8
- scala-library to 2.12.18
- cats-effect to 3.5.1
- scala-collection-compat to 2.11.0

---

## [:octicons-tag-24: Release 1.5.5](https://github.com/thatdot/quine/releases/tag/v1.5.5)

### Enhancemnts

- Ingest status is now preserved across restarts when restore-ingest configuration is set
- Improved error handling within Keyspaces persistor

### Bugfixes

- Valid Kafka configurations (e.g., `ssl.truststore.location`) are now accepted by the KafkaIngest stream configuration object
- Remove unnecessary log line

### Updates

- bootstrap to 5.3.0
- flatbuffers-java to 23.5.26
- sttp-client to 5.4.0
- pureconfig to 0.17.4
- commons-io to 2.12.0
- guava to 32.0.0-jre
- scalatest to 3.2.16
- sbt to 1.8.3

---

## [:octicons-tag-24: Release 1.5.4](https://github.com/thatdot/quine/releases/tag/v1.5.4)

Temporal types have had a spring cleaning! Time values created in Quine 1.5.3 and earlier (without timezones) are automatically converted to LocalTime values in Quine 1.5.4. No user action is required.

### Enhancements

- Added support for `localtime` function and values to the Quine graph and Cypher
- Updated documentation for Kinesis ingest to reflect the new name "Kinesis Data Streams"
- Technical preview for AWS Keyspaces persistor
- Improved configuration linting and error reporting for Kafka ingest
- Added support for `purgeNode` procedure to RocksDB and MapDB persistors
- Added automatic recovery (up to 3 attempts) of ingests that fail to start/restart due to upstream issues
- Simplified `reify.time`-generated nodes to use static labels for easier styling
- Improved documentation for administrative APIs
- Added scheduler checkpoint settings `maxBatchSize` and `maxBatchWait` to Kinesis Data Streams ingest

### Bugfixes

- Resuming a failed ingest now reports an HTTP error instead of timing out
- Time values in the Quine graph now correctly store and use offsets from UTC
- The `date` function now always correctly returns a date instead of a datetime.
- Errors that occur while submitting an invalid standing query are now reported as "Bad Request"

### Updates

- Base docker image to JVM 20
- sbt-scalajs-crossproject to 1.3.1
- scalajs to 1.13.1
- circe to 0.14.5
- pureconfig to 0.17.3
- endpoints4s to 3.8.15
- logback-classic to 1.4.7
- scala-collection-compat to 2.10.0

---

## [:octicons-tag-24: Release 1.5.3](https://github.com/thatdot/quine/releases/tag/v1.5.3)

### Enhancements

- A distinct "advertised"/canonical HTTP address used for self-referencing links may be set via `quine.webserver-advertise`
- Added `purgeNode` procedure, which may be used to permanently delete a node and all data owned by that node (including historical) from Quine, including the persistence layer

### Bugfixes

- Error message from Cypher compilation failures involving `int.add` now report the correct types
- Several typos throughout the codebase have been corrected, contributed by [@stavares843](https://github.com/stavares843)
- Slack notifications from Standing Queries now render with the intended rich formatting
- API documentation for DebugOps now correctly renders the usage disclaimer for those APIs

### Updates

- logback-classic to 1.4.6
- akka-http-backend, slf4j-backend to 3.8.14
- dropwizard metrics to 4.2.17
- circe-config to 0.10.0
- memeid4s to 0.8.0
- proto-google-common-protos to 2.14.2
- sbt-assembly to 2.1.1
- flatbuffers-java to 23.3.3
- rocksdbjni to 7.10.2

---

## [:octicons-tag-24: Release 1.5.2](https://github.com/thatdot/quine/releases/tag/v1.5.2)

### Enhancements

- Added first-class support to the Quine engine and the Cypher interpreter for new temporal types: `Date`, `Time`, `LocalDateTime`, and `Duration`
- Added `int.add()` (replacing incrementCounter), `float.add()`, `set.insert()`, and `set.union()` procedures for atomically updating properties of different types
- Added `castOrNull()` and `castOrThrow()` Cypher functions for providing type hints to the query compiler for values
- Replaced remaining uses of `ujson` with `circe` JSON for better support of 64-bit integers
- Replaced LiteralOps API with a new DebugOps API, focusing on presenting human-readable versions of node state
- Removed auto-commit option from Kafka ingest API: To continue using autocommit, pass `"autoCommitIntervalMs": 5000` in the kafkaProperties option instead

### Bugfixes

- Complex `strId()`-based queries are now executable without an all-node scan (contributed by @harpocrates)
- `MERGE` queries now correctly combine the scopes of their `CREATE` and `MATCH` possibilities (contributed by @harpocrates)
- Log messages from `IngestRoutes` no longer report as being logged by a different class (e.g., "QuineApp")

### Updates

- flatbuffers-java to 23.1.21
- jquery to 3.6.3
- kafka-clients to 3.3.2
- proguard-base to 7.3.1
- pulsar4s-akka-streams to 2.9.0
- rocksdbjni to 7.9.2
- sbt-jmh to 0.4.4
- scalajs-compiler to 1.13.0
  
---

## [:octicons-tag-24: Release 1.5.1](https://github.com/thatdot/quine/releases/tag/v1.5.1)

### Enhancements

- Switched JSON parser from `ujson` to `circe` to avoid loss of precision in large integers in standing queries (Further support for large integers coming soon)
- REST API documentation style and readability updates
- Added documentation for graph AI use cases
- Improved performance of Cassandra persistor by reducing load and latency during ingest

### Bugfixes

- Reduced log noise when performing more than 2 operations on a node as part of a single message

---

## [:octicons-tag-24: Release 1.5.0](https://github.com/thatdot/quine/releases/tag/v1.5.0)

### Enhancements

- Many API endpoints now support YAML request bodies: Convenient for copying pipelines from recipes to a Quine server!
- AWS client configurations (SNS, Kinesis, etc) separated "region" from "credentials"
- Added "Graph Algorithms" category to the REST API
- Added `random walk` graph algorithm (compatible with Node2Vec and GraphSAGE walk generation), REST endpoint, and Cypher procedure
- Administration REST API documentation style and readability updates
- Non-historical nodes can now be kept awake for a minimum duration after each message
- Kafka ingests may now include a settings block for the Kafka client, e.g., to specify credentials. Contributed by @charliemenke
- No longer require all fields to be specified in recipes, improving readability
- Performance improvements in MapDB, Cassandra, DistinctId standing queries
- Improve indexing of node edge collections for increased query performance

### Bugfixes

- Switched JSON parser from `ujson` to `circe` to avoid loss of precision in large integers in ingest streams (Further support for large integers coming soon)
- Resolved issues with clean shutdown of the Quine application via the API

---

## [:octicons-tag-24: Release 1.4.2](https://github.com/thatdot/quine/releases/tag/v1.4.2)

### Enhancements

- Added canonicalized mapping of QuineIds and DGBs to reduce heap usage
- Simplified node wakeup protocol
- Improved contributor experience when building with node version greater than 16
- Now support amd64 and arm64 architectures for Quine docker image

### Bugfixes

- Node debug endpoint / procedure now includes all DistinctId localEventIndex entries instead of just one

---

## [:octicons-tag-24: Release 1.4.1](https://github.com/thatdot/quine/releases/tag/v1.4.1)

### Bugfix

- Edge case where Akka complains of an actor name being re-used too quickly
is now retried.

---

## [:octicons-tag-24: Release 1.4.0](https://github.com/thatdot/quine/releases/tag/v1.4.0)

### Enhancements

- The `reify.time` procedure now yields only the finest-granularity reified period node for better query ergonomics
- Removed `nullary` and `unary` `locIdFrom` variants to simplify `locIdFrom` usage
- Enriched logging in edge cases involving shard resolution
- Restructured ingest streams for better consistency and handling of edge-cases involving charset transcoding
- Optimized storage of `DateTime` to use less space
- Simplified node wakeup protocol: edge cases involving simultaneous request to sleep and wake should now be more efficient
- Improved API documentation of ingest streams
- Add notification preview text for rich Slack messages used as Standing Query outputs
- Provide a warning when a full-node scan is detected in ingest query
- Added `text.urlencode` and `text.urldecode` Cypher functions
- Added atomic "count" return value to `incrementCounter` procedure
- Added support for decoding steps during ingest (base64, zlib, and gzip)
- Standing Queries can now use idFrom-based ID constraints, provided that all arguments to the idFrom are literal values
- Rewrote the serialization, persistence, and message passing system used by DistinctId Standing Queries
- Change default order of node effects to "persister first", applying in-memory effects only after the persistence operation succeeds
- Removed unused / experimental "node merging" behavior

### Bugfixes

- Cassandra persister batched writes now respect configured timeout and consistency options
- Long-typed QuineIdProviders will now interoperate correctly with the `getHost` procedure
- singleton-snapshot and journals may now be enabled at the same time
- Added a heuristic rate limiter to Kinesis ingest streams to improve stability when Quine reads faster than Kinesis can emit
- Improved cluster stability when cluster members experience temporary disconnections
- Improved shutdown behavior in failsafe case
- UUIDv5 and UUIDv3 id providers now generate correctly bit-masked identifiers
- MultipleValues standing queries are now properly restored on nodes where shutdown is started and then cancelled
- Node edge and property counts will now be correctly reflected in the metrics dashboard
- Improved use case for reify time to ensure reified time nodes have all properties set

### Updates

- Replaced `snakeyaml` with `snakeyaml-engine`, supporting YAML 1.2
- akka to 2.6.20 (contributed by @He-Pin)
- bootstrap to 5.2.2
- cassandra client to 4.15.0
- endpoints4s to 1.8.0
- endpoints4s (akka-http) to 7.0.0
- endpoints4s (openApi) to 4.2.0
- endpoints4s (xhr) to 5.1.0
- jquery to 3.6.1
- netty-nio-client to 2.17.288
- parboiled to 1.4.1
- scalajs to 1.11.0
- rocksdb to 7.6.0
- scalajs-dom to 2.3.0
- flatbuffers-java to 22.10.26
- dropwizard metrics to 4.2.12
- logback to 1.4.4
- apache commons to 1.10.0
- protobuf-java to 3.21.8
- Novelty: graal to 22.2.0
- Memeid4s to 0.7.0

---

## [:octicons-tag-24: Release 1.3.2](https://github.com/thatdot/quine/releases/tag/v1.3.2)

Release Quine 1.3.2

### Enhancements

- Improved logging of error and startup messages

### Bugfixes

- RocksDB unit tests will now clean up files they make (contributed by @dennylee)
- text.utf8Decode now accepts a 1-argument invocation

### Updates

- sbt to 1.7.1
- scala-collection-compat to 2.8.1
- scalatest to 3.2.13

---

## [:octicons-tag-24: Release 1.3.1](https://github.com/thatdot/quine/releases/tag/v1.3.1)

### Enhancements

- Added support for `QUINE_DATA` environment variable, which can be set to define the default persistence path.
- Improved consistency of prose in WARN and INFO log messages
- Improved Cypher query execution time

### Bugfixes

- Fixed an issue where messages output while running a recipe could be duplicated

### Updates

- classgraph to 4.8.149
- sbt to 1.7.0
- scala-collection-compat to 2.8.0
- scala-js-macrotask-executor to 1.1.0
- scopt to 4.1.0
- memeid4s to 0.6.0
- msgpack-core to 0.9.3

---

## [:octicons-tag-24: Release 1.3.0](https://github.com/thatdot/quine/releases/tag/v1.3.0)

### Features

- Added a pagination (SKIP/LIMIT) optimizer to the Cypher query engine for historical queries with no unaliased values
- Enabled journals by default when running a recipe
- Added support for using the Stoplight Elements interactive documentation behind an authentication proxy

### Bugfixes

- Fixed an issue where waking up a node would not correctly re-register its standing queries, potentially resulting in dropped results
- Fixed an issue where Cypher subqueries could be executed with too many variables in scope
- Fixed an issue where some Cypher constructs (notably: variable-length relationship patterns) could be executed with too many variables in scope
- Fixed a documentation rendering issue for Standing Query Outputs
- Renamed the metric "persistors.snapshot-sizes" to "persistor.snapshot-sizes" for consistency
- Fixed the behavior of DISTINCT during Cypher query execution, making it work correctly with SKIP and/or LIMIT

### Misc

- Simplified startup log messages
- Update some error messages to use the correct name for DistinctId Standing Queries
- Improved UX for API-issued historical queries near the present time
- Removed logback-config logging library: to configure logging, use standard logback.xml
- Added timestamps to node journal events in debug.node and node debug APIs
- Removed StandingQueryPattern.Graph API
- Improved distribution of randomly-generated partitioned IDs
- Documented metrics endpoint in openapi specification
- Added peephole optimization for property value comparsion
- Refactored to simplify DomainGraphBranch representation

### Updates

- rocksdbjni to 7.3.1
- msgpack-core to 0.9.2
- cats-core to 2.8.0
- metrics to 4.2.10
- scala-library to 2.12.16
- sbt-paradox to 0.10.2
- sbt-scalafix to 0.10.1
- scala-java-time to 2.4.0

---

## [:octicons-tag-24: Release 1.2.1](https://github.com/thatdot/quine/releases/tag/v1.2.1)

### Features

- Added `debug.sleep` Cypher procedure to request a node sleep
- Replaced swagger-ui REST API viewer with Stoplight Elements
- Integrated recipe-like command-line arguments with main configuration, allowing command-line arguments when running the main Quine application and reflecting those settings in the config REST API

### Bugfixes

- Fixed a bug where some node journal events were not deduplicated, improving throughput and reducing disk usage
- Fixed a minor typo in command-line argument parsing
- VOID procedures called as the last clause in a Cypher query now return 0 rows instead of 1 empty row

### Misc

- Switched to target JRE 11 during compilation, allowing more runtime optimizations
- Made process ID lookup more reliable cross-platform

### Updates

- rocksdbjni to 7.2.2

---

## [:octicons-tag-24: Release 1.2.0](https://github.com/thatdot/quine/releases/tag/v1.2.0)

This release brings several API improvements, new Cypher query features, and persistence backend bugfixes and optimizations to better support supernodes and other extreme-scale datasets. Some of these changes affect the data format on disk, and accordingly data from prior versions (<= 1.1.2) can not be used with Quine 1.2.0 without migration. Furthermore, this release increases the minimum supported Java version to 11.

### Features

- Add `reify.time`, a Cypher procedure to generate a uniform structured representation of timestamp data within the graph, to make time-related data analysis via Standing Queries easier and more consistent
- Add a REST API endpoint to request a specific node save a snapshot and go to sleep
- Added support for Cypher subqueries via the `CALL {}` syntax
- EXPLAINed query ASTs will now be logged at debug level, if enabled
- Improved serialization for nodes with an extremely large edge and/or property counts in the persistence backend
- Nodes with an extremely large edge and/or property count can now be correctly accessed via the Literal Operations REST APIs
- Iteration and sort order of Map-type values in Cypher queries are handled consistently
- Add a configuration option for writing updates to disk before applying effects in-memory: `quine.persistence.effect-order`
- Add a startup check to verify that Cassandra's configuration matches the provided Quine configuration, warning if there is a mismatch
- Calling the debug API on a node in a historical query now only includes journal events up to the time of the historical query

### Bugfixes

- Application start up can no longer log an opaque warning like "Current state = CODING, previous state = FLUSHED"
- Setting snapshot-singleton=true, snapshot-schedule=on-node-update, and journal-enabled=false no longer causes the most recent event on a node to be dropped
- Map-type values in Cypher hash to a consistent value, independent of how they were constructed
- Accessing nodes just as they fall out of the cache can no longer cause the system to enter a failsafe mode
- Historical queries that restore journals with noncommutative effects after a snapshot from Cassandra apply updates in order
- Fix an issue with RocksDB where some Standing Query states weren’t restored
- Added a minimum header width to recipe status query output
- Fix an issue where standing queries were not properly restored on startup

### Misc

- Remove support for DistinctId queries that do not specify a `DISTINCT` keyword
- Remove support for `toInt` in Cypher queries (use `toInteger` instead)
- Remove support for `filter` and `extract` in Cypher queries (use list comprehensions instead)
- ID values returned via the exploration UI are now pretty-printed
- Use Cypher for query UI edge queries for better performance and fewer timeouts when querying supernodes
- Make recipes and example queries more consistent in styling
- Make use of `datetime()` in Apache log recipe to parse timestamps
- When the system clock moves backwards, Quine handles the issue transparently. Change the log level for this occasion from `warn` to `info`.
- Rename `executionGuarantee` parameter on the CypherQuery Standing Query Output to `shouldRetry`, and set the default to true
- Increase resilience of persistence operations in deployments with unreliable persistors
- Improve performance of historical queries involving nodes with many edges
- Add persistor exceptions, including timeouts, to the list of exception types that are retried for ingest queries and when `shouldRetry = true`
- Rename Cassandra store options insert-time and select-timeout to write-timeout and read-timeout, respectively

### Updates

- akka-http, akka-http-xml to 10.2.9
- protobuf-java to 3.20.0
- rocksdbjni to 7.0.3
- ujson, upickle to 1.6.0
- shapeless to 2.3.9
- memeid4s to 0.5.0
- scalajs to 1.10.0
- scalatest to 3.2.12
- scalacheck to 1.16.0
- sbt-scalafix to 0.10.0
- classgraph to 4.8.146

---

## [:octicons-tag-24: Release 1.1.2](https://github.com/thatdot/quine/releases/tag/v1.1.2)

### Features

- Add support for additional Cypher aggregation functions: stDev, stDevP, percentileCont, and percentileDisc (#1629)
- Add an option to retry CypherQuery type Standing Query outputs (#1642)
- Add support for RocksDB on M1 Macs (#1659)
- Add a metric for "shared.valve.ingest" the "gauges" section of the `GET /api/v1/admin/metrics` API endpoint to report the backpressured status of ingest streams (#1653)
- Report all recipe substitution errors at once (#1650)

### Bugfixes

- Remove references to an endpoint that no longer exists (#1619)
- Retry loading application state in case of persistor failure (#1631)
- Print live-updating status messages for Recipes in a consistent order (#1609)
- Remove a duplicate copy of labels in certain queries for nodes (#1636)
- Enable ingest to recover from temporary failures (eg as caused by a flaky network) (#1627)

### Misc

- Rename "importers" package to "ingest" (#1621)
- Remove unused fields from build info API endpoint (#1554)
- Add additional context to some log messages (#1639)
- Log a warning when a node approaches becoming a supernode (#1641)
- Improve performance of some API endpoints (#1612)
- Update Akka version to 2.6.19 (#1656)

---

## [:octicons-tag-24: Release 1.1.0](https://github.com/thatdot/quine/releases/tag/v1.1.0)

### Features

- Add Websockets ingest adapter
- Add stdin ingest adapter
- Add pretty-printing of Status Query results to Recipe interpreter
- Add a "Run again as text query" link to the Query UI
- Add a static check to the Cypher query planner to identify queries which cannot fail

### Bugfixes

- Fix a race condition in the REST API which could cause multiple simultaneous API calls to report incorrect status, even when the system behaves correctly
- Fix cross-application config loading and re-add clustering config to Quine Enterprise
- Add extra checks to ensure failure to wake up a node does not leave the system in an inconsistent state
- Fixed a Cypher bug involving spurious errors during UDF compilation

### Misc

- Added OSS CI
- Fix 2.13 build for projects with 2.13 cross build
- Fix a typo
- Dependency updates: webjars-locator to 0.45; slinky to 0.7.2; classgraph to 4.8.141; akka-http, akka-http-xml to 10.2.8; scalajs to 1.9.0; sbt-buildinfo to 0.11.0; sbt-assembly to 1.2.0; proguard-base to 7.2.1

---

## [:octicons-tag-24: Release 1.0.0](https://github.com/thatdot/quine/releases/tag/v1.0.0)

Initial release <https://quine.io>


---
# Core Concepts
URL: https://quine.io/core-concepts/

# Core Concepts

## Graph Model for Data and Computation

Quine is built around a novel design choice: **to represent both the data model and the computational model using a graph**.

In fact, the same graph is used for both models. Using the graph structure for both the data model and the computational model allows Quine to provide remarkable new capabilities including the ability to ingest events from multiple source, materialize complex relationships as a single graph, and emit results with sub-millisecond latency.

### Data Model

A graph consists of nodes connected by edges. Each node is unique from other nodes and has a unique ID to distinguish it.

Nodes contain additional data from key-value pairs stored as properties on that node. In practice, this makes a node similar to a JSON object or a Python dictionary: it stores any arbitrary data.

Nodes are connected to other nodes by edges. This `Node-Edge-Node` pattern that makes up a graph is analogous to the `Subject-Predicate-Object` form used by most human languages.

This pattern makes a graph profoundly expressive, allowing it to represent both data and interrelationships between that data that might otherwise be stored in other systems like a relational database or NO-SQL document store.

The graph model surfaces the interconnected nature of data as a core part of the data model. It also provides the ideal sweet spot between structured data that can be efficiently traversed and schema-less flexibility for fitting in new data, which wasn't considered when the system was first set up.

### Computational Model

In the Quine system, nodes in the graph are backed by actors as needed to perform computation.

An actor is a lightweight computational unit, similar to a thread, but designed to be highly resource-efficient so that thousands or even millions of actors can run simulateneously in the same system.

Actors supervise other actors in a managed hierarchy to ensure that failure is well-contained and the system remains resilient in the presence of failure.

This mechanism, known as <a target="_blank" href="https://en.wikipedia.org/wiki/Actor_model">the Actor Model</a>, goes back to the 1970s and was a vital aspect of the resilient computer systems that made up the telecommunications infrastructure of the 1980s and 1990s. These brilliant ideas remained dormant for decades until the massive parallelism and complexity of modern Internet-scale computing have driven many cutting-edge tools to revive this battle-tested method for building robust, scalable systems.

## Selecting Nodes vs Searching for Nodes

With a graph data model, nodes are the primary unit of data — much like a "row" is the primary unit of data in a relational database. However, unlike traditional graph data systems, a Quine user never has to create a node directly. Instead, the system behaves as if all nodes exist but don't yet contain meaningful data.

> As data streams into the system, the node acquires meaningful data, and Quine begins to create a history for the node.

Quine adds an `idFrom` function to Cypher that takes any number of arguments and deterministically produces a node ID from that data. This is similar to a consistent-hashing strategy, except that the ID produced by this function is always a Quine node ID that matches the selected ID type in the configuration.

The `idFrom` function simplifies the selection of nodes within Cypher queries, enabling MATCH to assume the form:

```cypher
MATCH (n) WHERE id(n) = idFrom($that) SET n.line = $that
```

For guidance on designing graph structures and queries, see [Data Modeling and Query Design](data-modeling.md).

## Historical Versioning

By default, each node in the graph maintains a record of its historical changes over time. When a node's properties or edges change, the change event and timestamp are saved to an append-only log for that particular node. This technique is known as [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html), applied individually to each node. This historical log can be replayed up to any desired moment in time, allowing for the system to quickly answer questions using the state of the graph *as it was in the past.*

!!! note

    When you delete a node, it's not entirely gone, but instead, its history is updated to indicate that all its properties and edges were removed when the deletion occurred. At any given moment, if a node is empty, it is essentially the same as a non-existent node - the only difference being that because the node has a history, it will be included in the count returned by queries that perform a full node scan like MATCH (a) RETURN count(a).

## Real-Time Pattern Matching With Standing Queries

Work in Quine is done primarily through the expression of patterns to be found while incoming streams of data are being processed. First, you must declare the shape of data (the nodes and edges that will define your initial graph state). Then you should determine the sub-graph representing the interesting or valuable pattern of events that you want to find in the stream using Cypher. Finally, you will define the action you want to be triggered each time that pattern is found (this is accomplished using a standing query). The matching and discovery of every instance of each pattern happen automatically, and the corresponding data is used to trigger the desired action.

The action triggered may be one of many possible options upon matching a pattern.

* An update or annotation of the existing data can be triggered to enrich the data.
* An external service can be queried, and the result is fed into this system as another new event.
* The matching data can be packaged up (e.g., as a JSON object) and published to another message queue to be consumed by another service.
* Custom queries, algorithms, or code can even be triggered using data from the matching pattern to execute arbitrary actions.

## Backpressure

Inevitably, when streaming data producers outpace consumers, the consumer will become overwhelmed. Quine manages the data flow to avoid becoming overwhelmed using "backpressure."

The problem with buffering is that a buffer will eventually run out of space. The system must then decide what to do when the buffer is full: drop new results, drop old results, crash the system, or backpressure.

A backpressured system does not buffer, and it causes producers upstream to not send data at a rate greater than it can process. 

Backpressure is a [protocol](https://www.reactive-streams.org/) that defines how to send a logical signal back UP the stream with information about the downstream consumers readiness to receive more data. If downstream is not ready to consume, then upstream does not send new data.

Quine uses a reactive stream implementation of backpressure, [Pekko Streams](https://pekko.apache.org/docs/pekko/current/stream/stream-flows-and-basics.html#core-concepts), built on top of the actor model to ensure that the ingestion and processing of streams are resilient.

## Stateful Event-Driven Computation

All together, Quine is able to:

* Consume high-volume streaming event data
* Convert it into durable, versioned, connected data
* Monitor that connected data for complex structures or values
* Trigger arbitrary computation on the event of each match

This collection of capabilities is profoundly powerful. It represents a complete system for stateful event-driven arbitrary computation in a platform scalable to any size of data or desired throughput.

## Everything Needed Between Streams

Quine represents a significant new architectural component for enterprise data pipelines.

Quine is a stateful streaming graph interpreter. It consumes high volume data streams and publishes processed results to other streaming data consumers.

Quine eliminates the complex technical challenges of managing data ordering, time windowing, vertical and horizontal scalability, and the complex asynchronous processing needed to find compound objects or patterns spread across data streams.

Quine is easily integrated into existing [data pipelines](streaming-systems.md) and highly scalable across existing and next-generation enterprise infrastructure.


---
# Delivery Guarantees
URL: https://quine.io/core-concepts/delivery-guarantees/

# Delivery Guarantees

Quine provides different delivery guarantees for different parts of the data pipeline. Understanding these guarantees helps you design systems with appropriate reliability characteristics.

| Component                  | Guarantee     |
|:---------------------------|:--------------|
| **Ingest**                 | At-Least-Once |
| **Standing Query Outputs** | Best Effort   |

## Ingest: At-Least-Once

Ingest streams provide **at-least-once** delivery guarantees when properly configured. This means:

- Every ingested record will be processed at least once
- In failure scenarios (crash, restart), some records may be reprocessed
- No records will be silently dropped

### How It Works

Quine achieves at-least-once semantics by committing source offsets **after** successfully writing data to the graph:

```
Source → Deserialize → Write to Graph → Commit Offset
```

If Quine crashes after writing but before committing, the record will be reprocessed on restart. This is the standard trade-off for at-least-once delivery.

To achieve at-least-once guarantees, you need a source that supports offset tracking (such as Kafka, SQS, or Kinesis with KCL) configured to commit offsets only after successful processing.

### Idempotent Ingest Queries

Because at-least-once delivery may result in duplicate processing, your ingest queries should be **idempotent**. Processing the same record twice should produce the same result. The [`idFrom()` function](id-provider.md#idfrom) helps achieve this by generating deterministic node IDs from your data.

## Standing Query Outputs: Best Effort

Standing query outputs are delivered on a **best-effort** basis. While Quine includes [backpressure](index.md#backpressure) mechanisms to prevent result loss during normal operation, results can be dropped under certain conditions.

### Backpressure Mechanism

When standing query results are produced faster than outputs can consume them, Quine applies backpressure by pausing ingest. This keeps the result queue from growing unbounded and allows outputs to catch up.

However, the result queue has a maximum size. If the queue fills completely, new results are dropped rather than buffered indefinitely. This can happen in pathological cases such as patterns that trigger cascading matches.

### When Results May Be Lost

Standing query results can be lost in these scenarios:

- **Queue overflow**: New results are dropped when the result queue is full
- **Output failure**: If an output destination fails, the standing query is cancelled
- **Process restart**: In-flight results in the queue can be lost on restart

### Designing for Best-Effort Delivery

When building systems that consume standing query outputs:

1. **Treat outputs as notifications** rather than authoritative records
2. **Design for duplicate delivery**: The same result may be emitted more than once in edge cases
3. **Consider downstream buffering**: If guaranteed delivery is required, route outputs through a message queue that provides stronger guarantees


---
# Operational Considerations
URL: https://quine.io/core-concepts/operational-considerations/

# Operational Considerations

## Deploying: Basic Resource Planning

Quine is extremely flexible in its deployment model. It is robust and powerful enough to run on a server with hundreds of CPU cores and terabytes of RAM, but also small and lightweight enough to run in a container or on a Raspberry Pi. Custom deployment configurations (e.g. on-premise deployment, non-containerized deployments, alternate storage configurations, etc.) are available upon request.

Quine is a backpressured system; it will deliberately slow down when resource-constrained. This is a highly desirable quality because the only alternative is to crash the system. Making Quine run faster is essentially a matter of allocating resources to the machine/environment so that the slowest component is not a bottleneck.

### CPU

As a distributed system, Quine is designed from the ground up to take advantage of multiple CPU cores. Quine will automatically configure increased parallelism where there are more CPU cores available. This configuration can be customized using the Pekko configuration settings for the `default-dispatcher` when starting up Quine.

#### High CPU Utilization

By design, we expect Quine to utilize as many resources as it is allotted. High CPU usage alone, without additional symptoms, is desirable and reflects good utilization of the system.

#### Low CPU Utilization

Quine is designed to use backpressure to limit incoming data from producers. If any aspect of the graph is overwhelmed (e.g. the data storage layer is slow), Quine will backpressure incoming data to ensure all work can complete. This mechanism provides data safety by not dropping incoming records, and system resilience by not sending more data than the graph can handle. To understand if the ingest is backpressuring, view the `shared.valve.ingest.{ingest-name}.metric` to see if the value is increasing. A higher value means more backpressure.

If the ingest stream `parallelism` setting is too low, the system may not take advantage of all the parallel processing it is capable of. This setting determines how many items from the ingest source will be causing their effects in parallel. Too low of a `parallelism` setting will cause the CPU to be underutilized.

### RAM

System memory (RAM) is used by Quine primarily to keep nodes in the graph warm in the cache. When a node required for a query or operation is needed, if it is live in the cache, then calls to disk can often be entirely avoided. This significantly increases performance and throughput of the system overall. So in general, the more nodes live in the cache, the faster the system will run.

!!! note

    Quine uses a novel caching strategy that we call "semantic caching". The net result is that there is a decreasing benefit to having additional nodes kept in memory. There is only a performance improvement if those nodes are needed.

Since Quine runs in the JVM, you can have Quine use additional system RAM in the standard Java fashion by increasing the maximum heap size with the `-Xmx` flag when starting the application. E.g.: `java -Xmx6g -jar quine-2.1.1.jar`. In order for Quine to receive a benefit from increased RAM, you will probably also need to increase the limit of nodes kept in memory per shard by changing the `in-memory-soft-node-limit`, and possibly the `in-memory-hard-node-limit` sizes. See the [Configuration Reference](../reference/config/configuration.md) for details. The ideal setting for these limits depends on the use case and will vary. We suggest monitoring usage with the tools described below to determine the best settings for your application. In-memory node limits can be changed dynamically from the built-in REST API with the `shard-sizes` endpoint.

!!! warning

    Keep in mind that setting the maximum heap size is not the same as limiting the total memory used by the application. Some amount of additional off-heap memory will be used when the system is operating normally. Off-heap memory usage is usually trivial, but can become significant in some cases, e.g.: Lots of network traffic, or using memory mapped files. The latter case can become very significant when using the MapDB persistor, which defaults to using memory mapped files for local data storage.

#### Out of Memory Error

Quine can be configured to use any amount of RAM. The settings for `in-memory-soft-node-limit` and `in-memory-hard-node-limit` should be adjusted according to the workload in proportion to the amount of memory available on the system. If you encounter an Out-of-Memory error, we suggest lowering the soft limit. This setting can be adjusted while the system is running via an API call.

### Storage

Quine stores data using one or more [Persistor](../learn/persistors/index.md). Persistors are either local or remote. A local persistor will store data on the same machine where Quine is running, at a location in the filesystem determined by its configuration. A remote persistor will store data on a separate machine. By default, Quine will use the local RocksDB persistor.

For production deployments, we recommend using a remote persistor like [Cassandra](../learn/persistors/index.md#cassandra) to provide data redundancy and high-availability. Additionally, old data can be expired from disk using Cassandra's time-to-live setting in order to limit the total size of data stored.

## Monitoring

It is often helpful to watch an operating system to confirm it is behaving as anticipated. Quine includes multiple mechanisms for monitoring a running system.

For a set of production alert recommendations built on these metrics (what to watch and at what warning/critical levels), see [Recommended Alerts](../learn/metrics/recommended-alerts.md).

### Web Browser

The Quine web UI includes a metrics page (`/metrics`) with live-updating reports on memory usage, nodes in the cache (per shard), basic histograms of the number of properties and edges per node, and latency measurements of various important operations.

![Metrics](./core-concepts-images/metrics-oss.png)

### REST API

Quine serves a REST API from its built-in web server. That web server provides separate [Process Readiness: `GET /api/v2/system/readiness`](/reference/rest-api/?av=v2#/operations/get-readiness) and [Process Liveness: `GET /api/v2/system/liveness`](/reference/rest-api/?av=v2#/operations/get-liveness) API endpoints for monitoring the status of the Quine system. Detailed metrics can be retrieved from the [Metrics: `GET /api/v2/system/metrics`](/reference/rest-api/?av=v2#/operations/get-metrics) endpoint.

### JMX

Quine can be monitored and controlled live using JMX. There is a wide array of tools available for monitoring JVM applications through JMX. If you don't already have a preference, you might start with [VisualVM](https://visualvm.github.io).

<!--
### InfluxDB
-->

!!! tip "Quine Enterprise"

    Quine Enterprise adds clustering, failover, and Kubernetes Helm deployment. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).


---
# REST API
URL: https://quine.io/core-concepts/rest-api/

# REST API

Quine exposes a JSON REST API under the path prefix `/api/v2/`. This page describes the conventions every endpoint shares — URL shape, request and response format, error envelope, and common query parameters. For the per-endpoint reference, see the interactive [REST API Reference](/reference/rest-api/?av=v2).

## Quick start

A simple first request is `/system/systemInfo`, which returns version and build information about the running server:

```bash
curl http://localhost:8080/api/v2/system/systemInfo
```

The example assumes the default `localhost:8080` bind address — substitute your own host and port if you've configured the Quine webserver differently. The response is JSON with the resource at the top level of the body. The sections below describe list and error responses, the URL conventions every endpoint follows, and the common query parameters available to most endpoints.

## Fetching the system configuration

`GET /api/v2/system/config` returns information about the startup [configuration](../reference/config/configuration.md) of Quine (e.g., persistor, webserver, shard count, node limits, metrics reporters, default API version). Not all configuration details are returned; in particular, credentials and other secrets are always excluded.

## Interactive docs

The Quine web server hosts interactive API docs at `/docs`. The UI lists every endpoint, shows full request and response schemas (including the deeply-nested options on Standing Queries and Ingest Streams), and lets you fill in parameters and run requests directly from the page.

![API Docs](./core-concepts-images/quine-apiDocs.png)

The interactive view is powered by [Stoplight Elements](https://stoplight.io/open-source/elements), which renders the OpenAPI specification served at `/docs/openapi.json`. The same spec can be fed to [OpenAPI Generator](https://openapi-generator.tech/) to produce client libraries in your language of choice.

## URL conventions

Every v2 endpoint shares the same shape. Decoding a URL like `POST /api/v2/graph/quine/ingests/{ingestName}:pause` takes only a few rules:

- **Base path** — all v2 endpoints live under `/api/v2/`.
- **Graph-scoped resources** — operations on ingests, standing queries, Cypher, and algorithms are nested under `/graph/quine/`. Quine uses a single graph named `quine`.

- **System endpoints** — administrative endpoints (config, metrics, liveness, …) live under `/system/`.
- **RPC-style actions** — actions on a resource use a colon-separated verb (`/ingests/{ingestName}:pause`, `/system:shutdown`). Plain `GET`/`POST`/`PUT`/`DELETE` cover CRUD.
- **camelCase** — path segments are camelCase (`standingQueries`, `systemInfo`), matching JSON field names.
- **Plural collections** — collection paths are plural (`/ingests`); individual resources are singular (`/ingests/{ingestName}`).

These rules come from Google's [API Improvement Proposals](https://google.aip.dev/); see [Design Principles](../reference/upgrade/migrating-from-api-v1.md#design-principles) for the AIP citation behind each rule.

## Response format

Requests and responses are JSON. Endpoints that create or update a resource also accept YAML when the request `Content-Type` is `application/yaml`. Where a request body field is a tagged union (e.g., an ingest's `format`, an output destination's `type`), the `"type": "<Kind>"` discriminator uses bare PascalCase names like `"Kinesis"` or `"S3Bucket"`, not SCREAMING_SNAKE_CASE. AIP-126's convention applies to enum values; a type discriminator identifies a schema variant, and the value matches the corresponding OpenAPI component name shown in the interactive docs.

Request bodies are validated strictly: a request containing an unrecognized or misspelled field is rejected with a `400 INVALID_ARGUMENT` error that names the offending field and lists the valid ones (for example, `Unexpected field: [nam]; valid fields: name, source, query`), instead of the field being silently ignored and a default applied. This applies to both JSON and YAML bodies across the entire v2 API.

Successful responses return the resource (or collection) directly at the top level of the body:

```json
{
  "name": "my-ingest",
  "status": "RUNNING",
  "settings": { ... }
}
```

List endpoints wrap their results in an `items` envelope (per [AIP-158](https://google.aip.dev/158)):

```json
{
  "items": [ ... ]
}
```

Every list response currently fits in a single page.

| HTTP Status    | Response Body                                              |
|:---------------|:-----------------------------------------------------------|
| 200 OK         | Resource or collection                                     |
| 201 Created    | Created resource (may include `Warning` header)            |
| 202 Accepted   | Empty body                                                 |
| 204 No Content | Empty body                                                 |

## Error responses

Errors are returned as a structured `ApiError` envelope shaped after [AIP-193](https://google.aip.dev/193) / [`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto):

```json
{
  "error": {
    "code": 404,
    "status": "NOT_FOUND",
    "message": "Ingest stream 'my-ingest' does not exist",
    "details": []
  }
}
```

The `error` object contains:

| Field     | Description                                                                 |
|:----------|:----------------------------------------------------------------------------|
| `code`    | HTTP status code (e.g., `400`, `404`, `500`)                                |
| `status`  | Canonical AIP-193 status string (e.g., `NOT_FOUND`, `INVALID_ARGUMENT`, `INTERNAL`) |
| `message` | Human-readable error description                                            |
| `details` | Array of additional error details (`ErrorInfo`, `Help`, or `RequestInfo`)   |

Switch on `status` rather than parse `message`. `details[]` is a closed union of `ErrorInfo` (machine-readable classification with a stable `reason` code like `CYPHER_ERROR`), `Help` (free-form hints), and `RequestInfo` — `RequestInfo.requestId` lets you locate the matching server-side log entry for a 500 response.

## Query parameters

Common query parameters available across endpoints. Timestamps and durations follow [AIP-142](https://google.aip.dev/142): RFC 3339 timestamps and Go-style duration strings.

| Parameter     | Description                                                                    | Example                              |
|:--------------|:-------------------------------------------------------------------------------|:-------------------------------------|
| `atTime`      | Historical query at a specific time (RFC 3339 timestamp)                       | `?atTime=2026-04-27T15:30:00Z`      |
| `timeout`     | Operation timeout as a duration string                                         | `?timeout=20s`                       |

## Versioning

`/api/v2/` is the current API and is what this page describes. `/api/v1/` remains mounted by default for backwards compatibility, but **API v1 is planned for deprecation and will be removed in a future release.** See [Migrating from API v1](../reference/upgrade/migrating-from-api-v1.md) for the migration guide.


---
# Streaming Systems
URL: https://quine.io/core-concepts/streaming-systems/

# Streaming Systems

A streaming system processes events -- data -- that is **continuously generated**, often in **high volumes** and at **high velocity** in real time. A streaming event source typically consists of a continuous stream of timestamped logs that record events as they happen – such as a user logging in via an identity management system, or a web server logging page requests from an online application.

![Event Streaming System](./core-concepts-images/ValueData-Quine-3.svg)

Architecturally, Quine fits into a streaming event pipeline in between, or consuming from and writing back out to, streaming event processor like Kafka or Kinesis. Quine can shape, filter, analyze, or take action based on patterns matched in the event stream.

## Streaming vs Batch Processing

Event driven applications need to process events quickly, often within the time window of a single transaction. Quine allows you to analyze a stream of events in real time and match patterns, or sub-graphs of interest, using graph database techniques and, when it finds matches, to immediately execute a pre-defined action.

Historically, event analysis is done in batches, and by definition, batch processing does not provide real-time or near real-time results. A batch system analyses large groups of events well after the events occurred, typically with a goal to produce metrics, trends, or data for AI models. The insights discovered by batch processing detail "**what did happen**" in an event stream but they are not capable of describing "**what is happening**" __while it's still happening__.

Finding relationships between events with categorical data in real-time has significant implications for cyber security, fraud detection, observability, logistics, e-commerce, and really any use case graph is both well-suited for and must process high velocity data in real time.

Properties of a streaming system:

* Runs queries and algoritms in real-time
* Treats stream as infinite -- no beginning or end
* Detects interesting patterns
* Engineered to process high volumes of data and, often, for fault tolerance

## Deployment Patterns

Quine is designed to be deployed within an event streaming data pipeline.

* A single Quine instance, properly configured, can handle an infinite stream of data. Quine is limited only by how many values are available in the QuineId type that you choose, and it's possible to choose unbounded types.

* Quine currently supports RocksDB (default), Cassandra, and MapDB for persisting ingested event data. Of course you need a storage plan with enough capacity to hold the amount of data your application needs. Cassandra allows for one Quine host and many Cassandra systems whereas RocksDB and MapDB persistors are available to use as local storage.

* Because data is effectively infinite, Quine takes a unique approach to indexing, using a custom consistent hash Cypher function (`idFrom`). Use an event value to generate a Node ID that you can use to create or locate a node in the graph. In Quine, accessing data is very fast IF you know the ID of the node in the graph that you want to query (or start a query from). The `idFrom` built-in function is a convenient convention for referring to nodes using values to deterministically calculate a `nodeId` in the the graph.

* Quine ingest is backpressured which makes it very stable in a high-volume event stream.

* Standing query output results are also delivered in a backpressured stream.

* Standing queries, a feature unique to Quine, are Cypher queries that **persist** in the graph. Standing queries monitor the stream for important patterns or sub-graphs, maintain partial matches over time intervals you define, and execute instructions (e.g. updating the graph or publishing to a Kafka topic) the instant a full match is made.

* Standing queries can be created and canceled on the fly while ingest is happening. When you create a new standing query with live data already in the system, you can control how it is applied (e.g. lazily vs. proactively).

Typical Quine deployments:

1. Consume events from a source (e.g. Kafka or Kinesis) to detect known patterns and take action.
2. Aggregate event streams from different source types and produce a new event stream.
3. Ingest from a source, transform the events (for example, parameter isolation, reduction, or enrichment) and produce a new version of the event.

## Apache Kafka

Apache Kafka is a distributed streaming platform that is used to build real-time streaming data pipelines and applications.

### Ingesting from a Kafka topic

Ingesting from a Kafka topic as an event source is done by configuring an `ingest stream` using the `KafkaIngest` type. Quine supports ingest of raw bytes, JSON data, or structured data serialized in a protocol buffer.

More information is contained in the [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) API documentation. Be sure to select **Kafka Ingest Stream** in the body section.

```json
{
  "ingestStreams": [
    {
      "name": "kafka-ingest",
      "source": {
        "type": "Kafka",
        "topics": [
          "test-topic"
        ],
        "bootstrapServers": "localhost:9092",
        "saslJaasConfig": {
          "type": "PlainLogin",
          "username": "my-user",
          "password": "my-password"
        },
        "kafkaProperties": {
          "security.protocol": "SASL_SSL",
          "sasl.mechanism": "PLAIN"
        }
      },
      "query": "MATCH (n) WHERE id(n) = idFrom($that) SET n = $that"
    }
  ]
}
```

For details on SASL authentication types and SSL password configuration, see [Secure Kafka Configuration](../learn/ingest-sources/kafka.md#secure-kafka-configuration). Credential values are automatically redacted in API responses.

### Output to a Kafka topic

When Quine is positioned upstream in a data pipeline as an event source for Kafka, Quine publishes a record for each `StandingQueryResult` to a topic configured in the `Kafka` destination. Records can be serialized as JSON or Protocol Buffers before being published to Kafka. See the [Create Standing Query Output: `POST /api/v2/graph/quine/standingQueries/{standingQueryName}/outputs`](/reference/rest-api/?av=v2#/operations/create-standing-query-output) API documentation for details. Be sure to select **Publish to Kafka Topic** in the doc.

```json
{
  "standingQueries": [
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n) RETURN DISTINCT id(n) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "toKafka",
          "destinations": [
            {
              "type": "Kafka",
              "topic": "testToKafka",
              "bootstrapServers": "localhost:9092",
              "saslJaasConfig": {
                "type": "PlainLogin",
                "username": "my-user",
                "password": "my-password"
              },
              "kafkaProperties": {
                "security.protocol": "SASL_SSL",
                "sasl.mechanism": "PLAIN"
              },
              "format": {
                "type": "JSON"
              }
            }
          ]
        }
      ]
    }
  ]
}
```

## Amazon Kinesis

Amazon Kinesis is an Amazon Web Service designed to process large-scale data streams from a multitude of services in real time.

### Ingesting from a Kinesis stream

Ingesting from a Kinesis stream as an event source is done by configuring an `ingest stream` using the `Kinesis` type. Quine supports ingest of raw bytes, JSON data, or structured data serialized in a protocol buffer.

More information is contained in the [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) API documentation. Be sure to select **Kinesis Ingest Stream** in the body section.

```json
{
  "ingestStreams": [
    {
      "name": "kinesis-ingest",
      "source": {
        "type": "Kinesis",
        "streamName": "stream-feed",
        "credentials": {
          "region": "your_aws_region",
          "accessKeyId": "your_access_key_id",
          "secretAccessKey": "your_secret"
        },
        "streamingFormat": {
          "type": "Protobuf",
          "schemaUrl": "event-view.desc",
          "typeName": "EventView"
        }
      },
      "query": "CREATE ($that)"
    }
  ]
}
```

### Output to a Kinesis stream

Quine publishes a record for each `StandingQueryResult` to a stream configured in the `Kinesis` destination. Records can be serialized as JSON or Protocol Buffers before being published to Kinesis. See the [Create Standing Query Output: `POST /api/v2/graph/quine/standingQueries/{standingQueryName}/outputs`](/reference/rest-api/?av=v2#/operations/create-standing-query-output) API documentation for details. Be sure to select **Publish to Kinesis Stream** in the doc.

```json
{
  "standingQueries": [
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n) RETURN DISTINCT id(n) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "toKinesis",
          "resultEnrichment": {
            "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN n.timestamp, n",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Kinesis",
              "streamName": "quine-output",
              "credentials": {
                "region": "your_aws_region",
                "accessKeyId": "your_access_key_id",
                "secretAccessKey": "your_secret"
              },
              "format": {
                "type": "Protobuf",
                "schemaUrl": "event-view.desc",
                "typeName": "EventView"
              }
            }
          ]
        }
      ]
    }
  ]
}
```


---
# Supported Query Languages
URL: https://quine.io/core-concepts/supported-query-languages/

# Supported Query Languages

Quine supports several different ways of querying the graph data. This includes the established Cypher query language, as well as a novel mechanism for querying future data referred to here as Standing Queries.

## Cypher

[Cypher](https://s3.amazonaws.com/artifacts.opencypher.org/openCypher9.pdf) is the most widely
use query language for interacting with data in a property graph format. It is structurally and
syntactically similar to SQL, with the main difference being the `MATCH` clause. The idea of `MATCH`
is to focus on declaratively describing the graph shape (pattern) that you want and then to let the
query compiler pick a good execution plan. What would normally require multiple `JOIN`'s in a
relational model often just reduces to one `MATCH` with a pattern that has multiple edges:

```cypher
MATCH (n: Person)-[:has_parent]->(p: Person)-[:lives_in]->(c: City)
RETURN p.name AS name, c.name AS parentsCity
```

Compare the above Cypher to the equivalent SQL below:

```sql
SELECT n.name AS name, c.name AS parentsCity
FROM persons AS n
JOIN persons AS p ON n.parent = p.id
JOIN cities AS c ON p.city = c.id
```

Cypher queries can be issued to Quine in several ways:

  * entered in the query bar of the [Exploration UI](../getting-started/exploration-ui.md)

  * sent directly through the [REST API](rest-api.md)

## Gremlin (Deprecation Planned)

!!! warning "Gremlin is only available via API v1"
    Gremlin is reachable only through API v1 endpoints. API v1 is planned for deprecation and will be removed in a future release; Gremlin support will go with it. Use Cypher instead, which provides equivalent functionality with better performance and more complete feature support. See [Migrating from API v1](../reference/upgrade/migrating-from-api-v1.md#gremlin-endpoints) for details.

[Gremlin](https://tinkerpop.apache.org/gremlin.html) is another graph query language, but one
that is less declarative and more focused on letting users specify exactly the traversal they want.
The main strength that Gremlin has is that one of its focuses is traversals: instructions for how to
walk the graph structure given some starting points.

When API v1 is configured as the default (`default-api-version = "v1"`), the [Exploration UI](../getting-started/exploration-ui.md) supports Gremlin for quick queries. With the default setting of `"v2"`, the Exploration UI uses Cypher; Gremlin remains accessible only via direct calls to the `/api/v1/query/gremlin*` endpoints.

!!! note

    Support for Gremlin in Quine is much less complete than for Cypher. Even the parts of Gremlin that
    are implemented are not guaranteed to be compliant. Part of the difficulty here is that some parts of Gremlin were designed
    to be executed from inside a host language, most frequently Groovy, and don't extend naturally to remote execution (see for instance
    [this section](https://tinkerpop.apache.org/docs/3.4.7/reference/#_the_lambda_solution_3)
    of the Gremlin manual for some complexities around anonymous functions).


---
# Cypher builtin functions
URL: https://quine.io/generated/enterprise/cypher-builtin-functions/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th></tr></thead><tbody><tr><td><code>abs</code></td><td><code>abs(NUMBER?) :: NUMBER?</code></td><td><p>absolute value of a number</p></td></tr><tr><td><code>acos</code></td><td><code>acos(NUMBER?) :: FLOAT?</code></td><td><p>arcosine (in radians) of a number</p></td></tr><tr><td><code>asin</code></td><td><code>asin(NUMBER?) :: FLOAT?</code></td><td><p>arcsine (in radians) of a number</p></td></tr><tr><td><code>atan</code></td><td><code>atan(NUMBER?) :: FLOAT?</code></td><td><p>arctangent (in radians) of a number</p></td></tr><tr><td><code>atan2</code></td><td><code>atan2(NUMBER?, NUMBER?) :: FLOAT?</code></td><td><p>arctangent (in radians) of the quotient of its arguments</p></td></tr><tr><td><code>ceil</code></td><td><code>ceil(NUMBER?) :: FLOAT?</code></td><td><p>smallest integer greater than or equal to the input</p></td></tr><tr><td><code>coalesce</code></td><td><code>coalesce(ANY?, ..) :: ANY?</code></td><td><p>returns the first non-<code>null</code> value in a list of expressions</p></td></tr><tr><td><code>cos</code></td><td><code>cos(NUMBER?) :: FLOAT?</code></td><td><p>cosine of a number of radians</p></td></tr><tr><td><code>cot</code></td><td><code>cot(NUMBER?) :: FLOAT?</code></td><td><p>cotangent of a number of radians</p></td></tr><tr><td><code>degrees</code></td><td><code>degrees(NUMBER?) :: FLOAT?</code></td><td><p>convert radians to degrees</p></td></tr><tr><td><code>e</code></td><td><code>e() :: FLOAT?</code></td><td><p>mathematical constant <code>e</code></p></td></tr><tr><td><code>exp</code></td><td><code>exp(NUMBER?) :: FLOAT?</code></td><td><p>return the mathematical constant <code>e</code> raised to the power of the input</p></td></tr><tr><td><code>floor</code></td><td><code>floor(NUMBER?) :: FLOAT?</code></td><td><p>largest integer less than or equal to the input</p></td></tr><tr><td><code>haversin</code></td><td><code>haversin(NUMBER?) :: FLOAT?</code></td><td><p>half the versine of a number</p></td></tr><tr><td><code>head</code></td><td><code>head(LIST? OF ANY?) :: ANY?</code></td><td><p>extract the first element of a list</p></td></tr><tr><td><code>id</code></td><td><code>id(NODE?) :: ANY?</code></td><td><p>extract the ID of a node</p></td></tr><tr><td><code>keys</code></td><td><code>keys(ANY?) :: LIST? OF STRING?</code></td><td><p>extract the keys from a map, node, or relationship</p></td></tr><tr><td><code>lTrim</code></td><td><code>lTrim(STRING?) :: STRING?</code></td><td><p>original string with leading whitespace removed</p></td></tr><tr><td><code>labels</code></td><td><code>labels(ANY?) :: LIST? OF STRING?</code></td><td><p>extract the labels of a node or relationship</p></td></tr><tr><td><code>last</code></td><td><code>last(LIST? OF ANY?) :: ANY?</code></td><td><p>extract the last element of a list</p></td></tr><tr><td><code>left</code></td><td><code>left(STRING?, INTEGER?) :: STRING?</code></td><td><p>string containing the specified number of leftmost characters of the original string</p></td></tr><tr><td><code>length</code></td><td><code>length(PATH?) :: INTEGER?</code></td><td><p>length of a path (ie. the number of relationships in it)</p></td></tr><tr><td><code>log</code></td><td><code>log(NUMBER?) :: FLOAT?</code></td><td><p>natural logarithm of a number</p></td></tr><tr><td><code>log10</code></td><td><code>log10(NUMBER?) :: FLOAT?</code></td><td><p>common logarithm (base 10) of a number</p></td></tr><tr><td><code>nodes</code></td><td><code>nodes(PATH?) :: LIST? OF NODE?</code></td><td><p>extract a list of nodes in a path</p></td></tr><tr><td><code>pi</code></td><td><code>pi() :: FLOAT?</code></td><td><p>mathematical constant <code>π</code></p></td></tr><tr><td><code>properties</code></td><td><code>properties(ANY?) :: MAP?</code></td><td><p>extract the properties from a map, node, or relationship</p></td></tr><tr><td><code>rTrim</code></td><td><code>rTrim(STRING?) :: STRING?</code></td><td><p>original string with trailing whitespace removed</p></td></tr><tr><td><code>radians</code></td><td><code>radians(NUMBER?) :: FLOAT?</code></td><td><p>convert degrees to radians</p></td></tr><tr><td><code>rand</code></td><td><code>rand() :: FLOAT?</code></td><td><p>random float between 0 (inclusive) and 1 (exclusive)</p></td></tr><tr><td><code>range</code></td><td><code>range(start :: INTEGER, end :: INTEGER, step :: INTEGER?) :: LIST? OF INTEGER?</code></td><td><p>construct a list of integers representing a range</p></td></tr><tr><td><code>relationships</code></td><td><code>relationships(PATH?) :: LIST? OF RELATIONSHIP?</code></td><td><p>extract a list of relationships in a path</p></td></tr><tr><td><code>replace</code></td><td><code>replace(original :: STRING?, target :: STRING?, replacement :: STRING?) :: STRING?</code></td><td><p>replace every occurrence of a target string</p></td></tr><tr><td><code>reverse</code></td><td><code>reverse(ANY?) :: ANY?</code></td><td><p>reverse a string or list</p></td></tr><tr><td><code>right</code></td><td><code>right(STRING?, INTEGER?) :: STRING?</code></td><td><p>string containing the specified number of rightmost characters of the original string</p></td></tr><tr><td><code>round</code></td><td><code>round(input :: NUMBER?, precision :: INTEGER?, mode :: STRING?) :: FLOAT?</code></td><td><p>nearest number to the input</p></td></tr><tr><td><code>sign</code></td><td><code>sign(NUMBER?) :: INTEGER?</code></td><td><p>signum of a number</p></td></tr><tr><td><code>sin</code></td><td><code>sin(NUMBER?) :: FLOAT?</code></td><td><p>sine of a number of radians</p></td></tr><tr><td><code>size</code></td><td><code>size(ANY?) :: INTEGER?</code></td><td><p>number of elements in a list or characters in a string</p></td></tr><tr><td><code>split</code></td><td><code>split(input :: STRING?, delimiter :: STRING?) :: LIST? OF STRING?</code></td><td><p>split a string on every instance of a delimiter</p></td></tr><tr><td><code>sqrt</code></td><td><code>sqrt(NUMBER?) :: FLOAT?</code></td><td><p>square root of a number</p></td></tr><tr><td><code>substring</code></td><td><code>substring(original :: STRING?, start :: INTEGER? [, end :: INTEGER? ]) :: STRING?</code></td><td><p>substring of the original string, beginning with a 0-based index start and length</p></td></tr><tr><td><code>tail</code></td><td><code>tail(LIST? OF ANY?) :: LIST? OF ANY?</code></td><td><p>return the list without its first element</p></td></tr><tr><td><code>tan</code></td><td><code>tan(NUMBER?) :: FLOAT?</code></td><td><p>tangent of a number of radians</p></td></tr><tr><td><code>timestamp</code></td><td><code>timestamp() :: INTEGER?</code></td><td><p>number of milliseconds elapsed since midnight, January 1, 1970 UTC</p></td></tr><tr><td><code>toBoolean</code></td><td><code>toBoolean(STRING?) :: BOOLEAN?</code></td><td><p>convert a string into a boolean</p></td></tr><tr><td><code>toFloat</code></td><td><code>toFloat(ANY?) :: FLOAT?</code></td><td><p>convert a string or integer into a float</p></td></tr><tr><td><code>toInteger</code></td><td><code>toInteger(ANY?) :: INTEGER?</code></td><td><p>convert a string or float into an integer</p></td></tr><tr><td><code>toLower</code></td><td><code>toLower(STRING?) :: STRING?</code></td><td><p>convert a string to lowercase</p></td></tr><tr><td><code>toString</code></td><td><code>toString(ANY?) :: STRING?</code></td><td><p>convert a value to a string</p></td></tr><tr><td><code>toUpper</code></td><td><code>toUpper(STRING?) :: STRING?</code></td><td><p>convert a string to uppercase</p></td></tr><tr><td><code>trim</code></td><td><code>trim(STRING?) :: STRING?</code></td><td><p>removing leading and trailing whitespace from a string</p></td></tr><tr><td><code>type</code></td><td><code>type(RELATIONSHIP?) :: STRING?</code></td><td><p>return the name of a relationship</p></td></tr></tbody></table>


---
# Cypher user defined functions
URL: https://quine.io/generated/enterprise/cypher-user-defined-functions/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th></tr></thead><tbody><tr><td rowspan="1"><code>bytes</code></td><td><code>bytes(input :: STRING) :: BYTES</code></td><td><p>Returns bytes represented by a hexadecimal string</p></td></tr><tr><td rowspan="1"><code>castOrNull.boolean</code></td><td><code>castOrNull.boolean(value :: ANY) :: BOOLEAN</code></td><td><p>Casts the provided value to the type Bool. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.bytes</code></td><td><code>castOrNull.bytes(value :: ANY) :: BYTES</code></td><td><p>Casts the provided value to the type Bytes. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.datetime</code></td><td><code>castOrNull.datetime(value :: ANY) :: DATETIME</code></td><td><p>Casts the provided value to the type DateTime. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.duration</code></td><td><code>castOrNull.duration(value :: ANY) :: DURATION</code></td><td><p>Casts the provided value to the type Duration. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.float</code></td><td><code>castOrNull.float(value :: ANY) :: FLOAT</code></td><td><p>Casts the provided value to the type Floating. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.integer</code></td><td><code>castOrNull.integer(value :: ANY) :: INTEGER</code></td><td><p>Casts the provided value to the type Integer. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.list</code></td><td><code>castOrNull.list(value :: ANY) :: LIST OF ANY</code></td><td><p>Casts the provided value to the type List(Anything). If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.localdatetime</code></td><td><code>castOrNull.localdatetime(value :: ANY) :: LOCALDATETIME</code></td><td><p>Casts the provided value to the type LocalDateTime. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.map</code></td><td><code>castOrNull.map(value :: ANY) :: MAP</code></td><td><p>Casts the provided value to the type Map. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.node</code></td><td><code>castOrNull.node(value :: ANY) :: NODE</code></td><td><p>Casts the provided value to the type Node. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.path</code></td><td><code>castOrNull.path(value :: ANY) :: PATH</code></td><td><p>Casts the provided value to the type Path. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.relationship</code></td><td><code>castOrNull.relationship(value :: ANY) :: RELATIONSHIP</code></td><td><p>Casts the provided value to the type Relationship. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.string</code></td><td><code>castOrNull.string(value :: ANY) :: STRING</code></td><td><p>Casts the provided value to the type Str. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.boolean</code></td><td><code>castOrThrow.boolean(value :: ANY) :: BOOLEAN</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Bool. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.bytes</code></td><td><code>castOrThrow.bytes(value :: ANY) :: BYTES</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Bytes. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.datetime</code></td><td><code>castOrThrow.datetime(value :: ANY) :: DATETIME</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type DateTime. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.duration</code></td><td><code>castOrThrow.duration(value :: ANY) :: DURATION</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Duration. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.float</code></td><td><code>castOrThrow.float(value :: ANY) :: FLOAT</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Floating. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.integer</code></td><td><code>castOrThrow.integer(value :: ANY) :: INTEGER</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Integer. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.list</code></td><td><code>castOrThrow.list(value :: ANY) :: LIST OF ANY</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type List(Anything). This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.localdatetime</code></td><td><code>castOrThrow.localdatetime(value :: ANY) :: LOCALDATETIME</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type LocalDateTime. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.map</code></td><td><code>castOrThrow.map(value :: ANY) :: MAP</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Map. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.node</code></td><td><code>castOrThrow.node(value :: ANY) :: NODE</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Node. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.path</code></td><td><code>castOrThrow.path(value :: ANY) :: PATH</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Path. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.relationship</code></td><td><code>castOrThrow.relationship(value :: ANY) :: RELATIONSHIP</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Relationship. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.string</code></td><td><code>castOrThrow.string(value :: ANY) :: STRING</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Str. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>clusterPosition</code></td><td><code>clusterPosition() :: INTEGER</code></td><td><p>Returns the cluster position occupied by this member</p></td></tr><tr><td rowspan="16"><code>coll.max</code></td><td><code>coll.max(value :: LIST OF ANY) :: ANY</code></td><td><p>Computes the maximum of values in a list</p></td></tr><tr><td><code>coll.max(input0 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td rowspan="16"><code>coll.min</code></td><td><code>coll.min(value :: LIST OF ANY) :: ANY</code></td><td><p>Computes the minimum of values in a list</p></td></tr><tr><td><code>coll.min(input0 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td rowspan="1"><code>convert.stringToBytes</code></td><td><code>convert.stringToBytes(input :: STRING, encoding :: STRING) :: BYTES</code></td><td><p>Encodes a string into bytes according to the specified encoding</p></td></tr><tr><td rowspan="4"><code>date</code></td><td><code>date() :: DATE</code></td><td><p>Get the current local date</p></td></tr><tr><td><code>date(options :: MAP) :: DATE</code></td><td><p>Construct a local date from the options</p></td></tr><tr><td><code>date(date :: STRING) :: DATE</code></td><td><p>Parse a local date from a string</p></td></tr><tr><td><code>date(date :: STRING, format :: STRING) :: DATE</code></td><td><p>Parse a local date from a string using a custom format</p></td></tr><tr><td rowspan="4"><code>datetime</code></td><td><code>datetime() :: DATETIME</code></td><td><p>Get the current date time</p></td></tr><tr><td><code>datetime(options :: MAP) :: DATETIME</code></td><td><p>Construct a date time from the options</p></td></tr><tr><td><code>datetime(datetime :: STRING) :: DATETIME</code></td><td><p>Parse a date time from a string</p></td></tr><tr><td><code>datetime(datetime :: STRING, format :: STRING) :: DATETIME</code></td><td><p>Parse a local date time from a string using a custom format</p></td></tr><tr><td rowspan="2"><code>duration</code></td><td><code>duration(options :: MAP) :: DURATION</code></td><td><p>Construct a duration from the options</p></td></tr><tr><td><code>duration(duration :: STRING) :: DURATION</code></td><td><p>Parse a duration from a string</p></td></tr><tr><td rowspan="2"><code>duration.between</code></td><td><code>duration.between(date1 :: LOCALDATETIME, date2 :: LOCALDATETIME) :: DURATION</code></td><td><p>Compute the duration between two local dates</p></td></tr><tr><td><code>duration.between(date1 :: DATETIME, date2 :: DATETIME) :: DURATION</code></td><td><p>Compute the duration between two dates</p></td></tr><tr><td rowspan="2"><code>gen.boolean.from</code></td><td><code>gen.boolean.from(fromValue :: ANY) :: BOOLEAN</code></td><td><p>Deterministically generate a random boolean from the provided input.</p></td></tr><tr><td><code>gen.boolean.from(fromValue :: ANY, withSize :: INTEGER) :: BOOLEAN</code></td><td><p>Deterministically generate a random boolean from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.bytes.from</code></td><td><code>gen.bytes.from(fromValue :: ANY) :: BYTES</code></td><td><p>Deterministically generate a random bytes from the provided input.</p></td></tr><tr><td><code>gen.bytes.from(fromValue :: ANY, withSize :: INTEGER) :: BYTES</code></td><td><p>Deterministically generate a random bytes from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.float.from</code></td><td><code>gen.float.from(fromValue :: ANY) :: FLOAT</code></td><td><p>Deterministically generate a random float from the provided input.</p></td></tr><tr><td><code>gen.float.from(fromValue :: ANY, withSize :: INTEGER) :: FLOAT</code></td><td><p>Deterministically generate a random float from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.integer.from</code></td><td><code>gen.integer.from(fromValue :: ANY) :: INTEGER</code></td><td><p>Deterministically generate a random integer from the provided input.</p></td></tr><tr><td><code>gen.integer.from(fromValue :: ANY, withSize :: INTEGER) :: INTEGER</code></td><td><p>Deterministically generate a random integer from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.node.from</code></td><td><code>gen.node.from(fromValue :: ANY) :: NODE</code></td><td><p>Deterministically generate a random node from the provided input.</p></td></tr><tr><td><code>gen.node.from(fromValue :: ANY, withSize :: INTEGER) :: NODE</code></td><td><p>Deterministically generate a random node from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.string.from</code></td><td><code>gen.string.from(fromValue :: ANY) :: STRING</code></td><td><p>Deterministically generate a random string from the provided input.</p></td></tr><tr><td><code>gen.string.from(fromValue :: ANY, withSize :: INTEGER) :: STRING</code></td><td><p>Deterministically generate a random string from the provided input.</p></td></tr><tr><td rowspan="3"><code>getHost</code></td><td><code>getHost(node :: NODE) :: INTEGER</code></td><td><p>Compute which host a node should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td><code>getHost(nodeIdStr :: STRING) :: INTEGER</code></td><td><p>Compute which host a node ID (string representation) should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td><code>getHost(nodeIdBytes :: BYTES) :: INTEGER</code></td><td><p>Compute which host a node ID (bytes representation) should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td rowspan="16"><code>hash</code></td><td><code>hash() :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td rowspan="16"><code>idFrom</code></td><td><code>idFrom(input0 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY, input15 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td rowspan="2"><code>kafkaHash</code></td><td><code>kafkaHash(partitionKey :: STRING) :: INTEGER</code></td><td><p>Hashes a string to a (32-bit) integer using the same algorithm Apache Kafka uses for its DefaultPartitioner</p></td></tr><tr><td><code>kafkaHash(partitionKey :: BYTES) :: INTEGER</code></td><td><p>Hashes a bytes value to a (32-bit) integer using the same algorithm Apache Kafka uses for its DefaultPartitioner</p></td></tr><tr><td rowspan="15"><code>locIdFrom</code></td><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td rowspan="4"><code>localdatetime</code></td><td><code>localdatetime() :: LOCALDATETIME</code></td><td><p>Get the current local date time</p></td></tr><tr><td><code>localdatetime(options :: MAP) :: LOCALDATETIME</code></td><td><p>Construct a local date time from the options</p></td></tr><tr><td><code>localdatetime(datetime :: STRING) :: LOCALDATETIME</code></td><td><p>Parse a local date time from a string</p></td></tr><tr><td><code>localdatetime(datetime :: STRING, format :: STRING) :: LOCALDATETIME</code></td><td><p>Parse a local date time from a string using a custom format</p></td></tr><tr><td rowspan="4"><code>localtime</code></td><td><code>localtime() :: LOCALTIME</code></td><td><p>Get the current local time</p></td></tr><tr><td><code>localtime(options :: MAP) :: LOCALTIME</code></td><td><p>Construct a local time from the options</p></td></tr><tr><td><code>localtime(time :: STRING) :: LOCALTIME</code></td><td><p>Parse a local time from a string</p></td></tr><tr><td><code>localtime(time :: STRING, format :: STRING) :: LOCALTIME</code></td><td><p>Parse a local time from a string using a custom format</p></td></tr><tr><td rowspan="1"><code>map.dropNullValues</code></td><td><code>map.dropNullValues(argument :: MAP) :: MAP</code></td><td><p>Keep only non-null from the map</p></td></tr><tr><td rowspan="1"><code>map.fromPairs</code></td><td><code>map.fromPairs(entries :: LIST OF LIST OF ANY) :: MAP</code></td><td><p>Construct a map from a list of [key,value] entries</p></td></tr><tr><td rowspan="1"><code>map.merge</code></td><td><code>map.merge(first :: MAP, second :: MAP) :: MAP</code></td><td><p>Merge two maps</p></td></tr><tr><td rowspan="1"><code>map.removeKey</code></td><td><code>map.removeKey(map :: MAP, key :: STRING) :: MAP</code></td><td><p>remove the key from the map</p></td></tr><tr><td rowspan="1"><code>map.sortedProperties</code></td><td><code>map.sortedProperties(map :: MAP) :: LIST OF LIST OF ANY</code></td><td><p>Extract from a map a list of [key,value] entries sorted by the key</p></td></tr><tr><td rowspan="1"><code>meta.type</code></td><td><code>meta.type(value :: ANY) :: STRING</code></td><td><p>Inspect the (name of the) type of a value</p></td></tr><tr><td rowspan="1"><code>parseJson</code></td><td><code>parseJson(jsonStr :: STRING) :: ANY</code></td><td><p>Parses jsonStr to a Cypher value</p></td></tr><tr><td rowspan="1"><code>quineId</code></td><td><code>quineId(input :: STRING) :: BYTES</code></td><td><p>Returns the Quine ID corresponding to the string</p></td></tr><tr><td rowspan="1"><code>strId</code></td><td><code>strId(input :: NODE) :: STRING</code></td><td><p>Returns a string representation of the node&rsquo;s ID</p></td></tr><tr><td rowspan="2"><code>temporal.format</code></td><td><code>temporal.format(date :: DATETIME, format :: STRING) :: STRING</code></td><td><p>Convert date time into string</p></td></tr><tr><td><code>temporal.format(date :: LOCALDATETIME, format :: STRING) :: STRING</code></td><td><p>Convert local date time into string</p></td></tr><tr><td rowspan="1"><code>text.regexFirstMatch</code></td><td><code>text.regexFirstMatch(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Parses the string <code>text</code> using the regular expression <code>regex</code> and returns the first set of capture group matches</p></td></tr><tr><td rowspan="1"><code>text.regexGroups</code></td><td><code>text.regexGroups(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Parses the string <code>text</code> using the regular expression <code>regex</code> and returns all groups matching the given regular expression in the given text</p></td></tr><tr><td rowspan="1"><code>text.regexReplaceAll</code></td><td><code>text.regexReplaceAll(text :: STRING, regex :: STRING, replacement :: STRING) :: STRING</code></td><td><p>Replaces all instances of the regular expression <code>regex</code> in the string <code>text</code> with the <code>replacement</code> string. Numbered capture groups may be referenced with $1, $2, etc.</p></td></tr><tr><td rowspan="2"><code>text.split</code></td><td><code>text.split(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Splits the string around matches of the regex</p></td></tr><tr><td><code>text.split(text :: STRING, regex :: STRING, limit :: INTEGER) :: LIST OF STRING</code></td><td><p>Splits the string around the first <code>limit</code> matches of the regex</p></td></tr><tr><td rowspan="2"><code>text.urldecode</code></td><td><code>text.urldecode(text :: STRING) :: LIST OF STRING</code></td><td><p>URL-decodes (x-www-form-urlencoded) the provided string</p></td></tr><tr><td><code>text.urldecode(text :: STRING, decodePlusAsSpace :: BOOLEAN) :: LIST OF STRING</code></td><td><p>URL-decodes the provided string, using RFC3986 if decodePlusAsSpace = false</p></td></tr><tr><td rowspan="4"><code>text.urlencode</code></td><td><code>text.urlencode(text :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string; additionally percent-encoding quotes, angle brackets, and curly braces</p></td></tr><tr><td><code>text.urlencode(text :: STRING, usePlusForSpace :: BOOLEAN) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string; additionally percent-encoding quotes, angle brackets, and curly braces; optionally using <code>+</code> for spaces instead of <code>%20</code></p></td></tr><tr><td><code>text.urlencode(text :: STRING, encodeExtraChars :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string, additionally percent-encoding the characters enumerated in <code>encodeExtraChars</code></p></td></tr><tr><td><code>text.urlencode(text :: STRING, usePlusForSpace :: BOOLEAN, encodeExtraChars :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string, additionally percent-encoding the characters enumerated in <code>encodeExtraChars</code>, optionally using <code>+</code> for spaces instead of <code>%20</code></p></td></tr><tr><td rowspan="1"><code>text.utf8Decode</code></td><td><code>text.utf8Decode(bytes :: BYTES) :: STRING</code></td><td><p>Returns the bytes decoded as a UTF-8 String</p></td></tr><tr><td rowspan="1"><code>text.utf8Encode</code></td><td><code>text.utf8Encode(string :: STRING) :: BYTES</code></td><td><p>Returns the string encoded as UTF-8 bytes</p></td></tr><tr><td rowspan="4"><code>time</code></td><td><code>time() :: TIME</code></td><td><p>Get the current local time</p></td></tr><tr><td><code>time(options :: MAP) :: TIME</code></td><td><p>Construct a local time from the options</p></td></tr><tr><td><code>time(time :: STRING) :: TIME</code></td><td><p>Parse a local time from a string</p></td></tr><tr><td><code>time(time :: STRING, format :: STRING) :: TIME</code></td><td><p>Parse a local time from a string using a custom format</p></td></tr><tr><td rowspan="1"><code>toJson</code></td><td><code>toJson(x :: ANY) :: STRING</code></td><td><p>Returns x encoded as a JSON string</p></td></tr></tbody></table>


---
# Cypher user defined procedures
URL: https://quine.io/generated/enterprise/cypher-user-defined-procedures/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th><th>Mode</th></tr></thead><tbody><tr><td><code>create.relationship</code></td><td><code>create.relationship(from :: NODE, relType :: STRING, props :: MAP, to :: NODE) :: (rel :: RELATIONSHIP)</code></td><td><p>Create a relationship with a potentially dynamic name</p></td><td>WRITE</td></tr><tr><td><code>create.setLabels</code></td><td><code>create.setLabels(node :: NODE, labels :: LIST OF STRING) :: VOID</code></td><td><p>Set the labels on the specified input node, overriding any previously set labels</p></td><td>WRITE</td></tr><tr><td><code>create.setProperty</code></td><td><code>create.setProperty(node :: NODE, key :: STRING, value :: ANY) :: VOID</code></td><td><p>Set the property with the provided key on the specified input node</p></td><td>WRITE</td></tr><tr><td><code>cypher.do.case</code></td><td><code>cypher.do.case(conditionals :: LIST OF ANY, elseQuery :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Given a list of conditional/query pairs, execute the first query with a true conditional</p></td><td>WRITE</td></tr><tr><td><code>cypher.doIt</code></td><td><code>cypher.doIt(cypher :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Executes a Cypher query with the given parameters</p></td><td>WRITE</td></tr><tr><td><code>cypher.runTimeboxed</code></td><td><code>cypher.runTimeboxed(cypher :: STRING, params :: MAP, timeout :: INTEGER) :: (value :: MAP)</code></td><td><p>Executes a Cypher query with the given parameters but abort after a certain number of milliseconds</p></td><td>WRITE</td></tr><tr><td><code>db.indexes</code></td><td><code>db.indexes() :: (description :: ANY, indexName :: ANY, tokenNames :: ANY, properties :: ANY, state :: ANY, type :: ANY, progress :: ANY, provider :: ANY, id :: ANY, failureMessage :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>db.propertyKeys</code></td><td><code>db.propertyKeys() :: (propertyKey :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>db.relationshipTypes</code></td><td><code>db.relationshipTypes() :: (relationshipType :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>dbms.labels</code></td><td><code>dbms.labels() :: (label :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>debug.node</code></td><td><code>debug.node(node :: ANY) :: (atTime :: LOCALDATETIME, properties :: MAP, edges :: LIST OF ANY, latestUpdateMillisAfterSnapshot :: INTEGER, subscribers :: STRING, subscriptions :: STRING, multipleValuesStandingQueryStates :: LIST OF ANY, journal :: LIST OF ANY, graphNodeHashCode :: INTEGER)</code></td><td><p>Returns comprehensive internal state of a node including properties, edges, standing query states, and event journal. Useful for debugging why standing queries match or don&rsquo;t match.</p></td><td>READ</td></tr><tr><td><code>debug.sleep</code></td><td><code>debug.sleep(node :: ANY) :: VOID</code></td><td><p>Request a node sleep</p></td><td>READ</td></tr><tr><td><code>do.when</code></td><td><code>do.when(condition :: BOOLEAN, ifQuery :: STRING, elseQuery :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Depending on the condition execute ifQuery or elseQuery</p></td><td>WRITE</td></tr><tr><td><code>float.add</code></td><td><code>float.add(node :: NODE, key :: STRING, add :: FLOAT) :: (result :: FLOAT)</code></td><td><p>Atomically add to a floating-point property on a node by a certain amount (defaults to 1.0), returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>getFilteredEdges</code></td><td><code>getFilteredEdges(node :: ANY, edgeTypes :: LIST OF STRING, directions :: LIST OF STRING, allowedNodes :: LIST OF ANY) :: (edge :: RELATIONSHIP)</code></td><td><p>Get edges from a node filtered by edge type, direction, and/or allowed destination nodes</p></td><td>READ</td></tr><tr><td><code>getHost</code></td><td><code>getHost(node :: NODE) :: (host :: INTEGER)</code></td><td><p>Compute which host a node is currently located on</p></td><td>READ</td></tr><tr><td><code>help.functions</code></td><td><code>help.functions() :: (name :: STRING, signature :: STRING, description :: STRING)</code></td><td><p>List registered functions</p></td><td>READ</td></tr><tr><td><code>help.procedures</code></td><td><code>help.procedures() :: (name :: STRING, signature :: STRING, description :: STRING, mode :: STRING)</code></td><td><p>List registered procedures</p></td><td>READ</td></tr><tr><td><code>incrementCounter</code></td><td><code>incrementCounter(node :: NODE, key :: STRING, amount :: INTEGER) :: (count :: INTEGER)</code></td><td><p>Atomically increment an integer property on a node by a certain amount, returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>int.add</code></td><td><code>int.add(node :: NODE, key :: STRING, add :: INTEGER) :: (result :: INTEGER)</code></td><td><p>Atomically add to an integer property on a node by a certain amount (defaults to 1), returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>loadJsonLines</code></td><td><code>loadJsonLines(url :: STRING) :: (value :: ANY)</code></td><td><p>Load a line-base JSON file, emitting one record per line</p></td><td>READ</td></tr><tr><td><code>log</code></td><td><code>log(level :: STRING, value :: ANY) :: (log :: STRING)</code></td><td><p>Log a value to the system console during query execution. Supports levels: error, warn, info, debug, trace.</p></td><td>READ</td></tr><tr><td><code>parseProtobuf</code></td><td><code>parseProtobuf(bytes :: BYTES, schemaUrl :: STRING, typeName :: STRING) :: (value :: MAP)</code></td><td><p>Parses a protobuf message into a Cypher map value, or null if the bytes are not parseable as the requested type</p></td><td>READ</td></tr><tr><td><code>purgeNode</code></td><td><code>purgeNode(node :: ANY) :: VOID</code></td><td><p>Purge a node from history</p></td><td>WRITE</td></tr><tr><td><code>random.walk</code></td><td><code>random.walk(start :: ANY, depth :: INTEGER, return :: FLOAT, in-out :: FLOAT, seed :: STRING) :: (walk :: LIST OF STRING)</code></td><td><p>Randomly walk edges from a starting node for a chosen depth. Returns a list of node IDs in the order they were encountered.</p></td><td>READ</td></tr><tr><td><code>recentNodeIds</code></td><td><code>recentNodeIds(count :: INTEGER) :: (nodeId :: ANY)</code></td><td><p>Fetch the specified number of IDs of nodes from the in-memory cache</p></td><td>READ</td></tr><tr><td><code>recentNodes</code></td><td><code>recentNodes(count :: INTEGER) :: (node :: NODE)</code></td><td><p>Fetch the specified number of nodes from the in-memory cache</p></td><td>READ</td></tr><tr><td><code>reify.time</code></td><td><code>reify.time(timestamp :: DATETIME, periods :: LIST OF STRING) :: (node :: NODE)</code></td><td><p>Reifies the timestamp into a [sub]graph of time nodes, where each node represents one period (at the granularity of the period specifiers provided). Yields the reified nodes with the finest granularity.</p></td><td>WRITE</td></tr><tr><td><code>set.insert</code></td><td><code>set.insert(node :: NODE, key :: STRING, add :: ANY) :: (result :: LIST OF ANY)</code></td><td><p>Atomically add an element to a list property treated as a set. If one or more instances of <code>add</code> are already present in the list at node[key], this procedure has no effect.</p></td><td>WRITE</td></tr><tr><td><code>set.union</code></td><td><code>set.union(node :: NODE, key :: STRING, add :: LIST OF ANY) :: (result :: LIST OF ANY)</code></td><td><p>Atomically add set of elements to a list property treated as a set. The elements in <code>add</code> will be deduplicated and, for any that are not yet present at node[key], will be stored. If the list at node[key] already contains all elements of <code>add</code>, this procedure has no effect.</p></td><td>WRITE</td></tr><tr><td><code>standing.wiretap</code></td><td><code>standing.wiretap(options :: MAP) :: (data :: MAP, meta :: MAP)</code></td><td><p>Stream live results from a running standing query. Returns data and metadata for each match.</p></td><td>READ</td></tr><tr><td><code>subscribers</code></td><td><code>subscribers(node :: ANY) :: (queryId :: INTEGER, queryDepth :: INTEGER, receiverId :: STRING, lastResult :: ANY)</code></td><td><p>Returns nodes subscribed to this node for standing query updates. Useful for tracing standing query propagation.</p></td><td>READ</td></tr><tr><td><code>subscriptions</code></td><td><code>subscriptions(node :: ANY) :: (queryId :: INTEGER, queryDepth :: INTEGER, receiverId :: STRING, lastResult :: ANY)</code></td><td><p>Returns nodes this node subscribes to for standing query updates. Useful for tracing standing query propagation.</p></td><td>READ</td></tr><tr><td><code>toProtobuf</code></td><td><code>toProtobuf(value :: MAP, schemaUrl :: STRING, typeName :: STRING) :: (protoBytes :: BYTES)</code></td><td><p>Serializes a Cypher value into bytes, according to a protobuf schema. Returns null if the value is not serializable as the requested type</p></td><td>READ</td></tr><tr><td><code>util.sleep</code></td><td><code>util.sleep(duration :: INTEGER) :: VOID</code></td><td><p>Sleep for a certain number of milliseconds</p></td><td>READ</td></tr></tbody></table>


---
# Cypher builtin functions
URL: https://quine.io/generated/quine/cypher-builtin-functions/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th></tr></thead><tbody><tr><td><code>abs</code></td><td><code>abs(NUMBER?) :: NUMBER?</code></td><td><p>absolute value of a number</p></td></tr><tr><td><code>acos</code></td><td><code>acos(NUMBER?) :: FLOAT?</code></td><td><p>arcosine (in radians) of a number</p></td></tr><tr><td><code>asin</code></td><td><code>asin(NUMBER?) :: FLOAT?</code></td><td><p>arcsine (in radians) of a number</p></td></tr><tr><td><code>atan</code></td><td><code>atan(NUMBER?) :: FLOAT?</code></td><td><p>arctangent (in radians) of a number</p></td></tr><tr><td><code>atan2</code></td><td><code>atan2(NUMBER?, NUMBER?) :: FLOAT?</code></td><td><p>arctangent (in radians) of the quotient of its arguments</p></td></tr><tr><td><code>ceil</code></td><td><code>ceil(NUMBER?) :: FLOAT?</code></td><td><p>smallest integer greater than or equal to the input</p></td></tr><tr><td><code>coalesce</code></td><td><code>coalesce(ANY?, ..) :: ANY?</code></td><td><p>returns the first non-<code>null</code> value in a list of expressions</p></td></tr><tr><td><code>cos</code></td><td><code>cos(NUMBER?) :: FLOAT?</code></td><td><p>cosine of a number of radians</p></td></tr><tr><td><code>cot</code></td><td><code>cot(NUMBER?) :: FLOAT?</code></td><td><p>cotangent of a number of radians</p></td></tr><tr><td><code>degrees</code></td><td><code>degrees(NUMBER?) :: FLOAT?</code></td><td><p>convert radians to degrees</p></td></tr><tr><td><code>e</code></td><td><code>e() :: FLOAT?</code></td><td><p>mathematical constant <code>e</code></p></td></tr><tr><td><code>exp</code></td><td><code>exp(NUMBER?) :: FLOAT?</code></td><td><p>return the mathematical constant <code>e</code> raised to the power of the input</p></td></tr><tr><td><code>floor</code></td><td><code>floor(NUMBER?) :: FLOAT?</code></td><td><p>largest integer less than or equal to the input</p></td></tr><tr><td><code>haversin</code></td><td><code>haversin(NUMBER?) :: FLOAT?</code></td><td><p>half the versine of a number</p></td></tr><tr><td><code>head</code></td><td><code>head(LIST? OF ANY?) :: ANY?</code></td><td><p>extract the first element of a list</p></td></tr><tr><td><code>id</code></td><td><code>id(NODE?) :: ANY?</code></td><td><p>extract the ID of a node</p></td></tr><tr><td><code>keys</code></td><td><code>keys(ANY?) :: LIST? OF STRING?</code></td><td><p>extract the keys from a map, node, or relationship</p></td></tr><tr><td><code>lTrim</code></td><td><code>lTrim(STRING?) :: STRING?</code></td><td><p>original string with leading whitespace removed</p></td></tr><tr><td><code>labels</code></td><td><code>labels(ANY?) :: LIST? OF STRING?</code></td><td><p>extract the labels of a node or relationship</p></td></tr><tr><td><code>last</code></td><td><code>last(LIST? OF ANY?) :: ANY?</code></td><td><p>extract the last element of a list</p></td></tr><tr><td><code>left</code></td><td><code>left(STRING?, INTEGER?) :: STRING?</code></td><td><p>string containing the specified number of leftmost characters of the original string</p></td></tr><tr><td><code>length</code></td><td><code>length(PATH?) :: INTEGER?</code></td><td><p>length of a path (ie. the number of relationships in it)</p></td></tr><tr><td><code>log</code></td><td><code>log(NUMBER?) :: FLOAT?</code></td><td><p>natural logarithm of a number</p></td></tr><tr><td><code>log10</code></td><td><code>log10(NUMBER?) :: FLOAT?</code></td><td><p>common logarithm (base 10) of a number</p></td></tr><tr><td><code>nodes</code></td><td><code>nodes(PATH?) :: LIST? OF NODE?</code></td><td><p>extract a list of nodes in a path</p></td></tr><tr><td><code>pi</code></td><td><code>pi() :: FLOAT?</code></td><td><p>mathematical constant <code>π</code></p></td></tr><tr><td><code>properties</code></td><td><code>properties(ANY?) :: MAP?</code></td><td><p>extract the properties from a map, node, or relationship</p></td></tr><tr><td><code>rTrim</code></td><td><code>rTrim(STRING?) :: STRING?</code></td><td><p>original string with trailing whitespace removed</p></td></tr><tr><td><code>radians</code></td><td><code>radians(NUMBER?) :: FLOAT?</code></td><td><p>convert degrees to radians</p></td></tr><tr><td><code>rand</code></td><td><code>rand() :: FLOAT?</code></td><td><p>random float between 0 (inclusive) and 1 (exclusive)</p></td></tr><tr><td><code>range</code></td><td><code>range(start :: INTEGER, end :: INTEGER, step :: INTEGER?) :: LIST? OF INTEGER?</code></td><td><p>construct a list of integers representing a range</p></td></tr><tr><td><code>relationships</code></td><td><code>relationships(PATH?) :: LIST? OF RELATIONSHIP?</code></td><td><p>extract a list of relationships in a path</p></td></tr><tr><td><code>replace</code></td><td><code>replace(original :: STRING?, target :: STRING?, replacement :: STRING?) :: STRING?</code></td><td><p>replace every occurrence of a target string</p></td></tr><tr><td><code>reverse</code></td><td><code>reverse(ANY?) :: ANY?</code></td><td><p>reverse a string or list</p></td></tr><tr><td><code>right</code></td><td><code>right(STRING?, INTEGER?) :: STRING?</code></td><td><p>string containing the specified number of rightmost characters of the original string</p></td></tr><tr><td><code>round</code></td><td><code>round(input :: NUMBER?, precision :: INTEGER?, mode :: STRING?) :: FLOAT?</code></td><td><p>nearest number to the input</p></td></tr><tr><td><code>sign</code></td><td><code>sign(NUMBER?) :: INTEGER?</code></td><td><p>signum of a number</p></td></tr><tr><td><code>sin</code></td><td><code>sin(NUMBER?) :: FLOAT?</code></td><td><p>sine of a number of radians</p></td></tr><tr><td><code>size</code></td><td><code>size(ANY?) :: INTEGER?</code></td><td><p>number of elements in a list or characters in a string</p></td></tr><tr><td><code>split</code></td><td><code>split(input :: STRING?, delimiter :: STRING?) :: LIST? OF STRING?</code></td><td><p>split a string on every instance of a delimiter</p></td></tr><tr><td><code>sqrt</code></td><td><code>sqrt(NUMBER?) :: FLOAT?</code></td><td><p>square root of a number</p></td></tr><tr><td><code>substring</code></td><td><code>substring(original :: STRING?, start :: INTEGER? [, end :: INTEGER? ]) :: STRING?</code></td><td><p>substring of the original string, beginning with a 0-based index start and length</p></td></tr><tr><td><code>tail</code></td><td><code>tail(LIST? OF ANY?) :: LIST? OF ANY?</code></td><td><p>return the list without its first element</p></td></tr><tr><td><code>tan</code></td><td><code>tan(NUMBER?) :: FLOAT?</code></td><td><p>tangent of a number of radians</p></td></tr><tr><td><code>timestamp</code></td><td><code>timestamp() :: INTEGER?</code></td><td><p>number of milliseconds elapsed since midnight, January 1, 1970 UTC</p></td></tr><tr><td><code>toBoolean</code></td><td><code>toBoolean(STRING?) :: BOOLEAN?</code></td><td><p>convert a string into a boolean</p></td></tr><tr><td><code>toFloat</code></td><td><code>toFloat(ANY?) :: FLOAT?</code></td><td><p>convert a string or integer into a float</p></td></tr><tr><td><code>toInteger</code></td><td><code>toInteger(ANY?) :: INTEGER?</code></td><td><p>convert a string or float into an integer</p></td></tr><tr><td><code>toLower</code></td><td><code>toLower(STRING?) :: STRING?</code></td><td><p>convert a string to lowercase</p></td></tr><tr><td><code>toString</code></td><td><code>toString(ANY?) :: STRING?</code></td><td><p>convert a value to a string</p></td></tr><tr><td><code>toUpper</code></td><td><code>toUpper(STRING?) :: STRING?</code></td><td><p>convert a string to uppercase</p></td></tr><tr><td><code>trim</code></td><td><code>trim(STRING?) :: STRING?</code></td><td><p>removing leading and trailing whitespace from a string</p></td></tr><tr><td><code>type</code></td><td><code>type(RELATIONSHIP?) :: STRING?</code></td><td><p>return the name of a relationship</p></td></tr></tbody></table>


---
# Cypher user defined functions
URL: https://quine.io/generated/quine/cypher-user-defined-functions/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th></tr></thead><tbody><tr><td rowspan="1"><code>bytes</code></td><td><code>bytes(input :: STRING) :: BYTES</code></td><td><p>Returns bytes represented by a hexadecimal string</p></td></tr><tr><td rowspan="1"><code>castOrNull.boolean</code></td><td><code>castOrNull.boolean(value :: ANY) :: BOOLEAN</code></td><td><p>Casts the provided value to the type Bool. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.bytes</code></td><td><code>castOrNull.bytes(value :: ANY) :: BYTES</code></td><td><p>Casts the provided value to the type Bytes. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.datetime</code></td><td><code>castOrNull.datetime(value :: ANY) :: DATETIME</code></td><td><p>Casts the provided value to the type DateTime. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.duration</code></td><td><code>castOrNull.duration(value :: ANY) :: DURATION</code></td><td><p>Casts the provided value to the type Duration. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.float</code></td><td><code>castOrNull.float(value :: ANY) :: FLOAT</code></td><td><p>Casts the provided value to the type Floating. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.integer</code></td><td><code>castOrNull.integer(value :: ANY) :: INTEGER</code></td><td><p>Casts the provided value to the type Integer. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.list</code></td><td><code>castOrNull.list(value :: ANY) :: LIST OF ANY</code></td><td><p>Casts the provided value to the type List(Anything). If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.localdatetime</code></td><td><code>castOrNull.localdatetime(value :: ANY) :: LOCALDATETIME</code></td><td><p>Casts the provided value to the type LocalDateTime. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.map</code></td><td><code>castOrNull.map(value :: ANY) :: MAP</code></td><td><p>Casts the provided value to the type Map. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.node</code></td><td><code>castOrNull.node(value :: ANY) :: NODE</code></td><td><p>Casts the provided value to the type Node. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.path</code></td><td><code>castOrNull.path(value :: ANY) :: PATH</code></td><td><p>Casts the provided value to the type Path. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.relationship</code></td><td><code>castOrNull.relationship(value :: ANY) :: RELATIONSHIP</code></td><td><p>Casts the provided value to the type Relationship. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrNull.string</code></td><td><code>castOrNull.string(value :: ANY) :: STRING</code></td><td><p>Casts the provided value to the type Str. If the provided value is not already an instance of the requested type, this will return null. For functions that convert between types, see <code>toInteger</code> et al. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.boolean</code></td><td><code>castOrThrow.boolean(value :: ANY) :: BOOLEAN</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Bool. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.bytes</code></td><td><code>castOrThrow.bytes(value :: ANY) :: BYTES</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Bytes. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.datetime</code></td><td><code>castOrThrow.datetime(value :: ANY) :: DATETIME</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type DateTime. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.duration</code></td><td><code>castOrThrow.duration(value :: ANY) :: DURATION</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Duration. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.float</code></td><td><code>castOrThrow.float(value :: ANY) :: FLOAT</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Floating. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.integer</code></td><td><code>castOrThrow.integer(value :: ANY) :: INTEGER</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Integer. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.list</code></td><td><code>castOrThrow.list(value :: ANY) :: LIST OF ANY</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type List(Anything). This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.localdatetime</code></td><td><code>castOrThrow.localdatetime(value :: ANY) :: LOCALDATETIME</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type LocalDateTime. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.map</code></td><td><code>castOrThrow.map(value :: ANY) :: MAP</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Map. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.node</code></td><td><code>castOrThrow.node(value :: ANY) :: NODE</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Node. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.path</code></td><td><code>castOrThrow.path(value :: ANY) :: PATH</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Path. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.relationship</code></td><td><code>castOrThrow.relationship(value :: ANY) :: RELATIONSHIP</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Relationship. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="1"><code>castOrThrow.string</code></td><td><code>castOrThrow.string(value :: ANY) :: STRING</code></td><td><p>Adds a runtime assertion that the provided <code>value</code> is actually of type Str. This can be useful to recover type information in cases where the Cypher compiler is unable to fully track types on its own. This is most common when dealing with lists, due to the limited support for higher-kinded types within the Cypher language.</p></td></tr><tr><td rowspan="16"><code>coll.max</code></td><td><code>coll.max(value :: LIST OF ANY) :: ANY</code></td><td><p>Computes the maximum of values in a list</p></td></tr><tr><td><code>coll.max(input0 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td><code>coll.max(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Computes the maximum argument</p></td></tr><tr><td rowspan="16"><code>coll.min</code></td><td><code>coll.min(value :: LIST OF ANY) :: ANY</code></td><td><p>Computes the minimum of values in a list</p></td></tr><tr><td><code>coll.min(input0 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td><code>coll.min(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Computes the minimum argument</p></td></tr><tr><td rowspan="1"><code>convert.stringToBytes</code></td><td><code>convert.stringToBytes(input :: STRING, encoding :: STRING) :: BYTES</code></td><td><p>Encodes a string into bytes according to the specified encoding</p></td></tr><tr><td rowspan="4"><code>date</code></td><td><code>date() :: DATE</code></td><td><p>Get the current local date</p></td></tr><tr><td><code>date(options :: MAP) :: DATE</code></td><td><p>Construct a local date from the options</p></td></tr><tr><td><code>date(date :: STRING) :: DATE</code></td><td><p>Parse a local date from a string</p></td></tr><tr><td><code>date(date :: STRING, format :: STRING) :: DATE</code></td><td><p>Parse a local date from a string using a custom format</p></td></tr><tr><td rowspan="4"><code>datetime</code></td><td><code>datetime() :: DATETIME</code></td><td><p>Get the current date time</p></td></tr><tr><td><code>datetime(options :: MAP) :: DATETIME</code></td><td><p>Construct a date time from the options</p></td></tr><tr><td><code>datetime(datetime :: STRING) :: DATETIME</code></td><td><p>Parse a date time from a string</p></td></tr><tr><td><code>datetime(datetime :: STRING, format :: STRING) :: DATETIME</code></td><td><p>Parse a local date time from a string using a custom format</p></td></tr><tr><td rowspan="2"><code>duration</code></td><td><code>duration(options :: MAP) :: DURATION</code></td><td><p>Construct a duration from the options</p></td></tr><tr><td><code>duration(duration :: STRING) :: DURATION</code></td><td><p>Parse a duration from a string</p></td></tr><tr><td rowspan="2"><code>duration.between</code></td><td><code>duration.between(date1 :: LOCALDATETIME, date2 :: LOCALDATETIME) :: DURATION</code></td><td><p>Compute the duration between two local dates</p></td></tr><tr><td><code>duration.between(date1 :: DATETIME, date2 :: DATETIME) :: DURATION</code></td><td><p>Compute the duration between two dates</p></td></tr><tr><td rowspan="2"><code>gen.boolean.from</code></td><td><code>gen.boolean.from(fromValue :: ANY) :: BOOLEAN</code></td><td><p>Deterministically generate a random boolean from the provided input.</p></td></tr><tr><td><code>gen.boolean.from(fromValue :: ANY, withSize :: INTEGER) :: BOOLEAN</code></td><td><p>Deterministically generate a random boolean from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.bytes.from</code></td><td><code>gen.bytes.from(fromValue :: ANY) :: BYTES</code></td><td><p>Deterministically generate a random bytes from the provided input.</p></td></tr><tr><td><code>gen.bytes.from(fromValue :: ANY, withSize :: INTEGER) :: BYTES</code></td><td><p>Deterministically generate a random bytes from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.float.from</code></td><td><code>gen.float.from(fromValue :: ANY) :: FLOAT</code></td><td><p>Deterministically generate a random float from the provided input.</p></td></tr><tr><td><code>gen.float.from(fromValue :: ANY, withSize :: INTEGER) :: FLOAT</code></td><td><p>Deterministically generate a random float from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.integer.from</code></td><td><code>gen.integer.from(fromValue :: ANY) :: INTEGER</code></td><td><p>Deterministically generate a random integer from the provided input.</p></td></tr><tr><td><code>gen.integer.from(fromValue :: ANY, withSize :: INTEGER) :: INTEGER</code></td><td><p>Deterministically generate a random integer from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.node.from</code></td><td><code>gen.node.from(fromValue :: ANY) :: NODE</code></td><td><p>Deterministically generate a random node from the provided input.</p></td></tr><tr><td><code>gen.node.from(fromValue :: ANY, withSize :: INTEGER) :: NODE</code></td><td><p>Deterministically generate a random node from the provided input.</p></td></tr><tr><td rowspan="2"><code>gen.string.from</code></td><td><code>gen.string.from(fromValue :: ANY) :: STRING</code></td><td><p>Deterministically generate a random string from the provided input.</p></td></tr><tr><td><code>gen.string.from(fromValue :: ANY, withSize :: INTEGER) :: STRING</code></td><td><p>Deterministically generate a random string from the provided input.</p></td></tr><tr><td rowspan="3"><code>getHost</code></td><td><code>getHost(node :: NODE) :: INTEGER</code></td><td><p>Compute which host a node should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td><code>getHost(nodeIdStr :: STRING) :: INTEGER</code></td><td><p>Compute which host a node ID (string representation) should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td><code>getHost(nodeIdBytes :: BYTES) :: INTEGER</code></td><td><p>Compute which host a node ID (bytes representation) should be assigned to (null if unknown without contacting the graph)</p></td></tr><tr><td rowspan="16"><code>hash</code></td><td><code>hash() :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td><code>hash(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: INTEGER</code></td><td><p>Hashes the input arguments</p></td></tr><tr><td rowspan="16"><code>idFrom</code></td><td><code>idFrom(input0 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td><code>idFrom(input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY, input15 :: ANY) :: ANY</code></td><td><p>Hashes the input arguments into a valid ID</p></td></tr><tr><td rowspan="2"><code>kafkaHash</code></td><td><code>kafkaHash(partitionKey :: STRING) :: INTEGER</code></td><td><p>Hashes a string to a (32-bit) integer using the same algorithm Apache Kafka uses for its DefaultPartitioner</p></td></tr><tr><td><code>kafkaHash(partitionKey :: BYTES) :: INTEGER</code></td><td><p>Hashes a bytes value to a (32-bit) integer using the same algorithm Apache Kafka uses for its DefaultPartitioner</p></td></tr><tr><td rowspan="15"><code>locIdFrom</code></td><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td><code>locIdFrom(positionIdx :: INTEGER, input0 :: ANY, input1 :: ANY, input2 :: ANY, input3 :: ANY, input4 :: ANY, input5 :: ANY, input6 :: ANY, input7 :: ANY, input8 :: ANY, input9 :: ANY, input10 :: ANY, input11 :: ANY, input12 :: ANY, input13 :: ANY, input14 :: ANY) :: ANY</code></td><td><p>Generates a consistent (based on a hash of the arguments) ID. The ID created will be managed by the cluster member whose position corresponds to the provided position index given the cluster topology.</p></td></tr><tr><td rowspan="4"><code>localdatetime</code></td><td><code>localdatetime() :: LOCALDATETIME</code></td><td><p>Get the current local date time</p></td></tr><tr><td><code>localdatetime(options :: MAP) :: LOCALDATETIME</code></td><td><p>Construct a local date time from the options</p></td></tr><tr><td><code>localdatetime(datetime :: STRING) :: LOCALDATETIME</code></td><td><p>Parse a local date time from a string</p></td></tr><tr><td><code>localdatetime(datetime :: STRING, format :: STRING) :: LOCALDATETIME</code></td><td><p>Parse a local date time from a string using a custom format</p></td></tr><tr><td rowspan="4"><code>localtime</code></td><td><code>localtime() :: LOCALTIME</code></td><td><p>Get the current local time</p></td></tr><tr><td><code>localtime(options :: MAP) :: LOCALTIME</code></td><td><p>Construct a local time from the options</p></td></tr><tr><td><code>localtime(time :: STRING) :: LOCALTIME</code></td><td><p>Parse a local time from a string</p></td></tr><tr><td><code>localtime(time :: STRING, format :: STRING) :: LOCALTIME</code></td><td><p>Parse a local time from a string using a custom format</p></td></tr><tr><td rowspan="1"><code>map.dropNullValues</code></td><td><code>map.dropNullValues(argument :: MAP) :: MAP</code></td><td><p>Keep only non-null from the map</p></td></tr><tr><td rowspan="1"><code>map.fromPairs</code></td><td><code>map.fromPairs(entries :: LIST OF LIST OF ANY) :: MAP</code></td><td><p>Construct a map from a list of [key,value] entries</p></td></tr><tr><td rowspan="1"><code>map.merge</code></td><td><code>map.merge(first :: MAP, second :: MAP) :: MAP</code></td><td><p>Merge two maps</p></td></tr><tr><td rowspan="1"><code>map.removeKey</code></td><td><code>map.removeKey(map :: MAP, key :: STRING) :: MAP</code></td><td><p>remove the key from the map</p></td></tr><tr><td rowspan="1"><code>map.sortedProperties</code></td><td><code>map.sortedProperties(map :: MAP) :: LIST OF LIST OF ANY</code></td><td><p>Extract from a map a list of [key,value] entries sorted by the key</p></td></tr><tr><td rowspan="1"><code>meta.type</code></td><td><code>meta.type(value :: ANY) :: STRING</code></td><td><p>Inspect the (name of the) type of a value</p></td></tr><tr><td rowspan="1"><code>parseJson</code></td><td><code>parseJson(jsonStr :: STRING) :: ANY</code></td><td><p>Parses jsonStr to a Cypher value</p></td></tr><tr><td rowspan="1"><code>quineId</code></td><td><code>quineId(input :: STRING) :: BYTES</code></td><td><p>Returns the Quine ID corresponding to the string</p></td></tr><tr><td rowspan="1"><code>strId</code></td><td><code>strId(input :: NODE) :: STRING</code></td><td><p>Returns a string representation of the node&rsquo;s ID</p></td></tr><tr><td rowspan="2"><code>temporal.format</code></td><td><code>temporal.format(date :: DATETIME, format :: STRING) :: STRING</code></td><td><p>Convert date time into string</p></td></tr><tr><td><code>temporal.format(date :: LOCALDATETIME, format :: STRING) :: STRING</code></td><td><p>Convert local date time into string</p></td></tr><tr><td rowspan="1"><code>text.regexFirstMatch</code></td><td><code>text.regexFirstMatch(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Parses the string <code>text</code> using the regular expression <code>regex</code> and returns the first set of capture group matches</p></td></tr><tr><td rowspan="1"><code>text.regexGroups</code></td><td><code>text.regexGroups(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Parses the string <code>text</code> using the regular expression <code>regex</code> and returns all groups matching the given regular expression in the given text</p></td></tr><tr><td rowspan="1"><code>text.regexReplaceAll</code></td><td><code>text.regexReplaceAll(text :: STRING, regex :: STRING, replacement :: STRING) :: STRING</code></td><td><p>Replaces all instances of the regular expression <code>regex</code> in the string <code>text</code> with the <code>replacement</code> string. Numbered capture groups may be referenced with $1, $2, etc.</p></td></tr><tr><td rowspan="2"><code>text.split</code></td><td><code>text.split(text :: STRING, regex :: STRING) :: LIST OF STRING</code></td><td><p>Splits the string around matches of the regex</p></td></tr><tr><td><code>text.split(text :: STRING, regex :: STRING, limit :: INTEGER) :: LIST OF STRING</code></td><td><p>Splits the string around the first <code>limit</code> matches of the regex</p></td></tr><tr><td rowspan="2"><code>text.urldecode</code></td><td><code>text.urldecode(text :: STRING) :: LIST OF STRING</code></td><td><p>URL-decodes (x-www-form-urlencoded) the provided string</p></td></tr><tr><td><code>text.urldecode(text :: STRING, decodePlusAsSpace :: BOOLEAN) :: LIST OF STRING</code></td><td><p>URL-decodes the provided string, using RFC3986 if decodePlusAsSpace = false</p></td></tr><tr><td rowspan="4"><code>text.urlencode</code></td><td><code>text.urlencode(text :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string; additionally percent-encoding quotes, angle brackets, and curly braces</p></td></tr><tr><td><code>text.urlencode(text :: STRING, usePlusForSpace :: BOOLEAN) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string; additionally percent-encoding quotes, angle brackets, and curly braces; optionally using <code>+</code> for spaces instead of <code>%20</code></p></td></tr><tr><td><code>text.urlencode(text :: STRING, encodeExtraChars :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string, additionally percent-encoding the characters enumerated in <code>encodeExtraChars</code></p></td></tr><tr><td><code>text.urlencode(text :: STRING, usePlusForSpace :: BOOLEAN, encodeExtraChars :: STRING) :: LIST OF STRING</code></td><td><p>URL-encodes the provided string, additionally percent-encoding the characters enumerated in <code>encodeExtraChars</code>, optionally using <code>+</code> for spaces instead of <code>%20</code></p></td></tr><tr><td rowspan="1"><code>text.utf8Decode</code></td><td><code>text.utf8Decode(bytes :: BYTES) :: STRING</code></td><td><p>Returns the bytes decoded as a UTF-8 String</p></td></tr><tr><td rowspan="1"><code>text.utf8Encode</code></td><td><code>text.utf8Encode(string :: STRING) :: BYTES</code></td><td><p>Returns the string encoded as UTF-8 bytes</p></td></tr><tr><td rowspan="4"><code>time</code></td><td><code>time() :: TIME</code></td><td><p>Get the current local time</p></td></tr><tr><td><code>time(options :: MAP) :: TIME</code></td><td><p>Construct a local time from the options</p></td></tr><tr><td><code>time(time :: STRING) :: TIME</code></td><td><p>Parse a local time from a string</p></td></tr><tr><td><code>time(time :: STRING, format :: STRING) :: TIME</code></td><td><p>Parse a local time from a string using a custom format</p></td></tr><tr><td rowspan="1"><code>toJson</code></td><td><code>toJson(x :: ANY) :: STRING</code></td><td><p>Returns x encoded as a JSON string</p></td></tr></tbody></table>


---
# Cypher user defined procedures
URL: https://quine.io/generated/quine/cypher-user-defined-procedures/

<table><thead><tr><th>Name</th><th>Signature</th><th>Description</th><th>Mode</th></tr></thead><tbody><tr><td><code>parseProtobuf</code></td><td><code>parseProtobuf(bytes :: BYTES, schemaUrl :: STRING, typeName :: STRING) :: (value :: MAP)</code></td><td><p>Parses a protobuf message into a Cypher map value, or null if the bytes are not parseable as the requested type</p></td><td>READ</td></tr><tr><td><code>toProtobuf</code></td><td><code>toProtobuf(value :: MAP, schemaUrl :: STRING, typeName :: STRING) :: (protoBytes :: BYTES)</code></td><td><p>Serializes a Cypher value into bytes, according to a protobuf schema. Returns null if the value is not serializable as the requested type</p></td><td>READ</td></tr><tr><td><code>create.relationship</code></td><td><code>create.relationship(from :: NODE, relType :: STRING, props :: MAP, to :: NODE) :: (rel :: RELATIONSHIP)</code></td><td><p>Create a relationship with a potentially dynamic name</p></td><td>WRITE</td></tr><tr><td><code>create.setLabels</code></td><td><code>create.setLabels(node :: NODE, labels :: LIST OF STRING) :: VOID</code></td><td><p>Set the labels on the specified input node, overriding any previously set labels</p></td><td>WRITE</td></tr><tr><td><code>create.setProperty</code></td><td><code>create.setProperty(node :: NODE, key :: STRING, value :: ANY) :: VOID</code></td><td><p>Set the property with the provided key on the specified input node</p></td><td>WRITE</td></tr><tr><td><code>cypher.do.case</code></td><td><code>cypher.do.case(conditionals :: LIST OF ANY, elseQuery :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Given a list of conditional/query pairs, execute the first query with a true conditional</p></td><td>WRITE</td></tr><tr><td><code>cypher.doIt</code></td><td><code>cypher.doIt(cypher :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Executes a Cypher query with the given parameters</p></td><td>WRITE</td></tr><tr><td><code>cypher.runTimeboxed</code></td><td><code>cypher.runTimeboxed(cypher :: STRING, params :: MAP, timeout :: INTEGER) :: (value :: MAP)</code></td><td><p>Executes a Cypher query with the given parameters but abort after a certain number of milliseconds</p></td><td>WRITE</td></tr><tr><td><code>db.indexes</code></td><td><code>db.indexes() :: (description :: ANY, indexName :: ANY, tokenNames :: ANY, properties :: ANY, state :: ANY, type :: ANY, progress :: ANY, provider :: ANY, id :: ANY, failureMessage :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>db.propertyKeys</code></td><td><code>db.propertyKeys() :: (propertyKey :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>db.relationshipTypes</code></td><td><code>db.relationshipTypes() :: (relationshipType :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>dbms.labels</code></td><td><code>dbms.labels() :: (label :: ANY)</code></td><td></td><td>READ</td></tr><tr><td><code>debug.node</code></td><td><code>debug.node(node :: ANY) :: (atTime :: LOCALDATETIME, properties :: MAP, edges :: LIST OF ANY, latestUpdateMillisAfterSnapshot :: INTEGER, subscribers :: STRING, subscriptions :: STRING, multipleValuesStandingQueryStates :: LIST OF ANY, journal :: LIST OF ANY, graphNodeHashCode :: INTEGER)</code></td><td><p>Returns comprehensive internal state of a node including properties, edges, standing query states, and event journal. Useful for debugging why standing queries match or don&rsquo;t match.</p></td><td>READ</td></tr><tr><td><code>debug.sleep</code></td><td><code>debug.sleep(node :: ANY) :: VOID</code></td><td><p>Request a node sleep</p></td><td>READ</td></tr><tr><td><code>do.when</code></td><td><code>do.when(condition :: BOOLEAN, ifQuery :: STRING, elseQuery :: STRING, params :: MAP) :: (value :: MAP)</code></td><td><p>Depending on the condition execute ifQuery or elseQuery</p></td><td>WRITE</td></tr><tr><td><code>float.add</code></td><td><code>float.add(node :: NODE, key :: STRING, add :: FLOAT) :: (result :: FLOAT)</code></td><td><p>Atomically add to a floating-point property on a node by a certain amount (defaults to 1.0), returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>getFilteredEdges</code></td><td><code>getFilteredEdges(node :: ANY, edgeTypes :: LIST OF STRING, directions :: LIST OF STRING, allowedNodes :: LIST OF ANY) :: (edge :: RELATIONSHIP)</code></td><td><p>Get edges from a node filtered by edge type, direction, and/or allowed destination nodes</p></td><td>READ</td></tr><tr><td><code>help.functions</code></td><td><code>help.functions() :: (name :: STRING, signature :: STRING, description :: STRING)</code></td><td><p>List registered functions</p></td><td>READ</td></tr><tr><td><code>help.procedures</code></td><td><code>help.procedures() :: (name :: STRING, signature :: STRING, description :: STRING, mode :: STRING)</code></td><td><p>List registered procedures</p></td><td>READ</td></tr><tr><td><code>incrementCounter</code></td><td><code>incrementCounter(node :: NODE, key :: STRING, amount :: INTEGER) :: (count :: INTEGER)</code></td><td><p>Atomically increment an integer property on a node by a certain amount, returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>int.add</code></td><td><code>int.add(node :: NODE, key :: STRING, add :: INTEGER) :: (result :: INTEGER)</code></td><td><p>Atomically add to an integer property on a node by a certain amount (defaults to 1), returning the resultant value</p></td><td>WRITE</td></tr><tr><td><code>loadJsonLines</code></td><td><code>loadJsonLines(url :: STRING) :: (value :: ANY)</code></td><td><p>Load a line-base JSON file, emitting one record per line</p></td><td>READ</td></tr><tr><td><code>log</code></td><td><code>log(level :: STRING, value :: ANY) :: (log :: STRING)</code></td><td><p>Log a value to the system console during query execution. Supports levels: error, warn, info, debug, trace.</p></td><td>READ</td></tr><tr><td><code>purgeNode</code></td><td><code>purgeNode(node :: ANY) :: VOID</code></td><td><p>Purge a node from history</p></td><td>WRITE</td></tr><tr><td><code>random.walk</code></td><td><code>random.walk(start :: ANY, depth :: INTEGER, return :: FLOAT, in-out :: FLOAT, seed :: STRING) :: (walk :: LIST OF STRING)</code></td><td><p>Randomly walk edges from a starting node for a chosen depth. Returns a list of node IDs in the order they were encountered.</p></td><td>READ</td></tr><tr><td><code>recentNodeIds</code></td><td><code>recentNodeIds(count :: INTEGER) :: (nodeId :: ANY)</code></td><td><p>Fetch the specified number of IDs of nodes from the in-memory cache</p></td><td>READ</td></tr><tr><td><code>recentNodes</code></td><td><code>recentNodes(count :: INTEGER) :: (node :: NODE)</code></td><td><p>Fetch the specified number of nodes from the in-memory cache</p></td><td>READ</td></tr><tr><td><code>reify.time</code></td><td><code>reify.time(timestamp :: DATETIME, periods :: LIST OF STRING) :: (node :: NODE)</code></td><td><p>Reifies the timestamp into a [sub]graph of time nodes, where each node represents one period (at the granularity of the period specifiers provided). Yields the reified nodes with the finest granularity.</p></td><td>WRITE</td></tr><tr><td><code>set.insert</code></td><td><code>set.insert(node :: NODE, key :: STRING, add :: ANY) :: (result :: LIST OF ANY)</code></td><td><p>Atomically add an element to a list property treated as a set. If one or more instances of <code>add</code> are already present in the list at node[key], this procedure has no effect.</p></td><td>WRITE</td></tr><tr><td><code>set.union</code></td><td><code>set.union(node :: NODE, key :: STRING, add :: LIST OF ANY) :: (result :: LIST OF ANY)</code></td><td><p>Atomically add set of elements to a list property treated as a set. The elements in <code>add</code> will be deduplicated and, for any that are not yet present at node[key], will be stored. If the list at node[key] already contains all elements of <code>add</code>, this procedure has no effect.</p></td><td>WRITE</td></tr><tr><td><code>standing.wiretap</code></td><td><code>standing.wiretap(options :: MAP) :: (data :: MAP, meta :: MAP)</code></td><td><p>Stream live results from a running standing query. Returns data and metadata for each match.</p></td><td>READ</td></tr><tr><td><code>subscribers</code></td><td><code>subscribers(node :: ANY) :: (queryId :: INTEGER, queryDepth :: INTEGER, receiverId :: STRING, lastResult :: ANY)</code></td><td><p>Returns nodes subscribed to this node for standing query updates. Useful for tracing standing query propagation.</p></td><td>READ</td></tr><tr><td><code>subscriptions</code></td><td><code>subscriptions(node :: ANY) :: (queryId :: INTEGER, queryDepth :: INTEGER, receiverId :: STRING, lastResult :: ANY)</code></td><td><p>Returns nodes this node subscribes to for standing query updates. Useful for tracing standing query propagation.</p></td><td>READ</td></tr><tr><td><code>util.sleep</code></td><td><code>util.sleep(duration :: INTEGER) :: VOID</code></td><td><p>Sleep for a certain number of milliseconds</p></td><td>READ</td></tr></tbody></table>


---
# Getting Started
URL: https://quine.io/getting-started/

# Getting Started

The tutorials in this section will teach you to install Quine, connect an ingest stream to an event source in your data pipeline, shape events into a graph, inspect your data, and develop business logic using a standing query.

![Quine Architecture](../core-concepts/core-concepts-images//abstractQuine.png){ style="margin-top: -2rem;" }

| Tutorial                                             | Description                                                                                                                         |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| [**Quick Start**](quick-start.md)                    | Start here to walk through the basics and get Quine up and running quickly with no fuss.                                            |
| [**Installing Quine**](installing-quine-tutorial.md) | Quine is distributed multiple ways, start here and pick the distribution that is right for you.                                     |
| [**Ingest Streams**](ingest-streams-tutorial.md)     | After installing Quine, this tutorial walks you through connecting to an event source and forming a streaming graph.                |
| [**Standing Queries**](standing-queries-tutorial.md) | Now that you have events streaming into Quine, this tutorial teaches you how to put Quine to work in your streaming event pipeline. |
| [**Recipes**](recipes-tutorial.md)                   | Quine recipes are a convenient way to package together Quine config, graph logic/structure, and exploration UI customizations.      |
| [**Exploration UI**](exploration-ui.md)              | Quine’s Exploration UI is an interactive canvas for ad hoc, interactive exploration of your streaming event data.                   |
| [**Streams**](streams.md)                            | Create, monitor, and manage ingest streams and standing queries from a unified UI.                                                 |

Have a question, suggestion, or did you get stuck somewhere? 

We welcome your feedback! Please join the [Quine Community](https://that.re/chat) and let us know. The team is always happy to discuss Quine and answer your questions.


---
# Dashboard
URL: https://quine.io/getting-started/dashboard/

# Dashboard

![Dashboard](images/dashboard-oss.png)

The Quine dashboard is the landing page you see when you first open the UI. It shows a live flow diagram of your data pipeline, from ingest sources through the graph to standing query outputs, so you can see at a glance whether data is moving and where it is slowing down.

A summary bar in the page header shows live counts of active ingests and standing queries. Hovering over those badges gives a quick status breakdown.

## Status bar

The bar across the top of the flow diagram summarizes the state of the whole pipeline:

- **System Status**: the overall state of the pipeline, one of Flowing, Backpressured, Stopped, or No Data.
- **Throughput**: current events per second moving through the system.
- **Bottleneck**: the stage currently limiting throughput, if any. This compartment is hidden when viewing cluster-wide aggregates, since a bottleneck is a per-member property.
- **Global Ingest Valve**: whether the global valve is open, and how often it closed over the last minute. The valve is Quine's internal mechanism for pausing ingest when downstream stages cannot keep up.
- **Update**: the diagram refreshes every 5 seconds. Use the pause button to freeze it while you inspect a state.

## Flow diagram

The diagram lays the pipeline out in stages, left to right: **Ingest Source**, **Global Valve**, **Graph & Persistor**, **Standing Query Results**, and **Standing Query Outputs**. Each ingest stream, standing query, and output workflow appears by name, and the persistor card shows the database type (for example RocksDB or Cassandra) along with live read and write latency.

Animated flow lines connect the stages. Line width tracks events per second, and color tracks each stage's state: green for flowing, yellow for constrained, amber for backpressured, and red for stopped. When one stage is holding everything else back, the diagram marks it as the bottleneck.

## Cards below the diagram

**Host Metrics**: heap usage, shard count, and the soft and hard in-memory node limits.

## Other pages

The sidebar gives you access to the rest of the UI:

- **Exploration UI** (`/explorer`): interactive graph query and visualization

- **Streams** (`/streams`): create and manage ingest streams and standing queries
- **Metrics** (`/metrics`): deeper JVM and shard metrics with histograms
- **Interactive Docs** (`/docs`): interactive OpenAPI docs for the V2 API


---
# Exploration UI Settings
URL: https://quine.io/getting-started/exploration-ui-settings/

# Exploration UI Settings

The Exploration UI Settings editor configures how the [Exploration UI](exploration-ui.md) behaves for everyone using this Quine instance. Open it from the gear at the right end of the Exploration UI toolbar, then choose "Exploration UI Settings".

![Exploration UI Settings](./ex-ui/exploration-ui-settings.png)

The editor has four sections: Graph Feeds, Sample Queries, Quick Queries, and Node Appearances. Each section lists its existing entries with a search box, and has a button to create a new entry. All of these settings are saved to the server and shared with everyone, they are not per-user preferences.

## Graph Feeds

A graph feed watches a point in a [standing query](../learn/standing-queries/standing-queries.md)'s output pipeline and draws every matching result onto the canvas, live. Feeds turn the Exploration UI from a snapshot into a continuously updating view of what a standing query is matching right now.

Click "New Feed" to create one:

![Creating a graph feed](./ex-ui/graph-feed-editor.png)

- **Name and description**: how the feed appears in the feed list and on the canvas.
- **Draw from**: pick a standing query, then pick a single point in its output workflow to watch. The available tap points are the raw matches (every match before any transformation or enrichment) and, when the workflow defines them, the results after the transformation step and after the enrichment step.
- **Run for every result**: a Cypher query executed for each result arriving at the chosen point. It must return nodes, which are drawn onto the canvas. Each result is passed in as query parameters. For raw matches the returned columns are the fields of `$data`, alongside `$meta` match metadata. For transformed or enriched points, the columns of the transformed or enriched result are passed directly as parameters, for example `$id`.

Saved feeds appear as pills at the bottom of the Exploration UI canvas. Each pill has a toggle to start or stop drawing that feed's results:

![A graph feed pill on the canvas](./ex-ui/graph-feed-pills.png)

While a feed is live, matching nodes stream onto the canvas as they arrive:

![A live graph feed](./ex-ui/graph-feed-live.png)

## Sample Queries

Sample queries populate the drop-down under the Exploration UI query bar, giving users a set of starting points for exploring the graph. Each sample query has a display name and the Cypher text that is loaded into the query bar when chosen.

Sample queries can also be created directly from the query bar with "Bookmark as sample query" in the query menu.

## Quick Queries

Quick queries are the contextual queries offered when right-clicking a node on the canvas. Each quick query defines the node kinds it applies to, the query to run starting from the clicked node, whether it returns nodes or text, and optionally a synthetic edge label for rendering results. See [Interacting With Data](exploration-ui.md#interacting-with-data) for how quick queries behave on the canvas.

## Node Appearances

Node appearances control how nodes render on the canvas: the icon, color, size, and label shown for nodes matching a predicate. Use them to make different kinds of nodes visually distinct, for example coloring `:page` nodes differently from `:user` nodes.

## Configuring via the REST API

Everything in this editor is also configurable through the REST API, under the "UI Styling" heading in the [REST API documentation](../reference/rest-api.md). API configuration is useful for setting up the Exploration UI as part of automated deployment, or from a [recipe](../learn/recipe-ref-manual.md).


---
# Exploration UI
URL: https://quine.io/getting-started/exploration-ui/

# Exploration UI

Quine's Exploration UI is an interactive canvas for exploring specific sub-graphs of data. While Quine overall is a streaming graph interpreter, the Exploration UI gives a user the ability to explore subsets of data in a style more similar to a database. This tool is particularly useful for investigating specific data found via the streaming capabilities built into the rest of the system. It facilitates an ad hoc, interactive exploration of small subsets of data.

!!! Note
    The Exploration UI is fully supported in Chrome. Other browsers are likely to work for most cases, but may experience some unusual behavior. In particular, Safari occassionaly does not update or garbles text values returned in the interactive documentation. Safari also renders nodes more slowly than Chrome, so animating the layout of many nodes is likely to be slow in Safari.

## Components

The Exploration UI is composed of a toolbar along the top and a main canvas in white taking of most of the page. Data represented as nodes and edges will be rendered on the canvas in response to queries issued in the toolbar. A navigation bar along the left side will take you away from the Exploration UI to other views of the system.

![Components of the Exploration UI](./ex-ui/components.png)

### Toolbar

![Components of the Toolbar](./ex-ui/toolbar.png)

- **Query Bar**: A Cypher editor with syntax highlighting. Type a query here to render data into the canvas. If sample queries are defined, a drop-down menu listing them appears here.
- **Query Button**: Click to run the query. The arrow beside it opens a menu of more query actions, described below.
- **Undo**: Click to undo the last change to the canvas. Right-click for more options, including jumping to the previous checkpoint.
- **Animate**: Click to animate the layout of nodes in the canvas. Click again to stop animation.
- **Redo**: Click to redo the canvas change which was previously undone. Right-click for more options, including jumping to the next checkpoint.
- **Checkpoint**: Click to save the current canvas contents as a checkpoint. Right-click to list all available checkpoints and jump to a chosen checkpoint.
- **Download**: Click to download the current canvas (and canvas history) as a JSON object. Right-click for more options, including uploading a previously downloaded file to restore its state to the canvas.
- **Historical Query**: Click to set a historical moment to query the underlying data as it was at that moment. This will set all queries issued in the Exploration UI to query only that historical moment.
- **Tree or Graph**: Click to choose between rendering the nodes in the canvas in a free-form graph, or a hierarchical tree. Note: tree organization is not customizable.
- **Recenter**: Click to return the canvas viewport to the center position.
- **Settings and Maintenance**: The gear at the far right, providing access to the [Exploration UI Settings](exploration-ui-settings.md) and maintenance actions like clearing the canvas.

After a query completes, the number of nodes and edges it returned is shown briefly in the top right corner below the toolbar.

![Node and edge counts after a query](./ex-ui/count-indicator.png)

### Query Menu

The arrow on the right side of the Query button opens a menu with more query actions:

![Query menu](./ex-ui/query-menu.png)

- **Run as text query**: Run the query and display the results in tabular form in the results panel below, instead of displaying nodes in the graph view.
- **Run in background**: Dispatch the query as a [background query](../learn/background-queries/background-queries.md) that runs out-of-band, described below.
- **Multi-line editing**: Expand the query bar into a multi-line editor for longer queries. Pressing Shift+Enter in the query bar does the same thing.
- **Bookmark as sample query**: Save the current query into the sample query drop-down. Saving asks for confirmation, since sample queries are a global setting and changes apply to all users.
- **Standing Query Inspection**: Open a live view onto a running standing query, described below.

### Canvas

Query results are rendered onto the shared space of the canvas. New results do not replace the content on the canvas, but rather add to it. All edges between two nodes on the canvas will automatically be drawn on the canvas; nodes rendered on the canvas may have edges that are not drawn when the node at the edge's other endpoint is not rendered.

Nodes often contain properties, which can be [viewed and edited](#viewing-and-editing-node-properties). Queries in progress that have not yet returned results cause a spinner to display in the top right corner of the canvas. Nodes rendered in the UI represent the data on each node -at the time it was returned-. Updating the state of a node is accomplished by returning that node again (or using the "Refresh" quick query).

----------------------

When new results are returned to the canvas, the canvas animates briefly to layout all results. Nodes can be selected and moved around the canvas. Dragging a node automatically pins it to the canvas so that it does not move when the canvas is animated. Additional, contextually relevant queries can be easily initiated by right-clicking a node and choosing the appropriate query.

![Actions and Quick Queries shown when right-clicking a node](./ex-ui/quick-queries.png)

----------------------

Queries that return tabular results render into the results panel instead of the canvas, described in [Viewing Tabular Results](#viewing-tabular-results).

Node appearances, quick queries, and default sample queries in the query bar can be customized in the [Exploration UI Settings](exploration-ui-settings.md) editor, or via the corresponding [API calls documented](../reference/rest-api.md) under the heading of "UI Styling".

## How to Use the Exploration UI

### Running Queries

Queries are issued in one of three ways:

1. Typed into the Query Bar and executed by pressing Enter or clicking the "Query" button.
2. Chosen from the "Quick Query" context menu found by right-clicking a node (when defined). Note that this actually adds the query to the query bar and executes it.
3. Executed on page load when the URL for the Exploration UI ends with a hash (`#`) followed by a query.

!!! Warning

    Because the Exploration UI generates a snapshot of the graph, it can give Quine the appearance of database. This is just a convenient way to explore the data structures but make no mistake: Quine is a streaming graph interpreter designed to process what are effectively infinite streams of data. Quine is run without indices by default. If Quine is managing a large amount of data, some queries which require scanning all nodes can take a very long time and slow down other functionality. It is strongly recommended to use a node ID in each query or use built in functions like `idFrom(…)` or `recentNodes()` to efficiently pull out small amounts of data. [See this page for more guidance on querying infinite data](../core-concepts/id-provider.md).

Queries are written in [Cypher](../learn/cypher/index.md). Most Cypher syntax is supported, and queries can be entered directly into the query bar. The Up and Down arrow keys step back through your query history. For longer queries, choose "Multi-line editing" from the query menu (or press Shift+Enter) to expand the editor across multiple lines; Cmd+Enter (Ctrl+Enter on Windows and Linux) runs the query from any mode.

![Query editor with sample queries](./ex-ui/editor-sample-queries.png)

The Exploration UI can be configured with any set of queries pre-programmed and available via a drop-down menu from the query bar. These sample queries can be configured in the [Exploration UI Settings](exploration-ui-settings.md) editor, saved directly from the query menu with "Bookmark as sample query", or configured through the REST API. Sample queries can be parameterized with easily completed values to make pulling out complex patterns very easy when the proper starting value is identified.

By default, queries issued in the Exploration UI are expected to return nodes (and only nodes). Nodes returned from queries will be rendered in the canvas. After computing the nodes returned from a query, the Exploration UI automatically resolves the edges to display in the canvas. Edges which connect nodes rendered in the canvas (previously rendered nodes and newly returned node results) will all be displayed.

One query is _issued_ at a time, but any number of queries can be executing simultaneously. As new queries are issued, the in-progress spinner on the right reports the number of currently executing queries. Hovering the mouse over the spinner/counter will turn it into an "X" which can be clicked to cancel all in-progress queries. Since some queries can continue forever (e.g. wiretapping a Standing Query), clicking the "X" that appears in the progress spinner is the only way to gracefully end the query.

Quine includes many utility functions built in, and the ability to add in custom user-defined queries. To list all available functions which can be included in queries, you can execute the text query `CALL help.functions()` or `CALL help.procedures()` to print out the name, signature, and documentation for all currently supported functions and procedures.

### Viewing Tabular Results

In addition to the canvas, the results panel is another way to view the results of ad-hoc queries. The results panel displays results as a table rather than an interactive graph. While the canvas can only display the results of queries that return nodes, the results panel can display the results of any query. To run a query and display its results in the results panel, choose "Run as text query" from the query menu, or press Ctrl+Shift+Enter.

![The results panel](./ex-ui/result-card.png)

The results panel can be resized vertically by dragging its top edge. Only one panel is open at a time, and the open panel covers the minimized results, so minimize the current panel before opening another. Each panel shows the query it came from; the edit button loads that query back into the query bar for refinement, and re-running the same ad-hoc query reuses its existing panel instead of opening a duplicate. Results can continue to stream into a panel in perpetuity for long-running queries.

Minimized results collect at the bottom right corner of the canvas, listed with a search box to filter them, so past results stay reachable without cluttering the canvas.

![The minimized results drawer](./ex-ui/minimized-drawer.png)

Query and system errors are reported the same way, in a results panel marked as an error.

### Running a Query in the Background

A query that will take a long time, such as an all-node scan, a bulk update, a large export, does not have to be run interactively. Choosing **Run in background** from the query menu opens a dialog that dispatches the query bar's contents as a [background query](../learn/background-queries/background-queries.md): Quine accepts it, returns immediately, and runs it out-of-band, so nothing in the browser is waiting on it to finish.

![The Run in background dialog](./ex-ui/background-run-modal.png)

The query itself is taken from the editor buffer and shown read-only for confirmation. Everything else is generated from the server's own API schema, so the dialog offers the full set of destinations (Kafka, Kinesis, SNS, HTTP endpoints, files, Cypher queries, etc) with the same fields and validation as the Streams page forms.

Destinations default to **Drop**, which runs the query without writing its results anywhere. That is the usual choice from here, because the results panel sees every row before the destinations do.

Once the run starts, a results panel opens on it immediately and rows stream in live as the query produces them. The panel behaves like any other, except that a background query is a single finite run: when it ends, the panel keeps what it captured but cannot be resumed or fetched further, and it is not restored when the page reloads.

![Results streaming from a background query](./ex-ui/background-run-card.png)

The results panel is a best-effort live view with no delivery guarantees; rows are dropped rather than buffered if the browser cannot keep up. To see the authoritative row count, to cancel a run, or to find runs started elsewhere, use the Background Queries panel on the [Streams](streams.md#background-queries-panel) page.

### Inspecting Standing Queries

Choosing "Standing Query Inspection" from the query menu shows the output workflow of a running [standing query](../learn/standing-queries/standing-queries.md) as a diagram with tap points: the raw standing query matches, the results after any transformation step, and the results after any enrichment step.

![Standing Query Inspection](./ex-ui/sq-inspection-modal.png)

Clicking a tap point streams the results flowing through that point of the workflow. Results can be fetched a batch at a time ("Get more"), followed continuously ("Go live"), or stopped. A tap retains a bounded buffer of recent results, so it is safe to leave open against a busy standing query.

![Results streaming from a tap point](./ex-ui/sq-inspection-card.png)

### Viewing and Editing Node Properties

Right-clicking a node opens a context menu of actions and quick queries. Clicking "View properties" displays the properties of this node as well as the node's ID and labels.

![Viewing node properties](./ex-ui/node-properties.png)

Users can add, edit, or delete the properties and labels of the node by clicking the "Edit" button. Property values can be entered as text, a number, a boolean, or a JSON string, selectable using the selector to the left of each input box. If there is ever doubt about how the properties will be edited, clicking "Cypher query" will show the exact Cypher query used to edit the properties. No changes will be written to the graph until you click "Apply changes" and click "OK" in the confirmation popup.

![Editing node properties](./ex-ui/node-properties-edit.png)

### Interacting With Data

A quick query is a pre-programmed query that is made available for execution by right-clicking a relevant starting node. Any number of quick queries can be pre-programmed into the system, and each quick query can define on which kind of node it becomes available. The node on which the quick query is executed is bound to the variable `n` — for example, `MATCH (n)--(m) RETURN DISTINCT m` expands one hop from the clicked node. Quine automatically prepends `MATCH (n) WHERE id(n) = <clicked-node-id>` to the `querySuffix` you provide, so your query will use `n` as its starting point.

Quick queries that are configured with an `edgeLabel` will produce a "synthetic edge" in the Exploration UI. A synthetic edge is rendered as a purple dotted edge in the Exploration UI. A synthetic edge does not directly exist in the underlying data; it exists only in the canvas. A synthetic edge connects the starting node from a quick query with all the results returned from that quick query. Synthetic edges enable displaying a complex result (e.g. the result of a complex graph traversal) as a simple single edge.

![Synthetic Edges](./ex-ui/synthetic-edge.png)

Nodes rendered on the canvas represent the state of the node at the moment it was retrieved by Quine. To update the rendered node on the canvas with new properties, the node simply needs to be returned again. This is done with another query, or by right-clicking a node and choosing the built-in Quick Query to "Refresh" a node. If multiple nodes are selected (e.g. by pressing Ctrl-A, or holding shift to draw a box or multi-select), clicking the "Refresh" quick query for one will refresh all selected nodes.

When queries are executed and new results are added to the canvas, nodes are animated by simulating the physics of a force-directed graph. This animation helps to lay out nodes in a two-dimensional arrangement for quick interpretation and further interaction. Depending on the data, the short animation may not be enough to conveniently separate the rendered nodes. In that case, clicking the play button will allow the physics animation to continue indefinitely until the animation is paused again. Nodes are automatically "pinned" to the canvas and removed from the physics simulation when they are dragged. Pinned nodes remain pinned when dragged again. Nodes that have been removed from the physics simulation will have a subtle drop shadow behind their node and label. To unpin a node and allow it to animate freely again, hold Shift and click and hold on the node(s).

### Using the Toolbar Buttons

Interactive exploration of the data is usually an iterative process. Investigating one node often leads to the next, which leads to the next, and so on. As such, interactive exploration benefits from the ability to jump back and forth between what has been rendered at each step, as well as saving and sharing of progress. The toolbar includes buttons to undo and redo each step of the exploration. Each time results are rendered to the canvas, a new entry is added to the timeline managed by the Exploration UI.

At any point during exploration, clicking the checkpoint button will allow the user to create a named checkpoint in the exploration. Right-clicking the checkpoint button will show a list of all named checkpoints and where they fall in the timeline compared to the present state of the canvas.

Clicking the download button will download a JSON representation of the exploration history so far. The `history.json` file can be loaded back into the Exploration UI, using the upload option in the download button's right-click menu, on any machine running Quine, whether it is a part of the original system or not.

The Exploration UI is an interface into the underlying graph data managed by Quine. The underlying graph data is fully versioned, making it possible to run queries at past moments to see what the results would have been at previous points in time. Unlike the timeline used to undo and redo exploration history, historical queries are a feature of the underlying graph interpreter, and not the Exploration UI. The Exploration UI does provide an interface to querying past historical moments, however. A user can set the Exploration UI to query past moments by clicking the History button and following the prompt to choose a single historical moment to begin a new exploration session. Choosing a new historical moment to query with the history button will clear anything rendered on the canvas and in the Exploration UI timeline.

![Historical query](./ex-ui/historical-query.png)

Querying for a matched node is especially useful if there is a Cypher query registered as one of the outputs of the Standing Query and if that second query modifies the data—for instance, adding an edge connected to the node.

## Reference: Key Combinations

- **Enter**: when in the query bar, run the query; equivalent to clicking the "Query" button.
- **Shift-Enter**: when in the query bar, insert a newline and switch to multi-line editing.
- **Cmd-Enter (Ctrl-Enter on Windows and Linux)**: run the query, in single-line or multi-line mode.
- **Ctrl-Shift-Enter**: run the query as a text query with tabular results.
- **Up/Down arrows**: when in the single-line query bar, step through query history.
- **Ctrl-A**: select all nodes on the canvas.
- **Backspace**: remove the selected node(s) from the canvas.
- **Shift-click and drag**: draw a box to select multiple nodes.
- **Shift-click a node**: add a node to the selected nodes.
- **Right click a node**: bring up the menu of actions and quick queries available from the clicked node.
- **Drag a node**: pin it to the canvas so it does not animate with future queries.
- **Shift-click and hold a pinned node**: unpin it so it animates freely again.
- **Double click a node**: execute the node's default quick query (the first in the node's quick query list).
- **Shift-click thatDot logo**: download an SVG image of the canvas.

!!! tip "Quine Enterprise"

    For multi‑tenant role based access control, see Quine Enterprise. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).


---
# Ingest Stream Quickstart
URL: https://quine.io/getting-started/ingest-streams-tutorial/

# Ingest Stream Quickstart

Quine is specifically designed to ingest and process high volumes of event data from one or more streams, turning events into a graph to make it easy to detect complex patterns, and streaming the results out in real-time. There are two main structures that you need to properly use Quine: the [**ingest stream**](/reference/rest-api/?av=v2#/operations/create-ingest) and the [**standing query**](/reference/rest-api/?av=v2#/operations/create-standing-query). In this tutorial we will cover the ingest stream.

![Quine Event Streaming](images/quine-stream.png)

The ingest stream is where a streaming graph starts. It connects to data producers, transforms the data, then populates a streaming graph to be analyzed by standing queries.

Let's take a look at how ingest streams work.

Quine is fundamentally a stream-oriented data processor that uses a graph data model. This provides optimal integration with streaming data producers and consumers such as Kafka and Kinesis. Quine can also process batch data, consuming data from CSV files, databases and data lakes, or web APIs.

## Our Scenario

For the sake of this tutorial, assume that you need to separate human-generated events from bot-generated events in the english wikipedia database and send them to a destination in your data pipeline for additional processing. We will discuss concepts and provide code examples with this use case in mind.

## Ingest Stream Concepts

**What is an Ingest Stream?**

An *ingest stream* connects a data source to Quine and prepares the emitted data for the streaming graph. Within the ingest stream, an ingest query written in Cypher, updates the streaming graph nodes and edges as data is received.

**Quine Node IDs**

With a graph data model, nodes are the primary unit of data — much like a "row" is the primary unit of data in a relational database. However, unlike traditional graph data systems, a Quine user never has to create a node directly. Instead, using `idFrom`, *the graph functions as if all nodes exist.*

Here is some documentation on why we use `idFrom` due to how Quine handles the unique challenge of [maintaining state for a potentially infinite stream of data](../core-concepts/id-provider.md#idfrom).

Quine represents every possible node as an existing "empty node" with no history. As data streams into the system, previously empty nodes accumulate properties and connections to other nodes (edges). As new data arrives and nodes are updated, Quine keeps track of each node's history.

**Locating any node using `idFrom`**

`idFrom` is a Cypher function specific to Quine and allows you to retrieve node data without an index. `idFrom` takes any number of arguments and deterministically produces a node ID (hash) from that input. You will use `idFrom` in the ingest query portion of every ingest stream that you create.

For example, the absolute minimum ingest query to load any incoming data into the graph is simply a wrapper around the `idFrom` function.

```cypher
MATCH (n) WHERE id(n) = idFrom($that) SET n.line = $that
```

This query creates nodes but does not create any edges, so this is not a very interesting or useful graph. It does however demonstrate the basic mechanism for `idFrom`.

## Syntax and Structure

The first step when defining an ingest stream is to understand the patterns you are trying to find, or put another way, the questions you want the data to answer. This will determine the overall shape of your graph -- which fields are nodes, which are properties of nodes, and how all of these connect.

This is no different than designing a SQL database except that Quine's graph data structure conforms to the familiar *subject-predicate-object* structure found in most human languages. Properly structuring your Quine graph is an important step in making standing queries and quick queries productive and performant. For a  guide on this design process, see [Data Modeling and Query Design](../core-concepts/data-modeling.md).

An ingest query is defined by setting a `type` as described by the [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) API documentation. Quine supports many types of ingest streams. Each type has a unique form and requires a specific structure to configure properly.

For example, we can use a "server sent events" ingest stream to connect the live stream of page revisions on Wikipedia; [mediawiki.revision-create](https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_revision_create).

Note: You will need cURL and jq installed in your environment to use the tutorial commands.

Issue the following `curl` command in a separate terminal running on the machine were you started Quine.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests" \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "wikipedia-revision-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
  },
  "query": "CREATE ($that)"
}'
```

Take note of the structure of the ingest stream object carried in the body of the POST.

```json
{
  "name": "wikipedia-revision-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
  },
  "query": "CREATE ($that)"
}
```

* **name** - a unique identifier for the ingest stream
* **type** - identifies the type of ingest stream to create. In this case we created a "server sent events" ingest stream
* **url** - sets the URL for the event producer
* **format** - specifies the ingest query for the ingest stream (we will cover this in detail shortly)

!!! note "About the `format` choice"
    This tutorial uses `CypherJson` (and its v2 equivalent `Json`) throughout: each record is parsed as JSON and bound to `$that` as a `Map`, so the Cypher query can access fields like `$that.page_title` directly. The API also offers a `Raw` format, which binds the unparsed bytes to `$that` instead. See [Record Formats](../learn/ingest-sources/index.md#record-formats) for the full list and when to use each.

This ingest stream will load the stream of `revision-create` events from `mediawiki` into your graph. You may get a warning message about Quine being unable to confirm that the stream is idempotent, but simply ignore it for now. Without defining any relationships between node types, the nodes are all disconnected. Let's pause this ingest stream and consider how to create a graph from the events.

Send a POST to the [Pause Ingest Stream: `POST /api/v2/graph/quine/ingests/{ingestName}:pause`](/reference/rest-api/?av=v2#/operations/pause-ingest) endpoint to stop ingesting events.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests/wikipedia-revision-create:pause"
```

## The shape of your data

Before going too far with this example, we need to step back and understand the shape of the event data that we receive from the `revision-create` event stream. Although Quine does not require a pre-defined schema, understanding the shape of the data you are working with is tremendously helpful when writing Cypher queries.

Wikimedia defines a schema for the `revision-create` event in their [API documentation](https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_page_create), which uses an [OpenAPI](https://www.openapis.org/) specification to describe the API.  

Here is the sample `revision-create` event schema from the documentation:

```json
{
  "$schema": "/mediawiki/revision/create/1.2.0",
  "database": "examplewiki",
  "meta": {
    "domain": "test.wikipedia.org",
    "dt": "2020-06-10T18:57:16Z",
    "stream": "mediawiki.revision-create",
    "uri": "https://examplewiki.wikipedia.org/wiki/TestPage10"
  },
  "page_id": 123,
  "page_is_redirect": false,
  "page_namespace": 0,
  "page_title": "TestPage10",
  "performer": {
    "user_edit_count": 1,
    "user_groups": [
      "*",
      "user",
      "autoconfirmed"
    ],
    "user_id": 123,
    "user_is_bot": false,
    "user_registration_dt": "2016-01-29T21:13:24Z",
    "user_text": "example_user_text"
  },
  "rev_content_changed": true,
  "rev_content_format": "text/x-wiki",
  "rev_content_model": "wikitext",
  "rev_id": 123,
  "rev_is_revert": false,
  "rev_len": 3,
  "rev_minor_edit": false,
  "rev_parent_id": 122,
  "rev_sha1": "mr0szy90m5qbn6tek7ch3nebaild3tm",
  "rev_slots": {
    "main": {
      "rev_slot_content_model": "wikitext",
      "rev_slot_origin_rev_id": 123,
      "rev_slot_sha1": "2mx9qnkore72az8niqap1s3ycpu1jej",
      "rev_slot_size": 20
    }
  },
  "rev_timestamp": "2020-06-10T18:57:16Z"
}
```

By inspecting the sample `revision-create` object from the documentation, we identify several groups of parameters that will make for interesting graph analysis later.

* Information about the page
* Which database the page belongs to
* The user creating the page
* Information about the revision that was submitted

Using the information that we identified above, we can create a model of the graph from the `revision-create` event.

![Create event graph](images/page-create-graph.png)

## Define an ingest stream

Using this model from above we can create an ingest query in Cypher to shape and load the event into nodes in the graph.

```cypher
MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode)
WHERE id(revNode) = idFrom('revision', $that.rev_id) 
  AND id(pageNode) = idFrom('page', $that.page_id) 
  AND id(dbNode) = idFrom('db', $that.database)
  AND id(userNode) = idFrom('id', $that.performer.user_id) 
  AND id(parentNode) = idFrom('revision', $that.rev_parent_id)

SET revNode = $that,
    revNode.bot = $that.performer.user_is_bot,
    revNode:revision

SET parentNode.rev_id = $that.rev_parent_id

SET pageNode.id = $that.page_id, 
    pageNode.namespace = $that.page_namespace, 
    pageNode.title = $that.page_title, 
    pageNode.comment = $that.comment, 
    pageNode.is_redirect = $that.page_is_redirect, 
    pageNode:page 

SET dbNode.database = $that.database, 
    dbNode:db 

SET userNode = $that.performer, 
    userNode.name = $that.performer.user_text, 
    userNode:user 

CREATE (revNode)-[:TO]->(pageNode),
       (pageNode)-[:IN]->(dbNode),
       (userNode)-[:RESPONSIBLE_FOR]->(revNode),
       (parentNode)-[:NEXT]->(revNode)
```

Take a minute to understand this Cypher query.

The first thing that Quine does when it receives an event from an event source is to parse the object based on the ingest stream `type` setting and pass along the parsed object in the `$that` parameter.

The first `MATCH` statement locates the `revNode`, `pageNode`, `dbNode`, `userNode` and `parentNode` nodes by setting the node id to the id determined by the `idFrom` function.

```cypher
MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode)
WHERE id(revNode) = idFrom('revision', $that.rev_id) 
  AND id(pageNode) = idFrom('page', $that.page_id) 
  AND id(dbNode) = idFrom('db', $that.database)
  AND id(userNode) = idFrom('id', $that.performer.user_id) 
  AND id(parentNode) = idFrom('revision', $that.rev_parent_id)
```

The next section of the query uses `SET` to populate the properties into each node from the parsed data in `$that`.

```cypher
SET revNode = $that,
    revNode.bot = $that.performer.user_is_bot,
    revNode:revision

SET parentNode.rev_id = $that.rev_parent_id

SET pageNode.id = $that.page_id, 
    pageNode.namespace = $that.page_namespace, 
    pageNode.title = $that.page_title, 
    pageNode.comment = $that.comment, 
    pageNode.is_redirect = $that.page_is_redirect, 
    pageNode:page 

SET dbNode.database = $that.database, 
    dbNode:db 

SET userNode = $that.performer, 
    userNode.name = $that.performer.user_text, 
    userNode:user
```

And the final section of the query establishes the relationships between the nodes using `CREATE`.

```cypher
CREATE (revNode)-[:TO]->(pageNode),
       (pageNode)-[:IN]->(dbNode),
       (userNode)-[:RESPONSIBLE_FOR]->(revNode),
       (parentNode)-[:NEXT]->(revNode)
```

Now, lets update the `wikimedia-revision-create` ingest stream with this new ingest query.

* Remove the existing ingest stream using [Delete Ingest Stream: `DELETE /api/v2/graph/quine/ingests/{ingestName}`](/reference/rest-api/?av=v2#/operations/delete-ingest).

```shell
curl -X "DELETE" "http://127.0.0.1:8080/api/v2/graph/quine/ingests/wikipedia-revision-create"
```

* Create a new ingest stream configuration JSON object.

```json
{
  "name": "wikipedia-revision-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
  },
  "query": "MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode) WHERE id(revNode) = idFrom('revision', $that.rev_id) AND id(pageNode) = idFrom('page', $that.page_id) AND id(dbNode) = idFrom('db', $that.database) AND id(userNode) = idFrom('id', $that.performer.user_id) AND id(parentNode) = idFrom('revision', $that.rev_parent_id) SET revNode = $that, revNode.bot = $that.performer.user_is_bot, revNode:revision SET parentNode.rev_id = $that.rev_parent_id SET pageNode.id = $that.page_id, pageNode.namespace = $that.page_namespace, pageNode.title = $that.page_title, pageNode.comment = $that.comment, pageNode.is_redirect = $that.page_is_redirect, pageNode:page SET dbNode.database = $that.database, dbNode:db SET userNode = $that.performer, userNode.name = $that.performer.user_text, userNode:user CREATE (revNode)-[:TO]->(pageNode), (pageNode)-[:IN]->(dbNode), (userNode)-[:RESPONSIBLE_FOR]->(revNode), (parentNode)-[:NEXT]->(revNode)"
}
```

* POST the ingest stream JSON to [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest)

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests" \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "wikipedia-revision-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
  },
  "query": "MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode) WHERE id(revNode) = idFrom(\'revision\', $that.rev_id) AND id(pageNode) = idFrom(\'page\', $that.page_id) AND id(dbNode) = idFrom(\'db\', $that.database) AND id(userNode) = idFrom(\'id\', $that.performer.user_id) AND id(parentNode) = idFrom(\'revision\', $that.rev_parent_id) SET revNode = $that, revNode.bot = $that.performer.user_is_bot, revNode:revision SET parentNode.rev_id = $that.rev_parent_id SET pageNode.id = $that.page_id, pageNode.namespace = $that.page_namespace, pageNode.title = $that.page_title, pageNode.comment = $that.comment, pageNode.is_redirect = $that.page_is_redirect, pageNode:page SET dbNode.database = $that.database, dbNode:db SET userNode = $that.performer, userNode.name = $that.performer.user_text, userNode:user CREATE (revNode)-[:TO]->(pageNode), (pageNode)-[:IN]->(dbNode), (userNode)-[:RESPONSIBLE_FOR]->(revNode), (parentNode)-[:NEXT]->(revNode)"
}'
```

You can verify that the stream is active by sending `GET` to the [List Ingest Streams: `GET /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/list-ingests) endpoint.

```shell
curl "http://127.0.0.1:8080/api/v2/graph/quine/ingests" | jq '.'
```

Quine will return the status of all of the ingest streams that are configured. In our case, we only have one stream and it is running.

```json
{
  "items": [
    {
      "name": "wikipedia-revision-create",
      "status": "RUNNING",
      "settings": {
        "source": {
          "type": "ServerSentEvent",
          "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
        },
        "query": "MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode) WHERE id(revNode) = idFrom('revision', $that.rev_id) AND id(pageNode) = idFrom('page', $that.page_id) AND id(dbNode) = idFrom('db', $that.database) AND id(userNode) = idFrom('id', $that.performer.user_id) AND id(parentNode) = idFrom('revision', $that.rev_parent_id) SET revNode = $that, revNode.bot = $that.performer.user_is_bot, revNode:revision SET parentNode.rev_id = $that.rev_parent_id SET pageNode.id = $that.page_id, pageNode.namespace = $that.page_namespace, pageNode.title = $that.page_title, pageNode.comment = $that.comment, pageNode.is_redirect = $that.page_is_redirect, pageNode:page SET dbNode.database = $that.database, dbNode:db SET userNode = $that.performer, userNode.name = $that.performer.user_text, userNode:user CREATE (revNode)-[:TO]->(pageNode), (revNode)-[:IN]->(dbNode), (userNode)-[:RESPONSIBLE_FOR]->(revNode), (parentNode)-[:NEXT]->(revNode)"
      },
    "stats": {
      "ingestedCount": 1666,
      "rates": {
        "count": 1666,
        "oneMinute": 21.44527113707015,
        "fiveMinute": 15.367459925922011,
        "fifteenMinute": 13.715948876513574,
        "overall": 26.105181113725962
      },
      "byteRates": {
        "count": 2574561,
        "oneMinute": 32566.461096455438,
        "fiveMinute": 22733.450385686425,
        "fifteenMinute": 20032.299030301117,
        "overall": 40341.915896294115
      },
      "startTime": "2022-09-28T15:13:21.932384Z",
      "totalRuntime": 62921
    }
    }
  ]
}
```

## Next Steps

You have data streaming into Quine and forming a graph. Over time, the shape of the graph will become more connected as new events arrive.

In the [standing query](standing-queries-tutorial.md) tutorial we will separate human generated events from bot generated events in the english wikipedia database and send them to a destination in your data pipeline for additional processing.


---
# Installing Quine
URL: https://quine.io/getting-started/installing-quine-tutorial/

# Installing Quine

Quine is a key participant in a streaming event data pipeline that consumes data, builds it into a graph structure, runs computation on that graph to answer questions or compute results, and then stream them out. Quine combines the real-time event processing capabilities of systems like Flink and ksqlDB with the graph data structure found in graph databases like Neo4j and TigerGraph.

There are multiple ways for you to install Quine, select the one that is best for your environment from the list below.

| Method                                            | Description                                                                                                                                                                                  |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Docker Container**](#docker-container)         | If you already have Docker, this is the quickest way to get up and running with the minimum setup time and impact to your environment.                                                       |
| [**Download Java executable file**](#executable)  | Download the `jar` file to run locally on your laptop or development server. Running the `jar` file locally is the most flexible way to evaluate Quine.                                      |
| [**Build from source**](#source-code)             | Need to do a deeper evaluation of how Quine operates? Clone the open source repository from GitHub and dig in!                                                                               |
| [**Cloud hosted by thatDot**](#cloud-hosted-saas) | Are you interested in the [Quine Enterprise features](https://www.thatdot.com/product/pricing)? Contact our sales team to set up an evaluation environment in your cloud provider of choice. |

Note: You will need cURL and jq installed in your environment to use the tutorial commands.

Follow the steps below to install Quine for use in your environment.
---

## Docker Container

Docker allows you to install a containerized version of Quine for evaluation.

### Prerequisites

* **Host Environment** - You need a [Mac](https://docs.docker.com/desktop/install/mac-install/), [Windows](https://docs.docker.com/desktop/install/windows-install/), or [Unix](https://docs.docker.com/desktop/install/linux-install/) host server to run Docker. Please be sure that your host meets the system requirements outlined in the proceeding links.
* **Docker Desktop** - Instructions on how to install Docker can be found on the [official Docker website](https://docs.docker.com/get-docker/).

### Install and start the Quine container

The [Quine Docker image](https://hub.docker.com/r/thatdot/quine)  is distributed via Docker Hub and can be installed and launched with a single command.

1. Open a terminal window
2. With the Docker desktop application running, issue the following command

```shell
docker run -p 8080:8080 thatdot/quine
```

If successful, you will see a message similar to the following appear in the terminal.

```text
Unable to find image 'thatdot/quine:latest' locally
latest: Pulling from thatdot/quine
2408cc74d12b: Pull complete
3d4177d25912: Pull complete
84eef58e1007: Pull complete
7d414c479da8: Pull complete
ac0978c82c5c: Pull complete
5e38591d5629: Pull complete
Digest: sha256:8200a2ea46aaa021865cfa7e843c65bb3f6dded4d00329217800f1a26df36e14
Status: Downloaded newer image for thatdot/quine:latest
Graph is ready
Quine web server available at http://127.0.0.1:8080
```

!!! Tip

    If you want to use a recipe in docker, you’ll need to include the recipe file (and data file if required) in your docker container. The easiest way to do this is by mounting a [docker volume](https://docs.docker.com/storage/volumes/). 
    
    For example, `docker run -it -p 8080:8080 -v ~/quine-recipe.yml:/tmp/recipe.yml thatdot/quine:1.5.1 -r /tmp/recipe.yml` would mount the recipe at `~/quine-recipe.yml` on your host system as a volume at `/tmp/recipe.yml` in the container, and instruct quine to use that recipe via the startup flag.
    For cases when you need to pass multiple files to Quine from a single directory, `docker run -it -p 8080:8080 -v ~/quine/:/tmp/ thatdot/quine:1.5.1 -r /tmp/recipe.yml --recipe-value in_file=/tmp/recipe_file.json` would mount the directory `~/quine/` with recipe `~/quine/quine-recipe.yml` and data file `~/quine/recipe_file.json` on your host system as a volume at `/tmp/` in the container, and instruct quine to use the recipe and data files from `/tmp/` via the startup flags.

Verify that Quine is operating using the [System Information: `GET /api/v2/system/systemInfo`](/reference/rest-api/?av=v2#/operations/get-system-info) API endpoint.

```shell
❯ curl -s "http://127.0.0.1:8080/api/v2/system/systemInfo" | jq '.'
{
  "version": "1.3.2",
  "gitCommit": "6f8bb1b3a308d9c90cc71a2328858907ec341e74",
  "gitCommitDate": "2022-08-10T11:01:51-0700",
  "javaVersion": "OpenJDK 64-Bit Server VM 17.0.2 (Azul Systems, Inc.)",
  "persistenceWriteVersion": "12.0.0"
}
```

You can connect to the Quine exploration UI by entering `http://127.0.0.1:8080` into your browser.

### Shutdown Quine

POST to the [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) endpoint to gracefully shutdown Quine and stop the Docker container.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/system:shutdown"
```

Both Quine and the Docker container will shutdown and exit. If successful, you will see a message similar to the following appear in the terminal and the container will have a status of `exited` in the Docker desktop.

```text
Quine is shutting down...
Shutdown complete
```

---

## Executable

Using an executable file to run Quine locally provides the most flexibility …

* Evaluate Quine using your hardware of choice
* Make changes to the Quine [configuration](../reference/config/configuration.md)  file
* Launch Quine [recipes](../learn/recipe-ref-manual.md) 
* Benchmark Quine's performance running different versions of Java

### Prerequisites

You will need the following in order to run Quine in your environment.

* **Java JRE version 11 or greater** - instructions for how to install Java can be found on the [official Oracle/Java web site](https://www.oracle.com/java/technologies/downloads/) .

### Download the Quine Executable File

Quine is distributed as a pre-built Java `jar` executable that you download and run on your local laptop or server.

1. Download the `jar` file from the `quine.io` [download](../download.md)  page.
2. Store the `jar` file in a working directory

We recommend storing the Quine `jar` file in the same directory that you plan to use during your evaluation and development.

On a Mac, this process would look similar to this.

* Selecting the `JAR DOWNLOAD` button prompted the browser to download the latest version of Quine and store it in the `~/Downloads` directory.
* Once the download completes, issue the following commands to move Quine into a directory that you can use for evaluation and testing.

```shell
❯ mkdir gettingStarted
❯ cd gettingStarted
❯ mv ~/Downloads/quine-2.1.1.jar .
❯ ls -l
total 460088
-rw-r--r--@ 1 quine  staff  220704180 Aug 25 09:22 quine-2.1.1.jar
```

### Start Quine

Once you have the Quine package stored locally, you are ready to launch Quine for the first time.

To launch Quine, issue the following command.

```shell
❯ java -jar quine-2.1.1.jar
```

If successful, you will see a message similar to the following appear in the terminal.

```text
Graph is ready
Quine web server available at http://127.0.0.1:8080
```

Verify that Quine is operating using the [System Information: `GET /api/v2/system/systemInfo`](/reference/rest-api/?av=v2#/operations/get-system-info) API endpoint.

```shell
curl -s "http://127.0.0.1:8080/api/v2/system/systemInfo" | jq '.'
{
  "version": "1.3.2",
  "gitCommit": "6f8bb1b3a308d9c90cc71a2328858907ec341e74",
  "gitCommitDate": "2022-08-10T11:01:51-0700",
  "javaVersion": "OpenJDK 64-Bit Server VM 17.0.2 (Azul Systems, Inc.)",
  "persistenceWriteVersion": "12.0.0"
}
```

You can connect to the Quine exploration UI by entering `http://127.0.0.1:8080` into your browser.

### Shutdown Quine

You can stop Quine at any time by either typing `CTRL-c` into the terminal window or gracefully shutting down by issuing a POST to the [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) endpoint.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/system:shutdown"
```

---

## Source Code

Quine is an open source project and can be built from source code.

Build Quine from source code if …

* You want to review the code as part of your evaluation
* You are interested in contributing to the Quine open source project
* You need an enhancement or issue resolution that was delivered between releases

### Prerequisites

Quine is written and Scala. The UI is built on React and building it depends on NodeJS. Please review the [README.md](https://github.com/thatdot/quine) file in the GitHub repository for most up to date requirements to build Quine from source.

* Java JDK 11 or greater
* Scala SBT version 1.7.1 or greater
* Node version 16
* Yarn version 0.22.0 or greater

### Clone the Quine repository from GitHub

The Quine open source project is managed on GitHub, and the repo is located at [https://github.com/thatdot/quine](https://github.com/thatdot/quine) .

```shell
❯ mkdir quineEvaluation
❯ cd quineEvaluation
❯ git clone git@github.com:thatdot/quine.git
Cloning into 'quine'...
```

### Build Quine

```shell
❯ cd quine
❯ nvm use 16
Now using node v16.15.1 (npm v8.11.0)
❯ sbt quine/assembly
[info] welcome to sbt 1.7.1 (Homebrew Java 11.0.16.1)
[info] loading settings for project quine-build from plugins.sbt
... <build details removed> ...
[success] Total time: 78 s (01:18), completed Aug 25, 2022, 9:54:42 AM
```

If successful, the `sbt` tool will configure your environment, build, and package Quine into a `jar` file located in the target directory.

```shell
❯ find . -name "quine-*.jar"
./quine/target/scala-2.12/quine-assembly-1.3.2+n.jar
```

### Start Quine

Once that you have successfully built and assembled the Quine `jar` file, we recommend copying that file into a working directory to make it easier to evaluate Quine.

```shell
❯ cp $(find . -name "quine-assembly-*.jar") ..
❯ cd ..
❯ ls
quine/  quine-assembly-1.3.2+n.jar
```

Then you can launch Quine in the same way that you would launch the executable file.

```shell
❯ java -jar quine-assembly-1.3.2+n.jar
```

If successful, you will see a message similar to the following appear in the terminal.

```text
Graph is ready
Quine web server available at http://127.0.0.1:8080
```

Alternatively, you can run Quine directly from the source code with `sbt quine/run`.

```text
❯ sbt quine/run
[info] welcome to sbt 1.7.1 (Homebrew Java 11.0.16.1)
[info] loading settings for project quine-build from plugins.sbt ...
...
[info] running com.thatdot.quine.app.Main
Graph is ready
Quine web server available at http://127.0.0.1:8080
```

Regardless of the method that you choose to launch Quine, you can verify that it is operating using the [System Information: `GET /api/v2/system/systemInfo`](/reference/rest-api/?av=v2#/operations/get-system-info) API endpoint.

```shell
❯ curl -s "http://127.0.0.1:8080/api/v2/system/systemInfo" | jq '.'
{
  "version": "1.3.2+n",
  "gitCommit": "45dc9789b344b5c282cab227c560c45bc1a883b5",
  "gitCommitDate": "2022-08-24T19:29:11+0000",
  "javaVersion": "OpenJDK 64-Bit Server VM 11.0.16.1 (Homebrew)",
  "persistenceWriteVersion": "12.0.0"
}
```

You can connect to the Quine exploration UI by entering `http://127.0.0.1:8080` into your browser.

### Shutdown Quine

You can stop Quine at any time by either typing `CTRL-c` into the terminal window or gracefully shutting down by issuing a POST to the [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) endpoint.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/system:shutdown"
```

---

## Cloud Hosted / SaaS

Please contact the thatDot team using the link below to set up a call if you are interested in the Quine-Enterprise edition or a SaaS hosted option.

[Contact thatDot](https://www.thatdot.com/contact-us/pricing-request)


---
# Recipe Quickstart
URL: https://quine.io/getting-started/recipes-tutorial/

# Recipe Quickstart

In this article we will cover Quine recipes -- what are they for, their components, and how to use them to quickly iterate through the design-code-test portion of the development lifecycle.

## What is a Quine Recipe

A [recipe](../learn/recipe-ref-manual.md) is a collection of configurations that sets a Quine instance up for a specific purpose. A recipe is defined in a 'yaml' file and contains configuration for: ingest streams, standing queries, UI configuration, and some metadata about the recipe. Recipes are a great format for sharing what you've created, and for quickly iterating on your thoughts. Quine recipe components correspond directly to Quine's REST API elements. This means patterns and behaviors developed in a 'dev' environment can be applied to the production environment with only slight modifications.

### Some differences between using recipes vs using API calls:

API calls allow for naming of ingest streams and standing queries. In recipes, they are named for you in the format of INGEST-# or STANDING-#
By default, Quine launched with recipes creates a temporary persistent data store (in your tmp dir). Each subsequent launch of Quine w/ a recipe replaces this temporary data store

These differences between API calls and Recipes make them well suited for different things:

* Recipes are fantastic at quick iteration. No need to name your ingests and standing queries, and your data store is automatically cleaned up when Quine is restarted
* API calls are the way to setup Quine in production. You want to persist your data, even between Quine restarts, and you will want to name your ingest streams and standing queries

**A recipe allows you to:**

1. Configure ingest streams.
2. Configure standing queries.
3. Configure the Quine Exploration UI for graph analysis.
4. Iterate rapidly during the development phase.

Recipes are covered in detail inside the [recipe reference](../learn/recipe-ref-manual.md)

## Differences Between Recipes and the REST API

There are a couple of operational differences that you need to keep in mind when launching Quine along with a recipe.

* **API calls allow naming** - When you create an ingest stream or a standing query using the API, you choose the name for the object in the URL. The corresponding recipe object uses standardized names in the form of `INGEST-#` and `STANDING-#`.

* **Persistent storage** - Starting the Quine application `jar` without providing a configuration file creates a persistent data store in the local directory. That persistor retains the previous graph state and is appended to each time Quine starts. Alternatively, when Quine launches a recipe, a temporary persistent data store is created in the system `tmp` directory. Each subsequent launch of the recipe will replace the data store, and discard the graph from the previous run. You can override the default temporary storage behavior with your configuration settings by using the `--force-config` command line flag.

## Recipe Structure

A recipe is stored in a single `YAML` text file. The file must contain a single object with the following attributes:

| Attribute         | Type                                                                                                         | Description                                                                                 |
| ----------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| `version`         | Integer                                                                                                      | Schema versioning (1 or 2)                                                |
| `title`           | String                                                                                                       | Identifies the Recipe                                                                       |
| `contributor`     | String                                                                                                       | URL to social profile of the person or organization responsible for this Recipe             |
| `summary`         | String                                                                                                       | Brief information about this Recipe                                                         |
| `description`     | Text Block                                                                                                   | Long form description about this Recipe                                                     |
| `ingestStreams`   | Array of [IngestStream](/reference/rest-api/?av=v2#/operations/create-ingest) objects | Define how data is read from data sources, transformed, and loaded into the graph           |
| `standingQueries` | Array of [StandingQuery](/reference/rest-api/?av=v2#/operations/create-standing-query) objects  | Define both sub-graph patterns for Quine to match and subsequent output actions             |
| `nodeAppearances` | Array of [NodeAppearance](/reference/rest-api/?av=v2#/operations/replace-node-appearances) objects        | Customize node appearance in the exploration UI                                             |
| `quickQueries`    | Array of [QuickQuery](/reference/rest-api/?av=v2#/operations/replace-quick-queries) objects            | Add queries to node context menus in the exploration UI                                     |
| `sampleQueries`   | Array of [SampleQuery](/reference/rest-api/?av=v2#/operations/replace-sample-queries) objects                | Customize sample queries listed in the exploration UI                                       |
| `statusQuery`     | A [CypherQuery](/reference/rest-api/?av=v2#/operations/query-cypher) object                        | OPTIONAL Cypher query that is executed and reported to the terminal window during execution |

Use the `yaml` structure below as a starting point when developing your recipes.

```yaml
version: 2
title: The title of the recipe goes here
contributor: Your GitHub profile link here
summary: Single line summary of your recipe
description: |-
  Long format description of your recipe
ingestStreams: [ ]
standingQueries: [ ]
nodeAppearances: [ ]
quickQueries: [ ]
sampleQueries: [ ]
statusQuery: null
```

Now let's build a recipe to reproduce the use case from the [**Getting Started**](ingest-streams-tutorial.md) scenario. Remember our goal:

>For the sake of this tutorial, assume that you need to separate human-generated events from bot-generated events in the English Wikipedia database and send them to a destination in your data pipeline for additional processing.

## Recipe Metadata

The first few lines of a Quine recipe contain information about who wrote the recipe and what the recipe is intended to do. 

Starting out with the recipe template from above, we can fill in the `version`, `title`, `contributor`, `summary`, and `description` for our recipe.

```yaml
version: 2
title: Wikipedia non-bot page update event stream
contributor: https://github.com/maglietti
summary: Stream page-update events that were not created by bots
description: |-
  This recipe will separate human generated events from bot generated
  events in the english wikipedia database page-update event stream
  and store them for additional processing.
  API Reference: https://stream.wikimedia.org/?doc#/streams/get_v2_stream_mediawiki_revision_create
```

## Ingest Stream

Ok, now we need to transform the ingest stream object that we POSTed to [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) from JSON to YAML. The simplest way to convert the JSON API body to YAML is to use a tool like [YAML ❤ JSON](https://marketplace.visualstudio.com/items?itemName=hilleer.yaml-plus-json) in VSCode.  

Here's the JSON version of the ingest stream that we developed earlier in the [ingest streams](ingest-streams-tutorial.md) getting started tutorial.

```json
{
  "name": "wikipedia-revision-create",
  "source": {
    "type": "ServerSentEvent",
    "url": "https://stream.wikimedia.org/v2/stream/mediawiki.revision-create"
  },
  "query": "MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode) WHERE id(revNode) = idFrom('revision', $that.rev_id) AND id(pageNode) = idFrom('page', $that.page_id) AND id(dbNode) = idFrom('db', $that.database) AND id(userNode) = idFrom('id', $that.performer.user_id) AND id(parentNode) = idFrom('revision', $that.rev_parent_id) SET revNode = $that, revNode.bot = $that.performer.user_is_bot, revNode:revision SET parentNode.rev_id = $that.rev_parent_id SET pageNode.id = $that.page_id, pageNode.namespace = $that.page_namespace, pageNode.title = $that.page_title, pageNode.comment = $that.comment, pageNode.is_redirect = $that.page_is_redirect, pageNode:page SET dbNode.database = $that.database, dbNode:db SET userNode = $that.performer, userNode.name = $that.performer.user_text, userNode:user CREATE (revNode)-[:TO]->(pageNode), (pageNode)-[:IN]->(dbNode), (userNode)-[:RESPONSIBLE_FOR]->(revNode), (parentNode)-[:NEXT]->(revNode)"
}
```

And the converted YAML version, edited for readability and style

```yaml
ingestStreams:
  - name: wikipedia-revision-create
    source:
      type: ServerSentEvent
      url: https://stream.wikimedia.org/v2/stream/mediawiki.revision-create
    query: |-
      MATCH (revNode),(pageNode),(dbNode),(userNode),(parentNode)
      WHERE id(revNode) = idFrom('revision', $that.rev_id) 
        AND id(pageNode) = idFrom('page', $that.page_id) 
        AND id(dbNode) = idFrom('db', $that.database)
        AND id(userNode) = idFrom('id', $that.performer.user_id) 
        AND id(parentNode) = idFrom('revision', $that.rev_parent_id)
      
      SET revNode = $that,
          revNode.bot = $that.performer.user_is_bot,
          revNode:revision

      SET parentNode.rev_id = $that.rev_parent_id
      
      SET pageNode.id = $that.page_id, 
          pageNode.namespace = $that.page_namespace, 
          pageNode.title = $that.page_title, 
          pageNode.comment = $that.comment, 
          pageNode.is_redirect = $that.page_is_redirect, 
          pageNode:page 
      
      SET dbNode.database = $that.database, 
          dbNode:db 
      
      SET userNode = $that.performer, 
          userNode.name = $that.performer.user_text, 
          userNode:user 
      
      CREATE (revNode)-[:TO]->(pageNode),
             (pageNode)-[:IN]->(dbNode),
             (userNode)-[:RESPONSIBLE_FOR]->(revNode),
             (parentNode)-[:NEXT]->(revNode)
```

## Standing Query

We can transform the standing query from the [standing queries](standing-queries-tutorial.md) tutorial the same way that we transformed the ingest stream.

JSON:

```json
{
  "name": "not-a-bot",
  "pattern": {
    "query": "MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: 'enwiki'}) RETURN DISTINCT id(revNode) as id",
    "type": "Cypher"
  },
  "outputs": [
    {
      "name": "print-output",
      "resultEnrichment": {
        "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN properties(n)",
        "parameter": "that"
      },
      "destinations": [
        { "type": "StandardOut" }
      ]
    }
  ]
}
```

YAML:

```yaml
standingQueries:
  - name: not-a-bot
    pattern:
      query: |-
        MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: 'enwiki'})
        RETURN DISTINCT id(revNode) as id
      type: Cypher
    outputs:
      - name: print-output
        resultEnrichment:
          query: |-
            MATCH (n)
            WHERE id(n) = $that.data.id
            RETURN properties(n)
          parameter: that
        destinations:
          - type: StandardOut
```

At this point, our recipe will produce exactly the same running configuration that we accomplished with the API calls submitted during the previous sections in this getting started tutorial.

## Running a recipe

Recipes are launched by passing the `YAML` file as an argument to the Quine `jar` file using `-r`.

I saved the recipe elements that we created above into a file named [wikipedia-non-bot-revisions.yaml](https://github.com/thatdot/quine/blob/main/quine/recipes/wikipedia-non-bot-revisions.yaml)  on my laptop. The recipe file is in the same directory that I have the `quine-x.x.x.jar` file.

Launching Quine and the recipe.

```shell
❯ java -jar quine-2.1.1.jar -r wikipedia-non-bot-revisions.yaml
Graph is ready
Running Recipe Wikipedia non-bot page update event stream
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Quine web server available at http://127.0.0.1:8080
```

Notice that Quine announced the recipe title, provided the names it generated for the standing query and ingest stream, then immediately began outputting revision events to the console.

```text
2022-08-30 09:13:58,636 Standing query `print-output` match: {"meta":{"isPositiveMatch":true,"resultId":"9f4650d3-eb01-4a95-9836-fa17e932d430"},"data":{"properties(n)":{"$schema":"/mediawiki/revision/create/1.1.0","comment":"z","database":"enwiki","meta":{"domain":"en.wikipedia.org","dt":"2022-08-30T14:13:57Z","id":"63a721ff-a3d0-4ab8-a9c2-0890642a2696","offset":2843700515,"partition":0,"request_id":"de759ed9-afb4-4c78-812c-70255d2c6b6b","stream":"mediawiki.revision-create","topic":"eqiad.mediawiki.revision-create","uri":"https://en.wikipedia.org/wiki/User:Peter_I._Vardy/sandbox"},"page_id":8188575,"page_is_redirect":false,"page_namespace":2,"page_title":"User:Peter_I._Vardy/sandbox","parsedcomment":"z","performer":{"user_edit_count":205321,"user_groups":["autoreviewer","extendedconfirmed","reviewer","*","user","autoconfirmed"],"user_id":2675188,"user_is_bot":false,"user_registration_dt":"2006-11-06T14:56:58Z","user_text":"Peter I. Vardy"},"rev_content_changed":true,"rev_content_format":"text/x-wiki","rev_content_model":"wikitext","rev_id":1107535873,"rev_len":6319,"rev_minor_edit":false,"rev_parent_id":1107530092,"rev_sha1":"j67dl0l3cxot84gy2qft4rs2jnsb3cz","rev_slots":{"main":{"rev_slot_content_model":"wikitext","rev_slot_origin_rev_id":1107535873,"rev_slot_sha1":"j67dl0l3cxot84gy2qft4rs2jnsb3cz","rev_slot_size":6319}},"rev_timestamp":"2022-08-30T14:13:57Z"}}}
```

Let's pause the ingest stream for now to stop the scrolling in our terminal window. Remember that for recipes, the name of the ingest stream is assigned when the recipe is run using [Pause Ingest Stream: `POST /api/v2/graph/quine/ingests/{ingestName}:pause`](/reference/rest-api/?av=v2#/operations/pause-ingest).

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/ingests/INGEST-1:pause"
```

Also notice that interleaved with the revision events, Quine displays a running total of events processed by the ingest stream and standing query. These are easier to see once the ingest stream is paused.

```text
 | => STANDING-1 count 50
 | => INGEST-1 status is paused and ingested 715
```

Depending on your application, you may choose to leave the `nodeApperances`, `quickQueries` and `sampleQueries`  arrays empty. When configured, they can improve the readability and ease of analysis of the streaming graph within the exploration UI. We cover the Exploration UI in another tutorial.

## Status Query

Depending on your event stream, including an optional status query in your recipe can provide status updates to the terminal window while a less frequent event is waiting for a standing query match. Consider including a `statusQuery` in your recipe if you need to track more complex metrics about events processed by the ingest stream. Remember that a recipe will output the count of events processed by a standing query by default.

```yaml
statusQuery: 
  cypherQuery: MATCH (n) RETURN distinct labels(n), count(*)
```

Do not include the `statusQuery` attribute in your recipe file if you do not intend to use it.

## Next Steps

Great job making it this far. You should now have the fundamental knowledge to add Quine into an event streaming data pipeline.

As your recipes grow, the [Quine Recipe Analyzer](https://www.thatdot.com/quine/recipe-analyzer.html) can visualize a recipe's ingests, graph structure, and standing queries, and make recommendations for improving it.

Have a question, suggestion, or did you get stuck somewhere? We welcome your feedback! Please join the [Quine Community](https://that.re/chat) and let us know. The team is always happy to discuss Quine and answer your questions.


---
# Standing Queries Quickstart
URL: https://quine.io/getting-started/standing-queries-tutorial/

# Standing Queries Quickstart

A standing query is a feature unique to Quine. Standing queries monitor streams for specified patterns, maintaining partial matches, and executing user-specified actions the instant a full match is made. Actions can include anything from updating the graph itself by creating new nodes or edges, writing results out to Kafka or Kinesis, or posting results to a webhook. Whatever fits your streaming data pipeline.

Standing queries are a unique (and we think powerful) feature of Quine. Standing queries are key to how Quine can achieve sub-millisecond performance finding complex patterns (or subgraphs) in high volume event streams. Standing queries are also a key reason why Quine does not impose time windows, setting it apart from other event processing systems.

## Format of a standing query

A standing query is defined in two parts: a **pattern** and an **output**. The **pattern** defines what we want to match, expressed in Cypher using the form `MATCH … WHERE … RETURN …`. The **output** defines the action(s) to take for each result produced by the `RETURN`  in the pattern query.

In general, a standing query definition JSON object takes form as in this example.

```json
{
  "name": "kids-with-dads",
  "pattern": {
    "query": "MATCH (n)-[:has_father]->(m) WHERE n.name IS NOT NULL AND m.name IS NOT NULL RETURN DISTINCT strId(n) AS kidWithDad",
    "type": "Cypher"
  },
  "outputs": [
    {
      "name": "file-of-results",
      "destinations": [
        { "type": "File", "path": "kidsWithDads.jsonl" }
      ]
    }
  ]
}
```

A StandingQueryResult is an object with 2 sub-objects: `meta` and `data`. The `meta` object consists of:

* UUID `resultId`
* boolean `isPositiveMatch`

When the `pattern` query makes a positive match, the `data` object consists of the data returned by the Standing Query.

For example, a StandingQueryResult may look like the following:

```json
{
    "meta": {
        "resultId": "b3c35fa4-2515-442c-8a6a-35a3cb0caf6b",
        "isPositiveMatch": true
    },
    "data": {
        "kidWithDad": "a0f93a88-ecc8-4bd5-b9ba-faa6e9c5f95d"
    }
}
```

There are many standing query destination types. The [API documentation](/reference/rest-api/?av=v2#/operations/create-standing-query-output)  defines the format for each destination type.  

* `StandardOut` - Log JSON to Console
* `File` - Log JSON to File
* `HttpEndpoint` - POST to HTTP(S) Webhook
* `Kafka` - Publish to Kafka Topic
* `Kinesis` - Publish to Kinesis Stream
* `SNS` - Publish to SNS Topic
* `Slack` - Publish to Slack Webhook
* `CypherQuery` - Run a Cypher Query
* `Drop` - Discard results

Each output workflow can optionally include a `resultEnrichment` step to enrich data with a Cypher query before sending to destinations.

## Writing a standing query

Remember the scenario that we developed in the [ingest stream](ingest-streams-tutorial.md) tutorial:

>For the sake of this tutorial, assume that you need to separate human-generated events from bot-generated events in the English Wikipedia database and send them to a destination in your data pipeline for additional processing.

Our ingest stream is processing events as they arrive from the `mediawiki.revision-create` event source and manifesting them as nodes in the graph.

Our task is to find the human-generated events (not bots) in the stream, separate them from the rest of the stream and send them to another service for processing.

1. Write a Cypher query to match our scenario.

Using the Exploration UI we can develop a Cypher query that returns the last 10 `revision-create` event nodes.

```cypher
MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: 'enwiki'})
RETURN DISTINCT strid(userNode) as NodeID,
  revNode.page_title as Title,
  revNode.performer.user_text as User
LIMIT 10
```

![Quine Exploration UI](images/not-a-bot-query.png)

We can use this Cypher query to develop the standing query.

1. Write the `pattern`  portion of the standing query

```cypher
MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: 'enwiki'})
RETURN DISTINCT id(revNode) as id
```

2. Write the `output` portion of the standing query

```cypher
MATCH (n)
WHERE id(n) = $that.data.id
RETURN properties(n)
```

3. Form the standing query JSON object

```json
{
  "name": "not-a-bot",
  "pattern": {
    "query": "MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: 'enwiki'}) RETURN DISTINCT id(revNode) as id",
    "type": "Cypher"
  },
  "outputs": [
    {
      "name": "print-output",
      "resultEnrichment": {
        "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN properties(n)",
        "parameter": "that"
      },
      "destinations": [
        { "type": "StandardOut" }
      ]
    }
  ]
}
```

For simplicity, I am sending the output from the standing query to the console. You could easily send the output to another server via a webhook.

4. POST the standing query via `curl` to [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query) into Quine for processing.

```shell
curl -X "POST" "http://127.0.0.1:8080/api/v2/graph/quine/standingQueries" \
     -H 'Content-Type: application/json' \
     -d $'{
  "name": "not-a-bot",
  "pattern": {
    "query": "MATCH (userNode:user {user_is_bot: false})-[:RESPONSIBLE_FOR]->(revNode:revision {database: \'enwiki\'}) RETURN DISTINCT id(revNode) as id",
    "type": "Cypher"
  },
  "outputs": [
    {
      "name": "print-output",
      "resultEnrichment": {
        "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN properties(n)",
        "parameter": "that"
      },
      "destinations": [
        { "type": "StandardOut" }
      ]
    }
  ]
}'
```

Almost immediately you should see "non-bot" generated events appear in the terminal window similar to the one below.

```shell
2022-08-26 16:21:37,835 Standing query `print-output` match: {"meta":{"isPositiveMatch":true,"resultId":"bb9249d7-5102-4104-afce-c07a3d237546"},"data":{"properties(n)":{"$schema":"/mediawiki/revision/create/1.1.0","comment":"clean up","database":"enwiki","meta":{"domain":"en.wikipedia.org","dt":"2022-08-26T21:21:36Z","id":"7836bfc6-7372-4b6a-bd65-a3ed0a62f5de","offset":2838029929,"partition":0,"request_id":"b102aaf2-cd98-4829-a8fd-741663104215","stream":"mediawiki.revision-create","topic":"eqiad.mediawiki.revision-create","uri":"https://en.wikipedia.org/wiki/Luke_Harris"},"page_id":71585932,"page_is_redirect":false,"page_namespace":0,"page_title":"Luke_Harris","parsedcomment":"clean up","performer":{"user_edit_count":275949,"user_groups":["extendedconfirmed","*","user","autoconfirmed"],"user_id":14841472,"user_is_bot":false,"user_registration_dt":"2011-06-26T21:29:06Z","user_text":"Joeykai"},"rev_content_changed":true,"rev_content_format":"text/x-wiki","rev_content_model":"wikitext","rev_id":1106862807,"rev_len":3832,"rev_minor_edit":true,"rev_parent_id":1106729293,"rev_sha1":"ltx1i9nlezpw1wvak4mueahi6v3f155","rev_slots":{"main":{"rev_slot_content_model":"wikitext","rev_slot_origin_rev_id":1106862807,"rev_slot_sha1":"ltx1i9nlezpw1wvak4mueahi6v3f155","rev_slot_size":3832}},"rev_timestamp":"2022-08-26T21:21:36Z"}}}
```

## Next Steps

Congratulations! You have a stream of non-bot generated `revision-create` events isolated from the main event stream and you are displaying them in your terminal.

Up to this point, we've been using the REST API to interact with Quine. In the next section, we will collect all of the API calls that we've made and build a [recipe](recipes-tutorial.md).


---
# Streams
URL: https://quine.io/getting-started/streams/

# Streams

The Streams page is where you manage **ingest streams** and **standing queries** for your Quine data pipeline. It is accessible from the left sidebar.

!!! info "API v2 Required"
    The Streams page is available only when the v2 API is enabled. It does not appear in the sidebar when running in API v1 mode.

## Overview

The Streams page is organized into four panels:

- **Ingest Streams**: Create, monitor, pause, resume, and delete ingest streams that feed data into the streaming graph.
- **Standing Queries**: Create, monitor, and delete standing queries that watch the graph for patterns and route results to outputs.
- **Background Queries**: Run Cypher queries out-of-band, watch their results arrive, and cancel them.
- **Scheduled Jobs**: Create and delete jobs that dispatch a background query on a recurring schedule.

Each panel has a list view and a creation form. All panels auto-refresh to keep stats current.

![Streams page showing ingest streams in various states and a standing query](./streams-ui/streams-overview.png)

## Ingest Streams Panel

The Ingest Streams panel displays all configured ingest streams in a table with the following columns:

| Column     | Description                                                                 |
|:-----------|:----------------------------------------------------------------------------|
| **Name**   | The unique name assigned to the ingest stream                               |
| **Type**   | The source type (e.g., `Kafka`, `Kinesis`, `FileIngest`, `StdInput`)        |
| **Status** | Current state: `Running`, `Paused`, `Restored`, `Completed`, or `Failed`   |
| **Ingested** | Total number of records processed                                         |
| **Rate (1m)** | One-minute rolling ingest rate (records per second)                      |
| **Uptime** | How long the stream has been active                                         |
| **Actions** | Controls to pause, resume, or delete the stream                            |

### Ingest Stream Status

Each ingest stream displays a color-coded status badge:

- **Running** (green): The stream is actively consuming records.
- **Paused** / **Restored** (yellow): The stream is temporarily halted and can be resumed.
- **Completed** (gray): The source was fully consumed (e.g., a finite file or a number iterator with a limit).
- **Terminated** (gray): The stream was stopped by a user or API call.
- **Failed** (red): The stream encountered an error. The entire row highlights in red with the error message displayed. Only the delete action is available.

### Managing Ingest Streams

- **Pause**: Temporarily halt an ingest stream. The stream can be resumed later.
- **Resume**: Continue a paused or restored stream.
- **Delete**: Remove the stream entirely.

### Creating an Ingest Stream

Click **+ New Ingest** to switch to the creation form. The form uses a two-step workflow:

**Step 1: Source Type.** Select the type of system that will feed data into your stream. Available source types include Kafka, Kinesis, SQS, files, standard input, S3, server-sent events, WebSocket, and more.

![Step 1: Choose a source type from the available ingest sources](./streams-ui/ingest-source-types.png)

**Step 2: Configure.** Fill in the source-specific settings (such as topic names, bootstrap servers, or file paths), the ingest query, and any optional settings like record decoders or error handling. The form fields adapt to the source type selected in Step 1.

![Step 2: Configure the Kafka ingest stream with topics, bootstrap servers, format, and query](./streams-ui/ingest-configure-kafka.png)

Provide a unique **Name** for the stream and click **Create Ingest Stream** to submit.

Click **Back** at any step to return to the previous step or cancel the creation.

See [Ingest Streams](../learn/ingest-sources/index.md) for configuration options, source types, and error handling.

## Standing Queries Panel

The Standing Queries panel displays all configured standing queries in a table with the following columns:

| Column     | Description                                                        |
|:-----------|:-------------------------------------------------------------------|
| **Name**   | The unique name assigned to the standing query                     |
| **Pattern** | The Cypher pattern query (truncated in the table, full text on hover) |
| **Mode**   | The matching mode for the query                                    |
| **Outputs** | Number of output destinations configured                          |
| **Rate (1m)** | One-minute rolling match rate                                   |
| **Actions** | Control to delete the standing query                              |

### Creating a Standing Query

Click **+ New Standing Query** to switch to the creation form. The form collects:

- **Name**: A unique identifier for the standing query.
- **Pattern**: The Cypher `MATCH` / `RETURN` query that defines what graph structure to watch for.
- **Mode**: The matching mode (e.g., `Distinct ID`).
- **Outputs**: One or more output destinations with optional result enrichment queries.
- **Additional options**: Such as `includeCancellations` and `inputBufferSize`.

![Standing query creation form with name, pattern, mode, and outputs fields](./streams-ui/sq-create-form.png)

Click **Create Standing Query** to submit.

See [Standing Queries](../learn/standing-queries/standing-queries.md) for standing query patterns, outputs, and query enrichment.

## Background Queries Panel

The Background Queries panel lists one-off [background queries](../learn/background-queries/background-queries.md) cypher queries running out-of-band against the selected graph. In-flight runs are listed first, then the most recent.

Runs dispatched by a [scheduled job](../learn/background-queries/scheduled-jobs.md) are **not** listed here; they appear under their job in the Scheduled Jobs panel instead, so a frequent schedule cannot bury the handful of runs a person actually started.

![The Background Queries panel](./streams-ui/background-query-panel.png)

The table has the following columns:

| Column | Description |
|:-------|:------------|
| **Query** | The run's name if it has one, otherwise the Cypher text |
| **Status** | `Running`, `Completed`, `Failed`, `Cancelled`, or `Interrupted` |
| **Rows** | Total rows streamed, once the run completes |
| **Host** | The host executing the run |
| **Expires** | When the run's status record will be swept |
| **Actions** | Controls to cancel a running query, or delete its record |

### Running a Background Query

Click **+ New Background Query** to open the creation form. The form is generated from the server's own API schema, so it offers the full set of destinations (Kafka, Kinesis, SNS, HTTP endpoints, files, Cypher queries, etc) with the same fields and validation as the standing query output forms.

Destinations default to **Drop**, which runs the query without writing its results anywhere. That is usually what you want when you intend to watch the results here rather than route them somewhere.

![The background query creation form](./streams-ui/background-query-create-form.png)

### Inspecting Results

Expand a run to open an inspection: a live view of the rows the query is producing, in the same widget the Standing Queries panel uses for wiretaps. Results stream in as they are produced and the view reports when the run terminates.

Starting an inspection is a read, so it needs no write permission unlike cancelling the run.

![An expanded background query streaming its results](./streams-ui/background-query-inspection.png)

The same run can be inspected from more than one place at once (here and in the [Exploration UI](exploration-ui.md#running-a-query-in-the-background)) without either view interrupting the other.

!!! note "Results are best-effort"
    The live view is a diagnostic tool with no delivery guarantees: rows are dropped rather than buffered if the browser cannot keep up. The **Rows** column is the authoritative count.

## Scheduled Jobs Panel

The Scheduled Jobs panel lists [scheduled jobs](../learn/background-queries/scheduled-jobs.md) which are named schedules that dispatch a background query each time they fire.

![The Scheduled Jobs panel](./streams-ui/jobs-panel.png)

The table has the following columns:

| Column | Description |
|:-------|:------------|
| **Name** | The unique name identifying the job |
| **Type** | The kind of work the job dispatches |
| **Schedule** | The job's schedule, rendered from its definition |
| **Next fire** | When the job will next fire |
| **Last fire** | When the job most recently fired |
| **Status** | `Running` when a dispatched execution is currently in flight |
| **Actions** | Control to delete the job |

Expand a job to see the background-query executions it has dispatched against the selected graph. Each of those runs can be inspected, cancelled, and deleted exactly as in the Background Queries panel.

### Creating a Job

Click **+ New Job** to open the creation form, which collects:

- **Name**: A unique identifier for the job.
- **Schedule**: `Interval` for a fixed cadence, or `Hourly` / `Daily` / `Weekly` / `Monthly` for a wall-clock recurrence in a named timezone.
- **Action**: `BackgroundQuery`, with the target graph, the Cypher query, and its destinations.

![The scheduled job creation form](./streams-ui/job-create-form.png)

To edit a job one, delete it and create it again, or use `updateIfExists` on the [create endpoint](../learn/background-queries/scheduled-jobs.md#replacing-a-job).

See [Scheduled Jobs](../learn/background-queries/scheduled-jobs.md) for schedule types, timezone and daylight saving behavior, and delivery guarantees.

## Viewing Configuration

Click the chevron on any row in either panel to expand it and view the full JSON configuration for that resource.

![Expanded ingest stream row showing the full JSON configuration](./streams-ui/ingest-expanded-config.png)

For standing queries, the expanded view also includes:

- **Configuration**: The full JSON configuration, nested under a collapsible header.
- **Outputs**: A sub-table listing each output destination by name and type. Each output can be individually removed.

Click **+ Add Output** from the expanded view to add an output to an existing standing query.

![Expanded standing query showing the outputs table and Add Output button](./streams-ui/sq-expanded-outputs.png)

## Next Steps

- [Ingest Streams](../learn/ingest-sources/index.md): Detailed reference for all ingest source types, formats, error handling, and the API.
- [Standing Queries](../learn/standing-queries/standing-queries.md): How standing queries work, pattern constraints, output destinations, and result enrichment.
- [Background Queries](../learn/background-queries/background-queries.md): Running long Cypher queries out-of-band, streaming results, and watching them live.
- [Scheduled Jobs](../learn/background-queries/scheduled-jobs.md): Schedule types, timezone behavior, and delivery guarantees.
- [Exploration UI](exploration-ui.md): Explore the streaming graph interactively with Cypher queries on an interactive canvas.


---
# index
URL: https://quine.io/learn/

# Learn Quine

Quine is a stateful streaming graph interpreter. It consumes high volume data streams and publishes processed results to other streaming data consumers.

Quine eliminates the complex technical challenges of managing data ordering, time windowing, vertical and horizontal scalability, and the complex asynchronous processing needed to find compound objects or patterns spread across data streams.

Quine is easily integrated into existing data pipelines and highly scalable across existing and next-generation enterprise infrastructure.

-   [:material-download: __Install Quine__](../getting-started/installing-quine-tutorial.md)

    ---

    Install Quine by downloading a `jar` file, pulling a container from Docker, or building it form source code.

-   [:material-clock-fast: __Getting Started__](../getting-started/ingest-streams-tutorial.md)

    ---

    Connect to an event source, shape events into a graph, inspect data, and develop business logic in minutes.

-   [:material-book: __Core Concepts__](../core-concepts/index.md)

    ---

    If you're new to Quine, or just want a refresher on the fundamentals, you've come to the right place. These docs introduce you to the concepts behind Quine and its architecture.

-   [:material-api: __REST API Reference__](../reference/rest-api.md)

    ---

    Use the REST API endpoints to implement data pipelines, retrieve data, and operate a Quine instance.

-   [:material-graph: __OpenCypher Reference__](cypher/index.md)

    ---

    Cypher is the most widely use query language for interacting with data in a property graph format.

Please reach out to the team in [Discord](https://that.re/chat) if the reference docs do not provide what you are looking for.


---
# Gremlin Language
URL: https://quine.io/learn/gremlin-language/

# Gremlin Language

!!! warning "Gremlin is only available via API v1"
    Gremlin is reachable only through API v1 endpoints. API v1 is planned for deprecation and will be removed in a future release; Gremlin support will go with it. Use [Cypher](cypher/index.md) instead, which provides equivalent functionality with better performance and more complete feature support. See [Migrating from API v1](../reference/upgrade/migrating-from-api-v1.md#gremlin-endpoints) for details.

[Gremlin](https://tinkerpop.apache.org/gremlin.html) is a graph query language, but one that is less declarative than [Cypher Language](cypher/index.md) and more focused on letting users specify exactly the traversal they want. The main strength that Gremlin has is that one of its focuses is traversals: instructions for how to walk the graph structure given some starting points.

When API v1 is configured as the default (`default-api-version = "v1"`), the [Exploration UI](../getting-started/exploration-ui.md) supports querying nodes in Gremlin or Cypher interchangeably, and quick queries can be defined in Gremlin. With the default setting of `"v2"`, the Exploration UI uses Cypher; Gremlin remains accessible only via direct calls to the `/api/v1/query/gremlin*` endpoints.

!!! Note

    Quine supports only a subset of Gremlin, and uses a custom language parser to do so. It is much faster, but less feature-full than the Gremlin Server application provided in the Tinkerpop package. The source code defining what is supported in Quine's use of Gremlin is found in `GremlinParser.scala`. 

    The parts of Gremlin that are implemented are not guaranteed to be compliant. Part of the difficulty here is that some parts of Gremlin were designed to be executed form inside a host language, usually Groovy, and don't extend naturally to remote execution (see for instance [this section](https://tinkerpop.apache.org/docs/3.4.7/reference/#-the-lambda-solution-3) of the Gremlin manual for some complexities around anonymous functions).

## Query Start

All supported Gremlin queries begin in one of two ways:

- `g.`  This "g" refers to the "graph" and can take any traversal step following it. This is the primary use case.
- Assignment to a variable followed by a semi-colon, then another supported Gremlin query. E.g.: `x = 1234; g.V(x)`

## Expressions

- `idFrom( [values] )`
- literal value
- list of values: `[`value`,`value`,`…`]`

## Predicates

- `eq`
- `neq`
- `within`
- `regex`

## Traversal Steps

- `v()` or `V()`
- `v(id)` or `V(id)` where `id` is one or more node IDs.
- `recentV`
- `has`
- `hasNot`
- `hasLabel`
- `hasId`
- `eqToVar`
- `out`
- `outLimit`
- `in`
- `inLimit`
- `both`
- `bothLimit`
- `outE`
- `outELimit`
- `inE`
- `inELimit`
- `bothE`
- `bothELimit`
- `outV`
- `inV`
- `bothV`
- `values`
- `valueMap`
- `dedup`
- `as`
- `select`
- `limit`
- `id`
- `strId`
- `unrollPath`
- `count`
- `groupCount`
- `not`
- `where`
- `or`
- `and`
- `is`
- `union`


---
# Recipes
URL: https://quine.io/learn/recipe-ref-manual/

# Recipes

## What is a Quine Recipe

For Quine, a recipe is a document that contains all of the information necessary for Quine to execute any batch or streaming data process. Quine recipes are written in `yaml` and built from components including:

* [**Ingest Streams**](../learn/ingest-sources/index.md) to read streaming data from sources and update graph data
* [**Standing Queries**](../learn/standing-queries/standing-queries.md) to transform graph data, and to produce aggregates and other outputs
* [**Cypher expressions**](cypher/index.md) to implement graph operations such as querying and updating data
* [**Exploration UI**](../getting-started/exploration-ui.md) configuration to customize the web user interface for the use-case that is the subject of the recipe

> You can learn how to write a recipe in the [recipes tutorial](../getting-started/recipes-tutorial.md) section in the [getting started guide](../getting-started/index.md).

## When to use a Recipe

Recipes enable you to quickly iterate in Quine and to share what you've built with others so they can reproduce, explore and expand upon a solution.

**Consider writing a recipe:**

* When you are actively developing an event streaming solution.
* To preserve a solution whenever a development milestone is achieved.
* To configure the visual aspects of the Quine Exploration UI.
* To store `quick queries` and `sample queries` that aide in graph analysis.
* To share your solution with collaborators or the open source community.
* When interacting with Quine support.

**Recipes are NOT useful for production workloads. Once you are ready to put the recipe into production, you need to convert it to API calls because:**

* API calls allow for naming of ingest streams and standing queries. In recipes, they are named for you in the format of INGEST-# or STANDING-#.
* By default, Quine launched with recipes creates a temporary persistent data store (in your tmp dir). Each subsequent launch of Quine w/ a recipe replaces this temporary data store.

## Recipe Structure

A recipe is stored in a single `YAML` text file. The file contains a single object with the following attributes:

=== "v1"

    ```yaml
    version: 1
    title: recipe title
    contributor: https://github.com/example-user
    summary: ""
    description: ""
    ingestStreams: []
    standingQueries: []
    nodeAppearances: []
    quickQueries: []
    sampleQueries: []
    statusQuery: null
    ```

=== "v2"

    ```yaml
    version: 2
    title: recipe title
    contributor: https://github.com/example-user
    summary: ""
    description: ""
    ingestStreams: []
    standingQueries: []
    nodeAppearances: []
    quickQueries: []
    sampleQueries: []
    statusQuery: null
    ```

Each configuration object is defined by its corresponding API entity in the REST API. A v1 recipe uses the [v1 API](/reference/rest-api/?av=v1) entities, and a v2 recipe uses [v2 API](/reference/rest-api/?av=v2) entities. The two versions differ in structure, type names, and enum casing — see [Migrating from v1 Recipes](../reference/upgrade/migrating-from-recipe-v1.md) for a complete mapping. Follow the links in the table below for details regarding each attribute.

| Attribute | Type (v1) | Type (v2) | Description |
| --------- | --------- | --------- | ----------- |
| `version` | integer | integer | The recipe schema version (either `1` or `2`) |
| `title` | string | string | Identifies the recipe |
| `contributor` | string | string | URL to social profile of the person or organization responsible for this recipe |
| `summary` | string | string | Brief information about this recipe |
| `description` | string | string | Long form description about this recipe |
| `ingestStreams` | array of [IngestStreamConfiguration](/reference/rest-api/?av=v1#/paths/POST/api/v1/ingest/%7Bname%7D) | array of [QuineIngestConfiguration](/reference/rest-api/?av=v2#/operations/create-ingest) | Define how data is read from data sources, transformed, and loaded into the graph |
| `standingQueries` | array of [StandingQuery](/reference/rest-api/?av=v1#/paths/POST/api/v1/query/standing/%7Bstanding-query-name%7D) | array of [StandingQuery](/reference/rest-api/?av=v2#/operations/create-standing-query) | Define both sub-graph patterns for Quine to match and subsequent output actions |
| `nodeAppearances` | array of [UiNodeAppearance](/reference/rest-api/?av=v1#/paths/PUT/api/v1/query-ui/node-appearances) | array of [UiNodeAppearance](/reference/rest-api/?av=v2#/operations/replace-node-appearances) | Customize node appearance in the exploration UI |
| `quickQueries` | array of [QuickQueryAction](/reference/rest-api/?av=v1#/paths/PUT/api/v1/query-ui/quick-queries) | array of [QuickQuery](/reference/rest-api/?av=v2#/operations/replace-quick-queries) | Add queries to node context menus in the exploration UI. Each query's `querySuffix` receives the clicked node as variable `n`. |
| `sampleQueries` | array of [SampleQuery](/reference/rest-api/?av=v1#/paths/PUT/api/v1/query-ui/sample-queries) | array of [SampleQuery](/reference/rest-api/?av=v2#/operations/replace-sample-queries) | Customize sample queries listed in the exploration UI |
| `statusQuery` | [CypherQuery](/reference/rest-api/?av=v1#/paths/POST/api/v1/query/cypher) | [CypherQuery](/reference/rest-api/?av=v2#/operations/query-cypher) | OPTIONAL Cypher query that is executed and reported to the terminal window during execution |

!!! tip "Quick query `edgeLabel`"
    Single-hop quick queries that return a directly connected node should omit `edgeLabel` — the real edge already renders in the exploration UI. Only set a descriptive `edgeLabel` for 2+-hop traversals where the intermediate path isn't visible.

## Differences Between Recipes and the REST API

Recipes package together Quine config, graph logic/structure, and exploration UI customizations that are run automatically when Quine starts. A recipe file can define the graph and exploration UI configuration directly however, when a recipe is run, it infers a Quine configuration. There are a couple of operational differences that you need to keep in mind when launching Quine along with a recipe.

* **API calls allow naming** - When you create an ingest stream or a standing query using the API, you choose the name for the object in the URL. The corresponding recipe object uses standardized names in the form of `INGEST-#` and `STANDING-#`.

* **Persistent storage** - Starting the Quine application `jar` *without* providing a configuration file, Quine creates a RocksDB-based persistent data store in the local directory. That persistor retains the previous graph state and is appended to each time Quine starts. Alternatively, when Quine launches a recipe, a temporary persistent data store is created in the system `tmp` directory. Each subsequent launch of the recipe will replace the data store, discarding the graph from the previous run. You can override the default temporary storage behavior with your configuration settings by using the `--force-config` command line flag.

## Running a Recipe

Recipes are interpreted by `quine.jar` using command line arguments. For help on `quine.jar` command line arguments, use the following command:

```shell
❯ java -jar quine-2.1.1.jar -h
Quine universal program
Usage: quine [options]

  -W, --disable-web-service
                           disable Quine web service
  -p, --port <value>       web service port (default is 8080)
  -r, --recipe name, file, URL
                           follow the specified recipe
  -x, --recipe-value key=value
                           recipe parameter substitution
  --force-config           disable recipe configuration defaults
  --no-delete              disable deleting data file when process exits
  -h, --help
  -v, --version            print Quine program version
```

In order to launch Quine with a recipe, use `-r`, followed by either the short name of a sample recipe or a local YAML filename. Some recipes can expect input parameters. The parameter values are passed by using command line arguments with `-x` or `--recipe-value`.

## Recipe Parameters

A recipe may contain parameters that are used to pass information to the recipe. To use a parameter in a recipe file, a value in a recipe must start with the `$` character. The following example demonstrates a recipe with a parameter called `in-file`:

```yaml
--8<-- "recipes/assets/ingest.yaml"
```

Running the above example (without specifying the parameter value) causes an error:

```shell
❯ java -jar quine-2.1.1.jar -r ingest.yaml
Missing required parameter in-file; use --recipe-value in-file=
```

The error message indicates the command must be run with an additional command line argument that specifies the required parameter:

```shell
❯ java -jar quine-2.1.1.jar -r ingest.yaml --recipe-value in-file=my-file.txt
```

The parameter value is substituted into the recipe at runtime. Common examples for recipe parameters include file names, URLs, and host names.

## Additional Command Line Arguments

When Quine is started without command line arguments, its default behavior is to start the web service on port 8080. The following options are available to change this behavior:

* `-W, --disable-web-service`: Disables the web service
* `-p, --port`: Specify the TCP port of the web service

Quine is configurable as described in [Configuration](../reference/config/configuration.md). Normally when running a recipe, the configuration for `store` is overwritten with `PersistenceAgentType.RocksDb` and is configured to use a temporary file. This configuration is appropriate for most use cases of recipes and can be changed with the following command line arguments:

* `--force-config`: Quine will use the `store` that is configured via [Configuration](../reference/config/configuration.md) (instead of overwriting it as described above)
* `--no-delete`: Quine will not delete the DB file on exit and will print out that path to it

> **Note**: The `--force-config` and `--no-delete` options are mutually exclusive (only one of the two is allowed at a time).

RocksDB may not function on some platforms and you need to use MapDB instead by starting Quine using parameters as follows:

```shell
java -Dquine.store.type=map-db -jar quine-2.1.1.jar --force-config
```

## Recipe Analyzer

The [Quine Recipe Analyzer](https://www.thatdot.com/quine/recipe-analyzer.html) is a free browser-based tool that visualizes a recipe, or any set of ingest streams and standing queries, and makes recommendations for improving it. Paste in a recipe to see how its ingests, graph structure, and standing queries relate before running it.

## Recipe Repository

Complete recipes are useful as reference applications for use cases. [Quine's recipe repository](../recipes/index.md) highlights recipes shared by our community members. Additionally, you can view all of the shared recipes directory in the [Quine Github repository.](https://github.com/thatdot/quine/tree/main/quine/recipes/)

## Contribute

If you make a recipe and think others might find it useful, please consider contributing it back to the repository on Github so that others can use it. They could use it on their own data, or even just use it as a starting point for customization and remixing for other goals. To share with the community, [open a pull request on Github](https://github.com/thatdot/quine/pulls).


---
# Background Queries
URL: https://quine.io/learn/background-queries/background-queries/

# Background Queries

A **background query** runs a Cypher query out-of-band: Quine accepts the query, returns an execution id immediately, and runs the query independently of the request that started it. Nothing blocks on the query finishing, so a query that takes minutes or hours (for example an all-node scan, a bulk update, a large export) is no longer bounded by an HTTP or UI timeout.

Result rows are **streamed to destinations**, the same set of destinations that [standing query outputs](../standing-queries/standing-queries.md) use for a query run purely for its side effects. Rows are never stored by Quine itself. What *is* stored is a small **status record** per execution, whether it started, completed, failed, or was cancelled, how many rows it emitted, and what columns it returned. This is retained until an expiry and then swept.

While an execution runs, you can also watch its rows live over a [results tap WebSocket](#watching-results-live), cancel it, or poll its status.

To run a background query on a recurring schedule rather than once, see [Scheduled Jobs](scheduled-jobs.md).

## When to use a background query

Use a background query when the query's *results* are not what you are waiting for:

- **Long-running scans.** `MATCH (n) RETURN count(n)` over a large graph, or any query that must touch every node.
- **Bulk mutations.** Backfilling a property, relabeling nodes, or repairing data with `Drop` as the destination, since there is nothing to collect.
- **Exports.** Streaming a large result set straight into Kafka, S3, or a file without materializing it in a client.
- **Scheduled maintenance.** The same query, on a schedule. See [Scheduled Jobs](scheduled-jobs.md).

An ordinary interactive query is still the right tool when you want the rows back in the response.

!!! warning "Background does not mean cheap"
    A background query runs against the same graph as everything else. An all-node scan issued in the background still scans every node and still competes for the same resources. See [ID Provider](../../core-concepts/id-provider.md) for guidance on querying large graphs efficiently.

## Running a query in the background

`POST` the query and its destinations to the graph-scoped background queries endpoint:

```bash
curl -X POST "http://localhost:8080/api/v2/graph/quine/backgroundQueries" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "count-all-nodes",
    "query": "MATCH (n) RETURN count(n) AS total",
    "destinations": [{"type": "Drop"}]
  }'
```

The response is the execution id, returned as soon as the execution's status record exists, not when the query finishes:

```json
{"id": "6f1c0b3e-9c1a-4f5c-9b0e-1a2b3c4d5e6f"}
```

### Request fields

| Field | Required | Description |
|:------|:---------|:------------|
| `query` | Yes | The Cypher query to run. Compiled when the request is made, so a query that cannot compile is rejected with a `400` rather than failing later. |
| `destinations` | Yes | A non-empty list of destinations the result rows are streamed to. Use `[{"type": "Drop"}]` to discard rows. |
| `name` | No | A human-readable name, surfaced in the execution record and in the UI. |
| `parameters` | No | Cypher parameters, as a JSON object. Defaults to `{}`. |
| `statusExpiry` | No | How long the status record is retained *after the execution terminates*, as a duration string such as `"1h"` or `"168h"`. Defaults to one week. |

Because the endpoint is graph-scoped, the query runs in the graph named in the path.

!!! note "At-most-once"
    A run started this way is at-most-once: if the host executing it restarts or leaves the cluster mid-run, the run is lost and its record is reconciled to `interrupted`. Scheduled jobs, by contrast, re-fire an interrupted run. See [Scheduled Jobs](scheduled-jobs.md#delivery-guarantees).

## Execution status

Poll an execution by its id:

```bash
curl "http://localhost:8080/api/v2/graph/quine/backgroundQueries/6f1c0b3e-9c1a-4f5c-9b0e-1a2b3c4d5e6f"
```

```json
{
  "id": "6f1c0b3e-9c1a-4f5c-9b0e-1a2b3c4d5e6f",
  "jobName": null,
  "name": "count-all-nodes",
  "query": "MATCH (n) RETURN count(n) AS total",
  "status": "completed",
  "hostId": "0",
  "totalRowCount": 1,
  "columns": ["total"],
  "error": null,
  "expiresAt": "2026-09-08T14:02:11Z"
}
```

`status` is one of:

| Status | Meaning |
|:-------|:--------|
| `started` | The execution began and may still be running. |
| `completed` | The query finished. `totalRowCount` is the full number of rows emitted and `columns` the result column names. |
| `failed` | The query failed terminally. `error` carries the message. |
| `cancelled` | The execution was cancelled while in flight. Rows already streamed to destinations are not retracted. |
| `interrupted` | The executing host restarted or left the cluster mid-run, so the execution ended without recording an outcome. |

`jobName` is set when the execution was dispatched by a [scheduled job](scheduled-jobs.md); it is `null` for a run started directly.

### Listing executions

```bash
# Every unexpired execution in this graph
curl "http://localhost:8080/api/v2/graph/quine/backgroundQueries"

# Only executions dispatched by a particular job
curl "http://localhost:8080/api/v2/graph/quine/backgroundQueries?jobName=nightly-count"
```

Executions are scoped to the graph their query ran against, so this list never shows executions from another graph.

### Retention

`statusExpiry` is counted from the moment the execution *terminates*, not from when it started. A still-running execution therefore stays visible,  pollable, and cancellable for as long as it runs, no matter how short its expiry is. Once an execution terminates and its expiry passes, the record is hidden and swept.

Records are removed by expiry, or by an explicit `DELETE`. There is no other cleanup path: cancelling an execution is a state transition, not a deletion, and a cancelled record expires like any other.

Set `statusExpiry` longer than you expect the query to run *plus* however long you want to be able to read the outcome afterwards.

## Cancelling an execution

```bash
curl -X POST "http://localhost:8080/api/v2/graph/quine/backgroundQueries/{id}:cancel"
```

Cancellation aborts the query stream. The record transitions to `cancelled` as the executing host unwinds, so the record returned by the cancel call itself may still read `started`; poll for the transition. Cancelling an already-terminal execution is a no-op.

Destinations that were mid-write see the stream fail. Rows delivered before the cancel are not retracted.

## Deleting an execution record

```bash
curl -X DELETE "http://localhost:8080/api/v2/graph/quine/backgroundQueries/{id}"
```

Deleting drops the status record immediately instead of waiting for its expiry. If the execution is still running it is cancelled first, then its record is removed. Deleting a record that is already gone returns a `404`.

## Watching results live

!!! warning "Experimental Feature"
    The results tap is experimental and unstable. Its API and behavior may change without notice.

Each execution exposes a WebSocket that streams its result rows as they are produced:

```
ws://{host}/api/v2/graph/{graphName}/backgroundQueries/{id}:tap
```

Use `wss://` if your instance is served over TLS.

Each message is a JSON text frame. Row frames are plain objects keyed by result column name. After the run terminates, one final frame is sent under the key `__backgroundQueryComplete`:

```json
{
  "__backgroundQueryComplete": {
    "status": "completed",
    "totalRowCount": 3000,
    "droppedBufferedRows": 0,
    "error": null
  }
}
```

Because the completion frame carries the total row count, a consumer can report "showing N of M" even when it did not receive every row.

### Buffering and delivery

The executing host buffers the first **1024** rows until a subscriber attaches, and retains that buffer until **60 seconds** past termination. Connecting to the tap with the id the run endpoint just returned therefore still shows the head of the results, even for a query that finishes before your client connects. Rows beyond the buffer that are produced before anyone attaches are counted in `droppedBufferedRows`.

Once a subscriber attaches, the buffer flushes in order and subsequent rows stream live.

The tap is **best-effort and has no delivery guarantees**. It never applies backpressure to the query: a slow consumer drops frames rather than slowing the run down. Reconcile against `totalRowCount` in the status record or in the completion frame when the exact count matters.

A run cancelled before any subscriber ever attached discards its buffer entirely; cancelling disclaims the results.

### Example

Start a run that emits a burst of rows, then watch it:

```bash
ID=$(curl -s -X POST "http://localhost:8080/api/v2/graph/quine/backgroundQueries" \
  -H 'Content-Type: application/json' \
  -d '{"query": "UNWIND range(1, 3000) AS i RETURN i", "destinations": [{"type": "Drop"}]}' \
  | jq -r '.id')

websocat "ws://localhost:8080/api/v2/graph/quine/backgroundQueries/$ID:tap"
```

```json
{"i": 1}
{"i": 2}
{"i": 3}
...
{"__backgroundQueryComplete": {"status": "completed", "totalRowCount": 3000, "droppedBufferedRows": 0, "error": null}}
```

## Streaming results to a destination

To keep the results, give the query a real destination instead of `Drop`. Destinations use the same configuration as standing query outputs:

```bash
curl -X POST "http://localhost:8080/api/v2/graph/quine/backgroundQueries" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "export-accounts",
    "query": "MATCH (a:Account) RETURN a.id AS id, a.balance AS balance",
    "destinations": [
      {
        "type": "Kafka",
        "topic": "account-export",
        "bootstrapServers": "localhost:9092",
        "format": {"type": "JSON"}
      }
    ]
  }'
```

More than one destination can be given, and every row is delivered to all of them. See [Standing Queries](../standing-queries/standing-queries.md) for the full set of destination types and their options.

## Using the UI

Background queries can be started and watched from the browser without writing any HTTP calls:

- The **Streams** page has a Background Queries panel that lists the graph's runs, creates new ones, and inspects results as they arrive. See [Streams](../../getting-started/streams.md#background-queries-panel).
- The **Exploration UI** query menu has a **Run in background** action that dispatches the query bar's contents. See [Exploration UI](../../getting-started/exploration-ui.md#running-a-query-in-the-background).

## API reference

| Operation | Endpoint |
|:----------|:---------|
| [Run a background query](/reference/rest-api/?av=v2#/operations/run-background-query) | `POST /api/v2/graph/{graphName}/backgroundQueries` |
| [List background queries](/reference/rest-api/?av=v2#/operations/list-background-queries) | `GET /api/v2/graph/{graphName}/backgroundQueries` |
| [Get background query status](/reference/rest-api/?av=v2#/operations/get-background-query) | `GET /api/v2/graph/{graphName}/backgroundQueries/{id}` |
| [Cancel a background query](/reference/rest-api/?av=v2#/operations/cancel-background-query) | `POST /api/v2/graph/{graphName}/backgroundQueries/{id}:cancel` |
| [Delete a background query](/reference/rest-api/?av=v2#/operations/delete-background-query) | `DELETE /api/v2/graph/{graphName}/backgroundQueries/{id}` |
| [Background query results tap](/reference/rest-api/?av=v2#/operations/background-query-tap) | `GET /api/v2/graph/{graphName}/backgroundQueries/{id}:tap` (WebSocket) |

## Next steps

- [Scheduled Jobs](scheduled-jobs.md): run a background query on a recurring schedule.
- [Standing Queries](../standing-queries/standing-queries.md): destination types, formats, and result enrichment.
- [Standing Query Wiretap](../standing-queries/wiretap.md): the equivalent live view onto a standing query's output workflow.


---
# Scheduled Jobs
URL: https://quine.io/learn/background-queries/scheduled-jobs/

# Scheduled Jobs

A **job** is a named, recurring schedule paired with an **action** to perform each time it fires. Jobs are how you run the same work on a cadence such as  a nightly count, an hourly export, or a weekly cleanup without an external scheduler.

Jobs are **system-scoped**: one job list serves the whole Quine deployment, and each job is identified by a name you choose. The graph the work runs against is part of the *action*, not the job's URL.

Today the only action is a [background query](background-queries.md). Each time a job fires it dispatches one background-query execution, with its own execution id and its own status record.

## Jobs and executions

A job and a run of that job are different things, and they live at different endpoints:

| | Job | Execution |
|:--|:----|:----------|
| **Scope** | System-wide | One graph |
| **Identified by** | A name you choose | A server-generated UUID |
| **Endpoint** | `/api/v2/system/jobs` | `/api/v2/graph/{graphName}/backgroundQueries` |
| **Lifetime** | Until you delete it | Until its status record expires |

A job does not list its own runs. To find them, filter the executions of the graph the job targets by the job's name:

```bash
curl "http://localhost:8080/api/v2/graph/quine/backgroundQueries?jobName=nightly-count"
```

Each of those records is an ordinary background-query execution: pollable, tappable, and cancellable exactly as described in [Background Queries](background-queries.md).

## Creating a job

```bash
curl -X POST "http://localhost:8080/api/v2/system/jobs" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "nightly-count",
    "schedule": {
      "type": "Daily",
      "at": "02:30",
      "timezone": "America/New_York"
    },
    "action": {
      "type": "BackgroundQuery",
      "namespace": "quine",
      "query": "MATCH (n) RETURN count(n) AS total",
      "destinations": [{"type": "Drop"}]
    }
  }'
```

```json
{"name": "nightly-count"}
```

| Field | Required | Description |
|:------|:---------|:------------|
| `name` | Yes | Unique name identifying the job. Trimmed; must be non-empty and free of control characters. |
| `schedule` | Yes | When the job fires. See [Schedules](#schedules). |
| `action` | Yes | What the job does each time it fires. See [Actions](#actions). |
| `updateIfExists` | No | If a job with this name already exists: `true` replaces its definition in place, `false` (the default) rejects the request with a `400`. |

The schedule and the query are both validated when the job is created, so a malformed schedule or an uncompilable query is a `400` at creation rather than a job that fails on every fire.

### Replacing a job

Jobs are keyed by name, so creating a job whose name is already taken is rejected unless you pass `updateIfExists: true`, which replaces the definition in place. There is no separate update endpoint.

Replacing a job re-evaluates its schedule as if it were newly created, while preserving its run history. In particular, an `Interval` schedule that still omits `startAt` re-anchors to the replacement time and fires immediately again.

## Schedules

Schedules come in two families, distinguished by the `type` discriminator.

### Interval

`Interval` fires at a fixed cadence anchored to an instant: at `startAt`, then every `every` thereafter.

```json
{"type": "Interval", "every": "6h", "startAt": "2026-09-01T00:00:00Z"}
```

| Field | Required | Description |
|:------|:---------|:------------|
| `every` | Yes | The gap between fires, as a duration string such as `"1h30m"` or `"24h"`. **Minimum one hour.** |
| `startAt` | No | RFC-3339 anchor instant. If omitted, the anchor is the moment the job is created or replaced, so the job fires immediately and then every interval. |

Fires land on the grid `startAt + k × every`, so an interval schedule never drifts: a late or delayed fire does not push subsequent fires back. A future anchor waits; a past anchor collapses the missed slots and resumes on the next multiple.

`Interval` has no timezone. Its cadence is measured in absolute elapsed time, so it never skips or repeats a fire at a daylight saving transition. Instead it drifts against the local clock by the size of the transition. A 24-hour interval that fires at 09:00 local time fires at 10:00 local after a spring-forward.

### Wall-clock schedules

The other four schedules fire when the local clock in a named timezone reads the requested time.

=== "Hourly"

    ```json
    {"type": "Hourly", "minute": 15, "timezone": "UTC"}
    ```

    Fires once an hour at `minute` (0–59).

=== "Daily"

    ```json
    {"type": "Daily", "at": "09:30", "timezone": "America/New_York"}
    ```

    Fires once a day at `at`, given as `"HH:mm"` or `"HH:mm:ss"`.

=== "Weekly"

    ```json
    {"type": "Weekly", "dayOfWeek": "MONDAY", "at": "09:30", "timezone": "Europe/London"}
    ```

    Fires once a week on `dayOfWeek` (`MONDAY` … `SUNDAY`) at `at`.

=== "Monthly"

    ```json
    {"type": "Monthly", "dayOfMonth": 1, "at": "00:00", "timezone": "UTC"}
    ```

    Fires once a month on `dayOfMonth` (1–31) at `at`. Months that do not have that day are skipped: a `dayOfMonth` of `31` does not fire in February.

Every wall-clock schedule requires a `timezone`, so the time of day is never ambiguous.

## Time zones and daylight saving time

`timezone` accepts any zone id known to the tz database of the running JVM, matched **case-sensitively**:

- **Region ids** `America/New_York`, `Europe/London`, `Australia/Lord_Howe` observe that region's daylight saving rules.
- **Fixed-offset ids** `UTC`, `GMT`, `Z`, `-05:00`, `+05:30`, `Etc/GMT+5` never shift, and so never skip or repeat a fire.
- **Legacy aliases** such as `US/Eastern` and `EST5EDT` are accepted. Bare abbreviations such as `EST` or `PST` are **not**, and are rejected with a `400`.

Zone rules come from the tz database bundled with the running JVM, so a JVM or OS upgrade that revises a zone's rules also changes the future fire times of jobs already scheduled in it.

### Spring forward: a local time that does not exist

- `Daily`, `Weekly`, and `Monthly` **do not shift the fire, they skip it**. A `Daily` at `02:30` in `America/New_York` fires on 2026-03-07 and then on 2026-03-09; nothing runs on 2026-03-08. A `Weekly` at `SUNDAY 02:30` loses the entire week, and a `Monthly` whose `dayOfMonth` lands on the transition day loses the entire month.
- `Hourly` loses exactly one fire, leaving 23 that day.
- The skip covers the whole clock hour containing the transition, not only the minutes that literally do not exist. This is visible only in zones whose shift is not a whole hour: `Australia/Lord_Howe` moves 02:00 to 02:30, and a `Daily` anywhere in `02:00`–`02:59` is skipped that day including `02:45`, which does exist locally.

### Fall back: a local time that occurs twice

- `Daily`, `Weekly`, and `Monthly` fire **once**, at the first (pre-transition) occurrence. A `Daily` at `01:30` in `America/New_York` fires at `2026-11-01T05:30:00Z` and not again at `06:30Z`.
- `Hourly` fires at both occurrences, giving 25 fires that day.
- Firing once is a consequence of advancing past the repeated hour, so a job whose next fire is first computed from a moment *inside* that hour does land on the second occurrence. Creating the `01:30` daily job at `01:31` on the first pass fires it 59 minutes later, at `01:30` on the second pass.

### Consequences worth planning for

- **The absolute gap between wall-clock fires is not constant.** A `Daily` job fires 23 hours after its predecessor on a spring-forward day and 25 hours after it on a fall-back day. An action that assumes it covers exactly 24 hours of data will under- or over-cover on those two days.
- **`Interval` has the mirror-image behavior.** It holds its absolute spacing and drifts against the local clock. Use a wall-clock schedule to pin local time, `Interval` to pin elapsed time.
- **Missed fires collapse.** However many slots elapse while the scheduler is down, recovery produces a single catch-up fire and then resumes the schedule normally. Slots are not replayed. This is independent of DST, but compounds with it.
- **`nextFireAt` and `lastFireAt` are absolute UTC instants**, not local times. Convert them into the job's `timezone` before comparing against `at`.
- **A job that must never skip or double up belongs on `UTC`**, at the cost of its local firing time moving twice a year.

## Actions

An action is what the job does on each fire, discriminated by `type`. `BackgroundQuery` is currently the only one.

```json
{
  "type": "BackgroundQuery",
  "namespace": "quine",
  "query": "MATCH (n:Stale) DETACH DELETE n",
  "destinations": [{"type": "Drop"}],
  "name": "stale-cleanup",
  "parameters": {},
  "statusExpiry": "168h"
}
```

| Field | Required | Description |
|:------|:---------|:------------|
| `query` | Yes | The Cypher query to run on each fire. |
| `destinations` | Yes | A non-empty list of destinations the result rows are streamed to. Use `[{"type": "Drop"}]` for a pure side-effect run. |
| `namespace` | No | The graph the query runs in. Defaults to the default graph. |
| `name` | No | A human-readable name, surfaced in each execution's record. |
| `parameters` | No | Cypher parameters, as a JSON object. |
| `statusExpiry` | No | How long each execution's status record is retained after that execution terminates. Defaults to one week. |

Because jobs are system-scoped, the target graph is named here rather than in the URL. This is the one place a job's graph is recorded. The job status endpoints do not report it back.

Every field except `namespace` matches the [run-now request body](background-queries.md#request-fields) exactly.

## Job status

```bash
curl "http://localhost:8080/api/v2/system/jobs/nightly-count"
```

```json
{
  "name": "nightly-count",
  "jobType": "background-query",
  "schedule": {"type": "Daily", "at": "02:30", "timezone": "America/New_York"},
  "nextFireAt": "2026-09-02T06:30:00Z",
  "lastFireAt": "2026-09-01T06:30:00Z",
  "running": true
}
```

| Field | Description |
|:------|:------------|
| `jobType` | The kind of work this job dispatches. |
| `schedule` | The job's schedule, as submitted. |
| `nextFireAt` | The next scheduled fire, as an absolute UTC instant. Absent for a schedule that never fires again. |
| `lastFireAt` | The most recent fire, as an absolute UTC instant. |
| `running` | Whether a dispatched execution is currently in flight. |

!!! note "Job status does not report the action"
    A job's status reports its schedule but not its action. To change a job's query, re-create it with `updateIfExists: true`; to know what it runs, look at one of its executions, which carries the query text.

List every job with `GET /api/v2/system/jobs`.

## Deleting a job

```bash
curl -X DELETE "http://localhost:8080/api/v2/system/jobs/nightly-count"
```

Deleting a job removes it from the scheduler, cancels any of its executions that are still running, and erases its persisted state. Status records of past executions are left to expire on their own, and remain queryable by the job's name until they do. The response is the job's status as it was immediately before deletion.

## Delivery guarantees

Scheduled fires are **at-least-once**. A run that is interrupted (because the host executing it restarted) is re-fired on recovery. Design job queries to be idempotent, or to tolerate being run twice for the same slot.

This is deliberately different from a background query started directly, which is at-most-once: nothing re-runs it if its host dies. See [Delivery Guarantees](../../core-concepts/delivery-guarantees.md) for how this fits with the rest of Quine's guarantees.

Only one execution of a given job runs at a time. If a fire arrives while the previous one is still in flight, it does not start a second overlapping run.

## Using the UI

The **Streams** page has a Scheduled Jobs panel that lists jobs, creates them, expands a job to show its dispatched runs, and deletes them. See [Streams](../../getting-started/streams.md#scheduled-jobs-panel).

## API reference

| Operation | Endpoint |
|:----------|:---------|
| [Create a scheduled job](/reference/rest-api/?av=v2#/operations/create-job) | `POST /api/v2/system/jobs` |
| [List scheduled jobs](/reference/rest-api/?av=v2#/operations/list-jobs) | `GET /api/v2/system/jobs` |
| [Get a scheduled job's status](/reference/rest-api/?av=v2#/operations/get-job) | `GET /api/v2/system/jobs/{name}` |
| [Delete a scheduled job](/reference/rest-api/?av=v2#/operations/delete-job) | `DELETE /api/v2/system/jobs/{name}` |

## Next steps

- [Background Queries](background-queries.md): status, retention, cancellation, and watching results live.
- [Standing Queries](../standing-queries/standing-queries.md): destination types and formats.


---
# Cypher Language
URL: https://quine.io/learn/cypher/

# Cypher Language

!!! tip "Coming from Neo4j?"
    Quine uses a dialect of OpenCypher with important behavioral differences from Neo4j — including how nodes are addressed, how edges work, and what features are supported. See [Quine Cypher vs. Neo4j Cypher](quine-cypher-differences.md) for a complete guide with correct patterns and common mistakes to avoid.

Cypher is the most widely use query language for interacting with data in a property graph format. It is structurally and syntactically similar to SQL, with the main difference being the `MATCH` clause. The idea of `MATCH` is to focus on declaratively describing the graph shape (pattern) that you want and then to let the query compiler pick a good execution plan. What would normally require multiple `JOIN`s in a relational model often just reduces to one `MATCH` with a pattern that has multiple edges:

```cypher
MATCH (n: Person)-[:has-parent]->(p: Person)-[:lives-in]->(c: City)
RETURN p.name AS name, c.name AS parentsCity
```

Compare the above Cypher to the equivalent SQL below:

```sql
SELECT n.name AS name, c.name AS parentsCity
FROM persons AS n
JOIN persons AS p ON n.parent = p.id
JOIN cities AS c ON p.city = c.id
```

Cypher queries are used in Quine for several purposes:

  * Used to ingest events and create the graph
  * Set as standing queries to find live matches from the dynamically changing graph
  * Entered in the query bar of the [Exploration UI](../../getting-started/exploration-ui.md)
  * Set as "quick queries" in the [Exploration UI](../../getting-started/exploration-ui.md)
  * Sent directly through the [REST API](../../reference/rest-api.md)

## Language Specification
[The official OpenCypher language reference here](https://s3.amazonaws.com/artifacts.opencypher.org/openCypher9.pdf).

* Note that Quine supports a dialect of OpenCypher v9. Please check the documentation to learn about how Quine interprets Cypher in the context of a streaming graph.

## Unsupported Language Features
Almost all of the Cypher language is supported with a few notable exceptions:

* Nodes found in a Cypher `MATCH` statement do not include updates made in the same query following the match statement. Some particular cases where this tends to be more surprising are:
    * When updating a node's labels or properties with `SET n :MyLabel`, `SET n.foo = "bar"`, or `SET n += propertyMap`, the `n` variable representing **the node will not reflect the update**.
    * The same node can be aliased under two different names and those two aliases may be different if there have been intervening changes.
* The identity of an edge in Quine is determined entirely by the direction, label, and endpoints of the edge. This has several implications:
    * Edges do not have IDs. A query like `MATCH (n)-[e]->(m) RETURN id(e)` does not return a useful value. Looking up edges must be done from one of the nodes on either side.
    * There cannot be multiple edges with the same label and direction connecting the same two nodes. If a query such as `MATCH (n), (m) WHERE id(n) = idFrom(0) AND id(m) = idFrom(1) CREATE (n)-[:myEdge]->(m)` is run twice, the second `CREATE` will have no effect since the single possible edge already exists.
    * Properties on edges are not supported. Edges can always be equally represented as an `edge-node-edge` instead of just an `edge`. The node in the middle becomes the carrier for properties that are about the relationship. Quine does not suffer from graph traversal challenges that sometimes motivate users to limit query length, so this node amplification is not problematic in Quine.
* The runtime characteristics of setting a node label are exactly equivalent to using a property. Querying using labels does not enable more efficient scanning, it is just an additional filter condition. Even if there are only a handful of nodes with the `:Person` label on them, a query such as `MATCH (p:Person) RETURN DISTINCT p.name` will still need to scan all nodes to find those with the desired `:Person` label.
* `DETACH DELETE` does not work on a path.
* `shortestPath` cannot be used inside a `MATCH`/`MERGE` pattern — it can only be used as an expression (e.g., in `RETURN`, `WITH`, or `WHERE` clauses). The start and end nodes must be bound separately first:

    ```cypher
    MATCH (a), (b) WHERE id(a) = idFrom(1) AND id(b) = idFrom(2)
    RETURN shortestPath((a)-[*]->(b))
    ```

    The default maximum path length is 10 hops. Use the range syntax to override: `shortestPath((a)-[*..20]->(b))`.

    Note: `allShortestPaths` is not currently implemented.

* Variable-length patterns like `(a)-[*1..5]->(b)` can be used in `MATCH` clauses and return **all** matching paths within the length range. In contrast, `shortestPath()` returns only a single shortest path. Variable-length patterns cannot be used in `CREATE` statements or standing queries.
* Some aggregation functions are not yet implemented: `percentileCont`, `percentileDisc`, `stDev`, and `stDevP`. 
* `sum` and `avg` do not work on durations.
* Hints inside queries (an advanced Cypher feature) are not supported and are silently ignored.
* Cypher commands are not supported (used for system management in other systems).

## Query Execution Plans

Prefix any Cypher query with `EXPLAIN` to see the execution plan without running the query:

```cypher
EXPLAIN MATCH (n:Person)-[:KNOWS]->(m:Person)
WHERE n.age > 30
RETURN m.name
```

This returns a JSON tree structure showing how Quine will execute the query. The query is compiled but **not executed**, making it safe to use on production systems.

The plan includes:

* `operatorType`: The operation being performed (e.g., `Filter`, `Expand`, `AnchoredEntry`)
* `args`: Operator-specific arguments
* `identifiers`: Output columns available from this operator
* `children`: Nested sub-plans that execute first
* `isReadOnly`: Whether the query performs no writes
* `canContainAllNodeScan`: Whether the query may scan all nodes (performance warning)

For detailed information on interpreting execution plans, including a complete list of operators, see [Query Execution Plans](../../learn/troubleshooting/query-execution-plans.md).

## Railroad Diagrams

[Interactive railroad diagrams](https://s3.amazonaws.com/artifacts.opencypher.org/M16/railroad/Cypher.html) for the Cypher language syntax are a helpful in understanding query parts in Cypher.

Example:
![cypher railroad](./cypher-images/cypher-MultiPartQuery-railroad-diagram.png)


---
# Cypher Enhancements
URL: https://quine.io/learn/cypher/advanced-cypher/

# Cypher Enhancements

Quine extends OpenCypher with functionality unique to processing complex event streams. In addition to functions like `idFrom()` to calculate node IDs, the Quine Cypher interpreter includes functions and procedures used in data application development.

## Casting Property Types

OpenCypher defines three classes of data types; property types, composite types, and structural types. Quine supports storing `INTEGER`, `FLOAT`, `STRING`, and `BOOLEAN` property types as properties. As of release v1.5.2, Quine extends the specification to also include storing composite types `MAP` and `LIST OF ANY` as properties in nodes. Additionally, Quine provides functions to cast between these types in Cypher queries.  

Quine Cypher implements two families of functions to cast property values, `castOrThrow` and `castOrNull`. See the custom [Cypher functions](cypher-functions.md) page for the complete documentation.

The `castOrThrow` family of cast functions will cause the query to error if an argument of the wrong type is provided. This is the family you should use while writing pipelines to find errors sooner and when you know the structure of your data in advance.

The `castOrNull` family of cast functions will allow the query to continue processing by returning `null` if an argument of the wrong type is provided. This is useful when you don't know the structure of your data in advance, but your query must be more complex to handle possibly receiving a `null` as an indicator of a type mismatch.

For example, in Quine, a `MAP` property like `SET n.family = {dad: "Adam", mom: "Becca", brother: "Charlie"}` is allowed.

!!! note

    While Quine can store both property and composite types within the same node, it cannot store structural types like `NODE`, `RELATIONSHIP`, and `PATH` as properties, even as part of a composite.

Quine allows the use of composite property values, but the Cypher compiler may fail to detect their validity. Suppose we want to `UNWIND` the above family into individual rows. A natural query would be:

```cypher
MATCH (n) WHERE meta.type(n.family) = 'MAP'
WITH n.family AS family
UNWIND keys(family) AS relation
RETURN relation, family[relation] AS name
```

Running the above query will produce an error indicating `"Type mismatch: expected Map, Node or Relationship"` on `keys(family)`. This occurs when the Cypher compiler assumes `n.family` must be a standard property type and, therefore, can't be `MAP`, a composite type. For this query to run, we must _cast_ `n.family` as a `MAP` to pre-select the property type, so the compiler is not confused.

!!! note

    Cast functions are Cypher functions that have no effect at runtime but provide the compiler a hint about their argument type. Here are some examples of expressions using cast functions:

    - `castOrThrow.integer(1)` returns `1`, since `1` is a valid `INTEGER`
    - `castOrThrow.float(2)` fails the query, since the `INTEGER` `2` is not a `FLOAT`
    - `castOrNull.map(7)` returns `null`, since the `INTEGER` `7` is not a `MAP`
    - `castOrNull.float(3.0)` returns `3.0`, since `3.0` is a valid `FLOAT`

We know that `n.family` is a map (because `meta.type(n.family) = 'MAP'`); we use `castOrThrow.map` as a hint to the query compiler:

```cypher
MATCH (n) WHERE meta.type(n.family) = 'MAP'
WITH castOrThrow.map(n.family) AS family
UNWIND keys(family) AS relation
RETURN relation, family[relation] AS name
```

## Relationship Patterns as Predicates

A bare relationship pattern can be used directly as a boolean condition, without wrapping it in `exists(...)`. The pattern evaluates to `true` when at least one matching path exists. This works in every boolean position, including combined with `AND`, `OR`, and `NOT`, and inside `CASE WHEN`:

```cypher
MATCH (p { last: 'Weasley' })
WHERE NOT (p)-[:has_father]->({ last: 'Weasley' })
RETURN p.first
```

```cypher
MATCH (p { last: 'Weasley' })
RETURN p.first,
       CASE WHEN (p)-[:has_father]->({ last: 'Weasley' }) THEN true ELSE false END AS hasWeasleyFather
```

In list positions the same syntax keeps its list meaning, so expressions like `size((a)-->())` are unchanged.

!!! note

    This coercion applies to ad-hoc Cypher queries, including standing query outputs. Standing query **patterns** are a deliberate subset of Cypher and do not support pattern expressions in boolean positions.

## Atomic Property Updates

Stateful operations in a streaming system generally read data (like a property) or write to it. In most cases, this is all that is needed since more complicated operations can be broken down into individual reads and writes.

For example, `SET n.x = n.x + 1` can be broken down into a read operation (retrieving `n.x`) and a write operation (`SET`ing `n.x` to its new value). These stateful operations are _atomic_ -- that is, they lock the node and will either succeed (getting back or setting the value) or fail (getting back an error or a `null`).

Quine v1.5.2 introduced a family of atomic Cypher procedures for performing more complicated read-operate-write operations that lock the node for the duration of the transaction. These atomic procedures are:

- `int.add`, add an integer to a property, or set the property to the integer
- `float.add`, add a float to a property, or set the property to the float
- `set.insert`, insert an element to a list, treating that property like a set
- `set.union`, merge a set (represented as a list) into a list, treating the property as a set

Like read and write operations, a call to these procedures is _atomic_ and will lock the node against other operations while they execute. Use them to enforce the _consistency_ of properties affected by multiple parallel operations.

Suppose we want to count the number of nodes (spokes) connected to a central node (hub). We could register a standing query looking for the spoke-hub relationship and increment a counter on the hub node in response to the standing query matching a spoke. For example:

```cypher
MATCH (hub) WHERE id(hub) = $that["id(hub)"]
WITH coalesce(hub.numberOfMatches, 0) AS oldMatchCount
SET hub.numberOfMatches = oldMatchCount + 1
RETURN oldMatchCount + 1 AS result
```

This works as long as spokes are connected in strict sequence, always waiting for the `hub`'s `numberOfMatches` to be updated before connecting the next spoke. However, that approach requires a lot of waiting, which means latency for the overall data pipeline. The problem here is that if two spokes are added simultaneously, the `numberOfMatches` might only be increased by 1, depending on the order in which the two update streams execute the read and write events. (This is a classic example of a _race condition_).

!!! abstract "Hypothetical example of race condition"

    Two streams are both intending to increment `hub.numberOfMatches`

    - Stream 1 reads hub.numberOfMatches = 0
    - Stream 2 reads hub.numberOfMatches = 0
    - Stream 1 writes hub.numberOfMatches = 1
    - Stream 2 writes hub.numberOfMatches = 1

Alternatively, use `int.add` to increment the `numberOfMatches` property on `hub` node. This allows Quine to act on "increment" events instead of "read" and "write" events:

```cypher
MATCH (hub) WHERE id(hub) = $that["id(hub)"]
CALL int.add(hub, "numberOfMatches", 1) YIELD result
RETURN result
```

Or, more succinctly:

```cypher
CALL int.add($that["id(hub)"], "numberOfMatches", 1)
```

!!! abstract "Sequence of events with atomic update procedure"

    Two streams are both intending to increment `hub.numberOfMatches`

    - Stream 2 adds 1 to `hub.numberOfMatches`, yielding the value 1
    - Stream 1 adds 1 to `hub.numberOfMatches`, yielding the value 2

By performing the entire increment update atomically, there is no opportunity for a race condition to arise. Regardless of which stream performs an update first, completing both updates will leave the counter's final value at the correct value of `2`.

All of the functions introduced in v1.5.2 share this same principle of atomicity. For example, `set.union`, not only combines a set into another, it does so atomically. Thus, the query below adds `Paladin` and `Bard` (if they weren't already present) to the property `adventuringParty` on the node `dungeonId`:

```cypher
CALL set.union(dungeonId, "adventuringParty", ["Paladin", "Bard"])
```


---
# Quine Cypher Functions
URL: https://quine.io/learn/cypher/cypher-functions/

# Quine Cypher Functions

## Built-In Cypher Functions

These are functions that are part of the Cypher language.

### Predicate Functions

[Predicate](./functions/predicate.md) functions return either true or false for the given arguments.

| Function   | Syntax             | Description                                                                     |
| :--------- | :----------------- | :------------------------------------------------------------------------------ |
| ` IS NOT NULL` | `property IS NOT NULL` | Returns true if the specified property exists in the node, relationship or map. |

### Scalar Functions

Scalar functions return a single value.

| Function       | Syntax                                 | Description                                                                                                   |
| :------------- | :------------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| `coalesce()`   | `coalesce(expression [, expression]*)` | Returns the first non-null value in a list of expressions.                                                    |
| `endNode()`    | `endNode(relationship)`                | Returns the end node of a relationship.                                                                       |
| `head()`       | `head(list)`                           | Returns the first element in a list.                                                                          |
| `id()`         | `id(expression)`                       | Returns the id of a relationship or node.                                                                     |
| `last()`       | `last(expression)`                     | Returns the last element in a list.                                                                           |
| `length()`     | `length(path)`                         | Returns the length of a path.                                                                                 |
| `properties()` | `properties(expression)`               | Returns a map containing all the properties of a node or relationship.                                        |
| `size()`       | `size(list)`                           | Returns the number of items in a list.                                                                        |
| `size()`       | `size(pattern expression)`             | Applied to pattern expression, returns the number of sub-graphs matching the pattern expression.              |
| `size()`       | `size(string)`                         | Applied to string, returns the size of a string.                                                              |
| `startNode()`  | `startNode(relationship)`              | Returns the start node of a relationship.                                                                     |
| `timestamp()`  | `timestamp()`                          | Returns the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC. |
| `toBoolean()`  | `toBoolean(expression)`                | Converts a string value to a boolean value.                                                                   |
| `toFloat()`    | `toFloat(expression)`                  | Converts an integer or string value to a floating point number.                                               |
| `toInteger()`  | `toInteger(expression)`                | Converts a floating point or string value to an integer value.                                                |
| `type()`       | `type(relationship)`                   | Returns the string representation of the relationship type.                                                   |

### Aggregating Functions

These functions take multiple values as arguments, and calculate and return an aggregated value from them.

| Function           | Syntax                                   | Description                                                                                   |
| :----------------- | :--------------------------------------- | :-------------------------------------------------------------------------------------------- |
| `avg()`            | `avg(expression)`                        | Returns the average of a set of numeric values.                                               |
| `collect()`        | `collect(expression)`                    | Returns a list containing the values returned by an expression.                               |
| `count()`          | `count(expression)`                      | Returns the number of values or records.                                                      |
| `max()`            | `max(expression)`                        | Returns the maximum value in a set of values.                                                 |
| `min()`            | `min(expression)`                        | Returns the minimum value in a set of values.                                                 |
| `percentileCont()` | `percentileCont(expression, percentile)` | Returns the percentile of a value over a group using linear interpolation.                    |
| `percentileDisc()` | `percentileDisc(expression, percentile)` | Returns the nearest value to the given percentile over a group using a rounding method.       |
| `stDev()`          | `stDev(expression)`                      | Returns the standard deviation for the given value over a group for a sample of a population. |
| `stDevP()`         | `stDevP(expression)`                     | Returns the standard deviation for the given value over a group for an entire population.     |
| `sum()`            | `sum(expression)`                        | Returns the sum of a set of numeric values.                                                   |

### List Functions

These functions return lists of other values.

| Function          | Syntax                       | Description                                                                                                      |
| :---------------- | :--------------------------- | :--------------------------------------------------------------------------------------------------------------- |
| `keys()`          | `keys(expression)`           | Returns a list containing the string representations for all the property names of a node, relationship, or map. |
| `labels()`        | `labels(node)`               | Returns a list containing the string representations for all the labels of a node.                               |
| `nodes()`         | `nodes(path)`                | Returns a list containing all the nodes in a path.                                                               |
| `range()`         | `range(start, end [, step])` | Returns a list comprising all integer values within a specified range.                                           |
| `relationships()` | `relationships(path)`        | Returns a list containing all the relationships in a path.                                                       |
| `reverse()`       | `reverse(original)`          | Returns a list in which the order of all elements in the original list have been reversed.                       |
| `tail()`          | `tail(list)`                 | Returns all but the first element in a list.                                                                     |

### Mathematical Functions

These numeric functions all operate on numerical expressions only, and will return an error if used on any other values.

| Function  | Syntax                                  | Description                                                                                                               |
| :-------- | :-------------------------------------- | :------------------------------------------------------------------------------------------------------------------------ |
| `abs()`   | `abs(expression)`                       | Returns the absolute value of a number.                                                                                   |
| `ceil()`  | `ceil(expression)`                      | Returns the smallest floating point number that is greater than or equal to a number and equal to a mathematical integer. |
| `floor()` | `floor(expression)`                     | Returns the largest floating point number that is less than or equal to a number and equal to a mathematical integer.     |
| `rand()`  | `rand()`                                | Returns a random floating point number in the range from 0 (inclusive) to 1 (exclusive); i.e. [0,1).                      |
| `round()` | `round(expression [, precision, mode])` | Returns the floating point value of a number rounded to the nearest integer, or optionally as specified by `precision` and `mode`. `precision` should be an integer. [Rounding `mode`](https://docs.oracle.com/javase/8/docs/api/java/math/RoundingMode.html) should be one of: `UP`, `DOWN`, `CEILING`, `FLOOR`, `HALF_UP`, `HALF_DOWN`, `HALF_EVEN`, or `UNNECESSARY`. The default rounding mode is `HALF_UP`.                   |
| `sign()`  | `sign(expression)`                      | Returns the signum of a number: 0 if the number is 0, -1 for any negative number, and 1 for any positive number.          |

These logarithmic functions all operate on numerical expressions only, and will return an error if used on any other values.

| Function  | Syntax              | Description                                                                                               |
| :-------- | :------------------ | :-------------------------------------------------------------------------------------------------------- |
| `e()`     | `e()`               | Returns the base of the natural logarithm, e.                                                             |
| `exp()`   | `exp(expression)`   | Returns e^n, where e is the base of the natural logarithm, and n is the value of the argument expression. |
| `log()`   | `log(expression)`   | Returns the natural logarithm of a number.                                                                |
| `log10()` | `log10(expression)` | Returns the common logarithm (base 10) of a number.                                                       |
| `sqrt()`  | `sqrt(expression)`  | Returns the square root of a number.                                                                      |

All trigonometric functions operate on radians, unless otherwise specified.

| Function    | Syntax                            | Description                                                 |
| :---------- | :-------------------------------- | :---------------------------------------------------------- |
| `acos()`    | `acos(expression)`                | Returns the arccosine of a number in radians.               |
| `asin()`    | `asin(expression)`                | Returns the arcsine of a number in radians.                 |
| `atan()`    | `atan(expression)`                | Returns the arctangent of a number in radians.              |
| `atan2()`   | `atan2(expression1, expression2)` | Returns the arctangent2 of a set of coordinates in radians. |
| `cos()`     | `cos(expression)`                 | Returns the cosine of a number.                             |
| `cot()`     | `cot(expression)`                 | Returns the cotangent of a number.                          |
| `degrees()` | `degrees(expression)`             | Converts radians to degrees.                                |
| `pi()`      | `pi()`                            | Returns the mathematical constant pi.                       |
| `radians()` | `radians(expression)`             | Converts degrees to radians.                                |
| `sin()`     | `sin(expression)`                 | Returns the sine of a number.                               |
| `tan()`     | `tan(expression)`                 | Returns the tangent of a number.                            |

### String Functions

These functions are used to manipulate strings or to create a string representation of another value.

| Function      | Syntax                                  | Description                                                                                                                              |
| :------------ | :-------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- |
| `left()`      | `left(original, length)`                | Returns a string containing the specified number of leftmost characters of the original string.                                          |
| `lTrim()`     | `lTrim(original)`                       | Returns the original string with leading whitespace removed.                                                                             |
| `replace()`   | `replace(original, search, replace)`    | Returns a string in which all occurrences of a specified string in the original string have been replaced by another (specified) string. |
| `reverse()`   | `reverse(original)`                     | Returns a string in which the order of all characters in the original string have been reversed.                                         |
| `right()`     | `right(original, length)`               | Returns a string containing the specified number of rightmost characters of the original string.                                         |
| `rTrim()`     | `rTrim(original)`                       | Returns the original string with trailing whitespace removed.                                                                            |
| `split()`     | `split(original, splitDelimiter)`       | Returns a list of strings resulting from the splitting of the original string around matches of the given delimiter.                     |
| `substring()` | `substring(original, start [, length])` | Returns a substring of the original string, beginning with a 0-based index start and length.                                             |
| `toLower()`   | `toLower(original)`                     | Returns the original string in lowercase.                                                                                                |
| `toString()`  | `toString(expression)`                  | Converts an integer, float or boolean value to a string.                                                                                 |
| `toUpper()`   | `toUpper(original)`                     | Returns the original string in uppercase.                                                                                                |
| `trim()`      | `trim(original)`                        | Returns the original string with leading and trailing whitespace removed.                                                                |

## Custom Cypher Functions

These are the default additional functions that come with Quine:

--8<-- "generated/quine/cypher-user-defined-functions.md"


---
# Quine Cypher Procedures
URL: https://quine.io/learn/cypher/cypher-procedures/

# Quine Cypher Procedures

These are the default procedures that come with Quine.

--8<-- "generated/quine/cypher-user-defined-procedures.md"


---
# Purge Node
URL: https://quine.io/learn/cypher/purge-node/

`purgeNode` is a Cypher procedure available by default in Quine that "hard-deletes" a node, the data that node owns, and the history of data associated with the node from Quine. Data is deleted not only from the materialized graph, but also from the journals and snapshots in the persistor, as well as other auxiliary node managed data stores.

## Purpose

`purgeNode` may be preferred over soft-deletion (eg `DETACH DELETE`) when:

1. the node will not be necessary for future queries
2. reducing the volume of persistor-managed data is important (eg, given limited disk space)
3. historical queries are not in use (as `purgeNode` removes historical records)

## Application

As a cypher procedure, `purgeNode` can be invoked via typical `CALL` syntax:
`WITH n CALL purgeNode(n) RETURN "success"`, or, minimally, `CALL purgeNode(n)`. The argument `n` may be any of:

- a node ID (eg, from `id()` or `idFrom()`)
- a node variable (eg, from `MATCH (n) ...`)
- a stringified node ID (eg, from `strId()` or a string literal)

The Cypher query engine will delete the node data including snapshots, journals, and standing query bookkeeping, before the `CALL` completes.

Example for deleting a node by id:
`CALL purgeNode(idFrom('customer', 'abandoned-cart-1'))`

Example for (a slow, all-node scan) query deleting all nodes with a certain label
`MATCH (n:DeleteMe) CALL purgeNode(n)`


---
# Time Reification
URL: https://quine.io/learn/cypher/reify-time/

# Time Reification

`reify.time` is a Cypher procedure included with Quine. Its purpose is to facilitate the instantiation (reification) of
a graph of nodes representing time. `reify.time` is provided with a timestamp (current wall clock time as the default),
and a list of time periods. It adds time nodes representing that point in time at each period's level of precision.
It returns the node representing that point in time for the smallest among the given periods. `reify.time` does this by
determining which nodes must exist, and either reading them from the graph or creating them as necessary. Additionally,
`reify.time` relates the nodes it makes to each other and other nodes previously created by this function.

`reify.time` does not do anything Cypher can't do. In this sense, `reify.time` is unnecessary. So why does it need to exist?

* To reduce the boilerplate necessary to ingest time-series data usefully
* To create a modification point where in the future, changes can be made to how time-series data is modeled in the graph and have this change applied to all usages
* To organize data to be more useful in the Quine web UI
* To settle on a unified convention for representing time so that users don’t have to spend brainpower to create something bespoke (and inconsistent among different users)
* To create the persistent graph structure upon which time-series aggregate values can be stored or related

`reify.time` builds a hierarchy of related nodes for a single datetime value. Each node in this hierarchy represents a different period where the input datetime value belongs.

* Each node in the hierarchy is defined by a start datetime value and a period.
* Each node in the hierarchy is related to its parent node (except the largest).
* Each node in the hierarchy is also related to the next node in time (and the same period).

If you want to find a node created by `reify.time` in a later query, call it again with the datetime and the period of
the node you want.

## Supported Parameters

* ZonedDateTime (optional; defaults to now)
* Periods (optional list of strings; defaults to all periods)

Periods are:

* year
* month
* day
* hour
* minute
* second

## Return Values

`reify.time` returns the smallest period time node reified by this function. This function creates time nodes that do not exist and reuses time nodes that already exist.

## Examples

Call `reify.time` with default arguments, which will be to reify time nodes at all periods for the current system clock time:

```cypher
CALL reify.time() YIELD node AS secondNode RETURN secondNode
```

![Full Reified Time Graph](./cypher-images/reify-time-full.png)

Run with a time parsed from a string:

```cypher
CALL reify.time(datetime("2022-04-11T11:06:12Z"), ["month", "day"]) YIELD node AS dayNode RETURN dayNode
```

![Time Node Properties](./cypher-images/time-node-properties.png)

Reify Willard Van Orman Quine's birthday at the year and day level, then create a node representing Quine himself with
an edge to his birthday at the day level. Match the adjacent nodes created at all levels by reify.time to return
everything created by this query.

```cypher
CALL reify.time(datetime("1908-06-25T08:27:42-05:00"), ["year", "day"]) YIELD node AS dayNode
MATCH (q)
WHERE id(q) = idFrom("Quine")
SET q:Person, q.name = 'Willard Van Orman Quine'
CREATE (q)-[:BIRTHDAY]->(dayNode)
WITH dayNode AS birthday
MATCH
  (quine)-[:BIRTHDAY]->(birthday)<-[:DAY]-(birthyear {period: 'year'}),
  (previousYear)-[:NEXT]->(birthyear)-[:NEXT]->(nextYear),
  (previousDay)-[:NEXT]->(birthday)-[:NEXT]->(nextDay)
RETURN quine, previousYear, birthyear, nextYear, previousDay, birthday, nextDay
```

![Quine Birthday](./cypher-images/quine-birthday.png)

Use within a Recipe:

```yaml
--8<-- "recipes/assets/wikipedia.yaml"
```

The above Recipe consumes an event stream that describes new Wikipedia pages. Each event includes a timestamp, which is passed to `reify.time`.


---
# Temporal functions
URL: https://quine.io/learn/cypher/temporal-functions/

!!! Warning
    Quine relies on Java libraries when processing temporal functions. In most cases inconsistent or invalid parameters passed to Quine's temporal functions will result in errors, however there are some cases where Quine can return an unexpected result.

    For example, `dayOfQuarter = 93` returns `java.time.DateTimeException: Invalid value for DayOfQuarter (valid values 1 - 90/92): 93` but `dayOfQuarter=92` instead overflows and gives back April 1 (first day of the next quarter) if the quarter does not have 92 days.

!!! Note "Offsets vs Timezones"
    An offset refers to a number of minutes (Quine supports precision down to 15 minutes) ahead of or behind UTC. A timezone string like “US/Pacific” may be used to refer to an offset, but only the offset is stored, not the alias. Valid timezone strings are those recognized by the JVM as zone IDs, and may vary depending on which JVM is used to run Quine.

## DateTime

A DateTime represents an absolute moment at a specific offset from UTC. For example, May 21, 2002, at 2:30:00PM UTC-8, or July 30, 2039, at 12:00:00AM UTC. DateTimes with different offsets may refer to the same moment in time and are still distinct. For example, June 11, 4 PM UTC-8 is not the same DateTime as June 11, 7 PM UTC-5, even though they refer to the same moment in time (e.g., the same unix timestamp).

* `datetime()` - Returns the server’s current clock time in the server’s timezone.

??? example "RETURN datetime()"
    ```json
    {
        "columns": [
            "datetime()"
        ],
        "results": [
            [
                "2023-05-04T15:26:07.769696-05:00[America/Chicago]"
            ]
        ]
    }
    ```

* `datetime(datetime: string)` - Expects an ISO-8601 extended offset date-time string. For example, `2011-12-03T10:15:30+01:00`. The string may also include an optional timezone in square brackets, but the DateTime value created will track only the UTC offset, not the timezone name.

??? example "RETURN datetime("2011-12-03T10:15:30+01:00")"
    ```json
    {
        "columns": [
            "datetime(\"2011-12-03T10:15:30+01:00\")"
        ],
        "results": [
            [
                "2011-12-03T10:15:30+01:00"
            ]
        ]
    }
    ```

* `datetime(datetime: string, format: string)` - Expects a DateTime string parseable according to the format string. The format string is interpreted using Java’s syntax for datetime formats. Documentation available at [https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns)

??? example "`RETURN datetime("Wed, 1 May 2019 11:05:30 EST", "E, d MMM yyyy HH:mm:ss z")`"
    ```json
    {
        "columns": [
            "datetime(\"Wed, 1 May 2019 11:05:30 EST\", \"E, d MMM yyyy HH:mm:ss z\")"
        ],
        "results": [
            [
                "2019-05-01T11:05:30-04:00[America/New_York]"
            ]
        ]
    }
    ```

* `datetime(options: map)` - Options may be any combination of the following. If empty, the map is ignored, and the behavior is the same as `datetime()`
  * `date`: a `localdatetime` or datetime to use as a base date
    * When given a `localdatetime`: interpret the `localdatetime` as a datetime in the provided or default timezone (see below)
    * When given a datetime: interpret the datetime as a datetime in its original timezone, if no timezone is provided. If a timezone is provided, convert the provided datetime to the equivalent moment in the provided timezone
    * eg `datetime({date: datetime("2011-12-03T10:15:30+01:00"), timezone: "EST"}) = datetime("2011-12-03T04:15:30-05:00")`
    * When neither is provided, Jan 1 0000 is used
  * `timezone`: a string representing a timezone whose offset should be used (ex: PST or America/Los_Angeles). If the timezone can’t be parsed to an offset, the offset 0 (UTC) will be used. If no timezone is provided and no date is provided, the server timezone will be used. The zone provided must be a valid Java ZoneId (which may vary depending on the JVM)
  * `year`: integer representing the years since the base date
  * `quarter`: integer (1-4) representing the quarter within the year
  * `month`: integer (1-12) representing the month within the year
  * `week`: integer (1-53) representing the week within the (week-based) year
  * `dayOfQuarter`: integer (1-92) representing the day within the quarter
  * `day`: integer (1-31) representing the day within the month
  * `ordinalDay`: integer (1-366) representing the day within the year
  * `dayOfWeek`: integer (1-7) representing the day within the week
  * `hour`: integer (0-23) representing the hour within the day
  * `minute`: integer (0-59) representing the minute within the hour
  * `second`: integer (0-59) representing the second within the minute
  * `millisecond`: integer (0-999) representing the millisecond within the second
  * `microsecond`: integer (0-999,999) representing the microsecond within the second
  * `nanosecond`: integer (0-999,999,999) representing the nanosecond within the second
  * `epochMillis`: integer representing the number of milliseconds since the epoch
  * `epochSeconds`: integer representing the number of seconds since the epoch

??? example "WITH datetime({ year: 1984, month: 11, day: 11, hour: 12, minute: 31, second: 14, nanosecond: 645876123, timezone: "Europe/Stockholm" }) AS d RETURN d.year, d.quarter, d.month, d.week, d.weekYear, d.day, d.ordinalDay, d.dayOfWeek, d.dayOfQuarter"
    ```json
    {
        "columns": [
            "d.year",
            "d.quarter",
            "d.month",
            "d.week",
            "d.weekYear",
            "d.day",
            "d.ordinalDay",
            "d.dayOfWeek",
            "d.dayOfQuarter"
        ],
        "results": [
            [
                1984,
                4,
                11,
                45,
                null,
                11,
                316,
                7,
                42
            ]
        ]
    }
    ```

## LocalDateTime

A LocalDateTime represents a date and time, but no specific offset from UTC. For example, August 2nd, 2081 at 3:00:00PM. Note that as they lack a UTC offset, LocalDateTimes do not represent a specific moment, so questions like “how many seconds was August 2nd, 2018 at 3PM after the epoch?” are ill-defined.

* `localdatetime()` - Returns the server’s current clock time in the server’s timezone

??? example "RETURN localdatetime()"
    ```json
    {
        "columns": [
            "localdatetime()"
        ],
        "results": [
            [
                "2023-05-05T09:21:30.872348"
            ]
        ]
    }
    ```

* `localdatetime(datetime: string)` - Expects an ISO-8601 date-time string. For example, `2021-01-03T23:11:04`.

??? example "RETURN localdatetime("2021-01-03T23:11:04")"
    ```json
    {
        "columns": [
            "localdatetime(\"2021-01-03T23:11:04\")"
        ],
        "results": [
            [
                "2021-01-03T23:11:04"
            ]
        ]
    }
    ```

* `localdatetime(datetime: string, format: string)` - Expects a datetime string parseable according to the format string. The format string is interpreted using java’s syntax for datetime formats. Documentation available at [https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns)

??? example "`RETURN localdatetime("Wed, 1 May 2019 11:05:30 EST", "E, d MMM yyyy HH:mm:ss z")`"
    ```json
    {
      "columns": [
        "localdatetime(\"Wed, 1 May 2019 11:05:30 EST\", \"E, d MMM yyyy HH:mm:ss z\")"
      ],
      "results": [
        [
          "2019-05-01T11:05:30"
        ]
      ]
    }
    ```

* `localdatetime(options: map)`. Options may be any combination of the following. If empty, the behavior is the same as localdatetime(). Alternatively, a `timezone` may be provided as the only field, in which case the server’s clock time in the provided timezone will be returned (or UTC if the timezone could not be parsed).
  * `date`: a localdatetime or datetime to use as a base datetime
    * When given a datetime or localdatetime: interpret the datetime as a localdatetime by discarding any timezone component
    * When neither is provided, Jan 1 0000 is used
  * `year`: integer representing the years since the base date
  * `quarter`: integer (1-4) representing the quarter within the year
  * `month`: integer (1-12) representing the month within the year
  * `week`: integer (1-53) representing the week within the (week-based) year
  * `dayOfQuarter`: integer (1-92) representing the day within the quarter
  * `day`: integer (1-31) representing the day within the month
  * `ordinalDay`: integer (1-366) representing the day within the year
  * `dayOfWeek`: integer (1-7) representing the day within the week
  * `hour`: integer (0-23) representing the hour within the day
  * `minute`: integer (0-59) representing the minute within the hour
  * `second`: integer (0-59) representing the second within the minute
  * `millisecond`: integer (0-999) representing the millisecond within the second
  * `microsecond`: integer (0-999,999) representing the microsecond within the second
  * `nanosecond`: integer (0-999,999,999) representing the nanosecond within the second
  * `epochMillis`: integer representing the number of milliseconds since the epoch
  * `epochSeconds`: integer representing the number of seconds since the epoch

??? example "RETURN localdatetime({ year: 1995, month: 4, day: 25, hour: 5, minute: 1, second: 53 })"
    ```json
    {
      "columns": [
        "localdatetime({ year: 1995, month: 4, day: 25, hour: 5, minute: 1, second: 53 })"
      ],
      "results": [
        [
          "1995-04-25T05:01:53"
        ]
      ]
    }
    ```

## Date

Date represents an unzoned date. Examples of dates are February 2, 2009; November 28, 1941. Dates do not contain any offset information.

* `date()` - Returns the server’s current date

??? example "RETURN date()"
    ```json
    {
      "columns": [
        "date()"
      ],
      "results": [
        [
          "2023-05-05T09:36:25.941144"
        ]
      ]
    }
    ```

* `date(date: string)` - Expects an ISO-8601 date string. For example, '1986-06-07'

??? example "RETURN date("1986-06-07")"
    ```json
    {
      "columns": [
        "date(\"1986-06-07\")"
      ],
      "results": [
        [
          "1986-06-07"
        ]
      ]
    }
    ```

* `date(date: string, format: string)` - Expects a date string parseable according to the format string. The format string is interpreted using java’s syntax for datetime formats. Documentation available at [https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#patterns)

??? example "RETURN date("Wed, Jul 4, '01", "EEE, MMM d, ''yy")"
    ```json
    {
      "columns": [
        "date(\"Wed, Jul 4, '01\", \"EEE, MMM d, ''yy\")"
      ],
      "results": [
        [
          "2001-07-04"
        ]
      ]
    }
    ```

* `date(options: map)` - Options may be any combination of the following. All options modify a base date of Jan 1 0000
  * `year`: integer representing the years since the base date
  * `quarter`: integer (1-4) representing the quarter within the year
  * `month`: integer (1-12) representing the month within the year
  * `week`: integer (1-53) representing the week within the (week-based) year
  * `dayOfQuarter`: integer (1-92) representing the day within the quarter
  * `day`: integer (1-31) representing the day within the month
  * `ordinalDay`: integer (1-366) representing the day within the year
  * `dayOfWeek`: integer (1-7) representing the day within the week

??? example "RETURN date({ year: 1995, month: 4, day: 24 })"
    ```json
    {
      "columns": [
        "date({ year: 1995, month: 4, day: 24 })"
      ],
      "results": [
        [
          "1995-04-24"
        ]
      ]
    }
    ```

## Time

Time represents a time in at a specific offset from UTC. Examples of times are “1:27 AM UTC+1”, “4:05 PM UTC”, “11:22 PM UTC+9”. As with DateTime, multiple times may represent the same time within a day, but if they have different offsets, the time values are still considered distinct.

* `time()` - returns the server’s current clock time in the server’s timezone

??? example "RETURN time()"
    ```json
    {
      "columns": [
        "time()"
      ],
      "results": [
        [
          "09:47:05.660853"
        ]
      ]
    }
    ```

* `time(time: string)` - Expects an ISO-8601 extended time-string. For example, `19:45:03`. The offset portion will be truncated to a 15-minute interval.

??? example "RETURN time("19:45:03")"
    ```json
    {
      "columns": [
        "time(\"19:45:03\")"
      ],
      "results": [
        [
          "19:45:03"
        ]
      ]
    }
    ```

* `time(time: string, format: string)` -  Expects a time string parseable according to the format string. The format string is interpreted using java’s syntax for time formats. Documentation available at [https://docs.oracle.com/javase/8/docs/api/java/time/package-summary.html](https://docs.oracle.com/javase/8/docs/api/java/time/package-summary.html)

??? example "RETURN time("Apr 1, 11 oclock in '19", "MMM d, HH 'oclock in '''yy")"
    ```json
    {
      "columns": [
        "time(\"Apr 1, 11 oclock in '19\", \"MMM d, HH 'oclock in '''yy\")"
      ],
      "results": [
        [
          "11:00"
        ]
      ]
    }
    ```

* `time(options: map)` Options may be any combination of:
  * `hour`: integer (0-23) representing the hour within the day
  * `minute`: integer (0-59) representing the minute within the hour
  * `second`: integer (0-59) representing the second within the minute
  * `millisecond`: integer (0-999) representing the millisecond within the second
  * `microsecond`: integer (0-999,999) representing the microsecond within the second
  * `nanosecond`: integer (0-999,999,999) representing the nanosecond within the second
  * `offsetSeconds`: integer representing the number of seconds (-64800 to 64800 in 15-minute/900-second increments) offset from UTC

??? example "RETURN time({ hour: 10, minute: 4, second: 24, nanosecond: 110 })"
    ```json
    {
      "columns": [
        "time({ hour: 10, minute: 4, second: 24, nanosecond: 110 })"
      ],
      "results": [
        [
          "10:04:24.000000110"
        ]
      ]
    }
    ```

## LocalTime

LocalTime represents a time at no specific offset from UTC. Examples include “3:00 PM”, “6:40 AM”, and “5:05 PM”. LocalTimes contain no offset information

* `localtime()` - returns the server’s current clock time

* `localtime(time: string)` - Expects an ISO-8601 extended time-string. For example, `12:45:03`.

* `localtime(time: string, format: string)` -  Expects a time string parseable according to the format string. The format string is interpreted using java’s syntax for time formats. Documentation available at [https://docs.oracle.com/javase/8/docs/api/java/time/package-summary.html](https://docs.oracle.com/javase/8/docs/api/java/time/package-summary.html)

* `localtime(options: map)` Options may be any combination of:
  * `hour`: integer (0-23) representing the hour within the day
  * `minute`: integer (0-59) representing the minute within the hour
  * `second`: integer (0-59) representing the second within the minute
  * `millisecond`: integer (0-999) representing the millisecond within the second
  * `microsecond`: integer (0-999,999) representing the microsecond within the second
  * `nanosecond`: integer (0-999,999,999) representing the nanosecond within the second

## Duration

Duration represents a difference between 2 temporal values of the same type (either `datetime` or `localdatetime`) in the [ISO-8601 duration format](https://en.wikipedia.org/wiki/ISO_8601#Durations). For example, the duration between May 3 2020 and May 8 2020 is 120 hours. The duration between February 28 2000 and March 1 2000 is 48 hours, while the duration between February 28 2001 and March 1 2001 is 24 hours. The duration between 1:30 PM UTC-1 (from a datetime) and 2:00 PM (at no offset, from a localdatetime) is undefined. As a final example, the duration between May 29 1986, 2PM PST and May 30, 1986 12AM UTC is 3 hours.

??? example "RETURN duration({ days: 24 })"
    ```json
    {
      "columns": [
        "duration({ days: 24 })"
      ],
      "results": [
        [
          "PT576H"
        ]
      ]
    }
    ```

??? example "RETURN duration.between(localdatetime({day: 3, month: 5, year:2020}), localdatetime({day: 8, month: 5, year:2020}))"
    ```json
    {
      "columns": [
        "duration.between(localdatetime({day: 3, month: 5, year:2020}), localdatetime({day: 8, month: 5, year:2020}))"
      ],
      "results": [
        [
          "PT120H"
        ]
      ]
    }
    ```

??? example "RETURN duration.between(datetime({day: 28, month: 2, year:2000}), datetime({day: 1, month: 3, year:2000}))"
    ```json
    {
      "columns": [
        "duration.between(datetime({day: 28, month: 2, year:2000}), datetime({day: 1, month: 3, year:2000}))"
      ],
      "results": [
        [
          "PT48H"
        ]
      ]
    }
    ```

??? example "RETURN duration.between(datetime({day: 28, month: 2, year:2001}), datetime({day: 1, month: 3, year:2001}))"
    ```json
    {
      "columns": [
        "duration.between(datetime({day: 28, month: 2, year:2001}), datetime({day: 1, month: 3, year:2001}))"
      ],
      "results": [
        [
          "PT24H"
        ]
      ]
    }
    ```

??? example "RETURN duration.between(datetime({day: 29, month: 5, hour: 14, year: 1986, timezone: "PST"}), datetime({day: 30, month: 5, hour: 0, year: 1986, timezone: "UTC"}))"
    ```json
    {
      "columns": [
        "duration.between(datetime({day: 29, month: 5, hour: 14, year: 1986, timezone: \"PST\"}), datetime({day: 30, month: 5, hour: 0, year: 1986, timezone: \"UTC\"}))"
      ],
      "results": [
        [
          "PT3H"
        ]
      ]
    }
    ```

Durations may also be directly constructed as the sum of individual components.
The valid components of a duration are as follows:

* `years`
* `quarters`
* `months`
* `weeks`
* `days`
* `hours`
* `minutes`
* `seconds`
* `milliseconds`
* `microseconds`
* `nanoseconds`

Because temporal units at or above the scope of `days` are not fixed in length, an estimation of the duration in terms of smaller units is provided. For example, a duration of 1 day is estimated as 86400 seconds, even though some days have leap seconds. When an estimation is made, a message will be logged to warn the user. When directly constructing durations directly, each component must be an integer.

??? example "RETURN duration({ days: 1 })"
    ```json
    {
      "columns": [
        "duration({ days: 1 })"
      ],
      "results": [
        [
          "PT24H"
        ]
      ]
    }
    ```

??? example "RETURN duration({ hours: 25, minutes: 7, seconds: 20 })"
    ```json
    {
      "columns": [
        "duration({ hours: 25, minutes: 7, seconds: 20, milliseconds: 82 })"
      ],
      "results": [
        [
          "PT25H7M20.082S"
        ]
      ]
    }
    ```

??? example "RETURN duration({ years: 1 })"
    ```json
    {
      "columns": [
        "duration({ years: 1 })"
      ],
      "results": [
        [
          "PT8765H49M12S"
        ]
      ]
    }
    ```

Typically, durations are not used directly, but instead handled as a quantity of a familiar unit. In Cypher, this is represented using property-like syntax: Given a duration value named `d`, `d.years`, `d.quarters`, `d.months`, `d.weeks`, `d.days`, `d.hours`, `d.minutes`, `d.seconds`, `d.milliseconds`, `d.microseconds`, and `d.nanoseconds` are all valid expressions. When computing the length of a duration in terms of a unit, the result is rounded down to the nearest integer.

??? example "RETURN duration({ seconds: 3, milliseconds: 500, microseconds: 1700 }).milliseconds"
    ```json
    {
      "columns": [
        "duration({ seconds: 3, milliseconds: 500, microseconds: 700 }).milliseconds"
      ],
      "results": [
        [
          3501
        ]
      ]
    }
    ```

??? example "RETURN duration.between(datetime({day: 28, month: 2, year:2001}), datetime({day: 1, month: 3, year:2001})).days"
    ```json
    {
      "columns": [
        "duration.between(datetime({day: 28, month: 2, year:2001}), datetime({day: 1, month: 3, year:2001})).days"
      ],
      "results": [
        [
          1
        ]
      ]
    }
    ```


---
# Predicate Functions
URL: https://quine.io/learn/cypher/functions/predicate/

# Predicate Functions

## Introduction

Predicates are boolean functions that return true or false for a given set of non-null input. They are most commonly used to filter out paths in the WHERE part of a query.

The following graph is used in the example below:

![Example Graph](exampleGraph.png){ width=200px }

Run this Cypher in Quine to create the sample graph.

``` cypher
CREATE
  (alice:Person:Developer {name:'Alice', age: 38, eyes: 'brown'}),
  (bob {name: 'Bob', surname: 'Smith', age: 25, eyes: 'blue'}),
  (charlie {name: 'Charlie', surname: 'Brown', age: 53, eyes: 'green'}),
  (daniel {name: 'Daniel', age: 54, eyes: 'brown'}),
  (eskil {name: 'Eskil', age: 41, eyes: 'blue', array: ['one', 'two', 'three']}),
  (alice)-[:KNOWS]->(bob),
  (alice)-[:KNOWS]->(charlie),
  (bob)-[:KNOWS]->(daniel),
  (charlie)-[:KNOWS]->(daniel),
  (bob)-[:MARRIED]->(eskil)
```

---

## `property IS NOT NULL`

`property IS NOT NULL` returns true if the specified property exists in the node, relationship or map.

**Syntax**: `property IS NOT NULL`

**Returns**: A boolean

**Arguments**:

| Name       | Description                               |
| :--------- | :---------------------------------------- |
| `property` | A property (in the form 'variable.prop'). |

**Sample**:

``` cypher
MATCH (n)
WHERE n.surname IS NOT NULL
RETURN n.name AS name, n.surname AS surname
```

!!! example "Note"
    Be sure to submit the query as a text query using ++shift+enter++.

The names and surnames of all nodes with a surname property are returned.

``` json
{
  "columns": [
    "name",
    "surname"
  ],
  "results": [
    [
      "Charlie",
      "Brown"
    ],
    [
      "Bob",
      "Smith"
    ]
  ]
}
```


---
# Scalar Functions
URL: https://quine.io/learn/cypher/functions/scalar/

# Scalar Functions

## Introduction

Scalar functions return a single value.

The following graph is used in the examples below:

![Example Graph](exampleGraph.png){ width=200px }

Run this Cypher in Quine to create the sample graph.

``` cypher
CREATE
  (alice:Person:Developer {name:'Alice', age: 38, eyes: 'brown'}),
  (bob {name: 'Bob', surname: 'Smith', age: 25, eyes: 'blue'}),
  (charlie {name: 'Charlie', surname: 'Brown', age: 53, eyes: 'green'}),
  (daniel {name: 'Daniel', age: 54, eyes: 'brown'}),
  (eskil {name: 'Eskil', age: 41, eyes: 'blue', array: ['one', 'two', 'three']}),
  (alice)-[:KNOWS]->(bob),
  (alice)-[:KNOWS]->(charlie),
  (bob)-[:KNOWS]->(daniel),
  (charlie)-[:KNOWS]->(daniel),
  (bob)-[:MARRIED]->(eskil)
```

!!! example "Note"
    Be sure to submit the example queries below as a text query using ++shift+enter++.

---

## `coalesce()`  

`coalesce()` returns the first non-null value in the given list of expressions. 

**Syntax**: `coalesce(expression [, expression]*)`

**Returns**: The type of the value returned will be that of the first non-null expression. 

**Arguments**:

| Name       | Description                          |
| :--------- | :----------------------------------- |
| expression | An expression which may return null. |

**Considerations**: `null`` will be returned if all the arguments are null. 

**Sample**:

``` cypher
MATCH (a)
WHERE a.name = 'Alice'
RETURN coalesce(a.hairColor, a.eyes)
```

```json
{
  "columns": [
    "coalesce(a.hairColor, a.eyes)"
  ],
  "results": [
    [
      "brown"
    ]
  ]
}
```

---

## `endNode()`

`endNode()` returns the end node of a relationship. 

**Syntax**: `endNode(relationship)`

**Returns**: A Node.

**Arguments**:

| Name         | Description                                |
| :----------- | :----------------------------------------- |
| relationship | An expression that returns a relationship. |

**Considerations**: `endNode(null)` returns `null`. 

**Example**:

``` cypher
MATCH (x:Developer)-[r]-()
RETURN endNode(r)
```

``` json
{
  "columns": [
    "endNode(r)"
  ],
  "results": [
    [
      {
        "id": "00c43b90-63d2-4d43-a030-6621facdda18",
        "labels": [],
        "properties": {
          "age": 53,
          "eyes": "green",
          "name": "Charlie",
          "surname": "Brown"
        }
      }
    ],
    [
      {
        "id": "8d76cbd3-e386-4cbe-80ac-3f488fac9fc9",
        "labels": [],
        "properties": {
          "age": 25,
          "eyes": "blue",
          "name": "Bob",
          "surname": "Smith"
        }
      }
    ]
  ]
}
```

---

## `head()`

`head()` returns the first element in a list.

**Syntax:** `head(list)`

**Returns:** The type of the value returned will be that of the first element of list.

**Arguments:**

| Name | Description                        |
| :--- | :--------------------------------- |
| list | An expression that returns a list. |

**Considerations:** `head(null)` returns null. If the first element in list is null, head(list) will return null.

**Example:**

``` cypher
MATCH (a)
WHERE a.name = 'Eskil'
RETURN a.array, head(a.array)
```

The first element in the list is returned.

```json
{
  "columns": [
    "a.array",
    "head(a.array)"
  ],
  "results": [
    [
      [
        "one",
        "two",
        "three"
      ],
      "one"
    ]
  ]
}
```

---

## `id()`

`id()` returns the id of a relationship or node.

**Syntax:** `id(expression)`

**Returns:** A Quine ID.

**Arguments:**

| Name       | Description                                          |
| :--------- | :--------------------------------------------------- |
| expression | An expression that returns a node or a relationship. |

**Considerations:** `id(null)` returns `null`.

**Example:**

``` cypher
MATCH (a)
RETURN id(a)
```

The node id for each of the nodes is returned.

``` json
{
  "columns": [
    "id(a)"
  ],
  "results": [
    [
      "f21e0f55-9c40-44ae-8904-18ef5d74231d"
    ]
  ]
}
```

---

## `last()`

`last()` returns the last element in a list.

**Syntax:** `last(expression)`

**Returns:** The type of the value returned will be that of the last element of list.

**Arguments:**

| Name | Description                        |
| :--- | :--------------------------------- |
| list | An expression that returns a list. |

**Considerations:** `last(null)` returns `null`. If the last element in list is null, last(list) will return null.

**Example:**

``` cypher
MATCH (a)
WHERE a.name = 'Eskil'
RETURN a.array, last(a.array)
```

The last element in the list is returned.

``` json
{
  "columns": [
    "a.array",
    "last(a.array)"
  ],
  "results": [
    [
      [
        "one",
        "two",
        "three"
      ],
      "three"
    ]
  ]
}
```

---

## `length()`

---

## `properties()`

---

## List `size()`

---

## Pattern `size()`

---

## String `size()`

---

## `startNode()`

---

## `timestamp()`

---

## `toBoolean()`

---

## `toFloat()`

---

## `toInteger()`

---

## `type()`


---
# Graph Algorithms
URL: https://quine.io/learn/graph-algorithms/

# Graph Algorithms

This section contains the reference documentation for the graph algorithms included in Quine.

| Algorithm                     | Description                                                                                                                                  |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| [Random Walk](random-walk.md) | Random walks allow us to translate the possibly-infinite dimensions of graph data into a linear string we can feed to graph neural networks. |


---
# Random Walk
URL: https://quine.io/learn/graph-algorithms/random-walk/

# Random Walk

Random walks are often the central connection between graph-structured data and machine learning applications. A random walk starts at a graph node and follows one of its edges randomly to reach another node, then follows one of that node's edges randomly to reach another node, and so on. Random walks allow us to translate the possibly-infinite dimensions of graph data into a linear string we can feed to graph neural networks.

## Tuning the Walk Parameters

The random walk algorithms in Quine allow a user to tune the random walks as described in the [Node2Vec paper](https://arxiv.org/abs/1607.00653). The `return` parameter (sometimes called `p`) determines how likely a walk is to return one step back where it came from (to "backtrack" to the previous node). The `inOut` parameter (sometimes called `q`) determines whether a walk is more likely to explore the local region ("neighborhood") around a node or travel far afield to explore the graph far away. These parameters can tune the walks to learn different features of the graph and address different goals.

See the API documentation for a complete description of API parameters, types, allowed values, and description: [Save Random Walks: `POST /api/v2/graph/quine/algorithms/randomWalk:saveWalks`](/reference/rest-api/?av=v2#/operations/save-random-walks#Query-Parameters)

## Collecting Values While Walking

The standard approach to random walks returns only the ID of each node visited in the graph. This capability can be extended substantially by Quine's ability to run an arbitrary Cypher query at each point in the random walk. This capability supports more advanced algorithms like Graph Convolutional Networks or [GraphSAGE](https://arxiv.org/abs/1706.02216)

Quine's random walk algorithms include the ability to define an aggregation `query` for each node encountered in a random walk. This can be used to explore the local neighborhood and/or aggregate multiple properties which get automatically collected in to random walk output. This can be used instead of, or in addition to, collecting node IDs. 

The value of the `query` parameter should be a Cypher query fragment which returns the desired data. The simplest example of this is a simple RETURN statement. A RETURN statement can return any number of values, separated by `,`s. If returning the same value multiple times, you will need to alias subsequent values with `AS` so that column names are unique. If a list is returned, its content will be flattened out one level and concatenated with the rest of the aggregated values.

The provided query will have the following prefix prepended: `MATCH (thisNode) WHERE id(thisNode) = $n ` where `$n` evaluates to the ID of the node on which the query is executed. The default value of the `query` parameter is: `RETURN id(thisNode)`

## Point in Time Walks

Most use cases for Quine include continuously running data ingests, which continue to modify the graph. To correctly generate a set of random walks, you need a view of the graph at a specific moment — without the graph changing from under the random walker.

Use Quine's built-in historical query functionality by including an RFC 3339 timestamp (e.g., `2026-04-27T15:30:00Z`) in the `atTime` parameter in your query request to generate random walks from the graph at any fixed historical moment.

The rest of the graph can keep changing, and the walk algorithm will see a consistent view of the graph.

## Node-Anchored Walks

Generating a random walk from a specific node in Quine can be done either by calling a function in a Cypher query:

```cypher
MATCH (n)
WHERE n = {some_constraint}
  CALL random.walk(n, 10)
  YIELD walk
RETURN walk
```

Or through the [Generate Random Walk: `POST /api/v2/graph/quine/algorithms/randomWalk/nodes/{nodeId}:generateRandomWalk`](/reference/rest-api/?av=v2#/operations/generate-random-walk) endpoint once that you know the node's ID.

```bash
curl --request POST --url "http://localhost:8080/api/v2/graph/quine/algorithms/randomWalk/nodes/{nodeId}:generateRandomWalk"
```

## Full Graph Walks

Node-anchored walks start from one node and return one random walk. But most graph A.I. algorithms require building many walks from every node in the graph. To support this, Quine includes an API that will generate a stream of all random walks into a file for an entire graph—regardless of how large the graph is.

With an API `POST` to the [Save Random Walks: `POST /api/v2/graph/quine/algorithms/randomWalk:saveWalks`](/reference/rest-api/?av=v2#/operations/save-random-walks) endpoint, you can direct Quine to stream all the random walks from every node in the graph to a file stored locally or in an S3 bucket.

```bash
curl --request POST --url http://localhost:8080/api/v2/graph/quine/algorithms/randomWalk:saveWalks --header 'Content-Type: application/json' --data '{ "bucketName": "your-s3-bucket-name", "type": "S3Bucket" }'
```

### Graph Walk File Output

The output file is a CSV where each row is one random walk. The first column will always be the node ID where the walk originated. Each subsequent column will be either:

a.) by default, the ID of each node encountered (including the starting node ID again in the second
column), or

b.) optionally, the results of Cypher query executed from each node encountered on the walk; where
 multiple columns and rows returned from this query will be concatenated together sequentially into
 the aggregated walk results.
 
**The resulting CSV may have rows of varying length.**
 
The name of the output file is derived from the arguments used to generate it; or a custom file name can be specified in the API request body. If no custom name is specified, the following values are concatenated to produce the final file name:
 
  - the constant prefix: `graph-walk-`
  - the timestamp provided in `atTime` or else the current time when run. A trailing `_T` is appended if no timestamp was specified.
  - the `length` parameter followed by the constant `x`
  - the `count` parameter
  - the constant `-q` follow by the number of characters in the supplied `query` (`0` if not specified)
  - the `return` parameter followed by the constant `x`
  - the `inOut` parameter
  - the `seed` parameter or `_` if none was supplied
  - the constant suffix `.csv`
 
Example file name: `graph-walk-1675122348011_T-10x5-q0-1.0x1.0-_.csv`
 
The name of the actual file being written is returned in the API response body.

<!-- TODO
>Document local and S3 output data forms.
>Describe the nuances of using the s3 bucket … like authentication, or file path…
-->


---
# Files and Named Pipes
URL: https://quine.io/learn/ingest-sources/files-and-named-pipes/

# Files and Named Pipes

Files are a stream of data. It's easy to think of a file as a singular chunk, but on disk it is a linear sequence of bits. Reading data from a file is the process of starting at the beginning, and reading each bit in sequence, until you reach the end. Programs usually do this under the hood and deliver a single result when finished, but that data can be handled by a program before the end of file is reached. Reading a file until a particular sequence is found (the "delimiter") will yield a sequence of bytes which can be handled as a single event. This is how data is streamed into Quine from a file—making it a very natural and convenient way to load data into Quine.

## Security Controls

Quine implements directory allow listing and file enumeration controls to protect against path traversal vulnerabilities. File ingests are restricted to specific directories configured at startup.

Configure file ingest security in your configuration file:

```kconfig
quine.file-ingest {
  # Allow list of directories
  allowed-directories = ["/path/to/data"]

  # File resolution mode
  resolution-mode = "static"  # or "dynamic"
}
```

The `allowed-directories` setting specifies an allow list of directories from which files can be ingested. Relative paths are resolved against the working directory at startup. An empty list means no file ingests are allowed except from recipes. By default, open source Quine allows ingestion from the working directory (`["."]`) to provide low friction for development and testing, ensuring existing recipes work without configuration changes.

!!! warning "Symbolic links are not allowed"
    Symbolic links (symlinks) are not permitted within allowed directories. A symlink could be used by a bad actor to reference sensitive files outside the allowed directory — for example, linking to a password file — thereby tricking Quine into ingesting secret information. Quine will reject any file ingest that resolves to a symlink.

The `resolution-mode` setting controls which files within allowed directories can be ingested. Set to `"static"` to only allow files that were present at startup (more secure), or `"dynamic"` to allow any file in allowed directories, even files added after startup (more flexible). Open source Quine defaults to `"dynamic"` for development flexibility.

## File Ingest Format

Each file has one special feature which distinguishes it from a normal stream of data: *it has an end*. When ingesting data from one of these file types, the stream will continue until the end-of-file signal is reached. When the end-of-file signal is reached, the stream will finish and be marked by Quine as `COMPLETED`.

### JSON[L]

`"type" : "CypherJson"`

JSON data is ubiquitous; it is a very natural source of a data for a streaming system. The typical approach is to save records as JSON objects to a file, separated by new-lines: `\n`. It is very common to call that format a `.json` file. That format is not technically valid JSON, and so sometimes it is referred to as "[JSON Lines](https://jsonlines.org)" and given the `.jsonl` file extension.

Quine reads these `.json` or `.jsonl` files one line at a time, passing each parsed JSON object to the Cypher ingest query as `$that` to be used in the rest of the ingest query.

### CSV

`"type" : "CypherCsv"`

Comma-Separated Values are a convenient way to store tabular data. Quine reads CSV files natively, passing each parsed line into a Cypher query as: `$that`. Fields from each row are accessible by numeric index or by key, depending on whether `headers` are provided. The `headers` field can either be read from the first line of the CSV, provided manually as a list of strings, or disabled entirely. See the [REST API reference](../../reference/rest-api.md) for details.

### Text

`"type" : "CypherLine"`

Any text file can be used as as stream of input to Quine. Using the `CypherLine` ingest option will read in the specified file, passing each individual line as a string to the provided Cypher query, where it can be used directly as a string value, or parsed by hand into a more meaningful structure.

## Named Pipe

A named pipe is an operating-system/file-system abstraction typically used for inter-process communication. A named pipe has a file-like representation in the filesystem—so it looks like a file to most programs—but it does not persist data. The operating system uses this object that looks like a file as a communication channel. One program can write to a named pipe, and if another program is actively reading from the same named pipe, then the second program receives the data written by the first program.

Quine supports named pipes on Unix-like operating systems (MacOS, Linux, etc), but not on Windows. On most Unix-like operating systems, Quine will automatically detect whether the provided path refers to a regular file or a named pipe. Automatic detection can be overridden by setting the `fileIngestMode` setting in the ingest API.

When Quine has an active stream ingesting from a named pipe, it will consume data from that pipe until the stream is cancelled, or it encounters an error. The format of the incoming data from the named pipe can be any of the [File Ingest Formats](#file-ingest-format) listed above. Data written to the named pipe will be separated by new-line characters (`\n`) and passed to the provided Cypher query exactly as described above.


---
# Apache Kafka
URL: https://quine.io/learn/ingest-sources/kafka/

# Apache Kafka

## Reading Records from Kafka

Quine has full support for reading records from Apache Kafka topics. The means by which Quine interprets records into graph data is configurable via [REST API](../../reference/rest-api.md) and [Recipes](../../learn/recipe-ref-manual.md). The `format` field controls record decoding; see [Record Formats](index.md#record-formats) for the available formats and how to choose between them.

In addition to the API-mapped Kafka options, [arbitrary Kafka configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#ak-consumer-configurations-for-cp) can be provided via the `kafkaProperties` field.

In order to avoid confusion and mitigate certain vulnerabilities in the Kafka client libraries, certain configuration keys are disabled by a validation step. The `bootstrap.servers` property is disallowed because it duplicates the `bootstrapServers` field. Similarly, certain values for the `sasl.jaas.config` property are known to introduce vulnerabilities, so those property values are forbidden.

Because of the extreme variety of possible configuration combinations, we cannot provide a comprehensive guide on configuring Kafka. However, we recommend using `securityProtocol: "SSL"` wherever possible to encrypt requests between Quine and the Kafka broker.

## Secure Kafka Configuration

Quine supports typed "secret" parameters for Kafka security configuration. These parameters are redacted in API responses and logs (replaced with `Secret(****)`, usually, or `****` if a small part of a larger value).

### SSL/TLS Passwords

| Parameter               | Type            | Description                                  |
|-------------------------|-----------------|----------------------------------------------|
| `sslKeystorePassword`   | string (secret) | Password for the SSL keystore file           |
| `sslTruststorePassword` | string (secret) | Password for the SSL truststore file         |
| `sslKeyPassword`        | string (secret) | Password for the private key in the keystore |

### SASL Authentication

The `saslJaasConfig` parameter accepts the following authentication types:

#### PlainLogin

For SASL/PLAIN authentication:

```json
{
  "saslJaasConfig": {
    "type": "PlainLogin",
    "username": "my-username",
    "password": "my-password"
  }
}
```

#### ScramLogin

For SASL/SCRAM-SHA-256 or SCRAM-SHA-512 authentication:

```json
{
  "saslJaasConfig": {
    "type": "ScramLogin",
    "username": "my-username",
    "password": "my-password"
  }
}
```

#### OAuthBearerLogin

For SASL/OAUTHBEARER authentication:

```json
{
  "saslJaasConfig": {
    "type": "OAuthBearerLogin",
    "clientId": "my-client-id",
    "clientSecret": "my-client-secret",
    "scope": "optional-scope",
    "tokenEndpointUrl": "https://auth.example.com/oauth/token"
  }
}
```

| Field              | Required | Description                                 |
|--------------------|----------|---------------------------------------------|
| `clientId`         | Yes      | OAuth client identifier                     |
| `clientSecret`     | Yes      | OAuth client secret (redacted in responses) |
| `scope`            | No       | OAuth scope for the token request           |
| `tokenEndpointUrl` | No       | Token endpoint URL                          |

### Migrating from kafkaProperties

The typed Secret parameters take precedence over corresponding entries in `kafkaProperties`. When both are configured, a warning is logged. For example:

```
WARN - Kafka property 'ssl.keystore.password' in kafkaProperties will be overridden
by typed Secret parameter. Remove 'ssl.keystore.password' from kafkaProperties to
suppress this warning.
```

**Affected properties:**

| kafkaProperties key       | Typed parameter         |
|---------------------------|-------------------------|
| `ssl.keystore.password`   | `sslKeystorePassword`   |
| `ssl.truststore.password` | `sslTruststorePassword` |
| `ssl.key.password`        | `sslKeyPassword`        |
| `sasl.jaas.config`        | `saslJaasConfig`        |

**Before (unprotected):**

```json
{
  "type": "Kafka",
  "topic": "events",
  "bootstrapServers": "kafka:9093",
  "kafkaProperties": {
    "security.protocol": "SASL_SSL",
    "ssl.keystore.password": "keystore-secret",
    "sasl.jaas.config": "org.apache.kafka...PlainLoginModule required username=\"user\" password=\"pass\";"
  }
}
```

**After (protected):**

```json
{
  "type": "Kafka",
  "topic": "events",
  "bootstrapServers": "kafka:9093",
  "sslKeystorePassword": "keystore-secret",
  "saslJaasConfig": {
    "type": "PlainLogin",
    "username": "user",
    "password": "pass"
  },
  "kafkaProperties": {
    "security.protocol": "SASL_SSL"
  }
}
```

!!! note "Credential Redaction"
    The typed Secret parameters are automatically redacted in API responses, displaying as `Secret(****)`. Values in `kafkaProperties` are **not** redacted: migrate sensitive values to the typed parameters for protection.

## OAuth Bearer Token Authentication (SASL/OAUTHBEARER)

Quine can authenticate to Kafka brokers using OAuth 2.0 via the SASL/OAUTHBEARER mechanism. This uses the [client credentials grant](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4) to obtain access tokens from an OAuth identity provider (e.g. Keycloak, Azure Entra ID, Okta) and present them to the Kafka broker.

### Required Configuration

OAuth credentials (client ID and secret) should be provided via the typed [`saslJaasConfig`](#oauthbearerlogin) parameter with `type: OAuthBearerLogin`. This ensures credentials are automatically redacted in API responses. The remaining OAuth properties are configured through `kafkaProperties`:

| Property | Description |
|----------|-------------|
| `sasl.mechanism` | Must be set to `OAUTHBEARER`. |
| `sasl.login.callback.handler.class` | Must be set to `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler`. This tells the Kafka client how to exchange client credentials for an access token. |
| `sasl.oauthbearer.token.endpoint.url` | The OAuth token endpoint URL from your identity provider. |

For encrypted connections (recommended), the security protocol must also be set to `SASL_SSL`. See [TLS Trust Configuration](#tls-trust-configuration) below for the additional properties needed when using TLS.

### Ingest Stream Configuration

The `Kafka` source type supports `securityProtocol` as a field. Set it to `SASL_SSL`, provide OAuth credentials via `saslJaasConfig`, and the remaining properties in `kafkaProperties`:

```json
{
  "name": "my-kafka-ingest",
  "source": {
    "type": "Kafka",
    "topics": ["my-topic"],
    "bootstrapServers": "kafka-broker:9096",
    "groupId": "my-consumer-group",
    "securityProtocol": "SASL_SSL",
    "autoOffsetReset": "EARLIEST",
    "saslJaasConfig": {
      "type": "OAuthBearerLogin",
      "clientId": "my-client-id",
      "clientSecret": "my-client-secret",
      "tokenEndpointUrl": "https://my-idp.example.com/realms/kafka/protocol/openid-connect/token"
    },
    "kafkaProperties": {
      "sasl.mechanism": "OAUTHBEARER",
      "sasl.login.callback.handler.class": "org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler",
      "sasl.oauthbearer.token.endpoint.url": "https://my-idp.example.com/realms/kafka/protocol/openid-connect/token",
      "ssl.truststore.type": "PEM",
      "ssl.truststore.location": "/path/to/ca.crt"
    }
  },
  "query": "MATCH (n) WHERE id(n) = idFrom($that) SET n = $that"
}
```

!!! note "Token Endpoint URL"
    The `sasl.oauthbearer.token.endpoint.url` must be set in `kafkaProperties` even when `tokenEndpointUrl` is provided in `saslJaasConfig`. The Kafka `OAuthBearerLoginCallbackHandler` reads the token endpoint from the client configuration, not the JAAS string.

### Standing Query Output Configuration

The `Kafka` destination type does **not** have a top-level `securityProtocol` field. Instead, set `security.protocol` inside `kafkaProperties`. OAuth credentials are still provided via `saslJaasConfig`:

```json
{
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "to-kafka",
      "destinations": [
        {
          "type": "Kafka",
          "topic": "my-output-topic",
          "bootstrapServers": "kafka-broker:9096",
          "saslJaasConfig": {
            "type": "OAuthBearerLogin",
            "clientId": "my-client-id",
            "clientSecret": "my-client-secret",
            "tokenEndpointUrl": "https://my-idp.example.com/realms/kafka/protocol/openid-connect/token"
          },
          "kafkaProperties": {
            "security.protocol": "SASL_SSL",
            "sasl.mechanism": "OAUTHBEARER",
            "sasl.login.callback.handler.class": "org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler",
            "sasl.oauthbearer.token.endpoint.url": "https://my-idp.example.com/realms/kafka/protocol/openid-connect/token",
            "ssl.truststore.type": "PEM",
            "ssl.truststore.location": "/path/to/ca.crt"
          }
        }
      ]
    }
  ]
}
```

!!! warning "securityProtocol Asymmetry"
    Ingest streams accept `securityProtocol` as a top-level field on the ingest configuration, but standing query outputs do not. For outputs, you must set `security.protocol` inside `kafkaProperties`. This applies to all security protocols, not just `SASL_SSL`.

### TLS Trust Configuration

When using `SASL_SSL`, the Kafka client needs to trust the broker's TLS certificate. Configure trust via `kafkaProperties`:

| Property | Description |
|----------|-------------|
| `ssl.truststore.type` | Truststore format: `PEM` for a CA certificate file, or `JKS`/`PKCS12` for a keystore. |
| `ssl.truststore.location` | Path to the truststore file. For `PEM`, this is the CA certificate file (e.g. `/path/to/ca.crt`). |
| `ssl.truststore.password` | Password for the truststore (required for `JKS` and `PKCS12`; not needed for `PEM`). |

If the Kafka broker uses certificates signed by a well-known CA, no truststore configuration is needed: the JVM's default trust store will be used. Custom truststore configuration is typically required when:

- The broker uses certificates signed by an internal or private CA.
- The broker uses self-signed certificates.
- The OAuth identity provider's TLS certificate is also signed by a private CA (the truststore must include both the broker CA and the IdP CA).

### Identity Provider Requirements

The OAuth identity provider must be configured with a client that supports the client credentials grant. At minimum:

- **Client type**: Confidential (i.e. has a client secret)
- **Grant type**: Client credentials (`grant_type=client_credentials`)
- **Audience**: If the Kafka broker validates the `aud` claim in the access token, the identity provider must include an audience mapper that adds the expected audience value to the token.

### Example

In this example we will ingest messages from a Kafka topic and store them as nodes in the graph.

#### Preparation

For this example we will run Kafka locally. Because Kafka depends on ZooKeeper, we will start that too. [Download Kafka](https://kafka.apache.org/downloads), and extract the files to your local filesystem. Start each of ZooKeeper and Kafka in separate terminal sessions by running each of the following commands from the directory where you extracted Kafka.

```
bin/zookeeper-server-start.sh config/zookeeper.properties
```

```
bin/kafka-server-start.sh config/server.properties
```

With Kafka up and running, messages can be manually sent to the topic using the following command:

```
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic
>{"Message": "Hello, world."}
>^D
```

While `kafka-console-producer.sh` is running, messages are generated by inputting text followed by a new line. To end the program and stop generating messages, use Control-D.

#### Using a Recipe

The following is a simple Recipe that ingests each message from the Kafka topic as a node in the graph:

```yaml
--8<-- "recipes/assets/kafka-ingest.yaml"
```

To run this Recipe, run Quine as follows:

```
❯ java -jar quine-2.1.1.jar -r kafka-ingest.yaml
Graph is ready
Running Recipe Kafka Ingest
Running Ingest Stream INGEST-1
Quine app web server available at http://localhost:8080

 | => INGEST-1 status is running and ingested 0
```

Quine has downloaded the Recipe and begun execution. As shown above, use `kafka-console-producer.sh` to send a JSON record to the stream. Quine should immediately report that it has ingested the record.

```
| => INGEST-1 status is running and ingested 1
```

Results should already be available in the web UI at `https://<hostname>:8080`.


---
# AWS Kinesis
URL: https://quine.io/learn/ingest-sources/kinesis/

# AWS Kinesis

Quine provides two methods for ingesting data from Amazon Kinesis Data Streams. Configure Kinesis ingests via the [REST API](../../reference/rest-api.md).

| Ingest Type    | Use Case                                                                         | Features                                                              |
|----------------|----------------------------------------------------------------------------------|-----------------------------------------------------------------------|
| **Kinesis**    | Single-instance deployments, specific shard selection, lightweight setup         | Direct shard access, sequence number positioning                      |
| **KinesisKCL** | Multi-worker deployments, fault-tolerant processing, automatic shard rebalancing | Checkpointing, lease management, enhanced fan-out (requires DynamoDB) |

!!! tip "When to use KinesisKCL"
    Use **KinesisKCL** when you need checkpointing across restarts, distributed processing across multiple workers, or automatic handling of shard splits and merges. Note that KCL requires additional AWS resources (DynamoDB for leases, CloudWatch for metrics) which incur extra costs.

---

## Kinesis

The `Kinesis` type provides direct access to Kinesis shards without the overhead of lease management or checkpointing. Use this for single-instance deployments, lightweight setups, or when you need precise control over which shards to read.

### Example

In this example, we will register a multiple-shard Kinesis stream of JSON objects (one JSON object per Kinesis record) as a data source, creating a single node in the graph for each object.

#### Preparation

For the purposes of this example, you will need [a Kinesis data stream](https://console.aws.amazon.com/kinesis/home#/streams/create) and credentials (an access key ID and secret access key) for an [IAM User](https://console.aws.amazon.com/iam/home?#/users$new?step=details) with the following privileges:

- kinesis:RegisterStreamConsumer
- kinesis:DeregisterStreamConsumer
- kinesis:SubscribeToShard
- kinesis:DescribeStreamSummary
- kinesis:DescribeStreamConsumer
- kinesis:GetShardIterator
- kinesis:GetRecords
- kinesis:DescribeStream
- kinesis:ListTagsForStream

For our example, we'll assume we have such a user with access to the `json-logs` stream with access key ID `<ACCESS_KEY_ID>` and secret `<SECRET_ACCESS_KEY>`. These will be used to register the data source with Quine.

#### Registering Kinesis as a data source

To register Kinesis as a data source to Quine, we need to describe our stream via the ingest [REST API](../../reference/rest-api.md).

For example, we'll use the aforementioned Kinesis stream hosted in the region `us-west-2`, named `json-logs` and we'll give the Quine ingest stream the name `kinesis-logs`. Thus, we make our API request using [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) with the following payload:

```json
{
  "name": "kinesis-logs",
  "source": {
    "type": "Kinesis",
    "streamName": "json-logs",
    "shardIds": [],
    "credentials": {
      "accessKeyId": "<ACCESS_KEY_ID>",
      "secretAccessKey": "<SECRET_ACCESS_KEY>"
    },
    "region": "us-west-2",
    "iteratorType": "TrimHorizon"
  },
  "parallelism": 2,
  "query": "CREATE ($that)"
}
```

We pass in an empty list of shard IDs to specify that Quine should read from all shards in the stream. If we wanted to only read from particular shards, we would instead list out the shard IDs from which Quine should read.

Because the Kinesis stream is filled with JSON records, each record is read as a JSON object and passed as a `Map` to the Cypher query. The query accesses this object using the parameter `$that`. Thus, our configured query `CREATE ($that)` will create a node for each JSON record with the same property structure as the JSON record.

In this example, we use a Kinesis stream populated with JSON objects as records, though Quine offers other options for how to interpret records from a stream. These options are configurable via the same endpoint by using different `format`s in the above JSON payload. See [Record Formats](index.md#record-formats) for the full list and the difference between `Json` and `Raw`.

Finally, we choose to read all records from the Kinesis stream, including records already present in the stream when configuring the Quine data source. To get this behavior, we use a `TrimHorizon` Kinesis iterator type. If we wished to only read records written to the Kinesis stream *after* setting up the Quine data source, we would have used the `Latest` iterator type.

### Kinesis Configuration Reference

| Parameter          | Type          | Default     | Description                                                                                                                      |
|--------------------|---------------|-------------|----------------------------------------------------------------------------------------------------------------------------------|
| `streamName`       | string        | required    | Name of the Kinesis stream                                                                                                       |
| `shardIds`         | array         | [] (all)    | Specific shard IDs to read; empty for all shards                                                                                 |
| `format`           | object        | required    | Record format and Cypher query for processing                                                                                    |
| `parallelism`      | int           | 16          | Maximum concurrent database writes                                                                                               |
| `credentials`      | object        | env default | AWS credentials (see [AWS Credentials](#aws-credentials))                                                                        |
| `region`           | string        | env default | AWS region (see [AWS Region](#aws-region))                                                                                       |
| `iteratorType`     | string/object | `Latest`    | Starting position in the stream                                                                                                  |
| `numRetries`       | int           | 3           | Retry attempts on errors                                                                                                         |
| `maximumPerSecond` | int           | unlimited   | Rate limit for records processed per second                                                                                      |
| `recordDecoders`   | array         | []          | [Record decodings](index.md#record-decoding) to apply (e.g., `Zlib`, `Gzip`, `Base64`) |

### Iterator Type Options

The `iteratorType` determines where to start reading in the stream:

| Value                                                  | Description                              |
|--------------------------------------------------------|------------------------------------------|
| `"Latest"`                                             | Start with new records only              |
| `"TrimHorizon"`                                        | Start from the oldest available record   |
| `{"AtSequenceNumber": {"sequenceNumber": "..."}}`      | Start at a specific sequence number      |
| `{"AfterSequenceNumber": {"sequenceNumber": "..."}}`   | Start after a specific sequence number   |
| `{"AtTimestamp": {"millisSinceEpoch": 1234567890000}}` | Start at a Unix timestamp (milliseconds) |

---

## KinesisKCL

The KinesisKCL type uses the [AWS Kinesis Client Library (KCL) 3.x](https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html) to provide distributed stream processing with:

- **Checkpointing**: Automatic tracking of processed records via DynamoDB
- **Lease Management**: Distributed coordination of shard processing across multiple workers
- **Enhanced Fan-Out**: Dedicated throughput per consumer (optional)
- **Automatic Shard Handling**: Seamless processing of shard splits and merges
- **CloudWatch Metrics**: Built-in monitoring and observability

### Prerequisites

KinesisKCL requires additional AWS resources and permissions beyond basic Kinesis access.

#### IAM Permissions

Your IAM user or role needs permissions for Kinesis, DynamoDB (lease table), and CloudWatch (metrics):

**Kinesis permissions:**

- kinesis:DescribeStream
- kinesis:DescribeStreamSummary
- kinesis:GetRecords
- kinesis:GetShardIterator
- kinesis:ListShards
- kinesis:ListTagsForStream
- kinesis:SubscribeToShard (for Enhanced Fan-Out)
- kinesis:RegisterStreamConsumer (for Enhanced Fan-Out)
- kinesis:DescribeStreamConsumer (for Enhanced Fan-Out)
- kinesis:DeregisterStreamConsumer (for Enhanced Fan-Out)

**DynamoDB permissions (for lease table):**

- dynamodb:CreateTable
- dynamodb:DescribeTable
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
- dynamodb:Scan

**CloudWatch permissions (for metrics):**

- cloudwatch:PutMetricData

### Basic KinesisKCL Example

```json
{
  "name": "kinesis-kcl-logs",
  "source": {
    "type": "KinesisKCL",
    "kinesisStreamName": "json-logs",
    "applicationName": "quine-json-logs-processor",
    "credentials": {
      "accessKeyId": "<ACCESS_KEY_ID>",
      "secretAccessKey": "<SECRET_ACCESS_KEY>"
    },
    "region": "us-west-2",
    "initialPosition": "TrimHorizon"
  },
  "parallelism": 16,
  "query": "CREATE ($that)"
}
```

The `applicationName` serves as the identifier for your consumer application and is used as the default name for the DynamoDB lease table and CloudWatch metrics namespace.

### KinesisKCL Configuration Reference

#### Core Settings

| Parameter           | Type          | Default     | Description                                                                                                                                    |
|---------------------|---------------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| `kinesisStreamName` | string        | required    | Name of the Kinesis stream to ingest                                                                                                           |
| `applicationName`   | string        | required    | Unique application name; used as the DynamoDB lease table name and CloudWatch namespace                                                        |
| `format`            | object        | required    | Record format and Cypher query for processing                                                                                                  |
| `parallelism`       | int           | 16          | Maximum concurrent database writes                                                                                                             |
| `credentials`       | object        | env default | AWS credentials (see [AWS Credentials](#aws-credentials))                                                                                      |
| `region`            | string        | env default | AWS region (see [AWS Region](#aws-region))                                                                                                     |
| `initialPosition`   | string/object | `Latest`    | Where to start reading: `Latest`, `TrimHorizon`, or `AtTimestamp`                                                                              |
| `numRetries`        | int           | 3           | Number of retry attempts on Kinesis errors                                                                                                     |
| `maximumPerSecond`  | int           | unlimited   | Rate limit for records processed per second                                                                                                    |
| `recordDecoders`    | array         | []          | [Record decodings](index.md#record-decoding) applied to each record (e.g., `Zlib`, `Gzip`, `Base64`) |

#### Initial Position Options

The `initialPosition` determines where KCL begins reading when no checkpoint exists:

| Value                                                                                                 | Description                                                                          |
|-------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| `"Latest"`                                                                                            | Start with records added after the ingest begins                                     |
| `"TrimHorizon"`                                                                                       | Start from the oldest available record in the stream                                 |
| `{"AtTimestamp": {"year": 2025, "month": 4, "date": 15, "hourOfDay": 10, "minute": 30, "second": 0}}` | Start from records at or after the specified timestamp (month and day are 1-indexed) |

!!! note "InitialPosition vs IteratorType"
    KinesisKCL uses `initialPosition` which supports `Latest`, `TrimHorizon`, and `AtTimestamp` only. Kinesis uses `iteratorType` which also supports `AtSequenceNumber` and `AfterSequenceNumber`.

### Checkpoint Settings

Checkpointing tracks which records have been successfully processed, enabling recovery after failures. Configure via `checkpointSettings`:

```json
{
  "checkpointSettings": {
    "disableCheckpointing": false,
    "maxBatchSize": 1000,
    "maxBatchWaitMillis": 10000
  }
}
```

| Parameter              | Type    | Default | Description                                            |
|------------------------|---------|---------|--------------------------------------------------------|
| `disableCheckpointing` | boolean | false   | Set to `true` to disable checkpointing entirely        |
| `maxBatchSize`         | int     | none    | Maximum records to batch before checkpointing          |
| `maxBatchWaitMillis`   | long    | none    | Maximum time (ms) to wait before checkpointing a batch |

!!! warning "Disabling Checkpointing"
    Disabling checkpointing means records may be reprocessed after restarts. Only disable this for idempotent operations or development/testing scenarios.

### Scheduler Source Settings

Control the internal buffer and backpressure behavior via `schedulerSourceSettings`:

```json
{
  "schedulerSourceSettings": {
    "bufferSize": 1000,
    "backpressureTimeoutMillis": 60000
  }
}
```

| Parameter                   | Type | Default | Description                                                   |
|-----------------------------|------|---------|---------------------------------------------------------------|
| `bufferSize`                | int  | none    | Internal buffer size; must be > 0; use 1 to disable buffering |
| `backpressureTimeoutMillis` | long | none    | Timeout (ms) for backpressure before failing                  |

### Advanced KCL Configuration

For fine-grained control over KCL behavior, use the `advancedSettings` object. These settings map directly to the [KCL 3.x configuration options](https://docs.aws.amazon.com/streams/latest/dev/kcl-configuration.html).

| Configuration Group       | Purpose                                                                                                                                                                                     |
|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `configsBuilder`          | Custom lease table name and worker identifier                                                                                                                                               |
| `retrievalSpecificConfig` | Choose between Polling (shared throughput) or [Enhanced Fan-Out](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html) (dedicated 2 MB/s per consumer) |
| `leaseManagementConfig`   | Shard lease coordination and DynamoDB table settings                                                                                                                                        |
| `coordinatorConfig`       | Shard prioritization and sync behavior                                                                                                                                                      |
| `lifecycleConfig`         | Task retry and warning thresholds                                                                                                                                                           |
| `retrievalConfig`         | ListShards retry settings                                                                                                                                                                   |
| `metricsConfig`           | CloudWatch metrics level and dimensions                                                                                                                                                     |
| `processorConfig`         | Empty record list handling                                                                                                                                                                  |

See the [API documentation](/reference/rest-api/?av=v2#/operations/create-ingest) for complete parameter details.

### Complete KinesisKCL Example

Here's a comprehensive example with advanced settings for a production deployment using Enhanced Fan-Out:

```json
{
  "name": "kinesis-kcl-prod",
  "source": {
    "type": "KinesisKCL",
    "kinesisStreamName": "production-events",
    "applicationName": "quine-prod-processor",
    "credentials": {
      "accessKeyId": "<ACCESS_KEY_ID>",
      "secretAccessKey": "<SECRET_ACCESS_KEY>"
    },
    "region": "us-west-2",
    "initialPosition": "TrimHorizon",
    "numRetries": 5,
    "checkpointSettings": {
      "maxBatchSize": 1000,
      "maxBatchWaitMillis": 5000
    },
    "advancedSettings": {
    "configsBuilder": {
      "tableName": "quine-prod-leases"
    },
    "retrievalSpecificConfig": {
      "type": "FanOutConfig",
      "consumerName": "quine-prod-consumer"
    },
    "leaseManagementConfig": {
      "failoverTimeMillis": 10000,
      "maxLeasesForWorker": 50,
      "billingMode": "PAY_PER_REQUEST",
      "isGracefulLeaseHandoffEnabled": true,
      "gracefulLeaseHandoffTimeoutMillis": 30000
    },
    "metricsConfig": {
      "metricsLevel": "SUMMARY"
    }
  },
  "parallelism": 32,
  "maximumPerSecond": 10000,
  "query": "MATCH (n) WHERE id(n) = idFrom('event', $that.eventId) SET n = $that"
}
```

---

## AWS Credentials

### Explicit Credentials

```json
{
  "credentials": {
    "accessKeyId": "<ACCESS_KEY_ID>",
    "secretAccessKey": "<SECRET_ACCESS_KEY>"
  }
}
```

!!! note "Credential Redaction in API Responses"
    For security, `accessKeyId` and `secretAccessKey` values are automatically redacted in API responses. These fields display as `Secret(****)` instead of their actual values. This does not affect how credentials are stored or used internally—only the API response is redacted. When configuring credentials via POST or PUT requests, provide the actual plaintext values.

### Environment Credentials

If `credentials` is omitted, Quine uses the [default AWS credential provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

---

## AWS Region

### Explicit Region

```json
{
  "region": "us-west-2"
}
```

### Environment Region

If `region` is omitted, Quine uses the [region provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html#automatically-determine-the-aws-region-from-the-environment).

---

!!! tip "Need help?"

    See [Troubleshooting Ingest](../troubleshooting/ingest.md) for help with missing data, slow ingests, and other common issues.

## Additional Resources

- [AWS Kinesis Data Streams Documentation](https://docs.aws.amazon.com/streams/latest/dev/introduction.html)
- [KCL 3.x Configuration Reference](https://docs.aws.amazon.com/streams/latest/dev/kcl-configuration.html)
- [Enhanced Fan-Out Consumers](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html)
- [Kinesis Data Streams Pricing](https://aws.amazon.com/kinesis/data-streams/pricing/)


---
# Reactive Streams
URL: https://quine.io/learn/ingest-sources/reactive-streams/

# Reactive Streams

Reactive Streams enable TCP-based, backpressured communication between thatDot products. Use them to connect Quine Enterprise and Novelty for bidirectional data processing pipelines.

| Role | Description | Use Case |
|------|-------------|----------|
| **Publisher (Server)** | Binds to a port and broadcasts data to connected subscribers | Standing query outputs, observation results |
| **Subscriber (Client)** | Connects to a publisher to receive streamed data | Ingest from another product's output |

---

## ReactiveStream as Ingest Source

The `ReactiveStream` ingest source connects as a client to an existing reactive stream publisher. Use this to ingest data from another product's standing query or observation output.

### Example

In this example, we configure Quine to ingest data from a reactive stream server running on port 9002.

#### Registering via the API

To register a ReactiveStream ingest via [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest):

```json
{
  "name": "from-reactive-stream",
  "source": {
    "type": "ReactiveStream",
    "url": "localhost",
    "port": 9002,
    "format": {
      "type": "Json"
    }
  },
  "query": "CREATE (n:Event $that)",
  "parameter": "that",
  "parallelism": 1
}
```

#### Using a Recipe

The same ingest stream defined in a [Recipe](../../learn/recipe-ref-manual.md):

```yaml
ingestStreams:
  - name: reactive-stream-ingest
    source:
      type: ReactiveStream
      url: localhost
      port: 9002
    query: |-
      MATCH (n)
      WHERE id(n) = idFrom($that.id)
      SET n = $that
```

### Ingest Source Configuration Reference

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `type` | string | Yes | Must be `"ReactiveStream"` |
| `url` | string | Yes | Hostname of the reactive stream server to connect to |
| `port` | integer | Yes | Port of the reactive stream server |
| `format` | object | Yes | Record format (e.g., `Json`, `CypherJson`). See [Record Formats](index.md#record-formats) for the full list and the difference between `Json` and `Raw`. |

---

## ReactiveStream as Output Destination

The `ReactiveStream` output destination creates a server that broadcasts results to connected subscribers. Use this to publish standing query results for downstream processing by other products.

### Example

Configure a standing query to output results to a reactive stream on port 9001:

```json
{
  "name": "events-to-downstream",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n:Event) RETURN DISTINCT id(n) AS id",
    "mode": "DISTINCT_ID"
  },
  "outputs": [
    {
      "name": "to-reactive-stream",
      "destinations": [
        {
          "type": "ReactiveStream",
          "address": "0.0.0.0",
          "port": 9001,
          "format": {
            "type": "JSON"
          }
        }
      ]
    }
  ]
}
```

### Output Destination Configuration Reference

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `type` | string | Yes | — | Must be `"ReactiveStream"` |
| `address` | string | No | `localhost` | Address to bind the reactive stream server |
| `port` | integer | Yes | — | Port to bind the reactive stream server |
| `format` | object | Yes | — | Output format (`JSON` or `Protobuf`) |

!!! warning "Cluster Limitation"
    Reactive Stream outputs do not function correctly when running in a cluster. Use Kafka or Kinesis for clustered deployments.

---

## Connecting Quine Enterprise and Novelty

Reactive Streams enable powerful bidirectional data pipelines between Quine Enterprise and Novelty. This pattern allows you to:

1. Process graph data in Quine Enterprise
2. Stream results to Novelty for anomaly detection
3. Feed Novelty's observations back into Quine Enterprise for further analysis

### Architecture

```
┌─────────────────────┐                    ┌─────────────────────┐
│   Quine Enterprise  │                    │       Novelty       │
│                     │                    │                     │
│  ┌───────────────┐  │   ReactiveStream   │  ┌───────────────┐  │
│  │ Standing Query├──┼──── port 9001 ────►│──┤    Ingest     │  │
│  └───────────────┘  │                    │  └───────┬───────┘  │
│                     │                    │          │          │
│                     │                    │          ▼          │
│                     │                    │  ┌───────────────┐  │
│  ┌───────────────┐  │   ReactiveStream   │  │  Observations │  │
│  │    Ingest     │◄─┼──── port 9002 ─────┤──┤    Output     │  │
│  └───────────────┘  │                    │  └───────────────┘  │
│                     │                    │                     │
└─────────────────────┘                    └─────────────────────┘
```

### Step 1: Quine Enterprise Standing Query Output

Configure Quine Enterprise to output standing query results to a reactive stream:

```json
{
  "name": "events-to-novelty",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n:Event) RETURN DISTINCT id(n) AS id",
    "mode": "DISTINCT_ID"
  },
  "outputs": [
    {
      "name": "to-novelty",
      "resultEnrichment": {
        "query": "MATCH (n) WHERE id(n) = $that.data.id RETURN {eventId: id(n), type: n.type, timestamp: n.timestamp} AS result",
        "parameter": "that",
        "parallelism": 1
      },
      "destinations": [
        {
          "type": "ReactiveStream",
          "address": "0.0.0.0",
          "port": 9001,
          "format": {
            "type": "JSON"
          }
        }
      ]
    }
  ]
}
```

### Step 2: Create a Novelty Model

Before creating an ingest in Novelty, you must create a model to store observations. Model names must be 1–16 lowercase characters, start with a letter, and contain only lowercase letters and digits (no hyphens, underscores, or special characters).

```bash
curl -X POST http://localhost:8181/api/v2/system/models \
  -H 'Content-Type: text/plain' \
  -d 'eventanalysis'
```

Returns `201 Created` if the model is new, or `204 No Content` if it already exists.

### Step 3: Novelty Ingest from Quine Enterprise

Configure Novelty to ingest from Quine Enterprise's reactive stream and output observations to another reactive stream. The model is specified in the URL path (`/model/{modelName}/ingests`) — this model must already exist.

```json
{
  "name": "from-quine",
  "transformation": "event-transform",
  "source": {
    "type": "ReactiveStream",
    "url": "localhost",
    "port": 9001,
    "format": {
      "type": "Json"
    }
  },
  "parallelism": 1,
  "outputWorkflow": {
    "type": "EachResult",
    "destinations": [
      {
        "type": "ReactiveStream",
        "address": "0.0.0.0",
        "port": 9002,
        "format": {
          "type": "JSON"
        }
      }
    ]
  }
}
```

### Step 4: Quine Enterprise Ingest from Novelty

Configure Quine Enterprise to ingest Novelty's observations:

```json
{
  "name": "from-novelty",
  "source": {
    "type": "ReactiveStream",
    "url": "localhost",
    "port": 9002,
    "format": {
      "type": "Json"
    }
  },
  "query": "CREATE (n:NoveltyObservation {score: $that.score, observation: $that.observation, sequence: $that.sequence, processedAt: datetime()})",
  "parameter": "that",
  "parallelism": 1
}
```

---

## Troubleshooting

### Connection Refused

If you see connection errors when starting a ReactiveStream ingest:

- Verify the publisher (server) is running and bound to the expected port
- Check that the `url` and `port` in your ingest configuration match the publisher's `address` and `port`
- Ensure no firewall rules are blocking the connection

### No Data Received

If the connection succeeds but no data flows:

- Verify the format types match between publisher and subscriber (e.g., both using `Json`)
- Check that the standing query or observation output is actively producing results
- Use the standing query statistics API to verify results are being generated

### Backpressure

If data processing slows or stalls:

- The subscriber may be overwhelmed; increase `parallelism` or optimize your ingest query
- Check downstream systems (persistor, standing queries) for bottlenecks
- Monitor Quine metrics for queue depth and processing rates

---

## Additional Resources

- [Standing Queries](../standing-queries/standing-queries.md) - Configure standing query outputs
- [Ingest Streams Overview](index.md) - General ingest configuration
- [REST API Reference](/reference/rest-api/?av=v2) - Complete API documentation


---
# Managing Upstream Data Source Changes in Quine
URL: https://quine.io/learn/ingest-sources/schema_changes/

# Managing Upstream Data Source Changes in Quine

## Introduction

In dynamic data environments, upstream data sources can change unexpectedly, impacting the data ingest process and the structure of your graph in Quine. This guide provides strategies and best practices to help you manage these changes effectively, ensuring the integrity and performance of your graph analytics.

## Understanding Changes in Upstream Data Sources

Upstream data sources may evolve due to schema updates, format changes, or alterations in data content. These changes can manifest as:

* **Schema Modifications**: Addition or removal of fields, changes in data types, or alterations in field names.  
* **Data Format Changes**: Switching between formats like JSON, CSV, or XML.  
* **Content Variations**: Introduction of new categories, entities, or relationships within the data.

Understanding the nature of these changes is crucial for adapting your ingest pipelines and maintaining a consistent graph structure.

## Identifying Issues in the Graph Due to Upstream Changes

When upstream data sources change without notice, you might observe the following issues in your graph:

* **Incomplete or Missing Data**: Nodes or edges that should be present are absent due to failed ingest.  
* **Unexpected Node Structures**: Nodes have unexpected properties or lack essential ones.  
* **Erroneous Relationships**: Edges connect incorrect nodes, leading to faulty relationships.  
* **Warnings/Errors**: Increased error logs or exceptions during the ingest process.  
* **Standing Query Disruptions**: Standing queries fail to trigger or produce incorrect results.

Regular monitoring can help detect these issues early, allowing for prompt remediation.

## Strategies for Translating Upstream Changes into the Existing Graph Structure

To manage upstream changes effectively, consider the following strategies:

### 1\. Implement Schema Validation

* **Use Validation Tools**: Integrate schema validation in your ingest pipeline to catch discrepancies.  
* **Define Acceptable Variations**: Specify which changes are tolerable and which should trigger alerts.

### 2\. Employ Flexible Parsing Techniques

* **Dynamic Field Handling**: Use parsers that can handle optional fields or unknown properties gracefully.  
* **Format Agnostic Parsing**: Utilize tools that can adapt to different data formats with minimal configuration changes.

### 3\. Update Ingest Configurations Proactively

* **Version Control Configurations**: Keep your ingest configurations under version control to track changes.  
* **Automate Updates**: Use scripts or tools to update configurations in response to detected schema changes.

### 4\. Leverage Data Transformation Pipelines

* **Transform Data Upstream**: Use ETL (Extract, Transform, Load) processes to normalize data before it reaches Quine.  
* **Map New Fields**: Update your transformation logic to map new or changed fields into your existing graph schema.

### 5\. Communicate with Data Providers

* **Establish Notifications**: Set up alerts or notifications from data providers about upcoming changes.  
* **Collaborate on Changes**: Work with providers to understand changes and plan adaptations accordingly.

## Impacts of Graph Structure Changes on Standing Queries

Standing queries in Quine are continuous queries that react to changes in the graph. Changes in the graph structure can impact standing queries by:

* **Breaking Pattern Matches**: Altered node or edge structures may no longer satisfy query patterns.  
* **Causing False Negatives/Positives**: Queries might miss relevant data or trigger on incorrect data.  
* **Performance Degradation**: Inefficient execution due to unexpected graph configurations.

For detailed information on standing queries, refer to the [Standing Queries Documentation](../standing-queries/standing-queries.md).

## Managing Standing Queries Amid Graph Changes

To mitigate the impact on standing queries:

### 1\. Review and Update Query Patterns

* **Adjust Patterns**: Modify query patterns to accommodate new data structures.  
* **Use Wildcards and Variables**: Incorporate flexibility into queries to handle variations.

### 2\. Test Queries Against Sample Data

* **Use Test Datasets**: Validate queries against datasets that include the upstream changes.  
* **Simulate Changes**: Introduce controlled changes to assess query resilience.

### 3\. Monitor Query Performance

* **Set Performance Benchmarks**: Establish baseline metrics to detect deviations.  
* **Analyze Query Results**: Regularly review outputs to ensure accuracy.

## Other Considerations

### Monitoring and Alerting

* **Implement Logging**: Enhance logging to capture ingest and query processing details.  
* **Set Up Alerts**: Configure alerts for ingest errors, schema mismatches, and performance issues.

### Version Control and Documentation

* **Document Changes**: Maintain detailed records of schema versions, configurations, and changes.  
* **Use Version Control Systems**: Track changes to ingest pipelines and configurations using tools like Git.

### Testing and Staging Environments

* **Create Staging Environments**: Test changes in a non-production environment before deployment.  
* **Automate Testing**: Integrate automated tests to validate ingest and query functionalities.

## Conclusion

Managing changes in upstream data sources is critical for maintaining the reliability of your graph in Quine. By implementing proactive strategies, monitoring systems, and maintaining clear communication with data providers, you can ensure seamless adaptations to changes and minimize disruptions to your data ingest and analysis workflows.


---
# AWS SNS and SQS Support
URL: https://quine.io/learn/ingest-sources/sqs---sns/

# AWS SNS and SQS Support

AWS SNS (Simple Notification Service) acts as a broadcast hub for notifications that can be delivered over multiple channels such as texting (SMS), email, or programmatic queues. AWS SQS (Simple Queue Service) is one such programmatic queue, which can act as a recipient for SNS messages or be used as a standalone message queuing service. Quine can interact with SNS and SQS by publishing Standing Query results to an SNS topic, or by reading messages (records) from an SQS queue.

## Using SQS to Read Records From SNS

Both SNS and SQS can be used in conjunction with Quine to ingest messages published to an SNS topic. In order to do so, simply register a new SQS queue as a subscriber to the SNS topic via the [SNS console](https://console.aws.amazon.com/sns/v3/home#/create-subscription), being sure to check the "enable raw message delivery" box. Then, ingest from SQS as described in the SQS section of this page.

![Registering an SQS subscriber](aws-sns-sqs-register.png)

## Reading Records from SQS

Quine has full support for reading records from SQS Queues. The means by which Quine interprets a record is highly configurable via the [REST API](../../reference/rest-api.md).

### SQS Ingest Example

In this example, we will register an SQS queue as a data source, creating a single node in the graph for each object, and acknowledging receipt of each record from SQS to dequeue it.

#### Preparation for SQS Ingest

For the purposes of this tutorial, you will need [an SQS queue](https://console.aws.amazon.com/sqs/v2/home#/create-queue) and credentials (an access key ID and secret access key) for an [IAM User](https://console.aws.amazon.com/iam/home?#/users$new?step=details) with the following [privileges](https://docs.aws.amazon.com/IAM/latest/UserGuide/list-amazonsqs.html) for that queue:

- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:DeleteMessageBatch
- sqs:ChangeMessageVisibility
- sqs:ChangeMessageVisibilityBatch
- sqs:GetQueueAttributes

For our example, we'll assume there is such a user with access to the `json-events` stream with access key ID `AKIAMYACCESSKEY` and secret `AWSScRtACCessKeyAWS/ScRtACCessKey`. These will be used to register the data source with Quine. We'll assume the queue has the URL `https://sqs.us-west-2.amazonaws.com/507123456123/json-events` and contains JSON-encoded data.

#### Registering SQS as a data source

To register SQS as a data source to Quine, we need to describe our queue via the ingest [REST API](../../reference/rest-api.md).

For example, we'll use the previously-described `json-events` queue, and we'll give the Quine ingest stream the name `main-queue`. Thus, we make our API request using [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest) with the following payload:

```json
{
  "name": "main-queue",
  "source": {
    "type": "SQS",
    "queueUrl": "https://sqs.us-west-2.amazonaws.com/507123456123/json-events",
    "credentials": {
      "accessKeyId": "AKIAMYACCESSKEY",
      "secretAccessKey": "AWSScRtACCessKeyAWS/ScRtACCessKey"
    },
    "region": "us-west-2",
    "deleteReadMessages": true
  },
  "query": "CREATE ($that)"
}
```

!!! note "Credential Redaction in API Responses"
    For security, `accessKeyId` and `secretAccessKey` values are automatically redacted in API responses. These fields display as `Secret(****)` instead of their actual values. This does not affect how credentials are stored or used internally—only the API response is redacted. When configuring credentials via POST or PUT requests, provide the actual plaintext values.

Because the SQS queue is filled with JSON records, each record is read as a JSON object and passed as a `Map` to the Cypher query. The query accesses this object using the parameter `$that`. Thus, our configured query `CREATE ($that)` will create a node for each JSON record with the same property structure as the JSON record. For other record formats — including `Raw`, which binds the unparsed bytes to `$that` — see [Record Formats](index.md#record-formats).

We also choose to acknowledge receipt of each message successfully read off the queue, and thus set "deleteReadMessages" to true (this is the default!) If we wanted to read messages off the SQS queue, but leave them in the queue for SQS to re-issue to other consumers later, we would set deleteReadMessages to false.

## Writing Results to SNS

Quine also supports writing [Standing Query](../../learn/standing-queries/standing-queries.md) results to AWS SNS.

### SNS Write Example

In this example, we will register an SNS topic as a target for Standing Query matches, issuing a single message for each match.

#### Preparation for SNS Output

You will need [an SNS topic](https://console.aws.amazon.com/sns/v3/home#/homepage) and credentials (an access key ID and secret access key) for an [IAM User](https://console.aws.amazon.com/iam/home?#/users$new?step=details) with the `sns:Publish` [privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/list-amazonsns.html) for that topic. For this example, we'll assume the same user (with access key `AKIAMYACCESSKEY`) has sufficient privileges for a topic `sns-json`

#### Registering SNS as a Standing Query Output

To register SNS as a Standing Query output, we need to describe our output via the [REST API](../../reference/rest-api.md).

We'll use the topic `sns-json` with ARN `arn:aws:sns:us-west-2:507123456123:sns-json` and associate the output with an already-running Standing Query `match-usernames`. Thus, we'll make a POST request using [Create Standing Query Output: `POST /api/v2/graph/quine/standingQueries/match-usernames/outputs`](/reference/rest-api/?av=v2#/operations/create-standing-query-output) with the following body:

```json
{
  "name": "sns-output",
  "destinations": [
    {
      "type": "SNS",
      "topic": "arn:aws:sns:us-west-2:507123456123:sns-json",
      "credentials": {
        "accessKeyId": "AKIAMYACCESSKEY",
        "secretAccessKey": "AWSScRtACCessKeyAWS/ScRtACCessKey"
      },
      "region": "us-west-2"
    }
  ]
}
```

Note that as with all Standing Query outputs, this output could be registered on a new Standing Query or an already-running Standing Query.


---
# Standard In
URL: https://quine.io/learn/ingest-sources/stdin/

# Standard In

Quine fully supports reading from Standard In. Together with writing [Standing Queries to Standard Out](../../learn/standing-queries/standing-queries.md), Quine is a powerful tool for any command-line data processing task.

The following is a simple [Recipe](../recipe-ref-manual.md) that ingests each line of input from Standard In as a node in the graph. It also uses a Standing Query to write every to Standard Out:

```yaml
--8<-- "recipes/assets/pipe.yaml"
```

To run this Recipe, pipe data from other program into Quine. This example uses the Unix `find` program as a data source:

```
❯ java -jar quine-2.1.1.jar -r pipe.yaml
Graph is ready
Running Recipe Pipe
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
2022-02-22 15:33:21,995 Standing query `output-1` match: {"meta":{"isPositiveMatch":true,"resultId":"54754813-574b-b86e-5e8e-6968ef4ce2e5"},"data":{"line":"/dev/ptyu1"}}
2022-02-22 15:33:21,996 Standing query `output-1` match: {"meta":{"isPositiveMatch":true,"resultId":"e2783bf4-3328-1365-0fed-d9a9ab12489c"},"data":{"line":"/dev/ptyu4"}}
2022-02-22 15:33:21,997 Standing query `output-1` match: {"meta":{"isPositiveMatch":true,"resultId":"845cac7b-7994-baf8-982f-06adc4a818ff"},"data":{"line":"/dev/ptytf"}}
2022-02-22 15:33:21,998 Standing query `output-1` match: {"meta":{"isPositiveMatch":true,"resultId":"ccdb9fc6-969e-16a0-2a10-8a0e2b594ec1"},"data":{"line":"/dev/ttyu8"}}
...

Quine app web server available at http://localhost:8080
INGEST-1 status is completed and ingested 360

 | => STANDING-1 count 360
```


---
# Metrics
URL: https://quine.io/learn/metrics/

# Metrics

| Page                                                                                     | Description                                                |
|:-----------------------------------------------------------------------------------------|:-----------------------------------------------------------|
| [**Metrics Quick Start**](./quick-start.md)                                              | View metrics via REST API or JMX without external tooling  |
| [**Collected Metrics**](./metrics.md)                                                    | Reference of all metrics collected by Quine      |
| [**Recommended Alerts**](./recommended-alerts.md)                                        | Production alert recommendations based on emitted metrics   |
| [**Grafana + InfluxDB**](./influx-grafana.md)                                            | Set up monitoring dashboards locally using Grafana         |


---
# Grafana + InfluxDB
URL: https://quine.io/learn/metrics/influx-grafana/

# Grafana + InfluxDB

## Monitoring Data in Motion

There has been a significant increase in the popularity of event streaming and stream processing applications/technologies within the data engineering community. With the accelerating growth of big data, IoT, and cloud computing, more organizations are facing the challenge of extracting actionable insights earlier in the event pipeline. For historical reasons, operational tools for monitoring, alerting, and diagnosing system issues are oriented toward data at rest. That doesn’t mean they can’t be just as useful for monitoring data in motion. It just means adjusting your monitoring regime to a streaming mindset.

A good example of a next-gen streaming infrastructure element is Quine. Quine is a event streaming technology designed to process graph-shaped event streams and produce high-value events in real time.

In this guide, we'll guide you through setting up Grafana backed by InfluxDB to monitor a Quine instance. We'll show you how to configure Quine to send data to InfluxDB, create a dashboard in Grafana to visualize this data, and use Grafana's powerful features to detect issues and anomalies in real time. By the end of this guide, you'll have a solid understanding of how to monitor event stream pipelines using Grafana and InfluxDB, and you'll be equipped with the tools and knowledge needed to keep Quine running smoothly.

## Setting up Grafana and InfluxDB

[Grafana](https://grafana.com/) is a tool that helps you visualize and understand operational metrics data. It lets you create visual dashboards to monitor and analyze data from sources across your data infrastructure. DevOps teams use Grafana metrics dashboards to make informed decisions.

![Observability Stack](./observability-stack.png)

Above is an example of a typical development and testing environment when working on a [recipe](https://quine.io/recipes). The event sources and output sinks change depending on the scenario, but typically Quine runs on localhost, configured to push metrics to InfluxDB and visualize the observations in Grafana. Using Docker containers makes it easy to configure and clean up the environment quickly.

Some pre-work is needed before launching the Docker containers. The following example uses **docker-compose** to set up the environment. Your configuration may differ based on how Docker is installed on your host.

A recommended approach is to keep **docker-compose.yaml** files arranged inside their directories in a **docker** directory in **$HOME**. This helps keep things organized and makes sharing configs between machines easy.

A [**zip file**](https://quine-recipe-public.s3.us-west-2.amazonaws.com/quine-grafana-docker.zip) containing the configuration is available to download and use with this guide.

```bash
cd $HOME
wget https://quine-recipe-public.s3.us-west-2.amazonaws.com/quine-grafana-docker.zip
unzip quine-grafana-docker.zip
```

!!! note

    The zip archive includes a docker-compose file for Cassandra. Cassandra configuration is not covered in this guide, but the file is included as a reference if you choose to separate persistent storage from the application to avoid competing for server resources. See the [Cassandra Persistor](../persistors/cassandra-setup.md) docs for a sample configuration file.

You now have this directory structure in your **$HOME** dir.

```
docker
├── cassandra
│   └── docker-compose.yaml
└── grafana
    ├── docker-compose.yaml
    └── grafana-provisioning
        ├── dashboards
        │   ├── dashboard.yaml
        │   └── quine.json
        └── datasources
            └── datasource.yml
```

With Docker configured and the **quine-docker.zip** files loaded on the virtualization host, start the containers so that they are ready to receive data from Quine.

Change into the **grafana** directory and start the InfluxDB/Grafana stack:

```bash
docker compose up -d
```

Verify that the containers are running:

```bash
docker ps
NAMES                 STATUS           PORTS
grafana-grafana-1     Up 4 seconds     0.0.0.0:3000->3000/tcp
grafana-influxdb-1    Up 4 seconds     0.0.0.0:8086->8086/tcp
```

InfluxDB and Grafana are now running in separate containers and listening on their default ports.

## Configuring Quine to Send Metrics Data

Enable metrics reporting in Quine via configuration parameters that can be passed as Java system properties with -D or contained in a [Quine configuration file](../../reference/config/configuration.md). Quine can report metrics to jmx, csv, influxdb, and slf4j for analysis. The jmx metrics reporter is enabled by default.

```bash
java \
    -Xmx12G -Xms12G \
    -Dquine.metrics-reporters.1.type=influxdb \
    -Dquine.metrics-reporters.1.database=db0 \
    -Dquine.metrics-reporters.1.period=30s \
    -Dquine.metrics-reporters.1.host={container_host} \
    -jar quine-2.1.1.jar \
    -r wikipedia --force-config
```

A couple of things to note when passing configuration as system properties.

- The **-D** parameters must come before **-jar**
- When launching Quine with a recipe (**-r**) you also have to pass **--force-config**

Alternatively, you can pass the following configuration stored in **quine-metrics.conf** to Quine to accomplish the same thing.

Create a **quine-metrics.conf** file containing the HOCON configuration from the [documentation](../../reference/config/configuration.md).

```kconfig
quine {
  # where metrics collected by the application should be reported
  metrics-reporters = [
    {
      # Report metrics to an influxdb (version 1) database
      type = influxdb

      # required by influxdb - the interval at which new records will
      # be written to the database
      period = 30

      # Connection information for the influxdb database
      database = db0
      scheme = http
      host = {container_host}
      port = 8086

      # Authentication information for the influxdb database. Both
      # fields may be omitted
      # user = admin
      # password = admin
    }
  ]
}
```

!!! important
    Make sure to change out the `container_host` value for the actual container host value (like `localhost` for example)

Then launch Quine, passing the configuration file on the command line.

```bash
java -Dconfig.file=metrics.conf -jar quine-2.1.1.jar -r wikipedia --force-config
```

## Quine Metrics

Quine reports three classes of metrics; counters, timers, and gauges.

!!! tip
    When queried, the [Metrics: `GET /api/v2/system/metrics`](/reference/rest-api/?av=v2#/operations/get-metrics) API endpoint reports the same metrics as a metrics reporter.

### Counters

Quine uses counters to accumulate the number of times that events occur. Counters can return either a value or a histogram.

- **quine.node.edge-counts.***: Histogram-style summaries of edges per node
- **quine.node.property-counts.***: Histogram-style summaries of properties per node
- **quine.shard.shard-{n}.sleep-counters.***: Count the lifecycle state of nodes managed by a shard

### Timers

Quine reports the elapsed time in milliseconds it takes to perform persistor operations.

- **persistor.get-journal**: Time taken to read and deserialize a single node’s relevant journal
- **persistor.persist-event**: Time taken to serialize and persist one message’s worth of on-node events
- **persistor.get-latest-snapshot**: Time taken to read (but not deserialize) a single node snapshot

### Gauges

Quine gauges report metrics as a value.

- **memory.heap.***: JVM heap usage
- **memory.total**: JVM combined memory usage
- **shared.valve.ingest**: Number of current requests to slow ingest for another part of Quine to catch up
- **dgn-reg.count**: Number of in-memory registered DomainGraphNodes

## Create a Dashboard in Grafana

A dashboard in Grafana contains a series of panels that provide an at-a-glance view of how Quine is performing.

- Log into Grafana. The username and password for the container is admin:admin.
- Decide if you are going to keep the default password or skip changing it

If you launched Grafana using the **docker-compose** files from the **quine-docker.zip** file provided above, a dashboard called "Quine – Monitor a Recipe" will appear in the lower left hand corner of the Dashboards card. Click on that dashboard to open it. Initially, the dashboard will be empty. It will fill in as you run a recipe.

Let's start Quine with the Wikipedia recipe and the **metrics.conf** file from above to get familiar with each visualization.

```bash
java -Dconfig.file=metrics.conf -jar quine-2.1.1.jar -r wikipedia --force-config
```

Metrics will populate the dashboard after about 30 seconds once Quine is running. You may need to reload your browser to have Grafana pull all of the metrics from InfluxDB. Also, be sure to set the time range in the upper right corner of the dashboard to "Last 15 minutes" to ensure that you have a current time range selected to visualize.

Your dashboard will begin to populate like this:

![Grafana Dashboard](./grafana-dashboard.png)

A Grafana dashboard view for Quine running the Wikipedia ingest recipe.

Hover over each graph in the dashboard to expose a "three-dot" menu in the upper right hand corner of the panel. Click on the menu and select "edit" to review how each visualization is configured. Some visualizations use the query builder, and some are written directly as an InfluxDB query.

Please modify the dashboard to match your environment and satisfy your needs.

## Monitoring Best Practices

Monitoring a streaming graph is similar to any other database, with a few additional key metrics to watch.

Quine is backpressured, which means that the performance of the persistence subsystem affects the flow of events in the graph.
Java garbage collection impacts backpressure. It is normal for Quine ingest rates to fluctuate as Java manages the heap. Keep an eye on when heap consumption approaches the max memory configured for Java. Best performance is typically achieved when launching Quine with a 12G (**-Xmx12G -Xms12G**) memory allocation pool.

## Conclusion

The metrics dashboard built into the Exploration UI is good for understanding how Quine is currently operating. However, monitoring the performance of a recipe or solution over time requires a DevOps tool like Grafana. This guide will get you up and running with a sample dashboard that replicates all of the gauges in the Exploration UI that you can modify to suit your needs.


---
# Collected Metrics
URL: https://quine.io/learn/metrics/metrics/

# Collected Metrics

!!! tip "Upgrading from a previous version?"
    If you are updating dashboards or alerts after an upgrade, see [Upgrading](../../reference/upgrade/quine-2.0.0.md#metrics-prefix-changes) for migration steps and examples.

We expose a large number of JVM and application metrics via the [DropWizard Metrics library](https://metrics.dropwizard.io/4.2.0/manual/index.html).

They can be exported by periodically writing as CSV files, logging, to InfluxDB, and/or via JMX. By default only the JMX reporter is enabled. See the comments on the `metrics-reporters` setting in the [Config Ref Manual](../../reference/config/configuration.md) for how to enable / configure the others - i.e. the part on `one of [jmx, csv, influxdb, slf4j]`. Some metrics are also exposed in JSON on the HTTP endpoint [Metrics: `GET /api/v2/system/metrics`](/reference/rest-api/?av=v2#/operations/get-metrics).

## Available Metrics

Each metric reports the standard statistics for its type:

| Type | Reports |
|:----------|:-------------------------------------------------------------------------------|
| Counter   | A running `Count`.                                                             |
| Gauge     | A single instantaneous value.                                                 |
| Meter     | A `Count`, a mean rate, and 1-, 5-, and 15-minute rates (the one-minute rate is the `m1_rate` field in the JSON output). |
| Histogram | Min, mean, max, standard deviation, and percentiles: the 50th, 75th, 95th, 98th, 99th, and 99.9th. |
| Timer     | The percentiles of a histogram (as durations) plus the rates of a meter.       |

For example, the 95th-percentile latency of `persistor.persist-event` and the one-minute ingest rate of `quine.ingest.{name}.count` are both available.

The metrics that we explicitly measure in our code are as follows.

- **quine**
    - **shard.shard-{n}**
        - **sleep-counters**: Counters that track the sleep cycle (in aggregate) of nodes on the shard
            - **removed**
            - **slept-failure**
            - **slept-success**
            - **woken**
        - **sleep-timers**: Timers that measure the duration of sleep and wake operations on nodes
            - **slept**
            - **woken**
        - **nodes-evicted**: Meter tracking node evictions from memory (only emitted when `enableDebugMetrics` is set)
        - **unlikely**: Counters that track occurrences of supposedly unlikely (and generally bad) code paths
            - **wake-up-failed**: Despite repeated attempts, we cannot wakeup the requested node.
            - **wake-up-error**: An unexpected error was encountered when attempting to wake up a node; will retry.
            - **hard-limit-reached**: A node was blocked from being woken up because the hard limit for number of active nodes has been hit; will retry.
            - **actor-name-reserved**
            - **incomplete-shutdown**: A shard did not complete shutdown cleanly.
    - **node**: Bucketed counters
        - **edge-counts**: A counter for the numbers of edges on nodes, split into buckets
            - 1-7
            - 8-127
            - 128-2047
            - 2048-16383
            - 16384-infinity
        - **property-counts**: A counter for the numbers of properties on nodes, split into buckets
            - 1-7
            - 8-127
            - 128-2047
            - 2048-16383
            - 16384-infinity
        - **property-sizes**: A histogram of property sizes (in bytes) observed since startup
    - **ingest.{ingest-name}**
        - **count**: Number of records ingested
        - **bytes**: Number of bytes ingested (aggregate data payload size)
        - **query**: Timer measuring the duration of ingest query executions
        - **deserialization**: Timer measuring the duration of ingest record deserialization
    - **standing-queries**
        - **results.{standing-query-name}**: Meter of results that were produced for a named standing query on this member
        - **dropped.{standing-query-name}**: Counter of results that were dropped for a named standing query on this member due to an excess of messages already in-flight when the standing query backpressures. This should be zero.
        - **states.{standing-query-id}**: Histogram of the size (in bytes) of persistent standing query states.
        - **queue-time.{standing-query-name}**: Timer measuring how long SQ results spend in the result queue before being accepted for processing
- **persistor**: All are timers, except snapshot-sizes, which is a histogram.
    - **get-journal**: Measures how long it takes to query a node's journal from the persistor
    - **get-latest-snapshot**: Measures how long it takes to retrieve a node's snapshot from the persistor
    - **persist-event**: Measures how long it takes to persist a change to a node's state.
    - **persist-snapshot**: Measures how long it takes to persist a node's snapshot.
    - **set-standing-query-state**: Measures how long it takes to persist standing query state.
    - **get-standing-query-states**: Measures how long it takes to retrieve standing query states.
    - **snapshot-sizes**: A histogram that measures the serialized size (in bytes) of a node's persisted snapshot.
- **shard.shard-{n}**
    - **delivery-relay-deduplicated**: Counter of deduplicated message deliveries on this shard.
- **shared**
    - **valve.{name}**: A gauge representing how many operations are currently pausing an ingest due to backpressuring.
- **cache**
    - **{context}.insert**: Timer tracking insert operations into internal caches (e.g. `ingest-XYZ-deduplication`, `http-webpage-serve`).
- **node**
    - **mailbox-sizes**: A counter for the sizes of message mailboxes on nodes, split into buckets
        - 1-7
        - 8-127
        - 128-2047
        - 2048-16383
        - 16384-infinity
- **dgn-reg**
    - **count**: Gauge measuring the number of in-memory registered DomainGraphNodes.

### JVM Metrics

In addition to the application metrics above, standard JVM metrics are exported. These are not prefixed by the graph name.

- **memory**: JVM memory usage gauges
    - **heap.used**, **heap.max**, **heap.committed**, **heap.usage** (`heap.usage` is used divided by max)
    - **non-heap.used**, **non-heap.committed** (and the other `non-heap.*` gauges)
    - **total.used**, **total.max** (combined heap + non-heap)
- **gc**: Garbage-collector metrics, per collector (e.g. `count`, `time`)
- **buffers**: Direct and mapped buffer-pool usage

Other libraries we use also export metrics via this mechanism - e.g. the Cassandra client reports metrics relating to the usage of the Cassandra server, which can optionally be enabled in your config file: [https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/metrics/#enabling-specific-driver-metrics](https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/metrics/#enabling-specific-driver-metrics).


---
# Recommended Alerts
URL: https://quine.io/learn/metrics/recommended-alerts/

# Recommended Alerts

Recommended production alerts for Quine, expressed against the metrics it emits — so
they work with any dashboard or alerting backend. Each gives you **what to watch**, **warning**
and **critical** levels, and **why**.

Read the values from the [Metrics: `GET /api/v2/system/metrics`](/reference/rest-api/?av=v2#/operations/get-metrics)
endpoint, via JMX, or from your configured metrics reporter; metric names come from
[Collected Metrics](./metrics.md). The sections below follow the data path: ingest,
standing queries, persistor, graph, and host.

!!! tip "Three rules for every alert below"
    1. **Alert on _sustained_ conditions, not spikes.** Use a few-minute "for"/pending window
       so a GC pause or latency blip doesn't page.
    2. **Tune the numbers to your baseline.** These are starting points — adjust after watching
       steady state for a few days.
    3. **Poll volatile gauges directly.** Fast gauges like `shared.valve.ingest` can flap
       between scrapes; read them from the endpoint or JMX, not a slow-poll reporter.

When an alert fires and you need to find the underlying cause, see
[Diagnosing Bottlenecks](../troubleshooting/diagnosing-bottlenecks.md), which explains how to
read each metric and trace a symptom to its root cause.

!!! note "Metric names"
    Names below are written as they appear at the metrics endpoint. Metrics that are scoped to
    a graph are prefixed with its name (shown here as
    `quine`). Your metrics reporter may rewrite the `.` and `-` separators
    (for example to `_`). Some signals are emitted only as **log messages** rather than
    metrics; those are called out in the relevant section and marked as logs in the summary.

## Ingest Streams

### Ingest rate per stream

**What to watch:** `quine.ingest.{name}.count` (one-minute rate).

**Why it matters:** This is the live records-per-second rate for each stream. Two independent
failure modes are worth alerting on:

- **Critical — stalled stream:** rate `== 0` for ≥ 5 minutes on a stream that should be
  active. This catches a dead or stuck ingest immediately and needs no baseline.
- **Warning — degraded throughput:** rate sustained below a chosen fraction (for example,
  **50%**) of the stream's normal steady-state. Set this per stream from your observed
  historical rate — a stream that normally runs ~350/min dropping to ~150/min is worth a look
  even though it isn't zero.

The rate is an exponentially weighted moving average, so it is volatile at the start and end
of a stream; allow ~10 minutes for it to settle before drawing conclusions.

!!! warning "Rate alone does not catch every data problem"
    A mis-specified `idFrom` or an ingest race can *silently* produce missing or malformed data
    with no error and no drop in ingest rate. Pair these rate alerts with a node-count or
    output-volume baseline if data completeness matters.

## Standing Queries

### Standing query backpressure

**What to watch:** `shared.valve.ingest.{name}` (gauge, one per ingest stream).

**Why it matters:** This gauge reports how many standing queries are currently pausing an
ingest because the standing query result queue is filling up faster than results can be
processed. `0` means the valve is open (healthy). A non-zero value means Quine is
applying backpressure to protect itself — the ingest is waiting for standing query work to
catch up. That is the system working as designed, but a valve that stays closed means
something downstream of the match (the output query or its destination) can't keep up. Note
this is *not* data loss on its own — data is only lost once dropped results appear (see below).

- **Warning:** sustained non-zero on a stream that should be flowing freely.

### Dropped standing query results

**What to watch:** `quine.standing-queries.dropped.{name}` (counter).
As a leading indicator, also watch `quine.standing-queries.queue-time.{name}`
(timer) — a rising queue time means an output is getting slow *before* it starts dropping.

**Why it matters:** This counter records standing query results that were **irrecoverably
dropped**. Each drop is also accompanied by a WARN log explaining why. The backpressure valve
normally prevents the result queue from overflowing, so any sustained increase here means real
data loss.

- **Critical:** any sustained increase (the counter should stay flat).

## Persistor

### Persistor latency

**What to watch:** the persistor timers, weighting the **write path** most heavily because
that is where back-pressure first shows up and propagates back into ingest:

- `persistor.persist-event`, `persistor.persist-snapshot` — write path
- `persistor.get-journal`, `persistor.get-latest-snapshot` — read path

**Why it matters:** These measure how long persistence operations take. Watch both the average
and the 95th percentile — single-digit-millisecond p95 is healthy. As latency climbs into the
tens of milliseconds and beyond, the persistor becomes the bottleneck and back-pressure
propagates back into the ingest streams.

- **Warning:** p95 sustained **> 50 ms** (well above a < 10 ms healthy baseline).
- **Critical:** p95 sustained **> 100 ms**.

The *shape* of the latency tells you *why* the persistor is slow: a high average points to a
general persistor bottleneck, while a high p95 with a low average points to occasional slow
operations, often a supernode. See
[Diagnosing Bottlenecks](../troubleshooting/diagnosing-bottlenecks.md#persistor-latency) for
more on reading these, and — if your persistor is Cassandra — for the driver-side latency
metric (`s{n}.cql-requests`).

**Log signal — `Query timed out after PT2S` / `DriverTimeoutException`:** if your persistor is
Cassandra, this means a request hit the server-side timeout (default 2 s). If these correlate
with Cassandra GC events in its `gc.log`, the Cassandra JVM is pausing — consider ScyllaDB,
which has no GC pauses. **Severity: critical on recurring timeouts** (a lone timeout under heavy load can be transient).

## Graph Health

### Supernode edge counts

**What to watch:** the upper buckets of the edge-count histogram —
`quine.node.edge-counts.2048-16383` and
`quine.node.edge-counts.16384-infinity` (counters).

**Why it matters:** This histogram counts how many in-memory nodes fall into each edge-count
bucket — it is your supernode detector. Supernodes (nodes with very high edge counts) are
expensive: they are slow to wake, sleep, and snapshot, they increase persistor load, and they
serialize traversals through a single hot node.

- **Warning:** the `2048-16383` bucket becomes non-zero and stays populated — nodes with
  thousands of edges are accumulating.
- **Critical:** the `16384-infinity` bucket becomes non-zero — a live supernode with tens of
  thousands of edges.

!!! warning "This metric only sees awake nodes"
    The edge-count histogram counts only nodes that are **currently in memory** — and some
    failures never increment a counter at all — so pair it with the log signal below for
    durable detection.

**Log signal — `Node <id> has: <N> edges`:** emitted every 10,000 edges on a single node.
Because it fires regardless of whether the node is awake at scrape time, it catches a supernode
the histogram can miss. **Severity: warning, escalating as `N` grows.**

### Critical-node mailboxes

**What to watch:** the upper buckets of `node.mailbox-sizes` (counters).

**Why it matters:** This histogram counts how many in-memory node mailboxes hold each number
of queued messages. When the higher buckets populate, some nodes have become **critical
nodes** — they are receiving more work than they can keep up with. This is often driven by
supernodes, but can also point to a data-modeling or topology issue.

- **Warning:** the upper buckets become non-zero and stay populated.

### Oversized properties

**What to watch:** the top bucket of `quine.node.property-sizes` (histogram).

**Why it matters:** This tracks the serialized size of node properties. If the largest bucket
populates, some properties are extremely large and may approach the **1 MB** single-value size
guideline imposed by Cassandra — large values are slow to persist and can fail outright.

- **Warning:** the top bucket becomes non-zero.

## Host Metrics

### Heap memory usage

**What to watch:** the ratio of `memory.heap.used` to `memory.heap.max`, evaluated over a
moving average rather than instantaneously.

**Why it matters:** The JVM heap normally saw-tooths as garbage collection runs, so an
instantaneous reading near the top is often just pre-collection. Alert on *sustained* pressure
using a moving average to avoid false pages.

- **Warning:** heap used / max sustained **> 80%**.
- **Critical:** sustained **> 90%**.

!!! tip "Heap sizing context"
    A static heap of **12 GB** is recommended (**16 GB** maximum). If the JVM logs frequent long GC pauses,
    the heap is likely configured *too large*. Out-of-memory or OOM-killed errors indicate the
    opposite — too little memory for the instance, or an `in-memory-soft-node-limit` set too
    high. See [Operational Considerations](../../core-concepts/operational-considerations.md)
    for resource planning.

### Awake nodes versus capacity

**What to watch:** Quine keeps "hot" nodes in memory and sleeps the rest. The number
of nodes currently awake is not a single metric — it is derived from the per-shard sleep
counters using a conservation identity:

```
awake ≈ Σ over shards ( woken − slept-success − slept-failure − removed )
```

using `quine.shard.{shard}.sleep-counters.woken`, `.slept-success`,
`.slept-failure`, and `.removed`. (All four exit paths must be subtracted — subtracting only
some of them makes the figure grow without bound.)

**Why it matters:** This tracks how full Quine's in-memory capacity is. As awake
nodes approach capacity, node-sleeping can no longer keep pace and memory pressure builds.
Capacity is a function of your topology and configured node limits:

```
capacity_soft = shard-count × in-memory-soft-node-limit
capacity_hard = shard-count × in-memory-hard-node-limit
```

where `shard-count` is the number of shards (defaults to 4) and `in-memory-soft-node-limit` /
`in-memory-hard-node-limit` are the per-shard cache limits (defaulting to 10,000 and 75,000).

All of these are described in the
[Configuration Reference](../../reference/config/configuration.md).

- **Warning:** awake nodes sustained **above `capacity_soft`** — node-sleeping is no longer
  keeping up and memory pressure is building.
- **Critical:** awake nodes approaching **`capacity_hard`** — near the point where
  Quine applies back-pressure on waking new nodes.

A related early warning is rapid growth in
`quine.shard.{shard}.unlikely.incomplete-shutdown`: it means nodes are being
contacted just as they decide to sleep, wasting time serializing and persisting extra
snapshots (cache thrash).

## Summary

| Signal | Type | What to watch | Warning | Critical |
|:-------|:-----|:--------------|:--------|:---------|
| Ingest rate per stream | metric | `quine.ingest.{name}.count` | < ~50% of baseline | `== 0` for ≥ 5 min (active stream) |
| Standing query backpressure | metric | `shared.valve.ingest.{name}` | sustained non-zero | — |
| Dropped SQ results | metric | `quine.standing-queries.dropped.{name}` | — | any sustained increase |
| Persistor latency (p95) | metric | `persistor.persist-event` / `persist-snapshot` | > 50 ms sustained | > 100 ms sustained |
| Cassandra timeout | log | `Query timed out after PT2S` | — | recurring |
| Supernode edge counts | metric | `quine.node.edge-counts.*` | `2048-16383` populated | `16384-infinity` non-zero |
| Supernode (durable) | log | `Node <id> has: <N> edges` | populated | as `N` grows |
| Critical-node mailboxes | metric | `node.mailbox-sizes.*` | upper buckets populated | — |
| Oversized properties | metric | `quine.node.property-sizes` | top bucket populated | — |
| Heap memory usage | metric | `memory.heap.used` / `memory.heap.max` | > 80% sustained | > 90% sustained |
| Awake nodes vs capacity | metric | `quine.shard.{shard}.sleep-counters.*` | above `capacity_soft` | approaching `capacity_hard` |

!!! note "Know your baseline"
    The numbers above are starting points, not universal truths. Calibrate them to your own
    deployment — its steady-state ingest rate, persistor latency, heap size, and node counts.
    For broader resource-planning guidance (heap sizing, persistor-to-host ratios, scaling
    ceilings), see [Operational Considerations](../../core-concepts/operational-considerations.md).


---
# Persistors
URL: https://quine.io/learn/persistors/

# Persistors

The graph operates in memory, but saves its data to disk. Because data is durably stored, Quine does not need to define any time-windows for matching up data in memory. This data is managed automatically so that it is transparent to the operation of the graph, and saved in a way that is fast for streaming data.

Quine's primary unit of data is the graph node. A node is defined by its ID and serves as a collection of properties and edges. Changes to the collection of properties and edges are what is saved to disk. These changes, sometimes called "deltas" are very small units of data added to an append-only log using the strategy known as event-sourcing. The persistor is the agent that stores and retrieves event-sourced data.

The format of data saved on disk is conceptually a key-value pair, where the key is a node ID and the value is the append-only log of changes to the properties and edges of that node. While this is conceptually the storage format, in practice the choice of data storage medium can affect the actual format stored on disk. For instance, when using Cassandra as the backing store for the persistor, the key is a compound value of both the Node ID and a unique timestamp.

### Quine Persistence Event Configuration

Related to the persistence store configuration, the persistence section of our config has settings related to when to save data.

``` kconfig
# configuration for which data to save about nodes and when to do so
persistence {
    # whether to log updates in between writing node snapshots. Without
    # this, data may be lost in the event of a crash, and historical
    # queries will have less fine-grained history to work with, but
    # performance will be greater, and disk usage will be less.
    journal-enabled = true

    # one of [on-node-sleep, on-node-update, never]. When to save a
    # snapshot of a node's current state
    snapshot-schedule = on-node-sleep

    # whether only a single snapshot should be retained per-node. If false,
    # one snapshot will be saved at each timestamp against which a
    # historical query is made
    snapshot-singleton = false

    # when to save Standing Query partial result (only applies for the
    # `MULTIPLE_VALUES` mode -- `SingleId` Standing Queries always save when
    # a node saves a snapshot, regardless of this setting)
    standing-query-schedule = on-node-sleep

}
```

## Supported Persistors

Set with `quine.store.type`.

| Persistor | Configuration value | Description |
|:---|:---|:---|
| [RocksDB](#rocksdb) | `rocks-db` | An embedded log-structured merge tree on the local filesystem. The default, and the fastest choice for a single host. |
| [MapDB](#mapdb) | `map-db` | An embedded Java store on the local filesystem. The fallback where RocksDB has no native build for the host architecture. |
| [Apache Cassandra](cassandra-setup.md) | `cassandra` | A distributed database giving high throughput, replication, and failover. |
| [ScyllaDB](cassandra-setup.md) | `cassandra` | A Cassandra-compatible database, connected through the Cassandra persistor. |
| [Astra DB](cassandra-setup.md#astradb-configuration) | `cassandra` | DataStax's serverless Cassandra-compatible service. Needs an application token and a secure connect bundle. |
| [Amazon Keyspaces](cassandra-setup.md#amazon-keyspaces-configuration) | `keyspaces` | AWS's managed Cassandra-compatible service, for a distributed store without operating Cassandra yourself. |
| In-memory | `in-memory` | Holds everything in memory and writes nothing to disk. Useful for tests and short experiments; all data is lost on shutdown. |
| Empty | `empty` | Discards every write and returns nothing on read. No history, and nothing survives a restart. |

## Local Persistors

### RocksDB

[RocksDB](http://rocksdb.org) is a widely used implementation of a log-structured merge tree (LSMT). It is an ideal data store for Quine and is the default option for data storage locally.

Using RocksDB in Quine requires no special changes to use it, since it is the default. The setting that would choose RocksDB specifically is:

`quine.store.type=rocks-db`

!!! Warning

    RocksDB is distributed by its authors as a binary artifact built for specific architectures and used from JVM applications through the Java Native Interface (JNI). If you try to start Quine with the default settings on an unsupported platform, you will get an error suggesting that you run with the option to use MapDB instead: `-Dquine.store.type=map-db`.

#### Where RocksDB stores data

RocksDB stores the graph in a directory on the local filesystem. Quine resolves the location in this order:

1. The `filepath` setting, when configured:

    ```hocon
    quine.store = {
      type = rocks-db
      filepath = "/path/to/data"
    }
    ```

2. The `QUINE_DATA` environment variable, when set. The official Quine Docker image sets `QUINE_DATA=/var/quine` and declares that path as a volume, so a containerized Quine persists there by default.

3. Otherwise `quine.db`, relative to the process working directory.

In a container this means no storage configuration is needed for a working local persistor: mount storage at `/var/quine` (or point `QUINE_DATA` at your mount) and the default RocksDB store lands on it. The data then lives exactly as long as the mount does: on Kubernetes, for example, an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume gives storage with the pod's lifetime, while a PersistentVolume survives pod replacement.

### MapDB

[MapDB](https://mapdb.org) is an embedded database engine for the JVM that uses memory-mapped files. Since MapDB is written in a JVM language, it is included in Quine without any binary dependencies built for specific architectures. This makes MapDB the most portable option for a Persistor's data store.

MapDB does have some other limitations though. Memory mapped files are generally limited to 2GB in size. With MapDB, memory mapped files larger than 2GB will become very slow to use. Quine supports sharding the storage of a MapDB persistor into multiple files to work around this limitation. But even if sharded, memory mapped files will cause Quine to use off-heap memory, which in extreme circumstances can cause the process to use large amounts of RAM or lead to the operating system killing the process.

To use MapDB, set the following configuration setting:

`quine.store.type=map-db`

Unlike RocksDB, MapDB does not use the `QUINE_DATA` environment variable: when no `filepath` is configured, Quine uses a temporary file that is deleted when the process exits. Set `filepath` explicitly for MapDB data that must survive a restart.

<!--
### LMDB (Lightning database)
-->

## Remote Persistors

### Cassandra

[Apache Cassandra](https://cassandra.apache.org/) is a distributed NoSQL database. It is highly configurable and trusted by enterprise organizations around the world to manage very large amounts of data. Cassandra is an ideal data storage mechanism for a Quine persistor. Using Cassandra, Quine instances can achieve extremely high throughput, high-availability, data replication, and failover strategies needed for production operation in the enterprise.

The Cassandra persistence config also connects Quine to Cassandra compatible solutions like AstraDB and ScyllaDB. See the [Cassandra Setup](cassandra-setup.md) page for details on setting up and using Cassandra with Quine.

## Migration

Each version of Quine is associated with a persistence version. The persistence version identifies the conventions used to map the graph to the model of underlying persistor. Before it can become operational, Quine needs to modify the persisted data to match the conventions of the persistence version it will be using.

When possible, this will be done automatically. If manual intervention is required, a message will be logged describing what must be done to enable migrating persistence versions.

Persistence versions can only move from a lower version to a higher one. If there is more than one version between what is stored and what needs to be used by the running Quine instance, multiple migrations may be run in succession. The app version may advance without changing the persistence version.

## Backup and Export

Backup and export are delegated to the tools of the underlying persistor (e.g. [Cassandra Backups](https://cassandra.apache.org/doc/4.0/cassandra/operating/backups.html)).

Shutting down before backing up is a simple way to ensure consistency, at the cost of downtime

!!! tip "Quine Enterprise"

    Quine Enterprise adds operational controls and high availability clustering around Cassandra‑backed deployments. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).


---
# Cassandra Persistor
URL: https://quine.io/learn/persistors/cassandra-setup/

# Cassandra Persistor

The Cassandra persistor connects Quine to Cassandra and Cassandra compatible solutions like ScyllaDB, Astra DB, and Amazon Keyspaces.

## Quine Configuration

To use Cassandra as the persistence backend for Quine, you'll need to set the `quine.store` section to `type = cassandra` in the config.

``` kconfig
--8<-- "generated/quine/documented_cassandra_config.conf"
```

Where `endpoints` is a list of the address(es) of one or more Cassandra hosts in the cluster. If you need to specify a port other than 9042 (the default), you can use `host:portNum`.

Alternatively, you may specify the environment variable `CASSANDRA_ENDPOINTS` as a comma-separated list of hostnames, or host:ports, to be used if `endpoints` is not set in the config file.

## Cassandra Authentication

Quine communicates with Cassandra via the [DataStax Java Driver for Apache Cassandra](https://mvnrepository.com/artifact/com.datastax.oss/java-driver-core/4.17.0). You can configure authentication by adding `datastax-java-driver` configuration to your local config file file as described on the driver's [Authentication page](https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/authentication/).

For example, adding the following into a `quine.conf` file will set up basic authentication.

``` kconfig
quine.store {
  type = cassandra
}
datastax-java-driver {
  advanced {
    auth-provider {
      class = PlainTextAuthProvider
      username = user
      password = pass
    }
  }
}
```

Then launch Quine with the following command line:

```shell
java -Dconfig.file=quine.conf -jar quine.jar
```

## Automatic Creation of Keyspace and Tables

Quine has settings in the Cassandra section of the config, `should-create-keyspace` and `should-create-tables`. When enabled, Quine automatically creates the keyspace and/or tables at startup if they don't already exist.

!!! note
    Auto creation of the keyspace and tables is included as a development convenience and should never be used in production.

## Cassandra Schema

``` { .sql }
CREATE KEYSPACE quine WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};
USE quine;

CREATE TABLE domain_graph_nodes (
    dgn_id bigint PRIMARY KEY,
    data blob
);

CREATE TABLE domain_index_events (
    quine_id blob,
    timestamp bigint,
    data blob,
    dgn_id bigint,
    PRIMARY KEY (quine_id, timestamp)
) WITH CLUSTERING ORDER BY (timestamp ASC)
    AND compaction = {'class': 'TimeWindowCompactionStrategy'};

CREATE TABLE journals (
    quine_id blob,
    timestamp bigint,
    data blob,
    PRIMARY KEY (quine_id, timestamp)
) WITH CLUSTERING ORDER BY (timestamp ASC)
    AND compaction = {'class': 'TimeWindowCompactionStrategy'};

CREATE TABLE meta_data (
    key text PRIMARY KEY,
    value blob
);

CREATE TABLE snapshots (
    quine_id blob,
    timestamp bigint,
    multipart_index int,
    data blob,
    multipart_count int,
    PRIMARY KEY (quine_id, timestamp, multipart_index)
) WITH CLUSTERING ORDER BY (timestamp DESC, multipart_index ASC);

CREATE TABLE standing_queries (
    query_id uuid PRIMARY KEY,
    queries blob
);

CREATE TABLE standing_query_states (
    quine_id blob,
    standing_query_id uuid,
    standing_query_part_id uuid,
    data blob,
    PRIMARY KEY (quine_id, standing_query_id, standing_query_part_id)
) WITH CLUSTERING ORDER BY (standing_query_id ASC, standing_query_part_id ASC);

```

## AstraDB Configuration

Astra DB is a fully Cassandra compatible and serverless DbaaS that simplifies the development and deployment of high-growth applications.

It doesn't support setting compaction strategies other than `UnifiedCompactionStrategy`, so you'll need to remove the `AND compaction = {'class': 'TimeWindowCompactionStrategy'}` from the above schema.
It also requires setting a token from Astra DB, as well as the path to a secure connect bundle .zip file from them.

An example config for this is as follows:

``` kconfig
quine.store {
  # store data in Cassandra
  type = cassandra
  # the keyspace to use
  keyspace = quine
  should-create-keyspace = false
  replication-factor = 3
  local-datacenter = ${ASTRA_DB_REGION}
}
datastax-java-driver {
  advanced {
    auth-provider {
      class = PlainTextAuthProvider
      username = token
      password = "${ASTRA_DB_APP_TOKEN}"
    }
  }
  basic {
    cloud {
      secure-connect-bundle = "${SECURE-CONNECT-BUNDLE}.zip"
    }
  }
}
```

### Astra DB Relevant Settings

`type = cassandra` - Use the Cassandra persistor to connect to AstraDB

`should-create-keyspace = false` - Keyspaces can only be created in Astra via the dashboard.

`replication-factor = 3` - Defaults to 1 if not set.

`write-consistency = LOCAL_QUORUM` - Minimum consistency level required by Astra.

`read-consistency = LOCAL_QUORUM` - Any level is supported, though with replication-factor=3 and QUORUM writes, you'll want reads to be QUORUM as well if you want them to be immediately reflect writes.

`local-datacenter = "us-east1"` - Set your Astra DB cloud region as the local DC.

`username = "token"` - Leave it as the literal word "token."

`password` - A valid token for an Astra DB cluster.

`secure-connect-bundle` - A valid, local file location of a downloaded Astra secure connect bundle. The driver gets the Astra DB hostname from the secure bundle, so there is no need to specify endpoints separately.

## Amazon Keyspaces Configuration

Keyspaces is a service provided by AWS that is mostly compatible with Cassandra. It has a higher latency for responding to queries, which translates into higher latencies for many Quine operations. Because it doesn't support everything Cassandra does, and its configuration is a bit different, we have a different persistor type for it: `keyspaces`. At a minimum, setting `quine.store.type = keyspaces` should be all you need to use it.

The rest of the config options are the same as that for Cassandra, with the following differences:
#### Removals

`replication-factor` - Fixed to 3, not modifiable.

`write-consistency` - Fixed to `LOCAL_QUORUM`, not modifiable.

`local-datacenter` - Set to the same value as the AWS region.

#### Additions

`aws-region` - The AWS region to connect to Keyspaces in (e.g. `us-west-2`). If unspecified, the region sourced from one of the mechanisms on the AWS SDK's [DefaultAwsRegionProviderChain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html).

`aws-role-arn` - The ARN of an IAM role to assume. It must have read / write access to the desired keyspace, as well as `select` access to the system keyspaces `system`, `system_schema`, and `system_schema_mcs`.

AWS credentials are required, which should provided in the environment using one of the mechanisms on https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html

The full default config for Keyspaces is as follows:

``` kconfig
--8<-- "generated/quine/documented_keyspaces_config.conf"
```

## Data Expiration

The amount of data stored over time can be limited by setting the time to live on the tables used by Quine.

For example, this would set the time to live to one week using the cqlsh tool, assuming the name of the keyspace Quine is using is "quine":

```sql
ALTER TABLE quine.journals WITH default_time_to_live = 604800;
```

Each of the following tables could have a TTL set:

- domain_graph_nodes
- domain_index_events
- journals
- snapshots

- standing_queries
- standing_query_states

Quine also uses a `meta_data` table, which should not have a TTL set.


---
# Standing Query Wiretap
URL: https://quine.io/learn/standing-queries/wiretap/

!!! warning "Experimental Feature"
    This feature is experimental and unstable. Its API and behavior may change without notice.

# Standing Query Wiretap

The standing query wiretap enables observing a standing query's results in real time over a WebSocket connection. A standing query wiretap can observe any of the following three stages of a standing query result: before processing begins, after transformation, or after full enrichment.

This is a diagnostic and visualization tool. The wiretap has no delivery guarantees. Results are dropped rather than buffered when your client connection is slow or the cluster is overloaded. The stream stops as soon as you close the connection.

For background on standing queries and how outputs and enrichment stages work, see [Standing Queries](standing-queries.md). The Exploration UI offers the same tap points interactively through [Standing Query Inspection](../../getting-started/exploration-ui.md#inspecting-standing-queries), no WebSocket client required.

## Tap endpoints

All three endpoints are `GET` WebSocket connections. Use `wss://` instead of `ws://` if your instance is served over TLS (HTTPS).

### Raw tap

Streams every `StandingQueryResult` as soon as the match is found, before any output workflow runs.

```
ws://{host}/api/v2/graph/{graphName}/standingQueries/{sqName}:tap
```

Use when you want to see raw match events before any enrichment is applied.

### Pre-enrichment tap

Streams results after the `preEnrichmentTransformation` stage but before the Cypher `resultEnrichment` query runs.

```
ws://{host}/api/v2/graph/{graphName}/standingQueries/{sqName}/outputs/{outputName}:tap_pre_enrichment
```

Use when you have a transformation configured and want to inspect the result after preprocessing (such as metadata stripping) but before any enrichment is applied.

### Post-enrichment tap

Streams the processed data that your configured destinations receive after all workflow stages have run.

```
ws://{host}/api/v2/graph/{graphName}/standingQueries/{sqName}/outputs/{outputName}:tap
```

Use when you want to verify exactly what is being sent to your output destinations without having to set up a separate destination to inspect it.

## Message format

Each WebSocket message is a JSON string. The shape depends on the tap stage:

- **Raw tap:** a `StandingQueryResult` object with a `metadata` field and a `data` containing the data directly returned by your standing query.
- **Pre-enrichment tap:** the result after your `preEnrichmentTransformation` has been applied
- **Post-enrichment tap:** the fully-enriched result object, identical to what destinations receive

## Example uses

These examples require a running Quine instance with a standing query that is actively producing matches. A simple example would be a number iterator ingest limited to 1 per second, and a simple query that just labels a node.

```
{
  "name": "my-ingest",
  "source": {"type": "NumberIterator"},
  "query": "MATCH (n) WHERE id(n) = idFrom($that) SET n.num = $that, n : Number",
  "parameter": "that",
  "parallelism": 1,
  "maxPerSecond": 1
}
```

To be able to utilize the output taps, you should also add a destination with an enrichment query. Here is an example standing query you could use alongside the number iterator ingest above.

```
{
  "name": "numbers-sq",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n : Number) RETURN strId(n) AS id",
    "mode": "MULTIPLE_VALUES"
  },
  "outputs": [
    {
      "name": "myoutput",
      "preEnrichmentTransformation":{ 
        "type": "InlineData"
      },
      "resultEnrichment": {
        "query":"MATCH (n) WHERE id(n) = $that.id RETURN (n.num) as num"
      },
      "destinations": [
        {
          "type": "Drop"
        }
      ]
    }
  ]
}
```

## Command line standing wire tap

You can connect to the wiretap from any tool that supports websockets. For this example, we will use websocat. If you are using a HTTPS connection you will need to use wss instead of ws.

```
websocat ws://{host}/api/v2/graph/quine/standingQueries/numbers-sq:tap
```

This will provide a list of outputs that look something like:

```
{"meta":{"isPositiveMatch":true},"data":{"id":"0a362b50-12ff-3731-8d74-8f433a3b2338"}}
{"meta":{"isPositiveMatch":true},"data":{"id":"598319c2-9054-302b-8d0e-b9296c738ec4"}}
{"meta":{"isPositiveMatch":true},"data":{"id":"598319c2-9054-302b-8d0e-b9296c738ec4"}}
```

If you do not want the `meta` field and just want the id, you can tap into an output of a standing query that has `preEnrichmentTransformation` set to `InlineData` (like our example above). To do so, tap the pre-enrichment endpoint:

```
websocat ws://{host}/api/v2/graph/quine/standingQueries/numbers-sq/outputs/myoutput:tap_pre_enrichment
```

You will see results that look like this:
```
{"id":"0a362b50-12ff-3731-8d74-8f433a3b2338"}
{"id":"598319c2-9054-302b-8d0e-b9296c738ec4"}
{"id":"598319c2-9054-302b-8d0e-b9296c738ec4"}
```

To inspect the fully enriched result, tap the post-enrichment endpoint:

```
websocat ws://{host}/api/v2/graph/quine/standingQueries/numbers-sq/outputs/myoutput:tap
```

For our standing query enrichment query, the results will look like:
```
{"num": 1 }
{"num": 2 }
{"num": 3 }
```

## Bookmarklets

A bookmarklet is a browser bookmark that runs JavaScript instead of opening a URL. To use one, create a new bookmark, paste the code as the URL, then click it while you have the Quine Exploration UI open.

The bookmarklet below connects to the raw tap for `numbers-sq` and automatically runs a graph query in the Exploration UI whenever a new match arrives.

```js
  javascript: (() => {
    // Replace localhost:8080 with the URL of your quine web UI. 
    // If the url you are connected to starts with https, you will need to use wss:// instead of ws://
    const BASE = "ws://localhost:8080/";

    // Add entries here to watch additional standing queries.
    // onMatch receives the full standing query event, a runQuery function, and an addSyntheticEdge function.
    const WATCHERS = [
      {
        // `name` is the name of the standing query to watch, which must already be registered on the server.
        // This case the raw matches of the standing query, so the shape of the data will correspond to the pattern of the standing query. 
        // In this case, the standing query ended in `RETURN strId(n) AS even1, strId(m) AS even2`, so the data has fields even1 and even2 which are the string IDs of the matched nodes.
        name: "evens",
        onMatch: (e, runQuery, addSyntheticEdge) => {
          console.log(`evens match:`, e);
          // Runs a query in the UI query bar. In this case we run a simple query to get the matched nodes back into the UI, but you could run any query here to investigate the match further.
          runQuery(`MATCH (n),(m) WHERE strId(n) = "${e.data.even1}" AND strId(m) = "${e.data.even2}" RETURN n, m `);
          // addSyntheticEdge takes two string IDs and a label, and adds a synthetic (UI-only) edge to the UI between those nodes with that label. 
          addSyntheticEdge(e.data.even1, e.data.even2, "even_successor");
        },
      },
      {
        // You can also watch a specific output of a standing query by including the full output name here.
        // This will show the final output of the standing query after any transformations, which can be useful if you don't care about the full match details and just want to see the final results.
        // For example, if you are sending the output to Kafka and want to see what will actually be sent to Kakfa.
        // If you just want to log the output without running a query or adding edges, you can omit the runQuery and addSyntheticEdge calls and just do console.log here.
        name: "odds/outputs/odds_output", 
        onMatch: (e, runQuery, addSyntheticEdge) => {
          console.log(`odds match:`, e);
          runQuery(`MATCH (n),(m) WHERE strId(n) = "${e.data.odd1}" AND strId(m) = "${e.data.odd2}" RETURN n, m `);
          addSyntheticEdge(e.data.odd1, e.data.odd2, "odd_successor");
        },
      },
    ];

    // Everything below here is boilerplate for running standing query bookmarklets; the only thing you should need to change is above.

  const runQuery = (query) => {
    const queryBox = document.querySelector(".query-input-input");
    const nativeInputValueSetter = Object.getOwnPropertyDescriptor(
      window.HTMLInputElement.prototype,
      "value"
    ).set;
    nativeInputValueSetter.call(queryBox, query);
    queryBox.dispatchEvent(new Event("input", { bubbles: true }));
    setTimeout(() => queryBox.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true })), 0);
  };

    const addSyntheticEdge = (fromId, toId, label) => {
      if (!window.network) { console.error("[bookmarklet] vis network not ready"); return; }
      const edgeSet = window.network.body.data.edges;
      const id = `synthetic-${fromId}-${label}-${toId}`;
      edgeSet.remove(id);
      edgeSet.add({ id, from: fromId, to: toId, label, dashes: true, color: "purple", smooth: true, arrows: "to" });
    };

    WATCHERS.forEach(({ name, onMatch }) => {
      const ws = new WebSocket(`${BASE}/api/v2/graph/quine/standingQueries/${name}:tap`);
      ws.onmessage = (event) => {
        const e = JSON.parse(event.data);
        if (e.meta.isPositiveMatch) {
          onMatch(e, runQuery, addSyntheticEdge);
        } else {
        }
      };
      ws.onerror = (err) => console.error(`[${name}] ws error`, err);
    });
    alert(`Now monitoring ${WATCHERS.map((w) => w.name).join(", ")} ...`);
  })();

```

Replace `base` with your Quine host (e.g. `localhost:8080`), and use `wss://` instead of `ws://` if your instance is served over HTTPS.

**How it works:**

`new WebSocket(...)` opens a persistent connection to the raw tap for `numbers-sq`. Each time the standing query finds a new match, the server sends a message and `ws.onmessage` fires.

Each message has a `meta` field (with `isPositiveMatch`) and a `data` field. The `data` field contains one key for each column your standing query returns — in our case, `id` — because the pattern query uses `RETURN strId(n) AS id`. If the query instead returned `RETURN strId(n) AS identifier, n AS number`, the data fields would be `e.data.identifier` and `e.data.number`.

`runQuery` takes a query and submits it to the Exploration UI exactly as if you had typed it into the query bar and clicked the run button. The matched node appears in the graph after the query completes.


---
# Troubleshooting
URL: https://quine.io/learn/troubleshooting/

# Troubleshooting

This section provides guides for diagnosing and resolving common problems in Quine.

| Page | Description |
| ---- | ----------- |
| [**Queries**](queries.md) | Debug ad-hoc queries, ingest queries, and standing queries |
| [**Ingest Queries**](ingest.md) | Resolve missing data, race conditions, and slow ingest rates |
| [**Diagnosing Bottlenecks**](diagnosing-bottlenecks.md) | Use metrics to identify performance bottlenecks |
| [**Query Execution Plans**](query-execution-plans.md) | Understand `EXPLAIN` output and query plan operators |
| [**Miscellaneous**](miscellaneous.md) | Understand miscellaneous warnings, errors, and behaviors |


---
# Diagnosing Bottlenecks
URL: https://quine.io/learn/troubleshooting/diagnosing-bottlenecks/

# Diagnosing Bottlenecks

Quine is a fully backpressured system. When one component can't keep up, it slows down upstream components rather than dropping data. This makes the system resilient, but it also means that a bottleneck in one area can manifest as slowness elsewhere.

This guide helps you identify where bottlenecks are occurring so you can focus optimization efforts effectively.

!!! tip "Get ahead of these"
    To catch these conditions before they become incidents, set alerts on the same metrics. See [Recommended Alerts](../metrics/recommended-alerts.md).

## Common Symptoms

| Symptom                                   | Possible Bottleneck                                  |
|:------------------------------------------|:-----------------------------------------------------|
| Low ingest rate despite available CPU     | Standing queries or persistor |
| High CPU with low ingest rate             | Inefficient queries or supernodes                    |
| Ingest rate drops periodically            | Standing query backpressure                          |
| Standing query results delayed or dropped | Output destination or result queue overflow          |

## Key Metrics for Diagnosis

### Ingest Rate

**Metrics**: `quine.ingest.{name}.count`, `quine.ingest.{name}.bytes`

The ingest rate shows how many records per second are being processed. Low ingest rates can have many causes, so use other metrics to narrow down the bottleneck.

Note: Ingest rate is reported as an exponentially weighted moving average, which can be volatile at the beginning and end of a stream. Allow the rate to stabilize for at least 10 minutes before drawing conclusions.

### Standing Query Backpressure Valve

**Metric**: `shared.valve.ingest`

This is a key diagnostic metric. When this gauge shows a non-zero value, it means ingest is being paused because the standing query result queue is filling up faster than results can be processed and delivered.

Results flow from the queue through the output query and then to the destination. The most common cause of backpressure is **output queries that need optimization**. If the output query performs expensive operations (such as additional graph traversals or lookups), it can become a bottleneck. The second most common cause is **destination performance**, including slow network connections, rate-limited APIs, or destinations at capacity.

### Persistor Latency

**Metrics**: `persistor.{query-type}` timers

These metrics track how long persistence operations take. The *shape* of the latency tells you what kind of problem you have, so watch more than one statistic:

- **avg (average)**: If high across all query types, indicates a general persistor bottleneck.
- **p95 (95th percentile)**: If p95 is high but avg is low, a specific query is slow rather than the persistor as a whole — often a supernode that is occasionally matched in a standing query.
- **p99 (99th percentile)**: A more extreme measure that can reveal a rare, problematic query that p95 and avg both miss.

When persistor latency is the bottleneck, the cause is typically either I/O bound (disk throughput) or compute bound (CPU/memory on the persistor hosts).

If your persistor is Cassandra, the Cassandra driver also reports per-request latency at `s{n}.cql-requests` (for example `s0.cql-requests`); this is a separate, lower-level view than the graph-reported `persistor.*` timers above. A healthy value is roughly **1–2 ms**, so alert on sustained multiples of that. (Driver metrics are only present when they are enabled in your configuration — see [Collected Metrics](../metrics/metrics.md).) If these client-side latencies are high but Cassandra's own server-side latencies are low, Quine and Cassandra are likely not colocated — place them in the same region/availability zone.

### Edge Count Histogram

**Metrics**: `quine.node.edge-counts.{bucket}`

High counts in the larger buckets (2048-16383 or 16384-infinity) indicate the presence of supernodes. Supernodes are not inherently problematic, but they can cause performance issues in queries that traverse them.

### Resource Utilization

Resource metrics (CPU, memory, network) must be measured externally to Quine. See [Operational Considerations](../../core-concepts/operational-considerations.md) for detailed guidance on resource planning. In general:

- **High CPU utilization** is normal and indicates good resource usage
- **Low CPU utilization** with low ingest rates suggests the bottleneck is elsewhere

## Identifying the Bottleneck

### Step 1: Check Standing Query Backpressure

Start by checking `shared.valve.ingest`. If this metric shows non-zero values, standing query outputs are causing backpressure on ingest.

**Next steps**: First, review output query complexity and optimize any expensive operations. Second, check output destination throughput and capacity.

### Step 2: Check Persistor Latency

If standing query backpressure is not the issue, check persistor latency metrics.

**High average latency across all operations**: The persistor is generally overloaded. Consider:

- Adding persistor resources
- Reviewing persistor configuration (journaling, snapshot settings)
- Checking persistor host disk I/O and CPU

**High p95/p99 but normal average**: Occasional operations are slow, often due to supernodes. Check the edge count histogram for confirmation.

### Step 3: Check Resource Utilization

If neither standing queries nor the persistor appear to be the bottleneck:

- **Low CPU on Quine hosts**: Check network throughput.
- **High CPU on Quine hosts**: Review ingest query efficiency. Queries that don't anchor by ID cause expensive all-node scans. See [Troubleshooting Queries](index.md) for detailed query debugging techniques.

### Step 4: Check for Supernodes

If the edge count histogram shows significant counts in the high buckets, supernodes may be impacting performance. Supernodes affect:

- Query performance when traversing edges
- Persistor performance when reading/writing node state
- Memory usage for caching node state

!!! tip "Quine Enterprise"
    Quine Enterprise includes supernode mitigation capabilities for production deployments. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).

## When Metrics Aren't Enough

Some conditions don't show up cleanly in the metrics above and need a different signal.

**Logs catch what metrics miss.** The edge-count and mailbox histograms only count nodes that are **awake** at scrape time — a sleeping supernode disappears from them — and some failures never increment a counter at all. These conditions surface only in the logs:

| Log message | What it indicates |
|:------------|:------------------|
| `Node <id> has: <N> edges` | A node has crossed another 10,000-edge threshold — a supernode. |
| `Query timed out after PT2S` / `DriverTimeoutException` | A Cassandra request hit the server-side timeout, often from a Cassandra GC pause. |

**Some data problems are silent.** A mis-specified `idFrom` or an ingest race can produce missing or malformed data with **no error logged and no drop in ingest rate**. If data completeness matters, compare against a node-count or output-volume baseline rather than relying on error metrics alone. See [Troubleshooting Ingest Queries](ingest.md).

## Quick Reference

| Metric                                                           | Normal         | Indicates Problem                               |
|:-----------------------------------------------------------------|:---------------|:------------------------------------------------|
| `shared.valve.ingest`                                            | 0              | Non-zero values indicate SQ output backpressure |
| `persistor.*.avg`                                                | < 10ms         | > 50ms suggests persistor bottleneck            |
| `persistor.*.p95`                                                | Similar to avg | Much higher than avg suggests supernodes        |
| `quine.node.edge-counts.16384-infinity`       | 0 or low       | High values indicate supernodes                 |
| `quine.standing-queries.dropped.{name}`       | 0              | Non-zero means results are being lost           |


---
# Ingest Queries
URL: https://quine.io/learn/troubleshooting/ingest/

# Troubleshooting Ingest Queries

## Missing Data

If the graph doesn’t appear to include known ingested data, there can be several potential causes.

### 1\. Unstable or mismatched id selection

A common pattern in Quine is to create complex graph structures from multiple disparate sources of data. To ensure that desired data relationships are created, validate that the ids from the different data sources are being handled consistently. Here’s an example.

**Example order ingest query**

```cypher
WITH $that as orderData  
MATCH (order)  
WHERE id(order) = idFrom(“order”, orderData.orderId)  
SET order:Order,  
    order.qty = orderData.quantity,  
    order.partNumber = orderData.part
```        

**Example customer data ingest query**

```cypher
WITH $that as customerData  
MATCH (customer), (order)  
WHERE id(customer) = idFrom(“customer”, customerData.customerNumber)  
      AND id(order) = idFrom(customerData.myOrder)  
SET customer:Customer,  
    customer.name = customerData.name  
CREATE (customer)-[:ordered]->(order)
``` 

In the above example, the customer ingest query doesn’t match the order ingest query. In the order ingest query, the id for orders is being prefixed with the string `order`, to help ensure that there aren’t any collisions with ids for other types of nodes. However, in the customer ingest query, the idFrom expression being used to identify the order node doesn’t utilize any such prefix.

The outcome of the above misconfiguration would be that all orders wouldn’t have any associated customer nodes, and the customer nodes would all be connected to empty orders.

### 2\. Ingest race conditions

Due to the distributed nature of Quine, ingest queries have to be carefully crafted to avoid race conditions. Here’s an example.

**Example customer data ingest query**

```cypher
WITH $that as customerData  
MATCH (customer), (order)  
WHERE id(customer) = idFrom(“customer”, customerData.customerNumber)  
SET customer:Customer,  
    customer.name = customerData.name
```    

**Example order ingest query**

```cypher
WITH $that as orderData  
MATCH (order), (customer)  
WHERE id(order) = idFrom(“order”, orderData.orderId)  
      AND customer.name = orderData.forName  
SET order:Order,  
    order.qty = orderData.quantity,  
    order.partNumber = orderData.part  
CREATE (customer)-[:ordered]->(order)
```

In the above example, the order ingest query is attempting to create relationships between orders and customers. However, this query assumes that a certain customer node exists when this order data gets ingested. Since ingests are run in parallel, this might not be the case. 

The outcome of the above misconfiguration would be that some expected order data might not show up in the graph at all. If, at the time of ingested a specific order, the intended related customer node isn’t yet created, the ingest query will silently fail, resulting in no new data for that ingested record being present in the graph.

## Slow Ingests

### 1\. Parallelism configuration

Quine uses a default parallelism value of **16** for ingest sources. This value can be tweaked to  potentially increase throughput. The correct value depends on many factors that are outside the scope of this environment (your cluster configuration, data complexity, query structure). However, experimenting with this value may result in improved ingest rates.

### 2\. Poorly optimized ingest queries

Query optimization is a broad topic outside the scope of this document. However, here are some guidelines to consider when writing performant ingest queries.

#### Anchor by IDs when possible

Consider the following two ingest queries.

**Query 1**

```cypher
WITH $that as orderData  
MATCH (order), (customer)  
WHERE id(order) = idFrom(orderData.orderId)  
      AND customer.name = orderData.forName  
…
```

**Query 2**

```cypher
WITH $that as orderData  
MATCH (order), (customer)  
WHERE id(order) = idFrom(orderData.orderId)  
      AND id(customer) = idFrom(orderData.customerId)  
…
```

In the first query, Quine will have to perform an all node scan (think full table scan in a relational database) to find matching customer data. This is due to the fact that we are matching on an arbitrary property of the customer node (in this case, name). In the second query, Quine is able to anchor on specific node IDs (think index lookup in a relational database).

#### Consider supernodes

A supernode is any node that has a significant number of half-edges. These nodes require special consideration to ensure that Quine meets performance expectations. See [Diagnosing Bottlenecks](diagnosing-bottlenecks.md#step-4-check-for-supernodes) for more details.

#### Data locality issues

Ingest performance can be impacted by the amount of messages that require coordination across more than one host in the cluster. thatDot recommends structuring ingest queries in such a way that minimizes coordination between cluster nodes.

### 3\. Standing query bottlenecks

Quine is a fully backpressured system. This means that if standing queries are doing a lot of work ingest rates may slow (or even halt entirely). Refer to the [Troubleshooting Queries](queries.md) guide to improve the performance of standing queries.

### Other Ingest Errors

Look at the log output from Quine and follow the trail.


---
# Miscellaneous
URL: https://quine.io/learn/troubleshooting/miscellaneous/

# Miscellaneous

## "Unsupported HTTP method" warnings

When you visit Quine with a web browser, Quine may output log lines that look like this:

<pre style="white-space: pre-wrap;">
<code>WARN [org.apache.pekko.actor.ActorSystemImpl(graph-service)] [graph-service-pekko.actor.default-dispatcher-12] org.apache.pekko.actor.ActorSystemImpl - Illegal request, responding with status '400 Bad Request': Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method. Perhaps this was an HTTPS request sent to an HTTP endpoint?
</code>
</pre>

**This warning is likely harmless and requires no action.**

The warning means that Quine is serving its web UI over plain HTTP (the default behavior), but the browser is trying to connect over HTTPS. Many browsers, Safari in particular, will attempt to connect over HTTPS as an extra security measure, sometimes even when the URL explicitly starts with `http://`! When the browser receives back a 400 failure response, then it connects correctly over HTTP. If you think this is happening, then you can safely ignore these warnings.


---
# Queries
URL: https://quine.io/learn/troubleshooting/queries/

# Troubleshooting Queries

This guide covers how to debug and troubleshoot queries in Quine. Understanding the differences between query types helps you apply the right debugging approach.

## Query Types in Quine

Quine has three types of queries, each with different troubleshooting approaches:

| Query Type | Description | When It Runs | Key Debugging Tools |
|------------|-------------|--------------|---------------------|
| **Ad-hoc queries** | Interactive Cypher queries you run manually | On-demand | `EXPLAIN`, `recentNodes()` |
| **Ingest queries** | Queries that execute for each record during data ingestion | Per-record during ingest | `EXPLAIN`, `recentNodes()` |
| **Standing queries** | Incremental pattern matching that fires when data matches | Continuously as data changes | `standing.wiretap()`, `recentNodes()` |

### Ad-hoc Queries

Ad-hoc queries are interactive Cypher queries you run against the graph. Troubleshoot them when:

- Query returns unexpected results or no results
- Query is slow or times out
- Query causes high resource usage

**Debugging approach**: Use [EXPLAIN](query-execution-plans.md) to understand the execution plan.

### Ingest Queries

Ingest queries run for each record in your data pipeline. Troubleshoot them when:

- Data isn't appearing in the graph
- Nodes have wrong properties or missing edges
- Ingest is slow or causing backpressure

**Debugging approach**: Test the query as an ad-hoc query first with sample data. Use [recentNodes()](#recentnodes) to verify ingested data.

See also: [Troubleshooting Ingest](ingest.md)

### Standing Queries

Standing queries incrementally match patterns as data enters the graph. Troubleshoot them when:

- Pattern should match but doesn't fire
- Getting unexpected matches or cancellations
- Standing query causes performance issues

**Debugging approach**: First run the pattern as an ad-hoc query to verify it matches expected data. Use [standing.wiretap()](#standingwiretap) to stream live results.

See also: [Standing Queries](../standing-queries/standing-queries.md)

## Debugging Tools Reference

| Tool | Page | Use For |
|------|------|---------|
| `EXPLAIN` | [Query Execution Plans](query-execution-plans.md) | Understanding how queries execute |
| `standing.wiretap()` | [Debugging Procedures](#standingwiretap) | Streaming live standing query results |
| `recentNodes()` | [Debugging Procedures](#recentnodes) | Viewing recently accessed nodes |

## Debugging Workflow

Follow this systematic approach when queries don't behave as expected.

### Step 1: Validate Ingested Data

Before debugging queries, confirm that data is flowing into the graph correctly:

```cypher
// View recently ingested nodes
CALL recentNodes(10) YIELD node
RETURN node
```

In the Exploration UI, double-click nodes to view their edges and hover to see properties. Verify:

- Nodes exist with expected IDs
- Properties have correct values and types
- Edges connect the expected nodes

### Step 2: Run as Ad-Hoc Query

If debugging a standing query or ingest query, run it as a regular Cypher query first:

```cypher
// Your standing query pattern, run as ad-hoc
MATCH (order:Order)-[:PLACED_BY]->(customer:Customer)
WHERE order.total > 1000
RETURN order, customer
```

If this returns results but your standing query doesn't fire:

- The data may have been ingested before the standing query was registered
- Check if the query mode (`DISTINCT_ID` vs `MULTIPLE_VALUES`) is appropriate

### Step 3: Analyze Query Plan

Use [EXPLAIN](query-execution-plans.md) to understand how your query executes:

```cypher
EXPLAIN MATCH (order:Order)-[:PLACED_BY]->(customer:Customer)
WHERE order.total > 1000
RETURN order, customer
```

Look for:

- **AllNodesScan** in `AnchoredEntry`: Consider anchoring by ID
- **Filter operators**: Are conditions being applied efficiently?
- **canContainAllNodeScan: true**: Query may be slow on large graphs

## Common Failure Patterns

### Missing Data in Graph

If ingested data doesn't appear in the graph, common causes include inconsistent `idFrom` usage across data sources, ingest race conditions, and edge creation that depends on property matching instead of ID lookups.

**See**: [Troubleshooting Ingest](ingest.md) for detailed guidance on missing data and race conditions.

### Standing Query Not Matching

| Symptom | Cause | Solution |
|---------|-------|----------|
| Query works ad-hoc but not as standing query | Data ingested before standing query was registered | Propagate the standing query to existing data (`propagateTo=INCLUDE_SLEEPING` on create, or the `standingQueries:propagate` endpoint); see [Propagation](../standing-queries/standing-queries.md#propagation-to-existing-data) |
| Pattern should match but doesn't | Data shape doesn't match pattern exactly | Run pattern as ad-hoc query to verify data matches |
| Matches initially then stops | Negative match canceling positive match | Check for data updates that invalidate the pattern |

### Slow Query Performance

Slow queries typically result from full graph scans, standing query backpressure, supernodes, or cross-host messaging overhead.

**See**: [Diagnosing Bottlenecks](diagnosing-bottlenecks.md) for metrics-based diagnosis and [Troubleshooting Ingest](ingest.md#slow-ingests) for ingest-specific optimizations.

## Debugging Procedures

Quine provides several procedures for inspecting query and node state.

### standing.wiretap()

Streams live results from a running standing query. This procedure runs until the standing query is canceled, emitting results incrementally as they match.

```cypher
// Wiretap "hasMaternalGrandpaJoe" and return properties of matching nodes
CALL standing.wiretap({ name: "hasMaternalGrandpaJoe" }) YIELD meta, data
WHERE meta.isPositiveMatch
MATCH (n) WHERE id(n) = data.id
RETURN properties(n)
```

Results appear incrementally as they match. Cancel the query when you have the information you need.

!!! warning
    The `standing.wiretap` procedure only stops running if the standing query is canceled (since otherwise, it can never be certain that there won't be more forthcoming match results). This means that it is risky to use the procedure in the Cypher REST API or in other places where results are not reported incrementally and queries cannot be canceled.

#### Return Fields

| Field | Type | Description |
|-------|------|-------------|
| `meta` | Map | Metadata including `isPositiveMatch` (boolean) and `isInitialResult` (boolean) |
| `data` | Map | The data returned by the standing query pattern |

### Results API Endpoint

Quine can stream standing query results outside of the Exploration UI using the server-sent events (SSE) endpoint at [Standing Query Status: `GET /api/v2/graph/quine/standingQueries/{standingQueryName}`](/reference/rest-api/?av=v2#/operations/get-standing-query-status):

```
GET /api/v2/graph/quine/standingQueries/{standingQueryName}
```

The endpoint surfaces new matches as they are produced.

```bash
$ curl http://localhost:8080/api/v2/graph/quine/standingQueries/hasMaternalGrandpaJoe
data:

data:

data:{"data":{"id":"2756309260014435"},"meta":{"isInitialResult":true,"isPositiveMatch":true,"resultId":"8f408026-8fb3-3955-c81a-7259175f41b8"}}
event:result
id:8f408026-8fb3-3955-c81a-7259175f41b8

data:{"data":{"id":"7945274922095468"},"meta":{"isInitialResult":true,"isPositiveMatch":true,"resultId":"6a83dda3-08a1-e085-ee7d-14138398f336"}}
event:result
id:6a83dda3-08a1-e085-ee7d-14138398f336
```

Using the SSE output, you can query the matching nodes directly:

```cypher
// Query for children of nodes with IDs from the SSE endpoint above
UNWIND [2756309260014435, 7945274922095468, 6994090876991233] AS personId
MATCH (person)<-[:HAS_MOTHER|:HAS_FATHER]-(child) WHERE id(person) = personId
RETURN person.name, child.name, child.yearBorn
```

### recentNodes()

Fetches recently accessed nodes from the in-memory cache. Useful for quickly verifying that data is being ingested:

```cypher
// View recently ingested nodes
CALL recentNodes(10) YIELD node
RETURN node
```

In the Exploration UI, double-click nodes to view their edges and hover to see properties.


---
# Query Execution Plans
URL: https://quine.io/learn/troubleshooting/query-execution-plans/

# Query Execution Plans

Use the `EXPLAIN` command to understand how Quine executes your Cypher queries. This helps identify performance issues and optimize query patterns.

!!! tip "When to Use EXPLAIN"

    * **Ad-hoc queries**: Before running expensive queries on production data
    * **Ingest queries**: When optimizing data loading performance
    * **Standing queries**: To understand pattern matching behavior (though standing queries compile differently)

## Using EXPLAIN

Prefix any Cypher query with `EXPLAIN` to see the execution plan without running the query:

```cypher
EXPLAIN MATCH (n:Person)-[:KNOWS]->(m:Person)
WHERE n.age > 30
RETURN m.name
```

This returns a query plan showing how Quine will execute the query. The query is compiled but **not executed**, making it safe to use on production systems.

## Query Plan Structure

The execution plan is returned as a JSON tree structure:

```json
{
  "operatorType": "Filter",
  "args": {
    "condition": "n.age > 30"
  },
  "identifiers": ["n", "m"],
  "children": [
    {
      "operatorType": "Expand",
      "args": {
        "edgeName": "KNOWS",
        "direction": "Outgoing"
      },
      "identifiers": ["n", "m"],
      "children": [...]
    }
  ],
  "isReadOnly": true,
  "isIdempotent": true,
  "canContainAllNodeScan": true
}
```

Plans form a tree where each operator processes results from its children. Read the plan from the innermost children outward to understand execution order.

## Root-Level Flags

The root of the query plan includes metadata flags:

| Flag | Description |
|------|-------------|
| `isReadOnly` | `true` if the query performs no writes to the graph |
| `isIdempotent` | `true` if running the query multiple times produces the same result |
| `canContainAllNodeScan` | `true` if the query may scan all nodes in the graph (performance warning) |

!!! warning "Performance Warning"

    If `canContainAllNodeScan` is `true`, your query may be slow on large graphs. Consider anchoring your query by node ID using `WHERE id(n) = idFrom(...)`. See [Using IDs in a Query](../../core-concepts/id-provider.md) for details.

## Query Plan Operators

The following operators appear as `operatorType` in query plans:

### Data Scanning & Entry Points

| Operator | Description |
|----------|-------------|
| `AnchoredEntry` | Starts from a specific node ID or index lookup |
| `ArgumentEntry` | Starts from an externally provided node argument |

### Graph Traversal

| Operator | Description |
|----------|-------------|
| `Expand` | Follows edges with optional length bounds and direction constraints |
| `GetDegree` | Returns the count of edges matching specified constraints |
| `LocalNode` | Checks node labels and properties, optionally binding to a variable |

### Data Flow & Combination

| Operator | Description |
|----------|-------------|
| `Apply` | Executes one query then another sequentially (flatMap) |
| `Union` | Executes two queries and concatenates their results |
| `Or` | Executes the first query; uses second only if first returns nothing |
| `ValueHashJoin` | Joins two queries on matching property values |
| `SemiApply` | Filters results based on whether a sub-query succeeds |
| `Cross` | Calculates cross product of multiple query plan results |

### Filtering & Transformation

| Operator | Description |
|----------|-------------|
| `Filter` | Filters rows by a condition |
| `FilterMap` | Projects and optionally filters data from nested query results |
| `Optional` | Emits results, or input row if no results |
| `AdjustContext` | Adds, removes, or renames columns |
| `Unwind` | Expands a list into multiple rows |

### Aggregation & Ordering

| Operator | Description |
|----------|-------------|
| `EagerAggregation` | Groups and aggregates results |
| `Return` | Applies ORDER BY, DISTINCT, SKIP, LIMIT in one operator |
| `Skip` | Drops the first N results |
| `Limit` | Keeps only the first N results |
| `Sort` | Sorts results by expression(s) |
| `Distinct` | Removes duplicate rows |

### Data Modification

| Operator | Description |
|----------|-------------|
| `SetProperty` | Sets a single node property |
| `SetProperties` | Batch updates node properties |
| `SetLabels` | Adds or removes labels from a node |
| `SetEdge` | Creates or deletes an edge |
| `Delete` | Removes a node, relationship, or path |

### Special Operations

| Operator | Description |
|----------|-------------|
| `ProcedureCall` | Calls a user-defined procedure |
| `SubQuery` | Executes a sub-query and stitches results |
| `LocalProperty` | Generates a row containing a property value from a node |
| `LoadCSV` | Loads and iterates over CSV data |
| `Empty` | Returns no results |
| `Unit` | Returns input unchanged |

## Reading an Execution Plan

Here's how to interpret a query plan for the query:

```cypher
EXPLAIN MATCH (n:Person)-[:KNOWS]->(m) WHERE n.age > 30 RETURN m.name
```

1. **Start at the innermost operator** - This is typically `AnchoredEntry` or a scan operation
2. **Follow the tree outward** - Each parent operator processes its children's output
3. **Look for performance issues**:
    - `AnchoredEntry` with `AllNodesScan` indicates a full graph scan
    - Multiple `Cross` operators can indicate combinatorial explosion
    - `Filter` operators late in the plan may process many unnecessary rows

## Query Type Considerations

### Ad-hoc Queries

For interactive queries, use `EXPLAIN` before running expensive operations:

```cypher
// Check the plan first
EXPLAIN MATCH (n:Person)-[:KNOWS*1..5]->(m:Person)
WHERE n.name = "Alice"
RETURN m

// If plan looks reasonable, run the actual query
MATCH (n:Person)-[:KNOWS*1..5]->(m:Person)
WHERE n.name = "Alice"
RETURN m
```

### Ingest Queries

For ingest queries that run repeatedly, optimize the plan once during development. Run the query as an ad-hoc query first to test behavior and performance before deploying it as part of your ingest pipeline:

```cypher
// Test as ad-hoc query with sample data
MATCH (n) WHERE id(n) = idFrom("user", "test-user-123")
SET n.lastSeen = datetime()
RETURN n

// Then verify the plan anchors properly
EXPLAIN MATCH (n) WHERE id(n) = idFrom("user", $that.userId)
SET n.lastSeen = $that.timestamp
```

Look for `AnchoredEntry` rather than scans, since ingest queries execute for every record.

### Standing Queries

Standing queries compile into a different internal representation optimized for incremental matching. While `EXPLAIN` shows the logical plan, the actual execution differs.

Before registering a standing query, run the pattern as an ad-hoc query to verify it matches expected data:

```cypher
// Test the pattern as an ad-hoc query first
MATCH (order:Order)-[:PLACED_BY]->(customer:Customer)
WHERE order.total > 1000
RETURN order, customer
LIMIT 10

// Once satisfied, register as a standing query with the pattern
```

For debugging registered standing queries, use [debugging procedures](queries.md#debugging-procedures) to inspect runtime state.


---
# Recipes
URL: https://quine.io/recipes/

A [recipe](../learn/recipe-ref-manual.md) is a collection of configuration that sets a Quine instance up for a specific purpose. A recipe is defined in a `yaml` file and contains configuration for: ingest streams, standing queries, UI configuration, and some metadata about the recipe. They are also a great way to learn about Quine features like [ingest streams](../getting-started/ingest-streams-tutorial.md) that build the streaming graph, [standing queries](../getting-started/ingest-streams-tutorial.md) that find results you are looking for and take actions, and customizing the [exploration UI](../getting-started/exploration-ui.md).

To visualize what a recipe does and get recommendations before running it, paste it into the [Quine Recipe Analyzer](https://www.thatdot.com/quine/recipe-analyzer.html).

In order to launch Quine with a recipe, use `-r`, followed by either the short name of a sample recipe or a local YAML filename. Some recipes can expect input parameters. The parameter values are passed by using command line arguments with `-x` or `--recipe-value`.

For example, to run the [wikipedia](wikipedia.md) recipe[^1].

[^1]: When a recipe is launched that does not contain a file extension,
    Quine will fetch and launch the recipe by name from the
    [recipes](https://github.com/thatdot/quine/tree/main/quine/recipes/)
    repository on GitHub.
    For example, `java -jar quine-2.1.1.jar -r wikipedia` will download the `wikipedia.yaml` file from
    from GitHub and launch that version of the recipe, even if you have a copy of `wikipedia.yaml`
    your local directory. This method only works for example recipes that are distributed with Quine.

```shell
java -jar quine-2.1.1.jar -r wikipedia
```

## Quine Recipes

- #### [APT Detection](apt-detection.md)

    ---

    Endpoint logs and network traffic data merge to auto-detect exfiltration and alert for an IoB that matches a typical malicious data exfiltration pattern.

    <a href="https://github.com/rrwright" class="md-author" style="float: left;" title="@rrwright"> <img src="https://github.com/rrwright.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="rrwright"></a>
    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>
<!--
- #### [Financial Risk](finance.md)

    ---

    A streaming graph solution to the financial industry’s current approach to managing mandated operational risk capital requirements.

    <a href="https://github.com/7evenbridges" class="md-author" style="float: left;" title="@7evenbridges"> <img src="https://github.com/7evenbridges.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="7evenbridges"></a>
    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>
-->

- #### [CDN Cache Efficiency](cdn.md)

    ---

    Continuous and real-time computation of CDN cache node efficiency from Fastly CDN logs, materialized by ASN, Geo, Asset and PoP.

    <a href="https://github.com/7evenbridges" class="md-author" style="float: left;" title="@7evenbridges"> <img src="https://github.com/7evenbridges.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="7evenbridges"></a>
    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

- #### [Password Spraying](password-spraying.md)

    ---

    Detect password spraying attacks in real time. Ingests JSON-formatted IAM-style password authentication log file, creates graph, uses standing queries to sent alerts.

    <a href="https://github.com/7evenbridges" class="md-author" style="float: left;" title="@7evenbridges"> <img src="https://github.com/7evenbridges.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="7evenbridges"></a>
    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

- #### [Temporal Locality](duration.md)

    ---

    Relate email messages sent or received by a specific user within a 4-6 minute window.

    <a href="https://github.com/maglietti" class="md-author" style="float: left;" title="@maglietti"> <img src="https://github.com/maglietti.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="maglietti"></a>
    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

- #### [Monitor an MMO](planetside-2.md)

    ---

    Build a live event-driven model of what's currently happening in the "PlanetSide 2" MMOFPS video game. A great example of ingesting from Websockets.

    <a href="https://github.com/emanb29" class="md-author" style="float: left;" title="@emanb29"> <img src="https://github.com/emanb29.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="emanb29"></a>
    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

- #### [IMDB Movie Data](movieData.md)

    ---

    Explore a familiar graph data set using Quine to combine data from separate files, unwind nested data into unique nodes, populate node parameters, then generate a new stream from the combined data.

    <a href="https://github.com/maglietti" class="md-author" style="float: left;" title="@maglietti"> <img src="https://github.com/maglietti.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="maglietti"></a>
    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

- #### [Wikipedia Page Creation Feed](wikipedia.md)

    ---

    Wikipedia page creation events are instantiated in the graph with relationships to a refied time model Additionally, page creation event comments are echoed to standard output.

    <a href="https://github.com/landon9720" class="md-author" style="float: left;" title="@landon9720"> <img src="https://github.com/landon9720.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="landon9720"></a>
    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

- #### [Entity Resolution](./entity-resolution.md)

    ---

    Learn how real-time entity resolution – the deduplication of similar data – can drastically help with creating a comprehensive view of your data.

    <a href="https://github.com/rrwright" class="md-author" style="float: left;" title="@rrwright"> <img src="https://github.com/rrwright.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="rrwright"></a>
    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>

- #### [Webhook Data Enrichment](./webhook.md)

    ---

    Learn how to enrich the Quine graph from an external service using the Standing Query HTTP webhook output.

    <a href="https://github.com/mastapegs" class="md-author" style="float: left;" title="@mastapegs"> <img src="https://github.com/mastapegs.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="mastapegs"></a>
    <small>Shared by: [Matthew Pagan](https://github.com/mastapegs)</small>

- #### [Ethereum Tag Propagation](ethereum.md)

    ---

    Ingestion a live stream of events from the Ethereum Blockchain and demonstrate real-time "dirty money" tag propagation.

    <a href="https://github.com/emanb29" class="md-author" style="float: left;" title="@emanb29"> <img src="https://github.com/emanb29.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="emanb29"></a>
    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

- #### [Basic File Ingest](ingest.md)

    ---

    Ingest each line from a file passed as `$in_file` into a disconnected graph then fill each node with a property containing the line.

    <a href="https://github.com/landon9720" class="md-author" style="float: left;" title="@landon9720"> <img src="https://github.com/landon9720.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="landon9720"></a>
    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

- #### [Conway's Game of Life](conways-gol.md)

    ---

    Conway's Game of Life cellular automaton demonstrating Quine's Turing completeness through recursive standing queries.

    <a href="https://github.com/brackishman" class="md-author" style="float: left;" title="@brackishman"> <img src="https://github.com/brackishman.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="brackishman"></a>
    <small>Shared by: [Matthew Cullum](https://github.com/brackishman)</small>

- #### [Harry Potter](hpotter.md)

    ---

    Small graph of connected nodes that explore the familial relationships of Harry Potter characters.

    <a href="https://github.com/harpocrates" class="md-author" style="float: left;" title="@harpocrates"> <img src="https://github.com/harpocrates.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="harpocrates"></a>
    <small>Shared by: [Alec Theriault](https://github.com/harpocrates)</small>

- #### [Apache Log Analytics](apache_log.md)

    ---

    Example use of Quine's unique Standing Query function to parse incoming text for each line of an Apache web server access log into a graph.

    <a href="https://github.com/joshcody" class="md-author" style="float: left;" title="@joshcody"> <img src="https://github.com/joshcody.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="joshcody"></a>
    <small>Shared by: [Josh Cody](https://github.com/joshcody)</small>

- #### [Certstream Firehose](certstream-firehose.md)

    ---

    Reproduces the behavior of the [certstream website](https://certstream.calidog.io/) by connecting to the certstream firehose via SSL-encrypted websocket and printing to standard out each time a new certificate is detected.

    <a href="https://github.com/emanb29" class="md-author" style="float: left;" title="@emanb29"> <img src="https://github.com/emanb29.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="emanb29"></a>
    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

- #### [Quine Logs Recipe](quine-logs-recipe.md)

    ---

    Ingest Quine log lines into Quine!

    <a href="https://github.com/maglietti" class="md-author" style="float: left;" title="@maglietti"> <img src="https://github.com/maglietti.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="maglietti"></a>
    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

- #### [Approximating Pi](pi.md)

    ---

    Incrementally approximates pi using Leibniz' formula in Quine.

    <a href="https://github.com/emanb29" class="md-author" style="float: left;" title="@emanb29"> <img src="https://github.com/emanb29.png" class="skip-lightbox"  target="_blank" height="30px" width="30px" alt="emanb29"></a>
    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>


---
# Apache Log Analysis
URL: https://quine.io/recipes/apache_log/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Josh Cody](https://github.com/joshcody)</small>

    A simple recipe that parses incoming text for each line of an Apache web server access log and structures it into a graph. A useful introduction to standing queries, the powerful feature that makes Quine unique.

    ??? example "Apache Log Analysis Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/apache_log.yaml"
        ```

    [Download Recipe](assets/apache_log.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Josh Cody](https://github.com/joshcody)</small>

    A simple recipe that parses incoming text for each line of an Apache web server access log and structures it into a graph. A useful introduction to standing queries, the powerful feature that makes Quine unique.

    ??? example "Apache Log Analysis Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/apache_log.yaml"
        ```

    [Download Recipe](assets/v2/apache_log.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

This recipe loads a sample Apache log file and manifests disconnected nodes in a graph for basic analysis and metric reporting.

## Sample Data

!!! note

    Download the sample data to the same directory where you will run Quine.

A sample Apache web server access logs dataset downloaded from [`https://recipes.quine.io/sample_apache_logs`](https://recipes.quine.io/sample_apache_logs) or using the command below.

```shell
curl -L https://recipes.quine.io/sample_apache_logs -o apache.log
```

## How it Works

The recipe reads log entries from the sample data files using an [ingest stream](../learn/ingest-sources/index.md) to manifest a graph in Quine. A regular expression inside the ingest stream Cypher query parses the logline and populates parameters in the node.

The ingest stream processes the `apache.log` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: $in_file
        format:
          type: CypherLine
          query: |-
            WITH text.regexFirstMatch($that, '(\S+)\s+\S+\s+(\S+)\s+\[(.+)\]\s+"(.*)\s+(.*)\s+(.*)"\s+([0-9]+)\s+(\S+)\s+"(.*)"\s+"(.*)"')
            AS r
            CREATE ({
              sourceIp: r[1],
              user: r[2],
              time: datetime(r[3], 'dd/MMM/yyyy:HH:mm:ss Z'),
              verb: r[4],
              path: r[5],
              httpVersion: r[6],
              status: r[7],
              size: r[8],
              referrer: r[9],
              agent: r[10],
              type: 'log'
            })
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "$in_file",
      "format": {
        "type": "CypherLine",
        "query": "WITH text.regexFirstMatch($that, '(\\\\S+)\\\\s+\\\\S+\\\\s+(\\\\S+)\\\\s+\\\\[(.+)\\\\]\\\\s+\"(.*)\\\\s+(.*)\\\\s+(.*)\"\\\\s+([0-9]+)\\\\s+(\\\\S+)\\\\s+\"(.*)\"\\\\s+\"(.*)\"') AS r CREATE ({ sourceIp: r[1], user: r[2], time: datetime(r[3], 'dd/MMM/yyyy:HH:mm:ss Z'), verb: r[4], path: r[5], httpVersion: r[6], status: r[7], size: r[8], referrer: r[9], agent: r[10], type: 'log' })"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: apache-log-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Line
        query: |-
          WITH text.regexFirstMatch($that, '(\S+)\s+\S+\s+(\S+)\s+\[(.+)\]\s+"(.*)\s+(.*)\s+(.*)"\s+([0-9]+)\s+(\S+)\s+"(.*)"\s+"(.*)"')
          AS r
          CREATE ({
            sourceIp: r[1],
            user: r[2],
            time: datetime(r[3], 'dd/MMM/yyyy:HH:mm:ss Z'),
            verb: r[4],
            path: r[5],
            httpVersion: r[6],
            status: r[7],
            size: r[8],
            referrer: r[9],
            agent: r[10],
            type: 'log'
          })
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "apache-log-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Line"
        }
      },
      "query": "WITH text.regexFirstMatch($that, '(\\S+)\\s+\\S+\\s+(\\S+)\\s+\\[(.+)\\]\\s+\"(.*)\\s+(.*)\\s+(.*)\"\\s+([0-9]+)\\s+(\\S+)\\s+\"(.*)\"\\s+\"(.*)\"') AS r CREATE ({ sourceIp: r[1], user: r[2], time: datetime(r[3], 'dd/MMM/yyyy:HH:mm:ss Z'), verb: r[4], path: r[5], httpVersion: r[6], status: r[7], size: r[8], referrer: r[9], agent: r[10], type: 'log' })"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect nodes that have a `type` of `log` and then create relationships between the nodes and their verbs.

=== "YAML"

    ```yaml
      - pattern:
          type: Cypher
          query: MATCH (l) WHERE l.type = 'log' RETURN DISTINCT id(l) AS id
          mode: DistinctId
        outputs:
          verb:
            type: CypherQuery
            query: |-
              MATCH (l) WHERE id(l) = $that.data.id
              MATCH (v) WHERE id(v) = idFrom('verb', l.verb)
              SET v.type = 'verb',
                v.verb = l.verb
              CREATE (l)-[:verb]->(v)
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (l) WHERE l.type = 'log' RETURN DISTINCT id(l) AS id",
        "mode": "DistinctId"
      },
      "outputs": {
        "verb": {
          "type": "CypherQuery",
          "query": "MATCH (l) WHERE id(l) = $that.data.id MATCH (v) WHERE id(v) = idFrom('verb', l.verb) SET v.type = 'verb', v.verb = l.verb CREATE (l)-[:verb]->(v)"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: log-to-verb
        pattern:
          type: Cypher
          query: MATCH (l) WHERE l.type = 'log' RETURN DISTINCT id(l) AS id
          mode: DISTINCT_ID
        outputs:
          - name: verb
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (l) WHERE id(l) = $that.id
                MATCH (v) WHERE id(v) = idFrom('verb', l.verb)
                SET v.type = 'verb',
                  v.verb = l.verb
                CREATE (l)-[:verb]->(v)
                RETURN null
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "log-to-verb",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (l) WHERE l.type = 'log' RETURN DISTINCT id(l) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "verb",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (l) WHERE id(l) = $that.id MATCH (v) WHERE id(v) = idFrom('verb', l.verb) SET v.type = 'verb', v.verb = l.verb CREATE (l)-[:verb]->(v) RETURN null",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

## Running the Recipe

``` shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r apache_log.yaml --recipe-value in_file=apache.log
Graph is ready
Running Recipe: Apache Log Analytics
Using 1 sample queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Status query URL is http://localhost:8080#MATCH%20%28l%29%2D%5Brel%3Averb%5D%2D%3E%28v%29%20WHERE%20l%2Etype%20%3D%20%27log%27%20AND%20v%2Etype%20%3D%20%27verb%27%20AND%20v%2Everb%20%3D%20%27GET%27%20RETURN%20count%28rel%29%20AS%20get%5Fcount
Quine web server available at http://localhost:8080
INGEST-1 status is completed and ingested 10000
```

## Summary

The recipe contains a status query that will emit a link in the console window to view the results for the count of `GET` requests in the log file. The status query will also update the console with a running results count of `GET` entries as they are encountered.

``` json
Status query URL is http://localhost:8080#MATCH%20%28l%29%2D%5Brel%3Averb%5D%2D%3E%28v%29%20WHERE%20l%2Etype%20%3D%20%27log%27%20AND%20v%2Etype%20%3D%20%27verb%27%20AND%20v%2Everb%20%3D%20%27GET%27%20RETURN%20count%28rel%29%20AS%20get%5Fcount
```

At the time of writing this recipe, the sample data file contains 9951 `GET` requests.

``` text
---[ Status Query result 1 ]------------
get_count | 9951 count 10000
----------+-----------------------------
```

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query      | Node Type | Description                                       |
| :--------------- | :-------- | :------------------------------------------------ |
| Adjacent Nodes   | All       | Display the nodes that are adjacent to this node. |
| Refresh          | All       | Refresh the content stored in a node              |
| Local Properties | All       | Display the properties stored by the node         |

## Build your skills

What Cypher query could you write to return a count for other HTTP verbs in the log file?

??? success "Solution"

    We solved this by modifying the status query to be less specific and to return unique node verb parameters as part of the results.

    Enter this query into the Exploration UI and hit ++shift+enter++.

    ``` cypher
    MATCH (l)-[rel:verb]->(v)
    WHERE l.type = 'log' AND v.type = 'verb'
    RETURN DISTINCT v.verb, count(rel)
    ```

    Our results:

    | v.verb    | count(rel) |
    | :-------- | :--------- |
    | "OPTIONS" | 1          |
    | "POST"    | 5          |
    | "HEAD"    | 42         |
    | "GET"     | 9951       |


---
# APT Detection
URL: https://quine.io/recipes/apt-detection/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>

    This APT (Advanced Persistent Threat) detection recipe ingests EDR (Endpoint Detection and Response) and network traffic logs, while monitoring for an IoB (Indicator of Behavior) that matches malicious data exfiltration patterns.

    ??? example "APT Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/apt-detection.yaml"
        ```

    [Download Recipe](assets/apt-detection.yaml){ .md-button download="" .md-button--primary }

=== "Recipe v2"

    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>

    This APT (Advanced Persistent Threat) detection recipe ingests EDR (Endpoint Detection and Response) and network traffic logs, while monitoring for an IoB (Indicator of Behavior) that matches malicious data exfiltration patterns.

    ??? example "APT Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/apt-detection.yaml"
        ```

    [Download Recipe](assets/v2/apt-detection.yaml){ .md-button download="" .md-button--primary }

## Scenario

In this scenario, a malicious Excel macro collects personal data and stores it in a temporary file. The APT process `ntclean` infiltrated the system previously through an SSH exploit, and now reads from that temporary file and exfiltrates data from the network *hiding it as an HTTP GET request* before deleting the temporary file to cover its tracks.

Using a standing query, the recipe monitors for covert interprocess communication using a file to pass data. When that pattern is matched, with a network SEND event, we have our smoking gun and a URL is logged linking to the Quine Exploration UI with the full activity and context for investigation.

The source of the SSH exploit that planted the APT and the destination for exfiltrated data utilize the same IP address.

## Sample Data

Download the sample data to the same directory where Quine will be run.

* `endpoint.json` - [https://recipes.quine.io/apt-detection/endpoint-json](https://recipes.quine.io/apt-detection/endpoint-json)
* `network.json`  - [https://recipes.quine.io/apt-detection/network-json](https://recipes.quine.io/apt-detection/network-json)

## How it Works

The recipe reads observations from the two sample data files using [ingest streams](../learn/ingest-sources/index.md) to manifest a graph in Quine. A separate ingest stream is configured to process each file, each containing Cypher that parses the observations, manifests nodes, and relates them to each other in the graph.

INGEST-1 processes the `endpoints.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: endpoint.json
        format:
        type: CypherJson
        query: >-
            MATCH (proc), (event), (object)
            WHERE id(proc) = idFrom($that.pid)
              AND id(event) = idFrom($that)
              AND id(object) = idFrom($that.object)

            SET proc.id = $that.pid,
                proc: Process,
                event.type = $that.event_type,
                event: EndpointEvent,
                event.time = $that.time,
                object.data = $that.object

            CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "endpoint.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (proc), (event), (object) WHERE id(proc) = idFrom($that.pid) AND id(event) = idFrom($that) AND id(object) = idFrom($that.object) SET proc.id = $that.pid, proc: Process, event.type = $that.event_type, event: EndpointEvent, event.time = $that.time, object.data = $that.object CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: endpoint-events
        source:
          type: File
          path: $endpoint_file
          format:
            type: Json
        query: >-
          MATCH (proc), (event), (object)
          WHERE id(proc) = idFrom($that.pid)
            AND id(event) = idFrom($that)
            AND id(object) = idFrom($that.object)

          SET proc.id = $that.pid,
              proc: Process,
              event.type = $that.event_type,
              event: EndpointEvent,
              event.time = $that.time,
              object.data = $that.object

          CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "endpoint-events",
      "source": {
        "type": "File",
        "path": "$endpoint_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (proc), (event), (object) WHERE id(proc) = idFrom($that.pid) AND id(event) = idFrom($that) AND id(object) = idFrom($that.object) SET proc.id = $that.pid, proc: Process, event.type = $that.event_type, event: EndpointEvent, event.time = $that.time, object.data = $that.object CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)"
    }
    ```

INGEST-2 processes the `network.json` file:

=== "YAML"

    ```yaml
    - type: FileIngest
        path: network.json
        format:
        type: CypherJson
        query: >-
            MATCH (src), (dst), (event)
            WHERE id(src) = idFrom($that.src_ip+":"+$that.src_port)
              AND id(dst) = idFrom($that.dst_ip+":"+$that.dst_port)
              AND id(event) = idFrom('network_event', $that)

            SET src.ip = $that.src_ip+":"+$that.src_port,
                src: IP,
                dst.ip = $that.dst_ip+":"+$that.dst_port,
                dst: IP,
                event.proto = $that.proto,
                event.time = $that.time,
                event.detail = $that.detail,
                event: NetTraffic

            CREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-2"
    {
      "type": "FileIngest",
      "path": "network.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (src), (dst), (event) WHERE id(src) = idFrom($that.src_ip+\":\"+$that.src_port)\n  AND id(dst) = idFrom($that.dst_ip+\":\"+$that.dst_port)\n  AND id(event) = idFrom('network_event', $that)\n\nSET src.ip = $that.src_ip+\":\"+$that.src_port,\n    src: IP,\n    dst.ip = $that.dst_ip+\":\"+$that.dst_port,\n    dst: IP,\n    event.proto = $that.proto,\n    event.time = $that.time,\n    event.detail = $that.detail,\n    event: NetTraffic\n\nCREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: network-events
        source:
          type: File
          path: $network_file
          format:
            type: Json
        query: >-
          MATCH (src), (dst), (event)
          WHERE id(src) = idFrom($that.src_ip+":"+$that.src_port)
            AND id(dst) = idFrom($that.dst_ip+":"+$that.dst_port)
            AND id(event) = idFrom('network_event', $that)

          SET src.ip = $that.src_ip+":"+$that.src_port,
              src: IP,
              dst.ip = $that.dst_ip+":"+$that.dst_port,
              dst: IP,
              event.proto = $that.proto,
              event.time = $that.time,
              event.detail = $that.detail,
              event: NetTraffic

          CREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "network-events",
      "source": {
        "type": "File",
        "path": "$network_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (src), (dst), (event) WHERE id(src) = idFrom($that.src_ip+\":\"+$that.src_port) AND id(dst) = idFrom($that.dst_ip+\":\"+$that.dst_port) AND id(event) = idFrom('network_event', $that) SET src.ip = $that.src_ip+\":\"+$that.src_port, src: IP, dst.ip = $that.dst_ip+\":\"+$that.dst_port, dst: IP, event.proto = $that.proto, event.time = $that.time, event.detail = $that.detail, event: NetTraffic CREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect a WRITE->READ->SEND->DELETE pattern that is typical for this type of exflitration event.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: >-
            MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2),
                  (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)
            WHERE e1.type = "WRITE"
              AND e2.type = "READ"
              AND e3.type = "DELETE"
              AND e4.type = "SEND"
            RETURN DISTINCT id(f) as fileId

    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)\nWHERE e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\nRETURN DISTINCT id(f) as fileId"
      },
      "outputs": {
        "stolen-data": {
          "type": "CypherQuery",
          "query": "MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)\nWHERE id(f) = $that.data.fileId\n  AND e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\n  AND e1.time < e2.time\n  AND e2.time < e3.time\n  AND e2.time < e4.time\n\nCREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)\nWITH e1, e2, e3, e4, p1, p2, f, ip, \"http://localhost:8080/#MATCH\" + text.urlencode(\" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='\"+strId(p1)+\"' AND id(e1)='\"+strId(e1)+\"' AND id(f)='\"+strId(f)+\"' AND id(e2)='\"+strId(e2)+\"' AND id(p2)='\"+strId(p2)+\"' AND id(e3)='\"+strId(e3)+\"' AND id(e4)='\"+strId(e4)+\"' AND id(ip)='\"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip\") as URL RETURN URL",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: exfiltration-detection
        pattern:
          type: Cypher
          query: >-
            MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2),
                  (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)
            WHERE e1.type = "WRITE"
              AND e2.type = "READ"
              AND e3.type = "DELETE"
              AND e4.type = "SEND"
            RETURN DISTINCT id(f) as fileId
          mode: DISTINCT_ID
        outputs:
          - name: stolen-data
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: >-
                MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2),
                      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
                WHERE id(f) = $that.fileId
                  AND e1.type = "WRITE"
                  AND e2.type = "READ"
                  AND e3.type = "DELETE"
                  AND e4.type = "SEND"
                  AND e1.time < e2.time
                  AND e2.time < e3.time
                  AND e2.time < e4.time

                CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

                WITH e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
                RETURN URL
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "exfiltration-detection",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4) WHERE e1.type = \"WRITE\" AND e2.type = \"READ\" AND e3.type = \"DELETE\" AND e4.type = \"SEND\" RETURN DISTINCT id(f) as fileId",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "stolen-data",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip) WHERE id(f) = $that.fileId AND e1.type = \"WRITE\" AND e2.type = \"READ\" AND e3.type = \"DELETE\" AND e4.type = \"SEND\" AND e1.time < e2.time AND e2.time < e3.time AND e2.time < e4.time CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3) WITH e1, e2, e3, e4, p1, p2, f, ip, \"http://localhost:8080/#MATCH\" + text.urlencode(\" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='\"+strId(p1)+\"' AND id(e1)='\"+strId(e1)+\"' AND id(f)='\"+strId(f)+\"' AND id(e2)='\"+strId(e2)+\"' AND id(p2)='\"+strId(p2)+\"' AND id(e3)='\"+strId(e3)+\"' AND id(e4)='\"+strId(e4)+\"' AND id(ip)='\"+strId(ip)+\"' RETURN e1, e2, e3, e4, p1, p2, f, ip\") as URL RETURN URL",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

Once Quine detects the pattern, the event is sent to a standing query output for additional processing and action.

```yaml
    outputs:
      stolen-data:
        type: CypherQuery
        query: >-
          MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), 
                (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
          WHERE id(f) = $that.data.fileId
            AND e1.type = "WRITE"
            AND e2.type = "READ"
            AND e3.type = "DELETE"
            AND e4.type = "SEND"
            AND e1.time < e2.time
            AND e2.time < e3.time
            AND e2.time < e4.time

          CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

          With e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
          RETURN URL
        andThen:
          type: PrintToStandardOut
```

```yaml
    outputs:
      - name: stolen-data
        resultEnrichment:
          query: >-
            MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), 
                  (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
            WHERE id(f) = $that.fileId
              AND e1.type = "WRITE"
              AND e2.type = "READ"
              AND e3.type = "DELETE"
              AND e4.type = "SEND"
              AND e1.time < e2.time
              AND e2.time < e3.time
              AND e2.time < e4.time

            CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

            With e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
            RETURN URL
          parameter: that
        destinations:
          - type: StandardOut
```

The result once the pattern is detected is to output a link to the console that an analyst can use to review the event further within Quine's Exploration UI.

``` { .json linenums="1" }
2022-12-15 11:28:52,413 Standing query `stolen-data` match: {"meta":{"isPositiveMatch":true,"resultId":"5bd8beb7-78cc-de3a-bf69-8ad20b90cd11"},"data":{"URL":"http://localhost:8080/#MATCH%20%28e1%29%2C%28e2%29%2C%28e3%29%2C%28e4%29%2C%28p1%29%2C%28p2%29%2C%28f%29%2C%28ip%29%20WHERE%20id%28p1%29%3D%271ca87b55-a62a-3f13-bc2d-5752ca5f4143%27%20AND%20id%28e1%29%3D%2743f26672-0c93-3e2c-84bb-88adb84454ba%27%20AND%20id%28f%29%3D%27f00ae947-3dd5-3c92-a84f-118b401c80f1%27%20AND%20id%28e2%29%3D%275a6bb84a-ed7e-3982-83d8-ea9f7b0dee9d%27%20AND%20id%28p2%29%3D%2717d3fb9a-cd9b-39ba-a087-e2f577627873%27%20AND%20id%28e3%29%3D%279fc76f93-c93d-3eb3-8730-0bca7ee079c4%27%20AND%20id%28e4%29%3D%274cc1ccc4-6286-3a32-b691-d308ba1e68e7%27%20AND%20id%28ip%29%3D%27a1ce3be6-4501-3af2-b05b-2c6ac0936cc4%27%20RETURN%20e1%2C%20e2%2C%20e3%2C%20e4%2C%20p1%2C%20p2%2C%20f%2C%20ip"}}
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r apt-detection.yaml
Graph is ready
Running Recipe: APT Detection
Using 5 node appearances
Using 14 quick queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Running Ingest Stream INGEST-2
Quine web server available at http://localhost:8080
```

## Summary

When the standing query detects the WRITE->READ->SEND->DELETE pattern, it will output a link to the console that can be copied and pasted into a browser to explore the event in the Quine Exploration UI. Copy and paste the URL section of the match JSON from your console into your browser.

The nodes will be jumbled together when you first open the graph. Arrange the nodes to look similar to the image below before you start exploring.

![apt-detection-graph](images/apt-detection-graph.png)

The recipe includes a number of [Replace Quick Queries: `PUT /api/v2/queryUi/quickQueries`](/reference/rest-api/?av=v2#/operations/replace-quick-queries) to assist in exploring the event data. Right click on a node to bring up the quick query interface.

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query           | Node Type | Description                                            |
| :-------------------- | :-------- | :----------------------------------------------------- |
| Adjacent Nodes        | All       | Display the nodes that are adjacent to this node.      |
| Refresh               | All       | Refresh the content stored in a node                   |
| Local Properties      | All       | Display the properties stored by the node              |
| Files Read            | Process   | Load nodes representing the files read by this process |
| Files Written         | Process   | Load nodes representing files written by this process  |
| Read By               | data      | Load nodes that read data from this node               |
| Written By            | data      | Load nodes that wrote to this node                     |
| Received Data         | Process   | Where did this process receive data from               |
| Sent Data             | Process   | Where did this process send data to                    |
| Started By            | Process   | What started this process                              |
| Started Other Process | Process   | What process did this process start                    |
| Network Send          | IP        | Where did this IP node send data                       |
| Network Receive       | IP        | Whre did this IP node receive data from                |
| Network Communication | IP        | What other nodes did this IP node communicate with     |

Use the quick queries to explore the graph and uncover the timeline behind the entire event.


---
# CDN Observability
URL: https://quine.io/recipes/cdn/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    Real-time computation of CDN cache node efficiency from pseudonymized Fastly CDN logs, with graph association of each log entry to serving PoP, cache server, client, client ASN, asset and origin to identify potential root cause of issues.

    ??? example "Full CDN Observability Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/cdn.yaml"
        ```

    [Download Recipe](assets/cdn.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

=== "Recipe v2"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    Real-time computation of CDN cache node efficiency from pseudonymized Fastly CDN logs, with graph association of each log entry to serving PoP, cache server, client, client ASN, asset and origin to identify potential root cause of issues.

    ??? example "Full CDN Observability Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/cdn.yaml"
        ```

    [Download Recipe](assets/v2/cdn.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

## Scenario

Pseudonymized CDN log data is imported from a JSON file (`cdn_data_50k.json`) via a file ingest, and nodes are manifested for the elements associated with each event (e.g., client, server, pop, etc.).

Each of the manifested nodes increment counters to track the number of cache hits and misses at each level (e.g., source ASN, server, pop, etc.). Selecting any node allows you to query at each level to identify potential root cause of poor performance.

A standing query is defined to match consecutive cache misses within a configurable fixed period of time for the purpose of alerting.

## Sample Data

Download the sample data to the same directory where Quine will be run.

* `cdn_data_50k.json` - [https://that.re/cdn_data_50k](https://that.re/cdn_data_50k)

## How it Works

The recipe reads observations from the sample data file using [ingest streams](../learn/ingest-sources/index.md) to manifest a graph in Quine. An ingest stream is configured to process the data file, containing Cypher that parses the log entries, manifests nodes, and relates them to each other in the graph.

The log entries take the form of:

```json
{
    "backend_ip": "157.52.79.52",
    "backend_ttlb": 73.206,
    "business_unit": "68ae725c3fd8d6831735753269a727c9ce05baae6715bb0191ddb7f6d67842bd",
    "bytes_in": 377,
    "bytes_out": 902,
    "cached": false,
    "cache_shield": "false",
    "cache_status": "MISS-CLUSTER",
    "client_asn": 7922,
    "client_geo_country": "US",
    "client_ip": "1localhost",
    "client_ttfb": 73.198,
    "environment": "prod",
    "failover_status": "",
    "forward_for": "",
    "host": "682d313399617e3d194679e8422a6a5f2666b60c54ab6eead5908040443c793f",
    "if_modified_since": "",
    "if_none_match": "",
    "if_unmodified_since": "",
    "method": "GET",
    "path": "/flavi7d79/master/flavi7d79_6.m3u8",
    "pop": "SJC",
    "range_request": "(null)",
    "range_response": "",
    "request_id": "cache-sjc10025-SJC-2614091026",
    "restarts": 0,
    "retrans": 0,
    "role": "edge",
    "rtt_msecs": 28,
    "server_id": "cache-sjc10025-SJC",
    "server_ip": "2a04:4e42:a::645",
    "server_ttlb": 73.261,
    "shield_failover": "",
    "stream": 0,
    "status_code": 200,
    "timestamp": "2020-07-14 22:55:31.729734",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36",
    "workflow": "f2757f5a18302320de05caa509ac98370b6a9cec",
    "origin_request_id": "",
    "query": ""
}
```

INGEST-1 processes the `cdn_data_50k.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: cdn_data_50k.json
        format:
        type: CypherJson
        query: |-
          MATCH (event), (client), (asset), (asn), (server), (pop), (origin), (clientGeo)
          WHERE $that.cache_status IS NOT NULL
            AND id(event) = idFrom('event', $that.timestamp, $that.request_id)
            AND id(client) = idFrom('client', $that.client_ip, $that.business_unit)
            AND id(asset) = idFrom('asset', $that.path)
            AND id(asn) = idFrom('asn', toString($that.client_asn))
            AND id(server) = idFrom('server', $that.pop, $that.server_id)
            AND id(pop) = idFrom('pop', $that.pop)
            AND id(origin) = idFrom('origin', $that.backend_ip)
            AND id(clientGeo) = idFrom('clientGeo', $that.client_geo_country)

          ////////////////////////////////////////
          //Bucketing for HITs and MISSes counters
          ////////////////////////////////////////
          // RegEx deets here: https://regex101.com/r/uP0KMm/1
          WITH *, text.regexFirstMatch($that.cache_status, '(HIT|MISS(?!.*HIT)).*') AS hmp WHERE hmp[1] IS NOT NULL

          ////////////////////////////////////////
          // Bucketing for node type counters
          ////////////////////////////////////////
          CALL incrementCounter(client, "count",1) YIELD count AS clientCount
          CALL incrementCounter(client, toLower(hmp[1]),1) YIELD count AS clientHitMissCount
          CALL incrementCounter(asset, "count",1) YIELD count AS assetCount
          CALL incrementCounter(asset, toLower(hmp[1]),1) YIELD count AS assetHitMissCount
          CALL incrementCounter(asn, "count",1) YIELD count AS asnCount
          CALL incrementCounter(asn, toLower(hmp[1]),1) YIELD count AS asnHitMissCount
          CALL incrementCounter(server, "count",1) YIELD count AS serverCount
          CALL incrementCounter(server, toLower(hmp[1]),1) YIELD count AS serverHitMissCount
          CALL incrementCounter(pop, "count",1) YIELD count AS popCount
          CALL incrementCounter(pop, toLower(hmp[1]),1) YIELD count AS popHitMissCount
          CALL incrementCounter(clientGeo, "count",1) YIELD count AS clientGeoCount
          CALL incrementCounter(clientGeo, toLower(hmp[1]),1) YIELD count AS clientGeoHitMissCount
          CALL incrementCounter(origin, "count",1) YIELD count AS originGeoCount
          CALL incrementCounter(origin, toLower(hmp[1]),1) YIELD count AS originGeoHitMissCount

          ////////////////////////////////////////////////////////
          // Event
          ////////////////////////////////////////////////////////
          SET event = $that,
              event.cache_class = hmp[1],
              event: event

          ////////////////////////////////////////////////////////
          // Origin
          ////////////////////////////////////////////////////////
          SET origin.backend_ip = $that.backend_ip,
              origin: origin

          ////////////////////////////////////////////////////////
          // Client
          ////////////////////////////////////////////////////////
          SET client.client_geo_country = $that.client_geo_country,
              client.client_ip = $that.client_ip,
              client.user_agent = $that.user_agent,
              client: client

          // Extract Browser and Version
          // RegEx here: https://regex101.com/r/T0MThZ/2
          WITH *, text.regexFirstMatch($that.user_agent, '\\((.*?)\\)(\\s|$)|(.*?)\\/(.*?)(\\s|$)') AS cb
          SET client.browser = cb[3],
              client.browserVer = cb[4],
              client.first_seen = coll.min([$that.timestamp, coalesce(client.first_seen, $that.timestamp)]),
              client.last_seen = coll.max([$that.timestamp, coalesce(client.last_seen, $that.timestamp)])

          ////////////////////////////////////////////////////////
          // Client Geo
          ////////////////////////////////////////////////////////
          SET clientGeo.client_geo_country = $that.client_geo_country,
              clientGeo: clientGeo

          ////////////////////////////////////////////////////////
          // Asset
          ////////////////////////////////////////////////////////
          // RegEx here: https://regex101.com/r/tB8cd4/1
          WITH *, text.regexFirstMatch($that.path, '^(.+\\/)([^\\/]+)$') AS ap
          SET asset.path = ap[1],
              asset.name = ap[2],
              asset.full_path = $that.path,
              asset.if_modified_since = coll.max([$that.timestamp, coalesce(asset.if_modified_since, $that.timestamp)]),
              asset: asset

          ////////////////////////////////////////////////////////
          // ASN
          ////////////////////////////////////////////////////////
          SET asn.asn_id = toString($that.client_asn),
              asn: asn

          ////////////////////////////////////////////////////////
          // Server
          ////////////////////////////////////////////////////////
          SET server.server_id = $that.server_id,
              server.server_ip = $that.server_ip,
              server.cache_shield = $that.cache_shield,
              server.environment = $that.environment,
              server.host = $that.host,
              server.role = $that.role,
              server.pop = $that.pop,
              server: server

          ////////////////////////////////////////////////////////
          // PoP
          ////////////////////////////////////////////////////////
          SET pop.source = $that.pop,
              pop.environment = $that.environment,
              pop: pop

          ////////////////////////////////////////////////////////
          // Create relationship between nodes
          ////////////////////////////////////////////////////////
          CREATE (asset)<-[:REQUESTED]-(event)-[:REQUESTED_OVER]->(asn)-[:IN_CLIENT_GEO]->(clientGeo),
                 (origin)<-[:FROM]-(pop)<-[:WITHIN]-(server)<-[:TARGETED]-(event)<-[:ORIGINATED]-(client)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "cdn_data_50k.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (event), (client), (asset), (asn), (server), (pop), (origin), (clientGeo) WHERE $that.cache_status IS NOT NULL AND id(event) = idFrom('event', $that.timestamp, $that.request_id) AND id(client) = idFrom('client', $that.client_ip, $that.business_unit) AND id(asset) = idFrom('asset', $that.path) AND id(asn) = idFrom('asn', toString($that.client_asn)) AND id(server) = idFrom('server', $that.pop, $that.server_id) AND id(pop) = idFrom('pop', $that.pop) AND id(origin) = idFrom('origin', $that.backend_ip) AND id(clientGeo) = idFrom('clientGeo', $that.client_geo_country) WITH *, text.regexFirstMatch($that.cache_status, '(HIT|MISS(?!.*HIT)).*') AS hmp WHERE hmp[1] IS NOT NULL CALL incrementCounter(client, \"count\",1) YIELD count AS clientCount CALL incrementCounter(client, toLower(hmp[1]),1) YIELD count AS clientHitMissCount CALL incrementCounter(asset, \"count\",1) YIELD count AS assetCount CALL incrementCounter(asset, toLower(hmp[1]),1) YIELD count AS assetHitMissCount CALL incrementCounter(asn, \"count\",1) YIELD count AS asnCount CALL incrementCounter(asn, toLower(hmp[1]),1) YIELD count AS asnHitMissCount CALL incrementCounter(server, \"count\",1) YIELD count AS serverCount CALL incrementCounter(server, toLower(hmp[1]),1) YIELD count AS serverHitMissCount CALL incrementCounter(pop, \"count\",1) YIELD count AS popCount CALL incrementCounter(pop, toLower(hmp[1]),1) YIELD count AS popHitMissCount CALL incrementCounter(clientGeo, \"count\",1) YIELD count AS clientGeoCount CALL incrementCounter(clientGeo, toLower(hmp[1]),1) YIELD count AS clientGeoHitMissCount CALL incrementCounter(origin, \"count\",1) YIELD count AS originGeoCount CALL incrementCounter(origin, toLower(hmp[1]),1) YIELD count AS originGeoHitMissCount SET event = $that, event.cache_class = hmp[1], event: event SET origin.backend_ip = $that.backend_ip, origin: origin SET client.client_geo_country = $that.client_geo_country, client.client_ip = $that.client_ip, client.user_agent = $that.user_agent, client: client WITH *, text.regexFirstMatch($that.user_agent, '\\\\((.*?)\\\\)(\\\\s|$)|(.*?)\\\\/(.*?)(\\\\s|$)') AS cb SET client.browser = cb[3], client.browserVer = cb[4], client.first_seen = coll.min([$that.timestamp, coalesce(client.first_seen, $that.timestamp)]), client.last_seen = coll.max([$that.timestamp, coalesce(client.last_seen, $that.timestamp)]) SET clientGeo.client_geo_country = $that.client_geo_country, clientGeo: clientGeo WITH *, text.regexFirstMatch($that.path, '^(.+\\\\/)([^\\\\/]+)$') AS ap SET asset.path = ap[1], asset.name = ap[2], asset.full_path = $that.path, asset.if_modified_since = coll.max([$that.timestamp, coalesce(asset.if_modified_since, $that.timestamp)]), asset: asset SET asn.asn_id = toString($that.client_asn), asn: asn SET server.server_id = $that.server_id, server.server_ip = $that.server_ip, server.cache_shield = $that.cache_shield, server.environment = $that.environment, server.host = $that.host, server.role = $that.role, server.pop = $that.pop, server: server SET pop.source = $that.pop, pop.environment = $that.environment, pop: pop CREATE (asset)<-[:REQUESTED]-(event)-[:REQUESTED_OVER]->(asn)-[:IN_CLIENT_GEO]->(clientGeo),(origin)<-[:FROM]-(pop)<-[:WITHIN]-(server)<-[:TARGETED]-(event)<-[:ORIGINATED]-(client)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: cdn-file-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Json
        query: |-
          MATCH (event), (client), (asset), (asn), (server), (pop), (origin), (clientGeo)
          WHERE $that.cache_status IS NOT NULL
            AND id(event) = idFrom('event', $that.timestamp, $that.request_id)
            AND id(client) = idFrom('client', $that.client_ip, $that.business_unit)
            AND id(asset) = idFrom('asset', $that.path)
            AND id(asn) = idFrom('asn', toString($that.client_asn))
            AND id(server) = idFrom('server', $that.pop, $that.server_id)
            AND id(pop) = idFrom('pop', $that.pop)
            AND id(origin) = idFrom('origin', $that.backend_ip)
            AND id(clientGeo) = idFrom('clientGeo', $that.client_geo_country)

          WITH *, text.regexFirstMatch($that.cache_status, '(HIT|MISS(?!.*HIT)).*') AS hmp WHERE hmp[1] IS NOT NULL

          CALL incrementCounter(client, "count",1) YIELD count AS clientCount
          CALL incrementCounter(client, toLower(hmp[1]),1) YIELD count AS clientHitMissCount
          // ... (additional counter calls)

          SET event = $that,
              event.cache_class = hmp[1],
              event: event
          // ... (additional SET clauses)

          CREATE (asset)<-[:REQUESTED]-(event)-[:REQUESTED_OVER]->(asn)-[:IN_CLIENT_GEO]->(clientGeo),
                 (origin)<-[:FROM]-(pop)<-[:WITHIN]-(server)<-[:TARGETED]-(event)<-[:ORIGINATED]-(client)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "cdn-file-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (event), (client), (asset), (asn), (server), (pop), (origin), (clientGeo) WHERE $that.cache_status IS NOT NULL AND id(event) = idFrom('event', $that.timestamp, $that.request_id) ... CREATE (asset)<-[:REQUESTED]-(event)-[:REQUESTED_OVER]->(asn)-[:IN_CLIENT_GEO]->(clientGeo),(origin)<-[:FROM]-(pop)<-[:WITHIN]-(server)<-[:TARGETED]-(event)<-[:ORIGINATED]-(client)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to look for 10 consecutive cache MISS events involving the same server and asset pair within a defined duration.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: |-
            MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:"MISS"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:"MISS"})-[:TARGETED]->(server2:server)

            RETURN DISTINCT id(event1) AS event1
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:\"MISS\"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:\"MISS\"})-[:TARGETED]->(server2:server) RETURN DISTINCT id(event1) AS event1"
      },
      "outputs": {
        "cacheMissAlert": {
          "type": "CypherQuery",
          "query": "...",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: cache-miss-alert
        pattern:
          type: Cypher
          query: |-
            MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:"MISS"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:"MISS"})-[:TARGETED]->(server2:server)
            RETURN DISTINCT id(event1) AS event1
        outputs:
          - name: cacheMissAlert
            resultEnrichment:
              query: |-
                MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:"MISS"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:"MISS"})-[:TARGETED]->(server2:server)
                WHERE id(event1) = $that.data.event1
                  AND duration("PT45M") > duration.between(localdatetime(event1.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS"), localdatetime(event2.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS")) > duration("PT5M")
                  AND event1.client_asn = event2.client_asn
                  AND id(server1) = id(server2)
                  AND id(event1) <> id(event2)
                // ... additional processing
                RETURN 'http://localhost:8080/#...' AS Alert
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "cache-miss-alert",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:\"MISS\"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:\"MISS\"})-[:TARGETED]->(server2:server) RETURN DISTINCT id(event1) AS event1"
      },
      "outputs": [
        {
          "name": "cacheMissAlert",
          "resultEnrichment": {
            "query": "MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:\"MISS\"})... RETURN Alert",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

Once Quine detects the pattern, the event is sent to a standing query output for additional processing and action.

```yaml
    outputs:
      cacheMissAlert:
        type: CypherQuery
        query: |-
          query: |-
            // Add constraints to the cache MISS events match involving the same server and asset pair.
            MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:"MISS"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:"MISS"})-[:TARGETED]->(server2:server)
            WHERE id(event1) = $that.data.event1
              // Time between consecutive cache MISSes between 5-45 minutes expressed in ISO 8601 duration format (https://en.wikipedia.org/wiki/ISO_8601#Durations)
              // Feel free to alter the range to meet your requirements
              AND duration("PT45M") > duration.between(localdatetime(event1.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS"), localdatetime(event2.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS")) > duration("PT5M")
              AND event1.client_asn = event2.client_asn
              AND id(server1) = id(server2)
              AND id(event1) <> id(event2)

            ////////////////////////////////////////////////////////
            // missEvents
            ////////////////////////////////////////////////////////
            // Manifest missEvents node to track metadata relative to consecutive cache MISSes that match the previous constraints
            MATCH (missEvents) 
            WHERE id(missEvents) = idFrom('missEvents', server1.server_id, asset.full_path)
            SET missEvents.asset = event1.path, 
                missEvents.server = event1.server_id, 
                missEvents.pop = event1.pop, 
                missEvents.firstMiss = coll.min([event1.timestamp, coalesce(missEvents.firstMiss, event1.timestamp)]), 
                missEvents.latestMiss = coll.max([event1.timestamp, coalesce(missEvents.latestMiss, event1.timestamp)]), 
                missEvents: missEvents

            // Create subgraph from consecutive cache MISS events to provide a visualization in the Quine Exploration UI
            CREATE (asset)-[:HAD]->(missEvents)-[:FROM]->(server1)<-[:TARGETED]-(event1),
                   (server1)<-[:TARGETED]-(event2)

            // Increment the missEvents counter for the purpose of triggering an alert at a specified threshold
            WITH missEvents CALL incrementCounter(missEvents, "cumulativeCount", 1) YIELD count AS cumulativeCount

            // Trigger alert (RETURN clause) that prints URL to local running Quine instance
            MATCH (missEvents)
            // Threshold at which to emit alert
            // Feel free to alter it to meet your requirements
            WHERE missEvents.cumulativeCount = 10
            RETURN 'http://localhost:8080/#' + text.urlencode('MATCH(missEvents:missEvents) WHERE id(missEvents)="' + toString(strId(missEvents)) + '" MATCH (event {cache_class:"MISS"})-[:TARGETED]->(server)<-[:FROM]-(missEvents)<-[:HAD]-(asset)<-[:REQUESTED]-(event {cache_class:"MISS"}) RETURN DISTINCT missEvents, event, server, asset LIMIT 10') AS Alert
        andThen:
          type: PrintToStandardOut

```

```yaml
    outputs:
      - name: cacheMissAlert
        resultEnrichment:
          query: |-
            // Add constraints to the cache MISS events match involving the same server and asset pair.
            MATCH (server1:server)<-[:TARGETED]-(event1 {cache_class:"MISS"})-[:REQUESTED]->(asset)<-[:REQUESTED]-(event2 {cache_class:"MISS"})-[:TARGETED]->(server2:server)
            WHERE id(event1) = $that.data.event1
              // Time between consecutive cache MISSes between 5-45 minutes expressed in ISO 8601 duration format (https://en.wikipedia.org/wiki/ISO_8601#Durations)
              // Feel free to alter the range to meet your requirements
              AND duration("PT45M") > duration.between(localdatetime(event1.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS"), localdatetime(event2.timestamp, "yyyy-MM-dd HH:mm:ss.SSSSSS")) > duration("PT5M")
              AND event1.client_asn = event2.client_asn
              AND id(server1) = id(server2)
              AND id(event1) <> id(event2)

            ////////////////////////////////////////////////////////
            // missEvents
            ////////////////////////////////////////////////////////
            // Manifest missEvents node to track metadata relative to consecutive cache MISSes that match the previous constraints
            MATCH (missEvents) 
            WHERE id(missEvents) = idFrom('missEvents', server1.server_id, asset.full_path)
            SET missEvents.asset = event1.path, 
                missEvents.server = event1.server_id, 
                missEvents.pop = event1.pop, 
                missEvents.firstMiss = coll.min([event1.timestamp, coalesce(missEvents.firstMiss, event1.timestamp)]), 
                missEvents.latestMiss = coll.max([event1.timestamp, coalesce(missEvents.latestMiss, event1.timestamp)]), 
                missEvents: missEvents

            // Create subgraph from consecutive cache MISS events to provide a visualization in the Quine Exploration UI
            CREATE (asset)-[:HAD]->(missEvents)-[:FROM]->(server1)<-[:TARGETED]-(event1),
                   (server1)<-[:TARGETED]-(event2)

            // Increment the missEvents counter for the purpose of triggering an alert at a specified threshold
            WITH missEvents CALL incrementCounter(missEvents, "cumulativeCount", 1) YIELD count AS cumulativeCount

            // Trigger alert (RETURN clause) that prints URL to local running Quine instance
            MATCH (missEvents)
            // Threshold at which to emit alert
            // Feel free to alter it to meet your requirements
            WHERE missEvents.cumulativeCount = 10
            RETURN 'http://localhost:8080/#' + text.urlencode('MATCH(missEvents:missEvents) WHERE id(missEvents)="' + toString(strId(missEvents)) + '" MATCH (event {cache_class:"MISS"})-[:TARGETED]->(server)<-[:FROM]-(missEvents)<-[:HAD]-(asset)<-[:REQUESTED]-(event {cache_class:"MISS"}) RETURN DISTINCT missEvents, event, server, asset LIMIT 10') AS Alert
          parameter: that
        destinations:
          - type: StandardOut

```

The result once the pattern is detected is to output a link to the console that an analyst can use to review the event further within Quine's Exploration UI.

```json
2023-02-03 16:00:43,345 Standing query `cacheMissAlert` match: {"meta":{"isPositiveMatch":true,"resultId":"0e38c93e-338c-e964-8867-8487eb083e5b"},"data":{"Alert":"http://localhost:8080/#MATCH%28missEvents%3AmissEvents%29%20WHERE%20id%28missEvents%29%3D%2263c2f862-ea0f-3a3a-9f4a-09b11f176ad0%22%20MATCH%20%28event%20%7Bcache_class%3A%22MISS%22%7D%29-%5B%3ATARGETED%5D-%3E%28server%29%3C-%5B%3AFROM%5D-%28missEvents%29%3C-%5B%3AHAD%5D-%28asset%29%3C-%5B%3AREQUESTED%5D-%28event%20%7Bcache_class%3A%22MISS%22%7D%29%20RETURN%20DISTINCT%20missEvents%2C%20event%2C%20server%2C%20asset%20LIMIT%2010"}}
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r cdn.yaml --recipe-value in_file=cdn_data_50k.json
Graph is ready
Running Recipe: CDN Cache Efficiency By Segment
Using 11 node appearances
Using 14 quick queries
Using 9 sample queries
2023-02-03 17:05:00,342 WARN [NotFromActor] [graph-service-akka.quine.graph-shard-dispatcher-18] com.thatdot.quine.app.StandingQueryResultOutput$ - Could not verify that the provided Cypher query is idempotent. If timeouts or external system errors occur, query execution may be retried and duplicate data may be created. To avoid this, set shouldRetry = false in the Standing Query output
Running Standing Query STANDING-1
2023-02-03 17:05:00,847 WARN [NotFromActor] [graph-service-akka.quine.graph-shard-dispatcher-18] com.thatdot.quine.app.ingest.serialization.CypherJsonInputFormat - Could not verify that the provided ingest query is idempotent. If timeouts occur, query execution may be retried and duplicate data may be created.
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080

 | => STANDING-1 count 4248
 | => INGEST-1 status is running and ingested 5560
```

## Summary

When the standing query detects the cache miss pattern, it will output a link to the console that can be copied and pasted into a browser to explore the event in the Quine Exploration UI. Copy and paste the URL section of the match JSON from your console into your browser.

The nodes will be jumbled together when you first open the graph. Arrange the nodes to look similar to the image below before you start exploring.

![cdnCacheMissSqMatch.png](images/cdnCacheMissSqMatch.png)

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query                   | Node Type   | Description                                           |
| ----------------------------- | ----------- | ----------------------------------------------------- |
| Adjacent Nodes                | All         | Display the nodes that are adjacent to this node.     |
| Refresh                       | All         | Refresh the content stored in a node                  |
| Local Properties              | All         | Display the properties stored by the node             |
| Reset Counter                 | missedEvent | Deletes the missedEvent                               |
| Server Pop                    | server      | Displays the associated PoP                           |
| Cache Hit/Miss Percentage     | server      | Calculates the Hit/Miss percentages for the server    |
| PoP Hit/Miss Percentage       | pop         | Calculates the Hit/Miss percentages for the PoP       |
| PoP Origins                   | pop         | Displays the asset origins the PoP is serving         |
| Origin Hit/Miss Percentage    | origin      | Calculates the Hit/Miss percentages for the origin    |
| Client Hit/Miss Percentage    | client      | Calculates the Hit/Miss percentages for the client    |
| clientGeo Hit/Miss Percentage | clientGeo   | Calculates the Hit/Miss percentages for the clientGeo |
| Asset Hit/Miss Percentage     | asset       | Calculates the Hit/Miss percentages for the asset     |
| Client Geo                    | asn         | Displays the clientGeo associated with the ASN        |
| ASN Hit/Miss Percentage       | asn         |                                                       |


---
# Certstream Firehose
URL: https://quine.io/recipes/certstream-firehose/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Reproduces the behavior of the [certstream website](https://certstream.calidog.io/) by connecting to the certstream firehose via SSL-encrypted websocket and printing to standard out each time a new certificate is detected.

    ??? example "Certstream Firehose Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/certstream-firehose.yaml"
        ```

    [Download Recipe](assets/certstream-firehose.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

=== "Recipe v2"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Reproduces the behavior of the [certstream website](https://certstream.calidog.io/) by connecting to the certstream firehose via SSL-encrypted websocket and printing to standard out each time a new certificate is detected.

    ??? example "Certstream Firehose Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/certstream-firehose.yaml"
        ```

    [Download Recipe](assets/v2/certstream-firehose.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

## Scenario

CertStream is an intelligence feed that gives you real-time updates from the [Certificate Transparency Log](https://www.certificate-transparency.org/what-is-ct) network, allowing you to use it as a building block to make tools that react to new certificates being issued in real time.

This recipe connects to the curated public [Certstream](https://certstream.calidog.io/) aggregation service managed by the team at [Cali Dog Security](https://calidog.io/).

## Sample Data

This recipe connects to the live Certstream feed eliminating the need for sample data. However, below is a typical raw certificate update object for review.

``` json
  "data": {
    "cert_index": 160270422,
    "cert_link": "https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=160270422&end=160270422",
    "leaf_cert": {
      "all_domains": [
        "*.nyarkowiz.online",
        "nyarkowiz.online"
      ],
      "extensions": {
        "authorityInfoAccess": "CA Issuers - URI:http://pki.goog/repo/certs/gts1p5.der\nOCSP - URI:http://ocsp.pki.goog/s/gts1p5/fKV4K079ZKo\n",
        "authorityKeyIdentifier": "keyid:D5:FC:9E:0D:DF:1E:CA:DD:08:97:97:6E:2B:C5:5F:C5:2B:F5:EC:B8\n",
        "basicConstraints": "CA:FALSE",
        "certificatePolicies": "Policy: 1.3.6.1.4.1.11129.2.5.3\nPolicy: 2.23.140.1.2.1",
        "crlDistributionPoints": "Full Name:\n URI:http://crls.pki.goog/gts1p5/oE9rr3G5TqE.crl",
        "ctlPoisonByte": true,
        "extendedKeyUsage": "TLS Web server authentication",
        "keyUsage": "Digital Signature, Key Encipherment",
        "subjectAltName": "DNS:nyarkowiz.online, DNS:*.nyarkowiz.online",
        "subjectKeyIdentifier": "42:22:E3:A5:27:CB:93:B1:8F:C0:20:7C:CB:E6:11:ED:B3:A4:CB:BD"
      },
      "fingerprint": "B1:FE:F6:4C:D1:7E:A3:DB:A8:D9:92:EE:18:42:B7:1F:35:2F:75:68",
      "issuer": {
        "C": "US",
        "CN": "GTS CA 1P5",
        "L": null,
        "O": "Google Trust Services LLC",
        "OU": null,
        "ST": null,
        "aggregated": "/C=US/CN=GTS CA 1P5/O=Google Trust Services LLC",
        "emailAddress": null
      },
      "not_after": 1679365412,
      "not_before": 1671589413,
      "serial_number": "201DF51E883B4B37139BBB17CAEACE15",
      "signature_algorithm": "sha256, rsa",
      "subject": {
        "C": null,
        "CN": "*.nyarkowiz.online",
        "L": null,
        "O": null,
        "OU": null,
        "ST": null,
        "aggregated": "/CN=*.nyarkowiz.online",
        "emailAddress": null
      }
    },
    "seen": 1671638236.908937,
    "source": {
      "name": "DigiCert Nessie2023 Log",
      "url": "https://nessie2023.ct.digicert.com/log/"
    },
    "update_type": "PrecertLogEntry"
  },
  "message_type": "certificate_update"
}
```

## How it Works

The recipe is designed to rapidly load JSON objects into Quine producing as disconnected nodes. The [ingest stream](../learn/ingest-sources/index.md) connects using a `WebsocketClient` source type and parses each record as a JSON object accessible via `$that` in the Cypher query. 

INGEST-1 reads directly from the certstream web socket:

=== "YAML"

    ```yaml
        - type: WebsocketSimpleStartupIngest
          url: wss://certstream.calidog.io/
          format:
            type: CypherJson
            query: |-
              CREATE ($that)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "WebsocketSimpleStartupIngest",
      "url": "wss://certstream.calidog.io/",
      "format": {
        "type": "CypherJson",
        "query": "CREATE ($that)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: certstream-ingest
        source:
          type: WebsocketClient
          url: wss://certstream.calidog.io/
          format:
            type: Json
          initMessages: []
          characterEncoding: UTF-8
        query: |-
          CREATE ($that)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "certstream-ingest",
      "source": {
        "type": "WebsocketClient",
        "url": "wss://certstream.calidog.io/",
        "format": {
          "type": "Json"
        },
        "initMessages": [],
        "characterEncoding": "UTF-8"
      },
      "query": "CREATE ($that)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect new nodes in the graph and then print the event to the console.

=== "YAML"

    ```yaml
      - pattern:
          type: Cypher
          query: MATCH (n) RETURN DISTINCT id(n) AS id
        outputs:
          log-new-certs:
            type: CypherQuery
            query: |-
              MATCH (n)
              WHERE id(n) = $that.data.id
              RETURN n.data
            andThen:
              type: PrintToStandardOut
              logMode: FastSampling
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n) RETURN DISTINCT id(n) AS id"
      },
      "outputs": {
        "log-new-certs": {
          "type": "CypherQuery",
          "query": "MATCH (n)\nWHERE id(n) = $that.data.id\nRETURN n.data",
          "andThen": {
            "type": "PrintToStandardOut",
            "logMode": "FastSampling"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: log-new-certs
        pattern:
          type: Cypher
          query: MATCH (n) RETURN DISTINCT id(n) AS id
          mode: DISTINCT_ID
        outputs:
          - name: log-new-certs
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (n)
                WHERE id(n) = $that.id
                RETURN n.data AS data
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "log-new-certs",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n) RETURN DISTINCT id(n) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "log-new-certs",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (n) WHERE id(n) = $that.id RETURN n.data AS data",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

The recipe will stream events to the console similar to the sample event below.

``` { .json linenums="1" }
2022-12-21 10:32:54,863 Standing query `log-new-certs` match: {"meta":{"isPositiveMatch":true,"resultId":"e8709166-1a08-df62-3419-e030ce81d09a"},"data":{"n.data":{"cert_index":545387302,"cert_link":"https://ct.googleapis.com/logs/xenon2023/ct/v1/get-entries?start=545387302&end=545387302","leaf_cert":{"all_domains":["www.gamificationbook.com"],"extensions":{"authorityInfoAccess":"CA Issuers - URI:http://r3.i.lencr.org/\nOCSP - URI:http://r3.o.lencr.org\n","authorityKeyIdentifier":"keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6\n","basicConstraints":"CA:FALSE","certificatePolicies":"Policy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.letsencrypt.org","ctlPoisonByte":true,"extendedKeyUsage":"TLS Web server authentication, TLS Web client authentication","keyUsage":"Digital Signature, Key Encipherment","subjectAltName":"DNS:www.gamificationbook.com","subjectKeyIdentifier":"89:C8:C3:71:33:36:E7:37:BF:78:08:81:2F:4E:C7:74:DE:EF:9F:60"},"fingerprint":"D7:86:C7:29:54:AF:91:DF:DA:30:9D:8A:0A:AE:B0:17:1C:B6:F1:F8","issuer":{"C":"US","CN":"R3","L":null,"O":"Let's Encrypt","OU":null,"ST":null,"aggregated":"/C=US/CN=R3/O=Let's Encrypt","emailAddress":null},"not_after":1679412699,"not_before":1671636700,"serial_number":"3FFE0C620988AD6607BBFAB008D769E8BBF","signature_algorithm":"sha256, rsa","subject":{"C":null,"CN":"www.gamificationbook.com","L":null,"O":null,"OU":null,"ST":null,"aggregated":"/CN=www.gamificationbook.com","emailAddress":null}},"seen":1.671640374636495E9,"source":{"name":"Google 'Xenon2023' log","url":"https://ct.googleapis.com/logs/xenon2023/"},"update_type":"PrecertLogEntry"}}}
```

## Running the Recipe

``` shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r certstream-firehose.yaml
Graph is ready
Running Recipe: Certstream Firehose
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
```


---
# Conway's Game of Life
URL: https://quine.io/recipes/conways-gol/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Matthew Cullum](https://github.com/brackishman)</small>

    A complete implementation of [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) using Quine's standing queries to create a real-time cellular automaton. This recipe demonstrates that Quine is Turing complete by leveraging standing query recursion to evolve cell states across generations according to Conway's famous rules.

    ??? example "Conway's Game of Life Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/conways-gol.yaml"
        ```

    [Download Recipe](assets/conways-gol.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Matthew Cullum](https://github.com/brackishman)</small>

    A complete implementation of [Conway's Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) using Quine's standing queries to create a real-time cellular automaton. This recipe demonstrates that Quine is Turing complete by leveraging standing query recursion to evolve cell states across generations according to Conway's famous rules.

    ??? example "Conway's Game of Life Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/conways-gol.yaml"
        ```

    [Download Recipe](assets/v2/conways-gol.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

Conway's Game of Life is a classic cellular automaton invented by mathematician John Conway in 1970. Despite its simple rules, it can produce remarkably complex patterns and behaviors. This recipe implements the complete Game of Life in Quine, where each cell is a node in the graph that evaluates its neighbors and updates its state according to Conway's rules:

1. A live cell with 2-3 live neighbors survives
2. A dead cell with exactly 3 live neighbors becomes alive
3. All other cells die or stay dead

The implementation uses a two-wave standing query pattern to compute and apply state changes across all cells simultaneously, demonstrating Quine's capability for recursive, real-time graph computation.

## Sample Configurations

This recipe requires a configuration file that defines the grid size and initial pattern of alive cells. Each configuration is a JSON file specifying the grid dimensions and which cells start alive:

```json
{
  "name": "Small Conway's Game - Blinker Pattern",
  "description": "7x7 grid with a simple blinker pattern in the center",
  "gridWidth": 7,
  "gridHeight": 7,
  "initialPattern": [
    {"x": 3, "y": 2, "alive": true},
    {"x": 3, "y": 3, "alive": true},
    {"x": 3, "y": 4, "alive": true}
  ]
}
```

Three pre-configured patterns are included with the recipe:

### Blinker Pattern (Recommended for first run)

A simple 7x7 grid featuring a "blinker", a pattern that oscillates between horizontal and vertical orientations every generation. This is perfect for understanding the basic mechanics.

[Download blinker.json](assets/GoL/blinker/blinker.json){ .md-button download="" }

### Gosper Glider Gun

The famous 40x15 grid containing the original Gosper Glider Gun discovered in 1970. This pattern produces new gliders every 30 generations, demonstrating emergent complexity from simple rules.

[Download glider-gun.json](assets/GoL/glider-gun/glider-gun.json){ .md-button download="" }

### Explosion Pattern

A more chaotic initial configuration that creates dynamic, unpredictable evolution across the grid.

[Download explosion.json](assets/GoL/explosion/explosion.json){ .md-button download="" }

## How it Works

This recipe demonstrates **Quine's Turing completeness** through recursive standing queries that continuously evaluate and evolve the cellular automaton.

### Graph Structure

The recipe creates a graph where:

- Each **cell** is represented as a node with properties: `x`, `y`, `alive`, `generation`, and `state`
- Cells are connected via `[:NEIGHBOR]` relationships to their 8 adjacent cells
- A central **ready** node coordinates the computation waves and tracks the current generation

### Two-Wave Standing Query Pattern

The recipe uses a sophisticated two-phase approach to ensure all cells update simultaneously:

**Wave 1: Compute Next State**

Standing queries detect when all cells are ready to compute their next state. Each cell:

1. Counts its live neighbors
2. Applies Conway's rules to determine if it should be alive in the next generation
3. Stores the result in `nextAlive` without changing its current state

**Wave 2: Apply State Changes**

Once all cells have computed their next state, a second wave of standing queries:

1. Updates each cell's `alive` property to the computed `nextAlive` value
2. Marks cells that changed as `updated`
3. Increments the generation counter

**Wave Coordination**

Standing queries monitor the ready node to coordinate the waves:

- When Wave 1 completes → Start Wave 2
- When Wave 2 completes → Start next generation's Wave 1

This recursive pattern continues indefinitely, evolving the grid through successive generations. The standing queries act as the "rules engine" that recursively applies Conway's rules, demonstrating that Quine's standing query mechanism is Turing complete.

### Visual Configuration

The recipe includes node appearances that automatically style cells:

- **Live cells**: Large orange circles (●)
- **Dead cells**: Small gray circles (○)

## Prerequisites: Install the Bookmarklet

Before running the recipe, you need to install a browser bookmarklet that enables visualization of the Game of Life animation. The bookmarklet performs two critical functions:

1. **Enables unlimited node rendering** by automatically bypassing the browser's node limit prompts
2. **Monitors generation updates** by connecting to Quine's standing query WebSocket and automatically refreshing the view to show updated cells

Without the bookmarklet, you would need to manually approve rendering hundreds of nodes and manually refresh the query after each generation, making the animation impossible to watch in real-time.

### Installation Steps

1. Download the bookmarklet JavaScript file:

    [Download conways-gol-bookmarklet.js](assets/GoL/conways-gol-bookmarklet.js){ .md-button download="" }

2. Open the downloaded file in a text editor and copy the entire JavaScript code
3. Create a new bookmark in your browser (usually ++ctrl+d++ or ++cmd+d++)
4. Edit the bookmark and paste the JavaScript code as the bookmark URL
5. Name the bookmark "Quine GoL Monitor" (or any name you prefer)

Once Quine is running and you have loaded the cell nodes in the Exploration UI, click the bookmarklet in your browser's bookmark bar. You should see an alert confirming "Now monitoring Conway's Game of Life generations with unlimited node rendering..."

!!! warning "Bookmarklet Required"
    The bookmarklet must be activated before starting the game, or you will not see the animation. If you forget to activate it, simply click the bookmarklet and restart the game.

## Running the Recipe

### Step 1: Start Quine with the Recipe

Download one of the sample configurations (we'll use `blinker.json` for this walkthrough) and start Quine. Make sure the configuration file is in the same directory as your Quine JAR file, or provide the correct relative path:

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r conways-gol.yaml --recipe-value config_file=blinker.json
Graph is ready
Running Recipe: Conway's Game of Life
Using 2 node appearances
Using 2 quick queries
Using 1 sample query
Running Standing Query STANDING-1
Running Standing Query STANDING-2
Running Standing Query STANDING-3
Running Standing Query STANDING-4
Quine web server available at http://localhost:8080
```

!!! note
    The configuration file path is relative to where you run the `java` command. If you organize your files in subdirectories, adjust the path accordingly (e.g., `config_file=configs/blinker.json`).

### Step 2: Load the Cell Nodes

Open your browser to [http://localhost:8080](http://localhost:8080) and click the sample query **● Show All Cells** to load all cell nodes into the Exploration UI.

You should see all cells in the grid displayed as small gray circles (all dead initially) or a mix of orange (alive) and gray (dead) circles depending on your initial pattern.

![Initial Grid View](images/gol-initial-cells.png)

### Step 3: Load the Grid Layout

The cell nodes are currently displayed in a random arrangement. To visualize the Game of Life properly, you need to load a layout file that positions each cell at its correct x,y coordinates in a grid formation. Each configuration comes with a matching layout file that contains the precise coordinates for every cell node.

1. In the Exploration UI, click the layout dropdown menu (top right of the graph view)
2. Select "Load Layout from File"
3. Choose the corresponding layout file that matches your configuration (e.g., `blinker-layout.json` for the blinker configuration)

The cells will now be arranged in a proper grid formation, with each cell positioned at its x,y coordinate. Live cells will appear as large orange circles and dead cells as small gray circles.

![Grid With Layout Applied](images/gol-grid.png)

### Step 4: Activate the Bookmarklet

Click the bookmarklet you installed earlier in your browser's bookmark bar. You should see an alert confirming "Now monitoring Conway's Game of Life generations with unlimited node rendering..."

The bookmarklet is now connected to Quine and ready to automatically refresh the view as cells change.

### Step 5: Start the Game

Right-click on any cell node and select the **▶️ START Game** quick query.

![Start Game Quick Query](images/gol-context-start.png)

The game will immediately begin evolving! You'll see:

- Cells changing color as they become alive or dead
- The generation counter incrementing rapidly
- Patterns evolving according to Conway's rules

For the blinker pattern, you'll observe the three cells oscillating between horizontal and vertical orientations.

![Blinker Pattern Running](images/gol-running.png)

### Step 6: Stop the Game

To pause the evolution, right-click any cell and select the **⏸️ STOP Game** quick query.

### View Configuration Details

You can check the current game configuration and statistics at any time with the **📊 Show Game Configuration** sample query. Hold ++shift++ while clicking the sample query to view the results as tabular data rather than updating the exploration canvas.

```cypher
MATCH (ready) WHERE id(ready) = idFrom("ready")
MATCH (c:Cell)
RETURN
  ready.name AS setup,
  ready.description AS description,
  ready.gridWidth AS width,
  ready.gridHeight AS height,
  ready.totalCells AS totalCells,
  count(CASE WHEN c.alive = true THEN 1 END) AS liveCells,
  ready.generation AS currentGeneration
```

This displays the grid size, total cells, live cell count, and current generation as a table.

![Game Configuration Results](images/gol-config-query.png)

## Trying Other Patterns

To run different Game of Life patterns:

1. Stop the current game with the **⏸️ STOP Game** quick query
2. Shut down Quine by either typing ++ctrl+c++ in the terminal window or issuing a [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown):

    ```shell
    curl -X "POST" "http://127.0.0.1:8080/api/v2/system:shutdown"
    ```

3. Restart Quine with a different configuration file:

```shell
❯ java -jar quine-2.1.1.jar -r conways-gol.yaml --recipe-value config_file=glider-gun.json
```

4. Load the cells using the **● Show All Cells** sample query
5. Apply the corresponding layout file (`glider-gun-layout.json`)
6. Activate the bookmarklet again
7. Start the game with the **▶️ START Game** quick query

The Gosper Glider Gun is particularly fascinating to watch as it continuously creates gliders that move across the grid.

## Performance Considerations

For larger grids (50x50 or more), you may notice the visualization slowing down. This is **not** a limitation of Quine's computation - the standing queries continue to process generations at high speed. The performance bottleneck is primarily the browser UI continuously querying and rendering hundreds of nodes in real-time.

The Quine graph continues to evolve rapidly even when the UI struggles to keep up with the visualization. You can verify this by checking the generation count, which will continue incrementing quickly regardless of grid size.

!!! tip "Performance Tip"
    For very large grids, consider periodically stopping the game to examine the current state rather than trying to watch continuous animation.

## Creating Custom Patterns

Advanced users can create their own Game of Life configurations and layouts:

### Custom Configuration

Create a JSON file following this schema:

```json
{
  "name": "My Custom Pattern",
  "description": "Description of your pattern",
  "gridWidth": 20,
  "gridHeight": 20,
  "initialPattern": [
    {"x": 10, "y": 10, "alive": true},
    {"x": 11, "y": 10, "alive": true}
  ]
}
```

### Generating Layout Files

The included Python script can generate layout JSON files for your custom configurations. This tool queries a running Quine instance to discover all cell nodes and their x,y coordinates, then generates a layout file that positions each node properly in a grid formation.

[Download generate-conways-layout.py](assets/GoL/generate-conways-layout.py){ .md-button download="" }

To use the layout generator:

1. Start Quine with your custom configuration
2. Load all cells into the Exploration UI using the **● Show All Cells** sample query
3. Run the Python script: `python generate-conways-layout.py`
4. The script generates a layout JSON file mapping each cell to its grid coordinates
5. Load this layout file in the Exploration UI

This tool is useful when creating new patterns, as it automatically calculates proper spacing and positioning for any grid size.

## Summary

This recipe demonstrates a fundamental computer science concept - Turing completeness - through an elegant implementation of Conway's Game of Life. By using standing queries that recursively evaluate and update cell states, we prove that Quine can perform arbitrary computation. The two-wave pattern ensures synchronized updates across the entire grid, while the recursive nature of the standing queries drives the continuous evolution of the cellular automaton.

The Game of Life is a perfect demonstration of emergent complexity from simple rules, and implementing it in Quine showcases the power of recursive graph computation.

!!! tip "Turing Completeness"
    The ability to implement Conway's Game of Life demonstrates that Quine's standing query system is Turing complete. Since Game of Life itself is Turing complete, and we've implemented it entirely through standing queries, this proves that standing queries can perform any computable function.


---
# Temporal Locality
URL: https://quine.io/recipes/duration/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    This recipe looks for emails sent or received by `cto@company.com` within a sliding window as a means of highlighting a technique for matching on temporal locality of nodes in standing queries.

    ??? example "Temporal Locality Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/duration.yaml"
        ```

    [Download Recipe](assets/duration.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    This recipe looks for emails sent or received by `cto@company.com` within a sliding window as a means of highlighting a technique for matching on temporal locality of nodes in standing queries.

    ??? example "Temporal Locality Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/duration.yaml"
        ```

    [Download Recipe](assets/v2/duration.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

This scenario processes records containing metadata for almost 295,000 emails are ingested for the purpose of identifying emails to/from a specific email address within a sliding 2-minute window.

``` json
{
    "from": <sender>,
    "to": [<recipients>],
    "subject": <email subject line>,
    "time": <epoch time timestamp>,
    "sequence": <sequence number>
}
```

## Sample Data

Download the sample data to the same directory as the recipe and where Quine will be run.

[Download email.json](https://quine-recipe-public.s3.us-west-2.amazonaws.com/duration/email.json)

## How it Works

The recipe reads data from a sample data file using an [ingest stream](../learn/ingest-sources/index.md) and parses each line into sender, receiver and message nodes to manifest a graph in Quine.

INGEST-1 processes the `email.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: email.json
        format:
            type: CypherJson
            query: |-
                MATCH (sender), (message)
                WHERE id(sender) = idFrom('email', $that.from)
                AND id(message) = idFrom('message', $that)

                SET sender.email = $that.from,
                    sender: Email,
                    message.from = $that.from,
                    message.to = $that.to,
                    message.subject = $that.subject,
                    message.time = datetime({epochMillis: $that.time}),
                    message: Message

                CREATE (sender)-[:SENT_MSG]->(message)

                WITH $that AS t, message
                UNWIND t.to AS rcv
                MATCH (receiver)
                WHERE id(receiver) = idFrom('email', rcv)

                SET receiver.email = rcv,
                    receiver: Email

                CREATE (message)-[:RECEIVED_MSG]->(receiver)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "email.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (sender), (message) WHERE id(sender) = idFrom('email', $that.from) AND id(message) = idFrom('message', $that) SET sender.email = $that.from, sender: Email, message.from = $that.from, message.to = $that.to, message.subject = $that.subject, message.time = datetime({ epochMillis: $that.time}), message: Message CREATE (sender)-[:SENT_MSG]->(message) WITH $that as t, message UNWIND t.to AS rcv MATCH (receiver) WHERE id(receiver) = idFrom('email', rcv) SET receiver.email = rcv, receiver: Email CREATE (message)-[:RECEIVED_MSG]->(receiver)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: email-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Json
        query: |-
          MATCH (sender), (message)
          WHERE id(sender) = idFrom('email', $that.from)
            AND id(message) = idFrom('message', $that)

          SET sender.email = $that.from,
              sender: Email,
              message.from = $that.from,
              message.to = $that.to,
              message.subject = $that.subject,
              message.time = datetime({ epochMillis: $that.time}),
              message: Message

          CREATE (sender)-[:SENT_MSG]->(message)

          WITH $that as t, message
          UNWIND t.to AS rcv
          MATCH (receiver)
          WHERE id(receiver) = idFrom('email', rcv)

          SET receiver.email = rcv,
              receiver: Email

          CREATE (message)-[:RECEIVED_MSG]->(receiver)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "email-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (sender), (message) WHERE id(sender) = idFrom('email', $that.from) AND id(message) = idFrom('message', $that) SET sender.email = $that.from, sender: Email, message.from = $that.from, message.to = $that.to, message.subject = $that.subject, message.time = datetime({ epochMillis: $that.time}), message: Message CREATE (sender)-[:SENT_MSG]->(message) WITH $that as t, message UNWIND t.to AS rcv MATCH (receiver) WHERE id(receiver) = idFrom('email', rcv) SET receiver.email = rcv, receiver: Email CREATE (message)-[:RECEIVED_MSG]->(receiver)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect when emails that are sent or received by `cto@company.com` are within a two minute sliding window of one another. 

The pattern query matches each individual `(sender)-[:SENT_MSG]->(message)-[:RECEIVED_MSG]->(receiver)` pattern.

```yaml
- pattern:
    type: Cypher
    mode: MultipleValues
    query: |-
        MATCH (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r)
        WHERE n.email="cto@company.com" OR r.email="cto@company.com"
        RETURN id(n) as ctoId, id(m) as ctoMsgId, m.time as mTime, id(r) as recId

```

```yaml
- pattern:
    type: Cypher
    mode: MULTIPLE_VALUES
    query: |-
        MATCH (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r)
        WHERE n.email="cto@company.com" OR r.email="cto@company.com"
        RETURN id(n) as ctoId, id(m) as ctoMsgId, m.time as mTime, id(r) as recId

```

Once the pattern query matches the `(sender)-[:SENT_MSG]->(message)-[:RECEIVED_MSG]->(receiver)` pattern, an event is sent to an output query to calculate the temporal locality using the Cypher `duration.between()` function to establish a sliding window of interest.

``` cypher
duration("PT6M") > duration.between(m.time,thisMsg.time) > duration("PT4M")
```

The expression utilizes Cypher-defined temporal data types, based on [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format, PT6M and PT4M to represent 6 minutes < duration >4 minutes for the window.  We are able to use this because we converted the ingested epoch time formatted timestamp to datetime format in the ingest query.

``` cypher
message.time = datetime({epochMillis: $that.time})
```

Otherwise, we would have cast the data within the standing query:

``` cypher
AND duration("PT6M") > duration.between(datetime({epochMillis: m.time}),datetime({epochMillis: thisMsg.time})) > duration("PT4M")
```

Using the former pattern allows us to to express the standing query in a clean, simple manner.

```yaml
    outputs:
      withinFourToSixMinuteWindow:
        type: CypherQuery
        query: |-
          MATCH (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r), (thisMsg)
           WHERE id(n) = $that.data.ctoId
             AND id(r) = $that.data.recId
             AND id(thisMsg) = $that.data.ctoMsgId
             AND id(m) <> id(thisMsg)
             AND duration("PT6M") > duration.between(m.time,thisMsg.time) > duration("PT4M")
           
           CREATE (m)-[:IN_WINDOW]->(thisMsg)
           CREATE (m)<-[:IN_WINDOW]-(thisMsg)
           
           WITH n, m, r, "http://localhost:8080/#MATCH" + text.urlencode(' (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r) WHERE strId(n)="' + strId(n) + '"AND strId(r)="' + strId(r) + '" AND  strId(m)="' + strId(m) + '" RETURN n, r, m') as URL
          
           RETURN URL
        andThen:
          type: PrintToStandardOut

```

```yaml
    outputs:
      - name: withinFourToSixMinuteWindow
        resultEnrichment:
          query: |-
            MATCH (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r), (thisMsg)
             WHERE id(n) = $that.data.ctoId
               AND id(r) = $that.data.recId
               AND id(thisMsg) = $that.data.ctoMsgId
               AND id(m) <> id(thisMsg)
               AND duration("PT6M") > duration.between(m.time,thisMsg.time) > duration("PT4M")
             
             CREATE (m)-[:IN_WINDOW]->(thisMsg)
             CREATE (m)<-[:IN_WINDOW]-(thisMsg)
             
             WITH n, m, r, "http://localhost:8080/#MATCH" + text.urlencode(' (n)-[:SENT_MSG]->(m)-[:RECEIVED_MSG]->(r) WHERE strId(n)="' + strId(n) + '"AND strId(r)="' + strId(r) + '" AND  strId(m)="' + strId(m) + '" RETURN n, r, m') as URL
            
             RETURN URL
          parameter: that
        destinations:
          - type: StandardOut

```

When a complete pattern match is detected the recipe outputs a link to the console for an analyst to use when reviewing the event further inside Quine's Exploration UI.

``` shell
2023-02-23 12:04:24,981 Standing query `withinFourToSixMinuteWindow` match: {"meta":{"isPositiveMatch":true,"resultId":"628d6523-4d25-4cb4-aed2-9890ecf0cb9c"},"data":{"URL":"http://localhost:8080/#MATCH%20%28n%29-%5B%3ASENT_MSG%5D-%3E%28m%29-%5B%3ARECEIVED_MSG%5D-%3E%28r%29%20WHERE%20strId%28n%29%3D%22a69876bc-8876-37e9-b776-471507a080b9%22AND%20strId%28r%29%3D%2299eeafc4-3178-3aca-8c7c-f84d0781f3a1%22%20AND%20%20strId%28m%29%3D%229e66e020-45ad-3971-92ec-60be8fa844e7%22%20RETURN%20n%2C%20r%2C%20m"}}
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r duration.yaml
Graph is ready
Running Recipe: Temporal Locality Example
Using 3 node appearances
Using 5 quick queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
```

Almost immediately positive matches will begin to stream to your console window.

``` shell
2023-02-23 14:12:05,165 Standing query `withinFourToSixMinuteWindow` match: {"meta":{"isPositiveMatch":true,"resultId":"52c981a4-394f-48c3-8419-671fe5f0c1d5"},"data":{"URL":"http://localhost:8080/#MATCH%20%28n%29-%5B%3ASENT_MSG%5D-%3E%28m%29-%5B%3ARECEIVED_MSG%5D-%3E%28r%29%20WHERE%20strId%28n%29%3D%22a69876bc-8876-37e9-b776-471507a080b9%22AND%20strId%28r%29%3D%22c57f8f3b-b567-3b7e-b2a2-c516f030ce2f%22%20AND%20%20strId%28m%29%3D%221c0bc726-4a19-363d-a623-f5df12142a72%22%20RETURN%20n%2C%20r%2C%20m"}}
```

Copy and paste the URL into your browser (or ++"cmd+click"++ on a Mac) to open the Exploration UI and display an email event.

![email event](images/duration-email.png)

Right click on one of the email messages and select ++"[Node] Messages in Window"++ to find other messages that occured within the time window.

![messages in window](images/duration-inWindow.png)

Continue to right click and run queries to discover other messages that we sent but not did not occur within the time window.

![other email messages](images/duration-otherEmail.png)

## Summary

This simple recipe shows how the `duration.between()` temporal function is able to implement a sliding window for matching events that happen close to gether in time.

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query                 | Node Type | Description                                         |
| --------------------------- | --------- | --------------------------------------------------- |
| Adjacent Nodes              | All       | Display the nodes that are adjacent to this node.   |
| Refresh                     | All       | Refresh the content stored in a node                |
| Local Properties            | All       | Display the properties stored by the node           |
| Messages in Window          | Messages  | Display all message nodes within the sliding window |
| Table of Messages in Window | Messages  | Return a list of messages and time deltas           |


---
# Entity Resolution
URL: https://quine.io/recipes/entity-resolution/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>

    Learn how real-time entity resolution – the deduplication of similar data – can drastically help with creating a comprehensive view of your data.

    See the recipe in action on <a href="https://current.confluent.io/2024-sessions/streaming-entity-resolution-for-kafka-with-quine" target="_blank">Confluent</a>, and on <a href="https://www.youtube.com/watch?v=QWtFGcM7S4c" target="_blank">YouTube</a>.

    ??? example "Entity Resolution Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/entity-resolution.yaml"
        ```

    [Download Recipe](assets/entity-resolution.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Ryan Wright](https://github.com/rrwright)</small>

    Learn how real-time entity resolution – the deduplication of similar data – can drastically help with creating a comprehensive view of your data.

    See the recipe in action on <a href="https://current.confluent.io/2024-sessions/streaming-entity-resolution-for-kafka-with-quine" target="_blank">Confluent</a>, and on <a href="https://www.youtube.com/watch?v=QWtFGcM7S4c" target="_blank">YouTube</a>.

    ??? example "Entity Resolution Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/entity-resolution.yaml"
        ```

    [Download Recipe](assets/v2/entity-resolution.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

We will resolve entities fast enough to do it live in a data pipeline. Entities will always be resolved for downstream. We will stream in sample data from public address information, and add a `resolved` field to each data record. This field will answer, "How does this resolve? How does this one address as written, resolve to one physical place?" This is the entity resolution problem for addresses, which this recipe will demo.

## Sample Data

Before running this Recipe, download the dataset.

!!! note
    Download the sample data to the same directory where you will run Quine.

```shell
curl -L https://recipes.quine.io/public-record-addresses-2021 -o public-record-addresses-2021.ndjson
```

## How it Works

### Ingest Query

Here is the first record from the dataset to serve as an example:

```json
{
  "original": "America First Credit U\nPo Box 9199\nOgden, UT 84409-0000",
  "addressee": "america first credit u",
  "parts": {
    "house": "america first credit u",
    "poBox": "po box 9199",
    "city": "ogden",
    "state": "ut",
    "postcode": "84409"
  }
}
```

!!! note
    While this recipe's ingest query does ingest data from a file, this could just as easily be switched out for a Kafka data source, or any other streaming source of data.

For each record in the dataset, this recipe's ingest query manifests that data into a `record` node on the graph, along with potentially manifesting data into an `entity` node (many records can point to one entity if their addressee and parts are the same). Along with these two nodes, data is also manifested into several other nodes, representing the different parts of an entity's address.

The ingest query also creates relationships between the `record` and its `entity`, along with relationships between an `entity` and its address parts (`poBox`, `postcode`, `cityDistrict`, `road`, `country`, etc).

!!! note
    The standing queries discussed later on incrementally `MATCH` on the emergence of `:poBox` and `:postcode` edges from an `entity` node in the graph. These edges manifest from the ingest stream, triggering the standing queries **as data is ingested** into the graph, **the instant** these edges manifest.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: public-record-addresses-2021.ndjson
      format:
        type: CypherJson
        query: >-
          WITH $that.parts AS parts
          MATCH (record), (entity), (cityDistrict), (unit), (country), (state), (level), (suburb), (city), (road), (house), (houseNumber), (poBox), (category), (near), (stateDistrict), (staircase), (postcode)
          WHERE id(record) = idFrom($that)
            AND id(entity) = idFrom($that.addressee, parts)
            ...
          SET entity = parts,
              entity.addressee = $that.addressee,
              entity: Entity,
              record = $that,
              record: Record
          CREATE (record)-[:record_for_entity]->(entity)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/address-records"
    {
      "type": "FileIngest",
      "path": "public-record-addresses-2021.ndjson",
      "format": {
        "type": "CypherJson",
        "query": "WITH $that.parts AS parts MATCH (record), (entity), ... SET entity = parts, entity.addressee = $that.addressee, entity: Entity, record = $that, record: Record CREATE (record)-[:record_for_entity]->(entity)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: address-records
        source:
          type: File
          path: $in_file
          format:
            type: JsonL
        query: >-
          WITH $that.parts AS parts
          MATCH (record), (entity), (cityDistrict), (unit), (country), (state), (level), (suburb), (city), (road), (house), (houseNumber), (poBox), (category), (near), (stateDistrict), (staircase), (postcode)
          WHERE id(record) = idFrom($that)
            AND id(entity) = idFrom($that.addressee, parts)
            ...
          SET entity = parts,
              entity.addressee = $that.addressee,
              entity: Entity,
              record = $that,
              record: Record
          CREATE (record)-[:record_for_entity]->(entity)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "address-records",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "JsonL"
        }
      },
      "query": "WITH $that.parts AS parts MATCH (record), (entity), ... SET entity = parts, entity.addressee = $that.addressee, entity: Entity, record = $that, record: Record CREATE (record)-[:record_for_entity]->(entity)"
    }
    ```

!!! tip
    This recipe's ingest query uses Cypher's <a href="https://neo4j.com/developer/kb/conditional-cypher-execution/#_using_foreach_for_write_only_cypher" target="_blank">`FOREACH`</a> like a poor man's `IF` statement. You'll see it used to conditionally manifest a node property and set an edge to that node.

    Not every record in this dataset has the same set of `parts`, which is why we want some conditional cypher logic.

### Standing Queries

This recipe uses two [standing queries](../learn/standing-queries/standing-queries.md) to do the work of entity resolution based on an `entity`'s `poBox` and `postcode`, adding the `resolved` property to each record which references the resolved `entity`, and emitting these resolved records downstream.

#### First Standing Query

The first standing query incrementally `MATCH`es the pattern when an `entity` has `:poBox` and `:postcode` edges from itself to the respective nodes:

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        mode: MultipleValues
        query: >-
          MATCH (pb)<-[:poBox]-(e)-[:postcode]->(pc)
          RETURN id(e) AS entity, pb.poBox AS poBox, pc.postcode AS postcode
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/resolve-by-pobox-postcode"
    {
      "pattern": {
        "type": "Cypher",
        "mode": "MultipleValues",
        "query": "MATCH (pb)<-[:poBox]-(e)-[:postcode]->(pc) RETURN id(e) AS entity, pb.poBox AS poBox, pc.postcode AS postcode"
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: resolve-by-pobox-postcode
        pattern:
          type: Cypher
          mode: MULTIPLE_VALUES
          query: >-
            MATCH (pb)<-[:poBox]-(e)-[:postcode]->(pc)
            RETURN id(e) AS entity, pb.poBox AS poBox, pc.postcode AS postcode
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "resolve-by-pobox-postcode",
      "pattern": {
        "type": "Cypher",
        "mode": "MULTIPLE_VALUES",
        "query": "MATCH (pb)<-[:poBox]-(e)-[:postcode]->(pc) RETURN id(e) AS entity, pb.poBox AS poBox, pc.postcode AS postcode"
      }
    }
    ```

When that standing query finds this pattern, it passes down the `entity`'s id, along with the `poBox` and `postcode` to the standing query output, which creates a `canonical` node for each entity, and a relationship to that node (via the `:resolved` edge).

=== "YAML"

    ```yaml
    outputs:
      resolved:
        type: CypherQuery
        query: >-
          MATCH (e), (canonical)
          WHERE id(e) = $that.data.entity
            AND id(canonical) = idFrom($that.data.poBox, $that.data.postcode)
          SET canonical.canonical = {poBox: $that.data.poBox, postcode: $that.data.postcode},
              canonical: Canonical
          CREATE (e)-[:resolved]->(canonical)
    ```

=== "JSON"

    ```json
    {
      "outputs": {
        "resolved": {
          "type": "CypherQuery",
          "query": "MATCH (e), (canonical) WHERE id(e) = $that.data.entity AND id(canonical) = idFrom($that.data.poBox, $that.data.postcode) SET canonical.canonical = {poBox: $that.data.poBox, postcode: $that.data.postcode}, canonical: Canonical CREATE (e)-[:resolved]->(canonical)"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
        outputs:
          - name: resolved
            resultEnrichment:
              query: >-
                MATCH (e), (canonical)
                WHERE id(e) = $that.data.entity
                  AND id(canonical) = idFrom($that.data.poBox, $that.data.postcode)
                SET canonical.canonical = {poBox: $that.data.poBox, postcode: $that.data.postcode},
                    canonical: Canonical
                CREATE (e)-[:resolved]->(canonical)
                RETURN null
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json
    {
      "outputs": [
        {
          "name": "resolved",
          "resultEnrichment": {
            "query": "MATCH (e), (canonical) WHERE id(e) = $that.data.entity AND id(canonical) = idFrom($that.data.poBox, $that.data.postcode) SET canonical.canonical = {poBox: $that.data.poBox, postcode: $that.data.postcode}, canonical: Canonical CREATE (e)-[:resolved]->(canonical) RETURN null",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

#### Second Standing Query

The second standing query incrementally `MATCH`es the pattern involving the `:resolved` edge that is created when the first standing query matches the `poBox` and `postcode` pattern, specifically the moment an `entity` with a `record` is `resolved` to a `canonical` node.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        mode: MultipleValues
        query: >-
          MATCH (record)-[:record_for_entity]->(entity)-[:resolved]->(resolved)
          WHERE resolved.canonical IS NOT NULL
          RETURN id(record) AS record, id(resolved) AS resolved
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/emit-resolved-records"
    {
      "pattern": {
        "type": "Cypher",
        "mode": "MultipleValues",
        "query": "MATCH (record)-[:record_for_entity]->(entity)-[:resolved]->(resolved) WHERE resolved.canonical IS NOT NULL RETURN id(record) AS record, id(resolved) AS resolved"
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: emit-resolved-records
        pattern:
          type: Cypher
          mode: MULTIPLE_VALUES
          query: >-
            MATCH (record)-[:record_for_entity]->(entity)-[:resolved]->(resolved)
            WHERE resolved.canonical IS NOT NULL
            RETURN id(record) AS record, id(resolved) AS resolved
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "emit-resolved-records",
      "pattern": {
        "type": "Cypher",
        "mode": "MULTIPLE_VALUES",
        "query": "MATCH (record)-[:record_for_entity]->(entity)-[:resolved]->(resolved) WHERE resolved.canonical IS NOT NULL RETURN id(record) AS record, id(resolved) AS resolved"
      }
    }
    ```

When this standing query `MATCH`es the `resolved` pattern above, it passes down the id's of the `record` and the `resolved` `canonical` node to the standing query output.

=== "YAML"

    ```yaml
    outputs:
      resolved-record:
        type: CypherQuery
        query: >-
          MATCH (record)
          WHERE id(record) = $that.data.record
          WITH properties(record) as props
          RETURN props {.*, resolved: $that.data.resolved} AS resolved_entity
        andThen:
          type: WriteToFile
          path: "entities-resolved.ndjson"
    ```

=== "JSON"

    ```json
    {
      "outputs": {
        "resolved-record": {
          "type": "CypherQuery",
          "query": "MATCH (record) WHERE id(record) = $that.data.record WITH properties(record) as props RETURN props {.*, resolved: $that.data.resolved} AS resolved_entity",
          "andThen": {
            "type": "WriteToFile",
            "path": "entities-resolved.ndjson"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
        outputs:
          - name: resolved-record
            resultEnrichment:
              query: >-
                MATCH (record)
                WHERE id(record) = $that.data.record
                WITH properties(record) as props
                RETURN props {.*, resolved: $that.data.resolved} AS resolved_entity
              parameter: that
            destinations:
              - type: File
                path: "entities-resolved.ndjson"
    ```

=== "JSON"

    ```json
    {
      "outputs": [
        {
          "name": "resolved-record",
          "resultEnrichment": {
            "query": "MATCH (record) WHERE id(record) = $that.data.record WITH properties(record) as props RETURN props {.*, resolved: $that.data.resolved} AS resolved_entity",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "File",
              "path": "entities-resolved.ndjson"
            }
          ]
        }
      ]
    }
    ```

This standing query output emits to the `entities-resolved.ndjson` each record, along with their added `resolved` field, which contains the id of the `canonical` node which the `entity` resolves to.

!!! tip
    Note the use of the "all-properties selector" `.*` which is used to project all key-value pairs from the record.

#### Sample resolved record

```json
{
  "meta": { "isPositiveMatch": true },
  "data": {
    "resolved_entity": {
      "addressee": "pital one\npo \n ca",
      "original": "Capital One\nP.O. Box 60024\nCity Of Industry, CA 91716-0024",
      "parts": {
        "city": "city of industry",
        "house": "capital one po",
        "poBox": "po box 60024",
        "postcode": "91716",
        "state": "ca"
      },
      "resolved": "bcc26a64-20da-3816-bc4c-fe1ea6d1e1f8"
    }
  }
}
```

## Running the Recipe

### Start Recipe, trigger streaming input

```shell
java -jar quine-2.1.1.jar -r entity-resolution.yaml
```

![Recipe Running](./images/entity-resolution/recipe-running.png)

### Watch streaming output

```shell
tail -f entities-resolved.ndjson | jq
```

![Streaming Output](./images/entity-resolution/streaming-output.png)

### Experimenting with the Standing Queries

This recipe includes several sample queries and quick queries to aid in exploring the graph, along with experimenting with how the standing queries incrementally match for emergent patterns in the graph, and how they continously respond to these patterns, emitting results downstream.

Open up the Quine web server running at `http://127.0.0.1:8080/`, and then click on the empty Query input field to see several sample queries that we will be using.

| Sample Query          | Description                                                           |
| :-------------------- | :-------------------------------------------------------------------- |
| Recent node           | Renders the most recently modified/queried node                       |
| Show one record       | Renders a specific record which has a `postcode`, but not a `poBox`   |
| Missing PO Box        | Renders the entity for the specific record with no `poBox`            |
| Create missing PO BOX | Creates a `poBox` node, and creates the `:poBox` edge from the entity |

We will use several of these sample queries, along with several quick queries, to explore the graph, and see the effect of the continuously running Standing Queries.

First, use the second sample query to render a specific `record` node. This record has a `postcode` property, but it does not have a `poBox` property. This means that this record's `entity` does not resolve.

We can verify this by using the third sample query to render the `entity` node for this specific `record`, using that `entity` node's **Property Subgraph** quick query to show a missing `poBox`, and using that node's **Canonical Entity** quick query, which will correctly **NOT** render anything, since without a `poBox`, the entity can't resolve to a canonical entity.

![No canonical entity for record](./images/entity-resolution/no-canonical-entity.png)

We can trigger the resolution of the "edcboardwalk realtyan" `entity` by creating a `:poBox` edge from this node. Trigger the fourth **Create missing PO BOX** sample query, and that edge will be created, and its node rendered on the graph.

Since this fulfills the standing query for entity resolution for the "edcboardwalk realtyan" `entity`, the standing query will resolve this entity to a **Canonical Entity**, which will be immediately streamed out to the output file, and we can view it by issuing that **Canonical Entity** quick query again.

![Entity Resolved](./images/entity-resolution/entity-resolved.png)
![Streamed Entity Resolution](./images/entity-resolution/entity-resolution-streamed.png)

### Quick Queries

Here are the quick queries defined in this recipe.

| Quick Query       | Node Type        | Description                                                                        |
| :---------------- | :--------------- | :--------------------------------------------------------------------------------- |
| Adjacent Nodes    | All              | Display the nodes that are adjacent to this node.                                  |
| Refresh           | All              | Refresh the content stored in a node                                               |
| Local Properties  | All              | Display the properties stored by the node                                          |
| Property Subgraph | Entity           | Renders the address parts of an `entity` node                                      |
| Records           | Entity           | Renders the `records` for an entity node                                           |
| Resolved Entities | Entity           | Renders the `entity` nodes which resolve to the same `canonical` node              |
| Canonical Entity  | Entity           | Renders the `canonical` node which the `entity` resolves to                        |
| A.K.A.            | Entity/Canonical | Renders all the distinct `addressee` fields for the given entity or canonical node |

!!! Tip
    Quick Queries are available by right clicking on a node. The quick queries defined in this recipe are listed below.

## Summary

Entity resolution no longer needs to happen at the end of your stream processing pipeline. The value of real-time analysis on entity resolved data can now be unlocked by using Quine today.


---
# Ethereum Tag Propagation
URL: https://quine.io/recipes/ethereum/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    This recipe models data on the thoroughgoing Ethereum blockchain. Any transaction can be flagged as tainted causing a `tainted` tag to propagate into the graph to track the flow of transactions from the flagged and tainted accounts.

    ??? example "Ethereum Tag Propagation Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/ethereum.yaml"
        ```

    [Download Recipe](assets/ethereum.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    This recipe models data on the thoroughgoing Ethereum blockchain. Any transaction can be flagged as tainted causing a `tainted` tag to propagate into the graph to track the flow of transactions from the flagged and tainted accounts.

    ??? example "Ethereum Tag Propagation Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/ethereum.yaml"
        ```

    [Download Recipe](assets/v2/ethereum.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

  Newly-mined Ethereum transaction metadata is imported via a Server-Sent Events data
  source. Transactions are grouped by the block in which they were mined then imported
  into the graph. Each wallet address is represented by a node, linked by an edge
  to each transaction sent or received by that account, and linked by an edge to any
  blocks mined by that account. Quick queries allow marking an account as "tainted".
  The tainted flag is propagated along outgoing transaction paths via Standing Queries
  to record the least degree of separation between a tainted source and an account
  receiving a transaction.

!!! note

    The Ethereum diamond logo is property of the Ethereum Foundation, used under the terms of the Creative Commons Attribution 3.0 License.

## Sample Data

Sample data is continuously sampled from the Ethereum block chain and emitted as a [server sent event](https://en.wikipedia.org/wiki/Server-sent_events) for use in this demo.

## How it Works

The recipe installs two ingest queries. They are auto-named `INGEST-1` and `INGEST-2`. The `INGEST-1` query processes blocks, and `INGEST-2` processes mined transactions. In both queries, [idFrom is used](https://quine.io/core-concepts/id-provider/#idfrom) to identify nodes from unique identifiers present in the dataset. For accounts, the address is the identifier; for blocks, the block hash is the identifier; etc. Ethereum data uses hexadecimal strings for identifiers, sometimes with a [built-in capitalization checksum](https://eips.ethereum.org/EIPS/eip-55). This means the address `0x19975E29111a6c85E282eBe409C272c15492c6Ad` is the same address as `0x19975e29111a6c85e282ebe409c272c15492c6ad`, just written slightly differently. To account for these variations in the hex representation's capitalization, before resolving an id, `toLower` is used to convert the identifier to consistent lower-case representation.

### INGEST-1

The INGEST-1 query processes streaming data for `block_head` like:

```json
id: 14566607_head
event: block_head
data: {
    "number": 14566607,
    "hash": "0xf3dafdda16a884f6ff2b1b0c0325eaadc70db022363e3af74ab5994f8cbc1f12",
    "parentHash": "0xcd859249e97684f319173c284314307a11deaa2a708c8c5fcf377971e09abb01",
    "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "logsBloom": "0x0",
    "transactionsRoot": "0xa77b91fc4ee74bc1df28019e898a4ba17dd87fcc41c633cab25b4909ee56a60a",
    "stateRoot": "0xf7869b706a212bfa504520674c3ef3350b187d31ef207b155fa548a4e59169df",
    "receiptsRoot": "0x6669147c87b5cc857801372bed55ab6ddf3474d935b2b4e3b1ee1b95f4dc357b",
    "miner": "0x829BD824B016326A401d083B33D092293333A830",
    "difficulty": "13384256520560135",
    "extraData": "0xe4b883e5bda9e7a59ee4bb99e9b1bc4a1621",
    "gasLimit": 30029295,
    "gasUsed": 3117128,
    "timestamp": 1649710008,
    "baseFeePerGas": "0xfcf7d67a0",
    "nonce": "0xc1a22f3db05412ca",
    "mixHash": "0xfaafcc9e2be300ba795954bed57a38e415330e6131e48e58770e8e678a16e869"
}
```

The ingest query identifies `(BA)`, `(minerAcc)`, `(blk)`, and `(parentBlk)` nodes and loads them into the graph.

=== "YAML"

    ```yaml
        - format:
            query: |-
              MATCH (BA), (minerAcc), (blk), (parentBlk)
              WHERE
                id(blk) = idFrom('block', toLower($that.hash))
                AND id(parentBlk) = idFrom('block', toLower($that.parentHash))
                AND id(BA) = idFrom('block_assoc', toLower($that.hash))
                AND id(minerAcc) = idFrom('account', toLower($that.miner))
              CREATE
                (minerAcc)<-[:mined_by]-(blk)-[:header_for]->(BA),
                (blk)-[:preceded_by]->(parentBlk)
              SET
                BA:block_assoc,
                BA.number = $that.number,
                BA.hash = $that.hash,
                blk:block,
                blk = $that,
                minerAcc:account,
                minerAcc.address = $that.miner
            type: CypherJson
          url: https://ethereum.demo.thatdot.com/blocks_head
          type: ServerSentEventsIngest
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "format": {
        "query": "MATCH (BA), (minerAcc), (blk), (parentBlk)\nWHERE\n  id(blk) = idFrom('block', toLower($that.hash))\n  AND id(parentBlk) = idFrom('block', toLower($that.parentHash))\n  AND id(BA) = idFrom('block_assoc', toLower($that.hash))\n  AND id(minerAcc) = idFrom('account', toLower($that.miner))\nCREATE\n  (minerAcc)<-[:mined_by]-(blk)-[:header_for]->(BA),\n  (blk)-[:preceded_by]->(parentBlk)\nSET\n  BA:block_assoc,\n  BA.number = $that.number,\n  BA.hash = $that.hash,\n  blk:block,\n  blk = $that,\n  minerAcc:account,\n  minerAcc.address = $that.miner",
        "type": "CypherJson"
      },
      "url": "https://ethereum.demo.thatdot.com/blocks_head",
      "type": "ServerSentEventsIngest"
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: block-headers
        source:
          type: ServerSentEvent
          url: https://ethereum.demo.thatdot.com/blocks_head
          format:
            type: Json
        query: |-
          MATCH (BA), (minerAcc), (blk), (parentBlk)
          WHERE
            id(blk) = idFrom('block', toLower($that.hash))
            AND id(parentBlk) = idFrom('block', toLower($that.parentHash))
            AND id(BA) = idFrom('block_assoc', toLower($that.hash))
            AND id(minerAcc) = idFrom('account', toLower($that.miner))
          CREATE
            (minerAcc)<-[:mined_by]-(blk)-[:header_for]->(BA),
            (blk)-[:preceded_by]->(parentBlk)
          SET
            BA:block_assoc,
            BA.number = $that.number,
            BA.hash = $that.hash,
            blk:block,
            blk = $that,
            minerAcc:account,
            minerAcc.address = $that.miner
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "block-headers",
      "source": {
        "type": "ServerSentEvent",
        "url": "https://ethereum.demo.thatdot.com/blocks_head",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (BA), (minerAcc), (blk), (parentBlk) WHERE id(blk) = idFrom('block', toLower($that.hash)) AND id(parentBlk) = idFrom('block', toLower($that.parentHash)) AND id(BA) = idFrom('block_assoc', toLower($that.hash)) AND id(minerAcc) = idFrom('account', toLower($that.miner)) CREATE (minerAcc)<-[:mined_by]-(blk)-[:header_for]->(BA), (blk)-[:preceded_by]->(parentBlk) SET BA:block_assoc, BA.number = $that.number, BA.hash = $that.hash, blk:block, blk = $that, minerAcc:account, minerAcc.address = $that.miner"
    }
    ```

### INGEST-2

The INGEST-2 query receives `tx_mined` events like:

```json
id: 14566637: 0
event: tx_mined
data: {
    "blockHash": "0x0d7782556aef00f1391a05a18ab229a70720780fe3c92eaff74738dee59649d0",
    "blockNumber": 14566637,
    "from": "0x19975E29111a6c85E282eBe409C272c15492c6Ad",
    "gas": 42105,
    "gasPrice": "203940950410",
    "hash": "0x470294af9453f2cd1ec084456328da5c613585974e838fa088cef27246b2481e",
    "input": "0x",
    "nonce": 1,
    "r": "0x8b52f40f28db1627e82fea7352f6d2ba1133dcac081b6939bd03ff397370586d",
    "s": "0x8e7b2c69b1684873156090f238d42aad2c14315a08551a57dc5ed1aa45f0a76",
    "to": "0x732Ec041e4Dc8c01B541B237dE5Ce794c51cF838",
    "transactionIndex": 0,
    "type": "0x0",
    "v": "0x26",
    "value": "168930787638413525"
}
```

The ingest query identifies `(BA)`, `(toAcc)`, `(fromAcc)`, and `(tx)` and loads them into the graph.

=== "YAML"

    ```yaml
      - format:
          query: |-
            WITH true AS validTransactionRecord WHERE $that.to IS NOT NULL AND $that.from IS NOT NULL
            MATCH (BA), (toAcc), (fromAcc), (tx)
            WHERE
              id(BA) = idFrom('block_assoc', toLower($that.blockHash))
              AND id(toAcc) = idFrom('account', toLower($that.to))
              AND id(fromAcc) = idFrom('account', toLower($that.from))
              AND id(tx) = idFrom('transaction', toLower($that.hash))
            CREATE
              (tx)-[:defined_in]->(BA),
              (tx)-[:from]->(fromAcc),
              (tx)-[:to]->(toAcc)
            SET
              tx:transaction,
              BA:block_assoc,
              toAcc:account,
              fromAcc:account,
              tx = $that,
              fromAcc.address = $that.from,
              toAcc.address = $that.to
          type: CypherJson
        url: https://ethereum.demo.thatdot.com/mined_transactions
        type: ServerSentEventsIngest
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-2"
    {
      "format": {
        "query": "WITH true AS validTransactionRecord WHERE $that.to IS NOT NULL AND $that.from IS NOT NULL\nMATCH (BA), (toAcc), (fromAcc), (tx)\nWHERE\n  id(BA) = idFrom('block_assoc', toLower($that.blockHash))\n  AND id(toAcc) = idFrom('account', toLower($that.to))\n  AND id(fromAcc) = idFrom('account', toLower($that.from))\n  AND id(tx) = idFrom('transaction', toLower($that.hash))\nCREATE\n  (tx)-[:defined_in]->(BA),\n  (tx)-[:from]->(fromAcc),\n  (tx)-[:to]->(toAcc)\nSET\n  tx:transaction,\n  BA:block_assoc,\n  toAcc:account,\n  fromAcc:account,\n  tx = $that,\n  fromAcc.address = $that.from,\n  toAcc.address = $that.to",
        "type": "CypherJson"
      },
      "url": "https://ethereum.demo.thatdot.com/mined_transactions",
      "type": "ServerSentEventsIngest"
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: mined-transactions
        source:
          type: ServerSentEvent
          url: https://ethereum.demo.thatdot.com/mined_transactions
          format:
            type: Json
        query: |-
          WITH true AS validTransactionRecord WHERE $that.to IS NOT NULL AND $that.from IS NOT NULL
          MATCH (BA), (toAcc), (fromAcc), (tx)
          WHERE
            id(BA) = idFrom('block_assoc', toLower($that.blockHash))
            AND id(toAcc) = idFrom('account', toLower($that.to))
            AND id(fromAcc) = idFrom('account', toLower($that.from))
            AND id(tx) = idFrom('transaction', toLower($that.hash))
          CREATE
            (tx)-[:defined_in]->(BA),
            (tx)-[:from]->(fromAcc),
            (tx)-[:to]->(toAcc)
          SET
            tx:transaction,
            BA:block_assoc,
            toAcc:account,
            fromAcc:account,
            tx = $that,
            fromAcc.address = $that.from,
            toAcc.address = $that.to
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "mined-transactions",
      "source": {
        "type": "ServerSentEvent",
        "url": "https://ethereum.demo.thatdot.com/mined_transactions",
        "format": {
          "type": "Json"
        }
      },
      "query": "WITH true AS validTransactionRecord WHERE $that.to IS NOT NULL AND $that.from IS NOT NULL MATCH (BA), (toAcc), (fromAcc), (tx) WHERE id(BA) = idFrom('block_assoc', toLower($that.blockHash)) AND id(toAcc) = idFrom('account', toLower($that.to)) AND id(fromAcc) = idFrom('account', toLower($that.from)) AND id(tx) = idFrom('transaction', toLower($that.hash)) CREATE (tx)-[:defined_in]->(BA), (tx)-[:from]->(fromAcc), (tx)-[:to]->(toAcc) SET tx:transaction, BA:block_assoc, toAcc:account, fromAcc:account, tx = $that, fromAcc.address = $that.from, toAcc.address = $that.to"
    }
    ```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r ethereum.yaml
Graph is ready
Running Recipe: Ethereum Tag Propagation
Using 6 node appearances
Using 7 quick queries
Using 2 sample queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Running Ingest Stream INGEST-2
Quine web server available at http://localhost:8080 
```

Observe that Quine is running in the terminal window and that the ingest queries are receiving data.

```text
 | => STANDING-1 count 0
 | => INGEST-1 status is running and ingested 485
 | => INGEST-2 status is running and ingested 34820
```

## Reviewing chains

The nodes appearing in your graph are from the live Ethereum blockchain. They will continue to stream in as long as Quine is running the recipe.

Start exploring the graph by pulling a few recent blocks from the blockchain with the `Recently Accessed Blocks` sample query. Select the sample query in the query bar then click the Query button. The query returns a sub-graph of the recent blocks ordered by the block that preceded it.

??? Note
  
    Click on the query bar for a list of sample queries. 

    ![Recently Accessed Blocks](images/recentlyAccessedBlocks.png)

Take a moment to inspect a couple of the blocks to see the data stored as parameters.

![Blocks from the Ethereum Blockchain](images/ethereumBlocks.png)

Click back into the query bar and clear the query then submit the `Sent and Received ETH` sample query to see accounts that have sent and received transactions.

![Blocks that have sent and received Wei](images/ethereumWeiTransaction.png)

This query finds a series of Wei transactions chained from account to account. Arrange the graph so that you can see all of the nodes. Right-click on the node at the head of the chain and select "Outgoing Transactions" to create a synthetic edge between the accounts. Create a second synthetic edge between the second and third accounts.

!!! Tip

    Drag a node to lock its position in place. Hold shift and click and hold a pinned node to unlock it.

## Taint a Node

Right-click on the origin node again and select "Mark as Tainted." This adds a `tainted` parameter tag to the node and sets it to a value of 0. A node with `tainted=0` indicates that this is the source of taint in our graph.

Notice that you begin to receive updates in the terminal window where you launched Quine from. The **Standing Query** produces these notices from the recipe; let's look at it now.

A Standing Query is composed of two parts, the pattern query that detects a sub-graph shape and an output query that acts on the matched sub-graph.

??? "Standing Query"

    ``` { .yaml linenums="1" }
    --8<-- "recipes/assets/ethereum.yaml:68:98"
    ```

### Pattern

Cypher from the query pattern is always evaluating the stream of data looking for a match. When matched, it triggers the output query to process the event.

Our standing query is always looking for tainted nodes via the existence of a `tainted` parameter.

```cypher
MATCH
  (tainted:account)<-[:from]-(tx:transaction)-[:to]->(otherAccount:account),
  (tx)-[:defined_in]->(ba:block_assoc)
WHERE
  tainted.tainted IS NOT NULL
RETURN
  id(tainted) AS accountId,
  tainted.tainted AS oldTaintedLevel,
  id(otherAccount) AS otherAccountId
```

The results of the match pattern are sent to the output query.

The output query acts on the match to propagate the `tainted` tag. The value of `tainted` is equal to the shortest path to any tainted node.

### Output

```cypher
MATCH (tainted), (otherAccount)
WHERE
  tainted <> otherAccount
  AND id(tainted) = $that.data.accountId
  AND id(otherAccount) = $that.data.otherAccountId
WITH *, coll.min([($that.data.oldTaintedLevel + 1), otherAccount.tainted]) AS newTaintedLevel
SET otherAccount.tainted = newTaintedLevel
RETURN
  strId(tainted) AS taintedSource,
  strId(otherAccount) AS newlyTainted,
  newTaintedLevel
```

A standing query is capable of sending results to destinations configured in the API.

```json
  "andThen": {
    "type": "PrintToStandardOut"
  }
```

```json
  "destinations": [
    { "type": "StandardOut" }
  ]
```

In our case, the results from the match are printed to standard out. These are the messages that you now see in your terminal window.

```json
2022-04-13 11:05:14,877 Standing query `propagate-tainted` match: {"meta":{"isPositiveMatch":true,"resultId":"e3aa2a7c-b246-4896-b8b7-d4fea9904c91"},"data":{"taintedSource":"ed9899b5-e8a8-3a0b-9785-824f2cb1781b","newlyTainted":"981c7ef9-319a-35ba-90dd-401faf5de6a6","newTaintedLevel":3}}
```

## Tainted Tag Propagation

The recipe also installs a [graph feed](../getting-started/exploration-ui-settings.md#graph-feeds) named `taint-propagation-live` so that you can watch the taint spread instead of querying for it. The feed taps the `propagate-tainted` output after its enrichment query runs, which is the moment an account's `tainted` property is actually set.

```yaml
graphFeeds:
  - name: taint-propagation-live
    description: |-
      Draws each taint hop onto the canvas as it happens: the tainted source account,
      the transaction that carried the taint, and the account that just became tainted.
    standingQueryName: taint-propagation
    outputName: propagate-tainted
    query: |-
      MATCH (source)<-[:from]-(tx:transaction)-[:to]->(recipient)
      WHERE
        strId(source) = $taintedSource
        AND strId(recipient) = $newlyTainted
      RETURN source, tx, recipient
```

Every column the enrichment query returns arrives as a Cypher parameter of the same name, so the feed's query resolves `$taintedSource` and `$newlyTainted` back to their accounts and returns them along with the transaction that carried the taint. Each node the query returns is drawn onto the canvas.

Find the `taint-propagation-live` pill at the bottom of the Exploration UI and switch it on. Every hop the standing query records is added to your canvas as it happens.

Clear your explorer window using the '<<' button, then run the "Tainted Accounts" query. This query will find the original account or accounts responsible for the taint in the graph.

Right-click on a tainted account (appears fuchsia) and select "Outgoing Tainted Transactions" to find the accounts that this account tainted. Hover over the account to see the `tainted=1` property that indicates that this account is one hop away from the source of the taint.

![Tainted Node](images/ethereumTaintedNode.png)

Continue to taint and explore the graph as more of the nodes become tainted.

At any time, you can issue the following query to report the number of tainted nodes in the graph.

``` cypher
MATCH (n) 
WHERE n.tainted IS NOT NULL 
RETURN DISTINCT n.tainted, count(n) 
ORDER BY n.tainted
```


---
# Financial Risk Calculation
URL: https://quine.io/recipes/finance/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    The financial industry's current approach to managing mandated operational risk capital requirements, batch processing, often leads to over- or under-allocation of certain classes of funds, operating with tight time constraints, and slow reactions to changing market conditions.

    By responding to market changes in real time, organizations can provide adequate coverage for risk exposure while ensuring their regulatory compliance minimally affects their asset allocation.

    This recipe intends to show an example of conditionally adjusting data (investment value) based on the manifested nodes' property (investment class) before aggregating the adjusted values at multiple levels.  Further, it uses the adjusted aggregates to alert on threshold crossings (ratio of adjusted values of specific classes to the sum of all the adjusted values).

    The recipe highlights three technical strategies:

    1. Use of `NumberIteratorIngest` to generate sample transactions
    2. Conditional handling of data
    3. Real-time graph-based data (from #2) aggregated across multiple levels

    ??? example "Financial Risk Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/finance.yaml"
        ```

    [Download Recipe](assets/finance.yaml){ .md-button .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    The financial industry's current approach to managing mandated operational risk capital requirements, batch processing, often leads to over- or under-allocation of certain classes of funds, operating with tight time constraints, and slow reactions to changing market conditions.

    By responding to market changes in real time, organizations can provide adequate coverage for risk exposure while ensuring their regulatory compliance minimally affects their asset allocation.

    This recipe intends to show an example of conditionally adjusting data (investment value) based on the manifested nodes' property (investment class) before aggregating the adjusted values at multiple levels.  Further, it uses the adjusted aggregates to alert on threshold crossings (ratio of adjusted values of specific classes to the sum of all the adjusted values).

    The recipe highlights three technical strategies:

    1. Use of `NumberIteratorIngest` to generate sample transactions
    2. Conditional handling of data
    3. Real-time graph-based data (from #2) aggregated across multiple levels

    ??? example "Financial Risk Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/finance.yaml"
        ```

    [Download Recipe](assets/v2/finance.yaml){ .md-button .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

This recipe is modeled on regulatory monitoring requirements like the Basel III [Liquidity Coverage Ratio (LCR)](https://www.bis.org/publ/bcbs238.htm), [Net Stable Funding Ratio (NSFR)](https://www.bis.org/bcbs/publ/d295.htm), and liquidity risk monitoring tools as described in [https://www.bis.org/bcbs/basel3.htm](https://www.bis.org/bcbs/basel3.htm). This class of capital and liquidity requirements calls for, at a minimum, aggregation of trades/collateral/investments by class, as well as potentially adjusting valuations based on the potential for a haircut before rollups.

## How it Works

### INGEST

The recipe generates a hierarchy for an institution with a (configurable) fixed number of desks and investments in the ingest stream utilizing the ```NumberIteratorIngest```.  We limit the ingest to simulate one set of investments and associated desks in the recipe by configuring `ingestLimit: 1`.

![Ingest Flow](./images/financial.recipe.ingest.flow.png){ class="center-img" }

The first part of the ingest generates ten trading desks using a Cypher `UNWIND` clause and connects them to the parent institution node.

=== "YAML"

    ```yaml
      - type: NumberIteratorIngest
        ingestLimit: 1
        format:
         type: CypherLine
         query: |-
          WITH 0 AS institutionId
          // Generate 10 desks - change the range bound to alter the number of generated desks
          UNWIND range(1, 10) AS deskId
          MATCH (institution), (desk)
          WHERE id(institution) = idFrom('institution', institutionId)
           AND id(desk) = idFrom('desk', institutionId, deskId)

          SET institution:institution

          SET desk:desk,
           desk.deskNumber = deskId

          CREATE (institution)-[:HAS]->(desk)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "NumberIteratorIngest",
      "format": {
        "type": "CypherLine",
        "query": "WITH 0 AS institutionId\n// Generate 10 desks - change the range bound to alter the number of generated desks\nUNWIND range(1, 10) AS deskId\nMATCH (institution), (desk)\nWHERE id(institution) = idFrom('institution', institutionId)\n    AND id(desk) = idFrom('desk', institutionId, deskId)\n\nSET institution:institution\n\nSET desk:desk,\n    desk.deskNumber = deskId\n\nCREATE (institution)-[:HAS]->(desk)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: generate-finance-data
        source:
          type: NumberIterator
          startOffset: 0
          limit: 1
        query: |-
          WITH 0 AS institutionId
          // Generate 10 desks - change the range bound to alter the number of generated desks
          UNWIND range(1, 10) AS deskId
          MATCH (institution), (desk)
          WHERE id(institution) = idFrom('institution', institutionId)
              AND id(desk) = idFrom('desk', institutionId, deskId)

          SET institution:institution

          SET desk:desk,
              desk.deskNumber = deskId

          CREATE (institution)-[:HAS]->(desk)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "generate-finance-data",
      "source": {
        "type": "NumberIterator",
        "startOffset": 0,
        "limit": 1
      },
      "query": "WITH 0 AS institutionId\n// Generate 10 desks - change the range bound to alter the number of generated desks\nUNWIND range(1, 10) AS deskId\nMATCH (institution), (desk)\nWHERE id(institution) = idFrom('institution', institutionId)\n    AND id(desk) = idFrom('desk', institutionId, deskId)\n\nSET institution:institution\n\nSET desk:desk,\n    desk.deskNumber = deskId\n\nCREATE (institution)-[:HAS]->(desk)"
    }
    ```

!!! note
    You can change the number of trading desks by altering the upper bound of the `UNWIND` clause.  For example, if we wanted to generate 100 desks, we would modify the clause to be ```UNWIND range(1, 100) AS deskId```.

The second part of the ingest stream generates 1000 investments per desk using a Cypher `UNWIND` clause and connects them to the parent desk node.  There are two interesting techniques in this part of the recipe:

1. Both random (`investment.type`) and deterministic (`investment.code` and `investment.value`) property generation for the investment nodes; and
2. Assigning one of several potential property values based on the value of a previously generated property.

A value between 1-10 for `investment.type` is generated with a combination of the `rand` Cypher function to generate a random float between 0 (inclusive) and 1 (exclusive), multiplying that value by 10, casting it to an integer, then adding 1.

`investment.code` and `investment.value` utilize the gen class of custom Cypher functions to deterministically generate a value of a specific class from the provided input.  Because we generate the QuineId via the idFrom() function, which act as seeds for the gen classes in use (`gen.string.from()` and `gen.float.from()`).

Lastly, the `investment.class` is assigned based on the value of `investment.type` via a subquery that utilizes the `CASE` statement.  This value (which models the classes in LCR) is used later in the recipe to determine the adjustment to the investment value aggregated at the desk and institution levels.

``` cypher
WITH *
// Generate 1000 investments per desk- change the range bound to alter the number of investments generated per desk
UNWIND range(1, 1000) AS investmentId
MATCH (investment)
WHERE id(investment) = idFrom('investment', institutionId, deskId, investmentId)

SET investment:investment,
    investment.investmentId = toInteger(toString(deskId) + toString(investmentId)),
    investment.type = toInteger(rand() * 10) + 1,
    investment.code = gen.string.from(strId(investment), 25),
    investment.value = gen.float.from(strId(investment)) * 100

WITH id(investment) AS invId, desk, investment
CALL { 
      WITH invId
      MATCH (investment:investment) 
      WHERE id(investment) = invId
      SET investment.class = CASE
        WHEN investment.type <= 5 THEN '1'
        WHEN investment.type >= 6 AND investment.type <= 8 THEN '2a'
        WHEN investment.type >= 9 THEN '2b'
      END

      RETURN investment.type AS type
    }

CREATE (desk)-[:HOLDS]->(investment)
```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "NumberIteratorIngest",
      "format": {
        "type": "CypherLine",
        "query": "WITH *\n// Generate 1000 investments per desk- change the range bound to alter the number of investments generated per desk\nUNWIND range(1, 1000) AS investmentId\nMATCH (investment)\nWHERE id(investment) = idFrom('investment', institutionId, deskId, investmentId)\n\nSET investment:investment,\n    investment.investmentId = toInteger(toString(deskId) + toString(investmentId)),\n    investment.type = toInteger(rand() * 10) + 1,\n    investment.code = gen.string.from(strId(investment), 25),\n    investment.value = gen.float.from(strId(investment)) * 100\n\nWITH id(investment) AS invId, desk, investment\nCALL {\n      WITH invId\n      MATCH (investment:investment)\n      WHERE id(investment) = invId\n      SET investment.class = CASE\n        WHEN investment.type <= 5 THEN '1'\n        WHEN investment.type >= 6 AND investment.type <= 8 THEN '2a'\n        WHEN investment.type >= 9 THEN '2b'\n      END\n\n      RETURN investment.type AS type\n    }\n\nCREATE (desk)-[:HOLDS]->(investment)"
      }
    }
    ```

=== "YAML"

    ```yaml
    # This is part of the same ingest stream - the full query continues from above:
        WITH *
        // Generate 1000 investments per desk- change the range bound to alter the number of investments generated per desk
        UNWIND range(1, 1000) AS investmentId
        MATCH (investment)
        WHERE id(investment) = idFrom('investment', institutionId, deskId, investmentId)

        SET investment:investment,
            investment.investmentId = toInteger(toString(deskId) + toString(investmentId)),
            investment.type = toInteger(rand() * 10) + 1,
            investment.code = gen.string.from(strId(investment), 25),
            investment.value = gen.float.from(strId(investment)) * 100

        WITH id(investment) AS invId, desk, investment
        CALL {
              With invId
              MATCH (investment:investment)
              WHERE id(investment) = invId
              SET investment.class = CASE
                WHEN investment.type <= 5 THEN '1'
                WHEN investment.type >= 6 AND investment.type <= 8 THEN '2a'
                WHEN investment.type >= 9 THEN '2b'
              END

              RETURN investment.type AS type
            }

        CREATE (desk)-[:HOLDS]->(investment)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "generate-finance-data",
      "source": {
        "type": "NumberIterator",
        "startOffset": 0,
        "limit": 1
      },
      "query": "... WITH *\n// Generate 1000 investments per desk- change the range bound to alter the number of investments generated per desk\nUNWIND range(1, 1000) AS investmentId\nMATCH (investment)\nWHERE id(investment) = idFrom('investment', institutionId, deskId, investmentId)\n\nSET investment:investment,\n    investment.investmentId = toInteger(toString(deskId) + toString(investmentId)),\n    investment.type = toInteger(rand() * 10) + 1,\n    investment.code = gen.string.from(strId(investment), 25),\n    investment.value = gen.float.from(strId(investment)) * 100\n\nWITH id(investment) AS invId, desk, investment\nCALL {\n      WITH invId\n      MATCH (investment:investment)\n      WHERE id(investment) = invId\n      SET investment.class = CASE\n        WHEN investment.type <= 5 THEN '1'\n        WHEN investment.type >= 6 AND investment.type <= 8 THEN '2a'\n        WHEN investment.type >= 9 THEN '2b'\n      END\n\n      RETURN investment.type AS type\n    }\n\nCREATE (desk)-[:HOLDS]->(investment)"
    }
    ```

This generates a three-level tree structure with Institution at the top, ten desks next and 1,000 investments under each desk.

![Three-level tree](./images/finance.risk.tree.png){ class="center-img" }

### STANDING-1

#### Generate Adjusted Value

The first [standing query](../learn/standing-queries/standing-queries.md) generates a new property with a value based on another property. The standing query pattern is used to match every generated investment.

![Standing Query Flow](./images/financial.recipe.standing-1.flow.png){ class="center-img" style="width: 35%;" }

=== "YAML"

    ```yaml
        - pattern:
            type: Cypher
            query: |-
              MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
              WHERE investment.adjustedValue IS NOT NULL

              RETURN DISTINCT id(investment) AS id
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE investment.adjustedValue IS NOT NULL\n\nRETURN DISTINCT id(investment) AS id"
      },
      "outputs": {
        "adjustValues": {
          "type": "CypherQuery",
          "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n\nSET investment.adjustedValue = CASE\n      WHEN investment.class = '1' THEN investment.value\n      WHEN investment.class = '2a' THEN investment.value * .85\n      WHEN investment.class = '2b' AND investment.type = 9 THEN investment.value * .75\n      WHEN investment.class = '2b' AND investment.type = 10 THEN investment.value * .5\n    END"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: adjust-values
        pattern:
          type: Cypher
          query: |-
            MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
            RETURN DISTINCT id(investment) AS id
          mode: DISTINCT_ID
        outputs:
          - name: adjustValues
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
                WHERE id(investment) = $that.data.id

                SET investment.adjustedValue = CASE
                      WHEN investment.class = '1' THEN investment.value
                      WHEN investment.class = '2a' THEN investment.value * .85
                      WHEN investment.class = '2b' AND investment.type = 9 THEN investment.value * .75
                      WHEN investment.class = '2b' AND investment.type = 10 THEN investment.value * .5
                    END
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "adjust-values",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nRETURN DISTINCT id(investment) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "adjustValues",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n\nSET investment.adjustedValue = CASE\n      WHEN investment.class = '1' THEN investment.value\n      WHEN investment.class = '2a' THEN investment.value * .85\n      WHEN investment.class = '2b' AND investment.type = 9 THEN investment.value * .75\n      WHEN investment.class = '2b' AND investment.type = 10 THEN investment.value * .5\n    END",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

The standing query output then sets a property to track the adjusted value of the investment in a way that mimics Basel III LCR HQLA (High Quality Liquid Assets).

| **Class**   | **Factor** |
| ----------- | :--------: |
| 1           |    100%    |
| 2A          |    85%     |
| 2B RMBS     |    75%     |
| 2B Equities |    50%     |

Where the **Factor** represents the required adjustment to the investment class, for example, an investment of $100 will be rolled up with the following adjusted values:

| **Class**   | **Adjusted Value** |
| ----------- | :----------------: |
| 1           |        $100        |
| 2A          |        $85         |
| 2B RMBS     |        $75         |
| 2B Equities |        $50         |

This is accomplished via the use of a `CASE` statement, the last two of which utilize a combination of properties to determine the adjusted value.

```yaml
    outputs:
      rollUps:
        type: CypherQuery
        query: |-
          MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
          WHERE id(investment) = $that.data.id

          SET investment.adjustedValue = CASE
                WHEN investment.class = '1' THEN investment.value
                WHEN investment.class = '2a' THEN investment.value * .85
                WHEN investment.class = '2b' AND investment.type = 9 THEN investment.value * .75
                WHEN investment.class = '2b' AND investment.type = 10 THEN investment.value * .5
              END
```

![investment node properties](./images/finance.risk.investment.properties.png){ class="center-img" }

### STANDING-2

#### Calculate Rollups

The second [standing query](../learn/standing-queries/standing-queries.md) serves three purposes:

1. Aggregating the total adjusted values of investments to both the desk and institution levels;
2. Aggregating the per-class adjusted values of investments to both the desk and institution levels; and
3. Calculating the Class 2 and 2B composition of the total adjusted values at the institutional level

As with the first [standing query](../learn/standing-queries/standing-queries.md), the query pattern is used to match every generated investment.

![Standing Query Flow](./images/finance.recipe.standing-2.flow.png){ class="center-img" }

=== "YAML"

    ```yaml
        - pattern:
            type: Cypher
            query: |-
              MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
              WHERE investment.adjustedValue IS NOT NULL

              RETURN DISTINCT id(investment) AS id
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-2"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE investment.adjustedValue IS NOT NULL\n\nRETURN DISTINCT id(investment) AS id"
      },
      "outputs": {
        "rollUps": {
          "type": "CypherQuery",
          "query": "MATCH (investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND investment.adjustedValue IS NOT NULL\n\nUNWIND [[\"1\",\"adjustedValue1\"], [\"2a\",\"adjustedValue2a\"], [\"2b\",\"adjustedValue2b\"]] AS stuff\n\nWITH institution,investment,desk,stuff\nWHERE investment.class = stuff[0]\n\nCALL float.add(institution,stuff[1],investment.adjustedValue) YIELD result AS institutionAdjustedValueRollupByClass\nCALL float.add(institution,\"totalAdjustedValue\",investment.adjustedValue) YIELD result AS institutionAdjustedValueRollup\n\nCALL float.add(desk,stuff[1],investment.adjustedValue) YIELD result AS deskAdjustedValueRollupByClass\nCALL float.add(desk,\"totalAdjustedValue\",investment.adjustedValue) YIELD result AS deskAdjustedValueRollup\n\nSET institution.percentAdjustedValue2 = ((institution.adjustedValue2a + institution.adjustedValue2b)/institution.totalAdjustedValue) * 100,\n    institution.percentAdjustedValue2b = (institution.adjustedValue2b/institution.totalAdjustedValue) * 100"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: roll-ups
        pattern:
          type: Cypher
          query: |-
            MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
            WHERE investment.adjustedValue IS NOT NULL
            RETURN DISTINCT id(investment) AS id
          mode: DISTINCT_ID
        outputs:
          - name: rollUps
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
                WHERE id(investment) = $that.data.id
                  AND investment.adjustedValue IS NOT NULL

                UNWIND [["1","adjustedValue1"], ["2a","adjustedValue2a"], ["2b","adjustedValue2b"]] AS stuff

                WITH institution,investment,desk,stuff
                WHERE investment.class = stuff[0]

                CALL float.add(institution,stuff[1],investment.adjustedValue) YIELD result AS institutionAdjustedValueRollupByClass
                CALL float.add(institution,"totalAdjustedValue",investment.adjustedValue) YIELD result AS institutionAdjustedValueRollup

                CALL float.add(desk,stuff[1],investment.adjustedValue) YIELD result AS deskAdjustedValueRollupByClass
                CALL float.add(desk,"totalAdjustedValue",investment.adjustedValue) YIELD result AS deskAdjustedValueRollup

                SET institution.percentAdjustedValue2 = ((institution.adjustedValue2a + institution.adjustedValue2b)/institution.totalAdjustedValue) * 100,
                    institution.percentAdjustedValue2b = (institution.adjustedValue2b/institution.totalAdjustedValue) * 100
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "roll-ups",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE investment.adjustedValue IS NOT NULL\nRETURN DISTINCT id(investment) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "rollUps",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND investment.adjustedValue IS NOT NULL\n\nUNWIND [[\"1\",\"adjustedValue1\"], [\"2a\",\"adjustedValue2a\"], [\"2b\",\"adjustedValue2b\"]] AS stuff\n\nWITH institution,investment,desk,stuff\nWHERE investment.class = stuff[0]\n\nCALL float.add(institution,stuff[1],investment.adjustedValue) YIELD result AS institutionAdjustedValueRollupByClass\nCALL float.add(institution,\"totalAdjustedValue\",investment.adjustedValue) YIELD result AS institutionAdjustedValueRollup\n\nCALL float.add(desk,stuff[1],investment.adjustedValue) YIELD result AS deskAdjustedValueRollupByClass\nCALL float.add(desk,\"totalAdjustedValue\",investment.adjustedValue) YIELD result AS deskAdjustedValueRollup\n\nSET institution.percentAdjustedValue2 = ((institution.adjustedValue2a + institution.adjustedValue2b)/institution.totalAdjustedValue) * 100,\n    institution.percentAdjustedValue2b = (institution.adjustedValue2b/institution.totalAdjustedValue) * 100",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

The standing query output can be grouped by functionality:

##### Determine class per investment

We determine the class per investment for conditional processing in the next steps.

We create a list of tuples, operate on them with the `UNWIND` clause, and assign a new name for the inner values (Cypher requires this).

```cypher
UNWIND [["1","adjustedValue1"], ["2a","adjustedValue2a"], ["2b","adjustedValue2b"]] AS stuff
```

We then use the first value of the ordered pairs to match the ```investment.class``` by referring to its value by index.

```cypher
WITH institution,investment,desk,stuff
WHERE investment.class = stuff[0]
```

##### Generate property name based on class

We then use the second value of the ordered pairs to match the generated property name by referring to its value by index and aggregate the adjusted value at both the desk and institutional levels.  At the same time, we aggregate all of the classes' investments at the desk level (in the `desk.totalAdjustedValue` property) and institutional level (in the `institution.totalAdjustedValue` property).

```cypher
CALL float.add(institution,stuff[1],investment.adjustedValue) YIELD result AS institutionAdjustedValueRollupByClass
CALL float.add(institution,"totalAdjustedValue",investment.adjustedValue) YIELD result AS institutionAdjustedValueRollup

CALL float.add(desk,stuff[1],investment.adjustedValue) YIELD result AS deskAdjustedValueRollupByClass
CALL float.add(desk,"totalAdjustedValue",investment.adjustedValue) YIELD result AS deskAdjustedValueRollup
```

![desk rollup](./images/finance.risk.desk.properties.png){ class="center-img" }

##### Calculate composition of class 2 and 2b investments

Lastly, we utilize the aggregations to calculate the composition of class 2 and 2b investments at the desk and institutional levels for the third standing query for threshold-based alerting..

```cypher
SET institution.percentAdjustedValue2 = ((institution.adjustedValue2a + institution.adjustedValue2b)/institution.totalAdjustedValue) * 100,
    institution.percentAdjustedValue2b = (institution.adjustedValue2b/institution.totalAdjustedValue) * 100
```

![institution percentage](./images/finance.recipe.institution.percent.png){ class="center-img" }

### STANDING-3

#### Alert based on thresholds

The third [standing query](../learn/standing-queries/standing-queries.md) utilizes the aggregations from the previous standing query for threshold-based alerting.  As with the first [standing query](../learn/standing-queries/standing-queries.md), the query pattern matches every generated investment.

![Standing Query Flow](./images/finance.recipe.standing-3.flow.png){ class="center-img" }

=== "YAML"

    ```yaml
      - pattern:
          type: Cypher
          query: |-
            MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
            RETURN DISTINCT id(investment) AS id
          mode: DistinctId
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-3"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nRETURN DISTINCT id(investment) AS id",
        "mode": "DistinctId"
      },
      "outputs": {
        "class2CompositionAlert": {
          "type": "CypherQuery",
          "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)\n  AND institution.percentAdjustedValue2 > 40\n\nRETURN institution.percentAdjustedValue2 AS Class_2_Composition",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        },
        "class2bCompositionAlert": {
          "type": "CypherQuery",
          "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)\n  AND institution.percentAdjustedValue2b > 15\n\nRETURN institution.percentAdjustedValue2b AS Class_2b_Composition",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: composition-alerts
        pattern:
          type: Cypher
          query: |-
            MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
            RETURN DISTINCT id(investment) AS id
          mode: DISTINCT_ID
        outputs:
          - name: class2CompositionAlert
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
                WHERE id(investment) = $that.data.id
                  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)
                  AND institution.percentAdjustedValue2 > 40

                RETURN institution.percentAdjustedValue2 AS Class_2_Composition
              parameter: that
            destinations:
              - type: StandardOut
          - name: class2bCompositionAlert
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
                WHERE id(investment) = $that.data.id
                  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)
                  AND institution.percentAdjustedValue2b > 15

                RETURN institution.percentAdjustedValue2b AS Class_2b_Composition
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "composition-alerts",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nRETURN DISTINCT id(investment) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "class2CompositionAlert",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)\n  AND institution.percentAdjustedValue2 > 40\n\nRETURN institution.percentAdjustedValue2 AS Class_2_Composition",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        },
        {
          "name": "class2bCompositionAlert",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)\nWHERE id(investment) = $that.data.id\n  AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)\n  AND institution.percentAdjustedValue2b > 15\n\nRETURN institution.percentAdjustedValue2b AS Class_2b_Composition",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

For the sake of simplicity, we set two parameters for alert thresholding.  To mimic the Basel III LCR asset allocation requirements:

* Class 2 investments must not account for more than 40% of total HQLA; and
* Class2B investments must not account for more than 15% of total HQLA

by setting thresholds for `institution.percentAdjustedValue2` and `institution.percentAdjustedValue2b`.

To minimize the rate of alerts, we set thresholds on `institution.investments`.

```
MATCH (investment:investment)<-[:HOLDS]-(desk:desk)<-[:HAS]-(institution:institution)
WHERE id(investment) = $that.data.id
      AND (institution.investments = 2500 OR institution.investments = 5000 OR institution.investments = 10000)
```

We would use other thresholds in production, such as time of day.

STANDING-3 produces output in the terminal window like:

```shell
2023-05-11 17:56:58,207 Standing query `class2CompositionAlert` match: {"meta":{"isPositiveMatch":true,"resultId":"93868c2f-10fe-a76b-9195-01b4007e11a5"},"data":{"Class_2_Composition":43.66579601931527}}
2023-05-11 17:56:59,621 Standing query `class2CompositionAlert` match: {"meta":{"isPositiveMatch":true,"resultId":"59fdf2b4-093b-1755-458d-0c4934f328a6"},"data":{"Class_2b_Composition":17.5875533423801}}
```

## Running the Recipe

!!! warning
    This greatly simplified financial risk recipe simulates a single ingest cycle to demonstrate the capabilities of Quine. Do not make financial decisions based on its output.

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r finance_risk.yaml
Graph is ready
Running Recipe: Financial Risk Recipe
Using 12 node appearances
Using 5 quick queries
Using 4 sample queries
Running Standing Query STANDING-1
Running Standing Query STANDING-2
Running Standing Query STANDING-3
Running Ingest Stream INGEST-1
```

## Summary

Summary.

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query      | Node Type | Description                                       |
| :--------------- | :-------- | :------------------------------------------------ |
| Adjacent Nodes   | All       | Display the nodes that are adjacent to this node. |
| Refresh          | All       | Refresh the content stored in a node              |
| Local Properties | All       | Display the properties stored by the node         |
| Node Label       | All       | Return the label(s) for the selected node         |
| Parent Node      | All       | Retrieve the parent node                          |

## Build your skills

What would happen if we switched `ingestLimit: 1` in the ingest to `ingestLimit: 2`?

??? success "Answer"

    The ingest would run twice, generating the same institution, desks, and investment nodes.  The investment nodes would generate new types and values, affecting the total rollup value.

How could we add aggregating levels to the recipe?
??? success "Answer"

    By adding nodes to the ingest and adding additional `CALL float.add()` custom Cypher procedures to STANDING-2.

The recipe utilises `UNWIND` to generate desks and investments serially and deterministically.  How could we do the same in a non-serial form?

??? success "Answer"
    We could utilise other methods to generate the IDs for the desks and investments rather than relying on the `UNWIND`.  For example, generating hashes for each of the node classes based on properties:

    ```cypher
    WITH toInteger($that) AS x
    WITH *,
        x AS investmentID
    WITH *,
        toInteger(gen.float.from(hash(x, 'desk')) * 10) AS deskId
    ```

The recipe doesn't calculate net cash flow or the ratio of HQLA to net cash flow.  How could we do that?

??? success "Answer"

    We could configure additional ingests for the cash flow data and standing queries to generate the required calculations.


---
# Harry Potter
URL: https://quine.io/recipes/hpotter/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Alec Theriault](https://github.com/harpocrates)</small>

    Ingest a small JSON object to manifest a graph of connected nodes that explore the familial relationships of Harry Potter characters.

    ??? example "Harry Potter Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/hpotter.yaml"
        ```

    [Download Recipe](assets/hpotter.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

=== "Recipe v2"

    <small>Shared by: [Alec Theriault](https://github.com/harpocrates)</small>

    Ingest a small JSON object to manifest a graph of connected nodes that explore the familial relationships of Harry Potter characters.

    ??? example "Harry Potter Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/hpotter.yaml"
        ```

    [Download Recipe](assets/v2/hpotter.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

## Scenario

If you are new to Quine, let's start simple. This small graph of connected nodes allows you to  explore the familial relationships of Harry Potter characters in Quine. Use this recipe to follow along with the examples outlined in the [Exploration UI](../getting-started/exploration-ui.md) getting started guide.

## Sample Data

!!! note
    Download the sample data to the same directory where Quine will be run.

Before running this Recipe, download the dataset.

``` shell
curl https://quine.io/recipes/images/harry_potter_data.json -o harry_potter_data.json
```

## How it Works

This recipe connects an ingest stream to the `harry_potter_data.json` file, parses the JSON object, manifests parent (`p`) and child (`c`) nodes, and creates a relationship between the nodes.

The sample data JSON object is straightforward containing character `name`, `gender`, `birth_year`, and a list of `children`.

``` json
--8<-- "recipes/images/harry_potter_data.json"
```

INGEST-1 processes the `harry_potter_data.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: harry_potter_data.json
        format:
          type: CypherJson
          query: |-
            MATCH (p) WHERE id(p) = idFrom('name', $that.name)
            SET p = { name: $that.name, gender: $that.gender, birth_year: $that.birth_year },
              p: Person
            WITH $that.children AS childrenNames, p
            UNWIND childrenNames AS childName
            MATCH (c) WHERE id(c) = idFrom('name', childName)
            CREATE (c)-[:has_parent]->(p)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "harry_potter_data.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (p) WHERE id(p) = idFrom('name', $that.name)\nSET p = { name: $that.name, gender: $that.gender, birth_year: $that.birth_year },\n  p: Person\nWITH $that.children AS childrenNames, p\nUNWIND childrenNames AS childName\nMATCH (c) WHERE id(c) = idFrom('name', childName)\nCREATE (c)-[:has_parent]->(p)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: harry-potter-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Json
        query: |-
          MATCH (p) WHERE id(p) = idFrom('name', $that.name)
          SET p = { name: $that.name, gender: $that.gender, birth_year: $that.birth_year },
            p: Person
          WITH $that.children AS childrenNames, p
          UNWIND childrenNames AS childName
          MATCH (c) WHERE id(c) = idFrom('name', childName)
          CREATE (c)-[:has_parent]->(p)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "harry-potter-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (p) WHERE id(p) = idFrom('name', $that.name)\nSET p = { name: $that.name, gender: $that.gender, birth_year: $that.birth_year },\n  p: Person\nWITH $that.children AS childrenNames, p\nUNWIND childrenNames AS childName\nMATCH (c) WHERE id(c) = idFrom('name', childName)\nCREATE (c)-[:has_parent]->(p)"
    }
    ```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r hpotter.yaml
Graph is ready
Running Recipe: Harry Potter
Using 2 quick queries
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
INGEST-1 status is completed and ingested 13 
```

## Summary

Open your browser and navigate to `http://localhost:8080`. Click the query bar and select `CALL recentNodes(10)`. Change `10` to `20` to ensure that you pick up the entire graph, and click the `Query` button.

A jumbled graph will appear in your browser, click the "tree view" button (:fontawesome-solid-share-nodes:) to structure the graph before you begin exploring.

![Harry Potter Graph](images/hpotter-graph.png)

Explore the graph using the pre loaded quick queries or write Cypher queries in the query bar.

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query           | Node Type | Description                                            |
| :-------------------- | :-------- | :----------------------------------------------------- |
| Adjacent Nodes        | All       | Display the nodes that are adjacent to this node.      |
| Siblings              | All       | Show the `has sibling` relationship for this node.     |

Now that you have the graph loaded into Quine, head over to the [Exploration UI](../getting-started/exploration-ui.md) guide to learn more about how to use Quine's interface.


---
# File Ingest
URL: https://quine.io/recipes/ingest/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

    This recipe ingests each line in a file (`$in_file`) as graph node with property of "line" that contains the original line from the file.

    ??? example "File Ingest Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/ingest.yaml"
        ```

    [Download Recipe](assets/ingest.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

    This recipe ingests each line in a file (`$in_file`) as graph node with property of "line" that contains the original line from the file.

    ??? example "File Ingest Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/ingest.yaml"
        ```

    [Download Recipe](assets/v2/ingest.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

In this scenario, we read lines from a text file into the graph to create a cloud of disconnected nodes that can be operated on later or analyzed individually.

This recipe demonstrates the most basic of ingest streams possible in Quine.

## Sample Data

This recipe accepts any text file as input.

## How it Works

The recipe reads lines from the source data file using an [ingest stream](../learn/ingest-sources/index.md) to manifest a graph in Quine.

=== "YAML"

    ```yaml
      - type: FileIngest
        path: $in_file
        format:
          type: CypherLine
          query: |-
            MATCH (n)
            WHERE id(n) = idFrom($that)
            SET n.line = $that
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "$in_file",
      "format": {
        "type": "CypherLine",
        "query": "MATCH (n)\nWHERE id(n) = idFrom($that)\nSET n.line = $that"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: file-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Line
        query: |-
          MATCH (n)
          WHERE id(n) = idFrom($that)
          SET n.line = $that
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "file-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Line"
        }
      },
      "query": "MATCH (n)\nWHERE id(n) = idFrom($that)\nSET n.line = $that"
    }
    ```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r ingest.yaml --recipe-value in_file={$filename}
Graph is ready
Running Recipe: Ingest
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
INGEST-1 status is completed and ingested 112 
```

## Summary

Open your browser and navigate to `http://localhost:8080`. Click the query bar and select `CALL recentNodes(10)` from the drop down list, then click the "Query" button.

A jumbled graph will appear in your browser, hover over the nodes to see the text from the file stored in the "line" parameter.

![Ingest Nodes](images/ingest-nodes.png)


---
# IMDB Movie Data
URL: https://quine.io/recipes/movieData/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    Explore a standard graph data set using Quine to combine data from separate sources, then generates a new event stream from the combined data.

    ??? example "IMDB Movie Data Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/movieData.yaml"
        ```

    [Download Recipe](assets/movieData.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

=== "Recipe v2"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    Explore a standard graph data set using Quine to combine data from separate sources, then generates a new event stream from the combined data.

    ??? example "IMDB Movie Data Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/movieData.yaml"
        ```

    [Download Recipe](assets/v2/movieData.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

## Scenario

In this scenario, Quine combines data from multiple CSV files into one graph. As the graph is formed, a standing query matches and reports every instance of when a person is both the actor and director of a movie.

## Sample Data

The sample data for this recipe is provided in two csv files that were exported from a relational database.

![Movie Data](images/movieData.png){ style="display: block; padding: 5px; margin-left: auto; margin-right: auto; height: 350px;" }

**File 1: `movieData.csv`** contains the `Person`, `Movie`, and `Join` rows.

[Download movieData.csv](https://quine-recipe-public.s3.us-west-2.amazonaws.com/movieData.csv)

**File 2: `ratingData.csv`** contains rows of ratings.

[Download ratingData.csv](https://quine-recipe-public.s3.us-west-2.amazonaws.com/ratingData.csv)

Click on the buttons above to download the sample data into the same directory where Quine will be run.

## How it Works

This recipe parses the CSV files using [ingest streams](../learn/ingest-sources/index.md). We used multiple ingests streams to parse the `movieData.csv` file to highlight how the Cypher acts on each data structure. Creating a single ingest stream for the movie data would be more efficient.

The recipe ingests the CSV files to create the following graph shape.

![Movie Data Graph](images/movieDataGraph.png){ style="display: block; padding: 5px; margin-left: auto; margin-right: auto; height: 350px;" }

### Movie and Genre Nodes

The first ingest stream matches the `movieData.csv` file's rows containing `Movie` entities. These rows are parsed, turned into `Movie` and `Genre` nodes, and filled with properties. `Genre` nodes are connected to `Movie` nodes in the graph.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $movie_file
      format:
        type: CypherCsv
        headers: true
        query: |-
          WITH $that AS row
          MATCH (m)
          WHERE row.Entity = 'Movie'
            AND id(m) = idFrom("Movie", row.movieId)
          SET
            m:Movie,
            m.tmdbId = row.tmdbId,
            m.imdbId = row.imdbId,
            m.imdbRating = toFloat(row.imdbRating),
            m.released = row.released,
            m.title = row.title,
            m.year = toInteger(row.year),
            m.poster = row.poster,
            m.runtime = toInteger(row.runtime),
            m.countries = split(coalesce(row.countries,""), "|"),
            m.imdbVotes = toInteger(row.imdbVotes),
            m.revenue = toInteger(row.revenue),
            m.plot = row.plot,
            m.url = row.url,
            m.budget = toInteger(row.budget),
            m.languages = split(coalesce(row.languages,""), "|"),
            m.movieId = row.movieId
          WITH m,split(coalesce(row.genres,""), "|") AS genres
          UNWIND genres AS genre
          WITH m, genre
          MATCH (g)
          WHERE id(g) = idFrom("Genre", genre)
          SET g.genre = genre, g:Genre
          CREATE (m:Movie)-[:IN_GENRE]->(g:Genre)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "$movie_file",
      "format": {
        "type": "CypherCsv",
        "headers": true,
        "query": "WITH $that AS row\nMATCH (m) \nWHERE row.Entity = 'Movie' \n  AND id(m) = idFrom(\"Movie\", row.movieId)\nSET\n  m:Movie,\n  m.tmdbId = row.tmdbId,\n  m.imdbId = row.imdbId,\n  m.imdbRating = toFloat(row.imdbRating),\n  m.released = row.released,\n  m.title = row.title,\n  m.year = toInteger(row.year),\n  m.poster = row.poster,\n  m.runtime = toInteger(row.runtime),\n  m.countries = split(coalesce(row.countries,\"\"), \"|\"),\n  m.imdbVotes = toInteger(row.imdbVotes),\n  m.revenue = toInteger(row.revenue),\n  m.plot = row.plot,\n  m.url = row.url,\n  m.budget = toInteger(row.budget),\n  m.languages = split(coalesce(row.languages,\"\"), \"|\"),\n  m.movieId = row.movieId\nWITH m,split(coalesce(row.genres,\"\"), \"|\") AS genres\nUNWIND genres AS genre\nWITH m, genre\nMATCH (g) \nWHERE id(g) = idFrom(\"Genre\", genre)\nSET g.genre = genre, g:Genre\nCREATE (m:Movie)-[:IN_GENRE]->(g:Genre)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: movie-ingest
        source:
          type: File
          path: $movie_file
          format:
            type: CSV
            headers: true
        query: |-
          WITH $that AS row
          MATCH (m)
          WHERE row.Entity = 'Movie'
            AND id(m) = idFrom("Movie", row.movieId)
          SET
            m:Movie,
            m.tmdbId = row.tmdbId,
            m.imdbId = row.imdbId,
            m.imdbRating = toFloat(row.imdbRating),
            m.released = row.released,
            m.title = row.title,
            m.year = toInteger(row.year),
            m.poster = row.poster,
            m.runtime = toInteger(row.runtime),
            m.countries = split(coalesce(row.countries,""), "|"),
            m.imdbVotes = toInteger(row.imdbVotes),
            m.revenue = toInteger(row.revenue),
            m.plot = row.plot,
            m.url = row.url,
            m.budget = toInteger(row.budget),
            m.languages = split(coalesce(row.languages,""), "|"),
            m.movieId = row.movieId
          WITH m,split(coalesce(row.genres,""), "|") AS genres
          UNWIND genres AS genre
          WITH m, genre
          MATCH (g)
          WHERE id(g) = idFrom("Genre", genre)
          SET g.genre = genre, g:Genre
          CREATE (m:Movie)-[:IN_GENRE]->(g:Genre)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "movie-ingest",
      "source": {
        "type": "File",
        "path": "$movie_file",
        "format": {
          "type": "CSV",
          "headers": true
        }
      },
      "query": "WITH $that AS row\nMATCH (m)\nWHERE row.Entity = 'Movie'\n  AND id(m) = idFrom(\"Movie\", row.movieId)\nSET\n  m:Movie,\n  m.tmdbId = row.tmdbId,\n  m.imdbId = row.imdbId,\n  m.imdbRating = toFloat(row.imdbRating),\n  m.released = row.released,\n  m.title = row.title,\n  m.year = toInteger(row.year),\n  m.poster = row.poster,\n  m.runtime = toInteger(row.runtime),\n  m.countries = split(coalesce(row.countries,\"\"), \"|\"),\n  m.imdbVotes = toInteger(row.imdbVotes),\n  m.revenue = toInteger(row.revenue),\n  m.plot = row.plot,\n  m.url = row.url,\n  m.budget = toInteger(row.budget),\n  m.languages = split(coalesce(row.languages,\"\"), \"|\"),\n  m.movieId = row.movieId\nWITH m,split(coalesce(row.genres,\"\"), \"|\") AS genres\nUNWIND genres AS genre\nWITH m, genre\nMATCH (g)\nWHERE id(g) = idFrom(\"Genre\", genre)\nSET g.genre = genre, g:Genre\nCREATE (m:Movie)-[:IN_GENRE]->(g:Genre)"
    }
    ```

### Person Nodes

The second ingest stream matches the `movieData.csv` file's rows containing `Person` entities. These rows are parsed, turned into `Person` nodes, and filled with properties.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $movie_file
      format:
        type: CypherCsv
        headers: true
        query: |-
          WITH $that AS row
          MATCH (p)
          WHERE row.Entity = "Person"
            AND id(p) = idFrom("Person", row.tmdbId)
          SET
            p:Person,
            p.imdbId = row.imdbId,
            p.bornIn = row.bornIn,
            p.name = row.name,
            p.bio = row.bio,
            p.poster = row.poster,
            p.url = row.url,
            p.born = row.born,
            p.died = row.died,
            p.tmdbId = row.tmdbId,
            p.born = CASE row.born WHEN "" THEN null ELSE datetime(row.born + "T00:00:00Z") END,
            p.died = CASE row.died WHEN "" THEN null ELSE datetime(row.died + "T00:00:00Z") END
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-2"
    {
      "type": "FileIngest",
      "path": "$movie_file",
      "format": {
        "type": "CypherCsv",
        "headers": true,
        "query": "WITH $that AS row\nMATCH (p) \nWHERE row.Entity = \"Person\" \n  AND id(p) = idFrom(\"Person\", row.tmdbId)\nSET\n  p:Person,\n  p.imdbId = row.imdbId,\n  p.bornIn = row.bornIn,\n  p.name = row.name,\n  p.bio = row.bio,\n  p.poster = row.poster,\n  p.url = row.url,\n  p.born = row.born,\n  p.died = row.died,\n  p.tmdbId = row.tmdbId,\n  p.born = CASE row.born WHEN \"\" THEN null ELSE datetime(row.born + \"T00:00:00Z\") END,\n  p.died = CASE row.died WHEN \"\" THEN null ELSE datetime(row.died + \"T00:00:00Z\") END"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: person-ingest
        source:
          type: File
          path: $movie_file
          format:
            type: CSV
            headers: true
        query: |-
          WITH $that AS row
          MATCH (p)
          WHERE row.Entity = "Person"
            AND id(p) = idFrom("Person", row.tmdbId)
          SET
            p:Person,
            p.imdbId = row.imdbId,
            p.bornIn = row.bornIn,
            p.name = row.name,
            p.bio = row.bio,
            p.poster = row.poster,
            p.url = row.url,
            p.born = row.born,
            p.died = row.died,
            p.tmdbId = row.tmdbId,
            p.born = CASE row.born WHEN "" THEN null ELSE datetime(row.born + "T00:00:00Z") END,
            p.died = CASE row.died WHEN "" THEN null ELSE datetime(row.died + "T00:00:00Z") END
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "person-ingest",
      "source": {
        "type": "File",
        "path": "$movie_file",
        "format": {
          "type": "CSV",
          "headers": true
        }
      },
      "query": "WITH $that AS row\nMATCH (p)\nWHERE row.Entity = \"Person\"\n  AND id(p) = idFrom(\"Person\", row.tmdbId)\nSET\n  p:Person,\n  p.imdbId = row.imdbId,\n  p.bornIn = row.bornIn,\n  p.name = row.name,\n  p.bio = row.bio,\n  p.poster = row.poster,\n  p.url = row.url,\n  p.born = row.born,\n  p.died = row.died,\n  p.tmdbId = row.tmdbId,\n  p.born = CASE row.born WHEN \"\" THEN null ELSE datetime(row.born + \"T00:00:00Z\") END,\n  p.died = CASE row.died WHEN \"\" THEN null ELSE datetime(row.died + \"T00:00:00Z\") END"
    }
    ```

### Role Nodes

The third ingest stream matches the `movieData.csv` file's rows containing `Join` entities that have `Acting` in the `Work` column. These rows are parsed, turned into `Role` nodes, filled with properties, and connected to the graph. Additionally, the ACTED_IN relationship is set between the `Person` and `Movie` nodes.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $movie_file
      format:
        type: CypherCsv
        headers: true
        query: |-
          WITH $that AS row
          WITH row
          WHERE row.Entity = "Join"
            AND row.Work = "Acting"
          MATCH (p), (m), (r)
          WHERE id(p) = idFrom("Person", row.tmdbId)
            AND id(m) = idFrom("Movie", row.movieId)
            AND id(r) = idFrom("Role", row.tmdbId, row.movieId, row.role)
          SET
            r.role = row.role,
            r.movie = row.movieId,
            r.tmdbId = row.tmdbId,
            r:Role
          CREATE (p:Person)-[:PLAYED]->(r:Role)<-[:HAS_ROLE]-(m:Movie)
          CREATE (p:Person)-[:ACTED_IN]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-3"
    {
      "type": "FileIngest",
      "path": "$movie_file",
      "format": {
        "type": "CypherCsv",
        "headers": true,
        "query": "WITH $that AS row\nWITH row \nWHERE row.Entity = \"Join\" \n  AND row.Work = \"Acting\"\nMATCH (p), (m), (r) \nWHERE id(p) = idFrom(\"Person\", row.tmdbId)\n  AND id(m) = idFrom(\"Movie\", row.movieId)\n  AND id(r) = idFrom(\"Role\", row.tmdbId, row.movieId, row.role)\nSET \n  r.role = row.role, \n  r.movie = row.movieId, \n  r.tmdbId = row.tmdbId, \n  r:Role\nCREATE (p:Person)-[:PLAYED]->(r:Role)<-[:HAS_ROLE]-(m:Movie)\nCREATE (p:Person)-[:ACTED_IN]->(m:Movie)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: acting-ingest
        source:
          type: File
          path: $movie_file
          format:
            type: CSV
            headers: true
        query: |-
          WITH $that AS row
          WITH row
          WHERE row.Entity = "Join"
            AND row.Work = "Acting"
          MATCH (p), (m), (r)
          WHERE id(p) = idFrom("Person", row.tmdbId)
            AND id(m) = idFrom("Movie", row.movieId)
            AND id(r) = idFrom("Role", row.tmdbId, row.movieId, row.role)
          SET
            r.role = row.role,
            r.movie = row.movieId,
            r.tmdbId = row.tmdbId,
            r:Role
          CREATE (p:Person)-[:PLAYED]->(r:Role)<-[:HAS_ROLE]-(m:Movie)
          CREATE (p:Person)-[:ACTED_IN]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "acting-ingest",
      "source": {
        "type": "File",
        "path": "$movie_file",
        "format": {
          "type": "CSV",
          "headers": true
        }
      },
      "query": "WITH $that AS row\nWITH row\nWHERE row.Entity = \"Join\"\n  AND row.Work = \"Acting\"\nMATCH (p), (m), (r)\nWHERE id(p) = idFrom(\"Person\", row.tmdbId)\n  AND id(m) = idFrom(\"Movie\", row.movieId)\n  AND id(r) = idFrom(\"Role\", row.tmdbId, row.movieId, row.role)\nSET\n  r.role = row.role,\n  r.movie = row.movieId,\n  r.tmdbId = row.tmdbId,\n  r:Role\nCREATE (p:Person)-[:PLAYED]->(r:Role)<-[:HAS_ROLE]-(m:Movie)\nCREATE (p:Person)-[:ACTED_IN]->(m:Movie)"
    }
    ```

### Directed Nodes

The fourth ingest stream matches the `movieData.csv` file's rows containing `Join` entities that have `Directing` in the `Work` column. These rows are parsed and the DIRECTED relationship is created between the `Person` and `Movie` nodes.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $movie_file
      format:
        type: CypherCsv
        headers: true
        query: |-
          WITH $that AS row
          WITH row WHERE row.Entity = "Join" AND row.Work = "Directing"
          MATCH (p), (m)
          WHERE id(p) = idFrom("Person", row.tmdbId)
            AND id(m) = idFrom("Movie", row.movieId)
          CREATE (p:Person)-[:DIRECTED]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-4"
    {
      "type": "FileIngest",
      "path": "$movie_file",
      "format": {
        "type": "CypherCsv",
        "headers": true,
        "query": "WITH $that AS row\nWITH row WHERE row.Entity = \"Join\" AND row.Work = \"Directing\"\nMATCH (p), (m) \nWHERE id(p) = idFrom(\"Person\", row.tmdbId)\n  AND id(m) = idFrom(\"Movie\", row.movieId)\nCREATE (p:Person)-[:DIRECTED]->(m:Movie)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: directing-ingest
        source:
          type: File
          path: $movie_file
          format:
            type: CSV
            headers: true
        query: |-
          WITH $that AS row
          WITH row WHERE row.Entity = "Join" AND row.Work = "Directing"
          MATCH (p), (m)
          WHERE id(p) = idFrom("Person", row.tmdbId)
            AND id(m) = idFrom("Movie", row.movieId)
          CREATE (p:Person)-[:DIRECTED]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "directing-ingest",
      "source": {
        "type": "File",
        "path": "$movie_file",
        "format": {
          "type": "CSV",
          "headers": true
        }
      },
      "query": "WITH $that AS row\nWITH row WHERE row.Entity = \"Join\" AND row.Work = \"Directing\"\nMATCH (p), (m)\nWHERE id(p) = idFrom(\"Person\", row.tmdbId)\n  AND id(m) = idFrom(\"Movie\", row.movieId)\nCREATE (p:Person)-[:DIRECTED]->(m:Movie)"
    }
    ```

### Rating Nodes

The fifth ingest stream matches rows from the `ratingsData.csv` file to create `User` and `Rating` nodes, fill them with parameters, and connect them into the graph.

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $rating_file
      format:
        type: CypherCsv
        headers: true
        query: |-
          WITH $that AS row
          MATCH (m), (u), (rtg)
          WHERE id(m) = idFrom("Movie", row.movieId)
            AND id(u) = idFrom("User", row.userId)
            AND id(rtg) = idFrom("Rating", row.movieId, row.userId, row.rating)
          SET u.name = row.name, u:User
          SET rtg.rating = row.rating,
            rtg.timestamp = toInteger(row.timestamp),
            rtg:Rating
          CREATE (u:User)-[:SUBMITTED]->(rtg:Rating)<-[:HAS_RATING]-(m:Movie)
          CREATE (u:User)-[:RATED]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-5"
    {
      "type": "FileIngest",
      "path": "$rating_file",
      "format": {
        "type": "CypherCsv",
        "headers": true,
        "query": "WITH $that AS row\nMATCH (m), (u), (rtg) \nWHERE id(m) = idFrom(\"Movie\", row.movieId)\n  AND id(u) = idFrom(\"User\", row.userId)\n  AND id(rtg) = idFrom(\"Rating\", row.movieId, row.userId, row.rating)\nSET u.name = row.name, u:User\nSET rtg.rating = row.rating,\n  rtg.timestamp = toInteger(row.timestamp),\n  rtg:Rating\nCREATE (u:User)-[:SUBMITTED]->(rtg:Rating)<-[:HAS_RATING]-(m:Movie)\nCREATE (u:User)-[:RATED]->(m:Movie)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: rating-ingest
        source:
          type: File
          path: $rating_file
          format:
            type: CSV
            headers: true
        query: |-
          WITH $that AS row
          MATCH (m), (u), (rtg)
          WHERE id(m) = idFrom("Movie", row.movieId)
            AND id(u) = idFrom("User", row.userId)
            AND id(rtg) = idFrom("Rating", row.movieId, row.userId, row.rating)
          SET u.name = row.name, u:User
          SET rtg.rating = row.rating,
            rtg.timestamp = toInteger(row.timestamp),
            rtg:Rating
          CREATE (u:User)-[:SUBMITTED]->(rtg:Rating)<-[:HAS_RATING]-(m:Movie)
          CREATE (u:User)-[:RATED]->(m:Movie)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "rating-ingest",
      "source": {
        "type": "File",
        "path": "$rating_file",
        "format": {
          "type": "CSV",
          "headers": true
        }
      },
      "query": "WITH $that AS row\nMATCH (m), (u), (rtg)\nWHERE id(m) = idFrom(\"Movie\", row.movieId)\n  AND id(u) = idFrom(\"User\", row.userId)\n  AND id(rtg) = idFrom(\"Rating\", row.movieId, row.userId, row.rating)\nSET u.name = row.name, u:User\nSET rtg.rating = row.rating,\n  rtg.timestamp = toInteger(row.timestamp),\n  rtg:Rating\nCREATE (u:User)-[:SUBMITTED]->(rtg:Rating)<-[:HAS_RATING]-(m:Movie)\nCREATE (u:User)-[:RATED]->(m:Movie)"
    }
    ```

### Acted and Directed

A [standing query](../learn/standing-queries/standing-queries.md) detects when an actor (`Person`) has both the `ACTED_IN` and `DIRECTED` relationship to the same `Movie`.

When a pattern match is found, the `ActedDirected` relationship is created between the `Person` and `Movie` nodes in the graph, and an alert is written into the `ActorDirector.jsonl` file.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        mode: MultipleValues
        query: |-
          MATCH (a:Movie)<-[:ACTED_IN]-(p:Person)-[:DIRECTED]->(m:Movie)
          WHERE id(a) = id(m)
          RETURN id(m) as movieId, id(p) as personId
      outputs:
        set-ActedDirected:
          type: CypherQuery
          query: |-
            MATCH (m),(p)
            WHERE id(m) = $that.data.movieId
              AND id(p) = $that.data.personId
            WITH *
            CREATE (p:Person)-[:ActedDirected]->(m:Movie)
            RETURN id(m) as movieId, m.title as Movie, id(p) as personId, p.name as Actor
          andThen:
            type: WriteToFile
            path: "ActorDirector.jsonl"
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "mode": "MultipleValues",
        "query": "MATCH (a:Movie)<-[:ACTED_IN]-(p:Person)-[:DIRECTED]->(m:Movie) \nWHERE id(a) = id(m)\nRETURN id(m) as movieId, id(p) as personId"
      },
      "outputs": {
        "set-ActedDirected": {
          "type": "CypherQuery",
          "query": "MATCH (m),(p)\nWHERE id(m) = $that.data.movieId \n  AND id(p) = $that.data.personId\nWITH *\nCREATE (p:Person)-[:ActedDirected]->(m:Movie)\nRETURN id(m) as movieId, m.title as Movie, id(p) as personId, p.name as Actor",
          "andThen": {
            "type": "WriteToFile",
            "path": "ActorDirector.jsonl"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: actor-director-match
        pattern:
          type: Cypher
          mode: MULTIPLE_VALUES
          query: |-
            MATCH (a:Movie)<-[:ACTED_IN]-(p:Person)-[:DIRECTED]->(m:Movie)
            WHERE id(a) = id(m)
            RETURN id(m) as movieId, id(p) as personId
        outputs:
          - name: set-ActedDirected
            resultEnrichment:
              query: |-
                MATCH (m),(p)
                WHERE id(m) = $that.data.movieId
                  AND id(p) = $that.data.personId
                WITH *
                CREATE (p:Person)-[:ActedDirected]->(m:Movie)
                RETURN id(m) as movieId, m.title as Movie, id(p) as personId, p.name as Actor
              parameter: that
            destinations:
              - type: File
                path: "ActorDirector.jsonl"
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "actor-director-match",
      "pattern": {
        "type": "Cypher",
        "mode": "MULTIPLE_VALUES",
        "query": "MATCH (a:Movie)<-[:ACTED_IN]-(p:Person)-[:DIRECTED]->(m:Movie)\nWHERE id(a) = id(m)\nRETURN id(m) as movieId, id(p) as personId"
      },
      "outputs": [
        {
          "name": "set-ActedDirected",
          "resultEnrichment": {
            "query": "MATCH (m),(p)\nWHERE id(m) = $that.data.movieId\n  AND id(p) = $that.data.personId\nWITH *\nCREATE (p:Person)-[:ActedDirected]->(m:Movie)\nRETURN id(m) as movieId, m.title as Movie, id(p) as personId, p.name as Actor",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "File",
              "path": "ActorDirector.jsonl"
            }
          ]
        }
      ]
    }
    ```

The output object contains information about the match and the nodes matching the query.

``` json
{
       "meta": {
              "isPositiveMatch": true,
              "resultId": "d2008617-cc5c-4f81-8472-f3db277f8da2"
       },
       "data": {
              "Actor": "Clint Eastwood",
              "Movie": "Unforgiven",
              "movieId": "4a6d64c8-9c90-3362-b443-4d2e7b2fb9d1",
              "personId": "4638a820-3b68-3fc7-9fa7-341e876b701e"
       }
}
```

## Running the Recipe

```shell
java \
-jar quine-2.1.1.jar -r movieData.yaml\
--recipe-value movie_file=movieData.csv \
--recipe-value rating_file=ratingData.csv
```

??? Tip

    This recipe will create an `ActorDirector.jsonl` file in the local directory that you should remove before each run. We found it easier to launch Quine using the following shell script so that we didn't forget to clean up output from previous runs.

    ```shell title="run-recipe.sh"
    #!/bin/bash

    [ -f ActorDirector.jsonl ] && rm ActorDirector.jsonl

    java \
    -jar quine-2.1.1.jar -r movieData.yaml\
    --recipe-value movie_file=movieData.csv \
    --recipe-value rating_file=ratingData.csv
    ```

Launching Quine directly or using `run-recipe.sh` will produce output similar to this.

``` shell
❯ ./run_recipe.sh
Graph is ready
Running Recipe: Ingesting CSV Files
Using 6 node appearances
Using 8 sample queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Running Ingest Stream INGEST-2
Running Ingest Stream INGEST-3
Running Ingest Stream INGEST-4
Running Ingest Stream INGEST-5
Quine web server available at http://localhost:8080
INGEST-1 status is completed and ingested 74090
INGEST-2 status is completed and ingested 74090
INGEST-3 status is completed and ingested 74090
INGEST-4 status is completed and ingested 74090
INGEST-5 status is completed and ingested 100005

 | => STANDING-1 count 491
```

## Summary

Be sure to open Quine in your browser using the URL provided in your terminal window. Several sample queries are ready for you to use in the Exploration UI query bar. Click on the query bar and launch the queries by pressing the Query button.

!!! Tip

    Submit text queries with ++shift+enter++ to avoid the Exploration UI sending back an error.

| Sample Query                         | Type | Description                                                           |
| :----------------------------------- | :--- | :-------------------------------------------------------------------- |
| Sample of Nodes                      | Node | Return a sample of nodes from the graph                               |
| Count Nodes                          | Text | Count the types of nodes in the graph                                 |
| Count Relationships                  | Text | Count the types of relationships in the graph                         |
| Movie Genres                         | Node | Return the movie genres nodes from the graph                          |
| Person Acted In a movie              | Node | Return all of the actor nodes from the graph                          |
| Person Directed a movie              | Node | Return all of the director nodes from the graph                       |
| Person Acted In and Directed a movie | Node | Return all of the actor and movie nodes where the actor also directed |
| User Rated a movie                   | Node | Return all of the rating nodes from the graph                         |

The results from the `Person Acted In and Directed a movie` create an interesting shape you can explore further with your queries.

![Actor Director Graph](images/movieDataActorDirectors.png)


---
# Password Spraying Detection
URL: https://quine.io/recipes/password-spraying/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    Ingests JSON-formatted IAM-style password authentication log file and creates relationships to detect Password Spraying attacks.

    ??? example "Password Spraying Detection Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/password_spraying.yml"
        ```

    [Download Recipe](assets/password_spraying.yml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

=== "Recipe v2"

    <small>Shared by: [Allan Konar](https://github.com/7evenbridges)</small>

    Ingests JSON-formatted IAM-style password authentication log file and creates relationships to detect Password Spraying attacks.

    ??? example "Password Spraying Detection Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/password_spraying.yaml"
        ```

    [Download Recipe](assets/v2/password_spraying.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click"  }

## Scenario

In this scenario, Quine ingests password-based authentication logs modeled on the top IAM providers (hosted and on-prem) and generates a graph manifesting the following nodes:

* **attempt** - transaction representing a password authentication attempt
* **user** - user that originated the attempt
* **client** - client (computer/mobile/unknown) from which user originated the attempt
* **asn** - ASN from which user originated the attempt
* **asset** - asset (server, service, etc.) that the user targeted
* **time** - time of attempt

The first standing query uses the manifested graph structure to generate synthetic edges between sequential attempts for a user:

``` cypher
(attempt1)-[:NEXT]->(attempt2)-[:NEXT]->(attempt3)
```

The second standing query looks for four consecutive failed attempts followed by a successful attempt from a user to trigger an alert with a link to the subgraph that represents a potential password spraying attack.

``` cypher
(attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->
(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->
(attempt5 {outcomeResult:"SUCCESS"})-[:USING]->(client4)
```

## Sample Data

Ensure that the attempts.json file is in the same directory as Quine and issue the following command to begin:

[Download attempts.json](https://that.re/attempts)

## How it Works

A single [ingest stream](../learn/ingest-sources/index.md) does a lot of work for us to parse each line into multiple nodes.

We use `idFrom()` to create the node IDs from each with a unique namespace and event parameters to ensure uniqueness.

``` cypher title="Locate empty nodes in the graph"
MATCH (attempt), (client), (asn), (user), (asset)
WHERE id(attempt) = idFrom('attempt', $that.eventId, $that.timestamp)
  AND id(client) = idFrom('client', $that.user.id, $that.client.ipAddress)
  AND id(asn) = idFrom('asn', $that.client.asn)
  AND id(user) = idFrom('user', $that.user.id)
  AND id(asset) = idFrom('asset', $that.transaction.entityId)
```

A metric is set for the number of times an event occurs within the client, user, and asset nodes. This counter is used later to calculate the attempt success/fail ratio for specific assets.

``` cypher
//////////////////////////////
// Bucketing for counters
//////////////////////////////
CALL incrementCounter(client, "clientCount", 1) YIELD count AS clientCount
CALL incrementCounter(client, toLower($that.outcome.result), 1) YIELD count AS clientOutcomeCount
CALL incrementCounter(user, "userCount", 1) YIELD count AS userCount
CALL incrementCounter(user, toLower($that.outcome.result), 1) YIELD count AS userOutcomeCount
CALL incrementCounter(asset, "assetCount", 1) YIELD count AS assetCount
CALL incrementCounter(asset, toLower($that.outcome.result), 1) YIELD count AS assetOutcomeCount
```

Each node is then filled with parameters derived from the event itself.

=== "Client"

    ``` cypher title="Create parameters for Client nodes"
    //////////////////////////////
    // Client
    //////////////////////////////
    SET client.device = $that.client.device,
        client.ipAddress = $that.client.ipAddress,
        client.userAgent = $that.client.userAgent,
        client: client

    // Identify last time client seen across clients //
    SET client.lastseen = coll.max([$that.timestamp, coalesce(client.lastseen, $that.timestamp)])

    // Percentage of success vs. failure //
    SET client.successPercent = ceil(coalesce((client.success*1.0)/(client.count*1.0)*100.0, 0.0))
    SET client.failurePercent = floor(coalesce((client.failure*1.0)/(client.count*1.0)*100.0, 0.0))
    SET client.state = CASE
        // Set threshold ratios below for each of three cases //
        WHEN client.successPercent >= 90 THEN 'good'
        WHEN client.successPercent >= 75 AND client.successPercent < 90 THEN 'warn'
        WHEN client.successPercent < 75 THEN 'alarm'
        ELSE 'alarm'
      END
    ```

=== "User"

    ``` cypher title="Create parameters for User nodes"
    //////////////////////////////
    // User
    //////////////////////////////
    SET user.id = $that.user.id,
        user.alternateId = $that.user.alternateId,
        user.displayName = $that.user.displayName,
        user.type = $that.user.type,
        user: user

    // Identify last time user seen across users //
    SET user.lastseen = coll.max([$that.timestamp, coalesce(user.lastseen, $that.timestamp)])

    // Percentage of success vs. failure //
    SET user.successPercent = ceil(coalesce((user.success*1.0)/(user.count*1.0)*100.0, 0.0))
    SET user.failurePercent = floor(coalesce((user.failure*1.0)/(user.count*1.0)*100.0, 0.0))
    SET user.state = CASE
        // Set threshold ratios below for each of three cases //
        WHEN user.successPercent >= 90 THEN 'good'
        WHEN user.successPercent >= 75 AND user.successPercent < 90 THEN 'warn'
        WHEN user.successPercent < 75 THEN 'alarm'
        ELSE 'alarm'
      END
    ```

=== "Attempts"

    ``` cypher title="Create parameters for Attempts nodes"
    //////////////////////////////
    // Attempts
    //////////////////////////////
    SET attempt.schemaVersion = $that.schemaVersion,
        attempt.eventId = $that.eventId,
        attempt.transactionId = $that.transaction.id,
        attempt.timestamp = $that.timestamp,
        attempt.entityId = $that.transaction.entityId,
        attempt.eventType = $that.eventType,
        attempt.transactionType = $that.transaction.type,
        attempt.eventCode = $that.eventCode,
        attempt.displayMessage = $that.displayMessage,
        attempt.outcomeResult = $that.outcome.result,
        attempt.logLevel = $that.level,
        attempt.zone = $that.client.zone,
        attempt.client = $that.client.ipAddress,
        attempt.userSequence = coalesce(userCount,0),
        attempt.clientSequence = coalesce(clientCount,0),
        attempt: attempt
    ```

=== "ASN"

    ``` cypher title="Create parameters for ASN nodes"
    //////////////////////////////
    // ASN
    //////////////////////////////
    SET asn.id = $that.client.asn,
        asn: asn
    ```

=== "Asset"

    ``` cypher title="Create parameters for Asset nodes"
    //////////////////////////////
    // Asset
    //////////////////////////////
    SET asset.id = $that.transaction.entityId,
        asset.detail = $that.client.requestUri,
        asset: asset

    // Percentage of success vs. failure //
    SET asset.successPercent = ceil(coalesce((asset.success*1.0)/(asset.count*1.0)*100.0, 0.0))
    SET asset.failurePercent = floor(coalesce((asset.failure*1.0)/(asset.count*1.0)*100.0, 0.0))
    SET asset.state = CASE
        // Set threshold ratios below for each of three cases //
        WHEN asset.successPercent >= 90 THEN 'good'
        WHEN asset.successPercent >= 75 AND asset.successPercent < 90 THEN 'warn'
        WHEN asset.successPercent < 75 THEN 'alarm'
        ELSE 'alarm'
      END
    ```

Finally, relationships are created for all of the nodes generated from the event. 

``` cypher
//////////////////////////////
// Create relationship between nodes
//////////////////////////////
CREATE (user)-[:ORIGINATED]->(attempt)-[:USING]->(client),
        (client)<-[:USING]-(attempt)-[:TARGETED]->(asset),
        (user)-[:ORIGINATED]->(attempt)-[:TARGETED]->(asset),
        (attempt)-[:OVER]->(asn)
```

The complete INGEST-1 ingest stream configuration processes the `endpoints.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: attempts.json
        format:
          type: CypherJson
          query: >-
              MATCH (attempt), (client), (asn), (user), (asset)
              WHERE id(attempt) = idFrom('attempt', $that.eventId, $that.timestamp)
                AND id(client) = idFrom('client', $that.user.id, $that.client.ipAddress)
                AND id(asn) = idFrom('asn', $that.client.asn)
                AND id(user) = idFrom('user', $that.user.id)
                AND id(asset) = idFrom('asset', $that.transaction.entityId)

              CALL incrementCounter(client, "clientCount", 1) YIELD count AS clientCount
              CALL incrementCounter(client, toLower($that.outcome.result), 1) YIELD count AS clientOutcomeCount
              CALL incrementCounter(user, "userCount", 1) YIELD count AS userCount
              CALL incrementCounter(user, toLower($that.outcome.result), 1) YIELD count AS userOutcomeCount
              CALL incrementCounter(asset, "assetCount", 1) YIELD count AS assetCount
              CALL incrementCounter(asset, toLower($that.outcome.result), 1) YIELD count AS assetOutcomeCount

              SET client.device = $that.client.device,
                  client.ipAddress = $that.client.ipAddress,
                  client.userAgent = $that.client.userAgent,
                  client: client
              SET client.lastseen = coll.max([$that.timestamp, coalesce(client.lastseen, $that.timestamp)])
              SET client.successPercent = ceil(coalesce((client.success*1.0)/(client.count*1.0)*100.0, 0.0))
              SET client.failurePercent = floor(coalesce((client.failure*1.0)/(client.count*1.0)*100.0, 0.0))
              SET client.state = CASE
                  WHEN client.successPercent >= 90 THEN 'good'
                  WHEN client.successPercent >= 75 AND client.successPercent < 90 THEN 'warn'
                  WHEN client.successPercent < 75 THEN 'alarm'
                  ELSE 'alarm'
                END
              SET user.id = $that.user.id,
                  user.alternateId = $that.user.alternateId,
                  user.displayName = $that.user.displayName,
                  user.type = $that.user.type,
                  user: user
              SET user.lastseen = coll.max([$that.timestamp, coalesce(user.lastseen, $that.timestamp)])
              SET user.successPercent = ceil(coalesce((user.success*1.0)/(user.count*1.0)*100.0, 0.0))
              SET user.failurePercent = floor(coalesce((user.failure*1.0)/(user.count*1.0)*100.0, 0.0))
              SET user.state = CASE
                  WHEN user.successPercent >= 90 THEN 'good'
                  WHEN user.successPercent >= 75 AND user.successPercent < 90 THEN 'warn'
                  WHEN user.successPercent < 75 THEN 'alarm'
                  ELSE 'alarm'
                END
              SET attempt.schemaVersion = $that.schemaVersion,
                  attempt.eventId = $that.eventId,
                  attempt.transactionId = $that.transaction.id,
                  attempt.timestamp = $that.timestamp,
                  attempt.entityId = $that.transaction.entityId,
                  attempt.eventType = $that.eventType,
                  attempt.transactionType = $that.transaction.type,
                  attempt.eventCode = $that.eventCode,
                  attempt.displayMessage = $that.displayMessage,
                  attempt.outcomeResult = $that.outcome.result,
                  attempt.logLevel = $that.level,
                  attempt.zone = $that.client.zone,
                  attempt.client = $that.client.ipAddress,
                  attempt.userSequence = coalesce(userCount,0),
                  attempt.clientSequence = coalesce(clientCount,0),
                  attempt: attempt
              SET asn.id = $that.client.asn,
                  asn: asn
              SET asset.id = $that.transaction.entityId,
                  asset.detail = $that.client.requestUri,
                  asset: asset
              SET asset.successPercent = ceil(coalesce((asset.success*1.0)/(asset.count*1.0)*100.0, 0.0))
              SET asset.failurePercent = floor(coalesce((asset.failure*1.0)/(asset.count*1.0)*100.0, 0.0))
              SET asset.state = CASE
                  WHEN asset.successPercent >= 90 THEN 'good'
                  WHEN asset.successPercent >= 75 AND asset.successPercent < 90 THEN 'warn'
                  WHEN asset.successPercent < 75 THEN 'alarm'
                  ELSE 'alarm'
                END

              CREATE (user)-[:ORIGINATED]->(attempt)-[:USING]->(client),
                      (client)<-[:USING]-(attempt)-[:TARGETED]->(asset),
                      (user)-[:ORIGINATED]->(attempt)-[:TARGETED]->(asset),
                      (attempt)-[:OVER]->(asn)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "attempts.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (attempt), (client), (asn), (user), (asset) WHERE id(attempt) = idFrom('attempt', $that.eventId, $that.timestamp)\n  AND id(client) = idFrom('client', $that.user.id, $that.client.ipAddress)\n  AND id(asn) = idFrom('asn', $that.client.asn)\n  AND id(user) = idFrom('user', $that.user.id)\n  AND id(asset) = idFrom('asset', $that.transaction.entityId)\n\nCALL incrementCounter(client, \"clientCount\", 1) YIELD count AS clientCount CALL incrementCounter(client, toLower($that.outcome.result), 1) YIELD count AS clientOutcomeCount CALL incrementCounter(user, \"userCount\", 1) YIELD count AS userCount CALL incrementCounter(user, toLower($that.outcome.result), 1) YIELD count AS userOutcomeCount CALL incrementCounter(asset, \"assetCount\", 1) YIELD count AS assetCount CALL incrementCounter(asset, toLower($that.outcome.result), 1) YIELD count AS assetOutcomeCount\nSET client.device = $that.client.device,\n    client.ipAddress = $that.client.ipAddress,\n    client.userAgent = $that.client.userAgent,\n    client: client\nSET client.lastseen = coll.max([$that.timestamp, coalesce(client.lastseen, $that.timestamp)]) SET client.successPercent = ceil(coalesce((client.success*1.0)/(client.count*1.0)*100.0, 0.0)) SET client.failurePercent = floor(coalesce((client.failure*1.0)/(client.count*1.0)*100.0, 0.0)) SET client.state = CASE\n    WHEN client.successPercent >= 90 THEN 'good'\n    WHEN client.successPercent >= 75 AND client.successPercent < 90 THEN 'warn'\n    WHEN client.successPercent < 75 THEN 'alarm'\n    ELSE 'alarm'\n  END\nSET user.id = $that.user.id,\n    user.alternateId = $that.user.alternateId,\n    user.displayName = $that.user.displayName,\n    user.type = $that.user.type,\n    user: user\nSET user.lastseen = coll.max([$that.timestamp, coalesce(user.lastseen, $that.timestamp)]) SET user.successPercent = ceil(coalesce((user.success*1.0)/(user.count*1.0)*100.0, 0.0)) SET user.failurePercent = floor(coalesce((user.failure*1.0)/(user.count*1.0)*100.0, 0.0)) SET user.state = CASE\n    WHEN user.successPercent >= 90 THEN 'good'\n    WHEN user.successPercent >= 75 AND user.successPercent < 90 THEN 'warn'\n    WHEN user.successPercent < 75 THEN 'alarm'\n    ELSE 'alarm'\n  END\nSET attempt.schemaVersion = $that.schemaVersion,\n    attempt.eventId = $that.eventId,\n    attempt.transactionId = $that.transaction.id,\n    attempt.timestamp = $that.timestamp,\n    attempt.entityId = $that.transaction.entityId,\n    attempt.eventType = $that.eventType,\n    attempt.transactionType = $that.transaction.type,\n    attempt.eventCode = $that.eventCode,\n    attempt.displayMessage = $that.displayMessage,\n    attempt.outcomeResult = $that.outcome.result,\n    attempt.logLevel = $that.level,\n    attempt.zone = $that.client.zone,\n    attempt.client = $that.client.ipAddress,\n    attempt.userSequence = coalesce(userCount,0),\n    attempt.clientSequence = coalesce(clientCount,0),\n    attempt: attempt\nSET asn.id = $that.client.asn,\n    asn: asn\nSET asset.id = $that.transaction.entityId,\n    asset.detail = $that.client.requestUri,\n    asset: asset\nSET asset.successPercent = ceil(coalesce((asset.success*1.0)/(asset.count*1.0)*100.0, 0.0)) SET asset.failurePercent = floor(coalesce((asset.failure*1.0)/(asset.count*1.0)*100.0, 0.0)) SET asset.state = CASE\n    WHEN asset.successPercent >= 90 THEN 'good'\n    WHEN asset.successPercent >= 75 AND asset.successPercent < 90 THEN 'warn'\n    WHEN asset.successPercent < 75 THEN 'alarm'\n    ELSE 'alarm'\n  END\n\nCREATE (user)-[:ORIGINATED]->(attempt)-[:USING]->(client),\n        (client)<-[:USING]-(attempt)-[:TARGETED]->(asset),\n        (user)-[:ORIGINATED]->(attempt)-[:TARGETED]->(asset),\n        (attempt)-[:OVER]->(asn)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: attempts-file-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Json
        query: |-
          MATCH (attempt), (client), (asn), (user), (asset)
          WHERE id(attempt) = idFrom('attempt', $that.eventId, $that.timestamp)
            AND id(client) = idFrom('client', $that.user.id, $that.client.ipAddress)
            AND id(asn) = idFrom('asn', $that.client.asn)
            AND id(user) = idFrom('user', $that.user.id)
            AND id(asset) = idFrom('asset', $that.transaction.entityId)

          CALL incrementCounter(client, "clientCount", 1) YIELD count AS clientCount
          CALL incrementCounter(client, toLower($that.outcome.result), 1) YIELD count AS clientOutcomeCount
          CALL incrementCounter(user, "userCount", 1) YIELD count AS userCount
          CALL incrementCounter(user, toLower($that.outcome.result), 1) YIELD count AS userOutcomeCount
          CALL incrementCounter(asset, "assetCount", 1) YIELD count AS assetCount
          CALL incrementCounter(asset, toLower($that.outcome.result), 1) YIELD count AS assetOutcomeCount

          SET client.device = $that.client.device,
              client.ipAddress = $that.client.ipAddress,
              client.userAgent = $that.client.userAgent,
              client: client
          SET client.lastseen = coll.max([$that.timestamp, coalesce(client.lastseen, $that.timestamp)])
          SET client.successPercent = ceil(coalesce((client.success*1.0)/(client.count*1.0)*100.0, 0.0))
          SET client.failurePercent = floor(coalesce((client.failure*1.0)/(client.count*1.0)*100.0, 0.0))
          SET client.state = CASE
              WHEN client.successPercent >= 90 THEN 'good'
              WHEN client.successPercent >= 75 AND client.successPercent < 90 THEN 'warn'
              WHEN client.successPercent < 75 THEN 'alarm'
              ELSE 'alarm'
            END
          SET user.id = $that.user.id,
              user.alternateId = $that.user.alternateId,
              user.displayName = $that.user.displayName,
              user.type = $that.user.type,
              user: user
          SET user.lastseen = coll.max([$that.timestamp, coalesce(user.lastseen, $that.timestamp)])
          SET user.successPercent = ceil(coalesce((user.success*1.0)/(user.count*1.0)*100.0, 0.0))
          SET user.failurePercent = floor(coalesce((user.failure*1.0)/(user.count*1.0)*100.0, 0.0))
          SET user.state = CASE
              WHEN user.successPercent >= 90 THEN 'good'
              WHEN user.successPercent >= 75 AND user.successPercent < 90 THEN 'warn'
              WHEN user.successPercent < 75 THEN 'alarm'
              ELSE 'alarm'
            END
          SET attempt.schemaVersion = $that.schemaVersion,
              attempt.eventId = $that.eventId,
              attempt.transactionId = $that.transaction.id,
              attempt.timestamp = $that.timestamp,
              attempt.entityId = $that.transaction.entityId,
              attempt.eventType = $that.eventType,
              attempt.transactionType = $that.transaction.type,
              attempt.eventCode = $that.eventCode,
              attempt.displayMessage = $that.displayMessage,
              attempt.outcomeResult = $that.outcome.result,
              attempt.logLevel = $that.level,
              attempt.zone = $that.client.zone,
              attempt.client = $that.client.ipAddress,
              attempt.userSequence = coalesce(userCount,0),
              attempt.clientSequence = coalesce(clientCount,0),
              attempt: attempt
          SET asn.id = $that.client.asn,
              asn: asn
          SET asset.id = $that.transaction.entityId,
              asset.detail = $that.client.requestUri,
              asset: asset
          SET asset.successPercent = ceil(coalesce((asset.success*1.0)/(asset.count*1.0)*100.0, 0.0))
          SET asset.failurePercent = floor(coalesce((asset.failure*1.0)/(asset.count*1.0)*100.0, 0.0))
          SET asset.state = CASE
              WHEN asset.successPercent >= 90 THEN 'good'
              WHEN asset.successPercent >= 75 AND asset.successPercent < 90 THEN 'warn'
              WHEN asset.successPercent < 75 THEN 'alarm'
              ELSE 'alarm'
            END

          CREATE (user)-[:ORIGINATED]->(attempt)-[:USING]->(client),
                 (client)<-[:USING]-(attempt)-[:TARGETED]->(asset),
                 (user)-[:ORIGINATED]->(attempt)-[:TARGETED]->(asset),
                 (attempt)-[:OVER]->(asn)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "attempts-file-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (attempt), (client), (asn), (user), (asset) WHERE id(attempt) = idFrom('attempt', $that.eventId, $that.timestamp) AND id(client) = idFrom('client', $that.user.id, $that.client.ipAddress) AND id(asn) = idFrom('asn', $that.client.asn) AND id(user) = idFrom('user', $that.user.id) AND id(asset) = idFrom('asset', $that.transaction.entityId) CALL incrementCounter(client, \"clientCount\", 1) YIELD count AS clientCount CALL incrementCounter(client, toLower($that.outcome.result), 1) YIELD count AS clientOutcomeCount CALL incrementCounter(user, \"userCount\", 1) YIELD count AS userCount CALL incrementCounter(user, toLower($that.outcome.result), 1) YIELD count AS userOutcomeCount CALL incrementCounter(asset, \"assetCount\", 1) YIELD count AS assetCount CALL incrementCounter(asset, toLower($that.outcome.result), 1) YIELD count AS assetOutcomeCount SET client.device = $that.client.device, client.ipAddress = $that.client.ipAddress, client.userAgent = $that.client.userAgent, client: client SET client.lastseen = coll.max([$that.timestamp, coalesce(client.lastseen, $that.timestamp)]) SET client.successPercent = ceil(coalesce((client.success*1.0)/(client.count*1.0)*100.0, 0.0)) SET client.failurePercent = floor(coalesce((client.failure*1.0)/(client.count*1.0)*100.0, 0.0)) SET client.state = CASE WHEN client.successPercent >= 90 THEN 'good' WHEN client.successPercent >= 75 AND client.successPercent < 90 THEN 'warn' WHEN client.successPercent < 75 THEN 'alarm' ELSE 'alarm' END SET user.id = $that.user.id, user.alternateId = $that.user.alternateId, user.displayName = $that.user.displayName, user.type = $that.user.type, user: user SET user.lastseen = coll.max([$that.timestamp, coalesce(user.lastseen, $that.timestamp)]) SET user.successPercent = ceil(coalesce((user.success*1.0)/(user.count*1.0)*100.0, 0.0)) SET user.failurePercent = floor(coalesce((user.failure*1.0)/(user.count*1.0)*100.0, 0.0)) SET user.state = CASE WHEN user.successPercent >= 90 THEN 'good' WHEN user.successPercent >= 75 AND user.successPercent < 90 THEN 'warn' WHEN user.successPercent < 75 THEN 'alarm' ELSE 'alarm' END SET attempt.schemaVersion = $that.schemaVersion, attempt.eventId = $that.eventId, attempt.transactionId = $that.transaction.id, attempt.timestamp = $that.timestamp, attempt.entityId = $that.transaction.entityId, attempt.eventType = $that.eventType, attempt.transactionType = $that.transaction.type, attempt.eventCode = $that.eventCode, attempt.displayMessage = $that.displayMessage, attempt.outcomeResult = $that.outcome.result, attempt.logLevel = $that.level, attempt.zone = $that.client.zone, attempt.client = $that.client.ipAddress, attempt.userSequence = coalesce(userCount,0), attempt.clientSequence = coalesce(clientCount,0), attempt: attempt SET asn.id = $that.client.asn, asn: asn SET asset.id = $that.transaction.entityId, asset.detail = $that.client.requestUri, asset: asset SET asset.successPercent = ceil(coalesce((asset.success*1.0)/(asset.count*1.0)*100.0, 0.0)) SET asset.failurePercent = floor(coalesce((asset.failure*1.0)/(asset.count*1.0)*100.0, 0.0)) SET asset.state = CASE WHEN asset.successPercent >= 90 THEN 'good' WHEN asset.successPercent >= 75 AND asset.successPercent < 90 THEN 'warn' WHEN asset.successPercent < 75 THEN 'alarm' ELSE 'alarm' END CREATE (user)-[:ORIGINATED]->(attempt)-[:USING]->(client), (client)<-[:USING]-(attempt)-[:TARGETED]->(asset), (user)-[:ORIGINATED]->(attempt)-[:TARGETED]->(asset), (attempt)-[:OVER]->(asn)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) detects when new nodes enter the graph and creates `NEXT` relationships making it easier to follow event sequences during analysis.

=== "YAML"

    ```yaml
      - pattern:
        type: Cypher
        parallelism: 32
        query: |-
          MATCH (client2)<-[:USING]-(attempt1)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt2)-[:USING]->(client1)
          RETURN DISTINCT id(attempt2) AS attempt2
        mode: DistinctId
        outputs:
          sequence:
            type: CypherQuery
            query: |-
              MATCH (client2)<-[:USING]-(attempt2)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt1 {clientSequence: (attempt2.clientSequence-1)})-[:USING]->(client1)
              WHERE id(attempt2) = $that.data.attempt2
                AND id(client1) = id(client2)
              CREATE (attempt2)<-[:NEXT]-(attempt1)
            shouldRetry: false
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "parallelism": 32,
        "query": "MATCH (client2)<-[:USING]-(attempt1)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt2)-[:USING]->(client1)\nRETURN DISTINCT id(attempt2) AS attempt2",
        "mode": "DistinctId"
      },
      "outputs": {
        "sequence": {
          "type": "CypherQuery",
          "query": "MATCH (client2)<-[:USING]-(attempt2)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt1 {clientSequence: (attempt2.clientSequence-1)})-[:USING]->(client1)\nWHERE id(attempt2) = $that.data.attempt2\n  AND id(client1) = id(client2)\nCREATE (attempt2)<-[:NEXT]-(attempt1)",
          "shouldRetry": false
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: sequence-attempts
        pattern:
          type: Cypher
          query: |-
            MATCH (client2)<-[:USING]-(attempt1)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt2)-[:USING]->(client1)
            RETURN DISTINCT id(attempt2) AS attempt2
          mode: DISTINCT_ID
        outputs:
          - name: sequence
            preEnrichmentTransformation:
              type: InlineData
            destinations:
              - type: CypherQuery
                query: |-
                  MATCH (client2)<-[:USING]-(attempt2)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt1 {clientSequence: (attempt2.clientSequence-1)})-[:USING]->(client1)
                  WHERE id(attempt2) = $that.attempt2
                    AND id(client1) = id(client2)
                  CREATE (attempt2)<-[:NEXT]-(attempt1)
                parameter: that
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "sequence-attempts",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (client2)<-[:USING]-(attempt1)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt2)-[:USING]->(client1)\nRETURN DISTINCT id(attempt2) AS attempt2",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "sequence",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "destinations": [
            {
              "type": "CypherQuery",
              "query": "MATCH (client2)<-[:USING]-(attempt2)<-[:ORIGINATED]-(user)-[:ORIGINATED]->(attempt1 {clientSequence: (attempt2.clientSequence-1)})-[:USING]->(client1)\nWHERE id(attempt2) = $that.attempt2\n  AND id(client1) = id(client2)\nCREATE (attempt2)<-[:NEXT]-(attempt1)",
              "parameter": "that"
            }
          ]
        }
      ]
    }
    ```

A second [standing query](../learn/standing-queries/standing-queries.md) matches 4 consecutive failed attempts followed by a successful attempt and outputs a URL that can be copied and pasted into a browser to open the subgraph in Quine for exploration.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: |-
          MATCH (attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})-[:USING]->(client4)
          RETURN DISTINCT id(attempt1) AS attempt1
        mode: DistinctId
      outputs:
        alert:
          type: CypherQuery
          query: |-
            MATCH (attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})
            WHERE id(attempt1)=$that.data.attempt1
            RETURN 'Password Spraying Attack: ' + 'http://localhost:8080/#' + text.urlencode('MATCH (user)-[:ORIGINATED]->(attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})-[:USING]->(client) WHERE id(attempt1)="' + toString(strId(attempt1)) + '" RETURN DISTINCT user,attempt1,attempt2,attempt3,attempt4,attempt5,client') AS QuineUILink
          andThen:
            type: PrintToStandardOut
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-2"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})-[:USING]->(client4)\nRETURN DISTINCT id(attempt1) AS attempt1",
        "mode": "DistinctId"
      },
      "outputs": {
        "alert": {
          "type": "CypherQuery",
          "query": "MATCH (attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})\nWHERE id(attempt1)=$that.data.attempt1\nRETURN 'Password Spraying Attack: ' + 'http://localhost:8080/#' + text.urlencode('MATCH (user)-[:ORIGINATED]->(attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})-[:USING]->(client) WHERE id(attempt1)=\"' + toString(strId(attempt1)) + '\" RETURN DISTINCT user,attempt1,attempt2,attempt3,attempt4,attempt5,client') AS QuineUILink",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: detect-password-spraying
        pattern:
          type: Cypher
          query: |-
            MATCH (attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})-[:USING]->(client4)
            RETURN DISTINCT id(attempt1) AS attempt1
          mode: DISTINCT_ID
        outputs:
          - name: alert
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})
                WHERE id(attempt1)=$that.attempt1
                RETURN 'Password Spraying Attack: ' + 'http://localhost:8080/#' + text.urlencode('MATCH (user)-[:ORIGINATED]->(attempt1 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt2 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt3 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt4 {outcomeResult:"FAILURE"})-[:NEXT]->(attempt5 {outcomeResult:"SUCCESS"})-[:USING]->(client) WHERE id(attempt1)="' + toString(strId(attempt1)) + '" RETURN DISTINCT user,attempt1,attempt2,attempt3,attempt4,attempt5,client') AS QuineUILink
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "detect-password-spraying",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})-[:USING]->(client4)\nRETURN DISTINCT id(attempt1) AS attempt1",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "alert",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})\nWHERE id(attempt1)=$that.attempt1\nRETURN 'Password Spraying Attack: ' + 'http://localhost:8080/#' + text.urlencode('MATCH (user)-[:ORIGINATED]->(attempt1 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt2 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt3 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt4 {outcomeResult:\"FAILURE\"})-[:NEXT]->(attempt5 {outcomeResult:\"SUCCESS\"})-[:USING]->(client) WHERE id(attempt1)=\"' + toString(strId(attempt1)) + '\" RETURN DISTINCT user,attempt1,attempt2,attempt3,attempt4,attempt5,client') AS QuineUILink",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r password_spraying.yml
Graph is ready
Running Recipe: Password Spraying Detection
Using 36 node appearances
Using 18 quick queries
Using 12 sample queries
Running Standing Query STANDING-1
Running Standing Query STANDING-2
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
```

Quine will process the events looking for the subgraph pattern that we defined in STANDING-2. When it encounters the pattern, it will emit a link to the event in the Exploration UI.

``` shell
2023-02-10 15:34:54,460 Standing query `alert` match: {"meta":{"isPositiveMatch":true,"resultId":"7242b979-03c2-2bc3-9879-13661e8359b5"},"data":{"QuineUILink":"Password Spraying Attack: http://localhost:8080/#MATCH%20%28user%29-%5B%3AORIGINATED%5D-%3E%28attempt1%20%7BoutcomeResult%3A%22FAILURE%22%7D%29-%5B%3ANEXT%5D-%3E%28attempt2%20%7BoutcomeResult%3A%22FAILURE%22%7D%29-%5B%3ANEXT%5D-%3E%28attempt3%20%7BoutcomeResult%3A%22FAILURE%22%7D%29-%5B%3ANEXT%5D-%3E%28attempt4%20%7BoutcomeResult%3A%22FAILURE%22%7D%29-%5B%3ANEXT%5D-%3E%28attempt5%20%7BoutcomeResult%3A%22SUCCESS%22%7D%29-%5B%3AUSING%5D-%3E%28client%29%20WHERE%20id%28attempt1%29%3D%22cb73fb14-4686-3913-8cd8-7d4d608b53d5%22%20RETURN%20DISTINCT%20user%2Cattempt1%2Cattempt2%2Cattempt3%2Cattempt4%2Cattempt5%2Cclient"}}
```

![Event Pattern](images/pw-spraying-pattern.png)

## Summary

Take time to explore the graph in the Quine Exploration UI. Start by right clicking on the contractor node and selecting the `Attempts Timeline` quick query to generate the attack timeline.

![Event Timeline](images/pw-spraying-timeline.png)

The recipe contains a number of additional quick queries to view events.

| Quick Query                                      | Returns |
| :----------------------------------------------- | :------ |
| Adjacent Nodes                                   | Nodes   |
| Refresh                                          | Node    |
| Local Properties                                 | Text    |
| Admins that Targeted Asset                       | Nodes   |
| All User Types that Targeted Asset               | Nodes   |
| Attempts Timeline                                | Nodes   |
| Contractors that Failed Authentication for Asset | Nodes   |
| Contractors that Targeted Asset                  | Nodes   |
| Failed Password Authentication Attempts          | Nodes   |
| Guests that Targeted Asset                       | Nodes   |
| Next Attempt                                     | Nodes   |
| Previous Attempt                                 | Nodes   |
| Show Client and ASN                              | Nodes   |
| Targeted Assets                                  | Nodes   |
| Targeted Assets                                  | Nodes   |
| Users that Targeted Asset                        | Nodes   |
| Authentication attempts in chronological order   | Text    |
| Authentication attempts in chronological order   | Text    |


---
# Approximate Pi
URL: https://quine.io/recipes/pi/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Incrementally approximate pi using [Leibniz' formula for π](https://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80)

    ??? example "Pi Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/pi.yaml"
        ```

    [Download Recipe](assets/pi.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Incrementally approximate pi using [Leibniz' formula for π](https://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80)

    ??? example "Pi Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/pi.yaml"
        ```

    [Download Recipe](assets/v2/pi.yaml){ .md-button download="" .md-button--primary data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

Incrementally approximates pi using Leibniz' formula for π -- the arctangent function is incrementally (corecursively) computed along `:improved_by` edges, and each arctangent approximation is quadrupled to yield an approximation of pi.

## How it Works

The recipe is completely self contained. We take advantage of the unique [`match`](../learn/standing-queries/standing-queries.md#pattern-match-query), [`output action`](../learn/standing-queries/standing-queries.md#result-outputs) structure of a [`standing query`](../learn/standing-queries/standing-queries.md) to improve the approximation of pi by continuously streaming nodes back into the graph.

=== "YAML"

    ```yaml
      - pattern:
          type: Cypher
          query: MATCH (n:arctan) WHERE n.approximation IS NOT NULL AND n.denominator IS NOT NULL RETURN DISTINCT id(n) AS id
        outputs:
          # iterate over arctan
          iterate:
            type: CypherQuery
            query: |-
              MATCH (n)
              WHERE id(n) = $that.data.id
              WITH n, -sign(n.denominator)*(abs(n.denominator)+2) as nextDenom
              WITH n, nextDenom, n.approximation+(1/nextDenom) as nextApprox
              MATCH (next) WHERE id(next) = idFrom(nextDenom)
              SET next:arctan, next.denominator = nextDenom, next.approximation=nextApprox
              CREATE (n)-[:improved_by]->(next)
          # map arctan to piApprox
          piApprox:
            type: CypherQuery
            query: |-
              MATCH (arctan)
              WHERE id(arctan) = $that.data.id
              WITH arctan, arctan.denominator AS denominator, arctan.approximation*4 AS approximatedPi
              MATCH (approximation) WHERE id(approximation) = idFrom('approximation', denominator)
              SET approximation:piApproximation, approximation.approximatedPi = approximatedPi
              CREATE (arctan)-[:approximates]->(approximation)
              RETURN approximatedPi
            andThen:
              type: WriteToFile
              path: $out_file
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n:arctan) WHERE n.approximation IS NOT NULL AND n.denominator IS NOT NULL RETURN DISTINCT id(n) AS id"
      },
      "outputs": {
        "iterate": {
          "type": "CypherQuery",
          "query": "MATCH (n)\nWHERE id(n) = $that.data.id\nWITH n, -sign(n.denominator)*(abs(n.denominator)+2) as nextDenom\nWITH n, nextDenom, n.approximation+(1/nextDenom) as nextApprox\nMATCH (next) WHERE id(next) = idFrom(nextDenom)\nSET next:arctan, next.denominator = nextDenom, next.approximation=nextApprox\nCREATE (n)-[:improved_by]->(next)"
        },
        "piApprox": {
          "type": "CypherQuery",
          "query": "MATCH (arctan)\nWHERE id(arctan) = $that.data.id\nWITH arctan, arctan.denominator AS denominator, arctan.approximation*4 AS approximatedPi\nMATCH (approximation) WHERE id(approximation) = idFrom('approximation', denominator)\nSET approximation:piApproximation, approximation.approximatedPi = approximatedPi\nCREATE (arctan)-[:approximates]->(approximation)\nRETURN approximatedPi",
          "andThen": {
            "type": "WriteToFile",
            "path": "$out_file"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: arctan-processor
        pattern:
          type: Cypher
          query: MATCH (n:arctan) WHERE n.approximation IS NOT NULL AND n.denominator IS NOT NULL RETURN DISTINCT id(n) AS id
          mode: DISTINCT_ID
        outputs:
          # iterate over arctan
          - name: iterate
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (n)
                WHERE id(n) = $that.id
                WITH n, -sign(n.denominator)*(abs(n.denominator)+2) as nextDenom
                WITH n, nextDenom, n.approximation+(1/nextDenom) as nextApprox
                MATCH (next) WHERE id(next) = idFrom(nextDenom)
                SET next:arctan, next.denominator = nextDenom, next.approximation=nextApprox
                CREATE (n)-[:improved_by]->(next)
                RETURN null
              parameter: that
            destinations:
              - type: Drop
          # map arctan to piApprox
          - name: piApprox
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (arctan)
                WHERE id(arctan) = $that.id
                WITH arctan, arctan.denominator AS denominator, arctan.approximation*4 AS approximatedPi
                MATCH (approximation) WHERE id(approximation) = idFrom('approximation', denominator)
                SET approximation:piApproximation, approximation.approximatedPi = approximatedPi
                CREATE (arctan)-[:approximates]->(approximation)
                RETURN approximatedPi
              parameter: that
            destinations:
              - type: File
                path: $out_file
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "arctan-processor",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n:arctan) WHERE n.approximation IS NOT NULL AND n.denominator IS NOT NULL RETURN DISTINCT id(n) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "iterate",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (n)\nWHERE id(n) = $that.id\nWITH n, -sign(n.denominator)*(abs(n.denominator)+2) as nextDenom\nWITH n, nextDenom, n.approximation+(1/nextDenom) as nextApprox\nMATCH (next) WHERE id(next) = idFrom(nextDenom)\nSET next:arctan, next.denominator = nextDenom, next.approximation=nextApprox\nCREATE (n)-[:improved_by]->(next)\nRETURN null",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        },
        {
          "name": "piApprox",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (arctan)\nWHERE id(arctan) = $that.id\nWITH arctan, arctan.denominator AS denominator, arctan.approximation*4 AS approximatedPi\nMATCH (approximation) WHERE id(approximation) = idFrom('approximation', denominator)\nSET approximation:piApproximation, approximation.approximatedPi = approximatedPi\nCREATE (arctan)-[:approximates]->(approximation)\nRETURN approximatedPi",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "File",
              "path": "$out_file"
            }
          ]
        }
      ]
    }
    ```

We use a tag propagation technique set up in the standing query to perform the calculation.

Submitting the ++"[No Output] Run this query to begin processing."++ [sample query](/reference/rest-api/?av=v2#/operations/replace-sample-queries) creates a seed node `(n:arctan)` in the graph with an initial approximation of 1.0.

``` cypher
WITH 1 AS initialDenominator
MATCH (n)
WHERE id(n) = idFrom(1)
  SET n.denominator = toFloat(1),
      n.approximation = toFloat(1),
      n:arctan
```

Once the seed node is set in the graph, iteration over the approximation is done in several parts.

1. Detect when the seed node or its descendants enter the graph.
``` cypher
MATCH (n:arctan)
WHERE n.approximation IS NOT NULL
  AND n.denominator IS NOT NULL
RETURN DISTINCT id(n) AS id
```

1. Iterate over arctan.
``` cypher
MATCH (n)
WHERE id(n) = $that.data.id
WITH n, -sign(n.denominator)*(abs(n.denominator)+2) as nextDenom
WITH n, nextDenom, n.approximation+(1/nextDenom) as nextApprox
MATCH (next) WHERE id(next) = idFrom(nextDenom)
SET next:arctan, next.denominator = nextDenom, next.approximation=nextApprox
CREATE (n)-[:improved_by]->(next)
```

1. Map arctan to piApprox.
``` cypher
MATCH (arctan)
WHERE id(arctan) = $that.data.id
WITH arctan, arctan.denominator AS denominator, arctan.approximation*4 AS approximatedPi
MATCH (approximation) WHERE id(approximation) = idFrom('approximation', denominator)
SET approximation:piApproximation, approximation.approximatedPi = approximatedPi
CREATE (arctan)-[:approximates]->(approximation)
RETURN approximatedPi
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r pi.yaml -x out_file=approximation.log
Graph is ready
Running Recipe: Pi
Using 2 node appearances
Using 4 sample queries
Running Standing Query STANDING-1
Quine web server available at http://localhost:8080

 | => STANDING-1 count 0
```

Connect to Quine once it is started and submit the ++"[No Output] Run this query to begin processing."++ sample query.

![Run this query](images/pi-start-processing.png)

!!! Warning
    Once you submit the ++"[No Output] Run this query to begin processing."++ query, Quine will immediately begin to produce new approximations for pi. You must quit Quine (++ctrl+c++) to stop the sequence.

You will immediately see the count of `STANDING-1` matches increase and entries in the `approximation.log` fie.

``` shell title="Terminal Window"
 | => STANDING-1 count 5043
```

``` json title="approximation.log"
{"meta":{"isPositiveMatch":true,"resultId":"106f731f-be27-2650-af22-b3010744124c"},"data":{"approximatedPi":3.141791500277029}}
```

Submit the ++"[Node] Get Best Approximation (so far)"++ sample query to display the latest approximation of pi as a node.

![Get Best Approximation](images/pi-best-approximation.png)

Quine will manifest a graph similar to this.

![Graph structure](images/pi-graph-structure.png)

Submit the ++"[Text] Repeatedly Get Best Approximation (so far)"++ sample query with ++shift+enter++ to view the stream of updated approximations in the Exploration UI.

![Aproximation Stream](images/pi-approx.png)

## Build your skills

What ingest query could be added to replace the function of the ++"[No Output] Run this query to begin processing."++ sample query?

??? success "Solution"

    We solved this by modifying the ingest query to use the NumberIterator source type.

    Replace the empty ingest query with this one.

    ```yaml
      - name: seed-arctan
        source:
          type: NumberIterator
          startOffset: 1
          limit: 1
        query: |-
          WITH $that AS initialDenominator
          MATCH (n)
          WHERE id(n) = idFrom(1)

          SET n.denominator = toFloat(1),
              n.approximation = toFloat(1),
              n:arctan
    ```


---
# Monitor an MMO
URL: https://quine.io/recipes/planetside-2/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Model real-time player kill data from Planetside 2. Use API calls to supplement the killfeed graph with detailed information about the player characters and the weapons used.

    ??? example "Monitor an MMO Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/planetside-2.yaml"
        ```

    [Download Recipe](assets/planetside-2.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Ethan Bell](https://github.com/emanb29)</small>

    Model real-time player kill data from Planetside 2. Use API calls to supplement the killfeed graph with detailed information about the player characters and the weapons used.

    ??? example "Monitor an MMO Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/planetside-2.yaml"
        ```

    [Download Recipe](assets/v2/planetside-2.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

Ingest the killfeed [websocket](http://census.daybreakgames.com/#what-is-websocket) output from Daybreak Games' MMOFPS "PlanetSide 2", invoking the getJsonLines procedure to lazily fill out unknown static data. Replace all instances of `s:example` with a service-id acquired from [http://census.daybreakgames.com/#service-id](http://census.daybreakgames.com/#service-id)

## Sample Data

`Death` events will stream in from the websocket once the recipe is started. The events are JSON objects like the one below.

``` json
{
	"payload":{
		"attacker_character_id":"5428010618015189713",
		"attacker_fire_mode_id":"26103",
		"attacker_loadout_id":"15",
		"attacker_vehicle_id":"0",
		"attacker_weapon_id":"26003",
		"character_id":"5428168624838258657",
		"character_loadout_id":"6",
		"event_name":"Death",
		"is_headshot":"1",
		"timestamp":"1392056954",
		"vehicle_id":"0",
		"world_id":"1",
		"zone_id":"2"
	},
	"service":"event",
	"type":"serviceMessage"
}
```

## How it Works

The recipe connects an [ingest stream](../learn/ingest-sources/index.md) to the PS2 Event Streaming WebSocket to manifest `Death` events as `murder`, `victim`, `attacker`, `weapon`, and `character` nodes in Quine.

INGEST-1 processes events emitted form the websocket:

=== "YAML"

    ```yaml
    - type: WebsocketSimpleStartupIngest
      url: wss://push.planetside2.com/streaming?environment=ps2&service-id=s:example
      initMessages:
      - |-
        {
          "service":"event",
          "action":"subscribe",
          "worlds": ["all"],
          "characters":["all"],
          "eventNames":["Death"]
        }
      format:
        type: CypherJson
        query: |-
          WITH * WHERE $that.type = 'serviceMessage'
          CREATE (m:murder)
          SET m = COALESCE($that.payload, {})
          WITH id(m) as mId
          MATCH (murder) WHERE id(murder) = mId
          MATCH (victim) WHERE id(victim) = idFrom('character', murder.character_id)
          MATCH (attacker) WHERE id(attacker) = idFrom('character', murder.attacker_character_id)
          MATCH (weapon) WHERE id(weapon) = idFrom('weapon', murder.attacker_weapon_id)
          SET weapon.uninitialized = weapon.weapon_id IS NULL
          SET victim:character, attacker:character, weapon:weapon,
            victim.character_id = murder.character_id, attacker.character_id = murder.attacker_character_id,
            weapon.weapon_id = murder.attacker_weapon_id
          CREATE (victim)<-[:victim]-(murder)-[:attacker]->(attacker), (murder)-[:weapon]->(weapon)
          WITH murder, victim, attacker
          UNWIND [victim, attacker] AS character
          SET character.last_update = murder.timestamp
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "WebsocketSimpleStartupIngest",
      "url": "wss://push.planetside2.com/streaming?environment=ps2&service-id=s:example",
      "initMessages": [
        "{\n  \"service\":\"event\",\n  \"action\":\"subscribe\",\n  \"worlds\": [\"all\"],\n  \"characters\":[\"all\"],\n  \"eventNames\":[\"Death\"]\n}"
      ],
      "format": {
        "type": "CypherJson",
        "query": "WITH * WHERE $that.type = 'serviceMessage'\nCREATE (m:murder)\nSET m = COALESCE($that.payload, {})\nWITH id(m) as mId\nMATCH (murder) WHERE id(murder) = mId\nMATCH (victim) WHERE id(victim) = idFrom('character', murder.character_id)\nMATCH (attacker) WHERE id(attacker) = idFrom('character', murder.attacker_character_id)\nMATCH (weapon) WHERE id(weapon) = idFrom('weapon', murder.attacker_weapon_id)\nSET weapon.uninitialized = weapon.weapon_id IS NULL\nSET victim:character, attacker:character, weapon:weapon,\n  victim.character_id = murder.character_id, attacker.character_id = murder.attacker_character_id,\n  weapon.weapon_id = murder.attacker_weapon_id\nCREATE (victim)<-[:victim]-(murder)-[:attacker]->(attacker), (murder)-[:weapon]->(weapon)\nWITH murder, victim, attacker\nUNWIND [victim, attacker] AS character\nSET character.last_update = murder.timestamp"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: planetside-killfeed
        source:
          type: WebsocketClient
          url: wss://push.planetside2.com/streaming?environment=ps2&service-id=s:example
          format:
            type: Json
          initMessages:
            - |-
              {
                "service":"event",
                "action":"subscribe",
                "worlds": ["all"],
                "characters":["all"],
                "eventNames":["Death"]
              }
          characterEncoding: UTF-8
        query: |-
          WITH * WHERE $that.type = 'serviceMessage'
          CREATE (m:murder)
          SET m = COALESCE($that.payload, {})
          WITH id(m) as mId
          MATCH (murder) WHERE id(murder) = mId
          MATCH (victim) WHERE id(victim) = idFrom('character', murder.character_id)
          MATCH (attacker) WHERE id(attacker) = idFrom('character', murder.attacker_character_id)
          MATCH (weapon) WHERE id(weapon) = idFrom('weapon', murder.attacker_weapon_id)
          SET weapon.uninitialized = weapon.weapon_id IS NULL
          SET victim:character, attacker:character, weapon:weapon,
            victim.character_id = murder.character_id, attacker.character_id = murder.attacker_character_id,
            weapon.weapon_id = murder.attacker_weapon_id
          CREATE (victim)<-[:victim]-(murder)-[:attacker]->(attacker), (murder)-[:weapon]->(weapon)
          WITH murder, victim, attacker
          UNWIND [victim, attacker] AS character
          SET character.last_update = murder.timestamp
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "planetside-killfeed",
      "source": {
        "type": "WebsocketClient",
        "url": "wss://push.planetside2.com/streaming?environment=ps2&service-id=s:example",
        "format": {
          "type": "Json"
        },
        "initMessages": [
          "{\n  \"service\":\"event\",\n  \"action\":\"subscribe\",\n  \"worlds\": [\"all\"],\n  \"characters\":[\"all\"],\n  \"eventNames\":[\"Death\"]\n}"
        ],
        "characterEncoding": "UTF-8"
      },
      "query": "WITH * WHERE $that.type = 'serviceMessage'\nCREATE (m:murder)\nSET m = COALESCE($that.payload, {})\nWITH id(m) as mId\nMATCH (murder) WHERE id(murder) = mId\nMATCH (victim) WHERE id(victim) = idFrom('character', murder.character_id)\nMATCH (attacker) WHERE id(attacker) = idFrom('character', murder.attacker_character_id)\nMATCH (weapon) WHERE id(weapon) = idFrom('weapon', murder.attacker_weapon_id)\nSET weapon.uninitialized = weapon.weapon_id IS NULL\nSET victim:character, attacker:character, weapon:weapon,\n  victim.character_id = murder.character_id, attacker.character_id = murder.attacker_character_id,\n  weapon.weapon_id = murder.attacker_weapon_id\nCREATE (victim)<-[:victim]-(murder)-[:attacker]->(attacker), (murder)-[:weapon]->(weapon)\nWITH murder, victim, attacker\nUNWIND [victim, attacker] AS character\nSET character.last_update = murder.timestamp"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) matches each new character node to populate the character properties via the `census` API.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: MATCH (newCharacter:character) WHERE newCharacter.character_id IS NOT NULL RETURN DISTINCT id(newCharacter) AS id
      outputs:
        populate-fresh-character:
          type: CypherQuery
          query: |-
            MATCH (c)
            WHERE id(c) = $that.data.id
            CALL loadJsonLines("https://census.daybreakgames.com/s:example/get/ps2:v2/character/?character_id="+c.character_id) YIELD value
            SET c += COALESCE(value.character_list[0], {})
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (newCharacter:character) WHERE newCharacter.character_id IS NOT NULL RETURN DISTINCT id(newCharacter) AS id"
      },
      "outputs": {
        "populate-fresh-character": {
          "type": "CypherQuery",
          "query": "MATCH (c)\nWHERE id(c) = $that.data.id\nCALL loadJsonLines(\"https://census.daybreakgames.com/s:example/get/ps2:v2/character/?character_id=\"+c.character_id) YIELD value\nSET c += COALESCE(value.character_list[0], {})"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: populate-character-data
        pattern:
          type: Cypher
          query: MATCH (newCharacter:character) WHERE newCharacter.character_id IS NOT NULL RETURN DISTINCT id(newCharacter) AS id
          mode: DISTINCT_ID
        outputs:
          - name: populate-fresh-character
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (c)
                WHERE id(c) = $that.id
                CALL loadJsonLines("https://census.daybreakgames.com/s:example/get/ps2:v2/character/?character_id="+c.character_id) YIELD value
                SET c += COALESCE(value.character_list[0], {})
                RETURN null
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "populate-character-data",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (newCharacter:character) WHERE newCharacter.character_id IS NOT NULL RETURN DISTINCT id(newCharacter) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "populate-fresh-character",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (c)\nWHERE id(c) = $that.id\nCALL loadJsonLines(\"https://census.daybreakgames.com/s:example/get/ps2:v2/character/?character_id=\"+c.character_id) YIELD value\nSET c += COALESCE(value.character_list[0], {})\nRETURN null",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

The `character_list` object returned from the API is similar to the one below. 

``` json
{"character_list":[{"character_id":"5428010618020694593","name":{"first":"Dreadnaut","first_lower":"dreadnaut"},"faction_id":"1","head_id":"1","title_id":"97","times":{"creation":"1353434436","creation_date":"2012-11-20 18:00:36.0","last_save":"1508990907","last_save_date":"2017-10-26 04:08:27.0","last_login":"1508985950","last_login_date":"2017-10-26 02:45:50.0","login_count":"971","minutes_played":"101148"},"certs":{"earned_points":"206533","gifted_points":"9358","spent_points":"205791","available_points":"10100","percent_to_next":"0.002999999962901"},"battle_rank":{"percent_to_next":"43","value":"102"},"profile_id":"21","daily_ribbon":{"count":"5","time":"1508911200","date":"2017-10-25 06:00:00.0"},"prestige_level":"0"}],"returned":1}
```

A second [standing query](../learn/standing-queries/standing-queries.md) matches each new weapon node to populate the weapon properties via the `census` API.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: MATCH (weapon:weapon) WHERE weapon.uninitialized = true AND weapon.weapon_id IS NOT NULL RETURN DISTINCT id(weapon) AS id
      outputs:
        populate-weapon:
          type: CypherQuery
          query: |-
            MATCH (weapon) WHERE id(weapon) = $that.data.id
            CALL loadJsonLines("https://census.daybreakgames.com/s:example/get/ps2:v2/item?item_id="+weapon.weapon_id+"&c:join=weapon_datasheet") YIELD value
            SET weapon += COALESCE(value.item_list[0], {})
            REMOVE weapon.uninitialized
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-2"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (weapon:weapon) WHERE weapon.uninitialized = true AND weapon.weapon_id IS NOT NULL RETURN DISTINCT id(weapon) AS id"
      },
      "outputs": {
        "populate-weapon": {
          "type": "CypherQuery",
          "query": "MATCH (weapon) WHERE id(weapon) = $that.data.id\nCALL loadJsonLines(\"https://census.daybreakgames.com/s:example/get/ps2:v2/item?item_id=\"+weapon.weapon_id+\"&c:join=weapon_datasheet\") YIELD value\nSET weapon += COALESCE(value.item_list[0], {})\nREMOVE weapon.uninitialized"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: populate-weapon-data
        pattern:
          type: Cypher
          query: MATCH (weapon:weapon) WHERE weapon.uninitialized = true AND weapon.weapon_id IS NOT NULL RETURN DISTINCT id(weapon) AS id
          mode: DISTINCT_ID
        outputs:
          - name: populate-weapon
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (weapon) WHERE id(weapon) = $that.id
                CALL loadJsonLines("https://census.daybreakgames.com/s:example/get/ps2:v2/item?item_id="+weapon.weapon_id+"&c:join=weapon_datasheet") YIELD value
                SET weapon += COALESCE(value.item_list[0], {})
                REMOVE weapon.uninitialized
                RETURN null
              parameter: that
            destinations:
              - type: Drop
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "populate-weapon-data",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (weapon:weapon) WHERE weapon.uninitialized = true AND weapon.weapon_id IS NOT NULL RETURN DISTINCT id(weapon) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "populate-weapon",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (weapon) WHERE id(weapon) = $that.id\nCALL loadJsonLines(\"https://census.daybreakgames.com/s:example/get/ps2:v2/item?item_id=\"+weapon.weapon_id+\"&c:join=weapon_datasheet\") YIELD value\nSET weapon += COALESCE(value.item_list[0], {})\nREMOVE weapon.uninitialized\nRETURN null",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "Drop"
            }
          ]
        }
      ]
    }
    ```

The `item_list` object returned from the API is similar to the one below.

``` json
{"item_list":[{"item_id":"7","item_type_id":"26","item_category_id":"139","is_vehicle_weapon":"0","name":{"de":"Spawn -Leuchte","en":"Spawn Beacon","es":"Baliza de apariciones","fr":"Balise de réapparition","it":"Faro di rigenerazione","tr":"Animation Control Sign"},"description":{"de":"Eine Signalleuchte zum schnellen Absetzen von Truppmitgliedern in einen Bereich aus geringer Höhe.","en":"A signal-emitting beacon that allows squad members to drop pod into an area from low orbit.","es":"Una baliza que emite una señal y permite a los miembros del escuadrón aterrizar con cápsula dentro de una zona desde una órbita baja.","fr":"Une balise émettrice de signaux qui module de aux membres de l'es un déploiement à partir d'une orbite basse.","it":"Faro emettitore di segnali che permette ai membri della squadra di inserirsi a caldo nell'area.","tr":"It allows squad members to come from low orbit to instantaneous space is a scattering checkmark."},"faction_id":"0","max_stack_size":"1","image_set_id":"1568","image_id":"3056","image_path":"/files/ps2 /images/static/3056.png","is_default_attachment":"0"}],"returned":1}
```

## Running the Recipe

!!! Warning
    Before running this recipe beyond a few seconds, you'll need to apply for a [service ID](http://census.daybreakgames.com/#service-id) from Daybreak Games in order to access their API. The service ID "s:example" is available for casual use--it is throttled to 10 requests per minute per client IP address.

    Update a local copy of this recipe once you receive your personal service-id.

    Please don't share the updated recipe or your service ID with others.

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r planetside-2.yaml
Graph is ready
Running Recipe: Planetside 2
Using 4 node appearances
Running Standing Query STANDING-1
Running Standing Query STANDING-2
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
```

Once the recipe is running, you can explore the graph to find subgraphs formed around `Death` events. 

![Murder Event](images/mmo-murder.png)


---
# Quine Logs
URL: https://quine.io/recipes/quine-logs-recipe/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    This recipe processes Quine log lines using a regular expression.

    ??? example "Quine Logs Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/quine-logs-recipe.yaml"
        ```

    [Download Recipe](assets/quine-logs-recipe.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Michael Aglietti](https://github.com/maglietti)</small>

    This recipe processes Quine log lines using a regular expression.

    ??? example "Quine Logs Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/quine-logs-recipe.yaml"
        ```

    [Download Recipe](assets/v2/quine-logs-recipe.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

In this scenario, we process the Quine log output to manifest a graph to aide in troubleshooting.

## Sample Data

Sample data is created using Quine itself by launching Quine with the java property: `thatdot.loglevel=INFO`.

``` shell title="Launch Quine and redirect the console output to a file"
❯ java -Dthatdot.loglevel=DEBUG -jar quine-2.1.1.jar > quine.log
Graph is ready
Quine web server available at http://localhost:8080
```

Verify that Quine is running with the [Process Readiness: `GET /api/v2/system/readiness`](/reference/rest-api/?av=v2#/operations/get-readiness) API endpoint.

``` shell title="Query the endpoint using HTTPie or curl"
❯ http GET http://localhost:8080/api/v2/system/readiness
HTTP/1.1 204 No Content
```

Shutdown Quine using the [Graceful Shutdown: `POST /api/v2/system:shutdown`](/reference/rest-api/?av=v2#/operations/initiate-shutdown) API endpoint.

``` shell title="POST to the endpoint using HTTPie or curl"
❯ http POST http://localhost:8080/api/v2/system:shutdown
HTTP/1.1 202 Accepted
```

You now have a file containing a series of INFO events produced during Quine startup.

``` shell
2023-02-10 10:16:02,410 INFO [NotFromActor] [main] com.thatdot.quine.app.Main$ - Running 1.5.1 with 10 available cores and 12GiB max heap size.
2023-02-10 10:16:02,890 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-5] com.thatdot.quine.persistor.ExceptionWrappingPersistenceAgent - Persistence backend for: core quine data is at: Version(13.0.0), this is usable as-is by: Version(13.0.0)
2023-02-10 10:16:02,970 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-5] com.thatdot.quine.graph.GraphService - Adding a new local shard at idx: 0
2023-02-10 10:16:02,972 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-5] com.thatdot.quine.graph.GraphService - Adding a new local shard at idx: 1
2023-02-10 10:16:02,972 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-5] com.thatdot.quine.graph.GraphService - Adding a new local shard at idx: 2
2023-02-10 10:16:02,972 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-5] com.thatdot.quine.graph.GraphService - Adding a new local shard at idx: 3
2023-02-10 10:16:02,980 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-14] com.thatdot.quine.persistor.ExceptionWrappingPersistenceAgent - Persistence backend for: Quine app state is at: Version(1.1.0), this is usable as-is by: Version(1.1.0)
2023-02-10 10:16:19,364 INFO [NotFromActor] [graph-service-akka.actor.default-dispatcher-7] com.thatdot.quine.persistor.ExceptionWrappingPersistenceAgent - Persistence backend for: core quine data is at: Version(13.0.0), this is usable as-is by: Version(13.0.0)
```
## How it Works

The recipe reads Quine log events from a file using [ingest streams](../learn/ingest-sources/index.md) to manifest a graph in Quine. The filename is passed into Quine at runtime using `--recipe-value in_file={quine.log}`

INGEST-1 processes the `quine.log` file:

=== "YAML"

    ```yaml
    - type: FileIngest
      path: $in_file
      format:
        type: CypherLine
        query: |-
          WITH text.regexFirstMatch($that, "(^\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2},\\d{3}) (FATAL|ERROR|WARN|INFO|DEBUG) \\[(\\S*)\\] \\[(\\S*)\\] (\\S*) - (.*)") AS r WHERE r IS NOT NULL
          WITH r, split(r[3], "/") as path,
                  split(r[6], "(") as msgPts
          WITH r, path, msgPts, replace(COALESCE(split(path[2], "@")[-1], 'No host'),")","") as qh

          MATCH (actor), (msg), (class), (host)
          WHERE id(host)  = idFrom("host", qh)
            AND id(actor) = idFrom("actor", r[3])
            AND id(msg)   = idFrom("msg", r[0])
            AND id(class) = idFrom("class", r[5])

          SET host.address = split(qh, ":")[0],
              host.port = split(qh, ":")[-1],
              host.host = qh,
              host: Host

          SET actor.address = r[3],
              actor.id = replace(path[-1],")",""),
              actor.shard = path[-2],
              actor.type = path[-3],
              actor: Actor

          SET msg.msg = r[6],
              msg.path = path[0],
              msg.type = split(msgPts[0], " ")[0],
              msg.level = r[2],
              msg: Message

          SET class.class = r[5],
          class: Class

          WITH * CALL reify.time(datetime({date: localdatetime(r[1], "yyyy-MM-dd HH:mm:ss,SSS")})) YIELD node AS time

          CREATE (host)<-[:ON_HOST]-(actor)-[:SENT]->(msg),
                (actor)-[:OF_CLASS]->(class),
                (msg)-[:AT_TIME]->(time)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "FileIngest",
      "path": "$in_file",
      "format": {
        "type": "CypherLine",
        "query": "WITH text.regexFirstMatch($that, \"(^\\\\d{4}-\\\\d{2}-\\\\d{2} \\\\d{1,2}:\\\\d{2}:\\\\d{2},\\\\d{3}) (FATAL|ERROR|WARN|INFO|DEBUG) \\\\[(\\\\S*)\\\\] \\\\[(\\\\S*)\\\\] (\\\\S*) - (.*)\") AS r WHERE r IS NOT NULL \nWITH r, split(r[3], \"/\") as path,\n        split(r[6], \"(\") as msgPts\nWITH r, path, msgPts, replace(COALESCE(split(path[2], \"@\")[-1], 'No host'),\")\",\"\") as qh\n\nMATCH (actor), (msg), (class), (host)\nWHERE id(host)  = idFrom(\"host\", qh)\n  AND id(actor) = idFrom(\"actor\", r[3])\n  AND id(msg)   = idFrom(\"msg\", r[0])\n  AND id(class) = idFrom(\"class\", r[5])\n\nSET host.address = split(qh, \":\")[0],\n    host.port = split(qh, \":\")[-1],\n    host.host = qh,\n    host: Host\n\nSET actor.address = r[3],\n    actor.id = replace(path[-1],\")\",\"\"),\n    actor.shard = path[-2],\n    actor.type = path[-3],\n    actor: Actor\n\nSET msg.msg = r[6],\n    msg.path = path[0],\n    msg.type = split(msgPts[0], \" \")[0],\n    msg.level = r[2],\n    msg: Message\n\nSET class.class = r[5],\nclass: Class\n\nWITH * CALL reify.time(datetime({date: localdatetime(r[1], \"yyyy-MM-dd HH:mm:ss,SSS\")})) YIELD node AS time\n\nCREATE (host)<-[:ON_HOST]-(actor)-[:SENT]->(msg),\n      (actor)-[:OF_CLASS]->(class),\n      (msg)-[:AT_TIME]->(time)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: log-file-ingest
        source:
          type: File
          path: $in_file
          format:
            type: Line
        query: |-
          WITH text.regexFirstMatch($that, "(^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}) (FATAL|ERROR|WARN|INFO|DEBUG) \[(\S*)\] \[(\S*)\] (\S*) - (.*)") AS r WHERE r IS NOT NULL
          WITH r, split(r[3], "/") as path,
                  split(r[6], "(") as msgPts
          WITH r, path, msgPts, replace(COALESCE(split(path[2], "@")[-1], 'No host'),")","") as qh

          MATCH (actor), (msg), (class), (host)
          WHERE id(host)  = idFrom("host", qh)
            AND id(actor) = idFrom("actor", r[3])
            AND id(msg)   = idFrom("msg", r[0])
            AND id(class) = idFrom("class", r[5])

          SET host.address = split(qh, ":")[0],
              host.port = split(qh, ":")[-1],
              host.host = qh,
              host: Host

          SET actor.address = r[3],
              actor.id = replace(path[-1],")",""),
              actor.shard = path[-2],
              actor.type = path[-3],
              actor: Actor

          SET msg.msg = r[6],
              msg.path = path[0],
              msg.type = split(msgPts[0], " ")[0],
              msg.level = r[2],
              msg: Message

          SET class.class = r[5],
          class: Class

          WITH * CALL reify.time(datetime({date: localdatetime(r[1], "yyyy-MM-dd HH:mm:ss,SSS")})) YIELD node AS time

          CREATE (host)<-[:ON_HOST]-(actor)-[:SENT]->(msg),
                 (actor)-[:OF_CLASS]->(class),
                 (msg)-[:AT_TIME]->(time)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "log-file-ingest",
      "source": {
        "type": "File",
        "path": "$in_file",
        "format": {
          "type": "Line"
        }
      },
      "query": "WITH text.regexFirstMatch($that, \"(^\\d{4}-\\d{2}-\\d{2} \\d{1,2}:\\d{2}:\\d{2},\\d{3}) (FATAL|ERROR|WARN|INFO|DEBUG) \\[(\\S*)\\] \\[(\\S*)\\] (\\S*) - (.*)\") AS r WHERE r IS NOT NULL\nWITH r, split(r[3], \"/\") as path,\n        split(r[6], \"(\") as msgPts\nWITH r, path, msgPts, replace(COALESCE(split(path[2], \"@\")[-1], 'No host'),\")\",\"\") as qh\n\nMATCH (actor), (msg), (class), (host)\nWHERE id(host)  = idFrom(\"host\", qh)\n  AND id(actor) = idFrom(\"actor\", r[3])\n  AND id(msg)   = idFrom(\"msg\", r[0])\n  AND id(class) = idFrom(\"class\", r[5])\n\nSET host.address = split(qh, \":\")[0],\n    host.port = split(qh, \":\")[-1],\n    host.host = qh,\n    host: Host\n\nSET actor.address = r[3],\n    actor.id = replace(path[-1],\")\",\"\"),\n    actor.shard = path[-2],\n    actor.type = path[-3],\n    actor: Actor\n\nSET msg.msg = r[6],\n    msg.path = path[0],\n    msg.type = split(msgPts[0], \" \")[0],\n    msg.level = r[2],\n    msg: Message\n\nSET class.class = r[5],\nclass: Class\n\nWITH * CALL reify.time(datetime({date: localdatetime(r[1], \"yyyy-MM-dd HH:mm:ss,SSS\")})) YIELD node AS time\n\nCREATE (host)<-[:ON_HOST]-(actor)-[:SENT]->(msg),\n       (actor)-[:OF_CLASS]->(class),\n       (msg)-[:AT_TIME]->(time)"
    }
    ```

The regular expression parses each log file into parts.

``` python title="Regular Expression"
(^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}) (FATAL|ERROR|WARN|INFO|DEBUG) \[(\S*)\] \[(\S*)\] (\S*) - (.*)
```

``` text title="Matched Parts"
0: whole matched line
1: date time string
2: log level
3: actor address. Might be inside of `akka.stream.Log(…)`
4: thread name
5: logging class
6: Message
```

You can explore the regular expression in detail saved on [regex101](https://regex101.com/r/MAbux7/1).

![regex101](images/quine-log-regex.png)

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r quine-logs-recipe.yaml --recipe-value in_file=quine.log
Graph is ready
Running Recipe: Quine Log Reader
Using 4 node appearances
Using 8 quick queries
Using 2 sample queries
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080
INGEST-1 status is completed and ingested 8
```

![Quine Running](images/quine-log-running.png)

!!! Tip
    We've included a series of Quick Queries to help explore the graph. The are available by right clicking on a node displayed in the Exploration UI.

![Associated Host](images/quine-log-host.png)


---
# Recipe
URL: https://quine.io/recipes/recipePageTemplate/

## Full Recipe

<small>Shared by: [thatDot](https://github.com/thatdot)</small>

Abstract

??? example "Recipe"

    ```{ .yaml linenums="1" }
    --8<-- "recipes/assets/template-recipe.yaml"
    ```

[Download Recipe](assets/template-recipe.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

In this scenario, ...

## Sample Data

Download the sample data to the same directory where Quine will be run.

## How it Works

The recipe reads observations from the two sample data files using [ingest streams](../learn/ingest-sources/index.md) to manifest a graph in Quine. A separate ingest stream is configured to process each file, each containing Cypher that parses the observations, manifests nodes, and relates them to each other in the graph.

INGEST-1 processes the `endpoints.json` file:

=== "YAML"

    ```yaml
      - type: FileIngest
        path: endpoint.json
        format:
        type: CypherJson
        query: >-
            MATCH (proc), (event), (object)
            WHERE id(proc) = idFrom($that.pid)
            AND id(event) = idFrom($that)
            AND id(object) = idFrom($that.object)

            SET proc.id = $that.pid,
                proc: Process,
                event.type = $that.event_type,
                event: EndpointEvent,
                event.time = $that.time,
                object.data = $that.object

            CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "type": "FileIngest",
      "path": "endpoint.json",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (proc), (event), (object) WHERE id(proc) = idFrom($that.pid) AND id(event) = idFrom($that) AND id(object) = idFrom($that.object) SET proc.id = $that.pid, proc: Process, event.type = $that.event_type, event: EndpointEvent, event.time = $that.time, object.data = $that.object CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)"
      }
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect a WRITE->READ->SEND->DELETE pattern that is typical for this type of exflitration event.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: >-
            MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), 
                (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)
            WHERE e1.type = "WRITE"
            AND e2.type = "READ"
            AND e3.type = "DELETE"
            AND e4.type = "SEND"
            RETURN DISTINCT id(f) as fileId

    ```

=== "JSON"

    ```yaml title="POST /api/v2/graph/quine/standingQueries"
    [
        {
            "pattern": {
            "type": "Cypher",
            "query": "MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)\nWHERE e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\nRETURN DISTINCT id(f) as fileId"
            },
            "outputs": [
            {
                "name": "stolen-data",
                "resultEnrichment": {
                    "query": "MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)\nWHERE id(f) = $that.data.fileId\n  AND e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\n  AND e1.time < e2.time\n  AND e2.time < e3.time\n  AND e2.time < e4.time\n\nCREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)\nWITH e1, e2, e3, e4, p1, p2, f, ip, \"http://localhost:8080/#MATCH\" + text.urlencode(\" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='\"+strId(p1)+\"' AND id(e1)='\"+strId(e1)+\"' AND id(f)='\"+strId(f)+\"' AND id(e2)='\"+strId(e2)+\"' AND id(p2)='\"+strId(p2)+\"' AND id(e3)='\"+strId(e3)+\"' AND id(e4)='\"+strId(e4)+\"' AND id(ip)='\"+strId(ip)+\"' RETURN e1, e2, e3, e4, p1, p2, f, ip\") as URL RETURN URL",
                    "parameter": "that"
                },
                "destinations": [
                    { "type": "StandardOut" }
                ]
            }
            ]
        }
    ]
    ```

Once Quine detects the pattern, the event is sent to a standing query output for additional processing and action.

```yaml
    outputs:
      - name: stolen-data
        resultEnrichment:
          query: >-
            MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), 
                  (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
            WHERE id(f) = $that.data.fileId
              AND e1.type = "WRITE"
              AND e2.type = "READ"
              AND e3.type = "DELETE"
              AND e4.type = "SEND"
              AND e1.time < e2.time
              AND e2.time < e3.time
              AND e2.time < e4.time

            CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

            WITH e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
            RETURN URL
          parameter: that
        destinations:
          - type: StandardOut

```

The result once the pattern is detected is to output a link to the console that an analyst can use to review the event further within Quine's Exploration UI.

```json
result
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r .yaml
```

## Summary

Summary.

!!! Tip
    Quick Queries are available by right clicking on a node.

| Quick Query      | Node Type | Description                                       |
| :--------------- | :-------- | :------------------------------------------------ |
| Adjacent Nodes   | All       | Display the nodes that are adjacent to this node. |
| Refresh          | All       | Refresh the content stored in a node              |
| Local Properties | All       | Display the properties stored by the node         |

What is your call to action?


---
# Webhook Data Enrichment
URL: https://quine.io/recipes/webhook/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Matthew Pagan](https://github.com/mastapegs)</small>

    This recipe uses the `NumberIteratorIngest` to stream numbers into the graph. A Standing Query observes when numbers are manifested into the graph, logs them to the console, and then sends those numbers to an HTTP Endpoint. The service powering the HTTP endpoint will then enrich the graph by calculating factors of those numbers, and then creating edges between the number nodes and their factors in the graph.

    ??? example "Standing Query Output to HTTP Endpoint"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/webhook.yaml"
        ```

    [Download Recipe](assets/webhook.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Matthew Pagan](https://github.com/mastapegs)</small>

    This recipe uses the `NumberIteratorIngest` to stream numbers into the graph. A Standing Query observes when numbers are manifested into the graph, logs them to the console, and then sends those numbers to an HTTP Endpoint. The service powering the HTTP endpoint will then enrich the graph by calculating factors of those numbers, and then creating edges between the number nodes and their factors in the graph.

    ??? example "Standing Query Output to HTTP Endpoint"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/webhook.yaml"
        ```

    [Download Recipe](assets/v2/webhook.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

There may come a time when you want to enrich the graph's data from an external service. This is a great use-case for Quine's Standing Queries to output data to an HTTP endpoint. By sending the external service data needed to identify a node, we can perform Cypher Queries to enrich that node.

In this example, we'll simplify everything to demonstrate back-and-forth communication between Quine and an external service:

- A `NumberIteratorIngest` will be used to stream in 13 numbers, 1-13.
- A Standing Query monitoring for the creation of these numbers will then:
    - log them to the console
    - `POST` them to a python Flask service, where we can observe data coming in from Quine.
- The Flask service will then calculate the factors of the numbers, and create edges between numbers and their factors.
- The end result will be 13 `Number` nodes with edges between numbers and their factors.

## How it Works

The recipe uses the `NumberIteratorIngest` [ingest stream](../learn/ingest-sources/index.md) to stream 13 numbers, 1-13, into the graph.

=== "YAML"

    ```yaml
    - type: NumberIteratorIngest
      startAtOffset: 1
      ingestLimit: 13
      format:
        type: CypherLine
        query: |-
          WITH toInteger($that) AS number
          MATCH (n) WHERE id(n) = idFrom("Number", number)
          SET n:Number, n.number = number
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "NumberIteratorIngest",
      "startAtOffset": 1,
      "ingestLimit": 13,
      "format": {
        "type": "CypherLine",
        "query": "WITH toInteger($that) AS number\nMATCH (n) WHERE id(n) = idFrom(\"Number\", number)\nSET n:Number, n.number = number"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: number-iterator
        source:
          type: NumberIterator
          startOffset: 1
          limit: 13
        query: |-
          WITH toInteger($that) AS number
          MATCH (n) WHERE id(n) = idFrom("Number", number)
          SET n:Number, n.number = number
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "number-iterator",
      "source": {
        "type": "NumberIterator",
        "startOffset": 1,
        "limit": 13
      },
      "query": "WITH toInteger($that) AS number\nMATCH (n) WHERE id(n) = idFrom(\"Number\", number)\nSET n:Number, n.number = number"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to observe for the pattern of numbers being manifested into the graph.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        mode: DistinctId
        query: |-
          MATCH (n:Number)
          WHERE n.number IS NOT NULL
          RETURN DISTINCT id(n) AS id
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "mode": "DistinctId",
        "query": "MATCH (n:Number)\nWHERE n.number IS NOT NULL\nRETURN DISTINCT id(n) AS id"
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: number-processor
        pattern:
          type: Cypher
          mode: DISTINCT_ID
          query: |-
            MATCH (n:Number)
            WHERE n.number IS NOT NULL
            RETURN DISTINCT id(n) AS id
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "number-processor",
      "pattern": {
        "type": "Cypher",
        "mode": "DISTINCT_ID",
        "query": "MATCH (n:Number)\nWHERE n.number IS NOT NULL\nRETURN DISTINCT id(n) AS id"
      }
    }
    ```

When the pattern is detected, the recipe then defines 2 Standing Query Outputs to send the pattern results, one to the console, and the other to an HTTP endpoint.

**Log to Console**

=== "YAML"

    ```yaml
    log-to-console:
      type: CypherQuery
      query: |-
        MATCH (n:Number)
        WHERE id(n) = $that.data.id
        RETURN n.number AS number, $that.data.id AS id
      andThen:
        type: PrintToStandardOut
    ```

=== "JSON"

    ```json
    {
      "log-to-console": {
        "type": "CypherQuery",
        "query": "MATCH (n:Number)\nWHERE id(n) = $that.data.id\nRETURN n.number AS number, $that.data.id AS id",
        "andThen": {
          "type": "PrintToStandardOut"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    outputs:
      - name: log-to-console
        preEnrichmentTransformation:
          type: InlineData
        resultEnrichment:
          query: |-
            MATCH (n:Number)
            WHERE id(n) = $that.id
            RETURN n.number AS number, $that.id AS id
          parameter: that
        destinations:
          - type: StandardOut
    ```

=== "JSON"

    ```json
    {
      "outputs": [
        {
          "name": "log-to-console",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (n:Number)\nWHERE id(n) = $that.id\nRETURN n.number AS number, $that.id AS id",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

**POST to HTTP Endpoint**

=== "YAML"

    ```yaml
    post-to-webhook:
      type: CypherQuery
      query: |-
        MATCH (n:Number)
        WHERE id(n) = $that.data.id
        RETURN n.number AS number, $that.data.id AS id
      andThen:
        type: PostToEndpoint
        url: http://127.0.0.1:3000/webhook
    ```

=== "JSON"

    ```json
    {
      "post-to-webhook": {
        "type": "CypherQuery",
        "query": "MATCH (n:Number)\nWHERE id(n) = $that.data.id\nRETURN n.number AS number, $that.data.id AS id",
        "andThen": {
          "type": "PostToEndpoint",
          "url": "http://127.0.0.1:3000/webhook"
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    outputs:
      - name: post-to-webhook
        preEnrichmentTransformation:
          type: InlineData
        resultEnrichment:
          query: |-
            MATCH (n:Number)
            WHERE id(n) = $that.id
            RETURN n.number AS number, $that.id AS id
          parameter: that
        destinations:
          - type: HttpEndpoint
            url: http://127.0.0.1:3000/webhook
    ```

=== "JSON"

    ```json
    {
      "outputs": [
        {
          "name": "post-to-webhook",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (n:Number)\nWHERE id(n) = $that.id\nRETURN n.number AS number, $that.id AS id",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "HttpEndpoint",
              "url": "http://127.0.0.1:3000/webhook"
            }
          ]
        }
      ]
    }
    ```

**Python Flask HTTP Service**

This Python service defines the endpoint that Quine will send the standing query output.

- Retrieves the number property from the node sent by the standing query
- Uses that number to generate the node's id (via `idFrom`)
- Create edges between numbers and their factors by sending a Cypher Query back to Quine via the [Cypher Query: `POST /api/v2/graph/quine/cypher:query`](/reference/rest-api/?av=v2#/operations/query-cypher) endpoint.

=== "PYTHON"

```python
from flask import Flask, request
import requests
import time
import json

app = Flask(__name__)

def calculate_factors(number):
    factors = []
    for i in range(1, number):
        if number == i:
            continue
        if number % i == 0:
            factors.append(i)
    return factors

@app.route("/webhook", methods=["POST"])
def webhook():
    data = request.json
    print("Webhook received:", data)
    # {
    #     "meta": {
    #         "isPositiveMatch": True,
    #         "resultId": "0c89ce9e-16b0-71e1-ad1c-6ead813bed1b",
    #     },
    #     "data": {"number": 9, "id": "ddf60681-6476-3322-815b-ed093f5aa937"},
    # }

    number = data["data"]["number"]
    factors = calculate_factors(number)
    factors_list = json.dumps(factors)

    # Wait for Quine to be ready
    time.sleep(2)

    query = f"""\
UNWIND {factors_list} AS factor
MATCH (n), (m)
WHERE id(n) = idFrom("Number", {number})
AND id(m) = idFrom("Number", factor)
CREATE (m)-[:FACTOR_OF]->(n)\
"""

    requests.post(
        "http://localhost:8080/api/v2/graph/quine/cypher:query",
        headers={"Content-Type": "text/plain"},
        data=query,
    )

    return "Webhook received and processed", 200

if __name__ == "__main__":
    app.run(port=3000)

```

## Running the Recipe

**Start Python HTTP Service**

1. Copy over the Python code for the HTTP service and save it (`server.py` for example)
2. Create a new environment for Python script
```shell
python -m venv .venv
```
3. Install script dependencies
```shell
pip install flask requests
```
4. Run service
```shell
python server.py
```

**Start Recipe**

```shell
java -jar quine-2.1.1.jar -r webhook.yaml
```

This command serves the application on `http://127.0.0.1:8080`

**Observe Nodes Manifested in Graph**

After starting the Python service and running Quine with the recipe, load up the Exploration UI. The recipe includes a sample query that will load up all the `Number` nodes. You will observe that there are 13 nodes. These nodes were initially streamed into the graph ranging from 1-13, but the external python flask service, upon receiving the standing query POST, created edges between these numbers, and their factors.

Here is a picture of the relationships between the number nodes.

!!! tip
    Since the number `1` is a factor of every integer, after loading up the graph, I selected the `1` node and used the `DELETE` key to remove it from the Exploration UI so that the graph was a bit clearer to observe.

Note how all prime numbers have **NO** edges pointing **TO** them. They only either have no edges, or they are factors of other numbers themselves. Prime numbers have no factors (beyond themselves and 1), so it makes sense that there are no edges pointing **TO** them.

![number nodes](images/number-nodes.png)

## Summary

This recipe showed a simple example of how to facilitate communication between Quine, and an external service.

- `Quine -> Service` via Standing Query Output to an HTTP Endpoint
- `Service -> Quine` via Cypher Queries to [Cypher Query: `POST /api/v2/graph/quine/cypher:query`](/reference/rest-api/?av=v2#/operations/query-cypher)

While this example only calculated factors of numbers, it's not too much farther of a stretch to then add/enhance a node with data from another service, or a database.


---
# Wikipedia Page Create
URL: https://quine.io/recipes/wikipedia/

## Full Recipe

=== "Recipe v1"

    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

    Wikipedia page creation events are instantiated in the graph with relationships to a reified time model.

    ??? example "Wikipedia Page Create Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/wikipedia.yaml"
        ```

    [Download Recipe](assets/wikipedia.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

=== "Recipe v2"

    <small>Shared by: [Landon Kuhn](https://github.com/landon9720)</small>

    Wikipedia page creation events are instantiated in the graph with relationships to a reified time model.

    ??? example "Wikipedia Page Create Recipe"

        ```{ .yaml linenums="1" }
        --8<-- "recipes/assets/v2/wikipedia.yaml"
        ```

    [Download Recipe](assets/v2/wikipedia.yaml){ .md-button download="" .md-button--primary  data-category="Quine Recipe Detail" data-label="Download recipe yaml" data-action="button click" }

## Scenario

In this scenario, Quine consumes Wikipedia first revision page create events from the Mediawiki [EventStreams](https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams) service.  

## Sample Data

Data source documentation: [/streams/get_v2_stream_page_create](https://stream.wikimedia.org/?doc#/streams/get_v2_stream_page_create)

## How it Works

The recipe receives Server Sent Events ([SSE](https://en.wikipedia.org/wiki/Server-sent_events)) using an [ingest stream](../learn/ingest-sources/index.md) to manifest a graph in Quine.

INGEST-1 processes the SSE stream consisting of JSON records like:

``` json
{
  "$schema": "/mediawiki/revision/create/1.1.0",
  "meta": {
    "uri": "https://commons.wikimedia.org/wiki/User_talk:Florentin_Bart",
    "request_id": "c11b80bf-26ea-4e0f-9369-5f80ccaa276d",
    "id": "c34b93bc-14a8-4642-9aba-d4d07821ff33",
    "dt": "2023-02-07T21:03:26Z",
    "domain": "commons.wikimedia.org",
    "stream": "mediawiki.page-create",
    "topic": "eqiad.mediawiki.page-create",
    "partition": 0,
    "offset": 260591941
  },
  "database": "commonswiki",
  "page_id": 128505356,
  "page_title": "User_talk:Florentin_Bart",
  "page_namespace": 3,
  "rev_id": 730781127,
  "rev_timestamp": "2023-02-07T21:03:26Z",
  "rev_sha1": "en26ue963xtjl98402aitklflzn6srp",
  "rev_minor_edit": true,
  "rev_len": 238,
  "rev_content_model": "wikitext",
  "rev_content_format": "text/x-wiki",
  "performer": {
    "user_text": "Wikimedia Commons Welcome",
    "user_groups": [
      "autopatrolled",
      "*",
      "user",
      "autoconfirmed"
    ],
    "user_is_bot": false,
    "user_id": 302461,
    "user_registration_dt": "2008-05-28T14:23:02Z",
    "user_edit_count": 11295026
  },
  "page_is_redirect": false,
  "comment": "Adding [[Template:Welcome|welcome message]] to new user's talk page",
  "parsedcomment": "Adding <a href=\"/wiki/Template:Welcome\" title=\"Template:Welcome\">welcome message</a> to new user&#039;s talk page",
  "rev_slots": {
    "main": {
      "rev_slot_content_model": "wikitext",
      "rev_slot_sha1": "en26ue963xtjl98402aitklflzn6srp",
      "rev_slot_size": 238,
      "rev_slot_origin_rev_id": 730781127
    }
  }
}
```

The ingest query identifies `revNode`, `dbNode`, `userNode` nodes, loads them into the graph, and populates them with properties. The query also converts timestamps into `timeNode` nodes using the `reify.time` procedure for event bucketing.

=== "YAML"

    ```yaml
        - type: ServerSentEventsIngest
          url: https://stream.wikimedia.org/v2/stream/page-create
          format:
            type: CypherJson
            query: |-
              MATCH (revNode), (dbNode), (userNode)
              WHERE id(revNode) = idFrom("revision", $that.rev_id)
                AND id(dbNode) = idFrom("db", $that.database)
                AND id(userNode) = idFrom("id", $that.performer.user_id)

              // Set labels for nodes //
              CALL create.setLabels(revNode, ["rev:" + $that.page_title])
              CALL create.setLabels(dbNode, ["db:" + $that.database])
              CALL create.setLabels(userNode, ["user:" + $that.performer.user_text])

              // Create timeNode node to provide day/hour/minute bucketing and counting of revNodes //
              CALL reify.time(datetime($that.rev_timestamp), ["year", "month", "day", "hour", "minute", "second"]) YIELD node AS timeNode
              CALL incrementCounter(timeNode, "count", 1) YIELD count AS timeNodeCount

              // Set properties for nodes //
              SET revNode = $that,
                  revNode.type = "rev"

              SET dbNode.database = $that.database,
                  dbNode.type = "db"

              SET userNode = $that.performer,
                  userNode.type = "user"

              // Create edges between nodes //
              CREATE (revNode)-[:DB]->(dbNode),
                    (revNode)-[:BY]->(userNode),
                    (revNode)-[:AT]->(timeNode)
    ```

=== "JSON"

    ```json title="POST /api/v1/ingest/INGEST-1"
    {
      "type": "ServerSentEventsIngest",
      "url": "https://stream.wikimedia.org/v2/stream/page-create",
      "format": {
        "type": "CypherJson",
        "query": "MATCH (revNode), (dbNode), (userNode) \nWHERE id(revNode) = idFrom(\"revision\", $that.rev_id)\n  AND id(dbNode) = idFrom(\"db\", $that.database)\n  AND id(userNode) = idFrom(\"id\", $that.performer.user_id)\n\n// Set labels for nodes //\nCALL create.setLabels(revNode, [\"rev:\" + $that.page_title])\nCALL create.setLabels(dbNode, [\"db:\" + $that.database])\nCALL create.setLabels(userNode, [\"user:\" + $that.performer.user_text])\n\n// Create timeNode node to provide day/hour/minute bucketing and counting of revNodes //\nCALL reify.time(datetime($that.rev_timestamp), [\"year\", \"month\", \"day\", \"hour\", \"minute\", \"second\"]) YIELD node AS timeNode\nCALL incrementCounter(timeNode, \"count\", 1) YIELD count AS timeNodeCount\n\n// Set properties for nodes //\nSET revNode = $that,\n    revNode.type = \"rev\"\n\nSET dbNode.database = $that.database,\n    dbNode.type = \"db\"\n\nSET userNode = $that.performer,\n    userNode.type = \"user\"\n\n// Create edges between nodes //\nCREATE (revNode)-[:DB]->(dbNode),\n      (revNode)-[:BY]->(userNode),\n      (revNode)-[:AT]->(timeNode)"
      }
    }
    ```

=== "YAML"

    ```yaml
    ingestStreams:
      - name: wikipedia-page-create
        source:
          type: ServerSentEvent
          url: https://stream.wikimedia.org/v2/stream/page-create
          format:
            type: Json
        query: |-
          MATCH (revNode), (dbNode), (userNode)
          WHERE id(revNode) = idFrom("revision", $that.rev_id)
            AND id(dbNode) = idFrom("db", $that.database)
            AND id(userNode) = idFrom("id", $that.performer.user_id)

          // Set labels for nodes //
          CALL create.setLabels(revNode, ["rev:" + $that.page_title])
          CALL create.setLabels(dbNode, ["db:" + $that.database])
          CALL create.setLabels(userNode, ["user:" + $that.performer.user_text])

          // Create timeNode node to provide day/hour/minute bucketing and counting of revNodes //
          CALL reify.time(datetime($that.rev_timestamp), ["year", "month", "day", "hour", "minute", "second"]) YIELD node AS timeNode
          CALL incrementCounter(timeNode, "count", 1) YIELD count AS timeNodeCount

          // Set properties for nodes //
          SET revNode = $that,
              revNode.type = "rev"

          SET dbNode.database = $that.database,
              dbNode.type = "db"

          SET userNode = $that.performer,
              userNode.type = "user"

          // Create edges between nodes //
          CREATE (revNode)-[:DB]->(dbNode),
                 (revNode)-[:BY]->(userNode),
                 (revNode)-[:AT]->(timeNode)
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/ingests"
    {
      "name": "wikipedia-page-create",
      "source": {
        "type": "ServerSentEvent",
        "url": "https://stream.wikimedia.org/v2/stream/page-create",
        "format": {
          "type": "Json"
        }
      },
      "query": "MATCH (revNode), (dbNode), (userNode)\nWHERE id(revNode) = idFrom(\"revision\", $that.rev_id)\n  AND id(dbNode) = idFrom(\"db\", $that.database)\n  AND id(userNode) = idFrom(\"id\", $that.performer.user_id)\n\n// Set labels for nodes //\nCALL create.setLabels(revNode, [\"rev:\" + $that.page_title])\nCALL create.setLabels(dbNode, [\"db:\" + $that.database])\nCALL create.setLabels(userNode, [\"user:\" + $that.performer.user_text])\n\n// Create timeNode node to provide day/hour/minute bucketing and counting of revNodes //\nCALL reify.time(datetime($that.rev_timestamp), [\"year\", \"month\", \"day\", \"hour\", \"minute\", \"second\"]) YIELD node AS timeNode\nCALL incrementCounter(timeNode, \"count\", 1) YIELD count AS timeNodeCount\n\n// Set properties for nodes //\nSET revNode = $that,\n    revNode.type = \"rev\"\n\nSET dbNode.database = $that.database,\n    dbNode.type = \"db\"\n\nSET userNode = $that.performer,\n    userNode.type = \"user\"\n\n// Create edges between nodes //\nCREATE (revNode)-[:DB]->(dbNode),\n       (revNode)-[:BY]->(userNode),\n       (revNode)-[:AT]->(timeNode)"
    }
    ```

A [standing query](../learn/standing-queries/standing-queries.md) is configured to detect when new nodes are added to the graph and prints the event to standard out.

=== "YAML"

    ```yaml
    - pattern:
        type: Cypher
        query: |-
          MATCH (n)
          WHERE n.comment IS NOT NULL
          RETURN DISTINCT id(n) AS id
      outputs:
        output-1:
          type: CypherQuery
          query: |-
            MATCH (n)
            WHERE id(n) = $that.data.id
            RETURN n.comment AS line
          andThen:
            type: PrintToStandardOut
    ```

=== "JSON"

    ```json title="POST /api/v1/query/standing/STANDING-1"
    {
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n)\nWHERE n.comment IS NOT NULL\nRETURN DISTINCT id(n) AS id"
      },
      "outputs": {
        "output-1": {
          "type": "CypherQuery",
          "query": "MATCH (n)\nWHERE id(n) = $that.data.id\nRETURN n.comment AS line",
          "andThen": {
            "type": "PrintToStandardOut"
          }
        }
      }
    }
    ```

=== "YAML"

    ```yaml
    standingQueries:
      - name: comment-output
        pattern:
          type: Cypher
          query: |-
            MATCH (n)
            WHERE n.comment IS NOT NULL
            RETURN DISTINCT id(n) AS id
          mode: DISTINCT_ID
        outputs:
          - name: output-1
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (n)
                WHERE id(n) = $that.id
                RETURN n.comment AS line
              parameter: that
            destinations:
              - type: StandardOut
    ```

=== "JSON"

    ```json title="POST /api/v2/graph/quine/standingQueries"
    {
      "name": "comment-output",
      "pattern": {
        "type": "Cypher",
        "query": "MATCH (n)\nWHERE n.comment IS NOT NULL\nRETURN DISTINCT id(n) AS id",
        "mode": "DISTINCT_ID"
      },
      "outputs": [
        {
          "name": "output-1",
          "preEnrichmentTransformation": {
            "type": "InlineData"
          },
          "resultEnrichment": {
            "query": "MATCH (n)\nWHERE id(n) = $that.id\nRETURN n.comment AS line",
            "parameter": "that"
          },
          "destinations": [
            {
              "type": "StandardOut"
            }
          ]
        }
      ]
    }
    ```

The resulting event stream looks like this in the console. 

``` log
2023-02-07 15:39:37,967 Standing query `output-1` match: {"meta":{"isPositiveMatch":true,"resultId":"b995e3a0-12d2-2139-d349-4757801ad666"},"data":{"line":"Adding [[Template:Welcome|welcome message]] to new user's talk page"}}
```

## Running the Recipe

```shell hl_lines="1"
❯ java -jar quine-2.1.1.jar -r wikipedia.yaml
Graph is ready
Running Recipe: Ingest Wikipedia Page Create stream
Using 4 sample queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080 
```

## Summary

This recipe can serve as a boilerplate for other streaming recipes using the Wikipedia EventStreams source. We use variations of this recipe in our getting started guide and product demos.


---
# index
URL: https://quine.io/reference/

# Reference

-   [:octicons-package-dependents-24: __About Quine__](./about.md)

    ---

    Learn about our origin story and how it ties into our namesake, Willard Van Orman Quine.

-   [:material-api: __REST API__](./rest-api.md)

    ---

    The Interactive REST API.

-   [:simple-opentelemetry: __Telemetry__](./telemetry.md)

    ---

    Information about telemetry in Quine.

-   [:material-table-of-contents: __Glossary__](./glossary.md)

    ---

    Glossary of Quine terminology.

-   [:material-book: __Events__](./events.md)

    ---

    Meet us in person! We present at many of the data engineering focused events throughout the year.

-   [:fontawesome-solid-people-group: __Community__](./community/index.md)

    ---

    How you can contribute to Quine.

-   [:octicons-report-16: __Vulnerability Report__](./report-vulnerability.md)

    ---

    Report a Vulnerability w/ Quine

-   [:material-file-document: __Configuration__](../reference/config/configuration.md)

    ---

    How to configure Quine


---
# About Quine
URL: https://quine.io/reference/about/

# About Quine

Development for Quine began in 2014 when thatDot founder and CEO Ryan Wright
decided he'd rebuilt the same event processing microservice platform one too
many times. In 2015, Wright partnered with DARPA, through its Transparent
Computing program, to accelerate development.

The decision to open source the Quine graph streaming engine underscores the
thatDot team’s conviction that the best infrastructure software thrives within
an open, diverse community of contributors and that well-made software freely
available benefits everyone.

## Origin of the Project Name

![Willard Van Orman Quine](../assets/wvo-quine.png){ align=right loading=lazy width=200px }

Willard Van Orman Quine was a 20th century teacher and philosopher of logic and
science. Though influenced by logical positivism, Quine rejected the notion of
a distinction between analytical truths and empirical truths. Instead, he
maintained that both logic (and math) and science were subject to empirical
rules and could be refined through observation and the examination of
countervailing evidence.

It was Quine’s work [Two Dogmas of Empiricism](https://thereitis.org/quines-two-dogmas-of-empiricism/) that inspired the project’s name.
According to Ryan Wright, the original developer of Quine, "About 2 years into
the development of Quine the system, I started to realize that many of his
ideas had very direct parallels in the system I’d been developing. For example,
Quine’s interesting sidebar in 'Two Dogmas’ where he asks, ‘which points in
Ohio are starting points?’ impressed me as being related to indexes in Quine.
Hence the name."

For more on Quine’s work: [The Partially Examined Life Podcast Episode 66: Quine on Linguistic Meaning and Science](https://partiallyexaminedlife.com/product/ep-66-quine-on-linguistic-meaning-and-science/)


---
# Events
URL: https://quine.io/reference/events/

## Past Events

-   ![](../assets/images/events/current-2024.png){ class="event-center" }

    ---

    ### Current 2024: Streaming Entity Resolution for Kafka with Quine

    This lightning talk will highlight two approaches to real-time entity resolution on streaming data using the Quine streaming graph. We'll look at how to view your stream as a graph and why that's the key to: use event-triggered "standing queries" for real-time entity-resolution in graphs, and use the history of a stream to unlock AI-powered entity resolution with graph neural networks. In each case, a Kafka stream with messy data comes in, and a Kafka stream with clean "entity-resolved" data comes out.

    [:octicons-video-24: Watch ](https://current.confluent.io/2024-sessions/streaming-entity-resolution-for-kafka-with-quine)

-   ![](../assets/images/events/DoDIIS-2023.png){ class="event-center" }
    
    ---
    
    ### DoDIIS Worldwide 2023

    Are you attending DoDIIS 2023 this year? Stop by **booth 955** in the exhibitor hall to chat with our founder Ryan Wright and learn about our latest product Novelty Detector for AWS, a graph AI technique for AWS CloudTrail logs, providing real-time, priority-ranked threat alerts.

-   ![](../assets/images/events/Cassandra-Summit-Card.png){ class="event-center" }
    
    ---
    
    ### Cassandra Summit 2023

    December 12-13, 2023 in San Jose. Making stream processing easy, stable, and stateful with Quine + Cassandra. Quine continues to improve integrations with Cassandra for performance and stability.

    [:octicons-video-24: Watch ](https://youtu.be/pipF5yGQNeA)
    [:octicons-desktop-download-24: Slides ](https://static.sched.com/hosted_files/aidevcass23/d8/quine-cassandra-complex-event-processing-pdf.pdf)
    

-   ![](../assets/images/events/reInvent-2023.png){ class="event-center" }
    
    ---
    
    ### AWS re:Invent 2023

    thatDot is delighted to announce our participation at this year's AWS re:Invent. Stop by **booth 1518** in the Data Zone for an exclusive demonstration of Novelty Detector for AWS, a graph AI technique for AWS CloudTrail logs, providing real-time, priority-ranked threat alerts.

-   ![](../assets/images/events/ScyllaDB-Summit-Card.png){ class="event-center" }

    ---

    ### ScyllaDB Summit 2023

    Quine + ScyllaDB - The Easy Way to Build High-Performance, Stateful Event Stream Processing Pipelines.

    [:octicons-video-24: Watch](https://www.scylladb.com/scylladb-summit-2023/)

-   ![](../assets/images/events/Current22-Card-Event.png){ class="event-center" }  

    ---

    ### Current22: Quine + Kafka

    Build a Streaming Graph Pipeline on Apache Kafka with Quine. In this live-coding lightning talk, we'll start from scratch and build a streaming graph data pipeline from start to finish. (10 minutes)

    [:octicons-video-24: Watch](https://youtu.be/DKENFhSWzAI)

-   ![](../assets/images/events/Data-Stack-Show.jpg){ class="event-center" }  

    ---

    ### The Data Stack Show: What is Streaming Graph

    This week on The Data Stack Show, Eric and Kostas chat with Ryan Wright, philosopher-CEO at thatDot. During the episode, Ryan discusses all things graph databases, from use cases to scalability and more.

    [:octicons-video-24: Watch](https://datastackshow.com/podcast/what-is-streaming-graph-featuring-ryan-wright-of-thatdot/)

-   ![](../assets/images/events/reactive-summit.png){ class="event-center" }  

    ---

    ### Reactive Summit 2022

    Learn how Quine achieves high performance by putting the query execution inline with the event stream. High-volume events stream in, real-time answers to graph queries stream out.

    [:octicons-video-24: Watch](https://www.youtube.com/watch?v=Ywba1t6nsRA)

-   ![](../assets/images/events/datastax-event.png){ class="event-center" }  

    ---

    ### WORKSHOP: Using Quine + AstraDB

    Join us for a workshop where you learn by doing. We use Quine and Astra DB, built on Apache Cassandra, to work on a practical use case of detecting a password spraying attack, and help to keep your users safe!

    [:octicons-video-24: Watch](https://www.youtube.com/watch?v=IHgNmhPA7mA)

-   ![](../assets/images/events/cassandra-world-party.png){ class="event-center" }  

    ---

    ### Cassandra World Party

    Ryan Wright joins this one-day event that brings the global community together to celebrate the people and the technology behind Apache Cassandra. Lots of speakers, 5 minutes per talk. From July 20th, 2022.

    [:octicons-video-24: Watch](https://youtu.be/ID4YAU3Gnss)

-   ![](../assets/images/events/Cassandra-Corner-Quine.png){ class="event-center" }  

    ---

    ### The Cassandra Corner Podcast

    Ryan Wright, creator of Quine, joins the Cassandra Corner Podcast and talks about how Quine streaming graph database + Cassandra equals graph that scales to millions of events/second. (recorded June 13 '22)

    [:octicons-video-24: Watch](https://anchor.fm/cassandra-corner/episodes/ep3---Ryan-Wright-Quine-io-e1jntpe)


---
# Glossary
URL: https://quine.io/reference/glossary/

# Glossary

<!--
Conventions for writing this document: 
- Bold and capitalize each term.
- Multi-word terms should be written like a dictionary entry.
- Separate the term and the definition with ` - `
- The first phrase/sentence of the definition should begin with a lowercase and end with a period.
- Each additional sentence in the definition should be a complete sentence and use proper casing and punctuation.
- Alphabetize the list.
- Use "See:" to suggest further detailed reading on the same topic. Use "See also:" to suggest related topics.
-->

This page is an alphabetical list of technical terms which have a particular meaning for the Quine project.

- **Actor** - a unit of concurrent computation. An actor is a lightweight computational element which is single-threaded, encapsulates state, and communicates via message-passing.
- **Cypher** - a graph query language. It is used by Quine for ingesting data, running ad hoc queries, and setting standing queries. See: "[Cypher Language](../learn/cypher/index.md)"
- **Edge** - a relationship between exactly two nodes in a graph.
- **Event Sourcing** - the practice of saving -updates- to state instead of the total current state. [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) adds data to an append-only log.
- **Exploration UI** - an interactive user interface served by the Quine web server. See: "[Exploration UI](../getting-started/exploration-ui.md)"
- **Graph** - a logical data structure composed of Nodes and Edges.
- **ID Provider** - a user-configurable component of Quine which determines the type of ID used by each node in the graph. See: "[ID Provider](../core-concepts/id-provider.md)"
- **Member** - a computer that participates in a cluster. In other systems this is sometimes called a "node". Quine avoids using the term "node" to refer to a cluster member so as not to be confused with a node or vertex in a graph. See also: "Node"
- **Node** - a vertex in a graph. Node are uniquely defined by their ID and serve as a container for properties, connected to other nodes by edges.
- **Persistor** - a Quine-specific data storage mechanism used to persist data durably on disk. It may be on the same machine (even in the same process) as the Quine, or it may be served on a remote system. See: "[Persistor](../learn/persistors/index.md)"
- **Position** - a logical location in a Quine cluster which needs to be filled by a member. 
- **Property** - a key-value pair stored on a node. They key of a property is a string, the value is any one of the supported types in Quine.
- **Quick Query** - a query defined in the Exploration UI which enables easy graph exploration by right-clicking a node and selecting the quick queries enabled on that node. Quick queries are user customizable through the REST API or recipes.
- **QuineId** - the ID of a single node in the graph. See also: "ID Provider"
- **Recipe** - a set of configuration used to execute end-to-end functionality for a specific purpose. Recipes are contributed by community users and listed on the [Recipes](https://quine.io/recipes/) page.
- **Shard** - a logical division of the graph, responsible for managing a subset of nodes.
- **Standing Query** - a query set on the graph, which lives inside the graph, efficiently propagates, and produces results immediately.


---
# Recommended Operating Environment
URL: https://quine.io/reference/operating-env/

# Recommended Operating Environment

| Component | Recommendation |
|:----------|:---------------|
| Operating System | Ubuntu 22.04 LTS (recommended) |
| Quine Java | OpenJDK or Oracle 17 (requires at least 11) |
| Quine CPU | 8-32 Cores Recommended |
| Quine Memory | 16-20 GB RAM |

!!! info "EC2 instance‑type guidance"
    **Use fixed‑performance instances such as the `m7a` family for production clusters.**
    `m7a` nodes deliver *sustained* vCPU capacity.

    **Avoid burstable t‑series (`t3`, `t4g`, etc.) in production.**
    These instances rely on CPU‑credit buckets: once the credit balance is depleted, the vCPUs are throttled down to their low [baseline](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html#baseline_performance) (for example, about 30% of a core), causing sudden latency spikes and reduced throughput.

    Burstable nodes are fine for ad‑hoc testing or dev workloads that idle most of the time, but Quine's stream‑processing benefits from guaranteed compute.


---
# Report a Vulnerability
URL: https://quine.io/reference/report-vulnerability/

# Report a Vulnerability

If you discover a security vulnerability or have a security related concern about the software, please report it to: <security@quine.io>. Be sure to include all relevant details to understand and/or reproduce the vulnerability to ensure that your report receives the appropriate attention. Reports that do not include sufficient details are likely to be automatically filtered out as SPAM.


---
# REST API
URL: https://quine.io/reference/rest-api/




---
# Supported Connectors and Persistors
URL: https://quine.io/reference/supported-connectors-and-persistors/

# Supported Connectors and Persistors

## Persistors

The persistor is where Quine saves the graph. RocksDB is the default
and needs no configuration, so most deployments only revisit this choice when
one of the following applies:

- **RocksDB has no build for your architecture.** It ships as a native binary
  used through JNI, and startup fails on an unsupported platform. MapDB is the
  portable fallback, at the cost of a 2&nbsp;GB practical limit per memory-mapped
  file and off-heap memory use.
- **You need replication and failover.** Cassandra provides them, and
  reaches Cassandra-compatible services too. Amazon Keyspaces is the same
  trade with AWS operating it, at the cost of accepting only the `ONE`,
  `LOCAL_ONE`, and `LOCAL_QUORUM` read consistency levels.
- **Nothing needs to survive the run.** In-memory simulates a persistor without
  touching disk, and Empty makes every write a no-op, leaving only the node
  cache. Both are for tests and demos.

Set the choice with
`quine.store.type`.
[Persistors](../learn/persistors/index.md) covers each in depth, and backup is
delegated to whichever store you pick.

| Persistor | Configuration value | Description |
|:---|:---|:---|
| [RocksDB](../learn/persistors/index.md#rocksdb) | `rocks-db` | An embedded log-structured merge tree on the local filesystem. The default, and the fastest choice for a single host. |
| [MapDB](../learn/persistors/index.md#mapdb) | `map-db` | An embedded Java store on the local filesystem. The fallback where RocksDB has no native build for the host architecture. |
| [Apache Cassandra](../learn/persistors/cassandra-setup.md) | `cassandra` | A distributed database giving high throughput, replication, and failover. |
| [ScyllaDB](../learn/persistors/cassandra-setup.md) | `cassandra` | A Cassandra-compatible database, connected through the Cassandra persistor. |
| [Astra DB](../learn/persistors/cassandra-setup.md#astradb-configuration) | `cassandra` | DataStax's serverless Cassandra-compatible service. Needs an application token and a secure connect bundle. |
| [Amazon Keyspaces](../learn/persistors/cassandra-setup.md#amazon-keyspaces-configuration) | `keyspaces` | AWS's managed Cassandra-compatible service, for a distributed store without operating Cassandra yourself. |
| In-memory | `in-memory` | Holds everything in memory and writes nothing to disk. Useful for tests and short experiments; all data is lost on shutdown. |
| Empty | `empty` | Discards every write and returns nothing on read. No history, and nothing survives a restart. |

ScyllaDB and Astra DB are Cassandra-compatible, so they share the `cassandra`
configuration value and differ only in how you point it at them.

## Ingest sources

An ingest stream pairs one source with a Cypher `query` run on each record. The source decides only where the bytes
come from, so choose it by where your data already lives: a broker such as Kafka
or Kinesis for a continuous feed, a file or S3 object for data already at rest,
or Standard input and Number iterator to try something without setting up any
infrastructure.

Set it in the `source` block of
[Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest), or see
[Ingest Streams](../learn/ingest-sources/index.md) for how the rest of a stream
is built.

| Name | Configuration value | Formats | Compression (ZLIB, GZIP, BASE64) | Description |
|:---|:---|:---|:---|:---|
| [File Ingest](../../learn/ingest-sources/files-and-named-pipes/) | `File` | AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An active stream of data being ingested from a file on this Quine host. |
| [Kafka Ingest Stream](../../learn/ingest-sources/kafka/) | `Kafka` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kafka. |
| [Kinesis Data Stream](../../learn/ingest-sources/kinesis/) | `Kinesis` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis. |
| [Kinesis Data Stream Using Kcl lib](../../learn/ingest-sources/kinesis/) | `KinesisKCL` | Avro, Drop, Json, Protobuf, Raw | ✓ | A stream of data being ingested from Kinesis |
| Number Iterator Ingest | `NumberIterator` | — |  | An infinite ingest stream which requires no data source and just produces new sequential numbers every time the stream is (re)started. The numbers are Java `Long`s` and will wrap at their max value. |
| [Reactive Stream Ingest](../../learn/ingest-sources/reactive-streams/) | `ReactiveStream` | Avro, Drop, Json, Protobuf, Raw |  | A stream of data being ingested from a reactive stream. |
| S3 Ingest | `S3` | AvroContainer, CSV, Json, JsonL, Line, Parquet | ✓ | An ingest stream from a file in S3, newline delimited. This ingest source is experimental and its behavior is subject to change. It's best suited to continuously active streams; durability is not guaranteed once a stream has been inactive for 1 minute or more. |
| Server Sent Events Stream | `ServerSentEvent` | Avro, Drop, Json, Protobuf, Raw | ✓ | A server-issued event stream, as might be handled by the EventSource JavaScript API. Only consumes the `data` portion of an event. |
| [Simple Queue Service Queue](../../learn/ingest-sources/sqs---sns/) | `SQS` | Avro, Drop, Json, Protobuf, Raw | ✓ | An active stream of data being ingested from AWS SQS. |
| [Standard Input Ingest](../../learn/ingest-sources/stdin/) | `StdInput` | AvroContainer, CSV, Json, JsonL, Line, Parquet |  | An active stream of data being ingested from standard input to this Quine process. |
| WebSocket File Upload | `WebSocketFileUpload` | AvroContainer, CSV, Json, JsonL, Line, Parquet |  | Streamed file upload via WebSocket protocol. |
| Websockets Ingest Stream (Simple Startup) | `WebsocketClient` | Avro, Drop, Json, Protobuf, Raw |  | A websocket stream started after a sequence of text messages. |

## Standing query destinations

A standing query can have
one or more destinations, routed in parallel.
Run Cypher Query is the only one that acts back on the graph, enriching a
match or filtering out ones that don't qualify. The rest send results down stream,
so pick by the system that should receive them: a broker such as Kafka or
Kinesis to feed another pipeline, or an HTTP endpoint or Slack to notify a
service directly. Drop and Log JSON to Console are what you use while building,
before pointing anything at a real system.

Destinations are set in the
`outputs` of
[Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query),
and [Standing Queries](../learn/standing-queries/standing-queries.md) covers how
results are produced in the first place.

| Name | Configuration value | Formats | Description |
|:---|:---|:---|:---|
| [Broadcast to Reactive Stream](../../learn/standing-queries/standing-queries/#publish-to-reactive-stream) | `ReactiveStream` | JSON, Protobuf | Broadcasts data to a created Reactive Stream. Other thatDot products can subscribe to Reactive Streams. Reactive Stream outputs are only supported in standalone (single-host) deployments. |
| [Drop](../../learn/standing-queries/standing-queries/#drop) | `Drop` | — | Effectively no destination at all, this does nothing but forget the data sent to it. |
| [Log JSON to Console](../../learn/standing-queries/standing-queries/#log-json-to-standard-out) | `StandardOut` | JSON | Prints each result as a single-line JSON object to stdout on the application server. |
| [POST to HTTP[S] Webhook](../../learn/standing-queries/standing-queries/#post-to-webhook) | `HttpEndpoint` | JSON | Makes an HTTP[S] POST for each result. For the format of the result, see "Standing Query Result Output". |
| [Publish to Kafka Topic](../../learn/standing-queries/standing-queries/#publish-to-kafka-topic) | `Kafka` | JSON, Protobuf | Publishes provided data to the specified Apache Kafka topic. |
| [Publish to Kinesis Data Stream](../../learn/standing-queries/standing-queries/#publish-to-kinesis-stream) | `Kinesis` | JSON, Protobuf | Publishes provided data to the specified Amazon Kinesis stream. |
| [Publish to Slack Webhook](../../learn/standing-queries/standing-queries/#publish-to-slack) | `Slack` | Slack message | Sends a message to Slack via a configured webhook URL. See [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks). |
| [Publish to SNS Topic](../../learn/standing-queries/standing-queries/#publish-to-sns-topic) | `SNS` | JSON, Protobuf | Publishes an AWS SNS record to the provided topic. To guarantee delivery, writes that fail are retried indefinitely, so confirm the credentials and topic ARN before starting this output. An unfixable error (e.g., an invalid topic ARN or missing credentials) will retry forever without emitting results, which may stop the Standing Query this output is attached to. |
| [Run Cypher Query](../../learn/standing-queries/standing-queries/#cypher-query) | `CypherQuery` | — | Runs the `query`, where the given `parameter` is used to reference the data that is passed in. Runs at most `parallelism` queries simultaneously. |
| [Write JSON to File](../../learn/standing-queries/standing-queries/#log-json-to-a-file) | `File` | JSON | Writes each result as a single-line JSON record. For the format of the result, see "Standing Query Result Output". |


---
# Telemetry
URL: https://quine.io/reference/telemetry/

# Telemetry

As a company and an open-source community, our goal is to enhance the overall user experience for everyone. Your feedback is invaluable in helping us achieve this objective. To facilitate this, we have implemented telemetry for Quine, ensuring transparency and user privacy. The data we collect is *completely anonymous*.

Your participation in our telemetry contributes to the continuous improvement of Quine. Thank you for helping us make Quine better every day.

Participation in our telemetry is **anonymous** and optional (you may easily opt out).

## Why is telemetry important?

Without telemetry, collecting feedback is entirely manual. We value gathering feedback through community interactions, user interviews and community feedback, but they alone cannot give us a complete picture of how the community is interacting with Quine.

## What do we collect?

We collect basic usage information including:

- Product version
- Types of data sources and outputs configured
- Whether a recipe is in use
- General operational metrics

We do not collect any personally identifiable information, query content, or data flowing through the system.

## How to opt out

You can disable telemetry by setting `quine.help-make-quine-better` to `false` in your configuration:

```hocon title="quine.conf"
quine {
  help-make-quine-better = false
}
```

Or via system property:

```bash
java -Dquine.help-make-quine-better=false -jar quine-2.1.1.jar
```

## Questions

Have questions about our telemetry? Please reach out to the team on <a href="https://that.re/chat" target="_blank">Discord</a>. We are happy to hear any feedback that you have!


---
# Community
URL: https://quine.io/reference/community/

# Community

Quine is better because of contributions from developers like you. The articles in this section describe how you can join the community and participate in building something new and unique.

If you have questions, just click the Discord icon in the corner of the page.


---
# Code of Conduct
URL: https://quine.io/reference/community/code-of-conduct/

# Code of Conduct

## Code of Conduct for the Quine Community

thatDot is dedicated to providing the best community experience possible for the Quine community. Our goal is to provide the opportunity for community participants to learn, communicate, contribute and collaborate. The Community Code of Conduct governs how we all participate and behave. As such we are committed to creating a diverse, harassment-free experience for everyone, regardless of gender, sexual orientation, disability, physical appearance, body size, race, or religion. We do not tolerate harassment of community participants in any form. Any form of written, social or verbal communication that can be offensive or harassing to any community member, participant or staff is not allowed. Community participants violating these rules may be sanctioned or expelled from the community.

## Expectations for All Community Members

### Be kind.

All community participants should feel welcome, regardless of their personal background. Please be polite, courteous, and considerate to fellow participants. No offensive comments regarding to gender, sexual orientation, disability, physical appearance, body size, race, or religion will be tolerated.

### Be respectful.

We expect all participants to be respectful when communicating with other participants, even when differences of opinion arise. Participants are expected to work together to resolve disagreements constructively and respectfully. Disagreement is no excuse for poor manners. Please be patient.

### Reach out and ask for help.

Please inform our community operator or forum moderator if you feel a violation has taken place and our staff will address the situation. Ask questions if you are unsure and be helpful to those who ask. You can also contact [community@quine.io](mailto:community@quine.io)

### Communicate and collaborate.

The concept of the community is based on working together and participants will gain the most from the community by actively participating and communicating effectively. As such, we encourage collaboration and communication as long as they are conducted in a positive and constructive way.

### Continue.

This list is not exhaustive or complete. Please use your own good judgement on proper behavior and contribute to a productive and pleasant community experience.

## How To Report Inappropriate Behavior

If a community participant engages in harassing behavior, community staff may take any action they consider appropriate, including expulsion from the community. If you are being harassed or know of someone else is being harassed, please inform our community staff immediately by contacting [community@quine.io](mailto:community@quine.io)

We expect participants to abide by these rules at all community-related activities. Thank you for your cooperation.

## Privacy Policy

We understand that privacy is important to our community participants and users of these products and services. Our <a target="_blank" href="https://www.thatdot.com/privacy-policy/">privacy policy</a> explains how we collect, use, share, and protect personal information.


---
# Contributing
URL: https://quine.io/reference/community/contributing/

# Contributing

The community is the heart of all open-source projects. We welcome contributions from all people and strive to build a welcoming and open community of contributors, users, participants, speakers, lurkers, and anyone else who comes by.

## Code of Conduct

All community members must be good citizens; be sure to read the [Code of Conduct](code-of-conduct.md) page to understand what this means.

## Contributing Code

Code contributions can be made through Github. We welcome all contributors and any improvements to Quine, the website, recipes, etc.

## Contribution License

All contributions to the Quine repository are released under the same license as the Quine project overall. For details, see the license in the Github repository.


---
# Configuration
URL: https://quine.io/reference/config/configuration/

Configuration is supported by [Typesafe Config](https://github.com/lightbend/config), enabling
multiple ways to pass in options. Most commonly, configuration is provided via either Java system
properties (passed as command-line options) or via a
[HOCON](https://github.com/lightbend/config/blob/main/HOCON.md) config file. HOCON is a JSON-like
format that is very flexible and human-readable. The reference config below is in HOCON format.

```bash
# Example of setting configuration via configuration file
java \
  -Dconfig.file=quine.conf \
  -jar quine-2.1.1.jar

# Example of overriding configuration via system properties
java \
  -Dquine.webserver.port=9000 \
  -Dquine.id.type=uuid-3 \
  -jar quine-2.1.1.jar

# Example of overriding configuration via environment variables
CONFIG_FORCE_quine_webserver_port=9000 \
CONFIG_FORCE_quine_id_type=uuid-5 \
java \
  -Dconfig.override_with_env_vars=true \
  -jar quine-2.1.1.jar
```

## Memory Configuration

Quine caches nodes in memory as necessary. The setting for `in-memory-soft-node-limit` and `in-memory-hard-node-limit` along with the `shard-count` configuration determines how many nodes can be cached at a time, and therefore how much heap space the JVM needs available for processing. The shard count defaults to 4 and does not typically need to be changed. Each shard in a cluster member retains its own cache of in memory nodes. The soft limit (defaulting to 10000) determines the minimum size of this cache, and the hard limit (defaulting to 75000) determines the maximum size of the cache. The cache uses the flexible range between these values for nodes which are being put to sleep while other nodes are being rehydrated into memory (i.e. the difference between these values determines how many nodes can be going to sleep and waking up at the same time). The default settings would enable between 40,000 and 300,000 nodes in memory at a time. Depending on the use case, and the average memory footprint of a node, These values can be adjusted up or down to maximize usage of the memory on the machine and the JVM heap. 

!!! note
    The heap space requirement is primarily a function of the soft and hard node limit settings. Beyond that, leaving overhead for the operating system to manage direct memory requests will ensure the performance of the host server.

To avoid Garbage Collection (GC) pauses in the JVM heap, it is recommended that you set the memory allocated to Quine to a fixed size if possible. You can do this by setting `Xms` and `Xmx` to the same value, discouraging the JVM from dynamically resizing the heap. Resizing the heap triggers a full GC, which can mean a lengthy pause (meaning everything in the app is absolutely locked), depending on the size of the heap. Keep in mind that large heap spaces take longer in garbage collection so simply adding as much as possible could negatively affect performance. For this reason, it is not recommended to set the heap size larger than 16GB, and 12GB tends to be a good starting point for large graph implementations with high ingest requirements.

## Data Pipeline Security Recommendations

Data pipelines in Quine can be very flexible to almost any use-case, but with that flexibility comes potential for misuse. In order to keep Quine, its data, and its environment as secure as possible, we recommend you follow these best practices:

1. Use the latest version of Quine. We regularly release updates with security fixes.
2. Deploy Quine behind a reverse proxy that performs TLS termination, user authentication, and HTTP request logging. This will allow you to control access to your Quine instance and gives you more information about how and when your graph is being accessed.
3. Configure your data source according to that data source's best practices. For example, if using a Kafka data source, keep your Kafka servers up to date and use the `kafkaProperties` field in the Quine ingest configuration to enforce TLSv1.3 encryption between Quine and your Kafka cluster.

4. Do not pass unsanitized data to any Cypher procedures (functionality invoked with the `CALL` syntax). This can lead to query injection attacks (via procedures like `cypher.doIt` and `cypher.do.case`) or server-side request forgery (via procedures like `loadJsonLines`).
5. Configure file ingest security controls to restrict file access to specific directories. Use the `file-ingest.allowed-directories` setting to allow list permitted directories and set `file-ingest.resolution-mode` to `"static"` in production environments to only allow files present at startup. See the [Files and Named Pipes](../../learn/ingest-sources/files-and-named-pipes.md#security-controls) documentation for details.

## Loglevel Configuration

Quine uses the [Logback](https://logback.qos.ch/index.html) framework
for logging. Loglevel is configurable for several loggers via Java system properties:

| System property | Default | Affects |
|:----------------|:--------|:--------|
| `thatdot.loglevel` | `WARN` | Quine logs, plus the underlying Pekko actor system (actor lifecycle, remoting, and streams internals) |
| `root.loglevel` | `WARN` | All other loggers, including third-party dependencies |

Available log levels are `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`. See
documentation for more details:
[https://logback.qos.ch/manual/architecture.html#effectiveLevel](https://logback.qos.ch/manual/architecture.html#effectiveLevel)

Because `thatdot.loglevel` also controls the actor-system loggers, you can turn on verbose actor-system diagnostic logging with a single flag, without raising the log level for all third-party libraries:

```shell
java -Dthatdot.loglevel=DEBUG -jar ...
```

The two properties are independent: `root.loglevel` does not change the Quine or actor-system loggers, and `thatdot.loglevel` does not change third-party loggers.

Example, setting the level in a config file instead of a system property:

```kconfig
thatdot {
    loglevel = INFO
}
```

## Reference Documentation

Uncommented values are the defaults, unless otherwise noted. Unexpected configuration keys or
values in the `quine` block will report an error at startup; the error names the offending key and where it was found, so a typo in a config file points directly at the line to fix.

To see the configuration a running instance started with, send a `GET /api/v2/system/config` request. Not all configuration details are returned; in particular, credentials and other secrets are always excluded.

A single underscore `_` is used to indicate a required property with no default value. There are none of
these in the default configuration.

```kconfig
--8<-- "generated/quine/documented_config.conf"
```


---
# Webserver Config
URL: https://quine.io/reference/config/quine-webserver-advertise/

```kconfig
  # webserver binding configuration
  quine.webserver {
    # whether the webserver should be enabled
    enabled = true

    # Hostname or address of the interface to which the HTTP server should
    # be bound
    address = "0.0.0.0"

    # port to which the HTTP server should be bound
    # setting to `0` will choose an available port at random.
    port = 8080

    # Whether the webserver should perform TLS termination
    use-tls = no
  }
  # (optional) Configuration to use when advertising this server
  # quine.webserver-advertise {
    # Hostname or address using which the application should generate
    # user-facing hyperlinks to itself
    # address = "localhost"

    # port (on `address`) via which the HTTP server can be reached
    # port = 8080
  # }
```

## Serving Quine Behind a Reverse Proxy

Most enterprise deployments will put Quine behind a reverse proxy, load balancer, or other such network construct. Quine needs to bind to 0.0.0.0 (all interfaces), or a specific network interface, but the end-user reaches Quine via a very different address. This may be `localhost`, `quine.mycompany.com`, or any other address. Similarly, Quine may be configured to bind to one port, say, `8080`, but be exposed to the user on another port, say `80`. This is particularly common when using a containerized deployment of Quine. Any time Quine generates a self-referential URL (e.g., in the OpenAPI document's `servers` block), it should refer to an end-user resolvable address, rather than an interface's address.

Consider a simple production deployment of Quine in which the JVM is configured to bind to only the loopback interface on port 8080. That is,

```kconfig
quine.webserver {
  # whether the webserver should be enabled
  enabled = true

  # Hostname or address of the interface to which the HTTP server should
  # be bound
  address = "127.0.0.1"

  # port to which the HTTP server should be bound
  # setting to `0` will choose an available port at random.
  port = 8080

  # Whether the webserver should perform TLS termination
  use-tls = no
}
```

The Quine is then hosted behind a reverse proxy providing TLS termination and exposing the Quine instance at `quine.example.com`. The operator wishes to use a third-party API specification viewer. However, by default, the API specification document will render the server URL as `http://127.0.0.1:8080`, and trying to send API requests via documentation UI will cause errors as the useragent tries to connect to `127.0.0.1`, which resolves to the user's machine instead of the Quine host. To fix this, the Quine operator can set:

```kconfig
quine.webserver-advertise {
  # Hostname or address using which the application should generate
  # user-facing hyperlinks to itself
  address = "quine.example.com"

  # port (on `address`) via which the HTTP server can be reached
  port = 443
}
```

The documentation will now report the server's URL as `http://quine.example.com:443`. When the user's browser connects to this URL, their traffic is correctly routed through the TLS termination and reverse proxy to reach their Quine instance.

## Adding TLS Security

As of Quine 1.5.7, Quine supports TLS termination within the Quine application. While thatDot recommends customers deploy Quine behind a reverse proxy or load balancer rather than terminating TLS internally, Quine's own TLS termination can be used for prototyping or to encrypt the connection to the reverse proxy itself. In order to use Quine for TLS termination, you must have a Java keystore containing a key suitable for usage with your Java version's supported cipher suites.

Quine's webserver configuration is specified in the `quine.webserver.ssl` portion of the [Quine configuration](https://quine.io/reference/config/configuration/):

``` kconfig
quine {
  webserver {
    enabled = true
    address = "0.0.0.0"
    port = 443
    tls-enabled = true
  }
}
```

The system property `javax.net.ssl.keyStore` should be set to the path of a jks or PKCS12 keystore file on the machine running the Quine server. As Java keystores are secured by a password, the password should be provided for access via the `javax.net.ssl.keyStorePassword` system property.

As an alternative to the standard Java methods of TLS configuration, the environment variables `SSL_KEYSTORE_PATH` and `SSL_KEYSTORE_PASSWORD` may be used instead.

## Generating a keystore with a self-signed server key

For the best balance of performance, security, and compatibility with a wide variety of clients, we recommend generating an RSA2048 key. Under no circumstances should [RC4](https://blog.cloudflare.com/end-of-the-road-for-rc4/) be used. The most ergonomic way to generate a self-signed key is via the JDK's `keytool`, by using the following command and following the prompts that appear. This command will generate a 2048-bit RSA key so use as the server's private key, and convert it to the appropriate format to be consumed by Quine (a Java keystore file):

``` shell
keytool -genkey -alias server -keyalg RSA -keysize 4096 -keystore server_key.jks
```

Generally, the tooling used to interact with Quine's keystore is the same as the tooling used for managing any Java application's keystore; for example, Apache Tomcat, Kafka, or Cassandra. Many resources are available online describing different ways of interacting with Java keystores, including how to import and export keys from other common formats like X.509, or [generate certificate signing requests (CSRs)](https://www.digicert.com/kb/csr-ssl-installation/tomcat-keytool.htm) to endorse generated certificates elsewhere along a chain of trust. In addition, `keytool`'s built-in documentation (viewable with `man keytool`) provides a reference for many of the command-line arguments available.

!!! tip "Quine Enterprise"

    Quine Enterprise adds role based access control and other security & governance features. [Compare editions](https://www.thatdot.com/quine-open-source-vs-enterprise/).


---
# Upgrading
URL: https://quine.io/reference/upgrade/

# Upgrading

This guide covers breaking changes and migration steps for major Quine releases.

| Version                     | Key Changes                                        |
|:----------------------------|:---------------------------------------------------|
| [2.1.1](quine-2.1.1.md)    | Background queries and scheduled jobs, graph feeds via API and recipes, locale-independent `toLower`/`toUpper` |
| [2.1.0](quine-2.1.0.md)    | Strict API v2 validation, standing query propagation on create, redesigned Dashboard |
| [2.0.0](quine-2.0.0.md)    | API v2 default, metrics prefix change, recipe schema v2 |

## Migration Guides

- [Migrating from API v1](migrating-from-api-v1.md) — Endpoint mappings, design changes, and migration checklist for moving from API v1 to v2
- [Migrating from v1 Recipes](migrating-from-recipe-v1.md) — Steps and examples for updating recipe schema from v1 to v2


---
# Migrating from API v1
URL: https://quine.io/reference/upgrade/migrating-from-api-v1/

# Migrating from API v1

!!! warning "API v1 deprecation"
    API v1 is planned for deprecation and will be removed in a future release. New integrations should use v2; existing v1 integrations should plan to migrate.

Quine API v2 introduces improvements to response formats, error handling, and endpoint organization. This guide explains the key changes and how to migrate your integrations from v1 to v2.

## Why V2?

The v1 API evolved organically as Quine grew, resulting in inconsistencies that made the API harder to learn and use. V2 is a ground-up redesign that pins every decision to Google's [API Improvement Proposals (AIPs)](https://google.aip.dev/) — the same external design rules Google uses for its own public APIs. Anchoring v2 to a published standard means the rationale for every URL shape, status code, and wire format lives in one citable place rather than tribal knowledge, and well-known client patterns (pagination cursors, RFC 3339 timestamps, structured errors) work without per-endpoint surprises.

Quine is a JSON HTTP API, not a Protobuf/gRPC service, so a few AIP details are assumed by their gRPC tooling rather than written into the proposal text. Where an AIP relies on Protobuf semantics that don't apply to a JSON API (for example, the proto3 default-value-omission rule, or carrying `google.protobuf.Any` payloads inside an error response), v2 keeps the shape and intent of the AIP but adapts the encoding for plain JSON. Those deviations are called out next to the relevant principle below.

At a glance, v2 is focused on:

- **Predictable patterns** — Consistent naming conventions and HTTP method usage across all endpoints
- **Graph-scoped operations** — Ingest, query, and standing query endpoints are scoped under a named graph (`graph/quine`), making multi-graph support explicit in the URL

- **RPC-style actions** — Action endpoints use colon-separated verbs (`:pause`, `:resume`, `:shutdown`) to clearly distinguish actions from resource operations
- **Better error handling** — Structured error responses with actionable messages

## API Version Overview

| Version | Status                  | Base Path  | Notes                                                                                |
|:--------|:------------------------|:-----------|:-------------------------------------------------------------------------------------|
| v2      | Current (default)       | `/api/v2/` | Default API for all installations                                                    |
| v1      | Planned for deprecation | `/api/v1/` | Still mounted by default; will be removed in a future release |

Both API versions are currently available. V1 routes remain reachable for backwards compatibility. Migrate to v2 for all new development and plan migration of existing integrations.

## Design Principles

V2 follows REST conventions more strictly than v1. Each principle below cites the AIP that codifies it; the AIP itself is the authoritative source for edge cases and rationale.

**Graph-scoped resources** ([AIP-121](https://google.aip.dev/121) resource-oriented design, [AIP-122](https://google.aip.dev/122) resource names) — Operations on ingests, standing queries, Cypher, and algorithms are scoped under `/graph/quine/` in the URL path. Quine uses a single graph named `quine`. Scoping at the URL level (rather than via a query parameter) makes the parent–child relationship part of the resource name, so a single ingest or standing query has one canonical URL even across graphs.

**RPC-style action verbs** ([AIP-136](https://google.aip.dev/136) custom methods) — Action endpoints use a colon-separated verb suffix (e.g., `ingests/{ingestName}:pause`, `system:shutdown`). The colon syntax keeps actions on the same path as the resource they act on without misrepresenting them as sub-resources, so URL trees stay shaped around nouns while still naming verbs explicitly.

**camelCase path segments** ([AIP-122](https://google.aip.dev/122) resource names) — Path segments use camelCase (`standingQueries`, `shardSizeLimits`, `systemInfo`) instead of kebab-case (`standing-queries`, `shard-sizes`, `system-info`). This matches the camelCase JSON field convention from [AIP-140](https://google.aip.dev/140), so an identifier looks the same whether it appears in a URL or a request body.

**System endpoints at top level** — Administrative endpoints are grouped under `/system/` (renamed from `/admin/`). Treating "system" as a top-level category (rather than a peer of every per-graph collection) keeps `/graph/quine/…` reserved for graph-scoped resources.

**Plural resource names** ([AIP-122](https://google.aip.dev/122) resource names) — Collection endpoints use plural nouns (`/ingests` not `/ingest`) so the URL itself signals "this addresses many" vs. `/ingests/{ingestName}` for "this addresses one."

**Resource names in request body** ([AIP-133](https://google.aip.dev/133) standard `Create`) — When creating resources, the name is part of the resource representation in the request body, not the URL. The server validates the complete resource definition (name + configuration) atomically before accepting it, and the same JSON document round-trips through `GET`/`PUT`/`POST` without callers having to splice the name in and out of the URL.

**POST for actions, PUT for idempotent updates** ([AIP-133](https://google.aip.dev/133) / [AIP-134](https://google.aip.dev/134) / [AIP-136](https://google.aip.dev/136)) — Actions like `:pause`/`:resume` use POST because they trigger state changes and are not required to be idempotent. PUT is reserved for idempotent operations where repeating the request produces the same result; DELETE removes a resource and is also idempotent.

## Endpoint Path Changes

The tables below show the mapping from v1 to v2 endpoints. All v2 paths are relative to `/api/v2/`.

### System Endpoints

System endpoints (formerly "admin") manage system configuration, monitoring, and cluster operations. These have moved from `/admin/` to `/system/` and path segments now use camelCase.

| v1 Endpoint                           | v2 Endpoint                         | Notes                                |
|:--------------------------------------|:------------------------------------|:-------------------------------------|
| `GET /admin/build-info`               | `GET /system/systemInfo`            | Renamed, moved to `/system/`         |
| `GET /admin/config`                   | `GET /system/config`                | Moved to `/system/`                  |
| `GET /admin/graph-hash-code`          | `GET /graph/quine/hashCode`       | Graph-scoped, camelCase              |
| `GET /admin/liveness`                 | `GET /system/liveness`              | Moved to `/system/`                  |
| `GET /admin/metrics`                  | `GET /system/metrics`               | Moved to `/system/`                  |
| `GET /admin/readiness`                | `GET /system/readiness`             | Moved to `/system/`                  |
| `POST /admin/request-node-sleep/{id}` | —                                   | Removed                              |
| `POST /admin/shard-sizes`             | `GET /system/shardSizeLimits`       | Split into GET, moved, camelCase     |
| `POST /admin/shard-sizes`             | `PATCH /system/shardSizeLimits`     | Split into PATCH, moved, camelCase   |
| `POST /admin/shutdown`                | `POST /system:shutdown`             | RPC-style verb                       |

### Ingest Endpoints

Ingest endpoints manage data streaming into Quine. In v2, ingest operations are scoped under a named graph (`/graph/quine/ingests`). The stream name moved from the URL path to the request body when creating streams, and pause/resume operations use RPC-style verbs.

| v1 Endpoint                | v2 Endpoint                                            | Notes                            |
|:---------------------------|:-------------------------------------------------------|:---------------------------------|
| `GET /ingest`              | `GET /graph/quine/ingests`                           | Graph-scoped, pluralized         |
| `POST /ingest/{name}`      | `POST /graph/quine/ingests`                          | Name moved to request body       |
| `GET /ingest/{name}`       | `GET /graph/quine/ingests/{ingestName}`              | Graph-scoped, param renamed      |
| `DELETE /ingest/{name}`    | `DELETE /graph/quine/ingests/{ingestName}`            | Graph-scoped, param renamed      |
| `PUT /ingest/{name}/pause` | `POST /graph/quine/ingests/{ingestName}:pause`       | RPC-style verb, POST             |
| `PUT /ingest/{name}/start` | `POST /graph/quine/ingests/{ingestName}:resume`      | Renamed to `:resume`, POST       |

For ingest stream configuration and usage, see [Ingest Streams](../../learn/ingest-sources/index.md).

### Standing Query Endpoints

Standing queries are continuously-running pattern matchers that execute actions when patterns are detected in the graph. In v2, standing query operations are scoped under a named graph. Path segments use camelCase, and output management moves the output name into the request body.

| v1 Endpoint                                     | v2 Endpoint                                                                            | Notes                      |
|:------------------------------------------------|:---------------------------------------------------------------------------------------|:---------------------------|
| `GET /query/standing`                           | `GET /graph/quine/standingQueries`                                                   | Graph-scoped, camelCase    |
| `POST /query/standing/{name}`                   | `POST /graph/quine/standingQueries`                                                  | Name moved to request body |
| `GET /query/standing/{name}`                    | `GET /graph/quine/standingQueries/{standingQueryName}`                               | Graph-scoped, camelCase    |
| `DELETE /query/standing/{name}`                 | `DELETE /graph/quine/standingQueries/{standingQueryName}`                             | Graph-scoped, camelCase    |
| `POST /query/standing/{name}/output/{output}`   | `POST /graph/quine/standingQueries/{standingQueryName}/outputs`                      | Output name in body        |
| `DELETE /query/standing/{name}/output/{output}` | `DELETE /graph/quine/standingQueries/{standingQueryName}/outputs/{standingQueryOutputName}` | Graph-scoped, camelCase    |
| `POST /query/standing/control/propagate`        | `POST /graph/quine/standingQueries:propagate`                                        | RPC-style verb             |

For standing query configuration and output destinations, see [Standing Queries](../../learn/standing-queries/standing-queries.md).

### Cypher Query Endpoints

Cypher query endpoints execute ad-hoc queries against the graph. In v2, these are scoped under a named graph and use RPC-style verbs.

| v1 Endpoint                       | v2 Endpoint                                    | Notes                     |
|:----------------------------------|:-----------------------------------------------|:--------------------------|
| `POST /query/cypher`              | `POST /graph/quine/cypher:query`             | Graph-scoped, RPC verb    |
| `POST /query/cypher/nodes`        | `POST /graph/quine/cypher:queryNodes`        | Graph-scoped, RPC verb    |
| `POST /query/cypher/edges`        | `POST /graph/quine/cypher:queryEdges`        | Graph-scoped, RPC verb    |

### Algorithm Endpoints

Algorithm endpoints perform graph traversal operations like random walks. In v2, these are scoped under a named graph and use RPC-style verbs.

| v1 Endpoint                | v2 Endpoint                                                                   | Notes                               |
|:---------------------------|:------------------------------------------------------------------------------|:--------------------------------------|
| `GET /algorithm/walk/{id}` | `POST /graph/quine/algorithms/randomWalk/nodes/{nodeId}:generateRandomWalk` | Graph-scoped, RPC verb, param renamed |
| `PUT /algorithm/walk`      | `POST /graph/quine/algorithms/randomWalk:saveWalks`                         | Graph-scoped, RPC verb                |

### Query UI Endpoints

Query UI endpoints configure the Exploration UI with sample queries, quick queries, and node appearance customizations. These have moved to camelCase path segments.

| v1 Endpoint                      | v2 Endpoint                    | Notes     |
|:---------------------------------|:-------------------------------|:----------|
| `GET /query-ui/sample-queries`   | `GET /queryUi/sampleQueries`   | camelCase |
| `PUT /query-ui/sample-queries`   | `PUT /queryUi/sampleQueries`   | camelCase |
| `GET /query-ui/quick-queries`    | `GET /queryUi/quickQueries`    | camelCase |
| `PUT /query-ui/quick-queries`    | `PUT /queryUi/quickQueries`    | camelCase |
| `GET /query-ui/node-appearances` | `GET /queryUi/nodeAppearances` | camelCase |
| `PUT /query-ui/node-appearances` | `PUT /queryUi/nodeAppearances` | camelCase |

### Gremlin Endpoints

Gremlin was an alternative graph query language supported in v1. These endpoints are not available in v2. Cypher provides equivalent functionality with better performance and a more intuitive syntax:

- `POST /query/gremlin` (v1 only)
- `POST /query/gremlin/nodes` (v1 only)
- `POST /query/gremlin/edges` (v1 only)

Use Cypher query endpoints instead.

## Key Changes

This section describes behavioral changes that affect how you interact with the API, regardless of which endpoints you use.

### Response Format

Both **v1** and **v2** return data directly at the top level of the response body. List endpoints in **v2** return a paginated envelope with `items` and `nextPageToken` fields per [AIP-158](https://google.aip.dev/158) (pagination). **v2** 201 responses may include a `Warning` header with advisory messages.

The envelope wraps every list response over a user-managed collection (ingests, standing queries, namespaces, …) even when server-side paging is not currently implemented for that endpoint — the wrapper itself is the forward-compatibility contract. Changing from a bare array to an envelope later would be breaking, so v2 ships the envelope up front and `nextPageToken` is simply omitted when there are no further pages.

A small number of admin-curated configuration endpoints (`/queryUi/sampleQueries`, `/queryUi/quickQueries`, `/queryUi/nodeAppearances`) deliberately return bare arrays rather than a `Page` envelope. They are closed configuration blobs rather than user-managed collections, so paging serves no purpose and the bare-array shape is documented at the endpoints themselves.

See [Response Format](../../core-concepts/rest-api.md#response-format) for the complete response specification.

### Error Responses

**v1** used varied error formats depending on the error type. **v2** returns a unified `ApiError` envelope shaped after [AIP-193](https://google.aip.dev/193) / [`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto):

```json
{
  "error": {
    "code": 404,
    "status": "NOT_FOUND",
    "message": "Ingest stream 'my-ingest' does not exist",
    "details": []
  }
}
```

The `error` object includes `code` (the HTTP status code), `status` (the canonical AIP-193 status string such as `INVALID_ARGUMENT` or `NOT_FOUND`), `message` (the primary human-readable error), and `details` (additional structured context — request IDs, hints, machine-readable error classifications). Pinning the wire format to AIP-193 means clients can parse errors with one branch of code and get a stable `status` field for switching, instead of grepping `message` strings.

Because Quine is a JSON API rather than a Protobuf service, the `details[]` entries are modelled as a closed, type-discriminated union (`RequestInfo`, `Help`, `ErrorInfo`) instead of AIP-193's open `google.protobuf.Any` carrier. The shape mirrors `google.rpc.ErrorInfo` and friends for familiarity, but new variants are added by extending the union rather than packing arbitrary proto messages.

See [Error Responses](../../core-concepts/rest-api.md#error-responses) for the complete error format specification.

### Strict Request Validation

As of Quine 2.1.0, **v2** rejects request bodies containing unrecognized fields with a 400 `INVALID_ARGUMENT` error naming the offending field and listing the valid ones. **v1** silently ignores unknown fields, so misspelled optional settings that appeared to work under v1 (while silently not applying) will fail under v2. This applies to both JSON and YAML request bodies. See [Upgrading to 2.1.0](quine-2.1.0.md#stricter-api-v2-request-validation) for details.

### Enum Wire Format

Sealed-trait enumerations encode to SCREAMING_SNAKE_CASE strings per [AIP-126](https://google.aip.dev/126) (e.g. `RUNNING`, `INVALID_ARGUMENT`). The PascalCase identifiers stay in idiomatic Scala source; conversion happens at the codec boundary. Where the wire value must mirror an external system (e.g. Kafka's `PLAINTEXT`/`latest`), the codec uses an explicit literal instead of the AIP-126 default — those cases are documented at the field.

Type discriminators on sum types — the `"type": "<Kind>"` field on ingest, output, and credential configurations — are a deliberate exception: discriminator values use bare PascalCase names (`"type": "Kinesis"`, not `"type": "KINESIS"` or `"type": "KinesisIngest"`). AIP-126 covers enum *values*, not ADT *type tags*, and PascalCase discriminator values read more naturally next to the Scala class names they identify.

### Query Parameters

**v2** standardizes common query parameters using camelCase names (`atTime`, `timeout`) that work consistently across all graph-scoped endpoints that support them. Per [AIP-142](https://google.aip.dev/142) (time and duration), timestamps use RFC 3339 format (e.g., `2026-04-27T15:30:00Z`) instead of epoch milliseconds, and durations use Go-style strings (e.g., `20s`, `500ms`, `1.5m`) instead of millisecond integers. Both formats are self-describing and round-trip through standard JSON tooling. The `namespace` query parameter from previous versions is replaced by the graph name in the URL path (`/graph/{graphName}/...`). See [Query Parameters](../../core-concepts/rest-api.md#query-parameters) for usage details.

### Resource Creation Pattern

API v2 uses a consistent pattern for creating resources where the resource name is in the request body rather than the URL path, and the resource is scoped to a graph:

**v1:**

```text
POST /api/v1/ingest/my-ingest-name
Content-Type: application/json

{ "type": "FileIngest", ... }
```

**v2:**

```text
POST /api/v2/graph/quine/ingests
Content-Type: application/json

{ "name": "my-ingest-name", "type": "FileIngest", ... }
```

This applies to ingests and standing queries.

## Migration Checklist

1. **Update base paths** — `/admin/` is now `/system/`; data operations are now under `/graph/quine/`
2. **Update to camelCase** — Path segments changed from kebab-case to camelCase (e.g., `standing-queries` → `standingQueries`)
3. **Adopt RPC-style verbs** — Action endpoints use colon verbs (e.g., `/ingests/{name}/pause` → `/ingests/{ingestName}:pause`)
4. **Update HTTP methods** — Some endpoints changed from PUT/GET to POST
5. **Move resource names to body** — For create operations (ingests, standing queries)
6. **Update parameter names** — `name` → `ingestName`, `id` → `nodeId`, `standing-query-name` → `standingQueryName`
7. **Update query parameter formats** — `at-time` → `atTime` (RFC 3339 timestamp), `timeout` → `timeout` (duration string like `20s`)
8. **Update error handling** — Parse the new structured `ApiError` format with `code`, `status`, `message`, `details`
9. **Remove unrecognized request fields** — v2 rejects request bodies containing unknown fields (v1 silently ignored them)

## Related Documentation

- [REST API Reference](../rest-api.md) - Interactive API documentation


---
# Migrating from v1 Recipes
URL: https://quine.io/reference/upgrade/migrating-from-recipe-v1/

# Migrating from v1 Recipes

A v2 recipe uses v2 API entities instead of v1 API entities. The migration involves three kinds of changes:

1. **Structural changes** — ingest streams and standing query outputs are reorganized (names are required, config is nested differently).
2. **Type renames** — ingest source and destination type discriminators are renamed (e.g., `PrintToStandardOut` → `StandardOut`).
3. **Enum value changes** — enum values change from PascalCase to SCREAMING_SNAKE_CASE (e.g., `DistinctId` → `DISTINCT_ID`). See [Enum Wire Format](migrating-from-api-v1.md#enum-wire-format) for the convention behind this change.
4. Change the `version` attribute from `1` to `2`.

The top-level recipe attributes (`version`, `title`, `contributor`, `summary`, `description`, `ingestStreams`, `standingQueries`, `nodeAppearances`, `quickQueries`, `sampleQueries`, `statusQuery`) remain the same.

## Example

The following demonstrates migrating the [Data Enrichment with Webhooks](../../recipes/webhook.md) recipe from v1 to v2. The API v1 entities must be replaced with API v2 entities. For example, the `ingestStreams` attribute is an array of [Ingest Stream Configuration](/reference/rest-api/?av=v1#/paths/POST/api/v1/ingest/%7Bname%7D) objects in v1, but an array of [Quine Ingest Configuration](/reference/rest-api/?av=v2#/operations/create-ingest) objects in v2.

=== "Recipe v1"

    ```yaml
    --8<-- "recipes/assets/webhook.yaml"
    ```

=== "Recipe v2"

    ```yaml
    version: 2
    title: Data Enrichment with Webhooks
    contributor: https://github.com/mastapegs
    summary: Stream numbers into graph and notify HTTP endpoint to enrich graph
    description: |-
      This recipe will stream numbers into the graph and stream them out to an HTTP endpoint, which will
      then calculate the factors of those numbers, and create relationships between the numbers and their
      factors.
    ingestStreams:
      - name: number-iterator                 # ingest stream must have a name
        source:
          type: NumberIterator                # type moves under source
          startOffset: 1                      # startAtOffset → startOffset
          limit: 13                           # ingestLimit → limit
        query: |-                             # query moves out of format
          WITH toInteger($that) AS number
          MATCH (n) WHERE id(n) = idFrom("Number", number)
          SET n:Number, n.number = number
    standingQueries:
      - name: number-processor               # standing query must have a name
        pattern:
          type: Cypher
          mode: DISTINCT_ID
          query: |-
            MATCH (n:Number)
            WHERE n.number IS NOT NULL
            RETURN DISTINCT id(n) AS id
        outputs:                             # outputs changes from a map to an array
          - name: log-to-console             # output name moves to the `name` field
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-                      # query moves under resultEnrichment
                MATCH (n:Number)
                WHERE id(n) = $that.id
                RETURN n.number AS number, $that.id AS id
              parameter: that
            destinations:                    # andThen → destinations
              - type: StandardOut            # PrintToStandardOut → StandardOut
          - name: post-to-webhook
            preEnrichmentTransformation:
              type: InlineData
            resultEnrichment:
              query: |-
                MATCH (n:Number)
                WHERE id(n) = $that.id
                RETURN n.number AS number, $that.id AS id
              parameter: that
            destinations:
              - type: HttpEndpoint           # PostToEndpoint → HttpEndpoint
                url: http://127.0.0.1:3000/webhook
    nodeAppearances:
      - predicate:
          propertyKeys: []
          knownValues: {}
          dbLabel: Number
        label:
          type: Property
          key: number
          prefix: "Number: "
    quickQueries: []
    sampleQueries:
      - name: Return all Number nodes
        query: MATCH (n:Number) RETURN n
    ```

## Structural Changes

### Ingest Streams

In v1, each ingest stream is a flat object with `type` and `format` (which contains the Cypher query). In v2, ingest streams require a `name`, nest source config under `source`, and move the Cypher query to the top level.

| v1 | v2 |
| -- | -- |
| No name (assigned `INGEST-#`) | Required `name` field |
| Top-level `type` | `source.type` |
| `format: { type: CypherJson, query: ... }` | Top-level `query`; optional `fileFormat` or `streamingFormat` |
| `startAtOffset` | `source.startOffset` |
| `ingestLimit` | `source.limit` |

### Standing Query Outputs

In v1, `outputs` is a map of name → output config. In v2, `outputs` is an array of objects with a `name` field, and the output pipeline is restructured.

| v1 | v2 |
| -- | -- |
| `outputs: { name: { ... } }` (map) | `outputs: [ { name: "...", ... } ]` (array) |
| Top-level `query` in output | `resultEnrichment: { query: ..., parameter: ... }` |
| `andThen: { type: ... }` | `destinations: [ { type: ... } ]` |
| `$that.data.id` | `$that.id` |
| N/A | `preEnrichmentTransformation: { type: InlineData }` |

## Type and Enum Value Changes

Recipe v2 uses [API v2](/reference/rest-api/?av=v2) entities, which follow different naming conventions than v1. For the general convention, see [Enum Wire Format](migrating-from-api-v1.md#enum-wire-format).

### Enum Values

Enum values change from PascalCase to SCREAMING_SNAKE_CASE. The v2 API rejects v1-style values.

| Field | v1 | v2 |
| ----- | -- | -- |
| Standing query `mode` | `DistinctId` | `DISTINCT_ID` |
| Standing query `mode` | `MultipleValues` | `MULTIPLE_VALUES` |
| Quick query `sort` | `Node` | `NODE` |
| Quick query `sort` | `Text` | `TEXT` |
| `fileIngestMode` | `Regular` | `REGULAR` |
| `fileIngestMode` | `NamedPipe` | `NAMED_PIPE` |

### Ingest Source Types

Type discriminators remain PascalCase but are renamed to shorter forms. The type moves from the top level to `source.type`.

| v1 `type` | v2 `source.type` |
| --------- | ---------------- |
| `FileIngest` | `File` |
| `KafkaIngest` | `Kafka` |
| `KinesisIngest` | `Kinesis` |
| `KinesisKCLIngest` | `KinesisKCL` |
| `NumberIteratorIngest` | `NumberIterator` |
| `S3Ingest` | `S3` |
| `SQSIngest` | `SQS` |
| `ServerSentEventsIngest` | `ServerSentEvent` |
| `StandardInputIngest` | `StdInput` |
| `WebsocketSimpleStartupIngest` | `WebsocketClient` |

### Destination Types

v1 `andThen.type` values map to v2 `destinations[].type`:

| v1 | v2 |
| -- | -- |
| `PrintToStandardOut` | `StandardOut` |
| `PostToEndpoint` | `HttpEndpoint` |
| `PostToSlack` | `Slack` |
| `WriteToFile` | `File` |
| `WriteToKafka` | `Kafka` |
| `WriteToKinesis` | `Kinesis` |
| `WriteToSNS` | `SNS` |
| `CypherQuery` | `CypherQuery` |
| `Drop` | `Drop` |


---
# Upgrading to 2.0.0
URL: https://quine.io/reference/upgrade/quine-2.0.0/

# Upgrading to 2.0.0

This page covers what you need to know when upgrading to Quine 2.0.0.

## What's New

- **Redesigned Exploration UI** — modernized layout with sidebar navigation, a new Dashboard landing page, a Streams management page, JSON-LD graph export, and updated node interaction behavior. See [Exploration UI Changes](#exploration-ui-changes) for details.
- **API v2 is now the default** — API v1 endpoints remain available but are planned for removal
- **Recipe schema v2** with API v2 entities
- **Docker base image updated** to `eclipse-temurin:21.0.10_7-jre-noble` (Ubuntu 24.04 LTS)
- **All-node scan queries** now filter server-side for better performance
- **Content Security Policy headers** added by default
- **Credential redaction** in API responses

## Breaking Changes

### Metrics Prefix Changes

All graph-scoped metrics (shard, node, standing query, and ingest) are now prefixed with `quine.`. For example, `node.property-counts` becomes `quine.node.property-counts` and `ingest.my-stream.count` becomes `quine.ingest.my-stream.count`.

Update your monitoring infrastructure (Grafana dashboards, InfluxDB queries, alerting rules) to include the `quine.` prefix for all [graph-scoped metrics](../../learn/metrics/metrics.md). Global metrics (persistor, shared valve) remain unchanged.

### Updating Grafana Queries

The exact syntax depends on your metrics reporter.

**InfluxDB example:**

```sql
-- Previous query
SELECT mean("value") FROM "node_property_counts" WHERE $timeFilter

-- 2.0.0+ query
SELECT mean("value") FROM "quine_node_property_counts" WHERE $timeFilter
```

**Prometheus example:**

```promql
# Previous query
rate(node_property_counts[5m])

# 2.0.0+ query
rate(quine_node_property_counts[5m])
```

## Exploration UI Changes

The web interface has been redesigned with a modernized layout, sidebar navigation, and consistent theming across all pages.

### New Pages

- **Dashboard** — A new landing page with a system overview diagram, host metrics, ingest stream status, and standing query status.
- **Streams** — A management page for creating and managing ingest streams and standing queries directly through the UI.

### New Features

- **JSON-LD export** — The Exploration UI download menu now includes a JSON-LD graph export option.

### Behavior Changes

- **Node pinning** — Nodes are now pinned in place by default when dragged. To unpin a node, Shift+click-hold it.
- **Default node appearance** — Nodes now render the `name` property by default, regardless of label. This is customizable via the [UI Styling endpoints](/reference/rest-api/?av=v2#/operations/list-node-appearances).

## API v1 Deprecation

API v2 is now the default for all installations. API v1 remains available but is planned for deprecation and will be removed in a future release.

See [Migrating from API v1](migrating-from-api-v1.md) for the complete migration guide, including endpoint mappings, design changes, and a migration checklist.

## Recipe Schema v1 Deprecation

Recipes now support schema version 2, which uses API v2 entities. Version 1 recipes continue to work but should be migrated.

See [Migrating from v1 Recipes](migrating-from-recipe-v1.md) for migration steps and a before/after example.


---
# Upgrading to 2.1.0
URL: https://quine.io/reference/upgrade/quine-2.1.0/

# Upgrading to 2.1.0

This page covers what you need to know when upgrading to Quine 2.1.0.

## What's New

- **Redesigned Status Dashboard** — the home page is now built around a live diagram of how data flows through the system, from ingests through the write pipeline, on to standing queries, round-trip to the persistor, and out through outputs. Color coding shows whether each stage is flowing, constrained, backpressured, or stopped, and highlights the bottleneck stage. See the [Dashboard](../../getting-started/dashboard.md) page.

- **Rebuilt Cypher editor** — the query editor used across the UI now provides Cypher syntax highlighting and a multi-line editing mode for longer queries. See [Exploration UI](../../getting-started/exploration-ui.md).
- **Standing Query Inspections and Graph Feeds** — sample a standing query's output pipeline directly from the Exploration UI, or set up a feed that draws a standing query's live results onto the graph as they happen.
- **Exploration UI Settings** — a new settings panel for editing sample queries, quick queries, node appearances, and graph feeds directly in the UI, instead of only through the REST API.

- **Standing query propagation on creation** — the create standing query API can propagate to existing data in the same request. See [Behavior Changes](#behavior-changes).
- **Per-ingest restart policy** — retry count, backoff timing, and retry window are configurable per ingest via `onStreamError`. See [Ingest Sources](../../learn/ingest-sources/index.md).
- **Cypher pattern predicates** — relationship patterns used as a boolean condition, for example combined with `AND`/`OR`/`NOT` or inside `CASE WHEN`, now evaluate correctly. See [Cypher Enhancements](../../learn/cypher/advanced-cypher.md).

## Breaking Changes

### Stricter API v2 Request Validation

API v2 now rejects requests containing unrecognized or misspelled JSON (or YAML) fields with a `400` error naming the offending field, for example:

```json
{
  "error": {
    "code": 400,
    "status": "INVALID_ARGUMENT",
    "message": "Invalid value for: body (Unexpected field: [nam]; valid fields: name, source, query)",
    "details": []
  }
}
```

Previously, unknown fields were silently ignored and defaults were applied instead. Requests that used to succeed while carrying a misspelled field, and silently doing the wrong thing, now fail. Check any automation that constructs API v2 request bodies. API v1 is unaffected.

### System Configuration Endpoint Returns Operational Fields Only

`GET /api/v2/system/config` now returns an explicit, pre-approved set of operational fields (persistor type, webserver address, shard count, node limits, metrics reporter types, default API version) instead of the entire raw configuration with sensitive fields masked. Credentials and other secrets embedded in the configuration can no longer be exposed through this endpoint. Update any tooling that read other fields from this endpoint's response.

## Behavior Changes

### Standing Queries Propagate on Creation

By default, a newly created standing query now propagates to data already in the graph, matching against every node in the in-memory cache. In 2.0.x, a new standing query matched only data written after it was registered.

The behavior is controlled by a new `propagateTo` query parameter on `POST /api/v2/graph/quine/standingQueries`:

- `NONE`, no propagation; matches only data changed after registration (the 2.0.x behavior)
- `EXCLUDE_SLEEPING` (default), propagates to nodes currently in the in-memory cache
- `INCLUDE_SLEEPING`, propagates to all nodes, waking sleeping nodes from the persistor; tune with `wakeUpParallelism` (default 4)

Pass `propagateTo=NONE` to restore the previous behavior. See [Standing Queries](../../learn/standing-queries/standing-queries.md) for details.

### Web UI Updates Automatically After Deploys

The web UI now loads the latest version after a new release is deployed, without requiring a hard refresh to clear stale cached files. No action is needed; a stale browser tab from a pre-2.1.0 deployment may still need one final hard refresh.


---
# Upgrading to 2.1.1
URL: https://quine.io/reference/upgrade/quine-2.1.1/

# Upgrading to 2.1.1

This page covers what you need to know when upgrading to Quine 2.1.1.

## What's New

- **Background queries and scheduled jobs** — run long or recurring ad-hoc queries beyond the limits of a synchronous request. Background queries survive request and page timeouts, report status throughout, and can be cancelled on demand; results stream to any supported destination (Kafka, Kinesis, SNS, HTTP, Cypher, files), or to none for effect-only runs. Scheduled jobs fire on a fixed interval or wall-clock time, hourly through monthly, in any IANA time zone, survive restarts, and re-fire interrupted runs.
- **Graph feeds via API and recipes** — graph feeds now have an API endpoint and are supported in recipes, in addition to being configurable from the [Exploration UI](../../getting-started/exploration-ui.md).

- **Resizable result columns** — Exploration UI result table columns are drag-resizable, and long values, arrays, and objects stay on one line with the full value shown on hover.

## Behavior Changes

### `toLower` and `toUpper` Are Locale-Independent

The Cypher `toLower` and `toUpper` functions previously used the JVM's default locale, which meant the same query could return different results depending on the locale of the host running it. Both functions now use a fixed, locale-independent mapping.

The practical difference appears in locales with special case rules. Under a Turkish default locale, for example, `toUpper("i")` previously returned `İ` (dotted capital I) and now returns `I`. If you have data or queries that depended on the previous locale-sensitive behavior, review them before upgrading. Deployments running under an English or root locale see no change.


---
# Tutorials
URL: https://quine.io/tutorials/

# Tutorials

-   [:material-ethereum: __Ethereum Demo__](./ethereum-demo.md)

    ---

    Demo of using Quine to monitor and trace suspected fraudulent transactions on the
    Ethereum blockchain.

-   [:material-graph: __How to load data?__](./loading-data/loading-data.md)

    ---

    3 different ways of loading data into Quine.

-   [:material-reload: __Processing an event stream__](./3d-data/3d-data-ingest-sq.md)

    ---

    How to think about processing event streams with Quine.


---
# Ethereum Demo
URL: https://quine.io/tutorials/ethereum-demo/

Ryan Wright demos Quine streaming graph using the [Ethereum tag propagation recipe](../recipes/ethereum.md), to ingest live blockchain data and uses Quine's unique standing query feature to identify fraudulent transactions.

<iframe width="1280" height="720" src="https://www.youtube.com/embed/Z8pXVof9BfE?rel=0&controls=1&;autoplay=0&;mute=0&;start=0" title="Ethereum Demo on Quine.io" frameborder="0" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

Topics covered include: [ingest streams](../learn/ingest-sources/index.md), [standing queries](../learn/standing-queries/standing-queries.md) and, [recipes](../getting-started/recipes-tutorial.md).


---
# Processing an Event Stream
URL: https://quine.io/tutorials/3d-data/3d-data-ingest-sq/

# Processing an Event Stream

Finding complex patterns in streaming data often requires creating new elements that represent new types, composite values, judgments, or metrics. As new data streams in, these patterns are recognized immediately by standing queries and used to create more meaningful data elements tied to their underlying original data.

The result is:

* An interconnected graph of original data loaded into the system
* Associated new data which is smaller in size, but more meaningful
* Many possible "interpretations" of data living together happily in a streaming system

<video width="688" height=516 autoplay playsInline muted loop>
<source src="../3d-data-assets/3d_data.m4v" type="video/mp4" />
Your browser does not support video playback.
</video>

In this exploration, we will consume the public dataset of Enron email messages (500,000+ emails from 150 individuals released as part of Enron's prosecution for fraud) .

Email is ubiquitous and forms a natural web of interlinked data. However, the records in the dataset are just simple, raw email messages. In this exploration, we demonstrate how to read and store the original data, connect it into an interesting structure, and produce new data that builds toward answers of interest.

## Step #1: Plan for incoming data structure

The original dataset is in CSV format with the following columns:

```csv
"file", "message"
```

The `message` column from the original data set is all that interests us for this exploration.

For convenience, the dataset has been preprocessed and made available as a line-based JSON file [enron.json.zip](https://thatdot-public.s3.us-west-2.amazonaws.com/enron.json.zip) (385MB compressed, 1.6GB unzipped) with the following structure:

```json
{
  "sourceFile": "",
  "headers": [
    {
      "name": "",
      "value": ""
    }
  ],
  "from": "",
  "to": [],
  "subject": "",
  "timestamp": [],
  "body": ""
}
```

For the ingest steps below, we will make each message a node in a graph. The node ID will be deterministically generated from the email metadata. All JSON fields are stored on the node as key/value pairs (or "properties") on the respective node. For convenience, each node will be given a label of "Message".

This ingest plan will create a single node for each email message, entirely disconnected from any other node in the graph. This structure is directly analogous to each email message representing one row of a relational database, where the columns are the JSON object keys.

This can be accomplished with the following Cypher query used to ingest each JSON data record passed in as `$that`:

```cypher
MATCH (n)
WHERE id(n) = idFrom('message', $that.from, $that.to, $that.timestamp)
SET n = $that, n:Message
```

## Step #2: Set Standing Queries

_Quine_ has the unique ability to set a query which lives inside the graph, propagates automatically, and can trigger arbitrary action immediately for each result found when new data completes the query. standing queries can be applied to existing data, or only to new data coming in. We will use "universal standing queries" which are applied to new data to shape the graph. We will set the standing queries ahead of time and once they are all set up, we will begin ingesting the data.

### Standing Query #1: Connect Nodes via SENDER Email Address

Streaming data does not need to begin in a graph structure. Typical row or column-oriented data is a perfectly fine starting point, as is the de facto standard JSON format. But the building consensus of modern data processing is that connected data is more valuable than disconnected data. We’ll illustrate and make the connections here with a standing query.

We want a standing query that starts with this raw `message` node:

![Message Node](3d-data-assets/message_node.png)

and turns it into this connected set of nodes:

![Message With Address](3d-data-assets/message_with_address.png)

We can accomplish this with a standing query consisting of a pair of Cypher queries—one describing the pattern to find each node of interest:

```cypher
MATCH (n) WHERE n.from IS NOT NULL RETURN DISTINCT id(n) AS id
```

and the other describing the action we want to take (creating new data) for each matched result (`$that.data`):

```cypher
MATCH (n) WHERE id(n) = $that.data.id
MATCH (m) WHERE id(m) = idFrom('email', n.from)
CREATE (n)-[:from]->(m)
SET m.email = n.from, m:Address
```

When the queries and configuration above are combined together, the following JSON makes up the payload of a `POST` API call to [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query)

```json
{
  "name": "from",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) WHERE n.from IS NOT NULL RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "emailStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id MATCH (m) WHERE id(m) = idFrom('email', n.from) CREATE (n)-[:from]->(m) SET m.email = n.from, m:Address",
          "parameter": "that"
        }
      ]
    }
  ]
}
```

This API call can be issued via the built in API documentation page at `/docs` or with the following `curl` command at a Unix command line:

```bash
curl -X 'POST' \
  'https://localhost:8080/api/v2/graph/quine/standingQueries' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "from",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) WHERE n.from IS NOT NULL RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "emailStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id MATCH (m) WHERE id(m) = idFrom('\''email'\'', n.from) CREATE (n)-[:from]->(m) SET m.email = n.from, m:Address",
          "parameter": "that"
        }
      ]
    }
  ]
}'
```

### Standing Query #2: Connect Nodes via RECEIVER Email Address

Similar to the first standing query, now we want to pull out the email addresses in the "To:" field of each email message and connect the `Message` node to the nodes corresponding to each email address node. Unlike the "From:" field, there are often many addresses in the "To:" field.

We’d like to take data that looks like this:

![Message Node](3d-data-assets/message_node.png)

And turn it into data like this:

![Message Node](3d-data-assets/message_with_to_addresses.png)

As we did in the the first standing query, this is done with two Cypher queries. One to match the Message node, just as before, but with a "to" field:

```cypher
MATCH (n) WHERE n.to IS NOT NULL RETURN DISTINCT id(n) AS id
```

and another to update the graph with the results:

```cypher
MATCH (n) WHERE id(n) = $that.data.id
WITH n.to AS toAddys, n
UNWIND toAddys AS toAddy
MATCH (m) WHERE id(m) = idFrom('email', toAddy)
CREATE (n)-[:to]->(m)
SET m.email = toAddy, m:Address
```

The following JSON payload can be passed in to the `POST` endpoint at [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query):

```json
{
  "name": "to",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) WHERE n.to IS NOT NULL RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "toFromStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id WITH n.to AS toAddys, n UNWIND toAddys AS toAddy MATCH (m) WHERE id(m) = idFrom('email', toAddy) CREATE (n)-[:to]->(m) SET m.email = toAddy, m:Address",
          "parameter": "that"
        }
      ]
    }
  ]
}
```

Or issued via `curl` on the command line:

```bash
curl -X 'POST' \
  'https://localhost:8080/api/v2/graph/quine/standingQueries' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "to",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (n) WHERE n.to IS NOT NULL RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "toFromStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id WITH n.to AS toAddys, n UNWIND toAddys AS toAddy MATCH (m) WHERE id(m) = idFrom('\''email'\'', toAddy) CREATE (n)-[:to]->(m) SET m.email = toAddy, m:Address",
          "parameter": "that"
        }
      ]
    }
  ]
}'
```

### Standing Query #3: Identify SEND and RECEIVE

Analysis of streaming data often requires writing and deploying new microservices which can operate quickly on data streams and persist their results in ways useful to other services. In **Quine**, this is done with standing queries, just as before.

If we needed to find email addresses which were both sender and receiver in this dataset, we can define a new standing query that checks for that pattern (slightly more interesting than the previous patterns) and creates an edge associating those Address nodes with a new node that represents the set we care about to create a sub-graph.

We want to find node that look like this:

![Message Node](3d-data-assets/from_and_to.png)

And turn them into nodes which look like this:

![Message Node](3d-data-assets/send_and_receive_node.png)

Matching the pattern of interest can be accomplished with the following Cypher query:

```cypher
MATCH (to)-[:to]->(n)<-[:from]-(from) RETURN DISTINCT id(n) AS id
```

And transforming the data can be done with this Cypher query:

```cypher
MATCH (n) WHERE id(n) = $that.data.id
MATCH (m) WHERE id(m) = idFrom('sendAndReceive')
CREATE (n)-[:sendAndReceive]->(m)
SET m.name = 'Sends and Receives'
```

These can be assembled in the following JSON payload to be delivered via `POST` to the endpoint at [Create Standing Query: `POST /api/v2/graph/quine/standingQueries`](/reference/rest-api/?av=v2#/operations/create-standing-query)

```json
{
  "name": "s-r",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (to)-[:to]->(n)<-[:from]-(from) RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "sendAndReceiveStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id MATCH (m) WHERE id(m) = idFrom('sendAndReceive') CREATE (n)-[:sendAndReceive]->(m) SET m.name = 'Sends and Receives'",
          "parameter": "that"
        }
      ]
    }
  ]
}
```

Or issued via `curl` on the command line as follows:

```bash
curl -X 'POST' \
  'https://localhost:8080/api/v2/graph/quine/standingQueries' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "s-r",
  "pattern": {
    "type": "Cypher",
    "query": "MATCH (to)-[:to]->(n)<-[:from]-(from) RETURN DISTINCT id(n) AS id"
  },
  "outputs": [
    {
      "name": "sendAndReceiveStructure",
      "destinations": [
        {
          "type": "CypherQuery",
          "query": "MATCH (n) WHERE id(n) = $that.data.id MATCH (m) WHERE id(m) = idFrom('\''sendAndReceive'\'') CREATE (n)-[:sendAndReceive]->(m) SET m.name = '\''Sends and Receives'\''",
          "parameter": "that"
        }
      ]
    }
  ]
}'
```

## Step #3: Start Data Ingest

With the desired standing queries established, all new incoming data will advance the overall structure of the system’s data toward the patterns described in those standing queries. As new data comes in and results in matches being produced, the update will be applied automatically—regardless of the order in which the data arrives or the number of matches made, in progress, or incomplete so far. Quine supports many ingest sources, including some streaming systems such as AWS Kinesis, as well as local resources such as files.

Quine can read directly from the local filesystems via common data formats. To start data ingest from a line-based JSON file on the Quine host, we will use the same Cypher query (as mentioned in Step #1) to write a single node for each JSON object:

```cypher
MATCH (n)
WHERE id(n) = idFrom('message', $that.from, $that.to, $that.timestamp)
SET n = $that, n:Message
```

Note that the ID of each node is generated deterministically based on some of the content (the email's from and to addresses, plus the timestamp) which will result in a unique node for each unique email. Since nodes are defined by their ID alone, duplicate JSON objects received will result in attempts to produce the same content at the same node. The system interprets this as a no-op, and no duplicate data will be produced as part of the ingest process or from the standing queries defined above.

With the [pre-processed JSON file linked above](https://thatdot-public.s3.us-west-2.amazonaws.com/enron.json.zip) extracted to the Quine host's filesystem, say, at `/tmp/enron.json`, the following JSON payload can be issued to the `POST` endpoint at [Create Ingest Stream: `POST /api/v2/graph/quine/ingests`](/reference/rest-api/?av=v2#/operations/create-ingest)

```json
{
  "name": "enron-sample",
  "source": {
    "type": "File",
    "path": "/tmp/enron.json"
  },
  "query": "MATCH (n) WHERE id(n) = idFrom('message', $that.from, $that.to, $that.timestamp) SET n = $that, n:Message"
}
```

Or called at the command line with `curl`:

```bash
curl -X 'POST' \
  'https://localhost:8080/api/v2/graph/quine/ingests' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "enron-sample",
  "source": {
    "type": "File",
    "path": "/tmp/enron.json"
  },
  "query": "MATCH (n) WHERE id(n) = idFrom('\''message'\'', $that.from, $that.to, $that.timestamp) SET n = $that, n:Message"
}'
```

Upon issuing this REST API call, a local file ingest stream will begin to consume the data in the specified file. standing queries #1 and #2 will be automatically applied to the incoming records, and the new data will be created as specified. As that new data is created, it will also trigger matches for standing query #3. At each step, data gets more connected and closer to the specific answers desired.

## Step #4: Observe Matched Results

Quine can take many kinds of action when each standing query returns a result. The standing queries shown here updated the shape of the graph. Making use of the results in real-time as it is interpreted with the additional data written in makes the system profoundly powerful. To make use of real-time results, a standing query can be configured to publish data back to Kinesis or other event stream systems. Data can be sent out of Quine into other services or system components providing endless possibilities. The ability to interpret data by drawing connections, defining new levels in the data, and drilling down for answers is powerful!


---
# Loading data
URL: https://quine.io/tutorials/loading-data/loading-data/

---
canonical_url: https://quine.io/tutorials/loading-data/loading-data/
description: Three ways to load data into Quine: inline Cypher queries, file-based queries, and streaming ingest.
---

# How to load data?

There are several different ways of writing data into the system, optimized for different use cases.

## Inline queries

For ad-hoc experimenting, it is usually enough to create individual nodes and edges directly with
Cypher queries.

```cypher
CREATE
  (lilysr: Person { name: "Lily Potter", gender: "female", birth_year: 1960 }),
  (jamessr:Person { name: "James Potter", gender: "male", birth_year: 1960 }),
  (molly:Person { name: "Molly Weasley", gender: "female", birth_year: 1949 }),
  (arthur:Person { name: "Arthur Weasley", gender: "male", birth_year: 1950 }),
  (harry:Person { name: "Harry Potter", gender: "male", birth_year: 1980 }),
  (ginny:Person { name: "Ginny Weasley", gender: "female", birth_year: 1981 }),
  (ron:Person { name: "Ron Weasley", gender: "male", birth_year: 1980 }),
  (hermione:Person { name: "Hermione Granger", gender: "female", birth_year: 1979 }),
  (jamesjr:Person { name: "James Sirius Potter", gender: "male", birth_year: 2003 }),
  (albus:Person { name: "Albus Severus Potter", gender: "male", birth_year: 2005 }),
  (lilyjr:Person { name: "Lily Luna", gender: "female",  birth_year: 2007 }),
  (rose:Person { name: "Rose Weasley", gender: "female",  birth_year: 2005 }),
  (hugo:Person { name: "Hugo Weasley", gender: "male",  birth_year: 2008 }),
  (jamessr)<-[:has_father]-(harry)-[:has_mother]->(lilysr),
  (arthur)<-[:has_father]-(ginny)-[:has_mother]->(molly),
  (arthur)<-[:has_father]-(ron)-[:has_mother]->(molly),
  (harry)<-[:has_father]-(jamesjr)-[:has_mother]->(ginny),
  (harry)<-[:has_father]-(albus)-[:has_mother]->(ginny),
  (harry)<-[:has_father]-(lilyjr)-[:has_mother]->(ginny),
  (ron)<-[:has_father]-(rose)-[:has_mother]->(hermione),
  (ron)<-[:has_father]-(hugo)-[:has_mother]->(hermione);
```

This sort of query can be entered in the [Exploration UI](../../getting-started/exploration-ui.md), through
`cypher-shell`, or directly via the REST API (see the "Cypher query language" section). Entering the
above graph in the Exploration UI and then querying `MATCH (n) RETURN n` produces the following
graph.

![harry potter graph](harry_potter_graph.svg)

## Queries that read from files

For larger static datasets, it isn't always feasible or convenient to be constructing large Cypher
queries. If these datasets are CSVs or line-based JSON files that are publicly available on the web,
it is is possible to write Cypher queries that will iterate through the records in the file,
executing some query action for each entry.

For instance, consider the same Harry Potter dataset [in a JSON file](https://recipes.quine.io/harry-potter-json). Using
the custom `loadJsonLines` procedure to load data from either a file or web URL, we can iterate over each record and create a node for
it along with edges to its children.

```cypher
CALL loadJsonLines("https://recipes.quine.io/harry-potter-json") YIELD value AS person
MATCH (p) WHERE id(p) = idFrom('name', person.name)
SET p = { name: person.name, gender: person.gender, birth_year: person.birth_year }
SET p: Person
WITH person.children AS childrenNames, p
UNWIND childrenNames AS childName
MATCH (c) WHERE id(c) = idFrom('name', childName)
CREATE (c)-[:has_parent]->(p)
```

If the data is in a CSV format, you can use the `LOAD CSV` clause.

!!! note

    `idFrom` is a function that hashes its arguments into a valid ID. It takes an arbitrary number of
    arguments, so that multiple bits of data can factor into the deterministic ID. By convention, the
    first of these arguments is a string describing a namespace for the IDs being generated. This is
    important to avoid accidentally producing collisions in IDs that exist in different namespaces:
    `idFrom('year', 2000)` is different from `idFrom('part number', 2000)`.

## Streaming Data Ingest

_Quine_ is engineered first and foremost as a stream processing system. Data ingest pipelines are
almost always streams, and batch processing is something done for want of streaming capabilities.
Batch processing is often used to work around other limitations of an ingest system (eg.
slow query times and inability to properly trigger computation on new data). These are problems which we
believe can be avoided entirely in _Quine_ through judicious use of Standing Queries.

_Quine_ supports defining ingest streams that connect to existing industry streaming systems such as
Kafka and Kinesis. Since it is expected that ingest streams will run for long-periods of time, the
REST API is designed to make it easy to

  * list or lookup the configuration of currently running streams as well as their ingest progress
  * create fresh new ingest streams
  * halt currently running ingest streams

See the "Ingest streams" section of the REST API for more details.