Upgrading to 2.2.0

This page covers what you need to know when upgrading to Quine 2.2.0.

Back up your data store before upgrading

2.2.0 upgrades the persistence format in place, and stamps the store with the new format version the first time it starts. A 2.1.1 process will not start against a store carrying that stamp, so restoring a backup is the route back to 2.1.1. See The upgrade is one-way.

What's New

• Standing query results are independent of node sleep — DistinctId standing queries report the same results whatever the sleep and wake pattern of the nodes involved. See Behavior Changes.
• Tunable snapshot frequency — the new quine.persistence.snapshot-after-events setting lets a journaling deployment snapshot a node only once it has journaled enough to be worth it, cutting the write volume reaching your data store. See New configuration.
• Hot node metrics — new metrics rank individual nodes by mailbox backlog, message rate, and estimated wait. When a low-cardinality node on the critical path of a high-volume ingest is holding up the stream, these metrics name it by ID so you can rework the ingest query. See Metrics.
• Type-distinct result values in the UI — the results card renders values as syntax-highlighted JSON, so the number 42 and the string "42" are distinguishable at a glance, as are a null value and the word "null".
• Higher outbound HTTP connection limits — webhook and HTTP destinations reach the throughput their parallelism setting asks for.
• Streaming journal reads — a node's journal is now read as a stream rather than collected first, so walking a long history no longer costs memory proportional to its length.
• Nodes keep processing under clock skew — a node processes its messages whatever the offset between its own clock and its peers'. See max-catch-up-sleep is removed.

Upgrade procedure

1. Stop Quine.
2. Back up the data store. This is your route back to 2.1.1.
3. Remove max-catch-up-sleep from your configuration if it is set. 2.2.0 does not recognize the key and refuses to start while it is present.
4. Leave snapshot-singleton as it is. If you want to change it, do that in a separate restart once the upgrade has settled. See Change snapshot-singleton in its own step.
5. Start 2.2.0. The store's format version is stamped at startup, before any node wakes.

There is no bulk migration step and no deadline by which the store has to be converted. A node reads its 2.1.1 records when it wakes and writes the new format the next time it sleeps with something to save. A node that never wakes keeps its 2.1.1 records and reads them correctly every time.

Breaking Changes

The upgrade is one-way

Quine stamps its data store with the persistence format version it writes, and checks that stamp at startup before any node wakes. 2.2.0 raises the stamp:

• A 2.2.0 process accepts a store written by 2.1.1 and stamps it with the new version.
• A 2.1.1 process started against a store carrying that stamp stops immediately, reporting an incompatible version.

The startup check is what makes the upgrade safe to perform: an older build is stopped at the door rather than allowed to read a format it does not understand. It also means the upgrade is one-way. Take a backup before you start 2.2.0, because restoring it is the route back to 2.1.1.

Under the default time-keyed snapshots your 2.1.1 snapshot rows remain in the store, though the version stamp still prevents 2.1.1 from starting. Under snapshot-singleton = true, each node overwrites its single snapshot row the first time it sleeps, so the backup is the only copy of the 2.1.1 bytes.

What the upgrade does not change

No table, column family, keyspace, or column is added, renamed, or dropped by any persistor. Cassandra, Keyspaces, MapDB, RocksDB, and in-memory all keep the schema they had, so there is no DDL to run before upgrading, unlike the move to 2.1.1 which added two Cassandra tables.

max-catch-up-sleep is removed

Nodes now process their messages whatever the offset between their own clock and their peers', so no setting is needed to govern how long a node waits for a clock to catch up. quine.max-catch-up-sleep is removed.

2.2.0 does not recognize the key, and stops at startup with a configuration error naming it, as it does for any other unrecognized setting. Remove it from your configuration before upgrading.

Keep your clocks synchronized. Timestamps come from the system clock, so a clock that jumps is reflected in the history Quine records.

subscribers and subscriptions change their columns

These two procedures now report which standing queries each subscription is held for, and their columns change. Update any saved query that reads them.

Procedure	2.1.1 columns	2.2.0 columns
subscribers(node)	queryId, queryDepth, receiverId, lastResult	dgnId, subscriber, forQueries, lastResult
subscriptions(node)	queryId, queryDepth, receiverId, lastResult	dgnId, peer, forQueries, answer

New configuration: snapshot-after-events

One persistence key is added, along with a quine.metrics.hot-nodes block for the new metrics (see Configuration), and none is renamed. A configuration written for 2.1.1 is accepted unchanged unless it sets max-catch-up-sleep.

quine.persistence {
  # only snapshot a sleeping node once it has journaled at least this many
  # events since its last snapshot. 0, the default, snapshots on every sleep
  snapshot-after-events = 0
}

With journaling enabled, a snapshot bounds how much journal has to be replayed when a node next wakes. A node that journaled a handful of events replays quickly with or without one, so on a deployment with many lightly-touched nodes, snapshotting every one of them at every sleep is most of the write volume reaching your data store. Raising snapshot-after-events spends a little more replay time at wake to buy that write volume back.

The default is inert: at snapshot-after-events = 0, 2.2.0 snapshots on every sleep exactly as 2.1.1 did.

A non-zero value applies to sleep-time snapshots on a journaling deployment, and is accepted in the combinations where it decides something:

snapshot-after-events	journal-enabled	snapshot-singleton	snapshot-schedule	Outcome
0 (default)	any	any	any	Accepted — every 2.1.1 configuration lands here
negative	any	any	any	Refused
positive	false	any	any	Refused: without a journal, the snapshot is the durable record
positive	true	true	any	Refused: the single snapshot row is meant to hold the latest state
positive	true	false	not on-node-sleep	Refused: it decides only sleep-time snapshots
positive	true	false	on-node-sleep	Accepted

Each refusal names its reason at startup. See Persistors for the surrounding settings.

Behavior Changes

DistinctId standing query results are independent of node sleep

A DistinctId standing query now reports the same results whatever the sleep and wake pattern of the nodes involved. A node rebuilt from disk comes back holding what it held while awake, because replay records history rather than re-deciding it, and because each subscription and each cached answer now records which standing queries it is held for.

Two things follow for an upgrade:

• A match already delivered is not delivered again. What a node last reported carries forward from its 2.1.1 snapshot, or is re-derived when replaying a 2.1.1 journal.
• Your 2.1.1 state is read where it stands. Nothing is rewritten in bulk. Where a 2.1.1 record does not carry something 2.2.0 tracks, the value is derived from what the record does carry, erring toward keeping a subscription a little longer than needed. Each node narrows that to the exact set as it wakes.

Standing query state is saved on sleep independently of snapshots

MultipleValues standing query state is now written when a node sleeps, whether or not that sleep also writes a snapshot. Deployments running snapshot-schedule = never or on-node-update therefore keep this state across a node's sleep and wake, and no configuration change is needed to get it.

State written before journaling was turned off is read back

A node reads its journal at every wake. If journaling was enabled at any point in a deployment's life, the state written then comes back when the node next wakes, even if journal-enabled is false now.

For a deployment that has always run with journal-enabled = false, this is two additional data store reads per node wake, against the journals and domain_index_events tables, each returning nothing. On Cassandra that is two extra round trips per wake. If you run journaling off with a high node wake rate, measure the rate before and after the upgrade.

Change snapshot-singleton in its own step

Wake-time snapshot key resolution does nothing when the stored key shape already matches the setting, which is the case for any upgrade that leaves snapshot-singleton alone. Changing the setting moves snapshot rows to the other key shape: the bytes move verbatim rather than being re-encoded, and each write completes before its delete, so an interruption mid-move leaves two rows rather than none.

Change one thing at a time. Upgrade to 2.2.0 first, let it settle, then change snapshot-singleton in a later restart.

Server-Sent Events ingests run alongside other HTTP traffic

An SSE ingest holds its connection for the life of the stream. SSE ingests now use a connection path of their own, outside the shared outbound HTTP pool, so a long-lived SSE stream and the rest of a deployment's HTTP traffic to the same host no longer compete for the same connections.