Files and Named Pipes¶

Files are a stream of data. It's easy to think of a file as a singular chunk, but on disk it is a linear sequence of bits. Reading data from a file is the process of starting at the beginning, and reading each bit in sequence, until you reach the end. Programs usually do this under the hood and deliver a single result when finished, but that data can be handled by a program before the end of file is reached. Reading a file until a particular sequence is found (the "delimiter") will yield a sequence of bytes which can be handled as a single event. This is how data is streamed into Quine from a file—making it a very natural and convenient way to load data into Quine.

Security Controls¶

Quine implements directory allow listing and file enumeration controls to protect against path traversal vulnerabilities. File ingests are restricted to specific directories configured at startup.

Configure file ingest security in your configuration file:

quine.file-ingest {
  # Allow list of directories
  allowed-directories = ["/path/to/data"]

  # File resolution mode
  resolution-mode = "static"  # or "dynamic"
}

The allowed-directories setting specifies an allow list of directories from which files can be ingested. Relative paths are resolved against the working directory at startup. An empty list means no file ingests are allowed except from recipes. By default, open source Quine allows ingestion from the working directory (["."]) to provide low friction for development and testing, ensuring existing recipes work without configuration changes. A list you configure replaces the default rather than adding to it. Quine reads only files directly inside a listed directory, not files in its subdirectories.

A symbolic link is checked by where it points

Quine resolves a symbolic link (symlink) before it applies the allow list, and checks the file the link points to. A link to a file directly inside an allowed directory is accepted. A link to a file anywhere else is rejected, so a link placed in an allowed directory cannot trick Quine into ingesting a sensitive file outside it, such as a password file.

The resolution-mode setting controls which files within allowed directories can be ingested. Set to "static" to only allow files that were present at startup (more secure), or "dynamic" to allow any file in allowed directories, even files added after startup (more flexible). Open source Quine defaults to "dynamic" for development flexibility.

File Ingest Format¶

Each file has one special feature which distinguishes it from a normal stream of data: it has an end. When ingesting data from one of these file types, the stream will continue until the end-of-file signal is reached. When the end-of-file signal is reached, the stream will finish and be marked by Quine as COMPLETED.

The format field in the ingest source tells Quine how to split the file into records and how to parse each one. For example, to read a file of newline-delimited JSON objects from the default allowed directory via Create Ingest Stream: POST /api/v2/graph/quine/ingests:

{
  "name": "events-from-file",
  "source": {
    "type": "File",
    "path": "events.jsonl",
    "format": {
      "type": "JsonL"
    }
  },
  "query": "MATCH (n) WHERE id(n) = idFrom($that.id) SET n = $that"
}

The available file formats are:

Format	`$that` is...
`JsonL`	each line, parsed as a JSON value
`Json`	each element of a top-level array, or each of a series of concatenated objects
`CSV`	each row, as a list of strings or a map of header to string
`Line`	each line, as a string
`Parquet`	each row, as a map typed by the file's embedded schema
`AvroContainer`	each record of an Avro object container file

JSON Lines¶

"format": {"type": "JsonL"}

JSON data is ubiquitous; it is a very natural source of a data for a streaming system. The typical approach is to save records as JSON objects to a file, separated by new-lines: \n. It is very common to call that format a .json file. That format is not technically valid JSON, and so sometimes it is referred to as "JSON Lines" and given the .jsonl file extension.

Quine reads these files one line at a time, passing each parsed JSON value to the Cypher ingest query as $that to be used in the rest of the ingest query.

JSON¶

"format": {"type": "Json"}

Use Json for a file containing a single top-level array of objects, or a series of concatenated objects with optional commas or whitespace between them. Each object is passed to the ingest query as $that. Files whose newline-delimited values are not objects (arrays, for example) should use JsonL instead.

CSV¶

"format": {"type": "CSV", "headers": true}

Comma-Separated Values are a convenient way to store tabular data. Quine reads CSV files natively, passing each parsed row into a Cypher query as $that. Fields from each row are accessible by numeric index or by key, depending on whether headers are provided. The headers field can be true to read them from the first line of the file, a list of strings to provide them manually, or false (the default) to disable them. delimiter, quoteChar, and escapeChar are also configurable. See the CSV format in the request body of Create Ingest Stream in the REST API reference for details.

Text¶

"format": {"type": "Line"}

Any text file can be used as a stream of input to Quine. The Line format reads in the specified file, passing each individual line as a string to the provided Cypher query, where it can be used directly as a string value, or parsed by hand into a more meaningful structure.

Parquet and Avro¶

"format": {"type": "Parquet"} and "format": {"type": "AvroContainer"}

Parquet emits each row of an Apache Parquet file as a map, with column types taken from the file's embedded schema. AvroContainer emits each record of an Apache Avro object container file (.avro) using the schema in the file header; an optional schemaUrl supplies a reader schema for schema evolution. Parquet requires random access to the file's footer, so it is supported for file and S3 ingests only. AvroContainer is read as a stream, so it also works with standard input and named pipes.

Named Pipe¶

A named pipe is an operating-system/file-system abstraction typically used for inter-process communication. A named pipe has a file-like representation in the filesystem—so it looks like a file to most programs—but it does not persist data. The operating system uses this object that looks like a file as a communication channel. One program can write to a named pipe, and if another program is actively reading from the same named pipe, then the second program receives the data written by the first program.

Quine supports named pipes on Unix-like operating systems (MacOS, Linux, etc), but not on Windows. On most Unix-like operating systems, Quine will automatically detect whether the provided path refers to a regular file or a named pipe. Automatic detection can be overridden by setting fileIngestMode in the ingest source to REGULAR or NAMED_PIPE.

When Quine has an active stream ingesting from a named pipe, it will consume data from that pipe until the stream is cancelled, or it encounters an error. The format of the incoming data from the named pipe can be any of the File Ingest Formats listed above except Parquet, which needs random access to the file. Data written to the named pipe will be separated by new-line characters (\n) and passed to the provided Cypher query exactly as described above.