Skip to content

Certstream Firehose

Full Recipe

Shared by: Ethan Bell

Reproduces the behavior of the certstream website by connecting to the certstream firehose via SSL-encrypted websocket and printing to standard out each time a new certificate is detected.

Certstream Firehose Recipe 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
version: 1
title: Certstream Firehose
contributor: https://github.com/emanb29
summary: Log new SSL certificate registrations
description: |-
  Reproduces the behavior of the certstream website (https://certstream.calidog.io/) by
  connecting to the certstream firehose via SSL-encrypted websocket and printing to
  standard out each time a new certificate is detected
ingestStreams:
  - type: WebsocketSimpleStartupIngest
    url: wss://certstream.calidog.io/
    format:
      type: CypherJson
      query: |-
        CREATE ($that)
standingQueries:
  - pattern:
      type: Cypher
      query: MATCH (n) RETURN DISTINCT id(n) AS id
    outputs:
      log-new-certs:
        type: CypherQuery
        query: |-
          MATCH (n)
          WHERE id(n) = $that.data.id
          RETURN n.data
        andThen:
          type: PrintToStandardOut
          logMode: FastSampling
nodeAppearances: []
quickQueries: []
sampleQueries: []

Download Recipe

Scenario

CertStream is an intelligence feed that gives you real-time updates from the Certificate Transparency Log network, allowing you to use it as a building block to make tools that react to new certificates being issued in real time.

This recipe connects to the curated public Certstream aggregation service managed by the team at Cali Dog Security.

Sample Data

This recipe connects to the live Certstream feed eliminating the need for sample data. However, below is a typical raw certificate update object for review.

  "data": {
    "cert_index": 160270422,
    "cert_link": "https://nessie2023.ct.digicert.com/log/ct/v1/get-entries?start=160270422&end=160270422",
    "leaf_cert": {
      "all_domains": [
        "*.nyarkowiz.online",
        "nyarkowiz.online"
      ],
      "extensions": {
        "authorityInfoAccess": "CA Issuers - URI:http://pki.goog/repo/certs/gts1p5.der\nOCSP - URI:http://ocsp.pki.goog/s/gts1p5/fKV4K079ZKo\n",
        "authorityKeyIdentifier": "keyid:D5:FC:9E:0D:DF:1E:CA:DD:08:97:97:6E:2B:C5:5F:C5:2B:F5:EC:B8\n",
        "basicConstraints": "CA:FALSE",
        "certificatePolicies": "Policy: 1.3.6.1.4.1.11129.2.5.3\nPolicy: 2.23.140.1.2.1",
        "crlDistributionPoints": "Full Name:\n URI:http://crls.pki.goog/gts1p5/oE9rr3G5TqE.crl",
        "ctlPoisonByte": true,
        "extendedKeyUsage": "TLS Web server authentication",
        "keyUsage": "Digital Signature, Key Encipherment",
        "subjectAltName": "DNS:nyarkowiz.online, DNS:*.nyarkowiz.online",
        "subjectKeyIdentifier": "42:22:E3:A5:27:CB:93:B1:8F:C0:20:7C:CB:E6:11:ED:B3:A4:CB:BD"
      },
      "fingerprint": "B1:FE:F6:4C:D1:7E:A3:DB:A8:D9:92:EE:18:42:B7:1F:35:2F:75:68",
      "issuer": {
        "C": "US",
        "CN": "GTS CA 1P5",
        "L": null,
        "O": "Google Trust Services LLC",
        "OU": null,
        "ST": null,
        "aggregated": "/C=US/CN=GTS CA 1P5/O=Google Trust Services LLC",
        "emailAddress": null
      },
      "not_after": 1679365412,
      "not_before": 1671589413,
      "serial_number": "201DF51E883B4B37139BBB17CAEACE15",
      "signature_algorithm": "sha256, rsa",
      "subject": {
        "C": null,
        "CN": "*.nyarkowiz.online",
        "L": null,
        "O": null,
        "OU": null,
        "ST": null,
        "aggregated": "/CN=*.nyarkowiz.online",
        "emailAddress": null
      }
    },
    "seen": 1671638236.908937,
    "source": {
      "name": "DigiCert Nessie2023 Log",
      "url": "https://nessie2023.ct.digicert.com/log/"
    },
    "update_type": "PrecertLogEntry"
  },
  "message_type": "certificate_update"
}

How it Works

The recipe is designed to rapidly load JSON objects into Quine producing as disconnected nodes. The ingest stream connects using a WebsocketSimpleStartupIngest stream type and parsing $that using by using a CypherJson format.

INGEST-1 reads directly from the certstream web socket:

    - type: WebsocketSimpleStartupIngest
      url: wss://certstream.calidog.io/
      format:
        type: CypherJson
        query: |-
          CREATE ($that)
POST /api/v1/ingest/INGEST-1
[
  {
    "type": "WebsocketSimpleStartupIngest",
    "url": "wss://certstream.calidog.io/",
    "format": {
      "type": "CypherJson",
      "query": "CREATE ($that)"
    }
  }
]

A standing query is configured to detect new nodes in the graph and then print the event to the console.

  - pattern:
      type: Cypher
      query: MATCH (n) RETURN DISTINCT id(n) AS id
    outputs:
      log-new-certs:
        type: CypherQuery
        query: |-
          MATCH (n)
          WHERE id(n) = $that.data.id
          RETURN n.data
        andThen:
          type: PrintToStandardOut
          logMode: FastSampling

```yaml title="/api/v1/query/standing/STANDING-1" [ { "pattern": { "type": "Cypher", "query": "MATCH (n) RETURN DISTINCT id(n) AS id" }, "outputs": { "log-new-certs": { "type": "CypherQuery", "query": "MATCH (n)\nWHERE id(n) = $that.data.id\nRETURN n.data", "andThen": { "type": "PrintToStandardOut", "logMode": "FastSampling" } } } } ]

```

The recipe will stream events to the console similar to the sample event below.

1
2022-12-21 10:32:54,863 Standing query `log-new-certs` match: {"meta":{"isPositiveMatch":true,"resultId":"e8709166-1a08-df62-3419-e030ce81d09a"},"data":{"n.data":{"cert_index":545387302,"cert_link":"https://ct.googleapis.com/logs/xenon2023/ct/v1/get-entries?start=545387302&end=545387302","leaf_cert":{"all_domains":["www.gamificationbook.com"],"extensions":{"authorityInfoAccess":"CA Issuers - URI:http://r3.i.lencr.org/\nOCSP - URI:http://r3.o.lencr.org\n","authorityKeyIdentifier":"keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6\n","basicConstraints":"CA:FALSE","certificatePolicies":"Policy: 1.3.6.1.4.1.44947.1.1.1\n  CPS: http://cps.letsencrypt.org","ctlPoisonByte":true,"extendedKeyUsage":"TLS Web server authentication, TLS Web client authentication","keyUsage":"Digital Signature, Key Encipherment","subjectAltName":"DNS:www.gamificationbook.com","subjectKeyIdentifier":"89:C8:C3:71:33:36:E7:37:BF:78:08:81:2F:4E:C7:74:DE:EF:9F:60"},"fingerprint":"D7:86:C7:29:54:AF:91:DF:DA:30:9D:8A:0A:AE:B0:17:1C:B6:F1:F8","issuer":{"C":"US","CN":"R3","L":null,"O":"Let's Encrypt","OU":null,"ST":null,"aggregated":"/C=US/CN=R3/O=Let's Encrypt","emailAddress":null},"not_after":1679412699,"not_before":1671636700,"serial_number":"3FFE0C620988AD6607BBFAB008D769E8BBF","signature_algorithm":"sha256, rsa","subject":{"C":null,"CN":"www.gamificationbook.com","L":null,"O":null,"OU":null,"ST":null,"aggregated":"/CN=www.gamificationbook.com","emailAddress":null}},"seen":1.671640374636495E9,"source":{"name":"Google 'Xenon2023' log","url":"https://ct.googleapis.com/logs/xenon2023/"},"update_type":"PrecertLogEntry"}}}

Running the Recipe

❯ java -jar quine-1.7.3.jar -r certstream-firehose.yaml
Graph is ready
Running Recipe: Certstream Firehose
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Quine web server available at http://localhost:8080