Skip to content

APT Detection

Full Recipe

Shared by: Ryan Wright

This APT (Advanced Persistent Threat) detection recipe ingests EDR (Endpoint Detection and Response) and network traffic logs, while monitoring for an IoB (Indicator of Behavior) that matches malicious data exfiltration patterns.

APT Recipe 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
title: APT Detection
summary: Endpoint logs and network traffic data merge to auto-detect exfiltration
contributor: https://github.com/rrwright
version: 1
description: |-
  This APT (Advanced Persistent Threat) detection recipe ingests EDR (Endpoint 
  Detection and Response) and network traffic logs, while monitoring for an IoB 
  (Indicator of Behavior) that matches malicious data exfiltration patterns.

  SCENARIO:
  Using a standing query, the recipe monitors for covert interprocess
  communication using a file to pass data. When that pattern is matched, with a
  network SEND event, we have our smoking gun and a URL is logged linking to
  the Quine Exploration UI with the full activity and context for investigation.

  In this scenario, a malicious Excel macro collects personal data and stores 
  it in a temporary file. The APT process "ntclean" infiltrated the system 
  previously through an SSH exploit, and now reads from that temporary file 
  and exfiltrates data from the network--hiding it as an HTTP GET request--
  before deleting the temporary file to cover its tracks. 

  The source of the SSH exploit that planted the APT and the destination 
  for exfiltrated data utilize the same IP address.

  SAMPLE DATA:
    endpoint.json - https://recipes.quine.io/apt-detection/endpoint-json
     network.json - https://recipes.quine.io/apt-detection/network-json

  Download the sample data to the same directory where Quine will be run.

  RESULTS:
  When the standing query detects the WRITE->READ->SEND->DELETE pattern, it 
  will output a link to the console that can be copied and pasted into a 
  browser to explore the event in the Quine Exploration UI.

ingestStreams:
  - type: FileIngest
    path: endpoint.json
    format:
      type: CypherJson
      query: >-
        MATCH (proc), (event), (object)
        WHERE id(proc) = idFrom($that.pid)
          AND id(event) = idFrom($that)
          AND id(object) = idFrom($that.object)

        SET proc.id = $that.pid,
            proc: Process,
            event.type = $that.event_type,
            event: EndpointEvent,
            event.time = $that.time,
            object.data = $that.object

        CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)

  - type: FileIngest
    path: network.json
    format:
      type: CypherJson
      query: >-
        MATCH (src), (dst), (event)
        WHERE id(src) = idFrom($that.src_ip+":"+$that.src_port)
          AND id(dst) = idFrom($that.dst_ip+":"+$that.dst_port)
          AND id(event) = idFrom('network_event', $that)

        SET src.ip = $that.src_ip+":"+$that.src_port,
            src: IP,
            dst.ip = $that.dst_ip+":"+$that.dst_port,
            dst: IP,
            event.proto = $that.proto,
            event.time = $that.time,
            event.detail = $that.detail,
            event: NetTraffic

        CREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)

standingQueries:
  - pattern:
      type: Cypher
      query: >-
        MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), 
              (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)
        WHERE e1.type = "WRITE"
          AND e2.type = "READ"
          AND e3.type = "DELETE"
          AND e4.type = "SEND"
        RETURN DISTINCT id(f) as fileId
    outputs:
      stolen-data:
        type: CypherQuery
        query: >-
          MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), 
                (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
          WHERE id(f) = $that.data.fileId
            AND e1.type = "WRITE"
            AND e2.type = "READ"
            AND e3.type = "DELETE"
            AND e4.type = "SEND"
            AND e1.time < e2.time
            AND e2.time < e3.time
            AND e2.time < e4.time

          CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

          WITH e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
          RETURN URL
        andThen:
          type: PrintToStandardOut

nodeAppearances: 
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    icon: ion-load-a
    label:
      type: Property
      key: id
      prefix: "Process: "
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: IP
    icon: ion-ios-world
    label:
      type: Property
      key: ip
      prefix: ""
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: EndpointEvent
    icon: ion-android-checkmark-circle
    label:
      type: Property
      key: type
      prefix: ""
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: NetTraffic
    icon: ion-network
    label:
      type: Property
      key: proto
      prefix: ""
  - predicate:
      propertyKeys: []
      knownValues: {}
    icon: ion-ios-copy
    label:
      type: Property
      key: data
      prefix: ""

quickQueries: 
  - predicate:
      propertyKeys: []
      knownValues: {}
    quickQuery:
      name: Adjacent Nodes
      querySuffix: MATCH (n)--(m) RETURN DISTINCT m
      queryLanguage: Cypher
      sort: Node
  - predicate:
      propertyKeys: []
      knownValues: {}
    quickQuery:
      name: Refresh
      querySuffix: RETURN n
      queryLanguage: Cypher
      sort: Node
  - predicate:
      propertyKeys: []
      knownValues: {}
    quickQuery:
      name: Local Properties
      querySuffix: RETURN id(n), properties(n)
      queryLanguage: Cypher
      sort: Text
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Files Read
      querySuffix: MATCH (n)-[:EVENT]->(e)-[:EVENT]->(f) WHERE e.type = "READ" RETURN f
      queryLanguage: Cypher
      sort: Node
      edgeLabel: read
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Files Written
      querySuffix: MATCH (n)-[:EVENT]->(e)-[:EVENT]->(f) WHERE e.type = "WRITE" RETURN f
      queryLanguage: Cypher
      sort: Node
      edgeLabel: wrote
  - predicate:
      propertyKeys: 
        - data
      knownValues: {}
    quickQuery:
      name: Read By
      querySuffix: MATCH (n)<-[:EVENT]-(e)<-[:EVENT]-(p) WHERE e.type = "READ" RETURN p
      queryLanguage: Cypher
      sort: Node
      edgeLabel: written by
  - predicate:
      propertyKeys: 
        - data
      knownValues: {}
    quickQuery:
      name: Written By
      querySuffix: MATCH (n)<-[:EVENT]-(e)<-[:EVENT]-(p) WHERE e.type = "WRITE" RETURN p
      queryLanguage: Cypher
      sort: Node
      edgeLabel: written by
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Received Data
      querySuffix: MATCH (n)-[:EVENT]->(e)-[:EVENT]->(i) WHERE e.type = "RECEIVE" RETURN i
      queryLanguage: Cypher
      sort: Node
      edgeLabel: received
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Sent Data
      querySuffix: MATCH (n)-[:EVENT]->(e)-[:EVENT]->(i) WHERE e.type = "SEND" RETURN i
      queryLanguage: Cypher
      sort: Node
      edgeLabel: sent
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Started By
      querySuffix: MATCH (n)<-[:EVENT]-(e)<-[:EVENT]-(p) WHERE e.type = "SPAWN" RETURN p
      queryLanguage: Cypher
      sort: Node
      edgeLabel: parent process
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: Process
    quickQuery:
      name: Started Other Process
      querySuffix: MATCH (n)-[:EVENT]->(e)-[:EVENT]->(p) WHERE e.type = "SPAWN" RETURN p
      queryLanguage: Cypher
      sort: Node
      edgeLabel: child process
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: IP
    quickQuery:
      name: Network Send
      querySuffix: MATCH (n)-[:NET_TRAFFIC]->(net) RETURN net
      queryLanguage: Cypher
      sort: Node
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: IP
    quickQuery:
      name: Network Receive
      querySuffix: MATCH (n)<-[:NET_TRAFFIC]-(net) RETURN net
      queryLanguage: Cypher
      sort: Node
  - predicate:
      propertyKeys: []
      knownValues: {}
      dbLabel: IP
    quickQuery:
      name: Network Communication
      querySuffix: MATCH (n)-[:NET_TRAFFIC]-(net)-[:NET_TRAFFIC]-(ip) RETURN ip
      queryLanguage: Cypher
      sort: Node
      edgeLabel: Communication

sampleQueries: []

Download Recipe

Scenario

In this scenario, a malicious Excel macro collects personal data and stores it in a temporary file. The APT process ntclean infiltrated the system previously through an SSH exploit, and now reads from that temporary file and exfiltrates data from the network hiding it as an HTTP GET request before deleting the temporary file to cover its tracks.

Using a standing query, the recipe monitors for covert interprocess communication using a file to pass data. When that pattern is matched, with a network SEND event, we have our smoking gun and a URL is logged linking to the Quine Exploration UI with the full activity and context for investigation.

The source of the SSH exploit that planted the APT and the destination for exfiltrated data utilize the same IP address.

Sample Data

Download the sample data to the same directory where Quine will be run.

How it Works

The recipe reads observations from the two sample data files using ingest streams to manifest a graph in Quine. A separate ingest stream is configured to process each file, each containing Cypher that parses the observations, manifests nodes, and relates them to each other in the graph.

INGEST-1 processes the endpoints.json file:

  - type: FileIngest
    path: endpoint.json
    format:
    type: CypherJson
    query: >-
        MATCH (proc), (event), (object)
        WHERE id(proc) = idFrom($that.pid)
          AND id(event) = idFrom($that)
          AND id(object) = idFrom($that.object)

        SET proc.id = $that.pid,
            proc: Process,
            event.type = $that.event_type,
            event: EndpointEvent,
            event.time = $that.time,
            object.data = $that.object

        CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)
POST /api/v1/ingest/INGEST-1
{
  "type": "FileIngest",
  "path": "endpoint.json",
  "format": {
    "type": "CypherJson",
    "query": "MATCH (proc), (event), (object) WHERE id(proc) = idFrom($that.pid) AND id(event) = idFrom($that) AND id(object) = idFrom($that.object) SET proc.id = $that.pid, proc: Process, event.type = $that.event_type, event: EndpointEvent, event.time = $that.time, object.data = $that.object CREATE (proc)-[:EVENT]->(event)-[:EVENT]->(object)"
  }
}

INGEST-2 processes the network.json file:

- type: FileIngest
    path: network.json
    format:
    type: CypherJson
    query: >-
        MATCH (src), (dst), (event)
        WHERE id(src) = idFrom($that.src_ip+":"+$that.src_port)
          AND id(dst) = idFrom($that.dst_ip+":"+$that.dst_port)
          AND id(event) = idFrom('network_event', $that)

        SET src.ip = $that.src_ip+":"+$that.src_port,
            src: IP,
            dst.ip = $that.dst_ip+":"+$that.dst_port,
            dst: IP,
            event.proto = $that.proto,
            event.time = $that.time,
            event.detail = $that.detail,
            event: NetTraffic

        CREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)
POST /api/v1/ingest/INGEST-2
[
    {
        "type": "FileIngest",
        "path": "network.json",
        "format": {
        "type": "CypherJson",
        "query": "MATCH (src), (dst), (event) WHERE id(src) = idFrom($that.src_ip+\":\"+$that.src_port)\n  AND id(dst) = idFrom($that.dst_ip+\":\"+$that.dst_port)\n  AND id(event) = idFrom('network_event', $that)\n\nSET src.ip = $that.src_ip+\":\"+$that.src_port,\n    src: IP,\n    dst.ip = $that.dst_ip+\":\"+$that.dst_port,\n    dst: IP,\n    event.proto = $that.proto,\n    event.time = $that.time,\n    event.detail = $that.detail,\n    event: NetTraffic\n\nCREATE (src)-[:NET_TRAFFIC]->(event)-[:NET_TRAFFIC]->(dst)"
        }
    }
]

A standing query is configured to detect a WRITE->READ->SEND->DELETE pattern that is typical for this type of exflitration event.

- pattern:
    type: Cypher
    query: >-
        MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), 
              (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)
        WHERE e1.type = "WRITE"
          AND e2.type = "READ"
          AND e3.type = "DELETE"
          AND e4.type = "SEND"
        RETURN DISTINCT id(f) as fileId
/api/v1/query/standing/STANDING-1
[
    {
        "pattern": {
        "type": "Cypher",
        "query": "MATCH (e1)-[:EVENT]->(f)<-[:EVENT]-(e2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)\nWHERE e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\nRETURN DISTINCT id(f) as fileId"
        },
        "outputs": {
        "stolen-data": {
            "type": "CypherQuery",
            "query": "MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), \n      (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)\nWHERE id(f) = $that.data.fileId\n  AND e1.type = \"WRITE\"\n  AND e2.type = \"READ\"\n  AND e3.type = \"DELETE\"\n  AND e4.type = \"SEND\"\n  AND e1.time < e2.time\n  AND e2.time < e3.time\n  AND e2.time < e4.time\n\nCREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)\nWITH e1, e2, e3, e4, p1, p2, f, ip, \"http://localhost:8080/#MATCH\" + text.urlencode(\" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='\"+strId(p1)+\"' AND id(e1)='\"+strId(e1)+\"' AND id(f)='\"+strId(f)+\"' AND id(e2)='\"+strId(e2)+\"' AND id(p2)='\"+strId(p2)+\"' AND id(e3)='\"+strId(e3)+\"' AND id(e4)='\"+strId(e4)+\"' AND id(ip)='\"+strId(ip)+\"' RETURN e1, e2, e3, e4, p1, p2, f, ip\") as URL RETURN URL",
            "andThen": {
            "type": "PrintToStandardOut"
            }
        }
        }
    }
]

Once Quine detects the pattern, the event is sent to a standing query output for additional processing and action.

    outputs:
      stolen-data:
        type: CypherQuery
        query: >-
          MATCH (p1)-[:EVENT]->(e1)-[:EVENT]->(f)<-[:EVENT]-(e2)<-[:EVENT]-(p2), 
                (f)<-[:EVENT]-(e3)<-[:EVENT]-(p2)-[:EVENT]->(e4)-[:EVENT]->(ip)
          WHERE id(f) = $that.data.fileId
            AND e1.type = "WRITE"
            AND e2.type = "READ"
            AND e3.type = "DELETE"
            AND e4.type = "SEND"
            AND e1.time < e2.time
            AND e2.time < e3.time
            AND e2.time < e4.time

          CREATE (e1)-[:NEXT]->(e2)-[:NEXT]->(e4)-[:NEXT]->(e3)

          WITH e1, e2, e3, e4, p1, p2, f, ip, "http://localhost:8080/#MATCH" + text.urlencode(" (e1),(e2),(e3),(e4),(p1),(p2),(f),(ip) WHERE id(p1)='"+strId(p1)+"' AND id(e1)='"+strId(e1)+"' AND id(f)='"+strId(f)+"' AND id(e2)='"+strId(e2)+"' AND id(p2)='"+strId(p2)+"' AND id(e3)='"+strId(e3)+"' AND id(e4)='"+strId(e4)+"' AND id(ip)='"+strId(ip)+"' RETURN e1, e2, e3, e4, p1, p2, f, ip") as URL
          RETURN URL
        andThen:
          type: PrintToStandardOut

The result once the pattern is detected is to output a link to the console that an analyst can use to review the event further within Quine's Exploration UI.

1
2022-12-15 11:28:52,413 Standing query `stolen-data` match: {"meta":{"isPositiveMatch":true,"resultId":"5bd8beb7-78cc-de3a-bf69-8ad20b90cd11"},"data":{"URL":"http://localhost:8080/#MATCH%20%28e1%29%2C%28e2%29%2C%28e3%29%2C%28e4%29%2C%28p1%29%2C%28p2%29%2C%28f%29%2C%28ip%29%20WHERE%20id%28p1%29%3D%271ca87b55-a62a-3f13-bc2d-5752ca5f4143%27%20AND%20id%28e1%29%3D%2743f26672-0c93-3e2c-84bb-88adb84454ba%27%20AND%20id%28f%29%3D%27f00ae947-3dd5-3c92-a84f-118b401c80f1%27%20AND%20id%28e2%29%3D%275a6bb84a-ed7e-3982-83d8-ea9f7b0dee9d%27%20AND%20id%28p2%29%3D%2717d3fb9a-cd9b-39ba-a087-e2f577627873%27%20AND%20id%28e3%29%3D%279fc76f93-c93d-3eb3-8730-0bca7ee079c4%27%20AND%20id%28e4%29%3D%274cc1ccc4-6286-3a32-b691-d308ba1e68e7%27%20AND%20id%28ip%29%3D%27a1ce3be6-4501-3af2-b05b-2c6ac0936cc4%27%20RETURN%20e1%2C%20e2%2C%20e3%2C%20e4%2C%20p1%2C%20p2%2C%20f%2C%20ip"}}

Running the Recipe

❯ java -jar quine-1.8.2.jar -r apt-detection.yaml
Graph is ready
Running Recipe: APT Detection
Using 5 node appearances
Using 14 quick queries
Running Standing Query STANDING-1
Running Ingest Stream INGEST-1
Running Ingest Stream INGEST-2
Quine web server available at http://localhost:8080

Summary

When the standing query detects the WRITE->READ->SEND->DELETE pattern, it will output a link to the console that can be copied and pasted into a browser to explore the event in the Quine Exploration UI. Copy and paste the URL section of the match JSON from your console into your browser.

The nodes will be jumbled together when you first open the graph. Arrange the nodes to look similar to the image below before you start exploring.

apt-detection-graph

The recipe includes a number of quick queries to assist in exploring the event data. Right click on a node to bring up the quick query interface.

Tip

Quick Queries are available by right clicking on a node.

Quick Query Node Type Description
Adjacent Nodes All Display the nodes that are adjacent to this node.
Refresh All Refresh the content stored in a node
Local Properties All Display the properties stored by the node
Files Read Process Load nodes representing the files read by this process
Files Written Process Load nodes representing files written by this process
Read By data Load nodes that read data from this node
Written By data Load nodes that wrote to this node
Received Data Process Where did this process receive data from
Sent Data Process Where did this process send data to
Started By Process What started this process
Started Other Process Process What process did this process start
Network Send IP Where did this IP node send data
Network Receive IP Whre did this IP node receive data from
Network Communication IP What other nodes did this IP node communicate with

Use the quick queries to explore the graph and uncover the timeline behind the entire event.