</th>	Trust model</th>	Mistakes surface at</th>	Failure mode</th></tr></thead>
Windows</strong></td>	Ceremony — sign, register, enter</td>	Runtime</strong>, on the customer’s machine</td>	Loud and catastrophic (bugcheck)</td></tr>
Linux</strong></td>	Proof — submit to the verifier</td>	Load time</strong>, on your machine</td>	Loud and safe (rejection with a trace)</td></tr>
macOS</strong></td>	Brokerage — Apple looks for you</td>	Deployment</strong>, at organizational scale</td>	Silent</em> — an event class that simply never arrives</td></tr> </tbody></table> Windows fails at the worst time in the worst way, but tells you unambiguously. Linux fails at the best possible time, before anything is at risk, and the price is that you argue with a proof checker for an afternoon. macOS fails at the strangest time — long after the code is written and tested — and the failure is an absence, which is the hardest thing in the world to notice.</p> Every one of these is a coherent engineering position. But they aren’t interchangeable, and this is why an EDR agent cannot have one unified sensor abstraction that “handles platform differences.” The differences aren’t in the API surface. They’re in what the platform believes about you, and belief doesn’t refactor.</p> What can</em> be unified is everything downstream. Once an event exists — a process started, a file was written, a task port was acquired — the questions become identical everywhere: how do you not lose it, how do you name the thing it happened to, who is allowed to act on it. Three sensors, three trust models, one contract from there on.</p> The next essay follows that contract downstream and asks why telemetry pipelines keep converging on the same shape.</p> Building a Mini-EDR Agent - Monitoring Process Creation with ETW in C 2026-03-26T00:00:00+00:00 In my previous post, we built a mini-SIEM lab using Sysmon to gather high-quality telemetry from a Windows host. But how can an endpoint sensor observe process creation in real time without relying on Sysmon?</p> One useful source is Event Tracing for Windows (ETW)</strong></a>, Windows’ general-purpose tracing infrastructure. Production EDR products normally combine several user-mode and kernel-mode telemetry sources; ETW is one source rather than the mechanism behind every product.</p> In this post, we will ditch the pre-built collection tools and write a small EDR-style sensor in C. It will subscribe to the Windows Kernel Process provider and print an event whenever a new process starts.</p> Before diving into the code, let’s look at the three main parts of ETW:</p> Providers</strong> - The components generating logs (e.g. the Windows Kernel).</li> Controllers</strong> - Applications that start and configure tracing sessions.</li> Consumers</strong> - Applications that read events from real-time sessions or `.etl</code> files.</li> </ul> Our C program will act as both a controller (creating the trace session) and a consumer (parsing and printing events).</p> Let’s build it step-by-step.</p>`Phase 1: Setting up the ETW Session (The Controller)</a></h3> To receive events, we first need to start an ETW trace session and define the GUID of the provider we want to enable.</p> In our case, it is Microsoft-Windows-Kernel-Process</code>, whose GUID is {22FB2CD6-0E7B-422B-A0C7-2FAD1F0E716}</code></a>.</p> #define</span> _WIN32_WINNT</span> 0x0600</span></span> #include</span> <windows.h></span></span> #include</span> <stdio.h></span></span> #include</span> <evntrace.h></span></span> #include</span> <evntcons.h></span></span> #include</span> <tdh.h></span></span> </span> #pragma comment</span>(</span>lib</span>, </span>"tdh.lib"</span>)</span></span> #pragma comment</span>(</span>lib</span>, </span>"advapi32.lib"</span>)</span></span> </span> static const</span> GUID ProviderGuid </span>=</span></span> {</span> 0x22FB2CD6</span>,</span> 0x0E7B</span>,</span> 0x422B</span>, {</span> 0xA0</span>,</span> 0xC7</span>,</span> 0x2F</span>,</span> 0xAD</span>,</span> 0x1F</span>,</span> 0xD0</span>,</span> 0xE7</span>,</span> 0x16</span> } };</span></span></code></pre> In main</code>, we configure the EVENT_TRACE_PROPERTIES</code> structure. EVENT_TRACE_REAL_TIME_MODE</code> makes events available to a real-time consumer instead of writing them to a disk file.</p> // Setup session properties</span></span> TRACE_PROPERTIES sessionProps </span>=</span> {</span> 0</span> };</span></span> sessionProps.Properties.Wnode.BufferSize </span>=</span> sizeof</span>(TRACE_PROPERTIES);</span></span> sessionProps.Properties.Wnode.Flags </span>=</span> WNODE_FLAG_TRACED_GUID;</span></span> sessionProps.Properties.Wnode.ClientContext </span>=</span> 1</span>; </span></span> sessionProps.Properties.LogFileMode </span>=</span> EVENT_TRACE_REAL_TIME_MODE;</span> // Crucial for EDR</span></span> sessionProps.Properties.LoggerNameOffset </span>=</span> offsetof</span>(TRACE_PROPERTIES, SessionName);</span></span> </span> // Start the trace session (Requires Administrator privileges!)</span></span> ULONG status </span>=</span> StartTraceW</span>(</span>&</span>g_hSession</span>, g_SessionName,</span> &</span>sessionProps.Properties);</span></span></code></pre>Phase 2: Subscribing to Kernel Events</a></h3> Starting a session is not enough. We must enable our provider for that session with EnableTraceEx2</code>.</p> status </span>=</span> EnableTraceEx2</span>(</span></span> g_hSession,</span></span> &</span>ProviderGuid</span>,</span></span> EVENT_CONTROL_CODE_ENABLE_PROVIDER,</span></span> TRACE_LEVEL_VERBOSE,</span></span> 0x10</span>,</span> // WINEVENT_KEYWORD_PROCESS</span></span> 0</span>,</span></span> 0</span>,</span></span> NULL</span></span> );</span></span></code></pre>Phase 3: Consuming the Events (The Consumer)</a></h3> Now that events are flowing into the session, we need to read them. We set up an EVENT_TRACE_LOGFILEW</code> structure, point it to our session name, and provide an EventRecordCallback</code> function that ETW calls for each event.</p> Finally, we call ProcessTrace()</code>. This function blocks while it processes events and invokes our callback, until the trace is stopped.</p> EVENT_TRACE_LOGFILEW logFile </span>=</span> {</span> 0</span> };</span></span> logFile.LoggerName </span>=</span> g_SessionName;</span></span> logFile.ProcessTraceMode </span>=</span> PROCESS_TRACE_MODE_REAL_TIME </span>\|</span> PROCESS_TRACE_MODE_EVENT_RECORD;</span></span> logFile.EventRecordCallback </span>=</span> EventRecordCallback;</span> // The function that handles data</span></span> </span> TRACEHANDLE hTrace </span>=</span> OpenTraceW</span>(</span>&</span>logFile</span>);</span></span> </span> printf</span>(</span>"Listening for Process Starts... (Press Ctrl+C to stop)</span>\n</span>"</span>);</span></span> status </span>=</span> ProcessTrace</span>(</span>&</span>hTrace</span>,</span> 1</span>,</span> NULL</span>,</span> NULL</span>);</span> // Blocks and processes events</span></span></code></pre>Phase 4: Parsing the Payload with TDH</a></h3> Raw ETW data is an unformatted binary payload. Parsing it manually is error-prone, and its layout can change with Windows updates. Thankfully, Microsoft provides the Trace Data Helper (TDH)</strong> API, which uses the provider’s manifest to parse the binary payload into readable strings and integers.</p> In EventRecordCallback</code>, we verify the provider and select event ID 1</code>, which this provider’s manifest defines as process start. We then use TdhGetProperty</code> to extract fields such as ProcessID</code> and ImageName</code>. Event schemas can vary by provider version, so production code should inspect the event metadata rather than assume every Windows build exposes an identical payload.</p> VOID WINAPI </span>EventRecordCallback</span>(PEVENT_RECORD </span>pEvent</span>)</span></span> {</span></span> if</span> (</span>memcmp</span>(</span>&</span>pEvent</span>-></span>EventHeader</span>.</span>ProviderId</span>,</span> &</span>ProviderGuid,</span> sizeof</span>(GUID))</span> !=</span> 0</span>)</span> return</span>;</span></span> </span> // Event ID 1 == Process Start</span></span> if</span> (</span>pEvent</span>-></span>EventHeader</span>.</span>EventDescriptor</span>.</span>Id</span> ==</span> 1</span>)</span></span> {</span></span> printf</span>(</span>"</span>\n</span>[+] Process Started!</span>\n</span>"</span>);</span></span> printf</span>(</span>" ProcessID: </span>%lu</span>\n</span>"</span>,</span> GetUint32Property</span>(pEvent, L</span>"ProcessID"</span>));</span></span> printf</span>(</span>" ParentProcessID: </span>%lu</span>\n</span>"</span>,</span> GetUint32Property</span>(pEvent, L</span>"ParentProcessID"</span>));</span></span> </span> PrintStringProperty</span>(pEvent, L</span>"ImageName"</span>);</span></span> </span> printf</span>(</span>"--------------------------------------------------</span>\n</span>"</span>);</span></span> }</span></span> }</span></span></code></pre> The complete program should include GetUint32Property</code> and PrintStringProperty</code> helpers that wrap TdhGetProperty</code>, plus error handling for missing properties. The excerpts here focus on the ETW control flow.</em></p> Graceful Shutdown</a></h3> Real-time ETW sessions are system-wide objects. If our program exits without stopping its session, that session can remain active. To prevent this, we handle Ctrl+C</code> and stop it with ControlTraceW</code>.</p> BOOL WINAPI </span>ConsoleHandler</span>(DWORD </span>signal</span>) {</span></span> if</span> (signal </span>==</span> CTRL_C_EVENT </span>&&</span> g_hSession) {</span></span> printf</span>(</span>"</span>\n</span>Stopping trace session gracefully...</span>\n</span>"</span>);</span></span> // Tell the kernel to stop our session</span></span> ControlTraceW</span>(g_hSession, g_SessionName,</span> &</span>stopProps</span>.</span>Properties</span>, EVENT_TRACE_CONTROL_STOP);</span></span> return</span> TRUE</span>;</span></span> }</span></span> return</span> FALSE</span>;</span></span> }</span></span></code></pre>Compiling and Running</a></h3> To compile this sensor, use the MSVC compiler included with Visual Studio. Open the x64 Native Tools Command Prompt</strong> and run:</p> cl.</span>exe</span> /</span>O2</span> mini_edr.c</span></span></code></pre> Testing the Agent:</strong> Because ETW interacts directly with the Windows Kernel, you must run this executable as an Administrator</strong>.</p> Open an elevated command prompt, run mini_edr.exe</code>, and then open an application like notepad.exe</code> or calc.exe</code>. You will see your mini-EDR catch the execution:</p> PS > .\mini_edr.exe</span></span> Trace Session 'ProcMonTrace_18872' Started.</span></span> Listening for Process Starts... (Press Ctrl+C to stop)</span></span> </span> [+] Process Started!</span></span> ProcessID: 18956</span></span> ParentProcessID: 1600</span></span> ImageName: \Device\HarddiskVolume3\Windows\System32\dllhost.exe</span></span> --------------------------------------------------</span></span> </span> [+] Process Started!</span></span> ProcessID: 19012</span></span> ParentProcessID: 6692</span></span> ImageName: \Device\HarddiskVolume3\Program Files\WindowsApps\Microsoft.WindowsNotepad_11.2512.26.0_x64__8wekyb3d8bbwe\Notepad\Notepad.exe</span></span> --------------------------------------------------</span></span> </span> [+] Process Started!</span></span> ProcessID: 19200</span></span> ParentProcessID: 2040</span></span> ImageName: \Device\HarddiskVolume3\Program Files\WindowsApps\Microsoft.WindowsCalculator_11.2508.4.0_x64__8wekyb3d8bbwe\CalculatorApp.exe</span></span> --------------------------------------------------</span></span></code></pre>Conclusion</a></h3> Congratulations! You’ve just written a core component of an EDR engine.</p> Commercial EDRs add many layers: additional telemetry sources, tamper protection, buffering, enrichment, correlation, and remote analysis. This small consumer is not an EDR by itself, but it demonstrates one practical way to collect process telemetry on Windows.</p> In a future lab, we can compare this approach with a signed kernel driver that uses process-creation callbacks.</p> Building a Mini-SIEM Lab: Sysmon, Vector, Redpanda, ClickHouse & Grafana 2026-02-22T00:00:00+00:00 Building a Security Information and Event Management (SIEM) system from scratch is one of the best ways to understand how those enterprise system works.</p> In this lab, we will build a modern log pipeline using Windows host and WSL2 (Debian). Before diving into the code, let’s understand core architecture of our pipeline:</p> Ingestion (Sysmon</a> & Vector</a>):</strong> Gathering logs from the operating system is crucial step. Usually we gather logs from different system so we need to transform them into a schema format (ECS/OCSF)</li> Buffering (Redpanda</a>):</strong> A messaging queue that temporarily hold logs. This prevent database from crashing if a sudden spike in log volume occurs.</li> Storage (ClickHouse</a>):</strong> An OLAP Database which is adjusted to handling Real-Time Analytics. It is column-oriented and optimized for analyzing massive amounts of log data.</li> Visualization (Grafana</a>):</strong> The dashboard where security analysts run queries and visualize threats.</li> </ol> Let’s build it step-by-step</p> Phase 1: Ingestion (Sysmon & Vector)</a></h2> First we need something to generate high-quality security logs. For that we can use Sysmon</strong> Windows service that logs process creation, network connection and more. We can use some already existing config e.g. SwitftOnSecurity</a></p> Open an elevated PowerShell prompt on your Windows machine and run:</p> `# Download config to temp</span></span> Invoke-WebRequest -Uri https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml -OutFile C:\Windows\Temp\sysmonconfig.xml</span></span> </span> # Install sysmon with config</span></span> sysmon64.exe -accepteula -i C:\Windows\Temp\sysmonconfig.xml</span></span></code></pre>` Next step is installing `Vector</code> by DataDog to read those logs and send them to our pipeline. Note: You need to compile Vector from source to includePR #24305</a>. If you are compiling from Windows, ensure you have LLVM</a> and follow the Vector manual installation guide</a></em></p>` `Create your Vector configuration file. This config maps raw Windows Event Logs to the Elastic Common Schema (ECS) format, making the data clean and structured.</p>` # vector.yaml</span></span> data_dir: C:\Users\USERNAME\AppData\Local\Temp\vector-data</span></span> </span> api:</span></span> enabled: true</span></span> address: 127.0.0.1:8686</span></span> </span> sources:</span></span> sysmon_logs:</span></span> type: windows_event_log</span></span> channels:</span></span> - Microsoft-Windows-Sysmon/Operational</span></span> - Application</span></span> - System</span></span> - Security</span></span> </span> transforms:</span></span> remap_ecs:</span></span> type: remap</span></span> inputs:</span></span> - sysmon_logs</span></span> source: \|</span></span> # Metadata</span></span> .ecs.version = "8.11.0"</span></span> .event.kind = "event"</span></span> .event.code = .event_id</span></span> .event.provider = .provider_name</span></span> .event.dataset = .channel</span></span> </span> # Map Log Level</span></span> .log.level = .level</span></span> </span> # Timestamp</span></span> .timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S.%fZ")</span></span> .@timestamp = .timestamp</span></span> </span> # Host</span></span> .host.name = .computer</span></span> .host.os.family = "windows"</span></span> .host.os.platform = "windows"</span></span> </span> # Event_data as primary source of truth</span></span> ed = .event_data</span></span> </span> if .channel == "Microsoft-Windows-Sysmon/Operational" && !is_null(ed) {</span></span> .event.module = "sysmon"</span></span> </span> # Sysmon Event 1: Process Creation</span></span> if .event_id == 1 {</span></span> .event.category = "process"</span></span> .event.type = ["start"]</span></span> </span> .process.pid = to_int!(ed.ProcessId)</span></span> .process.entity_id = ed.ProcessGuid</span></span> .process.executable = ed.Image</span></span> .process.command_line = ed.CommandLine</span></span> .process.working_directory = ed.CurrentDirectory</span></span> .process.integrity = ed.IntegrityLevel</span></span> </span> if !is_null(ed.Image) {</span></span> path_parts = split!(ed.Image, "\\")</span></span> .process.name = path_parts[-1]</span></span> }</span></span> </span> # User Parsing</span></span> if !is_null(ed.User) {</span></span> user_parts = split!(ed.User, "\\")</span></span> if length(user_parts) == 2 {</span></span> .user.domain = user_parts[0]</span></span> .user.name = user_parts[1]</span></span> } else {</span></span> .user.name = ed.User</span></span> }</span></span> }</span></span> </span> # Hash Parsing</span></span> if !is_null(ed.Hashes) {</span></span> hashes = parse_key_value!(ed.Hashes, key_value_delimiter: "=", field_delimiter: ",")</span></span> if !is_null(hashes.MD5) { .process.hash.md5 = hashes.MD5 }</span></span> if !is_null(hashes.SHA1) { .process.hash.sha1 = hashes.SHA1 }</span></span> if !is_null(hashes.SHA256) { .process.hash.sha256 = hashes.SHA256 }</span></span> if !is_null(hashes.IMPHASH) { .process.hash.imphash = hashes.IMPHASH }</span></span> }</span></span> </span> # Parent Process</span></span> .process.parent.pid = to_int!(ed.ParentProcessId)</span></span> .process.parent.entity_id = ed.ParentProcessGuid</span></span> .process.parent.executable = ed.ParentImage</span></span> .process.parent.command_line = ed.ParentCommandLine</span></span> </span> if !is_null(ed.ParentImage) {</span></span> parent_parts = split!(ed.ParentImage, "\\")</span></span> .process.parent.name = parent_parts[-1]</span></span> }</span></span> }</span></span> </span> # Sysmon Event 3: Network Connection</span></span> if .event_id == 3 {</span></span> .event.category = "network"</span></span> .event.type = ["connection", "start"]</span></span> </span> .source.ip = ed.SourceIp</span></span> .source.port = to_int!(ed.SourcePort)</span></span> .source.domain = ed.SourceHostname</span></span> </span> .destination.ip = ed.DestinationIp</span></span> .destination.port = to_int!(ed.DestinationPort)</span></span> .destination.domain = ed.DestinationHostname</span></span> .network.transport = ed.Protocol</span></span> </span> .process.pid = to_int!(ed.ProcessId)</span></span> .process.entity_id = ed.ProcessGuid</span></span> .process.executable = ed.Image</span></span> if !is_null(ed.Image) {</span></span> img_parts = split!(ed.Image, "\\")</span></span> .process.name = img_parts[-1]</span></span> }</span></span> </span> if !is_null(ed.User) {</span></span> user_parts = split!(ed.User, "\\")</span></span> if length(user_parts) == 2 {</span></span> .user.domain = user_parts[0]</span></span> .user.name = user_parts[1]</span></span> }</span></span> }</span></span> }</span></span> } else {</span></span> # Handle standard Windows logs (Security, System, App)</span></span> if .channel == "Security" {</span></span> .event.category = "authentication"</span></span> if .event_id == 4624 {</span></span> .event.type = ["start", "authentication_success"]</span></span> .event.outcome = "success"</span></span> }</span></span> if .event_id == 4625 {</span></span> .event.type = ["start", "authentication_failure"]</span></span> .event.outcome = "failure"</span></span> }</span></span> }</span></span> if .channel == "System" { .event.category = "host" }</span></span> if .channel == "Application" { .event.category = "process" }</span></span> </span> if !is_null(ed) {</span></span> if !is_null(ed.TargetUserName) { .user.name = ed.TargetUserName }</span></span> if !is_null(ed.TargetDomainName) { .user.domain = ed.TargetDomainName }</span></span> if !is_null(ed.SubjectUserName) && ed.SubjectUserName != "-" { .user.effective.name = ed.SubjectUserName }</span></span> if !is_null(ed.IpAddress) && ed.IpAddress != "-" { .source.ip = ed.IpAddress }</span></span> if !is_null(ed.IpPort) && ed.IpPort != "-" { .source.port = to_int(ed.IpPort) ?? null }</span></span> if !is_null(ed.ProcessName) && ed.ProcessName != "-" {</span></span> .process.executable = ed.ProcessName</span></span> p_parts = split!(ed.ProcessName, "\\")</span></span> .process.name = p_parts[-1]</span></span> }</span></span> if !is_null(ed.ProcessId) { .process.pid = to_int(ed.ProcessId) ?? null }</span></span> }</span></span> }</span></span> </span> # Clean up raw fields to save bandwidth</span></span> del(.event_data)</span></span> del(.message)</span></span> del(.task)</span></span> del(.level_value)</span></span> del(.record_id)</span></span> del(.version)</span></span> </span> # Send to Redpanda (Kafka)</span></span> sinks:</span></span> to_redpanda:</span></span> type: kafka</span></span> inputs:</span></span> - remap_ecs</span></span> bootstrap_servers: 127.0.0.1:29092</span></span> topic: event-logs</span></span> encoding:</span></span> codec: json</span></span></code></pre>Phase 2, 3 & 4: Buffering, Storage, UI</a></h2> We will run our backend services in WSL2 (Debian) using Podman. Save following as docker-compose.yml</code></p> # docker-compose.yml</span></span> version: '3.8'</span></span> </span> services:</span></span> redpanda:</span></span> image: docker.redpanda.com/redpandadata/redpanda:latest</span></span> container_name: redpanda</span></span> command:</span></span> - redpanda</span></span> - start</span></span> - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092</span></span> # Advertise external to localhost so Windows Vector can reach it. Internal to redpanda:9092 so Clickhouse can reach it.</span></span> - --advertise-kafka-addr internal://redpanda:9092,external://127.0.0.1:29092</span></span> - --smp 1</span></span> - --memory 1G</span></span> - --mode dev-container</span></span> ports:</span></span> - "29092:29092"</span></span> - "9092:9092"</span></span> - "8081:8081"</span></span> - "9644:9644"</span></span> volumes:</span></span> - redpanda_data:/var/lib/redpanda/data</span></span> networks:</span></span> - siem_net</span></span> </span> clickhouse:</span></span> image: docker.io/clickhouse/clickhouse-server:latest</span></span> container_name: clickhouse</span></span> environment:</span></span> - CLICKHOUSE_PASSWORD=siemlab</span></span> ports:</span></span> - "8123:8123" # HTTP port (Grafana uses this)</span></span> - "9000:9000" # Native client port</span></span> volumes:</span></span> - clickhouse_data:/var/lib/clickhouse</span></span> - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql # Auto-initializes tables</span></span> depends_on:</span></span> - redpanda</span></span> networks:</span></span> - siem_net</span></span> </span> grafana:</span></span> image: docker.io/grafana/grafana:latest</span></span> container_name: grafana</span></span> ports:</span></span> - "3000:3000"</span></span> environment:</span></span> # Automatically install the ClickHouse plugin</span></span> - GF_INSTALL_PLUGINS=grafana-clickhouse-datasource</span></span> volumes:</span></span> - grafana_data:/var/lib/grafana</span></span> depends_on:</span></span> - clickhouse</span></span> networks:</span></span> - siem_net</span></span> </span> networks:</span></span> siem_net:</span></span> driver: bridge</span></span> </span> volumes:</span></span> redpanda_data:</span></span> clickhouse_data:</span></span> grafana_data:</span></span></code></pre> RedPanda does not require additional configuration but ClickHouse and Grafana does.</p> Initializing Database</a></h4> ClickHouse needs to know how to read the logs from Redpanda and where to store them. We do this using a three-step architecture.</p> Target Table:</strong> Where the data permanently lives.</li> Kafka Engine Table:</strong> A temporary consumer that pulls data from Redpanda</li> Materialized View: The pipeline connecting the consumer to the target table.</li> </ol> Let’s save this as init-db.sql</code> in the same directory as previous file so it will be initialized when database is initialized. Here ingestion is smaller than our Event source, but we can extend that according to our use case.</p> -- init-db.sql</span></span> CREATE DATABASE IF NOT EXISTS siem;</span></span> </span> -- 1. Storage Table: The actual storage for your ECS logs</span></span> CREATE TABLE IF NOT EXISTS siem.logs</span></span> (</span></span> `@timestamp` DateTime64(3),</span></span> `event.kind` String,</span></span> `event.category` String,</span></span> `event.type` Array(String),</span></span> `event.code` UInt32,</span></span> `host.name` String,</span></span> `process.name` String,</span></span> `process.pid` UInt32,</span></span> `source.ip` String,</span></span> `destination.ip` String,</span></span> `raw_message` String -- Storing the raw JSON is great for students to dig into missing fields</span></span> ) ENGINE = MergeTree()</span></span> ORDER BY `@timestamp`;</span></span> </span> -- 2. Consumer Table: Pulls raw JSON from Redpanda.</span></span> CREATE TABLE IF NOT EXISTS siem.logs_queue</span></span> (</span></span> message String</span></span> ) ENGINE = Kafka</span></span> SETTINGS kafka_broker_list = 'redpanda:9092',</span></span> kafka_topic_list = 'event-logs',</span></span> kafka_group_name = 'clickhouse_consumer_group',</span></span> kafka_format = 'JSONAsString';</span></span> </span> -- 3. Pipeline: The Materialized View translating the Queue into Storage</span></span> CREATE MATERIALIZED VIEW IF NOT EXISTS siem.logs_mv TO siem.logs AS</span></span> SELECT</span></span> -- Safely parse timestamp, default to current time if missing so the pipeline doesn't break</span></span> ifNull(parseDateTime64BestEffortOrNull(JSONExtractString(message, '@timestamp'), 3), now64(3)) AS `@timestamp`,</span></span> JSONExtractString(message, 'event', 'kind') AS `event.kind`,</span></span> JSONExtractString(message, 'event', 'category') AS `event.category`,</span></span> </span> -- Convert raw JSON array strings to actual ClickHouse Arrays</span></span> arrayMap(x -> replaceAll(x, '"', ''), JSONExtractArrayRaw(message, 'event', 'type')) AS `event.type`,</span></span> </span> JSONExtractUInt(message, 'event', 'code') AS `event.code`,</span></span> JSONExtractString(message, 'host', 'name') AS `host.name`,</span></span> JSONExtractString(message, 'process', 'name') AS `process.name`,</span></span> JSONExtractUInt(message, 'process', 'pid') AS `process.pid`,</span></span> JSONExtractString(message, 'source', 'ip') AS `source.ip`,</span></span> JSONExtractString(message, 'destination', 'ip') AS `destination.ip`,</span></span> message AS `raw_message`</span></span> FROM siem.logs_queue;</span></span></code></pre> Now we need to install podman</code> and podman-compose</code> then start the stack</p> sudo apt install podman podman-compose</span></span> sudo podman-compose up -d</span></span></code></pre>Troubleshooting WSL2 & Podman Issues</a></h2> When running Podman on WSL2, you might encounter a few known issue. Here is how to fix them:</p> Error:</strong> WARN: The cgroupv2 manager is set to systemd but there is no systemd user session available</code> Fix:</em> You need to enable systemd inside WSL2. Edit or create /etc/wsl.conf</code></p> [boot]</span></span> systemd=true</span></span></code></pre> Error:</strong> netavark: nftables error: nft did not return successfully while applying ruleset</code> Fix:</em> Switch your firewall to legacy iptables</code>:</p> sudo update-alternatives --set iptables /usr/sbin/iptables-legacy</span></span> sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy</span></span></code></pre> If that fails, force Podman to use iptables via its config</p> sudo mkdir -p /etc/containers</span></span> echo -e "[network]\nfirewall_driver = \"iptables\"" \| sudo tee -a /etc/containers/containers.conf</span></span></code></pre> Start Fresh</strong> If you mess up your configuration and need to wipe your state clean, run:</p> # Powershell</span></span> wsl --shutdown</span></span> </span> # WSL2</span></span> podman system reset -f</span></span> sudo podman network prune -f</span></span> sudo podman-compose down</span></span></code></pre>Phase 4: UI and Threat Hunting with Grafana</a></h2> If all services are running, it’s time to connect Grafana to ClickHouse</p> Open browser and go to http://localhost:3000</code></li> Navigate to Connections > Data Sources > Add data source</li> Select ClickHouse and fill in the details: Server Address:</strong> clickhouse</li> Protocol:</strong> HTTP</li> Secure (TLS):</strong> Off</li> Username:</strong> default</li> Password:</strong> siemlab</li> Default Database (Advanced):</strong> siem</li> </ol> </li> </ol> Click “Save & Test”. If it fails, check the logs with sudo podman logs grafana</code></p> Example Queries</a></h4> Create a new dashboard, add a visualization, and try these queries to analyze your ingested data.</p> Query 1: Find Noisy Processes</strong> Great for establishing a baseline of normal activity on host</p> SELECT</span></span> `process.name`,</span></span> count() AS total_events</span></span> FROM siem.logs</span></span> WHERE $__timeFilter(`@timestamp`) AND `process.name` != ''</span></span> GROUP BY `process.name`</span></span> ORDER BY total_events DESC</span></span> LIMIT 10</span></span></code></pre> </p> Query 2: Raw Event Log View</strong> A tabular view of recent events</p> SELECT</span></span> `@timestamp` AS Time,</span></span> `host.name` AS Host,</span></span> `event.code` AS EventID,</span></span> `process.name` AS Process,</span></span> `source.ip` AS SourceIP,</span></span> `destination.ip` AS DestIP</span></span> FROM siem.logs</span></span> WHERE $__timeFilter(`@timestamp`)</span></span> ORDER BY `@timestamp` DESC</span></span> LIMIT 100</span></span></code></pre> </p> Now you have local SIEM processing real-time Windows logs.</p>

Daniel's Detection Notes

Detection Is a Data-Modeling Problem

2026-09-17T00:00:00+00:00

Building an EDR from scratch, essay 7 of 13.

Detection engineering has a glamorous reputation. It’s the part with the attacker in it — behavioral analytics, threat hunting, the rule that catches the thing nobody else caught.

Having now built a small one, my honest assessment is that the clever rules are the easy part, and they’re not where the failures live. The failures live in the data model underneath: whether a rule can express what you mean, whether it means the same thing tomorrow, and — the one that nearly got me — whether the system tells you when it has quietly stopped working.

This essay is about that substrate.

One vocabulary, two purposes</a></h2>
The first decision sounds like an implementation detail and turns out to shape everything. From backend/src/detect.rs</code>:
//! Rules match the *normalized* EventRow (post-flatten), not raw proto: //! field names in a rule are the ClickHouse column names, so a rule that //! works in SQL works here and vice versa.</code></pre> Rules could have matched against the protobuf message — it’s what arrives on the wire, and it’s the natural object in the code. Instead they match the flattened database row, using database column names. The reason is a workflow, not a technical constraint. Real detection engineering goes like this: an analyst is hunting, writing SQL against historical events, and finds something. A query returns rows that shouldn’t exist. Now they want that query to run continuously against new data. If rules speak a different language than queries, that transition is a translation step — and translation is where meaning quietly drifts. The analyst’s WHERE reg_key LIKE '%\CurrentVersion\Run%'</code> becomes some registry.key_path</code> field in a rule DSL, and the two are almost the same, and the difference surfaces six months later as an alert that never fired. When both speak the same names, promoting a hunt to a rule is copying a condition. And it works in reverse, which is just as valuable: given an alert, you can paste its condition into SQL and ask “how often did this happen last month?” The investigation and the detection are the same sentence in the same language. Make the exploratory tool and the production tool speak the same vocabulary. The friction you remove is conceptual, not typing.
Correctness and volume are different problems</a></h2> The second idea is a separation that I think a lot of systems get wrong by conflating. Two things sound similar: don’t record the same alert twice and don’t spam the analyst. They are unrelated problems, and this engine solves them with unrelated mechanisms — deliberately. //! works in SQL works here and vice versa. Evaluation is pure — the same //! rows always yield the same alerts — which is what makes replay safe: //! an alert's identity is (agent_id, sequence, event_index, rule_id), so //! a replayed WAL batch re-derives byte-identical alert keys</code></pre> Alert identity is derived entirely from its inputs: which agent, which event, which rule. No timestamps of evaluation, no counters, no randomness. So when a WAL batch replays — which, per essay 2</a>, is routine — the same alerts are re-derived with the same keys and collapse in the database exactly like the events do. This is the dedupe token idea from the pipeline, applied one plane up, and it works for the same reason: identity computed from content is stable under repetition. Purity isn’t an aesthetic preference here. It’s the property that makes replay safe, and any impurity — an evaluation timestamp in the key, say — would silently break it. Noise suppression is the other problem, and note how differently it’s treated: //! Noise suppression is separate from replay dedupe and deliberately //! best-effort: an in-memory window per (rule_id, agent_id) that survives //! neither restarts nor multi-instance deployments — the deterministic //! alert key is what guarantees correctness, suppression only reduces //! volume.</code></pre> Suppression is explicitly allowed to be wrong. It lives in memory, dies on restart, and doesn’t coordinate across instances. If a backend restarts, you might see a duplicate alert you’d have preferred to suppress. Nobody’s correctness is violated; someone is mildly annoyed. That’s a completely legitimate place to be sloppy — provided you’re clear about which mechanism carries the guarantee. The failure I’ve seen in real systems is using the suppression cache as the deduplication mechanism, at which point a restart doesn’t produce a duplicate alert, it produces a missing one, and nobody notices because the alert that didn’t fire is invisible. Which leads directly to the subtlest line in the module: //! volume. The check/commit split is load-bearing: evaluate() only READS //! the window and commit() is called strictly after the alert insert //! succeeded, so an alert whose insert failed is never remembered as //! "already fired" and lost on the retry.</code></pre> Evaluation reads the suppression window. It does not update it. The update happens only after the alert has actually been written. Get that ordering backwards — mark it suppressed at evaluation time, then insert — and here’s your incident: the database write fails, the retry arrives, the suppression window says “already fired,” and the alert is dropped. It was never stored anywhere. The only trace is a metric counter that doesn’t match. The general rule is worth stating carefully because it’s violated everywhere: a record that says “I did X” must be written after X, never before. The window between attempt and confirmation is exactly where crashes and errors live, and a system that has already recorded success on entering that window cannot tell the difference between done and never-happened. Knowing what your data means</a></h2> Now my favorite rule-writing detail, because it isn’t about rules at all. From backend/rules/README.md</code>: Every string condition here uses the ASCII-case-insensitive form (`contains_i`, `eq_i`). The registry is a case-insensitive namespace, so the case recorded in the event is the *writer's* choice, not a fact about the key: a value that does not already exist is stored with whatever case created it. A case-sensitive `reg_value_name eq "Debugger"` would miss `debugger` for free, which is not a detection worth shipping.</code></pre> Sit with “the case recorded in the event is the writer’s choice, not a fact about the key.” The Windows registry is case-insensitive but case-preserving. Create a value named debugger</code> and the registry stores it as debugger</code> while treating it as identical to Debugger</code> for every lookup. Windows will happily use it. Your telemetry faithfully records the lowercase form. And a rule matching Debugger</code> exactly sees nothing. That’s not a bug in the sensor — the sensor reported precisely what happened. It’s a rule that mistook an incidental encoding for semantic content. And the bypass costs an attacker exactly one keystroke, with no tooling and no cleverness. You cannot find this class of bug by testing your rule, because your test will use the canonical casing, and it will pass. You find it by asking a question about the data itself: for this field, which parts of the value are meaningful and which are the writer’s arbitrary choice? The same question, asked elsewhere, produces: paths on case-insensitive filesystems, domain names, HTTP headers, Unicode normalization forms, trailing dots on FQDNs, /tmp</code> versus /private/tmp</code> on macOS. Every one is a place where two representations mean the same thing, and where a comparison that doesn’t know it becomes a free bypass. The packaged rule shows what this looks like in practice — note that both conditions use the _i</code> form: "where": [ {"field": "reg_key", "op": "contains_i", "value": "\\CurrentVersion\\Run"}, {"field": "reg_data", "op": "contains_i", "value": "\\AppData\\"} ],</code></pre> The key path is case-insensitive because the registry is. But reg_data</code> is a filesystem path the attacker chose, and C:\Users\bob\appdata\roaming\x.exe</code> works perfectly well on Windows. Any field an adversary controls the encoding of needs the same treatment, for the same reason. When healthy is a lie</a></h2> Now the story that reorganized how I think about validation, and it’s the reason this essay exists. The alerts table has an event_type</code> column, an enum listing every event type an alert can be about. It was created with 11 types. Meanwhile the events table grew — module loads, process access, TCC, exec-maps, registry — to 23. The alerts enum never followed. So a rule matching on, say, registry_value_set</code> produced an alert whose event_type</code> byte had no corresponding enum member. Here’s what happened next, from backend/schema/015_alerts_registry.sql</code>: -- an alert whose stored byte had no enum member. That does NOT fail -- the INSERT — RowBinary writes the byte unvalidated — it fails on -- READ, when the column is formatted (UNKNOWN_ELEMENT_OF_ENUM), -- breaking SELECT * on the table with nothing logged at ingest.</code></pre> Trace the observable state of the system. Batches are acknowledged. The alert counter climbs. No error is logged, anywhere, by anyone. The metrics dashboard is green. Every health check passes. And SELECT * FROM edr.alerts</code> throws. The detection pipeline was “working” by every signal it emitted, and the alerts were unreadable. The gap between the operation failing and anyone finding out was bounded only by how long until someone looked at the table. In this case, seven schema versions. Two things are worth extracting. The first is about where validation lives. RowBinary is a schema-less binary format: values are written positionally with no names and no types on the wire. That’s what makes it fast, and the speed is real. But “no schema on the wire” means the receiver cannot check anything — it writes the bytes you gave it. The validation you thought the database was doing was never happening; you had simply never sent it anything invalid before. Fast serialization formats buy their speed by trusting the sender. The trust doesn’t remove the need for validation, it relocates it — and if you don’t consciously relocate it, it lands nowhere. Which is why the fix isn’t only the migration: Two unit tests now pin the alerts enum’s membership and the struct’s column order against the DDL file itself. The check moved into the build, because that’s the last place both sides of the contract are visible at once. The second extraction is subtler and rather beautiful: -- Extending the enum also HEALS rows already written that way: the -- stored byte becomes a legal member again and reads back correctly, -- so no repair pass is needed after this runs.</code></pre> The old rows repair themselves. Because the byte was stored faithfully — ClickHouse wrote exactly what it was given — the data was never corrupt. It was uninterpretable, which is a different condition entirely. The value was right; the dictionary needed to read it was incomplete. Extend the dictionary and every historical row becomes readable. That distinction — lossless but unreadable versus lossy — is worth carrying around. Systems that store faithfully and interpret separately can recover from interpretation bugs. Systems that normalize on the way in cannot: they’ve thrown away the thing they’d need to fix it later. It’s the same instinct as essay 3</a>’s insistence that absence be representable. Preserve what you were given; decide what it means as late as you can. There’s a companion hazard right next to it, mentioned in the same file: RowBinary is positional, so the Rust struct’s field order and the DDL’s column order are a single contract split across two files in two languages. Get them out of step and every value shifts one column over — severity</code> lands in state</code>, agent_id</code> lands in sequence</code> — and again, nothing errors. Hence the comment on the struct: /// One edr.alerts row; field order matches the DDL exactly (RowBinary).</code></pre> A comment is not enforcement, which is why there’s a test. But the pattern to notice is that implicit contracts need explicit tests, and the most dangerous contracts are the ones expressed as an agreement about order, because they fail silently and totally. Retention makes you denormalize</a></h2> One more modeling decision, driven by an unglamorous force: -- Context copied from the matched event at detection time, so triage -- answers "what ran / what was touched / where did it connect" -- without a join and unaffected by the events TTL.</code></pre> An alert copies its context from the event that fired it. Normalization says don’t — store the reference, join when needed. The reason to break that rule is stated plainly in the migration: events live 30 days, alerts live 90, and an investigation routinely outlives its evidence. An alert that says “rule X fired on event (agent, 4821, 3)</code>” is, on day 45, a pointer into a void. The join returns nothing. The alert is technically still there and practically worthless. Normalization implicitly assumes both sides of a relationship live the same length of time. When they don’t, a foreign key is a promise that expires. Different retention periods force denormalization, and the copy isn’t redundancy — it’s the only way the alert stays meaningful for its own lifetime. Where the difficulty actually is</a></h2> None of the hard parts in this essay were about detection logic. The rules themselves are almost trivial — a few string comparisons ANDed together, matching persistence keys that have been documented for twenty years. The hard parts were: whether a rule can be expressed in the same language as a query; whether alert identity survives replay; which mechanism carries the guarantee and which is allowed to be sloppy; whether a comparison respects the semantics of the namespace it’s comparing in; whether a serialization format validates or merely stores; whether a foreign key outlives what it points to. Every one of those is data modeling. And every one of them can break a detection without breaking anything visible — which is the recurring theme of this whole series and the specific reason detection is unforgiving. A crashed system announces itself. A detection that stopped firing looks exactly like a quiet week. Which raises the next question. If silent failure is the enemy, and the code runs inside kernels on machines you’ll never touch, how do you ever know it works? The next essay tackles that problem.

What You Cannot See 2026-09-09T00:00:00+00:00 Building an EDR from scratch, essay 6 of 13. Every sensor in this series watches an operating system abstraction. Process creation. File open. Socket connect. Module load. The events are delivered because the kernel maintains those abstractions and offers hooks where they’re maintained. Which implies something uncomfortable, and it’s the honest starting point for this essay: anything that accomplishes its goal without going through the abstraction you’re watching is invisible to you. Not hard to see. Not low-fidelity. Invisible, structurally, in a way no amount of tuning fixes. An attacker who understands your sensor’s hook points is not trying to evade your detection logic. They’re trying to accomplish their goal via a path that never touches your hook. That’s a completely different game, and it’s why the interesting question for a telemetry system isn’t “what does it catch” but “what is it incapable of catching, and do we know?” This essay is about the boundary. It ends up being a lesson in epistemology more than in kernels. When the obvious field lies to you</a></h2> Start with something that looks like solid ground. A process starts, and you record the path of the executable. Every EDR does this; most detection logic is built on it. Then consider memfd_create</code>. It gives you a file descriptor backed by anonymous memory — a “file” that exists only in RAM, with no directory entry anywhere on disk. You can write an ELF binary into it and execve</code> it. The process runs. Nothing was ever written to the filesystem. What does your sensor report for the image path? Something like /dev/fd/3</code>. That’s not a lie exactly, but it’s useless: it’s a synthetic path naming a file descriptor in a process that is about to be replaced. It tells you nothing about what ran. Your hash of the image can’t be computed from it. Your allowlist can’t match it. Your analyst sees a path that resolves to nothing. The field is present, well-formed, and empty of meaning. Those are the worst kind of blind spot, because a null would at least announce itself. The fix, from linux/bpf/edr_probes.bpf.c</code>, is to stop asking the path and start asking the inode: if (BPF_CORE_READ(file, f_inode, i_nlink) == 0) ev->exec_flags |= EDR_LX_EXEC_UNLINKED;</code></pre> i_nlink</code> is the number of directory entries pointing at this inode. Zero means nothing on the filesystem names this file. It might be a memfd. It might be a normal binary that was unlinked after being opened — the classic “drop, execute, delete” that leaves a running process with no file behind it. Either way, one integer answers a question the path could not: is this executable reachable by name? A second read distinguishes the specific case: if (dn && bpf_probe_read_kernel(dname, sizeof(dname), dn) == 0 && dname[0] == 'm' && dname[1] == 'e' && dname[2] == 'm' && dname[3] == 'f' && dname[4] == 'd' && dname[5] == ':') ev->exec_flags |= EDR_LX_EXEC_MEMFD | EDR_LX_EXEC_UNLINKED;</code></pre> That’s a manually unrolled six-character string comparison, which looks absurd next to strncmp</code> until you remember it’s running under the eBPF verifier, where a fixed-size read at a constant offset is provable and a variable-length loop is an argument. The comment says exactly that: “A short fixed read keeps the verifier out of variable-offset territory.” But the technique is what I want to underline. The path was unreliable, so the probe moved closer to the mechanism — from a rendered string to the kernel structure the string was rendered from. Sensor design is full of this. When the convenient field is derived, ask what it’s derived from, and whether the source is harder to lie about. Two boolean flags, and a class of execution that used to be indistinguishable from a normal process start becomes a query you can write. The attacker’s goal was to run code without leaving a file. They achieved it. They just couldn’t do it without producing an inode with zero links, which is the one thing the technique fundamentally requires. Look for what the technique cannot avoid. Anything else is a signature, and signatures get bypassed. Absence as a signal</a></h2> There’s a stranger version of this, and it’s my favorite piece of design in the whole codebase. Loading a kernel module is the rootkit’s move — it’s the transition from “code running on the machine” to “code inside the machine.” Linux offers two syscalls for it. finit_module</code> takes a file descriptor. init_module</code> takes a memory buffer. The file-based path is straightforward to instrument, and the probe grabs the name: dn = BPF_CORE_READ(file, f_path.dentry, d_name.name); n = bpf_probe_read_kernel_str(ev->name, sizeof(ev->name), dn);</code></pre> The memory-based path has no file at all, and therefore nothing to name: /* init_module: the module image is a user buffer — no file, so no name. * The absence IS the signal (a fileless kernel-module load). */</code></pre> That comment is doing real philosophical work. There is no name to report, and rather than treating that as a data-quality problem to paper over, the design treats the shape of the missing data as the finding. An event that says “a kernel module was loaded and I cannot tell you from where” is not a degraded event. It is a high-signal one, because the legitimate path — modprobe</code>, insmod</code>, the distro’s own tooling — uses the file-based syscall. The nameless variant is the interesting one almost by definition. There’s an even neater trick layered on top, which relies on ordering rather than content. The attempt hooks fire when a load is requested; a separate tracepoint fires when a module is actually accepted. So an attempt with no matching acceptance means the kernel refused the load — a signature check failing, lockdown mode engaging, a malformed module. Nobody emits a “module load refused” event. That fact is constructed by noticing that a thing which should have followed didn’t. The general move: you can detect an event by observing the hole where its consequence should be. It requires knowing the normal causal chain well enough to notice a missing link, which is a much deeper form of instrumentation than pattern-matching on fields. Missed connections. Absent heartbeats. Requests without responses. Once you start looking for holes, they’re everywhere, and they’re often where the interesting things hide. Choosing your blindness</a></h2> Now the uncomfortable kind of blind spot: the one you build yourself, knowingly, because the alternative is worse. Watching executable memory mappings would be a superb detection surface. A DLL being mapped into a process is how sideloading works, how injection works, how a malicious plugin gets in. So: report every file mapped executable. Try it and the sensor dies. Every process launch maps dozens of system libraries — 30 to 100 DLLs on Windows, dyld’s whole flood on macOS. The event rate is orders of magnitude above anything interesting, the ring buffer overflows, and you lose events including the ones you cared about. A firehose of noise doesn’t just bury the signal; it evicts it. So you filter. And here’s the thing worth sitting with: filtering is deliberately choosing what to be blind to. The filter isn’t a performance detail, it’s a coverage decision with a security consequence, and it should be argued rather than tuned into place. The story of choosing this particular filter is recorded in the README</code>, and it’s a good one because the elegant idea lost: a pure W^X filter was tried first and rejected — `max_protection` is `RWX` for nearly every mapping on recent macOS, and a *current* RWX mapping is hardware-blocked on Apple Silicon</code></pre> The elegant idea was W^X: report mappings that are both writable and executable, since legitimate code is generally one or the other. Principled, mechanism-based, exactly the kind of filter this essay has been praising. It doesn’t work, for two independent reasons that only reality could have supplied. max_protection</code> — the maximum protection a mapping could ever be granted — is RWX for almost everything on modern macOS, so the field doesn’t discriminate. And current RWX is hardware-blocked on Apple Silicon, so the thing you’d actually want to catch can’t exist in that form anyway. The theory was clean and the platform simply doesn’t behave the way the theory assumed. What shipped instead is a crude path exclusion: skip /System</code> and /usr</code> on macOS, skip the Windows directory on Windows. It’s an inelegant heuristic and everyone involved knows it. The README says so plainly — Apple’s own developer tooling reports as a non-system code load, and on Windows every application’s own DLLs under Program Files</code> report too. But it’s honest about its own nature, it’s cheap enough to run in kernel context, and it reduces the volume enough that the interesting events survive. Sometimes the crude filter that works beats the beautiful one that doesn’t, and the professional skill is noticing which situation you’re in. One structural point, easy to miss and load-bearing: this filtering happens in the kernel, before the event is written to the ring. Filtering in user space would move the drop counter, not the load. The kernel would still produce the events, the ring would still overflow, and you’d still lose data — you’d just discard it after paying for it. A high-volume callback needs its filter at the source. The floor</a></h2> And finally, the blind spot with no clever workaround. From the README’s limitations: **Manually mapped ("reflective") images are invisible** on Windows: a DLL written into memory and relocated by hand creates no image section, so no load-image notification is ever delivered.</code></pre> Reflective DLL loading works by doing the loader’s job yourself. Allocate memory, copy in the PE, walk the relocation table, resolve imports, jump to the entry point. The result is functioning code in a process — and the operating system’s loader was never involved, so there is no image section, so PsSetLoadImageNotifyRoutine</code> has nothing to notify about. The hook isn’t failing. The event genuinely does not exist. macOS has the identical hole from the other direction: MAP_ANON</code> memory with shellcode in it is anonymous, file-backed by nothing, and there is no “file mapped executable” event to deliver. The README’s framing is the right one: in both cases the sensor sees code loads that pass through the OS loader, not code that bypasses it.</code></pre> That single sentence defines the sensor’s epistemic boundary better than any feature list could. You are not watching “code execution.” You are watching the operating system’s code-loading machinery, and the difference is exactly the set of techniques that skip it. Knowing the boundary changes what you do about it. You stop trying to make the file-mapping sensor catch reflective loading — it cannot, ever — and you start looking for the consequences an in-memory payload can’t avoid: the network connection it eventually makes, the process it injects into, the persistence it establishes, the child it spawns. The technique evades one sensor. It rarely evades the whole system, because it still has to accomplish something, and accomplishing things means using abstractions. Why writing them down is the actual skill</a></h2> The README</code> has a “Known limitations” section that runs for over a hundred lines. Memory-mapped writes attributed at mapping time rather than at write time. Linux MODIFY firing on open-intent rather than first write. DNS invisible on macOS loopback because content filters never see it. TCC events that need an entitlement the build doesn’t have. Windows UDPv6 decoded from documentation but never ground-truthed against a real capture. That list is, I’d argue, the most valuable artifact in the repository, and it’s worth being explicit about why. An analyst investigating an incident is constantly making inferences from absence. “There’s no network connection from that process, so it never called home.” “There’s no file write, so nothing was dropped.” Those inferences are only valid if the sensor would have seen it. Absence of evidence is evidence of absence only when you have coverage — and the limitations list is precisely the document that tells you when you don’t. This is the same principle as the counted ring drops from essay 2</a>, one level up. There, the pipeline made data loss known rather than silent. Here, the documentation makes coverage gaps known rather than implied. In both cases the underlying value is the same: a system that tells you what it doesn’t know is trustworthy in a way that a system implying total coverage never is. And there’s a professional dimension. Vendors are structurally incentivized to describe coverage and stay quiet about gaps, because gaps read as weakness in a bake-off. The result is a market full of tools whose users systematically overestimate what they can conclude from silence. Writing the gaps down, next to the code that causes them, is the opposite bet: it makes the tool less impressive and considerably more useful. Which sets up the next question. Given a stream of events with known coverage and known holes, how do you actually turn it into a claim that something bad is happening? That turns out to be much less about clever rules than about the shape of the data underneath them. That data model is the subject of the next essay. Cutting the Network from Inside 2026-09-02T00:00:00+00:00 Building an EDR from scratch, essay 5 of 13. Network isolation is the response action that looks easiest and is hardest. The pitch is simple enough that a manager can specify it in one sentence: a machine is compromised, cut it off the network, contain the damage. Every commercial EDR has the button. It usually has a reassuring icon. And the implementation is a paradox with a very short fuse. You are going to tell a machine to stop talking to the network — using the network. The instruction to sever connectivity arrives over the connectivity it severs. If you do the obvious thing, the machine goes silent mid-sentence and stays silent forever, because the only channel you had for saying “actually, let it back on” was the one you just cut. That’s not a race condition you’ll catch in code review. It’s a machine that needs someone to physically walk to it. Multiply by the number of hosts an analyst can click through in a panic, and a containment feature becomes an outage. Everything interesting about this action follows from that one problem. Isolation isn’t “block everything”</a></h2> The naive version — drop all traffic — is unusable, and the exemption list tells you a lot about how networks actually work. From shipper/src/isolate.rs</code>: //! Self-harm is the whole problem (see proto IsolateHostAction): a //! ruleset that fails to exempt the agent's own backend endpoint bricks //! the host — isolated, unreachable, and un-un-isolatable. So the //! executor refuses to apply ANYTHING unless it can parse and resolve //! the backend endpoint, and validates every operator CIDR before the //! kernel is touched (all-or-nothing).</code></pre> Start with the refusal, because it’s the most important line. If the agent cannot determine its own backend endpoint, it applies nothing at all. Not a best-effort ruleset, not a partial block. The action fails and reports failure. This runs against a reflex most of us have — partial progress feels responsible, and “we contained most of it” sounds better than “we did nothing.” Here it’s precisely backwards. A partially-applied isolation is a bricked host, which is strictly worse than an uncontained one: you’ve lost the machine and your visibility into it, and now the incident includes a site visit. When a failure mode is unrecoverable, refusing to start is the safe branch. The rest of the exemption list is a tour of the protocols people forget: //! kernel is touched (all-or-nothing). The exemption set is deliberately //! minimal: backend endpoint, loopback, DHCP (v4 + v6, or lease expiry //! takes the host off the network underneath the containment), and IPv6 //! neighbor discovery (ARP lives below IP, but NDP rides ICMPv6 — block //! it and v6 connectivity dies, including to the backend).</code></pre> Loopback — block it and half the software on the machine breaks, because local services talk to each other over 127.0.0.1</code> constantly. You’d contain the host by breaking it. DHCP is the one that gets people, and it’s a lovely example of a time-delayed failure. Block DHCP and everything works fine. For hours. Then the lease expires, the renewal is blocked, the interface loses its address, and the host drops off the network entirely — including the exempted backend path, which was exempted by IP on an interface that no longer has one. Your isolation silently upgrades itself to a brick, long after anyone is watching. IPv6 neighbor discovery is the same trap with a protocol-layering twist. On IPv4, address resolution is ARP, which sits below IP and isn’t touched by IP-layer filtering — you can block everything and ARP still works. IPv6 folded that function into NDP, which rides on ICMPv6, which is IP traffic. Apply the IPv4 mental model to an IPv6 host and you block neighbor discovery, and v6 connectivity dies — including to your backend, if it’s reached over v6. Two protocols doing the same job at different layers, and the difference is only visible if you know the history. What you refuse to exempt</a></h2> The omissions are more interesting than the inclusions, because each one is a deliberate acceptance of pain: //! it and v6 connectivity dies, including to the backend). Deliberately //! NOT exempted: established flows (an active C2 session must die) and //! DNS (re-resolution would open a tunnel; backend IPs are pinned at //! apply time — a backend failover during isolation needs an operator //! allow_cidrs entry planned ahead).</code></pre> Established flows must die. Almost every firewall ruleset on earth begins with “allow established and related” — it’s the first line of every tutorial, because without it stateful protocols break. But an active C2 session is an established connection. Exempt established flows and you have built a containment that specifically permits the thing you’re containing. The attacker’s live shell survives; only their ability to open new connections is blocked. This is, I’d bet, a real bug in real products. DNS must die, which hurts more than it sounds. Blocking name resolution breaks a lot of software in confusing ways. But DNS is a general-purpose bidirectional channel that every network permits by default — DNS tunneling is a mature, tooled technique, not a curiosity. Leaving a hole for it because “it’s just name resolution” defeats the exercise. The cost is stated plainly rather than hidden: backend IPs are pinned as addresses at the moment of application, so if the backend fails over to a different address while a host is isolated, that host can’t follow, and recovering it needs a CIDR that an operator planned in advance. That’s a genuine operational constraint. Writing it down in the module header, next to the decision that causes it, is what separates an accepted trade-off from a future incident report. Half-applied is worse than not applied</a></h2> Since a partially-applied ruleset is the disaster case, atomicity isn’t a nice-to-have. There must never be an instant where the block-everything rule exists and the exemption doesn’t. What I find genuinely satisfying is that all three platforms provide a way to do this, and all three arrived at it independently. Linux uses an nft -f</code> file load, which applies as a single transaction. macOS uses PF anchors, and the sequencing is spelled out step by step: /// sequenced so the policy only ever becomes visible complete: /// (1) `pfctl -E` enables PF with a reference token — rules without /// enforcement would be silent theater, PF ships disabled on macOS; /// (2) the whole ruleset lands in the anchor via one /// `pfctl -a edr_isolate -f -` load (an anchor load is a single commit, /// the `nft -f` analog); (3) the main ruleset gets the anchor /// attachment only if missing — attaching AFTER the load means the /// first evaluation already sees the full policy;</code></pre> Step 3 is the one to admire. The anchor is populated first, then attached to the main ruleset. Attach first and there’s a window — however brief — where an empty or half-loaded anchor is live. Ordering the operations so the dangerous window never opens is cheaper and more reliable than trying to make the window small. Windows uses the Windows Filtering Platform: /// mutations run inside one FwpmTransaction — commit or nothing, the /// `nft -f` atomicity.</code></pre> Three operating systems, three unrelated APIs, one requirement. When a constraint shows up identically across independent designs, it’s usually telling you something true about the problem rather than about the tools. Rules don’t apply to conversations already happening</a></h2> Here’s the concept I’d most want a reader to take away, because it generalizes far beyond firewalls and it’s counterintuitive until it isn’t. A stateful firewall doesn’t evaluate rules for every packet. That would be expensive. Instead, the first packet of a flow is evaluated against the rules, and the verdict is cached in a state table. Subsequent packets of that flow match the state and skip rule evaluation entirely. Which means: changing the rules does not affect conversations already in progress. You install a perfect block-everything ruleset, and the attacker’s existing session sails straight through it, because that session’s packets never reach the rules at all. /// first evaluation already sees the full policy; (4) `-F states` /// flushes the state table, because PF consults states BEFORE rules — /// a pre-isolation flow's state would ferry C2 packets straight past /// the block.</code></pre> You have to flush the state table. The ruleset is only half the enforcement; the cached verdicts are the other half, and they’re older than your policy. Windows has the same problem and solves it as a side effect, which is a pleasant piece of luck: /// table and macOS anchor exactly. Adding/deleting ALE filters makes /// BFE reauthorize established connections, so live C2 dies at apply /// with no extra work.</code></pre> WFP’s ALE (Application Layer Enforcement) layers reauthorize existing connections when filters change. The state invalidation is automatic. The general shape — a policy change must invalidate the caches that were built under the old policy — is everywhere once you look. Session tokens issued before a permission was revoked. Connection pools holding credentials rotated an hour ago. CDN caches serving content you’ve since made private. In every case the new rule is correct and completely bypassed, because something old is short-circuiting the check. Ask, always: what already decided this, before I changed the rules? Living in someone else’s house</a></h2> An enterprise machine’s firewall is not yours. It has corporate policy, maybe a VPN client, maybe firewalld</code>, maybe Docker rewriting rules whenever a container starts. You are a guest. The pattern that makes this survivable is an owned namespace: /// The agent-owned nftables table (Linux). Created, flushed, and deleted /// by name only — other tables (firewalld, docker) are never touched. pub const TABLE_NAME: &str = "edr_isolate";</code></pre>/// The agent-owned PF anchor (macOS) — the same owned-namespace idea as /// the Linux table: loaded, listed, and flushed by name only. The main /// ruleset is touched exactly once, to attach the anchor reference /// (see render_pf_main_ruleset), never to edit other rules.</code></pre> Windows gets the same treatment with a fixed provider and sublayer GUID. In all three cases the rule is: everything the agent creates lives under one name it owns, it only ever manipulates things under that name, and cleanup is “delete my container” rather than “remove the rules I think I added.” “Touched exactly once” is the discipline that makes it work. There’s one unavoidable interaction with shared state — the anchor has to be referenced from the main ruleset or it’s never consulted — and that single touch is called out explicitly, so the exception stays an exception. This is just namespacing, the same instinct that gives you Java packages and Kubernetes namespaces. It’s worth naming as a general technique for modifying systems you don’t own: claim a namespace, confine yourself to it, and make removal a single operation. Deciding what a reboot means</a></h2> One more decision, easy to skip past, that is actually a policy question disguised as an API flag: /// `nft -f` atomicity. The engine session is the default non-dynamic /// one and no filter carries FWPM_FILTER_FLAG_PERSISTENT: containment /// outlives the agent process but not a reboot, matching the Linux /// table and macOS anchor exactly.</code></pre> Should isolation survive a reboot? Yes is defensible: the machine was compromised, containment should be durable, and an attacker who can reboot shouldn’t be able to escape it. No is what’s implemented, and the reasoning is about recoverability. Isolation that persists across reboots means a mistake — a bad rule, a resolution failure, a mis-clicked host — needs manual intervention on the console. Isolation that dies at reboot means the worst case has a recovery path any user can perform, and the agent can simply re-apply it on startup if the host is genuinely still contained. I don’t think there’s a universal right answer; it depends on whether you fear attackers with reboot access more than you fear your own mistakes at 2am. What matters is that the question got asked and answered the same way on all three platforms. WFP filters can be persistent. The agent deliberately doesn’t make them so, specifically to match the Linux and macOS behavior. That’s the discipline that keeps cross-platform systems sane. Every platform offers slightly different capabilities, and the temptation is to use whatever each one provides. Do that and you get three subtly different products wearing one name, and an operator who learned the behavior on Linux is surprised by Windows during an incident. Choosing the same semantics everywhere — even when it means declining a feature — is what makes fleet behavior predictable. Testing the thing you can’t test</a></h2> Last piece, and it’s the one that made this action tractable at all: //! Policy is separated from mechanism so the dangerous decision logic is //! exhaustively unit-tested without ever touching a firewall: //! * `IsolateControl` — the OS primitive (atomically replace / remove //! the containment ruleset), one thin impl per platform. //! * `IsolateExecutor` — the pure guard/rendering logic over any //! `IsolateControl`, driven by a mock in tests.</code></pre> The dangerous part of this feature isn’t the firewall call. It’s the decisions: which exemptions to include, whether the backend resolved, what to do about a malformed CIDR, what the ruleset text should say. All of that is pure computation — inputs to strings — and pure computation can be tested exhaustively, on any machine, in milliseconds, with no root and no risk. So the rendering functions produce ruleset text that gets compared against golden files, and the guard logic runs against a mock. The part that actually touches the kernel shrinks to a thin layer whose only job is to hand a prepared string to nft</code>, pfctl</code>, or BFE. Which means most of this feature’s correctness was established on a Linux laptop, including the Windows and macOS behavior. Not all of it — you still need real drills to learn things like PF’s state-table ordering — but the decision logic, which is where the bricking bugs live, is verified without a firewall in sight. That inversion, and how far you can push it when your target is a kernel you cannot attach a debugger to, returns later in the series. But first, the uncomfortable inverse of everything this series has covered so far: not what the sensors see, but what they structurally cannot. That is the subject of the next essay. The Authority Problem 2026-08-26T00:00:00+00:00 Building an EDR from scratch, essay 4 of 13. Up to this point, everything in this series has been read-only. Sensors observe, the pipeline delivers, the backend stores. If the whole system were compromised tomorrow, the damage would be a serious data breach: an attacker learns what runs on your machines. Now we add the letter R. Response. Kill this process, quarantine that file, cut this host off the network. And the moment you do that, you have built something categorically different. You have deployed, to every endpoint in the organization, a privileged agent that accepts instructions from the network and carries them out. That is a description of a botnet. The only thing separating your security tool from the worst-case incident in your company’s history is the answer to one question: how does an endpoint know that an instruction is legitimate? Get that answer wrong and your EDR becomes the most efficient lateral movement infrastructure an attacker could ask for — pre-installed everywhere, running as root, already trusted, already exempt from your own monitoring. This essay is about the answer. Two questions that look like one</a></h2> The most common way to get this wrong is a confusion so natural that people make it without noticing. The agent connects to the backend over mutual TLS. Both sides present certificates, both verify, the channel is encrypted and authenticated. Excellent. Now a command arrives over that channel. Should the agent execute it? The tempting reasoning: the channel is authenticated, so the message came from the backend, so the message is legitimate. Each step is true. The conclusion does not follow — because “this message came from the backend” and “a human being with authority decided this should happen” are different propositions, and TLS only establishes the first. Say it more carefully: Transport security answers: who am I talking to right now?</li> Authorization answers: who decided this should happen, and were they allowed to?</li> </ul> A compromised backend passes the first test perfectly. It holds the legitimate server key. Every TLS check succeeds. If the agent’s only gate is “did this arrive over the authenticated channel,” then whoever owns the backend owns every endpoint, instantly and totally. And backends get compromised. They’re internet-adjacent, they run a stack of dependencies, they’re administered by humans. Designing as though the central server can never be breached is not a security posture, it’s a hope. Designing for your own compromise</a></h2> So the real design exercise is to assume the backend is compromised and ask what an attacker can then do. The answer this system gives is stated in the first paragraph of backend/src/tasking.rs</code>: //! The backend is a RELAY, never an authority (see proto AgentTasking): it //! decodes each SignedTask payload ONLY to read agent_id (routing) and //! task_id (dedupe/ack) — it never inspects the signature, so it holds no //! signing key and cannot mint a task an agent will run. All verification //! is the agent's (crate::task on the shipper side).</code></pre> Tasks are signed by an operator with an Ed25519 key. The backend never has that key. It cannot create a valid task, cannot modify one without breaking the signature, and — this is the elegant part — cannot even verify one, because it has no reason to. Note the deliberate refusal in “it never inspects the signature.” A backend that verified signatures would be no less secure, but it would blur the architecture: someone would eventually add a “trusted internal task” path for convenience, and the property would quietly die. Making the backend structurally incapable of the check keeps the boundary honest. So what can a compromised backend do? Exactly three things: withhold tasks, delay tasks, and replay tasks it has already seen. That is a real attack surface — a backend that silently drops every quarantine command is a serious problem — but it is a denial surface, not an execution surface. The attacker can prevent responses. They cannot cause them. That distinction is the entire point of the design, and it generalizes: when you’re deciding where to put a security check, ask what each component can do if it’s fully owned. Components that can’t do damage need less protection than components that can. Here, the backend was demoted from a trusted authority to an untrusted courier, and the amount of trust the system needs to place in its most exposed component dropped to nearly zero. Claims versus proof</a></h2> The second structural idea concerns identity, and it has the same flavor: never accept an assertion when you can require a proof. Every message on the wire contains an agent_id</code> field. An agent connecting to the task stream says “I am web-07</code>, give me my tasks.” The obvious implementation reads the field and routes accordingly, and it is trivially exploitable — any enrolled agent can claim to be any other agent and receive its commands. The fix is to treat the field as a claim that must match a proof, and the proof is the client certificate. From backend/src/authn.rs</code>: //! edr-agent://<agent_id> an agent; may publish telemetry and open a //! task stream only as <agent_id> //! edr-operator://<name> an operator; may submit tasks (<name> is //! for audit logging)</code></pre> The certificate carries a URI in its Subject Alternative Name, and the URI scheme is the role. An edr-agent://</code> certificate may open a task stream — but only for the exact agent_id</code> written into its own SAN. An edr-operator://</code> certificate may submit tasks and nothing else. An enrolled agent, even a fully compromised one, cannot submit tasks to other agents, because it doesn’t hold a certificate with the operator scheme. Two details in that module are worth dwelling on because they’re the kind of choice that separates a design from an implementation. The first is why the SAN rather than the Common Name, which is where most people’s instinct goes: //! Identity is a URI Subject Alternative Name, never the Subject CN: a DN //! may carry several CNs in several string encodings (RFC 6125 deprecates //! CN for name binding), while a scheme-scoped URI SAN is unambiguous and //! cannot collide with the DNS SANs of a server or web-PKI cert.</code></pre> A Distinguished Name can contain multiple CN components, in several string encodings, and different libraries will hand you different ones. An identifier that different parsers read differently is not an identifier — it’s a vulnerability with a waiting period. The web PKI learned this expensively enough that RFC 6125 deprecated the practice. A scheme-scoped URI has exactly one reading. The second is what happens when a certificate is ambiguous: //! The leaf must carry EXACTLY one EDR-scheme SAN: zero means the cert //! was not enrolled as an EDR identity, two or more is a misissued cert //! that could act as either — both are refused rather than guessed at.</code></pre> A certificate holding both an agent SAN and an operator SAN is refused outright. Not “prefer the more specific one,” not “take the first” — refused. Because a cert like that can only exist through an enrollment bug or an attack, and in either case the correct response to “I cannot tell what this is” is to stop. Every “pick a sensible default when the input is ambiguous” in a security path is a place where an attacker gets to choose which interpretation you take. Burning a key, not banning a name</a></h2> Certificates get stolen. So there has to be a way to say “that credential is no longer valid,” and the choice of what you revoke turns out to encode a philosophy. //! The list is keyed on the certificate SERIAL (the standard CRL key), //! not the identity: "this key is burned", so recovery is revoke the old //! serial and mint a fresh cert for the same agent_id.</code></pre> Revoking the serial says “this particular key is compromised.” Revoking the identity would say “this host is banned.” They diverge exactly when it matters: a laptop is stolen, you revoke, IT reimages the machine and reissues it. Under serial-keyed revocation, the new certificate for the same agent_id</code> just works, because it’s a different key. Under identity-keyed revocation, you’ve permanently poisoned a hostname and now you’re renaming machines to work around your own security tool. The general form: revoke the credential, not the principal. Compromise is a property of a secret, not of the person or machine that held it. There’s also a nicely pragmatic touch — serials are compared in canonical form, lowercase hex with leading zeros stripped, “so openssl x509 -serial -noout</code> output pastes straight into the file.” Revocation is an emergency procedure performed by a stressed human at an unpleasant hour. A format mismatch that silently fails to revoke is a security control defeated by usability. Meeting the operator where they already are is a security property. A valid signature is not a fresh authorization</a></h2> Now the deepest problem in the essay, and the one I found genuinely surprising. Signatures don’t expire. A message signed today verifies identically in five years — that’s what makes them useful. But it means an attacker who captures a signed message holds something permanently valid. Replay is therefore not an edge case; it’s the natural consequence of how signatures work. Worse, in this system, replay is also completely normal traffic. Delivery is at-least-once, so honest duplicates arrive routinely. The agent cannot treat repetition as an attack, because repetition is the protocol working correctly. shipper/src/task.rs</code> answers with a sequence of checks, each catching what the previous one can’t: //! Every SignedTask must pass, in order: Ed25519 signature over the raw //! payload bytes against the pinned operator key (the backend only //! relays — it can drop or delay tasks, never mint them), payload decode, //! agent binding (a task signed for another host is a cross-host replay), //! freshness of issued_at (temporal replay; the window is symmetric //! because agent clocks skew both ways), and task_id dedupe (at-least-once //! delivery makes benign re-sends normal; the same set stops malicious //! same-window replays).</code></pre> Signature: was this authored by the key holder. Agent binding: was it authored for this host — otherwise a task captured from one machine replays against another. Freshness: was it authored recently, bounding how long a captured task stays dangerous. Dedupe: have I already done this one. Four checks, four distinct attacks, none redundant. And the freshness window is symmetric because agent clocks drift in both directions — a detail that sounds pedantic until a host with a fast clock rejects every legitimate task it receives. Then comes the case that breaks the whole scheme, and it took me a while to even see it. From shipper/src/config.rs</code>: //! The policy lives INSIDE the signed payload, so the backend can withhold //! or delay policy but never mint it. The central hazard is stale-policy //! replay: a captured OLDER policy under a fresh task_id/issued_at is //! still validly signed, so the task-level replay defenses (dedupe, //! freshness) do not cover it.</code></pre> Read that twice. Configuration policy is signed. An attacker captures a policy document from six months ago — one that, say, monitored fewer registry keys, or set a heartbeat interval so long the agent looks alive while blind. They can’t modify it; the signature would break. But they don’t need to. They wrap that genuinely-signed old policy in a fresh envelope with a new task ID and a current timestamp. Every defense passes. The signature is real. The agent binding is right. The timestamp is current. The task ID has never been seen. The agent applies a six-month-old policy, and the attacker has rolled back its configuration using nothing but valid, correctly-signed messages. The envelope was fresh. The content was ancient. Freshness of transmission tells you nothing about currency of intent. The fix has to live in the content’s own semantics: //! freshness) do not cover it. `AgentPolicy.version` monotonicity is the //! identity guard: only version > current applies; version == current with //! identical content is success (goal state met — idempotence under //! at-least-once, surviving even a restart that emptied the result cache); //! equal version with different content, or any lower version, refuses.</code></pre> The policy carries a version, and the agent only ever moves forward. An old policy is refused on the strength of its own contents, no matter how fresh its wrapper. The general lesson is one I now apply everywhere: a signature proves authorship, not currency. If a document’s age matters to its safety, the document itself must say how old it is, in terms the recipient can order. You cannot bolt freshness onto the transport and expect it to protect the payload. Notice too the treatment of the equal-version case. Same version, identical content: success. Because under at-least-once delivery the agent will genuinely receive the same policy twice, and the honest question isn’t “did I process this message” but “is the world in the state this message asks for.” Idempotence expressed as goal-state rather than as message-history — which means it survives a restart that wiped every cache. Trusting your own disk, carefully</a></h2> One last move, because it inverts a rule the rest of the system follows strictly. Policy has to survive restarts, so it’s written to disk. Disk is not a trust boundary — an attacker on the host can edit files. What should the agent do at startup if the stored policy fails verification? //! validity. Disk is untrusted storage: tamper means the load refuses and //! the agent runs baked defaults at version 0 — never a startup failure, //! because bricking the sensor over a corrupt config file trades //! availability for nothing (an attacker with store write access already //! owns the host).</code></pre> Everywhere else in this system, bad security configuration means fail to start. Here it means fall back to defaults and keep running. The reasoning is a threat-model argument rather than a taste argument: the attacker who can corrupt that file already has host write access. Refusing to start doesn’t deny them anything — it just kills your telemetry, which is a gift. Running on safe defaults keeps the sensor reporting, from a host you now have reason to look at closely. The same header notes that load deliberately skips the freshness check, “because freshness gates admission over the NETWORK; a policy this agent itself admitted and persisted is durable state, not a new admission.” Which is exactly the transport-versus-content distinction again, seen from the other side: freshness was always a property of accepting a message, never of holding a fact. What authority actually requires</a></h2> Strip it down and the same principle produces every decision here. Authority must be provable at the point of action. Not asserted at the point of transmission, not implied by the channel, not inherited from a component’s position in the architecture. The agent — the thing that will actually terminate the process — has to be able to verify, from the message in front of it and a key it already holds, that someone with the right to order this ordered it, for this host, recently, and that it hasn’t already been done. Everything else follows. The backend is a relay because a courier needs no authority. Identity comes from certificates because claims need proofs. Revocation targets serials because compromise is a property of secrets. Policy carries a version because signatures don’t age. And the payoff is that you can say something concrete about the worst case. If the backend falls, the attacker gets to see telemetry and to withhold responses. They do not get to run commands on ten thousand machines. For a system with this much power distributed this widely, being able to state that bound — and point at the code that enforces it — is the difference between a security tool and a liability. Next comes the action with the largest blast radius of all: cutting a machine off the network, over the network, without cutting the connection you would need to undo it. Naming Things That Move 2026-08-19T00:00:00+00:00 Building an EDR from scratch, essay 3 of 13. An analyst looks at an alert and clicks kill this process. Between that click and the signal, a few hundred milliseconds pass. In those milliseconds the process exits on its own — it was a short-lived script, it was always going to exit — and the kernel, being a good steward of a small resource, hands its process ID to the next thing that asks. On a busy machine that might be a database backend, a session manager, or the SSH daemon holding the connection your incident responder is sitting in. The signal arrives. It is delivered, correctly and successfully, to something that has nothing to do with the alert. This is not an exotic race. It’s the ordinary consequence of a fact so familiar that it hides in plain sight: a PID is not an identity. It’s a name, drawn from a small pool, that gets recycled. Endpoint telemetry is full of this. Almost everything you want to talk about is short-lived, and almost every name you have for those things is either recycled, mutable, or both. Getting this wrong doesn’t produce a crash — it produces a system that is confidently, quietly, wrong about which thing it’s discussing. This essay is about the general shape of the problem and the handful of moves that solve it. What makes a name an identity</a></h2> Three properties separate a usable identity from a mere label: Unique at any instant. Two live things never share it.</li> Stable for the lifetime of what it names. It doesn’t change underneath you.</li> Non-recycled across lifetimes. Once a thing is gone, its identity is never issued to a different thing.</li> </ul> PIDs have the first two and spectacularly fail the third. Filesystem paths fail the second and the third — a path is a lookup, resolved fresh every time you use it, and someone else can change what it resolves to between your two uses. Names that fail the third property are the dangerous ones, because failure looks like success: you get an answer, it’s just about a different object than you meant. The recurring fix, in every case below, is the same: pair the unrecycled name with something that pins down which lifetime you meant. The pair that doesn’t recycle</a></h2> For processes, the second half of the pair is the start time. PIDs cycle; the combination of a PID and the moment that PID began doesn’t, because time doesn’t run backwards. The kill action in shipper/src/kill.rs</code> builds its entire safety story on this: //! PID reuse is the whole problem (see proto KillProcessAction): the task //! carries the intended victim's start time, and a kill proceeds only if the //! LIVE process at that pid has a matching start time (within tolerance). //! A recycled pid always starts later than the one the operator saw, so it //! never matches — the guard refuses instead of killing an innocent process.</code></pre> The operator’s task doesn’t say “kill PID 4823.” It says “kill PID 4823, which started at 14:22:07.431.” Before signalling, the agent reads the start time of whatever is currently living at 4823 and compares. Mismatch means the thing you meant is already gone, and the correct action is to refuse. Note the asymmetry that makes this airtight: a recycled PID always starts later than the one the operator observed. The comparison can never be fooled by a reused PID, because reuse only ever moves the timestamp forward, past any plausible measurement error. Which is what licenses something that would otherwise look sloppy: /// How far a live process's start time may differ from the requested one and /// still count as the same process. Absorbs the observation-vs-start skew and /// cross-platform rounding; it does NOT need to distinguish reuse, since /// reuse moves the start time by far more than any measurement jitter. const DEFAULT_TOLERANCE: Duration = Duration::from_secs(2);</code></pre> A fuzzy comparison in a security guard should make you nervous. It doesn’t here, and the comment explains exactly why: the tolerance absorbs jitter, and the threat it defends against operates at a completely different magnitude. Nobody’s PID gets recycled within two seconds and lands within two seconds of the original start time. The signal and the noise live orders of magnitude apart, so a sloppy-looking threshold is in fact a precise one. That’s a reusable habit for any threshold you’re tempted to add: don’t ask “is this value right,” ask “what’s the ratio between what I’m tolerating and what I’m detecting.” If it’s small, you don’t have a threshold problem, you have a design problem. There’s a subtler trap underneath, and it’s the kind of thing that only shows up in production. A start time is meaningless without knowing which clock produced it. Linux offers several — CLOCK_REALTIME</code> jumps when NTP corrects it, CLOCK_MONOTONIC</code> doesn’t count suspend, CLOCK_BOOTTIME</code> does. If the sensor stamps the event from one clock family and the kill guard reads the live process from another, your comparison drifts by whatever the difference happens to be that day, and your guard starts refusing legitimate kills after an NTP correction. The sensor here reads task->start_boottime</code> and applies the REALTIME−BOOTTIME offset precisely so both sides of the comparison end up in the same family. Two numbers being “a timestamp” is not enough for them to be comparable. A path is a question, not an answer</a></h2> Files have the same disease in a more virulent form. /tmp/payload.bin</code> is not a file. It’s a query — instructions for traversing directories to find an inode — and it’s re-evaluated every single time you use it. Between your stat()</code> and your open()</code>, anything with write access to any directory along that path can change the answer. That’s the TOCTOU (time-of-check to time-of-use) family, and for a security tool it’s not a theoretical concern: quarantining a file is precisely the moment an attacker most wants to redirect you. Imagine quarantine done naively. Hash the path to confirm it’s the malicious file. Then move the path into the quarantine store. In between, the attacker replaces the path with a symlink to /etc/shadow</code>. You have just used your elevated privileges to hash a known-bad file and quarantine a critical system file. Your logs will say the operation succeeded. The fix is to stop using the name after the first lookup. From shipper/src/quarantine.rs</code>: //! * `FileControl` — the OS primitives (pin a file to one inode/object,</code></pre> and, concretely: .custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC)</code></pre> O_NOFOLLOW</code> refuses to traverse a final symlink — so the swap-a-symlink attack fails at the door rather than succeeding silently. And once that open</code> returns, the file descriptor is the identity. It refers to one inode. Renaming the path doesn’t move it, deleting the path doesn’t invalidate it, and no amount of directory manipulation can make it point somewhere else. Every subsequent operation goes through the descriptor: // fchmod on the pinned inode: a path swap cannot redirect it.</code></pre> fchmod</code> rather than chmod</code>. fstat</code> rather than stat</code>. The pattern is consistent, and once you see it you’ll see it everywhere in well-written systems code: resolve the name exactly once, then work with the handle. The handle is the identity; the path was only ever a way to find it. The same code then does something I want to highlight because it’s a habit worth stealing: // instant; prove the blob IS our pinned inode.</code></pre> Having moved the file into the store, it verifies that the object now in the store is the same object it pinned — and rolls back if not. It doesn’t assume the operation it just performed did what it intended. In a system where an adversary is actively trying to make your operations mean something other than what you meant, that distinction between “I issued the correct call” and “the correct thing happened” is the whole game. Identity by content</a></h2> Sometimes you don’t want to identify a file at all — you want to identify a payload, wherever it happens to live. The same malware dropped at ten paths on ten machines is one thing worth tracking, not ten. That’s what a cryptographic hash gives you: an identity derived from content rather than location. Two files with the same SHA-256 are the same payload, no matter what they’re named or which host they’re on. The schema in backend/schema/001_events.sql</code> makes a small decision here that reveals something about designing for consumers rather than for storage: -- agent-computed sha256 of the executable image, lowercase hex; -- '' = not hashed (EXIT events, vanished file, over the size cap, -- or a sensor without enrichment). Hex, not raw: IOC lists join -- directly in SQL.</code></pre> Raw bytes would be half the size. Hex is stored anyway, because every threat intelligence feed on earth distributes hashes as lowercase hex, and an analyst with a list of indicators should be able to write a join and get an answer — not first write a conversion. Sixteen extra bytes per row buys frictionless use of the column for its actual purpose. Storage efficiency that makes data harder to use is a false economy. Note also that ''</code> means “no hash,” and the comment enumerates the reasons why. Which brings us to the theme that turns out to matter most of all. Absence is not zero</a></h2> If a value can be missing, its absence must be representable and must not be confusable with any real value. This sounds obvious. It is violated constantly, usually by accident, usually via a language’s default value. The uid column is the sharpest example: -- acting process's real/effective uid; 4294967295 = not captured -- (pre-uid WAL replay or platform without parity). uid 0 IS root, -- so absence must never look like it — rules must exclude the -- sentinel explicitly.</code></pre> The obvious default for a missing integer is 0. For a uid, 0 is root. Default an unknown user to 0 and every event from a sensor that couldn’t determine the user now claims to be a root action. Your detection rules — which quite reasonably treat root activity as more interesting — light up on a field that means “we don’t know.” You’d have manufactured a false positive generator out of a language default. The same reasoning appears for timestamps, and the choice of sentinel is neatly argued: -- identity (pids recycle, the pair doesn't). 1970-01-01 00:00:00 = -- not captured (pre-field agent) — no real process starts at the -- epoch, so the zero default is unambiguous, unlike uid.</code></pre> Epoch-0 is safe as a sentinel because no real process starts then. Zero is unsafe for uid because uid 0 is real and important. Identical mechanism, opposite verdicts — and the difference is entirely about the semantics of the domain, not about the type. There is no general rule like “use 0 for missing” or “use -1 for missing.” You have to look at what the values mean and pick something outside the real range. In-kernel or not at all</a></h2> Container attribution is where identity gets genuinely hard, and where the solution is instructive. The naive approach: an event arrives with a PID; read /proc/<pid>/cgroup</code>; parse out the container. It works when you test it. It fails in production for a reason you’ve already seen twice in this essay — by the time you read /proc</code>, the process may be gone, and the PID may belong to something else. Your container attribution is now not just missing but wrong, and short-lived processes are exactly the ones an attacker uses. shipper/src/cgroup.rs</code> refuses that whole approach: //! module turns that number into the runtime's container id. //! //! Why the inode: on cgroup v2 the id the kernel returns equals the cgroup //! directory's inode (its kernfs node id — a full 64-bit ino since 5.5; //! the agent's BTF-enabled floor sits there). So the mapping is a scan of //! /sys/fs/cgroup for the directory whose inode matches — keyed on the //! stable id, never on a pid that may already be gone (the whole reason //! the id is stamped in-kernel rather than re-read from /proc).</code></pre> The eBPF probe calls bpf_get_current_cgroup_id()</code> at event time, in kernel context, while the process is definitionally alive. That number rides along in the event. Resolution to a human-meaningful container id happens later, in user space, at leisure — because unlike the PID, the cgroup id is stable and doesn’t recycle out from under you. This is the general principle, and it’s the most portable lesson in the essay: capture identity at the moment of the event, in the context where it’s unambiguous. Resolve it to something readable whenever you like. Deferring capture is a correctness bug. Deferring resolution is just scheduling. There’s also a small piece of Linux trivia in there that I enjoy far more than I should. The cgroup id equals the directory’s inode number, because cgroups are backed by kernfs and the “id” is just the kernfs node id. That’s not a documented mapping so much as a consequence of how the thing is built — which means the resolver can find the cgroup by scanning for a matching inode, with no parsing and no API. Understanding the implementation gave a cleaner solution than the interface advertised. Making identity durable</a></h2> The final move is to take these carefully-constructed identities and turn them into something you can query, which is where backend/schema/006_process_index.sql</code> comes in: -- Backend-materialized process index: one row per process identity -- (agent_id, pid, start_time) — pids recycle, the pair doesn't.</code></pre> Two events describe a process — its creation and its exit — and they may arrive out of order, or twice, or hours apart if a WAL replayed. The index has to fold them into one row regardless: -- Replay contract: an insert deduplicated by token skips the view too. A -- replay that outlives the token window re-feeds the view, so every -- column is an idempotent fold (min/max) — replaying a byte-identical -- row cannot change the folded state. Out-of-order arrival (exit before -- create, WAL replay) folds the same way: merges are order-blind.</code></pre> Every column folds with min</code> or max</code>. Those operations are commutative, associative, and idempotent — which is the algebraic way of saying that order doesn’t matter, grouping doesn’t matter, and repetition doesn’t matter. Given those three properties, “exit arrived before create” and “this batch replayed four times” stop being cases you handle and become cases that cannot arise. And the sentinel discipline from earlier shows up again, now doing real work: -- uid/euid fold with min so the 4294967295 absence sentinel loses to -- any captured value — and 0 still IS root, never absence.</code></pre> Because absence is 4294967295</code> — the largest possible u32 — folding with min</code> means any real value beats it automatically. The choice of sentinel and the choice of fold operator were made together, and the result is that “prefer a real value over a missing one” needs no conditional logic at all. It’s a consequence of arithmetic. That’s the thing I’d point at if someone asked what good data modeling looks like. Not the absence of special cases, but special cases that dissolve into the structure so completely that there’s nothing left to get wrong. The tax and the return</a></h2> Look at what all this cost. Every process reference carries a timestamp. Every file operation carries a descriptor. Every event carries a cgroup id stamped in kernel context. Every nullable column carries an explicitly chosen sentinel and a comment defending it. It’s more fields, more plumbing, more to explain. What it buys is that a whole category of bug becomes impossible rather than unlikely. Not “we tested it and didn’t see the race” — actually impossible, because a recycled PID cannot match a start time, a descriptor cannot be redirected by a rename, and a min</code>-fold cannot be disturbed by a replay. Which matters especially here, because the next thing this system does with these identities is act on them: terminate the process, quarantine the file, cut the machine off the network. Once you’re taking destructive action on a remote host, “we’re pretty sure this is the right process” isn’t a standard anyone can live with. And that raises a question this essay has been carefully avoiding: who gets to decide that the action happens at all? That is the authority problem in the next essay. The Shape of a Pipeline 2026-08-12T00:00:00+00:00 Building an EDR from scratch, essay 2 of 13. Once the three sensors from the last essay are producing events, a suspiciously boring problem appears: get the events to a server. I say suspiciously because everyone underestimates it, including me. It sounds like plumbing. It reads like a weekend of work. And then you sit down to write it and discover that you are re-deriving, from scratch, the same structure that sits inside Kafka, inside PostgreSQL’s WAL, inside journald</code>, inside every message queue you’ve ever used, and — if you go back far enough — inside TCP. That convergence is the interesting thing. This essay is about why the shape is nearly forced, which decisions inside it are actually free, and the one place where I think the conventional answer is a lie people tell in marketing copy. Three things you can do when you can’t keep up</a></h2> Start with the only situation that matters. Data is arriving faster than you can move it along. You have exactly three options, and there is no fourth: Drop it. Throw away data. Fast, bounded, and lossy.</li> Buffer it. Hold it somewhere. Preserves data, and converts a throughput problem into a memory problem, which is a worse problem because it kills the whole process instead of one event.</li> Push back. Refuse to accept more until you’ve caught up, so the pressure propagates upstream to whoever is producing.</li> </ol> Every stage of every pipeline picks one. A system’s character is entirely determined by which stage picks which, and the failure you get in production is always the failure of whichever stage picked drop. Here is the thing that took me longest to internalize: backpressure is not a way to avoid dropping data. It’s a way to choose where the dropping happens. If the producer is a kernel probe watching a machine that is compiling a large C++ project, the events exist whether or not anyone is ready. Pressure has to terminate somewhere. Pushing back moves the overflow up the chain; it doesn’t delete it. So the real design question isn’t “how do we never lose events.” It’s where do we want the loss, and can that place be honest about it? Putting the loss where it can be counted</a></h2> The answer this agent settles on is stated as an invariant, and it’s the single most load-bearing sentence in the codebase: the kernel-adjacent ring is the only place allowed to drop events — and it counts what it drops. Both halves matter, and the second half is the one people forget. The kernel ring is the right place to lose data for unglamorous reasons. It is a fixed-size allocation that exists whether you like it or not. It sits at the boundary where the producer genuinely cannot be slowed down — you are not going to make a process-creation event wait while your userspace agent catches up, because that would mean the security tool is now a performance bottleneck in fork()</code>, which is how you get uninstalled. And critically, the kernel gives you a drop counter for free, because it knows exactly how many times it failed to write. That last property is what turns a data loss into a known data loss, and the difference between those two is the whole ballgame. A pipeline that loses 4% of events and reports “everything is fine” is worse than useless — it’s actively misleading, because an analyst will interpret the absence of an event as evidence that nothing happened. A pipeline that loses 4% and says so lets you make a decision: add capacity, tune the filters, or accept the gap knowingly. Everything downstream of the ring, therefore, is forbidden from dropping. It must push back instead. You can watch the invariant being enforced in one line in shipper/src/linux_sensor.rs</code>: //! events into the shipper's bounded channel. blocking_send provides the //! backpressure link: a stalled shipper stops this loop and overflow is</code></pre> blocking_send</code>, not try_send</code>. The distinction is four characters wide and it’s the entire policy. try_send</code> would return an error when the channel is full, and the natural thing to write next is a continue</code> — at which point you have quietly created a second drop site, uncounted, invisible, in user space. blocking_send</code> instead parks the reader thread, which stops draining the kernel ring, which pushes the overflow back to the one place designed to absorb it. A bounded channel with a blocking send is the backpressure. There’s no framework involved. The design lives in a function name. Why “exactly once” is a marketing term</a></h2> Now the events need to cross a network to a backend, and here we meet the most persistently misunderstood problem in distributed systems. You send a batch. You don’t get a response. What happened? Maybe the batch never arrived. Maybe it arrived, was written to the database, and the acknowledgement was lost on the way back. From where you sit those two situations are indistinguishable — the evidence available to you is identical — and yet they demand opposite actions. Resending is correct in the first case and creates a duplicate in the second. There is no protocol that fixes this. You cannot add a handshake, because the acknowledgement of the handshake can also be lost, forever, turtles all the way down. This is the Two Generals problem and it is not an engineering gap, it’s a proof. So “exactly once delivery” does not exist. What exists — and what every mature system actually does, whatever the brochure says — is at-least-once delivery plus idempotent processing. Send until acknowledged, accept duplicates as normal, and make the receiver’s handling of a duplicate a no-op. The system’s observable behavior is exactly-once. The delivery never was. The receiving half is a single line in backend/src/lib.rs</code>: .with_option("insert_deduplication_token", &token);</code></pre> The token is <agent_id>:<sequence></code>. ClickHouse remembers recently-seen tokens and silently discards an insert that repeats one. The agent is free to be dumb and persistent; the backend makes repetition harmless. The constraint nobody warns you about</a></h2> Here’s where it stops being textbook, and it’s my favorite thing in the whole pipeline. Once you’ve decided that identity is (agent_id, sequence)</code> attached to a batch, you have — without necessarily noticing — signed a contract that constrains you forever after. From shipper/src/queue.rs</code>: //! Because the backend dedupes on (agent_id, sequence), recovery must resend //! the exact batches as originally constructed — re-batching the same events //! under an old sequence would get them silently deduped away. So the WAL //! stores whole batches, and replay bypasses the batching logic entirely.</code></pre> Read that failure mode carefully, because it’s beautifully nasty. Suppose after a crash you replay your unsent events and, being tidy, you repack them into fuller batches. Batch 100 originally held events A and B; now batch 100 holds A, B, C, and D. You send it. The backend sees token agent:100</code>, recognizes it, and drops the insert as a duplicate. Events C and D are gone. Not delayed — gone, permanently, and with a successful acknowledgement returned to the agent. No error is logged anywhere. Your metrics are green. The data is missing. The lesson generalizes well beyond this codebase: when you make an identifier the basis of deduplication, you freeze the boundaries of whatever that identifier names. The batch stopped being a transient performance optimization the moment it became an identity. It’s now a durable structure you are not allowed to reorganize. Which is exactly why the write-ahead log stores whole encoded batches, and why replay is a separate code path that walks the disk rather than re-entering the batcher. The pipeline has a “no re-batching” rule the way a database has a “no rewriting history” rule, and for the same underlying reason. What durability actually costs</a></h2> The write-ahead log itself is the oldest trick in the book: write the data to disk and force it to physical media before you tell anyone you have it, and reclaim the space only after someone else has taken responsibility. The header spells the ordering out: //! Durable on-disk queue: a write-ahead log of EventBatch records so events //! survive a shipper restart. A batch is appended and fsync'd *before* the //! first send attempt and reclaimed only after the backend acks it.</code></pre> fsync</code> before the first attempt, reclaim after the ack. Those two bookends define a window in which the data exists in two places at once, and that redundancy is the entire point. Every durable system in existence is some variation on this trick. The part that separates a real WAL from a toy one is what happens when you find damage on startup: //! missing cursor is safe: everything replays and the backend dedupes. A //! torn record at the tail of the *last* segment is a normal crash artifact //! and is truncated away; corruption anywhere else is a hard error.</code></pre> This is a small masterclass in thinking about failure, so let me unpack the three cases separately. A torn record at the tail of the last segment is what a crash looks like. You were in the middle of appending; the machine lost power; the final record is half-written. That isn’t corruption, it’s the expected physical signature of an interruption. Truncate it and move on — the events in it were never acknowledged to anyone, so nothing is lost that anyone believes exists. A damaged record in the middle is a different animal entirely. Nothing in normal operation writes there. Its presence means a bad disk, a filesystem bug, or something modifying your files. Recovering from that by skipping the bad record would be the worst possible choice: you’d silently drop unknown data and continue as if healthy. So it’s a hard error, loudly. A missing ack cursor is safe to lose because losing it can only cause over-delivery — you replay batches the backend already has, and the deduplication makes that free. The cursor is an optimization, and it’s designed so that its failure mode lands on the harmless side. That’s the design principle underneath all three: for every piece of state, ask which direction its corruption pushes you, and arrange for the cheap direction to be the one that happens. Redundant work: fine. Silent data loss: never. Decoupling, and the luxury of a slow consumer</a></h2> The last structural idea is visible in the module header of shipper/src/ship.rs</code>: //! Batching gRPC publisher, decoupled into a batcher and a sender that //! meet at the durable queue: the batcher drains the channel and appends //! every batch to the WAL; the sender reads batches back off disk and //! delivers them at-least-once (same sequence until acked). A dead //! backend therefore parks only the sender — telemetry keeps flowing to //! disk up to Config::max_queue_bytes, past which the batcher stops //! consuming and backpressure walks channel -> reader thread -> kernel //! ring (still the only place designed to drop).</code></pre> Two independent components that never speak to each other; they share only a file. The batcher’s job is to get events onto disk. The sender’s job is to get things off disk and onto the network. Neither knows the other exists. The consequence is that a backend outage — which is the common case, not the exotic one; deploys happen, networks partition — degrades gracefully along a predictable path. The sender parks. The batcher keeps writing. Telemetry accumulates on disk, which is cheap and enormous. Only when the disk cap is reached does pressure resume its walk up the chain toward the kernel ring. There’s a nice systems-thinking observation buried in there. Introducing an intermediate storage layer didn’t just add buffering — it changed the failure mode from correlated to independent. Without the WAL, a network problem immediately becomes a kernel-ring problem, because the whole chain is rigidly coupled and pressure transmits instantly. With it, the two failures are separated by hours of disk capacity. Same components, one extra layer, and an outage that would have cost you telemetry now costs you nothing but disk space. Boring on purpose</a></h2> Look at what the finished pipeline actually is. A bounded channel. A blocking send. A size-and-time batcher. An append-only log with CRCs. A retry loop. A dedupe token. Not one of those is clever. There is no novel algorithm anywhere, nothing you couldn’t explain to a competent engineer in a minute each. And that’s the achievement rather than an apology for it — because this layer sits between kernel code that can crash a machine and a detection engine whose correctness depends utterly on its input being complete. It is the part of the system that must be reasoned about without effort, at 3am, by someone who didn’t write it. The interesting decisions were all about placement and honesty. Put the lossy stage where loss is unavoidable and countable. Put durability before the first send, not after. Keep failures independent by inserting a layer between them. Make every piece of state fail toward redundant work rather than toward silence. Do that, and what’s left is boring. Which means the thinking worked. The next problem doesn’t yield to boring, though. Once you have a complete, durable stream of events, you have to say what each one happened to — and on a running machine, the things you’re naming keep disappearing and the names keep getting reused. That identity problem is where the next essay begins. How an Operating System Decides to Let You Watch It 2026-08-05T00:00:00+00:00 Building an EDR from scratch, essay 1 of 13. Here is an awkward fact about endpoint security software: the program you want to install and the program you want to catch are asking the operating system for the same things. Both want to see every process that starts. Both want to know when files are written. Both want to watch the network. Both want to survive reboots, resist termination, and run before the user logs in. If you squint at the capability list of an EDR agent and the capability list of a competent rootkit, they are the same list. The difference is intent, and intent is not something a kernel can typecheck. So every operating system has had to answer a question that has no clean technical answer: how do we let someone watch everything, without letting everyone watch everything? Windows, Linux, and macOS each answered it differently. Not slightly differently — differently in kind, from different first principles, with different ideas about where trust comes from. And because the answer is a trust model rather than an API, it propagates upward. It shapes what your sensor can observe, what it costs to be wrong, and — this is the part I did not expect — when in the software lifecycle your mistakes become visible. That last one turned out to be the most useful lens I found. Let me get there the long way. Three answers to one question</a></h2> I’ll name the three models before defending them, because having the words makes the rest of this readable: Windows trusts by ceremony. You may enter the kernel. First, perform the rituals.</li> Linux trusts by proof. You may not be trusted, so submit your program for mechanical verification and it will run only if safety is provable.</li> macOS trusts by brokerage. You may not enter the kernel at all. Apple will go inside, look around, and tell you what it saw.</li> </ul> Each is internally coherent. Each is, I think, defensible. And each one extracts a completely different tax. Windows: the bet on ceremony</a></h2> The Windows model is the oldest and the most straightforward: security software runs in the kernel, with the same privileges as the kernel, because that is where the information is. There is no sandbox. If your driver dereferences a bad pointer at DISPATCH_LEVEL</code>, the machine bugchecks, and every user of that machine finds out at once. Given that, Windows does not try to make your code safe. It tries to make sure that only code someone was willing to sign their name to gets to be unsafe. Trust is established out of band — a signing certificate, an attestation process, a company with a legal identity — and enforced at load time. What I find architecturally interesting is how thoroughly that ceremony is woven into the APIs themselves. It isn’t a single check at the door. The privileged APIs individually refuse to work for a driver that hasn’t performed the ritual. From the header comment in windows/driver/edr_driver.c</code>: * MUST be linked /INTEGRITYCHECK or PsSetCreateProcessNotifyRoutineEx * fails with STATUS_ACCESS_DENIED (0xC0000022).</code></pre> /INTEGRITYCHECK</code> is a flag that marks the image as requiring signature verification at load. Note what’s happening: the process-notification API won’t talk to you unless your binary has asked to be held to a stricter standard. It’s an opt-in that gates a capability. You can’t accidentally get process-creation visibility; you have to declare, in your link line, that you intend to be the kind of code that is checked. The other half of the Windows model is stranger and more human. Filesystem minifilters — the mechanism for seeing file operations — stack in an order determined by a number called an altitude: * reg add HKLM\SYSTEM\CurrentControlSet\Services\<svc>\Instances\EdrAgentInstance * /v Altitude /t REG_SZ /d 385100 /f * (385100 sits in the FSFilter Activity Monitor range 360000-389999;</code></pre> Think about what an altitude actually is. Every filter driver on the system sits at a point in a global, numeric, totally-ordered namespace. Antivirus lives around 320000. Activity monitors live in 360000–389999. Encryption drivers live lower, so they see plaintext. Replication drivers live higher. Your position determines whether you observe a write before or after some other vendor’s product transforms it — which means it determines whether your product works in the presence of theirs. And the allocation of these numbers is not enforced by the kernel. It is administered by Microsoft, by request, as a registry of reserved ranges. It’s a social protocol wearing a REG_SZ</code>. I love this as a design artifact, because it’s an honest admission that the hard problem in kernel-mode security software was never memory safety — it was coordination between mutually distrustful vendors who all want to be first. Windows solved it with a phone book. My driver uses 385100 because it’s in the right neighborhood and nobody else on my lab machine is home; a shipping product would have that number assigned to it, in writing. The payoff for all this ceremony is that Windows gives you the sharpest semantics of the three platforms. When PsSetCreateProcessNotifyRoutineEx</code> fires, you are inside process creation, holding structures the kernel is using, before the first instruction of the new image runs. Nothing is inferred. Nothing is a heuristic. You are standing in the room where it happens. The tax is that the room has no railings. Linux: the bet on proof</a></h2> Linux, arriving at the same problem two decades later and with a very different culture, made close to the opposite bet. Rather than deciding who may run unsafe code in the kernel, it asked whether the code could be made safe enough that the question doesn’t matter. eBPF is the answer, and the verifier is the mechanism. You submit a program in a restricted instruction set; a static analyzer in the kernel simulates every reachable path and refuses to load anything it can’t prove terminates, stays in bounds, and touches only what it’s allowed to touch. If the proof succeeds, the program is JIT-compiled and runs at native speed in kernel context. If it fails, you get a rejection and a stack trace, and nothing happens to the machine. This inverts the cost structure completely. On Windows, the cost of a mistake is paid at runtime, by the customer, as a blue screen. On Linux, it’s paid at development time, by me, as an argument with a program that cannot be talked out of its position. What makes this more than a sandbox story is where the verifier’s constraints show up in the design of the sensor. Consider a snippet from the exec probe in linux/bpf/edr_probes.bpf.c</code>: /* Stack copies: key/value pointers into the ringbuf reservation are * a verifier gamble across kernels; the stack is always legal. */ pid = ev->pid; ppid = ev->ppid; bpf_map_update_elem(&exec_ppid, &pid, &ppid, BPF_ANY);</code></pre> There is a perfectly good copy of pid</code> sitting in the ring buffer reservation two lines up. Copying it to the stack first is redundant work in any normal program. But whether the verifier accepts a pointer into a ring-buffer reservation as a map key varies across kernel versions, and a program that loads on my laptop and is rejected on a customer’s 5.10 kernel is a support incident, not a bug report. So the code carries an extra assignment forever, as a compatibility hedge. That’s the real shape of eBPF development. You are not writing C for a machine; you are writing C for a proof checker whose exact reasoning powers differ between the kernels your program must run on. The verifier is a compiler audience with opinions. The second half of the Linux answer addresses a different problem, and it’s one Windows never had to solve. Kernel data structures on Linux are not a stable ABI. struct task_struct</code> is laid out differently in every kernel build, sometimes differently between two distributions of the same version. A sensor that reads task->start_boottime</code> at a hardcoded offset is correct exactly once. CO-RE — Compile Once, Run Everywhere — fixes this by shipping type information with the kernel (BTF) and having the loader relocate every field access at load time against the layout the running kernel actually has. BPF_CORE_READ(bprm, filename)</code> compiles to a relocation entry, not an offset. The kernel tells you where filename</code> lives on this machine, today. Step back and notice what those two mechanisms together represent. The verifier says: I don’t trust your code, so prove it. CO-RE says: I don’t trust your assumptions about me, so ask at runtime. Linux replaced the vendor-trust relationship with a pair of machine-checked contracts. That’s a genuinely different theory of how to make a kernel extensible, and it puts the burden on the extension author rather than on a certificate authority. macOS: the bet on brokerage</a></h2> Apple looked at kernel extensions and concluded the entire category was a mistake. Not “risky.” Not “in need of better tooling.” A mistake — third-party code in the kernel was a permanent, unfixable source of instability and compromise, and the right move was to remove the capability from the platform. Kexts were deprecated, then walled off behind boot-policy changes that a normal user will never make. But security software still needs the data. So Apple built Endpoint Security: Apple’s own code sits at the kernel hook points, and your agent — running in user space, as an ordinary process — subscribes to a stream of events over a framework API. You get the semantics of kernel instrumentation with the failure domain of a user process. If your ES client segfaults, your ES client segfaults. The machine keeps running. Here’s the subscription from the shim in macos/esf/edr_esf_shim.c</code>: es_event_type_t events[] = { ES_EVENT_TYPE_NOTIFY_EXEC, ES_EVENT_TYPE_NOTIFY_EXIT, ES_EVENT_TYPE_NOTIFY_CREATE, ES_EVENT_TYPE_NOTIFY_CLOSE, ES_EVENT_TYPE_NOTIFY_TRUNCATE, ES_EVENT_TYPE_NOTIFY_UNLINK, ES_EVENT_TYPE_NOTIFY_RENAME, ES_EVENT_TYPE_NOTIFY_MMAP,</code></pre> That reads like a config file. On Windows the equivalent capability is a signed kernel driver with a registered altitude and a callback running at elevated IRQL. On macOS it’s an array of enum values passed to a library function from a process that could, in principle, be a shell script’s child. The complexity didn’t vanish — Apple absorbed it, and now maintains it on your behalf. The tax is subtle and it comes due in two places. The first is that the broker decides what exists. On Windows or Linux you can, with enough determination, hook something nobody anticipated you’d want. Under ES, if Apple hasn’t modeled an event type, that event is not merely hard to observe — it is unobservable, and your options are a feature request and a wait measured in major OS releases. The interesting flip side is that a broker can also model things the other platforms don’t have words for. Two entries further down that same array: /* Mach task-port acquisition (macOS 10.15 / 12.0). CONTROL + READ * only — INSPECT/NAME are dominated by diagnostics tooling. */ ES_EVENT_TYPE_NOTIFY_GET_TASK, ES_EVENT_TYPE_NOTIFY_GET_TASK_READ,</code></pre> A Mach task port is a capability handle to another process’s address space. Acquiring one is the macOS injection primitive — the thing that stands where OpenProcess</code> + WriteProcessMemory</code> stands on Windows. There is no equivalent event on the other two platforms in this codebase, because Mach ports are a macOS abstraction and Apple chose to model access to them explicitly. The broker can be more expressive than raw hooking, when the broker understands its own OS’s semantics better than an outsider would. The second tax is the one that turned out to matter most in practice, and it’s the reason I ended up thinking about all of this in terms of time. Where your mistakes surface</a></h2> Under Endpoint Security, permission is not a runtime check that returns an error code. Permission is an entitlement: a cryptographic grant, issued by Apple, to a specific developer account, after a review. Your binary carries it in its signature. Development builds can fake this. With System Integrity Protection relaxed and amfi_get_out_of_my_way</code> set, an ad-hoc-signed client with a synthetic entitlement will be accepted by es_new_client</code>, and events flow. Every category in that array above is delivered on my dev VM. Except one. TCC-modification events — notifications that a privacy grant (camera, microphone, full disk access) changed — are gated more strictly than the rest. The client connects. The subscription succeeds. No error is returned anywhere. And no TCC_MODIFY</code> event is ever delivered, because the faked entitlement clears the door but not that particular room. There is no bug to find. The code is right. The only fix is a genuine Apple-issued entitlement, which is a business process, not an engineering one. And that, finally, is the pattern I want to leave you with. Line the three models up by when a mistake becomes visible: </th> Trust model</th> Mistakes surface at</th> Failure mode</th></tr></thead> Windows</td> Ceremony — sign, register, enter</td> Runtime, on the customer’s machine</td> Loud and catastrophic (bugcheck)</td></tr> Linux</td> Proof — submit to the verifier</td> Load time, on your machine</td> Loud and safe (rejection with a trace)</td></tr> macOS</td> Brokerage — Apple looks for you</td> Deployment, at organizational scale</td> Silent — an event class that simply never arrives</td></tr> </tbody></table> Windows fails at the worst time in the worst way, but tells you unambiguously. Linux fails at the best possible time, before anything is at risk, and the price is that you argue with a proof checker for an afternoon. macOS fails at the strangest time — long after the code is written and tested — and the failure is an absence, which is the hardest thing in the world to notice. Every one of these is a coherent engineering position. But they aren’t interchangeable, and this is why an EDR agent cannot have one unified sensor abstraction that “handles platform differences.” The differences aren’t in the API surface. They’re in what the platform believes about you, and belief doesn’t refactor. What can be unified is everything downstream. Once an event exists — a process started, a file was written, a task port was acquired — the questions become identical everywhere: how do you not lose it, how do you name the thing it happened to, who is allowed to act on it. Three sensors, three trust models, one contract from there on. The next essay follows that contract downstream and asks why telemetry pipelines keep converging on the same shape. Building a Mini-EDR Agent - Monitoring Process Creation with ETW in C 2026-03-26T00:00:00+00:00 In my previous post, we built a mini-SIEM lab using Sysmon to gather high-quality telemetry from a Windows host. But how can an endpoint sensor observe process creation in real time without relying on Sysmon? One useful source is Event Tracing for Windows (ETW)</a>, Windows’ general-purpose tracing infrastructure. Production EDR products normally combine several user-mode and kernel-mode telemetry sources; ETW is one source rather than the mechanism behind every product. In this post, we will ditch the pre-built collection tools and write a small EDR-style sensor in C. It will subscribe to the Windows Kernel Process provider and print an event whenever a new process starts. Before diving into the code, let’s look at the three main parts of ETW: Providers - The components generating logs (e.g. the Windows Kernel).</li> Controllers - Applications that start and configure tracing sessions.</li> Consumers - Applications that read events from real-time sessions or .etl</code> files.</li> </ul> Our C program will act as both a controller (creating the trace session) and a consumer (parsing and printing events). Let’s build it step-by-step. Phase 1: Setting up the ETW Session (The Controller)</a></h3> To receive events, we first need to start an ETW trace session and define the GUID of the provider we want to enable. In our case, it is Microsoft-Windows-Kernel-Process</code>, whose GUID is {22FB2CD6-0E7B-422B-A0C7-2FAD1F0E716}</code></a>. #define _WIN32_WINNT 0x0600 #include <windows.h> #include <stdio.h> #include <evntrace.h> #include <evntcons.h> #include <tdh.h> #pragma comment(lib, "tdh.lib") #pragma comment(lib, "advapi32.lib") static const GUID ProviderGuid = { 0x22FB2CD6, 0x0E7B, 0x422B, { 0xA0, 0xC7, 0x2F, 0xAD, 0x1F, 0xD0, 0xE7, 0x16 } };</code></pre> In main</code>, we configure the EVENT_TRACE_PROPERTIES</code> structure. EVENT_TRACE_REAL_TIME_MODE</code> makes events available to a real-time consumer instead of writing them to a disk file. // Setup session properties TRACE_PROPERTIES sessionProps = { 0 }; sessionProps.Properties.Wnode.BufferSize = sizeof(TRACE_PROPERTIES); sessionProps.Properties.Wnode.Flags = WNODE_FLAG_TRACED_GUID; sessionProps.Properties.Wnode.ClientContext = 1; sessionProps.Properties.LogFileMode = EVENT_TRACE_REAL_TIME_MODE; // Crucial for EDR sessionProps.Properties.LoggerNameOffset = offsetof(TRACE_PROPERTIES, SessionName); // Start the trace session (Requires Administrator privileges!) ULONG status = StartTraceW(&g_hSession, g_SessionName, &sessionProps.Properties);</code></pre>Phase 2: Subscribing to Kernel Events</a></h3> Starting a session is not enough. We must enable our provider for that session with EnableTraceEx2</code>. status = EnableTraceEx2( g_hSession, &ProviderGuid, EVENT_CONTROL_CODE_ENABLE_PROVIDER, TRACE_LEVEL_VERBOSE, 0x10, // WINEVENT_KEYWORD_PROCESS 0, 0, NULL );</code></pre>Phase 3: Consuming the Events (The Consumer)</a></h3> Now that events are flowing into the session, we need to read them. We set up an EVENT_TRACE_LOGFILEW</code> structure, point it to our session name, and provide an EventRecordCallback</code> function that ETW calls for each event. Finally, we call ProcessTrace()</code>. This function blocks while it processes events and invokes our callback, until the trace is stopped. EVENT_TRACE_LOGFILEW logFile = { 0 }; logFile.LoggerName = g_SessionName; logFile.ProcessTraceMode = PROCESS_TRACE_MODE_REAL_TIME | PROCESS_TRACE_MODE_EVENT_RECORD; logFile.EventRecordCallback = EventRecordCallback; // The function that handles data TRACEHANDLE hTrace = OpenTraceW(&logFile); printf("Listening for Process Starts... (Press Ctrl+C to stop)\n"); status = ProcessTrace(&hTrace, 1, NULL, NULL); // Blocks and processes events</code></pre>Phase 4: Parsing the Payload with TDH</a></h3> Raw ETW data is an unformatted binary payload. Parsing it manually is error-prone, and its layout can change with Windows updates. Thankfully, Microsoft provides the Trace Data Helper (TDH) API, which uses the provider’s manifest to parse the binary payload into readable strings and integers. In EventRecordCallback</code>, we verify the provider and select event ID 1</code>, which this provider’s manifest defines as process start. We then use TdhGetProperty</code> to extract fields such as ProcessID</code> and ImageName</code>. Event schemas can vary by provider version, so production code should inspect the event metadata rather than assume every Windows build exposes an identical payload. VOID WINAPI EventRecordCallback(PEVENT_RECORD pEvent) { if (memcmp(&pEvent->EventHeader.ProviderId, &ProviderGuid, sizeof(GUID)) != 0) return; // Event ID 1 == Process Start if (pEvent->EventHeader.EventDescriptor.Id == 1) { printf("\n[+] Process Started!\n"); printf(" ProcessID: %lu\n", GetUint32Property(pEvent, L"ProcessID")); printf(" ParentProcessID: %lu\n", GetUint32Property(pEvent, L"ParentProcessID")); PrintStringProperty(pEvent, L"ImageName"); printf("--------------------------------------------------\n"); } }</code></pre> The complete program should include GetUint32Property</code> and PrintStringProperty</code> helpers that wrap TdhGetProperty</code>, plus error handling for missing properties. The excerpts here focus on the ETW control flow. Graceful Shutdown</a></h3> Real-time ETW sessions are system-wide objects. If our program exits without stopping its session, that session can remain active. To prevent this, we handle Ctrl+C</code> and stop it with ControlTraceW</code>. BOOL WINAPI ConsoleHandler(DWORD signal) { if (signal == CTRL_C_EVENT && g_hSession) { printf("\nStopping trace session gracefully...\n"); // Tell the kernel to stop our session ControlTraceW(g_hSession, g_SessionName, &stopProps.Properties, EVENT_TRACE_CONTROL_STOP); return TRUE; } return FALSE; }</code></pre>Compiling and Running</a></h3> To compile this sensor, use the MSVC compiler included with Visual Studio. Open the x64 Native Tools Command Prompt and run: cl.exe /O2 mini_edr.c</code></pre> Testing the Agent: Because ETW interacts directly with the Windows Kernel, you must run this executable as an Administrator. Open an elevated command prompt, run mini_edr.exe</code>, and then open an application like notepad.exe</code> or calc.exe</code>. You will see your mini-EDR catch the execution: PS > .\mini_edr.exe Trace Session 'ProcMonTrace_18872' Started. Listening for Process Starts... (Press Ctrl+C to stop) [+] Process Started! ProcessID: 18956 ParentProcessID: 1600 ImageName: \Device\HarddiskVolume3\Windows\System32\dllhost.exe -------------------------------------------------- [+] Process Started! ProcessID: 19012 ParentProcessID: 6692 ImageName: \Device\HarddiskVolume3\Program Files\WindowsApps\Microsoft.WindowsNotepad_11.2512.26.0_x64__8wekyb3d8bbwe\Notepad\Notepad.exe -------------------------------------------------- [+] Process Started! ProcessID: 19200 ParentProcessID: 2040 ImageName: \Device\HarddiskVolume3\Program Files\WindowsApps\Microsoft.WindowsCalculator_11.2508.4.0_x64__8wekyb3d8bbwe\CalculatorApp.exe --------------------------------------------------</code></pre>Conclusion</a></h3> Congratulations! You’ve just written a core component of an EDR engine. Commercial EDRs add many layers: additional telemetry sources, tamper protection, buffering, enrichment, correlation, and remote analysis. This small consumer is not an EDR by itself, but it demonstrates one practical way to collect process telemetry on Windows. In a future lab, we can compare this approach with a signed kernel driver that uses process-creation callbacks. Building a Mini-SIEM Lab: Sysmon, Vector, Redpanda, ClickHouse & Grafana 2026-02-22T00:00:00+00:00 Building a Security Information and Event Management (SIEM) system from scratch is one of the best ways to understand how those enterprise system works. In this lab, we will build a modern log pipeline using Windows host and WSL2 (Debian). Before diving into the code, let’s understand core architecture of our pipeline: Ingestion (Sysmon</a> & Vector</a>): Gathering logs from the operating system is crucial step. Usually we gather logs from different system so we need to transform them into a schema format (ECS/OCSF)</li> Buffering (Redpanda</a>): A messaging queue that temporarily hold logs. This prevent database from crashing if a sudden spike in log volume occurs.</li> Storage (ClickHouse</a>): An OLAP Database which is adjusted to handling Real-Time Analytics. It is column-oriented and optimized for analyzing massive amounts of log data.</li> Visualization (Grafana</a>): The dashboard where security analysts run queries and visualize threats.</li> </ol> Let’s build it step-by-step Phase 1: Ingestion (Sysmon & Vector)</a></h2> First we need something to generate high-quality security logs. For that we can use Sysmon Windows service that logs process creation, network connection and more. We can use some already existing config e.g. SwitftOnSecurity</a> Open an elevated PowerShell prompt on your Windows machine and run: # Download config to temp Invoke-WebRequest -Uri https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml -OutFile C:\Windows\Temp\sysmonconfig.xml # Install sysmon with config sysmon64.exe -accepteula -i C:\Windows\Temp\sysmonconfig.xml</code></pre> Next step is installing Vector</code> by DataDog to read those logs and send them to our pipeline. Note: You need to compile Vector from source to include PR #24305</a>. If you are compiling from Windows, ensure you have LLVM</a> and follow the Vector manual installation guide</a> Create your Vector configuration file. This config maps raw Windows Event Logs to the Elastic Common Schema (ECS) format, making the data clean and structured. # vector.yaml data_dir: C:\Users\USERNAME\AppData\Local\Temp\vector-data api: enabled: true address: 127.0.0.1:8686 sources: sysmon_logs: type: windows_event_log channels: - Microsoft-Windows-Sysmon/Operational - Application - System - Security transforms: remap_ecs: type: remap inputs: - sysmon_logs source: | # Metadata .ecs.version = "8.11.0" .event.kind = "event" .event.code = .event_id .event.provider = .provider_name .event.dataset = .channel # Map Log Level .log.level = .level # Timestamp .timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%dT%H:%M:%S.%fZ") .@timestamp = .timestamp # Host .host.name = .computer .host.os.family = "windows" .host.os.platform = "windows" # Event_data as primary source of truth ed = .event_data if .channel == "Microsoft-Windows-Sysmon/Operational" && !is_null(ed) { .event.module = "sysmon" # Sysmon Event 1: Process Creation if .event_id == 1 { .event.category = "process" .event.type = ["start"] .process.pid = to_int!(ed.ProcessId) .process.entity_id = ed.ProcessGuid .process.executable = ed.Image .process.command_line = ed.CommandLine .process.working_directory = ed.CurrentDirectory .process.integrity = ed.IntegrityLevel if !is_null(ed.Image) { path_parts = split!(ed.Image, "\\") .process.name = path_parts[-1] } # User Parsing if !is_null(ed.User) { user_parts = split!(ed.User, "\\") if length(user_parts) == 2 { .user.domain = user_parts[0] .user.name = user_parts[1] } else { .user.name = ed.User } } # Hash Parsing if !is_null(ed.Hashes) { hashes = parse_key_value!(ed.Hashes, key_value_delimiter: "=", field_delimiter: ",") if !is_null(hashes.MD5) { .process.hash.md5 = hashes.MD5 } if !is_null(hashes.SHA1) { .process.hash.sha1 = hashes.SHA1 } if !is_null(hashes.SHA256) { .process.hash.sha256 = hashes.SHA256 } if !is_null(hashes.IMPHASH) { .process.hash.imphash = hashes.IMPHASH } } # Parent Process .process.parent.pid = to_int!(ed.ParentProcessId) .process.parent.entity_id = ed.ParentProcessGuid .process.parent.executable = ed.ParentImage .process.parent.command_line = ed.ParentCommandLine if !is_null(ed.ParentImage) { parent_parts = split!(ed.ParentImage, "\\") .process.parent.name = parent_parts[-1] } } # Sysmon Event 3: Network Connection if .event_id == 3 { .event.category = "network" .event.type = ["connection", "start"] .source.ip = ed.SourceIp .source.port = to_int!(ed.SourcePort) .source.domain = ed.SourceHostname .destination.ip = ed.DestinationIp .destination.port = to_int!(ed.DestinationPort) .destination.domain = ed.DestinationHostname .network.transport = ed.Protocol .process.pid = to_int!(ed.ProcessId) .process.entity_id = ed.ProcessGuid .process.executable = ed.Image if !is_null(ed.Image) { img_parts = split!(ed.Image, "\\") .process.name = img_parts[-1] } if !is_null(ed.User) { user_parts = split!(ed.User, "\\") if length(user_parts) == 2 { .user.domain = user_parts[0] .user.name = user_parts[1] } } } } else { # Handle standard Windows logs (Security, System, App) if .channel == "Security" { .event.category = "authentication" if .event_id == 4624 { .event.type = ["start", "authentication_success"] .event.outcome = "success" } if .event_id == 4625 { .event.type = ["start", "authentication_failure"] .event.outcome = "failure" } } if .channel == "System" { .event.category = "host" } if .channel == "Application" { .event.category = "process" } if !is_null(ed) { if !is_null(ed.TargetUserName) { .user.name = ed.TargetUserName } if !is_null(ed.TargetDomainName) { .user.domain = ed.TargetDomainName } if !is_null(ed.SubjectUserName) && ed.SubjectUserName != "-" { .user.effective.name = ed.SubjectUserName } if !is_null(ed.IpAddress) && ed.IpAddress != "-" { .source.ip = ed.IpAddress } if !is_null(ed.IpPort) && ed.IpPort != "-" { .source.port = to_int(ed.IpPort) ?? null } if !is_null(ed.ProcessName) && ed.ProcessName != "-" { .process.executable = ed.ProcessName p_parts = split!(ed.ProcessName, "\\") .process.name = p_parts[-1] } if !is_null(ed.ProcessId) { .process.pid = to_int(ed.ProcessId) ?? null } } } # Clean up raw fields to save bandwidth del(.event_data) del(.message) del(.task) del(.level_value) del(.record_id) del(.version) # Send to Redpanda (Kafka) sinks: to_redpanda: type: kafka inputs: - remap_ecs bootstrap_servers: 127.0.0.1:29092 topic: event-logs encoding: codec: json</code></pre>Phase 2, 3 & 4: Buffering, Storage, UI</a></h2> We will run our backend services in WSL2 (Debian) using Podman. Save following as docker-compose.yml</code> # docker-compose.yml version: '3.8' services: redpanda: image: docker.redpanda.com/redpandadata/redpanda:latest container_name: redpanda command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092 # Advertise external to localhost so Windows Vector can reach it. Internal to redpanda:9092 so Clickhouse can reach it. - --advertise-kafka-addr internal://redpanda:9092,external://127.0.0.1:29092 - --smp 1 - --memory 1G - --mode dev-container ports: - "29092:29092" - "9092:9092" - "8081:8081" - "9644:9644" volumes: - redpanda_data:/var/lib/redpanda/data networks: - siem_net clickhouse: image: docker.io/clickhouse/clickhouse-server:latest container_name: clickhouse environment: - CLICKHOUSE_PASSWORD=siemlab ports: - "8123:8123" # HTTP port (Grafana uses this) - "9000:9000" # Native client port volumes: - clickhouse_data:/var/lib/clickhouse - ./init-db.sql:/docker-entrypoint-initdb.d/init-db.sql # Auto-initializes tables depends_on: - redpanda networks: - siem_net grafana: image: docker.io/grafana/grafana:latest container_name: grafana ports: - "3000:3000" environment: # Automatically install the ClickHouse plugin - GF_INSTALL_PLUGINS=grafana-clickhouse-datasource volumes: - grafana_data:/var/lib/grafana depends_on: - clickhouse networks: - siem_net networks: siem_net: driver: bridge volumes: redpanda_data: clickhouse_data: grafana_data:</code></pre> RedPanda does not require additional configuration but ClickHouse and Grafana does. Initializing Database</a></h4> ClickHouse needs to know how to read the logs from Redpanda and where to store them. We do this using a three-step architecture. Target Table: Where the data permanently lives.</li> Kafka Engine Table: A temporary consumer that pulls data from Redpanda</li> Materialized View: The pipeline connecting the consumer to the target table.</li> </ol> Let’s save this as init-db.sql</code> in the same directory as previous file so it will be initialized when database is initialized. Here ingestion is smaller than our Event source, but we can extend that according to our use case. -- init-db.sql CREATE DATABASE IF NOT EXISTS siem; -- 1. Storage Table: The actual storage for your ECS logs CREATE TABLE IF NOT EXISTS siem.logs ( `@timestamp` DateTime64(3), `event.kind` String, `event.category` String, `event.type` Array(String), `event.code` UInt32, `host.name` String, `process.name` String, `process.pid` UInt32, `source.ip` String, `destination.ip` String, `raw_message` String -- Storing the raw JSON is great for students to dig into missing fields ) ENGINE = MergeTree() ORDER BY `@timestamp`; -- 2. Consumer Table: Pulls raw JSON from Redpanda. CREATE TABLE IF NOT EXISTS siem.logs_queue ( message String ) ENGINE = Kafka SETTINGS kafka_broker_list = 'redpanda:9092', kafka_topic_list = 'event-logs', kafka_group_name = 'clickhouse_consumer_group', kafka_format = 'JSONAsString'; -- 3. Pipeline: The Materialized View translating the Queue into Storage CREATE MATERIALIZED VIEW IF NOT EXISTS siem.logs_mv TO siem.logs AS SELECT -- Safely parse timestamp, default to current time if missing so the pipeline doesn't break ifNull(parseDateTime64BestEffortOrNull(JSONExtractString(message, '@timestamp'), 3), now64(3)) AS `@timestamp`, JSONExtractString(message, 'event', 'kind') AS `event.kind`, JSONExtractString(message, 'event', 'category') AS `event.category`, -- Convert raw JSON array strings to actual ClickHouse Arrays arrayMap(x -> replaceAll(x, '"', ''), JSONExtractArrayRaw(message, 'event', 'type')) AS `event.type`, JSONExtractUInt(message, 'event', 'code') AS `event.code`, JSONExtractString(message, 'host', 'name') AS `host.name`, JSONExtractString(message, 'process', 'name') AS `process.name`, JSONExtractUInt(message, 'process', 'pid') AS `process.pid`, JSONExtractString(message, 'source', 'ip') AS `source.ip`, JSONExtractString(message, 'destination', 'ip') AS `destination.ip`, message AS `raw_message` FROM siem.logs_queue;</code></pre> Now we need to install podman</code> and podman-compose</code> then start the stack sudo apt install podman podman-compose sudo podman-compose up -d</code></pre>Troubleshooting WSL2 & Podman Issues</a></h2> When running Podman on WSL2, you might encounter a few known issue. Here is how to fix them: Error: WARN: The cgroupv2 manager is set to systemd but there is no systemd user session available</code> Fix: You need to enable systemd inside WSL2. Edit or create /etc/wsl.conf</code> [boot] systemd=true</code></pre> Error: netavark: nftables error: nft did not return successfully while applying ruleset</code> Fix: Switch your firewall to legacy iptables</code>: sudo update-alternatives --set iptables /usr/sbin/iptables-legacy sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy</code></pre> If that fails, force Podman to use iptables via its config sudo mkdir -p /etc/containers echo -e "[network]\nfirewall_driver = \"iptables\"" | sudo tee -a /etc/containers/containers.conf</code></pre> Start Fresh If you mess up your configuration and need to wipe your state clean, run: # Powershell wsl --shutdown # WSL2 podman system reset -f sudo podman network prune -f sudo podman-compose down</code></pre>Phase 4: UI and Threat Hunting with Grafana</a></h2> If all services are running, it’s time to connect Grafana to ClickHouse Open browser and go to http://localhost:3000</code></li> Navigate to Connections > Data Sources > Add data source</li> Select ClickHouse and fill in the details: Server Address: clickhouse</li> Protocol: HTTP</li> Secure (TLS): Off</li> Username: default</li> Password: siemlab</li> Default Database (Advanced): siem</li> </ol> </li> </ol> Click “Save & Test”. If it fails, check the logs with sudo podman logs grafana</code> Example Queries</a></h4> Create a new dashboard, add a visualization, and try these queries to analyze your ingested data. Query 1: Find Noisy Processes Great for establishing a baseline of normal activity on host SELECT `process.name`, count() AS total_events FROM siem.logs WHERE $__timeFilter(`@timestamp`) AND `process.name` != '' GROUP BY `process.name` ORDER BY total_events DESC LIMIT 10</code></pre> Query 2: Raw Event Log View A tabular view of recent events SELECT `@timestamp` AS Time, `host.name` AS Host, `event.code` AS EventID, `process.name` AS Process, `source.ip` AS SourceIP, `destination.ip` AS DestIP FROM siem.logs WHERE $__timeFilter(`@timestamp`) ORDER BY `@timestamp` DESC LIMIT 100</code></pre> Now you have local SIEM processing real-time Windows logs.

Daniel's Detection Notes

Detection Is a Data-Modeling Problem

What You Cannot See

Cutting the Network from Inside

The Authority Problem

Naming Things That Move

What makes a name an identity</a></h2> Three properties separate a usable identity from a mere label:</p> Unique</strong> at any instant. Two live things never share it.</li> Stable</strong> for the lifetime of what it names. It doesn’t change underneath you.</li>

The Shape of a Pipeline

How an Operating System Decides to Let You Watch It

Building a Mini-EDR Agent - Monitoring Process Creation with ETW in C

Building a Mini-SIEM Lab: Sysmon, Vector, Redpanda, ClickHouse & Grafana