{"id":60348,"date":"2026-03-20T16:19:00","date_gmt":"2026-03-20T16:19:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/security\/blog\/?p=146042"},"modified":"2026-03-20T16:19:00","modified_gmt":"2026-03-20T16:19:00","slug":"cti-realm-a-new-benchmark-for-end-to-end-detection-rule-generation-with-ai-agents","status":"publish","type":"post","link":"https:\/\/www.threatshub.org\/blog\/cti-realm-a-new-benchmark-for-end-to-end-detection-rule-generation-with-ai-agents\/","title":{"rendered":"CTI-REALM: A new benchmark for end-to-end detection rule generation with AI agents"},"content":{"rendered":"

Excerpt:<\/strong> CTI-REALM is Microsoft\u2019s open-source benchmark for evaluating AI agents on real-world detection engineering\u2014turning cyber threat intelligence (CTI) into validated detections. Instead of measuring \u201cCTI trivia,\u201d CTI-REALM tests end-to-end workflows: reading threat reports, exploring telemetry, iterating on KQL queries, and producing Sigma rules and KQL-based detection logic that can be scored against ground truth across Linux, AKS, and Azure cloud environments.<\/p>\n

\n

Security is Microsoft\u2019s top priority. Every day, we process more than 100 trillion security signals across endpoints, cloud infrastructure, identity, and global threat intelligence. That\u2019s the scale modern cyber defense demands, and AI is a core part of how we protect Microsoft and our customers worldwide. At the same time, security is, and always will be, a team sport. <\/em><\/p>\n

That\u2019s why Microsoft is committed to AI model diversity and to helping defenders apply the latest AI responsibly. We created CTI\u2011REALM and open\u2011sourced it so the broader industry can test models, write better code, and build more secure systems together.<\/em><\/p>\n

\n

CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is Microsoft\u2019s open-source benchmark that evaluates AI agents on end-to-end detection engineering. Building on work like ExCyTIn-Bench<\/a>, which evaluates agents on threat investigation, CTI-REALM extends the scope to the next stage of the security workflow: detection rule generation. Rather than testing whether a model can answer CTI trivia or classify techniques in isolation, CTI-REALM places agents in a realistic, tool-rich environment and asks them to do what security analysts do every day: read a threat intelligence report, explore telemetry, write and refine KQL queries, and produce validated detection rules.<\/p>\n

We curated 37 CTI reports from public sources (Microsoft Security, Datadog Security Labs, Palo Alto Networks, and Splunk), selecting those that could be faithfully simulated in a sandboxed environment and that produced telemetry suitable for detection rule development. The benchmark spans three platforms: Linux endpoints, Azure Kubernetes Service (AKS), and Azure cloud infrastructure with ground-truth scoring at every stage of the analytical workflow.<\/p>\n

Why CTI-REALM exists<\/h3>\n

Existing cybersecurity benchmarks primarily test parametric knowledge: can a model name the MITRE technique behind a log entry, or classify a TTP from a report? These are useful signals. However, they miss the harder question: can an agent operationalize that knowledge into detection logic that finds attacks in production telemetry?<\/p>\n

No current benchmark evaluates this complete workflow. CTI-REALM fills that gap by measuring:<\/p>\n