Detect Search Open Technical Databases in Microsoft Sentinel
This detection identifies potential adversary reconnaissance activity involving queries to open technical databases such as Shodan, Censys, Certificate Transparency logs, passive DNS repositories, WHOIS services, and CDN enumeration tools. Because T1596 is a PRE-ATT&CK technique executed outside the victim network, direct detection is not possible. Instead, this rule focuses on second-order indicators: internal endpoints or service accounts making scripted bulk queries to known OSINT aggregator domains, automated certificate transparency monitoring tools being abused, and unusual DNS resolution patterns targeting passive DNS APIs. Detection of these behaviors may indicate an insider threat, a compromised endpoint conducting pre-pivot reconnaissance, or red team activity. Confidence is set to low due to high false positive potential from legitimate security research and threat intelligence workflows.
MITRE ATT&CK
- Tactic
- Reconnaissance
- Technique
- T1596 Search Open Technical Databases
- Canonical reference
- https://attack.mitre.org/techniques/T1596/
KQL Detection Query
let OSINTDomains = dynamic([
"shodan.io", "api.shodan.io",
"censys.io", "search.censys.io",
"dnsdumpster.com",
"securitytrails.com", "api.securitytrails.com",
"spyse.com",
"zoomeye.org",
"fofa.so", "fofa.info",
"binaryedge.io", "api.binaryedge.io",
"onyphe.io",
"hunter.io",
"intelx.io",
"passivedns.circl.lu",
"riddler.io",
"robtex.com",
"hackertarget.com",
"whoisxmlapi.com",
"domaintools.com",
"crt.sh",
"certspotter.com",
"urlscan.io",
"viewdns.info",
"threatcrowd.org"
]);
let SuspiciousProcesses = dynamic([
"curl.exe", "wget.exe",
"python.exe", "python3.exe", "py.exe",
"powershell.exe", "pwsh.exe",
"cmd.exe", "wscript.exe", "cscript.exe"
]);
DeviceNetworkEvents
| where TimeGenerated > ago(1h)
| where ActionType == "ConnectionSuccess" or ActionType == "HttpConnectionInspected"
| where RemoteUrl has_any (OSINTDomains)
or RemoteIPType != "Private" and RemotePort in (443, 80)
and (RemoteUrl contains "shodan" or RemoteUrl contains "censys" or RemoteUrl contains "crt.sh")
| where InitiatingProcessFileName in~ (SuspiciousProcesses)
or InitiatingProcessCommandLine has_any ("/api/", "apikey", "api_key", "--key", "-H 'API")
| summarize
RequestCount = count(),
UniqueOSINTDomains = dcount(RemoteUrl),
QueriedDomains = make_set(RemoteUrl, 20),
ProcessesUsed = make_set(InitiatingProcessFileName, 5),
CommandLines = make_set(InitiatingProcessCommandLine, 5),
FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated)
by DeviceName, InitiatingProcessAccountName, InitiatingProcessAccountDomain
| where RequestCount >= 5 or UniqueOSINTDomains >= 2
| extend
RiskScore = case(
UniqueOSINTDomains >= 5, 90,
UniqueOSINTDomains >= 3, 70,
RequestCount >= 20, 60,
40
),
Alert = "Possible open technical database reconnaissance from corporate endpoint"
| project
FirstSeen, LastSeen, DeviceName,
InitiatingProcessAccountName, InitiatingProcessAccountDomain,
RequestCount, UniqueOSINTDomains, QueriedDomains,
ProcessesUsed, CommandLines, RiskScore, Alert
| order by RiskScore descDetects scripted or bulk queries from corporate endpoints to known OSINT and open technical database services (Shodan, Censys, SecurityTrails, crt.sh, etc.), particularly via command-line tools like curl, Python, or PowerShell. Aggregates by user and device to surface bulk enumeration patterns that suggest automated reconnaissance against the organization's own infrastructure.
Data Sources
Required Tables
False Positives & Tuning
- Security analysts and threat intelligence teams performing legitimate asset discovery or exposure monitoring using these same OSINT tools
- Red team or penetration testing engagements querying Shodan/Censys to validate external attack surface
- Automated vulnerability management platforms (e.g., Tenable.io, Rapid7) that query third-party databases as part of external exposure scanning
- DevSecOps pipelines using crt.sh or SecurityTrails APIs for certificate transparency monitoring in CI/CD
- IT asset management tools that periodically validate domain and certificate configurations via WHOIS or passive DNS APIs
Other platforms for T1596
Testing Methodology
Validate this detection against 3 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Shodan CLI Reconnaissance of Organization IP Range
Expected signal: Sysmon EventCode 3 (Network Connection) to api.shodan.io:443 from python process; Sysmon EventCode 22 (DNS Query) for api.shodan.io; DeviceNetworkEvents showing ConnectionSuccess to shodan.io from python.exe or python3.exe
- Test 2Certificate Transparency Bulk Query via crt.sh API
Expected signal: Sysmon EventCode 1 (Process Create) for powershell.exe with Invoke-RestMethod in CommandLine; Sysmon EventCode 3 (Network Connection) to crt.sh:443; Windows Security EventCode 4688 (Process Creation) if Sysmon not available; DeviceNetworkEvents showing connection to crt.sh from powershell.exe
- Test 3Multi-Source OSINT Enumeration Script
Expected signal: Sysmon EventCode 3 (Network Connection) entries for hackertarget.com and viewdns.info from bash/curl; Sysmon EventCode 22 (DNS Query) for api.hackertarget.com, api.viewdns.info; proxy logs showing sequential requests to multiple OSINT domains within short time window
References (11)
- https://attack.mitre.org/techniques/T1596/
- https://attack.mitre.org/techniques/T1596/001/
- https://attack.mitre.org/techniques/T1596/002/
- https://attack.mitre.org/techniques/T1596/003/
- https://attack.mitre.org/techniques/T1596/004/
- https://attack.mitre.org/techniques/T1596/005/
- https://www.shodan.io/
- https://search.censys.io/
- https://crt.sh/
- https://securitytrails.com/
- https://dnsdumpster.com/
Unlock Pro Content
Get the full detection package for T1596 including response playbook, investigation guide, and atomic red team tests.