Detect Search Open Websites/Domains in Elastic Security
This detection identifies automated reconnaissance activity against your organization's public-facing web assets, which may indicate an adversary conducting pre-attack intelligence gathering via T1593. Since T1593 occurs externally (adversaries querying social media, search engines, and public websites), direct network-level detection from within the victim environment is impossible. This detection instead focuses on second-order observable indicators: anomalous automated scraping patterns against your web infrastructure (IIS, Apache, Nginx, Azure WAF), known OSINT/reconnaissance tool user agents in web access logs, high-velocity enumeration from single source IPs, and probing of sensitive disclosure paths such as /.git/, /robots.txt, sitemap.xml, and /admin. These patterns correlate with adversary pre-compromise reconnaissance workflows used by groups including Volt Typhoon, Mustang Panda, and Kimsuky prior to phishing or initial access operations.
MITRE ATT&CK
- Tactic
- Reconnaissance
- Technique
- T1593 Search Open Websites/Domains
- Canonical reference
- https://attack.mitre.org/techniques/T1593/
Elastic Detection Query
// T1593 — Search Open Websites
any where event.dataset : ("iis.access", "apache_http_server.access")
and (user_agent.original : ("python-requests*", "go-http-client*", "curl/*", "nuclei*",
"nikto*", "dirbuster*", "gobuster*", "scrapy*", "masscan*")
or url.path : ("/.git/*", "/.env", "/wp-admin/*", "/robots.txt", "/sitemap.xml",
"/.htaccess", "/web.config", "/backup/*", "/config/*"))Elastic EQL detection for Search Open Websites/Domains (T1593). Translates the Microsoft Sentinel KQL logic to Elastic Common Schema (ECS) field mappings for use in Elastic SIEM. Targets the same behavioral indicators across process creation, network, and authentication event types.
Data Sources
Required Tables
False Positives & Tuning
- Legitimate commercial web crawlers and search engine bots (Googlebot, Bingbot, DuckDuckGo) may match known user agent patterns — whitelist verified crawler IP ranges from respective ASNs
- Security vendors running authorized external attack surface scans (Qualys, Tenable, Rapid7) will produce reconnaissance-like patterns — maintain an allowlist of authorized scanner IPs
- Developers or internal teams using curl, Python requests, or httpx for legitimate API testing or load testing against production endpoints
- Content delivery networks and uptime monitoring services (Pingdom, UptimeRobot, StatusCake) making frequent automated HEAD/GET requests
Other platforms for T1593
Testing Methodology
Validate this detection against 3 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Automated Web Reconnaissance with Python Requests
Expected signal: Web server access logs will show 25+ requests from 127.0.0.1 with user agent 'python-requests/2.x.x' hitting sensitive paths including /.git/config, /.env, /wp-admin, and /wp-config.php. IIS W3CIISLog or Apache access_combined logs will capture all requests.
- Test 2Directory Enumeration with Gobuster (DNS/HTTP Mode)
Expected signal: Web server logs will show rapid sequential requests from 127.0.0.1 with user agent 'gobuster/3.x'. Each wordlist entry appears as a separate GET request. Requests arrive at ~5 concurrent requests/second. Response codes 200, 301, 302, 403, and 404 visible depending on what exists on the target.
- Test 3OSINT Reconnaissance with theHarvester Against Your Own Domain
Expected signal: DNS resolver logs and network flow logs will show multiple DNS queries for subdomains of the target domain originating from the test host. If your DNS logging infrastructure captures queries, these appear as sequential lookups for www.example.com, mail.example.com, api.example.com, etc. theHarvester queries are external to the target and logged by Bing/search infrastructure, not the victim — this validates the external nature of T1593.
References (8)
- https://attack.mitre.org/techniques/T1593/
- https://attack.mitre.org/techniques/T1593/001/
- https://attack.mitre.org/techniques/T1593/002/
- https://attack.mitre.org/techniques/T1593/003/
- https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-038a
- https://www.microsoft.com/en-us/security/blog/2023/05/24/volt-typhoon-targets-us-critical-infrastructure-with-living-off-the-land-techniques/
- https://www.greynoise.io/blog/understanding-mass-internet-scanners
- https://securitytrails.com/blog/google-hacking-techniques
Unlock Pro Content
Get the full detection package for T1593 including response playbook, investigation guide, and atomic red team tests.