T1593

Search Open Websites/Domains

Reconnaissance Last updated:

This detection identifies automated reconnaissance activity against your organization's public-facing web assets, which may indicate an adversary conducting pre-attack intelligence gathering via T1593. Since T1593 occurs externally (adversaries querying social media, search engines, and public websites), direct network-level detection from within the victim environment is impossible. This detection instead focuses on second-order observable indicators: anomalous automated scraping patterns against your web infrastructure (IIS, Apache, Nginx, Azure WAF), known OSINT/reconnaissance tool user agents in web access logs, high-velocity enumeration from single source IPs, and probing of sensitive disclosure paths such as /.git/, /robots.txt, sitemap.xml, and /admin. These patterns correlate with adversary pre-compromise reconnaissance workflows used by groups including Volt Typhoon, Mustang Panda, and Kimsuky prior to phishing or initial access operations.

What is T1593 Search Open Websites/Domains?

Search Open Websites/Domains (T1593) maps to the Reconnaissance tactic — the adversary is trying to gather information they can use to plan future operations in MITRE ATT&CK.

This page provides production-ready detection logic for Search Open Websites/Domains, covering the data sources and telemetry it touches: Microsoft Sentinel (IIS Logs via W3CIISLog), Azure Application Gateway WAF. The queries below are rated medium severity at low confidence, and ship for 7 SIEM platforms — KQL, SPL, Elastic, QRadar, Sumo, YARA-L, LogScale.

MITRE ATT&CK

Tactic
Reconnaissance
Technique
T1593 Search Open Websites/Domains
Canonical reference
https://attack.mitre.org/techniques/T1593/
Microsoft Sentinel / Defender
kusto 
let KnownReconUserAgents = dynamic(["python-requests", "python-urllib", "go-http-client", "curl/", "wget/", "nuclei", "nikto", "dirbuster", "gobuster", "feroxbuster", "ffuf", "sqlmap", "scrapy", "zgrab", "masscan", "shodan", "censys", "binaryedge", "nmap", "burpsuite", "zap", "httpx", "katana", "subfinder", "amass", "theHarvester", "mechanize", "httplib2", "libwww-perl"]);
let SensitivePaths = dynamic(["/.git", "/.env", "/wp-admin", "/phpmyadmin", "/admin", "/robots.txt", "/sitemap.xml", "/.htaccess", "/web.config", "/backup", "/config", "/.well-known", "/xmlrpc.php", "/wp-login"]);
W3CIISLog
| where TimeGenerated > ago(1h)
| where isnotempty(cIP)
| extend UserAgentLower = tolower(csUserAgent)
| extend IsReconUA = iff(
    csUserAgent has_any (KnownReconUserAgents) or isempty(csUserAgent),
    true, false)
| extend IsSensitivePath = iff(
    csUriStem has_any (SensitivePaths),
    true, false)
| summarize
    TotalRequests = count(),
    UniqueURIs = dcount(csUriStem),
    UniquePaths = make_set(csUriStem, 30),
    ReconUARequests = countif(IsReconUA == true),
    SensitivePathHits = countif(IsSensitivePath == true),
    StatusCodes = make_set(scStatus),
    UserAgents = make_set(csUserAgent, 10),
    FirstRequest = min(TimeGenerated),
    LastRequest = max(TimeGenerated)
    by cIP, bin(TimeGenerated, 1h)
| where TotalRequests > 30 or ReconUARequests > 5 or SensitivePathHits > 3 or UniqueURIs > 25
| extend RiskScore = case(
    ReconUARequests > 20 and SensitivePathHits > 5, "High",
    ReconUARequests > 5 or SensitivePathHits > 3 or UniqueURIs > 50, "Medium",
    "Low")
| project
    TimeGenerated,
    SourceIP = cIP,
    TotalRequests,
    UniqueURIs,
    ReconUARequests,
    SensitivePathHits,
    SampledPaths = UniquePaths,
    UserAgents,
    StatusCodes,
    RiskScore,
    FirstRequest,
    LastRequest
| order by RiskScore asc, TotalRequests desc

Detects automated reconnaissance against public-facing web assets by correlating known OSINT and scanning tool user agents in IIS access logs with high-velocity enumeration patterns, sensitive path probing (/.git, /.env, /admin, /wp-admin), and anomalously high unique URI counts from single source IPs. Targets pre-compromise intelligence gathering consistent with T1593 sub-techniques (social media, search engine dorking, code repository searches) that manifest as automated scraping when adversaries pivot to directly probing your infrastructure.

medium severity low confidence

Data Sources

Microsoft Sentinel (IIS Logs via W3CIISLog) Azure Application Gateway WAF

Required Tables

W3CIISLog

False Positives

  • Legitimate commercial web crawlers and search engine bots (Googlebot, Bingbot, DuckDuckGo) may match known user agent patterns — whitelist verified crawler IP ranges from respective ASNs
  • Security vendors running authorized external attack surface scans (Qualys, Tenable, Rapid7) will produce reconnaissance-like patterns — maintain an allowlist of authorized scanner IPs
  • Developers or internal teams using curl, Python requests, or httpx for legitimate API testing or load testing against production endpoints
  • Content delivery networks and uptime monitoring services (Pingdom, UptimeRobot, StatusCake) making frequent automated HEAD/GET requests
  • Partners or customers running automated integrations that access your web endpoints at high frequency

Sigma rule & cross-platform mapping

The detection logic for Search Open Websites/Domains (T1593) above is provided in a vendor-neutral form so you can deploy it on any SIEM. The same logic is shipped here as native KQL (Microsoft Sentinel / Defender), SPL (Splunk), Elastic (Elastic Security (EQL)), QRadar (IBM QRadar (AQL)), Sumo (Sumo Logic CSE), YARA-L (Google Chronicle / SecOps), LogScale (CrowdStrike LogScale (CQL)) queries. In Sigma terms, this detection targets the following logsource: 

logsource:
  product: azure

Browse the community-maintained Sigma rules for this technique:

SigmaHQ rules for T1593 Sigma rules on detection.fyi

Platform-specific guides for T1593

Detect in Microsoft Sentinel (KQL) Detect in Splunk (SPL) Detect in Elastic Security (Elastic) Detect in IBM QRadar (QRadar) Detect in Sumo Logic CSE (Sumo) Detect in Google Chronicle (YARA-L) Detect in CrowdStrike LogScale (LogScale)
Last updated: 2026-03-19 Research depth: deep
References (7)

Testing Methodology

Validate this detection against 3 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.

  1. Test 1Automated Web Reconnaissance with Python Requests

    Expected signal: Web server access logs will show 25+ requests from 127.0.0.1 with user agent 'python-requests/2.x.x' hitting sensitive paths including /.git/config, /.env, /wp-admin, and /wp-config.php. IIS W3CIISLog or Apache access_combined logs will capture all requests.

  2. Test 2Directory Enumeration with Gobuster (DNS/HTTP Mode)

    Expected signal: Web server logs will show rapid sequential requests from 127.0.0.1 with user agent 'gobuster/3.x'. Each wordlist entry appears as a separate GET request. Requests arrive at ~5 concurrent requests/second. Response codes 200, 301, 302, 403, and 404 visible depending on what exists on the target.

  3. Test 3OSINT Reconnaissance with theHarvester Against Your Own Domain

    Expected signal: DNS resolver logs and network flow logs will show multiple DNS queries for subdomains of the target domain originating from the test host. If your DNS logging infrastructure captures queries, these appear as sequential lookups for www.example.com, mail.example.com, api.example.com, etc. theHarvester queries are external to the target and logged by Bing/search infrastructure, not the victim — this validates the external nature of T1593.

Response Playbook

Triage

  1. Step 1: Identify the source IP(s) triggering the alert. Run the IP through threat intelligence sources (VirusTotal, Shodan, AbuseIPDB, GreyNoise) to determine if it is a known scanner, TOR exit node, VPN provider, or attributed to a known threat actor. Document findings.
  2. Step 2: Review the specific paths being enumerated. Distinguish between broad web crawling (fetching normal pages) vs. targeted sensitive path enumeration (/.git, /.env, /admin, /wp-login). Targeted enumeration is higher priority and may indicate adversary interest in a specific vulnerability or credential disclosure.
  3. Step 3: Analyze user agent strings. Known OSINT tools (nuclei, dirbuster, ffuf, theHarvester) indicate intentional reconnaissance. Empty or generic user agents combined with high request velocity suggest automated tooling. Cross-reference with any GeoIP data — unusual geographic origins combined with tool-like behavior elevate priority.
  4. Step 4: Determine the time window and duration. Brief high-burst scans may indicate automated scanning infrastructure. Prolonged low-rate crawls (hours to days) may indicate stealthier human-operated OSINT collection. Check if this IP has appeared in logs in the past 30 days.
  5. Step 5: Check if any sensitive paths returned non-404 responses (200, 301, 403). A 200 OK on /.env, /backup, or /.git indicates potential successful information disclosure — escalate immediately regardless of other criteria.
  6. Step 6: Correlate with other telemetry. Check DNS logs for the source IP resolving your domain names, check email security logs for phishing attempts from related infrastructure in the 7-14 days following the scan, and cross-reference against recent social media mentions of your organization via threat intel feeds.

Containment

  1. If the source IP is confirmed malicious or attributed to a known threat actor, add it to perimeter firewall block rules and WAF IP reputation block lists. Document the block with ticket reference and business justification.
  2. If sensitive paths (/.git, /.env, /backup) returned successful responses (non-404), initiate an emergency review of what data is exposed. If credentials or secrets are exposed, treat as an active credential compromise — rotate all exposed secrets immediately and notify security leadership.
  3. Enable enhanced logging on your WAF/CDN for the source IP range (ASN-level) to capture any follow-on activity from related infrastructure. Consider Cloudflare/Azure WAF challenge mode for the source ASN if the scan is persistent.
  4. If the reconnaissance is sustained and targeted (returning daily over multiple days), consider activating honeypot tokens on sensitive paths to track if the adversary successfully extracts data and uses it elsewhere.

Evidence Collection

  1. Export raw web server access logs for the identified source IP for the full reconnaissance window, including timestamps, URIs, HTTP methods, status codes, response sizes, and user agents. Preserve in original format with hash for chain of custody.
  2. Capture WAF/Application Gateway logs if available — these may include additional request headers (X-Forwarded-For, Referer, Accept-Language) that reveal automation tooling or true source IP behind proxies.
  3. Document all sensitive paths that were probed and their HTTP response codes. For any path returning 200, capture the response content to determine what was disclosed.
  4. Run WHOIS and BGP routing lookups on the source IP to identify ASN, hosting provider, and any known associations. Save output for threat intelligence correlation.
  5. If GreyNoise or Shodan show the source IP as a known scanner, document the classification and tags (e.g., 'nuclei', 'masscan') as supporting evidence for incident classification.

Escalation Criteria

  • ! Escalate immediately to Tier 2/Incident Response if any sensitive paths (/.git, /.env, /backup, /config, /web.config) returned HTTP 200 responses — this indicates successful information disclosure that may accelerate adversary timeline.
  • ! Escalate if the source IP or ASN appears in subsequent phishing campaign indicators (email headers, URL shorteners) within 14 days of the reconnaissance scan, as this suggests the scan was part of targeted pre-attack intelligence gathering.
  • ! Escalate if reconnaissance patterns match known APT tooling signatures (e.g., Kimsuky's use of custom Python scrapers, Volt Typhoon's living-off-the-land web enumeration) identified in threat intelligence feeds.
  • ! Escalate if the same source infrastructure is observed scanning multiple organizational assets (web, VPN portal, mail gateway, customer portal) within a short timeframe, indicating a coordinated attack surface mapping operation.

Investigation Guide

Forensic Artifacts

  • > Web server access logs (IIS: C:\inetpub\logs\LogFiles\, Apache: /var/log/apache2/access.log, Nginx: /var/log/nginx/access.log) containing source IP, user agent, URI, and response codes
  • > WAF/CDN logs from Azure Application Gateway, Cloudflare, AWS CloudFront, or Akamai showing blocked or allowed reconnaissance requests with full request metadata
  • > DNS query logs showing external resolution of your organization's subdomains and hostnames, useful for correlating scanner infrastructure
  • > NetFlow or firewall connection logs showing volume and pattern of inbound connections from reconnaissance source IPs
  • > Threat intelligence platform (TIP) enrichment artifacts: VirusTotal reports, GreyNoise classifications, Shodan scan history for the source IP

Tuning Guidance

This detection targets external reconnaissance against your web assets and requires significant environment-specific tuning. Begin by building a whitelist of known-good crawler IP ranges: download Googlebot, Bingbot, and other major search engine crawler IP lists and exclude them from alerting. For authorized security scanners (Qualys, Tenable, Rapid7, or internal pen test teams), maintain a dynamic allowlist of scanner IPs coordinated with your security team and update it before each authorized scan window. Adjust the TotalRequests and UniqueURIs thresholds based on your web traffic baseline — high-traffic public sites will need higher thresholds (100+ requests, 50+ URIs) while low-traffic sites can remain sensitive (30 requests, 20 URIs). For SensitivePathHits, a threshold of 1-2 hits may be appropriate if you do not serve those paths at all (every request is anomalous). Consider implementing GreyNoise integration to auto-suppress known benign mass scanners, which can eliminate 60-70% of false positives. Finally, correlate alerts over a 14-day window — a single low-confidence alert is informational, but the same source IP generating alerts multiple times across a fortnight warrants escalation to threat intelligence for attribution review.

Hunting Queries

Hunts for successful HTTP responses (200/206/304) from requests to sensitive credential and configuration disclosure paths. A non-404 response on these paths indicates actual data exposure to the scanning party — this is the highest-priority variant of T1593 reconnaissance.

Hunting — KQL
kql 
W3CIISLog
| where TimeGenerated > ago(7d)
| where csUriStem has_any ("/.git/config", "/.git/HEAD", "/.env", "/.env.local", "/.env.production", "/wp-config.php", "/config.php", "/database.yml", "/settings.py", "/.aws/credentials", "/.ssh/id_rsa", "/backup.zip", "/backup.sql", "/db_backup")
| where scStatus in ("200", "206", "304")
| project TimeGenerated, SourceIP = cIP, RequestedPath = csUriStem, ResponseCode = scStatus, BytesSent = scBytes, UserAgent = csUserAgent
| order by TimeGenerated desc
Hunting — SPL
spl 
index=web (sourcetype="iis" OR sourcetype="access_combined" OR sourcetype="nginx:plus:access") earliest=-7d
| search uri_path IN ("/.git/config", "/.git/HEAD", "/.env", "/.env.local", "/.env.production", "/wp-config.php", "/config.php", "/.aws/credentials", "/.ssh/id_rsa", "/backup.zip", "/backup.sql")
| where status IN ("200", "206", "304")
| table _time, src_ip, uri_path, status, bytes, http_user_agent
| sort - _time

Hunts for high-frequency URI enumeration patterns (>10 requests/minute across >15 unique paths in 10-minute windows) that indicate automated tooling conducting directory brute-forcing or sitemap crawling. Differs from main detection by focusing on request velocity and unique path density rather than user agent matching.

Hunting — KQL
kql 
W3CIISLog
| where TimeGenerated > ago(24h)
| summarize
    RequestCount = count(),
    UniqueURIs = dcount(csUriStem),
    AvgTimeBetweenRequests = (max(TimeGenerated) - min(TimeGenerated)) / count(),
    StatusCodes = make_set(scStatus)
    by cIP, bin(TimeGenerated, 10m)
| where UniqueURIs > 15 and RequestCount > 20
| extend RequestsPerMinute = RequestCount / 10.0
| extend IsHighFrequency = iff(RequestsPerMinute > 10, true, false)
| where IsHighFrequency == true
| project TimeGenerated, SourceIP = cIP, RequestCount, UniqueURIs, RequestsPerMinute, StatusCodes
| order by RequestsPerMinute desc
Hunting — SPL
spl 
index=web (sourcetype="iis" OR sourcetype="access_combined") earliest=-24h
| bin _time span=10m
| stats count as request_count, dc(uri_path) as unique_uris, values(status) as status_codes by _time, src_ip
| where unique_uris > 15 AND request_count > 20
| eval requests_per_minute=request_count / 10
| where requests_per_minute > 10
| table _time, src_ip, request_count, unique_uris, requests_per_minute, status_codes
| sort - requests_per_minute

Hunts for persistent, low-and-slow reconnaissance campaigns spanning multiple days from the same source IP. Adversaries like Volt Typhoon conduct deliberate, patient OSINT collection to avoid triggering rate-based alerts. This hunt identifies IPs active for 3+ days with sustained enumeration of >30 unique paths, revealing stealthy long-duration reconnaissance not caught by hourly detection windows.

Hunting — KQL
kql 
W3CIISLog
| where TimeGenerated > ago(30d)
| where isnotempty(cIP)
| summarize
    TotalRequests = count(),
    ActiveDays = dcount(bin(TimeGenerated, 1d)),
    UniqueURIs = dcount(csUriStem),
    EarliestSeen = min(TimeGenerated),
    LatestSeen = max(TimeGenerated),
    UserAgents = make_set(csUserAgent, 5)
    by cIP
| where ActiveDays >= 3 and TotalRequests > 100 and UniqueURIs > 30
| extend DailyRequestRate = TotalRequests / ActiveDays
| extend SpanDays = datetime_diff('day', LatestSeen, EarliestSeen)
| where SpanDays >= 3
| project EarliestSeen, LatestSeen, SourceIP = cIP, ActiveDays, TotalRequests, UniqueURIs, DailyRequestRate, UserAgents
| order by ActiveDays desc, TotalRequests desc
Hunting — SPL
spl 
index=web (sourcetype="iis" OR sourcetype="access_combined") earliest=-30d
| stats count as total_requests, dc(uri_path) as unique_uris, dc(date_mday) as active_days, min(_time) as first_seen, max(_time) as last_seen, values(http_user_agent) as user_agents by src_ip
| where active_days >= 3 AND total_requests > 100 AND unique_uris > 30
| eval daily_rate=total_requests / active_days
| eval span_days=(last_seen - first_seen) / 86400
| where span_days >= 3
| table src_ip, active_days, total_requests, unique_uris, daily_rate, first_seen, last_seen, user_agents
| sort - active_days total_requests

Atomic Red Team Tests

Test 1 Automated Web Reconnaissance with Python Requests
linux

Simulates adversary OSINT collection against a target web server using Python requests library — the most common tooling pattern for T1593 automated reconnaissance. Generates web access log entries with a recognizable user agent that should trigger the detection.

Command

bash 
python3 -c "
import requests
import time

target = 'http://localhost'
paths = ['/', '/robots.txt', '/sitemap.xml', '/.git/config', '/.env', '/admin', '/wp-admin', '/phpmyadmin', '/backup', '/config', '/api', '/login', '/dashboard', '/about', '/contact', '/wp-login.php', '/xmlrpc.php', '/wp-config.php', '/.htaccess', '/web.config', '/server-status', '/upload', '/uploads', '/files', '/docs', '/swagger', '/api/v1', '/api/v2']

for path in paths:
    try:
        r = requests.get(f'{target}{path}', timeout=3, verify=False)
        print(f'[{r.status_code}] {path}')
        time.sleep(0.1)
    except Exception as e:
        print(f'[ERR] {path}: {e}')
"

Cleanup

bash 
# No cleanup required — this generates only HTTP requests against a local server

Expected Telemetry

Web server access logs will show 25+ requests from 127.0.0.1 with user agent 'python-requests/2.x.x' hitting sensitive paths including /.git/config, /.env, /wp-admin, and /wp-config.php. IIS W3CIISLog or Apache access_combined logs will capture all requests.

Expected Detection

Detection rule should fire within the 1-hour evaluation window showing SourceIP=127.0.0.1, ReconUARequests>=25, SensitivePathHits>=8, UniqueURIs>=25, RiskScore=High

Test 2 Directory Enumeration with Gobuster (DNS/HTTP Mode)
linux

Executes gobuster in directory enumeration mode against a local web server, simulating the automated path brute-forcing used by adversaries to discover exposed files, backup archives, and administrative interfaces during T1593 reconnaissance.

Command

bash 
# Install gobuster if not present
which gobuster || (apt-get install -y gobuster 2>/dev/null || go install github.com/OJ/gobuster/v3@latest)

# Create a small wordlist of sensitive paths
cat > /tmp/atomic_recon_wordlist.txt << 'EOF'
robots.txt
sitemap.xml
.git/config
.git/HEAD
.env
.env.local
.env.production
admin
wp-admin
phpmyadmin
login
backup
backup.zip
backup.sql
config
config.php
wp-config.php
web.config
.htaccess
api
api/v1
swagger
swagger-ui.html
api-docs
upload
uploads
files
secret
private
test
staging
EOF

# Run gobuster against localhost
gobuster dir -u http://localhost -w /tmp/atomic_recon_wordlist.txt -t 5 -q 2>&1 | head -50

Cleanup

bash 
rm -f /tmp/atomic_recon_wordlist.txt

Expected Telemetry

Web server logs will show rapid sequential requests from 127.0.0.1 with user agent 'gobuster/3.x'. Each wordlist entry appears as a separate GET request. Requests arrive at ~5 concurrent requests/second. Response codes 200, 301, 302, 403, and 404 visible depending on what exists on the target.

Expected Detection

SPL and KQL detections should correlate gobuster user agent signature with high UniqueURIs count and SensitivePathHits. Alert fires with RiskScore=Medium or High depending on responsive paths found.

Test 3 OSINT Reconnaissance with theHarvester Against Your Own Domain
linux

Simulates the OSINT phase of T1593 using theHarvester to enumerate email addresses, subdomains, hostnames, and employee names from public sources (search engines, DNS). This is the external search engine and social media reconnaissance sub-technique (T1593.001/T1593.002) that leaves no direct footprint in victim logs but generates second-order indicators in DNS and web telemetry.

Command

bash 
# Install theHarvester
which theHarvester || pip3 install theHarvester 2>/dev/null

# Set your own domain for safe testing (replace with your organization's test/lab domain)
TARGET_DOMAIN="example.com"

# Run theHarvester against Bing (produces external search queries, not logged by target)
theHarvester -d $TARGET_DOMAIN -b bing -l 50 2>&1 | head -80

# Enumerate subdomains via DNS brute force (generates DNS queries observable in DNS logs)
echo "--- DNS subdomain enumeration ---"
for sub in www mail ftp dev staging api admin vpn remote portal; do
    result=$(dig +short $sub.$TARGET_DOMAIN 2>/dev/null | head -1)
    if [ -n "$result" ]; then
        echo "[FOUND] $sub.$TARGET_DOMAIN -> $result"
    else
        echo "[NXDOMAIN] $sub.$TARGET_DOMAIN"
    fi
done

Cleanup

bash 
# No cleanup required — DNS queries are ephemeral and no files are modified on target systems

Expected Telemetry

DNS resolver logs and network flow logs will show multiple DNS queries for subdomains of the target domain originating from the test host. If your DNS logging infrastructure captures queries, these appear as sequential lookups for www.example.com, mail.example.com, api.example.com, etc. theHarvester queries are external to the target and logged by Bing/search infrastructure, not the victim — this validates the external nature of T1593.

Expected Detection

DNS-based subdomain enumeration may trigger DNS analytics rules detecting sequential NXDOMAIN responses from a single source. Direct theHarvester queries against search engines are not detectable from within the victim environment — confirming the need for second-order indicator detection strategies.

Related Detections

Tactic Hub

TA0043Reconnaissance

Sub-techniques (3)

Related Techniques

T1593.001Social MediaT1593.002Search EnginesT1593.003Code RepositoriesT1592Gather Victim Host InformationT1596Search Open Technical DatabasesT1594Search Victim-Owned WebsitesT1566PhishingT1598Phishing for InformationT1585Establish AccountsT1133External Remote Services

Same Tactic: Reconnaissance

Popular Detections