T1594

Search Victim-Owned Websites

Reconnaissance Last updated:

This detection identifies adversary reconnaissance activity targeting victim-owned websites, including automated crawling, directory enumeration, and harvesting of sensitive pages such as robots.txt, sitemap.xml, staff/contact directories, and hidden paths. Because T1594 is a PRE-ATT&CK technique occurring outside the victim network, detection relies on web server access logs, WAF telemetry, and CDN logs ingested into SIEM. Detection focuses on high-volume requests from single source IPs, enumeration of employee/contact pages, known scraping tool user agents, and sequential access patterns indicative of automated reconnaissance tools used by groups like Kimsuky, Volt Typhoon, Silent Librarian, and Sandworm Team.

What is T1594 Search Victim-Owned Websites?

Search Victim-Owned Websites (T1594) maps to the Reconnaissance tactic — the adversary is trying to gather information they can use to plan future operations in MITRE ATT&CK.

This page provides production-ready detection logic for Search Victim-Owned Websites, covering the data sources and telemetry it touches: Microsoft Sentinel (W3C IIS Logs), Azure WAF Logs. The queries below are rated medium severity at low confidence, and ship for 7 SIEM platforms — KQL, SPL, Elastic, QRadar, Sumo, YARA-L, LogScale.

MITRE ATT&CK

Tactic
Reconnaissance
Technique
T1594 Search Victim-Owned Websites
Canonical reference
https://attack.mitre.org/techniques/T1594/
Microsoft Sentinel / Defender
kusto 
let timeWindow = 1h;
let requestThreshold = 100;
let errorThreshold = 20;
let suspiciousUAs = dynamic(["scrapy", "python-requests", "python-urllib", "wget", "nikto", "masscan", "nmap", "zgrab", "go-http-client", "libwww-perl", "java/", "curl/", "dirbuster", "gobuster", "feroxbuster", "wfuzz", "ffuf", "httprint", "sqlmap", "whatweb"]);
let reconPaths = dynamic(["/robots.txt", "/sitemap.xml", "/sitemap_index.xml", "/.well-known/security.txt", "/.git/", "/.env", "/wp-admin", "/admin", "/staff", "/team", "/employees", "/contact", "/about", "/management", "/leadership", "/directory"]);
W3CIISLog
| where TimeGenerated > ago(timeWindow)
| extend UALower = tolower(csUserAgent)
| extend IsReconUA = UALower has_any (suspiciousUAs)
| extend IsReconPath = csUriStem has_any (reconPaths)
| extend Is404 = (scStatus == 404)
| extend IsEmployeePage = csUriStem matches regex @"(?i)/(staff|team|employees|people|directory|contact|about|leadership|management|board)"
| where IsReconUA or IsReconPath or Is404 or IsEmployeePage
| summarize
    TotalRequests = count(),
    Count404 = countif(scStatus == 404),
    Count403 = countif(scStatus == 403),
    UniquePathsRequested = dcount(csUriStem),
    ReconPathHits = countif(IsReconPath),
    EmployeePageHits = countif(IsEmployeePage),
    SuspiciousUAUsed = countif(IsReconUA),
    UserAgents = make_set(csUserAgent, 5),
    AccessedPaths = make_set(csUriStem, 30),
    FirstRequest = min(TimeGenerated),
    LastRequest = max(TimeGenerated)
    by SourceIP = cIp, TargetSite = sSiteName
| extend ReconScore = 
    (case(Count404 > 50, 3, Count404 > 20, 2, Count404 > 5, 1, 0)) +
    (case(TotalRequests > 500, 3, TotalRequests > 200, 2, TotalRequests > 100, 1, 0)) +
    (case(ReconPathHits > 3, 2, ReconPathHits >= 1, 1, 0)) +
    (case(SuspiciousUAUsed > 0, 2, 0)) +
    (case(EmployeePageHits > 5, 2, EmployeePageHits >= 1, 1, 0))
| where ReconScore >= 3
| extend SessionDurationMinutes = datetime_diff('minute', LastRequest, FirstRequest)
| extend RequestsPerMinute = round(todouble(TotalRequests) / max_of(SessionDurationMinutes, 1), 1)
| project
    FirstRequest, LastRequest, SourceIP, TargetSite,
    TotalRequests, Count404, Count403, UniquePathsRequested,
    ReconPathHits, EmployeePageHits, SuspiciousUAUsed,
    SessionDurationMinutes, RequestsPerMinute,
    UserAgents, AccessedPaths, ReconScore
| sort by ReconScore desc, TotalRequests desc

Detects automated reconnaissance against victim-owned web properties by correlating IIS web server access logs for high-volume requests, directory enumeration patterns (404 spikes), access to reconnaissance-specific paths (robots.txt, sitemap.xml, .git, .env), employee/contact directory harvesting, and known scraping tool user agents. Assigns a composite ReconScore to prioritize high-confidence alerts.

medium severity low confidence

Data Sources

Microsoft Sentinel (W3C IIS Logs) Azure WAF Logs

Required Tables

W3CIISLog

False Positives

  • Legitimate search engine crawlers (Googlebot, Bingbot, DuckDuckBot) with high request volumes — filter by known crawler IP ranges and UA strings
  • Authorized penetration testing or red team engagements scheduled by the organization — cross-reference with change management records
  • Web archiving services such as archive.org (Internet Archive) performing scheduled snapshots
  • SEO audit tools used by the marketing team (Screaming Frog, Ahrefs, SEMrush bots)
  • Load testing tools (Apache JMeter, k6, Locust) run by the engineering team generating high 404 rates

Sigma rule & cross-platform mapping

The detection logic for Search Victim-Owned Websites (T1594) above is provided in a vendor-neutral form so you can deploy it on any SIEM. The same logic is shipped here as native KQL (Microsoft Sentinel / Defender), SPL (Splunk), Elastic (Elastic Security (EQL)), QRadar (IBM QRadar (AQL)), Sumo (Sumo Logic CSE), YARA-L (Google Chronicle / SecOps), LogScale (CrowdStrike LogScale (CQL)) queries. In Sigma terms, this detection targets the following logsource: 

logsource:
  product: azure

Browse the community-maintained Sigma rules for this technique:

SigmaHQ rules for T1594 Sigma rules on detection.fyi

Platform-specific guides for T1594

Detect in Microsoft Sentinel (KQL) Detect in Splunk (SPL) Detect in Elastic Security (Elastic) Detect in IBM QRadar (QRadar) Detect in Sumo Logic CSE (Sumo) Detect in Google Chronicle (YARA-L) Detect in CrowdStrike LogScale (LogScale)
Last updated: 2026-03-19 Research depth: deep
References (4)

Testing Methodology

Validate this detection against 3 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.

  1. Test 1Automated Website Crawling with wget Spider Mode

    Expected signal: Web server access logs showing rapid sequential GET requests from single IP with wget user agent. Multiple 200, 301, and 404 responses across diverse URL paths. Request rate 20-100 req/min.

  2. Test 2Reconnaissance Path Enumeration with robots.txt and sitemap.xml Harvest

    Expected signal: Sequential requests to robots.txt, sitemap.xml then employee-related paths. User agent 'python-requests' in all requests. Mix of 200 and 404 responses across 60-second window.

  3. Test 3Directory Enumeration with ffuf Wordlist Scanning

    Expected signal: Burst of 404 responses (12 requests, 1 per path) within 90 seconds. ffuf or spoofed browser UA. Requests for paths like /admin, /staff, /.git, /.env. Rate approximately 10 req/min.

Response Playbook

Triage

  1. Step 1: Identify the source IP(s) and check reputation against threat intel feeds (VirusTotal, Shodan, AbuseIPDB). Note ASN, country of origin, and hosting provider — adversary infrastructure often uses VPS providers (OVH, Vultr, DigitalOcean, Linode).
  2. Step 2: Examine the specific paths accessed. Distinguish between general crawling vs. targeted harvesting — sequential access to /team/, /staff/, /directory/ with extraction of named employee content indicates targeted recon vs. opportunistic scanning.
  3. Step 3: Review user agent string. Automated tools like Scrapy, Python-requests, wget, and Nikto are rarely used by legitimate browser users. Check if the UA rotates across requests (evasion technique used by Kimsuky and Volt Typhoon).
  4. Step 4: Determine if robots.txt and/or sitemap.xml were accessed early in the session — adversaries frequently retrieve these first to map the site before targeted crawling (as documented in EXOTIC LILY and Silent Librarian TTPs).
  5. Step 5: Check whether the source IP has previously triggered alerts on other systems (firewall, email gateway, DNS). Cross-reference with recent spear-phishing attempts targeting employees whose names appear on the scraped pages.
  6. Step 6: Calculate the request rate (requests/minute). Rates >50 req/min with systematic 404 patterns indicate automated tooling. Legitimate human browsing rarely exceeds 2-5 requests/minute.
  7. Step 7: Identify if WAF or CDN (Cloudflare, Akamai, Azure Front Door) blocked any subsequent requests from the same IP range — blocked requests after initial recon suggest active exploitation attempts following reconnaissance.

Containment

  1. If scraping is ongoing: add the source IP to the WAF block list and verify the block is applied at the CDN/edge layer, not just origin servers.
  2. If employee data was harvested (names, emails, org chart): notify HR and security awareness team to alert targeted employees about elevated spear-phishing risk.
  3. If hidden paths (/.git/, /.env, /wp-admin) were accessed and returned 200 responses: conduct emergency review of exposed content and rotate any credentials or secrets that may have been disclosed.
  4. Implement rate limiting on web server for non-authenticated endpoints: 30 requests/minute per IP with progressive CAPTCHA challenge, then temporary block.
  5. If sitemap.xml exposed internal application paths or staging environments: review and sanitize sitemap entries to remove non-public URLs.

Evidence Collection

  1. Export complete web server access logs for the identified source IP(s) covering a 72-hour window before and after detection — adversaries may have conducted low-and-slow recon before triggering thresholds.
  2. Capture full HTTP request headers (including Referer, Accept-Language, X-Forwarded-For) from WAF or load balancer logs — these can identify the scanning tool and potentially reveal VPN/proxy chain.
  3. Document all 200-response paths accessed by the source IP — this represents the complete set of information successfully exfiltrated during the recon session.
  4. Export DNS query logs for the victim domain to identify if adversaries pre-enumerated subdomains before web scanning (correlate with T1596 passive DNS lookups).
  5. Preserve CDN/WAF challenge and block logs as these establish timeline and demonstrate adversary persistence after initial blocking attempts.
  6. If applicable, capture PCAP from network tap or cloud flow logs (VPC Flow Logs, NSG Flow Logs) for the timeframe to preserve full session data.

Escalation Criteria

  • ! Escalate to Incident Response if source IP is attributed to a known APT group (e.g., Kimsuky, Volt Typhoon, Sandworm) in threat intel feeds.
  • ! Escalate if reconnaissance was followed within 72 hours by spear-phishing emails targeting employees whose information was available on the harvested pages.
  • ! Escalate if sensitive internal paths were successfully accessed (/.git/, /.env, /api/internal/, /admin/) and returned non-404 responses — potential exposure of credentials or application source code.
  • ! Escalate if recon is part of a coordinated campaign: same source IPs appearing across multiple owned domains or partner organization websites.
  • ! Escalate if employee contact information harvested correlates with recently registered lookalike domains (typosquatting) suggesting active phishing infrastructure preparation.

Investigation Guide

Forensic Artifacts

  • > Web server access logs: Apache (/var/log/apache2/access.log), Nginx (/var/log/nginx/access.log), IIS (C:\inetpub\logs\LogFiles\)
  • > WAF/CDN logs: Cloudflare Firewall Events, Azure Front Door logs, AWS CloudFront access logs, Akamai SIEM integration logs
  • > Network flow data: VPC Flow Logs, Azure NSG Flow Logs, on-premises NetFlow/IPFIX showing HTTP/HTTPS traffic patterns
  • > DNS query logs showing subdomain enumeration (*.victim.com) prior to web scraping activity
  • > robots.txt and sitemap.xml files (attacker-readable) — review for accidentally disclosed sensitive paths
  • > CDN challenge/block logs showing IP addresses that triggered rate limits or bot detection rules
  • > SSL/TLS handshake logs (JA3 fingerprints) that may identify scanning tool fingerprints beyond user agent strings

Tuning Guidance

Primary tuning challenge: distinguishing legitimate crawlers from adversarial reconnaissance. Start by building an allowlist of known search engine IP ranges (Googlebot: 66.249.x.x, Bingbot: 40.77.x.x, 157.55.x.x) and whitelisting by both IP range AND matching user agent (adversaries cannot spoof Googlebot without using Google IPs). Set ReconScore threshold to 5+ for high-fidelity alerting. For environments with active SEO programs, exclude user agents matching 'Screaming Frog', 'SEMrushBot', 'AhrefsBot' after confirming with marketing team. Increase the requestThreshold variable from 100 to 300 for high-traffic public sites. For low-and-slow detection, consider 24-hour windows rather than 1-hour for environments facing advanced persistent reconnaissance. Integrate with threat intel to auto-suppress known scanner IP ranges (Shodan, Censys, SecurityTrails) if their activity is not operationally concerning.

Hunting Queries

Hunts for targeted employee directory harvesting: source IPs systematically accessing multiple staff/team/leadership pages, indicating deliberate collection of personnel data for spear-phishing targeting.

Hunting — KQL
kql 
// Hunt: Sequential access to employee-identifying pages from single IPs suggesting targeted staff harvesting
W3CIISLog
| where TimeGenerated > ago(7d)
| where csUriStem matches regex @"(?i)/(staff|team|employees|people|directory|leadership|management|board|executives|about-us)"
| summarize
    UniqueEmployeePages = dcount(csUriStem),
    TotalHits = count(),
    PagesVisited = make_set(csUriStem, 20),
    UserAgents = make_set(csUserAgent, 5),
    SessionStart = min(TimeGenerated),
    SessionEnd = max(TimeGenerated)
    by SourceIP = cIp, Site = sSiteName
| where UniqueEmployeePages >= 3
| extend SessionMinutes = datetime_diff('minute', SessionEnd, SessionStart)
| extend HitsPerMinute = round(todouble(TotalHits) / max_of(SessionMinutes, 1), 2)
| where HitsPerMinute > 5 or TotalHits > 30
| project SessionStart, SessionEnd, SourceIP, Site, UniqueEmployeePages, TotalHits, HitsPerMinute, PagesVisited, UserAgents
| sort by UniqueEmployeePages desc
Hunting — SPL
spl 
index=* (sourcetype="access_combined" OR sourcetype="iis") earliest=-7d
| eval uri=coalesce(uri_path, cs_uri_stem)
| eval src=coalesce(src_ip, c_ip, clientip)
| where match(uri, "(?i)/(staff|team|employees|people|directory|leadership|management|board|executives|about-us)")
| bucket _time span=1h
| stats
    dc(uri) as unique_employee_pages,
    count as total_hits,
    values(uri) as pages_visited,
    values(http_user_agent) as user_agents
    by _time, src, host
| where unique_employee_pages >= 3
| sort -unique_employee_pages, -total_hits

Hunts for the classic recon sequence: adversary first fetches robots.txt/sitemap.xml to map site structure, then immediately begins directory enumeration (404 bursts). This ordered pattern is a strong indicator of automated reconnaissance tools.

Hunting — KQL
kql 
// Hunt: robots.txt and sitemap.xml access followed by directory enumeration within same session
let reconSessions = W3CIISLog
| where TimeGenerated > ago(7d)
| where csUriStem in ("/robots.txt", "/sitemap.xml", "/sitemap_index.xml", "/.well-known/security.txt")
| project SessionIP = cIp, SessionSite = sSiteName, ReconTime = TimeGenerated;
W3CIISLog
| where TimeGenerated > ago(7d)
| where scStatus == 404
| join kind=inner reconSessions on $left.cIp == $right.SessionIP and $left.sSiteName == $right.SessionSite
| where TimeGenerated between (ReconTime .. (ReconTime + 30min))
| summarize
    EnumerationAttempts = count(),
    UniquePathsTried = dcount(csUriStem),
    SamplePaths = make_set(csUriStem, 15),
    UserAgent = any(csUserAgent)
    by SourceIP = cIp, Site = sSiteName, ReconTime
| where EnumerationAttempts >= 10
| sort by EnumerationAttempts desc
Hunting — SPL
spl 
index=* (sourcetype="access_combined" OR sourcetype="iis") earliest=-7d
| eval uri=coalesce(uri_path, cs_uri_stem)
| eval src=coalesce(src_ip, c_ip, clientip)
| eval status=coalesce(status, sc_status)
| eval is_recon_seed=if(match(uri, "(robots\.txt|sitemap.*\.xml|\.well-known)"), 1, 0)
| eval is_404=if(status="404" OR status=404, 1, 0)
| bin _time span=30m
| stats
    sum(is_recon_seed) as recon_seeds,
    sum(is_404) as dir_enum_count,
    dc(uri) as unique_paths,
    values(uri) as paths
    by _time, src, host
| where recon_seeds >= 1 AND dir_enum_count >= 10
| sort -dir_enum_count

Hunts for low-and-slow reconnaissance designed to evade rate-based detection: activity spread across 4+ hours, staying below per-minute thresholds but systematically harvesting contact/employee information across many unique paths. This pattern is associated with nation-state actors (Volt Typhoon, Kimsuky) avoiding automated blocking.

Hunting — KQL
kql 
// Hunt: Low-and-slow scraping attempting to evade rate limits — spread across multiple hours
W3CIISLog
| where TimeGenerated > ago(24h)
| summarize
    HourlyBuckets = dcount(bin(TimeGenerated, 1h)),
    TotalRequests = count(),
    UniquePathsHarvested = dcount(csUriStem),
    Count404 = countif(scStatus == 404),
    ContactPageHits = countif(csUriStem matches regex @"(?i)/(contact|email|phone|staff|team)"),
    UserAgents = make_set(csUserAgent, 5)
    by SourceIP = cIp, Site = sSiteName
| where HourlyBuckets >= 4 and TotalRequests between (50 .. 2000)
| where ContactPageHits > 0 or UniquePathsHarvested > 50
| extend AvgRequestsPerActiveHour = round(todouble(TotalRequests) / HourlyBuckets, 1)
| where AvgRequestsPerActiveHour between (5 .. 100)
| project SourceIP, Site, HourlyBuckets, TotalRequests, AvgRequestsPerActiveHour, UniquePathsHarvested, Count404, ContactPageHits, UserAgents
| sort by ContactPageHits desc, UniquePathsHarvested desc
Hunting — SPL
spl 
index=* (sourcetype="access_combined" OR sourcetype="iis") earliest=-24h
| eval uri=coalesce(uri_path, cs_uri_stem)
| eval src=coalesce(src_ip, c_ip, clientip)
| eval status=coalesce(status, sc_status)
| eval is_contact=if(match(uri, "(?i)/(contact|email|phone|staff|team)"), 1, 0)
| eval is_404=if(status="404" OR status=404, 1, 0)
| bucket _time span=1h
| stats
    count as hourly_count,
    sum(is_contact) as contact_hits,
    sum(is_404) as error_count,
    dc(uri) as unique_paths
    by _time, src, host
| stats
    count as active_hours,
    sum(hourly_count) as total_requests,
    sum(contact_hits) as total_contact_hits,
    dc(unique_paths) as paths_harvested
    by src, host
| where active_hours >= 4 AND total_requests > 50 AND total_requests < 2000
| where total_contact_hits > 0 OR paths_harvested > 50
| eval avg_per_hour=round(total_requests/active_hours, 1)
| where avg_per_hour >= 5 AND avg_per_hour <= 100
| sort -total_contact_hits, -paths_harvested

Atomic Red Team Tests

Test 1 Automated Website Crawling with wget Spider Mode
linux

Simulates adversary using wget to recursively crawl a target website, harvesting links, page structure, and file paths — replicating Silent Librarian and Kimsuky reconnaissance techniques against victim web properties.

Command

bash 
wget --spider --recursive --level=3 --no-parent --output-file=/tmp/recon_crawl.log --user-agent='Mozilla/5.0 (compatible; research-bot/1.0)' https://example.com 2>&1 | head -100
grep -E '(200|301|302|404)' /tmp/recon_crawl.log | sort | uniq -c | sort -rn | head -20

Cleanup

bash 
rm -f /tmp/recon_crawl.log

Expected Telemetry

Web server access logs showing rapid sequential GET requests from single IP with wget user agent. Multiple 200, 301, and 404 responses across diverse URL paths. Request rate 20-100 req/min.

Expected Detection

Web recon detection fires on high request volume + wget user agent match. ReconScore >= 4 from suspicious UA (2pts) + request volume (2pts).

Test 2 Reconnaissance Path Enumeration with robots.txt and sitemap.xml Harvest
linux

Simulates the first step of adversary web reconnaissance: fetching robots.txt and sitemap.xml to map the site structure before targeted content harvesting. Followed by staff/contact page enumeration matching EXOTIC LILY and Volt Typhoon patterns.

Command

bash 
TARGET="https://example.com"
echo "[+] Fetching robots.txt"
curl -s -A 'python-requests/2.31.0' "${TARGET}/robots.txt"
echo "\n[+] Fetching sitemap.xml"
curl -s -A 'python-requests/2.31.0' "${TARGET}/sitemap.xml"
echo "\n[+] Enumerating staff/contact pages"
for path in /staff /team /employees /about /contact /leadership /management /board /directory /people; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" -A 'python-requests/2.31.0' "${TARGET}${path}")
  echo "${path}: HTTP ${STATUS}"
  sleep 0.5
done

Cleanup

bash 
No persistent artifacts. Network traffic logs preserved on server.

Expected Telemetry

Sequential requests to robots.txt, sitemap.xml then employee-related paths. User agent 'python-requests' in all requests. Mix of 200 and 404 responses across 60-second window.

Expected Detection

ReconPath access detection fires on robots.txt + sitemap.xml access (2pts). Employee page hunt query detects systematic /staff /team enumeration. Combined score >= 5.

Test 3 Directory Enumeration with ffuf Wordlist Scanning
linux

Simulates adversary using ffuf to enumerate hidden directories and files on victim web server, generating the characteristic 404 burst pattern used for detecting T1594 and T1595.003 activity.

Command

bash 
# Install ffuf if not present: go install github.com/ffuf/ffuf/v2@latest
# Use a small wordlist for testing
cat > /tmp/small_wordlist.txt << 'EOF'
admin
staff
team
employees
login
wp-admin
.git
.env
api
config
backup
private
internal
EOF
ffuf -w /tmp/small_wordlist.txt -u https://example.com/FUZZ -mc 200,301,302,403 -fc 404 -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36' -rate 10 -o /tmp/ffuf_results.json -of json 2>/dev/null
echo "Results saved to /tmp/ffuf_results.json"
cat /tmp/ffuf_results.json | python3 -c "import json,sys; data=json.load(sys.stdin); [print(r['url'], r['status']) for r in data.get('results', [])]"

Cleanup

bash 
rm -f /tmp/small_wordlist.txt /tmp/ffuf_results.json

Expected Telemetry

Burst of 404 responses (12 requests, 1 per path) within 90 seconds. ffuf or spoofed browser UA. Requests for paths like /admin, /staff, /.git, /.env. Rate approximately 10 req/min.

Expected Detection

404 burst threshold detection triggers (Count404 > 5). Directory enumeration hunting query fires on recon seed (robots.txt/sitemap.xml) + 404 burst within 30min session.

Related Detections

Tactic Hub

TA0043Reconnaissance

Related Techniques

T1592Gather Victim Host InformationT1589Gather Victim Identity InformationT1591Gather Victim Org InformationT1595.003Wordlist ScanningT1596Search Open Technical DatabasesT1598Phishing for InformationT1566PhishingT1585Establish AccountsT1199Trusted Relationship

Same Tactic: Reconnaissance

Popular Detections