Response playbooks, investigation guides, and Atomic Red Team tests are Pro-only. Upgrade to unlock the full detection package for THREAT-Capability-MaliciousAIToolAbuse.
Upgrade to ProDetect Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation in Google Chronicle
Adversaries increasingly obtain artificial-intelligence capabilities rather than developing them from scratch, subscribing to underground uncensored LLM services (WormGPT-style and FraudGPT-style offerings sold on cybercrime forums) or abusing jailbreak prompts against mainstream commercial AI APIs to generate polished phishing email copy, localized BEC lures, obfuscated malware/loader code, and voice-cloned audio for vishing pretexts. This lowers the skill floor for producing convincing, grammatically flawless social-engineering content and functional malicious code without in-house development expertise. From a defender's vantage point this capability-acquisition activity is largely invisible until it manifests downstream, so detection pivots to two observable surfaces: (1) corporate egress traffic to known underground-AI-service infrastructure or anomalous, high-volume API usage patterns against legitimate LLM providers from unexpected internal contexts (e.g., a build server or unmanaged workstation making sustained completions-API calls), and (2) content-based indicators in received email and documents consistent with AI-generated phishing (unusually well-formed but semantically generic lure text, absent from any known template library, arriving from a newly-observed sender at high volume). This detection focuses defenders on the egress/API-abuse side of the equation, since it is the most directly observable within typical enterprise telemetry.
MITRE ATT&CK
- Tactic
- Resource Development
YARA-L Detection Query
rule underground_ai_service_abuse {
meta:
author = "Detection Engineering"
description = "Detects outbound contact with underground/jailbroken AI-service infrastructure used to generate phishing lures or malware"
severity = "MEDIUM"
confidence = "low"
mitre_attack = "T1588.007"
reference = "https://attack.mitre.org/techniques/T1588/007/"
events:
$e.metadata.event_type = "NETWORK_HTTP"
re.regex($e.target.url, `(?i)(wormgpt|fraudgpt|worm-gpt|fraud-gpt|evilgpt|escapegpt)`)
condition:
$e
}Google Chronicle YARA-L 2.0 detection for outbound HTTP traffic to hostnames matching known underground/jailbroken AI-service branding patterns — flags adversary acquisition of AI capability for generating phishing content or malicious code.
Data Sources
Required Tables
False Positives & Tuning
- Authorized security research from a sanctioned analysis host
- Threat intelligence tooling crawling these services for research purposes
Other platforms for THREAT-Capability-MaliciousAIToolAbuse
Testing Methodology
Validate this detection against 2 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Outbound Connection Referencing Underground AI Service Naming Pattern
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: outbound connection attempt from powershell.exe to a hostname containing 'wormgpt'. DNS resolution will fail (NXDOMAIN) since the domain is a placeholder, but the connection attempt and DNS query are logged.
- Test 2Sustained High-Volume Requests to Legitimate LLM API Endpoint
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: approximately 55 outbound connections from powershell.exe to api.openai.com within roughly 15 seconds. Requests will return 401 Unauthorized due to the invalid test key, but the connection attempts are logged regardless.
References (5)
- https://attack.mitre.org/techniques/T1588/007/
- https://attack.mitre.org/tactics/TA0042/
- https://www.slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
- https://www.netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
Unlock playbooks & atomic tests with Pro
Get the full detection package for THREAT-Capability-MaliciousAIToolAbuse — response playbook and atomic red team tests, plus investigation guidance and hunting queries.
df00tech Pro — £29/user/month