Response playbooks, investigation guides, and Atomic Red Team tests are Pro-only. Upgrade to unlock the full detection package for THREAT-Capability-MaliciousAIToolAbuse.
Upgrade to ProDetect Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation in Elastic Security
Adversaries increasingly obtain artificial-intelligence capabilities rather than developing them from scratch, subscribing to underground uncensored LLM services (WormGPT-style and FraudGPT-style offerings sold on cybercrime forums) or abusing jailbreak prompts against mainstream commercial AI APIs to generate polished phishing email copy, localized BEC lures, obfuscated malware/loader code, and voice-cloned audio for vishing pretexts. This lowers the skill floor for producing convincing, grammatically flawless social-engineering content and functional malicious code without in-house development expertise. From a defender's vantage point this capability-acquisition activity is largely invisible until it manifests downstream, so detection pivots to two observable surfaces: (1) corporate egress traffic to known underground-AI-service infrastructure or anomalous, high-volume API usage patterns against legitimate LLM providers from unexpected internal contexts (e.g., a build server or unmanaged workstation making sustained completions-API calls), and (2) content-based indicators in received email and documents consistent with AI-generated phishing (unusually well-formed but semantically generic lure text, absent from any known template library, arriving from a newly-observed sender at high volume). This detection focuses defenders on the egress/API-abuse side of the equation, since it is the most directly observable within typical enterprise telemetry.
MITRE ATT&CK
- Tactic
- Resource Development
Elastic Detection Query
network where event.type == "connection" and network.direction == "outbound" and
(destination.domain : ("*wormgpt*", "*fraudgpt*", "*worm-gpt*", "*fraud-gpt*", "*evilgpt*", "*escapegpt*")
or (destination.domain : ("api.openai.com", "api.anthropic.com", "generativelanguage.googleapis.com", "api.mistral.ai") and
not host.name : ("BUILDSRV01", "AIPLATFORM01")))Elastic EQL detection for outbound connections to underground/jailbroken AI-service infrastructure by naming pattern, or to legitimate LLM API providers from hosts outside the organization's approved AI-development allowlist — surfacing adversary acquisition or abuse of AI capabilities for phishing/malware content generation.
Data Sources
Required Tables
False Positives & Tuning
- Authorized AI security research from a sanctioned host
- Approved AI/ML development or automation workloads not yet on the allowlist
- Newly onboarded engineering hosts using LLM APIs for legitimate development
Other platforms for THREAT-Capability-MaliciousAIToolAbuse
Testing Methodology
Validate this detection against 2 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Outbound Connection Referencing Underground AI Service Naming Pattern
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: outbound connection attempt from powershell.exe to a hostname containing 'wormgpt'. DNS resolution will fail (NXDOMAIN) since the domain is a placeholder, but the connection attempt and DNS query are logged.
- Test 2Sustained High-Volume Requests to Legitimate LLM API Endpoint
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: approximately 55 outbound connections from powershell.exe to api.openai.com within roughly 15 seconds. Requests will return 401 Unauthorized due to the invalid test key, but the connection attempts are logged regardless.
References (5)
- https://attack.mitre.org/techniques/T1588/007/
- https://attack.mitre.org/tactics/TA0042/
- https://www.slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
- https://www.netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
Unlock playbooks & atomic tests with Pro
Get the full detection package for THREAT-Capability-MaliciousAIToolAbuse — response playbook and atomic red team tests, plus investigation guidance and hunting queries.
df00tech Pro — £29/user/month