Response playbooks, investigation guides, and Atomic Red Team tests are Pro-only. Upgrade to unlock the full detection package for THREAT-Capability-MaliciousAIToolAbuse.
Upgrade to ProUnderground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation
Adversaries increasingly obtain artificial-intelligence capabilities rather than developing them from scratch, subscribing to underground uncensored LLM services (WormGPT-style and FraudGPT-style offerings sold on cybercrime forums) or abusing jailbreak prompts against mainstream commercial AI APIs to generate polished phishing email copy, localized BEC lures, obfuscated malware/loader code, and voice-cloned audio for vishing pretexts. This lowers the skill floor for producing convincing, grammatically flawless social-engineering content and functional malicious code without in-house development expertise. From a defender's vantage point this capability-acquisition activity is largely invisible until it manifests downstream, so detection pivots to two observable surfaces: (1) corporate egress traffic to known underground-AI-service infrastructure or anomalous, high-volume API usage patterns against legitimate LLM providers from unexpected internal contexts (e.g., a build server or unmanaged workstation making sustained completions-API calls), and (2) content-based indicators in received email and documents consistent with AI-generated phishing (unusually well-formed but semantically generic lure text, absent from any known template library, arriving from a newly-observed sender at high volume). This detection focuses defenders on the egress/API-abuse side of the equation, since it is the most directly observable within typical enterprise telemetry.
What is THREAT-Capability-MaliciousAIToolAbuse Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation?
Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation (THREAT-Capability-MaliciousAIToolAbuse) maps to the Resource Development tactic — the adversary is trying to establish resources they can use to support operations in MITRE ATT&CK.
This page provides production-ready detection logic for Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation, covering the data sources and telemetry it touches: Network Traffic: Network Connection Creation, Microsoft Defender for Endpoint (DeviceNetworkEvents). The queries below are rated medium severity at low confidence, and ship for 7 SIEM platforms — KQL, SPL, Elastic, QRadar, Sumo, YARA-L, LogScale.
MITRE ATT&CK
- Tactic
- Resource Development
// THREAT-Capability-MaliciousAIToolAbuse (T1588.007) - Underground/jailbroken AI service usage and anomalous LLM API abuse
// Pillar 1: Network connections to known underground AI-service hostnames/infrastructure patterns
let UndergroundAIIndicators = dynamic([
"wormgpt", "fraudgpt", "worm-gpt", "fraud-gpt", "evilgpt", "darkbard", "xxxgpt", "escapegpt"
]);
DeviceNetworkEvents
| where Timestamp > ago(24h)
| where RemoteUrl has_any (UndergroundAIIndicators)
| extend SuspicionReason = "UndergroundAIServiceContact"
| project Timestamp, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, RemoteUrl, RemoteIP, SuspicionReason
| union (
// Pillar 2: Anomalous sustained LLM completions-API usage from a non-developer/non-approved host
DeviceNetworkEvents
| where Timestamp > ago(24h)
| where RemoteUrl has_any ("api.openai.com", "api.anthropic.com", "generativelanguage.googleapis.com", "api.mistral.ai")
| where DeviceName !in ( // CUSTOMIZE: exclude your organization's approved AI-development/build hosts
"BUILDSRV01", "AIPLATFORM01"
)
| summarize CallCount=count(), FirstCall=min(Timestamp), LastCall=max(Timestamp) by DeviceName, InitiatingProcessFileName, RemoteUrl
| where CallCount >= 50
| extend SuspicionReason = "AnomalousLLMAPIVolumeFromUnapprovedHost"
| project Timestamp=LastCall, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine=strcat("CallCount=", tostring(CallCount)), RemoteUrl, RemoteIP="", SuspicionReason
)
| sort by Timestamp descTwo-pillar detection for adversary acquisition/use of AI capabilities. Pillar 1 flags any outbound network connection whose destination hostname matches known underground/jailbroken LLM-service naming patterns (WormGPT/FraudGPT-style branding commonly reused across cybercrime-forum offerings). Pillar 2 flags sustained high-volume API traffic to legitimate LLM providers (OpenAI, Anthropic, Google, Mistral) originating from hosts that are not on the organization's approved AI-development allowlist — a pattern consistent with an actor using a compromised host's egress or a stolen API key to mass-generate phishing/malware content rather than normal developer usage.
Data Sources
Required Tables
False Positives
- Legitimate AI-security-research or red-team activity investigating underground AI service offerings from an authorized research host
- Approved internal AI/ML development, chatbot, or automation workloads that legitimately generate high call volumes against OpenAI/Anthropic/Google APIs (must be captured in the exclusion list)
- Newly onboarded engineering teams or contractors using LLM APIs for approved development work from hosts not yet added to the allowlist
- SaaS integrations (e.g., a helpdesk or CRM plugin) that proxy LLM API calls through an on-prem gateway host
Sigma rule & cross-platform mapping
The detection logic for Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation (THREAT-Capability-MaliciousAIToolAbuse) above is provided in a vendor-neutral
form so you can deploy it on any SIEM. The same logic is shipped here as native
KQL (Microsoft Sentinel / Defender), SPL (Splunk), Elastic (Elastic Security (EQL)), QRadar (IBM QRadar (AQL)), Sumo (Sumo Logic CSE), YARA-L (Google Chronicle / SecOps), LogScale (CrowdStrike LogScale (CQL)) queries. In Sigma terms, this detection targets the
following logsource:
logsource:
category: network_connection
product: windowsBrowse the community-maintained Sigma rules for this technique:
Platform-specific guides for THREAT-Capability-MaliciousAIToolAbuse
References (5)
- https://attack.mitre.org/techniques/T1588/007/
- https://attack.mitre.org/tactics/TA0042/
- https://www.slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
- https://www.netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
Testing Methodology
Validate this detection against 2 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Outbound Connection Referencing Underground AI Service Naming Pattern
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: outbound connection attempt from powershell.exe to a hostname containing 'wormgpt'. DNS resolution will fail (NXDOMAIN) since the domain is a placeholder, but the connection attempt and DNS query are logged.
- Test 2Sustained High-Volume Requests to Legitimate LLM API Endpoint
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: approximately 55 outbound connections from powershell.exe to api.openai.com within roughly 15 seconds. Requests will return 401 Unauthorized due to the invalid test key, but the connection attempts are logged regardless.
Unlock playbooks & atomic tests with Pro
Get the full detection package for THREAT-Capability-MaliciousAIToolAbuse — response playbook and atomic red team tests, plus investigation guidance and hunting queries.
df00tech Pro — £29/user/month