Response playbooks, investigation guides, and Atomic Red Team tests are Pro-only. Upgrade to unlock the full detection package for THREAT-Capability-MaliciousAIToolAbuse.
Upgrade to ProDetect Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation in Microsoft Sentinel
Adversaries increasingly obtain artificial-intelligence capabilities rather than developing them from scratch, subscribing to underground uncensored LLM services (WormGPT-style and FraudGPT-style offerings sold on cybercrime forums) or abusing jailbreak prompts against mainstream commercial AI APIs to generate polished phishing email copy, localized BEC lures, obfuscated malware/loader code, and voice-cloned audio for vishing pretexts. This lowers the skill floor for producing convincing, grammatically flawless social-engineering content and functional malicious code without in-house development expertise. From a defender's vantage point this capability-acquisition activity is largely invisible until it manifests downstream, so detection pivots to two observable surfaces: (1) corporate egress traffic to known underground-AI-service infrastructure or anomalous, high-volume API usage patterns against legitimate LLM providers from unexpected internal contexts (e.g., a build server or unmanaged workstation making sustained completions-API calls), and (2) content-based indicators in received email and documents consistent with AI-generated phishing (unusually well-formed but semantically generic lure text, absent from any known template library, arriving from a newly-observed sender at high volume). This detection focuses defenders on the egress/API-abuse side of the equation, since it is the most directly observable within typical enterprise telemetry.
MITRE ATT&CK
- Tactic
- Resource Development
KQL Detection Query
// THREAT-Capability-MaliciousAIToolAbuse (T1588.007) - Underground/jailbroken AI service usage and anomalous LLM API abuse
// Pillar 1: Network connections to known underground AI-service hostnames/infrastructure patterns
let UndergroundAIIndicators = dynamic([
"wormgpt", "fraudgpt", "worm-gpt", "fraud-gpt", "evilgpt", "darkbard", "xxxgpt", "escapegpt"
]);
DeviceNetworkEvents
| where Timestamp > ago(24h)
| where RemoteUrl has_any (UndergroundAIIndicators)
| extend SuspicionReason = "UndergroundAIServiceContact"
| project Timestamp, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine, RemoteUrl, RemoteIP, SuspicionReason
| union (
// Pillar 2: Anomalous sustained LLM completions-API usage from a non-developer/non-approved host
DeviceNetworkEvents
| where Timestamp > ago(24h)
| where RemoteUrl has_any ("api.openai.com", "api.anthropic.com", "generativelanguage.googleapis.com", "api.mistral.ai")
| where DeviceName !in ( // CUSTOMIZE: exclude your organization's approved AI-development/build hosts
"BUILDSRV01", "AIPLATFORM01"
)
| summarize CallCount=count(), FirstCall=min(Timestamp), LastCall=max(Timestamp) by DeviceName, InitiatingProcessFileName, RemoteUrl
| where CallCount >= 50
| extend SuspicionReason = "AnomalousLLMAPIVolumeFromUnapprovedHost"
| project Timestamp=LastCall, DeviceName, InitiatingProcessFileName, InitiatingProcessCommandLine=strcat("CallCount=", tostring(CallCount)), RemoteUrl, RemoteIP="", SuspicionReason
)
| sort by Timestamp descTwo-pillar detection for adversary acquisition/use of AI capabilities. Pillar 1 flags any outbound network connection whose destination hostname matches known underground/jailbroken LLM-service naming patterns (WormGPT/FraudGPT-style branding commonly reused across cybercrime-forum offerings). Pillar 2 flags sustained high-volume API traffic to legitimate LLM providers (OpenAI, Anthropic, Google, Mistral) originating from hosts that are not on the organization's approved AI-development allowlist — a pattern consistent with an actor using a compromised host's egress or a stolen API key to mass-generate phishing/malware content rather than normal developer usage.
Data Sources
Required Tables
False Positives & Tuning
- Legitimate AI-security-research or red-team activity investigating underground AI service offerings from an authorized research host
- Approved internal AI/ML development, chatbot, or automation workloads that legitimately generate high call volumes against OpenAI/Anthropic/Google APIs (must be captured in the exclusion list)
- Newly onboarded engineering teams or contractors using LLM APIs for approved development work from hosts not yet added to the allowlist
- SaaS integrations (e.g., a helpdesk or CRM plugin) that proxy LLM API calls through an on-prem gateway host
Other platforms for THREAT-Capability-MaliciousAIToolAbuse
Testing Methodology
Validate this detection against 2 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Outbound Connection Referencing Underground AI Service Naming Pattern
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: outbound connection attempt from powershell.exe to a hostname containing 'wormgpt'. DNS resolution will fail (NXDOMAIN) since the domain is a placeholder, but the connection attempt and DNS query are logged.
- Test 2Sustained High-Volume Requests to Legitimate LLM API Endpoint
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: approximately 55 outbound connections from powershell.exe to api.openai.com within roughly 15 seconds. Requests will return 401 Unauthorized due to the invalid test key, but the connection attempts are logged regardless.
References (5)
- https://attack.mitre.org/techniques/T1588/007/
- https://attack.mitre.org/tactics/TA0042/
- https://www.slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
- https://www.netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
Unlock playbooks & atomic tests with Pro
Get the full detection package for THREAT-Capability-MaliciousAIToolAbuse — response playbook and atomic red team tests, plus investigation guidance and hunting queries.
df00tech Pro — £29/user/month