Response playbooks, investigation guides, and Atomic Red Team tests are Pro-only. Upgrade to unlock the full detection package for THREAT-Capability-MaliciousAIToolAbuse.
Upgrade to ProDetect Underground and Jailbroken LLM Services Obtained for Phishing Lure and Malware Generation in Splunk
Adversaries increasingly obtain artificial-intelligence capabilities rather than developing them from scratch, subscribing to underground uncensored LLM services (WormGPT-style and FraudGPT-style offerings sold on cybercrime forums) or abusing jailbreak prompts against mainstream commercial AI APIs to generate polished phishing email copy, localized BEC lures, obfuscated malware/loader code, and voice-cloned audio for vishing pretexts. This lowers the skill floor for producing convincing, grammatically flawless social-engineering content and functional malicious code without in-house development expertise. From a defender's vantage point this capability-acquisition activity is largely invisible until it manifests downstream, so detection pivots to two observable surfaces: (1) corporate egress traffic to known underground-AI-service infrastructure or anomalous, high-volume API usage patterns against legitimate LLM providers from unexpected internal contexts (e.g., a build server or unmanaged workstation making sustained completions-API calls), and (2) content-based indicators in received email and documents consistent with AI-generated phishing (unusually well-formed but semantically generic lure text, absent from any known template library, arriving from a newly-observed sender at high volume). This detection focuses defenders on the egress/API-abuse side of the equation, since it is the most directly observable within typical enterprise telemetry.
MITRE ATT&CK
- Tactic
- Resource Development
SPL Detection Query
| union
[
search index=proxy OR index=network sourcetype="*proxy*" OR sourcetype="*firewall*"
(url="*wormgpt*" OR url="*fraudgpt*" OR url="*worm-gpt*" OR url="*fraud-gpt*" OR url="*evilgpt*" OR url="*escapegpt*")
| eval SuspicionReason="UndergroundAIServiceContact"
| table _time, host, user, url, dest_ip, SuspicionReason
]
[
search index=proxy OR index=network sourcetype="*proxy*" OR sourcetype="*firewall*"
(url="*api.openai.com*" OR url="*api.anthropic.com*" OR url="*generativelanguage.googleapis.com*" OR url="*api.mistral.ai*")
NOT host IN ("BUILDSRV01", "AIPLATFORM01")
| stats count as CallCount, earliest(_time) as FirstCall, latest(_time) as LastCall by host, url
| where CallCount >= 50
| eval SuspicionReason="AnomalousLLMAPIVolumeFromUnapprovedHost"
| rename LastCall as _time
| table _time, host, url, CallCount, SuspicionReason
]
| sort - _timeTwo-source detection combining web proxy/firewall URL matching against known underground AI-service naming patterns with a volumetric analysis of legitimate LLM API traffic, excluding an organization-defined allowlist of approved AI-development hosts. Requires a proxy or firewall log source with URL visibility and per-host attribution.
Data Sources
Required Sourcetypes
False Positives & Tuning
- Authorized security research into underground AI marketplaces from a sanctioned analysis host
- Legitimate high-volume AI development or automation workloads not yet added to the host exclusion list
- Internal AI gateway/proxy hosts that aggregate LLM API traffic on behalf of multiple approved applications
- Vendor SaaS tools that call LLM APIs through an on-prem integration host
Other platforms for THREAT-Capability-MaliciousAIToolAbuse
Testing Methodology
Validate this detection against 2 adversary techniques from Atomic Red Team. Each test below lists the behaviour to exercise and the telemetry you should expect to see. Executable commands and cleanup steps are available with Pro.
- Test 1Outbound Connection Referencing Underground AI Service Naming Pattern
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: outbound connection attempt from powershell.exe to a hostname containing 'wormgpt'. DNS resolution will fail (NXDOMAIN) since the domain is a placeholder, but the connection attempt and DNS query are logged.
- Test 2Sustained High-Volume Requests to Legitimate LLM API Endpoint
Expected signal: Sysmon Event ID 3 / DeviceNetworkEvents: approximately 55 outbound connections from powershell.exe to api.openai.com within roughly 15 seconds. Requests will return 401 Unauthorized due to the invalid test key, but the connection attempts are logged regardless.
References (5)
- https://attack.mitre.org/techniques/T1588/007/
- https://attack.mitre.org/tactics/TA0042/
- https://www.slashnext.com/blog/wormgpt-the-generative-ai-tool-cybercriminals-are-using-to-launch-business-email-compromise-attacks/
- https://www.netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/
Unlock playbooks & atomic tests with Pro
Get the full detection package for THREAT-Capability-MaliciousAIToolAbuse — response playbook and atomic red team tests, plus investigation guidance and hunting queries.
df00tech Pro — £29/user/month