Automating Phishing Email Triage with Microsoft Security Copilot
This post was originally published on this site.
This blog details automating phishing email triage using Azure Logic Apps, Azure Function Apps, and Microsoft Security Copilot. Deployable in under 10 minutes, this solution primarily analyzes email intent without relying on traditional indicators of compromise, accurately classifying benign/junk, suspicious, and phishing emails. Benefits include reducing manual workload, improved threat detection, and (optional) integration seamlessly with Microsoft Sentinel – enabling analysts to see Security Copilot analysis within the incident itself.
Designed for flexibility and control, this Logic App is a customizable solution that can be self-deployed from GitHub. It helps automate phishing response at scale without requiring deep coding expertise, making it ideal for teams that prefer a more configurable approach and want to tailor workflows to their environment. The solution streamlines response and significantly reduces manual effort.
Access the full solution on the Security Copilot Github:
GitHub – UserReportedPhishing Solution.
For teams looking for a more sophisticated, fully integrated experience, the Security Copilot Phishing Triage Agent represents the next generation of phishing response. Natively embedded in Microsoft Defender, the agent autonomously triages phishing incidents with minimal setup. It uses advanced LLM-based reasoning to resolve false alarms, enabling analysts to stay focused on real threats. The agent offers step-by-step decision transparency and continuously learns from user feedback. Read the official announcement here.
Introduction: Phishing Challenges Continue to Evolve
Phishing continues to evolve in both scale and sophistication, but a growing challenge for defenders isn’t just stopping phishing, it’s scaling response. Thanks to tools like Outlook’s “Report Phishing” button and increased user awareness, organizations are now flooded with user-reported emails, many of which are ambiguous or benign. This has created a paradox: better detection by users has overwhelmed SOC teams, turning email triage into a manual, rotational task dreaded for its repetitiveness and time cost, often taking over 25 minutes per email to review.
Our solution addresses that problem, by automating the triage of user-reported phishing through AI-driven intent analysis. It’s not built to replace your secure email gateways or Microsoft Defender for Office 365; those tools have already done their job. This system assumes the email:
- Slipped past existing filters,
- Was suspicious enough for a user to escalate,
- Lacks typical IOCs like malicious domains or attachments.
As a former attacker, I spent years crafting high-quality phishing emails to penetrate the defenses of major banks. Effective phishing doesn’t rely on obvious IOCs like malicious domains, URLs, or attachments… the infrastructure often appears clean. The danger lies in the intent. This is where Security Copilot’s LLM-based reasoning is critical, analyzing structure, context, tone, and seasonal pretexts to determine whether an email is phishing, suspicious, spam, or legitimate.
What makes this novel is that it’s the first solution built specifically for the “last mile” of phishing defense, where human suspicion meets automation, and intent is the only signal left to analyze. It transforms noisy inboxes into structured intelligence and empowers analysts to focus only on what truly matters.
Solution Overview: How the Logic App Solution Works (and Why It’s Different)
Core Components:
- Azure Logic Apps: Orchestrates the entire workflow, from ingestion to analysis, and 100% customizable.
- Azure Function Apps: Parses and normalizes email data for efficient AI consumption.
- Microsoft Security Copilot: Performs sophisticated AI-based phishing analysis by understanding email intent and tactics, rather than relying exclusively on predefined malicious indicators.
Key Benefits:
- Rapid Analysis: Processes phishing alerts and, in minutes, delivers comprehensive reports that empower analysts to make faster, more informed triage decisions – compared to manual reviews that can take up to 30 minutes. And, unlike analysts, Security Copilot requires zero sleep!
- AI-driven Insights: LLM-based analysis is leveraged to generate clear explanations of classifications by assessing behavioral and contextual signals like urgency, seasonal threats, Business Email Compromise (BEC), subtle language clues, and otherwise sophisticated techniques. Most importantly, it identifies benign emails, which are often the bulk of reported emails.
- Detailed, Actionable Reports: Generates clear, human-readable HTML reports summarizing threats and recommendations for analyst review.
- Robust Attachment Parsing: Automatically examines attachments like PDFs and Excel documents for malicious content or contextual inconsistencies.
- Integrated with Microsoft Sentinel: Optional integration with Sentinel ensures central incident tracking and comprehensive threat management. Analysis is attached directly to the incident, saving analysts more time.
- Customization: Add, move, or replace any element of the Logic App or prompt to fit your specific workflows.
Deployment Guide: Quick, Secure, and Reliable Setup
The solution provides Azure Resource Manager (ARM) templates for rapid deployment:
Prerequisites:
- Azure Subscription with Contributor access to a resource group.
- Microsoft Security Copilot enabled.
- Dedicated Office 365 shared mailbox (e.g., phishing@yourdomain.com) with Mailbox.Read.Shared permissions.
- (Optional) Microsoft Sentinel workspace.
Refer to the up to date deployment instructions on the Security Copilot GitHub page.
Technical Architecture & Workflow:
The automated workflow operates as follows:
Email Ingestion:
- Monitors the shared mailbox via Office 365 connector.
- Triggers on new email arrivals every 3 minutes.
- Assumes that the reported email has arrived as an attachment to a “carrier” email.
Determine if the Email Came from Defender/Sentinel:
If the email came from Defender, it would have a prepended subject of “Phishing”, if not, it takes the “False” branch. Change as necessary.
Initial Email Processing:
- Exports raw email content from the shared mailbox.
- Determines if .msg or .eml attachments are in binary format and converts if necessary.
Email Parsing via Azure Function App:
- Extracts data from email content and attachments (URLs, sender info, email body, etc.) and returns a JSON structure.
- Prepares clean JSON data for AI analysis.
- This step is required to “prep” the data for LLM analysis due to token limits.
- Click on the “Parse Email” block to see the output of the Function App for any troubleshooting. You’ll also notice a number of JSON keys that are not used but provided for flexibility.
Security Copilot Advanced AI Reasoning:
- Analyzes email content using a comprehensive prompt that evaluates behavioral and seasonal patterns, BEC indicators, attachment context, and social engineering signals.
- Scores cumulative risk based on structured heuristics without relying solely on known malicious indicators.
- Returns validated JSON output (some customers are parsing this JSON and performing other action).
- This is where you would customize the prompt, should you need to add some of your own organizational situations if the Logic App needs to be tuned:
JSON Normalization & Error Handling:
- A “normalization” Azure Function ensures output matches the expected JSON schema.
- Sometimes LLMs will stray from a strict output structure, this aims to solve that problem.
- If you add or remove anything from the Parse Email code that alters the structure of the JSON, this and the next block will need to be updated to match your new structure.
Detailed HTML Reporting:
- Generates a detailed HTML report summarizing AI findings, indicators, and recommended actions.
- Reports are emailed directly to SOC team distribution lists or ticketing systems.
Optional Sentinel Integration:
Adds the reasoning & output from Security Copilot directly to the incident comments. This is the ideal location for output since the analyst is already in the security.microsoft.com portal. It waits up to 15 minutes for logs to appear, in situations where the user reports before an incident is created.
The solution works pretty well out of the box but may require some tuning, give it a test. Here are some examples of the type of Security Copilot reasoning.
Benign email detection:
Example of phishing email detection:
More sophisticated phishing with subtle clues:
Enhanced Technical Details & Clarifications
Attachment Processing:
- When multiple email attachments are detected, the Logic App processes each binary-format email sequentially.
- If PDF or Excel attachments are detected, they are parsed for content and are evaluated appropriately for content and intent.
Security Copilot Reliability:
- The Security Copilot Logic App API call uses an extensive retry policy (10 retries at 10-minute intervals) to ensure reliable AI analysis despite intermittent service latency.
- If you run out of SCUs in an hour, it will pause until they are refreshed and continue.
Sentinel Integration Reliability:
- Acknowledges inherent Sentinel logging delays (up to 15 minutes).
- Implements retry logic and explicit manual alerting for unmatched incidents, if the analysis runs before the incident is created.
Security Best Practices:
- Compare the Function & Logic App to your company security policies to ensure compliance.
- Credentials, API keys, and sensitive details utilize Azure Managed Identities or secure API connections. No secrets are stored in plaintext.
- Azure Function Apps perform only safe parsing operations; attachments and content are never executed or opened insecurely.
Be sure to check out how the Microsoft Defender for Office team is improving detection capabilities as well Microsoft Defender for Office 365’s Language AI for Phish: Enhancing Email Security | Microsoft Community Hub.
