Microsoft SC200 - Sentinel SIEM/SOAR Part 1

Microsoft Sentinel Overview

SIEM, or Security Information and Event Management, is a security solution that helps organizations identify and respond to potential security threats by collecting, analyzing, and correlating security events and data from various sources. Sentinel is Microsoft SIEM product. Comparable products to Sentinel include:

Splunk
Elastic
CrowdStrike Falcon scale
Google SecOps
Cortex XSIAM

One of the things that is important for SIEMs to be successful is the instrumentation and telemetry feeding it. At a high level there are two high level types of instrumentation systems that produce telemetry data for cyber security. They are Network Detection Response (NDR), and Endpoint Detection Response (EDR). NDR and EDR are the lower corners of the SOC Triad. With SIEM data, SOC analysts, threat hunters, and cybersecurity engineers can create queries to detect suspicious activity, investigate incidents, and proactively hunt for threats, uncovering potential risks before they escalate.

Key Technologies for SOC Success

Beyond SIEM, several critical technologies enhance SOC operations:

Security Orchestration, Automation & Response (SOAR)
Cyber Threat Intelligence (CTI)
Asset Management

The Role of SOAR

Having data in a SIEM is just the start, SOC analysts still face operational and technical challenges. A major issue is that SIEM alerts require investigation, which is time-consuming and often repetitive. This is where SOAR comes in, automating workflows to streamline responses. Many SIEM vendors have integrated SOAR capabilities, while third-party SOAR solutions offer independent integrations.

Integrated SIEM/SOAR Solutions:

Splunk – Acquired Phantom SOAR
Cortex XSIAM – Acquired Demisto SOAR
Microsoft Sentinel – Uses Logic Apps for automation

Third-Party SOARs:

Torq
Tines
Swimlane

The Role of CTI & Asset Management

CTI and SOAR accelerate SOC operations by enhancing detection, investigation, and response. Integrating CTI with SIEM ensures broader coverage for log sources lacking native threat intelligence and strengthens security insights beyond individual security tools.

Asset management, while not a direct SIEM/SOAR function, is fundamental. The first step in responding to a SIEM alert is identifying the asset, understanding its behavior, and determining its owner. A well-managed asset inventory improves incident response and security visibility across the organization.

The Evolution from SIEM to EDR and XDR in SOC Operations

As SOCs evolved, Endpoint Detection and Response (EDR) became a critical component of security stacks, often competing with traditional SIEM solutions. Unlike SIEM, which relies on log ingestion and queries, EDR operates with agents, providing superior telemetry, real-time protection, and deeper investigative capabilities for endpoint security.

SIEM vs. EDR: Shifting the SOC Workflow

While EDR alerts should be integrated into SIEM, meaningful investigation and response, such as timeline analysis, user activity tracking, and containment, still require analysts to pivot to the EDR console. This has led to SOCs operating with two primary consoles: SIEM for log-based detections and EDR for endpoint-centric investigations and actions.

For many SOCs, EDR has replaced SIEM as the primary tool for real-time monitoring and response.

The Rise of XDR and the SOC Tetrad

With the shift to cloud-based identity providers and applications, a new category of security tooling emerged: eXtended Detection and Response (XDR). XDR expands beyond endpoints, integrating network, cloud, and identity telemetry to provide holistic visibility and automated response capabilities. Given the significance of XDR in modern security operations, many argue it warrants its own category within SOC monitoring frameworks, proposing that the traditional SOC triad should evolve into a 'SOC Tetrad' or 'SOC Pentad', or 'SOC Hexad' to include other key source systems for cyber instrumentation and telemetry like 'Identity Threat Detection & response' (ITDR), or 'Application Detection & Response' (ADR).

Why SIEM Still Matters in an EDR/XDR-Driven SOC

While much of the action has shifted to EDR/XDR platforms, SIEM remains essential. Its primary value lies in alerting and correlation from non-endpoint sources such as firewalls, proxies, NDR tools, and other infrastructure logs.

During investigations, analysts often need to pivot to these log sources to gain broader context or uncover attack paths that EDR/XDR alone can't fully capture. SIEM provides that centralized visibility and log depth, still a critical piece of the modern SOC puzzle.

Microsoft Sentinel functions as a long-term retention and advanced query platform for Microsoft Defender XDR telemetry, enabling extended investigation, correlation, and compliance-driven data retention.

Microsoft Sentinel SIEM and Defender XDR

This Microsoft diagram illustrates the integration of Microsoft Defender XDR and Microsoft Sentinel, highlighting their roles in collecting and analyzing telemetry from various sources. Key takeaways include: distinct consoles for investigation and monitoring in Defender XDR and Sentinel; automatic forwarding of Defender XDR data and alerts into Sentinel for centralized analysis; Sentinel is the main source for collecting data for both Microsoft cloud and 3rd party cloud services because this is often cloud resource log data (Covered in defender for Cloud section), and ingestion of third-party log data directly into Sentinel for broader visibility and correlation.

Using Microsoft Defender XDR alongside a non-Microsoft SIEM like Elastic results in two separate platforms to monitor, requiring custom integrations for data exchange and workflow automation. Historically, even with Microsoft Sentinel, Defender XDR integration was limited, alerts flowed from Defender XDR to Sentinel, but investigations often required pivoting back to Defender XDR. Due to customer demand for tighter integration, Microsoft enhanced interoperability, shifting the primary investigation workflow to Defender XDR while allowing Sentinel to push its alerts and data into Defender XDR. While SIEM remains critical for aggregating and monitoring diverse log sources, including cloud resources, firewalls, proxies, and NDR, Defender XDR now serves as the central hub for security operations. One key benefit of integrating Microsoft Sentinel with Defender XDR in the Defender portal is the streamlined incident response workflow. It reduces operational overhead and minimizes the risk of errors associated with context switching between separate tools.

Microsoft Sentinel Configuration – Set up

In Microsoft’s ecosystem sentinel is a cloud native SIEM, that relies on data stored in Log Analytics Workspaces (LAW), which acts as the central repository. Microsoft Sentinel is then layered on top of the LAW, with each Sentinel instance connecting exclusively to a single LAW. This structure is similar to the relationship between Elastic and the Kibana Elastic Security App, where Elastic stores the raw log data and Kibana Elastic Security App serves as the analytical overlay.

Currently, Microsoft suggests separate log workspaces for every 100GB+ per day injest as well as different geographic regions. The diagram on the left displays a single workspace with a single Sentinel instance. The diagram on the right displays two geographic workspaces.

The below diagram displays multiple tenants and multiple workspaces under each tenant that are managed centrally with Azure Lighthouse.

Microsoft Sentinel Configuration – Configuration

Microsoft Sentinel follows the Microsoft Defender model, correlating related alerts from various sources into a single incident for streamlined investigation. A single incident in Sentinel can aggregate multiple alerts, providing a comprehensive view.

Upon logging into Sentinel, the left-hand navigation pane (blades) provides access to key features and configurations:

General – Overview, Logs, Search, News & Guides
Threat Management – Incidents, Workbooks, Hunting, Notebooks, Entity Behavior, Threat Intelligence, MITRE ATT&CK, SOC Optimization
Content Management – Content Hub, Repositories, Community
Configuration – Workspace Manager, Data Connectors, Analytics, Summary Rules, Watchlists, Automation, Settings

We'll start by reviewing the Configuration blade.

Log Integration – Connect and analyze logs in Microsoft Sentinel

For the SC-200 Security Analyst exam, you don’t need deep expertise in Microsoft log management, but understanding the fundamentals is essential. One of the most critical parts of any SIEM, especially Microsoft Sentinel, is log ingestion. Without the right data, you can’t generate meaningful alerts or support investigations and threat hunting.

In Sentinel, data used for detection and analysis comes from a Log Analytics Workspace (LAW). While Microsoft’s logging architecture can seem complex, a helpful analogy is to think of the LAW as a massive syslog server, and Sentinel as the security application using that data for alerting and analysis.

The diagram below from Microsoft's "How Azure Monitor Logs Work" illustrates four core components of this process:

Collect any data
Manage and optimize log data and cost
Retrieve data in near real-time
Use data flexibly

Other important points to understand are:

Understand the distinction between security telemetry, operational telemetry, and the overlap between them.
Sentinel typically involves log data from both Microsoft and non-Microsoft sources that is integrated using 'Data Connectors'.
Many Microsoft services collect telemetry by default, but not all logs are enabled out of the box.
To capture specific types of log data, you’ll often need to enable auditing or export settings within the source system itself.

Distinguishing Between Security and Operational Log Telemetry

A common pitfall in SIEM implementation is collecting too much data without considering its relevance. SIEM vendors and auditors often advocate for retaining all logs indefinitely—but they're rarely the ones managing the associated storage costs or performance impacts. The result? Many SIEMs become bloated with unnecessary logs while missing critical ones.

To avoid this, it’s essential to distinguish between security logs and operational logs early in your deployment:

Security logs are directly used for threat detection, investigation, and alerting.
Operational logs, such as firewall flow logs, proxy logs, NetFlow, and server diagnostics, may generate high volume but often have limited direct value for security detection.

Storing excessive operational logs can degrade query performance and drive-up storage costs. The cybersecurity team should lead the decision-making process to prioritize what’s ingested.

Microsoft’s model, outlined in the article “Leave No Data Behind: Using Summary Rules to Store Data Cost Effectively in Microsoft Sentinel”, categorizes telemetry into Primary Security Data and Secondary Security Data, providing a helpful framework for this selection process.

Primary vs. Secondary Data Sources in Microsoft Sentinel

Primary Data Sources

Primary data sources are essential for high-value detection and correlation use cases. These include:

Logs from antivirus and Endpoint Detection and Response (EDR) systems
Authentication logs (e.g., Azure AD Sign-ins)
Cloud platform audit trails (e.g., Azure Activity Logs)
Threat intelligence feeds
Alerts generated by security and external monitoring systems

These logs are typically actionable, high-fidelity, and central to security operations and incident detection.

Secondary Data Sources

Secondary log sources provide additional context but are often lower in fidelity or generate high volumes with less direct value. Examples include:

Cloud storage access logs
NetFlow or VPC flow logs
TLS/SSL certificate logs
Firewall and proxy logs
IoT telemetry logs

Fun Fact for Cyber Pros: The saying, “Any log with the word ‘flow’ is typically high volume, costly, and of low value,” is an inside joke in the SIEM and cybersecurity world—for good reason. Flow logs can be useful but often require heavy filtering to extract meaningful insight.

Managing Log Volume and Cost with Microsoft Sentinel Log Plans

Microsoft provides three log ingestion plans to balance retention, performance, and cost:

Analytics Logs Plan (Default)

Retention: Interactive retention for 90 days by default (extendable up to 2 years)
- Log Analytics workspace defaults retains data for 30 days except for log tables (example tables Sentinel uses), which have a default 90-day retention.
Querying: Unlimited, high-performance KQL querying with no per-query cost
Long-Term Retention: Optional retention up to 12 years at low storage cost
Accessing Archived Data:
- Use Restore to bring data back to interactive state
- Use Search jobs to scan archived data without restoring

Auxiliary Logs Plan

Designed for high-volume, lower-priority logs like NetFlow, DNS, and proxy logs.

Retention: Interactive for 30 days, at significantly reduced cost
Querying:
- Charged per GB scanned
- Limited to single-table queries
- Reduced performance
Use Case: Run summary rules to extract aggregate data into the Analytics plan for full querying capabilities
Post-Retention: After 30 days, data enters long-term retention (searchable via search jobs only)
- Restore is not supported in the Auxiliary plan

Basic Logs Plan (For select data types)

Intended for logs useful mainly for troubleshooting or general operations (not covered in detail here, but includes things like AppServiceHTTPLogs, AzureDiagnostics for specific services, etc.)

The diagram below shows the different log plan defaults as well as the transition from log analytics interactive log storage to long term cheaper storage.

The interactive retention period as well as the total retention period can be viewed by table in the log analytics workspace.

Sentinel volume ingestion and costs per GB ingest per day tiers can also be viewed under the Sentinel Configuration 'Settings' option.

Accessing Archived Data with Search Jobs or Restore

Once logs age out of the interactive retention window, they are archived in long-term storage (e.g., Azure Data Lake or Blob Storage). To access this data:

Use Search Jobs to query archived data without restoring it
Restore logs: Data cannot be queried interactively until restored (only supported in Analytics plan)

KQL and Archive Data Search in Microsoft Sentinel: Understanding the Options

Standard KQL (Kusto Query Language) queries are limited to searching within the interactive data retention period and come with a 10-minute timeout. This makes them less effective for querying large volumes of archived data stored on cost-effective, long-term storage.

For those scenarios, Search is the better tool. It divides a single query into multiple parallel jobs and allows for up to a 24-hour timeout, making it ideal for scanning massive datasets. Search works best when you're looking for log events that match specific search terms.

Alternatively, Log Restore is designed for restoring large chunks of archived data from a single table without needing to define a search term. This is useful when you want to bring historical logs back into the interactive environment for further analysis or investigation.

To restore logs in Microsoft Sentinel, you simply specify the table and time range. The restored data becomes temporarily available for high-performance querying in Log Analytics. Costs are calculated based on the volume of data restored and how long it remains active, billed per GB per day, with a minimum charge of 2TB per restore and a minimum duration of 12 hours.

Sentinel typically involves log data from both Microsoft and non-Microsoft sources that is integrated using 'Data Connectors'

Microsoft Sentinel uses Data Connectors to ingest log data from various sources. These connectors normalize incoming data during ingestion and often include built-in content such as analytics rules, workbooks, and hunting queries tailored to the specific log source.

Data Connectors in Sentinel are similar in function to Elastic Integrations and Splunk Connectors.

You can view and configure built-in connectors by navigating to "Data Connectors" under the Configuration blade in Sentinel.

Key Data Connector Concepts:

Connected Connectors: Actively ingesting data into Sentinel.
Onboarded Connectors: Configured and enabled to begin data ingestion.

Microsoft regularly updates connectors to include:

New log source integrations
Enhanced analytics rules
Additional out-of-the-box (OOTB) content

The Content Hub in Sentinel serves as a centralized location to discover and install pre-built content, including data connectors, playbooks, workbooks, and more, to accelerate deployment and improve threat detection coverage.

To simplify how Microsoft Sentinel Data Connectors work, they generally fall into the following categories:

Direct
- Used for Microsoft cloud services like Azure Activity Logs, Microsoft Defender for Cloud, and Microsoft 365 Defender.
- These services send data directly to Sentinel, often via built-in diagnostic settings or native integrations.
Agent-Based
- Relies on the Azure Monitor Agent (AMA) or Log Analytics Agent installed on virtual machines or endpoints.
- Collects logs such as Windows Event Logs, Sysmon, and performance counters from on-prem or cloud systems.
Syslog/CEF
- Syslog is a standard protocol supported by most network devices and Linux servers for log export.
- CEF (Common Event Format) is used by some security appliances and products for structured log output.
- Ideal for non-Windows systems or devices without a native agent.
Threat Intelligence (TI)
- Enables ingestion of external threat intel feeds, including indicators of compromise (IOCs), from trusted sources or custom feeds.
Custom/API
- Supports custom log ingestion via APIs, Logic Apps, or custom connectors for services without built-in Sentinel integrations such as code-less connectors.

Sentinel - Direct Connectors

Sentinel Direct Connectors are used for Microsoft products that generate and manage their own logs independently. The most common examples of Azure data sources that have security telemetry are:

Entra ID (sign-in and audit logs)
Azure Activity
Defender for Endpoint (EDR/XDR)
Defender for Cloud.

When an organization uses one of these Microsoft products, it typically has default access to 30 days of log data within that product as shown below in the left side Azure portal. However, to enable advanced use cases such as centralized monitoring and alerting, threat hunting, and longer-term log retention, organizations can forward that log data to Microsoft Sentinel via a Direct Connector.

Azure Event Hubs

Azure Event Hubs is a high-throughput, low-latency data streaming platform capable of ingesting millions of events per second. It supports integration with multiple systems and is natively compatible with Apache Kafka, allowing existing Kafka-based applications to run without code changes.

While not a built-in component of Microsoft Sentinel, Event Hubs plays a critical role in forwarding telemetry from Microsoft services to external systems, such as third-party SIEMs.

For example, if you're using Azure-native services and want to integrate with an Elastic SIEM, you would configure diagnostic settings on Microsoft resources (e.g., audit logs, activity logs) to stream logs to Event Hubs. Event Hubs then acts as a bridge, delivering that data to the third-party SIEM either by being pulled by the external system or pushed via log forwarding mechanisms.

Sentinel - Agent Connectors

Sentinel Agent Data Connectors use the Azure Monitor Agent (AMA) to collect and send telemetry from endpoints to Microsoft Sentinel. The agent is installed on the source system and enables telemetry collection directly from that machine.

In the broader industry, such agents are often referred to as log shippers, as they are designed to forward logs from the host to a centralized logging platform.

Key points about logging agents:

They are not limited to security telemetry; their primary use is often for operational data, such as performance metrics and diagnostics.
Logging agents typically provide granular control over what types of telemetry are collected and forwarded, helping optimize cost and relevance. More advanced agents also have a feature to centrally configure and control what log data the agent collects.
- Microsoft AMA Agent Data Collection Rules (DCR)'s
- Elastic Agent Fleet Policy
In many environments, logs are first sent to an intermediary system (e.g., a log collector or log forwarder) before being ingested by a SIEM. This architecture is especially common with Syslog-based systems.

Common industry logging agents.

Microsoft:
- Azure Monitor Agent (AMA)
Beats (Used with Elastic):
- Includes Filebeat, Winlogbeat, Auditbeat, and others.
- Website: Elastic Beats
Logstash:
- Website: Elastic Logstash
Snare:
- Website: Snare Solutions
Vector (Acquired by DataDog):
- Website: Vector
Splunk Forwarder/Heavy Forwarder:
- Used for sending logs to Splunk.
- Documentation: Splunk Forwarder
ArcSight Smart Connector:
- Used for sending logs to ArcSight.
- Website: ArcSight Smart Connector

Azure Monitor Agent (AMA)

The Azure Monitor Agent (AMA) collects telemetry from the guest OS of Azure VMs, on-premises servers, and hybrid environments. It sends this data to Azure Monitor and is commonly used by Microsoft Sentinel and Microsoft Defender for Cloud.

In Microsoft Sentinel, the AMA Data Connector streams logs from Windows servers (virtual, physical, or cloud-based) into the Sentinel workspace.

Note: You may encounter references to the older Log Analytics Agent (also known as the Microsoft Monitoring Agent, MMA). You will also see this old agent referred to as the Operations Management Suite (OMS) Agent, which was an MMA agent specifically for Linux systems. This legacy agent was officially retired in 2024, and organizations are encouraged to use AMA moving forward.

Data Collection Rules (DCR)

AMA relies on Data Collection Rules (DCRs) to define how data is collected and processed. A DCR specifies:

What data to collect (e.g., event logs, performance counters)
How to process the data (e.g., filter, transform, or shape)
Where to send the data (e.g., Log Analytics Workspace)

To activate a DCR for a specific machine, a Data Collection Rule Association (DCRA) links the DCR to the agent. Multiple agents can share the same DCR, and one agent can be associated with multiple DCRs.

Agents periodically check Azure Monitor for DCR updates, ensuring they always operate with the latest configuration. DCRs also enable Sentinel engineers to control log ingestion into standard or custom tables, depending on the use case.

The diagram highlights how DCR rules on different log flows can work to integrate data into different log tables.

Centralizing log collection on a device before Sentinel

As noted in the section discussing log shippers and agents a common log collection setup is to send logs to an intermediary device before ingesting inti SIEM. In these instances logs are sent to a central server via rsyslog or Windows Event Collection (WEC) for Windows and a single log agent is installed and managed on that intermediary collection system that process es the logs into SIEM.

The key inquiries regarding intermediary log shipping include:

1. System Compatibility:

Can the system you are logging from accommodate the installation of log shippers?
Does the system inherently support native log shipping?

2. Control Over Log injest Pipeline:

Does opting for intermediary log shipping provide the necessary control over the log injest pipeline?

3. Administrative Considerations:

Are system administrators willing to support additional software deployments across all systems under their purview?

Does the system you are logging from support the installation of log shippers?

For specific log sources, particularly network equipment such as routers, switches, and firewalls with operating systems that don't allow the installation of a standalone log shipper, a common alternative is available. These devices generally support sending logs to a central location through remote syslog. Therefore, if network logging is required from devices like firewalls, routers, and switches, it's likely necessary to have a dedicated server with a log shipper to receive and process the logs from these network devices.

Does the system you are logging from support native log shipping?

Historically, Windows systems lacked native capabilities to send their logs directly, in contrast to Linux systems and other network devices that could utilize "Remote Syslog" through rsyslog for native log shipping. To collect logs into the log database or SIEM, organizations had to install and manage a log shipper/agent on all Windows systems. Each individual Windows log shipper sent collected logs to the backend log database or SIEM, either directly or through an intermediary log shipper, provided the Windows log shipper/agent could convert logs to syslog. While some specialized log shipper tools could remotely log directly into Windows systems, these solutions were often suboptimal, relying on weak SMBv1-based logons and offering limited granularity in the types of logs that could be collected (e.g., all security logs and/or all system logs).

If you cannot install an AMA agent on every device you want to collect logs from you can leverage the native Syslog daemon on Linux-based systems and network appliances (e.g., routers, switches, firewalls, and other Linux servers) to forward local event logs to a centralized Syslog server. To integrate this with Microsoft Sentinel, install the Azure Monitor Agent (AMA) on the centralized Syslog server.

This setup allows the AMA to forward all collected Syslog data into Sentinel for analysis and monitoring.

To configure the integration:

Deploy the Syslog Data Connector in Microsoft Sentinel.
Install and onboard the AMA on the Syslog collector server.
During onboarding, provide the Log Analytics Workspace ID and either the primary or secondary key to authenticate the agent with Sentinel.

'Windows Event Collection' (WEC) & Windows Event Forwarding" (WEF),

Several years ago, Windows introduced WEC and WEF (Name depends upon push or pull option), enabling Windows systems to utilize native remoting to transport logs from end sources to a centralized Windows Log Collector. While a log shipper is still required on the WEF server, the introduction of WEF significantly diminishes the number of log shipper/agents that need deployment and management.

In summary to collect Windows logs from Windows servers there are two options:

Option 1 - Install an AMA agent on every server that processes events into Sentinel from each AMA agent on each server.
Option 2 - Have Windows servers/workers forward logs to a centralized Windows Event Collection/Forwarding (WEC/F) server and install an AMA agent on that central server that then processes the events collected centrally from all the servers to Sentinel.

Issues with Windows Event Collection (WEC) and entre joined machines

WEC is an excellent alternative for collecting logs from workstations that either lack Defender for Endpoint or use a different EDR solution. It’s a valuable supplement, especially if you want to enrich your detection capabilities with native Windows event logs.

Workstation telemetry remains crucial, after all, most intrusions originate on endpoints, and WEC offers an agentless solution to collect that data efficiently. It works by having a central collector system use native Windows Remote Management (WinRM) over port 5985 to pull logs from endpoints and servers.

These logs are then processed by a logging agent on the WEC server and forwarded to your organization’s SIEM for analysis. Authentication between the WEC collector and endpoints is Kerberos-based, assuming both are joined to Active Directory. Although the collection uses HTTP, communication is still encrypted using WinRM’s Kerberos-based encryption mechanism.

Source: Randy Franklin Smith's Ultimate Windows Security

As more organizations shift toward cloud-based identity models, traditional WEC (Windows Event Collection) setups face challenges. Increasingly, workstations are managed through Intune and authenticated using cloud-native Entra ID, rather than on-premises Active Directory. In these environments, WEC collectors cannot authenticate to endpoints using the default Kerberos-based WinRM authentication, since Kerberos requires traditional domain-joined infrastructure.

The solution is to implement certificate-based authentication. This can be achieved using the Simple Certificate Enrollment Protocol (SCEP), which enables devices to request and obtain certificates via a secure URL and shared secret from a PKI.

Organizations can configure SCEP policies within Intune to automatically deploy certificates to managed workstations. This allows the WEC collector to securely authenticate and collect logs over WinRM, even in cloud-only identity environments.

Randy Franklin Smith's Ultimate Windows Security

Certificate-based Windows Event Collection (WEC) operates over HTTPS (port 5986) and supports log collection from devices across the internet. However, if you're planning to collect logs from remote endpoints outside the corporate network, it's highly recommended to configure WEC to use port 443 instead. This ensures better compatibility with network firewalls and increases the likelihood that your WEC collector can successfully communicate with internet-facing devices.

Sentinel -Syslog/CEF Connectors

The default logging mechanism for most Linux systems is Syslog, a protocol originally developed in the 1980s by Eric Allman, the creator of Sendmail. Syslog was initially introduced to capture logging data related to email delivery but quickly became the de facto logging standard across Unix-like systems and network devices such as routers, switches, and firewalls.

An enhanced variant of Syslog, known as rsyslog (short for “remote syslog”), extends its functionality by enabling log messages to be sent to external systems—an essential capability for centralized logging.

In addition to syslog, Linux systems may also run auditd, which provides more granular, security-focused event logging. While syslog captures general system and application logs, auditd focuses on events related to authentication, file access, and user actions, making it highly valuable for forensic investigations and compliance monitoring.

Default Syslog Log File Locations on Linux servers

Syslog logs are typically stored in flat files on disk. Common locations include:

/var/log/messages
/var/log/auth.log
/var/log/secure
/var/log/<application>.log

Syslog Message Structure

A standard syslog message includes the following fields:

<PRI>1 2019-06-05T22:14:15.003Z server1 sshd - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2

Key components:

PRI (Priority) – A calculated value based on Facility and Severity.
Timestamp – The date and time of the event.
Source Host – The hostname where the event originated.
Source Process – The application or service generating the log.
Message – The actual log content (unstructured string data).

PRI is derived from this formula:

PRI = (Facility × 8) + Severity

Facility values range from 0–23 (e.g., kernel, mail, auth).
Severity values range from 0–7:
- 0: Emergency
- 1: Alert
- 2: Critical
- 3: Error
- 4: Warning
- 5: Notice
- 6: Informational
- 7: Debug

Challenges with Syslog

One major limitation of syslog is the lack of standardization within the message field. Because vendors are free to format log messages however they choose, and often alter formats between software versions, normalizing these logs for SIEM analysis becomes labor-intensive and error-prone.

This inconsistency can lead to:

Difficulty in creating reliable detection rules
Inefficient querying and alerting
Bloated log storage due to redundant or irrelevant entries

Common Event Format (CEF)

To address syslog’s inconsistencies, the industry introduced the Common Event Format (CEF) — a standardized log structure developed to improve parsing and correlation in SIEMs like Microsoft Sentinel.

Benefits of CEF:

Consistent key-value log structure
Easier to parse and analyze
Enhanced interoperability across platforms

However, CEF must be natively supported by the log source. Not all systems offer CEF export options. When they do, the logs are emitted in CEF format over Syslog, often to a designated forwarder.

CEF Ingestion in Microsoft Sentinel

Microsoft Sentinel uses a Linux-based log forwarder (configured with the Azure Monitor Agent) to receive logs via Syslog and convert them into CEF before ingestion. This forwarder acts as an intermediary between the source systems and Sentinel, ensuring that logs arrive in a standardized format ready for storage, correlation, and analysis within Log Analytics Workspaces (LAW).

This architecture, discussed earlier under Agent Connectors and AMA Agent, enables Sentinel to perform efficient parsing, normalization, and enrichment on logs stored in its tables. Below are diagrams showing this concept in action.

Sentinel - Threat Intelligence (TI) Connectors

Threat Intelligence (TI) data is typically imported into Microsoft Sentinel through built-in data connectors or via a REST API. These methods support integration with both Microsoft-generated and third-party TI sources. Below are the primary options for ingesting threat intelligence into Sentinel:

1. Microsoft Defender Threat Intelligence (MDTI)

This connector allows you to import Indicators of Compromise (IOCs) directly from Microsoft Defender Threat Intelligence into Sentinel. These IOCs can be used for real-time monitoring, alerting, and proactive threat hunting.

2. Threat Intelligence - TAXII

This connector supports the industry-standard STIX/TAXII protocol for ingesting structured threat intelligence feeds from external providers. It’s ideal for organizations already subscribed to TAXII-based threat feeds.

3. Threat Intelligence Upload API

This API enables seamless integration of curated TI feeds without requiring a traditional connector. It is commonly used for integrating third-party Threat Intelligence Platforms (TIPs) or custom TI solutions.

To integrate a TIP using the TI Upload API, follow these steps:

Register an application in Microsoft Entra ID and obtain the Application ID and Client Secret.
Configure your TIP or custom app with these credentials to authenticate with Sentinel.
Enable the TIP Upload API functionality in Microsoft Sentinel to begin streaming indicators.

4. Threat Intelligence Platform (TIP) Connector

This legacy connector supports basic ingestion of indicators from TIPs but is limited in functionality and is currently being phased out. Microsoft recommends transitioning to the Threat Intelligence Upload API to take advantage of enhanced performance, flexibility, and future support.

Sentinel - Custom Connectors

Beyond CEF and Syslog, many solutions integrate with Microsoft Sentinel through the Data Collector API, enabling the creation of custom log tables in the Log Analytics Workspace.

These integrations typically fall into three categories:

1. Logstash-Based Integrations

Some data sources support Logstash, which can use an output plugin to forward events to Microsoft Sentinel via the Data Collector API. This allows flexible processing and transformation of logs before ingestion.

2. API-Based Sources

For systems that expose log data via an API, there are two common methods for integration:

Indirect (Intermediate System):A middle-layer system (such as a Function App, virtual machine, or custom script) is configured to pull data from the source API and push it to Sentinel using the Data Collector API. This pull-push model is often necessary when the source system cannot be accessed directly from Sentinel.
Direct API Integration (Custom Connector Platform - CCP):Some log sources allow Sentinel to directly pull data from the API using a custom connector. This connector is configured with the API endpoint, authentication method, and data parsing logic. While more efficient and easier to manage, this method requires network connectivity between Sentinel (cloud-based) and the log source, which can be challenging or risky for on-premises systems due to firewall and security constraints.

Important Consideration: Direct API integrations can offer real-time data ingestion and lower latency, but when connecting to on-prem systems, security risks and connectivity limitations must be carefully managed. Many organizations opt for intermediary systems to maintain tighter control and reduce exposure.

Connector Installation

When setting up a data connector in Microsoft Sentinel, the configuration wizard will guide you through the installation steps. Once installed, the connector’s status will be visible in the Sentinel interface. A status of “Connected” confirms successful integration, and you’ll also see the timestamp of the most recent log ingestion, which helps verify data flow. The Related Content section provides additional resources tied to that connector, including:

Workbooks for visualizing the ingested data
Analytics Rules that leverage the data for threat detection
Analytic Rule Templates, which are prebuilt detection blueprints crafted by security experts. These templates are based on common attack vectors and known threat behaviors and can be customized to align with your specific environment.

Using Content Hub

The Content Hub in Microsoft Sentinel is used when a connector or solution isn’t available by default. It serves as a centralized location within Sentinel for discovering, deploying, and enabling out-of-the-box (OOTB) solutions and content—including data connectors, analytics rules, workbooks, and more.

The Content Hub has replaced the legacy Solutions Gallery, offering a more streamlined and integrated experience for managing third-party and Microsoft security solutions directly within Sentinel.

Example Sentinel Data Connector setups

Microsoft Defender for Cloud provides Cloud Security Posture Management (CSPM) capabilities by assessing the security state of cloud resources and identifying misconfigurations and risks.

By default, Defender for Cloud alerts are exported at scheduled intervals and are not streamed in real time. To enable continuous, real-time export of alerts and recommendations, you must configure the Continuous Export setting within the Defender for Cloud portal.

Steps to Enable Continuous Export:

In the Azure portal, navigate to Microsoft Defender for Cloud.
Select Environment settings, then choose the relevant subscription.
Under Monitoring, select Continuous export.
Choose the types of data to export (e.g., security alerts, recommendations).
Select the destination (e.g., Log Analytics, Event Hub, or Storage account).
Save the configuration.

Enabling continuous export ensures that Defender for Cloud data is sent to your selected destination in near real-time, improving detection, monitoring, and response capabilities in integrated solutions like Microsoft Sentinel.

SC-200 Exam Tip: Drag-and-Drop Order of Operations Questions

The SC-200 exam often includes drag-and-drop style questions that test your ability to correctly order steps in configuration processes. These can be challenging due to the absence of visual context, requiring a clear mental model of each task.

Example: Steps to Configure the Microsoft Entra ID Data Connector in Microsoft Sentinel

In the Azure portal, go to All services and select Microsoft Sentinel. Open the relevant Sentinel workspace.
From the left-hand navigation menu, select Configuration > Data connectors.
On the Data connectors page, locate and select Microsoft Entra ID.

Click Open connector page. Follow the on-screen instructions to connect to the data source.
Under the Configuration section, select the appropriate log types:
- Sign-in logs
- Audit logs
After configuration, verify that the connector is passing data:
- Check that the connector status is “Connected.”
- Confirm that logs are appearing in the relevant Sentinel tables (e.g., SigninLogs, AuditLogs).

Review: Palo Alto Networks Data Connector Setup in Microsoft Sentinel

To configure the Palo Alto Networks data connector in Microsoft Sentinel, follow these steps:

Navigate to the Content Hub
- In the Microsoft Sentinel workspace, go to the Content Hub under the Configuration section.
Search for “Palo Alto Networks”
- Use the search bar to locate the Palo Alto Networks solution.
Install and Configure the Data Connector
- Select the Palo Alto Networks content pack and click Install.
- Follow the provided setup instructions to configure the data connector, ensuring logs are being ingested into Sentinel.
Enable Related Content
- After installation, enable any relevant analytics rules, workbooks, and hunting queries included with the solution.
Validate Log Ingestion
- Confirm that real-time data from Palo Alto devices is flowing into Sentinel.
- Check that the connector status is Connected and verify log entries in the associated Sentinel tables.

Sentinel Tables

As data is ingested into Microsoft Sentinel, it is stored in structured tables. When ingesting data such as Azure activity logs, Microsoft Entra ID logs, and operating system logs (via the Azure Monitor Agent or Syslog), the following Log Analytics tables are commonly used:

Table	Description
AzureActivity	Entries from the Azure Activity log that provides insight into any subscription-level or management group level events that have occurred in Azure.
AzureDiagnostics	Stores resource logs for Azure services that use Azure Diagnostics mode. Resource logs describe the internal operation of Azure resources.
AuditLogs	Audit log for Microsoft Entra ID. Includes system activity information about user and group management, managed applications, and directory activities.
CommonSecurityLog	Syslog messages using the Common Event Format (CEF).
McasShadowItReporting	Microsoft Defender for Cloud Apps logs
OfficeActivity	Audit logs for Office 365 tenants collected by Microsoft Sentinel. Including Exchange, SharePoint and Teams logs.
SecurityEvent	Security events collected from windows machines by Azure Security Center or Microsoft Sentinel
SigninLogs	Azure Activity Directory Sign-in logs
Syslog	Syslog events on Linux computers using the Log Analytics agent.
Event	Sysmon Events collected from a Windows host.
WindowsFirewall	Windows Firewall Events

Common Sentinel Defender XDR log tables

Microsoft Defender XDR is a unified solution that brings together several Defender sub-products, including:

Microsoft Defender for Endpoint (MDE)
Microsoft Defender for Office 365 (MDO)
Microsoft Defender for Identity (MDI)
Microsoft Defender for Cloud Apps (MDCA)

Each of these products generates telemetry and logs accessible both within their respective portals and through the integrated Microsoft Defender XDR console.

When exporting Defender data to Microsoft Sentinel, the logs from each sub-product map to specific data tables in Log Analytics. In addition to these source-specific tables, Microsoft Defender XDR generates alerts that are stored in the AlertInfo and AlertEvidence tables, providing structured context for incidents and investigations.

Table name	Description	Defender log source
AlertInfo (Used to be SecurityAlert)		Defender XDR itself
AlertEvidence	Files, IP addresses, URLs, users, or devices associated with alerts	Defender XDR itself
DeviceInfo	Machine information, including OS information	Defender for Endpoint (MDE)
DeviceNetworkInfo	Network properties of devices, including physical adapters, IP and MAC addresses, as well as connected networks and domains	Defender for Endpoint (MDE)
DeviceProcessEvents	Process creation and related events	Defender for Endpoint (MDE)
DeviceLogonEvents	Sign-ins and other authentication events on devices	Defender for Endpoint (MDE)
DeviceNetworkEvents	Network connection and related events	Defender for Endpoint (MDE)
DeviceFileEvents	File creation, modification, and other file system events	Defender for Endpoint (MDE)
DeviceRegistryEvents	Creation and modification of registry entries	Defender for Endpoint (MDE)
DeviceLogonEvents	Sign-ins and other authentication events on devices	Defender for Endpoint (MDE)
DeviceImageLoadEvents	DLL loading events	Defender for Endpoint (MDE)
DeviceEvents	Multiple event types, including events triggered by security controls such as Windows Defender Antivirus and exploit protection	Defender for Endpoint (MDE)
DeviceFileCertificateInfo	Certificate information of signed files obtained from certificate verification events on endpoints	Defender for Endpoint (MDE)
EmailAttachmentInfo	Information about files attached to Office 365 emails	Microsoft Defender for Office (MDO)
EmailEvents	Microsoft 365 email events, including email delivery and blocking events	Microsoft Defender for Office (MDO)
EmailPostDeliveryEvents	Security events that occur post-delivery, after Microsoft 365 has delivered the emails to the recipient mailbox	Microsoft Defender for Office (MDO)
EmailUrlInfo	Information about URLs on emails	Microsoft Defender for Office (MDO)
UrlClickEvents	Information about files attached to Office 365 emails	Microsoft Defender for Office (MDO)
IdentityDirectoryEvents	Events involving an on-premises domain controller running Active Directory (AD). This table covers a range of identity-related events and system events on the domain controller.	Microsoft Defender for Identity (MDI)
IdentityLogonEvents	Authentication events on Active Directory and Microsoft online services	Microsoft Defender for Identity (MDI)
IdentityQueryEvents	Queries for Active Directory objects, such as users, groups, devices, and domains	Microsoft Defender for Identity (MDI)
CloudAppEvents	Events involving accounts and objects in Office 365 and other cloud apps and services	Microsoft Defender for Cloud Apps (MDCA)

When Microsoft Defender for Endpoint is running on a system, it collects telemetry using its advanced hunting schema rather than relying on raw Windows Event Logs. This is crucial to understand—especially when preparing for the SC-200 exam—because you won’t see traditional Windows Event IDs (like 4624 for successful logons) in Defender's native telemetry tables.

Many KQL-based exam questions will reference the SecurityEvent table and expect you to use Windows Event IDs. For example:

SecurityEvent
| where EventID == 4624
| where Account == "Bill.Loney"

However, Microsoft Defender for Endpoint does not log data in this way. Instead, it normalizes telemetry into dedicated tables like DeviceLogonEvents. Successful logons, for example, are represented by the ActionType field with a value of "LogonSuccess":

DeviceLogonEvents
| where ActionType == "LogonSuccess"
| where AccountName == "Bill.Loney"

It's important to remember that ActionType does not always directly map to a specific Windows Event ID, but often represents the same or similar activity in a more abstracted and enriched format.

Understanding this distinction will help you choose the correct tables and fields based on the data source—especially under exam conditions.

Note: A quick way to remember all the Microsoft defender for Endpoint (MDE) data tables in Sentinel is that they all start with Device.

ASIM Model

ASIM (Advanced Security Information Model) is a powerful normalization framework in Microsoft Sentinel that simplifies and unifies security data across diverse sources. It enables streamlined threat hunting, analytics, and detection rule creation by mapping disparate log formats to a consistent schema. ASIM is conceptually similar to Elastic's Elastic Common Schema (ECS).

ASIM is composed of two types of parsers:

Unifying Parsers
- These are the primary parsers users query.
- They provide schema-level abstraction, ensuring all relevant data—regardless of source—is accessed via a single interface.
- Unifying parsers delegate parsing tasks to source-specific parsers under the hood.
Source-Specific Parsers
- These perform the actual field mapping and normalization for individual log sources (e.g., Infoblox, Zscaler, Azure Firewall).
- They can also be used directly, particularly in source-specific workbooks or use cases.
- Example: vimDnsInfobloxNIOS

Naming Convention:

Built-in unifying parsers: _Im<Schema>
Workspace-deployed parsers: im<Schema>
Example: _ImDns, imDns

Core and Extended ASIM Schemas

Originally, ASIM began with four core schemas:

Function	Parser Name
Authentication	_ImAuthentication
DNS	_ImDns
Network Session	_ImNetworkSession
Web Session	_ImWebSession

It now includes additional schemas:

Category	Parser Name(s)
Audit Events	_Im_AuditEvent
Authentication	imAuthentication
DNS	_Im_Dns
File Events	imFileEvent
Network Session	_Im_NetworkSession
Process Events	imProcessCreate, imProcessTerminate
Registry Events	imRegistry
Web Session	_Im_WebSession

Benefits of ASIM

While ASIM introduces initial complexity due to parser configuration, it provides long-term efficiencies and scalability. For example, consider a scenario where you want to detect all DNS queries that return NXDOMAIN across your entire organization.

Without ASIM, you'd need separate queries per data source (Firewall, Zscaler, Endpoint, etc.).With ASIM, one unified query is sufficient:

_Im_Dns 
| where TimeGenerated > ago(1d) 
| where ResponseCodeName =~ "NXDOMAIN" 
| summarize count() by SrcIpAddr, bin(TimeGenerated, 15m)

Developing Custom ASIM Parsers

Creating a custom ASIM parser involves the following steps:

Collect Sample Logs
Identify the Appropriate Schema
Map Source Fields to ASIM Fields
Develop One or More ASIM Parsers
Test Your Parser Thoroughly
Deploy Parsers to Sentinel Workspace
Update the Unifying Parser to Reference the Custom Parser
(Optional) Contribute Your Parser to the Community

Performance Optimization

To improve query performance when using ASIM parsers:

Use Filtering Parameters: Many parsers support pre-filtering, reducing data volume before parsing begins.
Avoid Post-Parse Filtering: Query cost increases significantly if filtering happens after parsing.

With optimized queries, ASIM-based searches often outperform non-normalized queries,

particularly at scale.

Controlling Parser Updates

If you want to prevent automatic updates to built-in ASIM source-specific parsers:

Add the desired parser version (e.g., ImDns_AzureFirewallV02) to your custom unifying parser.
Define update exceptions:
To exclude all parsers for a specific CallerContext:
plaintext

SourceSpecificParser = "Any", CallerContext = "<YourContext>"

To exclude all built-in parsers:
plaintext

SourceSpecificParser = "Any", CallerContext = "Any"

Sentinel and Defender XDR Integration

Earlier in this section, we discussed the integration of Microsoft Defender XDR with Microsoft Sentinel. One of the key outcomes of this integration is that the log tables, queries, and functions within the connected Sentinel workspace become accessible in Advanced Hunting within Defender XDR.

However, there may be scenarios where you need to disconnect Defender XDR from Microsoft Sentinel—for example, during workspace migrations, testing, or configuration changes.

Steps to Disconnect Defender XDR from Microsoft Sentinel

Open Microsoft Sentinel. Navigate to your Sentinel workspace.
Locate the Defender XDR Connector. Go to Data connectors, find the Microsoft Defender XDR connector, and select it.
Disconnect the Connector. In the connector configuration pane, click Disconnect to remove the integration.
Disable Incident Creation Rules. After disconnection, return to Sentinel and disable any analytics rules or automation rules that were dependent on Defender XDR data—particularly those that created incidents based on logs from the connector.

Sentinel - Configuration - Workspace Manager

In this article, we explored Microsoft Sentinel’s Settings and Data Connectors, foundational elements for any successful deployment. In an upcoming article, we’ll dive deeper into Analytics, Watchlists, and Automation Rules.

Before we wrap up, let’s take a quick look at another important configuration area: Workspace Manager.

Workspace Manager provides a centralized interface for managing multiple Microsoft Sentinel workspaces across one or more Azure tenants, streamlining visibility and control from a single location.