Automating Sign-In Analysis with PowerShell and Microsoft Graph

Created October 21, 2024

Introduction

In this article, we'll walk through how to explore and analyze Conditional Access data in Microsoft Entra using the Developer Tools in your browser. The purpose of this walkthrough is to demonstrate how you can identify and extract the necessary Microsoft Graph API calls directly from the Entra portal, providing a foundation for automating this process with PowerShell.

We'll start by examining the Conditional Access Policy Summary dashboard and the "See all unprotected sign-ins" feature to understand how these sign-ins are logged. By leveraging Developer Mode, we'll trace the underlying API calls, dissect the filters used, and understand how these requests are structured. This knowledge allows us to recreate these calls using Microsoft Graph API and automate them with PowerShell.

This guide is ideal for IT administrators and PowerShell enthusiasts who want to streamline tenant monitoring, automate their tasks, and dive deeper into how Entra operates behind the scenes. Let's get started!

Clicking through the tenant

While exploring my Entra Test Tenant, I stumbled upon an interesting feature in the Conditional Access Policy Summary dashboard: a link labeled "See all unprotected sign-ins". Naturally, I wanted to dive in and explore further.

When I clicked the link, I initially didn’t see any entries for interactive user sign-ins—which, depending on how you look at it, could be either a good or a bad thing. However, switching to non-interactive sign-ins revealed a list of entries!

Importance of Monitoring Sign-In Behavior

Security Posture: Identifying potential security risks by understanding how users access resources helps protect against unauthorized access.
Policy Compliance: Ensuring that Conditional Access policies are enforced and detecting non-compliance issues.
Anomaly Detection: Recognizing unusual activity, such as logins from unexpected locations, can indicate account compromise.
Resource Utilization: Understanding the distribution of interactive vs. non-interactive sign-ins informs decisions about resource allocation and user experience.

Difference Between Interactive and Non-Interactive Sign-Ins

Interactive Sign-Ins: Occur when a user actively logs in via a user interface (e.g., entering credentials). This indicates user engagement.
Non-Interactive Sign-Ins: Happen automatically, often through scripts or services without user input. They are essential for background processes requiring resource access.

Monitoring both sign-in types provides a comprehensive view of user behavior and access patterns, enhancing security and compliance management.

In this blog post, I'll take you through my journey of analyzing the web interface, extracting the necessary information, and converting it into native Microsoft Graph API calls. We’ll then automate this process using PowerShell to extract the results as objects.

Finding the Data Source

The first question I asked myself was: Where can I find the source of the information I'm looking for? A simple search engine query could work, but since I was already in the Entra portal, I decided to explore the data directly from the portal itself.

To do this, I opened the Developer Tools in my browser, switched to the Network tab, and filtered on 'Fetch/XHR'.

After setting up the Developer Tools, I clicked on 'User sign-ins (interactive)' again. Even though no results appeared, I could still see the requests made in the background. Amid all the telemetry and token requests, there was one specific request that caught my attention.

Analyzing the Request

In the Headers section of the request, I discovered that it called the endpoint:

https://graph.microsoft.com/beta/auditLogs/signIns

Perfect. This gave us a solid starting point. Additionally, I could see which filters were applied, which is critical for our use case. However, as URLs are often encoded and hard to read, I switched to the Payload tab. This provided a more readable version of the filter:

(createdDateTime ge 2024-10-10T22:00:00.000Z and createdDateTime lt 2024-10-17T22:00:00.000Z and (status/errorCode eq 0 or (status/errorCode ne 16000 and status/errorCode ne 16001 and status/errorCode ne 16003 and status/errorCode ne 29200 and status/errorCode ne 50019 and status/errorCode ne 50055 and status/errorCode ne 50058 and status/errorCode ne 50059 and status/errorCode ne 50072 and status/errorCode ne 50074 and status/errorCode ne 50076 and status/errorCode ne 50097 and status/errorCode ne 50125 and status/errorCode ne 50127 and status/errorCode ne 50129 and status/errorCode ne 50140 and status/errorCode ne 50143 and status/errorCode ne 50144 and status/errorCode ne 50158 and status/errorCode ne 50209 and status/errorCode ne 51006 and status/errorCode ne 52004 and status/errorCode ne 65001 and status/errorCode ne 81010 and status/errorCode ne 81012 and status/errorCode ne 81014 and status/errorCode ne 165100 and status/errorCode ne 502031 and status/errorCode ne 50203 and status/errorCode ne 9002341)) and conditionalAccessStatus eq microsoft.graph.conditionalAccessStatus'notApplied')

This filter ignores certain error codes and focuses on a specific conditionalAccessStatus: 'notApplied'. With this information, I knew we could build a native PowerShell call. Almost there—but not quite!

Investigating Non-Interactive Sign-Ins

We also need to gather data on non-interactive sign-ins. Clicking on the corresponding option in the portal again revealed a different request:

https://graph.windows.net

Though it seemed similar, this is actually the Azure AD Graph endpoint (graph.windows.net), which is deprecated. Microsoft recommends using the Microsoft Graph API, as detailed in their migration guide here.

Building the Microsoft Graph Query

To switch over to Microsoft Graph for non-interactive sign-ins, we need to tweak our filter. Fortunately, I encountered a similar situation previously when I needed to export sign-in logs for enterprise applications. Here’s the filter that targets non-interactive sign-ins:

signInEventTypes/any(t: t ne 'interactiveUser')

Now, we can combine this additional filter with our existing query:

(createdDateTime ge 2024-10-10T22:00:00.000Z and createdDateTime lt 2024-10-17T22:00:00.000Z and (status/errorCode eq 0 or (status/errorCode ne 16000 and status/errorCode ne 16001 and status/errorCode ne 16003 and status/errorCode ne 29200 and status/errorCode ne 50019 and status/errorCode ne 50055 and status/errorCode ne 50058 and status/errorCode ne 50059 and status/errorCode ne 50072 and status/errorCode ne 50074 and status/errorCode ne 50076 and status/errorCode ne 50097 and status/errorCode ne 50125 and status/errorCode ne 50127 and status/errorCode ne 50129 and status/errorCode ne 50140 and status/errorCode ne 50143 and status/errorCode ne 50144 and status/errorCode ne 50158 and status/errorCode ne 50209 and status/errorCode ne 51006 and status/errorCode ne 52004 and status/errorCode ne 65001 and status/errorCode ne 81010 and status/errorCode ne 81012 and status/errorCode ne 81014 and status/errorCode ne 165100 and status/errorCode ne 502031 and status/errorCode ne 50203 and status/errorCode ne 9002341)) and conditionalAccessStatus eq microsoft.graph.conditionalAccessStatus'notApplied' and signInEventTypes/any(t: t ne 'interactiveUser'))

Automating the Process with PowerShell

With all this information at hand, we can now create a dynamic PowerShell function. The function should allow:

Selecting interactive and non-interactive sign-ins.
Specifying the date range.
Using concise listing for error codes to keep the script readable.
Selecting unprotected or protected sign-ins.

Here’s the final PowerShell function: show on Github

In Action

Get-ConditionalAccessSignIn -StartDate $((Get-Date).AddDays(-30)) -EndDate $(Get-Date) -SignInType non-Interactive -Unprotected | select UserDisplayname,appDisplayname,CreatedDatetime, @{n="ErrorCode";e={$_.Status.errorCode}},@{n="Reason";e={$_.Status.failureReason}},signInEventTypes,ConditionalAccessStatus

The Code

<#
.SYNOPSIS
    Retrieves Conditional Access Sign-In logs from Microsoft Graph API.

.DESCRIPTION
    The Get-ConditionalAccessSignIn function retrieves Conditional Access Sign-In logs from the Microsoft Graph API within a specified date range. 
    It allows filtering by sign-in type (Interactive or Non-Interactive) and conditional access status (Success or NotApplied).

.PARAMETER Unprotected
    Switch to indicate if only unprotected sign-ins should be retrieved. If specified, the conditional access status is set to "NotApplied".

.PARAMETER StartDate
    The start date for the sign-in logs retrieval. This parameter is mandatory.

.PARAMETER EndDate
    The end date for the sign-in logs retrieval. This parameter is mandatory and must be at least 1 day after the StartDate.

.PARAMETER SignInType
    Specifies the type of sign-in events to retrieve. Valid values are "Interactive" and "Non-Interactive". The default is "Interactive".

.EXAMPLE
    Get-ConditionalAccessSignIn -StartDate (Get-Date).AddDays(-7) -EndDate (Get-Date) -SignInType "Interactive"

    Retrieves interactive sign-in logs for the past week.

.EXAMPLE
    Get-ConditionalAccessSignIn -Unprotected -StartDate (Get-Date).AddDays(-7) -EndDate (Get-Date) -SignInType "Non-Interactive"

    Retrieves non-interactive unprotected sign-in logs for the past week.

.NOTES
    Author: Christian Ritter
    Date: 10/21/2024
#>


function Get-ConditionalAccessSignIn {
    [CmdletBinding()]
    param (
        [switch] $Unprotected,
        [Parameter(Mandatory=$true)]
        [datetime] $StartDate,
        [Parameter(Mandatory=$true)]
        [datetime] $EndDate,
        [ValidateSet("Interactive", "Non-Interactive")]
        [string[]] $SignInType = "Interactive"
    
    )
    
    begin {
        
        
        if($Unprotected){
            $ConditionalAccessStatus = "NotApplied"
        }else{
            $ConditionalAccessStatus = "Success"
        }
        #Enddate must be at least 1 day after startdate
        if($EndDate -lt $StartDate.AddDays(1)){
            throw "End date must be at least 1 day after startdate"
            return
        }

        $IgnorableUnprotectedStatusErrorCodes = @(
            9002341, 502031, 50209, 
            50203, 52004, 51006, 
            50158, 50144, 50143, 
            50140, 50129, 50127, 
            50125, 50097, 50076, 
            50074, 50072, 50059, 
            50058, 50055, 50019, 
            29200, 165100, 16003, 
            16001, 16000, 81014, 
            81012, 81010, 65001
        )
    }
    
    process {
        $returnObject = foreach($SignInTypeObject in @($SignInType)){
            #region build the filter dynamically
            $Filter = "(createdDateTime ge $($StartDate.ToString("yyyy-MM-dd"))T22:00:00.000Z and createdDateTime lt $($EndDate.ToString("yyyy-MM-dd"))T22:00:00.000Z)"

            $Filter += " and (status/errorCode eq 0 or ($($IgnorableUnprotectedStatusErrorCodes.ForEach({
                "status/errorCode ne $_"
            })-join " and ")))"
            

            
            if($SignInTypeObject -eq 'Non-Interactive'){
                $Filter += " and (signInEventTypes/any(t: t ne 'interactiveUser'))"
            }else {
                $Filter += " and (signInEventTypes/any(t: t eq 'interactiveUser')"
            }
            

            $Filter += " and (conditionalAccessStatus eq '$ConditionalAccessStatus')"
            #endregion build the filter dynamically
            
            # perform the request
            Invoke-MgGraphRequest -Method Get -Uri ("https://graph.microsoft.com/beta/auditLogs/signIns?`$Filter=$Filter") -OutputType PSObject

            Write-Verbose "Path: https://graph.microsoft.com/beta/auditLogs/signIns"
            Write-Verbose "Filter: $Filter"
        }


    }
    
    end {
        return $returnObject.Value
    }
}

This function streamlines the process, allowing you to pull detailed sign-in data directly through Microsoft Graph without needing to navigate the portal manually. This is especially useful for large-scale tenant monitoring or automation tasks!

Conclusion

Exploring and automating tenant monitoring with PowerShell and Microsoft Graph opens up countless opportunities. By digging into the web interface and converting the queries into native API calls, we not only save time but also gain more flexibility and power over our data.

Happy scripting!

Christian Ritter