Integrate Azure Data Explorer dashboards into Azure DevOps

A couple of months ago, I wrote a blog about the Azure Data Explorer dashboards. Since then, the dashboards came out of public preview and are now general available. It is even possible to create an Azure Data Explorer cluster using ARM or Bicep templates. The only thing lacking is the ability to integrate dashboards into Azure DevOps enabling Continuous Integration / Continuous Delivery, or is it?

** UPDATE 04-07-2023** 
Got in contact with Microsoft and the ADX dashboard API is officially not (yet) supported

Research

After a question on the Microsoft Q&A about this topic, I did a little research. Because in the Azure Data Explorer (ADX) portal there is a possibility to export and import dashboards:

Using the development tools of my browser, I found the requests send via the portal. And it was just a GET and POST request with a bearer token for authentication.

GET

To check if it would work using the Azure DevOps pipeline, I first created a Postman GET request. I started with the basics: a URL and the bearer token. The dashboard ID can be found in the url when you browse to the dashboard in the ADX portal or in the development tool of your browser. The bearer can be found in the development tool in the headers of the request. It would be better to retrieve the bearer token from the Azure API, but for this test, this is enough.

And right on the first try it worked. In the response we got a JSON with the dashboard.

POST

The next step is to send the JSON formatted dashboard back to the API. So I went back to the developer tools of the browser and found out it was a similar request. But in this case, it was a PUT request. So I duplicated the Postman GET request and changed it to a PUT request. To send the dashboard to the API, I added the JSON response of the GET request as the body of this PUT request. In Postman, it is important to change the format of the body to JSON (indicated with the arrow in the screenshot).

And again, on the first try it worked. But on the second try, it failed with the message:

After some investigation, I saw the key “eTag” in the JSON response.

In my experience with other API’s, I know that these should be sequential. So, if you do a GET request and the eTag is “abc123”, then the PUT request should contain the same eTag value “abc123”. When the PUT request has been processed, the eTag will change to something else. This way, it is not possible to overwrite previous versions.

After I changed the eTag in the body of the second request, the third attempt was a success. So it is important to retrieve the eTag before we upload a dashboard to the portal.

Interim conclusion

It is possible to export and import dashboards using Postman. So probably it is also possible to integrate this into Azure DevOps. In the next part of the article, I will try to integrate the requests into an Azure DevOps pipeline.

Integration into Azure DevOps

To integrate the requests into Azure DevOps, I will be using Powershell.

So first, I created the Powershell script below. In this example, I use the body of the retrieved dashboard and upload it again. In a real life situation, you can use a JSON formatted dashboard in your repository. To do so, you will need to edit the Powershell script. But don’t delete the retrieval of the eTag, because we will need this to send the PUT request to the portal.

# ------------------------------------------------
# Parameters
# ------------------------------------------------
param(
    [Parameter(Mandatory=$true)]
    [String]$dashboardid,
    [Parameter(Mandatory=$true)]
    [String]$token
)

$eTag = ''
$body = ''
$uri = "https://dashboards.kusto.windows.net/dashboards/" + $dashboardid
$headers = @{
    'Authorization' = $token
    'Content-Type' = 'application/json'
}

# ------------------------------------------------
# GET request to retreive the eTag
# ------------------------------------------------
    
try
{
    $response = Invoke-RestMethod -Method Get -Uri "$($uri)" -Headers $headers
    if ($response)
    {    
        $body = $response
        $eTag = $response.eTag
        Write-Host('eTag: ' + $eTag)
    }
}
catch
{
    Write-Host "Exception details GET: "
    $e = $_.Exception
    Write-Host ("`tMessage: " + $e.Message)
    Write-Host ("`tStatus code: " + $e.Response.StatusCode)
    Write-Host ("`tStatus description: " + $e.Response.StatusDescription)

    Write-Host "`tResponse: " -NoNewline
    $memStream = $e.Response.GetResponseStream()
    $readStream = New-Object System.IO.StreamReader($memStream)
    while ($readStream.Peek() -ne -1) {
        Write-Host $readStream.ReadLine()
    }
    $readStream.Dispose();
}


# ------------------------------------------------
# Change eTag in body
# ------------------------------------------------
$body.eTag = $eTag


# ------------------------------------------------
# PUT request to export the dashboard
# ------------------------------------------------
$jsonBody = $body | ConvertTo-Json -Depth 100 
try
{
    $response = Invoke-RestMethod -Method Put -Uri "$($uri)" -Headers $headers -Body $jsonBody
    if ($response.error)
    {    
        throw 'Error:' + $response.error
    } else {
         Write-Host "Success"
    }
}
catch
{
    Write-Host "Exception details PUT: "
    $e = $_.Exception
    Write-Host ("`tMessage: " + $e.Message)
    Write-Host ("`tStatus code: " + $e.Response.StatusCode)
    Write-Host ("`tStatus description: " + $e.Response.StatusDescription)

    Write-Host "`tResponse: " -NoNewline
    $memStream = $e.Response.GetResponseStream()
    $readStream = New-Object System.IO.StreamReader($memStream)
    while ($readStream.Peek() -ne -1) {
        Write-Host $readStream.ReadLine()
    }
    $readStream.Dispose();
}  

The best way to integrate this script into your Azure Devops is to save it to your repository and then use it within the pipeline. So, I created a repository, added the Powershell script and created a pipeline script:

stages:
  - stage: general
    displayName: General
    jobs:
      - job: deploy_dashboard 
        displayName: Deploy dashboard 
        steps:
          - task: PowerShell@2
            displayName: 'Powershell task'
            inputs:
              targetType: 'FilePath'
              filePath: 'Powershell/UploadDashboard.ps1'
              arguments: '-dashboardid "<Dashboardid>" -token "<Bearer token>"'

In this example, I used a hard-coded Bearer token in the pipeline. It is better to retrieve the token through the Azure API. On this Microsoft page you will find more information about that.

After I committed the files into the repository, I started the pipeline and it worked:

Conclusion

After a little bit of research, it was possible to deploy an Azure Data Explorer dashboard using Azure Devops. By using the repository and Azure DevOps, we can take advantage of all the benefits of version control and Continuous Integration / Continuous Delivery.

9 comments

  1. Can you give an example on the POST you make against login.microsoftonline.com to get a valid token?

    I’ve tried a couple of different methods but keep getting “(401) unauthorized” when trying to get the dashboard.

    I get a token when posting for using an app id with the Azure Data Explorer user_impersonation permission granted and access to the dashboard:
    client_id = APPID
    client_secret = APPSECRET
    scope = https://CLUSTERNAME.REGION.kusto.windows.net/.default
    grant_type = client_credentials

    But when trying to GET a dashboard from https://dashboards.kusto.windows.net/dashboards/ I get the 401.

    In additon I tried setting grant_type to password and providing username and password values to the body for a non-mfa user account with access to the dashboard, but no luck.

    1. Dear Mike,

      If I understand correctly you get a token but the token doesn’t work for retrieving the dashboard.

      Did you use the coorect URI off your cluster. You can get the URI from the overview page of your Azure Data Explorer cluster.

      Kind regards,
      Wilko

  2. This is the complete test-script. Maybe I’m missing something basic?
    $AccessToken = $null
    $Tenant = “[TENANTID]”
    $URL = “https://login.microsoftonline.com/$Tenant/oauth2/v2.0/token”
    $Body = @(
    “client_id = [APPID]”
    “client_secret = [APPSECRET]”
    “scope = https://[CLUSTER].[REGION].kusto.windows.net/.default
    “username = [NON-MFA USERNAME]@[domain].onmicrisoft.com”
    “password = [PASSWORD]”
    “grant_type = password”
    ) -join “&”

    $headers = @{
    “content-type” = “application/x-www-form-urlencoded”
    }

    $AccessToken = (Invoke-RestMethod -Uri $URL -Headers $headers -Body $Body -Method Post ).access_token

    $headers = @{
    “Authorization” = “Bearer $Accesstoken”
    “content-type” = “application/json”
    }
    $dash = Invoke-RestMethod -Uri https://dashboards.kusto.windows.net/dashboards/%5BGUID%5D -Headers $headers

    If I remove the ‘/.default’ part of the scope I get an invalid scope error (AADSTS70011)
    If I remove both the ‘/.default’ and ‘https://’ parts I get an invalid_grant error (AADSTS65001) the user or administrator has not consented to use the application with id [APPID].

    When looking at the token (using jwt.io) retrieved from Chrome dev tools, using the same credentials I’m trying to impersonate, the audience is set to 35e917a9-4d95-4062-9d97-5781291353b9 (the guid for the Data Explorer enterprise app), and the token works fine with REST requests.
    But I still get the AADSTS65001 error if I change the scope in my script to the ADX guid
    Checking in AAD the app registration has user_impersonation granted on the tenant for the API.

    1. Dear Mike,

      I did some checks with Postman and I also got a 401 when retrieving a dashboard. The token based on the service principal does work when querying data in the database. But it doesn’t work on the API for the dashboards. I have to do some additional research.

      Kind regards,
      Wilko

      1. Dear Mike,

        I have contacted Microsoft and the ADX dashboard API is officially not supported. So, we have to wait for a solution until the API is officially released.

        Kind regards,
        Wilko

  3. Hello Wilko,
    is your pipeline still working and any news from Microsoft?

    I tried two ways to get access token. One is using client_id, client_secret and the uri in your example; another way it to get uri with the azure CLI with login method.

    Both access tokens will show 401 with “”https://dashboards.kusto.windows.net/dashboards/{dashboard_id}”
    But both will show 200 with “https://dataexplorer.azure.com/dashboards/{dashboard_id}”.
    But when I further to do put with the second uri, it will show 405 (The page you are looking for cannot be displayed because an invalid method (HTTP verb) is being used.)

    I am wondering how did you make it work.
    Thanks

Leave a comment

Your email address will not be published. Required fields are marked *

Exit mobile version