On my latest project, we encountered some strange behavior in the Azure Storage Account. The number of transactions were extremely high, which resulted in unexpected costs. In this article, I will explain the situation and how we solved it.
Situation
We have an IoT hub with a “message route” which stores the DeviceConnectionStateEvents into an Azure Storage Account. The DeviceConnectionStateEvents are stored as blob files into a container. Then, a BlobTriggered Azure Function App picks up the blob files after begin triggered. The files are read and the content is sent to an Event Hub.
After a while, we noticed that the costs of the Storage account increased much more than expected. During the analysis, we found, due to the detailed cost analysis of the subscription, that “All Other Operations” were the cause of the increased costs. In the screenshot below is an example of the detailed cost analysis.
After further analysis we found this extremely high number of transactions in the metrics of the Storage Account. To be more specific, it was the API “GetBlobProperties”. The graph below is an example from Microsoft of the Azure Storage Account metric, split by API name, to identify the problem.
Test setup
So, I created a test setup to analyse this problem in more detail. To exclude the blob upload from the IoT hub to the Storage Account, I just created a Storage Account and a BlobTriggered Function App listening to the blob container.
When triggered, the Azure Function App will only print the name and size of the blob, so nothing special:
[FunctionName("ImportTelemetry")]
public void Run([BlobTrigger("telemetry/{name}", Connection = "BlobConnectionString")]Stream myBlob, string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
}
Then, I uploaded a file manually to the Storage Account. But there was no difference in the number of transactions, as seen in the graph below. The only thing we can see is that spike caused by uploading the file, as expected. After uploading a second file, I still saw no significant changes. But when I uploaded eighteen files at once, the sum of the transactions increased significantly, as seen in the third spike in the graph below. The number of transactions remained stable around seventy transactions until I manually deleted the blobs (fourth spike). During this period we didn’t execute any actions which could result in the transactions. This stable line of seventy transactions is related to only twenty files, let alone thousands of files.
We suspected this was due the Function App. To be sure I stopped the Azure Function App and repeated the scenario as seen above. The result is seen in the new graph below. In this case, the number of transactions only results in some spikes, as expected because of uploading the files. But we don’t experience that stable line of high transactions as seen in the previous scenario.
So, our conclusion is that the high transactions are caused by the Azure Function App.
Simple solution
In the test setup, the high number of transactions stopped once the files were deleted. So, we created a second container that we will use as an archive container. After creating that new container, we still use the original container as source for processing incoming files. After being triggered and processing the files, the Azure Function App will move the files to the archive container when processed successfully:
using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
namespace FunctionAppBlobTransactions
{
public class Function
{
string _connection = Environment.GetEnvironmentVariable("BlobConnectionString");
[FunctionName("ImportTelemetry")]
public void Run([BlobTrigger("telemetry/{name}", Connection = "BlobConnectionString")]Stream myBlob, string name, ILogger log)
{
log.LogInformation($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");
MoveBlob(name, log, _connection, "telemetry", "telemetry-archive");
}
public static void MoveBlob(string blobName, ILogger log, string BlobConnectionString, string sourceContainer, string targetContainer)
{
try
{
log.LogInformation($"Start moving file to archive");
BlobServiceClient blobClient = new BlobServiceClient(BlobConnectionString);
BlobContainerClient sourceBlobContainer = blobClient.GetBlobContainerClient(sourceContainer);
BlobContainerClient destBlobContainer = blobClient.GetBlobContainerClient(targetContainer);
BlobClient sourceBlob = sourceBlobContainer.GetBlobClient(blobName);
BlobClient destBlob = destBlobContainer.GetBlobClient(blobName);
CopyFromUriOperation ops = destBlob.StartCopyFromUri(sourceBlob.Uri);
//If file exists in the destination, we can delete the source file
if (destBlob.Exists())
{
sourceBlob.DeleteIfExists();
log.LogInformation($"Deleted file from {sourceContainer}: {blobName}");
}
else
{
log.LogInformation($"Destination blob does not exists");
}
}
catch (Exception e)
{
throw new ApplicationException($"Exception: {e.Message}");
}
finally
{
log.LogInformation($"End moving file to archive");
}
}
}
}
I published the Function App and confirmed the next incoming blob was indeed moved after processing. In between, we need to move all the historical files from the original container to the archive container. This is a two-step process. First, we copy the files to the other container and secondly, we delete the same files from the original container.
It is possible to copying the files by a BASH script:
az storage blob copy start-batch \ --destination-container <destinationContainer> \ --account-name <destinationStorageName> \ --account-key <destinationStorageKey> \ --source-account-name <sourceStorageName> \ --source-account-key <sourceStorageKey> \ --source-container <sourceContainer>
But the Azure cloud shell has a timeout of twenty minutes. When the source container has many files, the script may take longer than twenty minutes. This resulted in a timeout for me, the copy action continued in the background for a while but not till the end. When I checked the Folder statistics using “Microsoft Azure Storage Explorer”, I saw that not all files were copied. Therefore, I created a C# script for copying all files (click here for the script on GitHub).
When the C# script above was finished coping all files, I deleted all the old files by the BASH script as seen below. This somewhat particular script will delete all files that are not modified for the last two hours. This means recently added files are not deleted yet. I added this filter to prevent files being deleted when the Function App is processing them.
date=`date -d "2 hours ago" '+%Y-%m-%dT%H:%MZ'` az storage blob delete-batch \ --account-name <sourceStorageName> \ --account-key <sourceStorageKey> \ --source <sourceContainer> \ --if-unmodified-since $date
Note: You may need to re-run this script after 2 hours to remove all files.
Now, after all preparations are made, let’s check if it had any result. So, I repeated the same scenario again:
This resembles the behavior seen in the metrics of our production blob storage.
Conclusion
The investigation confirmed the unexpected high transactions in Azure Storage Account were caused by the Azure Function BlobTrigger. It can be solved by adding an archive container and having the Azure Function App moving the files from the original container to the archive container. This way we keep the original container clean.
An added advantage is that the original container only contains files ready to be processed or files which are being processed right now or files which failed to be processed.
The archive folder contains all files being proccessed successfully.
This way, you can also start counter measures for files not being processed (due to a disabled function) or incompatible files.