Azure function to store csv data in Cosmos DB

This is a http trigger function written in C# - .NET Core 3.1. It processed a csv file and store the data on Cosmos DB

GitHub repo : https://github.com/hansamaligamage/fncsvdatastore

Technology stack

• .NET Core 3.1 on Visual Studio 2019
• Azure functions v3 and Azure Cosmos DB table API

Some code snippets

Retrieve the database storage account

public static CloudStorageAccount RetieveStorageAccount(string storageConnectionString) 
{ 
    CloudStorageAccount storageAccount; 
    try 
    { 
       storageAccount = CloudStorageAccount.Parse(storageConnectionString); 
    } 
    catch (FormatException) 
    { 
       Console.WriteLine("Invalid storage account information provided"); 
       throw; 
    } 
    catch (ArgumentException) 
    { 
       Console.WriteLine("Please check the storage account details"); 
       Console.ReadLine(); 
       throw; 
    } 
    catch(Exception ex) 
    { 
       throw ex; 
    } 
    return storageAccount; 
 }

Create a table in Cosmos DB


public static string LoadConnectionDetails() 
{ 
    return Environment.GetEnvironmentVariable("StorageConnectionString"); 
}

public static async Task<CloudTable> CreateTableAsync(string tableName)
{
    CloudTable table;
    string storageConnectionString = LoadConnectionDetails();
    CloudStorageAccount storageAccount = RetieveStorageAccount(storageConnectionString);
    CloudTableClient tableClient = storageAccount.CreateCloudTableClient(new TableClientConfiguration());
    Console.WriteLine("Table client is created");
    try
    {
        table = tableClient.GetTableReference(tableName);
        if (await table.CreateIfNotExistsAsync())
        {
            Console.WriteLine("Table is created - {0}", tableName);
        }
        else
        {
            Console.WriteLine("Table {0} already exists", tableName);
        }
    }
    catch (Exception ex)
    {
        throw ex;
    }
    Console.WriteLine();
    return table;

}

Create a row in the Cosmos DB

public static async Task InsertOrUpdateEntityAsync(CloudTable table, Session session) 
{ 
   if (session == null) 
   { 
      throw new ArgumentNullException("Session is empty"); 
   } 
  try 
  { 
     TableOperation tableOperation = TableOperation.InsertOrMerge(session); 
     TableResult result = await table.ExecuteAsync(tableOperation); 
     Session newSession = result.Result as Session; 
     if (result.RequestCharge.HasValue) 
     { 
        Console.WriteLine("Request Charge of the operation: " + result.RequestCharge); 
     } 
     return newSession; 
 } 
 catch (StorageException e) 
 { 
    Console.WriteLine(e.Message); 
    Console.ReadLine(); 
    throw; 
 } 
 catch (Exception ex) 
 { 
    throw ex; 
 } 
}

Read a row in the Cosmos DB

public static async Task RetrieveEntityAsync(CloudTable table, string partitionKey, string rowKey) 
{ 
   try 
   { 
      TableOperation tableOperation = TableOperation.Retrieve(partitionKey, rowKey);
      TableResult result = await table.ExecuteAsync(tableOperation); 
      Session session = result.Result as Session; 
      if (session != null) 
      { 
         Console.WriteLine("\t{0}\t{1}\t{2}\t{3}", session.PartitionKey, session.RowKey,                               session.Points, session.IsWeekend); 
      } 
      if (result.RequestCharge.HasValue) 
      { 
         Console.WriteLine("Request Charge of Retrieve Operation: " +                                                     result.RequestCharge); 
      } 
      return session; 
  } 
  catch (StorageException e) 
  { 
     Console.WriteLine(e.Message); 
     Console.ReadLine(); 
     throw; 
  } 
 catch (Exception ex) 
 { 
    throw ex; 
 } 
}

Data types in Azure Cosmos DB SQL API

Check this video to understand how to model a Cosmos DB for a sample bookstore by going through the set of business requirements

Video : https://www.youtube.com/watch?v=Mz7FgEo3LFg&t=18s

Global Microsoft 365 Developer Bootcamp - 2019

Get your excel worksheet into Cosmos DB with Microsoft Graph

Event : https://www.eventbrite.com/e/2019-global-microsoft-365-developer-bootcamp-sri-lanka-colombo-tickets-79874337293#

Blog : https://hansamaligamage.blogspot.com/2018/08/net-core-process-excel-file-with.html

GitHub : https://github.com/hansamaligamage/msgraph-excel

Sri Lanka .NET Forum - November Meetup - 2019

Azure Media Services - A live video streaming site from scratch

Session Abstract : This session will give you a brief introduction to Azure Media services with hands on demo to encode and stream a live video

Event : https://www.meetup.com/Sri-Lanka-NET-Forum/events/266112882/

Global Azure Bootcamp - 2019

Azure Functions for Python Developers

Event : https://gabccolombo.azurewebsites.net/

Slides : https://www.slideshare.net/HansamaliGamage/azure-functions-for-python-developers

Azure Functions for Python Developers

1 Introduction

This blog post explains how to create a Azure function using Python 3.6, Azure functions support for Python is still in the preview mode, Let's try to explore some cool features available. Sample project uses Azure storage queue output binding to store the data in a queue. You will be able to get some hands on experience in creating a Python function with Visual Studio Code

2 Background

• If you want to see more things with Azure storage, check this article, Send smtp email with Azure blob storage attachments
• You can check this article if you want to know function app and Microsoft Graph integration, .NET Core: Process a Excel file with Microsoft Graph API & Azure Function
• This blog post explains how to work with a excel file availale on OneDrive and manipulate it using a Azure function .NET Core: Building Function app with Microsoft Graph API and Azure Functions
• Check this article, it shows how to create a Cosmos DB trigger in a function app, Azure Function App: Create a trigger on Cosmos DB

3 Prerequisites

• In this article, we are going to create a function app from Visual Studio Code. You should have installed Visual Studio Code in your machine 
• We are creating a function using Python 3.6.6, so you have to install it before starting the demo code.
• You should have a azure subscription to host your function app in Azure portal, If you dont have an active subscription try to create a free account and get a subscription

4 Azure serverless architecture

Azure serverless architecture focuses on developing applications faster without worrying about the application infrastructure. So we don’t have to handle server or infrastructure, no need to worry about the operating system its infrastructure or configurations. 

We can only focus on the function implementation, in fact server management & capacity planning is invisible to us and handled by Azure. Serverless code is event driven, it works with triggers & bindings. 

5 Why Serverless architecture

Let's see what are the advantages of using Azure Serverless architecture in your applicagions, in which kind of scenarios.

• Build apps faster using serverless functions

• No need to handle server or infrastructure, no administrative tasks

• We don’t have to worry about the operating system, its configurations or infrastructure

• We can focus only on the function implementation and business logic

• Applications can be scaled on demand without worrying about the infrastructure

• Applications will run with high availability

• Service billing is based on “pay as you go”, the amount of resources consumed or the time our code is running 

• Create modern applications with real time triggers and bindings 

• Integrate Azure services like Cosmos DB, Logic Apps or else use third party services like SendGrid, Twilio SMS API  

• Work with intelligent services like Azure machine learning & Cognitive services

6 Real World Case Studies on Azure Serverless architecture

We can use Azure serverless architecture in following scenarios like, when you want to develop a single page web application, a web application backend, mobile application backend, in a IOT backend. Also can use when you want to perform real time file processing, real time stream processing, task automation and SaaS Integration

7 Azure serverless platform

Azure serverless platform includes Azure Functions, Azure Logic Apps and Azure Event Grid. These services are going to connect together with other resources available in Azure. The serverless platform works with databases, storages, IOT devices and more advanced features like business analytics, machine learning and artificial intelligence. We can use azure serverless platform inside our application or else go for a hybrid solution that integrate with other cloud resources and on-premises services

7.1 Azure Functions

Azure function is invoked by a trigger; it can be a http endpoint or a timer. Function can be integarated with other azure services like queues, storages, databases or else with a azure logic app. There are two versions of a function app, first version (v1) is supported by .NET full framework and the other version (v2), still in the preview, is based on the cross platform .NET Core version that is compatible with macOS or a linux environment. We can use C#, F#, Javascript or experimental languages like Java, Python, php, typescript, batch, bash or powershell to write a function also can use Azure portal to write a basic function or else a IDE like Visual Studio or Visual Studio Code. You dont have to start from the beginning to develop a function app, you can use existing function templates.  

7.2 Logic Apps

Azure Logic Apps provides a serverless platform to build automated workflows to integrate applications and data between cloud services or on-premises systems. You can build workflows using a visual designer available in Azure portal or else can pick up an existing trigger template. You can trigger created workflows based on an event or a timer as per the requirement. Once the trigger is created, logic app offers many add ins like conditional statements, loops, steps, actions or parallel processors. A logic app can be integrated with Azure databases, storages, queues, Event grid, outlook, SharePoint lists, twitter, Facebook like social media applications or else with a function app. Logic App dashboard has a workflow run history, It’s very useful for troubleshooting since you can navigate to an each run and examine the data passed in a given step.

7.3 Event Grid

You can use Azure event grid for event based applications. You can publish an event to event grid and consume it from any platform. This is based on the publisher – subscriber architecture, you get the publishing and subscribing benefit without the overhead of setting up the necessary infrastructure. Event grid can support for events from blob storage or from any other azure resource. Event grid offers event routing, real time event delivery at larger scale and it integrates with azure services as well as other on-prem services outside azure. Event grid can trigger a azure function or a logic app and use push mechanism to send messages when an event occurs. Push mechanism consume less resources and scales better than polling mechanism, if we use polling it must check for updates on a regular interval which consumes lot of resources.

8 Azure function

Azure function is a serverless compute service that runs chunk of code or small piece of code on demand without having to provision or manage the infrastructure. We can use Azure function to run in response to variety of events in the cloud. Functions is going to make development productive, and we can use a development language from variety of choices like C#, F#, Javascript, Java, Python or php. You can pay only for the time your function code runs.

• Function development is easy
• You have variety of languages to select
• Set up continuous integration & deployment
• You can develop your functions from a Linux environment
• Pay for what you use
• Use your own dependencies - Nuget & NPM
• Integrate with other Azure services - Azure Cosmos DB, Azure Event Hubs, Azure Event Grid, Azure Notification Hubs, Azure service Bus, Azure storage (Blobs, Queues or Tables), Azure Logic Apps, Azure SignalR, Microsoft Graph, Power BI
• Integrate functions with third party services - Twilio API, SendGrid and other On-prem services
• Security - OAuth providers like Azure Active Directory, Facebook, Google, Twitter and Microsoft Account.
• Open source function runtime

9 Pricing in Azure functions

Azure function has a two type of pricing plans. Consumption plan & App service plan.

When you create a azure function, you have to define what is the hosting plan you want to select. You have to carefully go through the plans and finalize a suitable pricing plan based on your business needs. 

9.1 Consumption Plan

In Consumption plan, Azure provides all the computational resources a function to run, you don’t have to worry about the resource management, you only pay for the time your function runs. Consumption plan for Linux is now available in preview. Billing is based on no of executions, execution time and memory. Final bill will be calculated for all the functions inside a function app.

9.2 App service Plan

In App service plan, you have the option to run your functions as same as web applications. When you have already created a app service plan for other applications like web apps, API apps or mobile apps, you can run your function in the same plan without any additional cost. App service plan is supported for Linux. In App service plan, it has an hourly charge on the resources other than the free tier. You have to pay for the VM instance that you allocate, not for the no of execution, execution time or memory.

10 Create Azure Function in Python

Let's open Visual Studio Code & install Azure Functions extension, then you will be able to see Azure blade in the left side of the menu. You can click on the small icon available in the below picture and create a new function from Visual Studio Code.

Before creating a Azure function, you have to create a Azure function project, Let's click on Create new project or else you can open an existing project

We have to select a language for your azure function, let's select Python from the list of available languages, Python support for Azure functions are still in preview mode, but its worth to try it out 😀

Next, you can select a template to create your Azure function, Let's select Http trigger for this example,

You have to provide a name to create Http trigger, Let's give the name as OrderTrigger

You have to select Authorization level for your Azure function from the available list, let's select Anonymous since we are not planing to discuss security in this demonstration.

Let's open the solution in the current window,

You will get a notification to install pylint extension to check the syntax errors, let's install that to the project

I have already installed these extensions to the Visual Studio Code, Azure Account, Azure Functions are important for this demonstration, pip-packages are required to install python related packages. If you have missed any packages from the below list, install them to the editor

You can see the generated Http trigger function in Python, it simply takes a parameter and prints a message

Let's change the function as per our requirement,

• We are planing to create a simple order processing system, so should be able to place an order by calling this Azure function
• We should be able to pass order & location information to the function, as a query string or else from the method body
• When order and location both is passed to the function, order item is inserted to a queue

You can see the changed function as per the new requirement,

We have to add the output queue binding in function.json file as below with the connectionstring setting to connect to the storage account

Go to the setting file and add the connectionstring to the Azure queue storage

Let's run the function and navigate to the ordertrigger url as shown below, we haven't passed any parameters as the query string or else in the request body, so we get the following message

Let's pass order and location parameters in the query string, you can see it prints the delivery message as follows.

Now, let's try to pass these two parameters in the request body using Postman,

Let's navigate to the Queue storage we have configured, you will be able to see the queue message is inserted as below

Let's try to deploy the function to Azure cloud, click on Deploy icon

You can select an Azure subscription from the available list below,

In this demo, we are selecting an existing function app in the cloud, if not you can select a new function app

You can see function is deploying to Azure cloud

You can see the output window as below with the http trigger url hosted in Azure

You can see the function is deployed to the cloud as below, you wont be able to modify the function since it deployed using Visual Studio Code

We have to setup the connectionstring to the Azure account in the configuration section like this,

11 Download

11.1 GitHub

You can find the source code from here, httptriggerpython

12 Videos

I have created a video series to explain and highlight some cool thins in the demo, Please have a look when you have time

Create a python function & deploy to azure cloud

13 Conclusion

We have created and deployed an Azure function app in Python using Visual Studio Code. We have used Azure queue storage as an output binding in the function. Although Python support for Functions is still in the preview mode, we could get some experience with the language

TechNet Guru Awards February 2019 - Miscellaneous

TechNet Guru Awards February 2019 - Miscellaneous - SILVER

I won the SILVER medal for one of my article on TechNet Guru competition

Article: TypeScript: How to render a Zingchart component using Angular 6

Source Code 

    TechNet Gallery : angular-zingchart-gauge

    GutHub : angular-zingchart