Saturday, November 18, 2017

Azure Function App: Create a trigger on Cosmos DB

1 Introduction


This article is going to describe how to create a trigger/continuous checking on a data set using an Azure function app, this article has used Azure CosmosDB as a data set with multiple documents. Sample application shows how to get an alert or a notification when a vehicle is going to violate the speed limit in a road or in a highway. Let's dig into deep and how to use Azure function app to find a solution to this scenario

2 Background


If you are interested in Azure and its features, check these articles

3 Create a document in Azure Cosmos DB

3.1 Let's create Azure Cosmos DB account

Go to Azure Cosmos DB and navigate to Azure portal, Click on Add button, It will show Azure Cosmos DB account creation page. Add account name and select an API from list. In this example, we want to work with a simple document collection, so choose SQL as the data API




































You can select a subscription for billing of the account as below.
You can create a newresource group or use an existing one.
Select account location as a nearest region to your clients.
You can see a option to enable geo redundancy. At this point we are not going to replicate this data into another region, so don't tick that option. You can later replicate your data into different places to achieve high availability of your data.







































You will be redirected to Quick Start page, you can create a collection in various platforms, in this case let's use .NET Core
You can get an idea what is the default storage of your collection, what will be the throughput and estimated hourly billing of the collection. 
















3.2 Create a document in Azure Cosmos DB collection

Let's try to create a document in Azure Cosmos DB collection, Go to Data Explorer, your new database account is created. Create a new collection, We can create a document in that collection. Lets give some test data and create a document as below.



















3.3 Query a collection

Click on your collection and click on New SQL Query, In the New Query window, write the simplest query and execute it. You can see inserted document as a result, and it has required 2.28 request units to run the query

4 Create a document programmatically in Azure Cosmos DB

We created a document in Azure Cosmos DB collection using Azure portal, Let's see how we can create a document in Azure Cosmos DB programmatically

4.1 Create a console application

Open Visual Studio, This example has used Visual Studio 2017 with .NET Core 2.0.3 version to create the application. Select .NET Core and ASP.NET Core Web Application, You can give a name and click on OK button




























We need to add necessary references to access Cosmom DB,
Go to Tools -> NuGet Package Manager -> Manage NuGet Packages for solution, Search for Azure Document DB, it will show as below.
This version of Cosmos DB references are not compatible with .NET Core 2.0. So we have to find another one















Search for Microsoft.Azure.DocumentDB.Core, then you can find a reference that compatible with .NET Core. This is a special version of Document DB library designed for .NET Core application














4.2 Create Cosmos DB settings

Create a class named DBConnector and inside that, create a method as follows
First we need to create a DocumentClient to connect to the database account, we have to pass endpoint url and auth key


static class DBConnector 
{ 
   
  static DocumentClient client;
     
  public async static Task Create ()
  {
  
     try
     {

         string endPointUrl = "";

        string primaryKey = "";
      }
  }
}

4.3 Create a Document Client to connect to the database account


We have to instantiate a DocumentClient object to connect to Azure Cosmos DB account, in order to do that we have to pass service endpoint URL and auth key or a token, Let's see how we can find them












Go to Keys section in Azure Cosmos DB account, Get the URI and Primary Key to connect to Azure Cosmos DB account










static DocumentClient client; 

 public async static Task Create () 
{ 
   try 
   { 

       string endPointUrl = "https://highwaytraffic.documents.azure.com:443/"; 

       string primaryKey = "jSuZYTAvpOiw65P1O1TEPD7pAmjTTDUd5I9pudCkABUz97f3fxcnjg259o7jqDNCJXDyRAQrjOm6H2k24ZjyrQ=="; 
   }

Create a Document Client by passing endpoint URL and key






client = new DocumentClient(new Uri(endPointUrl), primaryKey);


4.4 Create the database


Create the database by passing database name








public async static Task CreateDatabase (string database) 
{ 
  var db = await client.CreateDatabaseIfNotExistsAsync(new Database { Id = database}); 
}

4.5 Create a Collection

Create a document collection by passing database name and collection name









 public async static Task CreateCollection(string database, string collection)
 {

    var databasecollection = await client.CreateDocumentCollectionIfNotExistsAsync(UriFactory.CreateDatabaseUri(database),
            new DocumentCollection { Id = collection });
  }

In the main method, pass database and collection name and call database and collection creation methods
















 string databaseName = "trafficdata";
string databaseCollection = "vehiclespeed"; 

 client = new DocumentClient(new Uri(endPointUrl), primaryKey);

 await CreateDatabase(databaseName); 

 await CreateCollection(databaseName, databaseCollection);

call database create method from main program as below,
















namespace highwaytraffic 
{ 
  class Program 
 { 
   static void Main(string[] args) 
   { 
     DBConnector.Create().Wait(); 
   } 
 } 
} 

Let's run the application and see database is created in Azure account, Go to Overview tab in Cosmos DB account and check  whether database is created. In Collections tab, it shows database and collectionid











4.6 Create a document in a collection

In order to create a document, we should have a object, In this example we try to find vehicles who exceeds certain speed limit, so need to create a class to store vehicle speed, Add a class called VehicleSpeed into the solution,



























 namespace highwaytraffic 
{ 
  public class VehicleSpeed 
  { 
    [JsonProperty(PropertyName = "vehicleNumber")] 
    public string VehicleNumber { get; set; } 

     [JsonProperty(PropertyName = "speed")] 
    public double Speed { get; set; } 

     [JsonProperty(PropertyName = "city")] 
    public string City { get; set; } 
 } 
} 

Create a document by passing database and collection ids, And specially you should pass a instance from a VehicleSpeed class








public async static Task CreateDocument (string database, string collection, VehicleSpeed vehicleSpeed) 
{ 
  await client.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(database,                                           collection), vehicleSpeed);
}

Create an instance from VehicleSpeed class, and pass it to the document create method






VehicleSpeed vehicleSpeed = new VehicleSpeed { VehicleNumber = "KJ -7788", 
                                               City = "Colombo", Speed = 100.78 }; 
await CreateDocument(databaseName, databaseCollection, vehicleSpeed);

Let's run this latest code and see whether document is created,
Click on Data Explorer, then it shows database document inside the collection, Click on Documents, you can see created document with few other meta data 

















4.7 Create multiple documents in a collection

We have to create multiple documents to demo this application, let's see how we can do that
Create a for loop and insert few documents with a random speed value as below











public async static Task CreateMultipleDocument(string database, string collection) 
{ 
  int documentCount = 500; 

   for (int i = 0; i < documentCount; i++) 
  { 
    double speed = new Random().NextDouble() * 100; 
    VehicleSpeed vehicleSpeed = new VehicleSpeed { VehicleNumber = "KJ -7788",
                                                  City = "Colombo", Speed = speed}; 
    await client.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(database, 
                                     collection), vehicleSpeed); 
  } 
}


Let's call this method and check whether it works,






await CreateMultipleDocument(databaseName, databaseCollection);

Let's run the application and see how it works, Go to Documents, it shows inserted documents as follows,

















Click on New SQL Query and Get a count of documents as follows,


























SELECT COUNT(1) FROM c

We created required no of documents for this demo, You can find the full code from below,

































static DocumentClient client; 

 public async static Task Create () 
{ 
  try 
  { 
    string endPointUrl = "https://highwaytraffic.documents.azure.com:443/"; 
    string primaryKey = "jSuZYTAvpOiw65P1O1TEPD7pAmjTTDUd5I9pudCkABUz97f3fxcnjg259o7jqDNCJXDyRAQrjOm6H2k24ZjyrQ==";

     string databaseName = "trafficdata"; 
    string databaseCollection = "vehiclespeed"; 

     client = new DocumentClient(new Uri(endPointUrl), primaryKey); 

     await CreateDatabase(databaseName); 

     await CreateCollection(databaseName, databaseCollection); 

     VehicleSpeed vehicleSpeed = new VehicleSpeed { VehicleNumber = "KJ -7788", 
                                                 City = "Colombo", Speed = 100.78 }; 
    await CreateDocument(databaseName, databaseCollection, vehicleSpeed); 

     await CreateMultipleDocument(databaseName, databaseCollection); 

  } 

  catch (DocumentClientException de) 
 { 

   Exception baseException = de.GetBaseException(); 
  Console.WriteLine("{0} error occurred: {1}, Message: {2}", de.StatusCode, de.Message,                        baseException.Message); 
 } 

  catch (Exception e) 
 { 
   Exception baseException = e.GetBaseException(); 
   Console.WriteLine("Error: {0}, Message: {1}", e.Message, baseException.Message); 
 } 
}


5 Create Azure Function App

5.1 Create a basic Function App

Let's see how we can create a azure function app in the portal, Click on New icon, 


In next screen, click on Compute and it shows Function App, click on it







































You can see the following create screen, 
Add application name, subscritption, resource group and select an operation system, 
You have to select App Service plan as the Hosting plan, Hosting plan is going to define how resources are going to allocate for your function app, 

If we select Consumption plan, resources are dynamically added to your application as per requirements, and you only pay for the time your function app runs, 

If you select App service plan, your function app runs on a dedicated VM similar to web apps, API apps and mobile apps, your function app is always running, Let's select App service plan as the hosting plan.

You have to create a storage account for your function app to use as below

We have defined all the configurations for function app to start, Let's go into the next step,





















































You will be prompted to this screen when function app created, You can find app curl, status of the function app from this screen















5.2 Create a new function

We have to create a new function inside this Function app, Click on + icon as below,






















You will see this screen, Click on Custom Function to view available templates to create a Function App

























You can see available templates as below,




















5.3 Create CosmosDB trigger

We are going to create a trigger on Azure Cosmos DB and call a function based on that, So select CosmosDBTrigger - C# template, Then you can what are the options you want to provide in the bottom

In Azure Cosmos DB trigger section, You can see Azure Cosmos DB account connection, we have to provide a Cosmos DB account to proceed the trigger, click on new link





























You can see available database accounts as below, Let's select our existing Azure Cosmos DB

























Give a name to your function, You can see new Cosmos DB account connection has been created as below,
































Click on show value in front of Azure Cosmos DB account connection, You can see endpoint URL to Cosmos DB account and the auth key as below


































Provide Database name & Collection name to the data set we are going to work on,
You can see Collection name for lease textbox,
Azure Cosmos DB Triger use Cosmos DB Change feed to listen to the changes, This trigger needs a secondary collection to store leases over the partitions,
Database collection and lease collection both should be available for the trigger to work


































Go to trafficdata database and expand the collections,
You can see leases collection has been created automatically,
You can see available documents as follows, Those documents contains metadata about data partitions


















5.4 Run CosmosDB trigger

If you go into the Function App, you can see new function is created, Let's Run this application and see what happens,
You can see Logs windows appears as below, it shows Function start and completion, And it logs no new traces available in each minute,



Go to Visual studio and modify Create() method to insert a new document into monitored collection, Let's see this function app can trigger that






You can see logs window, Since collection got changed, Function has been started, and it has gone inside the if condition and printed document id as below,














6 Change basic Function

Let's change function app logic to detect high speed of the vehicle as follows,
Add a for loop in Run method, its going to detect a high speed value and print it in the screen
If you want you can copy high speed vehicles into another collection, may be can store it in a blob



















Change the speed of the vehicle as 112.78, and run the solution







New document is created in the Cosmos DB collection and you can see the log window as below
It shows 'High speed detection' message.

We have created a trigger on Azure Cosmos DB and it fires when a change occur in a document,

 Cosmos DB has a new feature called Cosmos DB change feed, it can continuously read batch of changed entities, it can track list of changed documents, not only documents graphs, mongo db documents as well

When you query, as a result you get list of changed documents, list is going to depend on the throughput of the collection

Let's say you got a fatal error in the middle of the process, you can restart and continue from the recent change token, it's not going to miss any change








7 Download

7.1 Tech Net Gallery

You can download sample application from this location, https://gallery.technet.microsoft.com/Create-Cosmos-DB-Documents-e4d8ae00

7.2 GitHub

You can clone the source code from here, https://github.com/hansamaligamage/cosmoscollection-create

8 Conclusion

This article explains how to create a Azure Cosmos DB database, collection & documents using Azure Cosmos DB SDK. We can track changes on a collection by using Azure Cosmos DB change feed. Application explains how to run a trigger on Azure Cosmos DB data set and perform an action using a Azure Function app.

9 See Also


10 References

Posted by at
Labels: , , , , , ,

5 comments:

  1. UnknownAugust 6, 2018 at 2:34 AM

    Very informative!! Keep update and share Azure Online Training Hyderabad

    ReplyDelete
  2. AnonymousAugust 18, 2018 at 3:03 AM

    Is that possible to create Azure function without logging to azure portal manually? and also it possible to do a HTTP request on that trigger because we need to notify the server.

    Thanks!
    Sundar K

    ReplyDelete
  3. Azure DevOpsJanuary 10, 2019 at 1:48 AM

    Your thinking toward the respective issue is awesome also the idea behind the blog is very interesting which would bring a new evolution in respective field. Thanks for sharing.
    MS Azure Online Training

    ReplyDelete
  4. anirudhApril 9, 2019 at 2:18 AM

    Thanks for the great article this is very useful info thanks for the wonderful post.
    Best Devops Training Institute

    ReplyDelete
  5. vijayAugust 1, 2020 at 6:10 AM

    I am really happy with your blog because your article is very unique and powerful information....Very informative article.Thank you admin for you valuable points. Keep Rocking!!!!
    Salesforce Training in Chennai

    Salesforce Online Training in Chennai

    Salesforce Training in Bangalore

    Salesforce Training in Hyderabad

    Salesforce training in ameerpet

    Salesforce Training in Pune

    Salesforce Online Training

    Salesforce Training

    ReplyDelete

Subscribe to: Post Comments (Atom)