Running automated tests with the Azure Cosmos DB Emulator on AppVeyor

When developing an application relying on Azure Cosmos DB as the data backend, it’s always interesting to develop and run automated tests to check the integration between the business logic layer and the data layer. Nevertheless, creating a specific database and host it on, is an expensive and not so flexible solution. Hopefully, Microsoft is providing the Azure Cosmos DB Emulator to develop and test application interacting with Azure cosmos DB. To run our automated tests during the continuous build process, we can leverage this Emulator. As often, I’ll demonstrate the how-to with the Ci/CD service AppVeyor but this should be applicable to other CI/CD (on-premises or cloud-based).

The good news with AppVeyor is that a recent version of the Azure Cosmos DB Emulator is already provided in the build image. We don’t need to install it before anything. If you really want the last version of this emulator or if you’re using another CI/CD service, I’ll explain at the end of this blog post how to download and install the emulator (and also how to uninstall it).

The first step is to start the emulator. The emulator is located on C:\Program Files\Azure cosmos Db Emulator\CosmosDb.Emulator.exe and this executable is expecting arguments to define the action to execute. By default, without argument, you’ll just start the exe. In this case, we need to ask the emulator to start without displaying UI information and we can also ask it to not start the Data Explorer (that is dedicated to UI interactions that anyway we won’t be able to use during a build). The two arguments to pass to the exe are: /NoUI /NoExplorer.

By default in v2.7, the explorer is just enabling the SQL API. If you want to use other API such as Cassandra, Gremlin or MongoDB then you’ll have to enable the endpoints. In this case, we’ll enable the Gremlin endpoint by providing the argument /EnableGremlinEndpoint. It’s also possible to define on which port the emulator will be listening for this endpoint but I usually don’t change them. If you want to get an overview of all the flags that you can activate or disable in the emulator just check on your own computer by submitting the command CosmosDb.Emulator.exe /help.

To start the emulator with the default endpoint for Gremlin in the context of a CI/CD service, the final command is:

"C:\Program Files\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe"  /NoExplorer /NoUI /EnableGremlinEndpoint

You can place this command in a batch file named start-emulator.cmd

If you just submit this command in the install step of your CI/CD workflow, you’ll have to face difficult times. Indeed, the Emulator is just running and listening on ports, so this command is never ending. On a CI/CD build, it means that the command will run on the main process of the CI/CD build and block everything, eventually your build will hang out. To avoid this, you’ll need to execute this command in another process. With PowerShell, you can just do this with the cmdlet Start-Process "start-emulator.cmd".

The emulator always take a few seconds to fully start. It’s important to check that the port is open before effectively using it or you could have unexpected exceptions due to the unavailability of the emulator. To avoid this start the emulator as soon as possible (before your build) and use it as late as possible (run the tests relying on it as the last tests). Nevertheless, it’s always a good idea, to check if this tool is correctly running before using it. To perform this check, you can assess that the port is effectively listening. Following PowerShell script is doing the job by checking if it’s possible to connect to the port at regular interval as long as the port is not available. When the port is available, the script display a positive message and continue.

$attempt = 0; $max = 5
do {
	$client = New-Object System.Net.Sockets.TcpClient([System.Net.Sockets.AddressFamily]::InterNetwork)
	try {    
		$client.Connect("", 8901)
		write-host "Gremlin endpoint listening. Connection successful."
	catch {
		if($attempt -eq $max) {
			write-host "Gremlin endpoint is not listening. Aborting connection."
		} else {
			[int]$sleepTime = 5 * (++$attempt)
			write-host "Gremlin endpoint is not listening. Retry after $sleepTime seconds..."
			sleep $sleepTime;
}while(!$client.Connected -and $attempt -lt $max)

Now that the emulator is started, you’ll need to create a database and a collection. It’s important to note that even if you want to create a graph database (Gremlin) and the Gremlin API is listening on port 8901 (by default), you’ll have to connect to the SQL API at the port 8081 (by default) to create the database and the collection. This is valid for all the api kind and not just for Gremlin. In fact, if you want to manage your Azure Cosmos DB instance, you must connect to the SQL API.

To create a database and a collection (if they don’t exist), you’ll need the Microsoft.Azure.DocumentDB package. Once installed, you’ll be able to use the methods CreateDatabaseIfNotExistsAsync and CreateDocumentCollectionIfNotExistsAsync of the class DocumentClient. The response of these two methods returns information about what happened (the database/collection was already existing or has just been created).

using (var client = new DocumentClient(sqlApiEndpoint, authKey))
    var database = new Database() { Id = databaseId };
    var databaseResponse = client.CreateDatabaseIfNotExistsAsync(database).Result;
    switch (databaseResponse.StatusCode)
        case System.Net.HttpStatusCode.OK:
            Console.WriteLine($"Database {databaseId} already exists.");
        case System.Net.HttpStatusCode.Created:
            Console.WriteLine($"Database {databaseId} created.");
            throw new ArgumentException($"Can't create database {databaseId}: {databaseResponse.StatusCode}");

    var databaseUri = UriFactory.CreateDatabaseUri(databaseId);
    var collectionResponse = client.CreateDocumentCollectionIfNotExistsAsync(databaseUri, new DocumentCollection() { Id = collectionId }).Result;
    switch (collectionResponse.StatusCode)
        case System.Net.HttpStatusCode.OK:
            Console.WriteLine($"Collection {collectionId} already exists.");
        case System.Net.HttpStatusCode.Created:
            Console.WriteLine($"Database {collectionId} created.");
            throw new ArgumentException($"Can't create database {collectionId}: {collectionResponse.StatusCode}");

Once the database created, you’ll need to populate it before you can effectively run your tests on it. This is usually something that I code in the OneTimeSetUp of my test-suite. To proceed, you must create a few Gremlin commands to add vertices and edges such as:

g.addV('person').property('pk', 1).property('id', 'thomas').property('firstName', 'Thomas').property('age', 44)
g.addV('person').property('pk', 1).property('id', 'mary').property('firstName', 'Mary').property('lastName', 'Andersen').property('age', 39)
g.addV('person').property('pk', 1).property('id', 'ben').property('firstName', 'Ben').property('lastName', 'Miller')
g.addV('person').property('pk', 1).property('id', 'robin').property('firstName', 'Robin').property('lastName', 'Wakefield')

Then you’ll have to execute them. Tiny reminder: don’t forget that you’ll have to use the Gremlin API endpoint and not anymore the SQL API endpoint! The driver to connect to this API is the Gremlin.Net driver available on nuget and the source code is available on GitHub on the repository of TinkerPop. Once again, the usage is relatively straightforward, you first create an instance of a GremlinServer and then instantiate a GremlinClient to connect to this server. Using the method SubmitAsync of the class GremlinClient, you’ll send your commands to the Azure Cosmos DB Emulator.

var gremlinServer = new GremlinServer(hostname, port, enableSsl, username, password);

using (var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(), GremlinClient.GraphSON2MimeType))
    foreach (var statement in Statements)
        var query = gremlinClient.SubmitAsync(statement);
        Console.WriteLine($"Setup database: { statement }");

Now that your database is populated, you can freely connect to it and run your automated tests. Writting these test is another story (and another blog post)!

Back to the first step … as stated before I took the hypothesis that Azure Cosmos DB Emulator was already installed on your CI/CID image and that the version suited your needs. If it’s not the case, that’s not the end of the world, you can install it by yourself.

Before, you start to uninstall and install this software, it could be useful to check the version of the current installation. The following PowerShell script is just displaying this:

Write-Host "Version of ComosDB Emulator:" ([System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Program Files\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe").FileVersion)

If you need to install a new version of Azure Cosmos DB Emulator, the first action is to remove the existing instance of Azure Cosmos DB Emulator. If you don’t do it, the new installation will succeed but the emulator will never start correctly. The black magic to remove an installed software is to use the Windows Management Instrumentation Command-line. This command is surely not really fast but it’s doing the job.

wmic product where name="Azure Cosmos DB Emulator" call uninstall

Then, you can download the last version of Azure Cosmos DB Emulator at the following shortcut url:

Start-FileDownload '' -FileName 'c:\projects\cosmos-db.msi'

Once downloaded, the installation is started with the following command

cmd /c start /wait msiexec /i "c:\projects\cosmos-db.msi" /qn /quiet /norestart /log install.log

You should now be able to install, start and populate your Azure cosmos DB Emulator on your favorite CI/CD server … just a few tests to write and you’ll have some end-to-end tests running with Azure Cosmos DB Emulator.

The following appveyor.yml statements should do the job:

# Install the Azure CosmosDb Emaulator with the option /EnableGremlin
- wmic product where name="Azure Cosmos DB Emulator" call uninstall
- ps: Start-FileDownload '' -FileName 'c:\projects\cosmos-db.msi'
- cmd: cmd /c start /wait msiexec /i "c:\projects\cosmos-db.msi" /qn /quiet /norestart /log install.log  
- ps: Write-Host "Version of ComosDB Emulator:" ([System.Diagnostics.FileVersionInfo]::GetVersionInfo("C:\Program Files\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe").FileVersion)
- ps: Start-Process "start-emulator.cmd"

Don’t forget to check/wait for the Emulator before executing your tests:

- ps: .\wait-connect-cosmosdb.ps1

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s