A Use Case for DynamoDB

 DynamoDB is a noSQL database from Amazon. More information here.

Let us assume we are building a REST API that deals with three main resources: City, Team, Player. Each resource has different attributes.

Following are the endpoints for each resource:

• City:
• POST: AddCity
• PATCH: UpdateCity
• DELETE: DeleteCity
• GET: ListCities and GetCityByID
• Team:
• POST: AddTeam
• PATCH: UpdateTeam
• DELETE: DeleteTeam
• GET: ListTeams and GetTeamByID
• Player:
• POST: AddPlayer
• PATCH: UpdatePlayer
• DELETE:DeletePlayer
• GET: ListPlayers and GetPlayerByID

Based on the end points, it is evident that we will mostly be doing simple read/write operations and there will be no real need for any complex queries. Sometimes there can be a high volume of read operations, but thats about it. So fast read writes and scalability is our main criteria and of course cost optimization associated with software and hardware and usage.

• Implementation with some SQL database (ex: SQL Server)
• You will have to provision the servers and the software.
• Maintain the servers in the future.
• Manage the scaling as data grows. Might require additional servers.
• We will have to manage the replication, partitioning etc.
• We will need three tables for the three entities - city, team and player and associated referential integrity relationships between them. The database generates the keys for the tables and support for constraints is also built-in.
• Updates to the database schema are non-trivial. Adding additional attributes to a city or a team would require a good deal of work.
• No special support for fast read/write operations
• We can write complex queries (which is not our requirement)
• We will be charged irrespective of the amount of usage

• Implementation with DynamoDB
• It is a managed database, so no need for any software or hardware installations
• No maintenance of the software or hardware. Amazon will take care of this.
• We can use just one table for all the three entities. We store data as key-value pairs and as a result, we can store rows of varying attributes in DynamoDB. So we can store a row for the city, a row for the team and a row for the player in the same table. The only requirement is that each row should have a unique primary key attribute (which is defined in the schema definition). This is my favorite aspect of DynamoDB.
• Since we are storing the data as key-value pairs, adding additional attributes or removing existing attributes (other than primary key) is trivial. This makes DynamoDB very powerful for scenarios that require flexibility(with dynamic requirements).
• Supports fast read/write operations, which is our main requirement.
• Amazon will scale the table as it grows, we do not need to worry about it.
• We do not have to worry about partitioning or replication of data.
• Amazon does not charge for the amount of data, but for the number of reads/writes. We can optimize the costs by designing the schema based on our access patterns. This is the most important aspect to keep in mind when dealing with DynamoDB. For example - if we are trying to update a row with 100kb of data, DynamoDB will charge a lot more than if we were trying to update a row with 1kb of data. So we can perform optimizations like storing the data that will be updated, in a separate row and so on.
• We will have to generate the keys for the table. Other than enforcing the primary key constraint, DynamoDB does not support any other kind of uniqueness or constrains by default. We will need to implement those on our own. The beauty is that we can add manage additional constraints by adding additional rows to the same table and performing atomic operations in our code. For example - if we had a requirement that a city cannot have more than one team with the same name. As soon as we add new team for a city, within the same transaction, we can add another row with a primary key value that is a combination of cityID and team name. The next time we try to add another team with the same name in a city, it will violate the primary key constraint. So it is pretty easy to implement custom constraints in DynamoDB.

It is very clear that DynamoDB is better suited for our REST API requirements than any other SQL or noSQL databases. There are other noSQL options like MongoDB and Cassandra, with their own distinct advantages, but they do not make sense for our requirements. MongoDB for example supports a lot of data types and aggregate queries and transactions while Cassandra supports varying columns of data. 

Microservices - the good and the not-so-good

A microservice can be considered as a self contained unit of functionality, that is usually small to medium in size.  Since the beginning of last year, I have been working on a large web application that is is powered by microservices. This was my first exposure to microservices and I have come to realize that they have their own advantages and disadvantages.

The good:

• A microservice is a well defined unit of functionality. It does only one thing and does it well.
• A microservice is usually small in size and is easy to develop and test and maintain.
• A microservice is an independent unit of functionality, so any tech stack can be used to develop it
• If you decide to use the same tech stack for new microservices, you can copy/paste an existing microservice and reuse the configuration and deployment scripts and maybe even some code.
• Each microservice can have it's own security requirements
• Each microservice can be developed by a small dedicated team of engineers
• It is easy to make changes and deploy a microservice without fear of impacting other applications or services. A great deal of regression or smoke testing is not required like in the case of monolithic applications

The not-so-good:

• Troubleshooting an error in an application powered by microservices is not straight-forward. Each call can potentially traverse across several microservices before the required results are returned. It becomes necessary to pass certain request related data (like a Request ID) from the application to all the services in the chain so that all the errors can be logged with the same request and can be used to identify and troubleshoot errors.
• From a development point of view, each microservice is a separate project and a small dedicated team of engineers usually manage a few microservices. so it becomes hard to get up to speed and become familiar with many different projects. This becomes even more difficult if a different tech stack is used for different microservices
• All interactions with microservices happen via Http and it is a totally different paradigm compared to regular application development. The testing and debugging is done via tools like Postman. It takes some getting used to. 
• Since microservices are well defined units of functionality, they make calls to other microservices for additional information. As a result, there is a lot of dependency between microservices. This can get frustrating, especially during development stage, since some teams keep changing their interfaces constantly. This can also result in a lot of code rewrite.
• Since microservices depend on other microservices, the capability of a given microservice is limited by those dependencies. For example, a microservice may be able to process 50 requests per minute (RPM), but if one of the other dependency microservice can only handle 20 RPM. then this microservice will need to work with that speed.
• Since different microservices could potentially be developed and maintained by different teams in different time zones, a great deal of collaboration and understanding is required for completing a project. This can be very frustrating at times and can lead to a lot of friction.
• You will need to use tools like Splunk (there are several others) for logging and querying information about various microservices. This represents a learning curve, especially if you need to write complex queries for displaying information on dashboards. Splunk, for example, uses regular expressions to query data for meaningful insights. This is again a totally different paradigm compared to typical monolithic applications.
• You will need to document the functionality provided by each microservice via documentation tools like Swagger or Stoplight (there are others as well). This requires a good deal of work and good attention to detail since your documentation is your source of truth for the service users. Any changes need to be updated promptly as well.

Planning for Error Logging

The purpose of error logging is to log information about errors in a way which makes it easy to diagnose and troubleshoot issues when they arise. While error logging is necessary for all applications, it is indispensable when it comes to large distributed applications. 

In the past, I have written some error logging code but I had never worked on a extensive error logging requirement, that too, from scratch. Last month, as I started work on error logging for a large batch project, I quickly realized that a lack of planning and insight can lead to logs that are hard to track, analyze and troubleshoot and could have disastrous consequences on the outcome of the project. 

Here are some questions and thoughts to keep in mind while planning and coding for it:

• Questions
• Why do we want to log errors?
• What do we want to do with the logs?
• Are we going to just search and view them in a tool like Splunk?
• Are we going to build dashboards on top of the logs? if yes, what types of statistics do we want to show on the dashboards?
• what are the different types of errors that need to be logged to account for the various stats that we are interested in?
• What information needs to be logged with each error?
• If we are going to build dashboards, we need to be able to query the logs. So what format makes sense for querying?
• Thoughts
• Ensure that the format and structure of the data is concise and meaningful
• Ensure that the messages and formats are consistent across the board
• Try to use common libraries and classes to centralize the access to logging code for consistency and ease of maintenance.
• Create specific exception classes for each scenario, so that we can track, analyze and troubleshoot errors faster.
• Write extensive unit tests with excellent coverage including tests on specific error messages because it is very difficult to test all scenarios when code changes are made to the error logging code.
• If you are dealing with microservices, make sure to log all information required to track errors that span several services. RequestId is an example of one such piece of information that can be used to connect a request across various services and help troubleshoot issues.

Designing a Ticket Turnstile System at a Subway Train Station

Recently, I interviewed at a company where the interviewer asked me to explain how I would go about designing a ticket turnstile system at a subway train station. For some inexplicable reason, I just talked about it at a very high level, making incorrect assumptions along the way. I never made an attempt to first write down my thoughts on the notebook in front of me and then try to go about explaining my ideas to the interviewer (and to add to my bad luck, the interviewer also did not ask me to draw or explain anything on the white board).

After I came home, I decided to work on that design again, starting with a basic use case, with a pen and a notebook in hand. As i started writing down my thoughts, I realized that the requirements were not complex at all. By not writing down my ideas on paper, I had just complicated things for myself and that made me come across as an incompetent engineer, incapable of designing software systems. Enough venting and crying. In under 10 minutes this is what I was able to come up with :

• Consider a simple use case of a person, who has just bought a ticket for a ride from station X to station Y. I am therefore going to assume that there is a unique barcode generator of some sort and when the user bought a ticket, he was assigned a ticket with a unique barcode.
• Now that the user has bought the ticket, I am going to assume that there is a separate database of all tickets that are current and this particular ticket has been moved to that database. I am assuming that there is one row of information per ticket in the database. This row would include fields like the serial number,location, payment details, IsCheckedIn and IsCheckedOut.
• My assumption is that once the user scans the ticket at the starting point, the IsCheckedIn field is set to true and when the user exits from the destination station, the IsCheckedOut field is set to true.So considering a happy path, the user would checkin and checkout without any issues and the database would be updated accordingly and that would signal a completed transaction for that ticket.
• Because the data for each ticket requires just one row, given the fact that there is no other relational data, I am assuming that we do not need to use a relational database. We can use a no-sql database like ravendb where all the information can be stored in a single document or a JSON blob.
• Because each ticket is unique and there is nothing relational, there shouldn't be any performance issues due to locking of resources. All that needs to happen is the updating of the IsCheckedIn and IsCheckedOut fields for each ticket. As a result, the performance of the system would be a function of the processing power of the database server and the number of connections it can handle.There will be no need for any kind of database replications to keep the information up to date.
• Now, let us say, the user has bought a ticket and for some reason, he is not able to get into the station (or get out) because the scanner wont recognize his ticket for some reason. So I am going to assume that the user will go to an attendant at the station, who will try to look up the ticket on a computer. So the design would need to include a client, that will point to the same database of all current tickets.
• Once the user has checked out from the destination station, the IsCheckedOut field will be set to true, meaning that the transaction is complete. This record can then be moved to another database of all completed transactions, where they can stay for a mandated(by government, is my assumption) time period before the same barcodes can be recycled.
• For security reasons, I am also assuming that both the IsCheckedIn and IsCheckedout fields need to be set to true for a transaction to be considered as complete. My assumption is - the user can scan out only if the user has scanned in.

I still feel bad about not having come up with something along these lines during the interview. This is how I normally approach all my design projects (writing down my thoughts on paper), not sure why I kept talking in the air that day. Was I tense or careless or stressed out or just plain stupid? It still haunts me.

Azure App Services : Serverless functions

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Serverless functions are one of the services under the App Services Category.

A Serverless function is a piece of code that can be run independently. It can be triggered by certain events or invoked explicitly depending on the business requirements. Every time a serverless function runs it uses a certain amount of memory and Azure makes sure to run it on the appropriate server (depending on the language used to create the serverless function. You can create serverless functions in different languages like C#, Java, Javascript, Python and some others). So when you configure serverless functions in the Azure portal, you just specify the amount of memory you want to allocate. You do not have to be bothered about managing or scaling the server instances. Because of this, they are known as serverless functions. It is not because they don't run on servers. They do.

Here is an example of a serverless function :

There is a cool online shoe store where you go to buy your shoes. You go to their website and you want to get this brand new model xyz100, but that model is out of stock. So the store offers an option for you to be notified when the stock is replenished. You just need to provide your email which gets stored in their database. Behind the scenes, the store will create a serverless function that will be invoked whenever the model xyz100's stock is replenished. This function will then go to the database and get a list of people(including you) who have requested to be notified when this model is in stock and will send out emails to all of them.


In the case of serverless functions you are charged only when the function runs. You can always increase the amount of memory depending on your business needs. Since serverless functions are stand-alone and can be created in several different languages, they provide a lot of flexibility and freedom when it comes to the development process. You can have separate teams building and managing different functions. You can also choose the technology that best suits the requirement.

App Services in Microsoft Azure : Logic Apps

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Logic Apps are one of the services under the App Services Category. They are similar to workflows and can be used to perform a set of connected operations. Here is an example of a logic app :

Let us say there are a set of tables on two different database servers and you are trying to replicate data from one server to the other. Your goal is to monitor the destination tables and perform two specific actions when new data comes in :

• Call a stored proc that processes the new data and returns a result set.
• Invoke an API and pass the result set from the previous step.

Within a few minutes, you can create a logic app which can be scheduled to run at specified intervals. This logic app will have two connected nodes. The first one calls a stored procedure that queries the destination tables for new data and does some business rules processing and returns a result set and the second node can then take this result set and pass it to an API.

We can also scale up (i.e. increase the processing power) or scale out (increase the number of instances) very easily via the Azure portal. We do have to deal with any concurrency issues that may arise when several instances of the logic app are running simultaneously.

App Services in Microsoft Azure : Web Apps and Web Jobs

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Web Apps and Web Jobs are two of the services under the App Services category.

Before Azure Web Apps, in order to deploy web applications built using the .Net framework, we needed an infrastructure comprising of software and hardware. For example - we needed a machine with a windows operating system, along with the .Net framework components and an IIS web server. We could scale up, but if we needed to scale out, we had to replicate this process several times.

With Web Apps we can now deploy a web application to a Web App without having to worry about setting up any of the underlying infrastructure. Our web application can be up and running in just a few minutes. We can also scale up (i.e. increase the processing power) or scale out (increase the number of instances) very easily via the Azure portal. We can also use Web App as a pre-production environment as well in order to test our application before we go live. We can also do a/b testing with one Web App instance running the production code and another Web App instance running the new features that we are testing. We can then very easily swap the existing production instance with the pre-production instance if needed.

Azure Web jobs are used for deploying batch jobs without worrying about the underlying infrastructure. An example of a batch job that can be deployed to a Web Job would be a windows service. In the past, if we developed a windows service, we needed an infrastructure similar to the web application, in order to install it and run it. With Web Jobs, we just deploy our windows service .exe file to a Web Job and everything is ready to go. Windows Services usually work in tandem with web applications so a Web Job is deployed to the same instance as the related Web App. It is also very easy to debug the Web Apps and Web Jobs.

NOTE: Web Apps and Web jobs support other platforms and languages like Java and Python and others in addition to the .Net platform.