12/28/19

Designing a Ticket Turnstile System at a Subway Train Station

Recently, I interviewed at a company where the interviewer asked me to explain how I would go about designing a ticket turnstile system at a subway train station. For some inexplicable reason, I just talked about it at a very high level, making incorrect assumptions along the way. I never made an attempt to first write down my thoughts on the notebook in front of me and then try to go about explaining my ideas to the interviewer (and to add to my bad luck, the interviewer also did not ask me to draw or explain anything on the white board).

After I came home, I decided to work on that design again, starting with a basic use case, with a pen and a notebook in hand. As i started writing down my thoughts, I realized that the requirements were not complex at all. By not writing down my ideas on paper, I had just complicated things for myself and that made me come across as an incompetent engineer, incapable of designing software systems. Enough venting and crying. In under 10 minutes this is what I was able to come up with :

  • Consider a simple use case of a person, who has just bought a ticket for a ride from station X to station Y. I am therefore going to assume that there is a unique barcode generator of some sort and when the user bought a ticket, he was assigned a ticket with a unique barcode.
  • Now that the user has bought the ticket, I am going to assume that there is a separate database of all tickets that are current and this particular ticket has been moved to that database. I am assuming that there is one row of information per ticket in the database. This row would include fields like the serial number,location, payment details, IsCheckedIn and IsCheckedOut.
  • My assumption is that once the user scans the ticket at the starting point, the IsCheckedIn field is set to true and when the user exits from the destination station, the IsCheckedOut field is set to true.So considering a happy path, the user would checkin and checkout without any issues and the database would be updated accordingly and that would signal a completed transaction for that ticket.
  • Because the data for each ticket requires just one row, given the fact that there is no other relational data, I am assuming that we do not need to use a relational database. We can use a no-sql database like ravendb where all the information can be stored in a single document or a JSON blob.
  • Because each ticket is unique and there is nothing relational, there shouldn't be any performance issues due to locking of resources. All that needs to happen is the updating of the IsCheckedIn and IsCheckedOut fields for each ticket. As a result, the performance of the system would be a function of the processing power of the database server and the number of connections it can handle.There will be no need for any kind of database replications to keep the information up to date.
  • Now, let us say, the user has bought a ticket and for some reason, he is not able to get into the station (or get out) because the scanner wont recognize his ticket for some reason. So I am going to assume that the user will go to an attendant at the station, who will try to look up the ticket on a computer. So the design would need to include a client, that will point to the same database of all current tickets.
  • Once the user has checked out from the destination station, the IsCheckedOut field will be set to true, meaning that the transaction is complete. This record can then be moved to another database of all completed transactions, where they can stay for a mandated(by government, is my assumption) time period before the same barcodes can be recycled.
  • For security reasons, I am also assuming that both the IsCheckedIn and IsCheckedout fields need to be set to true for a transaction to be considered as complete. My assumption is - the user can scan out only if the user has scanned in.

I still feel bad about not having come up with something along these lines during the interview. This is how I normally approach all my design projects (writing down my thoughts on paper), not sure why I kept talking in the air that day. Was I tense or careless or stressed out or just plain stupid? It still haunts me.

12/2/19

Azure App Services : Serverless functions

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Serverless functions are one of the services under the App Services Category.

A Serverless function is a piece of code that can be run independently. It can be triggered by certain events or invoked explicitly depending on the business requirements. Every time a serverless function runs it uses a certain amount of memory and Azure makes sure to run it on the appropriate server (depending on the language used to create the serverless function. You can create serverless functions in different languages like C#, Java, Javascript, Python and some others). So when you configure serverless functions in the Azure portal, you just specify the amount of memory you want to allocate. You do not have to be bothered about managing or scaling the server instances. Because of this, they are known as serverless functions. It is not because they don't run on servers. They do.

Here is an example of a serverless function :

There is a cool online shoe store where you go to buy your shoes. You go to their website and you want to get this brand new model xyz100, but that model is out of stock. So the store offers an option for you to be notified when the stock is replenished. You just need to provide your email which gets stored in their database. Behind the scenes, the store will create a serverless function that will be invoked whenever the model xyz100's stock is replenished. This function will then go to the database and get a list of people(including you) who have requested to be notified when this model is in stock and will send out emails to all of them.


In the case of serverless functions you are charged only when the function runs. You can always increase the amount of memory depending on your business needs. Since serverless functions are stand-alone and can be created in several different languages, they provide a lot of flexibility and freedom when it comes to the development process. You can have separate teams building and managing different functions. You can also choose the technology that best suits the requirement.

App Services in Microsoft Azure : Logic Apps

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Logic Apps are one of the services under the App Services Category. They are similar to workflows and can be used to perform a set of connected operations. Here is an example of a logic app :

Let us say there are a set of tables on two different database servers and you are trying to replicate data from one server to the other. Your goal is to monitor the destination tables and perform two specific actions when new data comes in :

  • Call a stored proc that processes the new data and returns a result set.
  • Invoke an API and pass the result set from the previous step.

Within a few minutes, you can create a logic app which can be scheduled to run at specified intervals. This logic app will have two connected nodes. The first one calls a stored procedure that queries the destination tables for new data and does some business rules processing and returns a result set and the second node can then take this result set and pass it to an API.

We can also scale up (i.e. increase the processing power) or scale out (increase the number of instances) very easily via the Azure portal. We do have to deal with any concurrency issues that may arise when several instances of the logic app are running simultaneously.

10/25/19

App Services in Microsoft Azure : Web Apps and Web Jobs

Azure offers a set of PAAS(Platform-As-A-Service) offerings known as App Services. Web Apps and Web Jobs are two of the services under the App Services category.

Before Azure Web Apps, in order to deploy web applications built using the .Net framework, we needed an infrastructure comprising of software and hardware. For example - we needed a machine with a windows operating system, along with the .Net framework components and an IIS web server. We could scale up, but if we needed to scale out, we had to replicate this process several times.

With Web Apps we can now deploy a web application to a Web App without having to worry about setting up any of the underlying infrastructure. Our web application can be up and running in just a few minutes. We can also scale up (i.e. increase the processing power) or scale out (increase the number of instances) very easily via the Azure portal. We can also use Web App as a pre-production environment as well in order to test our application before we go live. We can also do a/b testing with one Web App instance running the production code and another Web App instance running the new features that we are testing. We can then very easily swap the existing production instance with the pre-production instance if needed.

Azure Web jobs are used for deploying batch jobs without worrying about the underlying infrastructure. An example of a batch job that can be deployed to a Web Job would be a windows service. In the past, if we developed a windows service, we needed an infrastructure similar to the web application, in order to install it and run it. With Web Jobs, we just deploy our windows service .exe file to a Web Job and everything is ready to go. Windows Services usually work in tandem with web applications so a Web Job is deployed to the same instance as the related Web App. It is also very easy to debug the Web Apps and Web Jobs.

NOTE: Web Apps and Web jobs support other platforms and languages like Java and Python and others in addition to the .Net platform.

4/14/17

Why do we need Integration Tests?

In the world of software development, every moderately complex software application will have a large code base consisting of several modules. There will be several software developers working on a given module. if you are a software developer, how would you ensure that your changes have not broken some existing functionality? How would you know if some other developer has made some changes that will end up breaking your code changes? One way is to keep testing the application 24/7 and making sure you have covered every possible use case. This is not only a preposterous but also an impossible approach to solving the problem at hand. This is exactly where Integration tests come into play. 

While developing the software application, we create integration tests for as many use cases as we can. At the very least, we need to ensure that we have tests for the core use cases, the bread and butter of the application. Once we have these tests in place, we run them before every new check-in that happens. If any of the existing tests fail, that will mean that the new changes need to reviewed before they can be checked-in. Every developer needs to make it a habit to run these tests before every check-in. 

There should also be continuous integration builds in place for the integration tests project. This will ensure that the tests will get run on different environments along the deployment cycle. This will ensure that there are no environment-specific dependencies that will cause the test to pass in one environment and fail in another. This will ensure that the tests are self-contained and can be run in any environment that we need to. If the company plans to move towards a continuous delivery model, then it becomes even more imperative that the integration tests are run during every build cycle.

Besides the functionality-breaking reasons, there is one other very important reason to have integration tests. Since the integration tests are self-contained, they are repeatable, which means they can be run over and over without causing any side effects. This is something that is not possible functionally. Let us say you are trying to troubleshoot a particular issue that requires certain data to be present. If you wanted to point your code to the application itself, then you would have to recreate the scenario and associated data every single time.This could be a very tiresome and time-consuming process that can be very frustrating. This can be compounded several times if your servers are on different machines across different networks.

With integration tests, you can simulate the issue via code on your local machine. Since the test is repeatable, the required data will need to be created only once. The test can keep creating and dropping the scenario every time it is run. This will eliminate the need to crank up the application on the web server and go through the hassle of creating data inside the application. There is no dependency on external servers and associated latency. We are not impacted if the external servers are brought down for maintenance or if the databases are refreshed.

In the process of troubleshooting the issue via integration tests, we will end up writing more and more integration tests that will remain in place once we have managed to get them to pass. This repeatable nature of integration tests makes them very powerful and valuable.

10/1/16

Designing a Vending Machine

I interact with Vending machines at work every single day, to purchase some snacks or get a drink and so on. I have always wanted to implement one just out of curiosity but never got around to doing it. Then one day I got an email from a software consulting firm asking me if i was interested in interviewing with them and if so, I should do a code exercise and send it back to them. One of the coding exercise options turned out to be a vending machine. It had a set of features that were expected. Though I had no interest in interviewing with that company, I decided that I was going to code the Vending Machine for fun anyways.

Following were the vending machine features in that coding exercise:
  • Accept Coins
    • As a vendor I want a vending machine that accepts coins So that I can collect money from the customer. The vending machine will accept valid coins (nickels, dimes, and quarters) and reject invalid ones (pennies). When a valid coin is inserted the amount of the coin will be added to the current amount and the display will be updated. When there are no coins inserted, the machine displays INSERT COIN. Rejected coins are placed in the coin return. 
    • NOTE: The temptation here will be to create Coin objects that know their value. However, this is not how a real vending machine works. Instead, it identifies coins by their weight and size and then assigns a value to what was inserted. You will need to do something similar
  • Select Product
    • As a vendor I want customers to select products So that I can give them an incentive to put money in the machine. 
    • There are three products: cola for $1.00, chips for $0.50, and candy for $0.65. When the respective button is pressed and enough money has been inserted, the product is dispensed and the machine displays THANK YOU. If the display is checked again, it will display INSERT COIN and the current amount will be set to $0.00. If there is not enough money inserted then the machine displays PRICE and the price of the item and subsequent checks of the display will display either INSERT COIN or the current amount as appropriate.
  • Make Change
    • As a vendor I want customers to receive correct change So that they will use the vending machine again.
    • When a product is selected that costs less than the amount of money in the machine, then the remaining amount is placed in the coin return.
  • Return Coins
    • As a customer I want to have my money returned So that I can change my mind about buying stuff from the vending machine.
    • When the return coins button is pressed, the money the customer has placed in the machine is returned and the display shows INSERT COIN.
  • Sold Out
    • As a customer I want to be told when the item I have selected is not available So that I can select another item.
    • When the item selected by the customer is out of stock, the machine displays SOLD OUT. If the display is checked again, it will display the amount of money remaining in the machine or INSERT COIN if there is no money in the machine.
  • Exact Change Only
    • As a customer I want to be told when exact change is required So that I can determine if I can buy something with the money I have before inserting it.
    • When the machine is not able to make change with the money in the machine for any of the items that it sells, it will display EXACT CHANGE ONLY instead of INSERT COIN.

My implementation included the following classes:

Client: This class represents the end user that will be interacting with the Vending Machine.

public class Client
    {
        public void Start()
        {
            var vm = new VendingMachineFacade(new CoinService(),new ProductService(new ProductRepository(),new ProductInventoryRepository()));

            //case 1 - invalid coins
            var response = vm.AcceptCoin(new InputCoin() { Weight = 1000, Size = 50 });
            if(response.IsSuccess == false)
            {
                //print response.Message to console
                return;
            }

            //case 2 - valid coins,return coins
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.ReturnCoins();

            //case 3 - valid coins, invalid product code
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("COOO1");

            //case 4 - valid coins, valid product code, exact change only
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("CO1");

            //case 5 - valid coins, valid product code, less amount entered
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("CO1");

            //case 6 - valid coins, valid product code, more amount entered, make change
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("CO1");

            //case 7 - valid coins, valid product code, correct(>=) amount entered, sold out
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("CO1");

            //case 8 - valid coins, valid product code, correct(>=) amount entered, sold out, return coins
            vm.AcceptCoin(new InputCoin() { Weight = 100, Size = 50 });
            vm.SelectProduct("CO1");
            vm.ReturnCoins();
        }
    }

VendingMachineFacade :  This class represents the vending machine interface that the client will be interacting with.

public class VendingMachineFacade
    {
        private double _cost;
        private CoinService _coinService;
        private ProductService _productService;
        public VendingMachineFacade(CoinService coinService,ProductService prodService)
        {
            _coinService = coinService;
            _productService = prodService;
        }

        //behaviors
        public VendingResponse AcceptCoin(InputCoin coin)
        {
            VendingResponse response = new VendingResponse();                       

            //check if the values correspond to an accepted coin            
            var currentCoin = _coinService.GetCoin(coin.Weight, coin.Size);

            //not a valid coin
            if (currentCoin == null)
            {
                response.Message = "Insert Coin";
                response.IsRejectedCoin = true;
                response.RejectedCoin = coin; //return rejected coin
                return response;                                
            }

            //valid coin
            _cost += currentCoin.Value;
            response.Message = _cost.ToString();
            response.IsRejectedCoin = false;
            return response;
        }
        public VendingResponse SelectProduct(string code)
        {
            var response = new VendingResponse();
            
            //check if the code is valid            
            //if no, return error object with details
            var product = _productService.GetProduct(code);

            //invalid code entered
            if(product == null)
            {
                response.Message = "Invalid Product Selected. Please try again";
                response.IsSuccess = false;
                return response;
            }

            //no coins entered, but selection pressed
            if (_cost == 0)
            {
                //if exact change item, message = "exact change only"
                response.Message = "Insert Coin";
                response.IsSuccess = false;
                return response;
            }

            //entered coins less than cost
            if (_cost < product.Cost)
            {
                response.Message = string.Format("Price : {0}", product.Cost);
                response.IsSuccess = false;
                return response;
            }

            //if exact change product            
            if (_productService.IsExactChangeOnlyProduct(product) && product.Cost != _cost)
            {
                response.Message = "Exact Change Only";
                response.IsSuccess = false;
                return response;
            }

            //all good, valid code and valid amount entered
            var quantity = _productService.GetProductQuantity(code);
            if (quantity > 0)
            {
                response.Message = "Thank You";
                response.IsSuccess = true;
                _productService.UpdateProductQuantity(code);
                MakeChange(_cost - product.Cost);
                return response;
            }
            else
            {
                response.Message = "SOLD OUT";
                response.IsSuccess = false;
                return response;
            }                        
        }
        public ItemChange ReturnCoins()
        {
            return MakeChange(_cost);
        }
        private ItemChange MakeChange(double input)
        {
            ItemChange itemchange = new ItemChange();
            var change = input - _cost;
            if (change == 0) return itemchange;
            double remainingAmount = 0;

            //get the number of quarters in the remaining amount
            var quarters = (int)(change / 0.25);
            if (quarters > 0)
            {
                itemchange.NoOfQuarters = quarters;
                remainingAmount = (change - (quarters * 0.25));
                if (remainingAmount == 0) return itemchange;
            }

            var nickels = (int)(change/ 0.10);
            if(nickels > 0)
            {
                itemchange.NoOfNickels = nickels;
                remainingAmount = (change - (nickels * 0.10));
                if (remainingAmount == 0) return itemchange;
            }            

            var dimes = (int)(change / 0.05);

            if (dimes > 0)
            {
                itemchange.NoOfDimes = dimes;
                remainingAmount = (change - (dimes * 0.05));
                if (remainingAmount == 0) return itemchange;
            }
                        
            return itemchange;
        }        
        private void SoldOut(string code)
        {
            //inventory manager checks quantity
        }
        private void ExactChangeOnly() { }
    }  


InputCoin: This would represent the coin inserted by the user

 public class InputCoin
    {
        public int Weight { get; set; }
        public int Size { get; set; }
    }


CoinService :  This class provides functionality related to Coins.

  public class CoinService
    {
        private static IEnumerable AcceptedCoins = new List() { new Quarter(), new Dime(), new Nickel() };

        public Coin GetCoin(int weight, int size)
        {
            return AcceptedCoins.Where(x => x.Weight == weight && x.Size == size).FirstOrDefault();
        }
    }


ProductService :  This class provides functionality related to the actual products that can be purchased.

  public class ProductService
    {
        private ProductRepository _productRepository;
        private ProductInventoryRepository _productInventoryRepository;
        public ProductService(ProductRepository repository, ProductInventoryRepository inventoryRepository)
        {
            _productRepository = repository;
            _productInventoryRepository = inventoryRepository;
        }

        public int GetProductQuantity(string code)
        {
            var quantities = _productInventoryRepository.GetInventory();
            return quantities[code];
        }

        public Product GetProduct(string code)
        {
            return GetAllProducts().Where(x => x.Code == code).First();
        }

        public IEnumerable GetAllProducts()
        {
            return _productRepository.GetProductList();
        }

        public void UpdateProductQuantity(string code)
        {
            //this should happen in a lock to handle concurrency
            _productInventoryRepository.UpdateInventory(code);
        }

        public bool IsExactChangeOnlyProduct(Product product)
        {
            if (product.Type == ProductType.Chips) return true;
            return false;
        }
    }


ProductRepository : This class is the acts as the ORM layer for interaction with the product datasources.

 public class ProductRepository
    {
        private static List _products;
        //reader writer lock
        public virtual IEnumerable GetProductList()
        {
            if (_products == null)
            {
                _products = new List();
                _products.Add(new Product() { Code = "CO1", Type = ProductType.Cola, Cost = 1.0 });
                _products.Add(new Product() { Code = "CO2", Type = ProductType.Cola, Cost = 1.0 });
                _products.Add(new Product() { Code = "CO3", Type = ProductType.Cola, Cost = 1.0 });

                _products.Add(new Product() { Code = "CH1", Type = ProductType.Chips, Cost = 0.50 });
                _products.Add(new Product() { Code = "CH2", Type = ProductType.Chips, Cost = 0.50 });
                _products.Add(new Product() { Code = "CH3", Type = ProductType.Chips, Cost = 0.50 });

                _products.Add(new Product() { Code = "CA1", Type = ProductType.Candy, Cost = 0.65 });
                _products.Add(new Product() { Code = "CA2", Type = ProductType.Candy, Cost = 0.65 });
                _products.Add(new Product() { Code = "CA3", Type = ProductType.Candy, Cost = 0.65 });
            }

            return _products;
        }
    }
  

ProductInventoryRepository: This class is the acts as the ORM layer for interaction with the product inventory related datasources.

public class ProductInventoryRepository
    {
        private static Dictionary _productQuantities;
        public Dictionary GetInventory()
        {
            if (_productQuantities == null)
            {
                _productQuantities = new Dictionary();
                _productQuantities.Add("CO1", 10);
                _productQuantities.Add("CO2", 10);
                _productQuantities.Add("CO3", 10);

                _productQuantities.Add("CH1", 10);
                _productQuantities.Add("CH2", 10);
                _productQuantities.Add("CH3", 10);

                _productQuantities.Add("CA1", 10);
                _productQuantities.Add("CA2", 10);
                _productQuantities.Add("CA3", 10);
            }

            return _productQuantities;
        }
        public void UpdateInventory(string code)
        {
            //sorround with reader writer lock
            var currentCount = _productQuantities[code];
            if(currentCount > 0)
                _productQuantities[code]--;
        }
    }



ItemChange: This class represents the return amount


public class ItemChange
{
    public int NoOfQuarters;
    public int NoOfNickels;
    public int NoOfDimes;
}


This represents my initial thought process in regards to a Vending Machine implementation. I will continue to tweak this code as time permits and as I get better ideas,

3/13/16

Process Monitor a.k.a Procmon

Process Monitor is an advanced monitoring tool for Windows that shows real-time file system, Registry and process/thread activity. You can get more information here

A couple of months ago, I ran into a very strange issue with some data update functionality involving two separate but related applications, one a WPF client and the other a console application running as a scheduled task.

The data update happened in two ways. In the first case, the user explicitly requested for an update using one of the context menu options inside the WPF Client. In the second case, the data update request was automatically queued up by the same Client when certain state changes happened. These queued requests were then processed by the scheduled console application at a later time.The code to perform the actual data update was in a separate dll and both these applications were referencing the same dll.

What was strange was that the manual request was updating the data successfully whereas the queued up request was failing to do so.

I stepped through the code but I was not able to find anything different. so I decided to use the process monitor to troubleshoot this issue. I ran procmon and monitored the manual data update request. Procmon shows you all the activities like registry access and files being loaded. I didn't notice any assembly loading failures or any other access failures.

I then repeated the same process with the console process. In this case, I noticed that there was an assembly loading failure. The console process was trying to load a particular assembly and it wasn't able to locate it after trying several locations. Procmon shows you all the paths that the process is looking at. The assembly that the process was trying to load was missing and that was the reason for the data update failure. Once we dropped the missing dll into a specific location, the process was able to load it and the data process worked just fine.

We later found out that the install team had forgotten to include that particular dll while building their installers. Once the installer got fixed, the data update issue went away.