Original post at Medium - How to estimate the cost of processing an API call?
Is there a way to use load testing and AWS cost reports to calculate the cost per API call?
Yes, and it can be done in four steps!
I recently came across the need to estimate the cost of processing one API call. After doing some research online, I quickly found out that this was not a topic easy to answer. There was no straightforward way to get it explained. At the same time, there are a lot of cloud-based business models today that rely on per transaction pricing which can be boiled down to billing every time an API is used.
If you run a business that uses an API-based model, you will at some point try to establish a price point for the API call. If you know the cost of a given API call, it would help calculate the margin for your price point.
In this article, I want to explain the methodology I have recently used to get an estimate of the cost of processing an API call which would lead to the “Cost of Goods Sold (COGS)” used for calculating margins. Here, I will only look at the infrastructure costs that are needed to serve a transaction (to get to a more precise margin calculation, additional costs would need to be estimated that are not covered by this methodology).
With cloud-based models, using a purely theoretical approach wouldn’t yield satisfying results given how complex some infrastructures can quickly become and the difficulty to isolate the processing of a specific transaction in the broader system.
I will describe an empirical method to estimate the cost of an API call. It can also be used to assess the impact of new features, or brand new products, on costs.
High-level approach
The methodology consists of four steps:
- Set up load testing scripts
- Measure the cloud costs in an isolated environment
- Run the load tests
- Compute the cost per API call
The idea is to run a series of tests to estimate the costs per transaction empirically. These load tests will be executed in an isolated environment for a period of time. The load tests help simulate the environment at scale. The isolated environment reflects the infrastructure used in a production environment.
The approach measures the costs generated by the load test during the testing period. At the same time, the volume of processed transactions is also recorded. The costs are finally determined by calculated a ratio.
Setting up load tests
To get started, the first step is to identify and set up the load test that corresponds to your business. The load test should be designed to answer one question: given a finite set of resources and a pre-defined period of time, how many transactions (API calls) can be processed by the infrastructure?
The concept of “finite resource” is a critical assumption here. We want the cost of the infrastructure to be capped. For instance, we shouldn’t have an auto-scaling process start.
The load test scripts should be created to simulate requests against the API. The tests should cover all the steps required to fulfill one transaction, the same way your production environment would serve the API call.
We then need to determine how much load to generate against the finite resource. This can be done by either identifying:
- what infrastructure resource saturates (e.g., when the CPU reaches 80% utilization); or
- when a specific SLA drops (e.g., API call response time gets above 100ms).
This can vary from organization to organization. If multiple resources seem to qualify, I would suggest starting with the most restricting one for your business.
Once this is identified, the load test should be configured in a way that generates enough API calls for the resources to reach 70–80% utilization, or for the drop in SLA to be triggered. This exercise might require running a few experiments with some trial and error before identifying the right configuration and being ready to move on.
If you wonder how to build those scripts, have a look at https://locust.io/.
Using AWS cost and usage reports to measure cloud costs
Once the load tests are created and ready to be used, we need to ensure that the costs can be measured. I will be using AWS as the cloud service provider as an example. A similar approach can be used for other cloud providers as well.
The best way to get results as close to production as possible is to set up a separate environment or AWS account that replicates the infrastructure used in your production environment. The load tests will run against that environment.
To measure the costs of the infrastructure, AWS has a service called AWS Cost & Usage Reporting (CUR). However, the finest granularity you can get from AWS CUR reports is one hour (e.g., the AWS generated costs from 9:00 am to 10:00 am). This means that at minimum a load test needs to run for one hour to get relevant results. You also need to consider any ramp-up time needed for your environment to replicate what you have in production.
AWS CUR is a bit more restrictive in the sense that costs are measured from the top of the hour to the next**.** So you need to plan for your load test to run at least for an entire calendar hour.
Running the tests
You have the load tests created. You have the environment set up. You are almost ready to go!
Now, the best approach is to automate the process as much as possible. You want to be able to repeat the tests automatically. For instance, this could mean generating the pipeline to set up the environment, spinning up a container to generate the traffic for the load test, tearing down the test environment, etc. Anything that can make this approach repeatable and scalable. This also means being able to automatically ramp up the load test for it to be ready at the beginning of the hour.
Before running the tests, there is one final step. We need to identify the key metrics to measure and collect data when the test is executed. These metrics are:
- the cost of the AWS infrastructure, this we will get from AWS CUR.
- the number of API calls (transactions) processed during that time; the load test script can record them or you can extract them from your logs.
- the error count, just in case something goes wrong and the data is unusable.
These measures reflect a simple environment. There are additional factors that you might want to consider if your environment is more complex (e.g., additional costs from a DB instance managed outside of AWS).
Now run the tests!
Calculating the cost
The final step is the easiest. We need to do some math using the metrics we identified above and the data we collected during the tests. If all went well, your estimated cost per transaction can be computed by dividing your AWS costs for the hour by the number of transactions processed during that same hour.
In other words, for the measured period the formula is:
Cost per API Call = AWS Costs / Number of API Calls
Wrapping up
I hope you found this read helpful. This method will help you get a good first estimation of your cost per API call. It is however not perfect and works under assumptions. The approach can further be refined by adding more complexity to the model.
Cloud costs are mostly fixed. For a given resource (e.g., an AWS EC2 instance) you will pay the same amount for an hour if you use it or not. Assuming there is no auto-scaling. One way to improve this approach, for instance, would be to take into consideration the impact that auto-scaling has on your costs at scale. Another would be to factor in savings plans.
Feel free to tweak and adjust the method to suit your needs.
Thanks for reading!