How to reduce AWS costs using machine learning driven EBS IOPS provisioning

Elastic Block Store (EBS) is an AWS service providing raw block-level storage volume that can be attached to Amazon EC2 instances. There are two major categories: SSD-backed storage and HDD-backed storage. The performance of SSD-backed storage for transactional workloads such as databases and boot volumes depends primarily on Input/output Operations Per Second (IOPS).

While Amazon charges by provisioned IOPS-month, not the actual usage, many customers still tend to provision large IOPS just to prevent under-provisioning. As a result, customers will often end up over-provisioning IOPS and therefore waste money on the underutilized IOPS.

This blog presents a machine learning approach for EBS IOPS optimization. It accurately predicts a customer’s consumed IOPS and changes provisioned IOPS accordingly. This allows customers to save money by only paying for IOPS they consume.

EBS IOPS optimization

The machine learning module we built for EBS IOPS optimization consists of two major components:

  • Prediction of consumed IOPS
  • Optimization of provisioned IOPS

For the first one, we aim to predict future consumed IOPS from historical data. This is a time series prediction problem. We have found that the time series of consumed IOPS has similar patterns to the time series of consumed read/write capacity units in DynamoDB. Therefore, the same time series model that we use for DynamoDB optimization is used to predict consumed IOPS.

With an accurate predictive model, the next challenge is to change provisioned IOPS. We need to decide when to make the change and how many IOPS to provision. There are two constraints:

The constraints imply that we cannot change IOPS provisioning anytime, and the change may not take effect immediately. To help customers save the most money, we developed an optimization algorithm that minimizes the total provisioned IOPS given the above constraints while satisfying the consumed IOPS. Note that optimizing provisioned IOPS is very different from optimizing provisioned capacity in DynamoDB because they have different constraints.

The following example shows performance of the machine learning module for EBS IOPS optimization. It accurately predicts consumed IOPS. Because of the constraints mentioned above, the red curve is the best IOPS provisioning we can achieve.

Other EBS related Blogs

About FittedCloud

FittedCloud is the industry’s leading public cloud resource optimization solution. It features machine learning algorithms that continuously analyze resource utilization and identify opportunities to reduce monthly recurring cloud infrastructure costs. Automated provisioning can adjust cloud resources according to load patterns, user-configured policies and other parameters. FittedCloud’s patented solution reduces costs up to 50% while eliminating complex manual provisioning processes and the risk of configuration errors. For more details please visit https://www.fittedcloud.com