Elastic Block Store (EBS) is an AWS service providing raw block-level storage volume that can be attached to Amazon EC2 instances. There are two major categories: SSD-backed storage and HDD-backed storage. The performance of SSD-backed storage for transactional workloads such as databases and boot volumes depends primarily on Input/output Operations Per Second (IOPS).
While Amazon charges by provisioned IOPS-month, not the actual usage, many customers still tend to provision large IOPS just to prevent under-provisioning. As a result, customers will often end up over-provisioning IOPS and therefore waste money on the underutilized IOPS.
This blog presents a machine learning approach for EBS IOPS optimization. It accurately predicts a customer’s consumed IOPS and changes provisioned IOPS accordingly. This allows customers to save money by only paying for IOPS they consume.
EBS IOPS optimization
The machine learning module we built for EBS IOPS optimization consists of two major components:
- Prediction of consumed IOPS
- Optimization of provisioned IOPS
For the first one, we aim to predict future consumed IOPS from historical data. This is a time series prediction problem. We have found that the time series of consumed IOPS has similar patterns to the time series of consumed read/write capacity units in DynamoDB. Therefore, the same time series model that we use for DynamoDB optimization is used to predict consumed IOPS.
With an accurate predictive model, the next challenge is to change provisioned IOPS. We need to decide when to make the change and how many IOPS to provision. There are two constraints:
- IOPS changes can be done only once every 6 hours. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/considerations.html)
- Changing IOPS takes time. A fully used 1 TiB volume takes about 6 hours to migrate to a new performance configuration. (https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-expand-volume.html)
The constraints imply that we cannot change IOPS provisioning anytime, and the change may not take effect immediately. To help customers save the most money, we developed an optimization algorithm that minimizes the total provisioned IOPS given the above constraints while satisfying the consumed IOPS. Note that optimizing provisioned IOPS is very different from optimizing provisioned capacity in DynamoDB because they have different constraints.
The following example shows performance of the machine learning module for EBS IOPS optimization. It accurately predicts consumed IOPS. Because of the constraints mentioned above, the red curve is the best IOPS provisioning we can achieve.
Click here to learn more about FittedCloud’s EBS optimization solution and start a free trial.