Elastic Block Store (EBS) is an AWS service providing raw block-level storage volume that can be attached to Amazon EC2 instances. There are two major categories: SSD-backed storage and HDD-backed storage. The performance of SSD-backed storage for transactional workloads such as databases and boot volumes depends primarily on Input/output Operations Per Second (IOPS).
While Amazon charges by provisioned IOPS-month, not the actual usage, many customers still tend to provision large IOPS just to prevent under-provisioning. As a result, customers will often end up over-provisioning IOPS and therefore waste money on the underutilized IOPS.
This blog presents a machine learning approach for EBS IOPS optimization. It accurately predicts a customer’s consumed IOPS and changes provisioned IOPS accordingly. This allows customers to save money by only paying for IOPS they consume.
EBS IOPS optimization
The machine learning module we built for EBS IOPS optimization consists of two major components:
- Prediction of consumed IOPS
- Optimization of provisioned IOPS
For the first one, we aim to predict future consumed IOPS from historical data. This is a time series prediction problem. We have found that the time series of consumed IOPS has similar patterns to the time series of consumed read/write capacity units in DynamoDB. Therefore, the same time series model that we use for DynamoDB optimization is used to predict consumed IOPS.
With an accurate predictive model, the next challenge is to change provisioned IOPS. We need to decide when to make the change and how many IOPS to provision. There are two constraints:
- IOPS changes can be done only once every 6 hours. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/considerations.html)
- Changing IOPS takes time. A fully used 1 TiB volume takes about 6 hours to migrate to a new performance configuration. (https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-expand-volume.html)
The constraints imply that we cannot change IOPS provisioning anytime, and the change may not take effect immediately. To help customers save the most money, we developed an optimization algorithm that minimizes the total provisioned IOPS given the above constraints while satisfying the consumed IOPS. Note that optimizing provisioned IOPS is very different from optimizing provisioned capacity in DynamoDB because they have different constraints.
The following example shows performance of the machine learning module for EBS IOPS optimization. It accurately predicts consumed IOPS. Because of the constraints mentioned above, the red curve is the best IOPS provisioning we can achieve.
Other EBS related Blogs
- An Open Source Tool to analyze wasted EBS capacity in your AWS environment
- How to reduce the size of EBS Windows boot volume in AWS?
- How to shrink EBS root volumes in AWS – with just one click
- AWS Elastic Volumes and FittedCloud EBS Optimizer
- FittedCloud AWS EBS Optimizer for Docker Containers
- An Open Source AWS EBS Cost Analyzer
- Is it possible to use EBS gp2 instead of io1, achieve same performance and save 50%?
- How to optimize AWS EBS using LVM and reduce cost
- How to create thin provisioned AWS EBS volumes and save a ton!
- AWS EBS Performance – Confused?
FittedCloud is the industry’s leading public cloud resource optimization solution. It features machine learning algorithms that continuously analyze resource utilization and identify opportunities to reduce monthly recurring cloud infrastructure costs. Automated provisioning can adjust cloud resources according to load patterns, user-configured policies and other parameters. FittedCloud’s patented solution reduces costs up to 50% while eliminating complex manual provisioning processes and the risk of configuration errors. For more details please visit https://www.fittedcloud.com