AWS EBS Optimization Use Case: Migrating from LVM
Traditionally a UNIX file system capacity is rigid, usually determined at its creation time, and not easily stretchable. This has largely improved since as most modern Linux file systems can be grown on the fly. However, the practice of resizing a file system is still uncommon, likely due to the rigidity of the underlying backing block devices.
Even with the advent of technologies such as the virtual block devices and the Logical Volume Manager (LVM), the process of growing a file system is still laborious and dangerous. Data could be lost if mistakes are made. Administrators have to monitor the file system space usage and be proficient in managing the LVM. Furthermore, shrinking file system size can only be done offline.
At work, I’ve been managing a build server that automatically pulls source codes from Github when new codes are pushed by developers, then the code gets compiled and built into installable packages. As the project grows, more components are added and more platforms are supported, the data has outgrown the file system capacity. To mitigate the issue, the file system’s underlying backing device have evolved from a simple fix-sized block device, to a logical volume using the LVM so new blocks can be added on the fly. Still, I have to keep a keen eye on the file system’s usage and be prudent about provisioning the storage space in order to limit both upfront and ongoing costs. This is especially true in a cloud environment since the cost is proportional to the capacity provisioned, for as long as the resources are allocated, comparing to a one time purchase cost of traditional physical disks. It would really be nice if this process can be automated. The FittedCloud Storage Optimization Agent does exactly that, and even better without the LVM (reducing administration complexities). Better yet, it trims un-used cloud disk space while file system is online.
This project at work initially needed only about 10GB of space. As the project grew I expected the need would grow by 20GB or so a year, and I probably wouldn’t need to worry about the need for more than 5 years. So a maximum of 100GB was what I needed. But I certainly didn’t want to pay the 100GB cost from day one. I would rather add disk space in 20GB increments and pay for what I would actually use as it went. So here was what I did with the help of the FittedCloud Storage optimization Agent.
It’s worth mentioning if this were a new project I would have thin-provisioned a 100GB disk with 20GB initial size using the FittedCloud GUI. But since this was an existing project with roughly 13GB worth of data in file system, I just installed the Agent and let it optimize for me. After a few minutes, the FittedCloud Agent determined a 20GB disk would be sufficient to hold the 13GB data, it allocated a new 20GB disk and migrated existing data from the 100GB disk to the new 20GB Disk. Once the data had been migrated successfully, it detached and deleted the 100GB disk. All this was done on the fly without un-mounting the file system or shutting down the application. The file system still thinks it has 100GB capacity. The FittedCloud Agent continues to monitor the file system space usage, and provision storage appropriately such as adding or deleting storage space in 20GB increments.
For me, I no longer need to worry about the file system space usage, or perform any of the LVM and file system growth procedures. Better yet, I only pay for the 20GB disk space I am actually using instead of the 100GB.
Other EBS related Blogs
- An Open Source Tool to analyze wasted EBS capacity in your AWS environment
- How to reduce the size of EBS Windows boot volume in AWS?
- How to shrink EBS root volumes in AWS – with just one click
- How to reduce AWS costs using machine learning driven EBS IOPS provisioning
- AWS Elastic Volumes and FittedCloud EBS Optimizer
- FittedCloud AWS EBS Optimizer for Docker Containers
- An Open Source AWS EBS Cost Analyzer
- Is it possible to use EBS gp2 instead of io1, achieve same performance and save 50%?
- How to optimize AWS EBS using LVM and reduce cost
- How to create thin provisioned AWS EBS volumes and save a ton!
- AWS EBS Performance – Confused?
FittedCloud is the industry’s leading public cloud resource optimization solution. It features machine learning algorithms that continuously analyze resource utilization and identify opportunities to reduce monthly recurring cloud infrastructure costs. Automated provisioning can adjust cloud resources according to load patterns, user-configured policies and other parameters. FittedCloud’s patented solution reduces costs up to 50% while eliminating complex manual provisioning processes and the risk of configuration errors. For more details, please visit https://www.fittedcloud.com