We came across a blog that was published on August 29th on AWS Partner Network Blogs titled ‘How to Build Sparse EBS Volumes for Fun and Easy Snapshotting’. A great article that describes how thin provisioned EBS volumes and optimized snapshots can be created.
It involves the following steps:
- Create an additional EBS volume of equivalent size
- DD contents to a file on new EBS volume
- Mount the new file image as a loopback device
- Run fstrim on new image
- Create another EBS volume of the same size
- Run ddpt to the new EBS volume
- Take snapshot which is optimized and uses only used data
Given that EBS doesn’t provide thin provisioned allocation, this is really great! However, it is clear that it takes quite a number of steps. If you want to apply this to a production environment with multiple EBS volumes per instance and a number of instances, clearly it will be a tedious process and not really practical. Complexity can be alleviated a bit by automating the steps, but it will still be very time consuming (you are copying entire data twice) and error prone. It gets complicated fairly quickly as one considers implementing it in a large scale environment. Perhaps the reason why the authors titled it ‘for Fun’!
An easier way – FittedCloud thin provisioned EBS volumes
Fortunately, there is a much easier way to create thin provisioned EBS volumes. FittedCloud, a Dynamic Cloud Resource Optimization solution provider offers an EBS optimizer (built using a patent pending technology) that supports thin provisioned EBS volumes with ZERO user action.
FittedCloud EBS Optimizer Software will monitor your environment and automatically adjust EBS provisioning to match your application usage. Unused disk space is deleted and returned to AWS, reducing your storage and snapshot costs. For example, if you have an EBS volume of 10TB and you are using only 100GB of storage, it will allocate a smaller EBS volume, migrate data and delete the unused 10TB of disk while maintaining the application and file system view of 10TB disk without interrupting applications. You pay for 100GB of storage and snapshots instead of 10TB! As the capacity utilization increases, it will automatically allocate additional storage from AWS to continue to support your application needs. Space optimization is done completely transparently to users and applications. No user action or intervention is needed.
It can optimize existing EBS volumes or create new thin provisioned volumes.
Following example shows how a 100GB thin provisioned volume named fca is created with a 20GB initial allocation using CLI.
FittedCloud Software Version 1.0, Build 1571
(c) 2015-2016, FittedCloud Inc.
All RIghts Reserved
An EBS volume of the specified size is being created. This may take a few mins. Please wait..
FC Volume fca created successfully. Total size 100GB, initial size 20GB
A new block device /dev/fca is created which looks like a 100GB block device, but only 20GB (/dev/xvdz in the screenshot below) of EBS storage is provisioned.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fca 252:0 0 100G 0 disk
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdcz 202:26368 0 20G 0 disk
How does it work?
FittedCloud EBS optimizer uses a patent pending technology to virtualize EBS volumes. It monitors space utilization continuously and based on utilization makes intelligent decisions on if an optimization is needed. If an optimization (scale up or down) is needed, necessary steps are taken to allocate or delete unused disks (hence the need for AWS credentials with volume create/delete/attach/detach privileges). FittedCloud Agent software manages the IO so that virtualized view of volumes is provided to file systems and application.
How would one install FittedCloud?
Installing FittedCloud software only takes a few minutes. Software can be accessed through our SaaS portal. Steps involved are the following:
- Register an account, login, provide AWS credentials
- Download and Install FittedCloud Agent on instances where EBS needs to be managed. Just drop in the software, no configuration is needed. (Or spin up a FittedCloud AMI with pre-installed agent software which we plan to release soon)
- Use WebGUI to monitor cost savings
A more detailed description can be found here.
Please contact us if you need more details or would like a demo!