An Open Source Tool to analyze wasted EBS capacity in your AWS environment

While AWS CloudWatch provides many useful storage statistics by default, such as the number of bytes written or read during a period of time, reads/writes per second, etc. a common metric customers want to know about is disk utilization. How much of EBS provisioned capacity is unused, resulting in wasted spending, is a question asked often.

As it’s very difficult to know how much storage an application will need over its lifetime, most people tend to over-provision their storage as an insurance policy.  In the AWS cloud, however, you pay for what you provision, not for what you actually use.  While we at FittedCloud solve this problem with our EBS Optimizer (which automatically and transparently provisions storage capacity for you) you may just want to simply know how much EBS space you are actually using.  Having this knowledge at your fingertips is helpful for cloud cost monitoring and cloud cost optimization.

Current approaches to determining your disk utilization involve running scripts on each instance that submit custom metrics to CloudWatch. This process involves:

  • Deploying scripts on each of your instances
  • Submitting custom metrics to CloudWatch on each instance
  • Configuring cron jobs to run the script periodically on each instance

Alternatively, you could use ssh to run commands remotely on Linux instances, however this approach would require juggling around key files and installing ssh servers on Windows instances.

At FittedCloud, we wanted a simpler, generic solution.

I’m pleased to announce the release of an open-source tool that you can use to fetch information about your disk utilization.  The script requires that Amazon’s AWS System Management Agent (SSM) is installed on your EC2 instances.  Among other things, the SSM Agent allows you to run shell commands remotely on your EC2 instances.

To use this tool:

1. Install AWS SSM Agent on the instances you wish to query.

If your instances are running Windows or Amazon Linux, SSM Agent is installed automatically so you can skip this step.  Otherwise, the instructions for installing the SSM Agent can be found here: https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html.

2. Enable IAM permission to run commands through SSM on your instances.

This can be done in the EC2 Management Console by selecting an EC2 Instance, under the Actions dropdown, select Instance Settings > Attach/Replace IAM Role, add the RunCommand role and click Apply.

3. Enable IAM permission for user to use SSM through the IAM Management Console.

Add the Actions SendCommand and GetCommandInvocations for the SSM Service for an individual user, or alternatively, create a new Role with SSM permissions (e.g., AmazonSSMFullAccess) and assign it to a user.

At the end of the day, using our new open-source tool means that you don’t have to write or install any third-party scripts on your instances or configure cron jobs to submit custom metric information to CloudWatch.  Furthermore, you don’t have to write any additional scripts to query CloudWatch for the new metrics.

You can download the open-source utility here: https://github.com/FittedCloud/diskanalyzer

Running diskanalyzer.py with the –help option will display its usage:

To simply query all of your instances to get their disk utilization, run:

$ python2 diskanalyzer.py -a <access-key> -s <secret-key>

Optionally, you may specify a region or comma-separated list of regions to limit the queries.  The output looks as follows:

You may also output in json format with the -j option, which includes additional fields such as the timestamp.

Other EBS related Blogs

How to reduce the size of EBS Windows boot volume in AWS?

How to shrink EBS root volumes in AWS – with just one click

How to reduce AWS costs using machine learning driven EBS IOPS provisioning

AWS Elastic Volumes and FittedCloud EBS Optimizer

FittedCloud AWS EBS Optimizer for Docker Containers

An Open Source AWS EBS Cost Analyzer

Is it possible to use EBS gp2 instead of io1, achieve same performance and save 50%?

How to optimize AWS EBS using LVM and reduce cost

How to create thin provisioned AWS EBS volumes and save a ton!

AWS EBS Performance – Confused?

 

FittedCloud Cloud Cost Optimization Solutions
FittedCloud offers machine learning based cloud cost optimization solutions that help customers reduce AWS spend significantly. Our current solution includes machine learning driven actionable advisories with click through actions for EC2, EBS, RDS, DynamoDB, ElastiCache, ElasticSearch, AutoScale, Lambda, etc. and full/lights out automation for EC2, EBS, DynamoDB and RDS. Our solution typically can save customers up to 50% of their cost on AWS. For more details, please visit https://www.fittedcloud.com/.