An Open Source AWS EBS Cost Analyzer

When creating EBS volumes, it’s difficult to predict what will be required of them in the future.  The thought process goes like this: “How many volumes will I need?  What type of volume should I use?  How big should they be?  How many IOPS should I provision?”  Most people will decide that “too much” is better than “not enough” and will over-provision on all fronts, which results in paying for unused resources.  At FittedCloud, we believe that “the right amount” is better than “too much” and “not enough”.

Common costly misconfigurations include selecting the wrong volume type, over-provisioning IOPS in the case of IO1 volumes, over-provisioning volume size, and paying for unattached storage.

The question then becomes, how do you know if you have any of the above issues?  I’m pleased to announce the release of an open-source tool that you can use to find EBS volumes that can be optimized for cost savings.

This tool will analyze all EBS volumes to achieve the following:

  1. Identify unattached EBS volumes.
  2. Identify EBS volumes that can be switched to another type to reduce cost, without impacting performance. For e.g. switching GP2 to ST1 or SC1.  Switching IO1 to GP2.
  3. Identify IOPS reductions based on last 2 weeks IOPS utilization.
  4. Identify possible capacity reduction for GP2, ST1, SC1 or magnetic (estimation only.  Typically capacity is over provisioned at least by a factor of 2).
  5. Total cost savings achieved by all of the above.

Note that root volumes are skipped as we cannot perform optimization on them yet.

You can download the EBS Cost Analyzer here: https://github.com/FittedCloud/ebscostanalyzer

Note: Python packages boto3, pytz, and arrow are needed. Please install them before running this tool. This tool has been tested with python 2.7.

Running EbsCostAnalyzer.py with no arguments or with the -h or –help argument will display its usage:

2017-02-03-screen2

AWS access key and secret key are the only required arguments.  Running the utility will result in human-readable output by default.  The below screenshot captures a single advisory, advising to migrate a GP2 volume to an SC1 volume.  It also displays the monthly cost savings potential from performing the migration.  Finally, a summary is displayed that details the number and types of volumes analyzed, the total number of advisories found, as well as the estimated projected monthly cost savings.  In this case, you can see that there were actually twelve advisories in this run.  The screenshot shown below only displays the last of the twelve advisories.

2017-02-03-screen1

 

Optional arguments:

You can specify a region using the -r or –regions option:

$ python EbsCostAnalyzer.py -a <access key> -s <secret key> -r us-west-2

You can also specify a list of regions as a comma-separated list:

$ python EbsCostAnalyzer.py -a <access key> -s <secret key> -r us-west-2,us-east-1,eu-central-1

You can use the -m or –mean option to use means (averages) of observed values rather than maximum observed values to determine advisories.

Finally, the -j or –json option will output the advisories in JSON format.  The JSON output is of the following form:

{
    “Advisories”: {
        “EBSMotion”: [ ... ],
        “Unattached”: [ ... ]
    }
}
{
    “Summary”: { ... }
}

An example of JSON output:

2017-02-03-json-full

Enjoy!  We hope you will find this utility useful.  All of the advisories identified by this utility can be implemented using FittedCloud EBSMotion feature without stopping applications or instances.  Please share your comments with us at support@fittedcloud.com.

Other Related Tools/Blogs

FittedCloud Cloud Cost Optimization Solutions
FittedCloud offers machine learning based cloud cost optimization solutions that help customers reduce AWS spend significantly. Our current solutions include machine learning driven actionable advisories with click through actions for EC2, EBS, RDS, DynamoDB, ElastiCache, ElasticSearch, AutoScale, Lambda, etc. and full/lights out automation for EC2, EBS, DynamoDB and RDS. Our solution typically can save customers up to 50% of their cost on AWS. For more details, please visit https://www.fittedcloud.com/.