Real-Time Anomaly Detection – Protect yourself from malicious spend attacks on clouds

As stated in our previous blogs, FittedCloud applies many kinds of AI-driven algorithms to analyze contextual data and behaviors in your Cloud environment to identify opportunities to optimize system’s performance, increase resource utilization, and reduce your total cost. In this blog, I highlight the powerful machine learning algorithms deployed in FittedCloud to process, analyze and detect anomalies in vast amounts of your sequential data. With these learning algorithms, FittedCloud can:

  • Detect unexpected data and behaviors in a real time manner
  • Provide automatic monitor of your Cloud environment (24/7)
  • Support contextual diagnosis of anomaly and offer advisory of actions
  • Enhance safety, increase resource utilization and cost efficiency, and improve customer satisfaction.

 

What is an anomaly in your Cloud?

An anomaly refers to an out-of-ordinary or unexpected data or event whose pattern does not conform to the normal behavior. Anomaly detection refers to the process of finding any anomaly in your Cloud data, the data we monitor and collect from your Cloud environment. It is critical to detect anomaly and analyze its root cause when you manage your Cloud account, as an anomaly may be relevant to a specific safety issue (e.g., attackers launch malicious attacks) or a cost saving opportunity (e.g., any big cost jump is detected and necessary advisory is provided to save your money).

(A)

(B)

Fig. 1: Two examples of anomalies in AWS Cloud, (A) an anomalous cost increase in April, and (B) an anomalous resource utilization increase on fourth Tuesday.

 

State-of-the-Art Anomaly Detection Algorithms

The study of anomaly detection has been studied in the statistics community as early as 1887 by Edgeworth who uses the law of frequency to detect “discordant observations”. Lately, a large number of anomaly detection algorithms have been developed in other communities, such as machine learning and data mining groups. While statisticians are interested in those model-based approaches where the data are assumed to follow some distribution models, researchers in machine learning communities now focus on those data-driven techniques where normal and abnormal patterns are discovered from the data itself.

Typically, there are two types of anomalies: point anomaly and contextual anomaly.

  • Point anomalies are those individual data instances that do not conform to the patterns of the rest of data. It is the simplest type of anomaly and widely studied by researchers in the past decades.
  • Contextual anomalies are those anomalous data instances within a specific context. A data instance might be considered as an anomaly in one context, but not in another different context.

In a Cloud environment, contextual anomalies are of great interest, as the data (e.g., resource utilization, total cost, etc.) is time-series data, sometimes spatial-temporal data (when the region of the data center where your application runs is considered). The time and location are the attributes, called contextual attributes, which determine the position of a data instance in the whole data sequence, thus specify the structure of the whole data set.

In FittedCloud, we deploy a variety of algorithms to detect contextual anomalies, including distribution-based approaches, distance-based approaches, and learning-based approaches.

  • Distribution-based approaches fit input data to a statistical model, in which normal data have high probabilities, and anomalies have low probabilities. In one of our previous blogs (https://www.fittedcloud.com/blog/anomaly-detection-in-dynamic-cloud-resource-optimization/), we discussed how to apply distribution-based approaches to detect an anomaly in Dynamic Cloud Resource Optimization.
  • Distance-based approaches use nearest neighbor analysis to examine the underlying structure of normal data, motivated by the fact that normal data appear in dense neighborhoods, while anomalies appear far from normal data or their closest neighbors. Various distance or similarity measures are examined and recent advanced approaches (e.g., relative density with kernels, connectivity outlier factors, mutual nearest neighbors, etc.) are used in FittedCloud.
  • Learning-based approaches use regression models to predict contextual output using historical data. Since the regression models capture patterns of historical data, an actual value which is far from the predicted value is considered as an out-of-ordinary data or an anomaly. As stated in our previousMachine Learning blogs, several powerful regression models are deployed in our FittedCloud products, and they can also be used here for anomaly detection.

As one can imagine, different approaches have their advantages and disadvantages. For example, the distribution-based approaches are pretty robust if the assumptions of underlying distribution models hold true. However, these approaches heavily rely on this assumption which might not be true, particularly for high-dimensional data sets in which the assumption is hard to justify. The distance-based approaches using nearest neighbors do not make any assumptions about data distribution, and they are data-driven. Also, this kind of approach can easily capture the local data structure, which is very suitable for contextual analysis. However, the computational cost for a new testing data is usually high. To improve its efficiency, fast nearest neighbor searching approaches have been published in the literature.

FittedCloud deploys all three types of anomaly detection techniques. An ensemble mechanism is used to combine them together to boost the performance, such that FittedCloud can automatically monitor your Cloud environment and detect all possible anomalous activities and pressing issues.

To learn more, read our new press release: FittedCloud Adds Anomaly Detection to AWS Cost Management Solutions.

Other Anomaly detection related Blogs

About FittedCloud

FittedCloud is the industry’s leading public cloud resource optimization solution. It features machine learning algorithms that continuously analyze resource utilization and identify opportunities to reduce monthly recurring cloud infrastructure costs. Automated provisioning can adjust cloud resources according to load patterns, user-configured policies and other parameters. FittedCloud’s patented solution reduces costs up to 50% while eliminating complex manual provisioning processes and the risk of configuration errors. For more details please visit https://www.fittedcloud.com