SECRET OF CSS

How To Optimize Cloud Cost With Block Storage and Snapshots | by Elvin Lari | Aug, 2022


Various options to keep your high-quality services without the high costs

1*sbtzbtV5Ee5JpjGY5XZ02Q
Designed by vectorjuice on Freepik

Companies that are new to the cloud usually overprovision resources to ensure performance resulting in unnecessary extra costs. Fortunately, several practices can help you reduce your cloud storage costs, such as block storage and snapshots.

Block storage, sometimes referred to as block-level storage, is a technology that is used to store data files on Storage Area Networks (SANs) or cloud-based storage environments. — IBM

Cloud providers like AWS and Digitalocean offer EC2 instances and droplets that have local hard disks. However, all data stored on the disk is lost when an instance shuts down.

Block storage solutions provide persistent storage volumes that can be attached to AWS EC2 instances or Digitalocean droplets. These volumes let you store data beyond the lifetime of a specific instance.

AWS block storage offering is called Elastic Block Store (EBS), which stores data in block volumes. EBS volumes can be used as a root partition for an EC2 instance. Since the volume persists after you delete the instance, it can be used as a backup. EBS volumes are also scalable and can be replicated automatically.

Optimizing EBS volume costs

By optimizing the usage of EBS, you can reduce their costs without affecting the application performance. You can use the following steps to optimize EBS volume costs.

Choosing the right EBS volume kind

AWS offers four types of EBS volumes that you can choose from depending on your budget and the performance characteristics of your application. They include General Purpose SSD volumes, Provisioned IOPS SSD volumes, Throughput Optimized HDD and Cold HDD volumes, and Previous generation Magnetic volumes.

Application architects should provision General Purpose SSD (gp2) volumes instead of Provisioned IOPS SSD (io1) volumes unless the running application is mission-critical and need more than 10000 IOPS or 160 MiB/s of throughput per volume. Administrators should monitor read-write operations of the provisioned volumes periodically.

If the throughput is low, they should downgrade the volume to reduce cost. This operation can be automated using vertical autoscaling by provisioning the volumes with Kubernetes.

Choosing the right EBS volume size

Factors to consider before arriving at the correct size of an EBS volume are IOPS, capacity, and throughput requirement of the application.

Leverage EBS snapshots

An EBS snapshot is a point-in-time “copy” of the data in EBS volume. EBS snapshots are billed at a lower rate than active EBS volumes. When an EBS block has low access volume, you can delete it after copying the information stored in it to an EBS Snapshot.

Delete unattached EBS volumes

Administrators who delete unused EC2 instances might forget to delete the attached EBS volumes. Even though these volumes are not connected to any instance, they would still be charged with AWS. Therefore, regularly checking if all the EBS volumes are being used and deleting unused volumes can lower the AWS monthly bill.

Delete old snapshots

Snapshots can be used for recovery in case of data loss or disaster. However, outdated snapshots of no practical value might still be among the EBS snapshots. The recovery procedures might need only the most recent snapshot from several snapshots backed up. Such EBS snapshot costs can snowball when many are provisioned.

While application architects are configuring the settings to back up snapshots automatically, they should also configure the settings to delete older snapshots periodically. Administrators must set a policy for how many snapshots should be retained per volume.

A snapshot is a way to back up and recover the data on a block storage volume. You can generate a volume snapshot to create a point-in-time backup of the data stored on the volume.

An EBS snapshot, sometimes called an AWS snapshot, is a way to back up and recover the data on an EBS volume. The user can restore the data from the latest snapshot in the event of a disaster. EBS snapshots only save changes from the last snapshot, functioning as incremental backups.

The snapshot subsystem backs up only changed blocks and creates logical links to blocks in the previous snapshots needed to restore the entire volume. Later, if those last snapshots are deleted, only the blocks not linked to any later snapshots are purged.

This provides the advantages of faster snapshots and the ability to purge old snapshots without worrying about subsequent backups that depend on previous backups. However, this limits you to using AWS S3 buckets because the snapshots are not saved like ordinary S3 Objects.

EBS snapshots pros

  1. Easy and flexible backup and recovery: Volumes can be restored to any snapshot.
  2. Cost-efficient. Snapshots take up less storage than EBS volumes and are priced at a lower cost. Using EBS snapshots enable users to save only the last versions, thus reducing redundancy.

EBS snapshots cons

  1. You cannot copy data outside AWS. EBS snapshots can only be stored in AWS S3; hence you cannot back up data outside AWS.
  2. Snapshots are region specific. When saving an EBS snapshot, you specify in which region to save, and you cannot access it from another region. You must copy it to those regions to make it available in additional regions. This limits scaling and incurs additional data transfer costs. It also requires you to pay for persistent storage of the snapshot for each region you copy it to.
  3. Accumulating snapshots end up taking too much storage hence increasing costs.
  4. Slower recovery. Recovery takes longer as compared to other storage options because the system works backward, restoring until the point in time that you select.

AWS EBS snapshots have two price components: data transfer fees and storage costs on Amazon S3, which differ from standard S3 pricing.

Amazon snapshots: Data transfer costs

Whenever a snapshot of an EBS Volume is created, AWS transfers the data intended to be saved in the snapshot from Amazon EC2 to an S3 bucket in an AWS region of your choice.

Below are the current AWS transfer costs, keeping in mind that AWS prices frequently change. Consult the AWS official pricing page for up-to-date pricing and additional details.

AWS S3 transfer costs

Amazon snapshots: Storage costs

AWS standard storage costs

EBS snapshots standard has a flat storage cost of $0.05 per GB-month. This is substantially more expensive than the S3 Standard storage tier, which costs between $0.021–0.023 per GB-month (in the US-East region).

How to cut snapshot costs

From the above analysis, it is clear that having a way to save EBS Volume snapshots in S3 can reduce a significant amount from the monthly AWS bill. You can achieve this by operating your cloud infrastructure in a Kubernetes Cluster.

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. — kubernetes.io

In this article, I would like to introduce you to a Kubernetes-native tool called ‘Stash.’ Stash is a tool for taking backups of your disks and helping with the crucial recovery plan. It is a Restic operator that accelerates the task of backing up and recovering your Kubernetes infrastructure.

Using Stash, you can back up Kubernetes volumes mounted in the following types of workloads:

  • StatefulSet
  • ReplicaSet
  • ReplicationController
  • Deployment
  • DaemonSet

“Stash supports various backends for storing data snapshots, e.g., GCS bucket, AWS S3, Azure Blob Storage, etc. or a Kubernetes persistent volume like HostPath, PersistentVolumeClaim, NFS, etc.”stash.

Stash supports S3 compatible storage services like AWS S3, tebi.io, Minio servers, Rook Object Store, and DigitalOceans Space for a backend.

Using stash, we can create snapshots of EBS volumes provisioned in K8S (Kubernetes) clusters and store them in AWS S3 as regular S3 objects. We can also keep them with any other S3 Object Storage providers. You can access such storage from any AWS region without storing the snapshot in the region. This also helps in making the application more scalable.

You can even use open source S3 providers like rook.io at no cost, depending on your project type. Providers like tebi.io offer a forever free tier (of 25GB storage) and a pay-as-you-go model ($0.02 per additional GB) per month.

You can leverage the above options to help cut your cloud storage costs efficiently without affecting the application’s performance.

Thanks for reading. I hope this was helpful. If you have any questions, feel free to leave a response.



News Credit

%d bloggers like this: