Configuring metricsd to setup a disk alarm

What is MetricsD?

Metricsd is a golang program that gathers metrics from instance an AWS EC2 node and reports these metrics to places such as AWS / CloudWatch. Metrics collected include disk space, cpu activity, memory allocation, Cassandra KPIs. MetricsD is most often run as a systemd process.

Disk Gatherer reports to AWS / CloudWatch, sets alarms or sends emails.

The Disk Gatherer reports disk state information to AWS / CloudWatch, sets alarms in AWS / CloudWatch or sends emails. It leverages the df command.

Continue reading

Notes on Cassandra OS setup and optimizations for deploying in EC2/AWS

Notes on Cassandra OS setup and optimizations for deploying in EC2/AWS

Disk concerns

These are important concepts for developers and DevOps who are responsible for developing Cassandra based applications and services.

Cassandra writes to four areas

• commit logs
• SSTable
• an index file
• a bloom filter

The compaction process of SSTable data makes heavy use of the disk. LeveledCompactionStrategy may need 10 to 20% overhead. SizeTieredCompactionStrategy worse case is 50% overhead needed to perform compaction. Keep this in mind while sizing disks. If you are doing a high-update use case, LeveledCompactionStrategy is the best solution if you want to limit the total disk size used at any point in time and to optimize reads as the row will be spread across less (up to ten times less) SSTables. LeveledCompactionStrategy requires more IO and processing time for compactions. If in doubt, use LeveledCompactionStrategy.

Continue reading

Backup/Recovery with EBS

Understanding what AWS provides for backing up EBS volumes is an important concept for DevOps.

Data safety with EBS - Backup/Recovery (Snapshots)

Amazon EBS allows you to easily backup data. You do this by taking snapshots. Snapshots are point-in-time backups. Data written to an EBS volume can be periodically used to create a snapshot. Snapshots provide incremental backups of your data. Snapshots just saves the blocks that have changed. Only changed blocks since the last snapshot are saved in the new snapshot.

Continue reading

Most bang for your buck with AWS Elastic Block Store (EBS)

Getting the most bang for your buck with AWS Elastic Block Store (EBS)

Understanding what AWS/EC2 provides for provisioning on-demand storage is critical for DevOps. Companies waste tons by over provisioning AWS.

Amazon Elastic Block Store

Amazon Web Services (AWS) provides Amazon Elastic Block Store (Amazon EBS) for EC2 instance storage. EBS is the virtual hard drives and SSDs for your servers running in the cloud. Amazon EBS volumes are automatically replicated, and it is easy to take snapshots of volumes to back them up in a known state. The replication happens within an availability zone (AZ).

Continue reading