Spark Tutorial: Spark Streaming with Kafka and MLib

In this part of Spark’s tutorial (part 3), we will introduce two important components of Spark’s Ecosystem: Spark Streaming and MLlib.

Display - Edit

Spark Streaming

By Fadi Maalouli and R.H.

Spark Streaming is a real-time processing tool, that has a high level API, is fault tolerant, and is easy to integrate with SQL DataFrames and GraphX.

On a high level Spark Streaming works by running receivers that receive data from for example S3, Cassandra, Kafka etc… and it divides these data into blocks, then pushes these blocks into Spark, then Spark will work with these blocks of data as RDDs, from here you get your results. The following diagram will demonstrate the process:

Continue reading

Setting up a Cassandra cluster with SSL for client and cluster transports for DevOps

Setting up client and cluster SSL transport for a Cassandra cluster

This articles is a Cassandra tutorial on Cassandra setup for SSL and CQL clients, as well as installing Cassandra with SSL configured on a series of Linux servers.

Cassandra allows you to secure the client transport (CQL) as well as the cluster transport (storage transport).

SSL/TLS have some overhead. This is especially true in the JVM world which is not as performant for handling SSL/TLS unless you are using Netty/OpenSSl integration.

Continue reading

Setting up a Cassandra cluster with cassandra image and cassandra cloud project with Vagrant for DevOps

The cassandra-image project creates CentOS Cassandra Database images for docker, virtualbox/vagrant and AWS/EC2 using best practices for Cassandra OS setup. It is nice to use vagrant and/or docker for local development. At this time it is hard to develop systemd services using Docker so we use Vagrant. Since we do a lot of that, we like to use Vagrant.

Vagrant is important for developers and DevOps not to mention Cassandra DBAs.

The cassandra-image project packages systemd utilities

Continue reading