What is Hadoop?

What is Hadoop? What are the basic components that make up a Hadoop (Big Data) system?

At its most simplest terms, these are the components that a Hadoop ecosystem contains.

Hadoop Distributed File System (HDFS)

Java based, distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster. Store and process vast quantities of data in a storage layer that scales linearly. It is designed to run across low-cost commodity hardware.

Hadoop YARN

A resource-management platform responsible for managing computing resources in clusters and using them for scheduling of users’ applications. It provides the resource management and a pluggable architecture for enabling a wide variety of data access methods to operate on data stored in Hadoop with predictable performance and service levels.

Hadoop MapReduce

A programming model for large scale data processing.

Hadoop Common 

Libraries and utilities needed by other Hadoop modules.

Reference:

https://github.com/hortonworks/tutorials/blob/hdp-2.5/tutorials/hortonworks/hello-hdp-an-introduction-to-hadoop/hello-hdp-section-2.md#13-apache-hadoop

Course Review - Apache Spark Streaming with Scala

I recently completed a course on Apache Spark Streaming with Scala. It was a great learning experience and gave me the skills necessary to complete a complex prototype in Big Data. The prototype involved reading a stream of information and processing them with Apache Spark framework.

Where is this course?

    This is a course on Udemy. https://www.udemy.com/taming-big-data-with-spark-streaming-hands-on/learn/v4/overview 

What does it cover?

   This course covers Apache Spark Streaming module. While the Spark framework is foundation for Big Data processing, its Streaming module is used for processing streaming information. This Streaming module supports both Python and Scala. This course is based on Scala language. There is an introductory section that explains Scala basics. The rest of the Course explains how Spark Streaming module works. A bonus feature is the lesson explaining Apache Kafka and Spark Streaming integration.

   The hands-on nature of the courses are great. Each course does not exceed 15 minutes' which makes learning easier even if you have short time every day.

What is the cost?

   The cost varies. It is not free.

How long is this course?

   The course duration is 6 hours. I took about 30 hours (learning 1 hour everyday) to complete the course.

What are the prerequisites?

   The learner must be familiar workign with Apache Spark and Scala. It would be ideal for the learner to have at least 5 years of programming experience and database knowledge before starting this course. Basic familiarity with UNIX commands or other CLI is needed. Basic knowledge in Java/Scala programming would be helpful, thought there are a few introductory lessons for Scala.

How will it benefit the learner?

  You will have enough knowledge to become an advanced Spark Scala developer in Big Data projects.

Course Review - Apache Spark with Scala

I recently completed a course on Apache Spark with Scala. It was a wonderful learning experience and gave me the skills necessary to complete a prototype in Big Data.

Where is this course?

    This is a course on Udemy. https://www.udemy.com/apache-spark-with-scala-hands-on-with-big-data/

What does it cover?

   This course covers a deep range of topics related to Apache Spark. While the Spark framework supports several languages, this course is based on Scala language. There is an introductory section that explains Scala basics, followed by Spark libraries. It just gives a brief introduction to Spark architecture.

   The hands-on nature of the courses are great. Each course does not exceed 15 minutes' which makes learning easier even if you have short time every day.

What is the cost?

   The cost varies. It is not free.

How long is this course?

   The course duration is 7.5 hours. I took about 30 hours (learning 1 hour everyday) to complete the course.

What are the prerequisites?

   It would be ideal for the learner to have at least 4 years of programming experience and database knowledge before starting this course. Basic familiarity with UNIX commands or other CLI would help. Basic knowledge in Java/Scala programming would be helpful, thought there are a few introductory lessons for Scala.

How will it benefit the learner?

  You will have enough knowledge to become an intermediate level Spark Scala developer in Big Data projects.

Course review - MongoDB - Udemy

I completed a course about MongoDB ("MongoDB Essentials - Understand the Basics of MongoDB").  This course covered the basics of MongoDB in a very simple and easy manner.

Where is this course?

This is a Udemy course. https://www.udemy.com/mongodb-essentials/learn/v4/overview

What does it cover?

Introduction to MongoDB.

What is the cost?

It is free.

How long is this course?

About 35 minutes. It would take about 2 hours to complete, if the learner runs every step in the tutorial.

What are the prerequisites?

You will need a computer with admin privileges and internet connection.

How will it benefit the learner?

After completing this course, the learner will become equipped to work on entry level tasks in MongoDB.

Course Review - Github Introduction - Udemy

I completed a course about Git and Github.  This course was brilliant in its simplicity and effectiveness in teaching about Git. The course focuses more on practical usage than architecture, which will help beginners to learn Git easily.

Where is this course?

This is a Udemy course. https://www.udemy.com/short-and-sweet-get-started-with-git-and-github-right-now/learn/v4/overview

What does it cover?

Basics of Git and Github.

What is the cost?

It is free.

How long is this course?

About 35 minutes. It would take about 2 hours to complete, if the learner practices every step from the tutorial.

What are the prerequisites?

You will need a computer with admin privileges and internet connection.

How will it benefit the learner?

The learner can become a beginner level Git user and can contribute to software projects.

Getting started with Hortonworks Data Platform (HDP) sandbox in Amazon Web Services (AWS)

A convenient way to start working on Big Data concepts is to use the Hortpnworks (HDP) Sandbox. I am not aware of a similar sandbox in other Big Data vendors. This sandbox encapsulates all the various components of the Big data ecosystem in a single VM, so it becomes easy to install. This Sandbox is available in 3 compact formats - VirtualBox, VMware and Docker.

I started working with VirtualBox, since that is (in my opinion) a compact and simple container.
But I faced issues because the Sandbox is very large (>10 GB) and it needs at least 8GB RAM. The installation just froze and I could not proceed.

I then decided to try it out in the AWS cloud, since I could spin up more powerful hosts machines. This worked. :)


In this post, I will explain how to get started.

What do we need to get started?

1. AWS account
2. Basic undesstanding of AWS concepts (I have glossed over the AWS instance setup, to keep this post compact)
3. An EC2 instance with minimum 128 GB disk storage and 16 GB RAM. (I used an instance type of m4.2xlarge). The larger, the better.

How much time do we need?

       The Hortonworks portal claims that you need just 15 minutes (!), but lets assume 1 hour.

Steps

     Step 1 - Launch the EC2 instance with the Amazon Linux AMI. Ensure that the security group is configured to allow the ports 8080, 888 and 22 to be accessible.

     Step 2 - Install Docker in AWS. This post from AWS will help -  http://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html

     Step 3 - Modify the Docker config file "/etc/sysconfig/docker-storage" to add the line:            

    DOCKER_STORAGE_OPTIONS="--storage-opt dm.basesize=20G"

          This step is needed to ensure that Docker is able to import the Sandbox. This is because by default Docker containers can have a maximum size of 10 GB and the Hortonworks Sandbox is about 13 GB.  Restart Docker ("sudo ").

     Step 4 - Download, unzip, load and start the Docker sandbox image from Hortonworks. This link has the details - http://hortonworks.com/hadoop-tutorial/hortonworks-sandbox-guide/#section_4


     While the Hortonworks guide is well written, it misses out the changes needed to increase the device memory.

      Please try out this installation and post your comments. Thanks for reading.

Common Issues

     If  Step 3 above was not run, the you will see the following error when loading the Docker image.

# docker load < HDP_2.5_docker.tar
b1b065555b8a: Loading layer 202.2 MB/202.2 MB
3901568415a3: Loading layer 10.37 GB/13.85 GB
ApplyLayer exit status 1 stdout:  stderr: write /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre/lib/amd64/server/libjvm.so: no space left on device

References

https://community.hortonworks.com/questions/64835/installing-hdp-sandbox-on-docker-for-aws-gives-no.html#answer-64852

Docker commands - Reference

Useful Docker commands

This blog post is a reference for Docker commands.

These commands can be got by typing "docker" or "man docker" in the CLI after successful installation of Docker.

Basic commands

    info  - Display system-wide information
    version -  Show the Docker version information
    start -  Start one or more stopped containers
    stop -  Stop a running container

    pull -  Pull an image or a repository from a registry
    push - Push an image or a repository to a registry

    cp -  Copy files/folders between a container and the local filesystem
    create -   Create a new container
    diff - Inspect changes on a container's filesystem
    restart -  Restart a container

Other commands

    attach  -  Attach to a running container
    build  -    Build an image from a Dockerfile
    commit -  Create a new image from a container's changes
    events -    Get real time events from the server
    exec  -    Run a command in a running container
    export -   Export a container's filesystem as a tar archive
    history -  Show the history of an image
    images  -  List images
    import  -  Import the contents from a tarball to create a filesystem image
    inspect -   Return low-level information on a container or image
    kill   -   Kill a running container
    load -   Load an image from a tar archive or STDIN
    login -  Log in to a Docker registry
    logout - Log out from a Docker registry
    logs - Fetch the logs of a container
    network -  Manage Docker networks
    pause -   Pause all processes within a container
    port -  List port mappings or a specific mapping for the CONTAINER
    ps -  List containers
    rename -   Rename a container
    rm -      Remove one or more containers
    rmi -     Remove one or more images
    run -     Run a command in a new container
    save -    Save one or more images to a tar archive
    search - Search the Docker Hub for images
    stats -   Display a live stream of container(s) resource usage statistics
    tag -     Tag an image into a repository
    top -     Display the running processes of a container
    unpause - Unpause all processes within a container
    update -  Update configuration of one or more containers
    volume -  Manage Docker volumes
    wait -    Block until a container stops, then print its exit code