1. Ray Burgemeestre
  2. hadoop-docker

Overview

HTTPS SSH

h1. Requirements

h2. Get docker 1.8.x from:

h2. Known issues:

  • I've had the problem with aufs backend for Docker not working properly, in that case modify the Docker Daemon settings to use a different backend like devicemapper or btrfs.

h1. Usage

Available scripts:

In cloudera-folder:

  • bash build_all.sh - build all cloudera docker images
  • bash start_all.sh - start all cloudera docker images

In hortonworks-folder:

  • bash build_all.sh - build all hortonworks docker images
  • bash start_all.sh - start all hortonworks docker images

use docker ps to see if everything started.

h2. More info

Cloudera Dockerfile's are based on instructions from http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_ig_install_path_b.html Hortonworks Dockerfile's are based on the manuals too and manual investigation.

h1. TODO

You should end up with something like this:

root@FIREFLY:/projects/docker-hadoop# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 307d148ee120 rayburgemeestre/hortonworks-slave "/usr/sbin/init" 3 minutes ago Up 3 minutes node105 0aa7fbc7683c rayburgemeestre/hortonworks-slave "/usr/sbin/init" 3 minutes ago Up 3 minutes node104 21a3de49e999 rayburgemeestre/hortonworks-slave "/usr/sbin/init" 3 minutes ago Up 3 minutes node103 3565b18caee2 rayburgemeestre/hortonworks-slave "/usr/sbin/init" 3 minutes ago Up 3 minutes 0.0.0.0:8088->8088/tcp node102 f6b6039d4d90 rayburgemeestre/hortonworks-slave "/usr/sbin/init" 3 minutes ago Up 3 minutes 0.0.0.0:16010->16010/tcp, 0.0.0.0:50070->50070/tcp node101 10a5ade72a83 rayburgemeestre/hortonworks-master "/usr/sbin/init" 4 minutes ago Up 4 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:8440-8441->8440-8441/tcp horton 9953b53a449d rayburgemeestre/cloudera-slave "/usr/sbin/init" 4 minutes ago Up 4 minutes node005 ec8069001ef0 rayburgemeestre/cloudera-slave "/usr/sbin/init" 4 minutes ago Up 4 minutes node004 f4e8351b713d rayburgemeestre/cloudera-slave "/usr/sbin/init" 4 minutes ago Up 4 minutes node003 f19d692df848 rayburgemeestre/cloudera-slave "/usr/sbin/init" 4 minutes ago Up 4 minutes node002 00607743ae67 rayburgemeestre/cloudera-slave "/usr/sbin/init" 4 minutes ago Up 4 minutes 0.0.0.0:8888->8888/tcp node001 265a2e2a10fc rayburgemeestre/cloudera-master "/usr/sbin/init" 5 minutes ago Up 5 minutes 0.0.0.0:7180->7180/tcp cloudera

Hosts file inside the cloudera master should look like this:

172.17.0.48 cloudera 172.17.0.48 cloudera.bridge 172.17.0.49 node001 172.17.0.49 node001.bridge 172.17.0.50 node002 172.17.0.50 node002.bridge 172.17.0.51 node003 172.17.0.51 node003.bridge 172.17.0.52 node004 172.17.0.52 node004.bridge 172.17.0.53 node005 172.17.0.53 node005.bridge 172.17.0.54 horton 172.17.0.54 horton.bridge 172.17.0.55 node101 172.17.0.55 node101.bridge 172.17.0.56 node102 172.17.0.56 node102.bridge 172.17.0.57 node103 172.17.0.57 node103.bridge 172.17.0.58 node104 172.17.0.58 node104.bridge 172.17.0.59 node105 172.17.0.59 node105.bridge

If you have bridge first cloudera manager will get confused. Because we create the instances with 5 seconds delay in between the hosts file should end up like the above output.