Big data learning and development platform

1. Hadoop

1. Purpose

- This documents describes how to set up and configure a single node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

2. Tools Used

The tool to be used is: Cloudera.

- Cloudera is revolutionizing enterprise data management by offering the first unified Platform for Bigdata: The Enterprise Data Hub. Cloudera offers enterprises one place to store, process, and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data.

- The cloudera was founded in 2008, and is currently, the leading provider and supporter of Apache Hadoop for the enterprise. Cloudera also offers software for business critical data challenges including storage, access, management, analysis, security, and search.

3. Requirements system.

• These 64-bit VMs require a 64-bit host OS and a virtualization product that can support a 64-bit guest OS.
• To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher:
• Player 4.x or higher
• Fusion 4.x or higher

• Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file), but some features in VMware Tools are not available.
• The amount of RAM required varies by the run-time option you choose:

CDH and Cloudera Manager Version	RAM Required by VM
CDH 5 (default)	4+ GiB*
Cloudera Express	8+ GiB*
Cloudera Enterprise (trial)	10+ GiB*

*Minimum recommended memory. If you are running workloads larger than the examples provided, consider allocating additional memory.

4. Installation

In the section we just show you how to installation the Cloudera QuickStart Virtual Machine.

- The first step is to download and install VirtualBox. Download here.

- The first step is to download the Cloudera quickstart by following this link.

- Use the 7-zip to extract the contents of downloaded zip file.

- Run the VirtualBox and then import Cloudera to VirtualBox.

5. Test Cloudera.

• Download WordCount.java from Sakai.
• Create new project in eclipse.
• Add references:
• File system/usr/lib/hadoop/client-0.20
• File system/usr/lib/hadoop
• File system/usr/lib/hadoop/lib
• Create folder input inside project and add document file with text.

• Run program.
• Check file in output folder to see result.

2. Eclipse

1. General about Eclipse.

Eclipse (@ www.eclipse.org) is an open-source Integrated Development Environment (IDE) supported by IBM. Eclipse is popular for Java application development (Java SE and Java EE) and Android apps. It also supports C/C++, PHP, Python, Perl, and other web project developments via extensible plug-ins. Eclipse is cross-platform and runs under Windows, Linux and Mac OS.

2. Installation.

• To use Eclipse for Java programming, you need to first install Java Development Kit (JDK). To read this link to know "How to install SDK (on Window)" .
• Download Eclipse from here.
• To install Eclipse, simply unzip the downloaded file into a directory of your choice (e.g., "d:\myproject").

3. Running eclipse.

• Create new project in Eclipse(HelloWorld).
• Customize your code and then run the project.
• 

3. References.

• https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/SingleCluster.html
• http://www.cloudera.com/about-cloudera/faqs.html.
• https://www.virtualbox.org/wiki/Downloads.
• https://www.ntu.edu.sg/home/ehchua/programming/howto/EclipseJava_HowTo.html
• http://www.eclipse.org/downloads/
• https://www.ntu.edu.sg/home/ehchua/programming/howto/JDK_Howto.html#jdk-install