Ansible and Hadoop

What is ansible ???

Ansible is a configuration management tool design on the top of python. It provides a way to automate the things . In simple term you can run the desired tasks remotely on a different operating system just be a single click. It helps in creating playbook which when run performs the functions you need.

What is hadoop ?

Apache Hadoop is software which works on master- slave topology . It is used to overcome the issues of big data by distributing the storage in its slave . The master also known as namenode and the slaves are known as datanodes.

This article tells how we can create hadoop cluster through ansible playbooks.

Assuming you have ansible already installed in your system . If not install using <<< pip3 install ansible>>>. Create a directory which holds all the playbook of the namenode and the datanode.

Make sure you add the remote ip’s of the system in the inventory . You can create a group in the file by using square brackets .

Create the playbooks of for the namenode and datanode as follows :

NAMENODE :

Explanation:

Use the copy module to copy the software to your namenode. << Make sure your controller node has both the software >>

Else you can also use the command module to install the softwares.

name : Installing the software

command : rpm -ivh /root/{{hadoop}} /root/{{jdk}}

Next step is to install the software . Now create the hdfs-site.xml and coe-site.xml files in your controller node with the exact configuration . Now copy them in to the namenode in the hadoop directory .

Create the directory for namenode . Format it before using . Start the services.

DATANODE :

Explanation :

Follow the similar step for datanode . << Make sure to give the source and destination a complete path >>>

To run the playbook use the command :

ansible-playbook -v namenode.yml

ansible-playbook -v datanode.yml

The hadoop cluster has been successfully configured :