Hadoop performance tuning (Hadoop-Hive) part -3

[Note: This post is Third part of Hadoop performance tuning, if you directly reached this page, please click here for part 1 and click here for part 2.]


Before going to see some configuration parameter for performance tuning, I like to ask you a question, have you ever observed job and task tracker WebUI, there you can see lot of jobs are being killed after few seconds or minutes before completion. Why so? Have you ever think of it? Of course, few of them know. Those who already know about this, please skip next paragraph.

[NOTE: To check WebUI of Hadoop cluster: Open the browser, type http://masternode-machineip(or)localhost:portnumber. We can also check this port number by changing the configuration parameter value to the portnumber we want.]

Name	Port	Configuration parameter
Jobtracker	50030	mapred.job.tracker.http.address
Task trackers	50060	mapred.task.tracker.http.address

As you know, our data to be processed is replicated in multiple node. So while processing data hadoop will also start processing same(replicated) data chunk in multiple node. The node which complete first will kill other jobs which is processing same data. advantage here is job will be completed soon. Everyone will be puzzled, how?. While processing if multiple node process same data, whichever process first will kill other, so first processed data is considers. After we killed on job in a node, that node start processing next job(next data chunk).By default hadoop works like this. Cause by default, following parameter value is set to true.

mapred.map.tasks.speculative.execution

mapred.reduce.tasks.speculative.execution

Is this situation applicable to all type of applications? No.

If data to be processed contain complex calculation, each job take more time cause of executing in multiple node. This time can be utilize by processing unique data chunks. Hence for the applications which has more complex operation to be perform, recommended to set this parameter value to false.

io.sort.mb (buffer size for sorting)

Default value is 100, means 100 MB of buffer memory for sorting. After map jobs are completed processing, hadoop will sort the map outputs for reducer. If map output are larger, then its recommended to increase the value. we should also consider our memory size while increasing this buffer value, anyhow it will take part in memory(RAM). This parameter will give good performance improvement because, if buffer size is large, then less amount of spill to disk. So it reduces operations like read/write of data spills to disks.

io.sort.factor (stream merging factor)

Default value for this parameter is 10, this value is recommended to increase for job with larger output similar like above. This value tell how many streams can merge at once while sorting. This will give performance improvement because its reduce time in number of intermediate merging.

Above suggestions are observed with Hadoop cluster with Hive querying, If any information discussed here is misinterpreted, please leave a suggestion in comments. Recommend this post by clicking  Facebook ‘Like’ button and ‘+1’ at bottom of this page. By clicking like button you got regular update about my post in your facebook updates.