Latest Hadoop Interview Questions -Part 14

What are the Side Data Distribution Techniques

Side data refers to extra static small data required by map reduce to perform job. Main challenge is the availability of side data on the node where the map would be executed. Hadoop provides two side data distribution techniques.

Using Job Configuration

An arbitrary Key value pair can be set in job configuration. very useful technique in case of small file. Suggested size of file to keep in configuration object is in KBs.Because conf object would be read by job tracker, task tracker and new child jvm. this would increase overhead at every front. A part from this side data would require serialization if it has non-primitive encoding.

Distributed Cache

Rather than serializing side data in the job configuration, it is preferable to distribute datasets using Hadoop’s distributed cache mechanism. This provides a service for copying files and archives to the task nodes in time for the tasks to use them when they run. To save network bandwidth, files are normally copied to any particular node once per job.

What is shuffleing in mapreduce?

Once map tasks started to complete, A communication from reducers is started. where map output sents to reducer, which is looking for the output data to process. at same time data nodes are still process multiple other tasks. The data transfer of mappers output to reducer known as shuffling.

What is partitioning?

Partitioning is a process to identify the reducer instance which would be used to supply the mappers output. Before mapper emits the data (Key Value) pair to reducer, mapper identify the reducer as an recipient of mapper output. All the key, no matter which mapper has generated this, must lie with same reducer.

Can we change the file cached by DistributedCache

No, DistributedCache tracks the caching with timestamp. cached file should not be changed during the job execution.

What is Distributed Cache in mapreduce framework?

Distributed cache is an important feature provide by map reduce framework. Distributed cache can cache text, archive, jars which could be used by application to improve performance. Application provide details of file to jobconf object to cache. Mapreduce framework would copy the specified file to data node before processing the job. Framework copy file only once for each job, and has the ability of archival. Application needs to specify the file path via http:// or hdfs:// to cache.

What is speculative execution in Hadoop?

It becomes very important when dealing with a very large cluster. Lets assume if you have thousands of machine in your cluster, and one of your data node is not performing well in comparison to others. It would degrade the over all performance of a job executed by whole cluster. Speculative execution is technique, hadoop runs multiple copies of MR task on other data nodes. Which machine would finish the execution of task, will be consider for result.

What if job tracker machine is down?

Single point failure from execution point of view.

Can we deploy job tracker other than name node?

Yes, in production it is highly recommended. For self development and learning you may setup according to your need.

What is a task tracker?

Task tracker is actual component which deployed the mapreduce jar on data nodes and responsible to execute the task given to mapreduce. It continuously executes the task and send updated report to job tracker.

What is a job tracker?

Job tracker is a background service executed on namenode for submitting and tracking a Job. Job in hadoop terminology refers to mapreduce jobs. It further break up the job into tasks. Which would be deployed every data node holding the required data. In a Hadoop cluster, Job tracker is master and task acts like child, acts, performs and revert the progress to job tracker through heartbeat.