Hortonworks Hortonworks Certifications APACHE-HADOOP-DEVELOPER Questions & Answers

• Question 61:

  In a MapReduce job, you want each of your input files processed by a single map task. How do you configure a MapReduce job so that a single map task processes each input file regardless of how many blocks the input file occupies?

  A. Increase the parameter that controls minimum split size in the job configuration.

  B. Write a custom MapRunner that iterates over all key-value pairs in the entire file.

  C. Set the number of mappers equal to the number of input files you want to process.

  D. Write a custom FileInputFormat and override the method isSplitable to always return false.

  Correct Answer: D

  Explanation: FileInputFormat is the base class for all file-based InputFormats. This provides a generic implementation of getSplits(JobContext). Subclasses of FileInputFormat can also override the isSplitable (JobContext, Path) method to ensure input-files are not split-up and are processed as a whole by Mappers.

  Reference: org.apache.hadoop.mapreduce.lib.input, Class FileInputFormat

• Question 62:

  Which Two of the following statements are true about hdfs? Choose 2 answers

  A. An HDFS file that is larger than dfs.block.size is split into blocks

  B. Blocks are replicated to multiple datanodes

  C. HDFS works best when storing a large number of relatively small files

  D. Block sizes for all files must be the same size

  Correct Answer: AB

• Question 63:

  Which one of the following statements is false about HCatalog?

  A. Provides a shared schema mechanism

  B. Designed to be used by other programs such as Pig, Hive and MapReduce

  C. Stores HDFS data in a database for performing SQL-like ad-hoc queries

  D. Exists as a subproject of Hive

  Correct Answer: C

• Question 64:

  You want to Ingest log files Into HDFS, which tool would you use?

  A. HCatalog

  B. Flume

  C. Sqoop

  D. Ambari

  Correct Answer: B

• Question 65:

  Which Hadoop component is responsible for managing the distributed file system metadata?

  A. NameNode

  B. Metanode

  C. DataNode

  D. NameSpaceManager

  Correct Answer: A

• Question 66:

  Which HDFS command displays the contents of the file x in the user's HDFS home directory?

  A. hadoop fs -Is x

  B. hdfs fs -get x

  C. hadoop fs -cat x

  D. hadoop fs -cp x

  Correct Answer: C

• Question 67:

  What types of algorithms are difficult to express in MapReduce v1 (MRv1)?

  A. Algorithms that require applying the same mathematical function to large numbers of individual binary records.

  B. Relational operations on large amounts of structured and semi-structured data.

  C. Algorithms that require global, sharing states.

  D. Large-scale graph algorithms that require one-step link traversal.

  E. Text analysis algorithms on large collections of unstructured text (e.g, Web crawls).

  Correct Answer: C

  Explanation: See 3) below.

  Limitations of Mapreduce ?where not to use Mapreduce

  While very powerful and applicable to a wide variety of problems, MapReduce is not the answer to every problem. Here are some problems I found where MapReudce is not suited and some papers that address the limitations of MapReuce.

  1.

  Computation depends on previously computed values If the computation of a value depends on previously computed values, then MapReduce cannot be used. One good example is the Fibonacci series where each value is summation of the previous two values. i.e., f(k+2) = f(k+1) + f(k). Also, if the data set is small enough to be computed on a single machine, then it is better to do it as a single reduce(map(data)) operation rather than going through the entire map reduce process.

  2.

  Full-text indexing or ad hoc searching The index generated in the Map step is one dimensional, and the Reduce step must not generate a large amount of data or there will be a serious performance degradation. For example, CouchDB's MapReduce may not be a good fit for full-text indexing or ad hoc searching. This is a problem better suited for a tool such as Lucene.

  3.

  Algorithms depend on shared global state Solutions to many interesting problems in text processing do not require global synchronization. As a result, they can be expressed naturally in MapReduce, since map and reduce tasks run independently and in isolation. However, there are many examples of algorithms that depend crucially on the existence of shared global state during processing, making them difficult to implement in MapReduce (since the single opportunity for global synchronization in MapReduce is the barrier between the map and reduce phases of processing)

  Reference: Limitations of Mapreduce ?where not to use Mapreduce

• Question 68:

  In the reducer, the MapReduce API provides you with an iterator over Writable values. What does calling the next () method return?

  A. It returns a reference to a different Writable object time.

  B. It returns a reference to a Writable object from an object pool.

  C. It returns a reference to the same Writable object each time, but populated with different data.

  D. It returns a reference to a Writable object. The API leaves unspecified whether this is a reused object or a new object.

  E. It returns a reference to the same Writable object if the next value is the same as the previous value, or a new Writable object otherwise.

  Correct Answer: C

  Explanation: Calling Iterator.next() will always return the SAME EXACT instance of IntWritable, with the contents of that instance replaced with the next value.

  Reference: manupulating iterator in mapreduce

• Question 69:

  Given the following Pig command:

  logevents = LOAD andapos;input/my.logandapos; AS (date:chararray, levehstring, code:int, message:string);

  Which one of the following statements is true?

  A. The logevents relation represents the data from the my.log file, using a comma as the parsing delimiter

  B. The logevents relation represents the data from the my.log file, using a tab as the parsing delimiter

  C. The first field of logevents must be a properly-formatted date string or table return an error

  D. The statement is not a valid Pig command

  Correct Answer: B

• Question 70:

  Determine which best describes when the reduce method is first called in a MapReduce job?

  A. Reducers start copying intermediate key-value pairs from each Mapper as soon as it has completed. The programmer can configure in the job what percentage of the intermediate data should arrive before the reduce method begins.

  B. Reducers start copying intermediate key-value pairs from each Mapper as soon as it has completed. The reduce method is called only after all intermediate data has been copied and sorted.

  C. Reduce methods and map methods all start at the beginning of a job, in order to provide optimal performance for map-only or reduce-only jobs.

  D. Reducers start copying intermediate key-value pairs from each Mapper as soon as it has completed. The reduce method is called as soon as the intermediate key-value pairs start to arrive.

  Correct Answer: B

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers , When is the reducers are started in a MapReduce job?