Hortonworks HCAHD APACHE-HADOOP-DEVELOPER Questions & Answers

• Question 1:

  When is the earliest point at which the reduce method of a given Reducer can be called?

  A. As soon as at least one mapper has finished processing its input split.

  B. As soon as a mapper has emitted at least one record.

  C. Not until all mappers have finished processing all records.

  D. It depends on the InputFormat used for the job.

  Correct Answer: C

  Explanation: In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.

  Note: The reduce phase has 3 steps: shuffle, sort, reduce. Shuffle is where the data is collected by the reducer from each mapper. This can happen while mappers are generating data since it is only a data transfer. On the other hand, sort and reduce can only start once all the mappers are done.

  Why is starting the reducers early a good thing? Because it spreads out the data transfer from the mappers to the reducers over time, which is a good thing if your network is the bottleneck.

  Why is starting the reducers early a bad thing? Because they "hog up" reduce slots while only copying data. Another job that starts later that will actually use the reduce slots now can't use them.

  You can customize when the reducers startup by changing the default value of mapred.reduce.slowstart.completed.maps in mapred-site.xml. A value of 1.00 will wait for all the mappers to finish before starting the reducers. A value of 0.0 will start the reducers right away. A value of 0.5 will start the reducers when half of the mappers are complete. You can also change mapred.reduce.slowstart.completed.maps on a job-by-job basis.

  Typically, keep mapred.reduce.slowstart.completed.maps above 0.9 if the system ever has multiple jobs running at once. This way the job doesn't hog up reducers when they aren't doing anything but copying data. If you only ever have one job running at a time, doing 0.1 would probably be appropriate.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, When is the reducers are started in a MapReduce job?

• Question 2:

  In Hadoop 2.0, which one of the following statements is true about a standby NameNode? The Standby NameNode:

  A. Communicates directly with the active NameNode to maintain the state of the active NameNode.

  B. Receives the same block reports as the active NameNode.

  C. Runs on the same machine and shares the memory of the active NameNode.

  D. Processes all client requests and block reports from the appropriate DataNodes.

  Correct Answer: B

• Question 3:

  Given the following Hive commands:

  

  Which one of the following statements Is true?

  A. The file mydata.txt is copied to a subfolder of /apps/hive/warehouse

  B. The file mydata.txt is moved to a subfolder of /apps/hive/warehouse

  C. The file mydata.txt is copied into Hive's underlying relational database 0.

  D. The file mydata.txt does not move from Its current location in HDFS

  Correct Answer: A

• Question 4:

  Consider the following two relations, A and B.

  

  A Pig JOIN statement that combined relations A by its first field and B by its second field would produce what output?

  A. 2 Jim Chris 2 3 Terry 3 4 Brian 4

  B. 2 cherry 2 cherry 3 orange 4 peach

  C. 2 cherry Jim, Chris 3 orange Terry

  4 peach Brian

  D. 2 cherry Jim 2 2 cherry Chris 2 3 orange Terry 3 4 peach Brian 4

  Correct Answer: D

• Question 5:

  Review the following data and Pig code:

  

  What command to define B would produce the output (M,62,95l02) when invoking the DUMP operator on B?

  A. B = FILTER A BY (zip = = '95102' AND gender = = M");

  B. B= FOREACH A BY (gender = = 'M' AND zip = = '95102');

  C. B = JOIN A BY (gender = = 'M' AND zip = = '95102');

  D. B= GROUP A BY (zip = = '95102' AND gender = = 'M');

  Correct Answer: A

• Question 6:

  You have written a Mapper which invokes the following five calls to the OutputColletor.collect method: output.collect (new Text ("Apple"), new Text ("Red") ) ;

  output.collect (new Text ("Banana"), new Text ("Yellow") ) ;

  output.collect (new Text ("Apple"), new Text ("Yellow") ) ;

  output.collect (new Text ("Cherry"), new Text ("Red") ) ;

  output.collect (new Text ("Apple"), new Text ("Green") ) ;

  How many times will the Reducer's reduce method be invoked?

  A. 6

  B. 3

  C. 1

  D. 0

  E. 5

  Correct Answer: B

  Explanation: reduce() gets called once for each [key, (list of values)] pair. To explain, let's say you called:

  out.collect(new Text("Car"),new Text("Subaru");

  out.collect(new Text("Car"),new Text("Honda");

  out.collect(new Text("Car"),new Text("Ford");

  out.collect(new Text("Truck"),new Text("Dodge");

  out.collect(new Text("Truck"),new Text("Chevy");

  Then reduce() would be called twice with the pairs reduce(Car, )

  reduce(Truck, )

  Reference: Mapper output.collect()?

• Question 7:

  You want to run Hadoop jobs on your development workstation for testing before you submit them to your production cluster. Which mode of operation in Hadoop allows you to most closely simulate a production cluster while using a single machine?

  A. Run all the nodes in your production cluster as virtual machines on your development workstation.

  B. Run the hadoop command with the –jt local and the –fs file:///options.

  C. Run the DataNode, TaskTracker, NameNode and JobTracker daemons on a single machine.

  D. Run simldooop, the Apache open-source software for simulating Hadoop clusters.

  Correct Answer: C

• Question 8:

  Can you use MapReduce to perform a relational join on two large tables sharing a key? Assume that the two tables are formatted as comma-separated files in HDFS.

  A. Yes.

  B. Yes, but only if one of the tables fits into memory

  C. Yes, so long as both tables fit into memory.

  D. No, MapReduce cannot perform relational operations.

  E. No, but it can be done with either Pig or Hive.

  Correct Answer: A

  Explanation: Note:

  *

  Join Algorithms in MapReduce A) Reduce-side join B) Map-side join

  C) In-memory join / Striped Striped variant variant / Memcached variant

  *

  Which join to use? / In-memory join > map-side join > reduce-side join / Limitations of each? In-memory join: memory Map-side join: sort order and partitioning Reduce-side join: general purpose

• Question 9:

  Which one of the following statements describes a Hive user-defined aggregate function?

  A. Operates on multiple input rows and creates a single row as output

  B. Operates on a single input row and produces a single row as output

  C. Operates on a single input row and produces a table as output

  D. Operates on multiple input rows and produces a table as output

  Correct Answer: A

• Question 10:

  Assuming the following Hive query executes successfully:

  

  Which one of the following statements describes the result set?

  A. A bigram of the top 80 sentences that contain the substring "you are" in the lines column of the input data A1 table.

  B. An 80-value ngram of sentences that contain the words "you" or "are" in the lines column of the inputdata table.

  C. A trigram of the top 80 sentences that contain "you are" followed by a null space in the lines column of the inputdata table.

  D. A frequency distribution of the top 80 words that follow the subsequence "you are" in the lines column of the inputdata table.

  Correct Answer: D