Cloudera CCDH CCD-410 Questions & Answers

• Question 1:

  What is a SequenceFile?

  A. A SequenceFile contains a binary encoding of an arbitrary number of homogeneous writable objects.

  B. A SequenceFile contains a binary encoding of an arbitrary number of heterogeneous writable objects.

  C. A SequenceFile contains a binary encoding of an arbitrary number of WritableComparable objects, in sorted order.

  D. A SequenceFile contains a binary encoding of an arbitrary number key-value pairs. Each key must be the same type. Each value must be same type.

  Correct Answer: D

  SequenceFile is a flat file consisting of binary key/value pairs.

  There are 3 different SequenceFile formats:

  Uncompressed key/value records.

  Record compressed key/value records - only 'values' are compressed here.

  Block compressed key/value records - both keys and values are collected in 'blocks' separately and compressed. The size of the 'block' is configurable.

  Reference: http://wiki.apache.org/hadoop/SequenceFile

• Question 2:

  You want to run Hadoop jobs on your development workstation for testing before you submit them to your production cluster. Which mode of operation in Hadoop allows you to most closely simulate a production cluster while using a single machine?

  A. Run all the nodes in your production cluster as virtual machines on your development workstation.

  B. Run the hadoop command with the jt local and the fs file:///options.

  C. Run the DataNode, TaskTracker, NameNode and JobTracker daemons on a single machine.

  D. Run simldooop, the Apache open-source software for simulating Hadoop clusters.

  Correct Answer: C

• Question 3:

  Your cluster's HDFS block size in 64MB. You have directory containing 100 plain text files, each of which is 100MB in size. The InputFormat for your job is TextInputFormat. Determine how many Mappers will run?

  A. 64

  B. 100

  C. 200

  D. 640

  Correct Answer: C

  Each file would be split into two as the block size (64 MB) is less than the file size (100 MB), so 200 mappers would be running.

  Note:

  If you're not compressing the files then hadoop will process your large files (say 10G), with a number of mappers related to the block size of the file.

  Say your block size is 64M, then you will have ~160 mappers processing this 10G file (160*64 ~= 10G). Depending on how CPU intensive your mapper logic is, this might be an acceptable blocks size, but if you find that your mappers are executing in sub minute times, then you might want to increase the work done by each mapper (by increasing the block size to 128, 256, 512m - the actual size depends on how you intend to process the data).

  Reference: http://stackoverflow.com/questions/11014493/hadoop-mapreduce-appropriate-input-files-size (first answer, second paragraph)

• Question 4:

  You want to perform analysis on a large collection of images. You want to store this data in HDFS and process it with MapReduce but you also want to give your data analysts and data scientists the ability to process the data directly from HDFS with an interpreted high-level programming language like Python. Which format should you use to store this data in HDFS?

  A. SequenceFiles

  B. Avro

  C. JSON

  D. HTML

  E. XML

  F. CSV

  Correct Answer: B

  Reference: Hadoop binary files processing introduced by image duplicates finder

• Question 5:

  When can a reduce class also serve as a combiner without affecting the output of a MapReduce program?

  A. When the types of the reduce operation's input key and input value match the types of the reducer's output key and output value and when the reduce operation is both communicative and associative.

  B. When the signature of the reduce method matches the signature of the combine method.

  C. Always. Code can be reused in Java since it is a polymorphic object-oriented programming language.

  D. Always. The point of a combiner is to serve as a mini-reducer directly after the map phase to increase performance.

  E. Never. Combiners and reducers must be implemented separately because they serve different purposes.

  Correct Answer: A

  You can use your reducer code as a combiner if the operation performed is commutative and associative.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, What are combiners? When should I use a combiner in my MapReduce Job?

• Question 6:

  Which best describes what the map method accepts and emits?

  A. It accepts a single key-value pair as input and emits a single key and list of corresponding values as output.

  B. It accepts a single key-value pairs as input and can emit only one key-value pair as output.

  C. It accepts a list key-value pairs as input and can emit only one key-value pair as output.

  D. It accepts a single key-value pairs as input and can emit any number of key-value pair as output, including zero.

  Correct Answer: D

  public class Mapper

  extends Object

  Maps input key/value pairs to a set of intermediate key/value pairs.

  Maps are the individual tasks which transform input records into a intermediate records. The transformed intermediate records need not be of the same type as the input records. A given input pair may map to zero or many output pairs. Reference: org.apache.hadoop.mapreduce

  Class Mapper

• Question 7:

  Workflows expressed in Oozie can contain:

  A. Sequences of MapReduce and Pig. These sequences can be combined with other actions including forks, decision points, and path joins.

  B. Sequences of MapReduce job only; on Pig on Hive tasks or jobs. These MapReduce sequences can be combined with forks and path joins.

  C. Sequences of MapReduce and Pig jobs. These are limited to linear sequences of actions with exception handlers but no forks.

  D. Iterntive repetition of MapReduce jobs until a desired answer or state is reached.

  Correct Answer: A

  Oozie workflow is a collection of actions (i.e. Hadoop Map/Reduce jobs, Pig jobs) arranged in a control dependency DAG (Direct Acyclic Graph), specifying a sequence of actions execution. This graph is specified in hPDL (a XML Process Definition Language).

  hPDL is a fairly compact language, using a limited amount of flow control and action nodes. Control nodes define the flow of execution and include beginning and end of a workflow (start, end and fail nodes) and mechanisms to control the workflow execution path ( decision, fork and join nodes).

  Workflow definitions Currently running workflow instances, including instance states and variables

  Reference: Introduction to Oozie

  Note: Oozie is a Java Web-Application that runs in a Java servlet-container - Tomcat and uses a database to store:

• Question 8:

  In a large MapReduce job with m mappers and n reducers, how many distinct copy operations will there be in the sort/shuffle phase?

  A. mXn (i.e., m multiplied by n)

  B. n

  C. m

  D. m+n (i.e., m plus n)

  E. mn (i.e., m to the power of n)

  Correct Answer: A

  A MapReduce job with m mappers and r reducers involves up to m * r distinct copy operations, since each mapper may have intermediate output going to every reducer.

• Question 9:

  You need to move a file titled "weblogs" into HDFS. When you try to copy the file, you can't. You know you have ample space on your DataNodes. Which action should you take to relieve this situation and store more files in HDFS?

  A. Increase the block size on all current files in HDFS.

  B. Increase the block size on your remaining files.

  C. Decrease the block size on your remaining files.

  D. Increase the amount of memory for the NameNode.

  E. Increase the number of disks (or size) for the NameNode.

  F. Decrease the block size on all current files in HDFS.

  Correct Answer: D

• Question 10:

  Analyze each scenario below and indentify which best describes the behavior of the default partitioner?

  A. The default partitioner assigns key-values pairs to reduces based on an internal random number generator.

  B. The default partitioner implements a round-robin strategy, shuffling the key-value pairs to each reducer in turn. This ensures an event partition of the key space.

  C. The default partitioner computes the hash of the key. Hash values between specific ranges are associated with different buckets, and each bucket is assigned to a specific reducer.

  D. The default partitioner computes the hash of the key and divides that valule modulo the number of reducers. The result determines the reducer assigned to process the key-value pair.

  E. The default partitioner computes the hash of the value and takes the mod of that value with the number of reducers. The result determines the reducer assigned to process the key-value pair.

  Correct Answer: D

  The default partitioner computes a hash value for the key and assigns the partition based on this result.

  The default Partitioner implementation is called HashPartitioner. It uses the hashCode() method of the key objects modulo the number of partitions total to determine which partition to send a given (key, value) pair to.

  In Hadoop, the default partitioner is HashPartitioner, which hashes a record's key to determine which partition (and thus which reducer) the record belongs in.The number of partition is then equal to the number of reduce tasks for the job.

  Reference: Getting Started With (Customized) Partitioning