Cloudera CCDH CCD-410 Questions & Answers

• Question 41:

  You have user profile records in your OLPT database, that you want to join with web logs you have already ingested into the Hadoop file system. How will you obtain these user records?

  A. HDFS command

  B. Pig LOAD command

  C. Sqoop import

  D. Hive LOAD DATA command

  E. Ingest with Flume agents F. Ingest with Hadoop Streaming

  Correct Answer: C

  Reference: Hadoop and Pig for Large-Scale Web Log Analysis

• Question 42:

  You have the following key-value pairs as output from your Map task:

  (the, 1) (fox, 1) (faster, 1) (than, 1) (the, 1) (dog, 1)

  How many keys will be passed to the Reducer's reduce method?

  A. Six

  B. Five

  C. Four

  D. Two

  E. One

  F. Three

  Correct Answer: B

  Only one key value pair will be passed from the two (the, 1) key value pairs.

• Question 43:

  For each input key-value pair, mappers can emit:

  A. As many intermediate key-value pairs as designed. There are no restrictions on the types of those key-value pairs (i.e., they can be heterogeneous).

  B. As many intermediate key-value pairs as designed, but they cannot be of the same type as the input key-value pair.

  C. One intermediate key-value pair, of a different type.

  D. One intermediate key-value pair, but of the same type.

  E. As many intermediate key-value pairs as designed, as long as all the keys have the same types and all the values have the same type.

  Correct Answer: E

  Mapper maps input key/value pairs to a set of intermediate key/value pairs.

  Maps are the individual tasks that transform input records into intermediate records. The transformed intermediate records do not need to be of the same type as the input records. A given input pair may map to zero or many output pairs.

  Reference: Hadoop Map-Reduce Tutorial

• Question 44:

  Which best describes how TextInputFormat processes input files and line breaks?

  A. Input file splits may cross line breaks. A line that crosses file splits is read by the RecordReader of the split that contains the beginning of the broken line.

  B. Input file splits may cross line breaks. A line that crosses file splits is read by the RecordReaders of both splits containing the broken line.

  C. The input file is split exactly at the line breaks, so each RecordReader will read a series of complete lines.

  D. Input file splits may cross line breaks. A line that crosses file splits is ignored.

  E. Input file splits may cross line breaks. A line that crosses file splits is read by the RecordReader of the split that contains the end of the broken line.

  Correct Answer: E

  As the Map operation is parallelized the input file set is first split to several pieces called FileSplits. If an individual file is so large that it will affect seek time it will be split to several Splits. The splitting does not know anything about the input file's internal logical structure, for example line-oriented text files are split on arbitrary byte boundaries. Then a new map task is created per FileSplit.

  When an individual map task starts it will open a new output writer per configured reduce task. It will then proceed to read its FileSplit using the RecordReader it gets from the specified InputFormat. InputFormat parses the input and generates key-value pairs. InputFormat must also handle records that may be split on the FileSplit boundary. For example TextInputFormat will read the last line of the FileSplit past the split boundary and, when reading other than the first FileSplit, TextInputFormat ignores the content up to the first newline.

  Reference: How Map and Reduce operations are actually carried out

• Question 45:

  Identify the MapReduce v2 (MRv2 / YARN) daemon responsible for launching application containers and monitoring application resource usage?

  A. ResourceManager

  B. NodeManager

  C. ApplicationMaster

  D. ApplicationMasterService

  E. TaskTracker

  F. JobTracker

  Correct Answer: B

  Reference: Apache Hadoop YARN Concepts and Applications

• Question 46:

  You are developing a MapReduce job for sales reporting. The mapper will process input keys representing the year (IntWritable) and input values representing product indentifies (Text). Indentify what determines the data types used by the Mapper for a given job.

  A. The key and value types specified in the JobConf.setMapInputKeyClass and JobConf.setMapInputValuesClass methods

  B. The data types specified in HADOOP_MAP_DATATYPES environment variable

  C. The mapper-specification.xml file submitted with the job determine the mapper's input key and value types.

  D. The InputFormat used by the job determines the mapper's input key and value types.

  Correct Answer: D

  The input types fed to the mapper are controlled by the InputFormat used. The default input format,

  "TextInputFormat," will load data in as (LongWritable, Text) pairs. The long value is the byte offset of the

  line in the file. The Text object holds the string contents of the line of the file.

  Note: The data types emitted by the reducer are identified by setOutputKeyClass()

  andsetOutputValueClass(). The data types emitted by the reducer are identified by setOutputKeyClass()

  and setOutputValueClass().

  By default, it is assumed that these are the output types of the mapper as well. If this is not the case, the

  methods setMapOutputKeyClass() and setMapOutputValueClass() methods of the JobConf class will

  override these.

  Reference: Yahoo! Hadoop Tutorial, THE DRIVER METHOD

• Question 47:

  You need to run the same job many times with minor variations. Rather than hardcoding all job configuration options in your drive code, you've decided to have your Driver subclass org.apache.hadoop.conf.Configured and implement the org.apache.hadoop.util.Tool interface. Indentify which invocation correctly passes.mapred.job.name with a value of Example to Hadoop?

  A. hadoop "mapred.job.name=Example" MyDriver input output

  B. hadoop MyDriver mapred.job.name=Example input output

  C. hadoop MyDrive D mapred.job.name=Example input output

  D. hadoop setproperty mapred.job.name=Example MyDriver input output

  E. hadoop setproperty ("mapred.job.name=Example") MyDriver input output

  Correct Answer: C

  Configure the property using the -D key=value notation:

  -D mapred.job.name='My Job'

  You can list a whole bunch of options by calling the streaming jar with just the -info argument

  Reference: Python hadoop streaming : Setting a job name

• Question 48:

  MapReduce v2 (MRv2/YARN) is designed to address which two issues?

  A. Single point of failure in the NameNode.

  B. Resource pressure on the JobTracker.

  C. HDFS latency.

  D. Ability to run frameworks other than MapReduce, such as MPI.

  E. Reduce complexity of the MapReduce APIs.

  F. Standardize on a single MapReduce API.

  Correct Answer: BD

  YARN (Yet Another Resource Negotiator), as an aspect of Hadoop, has two major kinds of benefits:

  *

  (D) The ability to use programming frameworks other than MapReduce. / MPI (Message Passing Interface) was mentioned as a paradigmatic example of a MapReduce alternative

  *

  Scalability, no matter what programming framework you use. Note:

  *

  The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.

  *

  (B) The central goal of YARN is to clearly separate two things that are unfortunately smushed together in current Hadoop, specifically in (mainly) JobTracker:

  / Monitoring the status of the cluster with respect to which nodes have which resources available. Under YARN, this will be global. / Managing the parallelization execution of any specific job. Under YARN, this will be done separately for each job. The current Hadoop MapReduce system is fairly scalable -- Yahoo runs 5000 Hadoop jobs, truly concurrently, on a single cluster, for a total 1.5 2 millions jobs/cluster/month. Still, YARN will remove scalability bottlenecks

  Reference: Apache Hadoop YARN Concepts and Applications

• Question 49:

  You want to understand more about how users browse your public website, such as which pages they visit

  prior to placing an order. You have a farm of 200 web servers hosting your website.

  How will you gather this data for your analysis?

  A. Ingest the server web logs into HDFS using Flume.

  B. Write a MapReduce job, with the web servers for mappers, and the Hadoop cluster nodes for reduces.

  C. Import all users' clicks from your OLTP databases into Hadoop, using Sqoop.

  D. Channel these clickstreams inot Hadoop using Hadoop Streaming.

  E. Sample the weblogs from the web servers, copying them into Hadoop using curl.

  Correct Answer: A

• Question 50:

  You have just executed a MapReduce job. Where is intermediate data written to after being emitted from the Mapper's map method?

  A. Intermediate data in streamed across the network from Mapper to the Reduce and is never written to disk.

  B. Into in-memory buffers on the TaskTracker node running the Mapper that spill over and are written into HDFS.

  C. Into in-memory buffers that spill over to the local file system of the TaskTracker node running the Mapper.

  D. Into in-memory buffers that spill over to the local file system (outside HDFS) of the TaskTracker node running the Reducer

  E. Into in-memory buffers on the TaskTracker node running the Reducer that spill over and are written into HDFS.

  Correct Answer: C

  The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, Where is the Mapper Output (intermediate kay-value data) stored ?