Hortonworks Hortonworks Certifications HDPCD Questions & Answers

• Question 81:

  Which two of the following statements are true about Pig's approach toward data? Choose 2 answers

  A. Accepts only data that has a key/value pair structure

  B. Accepts data whether it has metadata or not

  C. Accepts only data that is defined by metadata tables stored in a database

  D. Accepts tab-delimited text data only

  E. Accepts any data: structured or unstructured

  Correct Answer: BE

• Question 82:

  When is the earliest point at which the reduce method of a given Reducer can be called?

  A. As soon as at least one mapper has finished processing its input split.

  B. As soon as a mapper has emitted at least one record.

  C. Not until all mappers have finished processing all records.

  D. It depends on the InputFormat used for the job.

  Correct Answer: C

  Explanation: In a MapReduce job reducers do not start executing the reduce method until the all Map jobs have completed. Reducers start copying intermediate key-value pairs from the mappers as soon as they are available. The programmer defined reduce method is called only after all the mappers have finished.

  Note: The reduce phase has 3 steps: shuffle, sort, reduce. Shuffle is where the data is collected by the reducer from each mapper. This can happen while mappers are generating data since it is only a data transfer. On the other hand, sort and reduce can only start once all the mappers are done.

  Why is starting the reducers early a good thing? Because it spreads out the data transfer from the mappers to the reducers over time, which is a good thing if your network is the bottleneck.

  Why is starting the reducers early a bad thing? Because they "hog up" reduce slots while only copying data. Another job that starts later that will actually use the reduce slots now can't use them.

  You can customize when the reducers startup by changing the default value of mapred.reduce.slowstart.completed.maps in mapred-site.xml. A value of 1.00 will wait for all the mappers to finish before starting the reducers. A value of 0.0 will start the reducers right away. A value of 0.5 will start the reducers when half of the mappers are complete. You can also change mapred.reduce.slowstart.completed.maps on a job-by-job basis.

  Typically, keep mapred.reduce.slowstart.completed.maps above 0.9 if the system ever has multiple jobs running at once. This way the job doesn't hog up reducers when they aren't doing anything but copying data. If you only ever have one job running at a time, doing 0.1 would probably be appropriate.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, When is the reducers are started in a MapReduce job?

• Question 83:

  Which describes how a client reads a file from HDFS?

  A. The client queries the NameNode for the block location(s). The NameNode returns the block location(s) to the client. The client reads the data directory off the DataNode(s).

  B. The client queries all DataNodes in parallel. The DataNode that contains the requested data responds directly to the client. The client reads the data directly off the DataNode.

  C. The client contacts the NameNode for the block location(s). The NameNode then queries the DataNodes for block locations. The DataNodes respond to the NameNode, and the NameNode redirects the client to the DataNode that holds the requested data block(s). The client then reads the data directly off the DataNode.

  D. The client contacts the NameNode for the block location(s). The NameNode contacts the DataNode that holds the requested data block. Data is transferred from the DataNode to the NameNode, and then from the NameNode to the client.

  Correct Answer: A

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, How the Client communicates with HDFS?

• Question 84:

  Which HDFS command uploads a local file X into an existing HDFS directory Y?

  A. hadoop scp X Y

  B. hadoop fs -localPut X Y

  C. hadoop fs-put X Y

  D. hadoop fs -get X Y

  Correct Answer: C

• Question 85:

  You have just executed a MapReduce job. Where is intermediate data written to after being emitted from the Mapper's map method?

  A. Intermediate data in streamed across the network from Mapper to the Reduce and is never written to disk.

  B. Into in-memory buffers on the TaskTracker node running the Mapper that spill over and are written into HDFS.

  C. Into in-memory buffers that spill over to the local file system of the TaskTracker node running the Mapper.

  D. Into in-memory buffers that spill over to the local file system (outside HDFS) of the TaskTracker node running the Reducer

  E. Into in-memory buffers on the TaskTracker node running the Reducer that spill over and are written into HDFS.

  Correct Answer: C

  Explanation: The mapper output (intermediate data) is stored on the Local file system (NOT HDFS) of each individual mapper nodes. This is typically a temporary directory location which can be setup in config by the hadoop administrator. The intermediate data is cleaned up after the Hadoop Job completes.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, Where is the Mapper Output (intermediate kay-value data) stored ?

• Question 86:

  Examine the following Hive statements:

  

  Assuming the statements above execute successfully, which one of the following statements is true?

  A. Each reducer generates a file sorted by age

  B. The SORT BY command causes only one reducer to be used

  C. The output of each reducer is only the age column

  D. The output is guaranteed to be a single file with all the data sorted by age

  Correct Answer: A

• Question 87:

  MapReduce v2 (MRv2/YARN) splits which major functions of the JobTracker into separate daemons?

  Select two.

  A. Heath states checks (heartbeats)

  B. Resource management

  C. Job scheduling/monitoring

  D. Job coordination between the ResourceManager and NodeManager

  E. Launching tasks

  F. Managing file system metadata

  G. MapReduce metric reporting

  H. Managing tasks

  Correct Answer: BC

  Explanation: The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker,

  resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global

  ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in

  the classical sense of Map- Reduce jobs or a DAG of jobs.

  Note:

  The central goal of YARN is to clearly separate two things that are unfortunately smushed together in

  current Hadoop, specifically in (mainly) JobTracker:

  / Monitoring the status of the cluster with respect to which nodes have which resources available. Under

  YARN, this will be global.

  / Managing the parallelization execution of any specific job. Under YARN, this will be done separately for

  each job.

  Reference: Apache Hadoop YARN ?Conceptsand; Applications

• Question 88:

  On a cluster running MapReduce v1 (MRv1), a TaskTracker heartbeats into the JobTracker on your cluster, and alerts the JobTracker it has an open map task slot.

  What determines how the JobTracker assigns each map task to a TaskTracker?

  A. The amount of RAM installed on the TaskTracker node.

  B. The amount of free disk space on the TaskTracker node.

  C. The number and speed of CPU cores on the TaskTracker node.

  D. The average system load on the TaskTracker node over the past fifteen (15) minutes.

  E. The location of the InsputSplit to be processed in relation to the location of the node.

  Correct Answer: E

  Explanation: The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the JobTracker that it is still alive. These message also inform the JobTracker of the number of available slots, so the JobTracker can stay up to date with where in the cluster work can be delegated. When the JobTracker tries to find somewhere to schedule a task within the MapReduce operations, it first looks for an empty slot on the same server that hosts the DataNode containing the data, and if not, it looks for an empty slot on a machine in the same rack.

  Reference: 24 Interview Questions and Answers for Hadoop MapReduce developers, How JobTracker schedules a task?

• Question 89:

  Given the following Hive command:

  INSERT OVERWRITE TABLE mytable SELECT * FROM myothertable;

  Which one of the following statements is true?

  A. The contents of myothertable are appended to mytable

  B. Any existing data in mytable will be overwritten

  C. A new table named mytable is created, and the contents of myothertable are copied into mytable

  D. The statement is not a valid Hive command

  Correct Answer: B

• Question 90:

  You have a directory named jobdata in HDFS that contains four files: _first.txt, second.txt, .third.txt and #data.txt. How many files will be processed by the FileInputFormat.setInputPaths () command when it's given a path object representing this directory?

  A. Four, all files will be processed

  B. Three, the pound sign is an invalid character for HDFS file names

  C. Two, file names with a leading period or underscore are ignored

  D. None, the directory cannot be named jobdata

  E. One, no special characters can prefix the name of an input file

  Correct Answer: C

  Explanation: Files starting with '_' are considered 'hidden' like unix files starting with '.'. # characters are allowed in HDFS file names.