Amazon DATA-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Amazon DATA-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 361:

  A company has a data lake in Amazon 53. The company uses AWS Glue to catalog data and AWS Glue Studio to implement data extract, transform, and load (ETL) pipelines.

  The company needs to ensure that data quality issues are checked every time the pipelines run. A data engineer must enhance the existing pipelines to evaluate data quality rules based on predefined thresholds.

  Which solution will meet these requirements with the LEAST implementation effort?

  A. Add a new transform that is defined by a SQL query to each Glue ETL job. Use the SQL query to implement a ruleset that includes the data quality rules that need to be evaluated.
  B. Add a new Evaluate Data Quality transform to each Glue ETL job. Use Data Quality Definition Language (DQDL) to implement a ruleset that includes the data quality rules that need to be evaluated.
  C. Add a new custom transform to each Glue ETL job. Use the PyDeequ library to implement a ruleset that includes the data quality rules that need to be evaluated.
  D. Add a new custom transform to each Glue ETL job. Use the Great Expectations library to implement a ruleset that includes the data quality rules that need to be evaluated.
  B. Add a new Evaluate Data Quality transform to each Glue ETL job. Use Data Quality Definition Language (DQDL) to implement a ruleset that includes the data quality rules that need to be evaluated.

  ---

  Explanation

  Problem Analysis:

  The company uses AWS Glue for ETL pipelines and must enforce data quality checks during pipeline execution.

  The goal is to implement quality checks with minimal implementation effort .

  Key Considerations:

  AWS Glue provides an Evaluate Data Quality transform that allows for defining quality checks directly in the pipeline.

  DQDL (Data Quality Definition Language)simplifies the process by allowing declarative rule definitions.

  Solution Analysis:

  Option A: SQL Transform SQL queries can implement rules but require manual effort for each rule and do not integrate natively with Glue.

  Option B: Evaluate Data Quality Transform + DQDL AWS Glue's built-in Evaluate Data Quality transform is designed for this use case.

  Allows defining thresholds and rules in DQDL with minimal coding effort.

  Option C: Custom Transform with PyDeequ PyDeequ is a powerful library but adds unnecessary complexity compared to Glue's native features.

  Option D: Custom Transform with Great Expectations Similar to PyDeequ, Great Expectations adds operational complexity and external dependencies.

  Final Recommendation:

  Use the Evaluate Data Quality transform with DQDL to implement data quality rules in AWS Glue pipelines.

  References:

  AWS Glue Data Quality

  DQDL Syntax and Examples

  AWS Glue Studio Documentation

• Question 362:

  A data engineer is configuring an AWS Glue job to read data from an Amazon S3 bucket. The data engineer has set up the necessary AWS Glue connection details and an associated IAM role. However, when the data engineer attempts to run the AWS Glue job, the data engineer receives an error message that indicates that there are problems with the Amazon S3 VPC gateway endpoint.

  The data engineer must resolve the error and connect the AWS Glue job to the S3 bucket.

  Which solution will meet this requirement?

  A. Update the AWS Glue security group to allow inbound traffic from the Amazon S3 VPC gateway endpoint.
  B. Configure an S3 bucket policy to explicitly grant the AWS Glue job permissions to access the S3 bucket.
  C. Review the AWS Glue job code to ensure that the AWS Glue connection details include a fully qualified domain name.
  D. Verify that the VPC's route table includes inbound and outbound routes for the Amazon S3 VPC gateway endpoint.
  D. Verify that the VPC's route table includes inbound and outbound routes for the Amazon S3 VPC gateway endpoint.

  ---

  Explanation

  The error message indicates that the AWS Glue job cannot access the Amazon S3 bucket through the VPC endpoint. This could be because the VPC's route table does not have the necessary routes to direct the traffic to the endpoint. To fix this, the data engineer must verify that the route table has an entry for the Amazon S3 service prefix (com.amazonaws.region.s3) with the target as the VPC endpoint ID. This will allow the AWS Glue job to use the VPC endpoint to access the S3 bucket without going through the internet or a NAT gateway. For more information, see Gateway endpoints.

• Question 363:

  A company uses Amazon Redshift as its data warehouse service. A data engineer needs to design a physical data model.

  The data engineer encounters a de-normalized table that is growing in size. The table does not have a suitable column to use as the distribution key.

  Which distribution style should the data engineer use to meet these requirements with the LEAST maintenance overhead?

  A. ALL distribution
  B. EVEN distribution
  C. AUTO distribution
  D. KEY distribution
  C. AUTO distribution

• Question 364:

  A data engineer needs to use Amazon Neptune to develop graph applications.

  Which programming languages should the engineer use to develop the graph applications? (Choose Two.)

  A. Gremlin
  B. SQL
  C. ANSI SQL
  D. SPARQL
  E. Spark SQL
  A. Gremlin
  D. SPARQL

  ---

  Explanation

  Amazon Neptune supports graph applications using Gremlin and SPARQL as query languages. Neptune is a fully managed graph database service that supports both property graph and RDF graph models.

  Option A: GremlinGremlin is a query language for property graph databases, which is supported by Amazon Neptune. It allows the traversal and manipulation of graph data in the property graph model.

  Option D: SPARQLSPARQL is a query language for querying RDF graph data in Neptune. It is used to query, manipulate, and retrieve information stored in RDF format.

  Other options:

  SQL (Option B)and ANSI SQL (Option C) are traditional relational database query languages and are not used for graph databases.

  Spark SQL (Option E)is related to Apache Spark for big data processing, not for querying graph databases.

• Question 365:

  A company built a data lake and a data warehouse on AWS. The company wants to implement a data catalog to enhance the current data storage solutions. The company wants to have the capability to add business metadata and glossary information to the data catalog for every asset.

  Which solution will meet these requirements with the LEAST operational overhead?

  A. Use AWS Glue Catalog. Create a user table for the business glossary. Use the AWS Glue API to change table properties to add business metadata. Create a web application to access the metadata.
  B. Use an Apache Hive metastore. Create a user table for the business glossary. Use the ALTER TABLE command to change table properties to add business metadata. Create a web application to access the metadata.
  C. Use Amazon DataZone. Create the business glossaries. Create metadata forms. Use the Amazon DataZone data portal to access the metadata.
  D. Use Amazon OpenSearch Service. Create an index for the business glossary. Create a second index for the business metadata. Use the OpenSearch Service dashboard to access the metadata.
  C. Use Amazon DataZone. Create the business glossaries. Create metadata forms. Use the Amazon DataZone data portal to access the metadata.

  ---

  Explanation

  Amazon DataZone provides a fully managed data catalog and governance service with built-in support for business glossaries, custom metadata forms, and a user-friendly data portal. You can onboard your existing data assets, define glossaries, and capture business metadata without building or maintaining custom tables, APIs, or applications, minimizing operational overhead.

• Question 366:

  A data engineer is building a data pipeline. A large data file is uploaded to an Amazon S3 bucket once each day at unpredictable times. An AWS Glue workflow uses hundreds of workers to process the fileand load the data into Amazon Redshift. The company wants to process the file as quickly as possible.

  Which solution will meet these requirements?

  A. Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Lambda function that runs every 15 minutes to check the S3 bucket for the daily file. Configure the function to start the AWS Glue workflow if the file is present.
  B. Create an event-based AWS Glue trigger to start the workflow. Configure Amazon S3 to log events to AWS CloudTrail. Create a rule in Amazon EventBridge to forward PutObject events to the AWS Glue trigger.
  C. Create a scheduled AWS Glue trigger to start the workflow. Create a cron job that runs the AWS Glue job every 15 minutes. Set up the AWS Glue job to check the S3 bucket for the daily file. Configure the job to stop if the file is not present.
  D. Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Database Migration Service (AWS DMS) migration task. Set the DMS source as the S3 bucket. Set the target endpoint as the AWS Glue workflow.
  B. Create an event-based AWS Glue trigger to start the workflow. Configure Amazon S3 to log events to AWS CloudTrail. Create a rule in Amazon EventBridge to forward PutObject events to the AWS Glue trigger.

  ---

  Explanation

  An event-based trigger driven by Amazon EventBridge provides the lowest-latency, lowest-overhead way to kick off your Glue workflow as soon as the file arrives. By enabling S3 data events in CloudTrail and writing a simple EventBridge rule that matches the PutObject event for your bucket, you can target your Glue workflow's trigger directly, eliminating continuous polling or extra Lambda or DMS infrastructure.

• Question 367:

  A company wants to ingest streaming data into an Amazon Redshift data warehouse from an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster. A data engineer needs to develop a solution that provides low data access time and that optimizes storage costs.

  Which solution will meet these requirements with the LEAST operational overhead?

  A. Create an external schema that maps to the MSK cluster. Create a materialized view that references the external schema to consume the streaming data from the MSK topic.
  B. Develop an AWS Glue streaming extract, transform, and load (ETL) job to process the incoming data from Amazon MSK. Load the data into Amazon S3. Use Amazon Redshift Spectrum to read the data from Amazon S3.
  C. Create an external schema that maps to the streaming data source. Create a new Amazon Redshift table that references the external schema.
  D. Create an Amazon S3 bucket. Ingest the data from Amazon MSK. Create an event-driven AWS Lambda function to load the data from the S3 bucket to a new Amazon Redshift table.
  B. Develop an AWS Glue streaming extract, transform, and load (ETL) job to process the incoming data from Amazon MSK. Load the data into Amazon S3. Use Amazon Redshift Spectrum to read the data from Amazon S3.

  ---

  Explanation

  By using a serverless Glue streaming job to continuously pull your MSK records, transform them as needed, and land them in Parquet (or another columnar) files in S3, you:

  1. Optimize storage costs ?your data sits in S3, where you pay pennies per GB-month and can tier it further with lifecycle rules.

  2. Get low-latency access ?Redshift Spectrum lets you query S3-backed tables with millisecond planning time, so freshly landed data becomes queryable almost immediately.

  3. Minimize ops overhead ?you don't have to stand up or manage any EC2-based brokers, Lambda polling loops, or custom connector infrastructure. Glue's managed streaming runtime handles checkpointing, autoscaling, and fault tolerance for you.

  Once the data lands in S3, you simply define an external schema in Redshift that points at the Glue Data Catalog database where your streaming job writes tables. Analysts can then query the "live" dataset via Spectrum as if it were inside Redshift, meeting both your performance and cost goals with minimal operational effort.

• Question 368:

  A company built a data lake and a data warehouse on AWS. The company wants to implement a data catalog to enhance the current data storage solutions. The company wants to have the capability to add busine metadata and gloary information to the data catalog for every aet.

  Which solution will meet these requirements with the LEAST operational overhead?

  A. Use AWS Glue Catalog. Create a user table for the busine gloary. Use the AWS Glue API to change table properties to add busine metadata. Create a web application to acce the metadata.
  B. Use an Apache Hive metastore. Create a user table for the busine gloary. Use the ALTER TABLE command to change table properties to add busine metadata. Create a web application to acce the metadata.
  C. Use Amazon DataZone. Create the busine gloaries. Create metadata forms. Use the Amazon DataZone data portal to acce the metadata.
  D. Use Amazon OpenSearch Service. Create an index for the busine gloary. Create a second index for the busine metadata. Use the OpenSearch Service dashboard to acce the metadata.
  C. Use Amazon DataZone. Create the busine gloaries. Create metadata forms. Use the Amazon DataZone data portal to acce the metadata.

  ---

  Explanation

  Amazon DataZone provides a fully managed data catalog and governance service with built-in support for busine gloaries, custom metadata forms, and a user-friendly data portal. You can onboard your existing data aets, define gloaries, and capture busine metadata without building or maintaining custom tables, APIs, or applications, minimizing operational overhead.

• Question 369:

  A data engineer configured an AWS Glue Data Catalog for data that is stored in Amazon S3 buckets. The data engineer needs to configure the Data Catalog to receive incremental updates.

  The data engineer sets up event notifications for the S3 bucket and creates an Amazon Simple Queue Service (Amazon SQS) queue to receive the S3 events.

  Which combination of steps should the data engineer take to meet these requirements with LEAST operational overhead? (Choose Two.)

  A. Create an S3 event-based AWS Glue crawler to consume events from the SQS queue.
  B. Define a time-based schedule to run the AWS Glue crawler, and perform incremental updates to the Data Catalog.
  C. Use an AWS Lambda function to directly update the Data Catalog based on S3 events that the SQS queue receives.
  D. Manually initiate the AWS Glue crawler to perform updates to the Data Catalog when there is a change in the S3 bucket.
  E. Use AWS Step Functions to orchestrate the process of updating the Data Catalog based on 53 events that the SQS queue receives.
  A. Create an S3 event-based AWS Glue crawler to consume events from the SQS queue.
  C. Use an AWS Lambda function to directly update the Data Catalog based on S3 events that the SQS queue receives.

  ---

  Explanation

  The requirement is to update the AWS Glue Data Catalog incrementally based on S3 events. Using an S3 event-based approach is the most automated and operationally efficient solution.

  A. Create an S3 event-based AWS Glue crawler: An event-based Glue crawler can automatically update the Data Catalog when new data arrives in the S3 bucket. This ensures incremental updates with minimal operational overhead.

• Question 370:

  A media company uses software as a service (SaaS) applications to gather data by using third-party tools.

  The company needs to store the data in an Amazon S3 bucket. The company will use Amazon Redshift to perform analytics based on the data.

  Which AWS service or feature will meet these requirements with the LEAST operational overhead?

  A. Amazon Managed Streaming for Apache Kafka (Amazon MSK)
  B. Amazon AppFlow
  C. AWS Glue Data Catalog
  D. Amazon Kinesis
  B. Amazon AppFlow

  ---

  Explanation

  Amazon AppFlow is a fully managed integration service that enables you to securely transfer data between SaaS applications and AWS services like Amazon S3 and Amazon Redshift. Amazon AppFlow supports many SaaS applications as data sources and targets, and allows you to configure data flows with a few clicks. Amazon AppFlow also provides features such as data transformation, filtering, validation, and encryption to prepare and protect your data. Amazon AppFlow meets the requirements of the media company with the least operational overhead, as it eliminates the need to write code, manage infrastructure, or monitor data pipelines.