Amazon DATA-ENGINEER-ASSOCIATE Online Practice Questions and Exam Preparation

Amazon DATA-ENGINEER-ASSOCIATE Online Questions & Answers

• Question 111:

  A company is migrating an on-premises PostgreSQL database to Amazon Redshift. The target must receive the existing rows first and then continue to receive source database changes until cutover.

  Which AWS service and task type should the data engineer use?

  A. AWS Database Migration Service with a full-load-and-CDC migration task
  B. AWS Glue crawler with a daily schedule
  C. AWS DataSync with a task that transfers changed files
  D. Amazon AppFlow with a scheduled flow from PostgreSQL to Redshift
  A. AWS Database Migration Service with a full-load-and-CDC migration task

  ---

  Explanation

  AWS DMS full load plus change data capture migrates existing data and then applies ongoing changes from database logs to the target. A Glue crawler catalogs metadata but does not replicate database changes. DataSync transfers files and file-system data, not relational change logs. AppFlow is not the standard service for continuous database migration into Redshift.

• Question 112:

  A gaming company uses a NoSQL database to store customer information. The company is planning to migrate to AWS.

  The company needs a fully managed AWS solution that will handle high online transaction processing (OLTP) workload, provide single-digit millisecond performance, and provide high availability around the world.

  Which solution will meet these requirements with the LEAST operational overhead?

  A. Amazon Keyspaces (for Apache Cassandra)
  B. Amazon DocumentDB (with MongoDB compatibility)
  C. Amazon DynamoDB
  D. Amazon Timestream
  C. Amazon DynamoDB

• Question 113:

  A company needs to build an extract, transform, and load (ETL) pipeline that has separate stages for batch data ingestion, transformation, and storage. The pipeline must store the transformed data in an Amazon

  S3 bucket. Each stage must automatically retry failures. The pipeline must provide visibility into the success or failure of individual stages.

  Which solution will meet these requirements with the LEAST operational overhead?

  A. Chain AWS Glue jobs that perform each stage together by using job triggers. Set the MaxRetries field to 0.
  B. Deploy AWS Step Functions workflows to orchestrate AWS Lambda functions that ingest data. Use AWS Glue jobs to transform the data and store the data in the S3 bucket.
  C. Build an Amazon EventBridge based pipeline that invokes AWS Lambda functions to perform each stage.
  D. Schedule Apache Airflow directed acyclic graphs (DAGs) on Amazon Manages Workflows tor Apache Airflow (Amazon MWAA) to orchestrate pipeline steps. Use Amazon Simple Queue Service (Amazon SQS) to ingest data. Use AWS Glue jobs to transform data and store the data in the S3 bucket.
  B. Deploy AWS Step Functions workflows to orchestrate AWS Lambda functions that ingest data. Use AWS Glue jobs to transform the data and store the data in the S3 bucket.

• Question 114:

  A data engineer is configuring an AWS Glue Apache Spark extract, transform, and load (ETL) job. The job contains a sort-merge join of two large and equally sized DataFrames. The job is failing with the following error: No space left on device.

  Which solution will resolve the error?

  A. Use the AWS Glue Spark shuffle manager.
  B. Deploy are Amazon Elastic Block Store (Amazon EBS) volume for the job to use.
  C. Convert the sort-merge join in the job to be a broadcast join.
  D. Convert the DataFrames to DynamicFrames, and perform a DynamicFrame join in the job.
  A. Use the AWS Glue Spark shuffle manager.

  ---

  Explanation

  The error "No space left on device" during a sort-merge join typically results from insufficient disk space during the shuffle phase of a Spark job. Using the AWS Glue Spark shuffle manager (optimized shuffle manager) helps reduce disk usage and improves shuffle performance by optimizing how intermediate data is written and read. This directly addresses the issue without requiring job logic changes or manual resource management.

• Question 115:

  A company has an application that uses a microservice architecture. The company hosts the application on an Amazon Elastic Kubernetes Services (Amazon EKS) cluster.

  The company wants to set up a robust monitoring system for the application. The company needs to analyze the logs from the EKS cluster and the application. The company needs to correlate the cluster's logs with the application's traces to identify points of failure in the whole application request flow.

  Which combination of steps will meet these requirements with the LEAST development effort? (Choose Two.)

  A. Use FluentBit to collect logs. Use OpenTelemetry to collect traces.
  B. Use Amazon CloudWatch to collect logs. Use Amazon Kinesis to collect traces.
  C. Use Amazon CloudWatch to collect logs. Use Amazon Managed Streaming for Apache Kafka (Amazon MSK) to collect traces.
  D. Use Amazon OpenSearch to correlate the logs and traces.
  E. Use AWS Glue to correlate the logs and traces.
  A. Use FluentBit to collect logs. Use OpenTelemetry to collect traces.
  D. Use Amazon OpenSearch to correlate the logs and traces.

  ---

  Explanation

  Step 1: Log Collection (FluentBit and CloudWatch)

  Option Asuggests using FluentBit to collect logs and OpenTelemetry to collect traces.

  FluentBitis a lightweight log processor that integrates with Amazon EKS to collect and forward logs from Kubernetes clusters. It is widely used with minimal overhead, making it an ideal choice for log collection in this scenario. FluentBit is also natively compatible with AWS services.

  OpenTelemetryis a popular framework to collect traces from distributed applications. It provides observability, making it easier to monitor microservices. This combination allows you to effectively gather both logs and traces with minimal setup and configuration, aligning with the goal of least development effort. CloudWatchcan be used to monitor logs (Option B and C). However, for applications that need more custom and fine-grained control over logging mechanisms, FluentBit and OpenTelemetry are the preferred choice in microservice environments.

  Step 2: Log and Trace Correlation (Amazon OpenSearch)

  Option D(Amazon OpenSearch) is specifically designed to search, analyze, and visualize logs, metrics, and traces in real-time. OpenSearch allows you to correlate logs and traces effectively. With Amazon OpenSearch , you can set up dashboards that help in visualizing both logs and traces together, which assists in identifying any failure points across the entire request flow. It offers integrations with FluentBit and OpenTelemetry , ensuring that both logs from the EKS cluster and application traces are centrally collected, stored, and correlated without additional heavy development.

  Step 3: Why Other Options Are Not Suitable

  Option B (Amazon Kinesis)is designed for real-time data streaming and analytics but is not as well-suited for tracing microservice requests and logs correlation compared to OpenSearch.

  Option C (Amazon MSK) provides a managed Kafka streaming service, but this adds complexity when trying to integrate and correlate logs and traces from a microservice environment. Setting up Kafka requires more development effort compared to using FluentBit and OpenTelemetry.

  Option E (AWS Glue)is primarily an ETL (Extract, Transform, Load) service. While Glue is powerful for data processing, it is not a native tool for log and trace correlation, and using it would add unnecessary complexity for this use case.

  Conclusion:

  To meet the requirements with the least development effort: Use FluentBit for log collection and OpenTelemetry for tracing (Option A).

  Correlate logs and traces using Amazon OpenSearch (Option D).

  This approach leverages AWS-native services designed for seamless integration with microservices hosted on Amazon EKS and ensures effective monitoring with minimal overhead.

• Question 116:

  An ecommerce company operates a complex order ful lment process that spans several operational systems hosted in AWS. Each of the operational systems has a Java Database Connectivity (JDBC)- compliant relational database where the latest processing state is captured.

  The company needs to give an operations team the ability to track orders on an hourly basis across the entire ful llment process.

  Which solution will meet these requirements with the LEAST development overhead?

  A. Use AWS Glue to build ingestion pipelines from the operational systems into Amazon Redshift Build dashboards in Amazon QuickSight that track the orders.
  B. Use AWS Glue to build ingestion pipelines from the operational systems into Amazon DynamoDBuild dashboards in Amazon QuickSight that track the orders.
  C. Use AWS Database Migration Service (AWS DMS) to capture changed records in the operational systems. Publish the changes to an Amazon DynamoDB table in a different AWS region from the source database. Build Grafana dashboards that track the orders.
  D. Use AWS Database Migration Service (AWS DMS) to capture changed records in the operational systems. Publish the changes to an Amazon DynamoDB table in a different AWS region from the source database. Build Amazon QuickSight dashboards that track the orders.
  A. Use AWS Glue to build ingestion pipelines from the operational systems into Amazon Redshift Build dashboards in Amazon QuickSight that track the orders.

  ---

  Explanation

• Question 117:

  A company uses a data lake that is based on an Amazon S3 bucket. To comply with regulations, the company must apply two layers of server-side encryption to les that are uploaded to the S3 bucket. The company wants to use an AWS Lambda function to apply the necessary encryption.

  Which solution will meet these requirements?

  A. Use both server-side encryption with AWS KMS keys (SSE-KMS) and the Amazon S3 Encryption Client.
  B. Use dual-layer server-side encryption with AWS KMS keys (DSSE-KMS).
  C. Use server-side encryption with customer-provided keys (SSE-C) before les are uploaded.
  D. Use server-side encryption with AWS KMS keys (SSE-KMS).
  B. Use dual-layer server-side encryption with AWS KMS keys (DSSE-KMS).

• Question 118:

  A company stores its processed data in an S3 bucket. The company has a strict data access policy. The company uses IAM roles to grant teams within the company different levels of access to the S3 bucket.

  The company wants to receive notifications when a user violates the data access policy. Each notification must include the username of the user who violated the policy.

  Which solution will meet these requirements?

  A. Use AWS Config rules to detect violations of the data access policy. Set up compliance alarms.
  B. Use Amazon CloudWatch metrics to gather object-level metrics. Set up CloudWatch alarms.
  C. Use AWS CloudTrail to track object-level events for the S3 bucket. Forward events to Amazon CloudWatch to set up CloudWatch alarms.
  D. Use Amazon S3 server access logs to monitor access to the bucket. Forward the access logs to an Amazon CloudWatch log group. Use metric filters on the log group to set up CloudWatch alarms.
  C. Use AWS CloudTrail to track object-level events for the S3 bucket. Forward events to Amazon CloudWatch to set up CloudWatch alarms.

  ---

  Explanation

  The requirement is to detect violations of data access policies and receive notifications with the username of the violator. AWS CloudTrail can provide object-level tracking for S3 to capture detailed API actions on specific S3 objects, including the user who performed the action.

  AWS CloudTrail:

  CloudTrail can monitor API calls made to an S3 bucket, including object-level API actions such as GetObject, PutObject, and DeleteObject. This will help detect access violations based on the API calls made by different users. CloudTrail logs include details such as the user identity , which is essential for meeting the requirement of including the username in notifications. The CloudTrail logs can be forwarded to Amazon CloudWatch to trigger alarms based on certain access patterns (e.g., violations of specific policies).

• Question 119:

  A company has multiple applications that use datasets that are stored in an Amazon S3 bucket. The company has an ecommerce application that generates a dataset that contains personally identifiable information (PII). The company has an internal analytics application that does not require access to the PII.

  To comply with regulations, the company must not share PII unnecessarily. A data engineer needs to implement a solution that with redact PII dynamically, based on the needs of each application that accesses the dataset.

  Which solution will meet the requirements with the LEAST operational overhead?

  A. Create an S3 bucket policy to limit the access each application has. Create multiple copies of the dataset. Give each dataset copy the appropriate level of redaction for the needs of the application that accesses the copy.
  B. Create an S3 Object Lambda endpoint. Use the S3 Object Lambda endpoint to read data from the S3 bucket. Implement redaction logic within an S3 Object Lambda function to dynamically redact PII based on the needs of each application that accesses the data.
  C. Use AWS Glue to transform the data for each application. Create multiple copies of the dataset. Give each dataset copy the appropriate level of redaction for the needs of the application that accesses the copy.
  D. Create an API Gateway endpoint that has custom authorizers. Use the API Gateway endpoint to read data from the S3 bucket. Initiate a REST API call to dynamically redact PII based on the needs of each application that accesses the data.
  B. Create an S3 Object Lambda endpoint. Use the S3 Object Lambda endpoint to read data from the S3 bucket. Implement redaction logic within an S3 Object Lambda function to dynamically redact PII based on the needs of each application that accesses the data.

  ---

  Explanation

  Option B is the best solution to meet the requirements with the least operational overhead because S3 Object Lambda is a feature that allows you to add your own code to process data retrieved from S3 before returning it to an application. S3 Object Lambda works with S3 GET requests and can modify both the object metadata and the object data. By using S3 Object Lambda, you can implement redaction logic within an S3 Object Lambda function to dynamically redact PII based on the needs of each application that accesses the data. This way, you can avoid creating and maintaining multiple copies of the dataset with different levels of redaction.

  Option A is not a good solution because it involves creating and managing multiple copies of the dataset with different levels of redaction for each application. This option adds complexity and storage cost to the data protection process and requires additional resources and configuration. Moreover, S3 bucket policies cannot enforce fine-grained data access control at the row and column level, so they are not sufficient to redact PII.

  Option C is not a good solution because it involves using AWS Glue to transform the data for each application. AWS Glue is a fully managed service that can extract, transform, and load (ETL) data from various sources to various destinations, including S3. AWS Glue can also convert data to different formats, such as Parquet, which is a columnar storage format that is optimized for analytics. However, in this scenario, using AWS Glue to redact PII is not the best option because it requires creating and maintaining multiple copies of the dataset with different levels of redaction for each application. This option also adds extra time and cost to the data protection process and requires additional resources and configuration.

  Option D is not a good solution because it involves creating and configuring an API Gateway endpoint that has custom authorizers. API Gateway is a service that allows you to create, publish, maintain, monitor, and secure APIs at any scale. API Gateway can also integrate with other AWS services, such as Lambda, to provide custom logic for processing requests. However, in this scenario, using API Gateway to redact PII is not the best option because it requires writing and maintaining custom code and configuration for the API endpoint, the custom authorizers, and the REST API call. This option also adds complexity and latency to the data protection process and requires additional resources and configuration.

• Question 120:

  A data engineer needs to use an Amazon QuickSight dashboard that is based on Amazon Athena queries on data that is stored in an Amazon S3 bucket. When the data engineer connects to the QuickSight dashboard, the data engineer receives an error message that indicates insufficient permissions.

  Which factors could cause to the permissions-related errors? (Choose two.)

  A. There is no connection between QuickSgqht and Athena.
  B. The Athena tables are not cataloged.
  C. QuickSiqht does not have access to the S3 bucket.
  D. QuickSight does not have access to decrypt S3 data.
  E. There is no IAM role assigned to QuickSight.
  C. QuickSiqht does not have access to the S3 bucket.
  D. QuickSight does not have access to decrypt S3 data.

  ---

  Explanation

  QuickSight does not have access to the S3 bucket and QuickSight does not have access to decrypt S3 data are two possible factors that could cause the permissions-related errors. Amazon QuickSight is a business intelligence service that allows you to create and share interactive dashboards based on various data sources, including Amazon Athena. Amazon Athena is a serverless query service that allows you to analyze data stored in Amazon S3 using standard SQL. To use an Amazon QuickSight dashboard that is based on Amazon Athena queries on data that is stored in an Amazon S3 bucket, you need to grant QuickSight access to both Athena and S3, as well as any encryption keys that are used to encrypt the S3 data. If QuickSight does not have access to the S3 bucket or the encryption keys, it will not be able to read the data from Athena and display it on the dashboard, resulting in an error message that indicates insufficient permissions.

  The other options are not factors that could cause the permissions-related errors. Option A, there is no connection between QuickSight and Athena, is not a factor, as QuickSight supports Athena as a native data source, and you can easily create a connection between them using the QuickSight console or the API. Option B, the Athena tables are not cataloged, is not a factor, as QuickSight can automatically discover the Athena tables that are cataloged in the AWS Glue Data Catalog, and you can also manually specify the Athena tables that are not cataloged. Option E, there is no IAM role assigned to QuickSight, is not a factor, as QuickSight requires an IAM role to access any AWS data sources, including Athena and S3, and you can create and assign an IAM role to QuickSight using the QuickSight console or the API.

  References:

  Using Amazon Athena as a Data Source Granting Amazon QuickSight Access to AWS Resources Encrypting Data at Rest in Amazon S3