DATA-ENGINEER-ASSOCIATE Web TestEngine demo

Create an AWS Lambda function to check data quality and to raise exceptions in the code. Run the function when data is added to the S3 bucket. Create an Amazon CloudWatch alarm for exceptions in the code.

Create an AWS Glue DataBrew project for the data in the S3 bucket. Create a ruleset for the data quality rules.
Create a profile job to run the data quality rules. Use Amazon EventBridge to run the profile job when data is added to the S3 bucket.

Create an Amazon EMR provisioned cluster. Add a Python open source data quality package to the
EMR cluster.
Use the Python package to write code for data quality rules and to copy the data from the S3 bucket to the EMR cluster. Copy the data from the S3 bucket to the EMR cluster. Run the data quality rules.

Create AWS Lambda functions to evaluate data quality rules. Use AWS Step Functions to orchestrate a workflow that publishes notifications when the data fails to meet data quality rules.