Google Google Certifications PROFESSIONAL-DATA-ENGINEER Questions & Answers

• Question 301:

  You are designing a data mesh on Google Cloud with multiple distinct data engineering teams building data products. The typical data curation design pattern consists of landing files in Cloud Storage, transforming raw data in Cloud Storage and BigQuery datasets. and storing the final curated data product in BigQuery datasets You need to configure Dataplex to ensure that each team can access only the assets needed to build their data products. You also need to ensure that teams can easily share the curated data product. What should you do?

  A. 1 Create a single Dataplex virtual lake and create a single zone to contain landing, raw.and curated data. 2 Provide each data engineering team access to the virtual lake.

  B. 1 Create a single Dataplex virtual lake and create a single zone to contain landing, raw.and curated data. 2 Build separate assets for each data product within the zone.

  3. Assign permissions to the data engineering teams at the zone level.

  C. 1 Create a Dataplex virtual lake for each data product, and create a single zone to contain landing, raw, and curated data.

  2. Provide the data engineering teams with full access to the virtual lake assigned to their data product.

  D. 1 Create a Dataplex virtual lake for each data product, and create multiple zones for landing, raw. and curated data.

  2. Provide the data engineering teams with full access to the virtual lake assigned to their data product.

  Correct Answer: D

  This option is the best way to configure Dataplex for a data mesh architecture, as it allows each data engineering team to have full ownership and control over their data products, while also enabling easy discovery and sharing of the curated data across the organization12. By creating a Dataplex virtual lake for each data product, you can isolate the data assets and resources for each domain, and avoid conflicts and dependencies between different teams3. By creating multiple zones for landing, raw, and curated data, you can enforce different security and governance policies for each stage of the data curation process, and ensure that only authorized users can access the data assets45. By providing the data engineering teams with full access to the virtual lake assigned to their data product, you can empower them to manage and monitor their data products, and leverage the Dataplex features such as tagging, quality, and lineage. Option A is not suitable, as it creates a single point of failure and a bottleneck for the data mesh, and does not allow for fine-grained access control and governance for different data products2. Option B is also not suitable, as it does not isolate the data assets and resources for each data product, and assigns permissions at the zone level, which may not reflect the different roles and responsibilities of the data engineering teams34. Option C is better than option A and B, but it does not create multiple zones for landing, raw, and curated data, which may compromise the security and quality of the data products5. References:

  1: Building a data mesh on Google Cloud using BigQuery and Dataplex | Google Cloud Blog

  2: Data Mesh - 7 Effective Practices to Get Started - Confluent

  3: Best practices | Dataplex | Google Cloud

  4: Secure your lake | Dataplex | Google Cloud

  5: Zones | Dataplex | Google Cloud [6]: Managing a Data Mesh with Dataplex ?ROI Training

• Question 302:

  Your company is implementing a data warehouse using BigQuery, and you have been tasked with designing the data model You move your on-premises sales data warehouse with a star data schema to BigQuery but notice performance issues when querying the data of the past 30 days Based on Google's recommended practices, what should you do to speed up the query without increasing storage costs?

  A. Denormalize the data

  B. Shard the data by customer ID

  C. Materialize the dimensional data in views

  D. Partition the data by transaction date

  Correct Answer: C

• Question 303:

  You have an Apache Kafka Cluster on-prem with topics containing web application logs. You need to replicate the data to Google Cloud for analysis in BigQuery and Cloud Storage. The preferred replication method is mirroring to avoid deployment of Kafka Connect plugins.

  What should you do?

  A. Deploy a Kafka cluster on GCE VM Instances. Configure your on-prem cluster to mirror your topics to the cluster running in GCE. Use a Dataproc cluster or Dataflow job to read from Kafka and write to GCS.

  B. Deploy a Kafka cluster on GCE VM Instances with the PubSub Kafka connector configured as a Sink connector. Use a Dataproc cluster or Dataflow job to read from Kafka and write to GCS.

  C. Deploy the PubSub Kafka connector to your on-prem Kafka cluster and configure PubSub as a Source connector. Use a Dataflow job to read fron PubSub and write to GCS.

  D. Deploy the PubSub Kafka connector to your on-prem Kafka cluster and configure PubSub as a Sink connector. Use a Dataflow job to read fron PubSub and write to GCS.

  Correct Answer: A

• Question 304:

  You are designing a pipeline that publishes application events to a Pub/Sub topic. You need to aggregate events across hourly intervals before loading the results to BigQuery for analysis. Your solution must be scalable so it can process and load large volumes of events to BigQuery. What should you do?

  A. Create a streaming Dataflow job to continually read from the Pub/Sub topic and perform the necessary aggregations using tumbling windows

  B. Schedule a batch Dataflow job to run hourly, pulling all available messages from the Pub-Sub topic and performing the necessary aggregations

  C. Schedule a Cloud Function to run hourly, pulling all avertable messages from the Pub/Sub topic and performing the necessary aggregations

  D. Create a Cloud Function to perform the necessary data processing that executes using the Pub/Sub trigger every time a new message is published to the topic.

  Correct Answer: A

• Question 305:

  You are designing a Dataflow pipeline for a batch processing job. You want to mitigate multiple zonal failures at job submission time. What should you do?

  A. Specify a worker region by using the --region flag.

  B. Set the pipeline staging location as a regional Cloud Storage bucket.

  C. Submit duplicate pipelines in two different zones by using the --zone flag.

  D. Create an Eventarc trigger to resubmit the job in case of zonal failure when submitting the job.

  Correct Answer: A

  By specifying a worker region, you can run your Dataflow pipeline in a multi- zone or multi-region configuration, which provides higher availability and resilience in case of zonal failures1. The --region flag allows you to specify the regional endpoint for your pipeline, which determines the location of the Dataflow service and the default location of the Compute Engine resources1. If you do not specify a zone by using the --zone flag, Dataflow automatically selects a zone within the region for your job workers1. This option is recommended over submitting duplicate pipelines in two different zones, which would incur additional costs and complexity. Setting the pipeline staging location as a regional Cloud Storage bucket does not affect the availability of your pipeline, as the staging location only stores the pipeline code and dependencies2. Creating an Eventarc trigger to resubmit the job in case of zonal failure is not a reliable solution, as it depends on the availability of the Eventarc service and the zonal resources at the time of resubmission. References:

  1: Pipeline troubleshooting and debugging | Cloud Dataflow | Google Cloud

  3: Regional endpoints | Cloud Dataflow | Google Cloud

• Question 306:

  You are loading CSV files from Cloud Storage to BigQuery. The files have known data quality issues, including mismatched data types, such as STRINGS and INT64s in the same column, and inconsistent formatting of values such as phone numbers or addresses. You need to create the data pipeline to maintain data quality and perform the required cleansing and transformation. What should you do?

  A. Use Data Fusion to transform the data before loading it into BigQuery.

  B. Load the CSV files into a staging table with the desired schema, perform the transformations with SQL. and then write the results to the final destination table.

  C. Create a table with the desired schema, toad the CSV files into the table, and perform the transformations in place using SQL.

  D. Use Data Fusion to convert the CSV files lo a self-describing data formal, such as AVRO. before loading the data to BigOuery.

  Correct Answer: A

  Data Fusion's advantages:

  Visual interface: Offers a user-friendly interface for designing data pipelines without extensive coding, making it accessible to a wider range of users. Built-in transformations: Includes a wide range of pre-built transformations to handle

  common data quality issues, such as:

  Data type conversions

  Data cleansing (e.g., removing invalid characters, correcting formatting) Data validation (e.g., checking for missing values, enforcing constraints) Data enrichment (e.g., adding derived fields, joining with other datasets) Custom

  transformations: Allows for custom transformations using SQL or Java code for more complex cleaning tasks.

  Scalability: Can handle large datasets efficiently, making it suitable for processing CSV files with potential data quality issues.

  Integration with BigQuery: Integrates seamlessly with BigQuery, allowing for direct loading of transformed data.

• Question 307:

  You have a data processing application that runs on Google Kubernetes Engine (GKE). Containers need to be launched with their latest available configurations from a container registry. Your GKE nodes need to have GPUs. local SSDs, and 8 Gbps bandwidth. You want to efficiently provision the data processing infrastructure and manage the deployment process. What should you do?

  A. Use Compute Engi.no startup scriots to pull container Images, and use gloud commands to provision the infrastructure.

  B. Use GKE to autoscale containers, and use gloud commands to provision the infrastructure.

  C. Use Cloud Build to schedule a job using Terraform build to provision the infrastructure and launch with the most current container images.

  D. Use Dataflow to provision the data pipeline, and use Cloud Scheduler to run the job.

  Correct Answer: C

  https://cloud.google.com/architecture/managing-infrastructure-as-code

• Question 308:

  You are troubleshooting your Dataflow pipeline that processes data from Cloud Storage to BigQuery. You have discovered that the Dataflow worker nodes cannot communicate with one another Your networking team relies on Google Cloud network tags to define firewall rules You need to identify the issue while following Google-recommended networking security practices. What should you do?

  A. Determine whether your Dataflow pipeline has a custom network tag set.

  B. Determine whether there is a firewall rule set to allow traffic on TCP ports 12345 and 12346 for the Dataflow network tag.

  C. Determine whether your Dataflow pipeline is deployed with the external IP address option enabled.

  D. Determine whether there is a firewall rule set to allow traffic on TCP ports 12345 and 12346 on the subnet used by Dataflow workers.

  Correct Answer: B

  Dataflow worker nodes need to communicate with each other and with the Dataflow service on TCP ports 12345 and 12346. These ports are used for data shuffling and streaming engine communication. By default, Dataflow assigns a network tag called dataflow to the worker nodes, and creates a firewall rule that allows traffic on these ports for the dataflow network tag. However, if you use a custom network tag for your Dataflow pipeline, you need to create a firewall rule that allows traffic on these ports for your custom network tag. Otherwise, the worker nodes will not be able to communicate with each other and the Dataflow service, and the pipeline will fail. Therefore, the best way to identify the issue is to determine whether there is a firewall rule set to allow traffic on TCP ports 12345 and 12346 for the Dataflow network tag. If there is no such firewall rule, or if the firewall rule does not match the network tag used by your Dataflow pipeline, you need to create or update the firewall rule accordingly. Option A is not a good solution, as determining whether your Dataflow pipeline has a custom network tag set does not tell you whether there is a firewall rule that allows traffic on the required ports for that network tag. You need to check the firewall rule as well. Option C is not a good solution, as determining whether your Dataflow pipeline is deployed with the external IP address option enabled does not tell you whether there is a firewall rule that allows traffic on the required ports for the Dataflow network tag. The external IP address option determines whether the worker nodes can access resources on the public internet, but it does not affect the internal communication between the worker nodes and the Dataflow service. Option D is not a good solution, as determining whether there is a firewall rule set to allow traffic on TCP ports 12345 and 12346 on the subnet used by Dataflow workers does not tell you whether the firewall rule applies to the Dataflow network tag. The firewall rule should be based on the network tag, not the subnet, as the network tag is more specific and secure. References: Dataflow network tags | Cloud Dataflow | Google Cloud, Dataflow firewall rules | Cloud Dataflow | Google Cloud, Dataflow network configuration | Cloud Dataflow | Google Cloud, Dataflow Streaming Engine | Cloud Dataflow | Google Cloud.

• Question 309:

  You have several Spark jobs that run on a Cloud Dataproc cluster on a schedule. Some of the jobs run in sequence, and some of the jobs run concurrently. You need to automate this process. What should you do?

  A. Create a Cloud Dataproc Workflow Template

  B. Create an initialization action to execute the jobs

  C. Create a Directed Acyclic Graph in Cloud Composer

  D. Create a Bash script that uses the Cloud SDK to create a cluster, execute jobs, and then tear down the cluster

  Correct Answer: C

• Question 310:

  You need to modernize your existing on-premises data strategy. Your organization currently uses.

  1.

  Apache Hadoop clusters for processing multiple large data sets, including on-premises Hadoop Distributed File System (HDFS) for data replication.

  2.

  Apache Airflow to orchestrate hundreds of ETL pipelines with thousands of job steps.

  You need to set up a new architecture in Google Cloud that can handle your Hadoop workloads and requires minimal changes to your existing orchestration processes. What should you do?

  A. Use Dataproc to migrate Hadoop clusters to Google Cloud, and Cloud Storage to handle any HDFS use cases Convert your ETL pipelines to Dataflow.

  B. Use Bigtable for your large workloads, with connections to Cloud Storage to handle any HDFS use cases Orchestrate your pipelines with Cloud Composer.

  C. Use Dataproc to migrate your Hadoop clusters to Google Cloud, and Cloud Storage to handle any HDFS use cases. Use Cloud Data Fusion to visually design and deploy your ETL pipelines.

  D. Use Dataproc to migrate Hadoop clusters to Google Cloud, and Cloud Storage to handle any HDFS use cases. Orchestrate your pipelines with Cloud Composer..

  Correct Answer: D

  Dataproc is a fully managed service that allows you to run Apache Hadoop and Spark workloads on Google Cloud. It is compatible with the open source ecosystem, so you can migrate your existing Hadoop clusters to Dataproc with minimal changes. Cloud Storage is a scalable, durable, and cost-effective object storage service that can replace HDFS for storing and accessing data. Cloud Storage offers interoperability with Hadoop through connectors, so you can use it as a data source or sink for your Dataproc jobs. Cloud Composer is a fully managed service that allows you to create, schedule, and monitor workflows using Apache Airflow. It is integrated with Google Cloud services, such as Dataproc, BigQuery, Dataflow, and Pub/Sub, so you can orchestrate your ETL pipelines across different platforms. Cloud Composer is compatible with your existing Airflow code, so you can migrate your existing orchestration processes to Cloud Composer with minimal changes. The other options are not as suitable as Dataproc and Cloud Composer for this use case, because they either require more changes to your existing code, or do not meet your requirements. Dataflow is a fully managed service that allows you to create and run scalable data processing pipelines using Apache Beam. However, Dataflow is not compatible with your existing Hadoop code, so you would need to rewrite your ETL pipelines using Beam. Bigtable is a fully managed NoSQL database service that can handle large and complex data sets. However, Bigtable is not compatible with your existing Hadoop code, so you would need to rewrite your queries and applications using Bigtable APIs. Cloud Data Fusion is a fully managed service that allows you to visually design and deploy data integration pipelines using a graphical interface. However, Cloud Data Fusion is not compatible with your existing Airflow code, so you would need to recreate your orchestration processes using Cloud Data Fusion UI. References: Dataproc overview Cloud Storage connector for Hadoop Cloud Composer overview