An organization processes customer data from web and mobile applications. Data includes names, emails phone numbers, and location history. Data arrives both as Batch files from an SFTP drop (daily) and Streaming JSON events from Kafka (real-time).
To comply with internal data privacy policies, the following requirements must be met: Personally identifiable information (PII) like email, phone_number, and ip_address must be masked or anonymized before storage Both batch and streaming pipelines must apply consistent PII handling Masking logic must be auditable and reproducible.
The masked data must still be usable for downstream analytics.
How should the data engineer design a compliant data pipeline on Databricks that supports both batch and streaming modes, applies data masking to PII, and maintains traceability of transformations for audits?