BenchMark Quizzes by MDFT Pro

Consolidate Parquet Files in Delta Table

You work for MDFT Pro, a well-known training agency that tracks student course activity in real-time. Claire, a Data Engineering Specialist at MDFT Pro, manages a Fabric lakehouse that contains a notebook named ActivityProcessor and a table named StudentActivityLog. The notebook reads streaming data about student course interactions into a DataFrame from StudentActivityLog, applies transformation logic to calculate engagement metrics and learning progress indicators, then writes the results to a new Delta table named StudentEngagementMetrics using a merge operation. This merge process runs every 15 minutes throughout the day, creating many small Parquet files in StudentActivityLog as new activity data continuously arrives from the learning management system. Claire has noticed that query performance against StudentActivityLog has degraded due to the accumulation of small files. She needs to consolidate the underlying Parquet files in StudentActivityLog to improve read performance for downstream analytics and reporting.

Which command should Claire run to consolidate the underlying Parquet files in StudentActivityLog?

Choose the correct answer from the options below.

Explanations for each answer:

VACUUM is incorrect. VACUUM removes old files that are no longer referenced by the Delta table transaction log, cleaning up files that have been marked for deletion after retention periods expire. While VACUUM is important for storage management, it doesn't consolidate or merge small Parquet files into larger ones.
BROADCAST is incorrect. BROADCAST is a join hint used in Spark SQL to optimize query performance by broadcasting small tables to all executors during join operations. It's not related to file consolidation or Delta table maintenance tasks.
OPTIMIZE is correct. The OPTIMIZE command performs bin-compaction by merging multiple small Parquet files into larger, consolidated files (default target size is 1GB in Fabric). This is essential because reading many small files creates overhead and reduces performance, while larger files provide better data distribution and compression. OPTIMIZE can be run from the Lakehouse UI maintenance menu or through a notebook, and you can optionally apply V-Order during optimization to maximize read speeds in Fabric.
CACHE is incorrect. CACHE stores DataFrame data in memory to improve performance of repeated queries on the same data during a Spark session. It's a runtime optimization for in-memory processing and doesn't modify the underlying Parquet file structure or consolidate files on disk.

Learn more about Delta Lake table optimization and V-Order:

Delta Lake Table Optimization and V-Order

Practice Exam

Consolidate Parquet Files in Delta Table

Discuss this question on social media: