BenchMark Quizzes by MDFT Pro

Attach Notebook to Same Spark Session

You work for MDFT Pro, a well-known training agency that uses notebooks for data processing and analysis. Claire, a Data Engineering Lead at MDFT Pro, manages a Fabric workspace called AnalyticsHub containing multiple notebooks for processing student enrollment data. She has an existing notebook called EnrollmentProcessor that performs daily student data transformations and has an active Apache Spark session running. To improve development efficiency and reduce costs, Claire wants to create a new notebook called EnrollmentValidator that can attach to the same Spark session as EnrollmentProcessor, eliminating the 30-60 second wait time for starting a new session and avoiding duplicate resource consumption. Both notebooks will run under Claire’s user account, use the same default lakehouse (StudentData), and share identical Spark compute configurations. The solution should provide near-instant session startup for the second notebook while maintaining proper isolation between notebook executions.

What should Claire do to ensure that EnrollmentValidator can attach to the same Apache Spark session as EnrollmentProcessor?

Choose the correct answer from the options below.

Explanations for each answer:

Enable high concurrency for notebooks is correct. Enabling high concurrency mode allows multiple notebooks to share the same Spark session. When high concurrency is enabled, notebooks run by the same user with the same default lakehouse and Spark configurations can attach to an existing session rather than starting a new one, providing near-instant session startup (~5 seconds) and reducing costs.
Enable dynamic allocation for the Spark pool is incorrect. Dynamic allocation controls how Spark executors are added or removed based on workload, but it doesn't enable session sharing between notebooks. Dynamic allocation helps with resource optimization within a session but doesn't allow multiple notebooks to attach to the same session.
Change the runtime version is incorrect. While notebooks must have the same runtime version to share a session, simply changing the runtime version doesn't enable session sharing. High concurrency mode must be enabled first, and notebooks must match on multiple criteria (user, lakehouse, configurations) to share sessions.
Increase the number of executors is incorrect. Increasing executors affects the computational capacity of a Spark session but doesn't enable session sharing. The number of executors determines how many parallel tasks can run within a session, not whether multiple notebooks can use the same session.

Learn more about high concurrency mode:

High Concurrency Mode

Practice Exam

Attach Notebook to Same Spark Session

Discuss this question on social media: