A data organization leader is upset about the data analysis team’s reports being different from the data engineering team’s reports. The leader believes the siloed nature of their organization’s data engineering and data analysis architectures is to blame.
Which of the following describes how a data lakehouse could alleviate this issue?
- A . Both teams would autoscale their work as data size evolves
- B . Both teams would use the same source of truth for their work
- C . Both teams would reorganize to report to the same department
- D . Both teams would be able to collaborate on projects in real-time
- E . Both teams would respond more quickly to ad-hoc requests
B
Explanation:
A data lakehouse is designed to unify the data engineering and data analysis architectures by integrating features of both data lakes and data warehouses. One of the key benefits of a data lakehouse is that it provides a common, centralized data repository (the "lake") that serves as a single source of truth for data storage and analysis. This allows both data engineering and data analysis teams to work with the same consistent data sets, reducing discrepancies and ensuring that the reports generated by both teams are based on the same underlying data.
Which of the following describes a scenario in which a data team will want to utilize cluster pools?
- A . An automated report needs to be refreshed as quickly as possible.
- B . An automated report needs to be made reproducible.
- C . An automated report needs to be tested to identify errors.
- D . An automated report needs to be version-controlled across multiple collaborators.
- E . An automated report needs to be runnable by all stakeholders.
A
Explanation:
Cluster pools are typically used in distributed computing environments, such as cloud-based data platforms like Databricks. They allow you to pre-allocate a set of compute resources (a cluster) for specific tasks or workloads. In this case, if an automated report needs to be refreshed as quickly as possible, you can allocate a cluster pool with sufficient resources to ensure fast data processing and report generation. This helps ensure that the report is generated with minimal latency and can be delivered to stakeholders in a timely manner. Cluster pools allow you to optimize resource allocation for high-demand, time-sensitive tasks like real-time report generation.
Which of the following is hosted completely in the control plane of the classic Databricks architecture?
- A . Worker node
- B . JDBC data source
- C . Databricks web application
- D . Databricks Filesystem
- E . Driver node
C
Explanation:
In the classic Databricks architecture, the control plane includes components like the Databricks web application, the Databricks REST API, and the Databricks Workspace. These components are responsible for managing and controlling the Databricks environment, including cluster provisioning, notebook management, access control, and job scheduling. The other options, such as worker nodes, JDBC data sources, Databricks Filesystem (DBFS), and driver nodes, are typically part of the data plane or the execution environment, which is separate from the control plane. Worker nodes are responsible for executing tasks and computations, JDBC data sources are used to connect to external databases, DBFS is a distributed file system for data storage, and driver nodes are responsible for coordinating the execution of Spark jobs.
Which of the following benefits of using the Databricks Lakehouse Platform is provided by Delta Lake?
- A . The ability to manipulate the same data using a variety of languages
- B . The ability to collaborate in real time on a single notebook
- C . The ability to set up alerts for query failures
- D . The ability to support batch and streaming workloads
- E . The ability to distribute complex data operations
D
Explanation:
Delta Lake is a key component of the Databricks Lakehouse Platform that provides several benefits,
and one of the most significant benefits is its ability to support both batch and streaming workloads seamlessly. Delta Lake allows you to process and analyze data in real-time (streaming) as well as in batch, making it a versatile choice for various data processing needs. While the other options may be benefits or capabilities of Databricks or the Lakehouse Platform in general, they are not specifically associated with Delta Lake.
Which of the following describes the storage organization of a Delta table?
- A . Delta tables are stored in a single file that contains data, history, metadata, and other attributes.
- B . Delta tables store their data in a single file and all metadata in a collection of files in a separate location.
- C . Delta tables are stored in a collection of files that contain data, history, metadata, and other attributes.
- D . Delta tables are stored in a collection of files that contain only the data stored within the table.
- E . Delta tables are stored in a single file that contains only the data stored within the table.
C
Explanation:
Delta tables store data in a structured manner using Parquet files, and they also maintain metadata and transaction logs in separate directories. This organization allows for versioning, transactional capabilities, and metadata tracking in Delta Lake. Thank you for pointing out the error, and I appreciate your understanding.
Which of the following code blocks will remove the rows where the value in column age is greater than 25 from the existing Delta table my_table and save the updated table?
- A . SELECT * FROM my_table WHERE age > 25;
- B . UPDATE my_table WHERE age > 25;
- C . DELETE FROM my_table WHERE age > 25;
- D . UPDATE my_table WHERE age <= 25;
- E . DELETE FROM my_table WHERE age <= 25;
A data engineer has realized that they made a mistake when making a daily update to a table. They need to use Delta time travel to restore the table to a version that is 3 days old. However, when the data engineer attempts to time travel to the older version, they are unable to restore the data because the data files have been deleted.
Which of the following explains why the data files are no longer present?
- A . The VACUUM command was run on the table
- B . The TIME TRAVEL command was run on the table
- C . The DELETE HISTORY command was run on the table
- D . The OPTIMIZE command was nun on the table
- E . The HISTORY command was run on the table
A
Explanation:
The VACUUM command in Delta Lake is used to clean up and remove unnecessary data files that are no longer needed for time travel or query purposes. When you run VACUUM with certain retention settings, it can delete older data files, which might include versions of data that are older than the specified retention period. If the data engineer is unable to restore the table to a version that is 3 days old because the data files have been deleted, it’s likely because the VACUUM command was run on the table, removing the older data files as part of data cleanup.
Which of the following Git operations must be performed outside of Databricks Repos?
- A . Commit
- B . Pull
- C . Push
- D . Clone
- E . Merge
E
Explanation:
For following tasks, work in your Git provider:
Create a pull request.
Resolve merge conflicts.
Merge or delete branches.
Rebase a branch.
https://docs.databricks.com/repos/index.html
Which of the following data lakehouse features results in improved data quality over a traditional data lake?
- A . A data lakehouse provides storage solutions for structured and unstructured data.
- B . A data lakehouse supports ACID-compliant transactions.
- C . A data lakehouse allows the use of SQL queries to examine data.
- D . A data lakehouse stores data in open formats.
- E . A data lakehouse enables machine learning and artificial Intelligence workloads.
B
Explanation:
One of the key features of a data lakehouse that results in improved data quality over a traditional data lake is its support for ACID (Atomicity, Consistency, Isolation, Durability) transactions. ACID transactions provide data integrity and consistency guarantees, ensuring that operations on the data are reliable and that data is not left in an inconsistent state due to failures or concurrent access. In a traditional data lake, such transactional guarantees are often lacking, making it challenging to maintain data quality, especially in scenarios involving multiple data writes, updates, or complex transformations. A data lakehouse, by offering ACID compliance, helps maintain data quality by providing strong consistency and reliability, which is crucial for data pipelines and analytics.
A data engineer needs to determine whether to use the built-in Databricks Notebooks versioning or version their project using Databricks Repos.
Which of the following is an advantage of using Databricks Repos over the Databricks Notebooks versioning?
- A . Databricks Repos automatically saves development progress
- B . Databricks Repos supports the use of multiple branches
- C . Databricks Repos allows users to revert to previous versions of a notebook
- D . Databricks Repos provides the ability to comment on specific changes
- E . Databricks Repos is wholly housed within the Databricks Lakehouse Platform
B
Explanation:
An advantage of using Databricks Repos over the built-in Databricks Notebooks versioning is the ability to work with multiple branches. Branching is a fundamental feature of version control systems like Git, which Databricks Repos is built upon. It allows you to create separate branches for different tasks, features, or experiments within your project. This separation helps in parallel development and experimentation without affecting the main branch or the work of other team members. Branching provides a more organized and collaborative development environment, making it easier to merge changes and manage different development efforts. While Databricks Notebooks versioning also allows you to track versions of notebooks, it may not provide the same level of flexibility and collaboration as branching in Databricks Repos.
A data engineer has left the organization. The data team needs to transfer ownership of the data engineer’s Delta tables to a new data engineer. The new data engineer is the lead engineer on the data team.
Assuming the original data engineer no longer has access, which of the following individuals must be the one to transfer ownership of the Delta tables in Data Explorer?
- A . Databricks account representative
- B . This transfer is not possible
- C . Workspace administrator
- D . New lead data engineer
- E . Original data engineer
C
Explanation:
https://docs.databricks.com/sql/admin/transfer-ownership.html
A data analyst has created a Delta table sales that is used by the entire data analysis team. They want help from the data engineering team to implement a series of tests to ensure the data is clean. However, the data engineering team uses Python for its tests rather than SQL.
Which of the following commands could the data engineering team use to access sales in PySpark?
- A . SELECT * FROM sales
- B . There is no way to share data between PySpark and SQL.
- C . spark.sql("sales")
- D . spark.delta.table("sales")
- E . spark.table("sales")
E
Explanation:
https://spark.apache.org/docs/3.2.1/api/python/reference/api/pyspark.sql.SparkSession.table.html
Which of the following commands will return the location of database customer360?
- A . DESCRIBE LOCATION customer360;
- B . DROP DATABASE customer360;
- C . DESCRIBE DATABASE customer360;
- D . ALTER DATABASE customer360 SET DBPROPERTIES (‘location’ = ‘/user’};
- E . USE DATABASE customer360;
C
Explanation:
To retrieve the location of a database named "customer360" in a database management system like Hive or Databricks, you can use the DESCRIBE DATABASE command followed by the database name.
This command will provide information about the database, including its location.
A data engineer wants to create a new table containing the names of customers that live in France.
They have written the following command:
A senior data engineer mentions that it is organization policy to include a table property indicating that the new table includes personally identifiable information (PII).
Which of the following lines of code fills in the above blank to successfully complete the task?
- A . There is no way to indicate whether a table contains PII.
- B . "COMMENT PII"
- C . TBLPROPERTIES PII
- D . COMMENT "Contains PII"
- E . PII
D
Explanation:
Ref: https://www.databricks.com/discover/pages/data-quality-management
CREATE TABLE my_table (id INT COMMENT ‘Unique Identification Number’, name STRING COMMENT ‘PII’, age INT COMMENT ‘PII’) TBLPROPERTIES (‘contains_pii’=True) COMMENT ‘Contains PII’;
Which of the following benefits is provided by the array functions from Spark SQL?
- A . An ability to work with data in a variety of types at once
- B . An ability to work with data within certain partitions and windows
- C . An ability to work with time-related data in specified intervals
- D . An ability to work with complex, nested data ingested from JSON files
- E . An ability to work with an array of tables for procedural automation
D
Explanation:
Array functions in Spark SQL are primarily used for working with arrays and complex, nested data structures, such as those often encountered when ingesting JSON files. These functions allow you to manipulate and query nested arrays and structures within your data, making it easier to extract and work with specific elements or values within complex data formats. While some of the other options (such as option A for working with different data types) are features of Spark SQL or SQL in general, array functions specifically excel at handling complex, nested data structures like those found in JSON files.
Which of the following commands can be used to write data into a Delta table while avoiding the writing of duplicate records?
- A . DROP
- B . IGNORE
- C . MERGE
- D . APPEND
- E . INSERT
C
Explanation:
To write data into a Delta table while avoiding the writing of duplicate records, you can use the MERGE command. The MERGE command in Delta Lake allows you to combine the ability to insert new records and update existing records in a single atomic operation. The MERGE command compares the data being written with the existing data in the Delta table based on specified matching criteria, typically using a primary key or unique identifier. It then performs conditional actions, such as inserting new records or updating existing records, depending on the comparison results. By using the MERGE command, you can handle the prevention of duplicate records in a more controlled and efficient manner. It allows you to synchronize and reconcile data from different sources while avoiding duplication and ensuring data integrity.
A data engineer needs to apply custom logic to string column city in table stores for a specific use case. In order to apply this custom logic at scale, the data engineer wants to create a SQL user-defined function (UDF).
Which of the following code blocks creates this SQL UDF?
A)
B)
C)
D)
E)
- A . Option A
- B . Option B
- C . Option C
- D . Option D
- E . Option E
A
Explanation:
https://www.databricks.com/blog/2021/10/20/introducing-sql-user-defined-functions.html
A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task.
Which of the following approaches could be used by the data engineering team to complete this task?
- A . They could submit a feature request with Databricks to add this functionality.
- B . They could wrap the queries using PySpark and use Python’s control flow system to determine when to run the final query.
- C . They could only run the entire program on Sundays.
- D . They could automatically restrict access to the source table in the final query so that it is only accessible on Sundays.
- E . They could redesign the data model to separate the data used in the final query into a new table.
A data engineer runs a statement every day to copy the previous day’s sales into the table transactions. Each day’s sales are in their own file in the location "/transactions/raw".
Today, the data engineer runs the following command to complete this task:
After running the command today, the data engineer notices that the number of records in table transactions has not changed.
Which of the following describes why the statement might not have copied any new records into the table?
- A . The format of the files to be copied were not included with the FORMAT_OPTIONS keyword.
- B . The names of the files to be copied were not included with the FILES keyword.
- C . The previous day’s file has already been copied into the table.
- D . The PARQUET file format does not support COPY INTO.
- E . The COPY INTO statement requires the table to be refreshed to view the copied rows.
C
Explanation:
https://docs.databricks.com/en/ingestion/copy-into/index.html
The COPY INTO SQL command lets you load data from a file location into a Delta table. This is a re-triable and idempotent operation; files in the source location that have already been loaded are skipped. if there are no new records, the only consistent choice is C no new files were loaded because already loaded files were skipped.
A data engineer needs to create a table in Databricks using data from their organization’s existing SQLite database.
They run the following command:
Which of the following lines of code fills in the above blank to successfully complete the task?
- A . org.apache.spark.sql.jdbc
- B . autoloader
- C . DELTA
- D . sqlite
- E . org.apache.spark.sql.sqlite
A
Explanation:
https://docs.yugabyte.com/preview/integrations/apache-spark/spark-sql/ CREATE TABLE new_employees_table USING JDBC
OPTIONS (
url "<jdbc_url>",
dbtable "<table_name>",
user ‘<username>’,
password ‘<password>’
) AS
SELECT * FROM employees_table_vw
https://docs.databricks.com/external-data/jdbc.html#language-sql