The current ELT pipeline is receiving data from the operations team once a day so you had setup an AUTO LOADER process to run once a day using trigger (Once = True) and scheduled a job to run once a day, operations team recently rolled out a new feature that allows them to send data every 1 min, what changes do you need to make to AUTO LOADER to process the data every 1 min.

The current ELT pipeline is receiving data from the operations team once a day so you had setup an AUTO LOADER process to run once a day using trigger (Once = True) and scheduled a job to run once a day, operations team recently rolled out a new feature that allows them to send data every 1 min, what changes do you need to make to AUTO LOADER to process the data every 1 min.
A . Convert AUTO LOADER to structured streaming
B . Change AUTO LOADER trigger to .trigger(ProcessingTime = "1 minute")
C . Setup a job cluster run the notebook once a minute
D . Enable stream processing
E . Change AUTO LOADER trigger to ("1 minute")

Answer: B

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments