Scalable Ingestion and The Listing Problem.

In Cloud Object Stores (S3, ADLS), the LIST operation is expensive. Its latency scales linearly (or worse) with the number of objects in the prefix. For a data lake with millions of files, identifying "What is new?" by listing everything is computationally prohibitive and creates high latency. This is the "Listing Bottleneck". Furthermore, "Exactly-Once" semantics require a persistent State Store to remember history.

Application: Databricks Auto Loader (cloudFiles) is the architectural answer to these two theoretical problems.

Solving the Listing Problem: It bypasses the LIST API entirely by using File Notification Mode. Cloud providers allow buckets to publish events ("File X was created") to a Queue. Auto Loader treats this Queue as the stream source. This is $O(1)$ complexity relative to total bucket size, ensuring consistent low latency even as the data grows to petabytes.

Solving the State Problem: It maintains an internal RocksDB database within the checkpoint directory. This acts as the deduplication ledger. Even if the upstream system creates a "Retry Storm" and re-uploads thousands of files, Auto Loader's state check is a fast local lookup, protecting the downstream table from duplicate records. This combination provides a robust, "set it and forget it" ingestion primitive.

Topic: Autoloader

Scalability: It uses \"File Notification\" mode (subscribing to cloud storage events like SQS/Event Grid) to avoid listing massive directories, making it performant even with millions of files.
Schema Evolution: It has advanced features to handle drifting schemas (new columns) and data quality issues (type mismatches) by using a \"rescued data\" column, ensuring pipelines don't break when data changes.