spark-sql-streaming-CheckpointFileManager.adoc

CheckpointFileManager Contract

CheckpointFileManager is the abstraction of checkpoint managers that can FIXME.

Table 1. CheckpointFileManager Contract

Method	Description
createAtomic	createAtomic( path: Path, overwriteIfPossible: Boolean): CancellableFSDataOutputStream Used when: HDFSMetadataLog is requested to store metadata for a batch (that writeBatchToFile) StreamMetadata helper object is requested to persist metadata HDFSBackedStateStore is requested for the deltaFileStream HDFSBackedStateStoreProvider is requested to writeSnapshotFile
delete	delete(path: Path): Unit Deletes a path recursively (if it exists) Used when: RenameBasedFSDataOutputStream is requested to cancel CompactibleFileStreamLog is requested to store metadata for a batch (that deleteExpiredLog) HDFSMetadataLog is requested to remove expired metadata HDFSBackedStateStoreProvider is requested to do maintenance (that clean up)
exists	exists(path: Path): Boolean Used when…​FIXME
isLocal	isLocal: Boolean Does not seem to be used.
list	list(path: Path): Array[FileStatus] list(path: Path, filter: PathFilter): Array[FileStatus] Used when…​FIXME
mkdirs	mkdirs(path: Path): Unit Used when…​FIXME
open	open(path: Path): FSDataInputStream Opens a file for reading Used when: HDFSMetadataLog is requested to retrieve metadata for batch HDFSBackedStateStoreProvider is requested to retrieve the state store for a specified version (that updateFromDeltaFile)

Table 2. CheckpointFileManagers

CheckpointFileManager	Description
FileContextBasedCheckpointFileManager
FileSystemBasedCheckpointFileManager