awesome-python/website/data/category_intros/file-manipulation.md
Vinta Chen 3856003a30 docs: add archive libraries to File Manipulation intro
Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-15 00:54:55 +08:00

8.7 KiB

Without installing anything, two Python file manipulation libraries cover paths and file types by name: pathlib and mimetypes. python-magic reads the bytes.

How to choose:

  • Building and joining paths, and listing directories: pathlib
  • A MIME type from a file name or URL: mimetypes
  • A file's type from its content, like the Unix file command: python-magic
  • Rerunning or reloading code when files change: watchfiles
  • Your own handler for each created, modified, or moved file: watchdog
  • Telling text file types apart, like source code from config files: Magika
  • Listing, globbing, and copying files on cloud or remote storage: fsspec
  • Streaming one large remote file through a drop-in open(): smart_open
  • Zstandard files and data: compression.zstd, or zstandard for multi-threaded compression
  • Reading and writing .gz files: gzip
  • Denser compression than gzip at a similar speed: brotli
  • Compressing and decompressing as fast as possible: lz4
  • Tar and ZIP archives: tarfile and zipfile, or pyzipper for AES-encrypted ZIP files
  • 7-Zip and RAR archives: py7zr, or rarfile to read .rar files

If you aren't sure which pathlib class you need, the docs say Path is most likely it: it makes a concrete path for the platform your code runs on. Join paths with the / operator, and list a directory with iterdir() or glob().

mimetypes maps a file name's extension to a MIME type, and a MIME type back to extensions. A type guess returns 2 values: a type for the Content-Type header, and an encoding like gzip for the Content-Encoding header. The type is None for a missing or unknown suffix.

python-magic wraps libmagic, which identifies file types by checking their headers, the way the Unix file command does. It's a thin wrapper, so install libmagic too, with apt-get install libmagic1 or brew install libmagic. Call magic.from_file(path, mime=True) for a MIME type, or magic.from_buffer() on bytes you already have.

watchfiles is built for file watching and code reload, and its Rust core groups changes into batches instead of firing once per file. watch() is a generator that yields sets of changes, and awatch() is the async version. To restart code when files change, use run_process() with a function or a command. From a shell, the watchfiles CLI does the same: watchfiles --filter python 'pytest --lf' src tests.

watchdog gives you an API and a shell tool for monitoring directories. Its quickstart has you subclass FileSystemEventHandler, override methods like on_created() and on_modified(), schedule the handler on an Observer, and start that thread. An observer skips subdirectories unless you pass recursive=True to schedule().

Magika runs a compact deep-learning model on a few chunks of a file, without loading large files entirely into memory. On its own test set, it outperforms existing approaches, especially on textual content types. Create a Magika() instance, then call identify_path() on a file or identify_bytes() on bytes you already have. Check result.ok before you read result.output.label or result.output.mime_type.

fsspec gives you one API for local, remote, and embedded file systems, and it works the same whatever the storage backend. Get a file system by protocol name, like fsspec.filesystem("s3"), then call methods that follow posix naming and the os module: ls(), glob(), cp(), and rm(). For one file, fsspec.open() picks the backend from the URL, and opens the file only when you enter its with block.

smart_open is a drop-in replacement for Python's built-in open() that streams very large files from and to S3, GCS, Azure Blob Storage, HDFS, HTTP, and SFTP. Import its open in place of the built-in, and pass it a URL like s3://bucket/key. A file name ending in .gz, .bz2, or .zst gets compressed or decompressed on the fly. To use your own credentials or settings, pass the storage service's client object in transport_params.

compression.zstd covers files, bytes, and streams. compression.zstd.open() reads and writes .zst files as file objects, compress() and decompress() handle bytes in one call, and ZstdCompressor and ZstdDecompressor work through data a chunk at a time.

gzip opens .gz files like regular files: gzip.open() takes 'rb' or 'wb' for bytes, and 'rt' or 'wt' for text. For data already in memory, call gzip.compress() and gzip.decompress().

zstandard is the most feature-rich zstd binding. Still, its own docs say the standard library is probably the easiest way to use zstd, if it works for you. Use its one-shot APIs for data that fits in memory, and its streaming APIs for large inputs. Pass threads to ZstdCompressor to compress on several cores, in C threads that run outside the GIL.

brotli is the Python module of the Brotli project, whose format is similar in speed to deflate but denser. Call brotli.compress() and brotli.decompress() on bytes, or feed a brotli.Compressor piece by piece with process(), then call finish().

lz4 binds the LZ4 library, built for extremely fast compression and an extremely fast decoder. Its docs recommend the lz4.frame bindings, since the frame format works with other LZ4 implementations and language bindings. lz4.frame.open() is the most convenient way to work with compressed files, and it works like gzip.open().

tarfile reads and writes tar archives through tarfile.open(), and a mode like 'r:gz' or 'w:zst' picks the compression. To extract, use extractall() with the 'data' filter, which is meant for cross-platform data archives. Still, never extract an archive from an untrusted source without inspecting it first.

zipfile works through ZipFile as a context manager: ZipFile('spam.zip', 'w') to write, then write() each file. Before you call extractall() on an archive you didn't make, inspect it first, since members with absolute paths or .. can land outside the target folder.

py7zr opens 7-Zip archives with SevenZipFile in a with block: call extractall() in 'r' mode, or writeall() in 'w' mode to pack a whole folder.

pyzipper is a replacement for zipfile that reads and writes AES-encrypted ZIP files, with the same API. Open the archive with pyzipper.AESZipFile, pass encryption=pyzipper.WZ_AES when writing, and call setpassword() with a bytes password.

rarfile reads RAR archives through an interface in the style of zipfile. It parses archives in pure Python, but extracts compressed files with an external tool, so install unrar, the one it recommends.