From 550a5e81f6c3e138537c7a19485f2cca6c11e3f1 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Fri, 16 Oct 2026 18:24:58 +0800 Subject: [PATCH] docs: add cloudpickle and dill to Serialization intro Co-Authored-By: Claude --- website/data/category_intros/serialization.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/website/data/category_intros/serialization.md b/website/data/category_intros/serialization.md index de3f8695..ab864163 100644 --- a/website/data/category_intros/serialization.md +++ b/website/data/category_intros/serialization.md @@ -6,6 +6,8 @@ How to choose: - Faster JSON that still gives you dicts and lists: orjson - Compact binary data other languages can read: msgpack - Schemas that convert ORM or app objects for a web API: marshmallow +- Shipping lambdas and notebook functions to remote workers: cloudpickle +- Saving a whole interpreter session, or objects pickle refuses: dill msgspec works as a faster JSON or MessagePack library on its own. Still, its docs recommend it for the [full serialization and validation workflow](https://github.com/msgspec/msgspec): define your schemas with type annotations, encode them, then decode them back with validation. [Structs are the preferred way](https://msgspec.dev/structs) to define those types. Pass the type when you decode, and msgspec [validates while decoding](https://msgspec.dev/usage#typed-decoding) at no added runtime cost: `msgspec.json.decode(data, type=User)`. A consistent interface covers MessagePack, YAML, and TOML too. @@ -15,4 +17,8 @@ msgpack reads and writes MessagePack, a binary format that [lets you exchange da marshmallow [makes no assumption about your web framework or database layer](https://marshmallow.readthedocs.io/en/latest/why.html#agnostic), so its schemas work with just about any ORM, or none. [Declare a schema](https://marshmallow.readthedocs.io/en/latest/quickstart.html#declaring-schemas) as a class that maps attribute names to fields. Its `dump` method turns your objects into primitive Python types, and `load` [validates and deserializes](https://marshmallow.readthedocs.io/en/latest/quickstart.html#deserializing-objects-loading) incoming data, raising `ValidationError` on invalid input. To get objects back instead of dicts, [decorate a schema method with `post_load`](https://marshmallow.readthedocs.io/en/latest/quickstart.html#deserializing-to-objects). +cloudpickle is built for [cluster computing, where Python code is shipped over the network](https://github.com/cloudpipe/cloudpickle) to run on remote hosts. It pickles lambdas, plus functions and classes you define in a script, a shell, or a Jupyter notebook, and the result loads back with the standard `pickle.loads`. Use it only between the [exact same version of Python](https://pypi.org/project/cloudpickle/), never for long-term storage. + +dill is a [drop-in replacement for `pickle`](https://dill.readthedocs.io/en/latest/#basic-usage): write `import dill as pickle`, and nested functions, lambdas, and more exotic types pickle too. Call [`dill.dump_module()`](https://dill.readthedocs.io/en/latest/dill.html#dill.session.dump_module) to save your interpreter session to a file, and `dill.load_module()` to restore it later. dill [isn't meant to be secure against maliciously constructed data](https://dill.readthedocs.io/en/latest/#about-dill), and no pickle is. Never load one from a source you don't trust: use a data format instead. + orjson, msgspec, and msgpack all pitch speed, but msgspec's own benchmark page [encourages you to write your own benchmarks](https://msgspec.dev/benchmarks) before deciding.