mirror of
https://github.com/vinta/awesome-python.git
synced 2026-11-20 06:40:51 +00:00
docs: add cloudpickle and dill to Serialization intro
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
83bc124112
commit
550a5e81f6
1 changed files with 6 additions and 0 deletions
|
|
@ -6,6 +6,8 @@ How to choose:
|
|||
- Faster JSON that still gives you dicts and lists: orjson
|
||||
- Compact binary data other languages can read: msgpack
|
||||
- Schemas that convert ORM or app objects for a web API: marshmallow
|
||||
- Shipping lambdas and notebook functions to remote workers: cloudpickle
|
||||
- Saving a whole interpreter session, or objects pickle refuses: dill
|
||||
|
||||
msgspec works as a faster JSON or MessagePack library on its own. Still, its docs recommend it for the [full serialization and validation workflow](https://github.com/msgspec/msgspec): define your schemas with type annotations, encode them, then decode them back with validation. [Structs are the preferred way](https://msgspec.dev/structs) to define those types. Pass the type when you decode, and msgspec [validates while decoding](https://msgspec.dev/usage#typed-decoding) at no added runtime cost: `msgspec.json.decode(data, type=User)`. A consistent interface covers MessagePack, YAML, and TOML too.
|
||||
|
||||
|
|
@ -15,4 +17,8 @@ msgpack reads and writes MessagePack, a binary format that [lets you exchange da
|
|||
|
||||
marshmallow [makes no assumption about your web framework or database layer](https://marshmallow.readthedocs.io/en/latest/why.html#agnostic), so its schemas work with just about any ORM, or none. [Declare a schema](https://marshmallow.readthedocs.io/en/latest/quickstart.html#declaring-schemas) as a class that maps attribute names to fields. Its `dump` method turns your objects into primitive Python types, and `load` [validates and deserializes](https://marshmallow.readthedocs.io/en/latest/quickstart.html#deserializing-objects-loading) incoming data, raising `ValidationError` on invalid input. To get objects back instead of dicts, [decorate a schema method with `post_load`](https://marshmallow.readthedocs.io/en/latest/quickstart.html#deserializing-to-objects).
|
||||
|
||||
cloudpickle is built for [cluster computing, where Python code is shipped over the network](https://github.com/cloudpipe/cloudpickle) to run on remote hosts. It pickles lambdas, plus functions and classes you define in a script, a shell, or a Jupyter notebook, and the result loads back with the standard `pickle.loads`. Use it only between the [exact same version of Python](https://pypi.org/project/cloudpickle/), never for long-term storage.
|
||||
|
||||
dill is a [drop-in replacement for `pickle`](https://dill.readthedocs.io/en/latest/#basic-usage): write `import dill as pickle`, and nested functions, lambdas, and more exotic types pickle too. Call [`dill.dump_module()`](https://dill.readthedocs.io/en/latest/dill.html#dill.session.dump_module) to save your interpreter session to a file, and `dill.load_module()` to restore it later. dill [isn't meant to be secure against maliciously constructed data](https://dill.readthedocs.io/en/latest/#about-dill), and no pickle is. Never load one from a source you don't trust: use a data format instead.
|
||||
|
||||
orjson, msgspec, and msgpack all pitch speed, but msgspec's own benchmark page [encourages you to write your own benchmarks](https://msgspec.dev/benchmarks) before deciding.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue