djblets.cache.backend¶

Utility functions for working with memory caching backends.

These functions are designed to integrate with a cache backend using Django’s cache framework. They handle creating caching keys unique to the install and caching more complex data efficiently (such as the results of iterators and large data normally too big for the cache).

cache_memoize_iter(key: str, items_or_callable: Union[Callable[[], Iterable[_T]], Iterable[_T]], *, expiration: int = 2592000, force_overwrite: bool = False, compress_large_data: bool = True, use_encryption: Optional[bool] = None, encryption_key: Optional[bytes] = None) → Iterator[_T][source]¶

Memoize an iterable list of items inside the configured cache.

If the provided list of items is a function, the function must return a an iterable object, such as a list or a generator.

If a generator is provided, directly or through a function, then each item will be immediately yielded to the caller as they’re retrieved, and the cached entries will be built up as the items are processed.

The data is assumed to be big enough that it must be pickled, optionally compressed, and stored as chunks in the cache.

Data can be encrypted using AES encryption, for safe storage of potentially sensitive information or state. This adds to the processing time slightly, but can be important for cache keys containing sensitive information or that impact access control, particularly in the event that the cache is compromised or shared between services.

The result from this function is always a generator. Note that it’s important that the generator be allowed to continue until completion, or the data won’t be retrievable from the cache.

Changed in version 5.1:

All arguments (except key and items_or_callable) are now keyword-only arguments. Passing these as positional arguments is deprecated and will be removed in Djblets 7.
Added support for type hints.

Changed in version 3.0: Added support for encrypting keys and data through the use_encryption and encryption_key arguments.

Parameters:

key (str) – The key to use in the cache.
items_or_callable (list or callable) – A list of items or callable returning a list of items to cache, if the key is not already found in cache.
expiration (int, optional) – The expiration time for the key, in seconds.
force_overwrite (bool, optional) – If True, the value will always be computed and stored regardless of whether it exists in the cache already.
compress_large_data (bool, optional) – If True, the data will be zlib-compressed.
use_encryption (bool, optional) –
Whether to use encryption when storing or reading data.

If reading data, and if the data cannot be decrypted with the given key, then the data will be considered to have fallen out of cache.

This defaults to False, but can be forced on for all cached data by setting settings.DJBLETS_CACHE_FORCE_ENCRYPTION=True.

New in version 3.0.
encryption_key (bytes, optional) –
An explicit AES encryption key to use when passing use_encryption=True.

This defaults the value in settings.DJBLETS_CACHE_DEFAULT_ENCRYPTION_KEY, if set, or to the default AES encryption key for the server as provided by djblets.secrets.crypto.get_default_aes_encryption_key().

New in version 3.0.

Yields:

object – The list of items from the cache or from items_or_callable if uncached.

Raises:

Exception – An error occurred while yielding results. This could be a failure with the cache backend. Callers are encouraged to catch exceptions and try to handle any mid-result failures gracefully.

cache_memoize(key: str, lookup_callable: Callable[[], _T], *, expiration: int = 2592000, force_overwrite: bool = False, large_data: bool = False, compress_large_data: bool = True, use_generator: bool = False, use_encryption: Optional[bool] = None, encryption_key: Optional[bytes] = None) → _T[source]¶

Memoize the results of a callable inside the configured cache.

Changed in version 5.1:

All arguments (except key and lookup_callable) are now keyword-only arguments. Passing these as positional arguments is deprecated and will be removed in Djblets 7, along with the deprecated use_generator argument.
Added support for type hints.

Changed in version 3.0: Added support for encrypting keys and data through the use_encryption and encryption_key arguments.

Changed in version 2.2.4: Added support for non-iterable value types.

Parameters:

key (str) – The key to use in the cache.
lookup_callable (callable) – A callable to execute in the case where the data did not exist in the cache.
expiration (int, optional) – The expiration time for the key, in seconds.
force_overwrite (bool, optional) – If True, the value will always be computed and stored regardless of whether it exists in the cache already.
large_data (bool, optional) – If True, the resulting data will be pickled, gzipped, and (potentially) split up into megabyte-sized chunks. This is useful for very large, computationally intensive hunks of data which we don’t want to store in a database due to the way things are accessed.
compress_large_data (bool, optional) – Compresses the data with zlib compression when large_data is True.
use_generator (bool, deprecated) – This parameter is deprecated and was scheduled for removal in Djblets 3. It will be removed in Djblets 7.
use_encryption (bool, optional) –
Whether to use encryption when storing or reading data.

If reading data, and if the data cannot be decrypted with the given key, then the data will be considered to have fallen out of cache.

This defaults to False, but can be forced on for all cached data by setting settings.DJBLETS_CACHE_FORCE_ENCRYPTION=True.

New in version 3.0.
encryption_key (bytes or str, optional) –
An explicit AES encryption key to use when passing use_encryption=True.

This defaults the value in settings.DJBLETS_CACHE_DEFAULT_ENCRYPTION_KEY, if set, or to the default AES encryption key for the server as provided by djblets.secrets.crypto.get_default_aes_encryption_key().

New in version 3.0.

Returns:

The cached data, or the result of lookup_callable if uncached.

Return type:

object

make_cache_key(key: str, *, use_encryption: Optional[bool] = None, encryption_key: Optional[bytes] = None) → str[source]¶

Create a cache key guaranteed to avoid conflicts and size limits.

The cache key will be prefixed by the site’s domain, and will be changed to an SHA256 hash if it’s larger than the maximum key size or contains characters not compatible with the cache backend.

Changed in version 5.1:

All arguments (except key) are now keyword-only arguments. Passing these as positional arguments is deprecated and will be removed in Djblets 7.
Added support for type hints.

Changed in version 3.0:

Added support for encrypting keys through the use_encryption and encryption_key arguments.
Changed the hash format for keys to use SHA256 instead of MD5. This will invalidate all old keys in cache, but reduces chances of collision.
Keys will automatically use the hash format if they contain characters unsupported by the cache backend.
The return type is now str, to generate keys that are more suitable for modern versions of Django.

Parameters:

key (str) – The base key to generate a cache key from.
use_encryption (bool, optional) –
Whether to generate an encrypted key.

This will generate a HMAC digest of the key. There will be no identifying information in the resulting key.

This defaults to False, but can be forced on for all cached data by setting settings.DJBLETS_CACHE_FORCE_ENCRYPTION=True.

New in version 3.0.
encryption_key (bytes, optional) –
An explicit AES encryption key to use when passing use_encryption=True.

This defaults the value in settings.DJBLETS_CACHE_DEFAULT_ENCRYPTION_KEY, if set, or to the default AES encryption key for the server as provided by djblets.secrets.crypto.get_default_aes_encryption_key().

New in version 3.0.

Returns:

A cache key suitable for use with the cache backend.

Return type:

str