Skip to content

pyroid.datasketch

An exact drop-in for datasketch 2.0.0's MinHash family: MinHash, LeanMinHash, MinHashLSH, MinHashLSHForest and MinHashLSHEnsemble. Guide: pyroid.datasketch.

pyroid.datasketch

An exact drop-in for datasketch 2.0.0's MinHash family, with the work in Rust.

from pyroid.datasketch import MinHash, MinHashLSH  # instead of: from datasketch import ...

Hash values, LeanMinHash bytes, the LSH (b, r) parameters and query results are datasketch 2.0.0's, for all three permutation schemes. The submodules mirror datasketch's (pyroid.datasketch.minhash, .lean_minhash, .lsh, ...), so from datasketch.lsh import MinHashLSH becomes from pyroid.datasketch.lsh import MinHashLSH. numpy is required, as for datasketch; scipy is not.

pyroid extensions run on pyroid's thread pool: MinHash.bulk(..., threads=), minhash_batch (a signature matrix from token lists or texts) and the MinHashLSH batch methods.

datasketch classes that pyroid does not port (bBitMinHash, HNSW, MinHashLSHBloom, AsyncMinHashLSH, WeightedMinHash, WeightedMinHashGenerator, HyperLogLog, HyperLogLogPlusPlus) are datasketch's own: getting one from this module imports datasketch, with a one-time pyroid.FallbackWarning (pyroid.PyroidError if datasketch is not installed).

WeightedMinHashLSH module-attribute 

WeightedMinHashLSH = MinHashLSH

WeightedMinHashLSHForest module-attribute 

WeightedMinHashLSHForest = MinHashLSHForest

DATASKETCH_VERSION module-attribute 

DATASKETCH_VERSION: Final[str]

LeanMinHash 

LeanMinHash(
    minhash: MinHash | None = None,
    seed: int | None = None,
    hashvalues: Iterable[int] | None = None,
    scheme: str | None = None,
)

Bases: MinHash

A MinHash with its state frozen: no update.

Parameters:

  • minhash (MinHash | None, default: None ) –

    The MinHash to take the seed, hash values and scheme from.

  • seed (int | None, default: None ) –

    With hashvalues and scheme: the seed of the permutations.

  • hashvalues (Iterable[int] | None, default: None ) –

    With seed and scheme: the hash values.

  • scheme (str | None, default: None ) –

    The scheme of hashvalues (required with them; "legacy" for values from datasketch before 2.0.0). With minhash it may only repeat the MinHash's scheme.

Raises:

  • ValueError –

    Neither minhash nor seed and hashvalues; no or conflicting scheme; empty hashvalues for an affine scheme.

update 

update(b: Any) -> None

Raise TypeError: a LeanMinHash cannot be updated.

copy 

copy() -> LeanMinHash

Return a copy.

bytesize 

bytesize(byteorder: str = '@') -> int

Return the size of the serialized form in bytes.

Parameters:

  • byteorder (str, default: '@' ) –

    A struct byte order character: @ (native, the default), =, <, > or !.

serialize 

serialize(buf: Buffer, byteorder: str = '@') -> None

Write this LeanMinHash into the writable buffer buf, from offset 0.

Layout ("legacy", as before datasketch 2.0.0): seed (8 bytes), the number of hash values (4 bytes), then each value in 4 bytes. Affine schemes: seed, the negated number of values, a scheme code byte (1 for "affine32", 2 for "affine64"), then each value in 4 or 8 bytes.

Raises:

deserialize classmethod 

deserialize(
    buf: Buffer, byteorder: str = "@"
) -> LeanMinHash

Read a LeanMinHash written by serialize (pyroid's or datasketch's).

Buffers written before datasketch 2.0.0 come back with scheme="legacy".

Raises:

union classmethod 

union(*lmhs: LeanMinHash) -> LeanMinHash

Return a new LeanMinHash, the union of lmhs (at least two).

MinHashLSH 

MinHashLSH(
    threshold: float = 0.9,
    num_perm: int = 128,
    weights: tuple[float, float] = (0.5, 0.5),
    params: tuple[int, int] | None = None,
    storage_config: dict[str, Any] | None = None,
    prepickle: bool | None = None,
    hashfunc: Callable[[bytes], bytes] | None = None,
)

The MinHash LSH index, for queries with a Jaccard similarity threshold.

Parameters:

  • threshold (float, default: 0.9 ) –

    The Jaccard similarity threshold, in [0.0, 1.0]. The band layout is chosen to minimize the weighted false positive and false negative probabilities at this threshold.

  • num_perm (int, default: 128 ) –

    The number of permutations of the MinHashes to index (the sample_size of a weighted MinHash).

  • weights (tuple[float, float], default: (0.5, 0.5) ) –

    (false_positive_weight, false_negative_weight); they sum to 1.0.

  • params (tuple[int, int] | None, default: None ) –

    (b, r), the number of bands and rows per band. Skips the search; threshold and weights are then ignored.

  • storage_config (dict[str, Any] | None, default: None ) –

    {"type": "dict"} (the default) is the in-memory index. "redis" and "cassandra" are delegated to datasketch. "basename" names the storage.

  • prepickle (bool | None, default: None ) –

    Pickle keys to bytes before storing them. Default: true for Redis storage only.

  • hashfunc (Callable[[bytes], bytes] | None, default: None ) –

    Hash function applied to band keys. Delegated to datasketch.

Raises:

  • ValueError –

    For the same invalid arguments as datasketch, with its messages ("Unknown storage type: ..." included).

Attributes:

h instance-attribute 

h: int

b instance-attribute 

b: int

r instance-attribute 

r: int

hashranges instance-attribute 

hashranges: list[tuple[int, int]]

prepickle instance-attribute 

prepickle: bool

hashfunc instance-attribute 

hashfunc: Callable[[bytes], bytes] | None

hashtables instance-attribute 

hashtables: list[UnorderedStorage]

keys instance-attribute 

keys: OrderedStorage

buffer_size property writable 

buffer_size: int

The storage buffer size (used by Redis and Cassandra storage).

insert 

insert(
    key: Hashable,
    minhash: _Sketch,
    check_duplication: bool = True,
) -> None

Insert key with the MinHash (or weighted MinHash) of its set.

Parameters:

  • key (Hashable) –

    The key of the set (any hashable; keys compare with ==).

  • minhash (_Sketch) –

    Its MinHash, of length num_perm.

  • check_duplication (bool, default: True ) –

    Raise if key is already in the index. With False, inserting a key again adds the new band keys to its entry, as datasketch does.

Raises:

  • ValueError –

    The MinHash has the wrong length or another permutation scheme than earlier ones, or key exists.

insert_batch 

insert_batch(
    keys: Iterable[Hashable],
    minhashes: Sequence[_Sketch] | Any,
    *,
    check_duplication: bool = True,
    threads: int | None = None,
) -> None

Insert keys[i] with minhashes[i] for every i (pyroid extension).

The index ends up as after insert of each item in order, but nothing is inserted if an item is invalid: the error is the one insert would raise for the first invalid item, with a note naming it. Band tables are filled in parallel.

Parameters:

  • keys (Iterable[Hashable]) –

    The keys.

  • minhashes (Sequence[_Sketch] | Any) –

    MinHash-like objects, or a numpy array whose rows (axis 0) are hashvalues arrays, e.g. (n, num_perm) uint64 or uint32. Array rows have no scheme, so they are not checked against it (as weighted MinHashes).

  • check_duplication (bool, default: True ) –

    As for insert; a key repeated within the batch is a duplicate too.

  • threads (int | None, default: None ) –

    At most this many tasks at once on pyroid's thread pool (None: the whole pool, 1: the calling thread).

Raises:

  • ValueError –

    keys and minhashes differ in length, or as insert.

  • TypeError –

    An array of another dtype than integers or floats.

merge 

merge(
    other: MinHashLSH, check_overlap: bool = False
) -> None

Merge other into this index, making this one the union of both.

Only num_perm, b and r are compared, as in datasketch.

Parameters:

  • other (MinHashLSH) –

    Another pyroid MinHashLSH.

  • check_overlap (bool, default: False ) –

    Raise if a key is in both indexes (nothing is merged then).

Raises:

  • ValueError –

    The indexes differ in type, parameters or MinHash scheme, or check_overlap found a common key.

insertion_session 

insertion_session(
    buffer_size: int = 50000,
) -> MinHashLSHInsertionSession

Return a context manager whose insert inserts into this index.

deletion_session 

deletion_session(
    buffer_size: int = 50000,
) -> MinHashLSHDeletionSession

Return a context manager whose remove removes from this index.

query 

query(minhash: _Sketch) -> list[Hashable]

Return the keys whose sets likely have a Jaccard similarity above the threshold.

Parameters:

  • minhash (_Sketch) –

    The MinHash of the query set.

Returns:

  • list[Hashable] –

    The keys that share a band with minhash, each once, in no particular order.

Raises:

  • ValueError –

    The MinHash has the wrong length or another scheme.

query_batch 

query_batch(
    minhashes: Sequence[_Sketch] | Any,
    *,
    threads: int | None = None,
) -> list[list[Hashable]]

Return [self.query(m) for m in minhashes], in parallel (pyroid extension).

Parameters:

add_to_query_buffer 

add_to_query_buffer(minhash: _Sketch) -> None

Buffer a query for collect_query_buffer.

The buckets are read now (a snapshot), not when the buffer is collected.

collect_query_buffer 

collect_query_buffer() -> list[Hashable]

Run the buffered queries; return the keys every one of them found.

remove 

remove(key: Hashable) -> None

Remove key from the index.

Raises:

is_empty 

is_empty() -> bool

Return True if some band table is empty (datasketch's definition).

get_counts 

get_counts() -> list[dict[Hashable, int]]

Return, per band table, {band key: number of keys in the bucket}.

get_subset_counts 

get_subset_counts(
    *keys: Hashable,
) -> list[dict[Hashable, int]]

Return get_counts restricted to keys.

MinHashLSHEnsemble 

MinHashLSHEnsemble(
    threshold: float = 0.9,
    num_perm: int = 128,
    num_part: int = 16,
    m: int = 8,
    weights: tuple[float, float] = (0.5, 0.5),
    storage_config: dict[str, Any] | None = None,
    prepickle: bool | None = None,
)

The LSH Ensemble index for containment queries.

Parameters:

  • threshold (float, default: 0.9 ) –

    The containment threshold, in [0.0, 1.0].

  • num_perm (int, default: 128 ) –

    The number of permutations of the indexed MinHash.

  • num_part (int, default: 16 ) –

    The number of partitions by set size.

  • m (int, default: 8 ) –

    The maximum band size r over the inner LSH indexes, in [2, num_perm].

  • weights (tuple[float, float], default: (0.5, 0.5) ) –

    (false_positive_weight, false_negative_weight), summing to 1.0.

  • storage_config (dict[str, Any] | None, default: None ) –

    As for pyroid.datasketch.MinHashLSH.

  • prepickle (bool | None, default: None ) –

    As for pyroid.datasketch.MinHashLSH.

Raises:

  • ValueError –

    As datasketch, for parameters out of range.

threshold instance-attribute 

threshold: float

h instance-attribute 

h: int

m instance-attribute 

m: int

xqs instance-attribute 

xqs: NDArray[float64]

params instance-attribute 

params: NDArray[Any]

indexes instance-attribute 

indexes: list[dict[int, MinHashLSH]]

lowers instance-attribute 

lowers: list[Any]

uppers instance-attribute 

uppers: list[Any]

index 

index(
    entries: Iterable[tuple[Hashable, MinHash, int]],
) -> None

Index every (key, minhash, size) of entries; callable once.

Raises:

  • ValueError –

    The index is not empty, entries is empty, or a size is not positive.

query 

query(minhash: MinHash, size: int) -> Generator[Hashable]

Yield the keys of sets likely to contain threshold of the query set.

Parameters:

  • minhash (MinHash) –

    The MinHash of the query set.

  • size (int) –

    The number of distinct items in the query set.

is_empty 

is_empty() -> bool

Return whether nothing has been indexed.

MinHashLSHForest 

MinHashLSHForest(num_perm: int = 128, l: int = 8)

The LSH Forest for MinHash: top-k queries by Jaccard similarity.

Band keys are kept in sorted arrays, one per prefix tree, as in datasketch. It also works with weighted MinHashes.

Parameters:

  • num_perm (int, default: 128 ) –

    The number of permutations of the MinHashes to index (the sample_size of a weighted MinHash).

  • l (int, default: 8 ) –

    The number of prefix trees.

Raises:

  • ValueError –

    num_perm or l is not positive, or l > num_perm.

Attributes:

l instance-attribute 

l: int

k instance-attribute 

k: int

hashranges instance-attribute 

hashranges: list[tuple[int, int]]

keys instance-attribute 

keys: MutableMapping[Hashable, list[bytes]]

hashtables instance-attribute 

hashtables: list[Mapping[bytes, list[Hashable]]]

sorted_hashtables instance-attribute 

sorted_hashtables: list[Sequence[bytes]]

add 

add(key: Hashable, minhash: _Sketch) -> None

Add key with the MinHash of its set.

The key is searchable after the next index (or now, through band keys that are already indexed).

Raises:

  • ValueError –

    The MinHash is shorter than k * l, has another scheme than earlier ones, or key was already added.

index 

index() -> None

Index all the keys added so far and make them searchable.

query 

query(minhash: _Sketch, k: int) -> list[Hashable]

Return up to k keys with (approximately) the highest Jaccard similarity.

Parameters:

  • minhash (_Sketch) –

    The MinHash of the query set.

  • k (int) –

    The maximum number of keys to return.

Returns:

Raises:

  • ValueError –

    k <= 0, or the MinHash is too short or has another scheme.

get_minhash_hashvalues 

get_minhash_hashvalues(key: Hashable) -> Any

Return the hash values stored for key (a numpy array).

Raises:

  • KeyError –

    key is not in the forest.

  • ValueError –

    The stored band keys are not whole 4- or 8-byte values.

is_empty 

is_empty() -> bool

Return True if some tree has nothing searchable (before index).

MinHash 

MinHash(
    num_perm: int = 128,
    seed: int = 1,
    gpu_mode: Literal[
        "disable", "detect", "always"
    ] = "disable",
    hashfunc: Callable[[Any], Any] | None = None,
    hashobj: object | None = None,
    hashvalues: ArrayLike | None = None,
    permutations: tuple[ArrayLike, ArrayLike]
    | ArrayLike
    | None = None,
    scheme: _Scheme | None = None,
)

MinHash: estimate the Jaccard similarity and cardinality of sets.

Parameters:

  • num_perm (int, default: 128 ) –

    Number of random permutation functions. Ignored if hashvalues is given.

  • seed (int, default: 1 ) –

    The random seed of the permutation functions.

  • gpu_mode (Literal['disable', 'detect', 'always'], default: 'disable' ) –

    "disable", "detect" or "always". pyroid computes on the CPU in every mode; "always" raises RuntimeError if CuPy and a CUDA device are not available, as in datasketch.

  • hashfunc (Callable[[Any], Any] | None, default: None ) –

    The hash function: takes a value given to update and returns an int of 32 bits ("affine32", "legacy") or 64 bits ("affine64"). Default: sha1_hash32, or sha1_hash64 for "affine64". Other functions run in Python (slow path).

  • hashobj (object | None, default: None ) –

    Deprecated since datasketch 1.4.0; a no-op that warns.

  • hashvalues (ArrayLike | None, default: None ) –

    Initial hash values (another MinHash's hashvalues). scheme must then be given.

  • permutations (tuple[ArrayLike, ArrayLike] | ArrayLike | None, default: None ) –

    Permutation parameters (another MinHash's permutations). scheme must then be given.

  • scheme (_Scheme | None, default: None ) –

    "affine32" (the default), "affine64" or "legacy" (datasketch before 2.0.0). MinHash of different schemes cannot be compared, merged or unioned.

Raises:

  • ValueError –

    As datasketch: an unknown scheme, num_perm out of range, hashvalues/permutations without scheme, mismatched lengths, values out of the scheme's range.

scheme instance-attribute 

scheme: str

seed instance-attribute 

seed: Any

num_perm instance-attribute 

num_perm: int

hashfunc instance-attribute 

hashfunc: Callable[[Any], Any]

hashvalues instance-attribute 

hashvalues: NDArray[Any]

permutations instance-attribute 

permutations: Any

update 

update(b: Any) -> None

Add one value to the set.

With the default hash function the value must be bytes-like, e.g. m.update("text".encode("utf-8")).

Parameters:

update_batch 

update_batch(b: Iterable[Any]) -> None

Add many values to the set (faster than update per value).

Parameters:

  • b (Iterable[Any]) –

    The values, each passed to hashfunc. An empty batch leaves the hash values unchanged.

Raises:

  • RuntimeError –

    gpu_mode="always" without CuPy and a CUDA device.

jaccard 

jaccard(other: MinHash) -> float

Estimate the Jaccard similarity with other.

Raises:

  • ValueError –

    Different schemes, seeds or numbers of permutations.

count 

count() -> float

Estimate the cardinality of the set (a numpy.float64).

merge 

merge(other: MinHash) -> None

Make this MinHash the union of itself and other.

Raises:

  • ValueError –

    Different schemes, seeds or numbers of permutations.

digest 

digest() -> NDArray[Any]

Return a copy of hashvalues.

is_empty 

is_empty() -> bool

Return whether the MinHash is in its initial state.

clear 

clear() -> None

Reset the hash values to the initial state.

copy 

copy() -> MinHash

Return a copy (the permutations are shared, as in datasketch).

union classmethod 

union(*mhs: MinHash) -> MinHash

Return a new MinHash, the union of mhs (at least two).

Raises:

  • ValueError –

    Fewer than two, or different seeds, sizes or schemes.

bulk classmethod 

bulk(
    b: Iterable[Iterable[Any]],
    *,
    threads: int | None = None,
    **minhash_kwargs: Any,
) -> list[MinHash]

Compute one MinHash per document of b.

Equal to [m.copy() updated with update_batch(doc) for doc in b] with m = MinHash(**minhash_kwargs), as in datasketch. Documents that are lists or tuples of bytes are hashed in parallel on pyroid's thread pool; others run datasketch's code.

Parameters:

  • b (Iterable[Iterable[Any]]) –

    The documents, each an iterable of values for hashfunc.

  • threads (int | None, default: None ) –

    pyroid extension: at most this many threads (None: the whole pool, 1: the calling thread).

  • **minhash_kwargs (Any, default: {} ) –

    Arguments for MinHash, used for all.

Example 
data = [[b"token1", b"token2", b"token3"], [b"token4", b"token5"]]
minhashes = MinHash.bulk(data, num_perm=64)

generator classmethod 

generator(
    b: Iterable[Iterable[Any]],
    *,
    threads: int | None = None,
    **minhash_kwargs: Any,
) -> Generator[MinHash]

Yield one MinHash per document of b (see bulk).

pyroid reads up to 2,048 documents ahead to hash them in parallel; the MinHash objects and any error come out in datasketch's order.

sha1_hash32 

sha1_hash32(data: Buffer) -> int

Return the first 4 bytes of sha1(data) as a little-endian int.

sha1_hash64 

sha1_hash64(data: Buffer) -> int

Return the first 8 bytes of sha1(data) as a little-endian int.

minhash_batch 

minhash_batch(
    docs: str
    | list[str]
    | tuple[str, ...]
    | ArrowStrings
    | Sequence[Sequence[bytes | str]],
    *,
    num_perm: int = 128,
    seed: int = 1,
    scheme: _Scheme | None = None,
    ngram: int = 1,
    threads: int | None = None,
) -> NDArray[uint32] | NDArray[uint64]

Compute MinHash signatures for many documents, as a matrix.

Row i equals MinHash(num_perm, seed, scheme=scheme) updated with the tokens of document i, i.e. its MinHash.hashvalues (uint32 for "affine32", uint64 otherwise). Documents are hashed in parallel on pyroid's thread pool. A pyroid extension.

Parameters:

  • docs (str | list[str] | tuple[str, ...] | ArrowStrings | Sequence[Sequence[bytes | str]]) –

    Token lists (a list or tuple of lists/tuples of bytes or str tokens; str is hashed as UTF-8), or texts (str, list[str], tuple[str, ...] or an Arrow string array). A text's tokens are its word ngram-grams: " ".join(words[i:i + ngram]) over text.split(), UTF-8 encoded; a text with fewer words has no tokens. Arrow nulls give the empty signature.

  • num_perm (int, default: 128 ) –

    Number of permutations.

  • seed (int, default: 1 ) –

    The permutation seed.

  • scheme (_Scheme | None, default: None ) –

    "affine32" (default), "affine64" or "legacy".

  • ngram (int, default: 1 ) –

    Words per shingle, for texts only.

  • threads (int | None, default: None ) –

    At most this many threads (None: the whole pool).

Returns:

  • NDArray[uint32] | NDArray[uint64] –

    A (len(docs), num_perm) array, or (num_perm,) for a single str.

Raises:

Example 
from pyroid.datasketch import MinHashLSH, minhash_batch

sigs = minhash_batch(texts, ngram=5, threads=8)  # (len(texts), 128) uint32
lsh = MinHashLSH(threshold=0.8)
lsh.insert_batch(range(len(texts)), sigs)

pyroid.datasketch.minhash

datasketch's MinHash (datasketch.minhash), with the hashing in Rust.

from pyroid.datasketch import MinHash replaces from datasketch import MinHash. A pyroid MinHash has datasketch 2.0.0's attributes (hashvalues, permutations, seed, scheme, hashfunc, ...), the same hash values for all three permutation schemes ("affine32", the default, "affine64" and "legacy"), the same pickled state and the same errors.

What runs in Rust: drawing the permutation parameters (numpy's RandomState, reproduced and cached per seed), SHA1 hashing with the default hash functions, and applying the permutations. MinHash.bulk and MinHash.generator hash many documents in parallel (threads=), and minhash_batch returns a signature matrix for token lists or the word n-grams of texts.

Example 
from pyroid.datasketch import MinHash

m1, m2 = MinHash(num_perm=128), MinHash(num_perm=128)
m1.update_batch([b"minhash", b"is", b"a", b"probabilistic", b"data", b"structure"])
m2.update_batch([b"minhash", b"is", b"a", b"probability", b"data", b"structure"])
m1.jaccard(m2)  # 0.65625, as with datasketch

sigs = MinHash.bulk([[b"a", b"b"], [b"b", b"c"]], num_perm=64, threads=8)

Differences from datasketch

  • A hashfunc other than sha1_hash32/sha1_hash64 (pyroid's or datasketch's) is called from Python for every token, as in datasketch, with a one-time pyroid.FallbackWarning; the permutations still run in Rust when it returns ints in range.
  • Tokens that are not bytes (bytearray, memoryview, numpy scalars), and hash values or permutations that are not the scheme's numpy dtype, run datasketch's numpy code (same results, datasketch's speed).
  • gpu_mode is accepted, but pyroid always computes on the CPU; the values are the same. gpu_mode="always" still raises RuntimeError without CuPy and a CUDA device, as datasketch does.
  • CuPy is imported only when a MinHash with gpu_mode other than "disable" updates, not when the module is imported.

hashvalue_byte_size module-attribute 

hashvalue_byte_size: int

MinHash 

MinHash(
    num_perm: int = 128,
    seed: int = 1,
    gpu_mode: Literal[
        "disable", "detect", "always"
    ] = "disable",
    hashfunc: Callable[[Any], Any] | None = None,
    hashobj: object | None = None,
    hashvalues: ArrayLike | None = None,
    permutations: tuple[ArrayLike, ArrayLike]
    | ArrayLike
    | None = None,
    scheme: _Scheme | None = None,
)

MinHash: estimate the Jaccard similarity and cardinality of sets.

Parameters:

  • num_perm (int, default: 128 ) –

    Number of random permutation functions. Ignored if hashvalues is given.

  • seed (int, default: 1 ) –

    The random seed of the permutation functions.

  • gpu_mode (Literal['disable', 'detect', 'always'], default: 'disable' ) –

    "disable", "detect" or "always". pyroid computes on the CPU in every mode; "always" raises RuntimeError if CuPy and a CUDA device are not available, as in datasketch.

  • hashfunc (Callable[[Any], Any] | None, default: None ) –

    The hash function: takes a value given to update and returns an int of 32 bits ("affine32", "legacy") or 64 bits ("affine64"). Default: sha1_hash32, or sha1_hash64 for "affine64". Other functions run in Python (slow path).

  • hashobj (object | None, default: None ) –

    Deprecated since datasketch 1.4.0; a no-op that warns.

  • hashvalues (ArrayLike | None, default: None ) –

    Initial hash values (another MinHash's hashvalues). scheme must then be given.

  • permutations (tuple[ArrayLike, ArrayLike] | ArrayLike | None, default: None ) –

    Permutation parameters (another MinHash's permutations). scheme must then be given.

  • scheme (_Scheme | None, default: None ) –

    "affine32" (the default), "affine64" or "legacy" (datasketch before 2.0.0). MinHash of different schemes cannot be compared, merged or unioned.

Raises:

  • ValueError –

    As datasketch: an unknown scheme, num_perm out of range, hashvalues/permutations without scheme, mismatched lengths, values out of the scheme's range.

scheme instance-attribute 

scheme: str

seed instance-attribute 

seed: Any

num_perm instance-attribute 

num_perm: int

hashfunc instance-attribute 

hashfunc: Callable[[Any], Any]

hashvalues instance-attribute 

hashvalues: NDArray[Any]

permutations instance-attribute 

permutations: Any

update 

update(b: Any) -> None

Add one value to the set.

With the default hash function the value must be bytes-like, e.g. m.update("text".encode("utf-8")).

Parameters:

update_batch 

update_batch(b: Iterable[Any]) -> None

Add many values to the set (faster than update per value).

Parameters:

  • b (Iterable[Any]) –

    The values, each passed to hashfunc. An empty batch leaves the hash values unchanged.

Raises:

  • RuntimeError –

    gpu_mode="always" without CuPy and a CUDA device.

jaccard 

jaccard(other: MinHash) -> float

Estimate the Jaccard similarity with other.

Raises:

  • ValueError –

    Different schemes, seeds or numbers of permutations.

count 

count() -> float

Estimate the cardinality of the set (a numpy.float64).

merge 

merge(other: MinHash) -> None

Make this MinHash the union of itself and other.

Raises:

  • ValueError –

    Different schemes, seeds or numbers of permutations.

digest 

digest() -> NDArray[Any]

Return a copy of hashvalues.

is_empty 

is_empty() -> bool

Return whether the MinHash is in its initial state.

clear 

clear() -> None

Reset the hash values to the initial state.

copy 

copy() -> MinHash

Return a copy (the permutations are shared, as in datasketch).

union classmethod 

union(*mhs: MinHash) -> MinHash

Return a new MinHash, the union of mhs (at least two).

Raises:

  • ValueError –

    Fewer than two, or different seeds, sizes or schemes.

bulk classmethod 

bulk(
    b: Iterable[Iterable[Any]],
    *,
    threads: int | None = None,
    **minhash_kwargs: Any,
) -> list[MinHash]

Compute one MinHash per document of b.

Equal to [m.copy() updated with update_batch(doc) for doc in b] with m = MinHash(**minhash_kwargs), as in datasketch. Documents that are lists or tuples of bytes are hashed in parallel on pyroid's thread pool; others run datasketch's code.

Parameters:

  • b (Iterable[Iterable[Any]]) –

    The documents, each an iterable of values for hashfunc.

  • threads (int | None, default: None ) –

    pyroid extension: at most this many threads (None: the whole pool, 1: the calling thread).

  • **minhash_kwargs (Any, default: {} ) –

    Arguments for MinHash, used for all.

Example 
data = [[b"token1", b"token2", b"token3"], [b"token4", b"token5"]]
minhashes = MinHash.bulk(data, num_perm=64)

generator classmethod 

generator(
    b: Iterable[Iterable[Any]],
    *,
    threads: int | None = None,
    **minhash_kwargs: Any,
) -> Generator[MinHash]

Yield one MinHash per document of b (see bulk).

pyroid reads up to 2,048 documents ahead to hash them in parallel; the MinHash objects and any error come out in datasketch's order.

minhash_batch 

minhash_batch(
    docs: str
    | list[str]
    | tuple[str, ...]
    | ArrowStrings
    | Sequence[Sequence[bytes | str]],
    *,
    num_perm: int = 128,
    seed: int = 1,
    scheme: _Scheme | None = None,
    ngram: int = 1,
    threads: int | None = None,
) -> NDArray[uint32] | NDArray[uint64]

Compute MinHash signatures for many documents, as a matrix.

Row i equals MinHash(num_perm, seed, scheme=scheme) updated with the tokens of document i, i.e. its MinHash.hashvalues (uint32 for "affine32", uint64 otherwise). Documents are hashed in parallel on pyroid's thread pool. A pyroid extension.

Parameters:

  • docs (str | list[str] | tuple[str, ...] | ArrowStrings | Sequence[Sequence[bytes | str]]) –

    Token lists (a list or tuple of lists/tuples of bytes or str tokens; str is hashed as UTF-8), or texts (str, list[str], tuple[str, ...] or an Arrow string array). A text's tokens are its word ngram-grams: " ".join(words[i:i + ngram]) over text.split(), UTF-8 encoded; a text with fewer words has no tokens. Arrow nulls give the empty signature.

  • num_perm (int, default: 128 ) –

    Number of permutations.

  • seed (int, default: 1 ) –

    The permutation seed.

  • scheme (_Scheme | None, default: None ) –

    "affine32" (default), "affine64" or "legacy".

  • ngram (int, default: 1 ) –

    Words per shingle, for texts only.

  • threads (int | None, default: None ) –

    At most this many threads (None: the whole pool).

Returns:

  • NDArray[uint32] | NDArray[uint64] –

    A (len(docs), num_perm) array, or (num_perm,) for a single str.

Raises:

Example 
from pyroid.datasketch import MinHashLSH, minhash_batch

sigs = minhash_batch(texts, ngram=5, threads=8)  # (len(texts), 128) uint32
lsh = MinHashLSH(threshold=0.8)
lsh.insert_batch(range(len(texts)), sigs)

pyroid.datasketch.lean_minhash

datasketch's LeanMinHash (datasketch.lean_minhash).

A frozen pyroid.datasketch.MinHash without permutations or hash function, for storage and indexing. LeanMinHash.serialize writes datasketch 2.0.0's byte format (the pre-2.0.0 format for "legacy"), so the bytes are interchangeable with datasketch's in both directions; pickles hold the same bytes.

Example 
from pyroid.datasketch import LeanMinHash

lean = LeanMinHash(minhash)
buf = bytearray(lean.bytesize())
lean.serialize(buf)
LeanMinHash.deserialize(buf) == lean  # True

LeanMinHash 

LeanMinHash(
    minhash: MinHash | None = None,
    seed: int | None = None,
    hashvalues: Iterable[int] | None = None,
    scheme: str | None = None,
)

Bases: MinHash

A MinHash with its state frozen: no update.

Parameters:

  • minhash (MinHash | None, default: None ) –

    The MinHash to take the seed, hash values and scheme from.

  • seed (int | None, default: None ) –

    With hashvalues and scheme: the seed of the permutations.

  • hashvalues (Iterable[int] | None, default: None ) –

    With seed and scheme: the hash values.

  • scheme (str | None, default: None ) –

    The scheme of hashvalues (required with them; "legacy" for values from datasketch before 2.0.0). With minhash it may only repeat the MinHash's scheme.

Raises:

  • ValueError –

    Neither minhash nor seed and hashvalues; no or conflicting scheme; empty hashvalues for an affine scheme.

update 

update(b: Any) -> None

Raise TypeError: a LeanMinHash cannot be updated.

copy 

copy() -> LeanMinHash

Return a copy.

bytesize 

bytesize(byteorder: str = '@') -> int

Return the size of the serialized form in bytes.

Parameters:

  • byteorder (str, default: '@' ) –

    A struct byte order character: @ (native, the default), =, <, > or !.

serialize 

serialize(buf: Buffer, byteorder: str = '@') -> None

Write this LeanMinHash into the writable buffer buf, from offset 0.

Layout ("legacy", as before datasketch 2.0.0): seed (8 bytes), the number of hash values (4 bytes), then each value in 4 bytes. Affine schemes: seed, the negated number of values, a scheme code byte (1 for "affine32", 2 for "affine64"), then each value in 4 or 8 bytes.

Raises:

deserialize classmethod 

deserialize(
    buf: Buffer, byteorder: str = "@"
) -> LeanMinHash

Read a LeanMinHash written by serialize (pyroid's or datasketch's).

Buffers written before datasketch 2.0.0 come back with scheme="legacy".

Raises:

union classmethod 

union(*lmhs: LeanMinHash) -> LeanMinHash

Return a new LeanMinHash, the union of lmhs (at least two).

pyroid.datasketch.lsh

MinHash LSH (pyroid.datasketch.lsh), as datasketch 2.0.0's datasketch.lsh.

MinHashLSH has datasketch's constructor, methods, attributes and error messages, and returns the same query results (as sets: result lists are in no particular order, as in datasketch). The in-memory index is in Rust:

from pyroid.datasketch.lsh import MinHashLSH

lsh = MinHashLSH(threshold=0.5, num_perm=128)
for key, m in minhashes.items():
    lsh.insert(key, m)
lsh.query(minhashes["doc-1"])
lsh.insert_batch(keys, list_of_minhashes, threads=8)  # pyroid extension
lsh.query_batch(signatures_2d_array)  # pyroid extension

The index reads hashvalues, len() and scheme of the sketches it is given, so datasketch's MinHash, LeanMinHash and WeightedMinHash work as well as pyroid's. Band keys are computed in Rust when hashvalues is a C-contiguous numpy array of an integer or float dtype in native byte order; anything else runs datasketch's own expression bytes(hashvalues[start:end].byteswap().data).

Differences from datasketch:

  • storage_config of type "redis" or "cassandra", and a hashfunc, are handled by datasketch: MinHashLSH(...) returns a datasketch.MinHashLSH and emits a pyroid.FallbackWarning (once).
  • keys and hashtables[i] are read-only views (pyroid.datasketch.storage.DictListStorageView, pyroid.datasketch.storage.DictSetStorageView). Their values are new lists and sets on every access (a snapshot), so a query buffered with MinHashLSH.add_to_query_buffer sees the index as it was then. The mutating storage methods raise NotImplementedError. Band tables iterate in no particular order (keys iterates in insertion order, as a dict).
  • When several equal keys of different types are involved (1, 1.0, True), the object returned may be another one of the equal keys than datasketch returns.

The (b, r) search for threshold and weights is a Rust port of datasketch's (scipy.integrate.quad) and picks the same parameters; it needs neither datasketch nor scipy.

MinHashLSH 

MinHashLSH(
    threshold: float = 0.9,
    num_perm: int = 128,
    weights: tuple[float, float] = (0.5, 0.5),
    params: tuple[int, int] | None = None,
    storage_config: dict[str, Any] | None = None,
    prepickle: bool | None = None,
    hashfunc: Callable[[bytes], bytes] | None = None,
)

The MinHash LSH index, for queries with a Jaccard similarity threshold.

Parameters:

  • threshold (float, default: 0.9 ) –

    The Jaccard similarity threshold, in [0.0, 1.0]. The band layout is chosen to minimize the weighted false positive and false negative probabilities at this threshold.

  • num_perm (int, default: 128 ) –

    The number of permutations of the MinHashes to index (the sample_size of a weighted MinHash).

  • weights (tuple[float, float], default: (0.5, 0.5) ) –

    (false_positive_weight, false_negative_weight); they sum to 1.0.

  • params (tuple[int, int] | None, default: None ) –

    (b, r), the number of bands and rows per band. Skips the search; threshold and weights are then ignored.

  • storage_config (dict[str, Any] | None, default: None ) –

    {"type": "dict"} (the default) is the in-memory index. "redis" and "cassandra" are delegated to datasketch. "basename" names the storage.

  • prepickle (bool | None, default: None ) –

    Pickle keys to bytes before storing them. Default: true for Redis storage only.

  • hashfunc (Callable[[bytes], bytes] | None, default: None ) –

    Hash function applied to band keys. Delegated to datasketch.

Raises:

  • ValueError –

    For the same invalid arguments as datasketch, with its messages ("Unknown storage type: ..." included).

Attributes:

h instance-attribute 

h: int

b instance-attribute 

b: int

r instance-attribute 

r: int

hashranges instance-attribute 

hashranges: list[tuple[int, int]]

prepickle instance-attribute 

prepickle: bool

hashfunc instance-attribute 

hashfunc: Callable[[bytes], bytes] | None

hashtables instance-attribute 

hashtables: list[UnorderedStorage]

keys instance-attribute 

keys: OrderedStorage

buffer_size property writable 

buffer_size: int

The storage buffer size (used by Redis and Cassandra storage).

insert 

insert(
    key: Hashable,
    minhash: _Sketch,
    check_duplication: bool = True,
) -> None

Insert key with the MinHash (or weighted MinHash) of its set.

Parameters:

  • key (Hashable) –

    The key of the set (any hashable; keys compare with ==).

  • minhash (_Sketch) –

    Its MinHash, of length num_perm.

  • check_duplication (bool, default: True ) –

    Raise if key is already in the index. With False, inserting a key again adds the new band keys to its entry, as datasketch does.

Raises:

  • ValueError –

    The MinHash has the wrong length or another permutation scheme than earlier ones, or key exists.

insert_batch 

insert_batch(
    keys: Iterable[Hashable],
    minhashes: Sequence[_Sketch] | Any,
    *,
    check_duplication: bool = True,
    threads: int | None = None,
) -> None

Insert keys[i] with minhashes[i] for every i (pyroid extension).

The index ends up as after insert of each item in order, but nothing is inserted if an item is invalid: the error is the one insert would raise for the first invalid item, with a note naming it. Band tables are filled in parallel.

Parameters:

  • keys (Iterable[Hashable]) –

    The keys.

  • minhashes (Sequence[_Sketch] | Any) –

    MinHash-like objects, or a numpy array whose rows (axis 0) are hashvalues arrays, e.g. (n, num_perm) uint64 or uint32. Array rows have no scheme, so they are not checked against it (as weighted MinHashes).

  • check_duplication (bool, default: True ) –

    As for insert; a key repeated within the batch is a duplicate too.

  • threads (int | None, default: None ) –

    At most this many tasks at once on pyroid's thread pool (None: the whole pool, 1: the calling thread).

Raises:

  • ValueError –

    keys and minhashes differ in length, or as insert.

  • TypeError –

    An array of another dtype than integers or floats.

merge 

merge(
    other: MinHashLSH, check_overlap: bool = False
) -> None

Merge other into this index, making this one the union of both.

Only num_perm, b and r are compared, as in datasketch.

Parameters:

  • other (MinHashLSH) –

    Another pyroid MinHashLSH.

  • check_overlap (bool, default: False ) –

    Raise if a key is in both indexes (nothing is merged then).

Raises:

  • ValueError –

    The indexes differ in type, parameters or MinHash scheme, or check_overlap found a common key.

insertion_session 

insertion_session(
    buffer_size: int = 50000,
) -> MinHashLSHInsertionSession

Return a context manager whose insert inserts into this index.

deletion_session 

deletion_session(
    buffer_size: int = 50000,
) -> MinHashLSHDeletionSession

Return a context manager whose remove removes from this index.

query 

query(minhash: _Sketch) -> list[Hashable]

Return the keys whose sets likely have a Jaccard similarity above the threshold.

Parameters:

  • minhash (_Sketch) –

    The MinHash of the query set.

Returns:

  • list[Hashable] –

    The keys that share a band with minhash, each once, in no particular order.

Raises:

  • ValueError –

    The MinHash has the wrong length or another scheme.

query_batch 

query_batch(
    minhashes: Sequence[_Sketch] | Any,
    *,
    threads: int | None = None,
) -> list[list[Hashable]]

Return [self.query(m) for m in minhashes], in parallel (pyroid extension).

Parameters:

add_to_query_buffer 

add_to_query_buffer(minhash: _Sketch) -> None

Buffer a query for collect_query_buffer.

The buckets are read now (a snapshot), not when the buffer is collected.

collect_query_buffer 

collect_query_buffer() -> list[Hashable]

Run the buffered queries; return the keys every one of them found.

remove 

remove(key: Hashable) -> None

Remove key from the index.

Raises:

is_empty 

is_empty() -> bool

Return True if some band table is empty (datasketch's definition).

get_counts 

get_counts() -> list[dict[Hashable, int]]

Return, per band table, {band key: number of keys in the bucket}.

get_subset_counts 

get_subset_counts(
    *keys: Hashable,
) -> list[dict[Hashable, int]]

Return get_counts restricted to keys.

MinHashLSHInsertionSession 

MinHashLSHInsertionSession(
    lsh: MinHashLSH, buffer_size: int
)

Context manager for inserting many keys into a MinHashLSH.

lsh instance-attribute 

lsh: MinHashLSH

close 

close() -> None

Flush the buffers (nothing to do for the in-memory index).

insert 

insert(
    key: Hashable,
    minhash: _Sketch,
    check_duplication: bool = True,
) -> None

Insert key with its MinHash, as MinHashLSH.insert.

MinHashLSHDeletionSession 

MinHashLSHDeletionSession(
    lsh: MinHashLSH, buffer_size: int
)

Context manager for removing many keys from a MinHashLSH.

lsh instance-attribute 

lsh: MinHashLSH

close 

close() -> None

Flush the buffers (nothing to do for the in-memory index).

remove 

remove(key: Hashable) -> None

Remove key, as MinHashLSH.remove.

pyroid.datasketch.lshforest

MinHash LSH Forest (pyroid.datasketch.lshforest), as datasketch 2.0.0's.

MinHashLSHForest has datasketch's methods, attributes and errors, and query(minhash, k) returns the same key set as datasketch, including where it stops at k keys: buckets and their keys are visited in datasketch's order. The index is in Rust:

from pyroid.datasketch.lshforest import MinHashLSHForest

forest = MinHashLSHForest(num_perm=128)
for key, m in minhashes.items():
    forest.add(key, m)
forest.index()
forest.query(minhashes["doc-1"], 10)

Keys added after MinHashLSHForest.index are found when they share an indexed band key, as in datasketch.

Differences from datasketch: keys, hashtables[i] and sorted_hashtables[i] are views of the native index with the API of a dict, a defaultdict(list) and a list. Values read from them are new lists. keys accepts assignment and deletion of entries, whose values must be lists of bytes; reading a missing bucket of hashtables[i] returns [] without creating it.

MinHashLSHForest 

MinHashLSHForest(num_perm: int = 128, l: int = 8)

The LSH Forest for MinHash: top-k queries by Jaccard similarity.

Band keys are kept in sorted arrays, one per prefix tree, as in datasketch. It also works with weighted MinHashes.

Parameters:

  • num_perm (int, default: 128 ) –

    The number of permutations of the MinHashes to index (the sample_size of a weighted MinHash).

  • l (int, default: 8 ) –

    The number of prefix trees.

Raises:

  • ValueError –

    num_perm or l is not positive, or l > num_perm.

Attributes:

l instance-attribute 

l: int

k instance-attribute 

k: int

hashranges instance-attribute 

hashranges: list[tuple[int, int]]

keys instance-attribute 

keys: MutableMapping[Hashable, list[bytes]]

hashtables instance-attribute 

hashtables: list[Mapping[bytes, list[Hashable]]]

sorted_hashtables instance-attribute 

sorted_hashtables: list[Sequence[bytes]]

add 

add(key: Hashable, minhash: _Sketch) -> None

Add key with the MinHash of its set.

The key is searchable after the next index (or now, through band keys that are already indexed).

Raises:

  • ValueError –

    The MinHash is shorter than k * l, has another scheme than earlier ones, or key was already added.

index 

index() -> None

Index all the keys added so far and make them searchable.

query 

query(minhash: _Sketch, k: int) -> list[Hashable]

Return up to k keys with (approximately) the highest Jaccard similarity.

Parameters:

  • minhash (_Sketch) –

    The MinHash of the query set.

  • k (int) –

    The maximum number of keys to return.

Returns:

Raises:

  • ValueError –

    k <= 0, or the MinHash is too short or has another scheme.

get_minhash_hashvalues 

get_minhash_hashvalues(key: Hashable) -> Any

Return the hash values stored for key (a numpy array).

Raises:

  • KeyError –

    key is not in the forest.

  • ValueError –

    The stored band keys are not whole 4- or 8-byte values.

is_empty 

is_empty() -> bool

Return True if some tree has nothing searchable (before index).

pyroid.datasketch.lshensemble

LSH Ensemble, as datasketch 2.0.0's datasketch.lshensemble.

MinHashLSHEnsemble answers containment queries: which indexed sets contain at least threshold of the query set. It is datasketch's class on top of pyroid's pyroid.datasketch.MinHashLSH, so the partitions, the (b, r) parameters per partition and the query results are datasketch's. The parameter search (scipy's quad in datasketch) runs in Rust, ten searches in parallel, so constructing an ensemble takes milliseconds.

Example 
from pyroid.datasketch import MinHashLSHEnsemble

ens = MinHashLSHEnsemble(threshold=0.8, num_perm=128, num_part=16)
ens.index([(key, minhash, size) for key, minhash, size in entries])
set(ens.query(query_minhash, query_size))

Redis and Cassandra storage go through pyroid's MinHashLSH, which hands them to datasketch (with a one-time pyroid.FallbackWarning).

logger module-attribute 

logger: Logger

MinHashLSHEnsemble 

MinHashLSHEnsemble(
    threshold: float = 0.9,
    num_perm: int = 128,
    num_part: int = 16,
    m: int = 8,
    weights: tuple[float, float] = (0.5, 0.5),
    storage_config: dict[str, Any] | None = None,
    prepickle: bool | None = None,
)

The LSH Ensemble index for containment queries.

Parameters:

  • threshold (float, default: 0.9 ) –

    The containment threshold, in [0.0, 1.0].

  • num_perm (int, default: 128 ) –

    The number of permutations of the indexed MinHash.

  • num_part (int, default: 16 ) –

    The number of partitions by set size.

  • m (int, default: 8 ) –

    The maximum band size r over the inner LSH indexes, in [2, num_perm].

  • weights (tuple[float, float], default: (0.5, 0.5) ) –

    (false_positive_weight, false_negative_weight), summing to 1.0.

  • storage_config (dict[str, Any] | None, default: None ) –

    As for pyroid.datasketch.MinHashLSH.

  • prepickle (bool | None, default: None ) –

    As for pyroid.datasketch.MinHashLSH.

Raises:

  • ValueError –

    As datasketch, for parameters out of range.

threshold instance-attribute 

threshold: float

h instance-attribute 

h: int

m instance-attribute 

m: int

xqs instance-attribute 

xqs: NDArray[float64]

params instance-attribute 

params: NDArray[Any]

indexes instance-attribute 

indexes: list[dict[int, MinHashLSH]]

lowers instance-attribute 

lowers: list[Any]

uppers instance-attribute 

uppers: list[Any]

index 

index(
    entries: Iterable[tuple[Hashable, MinHash, int]],
) -> None

Index every (key, minhash, size) of entries; callable once.

Raises:

  • ValueError –

    The index is not empty, entries is empty, or a size is not positive.

query 

query(minhash: MinHash, size: int) -> Generator[Hashable]

Yield the keys of sets likely to contain threshold of the query set.

Parameters:

  • minhash (MinHash) –

    The MinHash of the query set.

  • size (int) –

    The number of distinct items in the query set.

is_empty 

is_empty() -> bool

Return whether nothing has been indexed.

pyroid.datasketch.storage

Storage containers (pyroid.datasketch.storage), as datasketch.storage.

Storage, OrderedStorage, UnorderedStorage, DictListStorage, DictSetStorage, ordered_storage and unordered_storage are datasketch's in-memory classes and factories. "redis" and "cassandra" configurations are passed to datasketch's factories (a pyroid.FallbackWarning).

MinHashLSH keeps its index in Rust; its keys and hashtables[i] are the read-only views DictListStorageView and DictSetStorageView, with the same read API. Their values are new lists and sets on every call (a snapshot).

Storage

Bases: ABC

Base class for key, value containers where the values are sequences.

buffer_size property writable 

buffer_size: int

The buffer size (used by the Redis and Cassandra storages).

keys abstractmethod 

keys() -> Any

Return the keys.

get abstractmethod 

get(key: Any) -> Any

Return the values of key, empty if it is absent.

getmany 

getmany(*keys: Any) -> list[Any]

Return [self.get(key) for key in keys].

insert abstractmethod 

insert(key: Any, *vals: Any, **kwargs: Any) -> None

Add vals to the values of key.

remove abstractmethod 

remove(*keys: Any, **kwargs: Any) -> None

Remove keys.

remove_val abstractmethod 

remove_val(key: Any, val: Any) -> None

Remove val from the values of key.

size abstractmethod 

size() -> int

Return the number of keys.

itemcounts abstractmethod 

itemcounts(**kwargs: Any) -> dict[Any, int]

Return the number of values of each key.

has_key abstractmethod 

has_key(key: Any) -> bool

Return whether key is stored.

status 

status() -> dict[str, Any]

Return {"keyspace_size": len(self)}.

empty_buffer 

empty_buffer() -> None

Flush buffered writes (nothing to do in memory).

add_to_select_buffer 

add_to_select_buffer(keys: Any) -> None

Look up keys and buffer the results.

collect_select_buffer 

collect_select_buffer() -> list[Any]

Return the buffered results and clear the buffer.

OrderedStorage

Bases: Storage

A storage whose values are lists (defaultdict(list)).

UnorderedStorage

Bases: Storage

A storage whose values are sets (defaultdict(set)).

DictListStorage 

DictListStorage(config: Any)

Bases: OrderedStorage

defaultdict(list) with the Storage API.

keys 

keys() -> Any

Return the keys (a dict keys view).

get 

get(key: Any) -> Any

Return the list stored under key, [] if there is none.

remove 

remove(*keys: Any, **kwargs: Any) -> None

Delete keys (KeyError if one is absent).

remove_val 

remove_val(key: Any, val: Any, **kwargs: Any) -> None

Remove val from the values of key.

insert 

insert(key: Any, *vals: Any, **kwargs: Any) -> None

Add vals to the values of key.

size 

size() -> int

Return the number of keys.

itemcounts 

itemcounts(**kwargs: Any) -> dict[Any, int]

Return {key: number of values}.

has_key 

has_key(key: Any) -> bool

Return whether key is stored.

DictSetStorage 

DictSetStorage(config: Any)

Bases: UnorderedStorage, DictListStorage

defaultdict(set) with the Storage API.

get 

get(key: Any) -> Any

Return the set stored under key, set() if there is none.

insert 

insert(key: Any, *vals: Any, **kwargs: Any) -> None

Add vals to the set of key.

DictListStorageView 

DictListStorageView(index: Any)

Bases: OrderedStorage

Read-only view of MinHashLSH.keys: key -> its band keys.

It has DictListStorage's API. insert, remove and remove_val raise NotImplementedError.

keys 

keys() -> list[Hashable]

Return the keys in insertion order (a new list).

get 

get(key: Hashable) -> list[bytes]

Return the band keys of key (a new list), [] if it is absent.

insert 

insert(key: Any, *vals: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

remove 

remove(*keys: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

remove_val 

remove_val(key: Any, val: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

size 

size() -> int

Return the number of keys.

itemcounts 

itemcounts(**kwargs: Any) -> dict[Hashable, int]

Return {key: number of band keys}.

has_key 

has_key(key: Hashable) -> bool

Return whether key is in the index.

DictSetStorageView 

DictSetStorageView(index: Any, table: int)

Bases: UnorderedStorage

Read-only view of one band table, MinHashLSH.hashtables[i].

It has DictSetStorage's API: band key (bytes) -> the set of keys in that bucket. Buckets iterate in no particular order. insert, remove and remove_val raise NotImplementedError.

keys 

keys() -> list[bytes]

Return the band keys of the buckets (a new list).

get 

get(key: Any) -> set[Hashable]

Return the keys in bucket key (a new set), set() if it is absent.

insert 

insert(key: Any, *vals: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

remove 

remove(*keys: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

remove_val 

remove_val(key: Any, val: Any, **kwargs: Any) -> None

Raise NotImplementedError: the view is read-only.

size 

size() -> int

Return the number of buckets.

itemcounts 

itemcounts(**kwargs: Any) -> dict[bytes, int]

Return {band key: number of keys in the bucket}.

has_key 

has_key(key: Any) -> bool

Return whether there is a bucket key.

ordered_storage 

ordered_storage(
    config: dict[str, Any], name: bytes | None = None
) -> OrderedStorage

Return an ordered storage (values are lists) for config.

Raises:

unordered_storage 

unordered_storage(
    config: dict[str, Any], name: bytes | None = None
) -> UnorderedStorage

Return an unordered storage (values are sets) for config.

Raises:

pyroid.datasketch.hashfunc

datasketch's SHA1-based hash functions (datasketch.hashfunc), unchanged.

pyroid.datasketch.MinHash recognises these two (and datasketch's own) and hashes natively. Called directly they are plain hashlib code, so they take and reject what datasketch's take and reject (bytes-like data only).

sha1_hash32 

sha1_hash32(data: Buffer) -> int

Return the first 4 bytes of sha1(data) as a little-endian int.

sha1_hash64 

sha1_hash64(data: Buffer) -> int

Return the first 8 bytes of sha1(data) as a little-endian int.