Analysis Provenance

Use provenance manifests to identify the exact DataFrame and configuration supplied to an analysis without storing the raw records in the manifest itself.

Deterministic provenance utilities for missing-data analyses.

The functions in this module create content-derived identifiers without persisting the caller’s raw records. They deliberately include pandas dtype, index, and categorical metadata because those properties can change the meaning or result of an imputation workflow even when printed values look the same.

missingly.provenance.analysis_provenance(df, operation, parameters=None)[source]

Build a replay-oriented provenance manifest for an analysis.

Parameters:

  • df (pandas.DataFrame) – Exact input data supplied to the analytical operation.

  • operation (str) – Stable public operation name, such as "impute_mice".

  • parameters (mapping of str to Any, optional) – Deterministic operation settings, including any random seed.

Returns:

JSON-compatible manifest containing schema and package versions, the normalized settings, an input-only digest, and a configured-analysis digest.

Return type:

dict

Raises:

  • TypeError – If the input or context violates the deterministic serialization contract.

  • ValueError – If operation is empty or contains only whitespace.

Examples

>>> import pandas as pd
>>> from missingly.provenance import analysis_provenance
>>> frame = pd.DataFrame({"age": [20.0, None, 40.0]})
>>> manifest = analysis_provenance(
...     frame,
...     "impute_mice",
...     {"m": 5, "random_state": 42},
... )
>>> manifest["package"]
'missingly'
>>> len(manifest["analysis_sha256"])
64
missingly.provenance.dataframe_fingerprint(df, *, operation=None, parameters=None)[source]

Return a deterministic SHA-256 identity for a DataFrame and context.

The fingerprint covers column order and labels, dtype metadata, index structure and values, cell values, DataFrame.attrs, and—when supplied—the analytical operation and its parameters. The function does not modify df and does not write raw data anywhere.

Parameters:

  • df (pandas.DataFrame) – Input data whose exact analytical identity should be recorded.

  • operation (str, optional) – Stable operation name, for example "impute_mice". Include it when the identifier should distinguish different workflows on the same input.

  • parameters (mapping of str to Any, optional) – Operation settings such as seeds, method names, and iteration counts. Values must consist of deterministic scalar or container types.

Returns:

A lowercase, 64-character hexadecimal SHA-256 digest.

Return type:

str

Raises:

TypeError – If df is not a DataFrame, the context types are invalid, or a value cannot be represented deterministically.

Examples

>>> import pandas as pd
>>> from missingly.provenance import dataframe_fingerprint
>>> frame = pd.DataFrame({"age": [20.0, None, 40.0]})
>>> digest = dataframe_fingerprint(
...     frame,
...     operation="impute_simple",
...     parameters={"strategy": "mean"},
... )
>>> len(digest)
64
>>> digest == dataframe_fingerprint(
...     frame,
...     operation="impute_simple",
...     parameters={"strategy": "mean"},
... )
True