Report Module

The xpyrment.report module contains submodules and components for report.

report

Experimental reporting, lifecycle tracking, and scientific visualization.

This package provides logging, recording, and charting tools to summarize experimental phases. It ensures that setups, runtime quality checks, and analytical inferences are aggregated and presented in standard-compliant, publication-ready formats.

Submodules: - card: Compiles standard, machine-readable Experiment Cards for metadata cataloging. - audit: Logs and chains immutable lifecycle events for audit and governance compliance. - export: Generates high-quality statistical plots (Forest plots, Power curves).

MODULE	DESCRIPTION
`audit`	Immutable, compliance-ready experimental audit trails.
`card`	Unified, standard-compliant Experiment Cards and meta-ledgers.
`export`	Statistical visualization and plot generation for experimental reports.
`generator`

CLASS	DESCRIPTION
`ExperimentCard`	Consumes metadata, planning state, and calculations to compile a unified report card.
`AuditTrail`	Maintains an immutable, compliance-ready audit trail of experiment phase transition events.
`ExperimentReportGenerator`	Generates premium standalone Markdown and HTML reports from experiment AnalysisResult instances.

FUNCTION	DESCRIPTION
`plot_forest`	Generates a horizontal forest plot visualizing relative lift and confidence intervals.
`plot_power_curve`	Plots required sample size per variant across a range of Minimum Detectable Effects (MDE).

ExperimentCard

ExperimentCard(
    experiment_id: str,
    plan_spec: dict,
    validation_spec: dict,
    analysis_summary: dict,
)

Consumes metadata, planning state, and calculations to compile a unified report card.

An Experiment Card (inspired by Model Cards, Mitchell et al. 2019) is the definitive, unified record and metadata registry of an experiment. It acts as a standardized document that records the design, execution, and results of an experiment in a machine-readable format. This makes it possible to search, catalog, and run large-scale meta-analyses across thousands of past experiments (e.g., tracking cumulative lift, estimating p-value distributions, or measuring historical power).

The Experiment Card schema unifies three core lifecycle stages

Planning & Setup Specification (plan_spec):
mde: Minimum Detectable Effect (relative or absolute).
alpha: Nominal Type I error rate (e.g., $0.05$).
power: Target statistical power ($1 - \\beta$, e.g., $0.80$).
target_sample_size: Calculated sample size requirement.
metric_registry: Names and types of registered primary, secondary, and guardrail metrics.
Runtime Validation & Diagnostics (validation_spec):
srm_p_value: Pearson Chi-Square goodness-of-fit p-value for sample allocation ratio mismatches.
covariate_balance: Standardized Mean Differences (SMDs) confirming unbiased random assignments.
Statistical Analysis Outcomes (analysis_summary):
treatment_effect: Relative and absolute lifts, standard errors, and confidence intervals.
p_values: Observed p-values (with any multiple-testing adjustments applied).
recommendation: Automated decision outcome (e.g., "SHIP", "NO-SHIP", "INCONCLUSIVE").

ATTRIBUTE	DESCRIPTION
`experiment_id`	Unique tracking identifier for the experiment. TYPE: `str`
`plan_spec`	Setup configurations, expected metrics, and calculated power parameters. TYPE: `dict`
`validation_spec`	Summary of SRM and covariate balance diagnostics. TYPE: `dict`
`analysis_summary`	Calculated point estimates, confidence intervals, and launch recommendations. TYPE: `dict`

PARAMETER	DESCRIPTION
`experiment_id`	The unique ID of the experiment. TYPE: `str`
`plan_spec`	Setup and design characteristics dictionary. TYPE: `dict`
`validation_spec`	Pre-analysis quality check outcomes. TYPE: `dict`
`analysis_summary`	Post-analysis statistical summaries. TYPE: `dict`

METHOD	DESCRIPTION
`to_dict`	Serializes the experiment card metadata to a standard python dictionary.
`to_json`	Dumps the card as a formatted JSON document.

Source code in src\xpyrment\report\card.py

def __init__(self, experiment_id: str, plan_spec: dict, validation_spec: dict, analysis_summary: dict):
    """Initializes a new ExperimentCard.

    Args:
        experiment_id (str): The unique ID of the experiment.
        plan_spec (dict): Setup and design characteristics dictionary.
        validation_spec (dict): Pre-analysis quality check outcomes.
        analysis_summary (dict): Post-analysis statistical summaries.
    """
    self.experiment_id = experiment_id
    self.plan_spec = plan_spec
    self.validation_spec = validation_spec
    self.analysis_summary = analysis_summary

to_dict

to_dict() -> dict

Serializes the experiment card metadata to a standard python dictionary.

RETURNS	DESCRIPTION
`dict`	The nested dictionary of card metadata. TYPE: `dict`

Source code in src\xpyrment\report\card.py

def to_dict(self) -> dict:
    """Serializes the experiment card metadata to a standard python dictionary.

    Returns:
        dict: The nested dictionary of card metadata.
    """
    return {
        "experiment_id": self.experiment_id,
        "plan_spec": self.plan_spec,
        "validation_spec": self.validation_spec,
        "analysis_summary": self.analysis_summary
    }

to_json

to_json() -> str

Dumps the card as a formatted JSON document.

RETURNS	DESCRIPTION
`str`	Indented, pretty-printed JSON string of the complete experiment card ledger. TYPE: `str`

Source code in src\xpyrment\report\card.py

def to_json(self) -> str:
    """Dumps the card as a formatted JSON document.

    Returns:
        str: Indented, pretty-printed JSON string of the complete experiment card ledger.
    """
    return json.dumps(self.to_dict(), indent=2)

AuditTrail

AuditTrail(experiment_id: str, db_path: str = None)

Maintains an immutable, compliance-ready audit trail of experiment phase transition events.

In enterprise, financial, and clinical environments, maintaining a rigorous record of an experiment's history is critical for governance, auditing, and scientific reproducibility. An audit trail acts as a tamper-evident, chronological log tracking every key lifecycle change, modification to allocation parameters, and analytical peeking event.

Cryptographic Verification and State-Chaining

To satisfy strict regulatory compliance frameworks, the audit log entries are structured as a linear hash chain: - Each log entry is represented as a state block $B_k = (t_k, a_k, d_k, h_{k-1})$ where: - $t_k$: Coordinated Universal Time (ISO 8601 UTC timestamp). - $a_k$: The action or state transition executed (e.g., "ALLOCATION_SHIFT"). - $d_k$: Detailed parameter changes (e.g., altering treatment allocation from $10\\%$ to $50\\%$). - $h_{k-1}$: The SHA-256 cryptographic hash of the preceding block $B_{k-1}$. - The hash of the current block $h_k$ is computed as: $$ h_k = H(t_k \parallel a_k \parallel d_k \parallel h_{k-1}) $$ where $\\parallel$ denotes string concatenation, and $H$ is the SHA-256 secure hash function. - Because of this chaining, any retroactive modification of historical logs immediately breaks the hash chain, making the log highly secure and tamper-evident.

ATTRIBUTE	DESCRIPTION
`experiment_id`	The unique identifier of the experiment under audit. TYPE: `str`
`logs`	List of chronological, cryptographically linked log events. TYPE: `List[Dict[str, str]]`

PARAMETER	DESCRIPTION
`experiment_id`	The unique ID of the target experiment. TYPE: `str`
`db_path`	Optional SQLite database path for tamper-proof persistence. TYPE: `str` DEFAULT: `None`

METHOD	DESCRIPTION
`log_event`	Appends a new event with an active timestamp to the audit trail log.
`verify_integrity`	Verifies the complete cryptographic chain of the audit trail ledger.
`get_logs`	Returns the full list of chronological logs in the audit ledger.

Source code in src\xpyrment\report\audit.py

def __init__(self, experiment_id: str, db_path: str = None):
    """Initializes an AuditTrail log.

    Args:
        experiment_id (str): The unique ID of the target experiment.
        db_path (str, optional): Optional SQLite database path for tamper-proof persistence.
    """
    self.experiment_id = experiment_id
    self.db_path = db_path
    self.logs: List[Dict[str, str]] = []

    if self.db_path:
        import sqlite3
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            cursor.execute(
                '''CREATE TABLE IF NOT EXISTS audit_logs (
                    experiment_id TEXT,
                    timestamp TEXT,
                    action TEXT,
                    details TEXT,
                    prev_hash TEXT,
                    hash TEXT,
                    signature TEXT,
                    public_key TEXT
                )'''
            )
            conn.commit()

log_event

log_event(
    action: str,
    details: str,
    signature: str = None,
    public_key: str = None,
)

Appends a new event with an active timestamp to the audit trail log.

Calculates timestamps in strict UTC, hashes the event details with the prior block's hash, and appends the entry to the ledger. Optionally attaches RSA/ECDSA digital signatures.

PARAMETER	DESCRIPTION
`action`	The action category (e.g., `"PHASE_TRANSITION"`, `"ALLOCATION_MODIFIED"`). TYPE: `str`
`details`	Detailed text or JSON payload describing the parameters or user that initiated the change. TYPE: `str`
`signature`	Cryptographic signature of the event hash. TYPE: `str` DEFAULT: `None`
`public_key`	Public key string to verify the signature. TYPE: `str` DEFAULT: `None`

Source code in src\xpyrment\report\audit.py

def log_event(self, action: str, details: str, signature: str = None, public_key: str = None):
    """Appends a new event with an active timestamp to the audit trail log.

    Calculates timestamps in strict UTC, hashes the event details with the prior block's hash,
    and appends the entry to the ledger. Optionally attaches RSA/ECDSA digital signatures.

    Args:
        action (str): The action category (e.g., `"PHASE_TRANSITION"`, `"ALLOCATION_MODIFIED"`).
        details (str): Detailed text or JSON payload describing the parameters or user that initiated the change.
        signature (str, optional): Cryptographic signature of the event hash.
        public_key (str, optional): Public key string to verify the signature.
    """
    import hashlib

    timestamp = datetime.datetime.now(datetime.UTC).isoformat()
    prev_hash = "0" * 64 if len(self.logs) == 0 else self.logs[-1]["hash"]

    safe_signature = signature or ""
    safe_public_key = public_key or ""
    # Formulate canonical block string for SHA-256 hashing
    data_str = f"{timestamp}||{action}||{details}||{prev_hash}||{safe_signature}||{safe_public_key}"
    current_hash = hashlib.sha256(data_str.encode("utf-8")).hexdigest()

    log_entry = {
        "timestamp": timestamp,
        "action": action,
        "details": details,
        "prev_hash": prev_hash,
        "hash": current_hash,
        "signature": safe_signature,
        "public_key": safe_public_key
    }
    self.logs.append(log_entry)

    if self.db_path:
        import sqlite3
        with sqlite3.connect(self.db_path) as conn:
            cursor = conn.cursor()
            cursor.execute(
                '''INSERT INTO audit_logs (experiment_id, timestamp, action, details, prev_hash, hash, signature, public_key)
                   VALUES (?, ?, ?, ?, ?, ?, ?, ?)''',
                (self.experiment_id, timestamp, action, details, prev_hash, current_hash, log_entry["signature"], log_entry["public_key"])
            )
            conn.commit()

verify_integrity

verify_integrity() -> bool

Verifies the complete cryptographic chain of the audit trail ledger.

RETURNS	DESCRIPTION
`bool`	True if the hash chain is fully intact and unmodified, False otherwise. TYPE: `bool`

Source code in src\xpyrment\report\audit.py

def verify_integrity(self) -> bool:
    """Verifies the complete cryptographic chain of the audit trail ledger.

    Returns:
        bool: True if the hash chain is fully intact and unmodified, False otherwise.
    """
    import hashlib

    for i in range(len(self.logs)):
        block = self.logs[i]
        expected_prev = "0" * 64 if i == 0 else self.logs[i-1]["hash"]

        if block["prev_hash"] != expected_prev:
            return False

        # Recalculate block hash
        data_str = f"{block['timestamp']}||{block['action']}||{block['details']}||{block['prev_hash']}||{block.get('signature', '')}||{block.get('public_key', '')}"
        actual_hash = hashlib.sha256(data_str.encode("utf-8")).hexdigest()

        if block["hash"] != actual_hash:
            return False

    return True

get_logs

get_logs() -> List[Dict[str, str]]

Returns the full list of chronological logs in the audit ledger.

RETURNS	DESCRIPTION
`List[Dict[str, str]]`	List[Dict[str, str]]: A list of dictionary objects representing the serialized ledger blocks.

Source code in src\xpyrment\report\audit.py

def get_logs(self) -> List[Dict[str, str]]:
    """Returns the full list of chronological logs in the audit ledger.

    Returns:
        List[Dict[str, str]]: A list of dictionary objects representing the serialized ledger blocks.
    """
    return self.logs

ExperimentReportGenerator

ExperimentReportGenerator(
    result: AnalysisResult,
    experiment_name: str = "A/B Experiment Report",
)

Generates premium standalone Markdown and HTML reports from experiment AnalysisResult instances.

PARAMETER	DESCRIPTION
`result`	The completed analysis result object. TYPE: `AnalysisResult`
`experiment_name`	The logical name of the experiment. TYPE: `str` DEFAULT: `'A/B Experiment Report'`

RAISES	DESCRIPTION
`ValueError`	If the analysis results are empty or invalid.

METHOD	DESCRIPTION
`generate_markdown`	Generates a complete, beautiful GitHub-compatible Markdown summary card.
`generate_html`	Generates a premium, self-contained interactive HTML dashboard of the results.
`save_html`	Saves the beautifully compiled HTML dashboard report to a local file.
`save_markdown`	Saves the GitHub-compatible Markdown summary report to a local file.