# Measurement, instrumentation, and reproducibility protocol

**Status:** proposal-derived protocol specification. It is not evidence that experiments have been run or that candidate metrics are validated.

## 1. Experimental unit and comparison

The experimental unit is **Agent × Task × Run × Condition**. The primary comparison, where authorized and methodologically feasible, is the existing evaluation setting versus that setting augmented by a longitudinal measurement layer. This is not a comparison of MENS-SCCA against APEX and does not require adoption of MENS-SCCA internals by the host environment.

A run is represented through observable records: initial state, action, tool interaction, state transformation, evidence reference, error or controlled perturbation, recovery event, final deliverable, and evaluation. The record supports reconstruction of relevant execution history without collecting private chain-of-thought.

## 2. Adapter and canonical record

The proposed flow is:

`authorized task/run → authorized observable extraction → ingestion adapter → canonical trajectory → metric calculation and versioned registry → aggregate analysis → authorized publication`.

The adapter is deliberately decoupled and independently reviewable or replaceable. A minimal canonical record contains:

| Field | Meaning / control |
|---|---|
| `task_id`, `run_id` | Stable identifiers within the authorized environment; avoid embedding confidential content in identifiers. |
| `agent_model_id` | Agent/model identity at the granularity permitted by the data agreement. |
| `condition` | Baseline, perturbation, recovery, or transfer condition as preregistered. |
| `environment_version`, `tool_config` | Versions and configuration necessary to interpret execution. |
| `initial_state` | Authorized, observable starting state or a protected reference to it. |
| `event_sequence`, `state_transitions` | Ordered observable actions and state changes, not private reasoning traces. |
| `evidence_refs` | References to evidence, subject to access control and retention rules. |
| `perturbation_marker`, `recovery_marker` | Whether and where a controlled perturbation/recovery condition occurred. |
| `final_artifact_ref` | Protected reference to the deliverable; publication requires explicit authorization. |
| `rubric_version`, `judge_version`, `evaluation_result` | Versioned evaluation context and output. Preserve expert and model-judge outputs separately. |
| `provenance` | Source class, transformations, access, and publication status. |

Missing or unauthorized fields must be marked as unavailable, not silently inferred. The schema is a research specification, not a demand for privileged access to production systems.

## 3. Candidate measures (exploratory, not validated)

| Measure | Candidate operationalization | Interpretation cautions |
|---|---|---|
| Outcome score | Weighted criterion score when the rubric supports it. | Depends on rubric validity and evaluator; does not describe the path by itself. |
| Trajectory Integrity (TI) | Proportion of required observable properties preserved across the trajectory. | Requires a preregistered definition of required properties and observability. |
| Transformation Fidelity (FT) | Preserved relevant properties divided by preserved + lost + ambiguous properties. | Denominator and treatment of unobservable properties must be declared; undefined cases must not be silently set to zero. |
| Failure Propagation (FP) | Downstream affected properties divided by downstream properties at risk. | Requires explicit dependency and risk sets; association is not automatically causal attribution. |
| Recovery Ratio (RR) | Post-recovery performance relative to an appropriate baseline. | Baseline, scale, and recovery window must be fixed before confirmatory analysis; report cases where ratio is undefined. |
| Transfer Loss (TL) | One minus normalized preservation under transfer. | Normalization and source/target task comparability require justification. |
| Consistency Loss (CL) | One minus normalized cross-run consistency. | Runs must be comparable; report variability and uncertainty, not only a point estimate. |

These definitions are candidate operationalizations from the technical proposal. Their stability, sensitivity, redundancy, reproducibility, agreement with expert assessment, and dependence on instrumentation choices must be tested before any composite score is considered. No universal validity is presumed.

## 4. Evaluation and disagreement

Use deterministic checks for objectively testable constraints; expert-authored rubrics for professional and domain-specific quality; and model-based judges only as auxiliary instruments. Retain the outputs separately. Analyze agreement and disagreement, judge dependence, and critical safety, privacy, compliance, or validity failures explicitly. Do not average critical failures away into an aggregate score.

## 5. Validation sequence

1. Specify constructs, observable properties, denominators, missingness rules, and analysis choices.
2. Establish governance and preregister the confirmatory questions and conditions.
3. Validate the instrument with an authorized API micro-pilot if access is available; otherwise use synthetic fixtures for instrument validation, without presenting them as benchmark evidence.
4. Pilot and calibrate; document changes and version them.
5. Establish baseline measurements.
6. Examine controlled perturbation, recovery, and transfer conditions.
7. Confirm and test robustness across tasks, models, runs, judges, and instrumentation choices as permitted by the design.
8. Synthesize results and disseminate only authorized artifacts.

The proposal's planned sequence is Month 1 specification/governance/preregistration and initial validation; Month 2 pilot/calibration; Month 3 baseline; Month 4 perturbation/recovery/transfer; Month 5 confirmation/robustness; Month 6 synthesis/dissemination.

## 6. Reproducibility registry

For each derived result, retain—within authorized access boundaries—the source observable record or protected reference, task/run/condition identity, environment and tool versions, rubric and judge versions, metric definition/version, transformation history, analysis code/version, and publication authorization. A result is reproducible only to the extent that its required inputs and transformations are available to an authorized reviewer. Public reporting may use aggregates or synthetic fixtures when primary data cannot be released; this limitation must be disclosed.

## 7. Contamination and data governance

Maintain four distinct zones: (A) confidential benchmark materials; (B) expert/reference materials; (C) researcher-controlled MENS-SCCA infrastructure; (D) explicitly authorized derived/public artifacts. No zone transfers into another automatically. Primary data remain in authorized environments; extraction is minimum-necessary.

Where permitted, provenance records should include `artifact_id`, `source_class`, `creation_date`, `access_group`, `task_id`, `run_id`, `benchmark_split`, `training_exposure`, `publication_status`, `license_status`, and `transformation_history`. Keep development and confirmatory data separate. Record known or suspected training exposure and contamination risks; do not claim absence of contamination where exposure cannot be established. Publication eligibility is a controlled status, not an assumption arising from aggregation or anonymization.

Public repositories must not contain reserved tasks, confidential documents, credentials, unnecessary personal information, proprietary content, or trajectories from which protected materials can be reconstructed.

## 8. Falsification and reporting

The proposed incremental value is weakened or unsupported if candidate measures are redundant with existing scores, unstable across runs, dependent on a particular judge, excessively sensitive to instrumentation, or fail to improve identification or explanation of failures. Supporting evidence would require stability, reproducibility, meaningful discrimination of relevant trajectory classes, incremental information beyond outcome scores, and interpretable relation to expert evaluation. Null and negative results are legitimate findings. Prior architecture, prototypes, tests, and registration records establish existence/authorship or technical development—not external predictive validity, superiority, or generalization.