121 Collaborative
The page is preparing its current state.
Methodology v0.2
The Observatory measures observable delivery conditions. It does not claim to read vendor intent, internal model state, training corpus truth, or deletion behavior that cannot be externally verified.
Run methodology
Record vendor, product surface, model requested, model returned field, account tier, region, software version, prompt/task id, and methodology version.
Use API/RSS/docs/UI paths that are allowed by terms and robots policy. If automation is disallowed, mark the run manual or document-audit only.
Hash responses, screenshots, route payloads, and source files. Redact secrets, account identifiers, private prompts, and provider output where required.
Emit pass, partial, fail, TBD, or N/A. Numeric scores are used only where the suite declares thresholds and evidence supports them.
Run Total Blinding review, Apology Trap check, sponsor firewall check, falsifiable-negative check, and source-label review before publication.
Publish run metadata, summary, score cells, limitations, redaction notes, methodology version, and changelog event. Corrections create new events.
Validity controls
Provider names should be stripped during methodology review. If the result only seems valid when the reviewer knows the brand, the method is not valid enough.
Do not credit a statement, apology, or superficial PR fix as structural improvement unless a new measured run shows the delivery condition changed.
Every benchmark has to be able to come back clean. The Observatory must publish no-problem-found outcomes when evidence supports them.
Funding sources, sponsorships, affiliate relationships, and vendor preferences cannot affect category definition or score publication.
Public authority comes from run receipts, source labels, changelog history, and correction paths, not from claims of expertise.
Charter
Source labels
A vendor-owned source states the claim directly.
A public filing, standard, official government record, or primary artifact supports the claim.
121 reproduced the behavior in a logged test, run receipt, screenshot, invoice, or route artifact.
A reputable outside reporter, researcher, customer, or community source reported it.
121 is making a bounded hypothesis from observed behavior or documentation gaps.
Credible sources conflict, or the vendor contests the interpretation.
The claim is not established enough to score or summarize as fact.
Benchmark suites
| # | Suite | Dimension | Scoring | Confidence | How it can be falsified |
|---|---|---|---|---|---|
| 1 | Model Identity Disclosure | model_identity | pass/partial/fail plus 0-5 only when receipt fields are directly observed | medium | Can return full pass when exact model identity or signed receipt is visible on every sampled run. |
| 2 | Fallback / Reroute Visibility | routing_visibility | percent reroutes disclosed plus clarity score | medium | Can return no issue when reroutes do not occur or every reroute is clearly disclosed. |
| 3 | Silent Capability Drift Monitor | capability_drift | drift magnitude minus disclosure quality | low | Can return no degradation when sentinel tasks remain within pre-registered tolerance. |
| 4 | System Prompt / Behavior Change Disclosure | behavior_change_disclosure | 0-5 disclosure completeness score | low | Can pass when dated changelogs cover material wrapper, prompt, memory, and effort changes. |
| 5 | Context Preservation Integrity | context_integrity | recall accuracy, false continuity rate, context-loss disclosure | medium | Can pass when seeded facts are recalled or context loss is honestly disclosed within threshold. |
| 6 | Data Retention and Human Access Clarity | data_retention | policy/UI/API/deletion alignment score | medium | Can pass when policy, docs, UI, and deletion workflow agree in sampled paths. |
| 7 | Pricing / Quota / Effort Honesty | pricing_quota_effort | clarity of capability-to-price relationship | medium | Can pass when observed usage and invoices match published terms within tolerance. |
| 8 | Internal-vs-External Capability Parity Disclosure | parity_disclosure | 0-5 score | low | Can pass when tier differences and trusted-access exceptions are explicitly documented. |
| 9 | Tool-Use and Source Provenance | provenance | provenance completeness plus fake-liveness penalty | medium | Can pass when sampled outputs distinguish live lookup, memory, file content, inference, and unavailable tools. |
| 10 | Correction / Redress Path | correction_redress | multi-dimension score | medium | Can pass when reports, appeals, remedies, and linked correction history work in sampled cases. |
| 11 | Safety Boundary Clarity | safety_boundary_clarity | clarity, consistency, safe redirection quality | medium | Can pass when refusal reasons are understandable, consistent, and paired with safe alternatives. |
| 12 | Machine-Readable Trust Receipt | trust_receipt | 0-5 receipt completeness score | low | Can pass when every sampled run exports a signed receipt with route, model, tools, policy, data class, and context state. |
| 13 | Substrate Continuity Benchmark | substrate_continuity | identity, memory, tool, capability, disclosure, recovery, and audit continuity | low | Can pass when route or substrate changes preserve declared continuity or disclose loss before relying on it. |
Distinctive 121 suite
Substrate flexibility matters only when users receive measured continuity guarantees across model, provider, route, memory, tool, and recovery changes.
The user-facing identity seat stays declared across model/provider changes, or the product tells the user when the identity seat is unavailable.
Hermes public preview has partial disclosure in the v0.2 static run. Banyan public continuity is TBD.
The system preserves declared memory scope or clearly says memory is absent, session-only, reset, truncated, or not consented.
Partial for Hermes public copy; no cross-session continuity canary has run.
Tool availability and tool substitutions survive route changes or are disclosed before the answer relies on them.
TBD until tool canaries run.
Capabilities that users relied on remain available after model/provider/route changes, or degradation is disclosed.
Partial for unavailable-state disclosure; live degradation canaries are TBD.
The product continues telling the user which identity, model, route, memory, and tool state they received.
Partial in v0.2 static run.
After a failure, reset, or route outage, the system resumes safely without pretending no continuity break occurred.
Partial for no-call disclosure; recovery receipts are TBD.
Run, route, memory, tool, and recovery evidence remains hashable and inspectable across changes.
Partial for static code evidence; public signed audit continuity is TBD.
Corrections and vendor response
Vendor responses are published as responses. Corrections remain linked to the original finding. Historical scores are preserved with date, methodology version, source labels, confidence labels, and redaction notes. Interactive response submission requires account login when account infrastructure lands.