◇ Open Values Standardfor builders

Spec / open-values-challenge-index

Open Values Challenge Index v0.1

A compact version of the challenge set for sites to read: each worked example with its top picks recomputed from the current category data and the chosen sourced facts filled in. pipeline/build_challenge.js builds it from content/challenge/index.json into app/data/challenge-index.json; change the challenge set, not this file.

Draft. It may still change before it is called stable. A file a build step makes from other files. Change the sources, not this file.

Who writes it and who reads it

Written by pipeline/build_challenge.js. Read by future challenge surface, Kosplora configuration, Socrates Colosseum format seed.

Fields

FieldWhat it meansType
format requiredAlways "open-values-challenge-index". Tells a reader which Open Values format this is.always "open-values-challenge-index"
version requiredThe version of the index format this file follows. Must start with 0.1.0.string
standard requiredAlways "open-values-standard", the standard this format belongs to.always "open-values-standard"
built requiredThe date of the challenge set this index was built from, copied from that file's updated field, as YYYY-MM-DD.string
purpose requiredA sentence saying what this index is for.see nonEmptyString
generatedFrom requiredThe challenge set file this index was built from, its date, and its hash, so anyone can check the index against it.see generatedFrom
counts requiredHow many sites, methods, cases, contrary tags and categories the challenge set holds.see counts
principles requiredCopied unchanged from the challenge set: the rules challenge features follow.list
algorithms requiredThe methods from the challenge set, with only their id, label and summary.list
instances requiredThe sites and apps from the challenge set, copied with their settings.list
categories requiredEvery category that has at least one case, sorted by id, with its criteria.list
cases requiredEvery case from the challenge set, with each view reranked against the current category data and the chosen sourced facts filled in.list

A reader MUST ignore fields it does not know, so a file can carry more than this list and still be read. A writer MUST NOT change what a listed field means.

A file that passes

Challenge indexes expose generated instance defaults, category summaries, replayed top picks, and selected sourced contrary tags.

{
  "format": "open-values-challenge-index",
  "version": "0.1.0",
  "standard": "open-values-standard",
  "built": "2026-07-07",
  "purpose": "Generated compact challenge index.",
  "generatedFrom": {
    "path": "content/challenge/index.json",
    "updated": "2026-07-07",
    "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
  },
  "counts": {
    "instances": 3,
    "algorithms": 2,
    "cases": 1,
    "contraryTags": 2,
    "categories": 1
  },
  "principles": [
    "calm challenge",
    "sourced facts"
  ],
  "algorithms": [
    {
      "id": "opposite-view",
      "label": "Opposite view",
      "summary": "Rank under opposite weights."
    },
    {
      "id": "steelman",
      "label": "Steelman",
      "summary": "Surface a sourced contrary fact."
    }
  ],
  "instances": [
    {
      "id": "conscious-consuming",
      "label": "Conscious Consuming",
      "role": "reference",
      "challengeDefault": "available",
      "surfaces": [
        "verdict"
      ]
    },
    {
      "id": "kosplora",
      "label": "Kosplora",
      "role": "learning",
      "challengeDefault": "prominent",
      "surfaces": [
        "resource"
      ]
    },
    {
      "id": "socrates-colosseum",
      "label": "Socrates Colosseum",
      "role": "dissent",
      "challengeDefault": "primary",
      "surfaces": [
        "position"
      ]
    }
  ],
  "categories": [
    {
      "id": "ai-assistants",
      "label": "AI assistants",
      "type": "Services",
      "file": "app/data/ai-assistants.json",
      "n": 2,
      "criteria": [
        {
          "key": "openness",
          "label": "Openness"
        },
        {
          "key": "safety",
          "label": "Safety"
        }
      ]
    }
  ],
  "cases": [
    {
      "id": "sample-challenge",
      "category": "ai-assistants",
      "label": "Sample challenge",
      "reads": "The strongest case the other way: ChatGPT gives managed safety receipts.",
      "userView": {
        "label": "Open first",
        "weights": {
          "openness": 5,
          "safety": 1
        },
        "expectedTop": "local-open",
        "top": {
          "code": "local-open",
          "name": "Local Open",
          "score": 82,
          "coverage": 1,
          "why": [
            "Openness"
          ]
        }
      },
      "oppositeView": {
        "label": "Safety first",
        "weights": {
          "openness": 1,
          "safety": 5
        },
        "expectedTop": "chatgpt",
        "top": {
          "code": "chatgpt",
          "name": "ChatGPT",
          "score": 80,
          "coverage": 1,
          "why": [
            "Safety"
          ]
        },
        "sourceTag": "ai-assistants/chatgpt/safety",
        "selectedTag": {
          "tag": "ai-assistants/chatgpt/safety",
          "axis": "safety",
          "score": 82,
          "summary": "Managed services publish safety evaluations.",
          "source": {
            "note": "System cards document safety evaluations.",
            "source": "https://example.com/system-card",
            "asof": "2026"
          }
        },
        "receipt": {
          "tag": "ai-assistants/chatgpt/safety",
          "axis": "safety",
          "score": 82,
          "summary": "Managed services publish safety evaluations.",
          "source": {
            "note": "System cards document safety evaluations.",
            "source": "https://example.com/system-card",
            "asof": "2026"
          }
        }
      },
      "steelman": {
        "forEntity": "local-open",
        "axis": "safety",
        "sourceTag": "ai-assistants/local-open/safety",
        "framing": "The strongest counterpoint is managed safety.",
        "line": "The strongest counterpoint is managed safety.",
        "selectedTag": {
          "tag": "ai-assistants/local-open/safety",
          "axis": "safety",
          "score": 42,
          "summary": "Local models vary by configuration.",
          "sour
…

A file that must be rejected

A generated challenge index must carry the selected sourced tag for the steelman fact.

{
  "format": "open-values-challenge-index",
  "version": "0.1.0",
  "standard": "open-values-standard",
  "built": "2026-07-07",
  "purpose": "Generated compact challenge index.",
  "generatedFrom": {
    "path": "content/challenge/index.json",
    "updated": "2026-07-07",
    "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
  },
  "counts": {
    "instances": 1,
    "algorithms": 1,
    "cases": 1,
    "contraryTags": 1,
    "categories": 1
  },
  "principles": [
    "calm challenge"
  ],
  "algorithms": [
    {
      "id": "steelman",
      "label": "Steelman",
      "summary": "Surface a sourced contrary fact."
    }
  ],
  "instances": [
    {
      "id": "conscious-consuming",
      "label": "Conscious Consuming",
      "role": "reference",
      "challengeDefault": "available",
      "surfaces": [
        "verdict"
      ]
    }
  ],
  "categories": [
    {
      "id": "ai-assistants",
      "label": "AI assistants",
      "type": "Services",
      "file": "app/data/ai-assistants.json",
      "criteria": [
        {
          "key": "safety",
          "label": "Safety"
        }
      ]
    }
  ],
  "cases": [
    {
      "id": "sample-challenge",
      "category": "ai-assistants",
      "label": "Sample challenge",
      "userView": {
        "label": "Open first",
        "weights": {
          "safety": 1
        },
        "expectedTop": "local-open",
        "top": {
          "code": "local-open",
          "name": "Local Open",
          "score": 82,
          "coverage": 1,
          "why": [
            "Safety"
          ]
        }
      },
      "oppositeView": {
        "label": "Safety first",
        "weights": {
          "safety": 5
        },
        "expectedTop": "chatgpt",
        "top": {
          "code": "chatgpt",
          "name": "ChatGPT",
          "score": 80,
          "coverage": 1,
          "why": [
            "Safety"
          ]
        },
        "sourceTag": "ai-assistants/chatgpt/safety",
        "selectedTag": {
          "tag": "ai-assistants/chatgpt/safety",
          "axis": "safety",
          "score": 82,
          "summary": "Managed services publish safety evaluations.",
          "source": {
            "note": "System cards document safety evaluations.",
            "source": "https://example.com/system-card",
            "asof": "2026"
          }
        }
      },
      "steelman": {
        "forEntity": "local-open",
        "axis": "safety",
        "sourceTag": "ai-assistants/local-open/safety",
        "framing": "The strongest counterpoint is managed safety."
      }
    }
  ]
}

The schema

Its address is https://openvaluesstandard.org/schema/challenge-index-v0.1.schema.json. Validators can fetch it there; the file is also in the repository at app/data/standard/schemas/challenge-index.schema.json.

The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are used as described in RFC 2119 and RFC 8174 when, and only when, they appear in capitals.

Whether this format fits your app is your call. The spec says what a file MUST hold. Which formats you read and write, and what your app does with them, is yours to decide.