Write a reproducibility appendix for a software white paper
Add a reproducibility appendix when a software white paper relies on an executed example, integration test or evaluation. Record the versions, inputs, procedure and observed result needed to understand the claim.
Keep those references beside the evidence, even if the main text presents only a short account of the experiment.
Make one result traceable from claim to output
A software white paper may show a successful extraction, an integration response or an evaluation chart. To assess the result, another reader needs to know what was run, with which input and under which conditions. A repository link without a revision can lead to code that no longer produces the published example.
Start with one material claim and work backwards. Identify the output that supports it, the command or procedure that produced the output, the input and the software reference. Record conditions outside the code, such as an enabled option or an unavailable dependency.
| Part of the appendix | Question a reviewer should be able to answer |
|---|---|
| Claim reference | Which sentence or figure does this support? |
| Version | Which code, model or specification was used? |
| Input | What data and configuration entered the run? |
| Procedure | What steps produced the observation? |
| Output | What was actually obtained? |
| Limit | Which conclusions are outside the experiment? |
Explain what can be reproduced and what can only be inspected
A public sample may be reproducible when the necessary code and inputs are available. A result based on restricted customer data may only be inspectable through an authorised review. State which situation applies.
If you provide a synthetic substitute, explain what it demonstrates and what it omits. It may show the interface or transformation without reproducing the original distribution of customer cases. Do not label it a reproduced customer result merely because the output format looks similar.
Likewise, a mocked external service can test client behaviour under a controlled response. It does not establish interoperability with the live service. Keep the boundary next to the procedure, where a reviewer will see it before interpreting the output.
Preserve version references that can move
The UCP versioning policy provides a useful example: it describes maintained release branches and a backport process. A document discussing an integration should preserve the exact reference used, rather than assuming a release label or a latest-documentation link will always point to identical artefacts.
Record the revision or digest where available, together with the observation date. The digest helps establish which local artefact was used; it does not establish that the artefact was correct or that the test was well designed. Those questions still require review of the procedure and result.
When updating the white paper, identify which evidence has been rerun and which is retained from the previous version. A new cover date should not make an old experiment appear current. If the new release changes a dependency relevant to the result, either repeat the affected work or preserve the earlier scope explicitly.
Label examples according to their origin
A synthetic example can be useful for explaining an interface. A controlled test can establish behaviour under recorded conditions. A customer result requires authentic supporting evidence and appropriate permission to disclose it.
Keep these categories explicit. Do not turn an illustrative example into a case study by adding an invented company name, testimonial or performance number.
The NIST's AI measurement and evaluation work is relevant when the paper concerns AI: the evaluated property and context need to be defined before the result can be interpreted.
Include failures that define the boundary
If an example only works with a particular option or excludes certain cases, say so. A limitation is part of the result, not an embarrassing detail to hide in a private notebook.
Retain unsuccessful cases when they explain the scope. Avoid publishing secrets, personal data or confidential customer material in the name of reproducibility. Describe the restricted part and provide an authorised substitute where that still supports the claim.
Update references without disguising old results
Assign responsibility for checking the references when the paper is updated. If a result has not been rerun, do not imply that changing the publication date refreshed the experiment.
The white-paper examples page, available in French, helps choose the broader format. Place the appendix where readers can move directly from a technical claim to its execution record.
Before release, ask a qualified reviewer to reconstruct one result from the appendix. If a necessary input, version or instruction is missing, add it or narrow the claim to what the evidence actually supports.
The Verity Score UCP reference provides a merchant-oriented introduction. It can help a non-specialist reader with terminology; the normative reference remains necessary for an implementation claim.
