Science
Why so many published signatures fail to replicate, how we avoid it, and answers to the questions we hear most often.
Limits of current approaches
Molecular signatures have a well-known reproducibility problem. Six causes are well documented in the literature; our platform is built to address each one.
Reproducibility
A team tried to reproduce 18 gene expression analyses published in Nature Genetics: only 2 could be reproduced in principle, 10 not at all.
Our answer: each panel is locked, versioned, then tested as is on independent cohorts.
Read · Ioannidis et al., 2009Chance
In breast cancer, most signatures made of randomly chosen genes are significantly associated with outcome. An association does not prove that a signature measures anything specific.
Our answer: every signal is compared with random panels and with a generic inflammation score.
Read · Venet, 2011Evaluation
When the same data are used to choose a model and to measure its performance, the error is underestimated, sometimes badly.
Our answer: nested cross-validation, then external replication without retraining.
Read · Varma & Simon, 2006Batch effects
Processing date, reagent batch or measurement platform create spurious signals that can pass for biology.
Our answer: platform-by-platform harmonisation and replication in other laboratories, on other technologies.
Read · Leek et al., 2010Cell composition
A shift in blood cell proportions can mimic a disease signal in a bulk measurement.
Our answer: systematic control of cell composition and of usual inflammation markers.
Read · Shen-Orr & Gaujoux, 2013Single cohort
Signatures derived from a single cohort generalise poorly; analysing several heterogeneous cohorts markedly improves reproducibility.
Our answer: discovery and mapping across hundreds of harmonised cohorts.
Read · Sweeney et al., 2017Our method
For R&D teams who want to know exactly what we do. Details of each signature are shared under a confidentiality agreement.
Each cohort imported and annotated, probes reconciled at gene level, normalised by platform, duplicate patients removed.
A fast probe checks that a signal exists before any heavy computation. Leads without signal stop early.
Sparse, regularised AI models, gene selection in nested cross-validation, stability measured fold by fold.
AUC with bootstrap confidence interval, permutation p-value, score calibration.
Locked panel, frozen normalisation parameters, applied without retraining to independent cohorts.
Specificity maps against neighbouring diseases, documented negative boundaries, comparison with existing tests.
Questions
Can't find yours? Write to us.
A locked panel of genes, proteins or microbial markers, measured at once and combined into one score. Where a single biomarker captures one mechanism, a signature reads a fuller biological state.
The area under the ROC curve measures how well a score separates two groups, for example responders and non-responders. 0.5 is chance, 1 is perfect separation. We always report it with its confidence interval.
AI selects the most informative combinations of genes among tens of thousands of measurements. It proposes; only replication on independent cohorts decides what is kept.
It must keep its performance on patients, laboratories and sometimes technologies it has never seen, without retraining. We also check that it cannot be reduced to age, sex, cell composition or generic inflammation.
Not always. Many projects start from biobanked samples or existing data. We tell you at scoping whether new measurements are needed.
Standard laboratory technologies such as RNA-seq or NanoString, with no new instrument. The choice depends on your setting and the number of markers.
No. They are research tools. Moving to a diagnostic device follows its own regulatory pathway, run with an industrial partner.
We tell you. A well-established negative result avoids investing in the wrong lead; it is a result too.
Exchanges take place under a confidentiality agreement, data are analysed in coded form and are not reused outside your project.
Yes. The same method applies to animal health, from herd microbiome to veterinary treatments, and to cosmetics, for the mode of action of actives.
Reading
