Experiments Log

Running record of experiments on the living project, trigger, method, result, interpretation, confidence. Newest first.

2026-08-28. Experiment #2: which genera generalize? Direction, not strength, fails to transfer#

Trigger. The South Africa reversal showed Atopobium flips its diabetes direction (control-associated in the training populations, case-associated in SA). That convicts a systematic question across ALL six populations: which oral genera keep a consistent T2D direction (trustworthy markers) and which flip? A model can only transfer on the stable ones.

Method. Six populations placed in the shared 139-genus space: NHANES (US) and China from the training pool (split by study_labels), Portugal from the geographic holdout, and India/Mumbai, Mexico, South Africa as external cohorts, each per-sample rank-transformed exactly as at inference (leakage-free). Per genus per cohort: point-biserial correlation between the sample's rank for that genus and the T2D label (sign = direction, |r| = strength). Stability is defined by DIRECTION consistency, not magnitude: NHANES (n=9683) has a diffuse signal (median |r| = 0.01), so a fixed magnitude gate would wrongly mark the largest cohort uninformative. A cohort casts a sign vote once |r| >= 0.05; a genus is "robust" when its sign agrees across >= 5 of 6 cohorts, a "flipper" when it clears |r| >= 0.15 in both directions. scripts/experiment_marker_stability.py.

Result.

• 71% of genera informative (|r| >= 0.15) in >= 2 populations reverse sign across them (57 of 80). This figure excludes NHANES, so it does not depend on the magnitude artifact.
• Only 15 genera hold a single direction across >= 5 populations, and all are weak (|mean r| 0.03 to 0.12). Strongest stable case-associated: Streptococcus (+0.12), Peptoniphilus, Enterococcus, Bifidobacterium. Stable control-associated: Fretibacterium, Johnsonella, Treponema, Cloacibacterium.
• 56 flippers, Atopobium among them (NHANES +0.02, China -0.04, Portugal -0.07, India +0.15, Mexico -0.19, SA +0.47). Others: Faecalibacterium, Campylobacter, Bacteroides, Actinomyces.
• Effect: marker direction predicts transfer. Of the pooled model's 20 strongest training markers, the fraction that flip sign tracks how well the model transfers: Portugal 10% (transfer AUC 0.698), India 40% (0.621), Mexico 50% (0.564), South Africa 40% with the highest-importance marker among them (reversed to 0.320).

Interpretation. The cross-population transfer gap is a direction problem, not a strength problem. The diabetes signal is real and often strong inside each population, but the SIGN of almost every genus-to-diabetes association is population-specific. South Africa is not an outlier so much as the extreme of a pervasive instability. This is why a pooled model degrades smoothly with population distance and then breaks entirely on SA: it is averaging directions that do not agree. The few direction-stable genera are too weak to carry a portable model on their own.

Confidence. The 71% reversal prevalence and the marker-flip-vs-transfer effect are robust to the NHANES magnitude artifact (the effect uses only sign, the prevalence excludes NHANES). GRADED DOWN: these are associations in modest per-cohort samples (n = 50 to 88 for four of six cohorts), not causal or diagnostic claims, and several flippers are low-abundance or environmental/reagent-associated taxa (Serratia, Burkholderia, Delftia, Paenibacillus) whose signs may reflect contamination rather than biology. The "robust" panel is direction-consistent but weak, not a validated biomarker set.

Implication. Points the modeling work at portability rather than accuracy: a model built on direction-stable genera, or one that adapts sign per population, is the honest path to transfer, and any future African/SCCS cohort gives a direct test of whether the flippers (Atopobium first) flip the same way. It also reframes the flagship claim: the oral-microbiome T2D signal is real but locally oriented, so a single global screening model is the wrong target.

2026-08-27. The SA reversal is broad and led by Atopobium (the model's top marker)#

Trigger. South Africa reverses (strong within-cohort signal, backwards transfer). Which genera flip, and is it a few markers or a global inversion?

Method. For each of the 139 model genera, point-biserial correlation of its (rank-transformed) abundance with the T2D label, computed in the TRAINING populations and in South Africa; then correlate the two per-feature association vectors and inspect the model's most important features. scripts/experiment_feature_reversal.py.

Result.

• Per-feature T2D associations, training vs SA: r = -0.237 (p = 0.005), a significant broad anti-correlation.
• The model's dominant marker Atopobium (importance ~19, several times the next) flips hard: assoc_train -0.401 (lower in diabetics) to assoc_SA +0.467 (higher in diabetics).
• 7 of the top 12 model genera flip diabetes-direction in SA (Atopobium, Phocaeicola, Alloprevotella, Treponema, Neisseria, Mogibacterium, Rhodobacter).

Interpretation. The reversal is broad (significant negative association-correlation) and concentrated in the genera the model relies on most, led by Atopobium. Because Atopobium alone carries ~19% importance and its diabetes association strongly flips, the pooled model is confidently backwards on SA. It is not a single-feature artifact: most of the model's reasoning inverts. Notably, Atopobium is the preprint's headline T2D marker, so its reversal in South Africa is exactly the finding to scrutinize.

Confidence. The Atopobium flip and the significant negative correlation are clear. Whether this reflects population biology or a SA-study batch effect is still unresolved, a systematic case/control batch difference could also flip associations broadly.

Implication. Names the mechanism (Atopobium-led broad reversal) and gives a concrete, pre-registered thing to check in any future African or SCCS cohort: does Atopobium's diabetes direction flip there too? If it does, population biology gains support; if not, the SA study is idiosyncratic.

2026-08-27. No second public African oral cohort to replicate the reversal#

Trigger. South Africa shows a strong, reversed T2D signal. To decide whether that is population biology or specific to one study, we need an independent African (or African-descent) oral 16S cohort with diabetes labels.

Method. Systematic search across NCBI SRA/BioProject, ENA, GEO, Qiita, PubMed, Google Scholar, and African-microbiome/H3Africa angles (2019-2026), filtering for oral (not gut) 16S with per-subject T2D or glycemic labels.

Result. No clean match exists. Public African oral 16S with T2D labels is essentially a single cohort, the one we already have (PRJNA723337). Nearest options, each caveated:

• Uganda PRJNA1087153 (public): gingival crevicular fluid, Nanopore full-length 16S, n=45 (26 diabetes / 19 non-diabetes), diabetes diagnosis but no HbA1c. Different specimen and platform, so only a weak directional check, not a like-for-like replication.
• SCCS (African-American, US): mouth rinse, Illumina V4, n=294, real T2D labels, the methodologically closest resource, but CONTROLLED ACCESS (requires a data-use request).
• Ruled out: Nigeria (samples pooled, no deposition), Egypt PRJNA1475433 (healthy-only), Angola/Zimbabwe saliva (no diabetes labels). A "SCCS on SRA" lead was a mislabel, the accession is a rhesus-macaque project.

Interpretation. The population-vs-study question CANNOT be settled with off-the-shelf public data today. This is a genuine data-availability limitation, not a failure to search: African oral-microbiome + diabetes data is thin (most African diabetes-microbiome work is gut/stool).

Confidence. High on the availability conclusion (convergent search across repositories).

Implication. Paths forward: (a) run the Uganda cohort as an explicitly weak, cross-platform directional check; (b) apply for SCCS controlled access, the rigorous answer; (c) partner for new data (e.g. H3Africa-linked oral cohort). Until then, the SA reversal is reported as a real, verified, single-cohort finding whose population-generality is unresolved.

2026-08-27. South Africa's inversion is a REAL, reversed signal (not noise)#

Trigger. Depth was ruled out. Is SA's inversion (AUC 0.320) a genuine reversed signal, or just small-sample noise that happens to land below 0.5?

Method. For each new cohort, in the SAME 139-feature rank space, compare the TRANSFER AUC (pooled model scored on the cohort) with a WITHIN-COHORT AUC (5-fold CV of a model trained and tested inside the cohort). scripts/experiment_within_cohort.py.

Result.

Interpretation. Three distinct regimes:

• South Africa has a strong, learnable T2D signal (within-CV 0.853) that is reversed relative to the training populations, which is exactly why the pooled model lands at 0.320. Flip the predictions in SA and the same model would score ~0.85. So the inversion is real signal pointing the opposite way, not noise and not depth.
• Mexico is ~chance both ways (0.566 within, 0.564 transfer): little learnable T2D signal at all, consistent with its 100%-periodontitis composition masking the contrast.
• India transfers about as well as it is internally learnable (0.638 vs 0.621).

So "cross-population failure" is not one thing: it has at least two modes, no signal (Mexico) and reversed signal (South Africa).

Confidence. The SA reversal is clear: within-CV 0.853 is far above chance, so even allowing for small-n CV optimism there is strong, real signal that the pooled model gets backwards. What one African cohort CANNOT separate: population-biological reversal vs a study/batch-specific reversal (SA is a single study).

Implication. The oral-microbiome-to-T2D relationship can genuinely differ, even reverse, across populations. A pooled model is not merely unreliable across populations, it can be confidently wrong, which argues for population-specific calibration (or sign checks) before any deployment. Next: a second African (or African-descent) cohort to test whether the reversal reproduces (population) or is specific to this study.

2026-08-27. South Africa inversion is NOT a sequencing-depth artifact#

Trigger. SA transferred at AUC 0.320 (inverted). The leading suspect was sequencing depth: SA is deep (~218k mapped reads/sample) vs the shallower cohorts, and deep sequencing detects more rare genera → a denser per-sample rank profile that could be out-of-distribution for the training data.

Method. Rarefied each SA sample's genus-count vector to a ladder of shallower depths (50k → 1k, multinomial subsample, 3 seeds averaged), re-applied the identical per-sample rank transform, and re-scored with the pooled-cohort CatBoost. scripts/experiment_sa_rarefaction.py.

Result.

Total AUC swing across depth: 0.048, and it stays inverted (well below 0.5) at every depth, down to 1,000 reads.

Interpretation (hypothesis rejected). Sequencing depth is not the driver of the inversion. Rarefying to matched-or-shallower depth nudges AUC up by at most ~0.05 but never uncrosses 0.5. So the model's T2D-associated oral-microbiome features genuinely anti-correlate with T2D in this African cohort, a real cross-population reversal, not a technical/depth artifact.

Confidence. Solid rejection of the depth hypothesis (consistent across six depths × three seeds). What remains open is why it inverts, population biology vs a study/batch confound vs cohort composition, which one deep cohort can't separate.

Implication. Strengthens the core thesis: cross-population failure is real and can be directional, not just noisy. Next suspects to test: study/batch effect (SA is a single-study cohort) and whether a second African/saliva cohort reproduces the sign.

2026-08-23. South Africa (PRJNA723337): signal INVERSION (AUC 0.320, verified)#

Trigger. Third plaque cohort, completing the geographic sweep (Africa gap, the only public African oral 16S T2DM data).

Method. PRJNA723337, 88 subgingival-plaque samples (57 diabetic [DM+Known_DM] / 31 control [Normal]; Pre-DM excluded). Single-end, deep (~227k reads/sample). Required --fastq_qmax 50 (SA reads hit Q43, above vsearch's default 41, a real pipeline fix, now in the generic processor). Labels verified against NCBI BioSample isolate (spot-checked: DM_13→diabetic, Known_DM_23→diabetic, etc., all correct). Same 139-feature → rank → CatBoost(existing pool) transfer test.

Result (verified inversion).

Four-cohort gradient (all new cohorts this session):

Interpretation. A real, verified systematic inversion, the extreme end of cross-population failure: the model's T2DM-associated features anti-correlate with T2DM in South Africa. The gradient above roughly tracks population distance from the US+Asian training set. BUT the cause is entangled and must not be overclaimed as biological: SA differs on four axes at once, population (African), specimen (plaque), sequencing depth (227k reads → far denser genus detection → a very different rank-profile distribution than the shallow training cohorts), and clinic composition. The compressed, uniformly-high predictions (0.83–0.88) point to a severe distribution shift where the rank transform interacts badly with the depth difference.

Confidence. Inversion is verified and systematic (labels correct, flip=0.68). Attribution is NOT established, do not claim "African oral T2DM signal is inverted."

Implication. (1) Naive cross-population application isn't just unreliable, it can be actively wrong, a strong argument for mandatory population-specific calibration before any deployment. (2) New concrete suspect: sequencing-depth harmonization. Testable next step, rarefy/normalize SA to comparable depth before the rank transform and re-run; if the inversion softens, depth (not population) is a major driver.

2026-08-23. Mexico (PRJNA1208116) transfer test REFUTES the specimen hypothesis#

Trigger. India (plaque) transferred only partially and naive pooling hurt; the leading hypothesis was that specimen mismatch (plaque vs saliva) was the culprit. Mexico is saliva, a specimen-matched, balanced, new-continent test to isolate population effect from the plaque confound.

Method. PRJNA1208116, 87 saliva samples (41 T2DM / 46 control; labels recovered from the paper's Supplementary Table S1, CC-BY). Same VSEARCH → 139-feature → per-sample rank → CatBoost(existing pool) transfer test. scripts/integrate_cohort.py.

Result.

Mexico had the best feature overlap (109/139) yet transferred worst, near random.

Interpretation (hypothesis refuted). Specimen match did NOT rescue transfer: saliva Mexico (0.564) transferred worse than plaque India (0.621). So specimen type is not the dominant barrier. BUT the result is confounded: Mexico is a 100%-periodontitis cohort (Severe 48 / Moderate 31 / Mild 21), so the T2DM contrast is within periodontitis, where the diabetes signal is subtle relative to the dominant periodontitis-driven microbiome shift, and this differs from the general-population (NHANES-heavy) training set. Two entangled explanations remain: (a) population distance (Latin American), (b) cohort composition (all-periodontitis clinic case-control vs general-population survey). One cohort cannot separate them.

Confidence. Result solid (near-random, n=87, balanced). Interpretation graded: specimen-hypothesis refutation is well-supported; population-vs-periodontitis remains open.

Implication (reframes the roadmap). The clinic-recruited, periodontitis-focused cohorts (India, Mexico) test a different, harder, confounded task than general-population T2DM screening. To measure pure population transfer, prioritize general-population survey cohorts (like NHANES itself, or the Qatar Biobank) over periodontitis clinic case-controls. This is now the sharper dataset-search criterion.

2026-08-23. Does pooling India into training help the existing holdouts? (NO)#

Trigger. After the India transfer test, ask the complementary question: does adding India to the training pool improve generalization to the current holdouts?

Method. Baseline CatBoost (existing pooled cohorts) vs. the same with India's 60 plaque samples appended to training (rank-transformed into the 139-feature space). Re-scored the Portugal and 3-cohort holdouts. scripts/experiment_india_in_pool.py.

Result (negative).

Interpretation. Naive pooling of the India plaque cohort degrades transfer to the (saliva) holdouts by 0.05–0.07 AUC. The specimen mismatch (plaque vs saliva) plus partial feature overlap acts as a domain shift that adds noise rather than useful diversity. Consistent with the project's recurring finding that naive cross-study pooling hurts generalization (prior domain-adaptation attempts also degraded holdout).

Confidence. Clear directional result, both holdouts drop materially and in the same direction. Caveat: single cohort, n=60, specimen-confounded.

Implication. Don't naively pool across specimen types. India (plaque) needs site-matching or explicit domain handling; specimen type is a confound to control before added geography can help. Motivates seeking a saliva cohort for new geography.

2026-08-23. India/Mumbai (PRJNA1240053) cross-population transfer test#

Trigger. Living-project goal "close the cross-population transfer gap": test whether the existing pooled model generalizes to a never-seen population.

Method. Downloaded the India/Mumbai plaque 16S cohort (60 samples, labels from SRA aliases: 40 diabetic [T2DM + T2DM+perio] / 20 non-diabetic [healthy + perio]). Processed through the same VSEARCH pipeline (merge → maxee 1.0/minlen 200 filter → 97% closed-ref vs HOMD v16.03 → genus). Mapped to the 139 model genera, applied the identical leakage-free per-sample rank transform, and scored with CatBoost (depth 4, iters 200, lr 0.1, balanced, seed 42) trained on the existing pooled cohorts. India was never seen in training, a clean LOSO-style holdout. Scripts: scripts/process_india_vsearch.py, scripts/integrate_india.py.

Result.

Only 76/139 model genera were observed in India (plaque community + region differences); the 63 absent genera tie at the bottom rank and add little signal.

Interpretation. Partial transfer, above the near-random cross-continental failure floor (0.541) and above chance, but below the European holdouts. The model retains some signal on a new population, which is notable given three stacked disadvantages: (1) never-seen population, (2) plaque, not saliva (specimen mismatch), (3) only 55% feature overlap. The plaque mismatch likely understates population transfer, so 0.621 is a conservative floor for "does the signal reach India."

Confidence. Suggestive, not definitive, n=60, single cohort, specimen mismatch. Not strong enough to claim India generalization; strong enough to justify adding saliva Indian data and testing whether site-matching lifts it.

Next. (a) Get a saliva Indian cohort to separate site effect from population effect. (b) Add India to the training pool and re-check whether it helps other holdouts (diversity gain). (c) Pending Mexico/South Africa labels for more geography.