Oral Microbiome Models of Diabetes Status: A Retrospective Audit of Cross-Study Transfer

Jonathan Scarpelli and Ayesha Akhter

Working manuscript, revised 2026-10-18. Not peer reviewed.

Abstract

We retrospectively audited oral-microbiome diabetes-status prediction in six source studies. Removing 129 fecal specimens mislabeled oral leaves 10,228 source specimens (1,251 cases). NHANES labels encode self-reported diabetes, not a verified subtype; clinical cohorts use deposited diabetes labels. Related India and Shanghai evaluations exclude their entire corresponding source study. Fixed CatBoost yields pooled-source OOF AUC 0.793, Portugal AUC 0.646, and South Africa AUC 0.449 (conditional 95% interval 0.323–0.578). The South African interval includes chance, withdrawing the earlier confident-reversal claim. Source-only nested panel selection yields macro-study AUC 0.661; 30 genera do not improve Portugal. A consistent within-study permutation test gives observed mean fold AUC 0.794 versus null 0.660, p=0.0099. Disjoint-row isotonic calibration reduces source ECE from 0.194 to 0.006, but external miscalibration remains. Subgroup estimates replace training-row scores with OOF evaluation. Association directions vary within and between countries without identifying a biological or technical cause. All external cohorts informed development; these are exploratory, not confirmatory estimates. No clinical screening tool, biomarker panel, validated failure gate, or harm-reduction benefit is established.

1. Motivation and prior work

The International Diabetes Federation estimates 589 million adults aged 20–79 lived with diabetes in 2024, across all diabetes types, with approximately 252 million undiagnosed [1]. This motivates research but does not establish benefit from microbiome screening. HbA1c does not require fasting, unlike fasting plasma glucose [2]. Saliva collection is noninvasive; shipping and storage depend on the protocol and assay. Sequencing requires laboratory infrastructure and quality control. No cost advantage or clinical decision benefit is established here.

Gao et al. analyzed five salivary studies, including Portugal, and reported pooled random-forest AUC 0.92 and evaluation on an independent Russian subgingival cohort [3]. Yang et al. reported 83.3% accuracy for a four-bacterium classifier in a study incorporating periodontitis [4]. Accuracy and pooled AUC are not directly comparable to our external-study AUCs. Cross-cohort analysis is not novel in itself; this work contributes a reproducible audit of transfer, eligibility, and validation on a heterogeneous survey/clinical mixture. No first European or first cross-continental validation claim is made.

Our question is whether scores and association directions reproduce across studies. Country labels describe cohort provenance, not measured ancestry or an experimentally isolated population effect. Study, specimen, recruitment, periodontitis, treatment, and geography are entangled.

Audit findings and consequences

The table records workflow repairs, not causal treatment effects. Historical and current AUCs differ in eligibility and analysis as well as procedure; subtracting them would not isolate the benefit of a repair.

2. Data eligibility and provenance

The pipeline identifiers china_shandong_sun and china_anhui_yang map to PRJNA601054 and PRJNA561495, respectively. Their local region names do not establish recruitment geography; the table uses accessions as study identities. NHANES identifiers refer to the official oral-microbiome project and DIQ_F/G files [13,14], not a new unrestricted SRA deposit. Specimens and deposited case/control definitions vary; the PRJNA782768 oral subsite remains unverified. South African cases combine DM and Known_DM; Pre-DM rows are excluded.

The historical matrix contained 10,357 source rows. PRJNA910326 deposits tongue and fecal specimens [5]; all 347 rows had been labeled tongue coating locally. Deposited DT/HT aliases identify oral specimens and DF aliases fecal specimens. We exclude all 129 DF rows, retain 218 DT/HT rows, and preserve the original matrix as a historical artifact. The paper reports 183 oral diabetic specimens whereas deposited DT aliases yield 184; this single-row discrepancy remains unresolved. We do not guess which row to remove.

India/Mumbai (PRJNA1240053) contains the same 50 SRR identifiers used in the Indian source subset, plus ten controls. Every India evaluation fit excludes the entire 50-row Indian source study. PRJNA782768 and PRJNA910326 were deposited by the same Shanghai publication [5]; different run identifiers do not establish different participants. All 218 Shanghai source rows are excluded from fitting, SHAP ranking, and probability calibration when scoring PRJNA782768. This is related-study re-evaluation, not independent replication. PRJNA664107 is from Moscow, Russia, with gingival-sulcus specimens [6], not China. These safeguards cannot undo benchmark reuse or establish participant independence across related publications.

Public SRA-derived counts use the historical VSEARCH closed-reference workflow against HOMD v16.03 [7]. NHANES uses public-use DADA2-derived genus tables; restricted NHANES reads were not accessed. Raw processing is not uniform across all studies. Audited reruns retain the fixed historical 139-genus vocabulary for retrospective comparability, reindex missing genera to zero, and transform each row to average ranks divided by 139. This repairs eligibility, not a newly validated oral-only assay vocabulary. Future confirmation must freeze eligibility, raw processing, and vocabulary derivation independently.

NHANES retained labels correspond exactly to DIQ010: questionnaire code 1 (yes) maps to label 1 for all 830 cases, and code 2 (no) to label 0 for all 8,853 controls; no retained row has borderline, missing or refused/unknown codes [13]. DIQ010 concerns self-reported doctor-diagnosed diabetes and does not establish Type 2 subtype. No additional subtype adjudication rule was verified in the available provenance, so the source-mixture target is diabetes status. A negative self-report is not biochemical confirmation of normoglycaemia. Clinical cohorts retain their deposited labels, with no harmonized subtype/diagnostic adjudication claimed. Recruitment and case prevalence differ greatly. Harmonized adjustment for medication, age, periodontitis, and other confounders is unavailable. Public sample identities are checked, but specimen-level analyses do not prove unique participants or eliminate repeated-subject dependence. Labels are not standardized incident-disease outcomes.

3. Methods and estimands

3.1 Fixed retrospective estimators

The primary CatBoost estimator [8] uses 200 iterations, depth 4, learning rate 0.1, balanced class weights, seed 42, four CPU threads, and no training-file writes. Its fitted CPU configuration uses Plain, not Ordered, boosting; the earlier algorithmic explanation is withdrawn. Estimator and parameter choices were developed using older matrices and inspected external results. Historical comparisons do not establish unbiased superiority.

The adaptation probe is balanced logistic regression, C=1. Local fits minimize logistic loss with a quadratic penalty toward source coefficients, lambda=2. Coefficient sign describes a fitted direction in rank space, not a causal bacterial effect. These AUCs are separate from CatBoost’s leaderboard; outputs are not calibrated disease risks.

3.2 Source and benchmark evaluation

Pooled five-fold stratified OOF AUC estimates discrimination for this source mixture. Nested leave-one-source-study-out (LOSO) uses six outer studies; inner LOSO selects K=30 or 139 by macro-study AUC, retaining 139 on an exact tie. Every reduced fit ranks all 139 columns by mean absolute CatBoost SHAP [9] on at most 1,024 fitting rows. Outer labels do not enter fitting, ranking, or K selection. Model-family and parameter selection remain retrospective.

For 50 paired seeds, 500 NHANES rows are sampled without replacement and all other oral-only source rows retained; 30/139-feature fits are scored on Portugal. Seed intervals summarize algorithm/subsample variability on the same 50 specimens, not patient confidence intervals or independent replications.

External cohorts were repeatedly inspected during development. They are exploratory development benchmarks, not untouched validation. India and related Shanghai use source-study-excluded fits. The descriptive three-cohort pooled score concatenates Portugal, Russia, and related Shanghai predictions from different source fits; it is not a single frozen classifier’s transfer estimate. Cohort-specific results are primary. Within-cohort CatBoost AUC comes from a separate five-fold OOF fit and is not a clinical ceiling.

AUC intervals use 10,000 stratified case/control bootstrap draws, seed 42, conditional on the fitted score and observed specimens. They omit model/study-selection uncertainty, multiplicity, and unresolved participant clustering. Below-chance AUC indicates reversed ranking, not confidence of individual predictions or demonstrated harm reduction when inversion disappears.

3.3 Calibration, permutation, and subgroups

Five outer stratified folds separate fitting, isotonic calibration, and evaluation. Development rows split 75% fitting / 25% calibration; neither uses evaluation labels. External scores average the five fits. Related Shanghai source rows are excluded from fitting and calibration. ECE uses ten equal-width bins, including probability 1; Brier score is also reported. Calibration targets source-mixture prevalence, not destination prevalence.

Observed and permuted statistics use identical five-fold mean-AUC procedures. One hundred within-study label shuffles preserve study case counts and composition. The null assumes within-study exchangeability; residual confounding or repeated participants may violate it. This tests conditional association, not causality or transfer.

NHANES subgroup estimates join source OOF predictions to public CDC DEMO_F/G data by SEQN. Two thousand stratified bootstrap draws give conditional AUC intervals. Sensitivity/specificity use raw threshold 0.5 descriptively. No survey weights, intersectional analysis, clinical threshold, or fairness certification is claimed.

4. Corrected results

4.1 Source discrimination and cohort transfer

Pooled-source OOF AUC is 0.793. The descriptive pooled benchmark AUC is 0.594 on 176 specimens, with Shanghai-excluded fitting for its related benchmark.

Russia/Moscow AUC 0.639; related Shanghai AUC 0.546.

South Africa has a below-chance point estimate, but its interval includes 0.5. The historical confident reversal does not survive oral-only reconstruction. Within-cohort structure remains measurable without proving a transferable disease signal.

4.2 Source-only feature selection and seed sensitivity

Previous claims that 30 genera improved Portugal to 0.683 and that 50 seeds averaged 0.718 with 48 wins lacked matching generated runs and are withdrawn. Historical 49-specimen feature-selection artifacts, older model comparisons, and mixed-specimen results remain research history, not current evidence.

Nested source-study selection yields macro LOSO AUC 0.661; the all-source inner comparison selects K=30. The fully ranked 30-genus model scores 0.578 in Portugal versus 0.646 for 139 genera, and 0.624 versus 0.594 on the three-cohort benchmark. Source-study selection does not imply external improvement.

Across 50 paired seeds, NHANES=500 plus all other source studies gives a 30-genus mean Portugal AUC 0.631 (SD 0.048) versus 0.616 (SD 0.057) for all features; the panel wins 29/50. Its seed 2.5th to 97.5th percentiles are 0.547 to 0.722. These are correlated sensitivity runs on the same 50-person benchmark, not a patient confidence interval. No reduced assay or clinical panel is validated. Full rankings, nested fold choices, and paired runs are in data/processed/validation_panel.json.

4.3 Conditional permutation

The reconstructed within-study permutation test on the final matrix gives observed mean fold AUC 0.794 versus a conditional null mean 0.660 (SD 0.008); none of 100 shuffles reaches the observed statistic (corrected p=1/101=0.0099). The null is well above chance because study case counts and study-associated features are preserved. This supports within-study association beyond study composition under the permutation assumptions, but does not establish disease causality or external transfer. See data/processed/validation_permutation.json.

4.4 Disjoint-row probability calibration

On disjoint outer evaluation rows, source-mixture ECE (10 equal-width bins) changes from 0.194 to 0.006, while Brier score changes from 0.126 to 0.079. In Portugal the ensemble’s ECE changes from 0.319 to 0.169; on the three-cohort benchmark it changes from 0.317 to 0.176. Source calibration thus improves the fitted source mixture without eliminating external miscalibration. Full scores, prevalences, and fold metrics are in data/processed/validation_calibration.json.

4.5 NHANES subgroups

Earlier approximately 0.9 subgroup AUCs evaluated training rows and are withdrawn. These OOF estimates concern NHANES in a pooled-source model, not external demographic groups. Comparisons are descriptive, not tests of equivalent error rates.

The raw threshold is fixed for description, not a clinical recommendation. No survey weighting, subgroup-specific threshold selection, prospective calibration, or fairness certification is claimed.

5. Exploratory transfer investigations

These investigations reuse the benchmarks. They are retrospective modeling tests, not causal identification.

Marker directions: 68% of 88 informative genera reverse among geographic groups excluding NHANES; 14 meet the retrospective stable-sign rule and 58 meet the broader flipper rule. Different rules produce different denominators.

Portability restriction: mean four-cohort AUC changes from 0.569 to 0.556. South Africa changes from 0.449 to 0.486, remaining below chance as a point estimate. Dropping Atopobium changes it to 0.340; the previous beneficial-ablation interpretation is withdrawn.

Local adaptation: South Africa’s logistic probe changes from 0.304 with no local labels to 0.615 at 40 labels; its within-cohort logistic reference is 0.859. These point estimates do not establish clinical calibration or an optimal label requirement.

Distribution diagnostics: domain AUC spans 0.991 to 1.000. Mahalanobis distances are Portugal 9.49, India/Mumbai 10.25, Mexico 10.76, South Africa 21.59. The largest distance is not proof of a calibrated failure detector.

Marker properties: median source-only prevalence is 19.8% for stable genera versus 2.1% for flippers (Mann–Whitney p=0.1881). This underpowered association does not qualify a panel.

Diversity: mean transfer AUC is 0.400 versus 0.569 for taxa; 3/4 diversity point estimates are below chance.

Study/geography comparison: within-China sign reversals 58% of 114 informative genera versus cross-geography 67% of 87. Association-vector correlations average 0.170 and -0.026, respectively. This nested comparison does not estimate a causal variance decomposition.

Marker stability uses point-biserial correlations of ranks with deposited labels. The cross-group reversal fraction excludes NHANES and requires absolute correlation at least 0.15 in at least two of five remaining geographic groups. Stable features require at least five votes at absolute correlation 0.05 and 80% sign agreement across six groups. These retrospective thresholds are not clinical qualification. The association-ranked top-20 list is not SHAP/model importance, excludes Atopobium, and does not monotonically predict transfer.

Portability restriction chooses stable features from groups other than the target before source fitting. Other inspected benchmark labels inform selection: this differs from source-only nested panel selection. Atopobium deletion is model ablation, not microbial intervention. Movement toward chance does not prove safer decisions or useful discrimination.

Adaptation repeats five-fold local splits across 12 seeds, sampling K=0,5,10,20,40 fitting labels only from local development rows. Evaluation rows are disjoint; folds/seeds are correlated. Forty labels consume nearly half the 88-specimen South African cohort. Logistic within-cohort fits are references, not ceilings.

Label-free diagnostics use a balanced domain classifier and regularized Mahalanobis centroid distance in rank space. A retrospective outlier does not establish a refusal threshold or validated failure gate. Four cohorts cannot calibrate such a rule. Distances are model diagnostics, not population distances.

Diversity comprises Shannon, Simpson, and richness within 139 genera, not the full microbiome. Diversity/taxon comparisons differ in fitting family and representation and do not isolate coarse-graining causally. Marker-property comparisons are underpowered; nonsignificance is not equivalence. Within-China/cross-geography reversal comparisons are nested, use different informative denominators, and confound specimens, recruitment, periodontitis, treatment, processing, and sampling noise. They do not partition variance or show study effects dominate biology.

6. Interpretation and limitations

Source discrimination on heterogeneous diabetes labels is not evidence of a portable Type 2 diabetes assay. The elevated conditional permutation null shows study composition can itself support substantial AUC. Local fits and inconsistent directions illustrate instability, not a unique explanation.

The audit corrected mixed specimens, related-study evaluation, country provenance, incomplete feature ranking, inconsistent permutation statistics, training-row subgroup evaluation, and overlapping calibration/evaluation labels. These repairs improve traceability but do not create untouched test sets. Residual limits include imperfect labels, unresolved participant linkage and DT count discrepancy, historical vocabulary, heterogeneous specimens/processing, unmeasured confounding, small studies, and adaptive reuse. Raw-read reprocessing of every source and unbiased reruns of every historical model family were not performed.

Clinical use is out of scope. No validated threshold, destination-risk calibration, prospective utility, screening interval, cost, number-needed-to-screen, or qualified qPCR panel is established. Missed cases could delay follow-up; false positives could cause anxiety, unnecessary tests, cost, and unequal access burdens. Research scores must not replace established diagnosis or care.

Confirmation requires a frozen eligibility/vocabulary/estimator/calibration/threshold protocol before receiving a new cohort, participant-level separation, multiple harmonized studies per population, and local decision analysis. These are research requirements, not completed actions.

7. Ethics, availability, and contributions

This independent secondary analysis uses public-use survey tables and publicly deposited sequence-derived counts, without new participant contact, specimen collection, restricted NHANES access, or re-identification. Public availability alone is not a universal exemption. Applicable review and original consent/reuse terms must be assessed; a formal determination is required only where the relevant rules apply [10]. No new institutional review requirement has been established for the described activity, and no formal exemption or not-human-subject determination is claimed. Source publications report original ethics review and, where accessible, consent. For the Sichuan source study, the publisher’s indexed text reports ethics approval and public sequence deposition, but the original consent wording could not be verified through accessible legitimate routes. This documentary gap remains disclosed and unresolved.

The public dashboard accompanies this manuscript with our aggregate statistical summaries; a separately prepared analysis-code package remains private. It excludes sample-level matrices, labels, demographic linkages, raw reads, reference databases, fitted models, and repository history. NCHS public-use terms restrict use to statistical analysis/reporting and prohibit identification attempts or linkage to individually identifiable data [11]. NCBI places no restrictions of its own on molecular-data use or distribution but cannot grant third-party rights [12]. These policies support the described statistical-use route; they do not certify original consent scope or provide blanket derivative-data redistribution permission. Source-specific and exact-artifact review is documented in the private cohort dossier; no project-wide release clearance is claimed.

Contributions: Jonathan Scarpelli, study design, baseline/CatBoost analyses, interpretation, transfer investigations, and manuscript; Ayesha Akhter, preprocessing, exploratory analysis, feature engineering, and gradient-boosting comparison. Both contributed to methods and experiments.

The working manuscript and dashboard aggregates are available at https://oralbiome.scarpelli.ai/preprint.html (HTML), https://oralbiome.scarpelli.ai/preprint.pdf (PDF), and https://oralbiome.scarpelli.ai/results.json (aggregate results). A separate analysis-code review package remains private. Sample-level derived inputs, labels, demographic linkages, raw reads, reference databases, fitted models and repository history are excluded. Numerical reconstruction requires the exact permitted historical inputs identified by hashes; full raw-to-paper reproduction is not established. Reproduction instructions distinguish the audited entrypoints from historical scripts. No DOI or peer review is claimed.

References

[1] International Diabetes Federation. “Over 250 million people worldwide unaware they have diabetes, according to new IDF research.” 2025-04-07; release of IDF Diabetes Atlas, 11th edition (2024 estimates). https://idf.org/news-and-resources/news/idf-diabetes-atlas-11th-edition/.

[2] NIDDK. “The A1C Test & Diabetes.” https://www.niddk.nih.gov/health-information/diagnostic-tests/a1c-test.

[3] Gao, C., Guo, Y. & Chen, F. “Cross-Cohort Microbiome Analysis of Salivary Biomarkers in Patients With Type 2 Diabetes Mellitus.” Front. Cell. Infect. Microbiol. 12 (2022): 816526. https://doi.org/10.3389/fcimb.2022.816526.

[4] Yang, Y. et al. “Changes of saliva microbiota in the onset and after the treatment of diabetes in patients with periodontitis.” Aging 12.13 (2020): 13090–13114. https://doi.org/10.18632/aging.103399.

[5] Guo, X.J. et al. “Distribution characteristics of oral microbiota and its relationship with intestinal microbiota in patients with type 2 diabetes mellitus.” Front. Endocrinol. 14 (2023): 1119201. https://doi.org/10.3389/fendo.2023.1119201.

[6] “Drift of Subgingival Periodontal Microbiome during Chronic Periodontitis in Type 2 Diabetes Mellitus Patients.” Pathogens 10.5 (2021): 504. https://doi.org/10.3390/pathogens10050504.

[7] Rognes, T. et al. “VSEARCH: a versatile open source tool for metagenomics.” PeerJ 4 (2016): e2584.

[8] Prokhorenkova, L. et al. “CatBoost: unbiased boosting with categorical features.” NeurIPS 31 (2018): 6638–6648.

[9] Lundberg, S.M. & Lee, S.I. “A unified approach to interpreting model predictions.” NeurIPS 30 (2017): 4765–4774.

[10] HHS Office for Human Research Protections. “Coded Private Information or Biospecimens Used in Research, Guidance” (2018). https://www.hhs.gov/ohrp/coded-private-information-or-biospecimens-used-research.html.

[11] CDC National Center for Health Statistics. “Data User Agreement.” https://www.cdc.gov/nchs/policy/data-user-agreement.html. Accessed 2026-10-18.

[12] NCBI. “Policies and Disclaimers,” Molecular Data Usage. https://www.ncbi.nlm.nih.gov/home/about/policies/. Accessed 2026-10-18.

[13] CDC/NCHS. NHANES Diabetes Questionnaire, DIQ_F (2009–2010) and DIQ_G (2011–2012), DIQ010. https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2009/DataFiles/DIQ_F.htm; https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2011/DataFiles/DIQ_G.htm.

[14] CDC/NCHS. NHANES 2009–2012 Oral Microbiome: data documentation and public-use files. https://wwwn.cdc.gov/Nchs/Nhanes/omp/.