Enhancement of hidden Markov model analyses for improved inference of archaic introgression in modern humans

Abstract

Insights into the admixture history between modern and archaic humans require accurately inferred introgressed fragments within modern genomes. Here, we introduce two enhancements to hidden Markov models (HMMs) implemented in hmmix. First, we develop a method for sampling hidden state sequences conditional on observed genomic data, enabling robust estimation of admixture summary statistics—such as admixture proportion and fragment length distributions. This represents an improvement compared to relying solely on point estimates as provided by classical decoding methods. Additionally, we integrate the Finite Markov Chain Imbedding (FMCI) framework, allowing exact analytical calculation of these admixture statistics, tailored to large scale human genomes. Second, we implement a novel hybrid decoding method which combines the strengths of Viterbi and Posterior decoding methods, substantially improving the reliability of archaic fragments identified. We validate these improvements on data from the 1000 Genomes Project and demonstrate that our sampling method yields more accurate admixture estimates from single individuals compared to existing approaches requiring extensive population-level datasets. Moreover, we show how hybrid decoding can be instrumental in resolving the inference of local archaic haplotype structure in modern human genomes. These methodological advancements will enhance HMM-based analyses in any field of science and will provide deeper insight into the complex history of genetic interactions between archaic and modern human populations. Keywords: hidden Markov models, inhomogeneous Markov chain, sampling from the posterior, Finite Markov Chain Imbeding, hybrid decoding, hmmix, archaic introgression