Mathematical Techniques for Parameter Estimation in Bayesian Inference

Mohammed Shakir Zghair

doi:10.47134/ppm.v3i1.2080

Authors

Mohammed Shakir Zghair Ardabil University

DOI:

https://doi.org/10.47134/ppm.v3i1.2080

Keywords:

Bayesian Inference, Parameter Estimation, Markov Chain Monte Carlo (MCMC), Variational Inference

Abstract

Combining observed data with previous knowledge, Bayesian inference is a strong statistical technique for parameter estimation. Parameters are seen as random variables; previous opinions are updated using Bayes' theorem to generate the posterior distribution. By means of this approach, model parameters can be uncertain and change with additional data. Still, calculating the posterior analytically is sometimes impossible, mostly in complex models with high dimensional data. Markov Chain Monte Carlo (MCMC) methods, such as Metropolis Hastings and Gibbs sampling, use repeated processes produce samples from the posterior distribution thereby addressing this issue. Varitional inference offers a faster, deterministic option by approximating the posterior with a simpler distribution. The Laplace approximation uses local curvature for a Gaussian approximation. Common uses of these methods are statistics and machine learning for parameter estimation, model selection, and uncertainty analysis. The study evaluates each approach's effectiveness, showing that MCMC offers the best accuracy but variational inference and Laplace approximations offer quicker but less precise substitutes. The results emphasize the importance of choosing the appropriate method depending on the complexity of the data and the computational efficiency

References

Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.

Flegal, J. M., & Haran, M. (2015). A Review of Markov Chain Monte Carlo Methods. International Statistical Review.

Furnstahl, R. J. (2020). Efficient emulators for scattering using eigenvector continuation. Physics Letters Section B Nuclear Elementary Particle and High Energy Physics, 809. https://doi.org/10.1016/j.physletb.2020.135719 DOI: https://doi.org/10.1016/j.physletb.2020.135719

Gabbard, H. (2022). Bayesian parameter estimation using conditional variational autoencoders for gravitational-wave astronomy. Nature Physics, 18(1), 112–117. https://doi.org/10.1038/s41567-021-01425-7 DOI: https://doi.org/10.1038/s41567-021-01425-7

Gelman, A., Carlin, J., Stern, H., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis. CRC Press. DOI: https://doi.org/10.1201/b16018

Gelman, A., et al. (2020). Bayesian Data Analysis. CRC Press.

Hashemi, M. (2020). The Bayesian Virtual Epileptic Patient: A probabilistic framework designed to infer the spatial map of epileptogenicity in a personalized large-scale brain model of epilepsy spread. Neuroimage, 217. https://doi.org/10.1016/j.neuroimage.2020.116839 DOI: https://doi.org/10.1016/j.neuroimage.2020.116839

Hirose, O. (2021). A Bayesian Formulation of Coherent Point Drift. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7), 2269–2286. https://doi.org/10.1109/TPAMI.2020.2971687 DOI: https://doi.org/10.1109/TPAMI.2020.2971687

Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. ICLR.

Lee, A., & Heskes, T. (2003). Variational Inference: An Overview. Journal of Machine Learning.

Li, D. (2021). Bayesian LSTM With Stochastic Variational Inference for Estimating Model Uncertainty in Process-Based Hydrological Models. Water Resources Research, 57(9). https://doi.org/10.1029/2021WR029772 DOI: https://doi.org/10.1029/2021WR029772

Liu, Y. (2019). Deriving reservoir operation rule based on Bayesian deep learning method considering multiple uncertainties. Journal of Hydrology, 579. https://doi.org/10.1016/j.jhydrol.2019.124207 DOI: https://doi.org/10.1016/j.jhydrol.2019.124207

Liu, Y. (2020). Probabilistic spatiotemporal wind speed forecasting based on a variational Bayesian deep learning model. Applied Energy, 260. https://doi.org/10.1016/j.apenergy.2019.114259 DOI: https://doi.org/10.1016/j.apenergy.2019.114259

Liu, Y. (2020). Uplink-Aided High Mobility Downlink Channel Estimation over Massive MIMO-OTFS System. IEEE Journal on Selected Areas in Communications, 38(9), 1994–2009. https://doi.org/10.1109/JSAC.2020.3000884 DOI: https://doi.org/10.1109/JSAC.2020.3000884

MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press.

Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.

Neal, R. M. (1993). Probabilistic Inference Using Markov Chain Monte Carlo Methods. Technical Report 2, University of Toronto.

Nguyen, C. (2020). Uncertainty in model-agnostic meta-learning using variational inference. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 3079–3089. https://doi.org/10.1109/WACV45572.2020.9093536 DOI: https://doi.org/10.1109/WACV45572.2020.9093536

Ni, P. (2021). Probabilistic model updating via variational Bayesian inference and adaptive Gaussian process modeling. Computer Methods in Applied Mechanics and Engineering, 383. https://doi.org/10.1016/j.cma.2021.113915 DOI: https://doi.org/10.1016/j.cma.2021.113915

O'Hagan, A., & Forster, J. (2004). Bayesian Inference. Wiley.

Robert, C. P., & Casella, G. (2004). Monte Carlo Statistical Methods. Springer. DOI: https://doi.org/10.1007/978-1-4757-4145-2

Yue, Z. (2019). Variational denoising network: Toward blind noise modeling and removal. Advances in Neural Information Processing Systems, 32.

Zhang, T. (2023). Bayesian Inference for State-Space Models With Student-t Mixture Distributions. IEEE Transactions on Cybernetics, 53(7), 4435–4445. https://doi.org/10.1109/TCYB.2022.3183104 DOI: https://doi.org/10.1109/TCYB.2022.3183104

Zhao, S. (2023). Tuning-Free Bayesian Estimation Algorithms for Faulty Sensor Signals in State-Space. IEEE Transactions on Industrial Electronics, 70(1), 921–929. https://doi.org/10.1109/TIE.2022.3153814 DOI: https://doi.org/10.1109/TIE.2022.3153814

Zhu, W. (2023). Uncertainty quantification of proton-exchange-membrane fuel cells degradation prediction based on Bayesian-Gated Recurrent Unit. Etransportation, 16. https://doi.org/10.1016/j.etran.2023.100230 DOI: https://doi.org/10.1016/j.etran.2023.100230