Sampling algorithms are commonplace in statistics and machine learning – in particular, in Bayesian computation – and have been used for decades to enable inference, prediction and model comparison in many different settings.  They are also widely used in statistical physics, where many popular sampling algorithms first originated [1, 2].  At a high level, the goals within each discipline are the same – to sample from and approximate statistical expectations with respect to some probability distribution – but the motivations, nomenclature and methods of explanation differ significantly.  This has led to challenges in communicating between the fields, and indeed the fundamental goals of one field are often misunderstood in the other.  In this talk, we elucidate statistical physics for the statistician, with a particular emphasis on phase transitions in order to demonstrate that physicists tend to study probability models as functions of thermodynamic hyperparameters such as the temperature.

We then move on to sampling algorithms in general, with a particular focus on the Metropolis [1] and molecular-dynamics [2] algorithms.  Indeed, certain key aspects of statistical physics have led to innovations in sampling algorithms that inform the Bayesian world.  In particular, one could argue that the Swendsen-Wang [3], Wolff [4] and event-chain Monte Carlo [5, 6] algorithms were all developed in response to the onset of nonergodicity (with respect to both physical and Metropolis dynamics) at certain phase transitions.  The final part of this talk therefore focusses on ergodicity breaking with respect to the Metropolis algorithm, and how these alternative ergodic sampling algorithms were developed in response to this phenomenon.  The aim here is to demonstrate that this key aspect of statistical physics has informed a fundamental goal of Bayesian computation — the development of efficient, multi-purpose and ergodic sampling algorithms.

 

[1] Metropolis, Rosenbluth, Rosenbluth, Teller and Teller, J. Chem. Phys. 21 1087 (1953)

[2] Alder and Wainwright, J. Chem. Phys. 27 1208 (1957)

[3] Swendsen and Wang, Phys. Rev. Lett. 58 86  (1987)

[4] Wolff, Phys. Rev. Lett. 62 361 (1989)

[5] Bernard, Krauth and Wilson, Phys. Rev. E 80 056704 (2009)

[6] Michel, Mayer and Krauth, EPL (Europhys. Lett.) 112 20003 (2015)