Bayesian Mechanics for Stationary Processes

This paper develops a Bayesian mechanics for adaptive systems. Firstly, we model the interface between a system and its environment with a Markov blanket. This affords conditions under which states internal to the blanket encode information about external states. Second, we introduce dynamics and represent adaptive systems as Markov blankets at steady-state. This allows us to identify a wide class of systems whose internal states appear to infer external states, consistent with variational inference in Bayesian statistics and theoretical neuroscience. Finally, we partition the blanket into sensory and active states. It follows that active states can be seen as performing active inference and well-known forms of stochastic control (such as PID control), which are prominent formulations of adaptive behaviour in theoretical biology and engineering.

(a) Outline of paper This paper has three parts, each of which introduces a simple, but fundamental, move.
(i) The first is to partition the world into internal and external states whose boundary is modelled with a Markov blanket [3,4]. This allows us to identify conditions under which internal states encode information about external states. (ii) The second move is to equip this partition with stochastic dynamics. The key consequence of this is that internal states can be seen as continuously inferring external states, consistent with variational inference in Bayesian statistics and with predictive processing accounts of biological neural networks in theoretical neuroscience. (iii) The third move is to partition the boundary into sensory and active states. It follows that active states can be seen as performing active inference and stochastic control, which are prominent descriptions of adaptive behaviour in biological agents, machine learning and robotics.

(b) Related work
The emergence and sustaining of complex (dissipative) structures have been subjects of long-standing research starting from the work of Prigogine [5,6], followed notably by Haken's synergetics [7], and in recent years, the statistical physics of adaptation [8]. A central theme of these works is that complex systems can only emerge and sustain themselves far from equilibrium [9][10][11].
Information processing has long been recognised as a hallmark of cognition in biological systems. In light of this, theoretical physicists have identified basic instances of information processing in systems far from equilibrium using tools from information theory, such as how a drive for metabolic efficiency can lead a system to become predictive [12][13][14][15].
A fundamental aspect of biological systems is a self-organisation of various interacting components at several spatial scales [1,2]. Much research currently focuses on multipartite processes-modelling interactions between various sub-components that form biological systems-and how their interactions constrain the thermodynamics of the whole [16][17][18][19][20].
At the confluence of these efforts, researchers have sought to explain cognition in biological systems. Since the advent of the 20th century, Bayesian inference has been used to describe various cognitive processes in the brain [21][22][23][24][25]. In particular, the free energy principle [23], a prominent theory of self-organisation from the neurosciences, postulates that Bayesian inference can be used to describe the dynamics of multipartite, persistent systems modelled as Markov blankets at nonequilibrium steady-state [26][27][28][29][30].
This paper connects and develops some of the key themes from this literature. Starting from fundamental considerations about adaptive systems, we develop a physics of things that hold beliefs about other things-consistently with Bayesian inference-and explore how it relates to known descriptions of action and behaviour from the neurosciences and engineering. Our contribution is theoretical: from a biophysicist's perspective, this paper describes how Bayesian descriptions of biological cognition and behaviour can emerge from standard accounts of physics. From an engineer's perspective this paper contextualises some of the most common stochastic control methods and reminds us how these can be extended to suit more sophisticated control problems.

(c) Notation
Let Π ∈ R d×d be a square matrix with real coefficients. Let η, b, µ denote a partition of the states [ [1, d]], so that We denote principal submatrices with one index only (i.e., we use Πη instead of Πηη). Similarly, principal submatrices involving various indices are denoted with a colon When a square matrix Π is symmetric positive-definite we write Π 0. ker, Im and · − respectively denote the kernel, image and Moore-Penrose pseudo-inverse of a linear map or matrix, e.g., a non-necessarily square matrix such as Π µb . In our notation, indexing takes precedence over (pseudo) inversion, for example,

Markov blankets
The section formalises the notion of boundary between a system and its environment as a Markov blanket [3,4], depicted graphically in Figure 1. Intuitive examples of a Markov blanket are that of a cell membrane, mediating all interactions between the inside and the outside of the cell, or that of sensory receptors and muscles that bound the nervous system. To formalise this intuition, we model the world's state as a random variable x with corresponding probability distribution p over a state-space X = R d . We partition the state-space of x into external, blanket and internal states: External, blanket and internal state-spaces (E, B, I) are taken to be Euclidean spaces for simplicity.
A Markov blanket is a statement of conditional independence between internal and external states given blanket states.

Definition 2.1 (Markov blanket).
A Markov blanket is defined as Figure 1. Markov blanket depicted graphically as an undirected graphical model, also known as a Markov random field [4,31]. (A Markov random field is a Bayesian network whose directed arrows are replaced by undirected arrows). The circles represent random variables. The lines represent conditional dependencies between random variables. The Markov blanket condition means that there is no line between µ and η. This means that, µ and η are conditionally independent given b. In other words, knowing the internal state µ, does not afford additional information about the external state η when the blanket state b is known. Thus blanket states act as an informational boundary between internal and external states.
The existence of a Markov blanket can be expressed in several equivalent ways For now, we will consider a (non-degenerate) Gaussian distribution p encoding the distribution of states of the world p(x) := N (x; 0, Π −1 ), Π 0, with associated precision (i.e., inverse covariance) matrix Π. Throughout, we will denote the (positive definite) covariance by Σ := Π −1 . Unpacking (2.1) in terms of Gaussian densities, we find that a Markov blanket is equivalent to a sparsity in the precision matrix (M.B.) ⇐⇒ Πηµ = Πµη = 0. (2.2)

(a) Expected internal and external states
Blanket states act as an information boundary between external and internal states. Given a blanket state, we can express the conditional probability densities over external and internal states (using (2.1) and [32,Prop. 3.13] This enables us to associate to any blanket state its corresponding expected external and expected internal states: Pursuing the example of the nervous system, each sensory impression on the retina and oculomotor orientation (blanket state) is associated with an expected scene that caused sensory input (expected external state) and an expected pattern of neural activity in the visual cortex (expected internal state) [33].

(b) Synchronisation map
A central question is whether and how expected internal states encode information about expected external states. For this, we need to characterise a synchronisation function σ, mapping the expected internal state to the expected external state, given a blanket state σ(µ(b)) = η(b). This is summarised in the following commutative diagram: The existence of σ is guaranteed, for instance, if the expected internal state completely determines the blanket state-that is, when no information is lost in the mapping b → µ(b) in virtue of it being one-to-one. In general, however, many blanket states may correspond to an unique expected internal state. Intuitively, consider the various neural pathways that compress the signal arriving from retinal photoreceptors [34], thus many different (hopefully similar) retinal impressions lead to the same signal arriving in the visual cortex.

i Existence
The key for the existence of a function σ mapping expected internal states to expected external states given blanket states, is that for any two blanket states associated with the same expected internal state, these be associated with the same expected external state. This non-degeneracy means that the internal states (e.g., patterns of activity in the visual cortex) have enough capacity to represent all possible expected external states (e.g., 3D scenes of the environment). We formalise this in the following Lemma: rspa.royalsocietypublishing.org Proc R Soc A 0000000

(ii) For any two blanket states
See Appendix A for a proof of Lemma 2.1.

Example 2.2.
• When external, blanket and internal states are one-dimensional, the existence of a synchronisation map is equivalent to Π µb = 0 or Π µb = Π ηb = 0.
• If Π µb is chosen at random-its entries sampled from a non-degenerate Gaussian or uniform distribution-then Π µb has full rank with probability 1. If furthermore the blanket state-space B has lower or equal dimensionality than the internal state-space I, we obtain that Π µb is oneto-one (i.e., ker Π µb = 0) with probability 1. Thus, in this case, the conditions of Lemma 2.1 are fulfilled with probability 1.

ii Construction
The key idea to map an expected internal state µ(b) to an expected external state η(b) is to: 1) find a blanket state that maps to this expected internal state (i.e., by inverting µ) and 2) from this blanket state, find the corresponding expected external state (i.e., by applying η): We now proceed to solving this problem. Given an internal state µ, we study the set of blanket states b such that µ(b) = µ Here the inverse on the right hand side of (2.4) is understood as the preimage of a linear map. We know that this system of linear equations has a vector space of solutions given by [35]

Definition 2.2 (Synchronisation map).
We define a synchronisation function that maps to an internal state a corresponding most likely internal state 23 The expression in terms of the precision matrix is a byproduct of Appendix A.
Note that we can always define such σ, however, it is only when the conditions of Lemma 2.1 are fulfilled that σ maps expected internal states to expected external states σ(µ(b)) = η(b). When this is not the case, the internal states do not fully represent external states, which leads to a partly degenerate type of representation, see Figure 2 for a numerical illustration obtained by 7 rspa.royalsocietypublishing.org Proc R Soc A 0000000 sampling from a Gaussian distribution, in the non-degenerate (left) and degenerate cases (right), respectively.

Bayesian mechanics
In order to study the time-evolution of systems with a Markov blanket, we introduce dynamics into the external, blanket and internal states. Henceforth, we assume a synchronisation map under the conditions of Lemma 2.1.

(a) Processes at a Gaussian steady-state
We consider stochastic processes at a Gaussian steady-state p with a Markov blanket. The steady-state assumption means that the system's overall configuration persists over time (e.g., it does not dissipate). In other words, we have a Gaussian density p = N (0, Π −1 ) with a Markov blanket (2.2) and a stochastic process distributed according to p at every point in time Recalling our partition into external, blanket and internal states, this affords a Markov blanket that persists over time, see Figure 3 x Note that we do not require x t to be independent samples from the steady-state distribution p. On the contrary, x t may be generated by extremely complex, non-linear, and possibly stochastic equations of motion. See Example 3.1 and Figure 4 for details.    Figure 2, only, here, the samples are drawn from trajectories of a diffusion process (3.2) with a Markov blanket. Although this is not the case here, one might obtain a slightly noisier correspondence between predictions σ(µ(bt)) and expected external states η(bt)compared to Figure 2-in numerical discretisations of a diffusion process. This is because the steady-state of a numerical discretisation usually differs slightly from the steady-state of the continuous-time process [37]. Right: This panel plots the transition probabilities of the same diffusion process (3.2), for the blanket state at two different times. The joint distribution (depicted as a heat map) is not Gaussian but its marginals-the steady-state density-are Gaussian. This shows that in general, processes at a Gaussian steady-state are not Gaussian processes. In fact, the Ornstein-Uhlenbeck process is the only stationary diffusion process (3.2) that is a Gaussian process, so the transition probabilities of non-linear diffusion processes (3.2) are never multivariate Gaussians.
• Stationary diffusion processes, with initial condition x 0 ∼ p. Their time-evolution is given by an Itô stochastic differential equation (see Appendix B): Here, W t is a standard Brownian motion (a.k.a., Wiener process) [38,39] and ς, Γ, Q are sufficiently well-behaved matrix fields (see Appendix B). Namely, Γ is the diffusion tensor (half the covariance of random fluctuations), which drives dissipative flow; Q is an arbitrary antisymmetric matrix field which drives conservative (i.e., solenoidal) flow. We emphasise that there are no nondegeneracy conditions on the matrix field ς-in particular, the process is allowed to be non-ergodic or even completely deterministic (i.e., ς ≡ 0). Also, ∇· denotes the divergence of a matrix field defined as (∇ · (Γ + Q)) i := j ∂ ∂xj (Γ + Q) ij . • More generally, x t could be generated by any Markov process at steady-state p, such as the zig-zag process or the bouncy particle sampler [40-42], by any mean-zero Gaussian process at steady-state p [43] or by any random dynamical system at steady-state p [44].
Remark 3.1. When the dynamics are given by an Itô stochastic differential equation (3.2), a Markov blanket of the steady-state density (2.2) does not preclude reciprocal influences between internal and external states [45,46]. For example, Conversely, the absence of reciprocal coupling between two states in the drift in some instances, though not always, leads to conditional independence [30,36,45].

(b) Maximum a posteriori estimation
The Markov blanket (3.1) allows us to harness the construction of Section 2 to determine expected external and internal states given blanket states Note that η, µ are linear functions of blanket states; since b t generally exhibits rough sample paths, η t , µ t will also exhibit very rough sample paths. We can view the steady-state density p as specifying the relationship between external states (η, causes) and particular states (b, µ, consequences). In statistics, this corresponds to a generative model, a probabilistic specification of how (external) causes generate (particular) consequences.
By construction, the expected internal states encode expected external states via the synchronisation map which manifests a form of generalised synchrony across the Markov blanket [47][48][49]. Moreover, the expected internal state µ t effectively follows the most likely cause of its sensations This has an interesting statistical interpretation as expected internal states perform maximum a posteriori (MAP) inference over external states.

(c) Predictive processing
We can go further and associate to each internal state µ a probability distribution over external states, such that each internal state encodes beliefs about external states We will call qµ the approximate posterior belief associated with the internal state µ due to the forecoming connection to inference. Under this specification, the mean of the approximate posterior depends upon the internal state, while its covariance equals that of the true posterior w.
Note a potential connection with epistemic accounts of quantum mechanics; namely, a world governed by classical mechanics (σ ≡ 0 in (3.2)) in which each agent encodes Gaussian beliefs about external states could appear to the agents as reproducing many features of quantum mechanics [50].
Under this specification (3.4), expected internal states are the unique minimiser of a Kullback-Leibler divergence [51] that measures the discrepancy between beliefs about the external world qµ(η) and the posterior distribution over external variables. Computing the KL divergence (see Appendix C), we obtain In the neurosciences, the right hand side of (3.5) is commonly known as a (squared) precisionweighted prediction error: the discrepancy between the prediction and the (expected) state of the environment is weighted with a precision matrix [24,52,53] that derives from the steadystate density. This equation is formally similar to that found in predictive coding formulations of biological function [24,[54][55][56], which stipulate that organisms minimise prediction errors, and in doing so optimise their beliefs to match the distribution of external states.

(d) Variational Bayesian inference
We can go further and associate expected internal states to the solution to the classical variational inference problem from statistical machine learning [59] and theoretical neurobiology [52,60]. Expected internal states are the unique minimiser of a free energy functional (i.e., an evidence bound [59,61]) (3.6) The last line expresses the free energy as a difference between energy and entropy: energy or accuracy measures to what extent predicted external states are close to the true external states, while entropy penalises beliefs that are overly precise. At first sight, variational inference and predictive processing are solely useful to characterise the average internal state given blanket states at steady-state. It is then surprising to see that the free energy says a great deal about a system's expected trajectories as it relaxes to steady-state. Figure 5 and 6 illustrate the time-evolution of the free energy and prediction errors after exposure to a surprising stimulus. In particular, Figure 5 averages internal variables for any blanket state: In the neurosciences, perhaps the closest analogy is the event-triggered averaging protocol, where neurophysiological responses are averaged following a fixed perturbation, such a predictable neural input or an experimentally-controlled sensory stimulus (e.g., spike-triggered averaging, event-related potentials) [62][63][64].
The most striking observation is the nearly monotonic decrease of the free energy as the system relaxes to steady-state. This simply follows from the fact that regions of high density under the steady-state distribution have a low free energy. This overall decrease in free energy is the essence of the free-energy principle, which describes self-organisation at non-equilibrium steady-state [23,28,29]. Note that the free energy, even after averaging internal variables, may decrease nonmonotonically. See the explanation in Figure 5.

Active inference and stochastic control
In order to model agents that interact with their environment, we now partition blanket states into sensory and active states b t = (s t , a t ) x t = (η t , s t , a t , µ t ).
Intuitively, sensory states are the sensory receptors of the system (e.g., olfactory or visual receptors) while active states correspond to actuators through which the system influences the environment (e.g., muscles). See  states (i.e., active and internal states) respond adaptively to sensory perturbations in order to maintain the steady-state, which we interpret as the agent's preferences or goal. This allows us to relate the dynamics of autonomous states to active inference and stochastic control, which are well-known formulations of adaptive behaviour in theoretical biology and engineering.

(a) Active inference
We now proceed to characterise autonomous states, given sensory states, using the free energy. Unpacking blanket states, the free energy (3.6) reads Crucially, it follows that the expected autonomous states minimise free energy F (s t , a t , µ t ) ≥ F (s t , a t , µ t ),   averaging internal variables for any blanket state. We simulated an Ornstein-Uhlenbeck process with two external, one sensory, one active and two internal variables, initialised at the steady-state density conditioned upon an improbable sensory state p(x 0 |s 0 ). Left: The white line shows the expected active state given sensory states: this is the action that performs active inference and optimal stochastic control. As the process experiences a surprising sensory state, it initially relaxes to steady-state in a winding manner due to the presence of solenoidal flow. Even though solenoidal flow drives the actions away from the optimal action initially, it allows the process to converge faster to steady-state [57,58,73] where the actions are again close to the optimal action from optimal control. Right: We plot the free energy of the expected internal state, averaged over multiple trajectories. In this example, the average free energy does not decrease monotonically-see Figure 5 for an explanation.
where a t denotes the expected active states given sensory states, which is the mean of p(a t |s t ). This result forms the basis of active inference, a well-known framework to describe and generate adaptive behaviour in neuroscience, machine learning and robotics [25,60,[65][66][67][68][69][70][71][72]. See Figure 8.

(b) Multivariate control
Active inference is used in various domains to simulate control [65,69,71,72,[74][75][76][77], thus, it is natural that we can relate the dynamics of active states to well-known forms of stochastic control.
By computing the free energy explicitly (see Appendix C), we obtain that   where we denoted by K the concentration (i.e., precision) matrix of p(s, a, µ). We may interpret (a, µ) as controlling how far particular states [s, a, µ] are from their target set-point of [0, 0, 0], where the error is weighted by the precision matrix K. See Figure 9. (Note that we could choose any other set-point by translating the frame of reference or equivalently choosing a Gaussian steady-state centred away from zero). In other words, there is a cost associated to how far away s, a, µ are from the origin and this cost is weighed by the precision matrix, which derives from the stationary covariance of the steady-state. In summary, the expected internal and active states can be seen as performing multivariate stochastic control, where the matrix K encodes control gains. From a biologist's perspective, this corresponds to a simple instance of homeostatic regulation: maintaining physiological variables within their preferred range.

(c) Stochastic control in an extended state-space
More sophisticated control methods, such as PID (Proportional-Integral-Derivative) control [77,80], involve controlling a process and its higher orders of motion (e.g., integral or derivative terms). So how can we relate the dynamics of autonomous states to these more sophisticated control methods? The basic idea involves extending the sensory state-space to replace the sensory process s t by its various orders of motions t = s

i The stochastic realisation problem
Recall that the sensory process s t is a stationary stochastic process (with a Gaussian steadystate). The following is a central problem in stochastic systems theory: Given a stationary stochastic process s t , find a Markov processs t , called the state process, and a function f such that Moreover, find an Itô stochastic differential equation whose unique solution is the state process s t . The problem of characterising the family of all such representations is known as the stochastic realisation problem [81]. What kind of processes s t can be expressed as a function of a Markov process (4.2)? There is a rather comprehensive theory of stochastic realisation for the case where s t is a Gaussian process (which occurs, for example, when x t is a Gaussian process). This theory expresses s t as a linear map of an Ornstein-Uhlenbeck process [39, 82,83]. The idea is as follows: as a mean-zero Gaussian process, s t is completely determined by its autocovariance function C(t − r) = E [s t ⊗ sr], which by stationarity only depends on |t − r|. It is well known that any mean-zero stationary Gaussian process with exponentially decaying autocovariance function is an Ornstein-Uhlenbeck process (a result sometimes known as Doob's theorem) [39, [84][85][86]. Thus if C equals a finite sum of exponentially decaying functions, we can express s t as a linear function of several nested Ornstein-Uhlenbeck processes, i.e., as an integrator chain from control theory [87,88] t .

(4.3)
In this example, f, f i are suitably chosen linear functions, ς i are matrices and W (i) are standard Brownian motions. Thus, we can see s t as the output of a continuous-time hidden Markov model, whose (hidden) states s (i) t encode its various orders of motion: position, velocity, jerk etc. These are known as generalised coordinates of motion in the Bayesian filtering literature [89][90][91]. See Figure 10.
More generally, the state processs t and the function f need not be linear, which enables to realise non-linear, non-Gaussian processes s t [89,92,93]. Technically, this follows as Ornstein-Uhlenbeck processes are the only stationary Gaussian Markov processes. Note that stochastic realisation theory is not as well developed in this general case [81,89,[93][94][95].
ii Stochastic control of integrator chains Henceforth, we assume that we can express s t as a function of a Markov processs t (4.2). Inserting (4.2) into (4.1), we now see that the expected autonomous states minimise how far themselves and f (s t ) are from their target value of zero  iii PID-like control Proportional-integral-derivative (PID) control is a well-known control method in engineering [77,80]. More than 90% of controllers in engineered systems implement either PID or PI (no derivative) control. The goal of PID control is to control a signal s (1) t , its integral s (0) t , and its derivative s (2) t close to a pre-specified target value [77]. This turns out to be exactly what happens here when we consider the stochastic control of an integrator chain (4.4) with three orders of motion (n = 2). When f is linear, expected autonomous states control integral, proportional and derivative processes s t , s (2) t , respectively, from their target value of zero. Crucially, these control gains are simple by-products of the steady-state density and the stochastic realisation problem.
Why restrict ourselves to PID control when stochastic control of integrator chains is available? It turns out that when sensory states s t are expressed as a function of an integrator chain (4.3), one may get away by controlling an approximation of the true (sensory) process, obtained by truncating high orders of motion as these have less effect on the dynamics, though knowing when this is warranted is a problem in approximation theory. This may explain why integral feedback control (n = 0), PI control (n = 1) and PID control (n = 2) are the most ubiquitous control methods in engineering applications. However, when simulating biological control-usually with highly non-linear dynamics-it is not uncommon to consider generalised motion to fourth (n = 4) or sixth (n = 6) order [92,96].
It is worth mentioning that PID control has been shown to be implemented in simple molecular systems and is becoming a popular mechanistic explanation of behaviours such as bacterial chemotaxis and robust homeostatic algorithms in biochemical networks [77,97,98]. We suggest that this kind of behaviour emerges in Markov blankets at non-equilibrium steadystate. Indeed, stationarity means that autonomous states will look as if they respond adaptively to external perturbations to preserve the steady-state, and we can identify these dynamics as implementations of various forms of stochastic control (including PID-like control).

Discussion
In this paper, we considered the consequences of a boundary mediating interactions between states internal and external to a system. On unpacking this notion, we found that the states internal to a Markov blanket look as if they perform variational Bayesian inference, optimising beliefs about their external counterparts. When subdividing the blanket into sensory and active states, we found that autonomous states perform active inference and various forms of stochastic control (i.e., generalisations of PID control).
Interacting Markov blankets: The sort of inference we have described could be nuanced by partitioning the external state-space into several systems that are themselves Markov blankets (such as Markov blankets nested at several different scales [1]). From the perspective of internal states, this leads to a more interesting inference problem, with a more complex generative model. It may be that the distinction between the sorts of systems we generally think of as engaging in cognitive, inferential, dynamics [99] and simpler systems rest upon the level of structure of the generative models (i.e., steady-state densities) that describe their inferential dynamics.
Temporally deep inference: This distinction may speak to a straightforward extension of the treatment on offer, from simply inferring an external state to inferring the trajectories of external states. This may be achieved by representing the external process in terms of its higher orders of motion by solving the stochastic realisation problem. By repeating the analysis above, internal states may be seen as inferring the position, velocity, jerk, etc of the external process, consistently with temporally deep inference in the sense of a Bayesian filter [91] (a special case of which is an extended Kalman-Bucy filter [100]).
Bayesian mechanics in non-Gaussian steady-states: The treatment from this paper extends easily to non-Gaussian steady-states, in which internal states appear to perform approximate Bayesian inference over external states. Indeed, any arbitrary (smooth) steady-state density may be approximated by a Gaussian density at one of its modes using a so-called Laplace approximation. This Gaussian density affords one with a synchronisation map in closed form 4 that maps the expected internal state to an approximation of the expected external state. It follows that the system can be seen as performing approximate Bayesian inference over external states-precisely, an inferential scheme known as variational Laplace [101]. We refer the interested reader to a worked-out example involving two sparsely coupled Lorenz systems [30]. Note that variational Laplace has been proposed as an implementation of various cognitive processes in biological systems [25,52,60] accounting for several features of the brain's functional anatomy and neural message passing [53,70,99,102,103].
Modelling real systems: The simulations presented here are as simple as possible and are intended to illustrate general principles that apply to all stationary processes with a Markov blanket (3.1). These principles have been used to account for synthetic data arising in more refined (and more specific) simulations of an interacting particle system [27] and synchronisation between two sparsely coupled stochastic Lorenz systems [30]. Clearly, an outstanding challenge is to account for empirical data arising from more interesting and complex structures. To do this, one would have to collect time-series from an organism's internal states (e.g., neural activity), its surrounding external states, and its interface, including sensory receptors and actuators. Then, one could test for conditional independence between internal, external and blanket states (3.1) [104]. One might then test for the existence of a synchronisation map (using Lemma 2.1). This speaks to modelling systemic dynamics using stochastic processes with a Markov blanket. For example, one could learn the volatility, solenoidal flow and steady-state density in a stochastic differential equation (3.2) from data, using supervised learning [105].

Conclusion
This paper outlines some of the key relationships between stationary processes, inference and control. These relationships rest upon partitioning the world into those things that are internal or external to a (statistical) boundary, known as a Markov blanket. When equipped with dynamics, the expected internal states appear to engage in variational inference, while the expected active states appear to be performing active inference and various forms of stochastic control.
The rationale behind these findings is rather simple: if a Markov blanket derives from a steadystate density, the states of the system will look as if they are responding adaptively to external perturbations in order to recover the steady-state. Conversely, well-known methods used to build adaptive systems implement the same kind of dynamics, implicitly so that the system maintains a steady-state with its environment.
Data Accessibility. All data and numerical simulations can be reproduced with code freely available at https://github.com/conorheins/bayesian-mechanics-sdes. Acknowledgements. LD would like to thank Kai Ueltzhöffer, Toby St Clere Smithe and Thomas Parr for interesting discussions. We are grateful to our two anonymous reviewers for feedback which substantially improved the manuscript.

A Existence of synchronisation map: proof
We prove Lemma 2.1.
Proof. (i) ⇐⇒ (ii) follows by definition of a function.

B The Helmholtz decomposition
We consider a diffusion process x t on R d satisfying an Itô stochastic differential equation (SDE) [39, 107,108], We now recall an important result from the theory of stationary diffusion processes, known as the Helmholtz decomposition. It consists of splitting the dynamic into time-reversible (i.e., dissipative) and time-irreversible (i.e., conservative) components. The importance of this result in non-equilibrium thermodynamics was originally recognised by Graham in 1977 [109] and has been of great interest in the field since [39, [110][111][112]. Furthermore, the Helmholtz decomposition is widely used in statistical machine learning to generate Monte-Carlo sampling schemes [39, 73,[113][114][115][116].
Lemma B.1 (Helmholtz decomposition). For a diffusion process (A 1) and a smooth probability density p > 0, the following are equivalent: (i) p is a steady-state for x t . (ii) We can write the drift as where Γ = ςς /2 is the diffusion tensor and Q = −Q is a smooth antisymmetric matrix field. ∇· denotes the divergence of a matrix field defined as (∇ · Q) i := j ∂ ∂xj Q ij . Furthermore, frev is invariant under time-reversal, while f irrev changes sign under time-reversal.
In the Helmholtz decomposition of the drift (A 2), the diffusion tensor Γ mediates the dissipative flow, which flows towards the modes of the steady-state density, but is counteracted by random fluctuations W t , so that the system's distribution remains unchanged-together these form the time-reversible part of the dynamics. In contrast, Q mediates the solenoidal flow-whose direction is reversed under time-reversal-which consists of conservative (i.e., Hamiltonian) dynamics that flow on the level sets of the steady-state. See Figure 11 for an illustration. Note that the terms time- sense that time-reversibility denotes invariance under time-reversal. This is opposite to reversible and irreversible in a classical physics sense, which respectively mean energy preserving (i.e., conservative) and entropy creating (i.e., dissipative). The lower panels plot sample paths of the time-reversible (lower left) and time-irreversible (lower right) parts of the dynamics. Purely conservative dynamics (lower right panel) are reminiscent of the trajectories of massive bodies (e.g., planets) whose random fluctuations are negligible, as in Newtonian mechanics. The lower panels help illustrate the meaning of time-irreversibility: If we were to reverse time (c.f., (A 3)), the trajectories the time-reversible process would be, on average, no different, while the trajectories of the time-irreversible process would flow, say, clockwise instead of counterclockwise, which would clearly be distinguishable. Here, the full process (upper left panel) is a combination of both dynamics. As we can see the time-reversible part affords the stochasticity while the time-irreversible part characterises non-equilibria and the accompanying wandering behaviour that characterises life-like systems [11,117].

Proof.
"⇒" It is well-known that when x t is stationary at p, its time-reversal is also a diffusion process that solves the following Itô SDE [118] dx This enables us to write the drift f as a sum of two terms: one that is invariant under time reversal, another that changes sign under time-reversal It is straightforward to identify the time-reversible term frev = p −1 ∇ · (Γ p) = p −1 Γ ∇p + p −1 p∇ · Γ = Γ ∇ log p + ∇ · Γ.
Decomposing the drift into time-reversible and time-irreversible parts, we obtain that the time-irreversible part produces divergence-free (i.e., conservative) flow w.r.t. the steadystate distribution ∇ · (f irrev p) = 0.
Now recall that any (smooth) divergence-free vector field is the divergence of a (smooth) antisymmetric matrix field A = −A T [109,110,120] f irrev p = ∇ · A.
We define a new antisymmetric matrix field Q := p −1 A. It follows from the product rule for divergences that we can rewrite the time-irreversible drift as required f irrev = Q∇ log p + ∇ · Q.

(A 5)
In addition, we define the auxiliary antisymmetric matrix field A := pQ and use the product rule for divergences to simplify the expression of the time-irreversible part f irrev = p −1 ∇ · A.

C Free energy computations
The free energy reads (3.6)