Optimal foraging and the information theory of gambling

At a macroscopic level, part of the ant colony life cycle is simple: a colony collects resources; these resources are converted into more ants, and these ants in turn collect more resources. Because more ants collect more resources, this is a multiplicative process, and the expected logarithm of the amount of resources determines how successful the colony will be in the long run. Over 60 years ago, Kelly showed, using information theoretic techniques, that the rate of growth of resources for such a situation is optimized by a strategy of betting in proportion to the probability of pay-off. Thus, in the case of ants, the fraction of the colony foraging at a given location should be proportional to the probability that resources will be found there, a result widely applied in the mathematics of gambling. This theoretical optimum leads to predictions as to which collective ant movement strategies might have evolved. Here, we show how colony-level optimal foraging behaviour can be achieved by mapping movement to Markov chain Monte Carlo (MCMC) methods, specifically Hamiltonian Monte Carlo (HMC). This can be done by the ants following a (noisy) local measurement of the (logarithm of) resource probability gradient (possibly supplemented with momentum, i.e. a propensity to move in the same direction). This maps the problem of foraging (via the information theory of gambling, stochastic dynamics and techniques employed within Bayesian statistics to efficiently sample from probability distributions) to simple models of ant foraging behaviour. This identification has broad applicability, facilitates the application of information theory approaches to understand movement ecology and unifies insights from existing biomechanical, cognitive, random and optimality movement paradigms. At the cost of requiring ants to obtain (noisy) resource gradient information, we show that this model is both efficient and matches a number of characteristics of real ant exploration.

< 1. In both of these cases, the full capital or proportion of foragers should be allocated across foraging sites to maximise the long-term wealth, although the Kelly strategy may also be followed by betting a fixed proportion of full bets. This so-called -Kelly strategy has a much slower rate of growth in wealth (MacLean et al., 2010), however, short-term volatility in wealth (or biomass) is reduced by a factor ( (Rising et al., 2012). Animals are often risk averse to variable rewards (Kacelnik and Bateson, 1996) and this may be seen at the level of the colony; in practice only a proportion of ants forage, with some staying purposefully inactive for significant periods of time (Charbonneau and Dornhaus, 2015). In a colony of, say, 200 workers, 50 might forage, which would correspond to = 0.25; but for the purpose of this paper we do not consider this further.
When ∑ " # $ > 1 subfair odds are said to be available, and it is better to hold back some proportion of the ant foragers (or a gambler's wealth) and try to bet on outcomes offering the most favourable expected return 0 0 . In this case the optimal betting amount is solved using a 'waterfilling' method and is dealt with in Erkip (1996).

Metropolis-Hastings -further details and comparison with ant data
The M-H method makes use of a proposal density (which depends on the current state ) to create a new proposal state to potentially sample from. Q can be simply a uniform distribution: in a discretized environment these can be (5) + [−1,0,1] with equal probability. To decide whether to accept the new state as the next step in the sample, we compute the quantity = * ( @ ) * ( (5) ) ( (5) ; @ ) ( @ ; (5) ) If ≥ 1 then the new state is accepted; otherwise it is accepted with probability . If the step is accepted, we set (5C") = @ ; if it is rejected then (5C") = (5) (the current state is taken forward to the next step). The M-H method is widely used for sampling from high-dimensional problems, but it has a major disadvantage that it explores the probability distribution by a random walk, which can take many steps to move through the space, according to √ where T is the number of steps and is the step length.
The Metropolis-Hastings algorithm was used to sample from a sparse probability distribution. This is generated by combining a gamma-distributed background noise (shape parameter=0.2, scale parameter=1) on a 100×100 grid given a Gaussian blur ( = 3, filter size 100×100) Using = 18 simulated walkers exploring for 600 iterations, the exponent was found to be 0.487, 95% confidence interval (0.444 0.529), i.e. not significantly different to 0.5 and so engaging in a standard diffusive search, as expected. We can use the M-H walkers as a model of exploring ants, and so to assess its validity we can examine trajectory data of real ants exploring a uniform empty space (Hunt et al., 2016a) and compare it with simulations from this model, to see whether ants are engaged in a standard diffusive (Brownian) exploration of space. The root mean square (r.m.s.) displacement for = 18 ants was found for the first 60 seconds of their exploration of an unfamiliar arena (600 trajectory points with a sampling period of 0.1s, a time short enough that they would not reach the edge of the arena), and its log was regressed on log time. The gradient was found to be 0.59 with a 95% confidence interval of (0.55,0.63), which is greater than 0.5 and so the ants are engaged in superdiffusive search.

Pseudocode for Hamiltonian Monte Carlo method
input: target probability distribution P, leapfrog steps L, stepsize ε output: sequence of samples from P in proportion to probability density (after 'burn in' period) set initial state [x 0 , p 0 ] for t = 1: simulation time sample new random initial momentum p 0 from standard normal distribution run Leapfrog method for L steps with stepsize ε to obtain proposed states x * and p * calculate acceptance probability a = min(1, exp(−E(x * ) + E(x 0 ) − K(p * ) + K(p 0 ))) if a > 1, accept = 1; elseif a > rand(1), accept = 1; else accept = 0 See also p.388 of Mackay (2003): http://www.inference.org.uk/itprnn/book.pdf

Leapfrog method
In two of the MCMC methods presented, HMC and PMR, there are two key parameters ε and that must be chosen suitably to generate a sample efficiently. See Brooks et al. (2011) for a full treatment of these issues.
Although the dynamics of a system in Hamiltonian mechanics is described using differential equations for continuous time and space, for implementation on a computer the equations must be approximated by discretizing time using a small step length ε. We may assume a Hamiltonian of the form ( , ) = ( ) + ( ) and a kinetic energy ( ) = Q S" /2, and a diagonal M with Then the leapfrog method proceeds as follows: We start with a half step for the momentum variables, a full step follows for the position variables using those new momentum variables, and finally we do another half step for the momentum variables using the new values for the position variables. This process is iterated a certain number of times , and preserves volume exactly. It is also reversible by negating and applying the same number of steps , then negating p again. It gains its name from the use of interleaved time points such that 0 and 0 'leapfrog' over each other.
The leapfrog step length was set to be ε = 0.3 in the simulations presented in the main text.
Selecting a suitable step length is vital, because if it is too large there will be a very low acceptance rate for the states proposed by the simulated trajectories. If it is too small it will waste computation time, as the trajectories will make unnecessarily short progress through the space, or it can lead to slow exploration by a random walk through the space if the trajectory length ε is too small, which defeats the object of the method. In the first instance reasonably effective ε and can be found through trial and error.