Spatio-temporal reconstruction of emergent flash synchronization in firefly swarms via stereoscopic 360-degree cameras

During mating season, males of synchronous firefly species flash in unison within swarms of thousands of individuals. These strongly correlated collective displays have inspired numerous mathematical models to explain how global synchronous patterns emerge from local interactions. Yet, experimental data to validate these models remain sparse. To address this gap, we develop a method for three-dimensional tracking of firefly flashes, using a stereoscopic set-up of 360-degree cameras. We apply this method to record flashing displays of the North American synchronous species Photinus carolinus in its natural habitat as well as within controlled environments, and obtain the three-dimensional reconstruction of flash occurrences in the swarm. Our results show that even a small number of interacting males synchronize their flashes; however, periodic flash bursts only occur in groups larger than 15 males. Moreover, flash occurrences are correlated over several metres, indicating long-range interactions. While this suggests emergent collective behaviour and cooperation, we identify distinct individual trajectories that hint at additional competitive mechanisms. These reveal possible behavioural differentiation with early flashers being more mobile and flashing longer than late followers. Our experimental technique is inexpensive and easily implemented. It is extensible to tracking light communication in various firefly species and flight trajectories in other insect swarms.

1 Provided Software

Matlab Code
A set of Matlab functions to perform three-dimensional reconstruction from pairs of 360-cameras is made available at http://www.github.com/rapsar/stereo360. These functions mirror those of the Matlab Computer Vision Toolbox for standard stereo-reconstruction. Below is a brief description of our main functions (see code for more details).
xy2alpha: returns the spherical projections α of world points given their (x, y) coordinates in equirectangular movie frames (the size of movie frames in pixels needs to be specified); estimate360CameraParameters: estimates the relative camera pose ( t, R) given a set of matched projections α 1 , α 2 (different methods are possible); matchPoints: finds optimal pairing for two lists of (x, y) points given ( t, R); triangulate360: triangulates pairs of matched projections given ( t, R).

FireflyNavigator
Our standalone application FireflyNavigator, made in Unity, is available at http://www.github. com/elie-s/FireflyNavigator. The repository also contains a short user guide with instructions, and two datasets of 3D reconstructed swarms: one for the swarm in the tent (xyzt_tent.csv), and the other for the swarm in its natural habitat (xyzt_wild.csv). Length units are meters.

Supplemental Movies
Movie S1: 360-degree movie showing flash occurrences over 1min, corresponding to the recordings of gp1 presented in figure 2. This movie is an enhanced rendering of actual data. While flashes were recorded in absolute darkness, they are shown overlaid onto the corresponding background (recorded at an earlier time) for orientation. Flashes have also been slightly dilated for better visualization, and coloured in yellow. The movie is rendered in equirectangular frames, and can be visualized as a 2D projection or as a 360-degree movie with Windows' Movies & TV or in a Virtual Reality Headset.
Movie S2: movie of reconstructed flash occurrences over 1min, corresponding to the reconstructed swarm presented in figure 2. The movie is played in real time, and the last few seconds all triangulated flash occurrences.
Movie S3: movie of reconstructed trajectories corresponding to flash bursts in the tent at n = 15 presented in figure 9. Coloured dots of triangulated flash positions, and black dots indicate interpolated flight paths. For better visualization, flash bursts are separated by 2s.

Additional Experimental Results
To substantiate the findings presented in the Main Text, we provide here additional experimental results.

Firefly Identification
Fireflies introduced in the tent for our controlled experiments were gently captured using insect nets, and placed in small plastic boxes for inspection. We visually verified that they were Photinus carolinus males, using GSMNP's specimen collections as a reference (figure S1).

Absence of Burst Periodicity at Low Firefly Density
In figure S2 we report additional time series of number of recorded flashes for 1, 2, or 3 fireflies in isolation, from three different sets of controlled experiments. For consistency and clarity, we show intervals of 3min (180s), although experiments typically lasted 5min to 10min. In accordance with the dataset presented in the Main Text, flash bursts do not occur at well-defined intervals, contrary to what is observed when several fireflies are allowed to interact.

Flash Intensity Patterns
In our results in the Main Text, we effectively consider firefly flashes as being either "on" or "off" (rectangular pattern), depending on whether a certain pixel brightness is reached. Of course, flash intensities exhibit temporal variations, and notably ramp up/down phases. However, these phases are fast compared to the typical duration of a flash, and below our time resolution of 30fps or 60fps, as shown in Fig. S4 and with higher precision in Ref. [2]. Therefore, our approximation is justified.  Figure S4: Average pixel brightness (in arbitrary units) in the vicinity of 10 independent flash occurrences, recorded at 30fps in the wild. The threshold value to consider a firefly "on" or "off" is indicated by the red line.

Temporal Pattern of Collective Flashing
In figure S5, we present a schematic to illustrate the emergence of the triangular shape in bursts of collective flashing. Assuming different fireflies trigger their flash trains at different times, the number of flashes in the swarms over a burst increases then decreases. Figure S5: Collective flashing with out-of-phase flash trains leads to triangular flash bursts.

From Positions to Streaks and Trajectories
Individual 3D points can be concatenated into streaks, trajectories, and bursts. Points appearing in consecutive frames are first connected into streaks (spatial localization of a single, continuous flash). This relies on a simple matching algorithm based on minimization of the sum of squared distances within a defined threshold [1]. Based on the flash pattern of P. carolinus, it is possible to connect successive streaks into trajectories, notably thanks to their regular interflash interval, between 0.45s and 0.5s [2,3]. Therefore, for two streaks to be potentially connected, they must occur within that time period. The second condition to connect two streaks is that they are spatially compatible. Since fireflies can be either stationary or fast-moving (see below), this can mean one of two different things. If two streaks occur within a short distance (e.g., 5cm), they are connected. The other possibility to connect two streaks is if they can be linked by a "reasonable" flight path. Specifically, the path between two successive streaks is approximated using a polynomial interpolation (degree = 3) of each spatial coordinate. To be valid, this "likely" flight path must satisfy two criteria: 1) velocities must be consistent with velocities in the recorded streaks; 2) accelerations must be below a certain threshold. Notably, flight paths that would involve high velocities or complicated turns are rejected.
Transitively connected streaks form trajectories. When ambiguities occur, for example a streak potentially connected to two successors, they are resolved by removing edges in the directed graph of streak connections based on a "three-point" algorithm. This means that for two possible 3-node paths going through the same node, the path that will be considered is the one that minimizes the mean acceleration of the interpolated flight.
Finally, a burst is defined as a set of transitively (not mutually) simultaneous trajectories. This is based on the fact that P. carolinus collectively flash in bursts separated by a few seconds of total darkness, as detailed below [2,3]. It is not possible to extend trajectories across bursts, as too much localization information is lost in those few seconds. To estimate theoretical limits to triangulation resolution, we aim to express the distance r 1 between a world point X and camera C 1 : r 1 = f (θ 1 , θ 2 ). Then, we assess how small incertainties on θ 1 , θ 2 affect 3D reconstruction. Without loss of generality we consider the upper half-plane, 0 < θ 1 < π, and triangulation requires θ 1 < θ 2 < π. From the geometrical considerations of figure S4, we have r 1 = s 1 / cos θ 1 , s 1 = 1 + s 2 , s 2 = o/ tan(θ 2 ), and o = s 1 tan θ 1 . Consequently, s 1 = s + s 1 tan θ 1 / tan θ 2 , so that s 1 = s(tan θ 2 − tan θ 1 )/ tan θ 2 . Finally: Using Wolfram Alpha, we calculate the partial derivatives: Assuming now |dθ 1 | |dθ 2 | =dθ > 0, we can approximate the errord r 1 > 0 on r 1 as a function of θ 1 , θ 2 as:d Taking the separation between the two cameras as s = 1m and assumingdθ = 10 −3 rad (see Main Text), we solve numerically ford r 1 . The results are presented in figure S7. As expected, resolution is maximum when the rays coming from C 1 , C 2 intersect at close to 90 • ; it is minimum when the rays intersect at sharp angles, which is both at large distances, and close to the cameras' connecting line.

Experimental Reconstruction
Calibration Starting from a set of matched points, we propose and implement two general methods for estimating the camera pose ( t, R): optimization search, and fundamental matrix computation (see Main Text). Fundamental Matrix computation uses a Matlab built-in function, which can Triangulation Inaccuracies When performing the 3D reconstruction of recorded flashes, we noticed some obvious inaccuracies, despite a generally very reliable technique. We briefly mention them here. We also insist that similar inaccuracies can occur in traditional stereo-reconstruction. First, in the reconstruction of flashes in the tent, we observed about 1% of points outside the volume of the tent (figure S9a). These points, curiously, are aligned on two specific planes connecting the cameras to the edges of the tent's curved roof. We believe these anomalous triangulations occur for flashes that take place in those narrow corners, and which may be reflected on the tent's fabric, hence changing their detected localization from their real localization.
Second, in field experiments, we observed that streaks reconstructed far from the cameras tend to exhibit a"seeping" effect, for which streaks tend to align with the direction leading to the camera (figure S9b). This is a direct consequence of deteriorating resolution at large distances, as described above.