An artificial elementary eye with optic flow detection and compositional properties
Abstract
We describe a 2 mg artificial elementary eye whose structure and functionality is inspired by compound eye ommatidia. Its optical sensitivity and electronic architecture are sufficient to generate the required signals for the measurement of local optic flow vectors in multiple directions. Multiple elementary eyes can be assembled to create a compound vision system of desired shape and curvature spanning large fields of view. The system configurability is validated with the fabrication of a flexible linear array of artificial elementary eyes capable of extracting optic flow over multiple visual directions.
1. Introduction
Insects have evolved vision systems with a wide variety of morphologies and distribution on the animal head tailored to assist specific behaviours in their ecological niche [1]. The advantage of this variable spatial design with respect to vertebrate eyes is that it can offer such customized sensing over large fields of view combined with relatively small packages and fast extraction of motion information. Insect vision organs can be categorized in two main types: insect simple eyes or ocelli are individual organs made of a corneal refracting interface, often underlying a lens, which focuses light onto a variable number of photoreceptors, ranging from a handful in the stemmata of polychaete annelid worms to tens of thousands in some flying insect ocelli [1]. Ocelli can detect change in patterns of light intensity [2–4] and help in tasks such as attitude stabilization [5]. On the other hand, compound eyes consist of an arrangement of tiny optical units called ommatidia, each composed of a microscopic lens that focuses light on a small set of photoreceptors. Compound eyes are specialized in optic flow extraction from wide fields of view to assist in tasks like collision-free navigation, flight stabilization, and prey, predator or mate detection [1].
Vision systems that can be configured like insect eyes to suit specific tasks could provide useful information in an efficient manner for many applications where energy and payload constraints are drastic, such as microflying robot navigation [6–8]. Curved artificial compound eyes made of hundreds of ommatidia have been recently described [9,10] and tested for basic navigation control [8,9]. These prototypes possess a fixed design as optic flow extraction over adjacent ommatidia puts strong constraints on the interommatidial angle and thus geometry of the compound eye. An arrangement of four phototransistors inspired by the three dorsal ocelli found in most flying insects was recently used to stabilize the attitude of a microflyer [11]. The four photodetectors used to compute the angular velocities of the robot encompassed a volume of 4 × 4 × 3.3 mm and yielded a non-customizable design. In another work, a Centeye's Tam4 optic flow sensor was used to control the altitude of a tethered microflyer, although this sensor extracts motion in only a single direction and has a size of approximately 3 × 3 mm2 [12].
Here, we describe the design, fabrication, configurability and use of an artificial elementary eye, partly inspired by the insect ommatidium, that can be combined in arrays with different geometries. In contrast to a single insect ommatidium, the artificial elementary eye can extract optic flow vectors in multiple directions by means of several suitably arranged photodetectors. This feature, which is loosely equivalent to a small cluster of neighbouring ommatidia or to a single ocellus, makes the elementary eye usable either as a standalone sensor or as an element of an artificial compound eye without constraints on the visual angle between adjacent elementary eyes. The sensor yields three photoreceptors under a single lens and bears a size of 1925 × 475 × 860 µm and a weight of 2 mg, which includes the optics and the photosensitive area with a total footprint diameter of 175 µm, an analogue signal processing circuit for illuminance adaptation, an analogue-to-digital signal conversion circuit, and a communication interface. The signal acquisition frequency of the elementary eye is 300 Hz, which allows for fast motion extraction. The photodetector sensitivity, analogue processing circuitry, analogue-to-digital converter (ADC) components and communication interface of the sensor are similar to the ones of the artificial ommatidia columns in the sensor presented in Floreano et al. [9].
2. Structure of the elementary eye
The artificial elementary eye was designed finding inspiration in the ommatidia of insect apposition compound eyes as natural local motion sensors. Each natural ommatidium consists of a lens, an elongated waveguide space known as the rhabdom, and a small set of photoreceptors along the rhabdom. In the elementary eye (figure 1 and electronic supplementary material, figure S1), the focusing element consists of a moulded polymer microlens (Ormocomp, Micro Resist Technology GmbH, Berlin, Germany) placed on a 560 µm thick glass stack (display glass AF32, Schott AG), which guides light to the photosensitive areas of an optoelectronic silicon chip (figure 1b). A characteristic of the elementary eye is that the optics have been designed such that the projected light on the photodetectors bears the same area as the photodetector footprint in order to optimize its sensitivity.
Figure 1. The artificial elementary eye yielding three photodetectors. (a) Top-view microscopy image; (b) detailed scheme of the elementary eye and its components; (c) image of the elementary eye mounted on the PCB surface before and (d) after glop-top encapsulation; (e) zoomed image of the photodetector triplet without optics and (f) lateral schematic view of the elementary eye.
Models of motion detection [13,14] in the retina of mammals [15] as well as in the compound eyes of arthropods [16] suggest that isolated pairs of photoreceptors provide information about motion along a single direction. Recently, it has been observed that the fly visual system extracts directional motion in the four cardinal directions using eight different local motion detectors, being the major or unique signal inputs to downstream neuropiles and motion-driven behaviours [16]. Inspired by these findings, we have endowed our elementary eye with a triplet of hexagonal photodetectors arranged so that their centres form the vertices of an equilateral triangle (figure 1e and electronic supplementary material, figure S1). The unidirectional motion values issued by the three photodetector pairs can be combined to compute an optic flow vector in any direction of the image plane from the vision field of the elementary eye. A higher number of photodetectors could increase the precision of the optic flow extraction, but it would come at the cost of significantly increasing the elementary eye footprint. A lower number would not provide motion information in more than a single direction.
Diurnal flying insects perceive highly contrasted images of the surrounding world as a result of the 1 : 1 ratio between the interommatidial and the acceptance angle of their compound eyes. This characteristic provides them with an efficient motion extraction at fast speeds [17]. Our elementary eye was designed to yield acceptance and inter-detector angles inspired by the optical properties of the ommatidia in Drosophila melanogaster, where each ommatidium has an acceptance angle of 4.5° [18] and an interommatidial angle of 4.7°–5.5° [19] (see the electronic supplementary material for further details on Microoptics design).
Natural ommatidia bear pigmented sidewalls that prevent light rays other than those refracted by the lens from arriving at the photodetectors. In the elementary eye, this function is played by a diaphragm layer inside the glass stack (figure 1b) and an opaque glob-top encapsulation polymer on the sidewalls (figure 1d) blocking light that does not come directly from the upper optical aperture.
In a natural visual scene, illumination may have a dynamic range up to eight decades. To avoid saturation due to illumination changes, each photodetector is connected to an adaptive analogue VLSI circuit featuring a large dynamic range at ultra-low-power processing [20] (electronic supplementary material, figure S1a,b). A first-order, low-pass filter prevents temporal aliasing and keeps the photodetector bandwidth at 300 Hz. The electronic die also includes a bias circuit, a 10-bit ADC, and a logic circuit implementing a serial interface for signal read-out. The self-contained elementary eye has a total surface of 0.9 mm2, is 860 µm high, consumes 444 µA and weighs less than 2 mg.
The optical and electronic components of the elementary eye are separately fabricated in large planar arrays, precisely aligned on top of each other so that each microlens focuses light only onto its corresponding photodetector triplet, individually cut and attached to a printed circuit board (PCB), following the same manufacturing procedure used in a previously described artificial compound eye [9,21]. Every eye in the array has individual bond pads wired to the PCB for independent analogue and digital voltage feed, as well as synchronization, clock and data lines for serial read-out (electronic supplementary material, figure S1a). The PCB, wires and encapsulation material add 54 mg to each artificial elementary eye, and could be further reduced to fit the requirements of specific applications.
3. Sensitivity to light incidence angle and intensity
To characterize the optics of the artificial elementary eye, we measured the angular sensitivity function (ASF) of each of the three photodetectors of a single prototype (figure 2a; see also Characterization methods in electronic supplementary material), which yield Gaussian profiles. We then measured the acceptance angles and inter-detector angles from the measured ASFs. The acceptance angle Δρ is defined by the full width at half maximum of the ASF. The inter-detector angle Δφ is defined by the angular position of the peak of two adjacent ASFs (figure 2a). The mean acceptance angle Δρ was 4.78° ± 0.22° and the mean inter-detector angle Δφ was 5.52° ± 0.02°. The obtained 1.15 : 1 ratio lies in the range found in most diurnal flying insects, providing the elementary eye with adequate optical characteristics for optic flow extraction [17].
Figure 2. (a) Measured angular sensitivity functions (ASF) of the three pairs of photodetectors of the artificial elementary eye. Each colour corresponds to the signal of one of the photodetectors. (b) Auto-adaptation of the artificial elementary eye photodetector to ambient light. Measured steady-state (red dots) and transient responses (green dots) of the adaptive analogue VLSI photodetectors [20]. Each of the six dynamic operating curves shows the V(log I) response to step increments and decrements of irradiance over about six decades.
Figure 2b shows the mean steady-state and transient responses of the elementary eye photodetectors to illuminance step increments and decrements presented at six different steady levels and covering nearly six decades of light irradiance. At five of these six levels (red circles in figure 2b), the output response of the individual photodetector to light steps yields an S-shaped operating curve in a semi-log plot. The five S-shaped curves yield a similar slope, which demonstrates an analogous dynamic sensitivity at the five decades to sudden changes of illumination of about 1100 mV per decade in the linear part. Similar S-shape curves shifting for different background illuminations were also observed in biological photoreceptors [22,23], preventing saturation by bright spots in the visual scene and enabling adaptation to sudden changes of illumination. The curve at the lowest irradiance (10−8 W cm−2) was obtained by means of an additional neutral density filter with an attenuating factor of 10−4 (see Characterization methods in electronic supplementary material) yielding a response at only 0.1 lx. The different shape of this last curve is probably because of the influence of the photodetector dark current at this luminosity level.
4. Optic flow extraction
The artificial elementary eye has been designed following optical parameters of fruit fly ommatidia that enable it to extract the optic flow. We characterize this capability by making it rotate in the middle of an arena surrounded by a circular wall covered with random pattern images (figure 3a). Theoretically, the optic flow under such pure rotational motion must be directly proportional to the rotational speed [24]. In the experimental set-up, the elementary eye is embedded in a robotic device with the photodetector triplet plane perpendicular to the ground and parallel to the arena walls (figure 3a). The sensor can be fixed at four different angles α around its roll axis with respect to the arena walls (figures 3a and 4a) so that the optic flow extraction in four different directions at the triplet plane can be assessed. In the experiment, the robot performs yaw rotations for every eye angle position α at several constant speeds ω while the visual signals are registered in a memory card. Frames containing the three values of the triplet signals were recorded at a rate of 350 frames s–1 during the experiments. The optic flow is calculated out of the recorded frames offline in an external computer using Matlab (Mathworks, MA, USA).
Figure 3. (a) Picture of the artificial elementary eye set-up mounted on an e-puck robot for an optic flow characterization experiment. The elementary eye is rotated at a roll angle of 30°. (b) Graph displaying the output signal of the three photodetectors of the artificial elementary eye versus the frame number during a characterization experiment at 350 frames s−1. Δfij shows the frame delay between photodetector signals, which is used to calculate the optic flow. The inset displays the orientation of the three photodetectors roll-rotated 30° during the experiment shown in the graph as well as the unit distance vectors rij for every pair of photodetectors. Figure 4. (a) Schematic display of the optic flow characterization set-up, showing the photodetector triplet and optic flow vector OF measured during yaw rotational motion at angular velocity ω for different α roll angles. (b) Optic flow vector norm as a function of yaw rotational speed ω at four α roll angles. Each data point indicates mean and standard deviation of the OF vector norm computed from signals collected during 30 s at a frame rate of 350 frames s−1. The dashed line shows the theoretical equivalence between OF norm and rotational speed. (c) Optic flow vector angle with respect to α = 0°as a function of yaw rotational speed ω at four roll angles α. Each data point indicates mean and standard deviation of OF vector angle values computed from signals collected during 30 s at a frame rate of 350 frames s−1. The dashed line shows the nominal sensor roll angle α for positive rotational speeds and α + 180°for negative ones.

In this work, an ad hoc method is used to calculate the optic flow. For this, the artificial elementary eye is broken down into three unidirectional motion sensors formed by the three pairs of photodetectors, which bear a fixed inter-detector angle Δφ between their viewing directions (figure 2a). It is considered that a photodetector unit i of the pair will sample scene sections during motion with a time difference Δtij with respect to the contiguous paired photodetector j (figure 3b). At an inter-detector angle Δφ, the signal time delay Δtij at every pair is inversely proportional to the optic flow
projected onto the vector
which connects the photodetector pair (figure 3b, inset)

The OFx and OFy optic flow vector components, which are the final outcomes sought, can be expressed using the X and Y components of the pair connector
and
as

If we apply this expression to the three pairs of photodetectors, we obtain a system of three equations and two unknowns, OFx and OFy:



From these, the optic flow vector can be calculated using the least-squares method:

Δtij can be obtained from the frame difference Δfij (figure 3b) using the sensor frame rate in the expression:

In the experimental visual data, we search Δfij that minimizes the mean squared error between the photodetector signals of the ij pair in data windows of 350 points, that is, one experimental second, for the whole 30 s of the experiment. The optic flow values identified in the 30-s period are collected, and the mean and standard deviations for the optic flow vector module and for the angle are plotted against the constant rotational speed of the robot (figure 4b,c).
As predicted by theory [24], the norm of the measured optic flow matches the yaw rotational speed, regardless of the roll angle at which the elementary eye is fixed (figure 4b), and the optic flow angle matches well the roll angle in the four studied cases (figure 4c). The increasing standard deviation of the OF angles at lower rotational speeds are likely because of the light adaptation mechanism of the analogue VLSI circuit [20,25], which yields an exponential decay of the signal over time in the absence of visual contrast. At low rotational speeds, there is a higher probability of spending longer periods pointing at featureless areas of the visual scene, thus yielding small errors in the extraction of optic flow angle.
The capability of the artificial elementary eye to extract vectors of optic flow is demonstrated in an environment displaying natural-like features and of about 400 lx of illuminance, which corresponds approximately to the illumination of a standard office. Nevertheless, the light sensitivity of the elementary eye is practically similar along five decades of irradiance owing to the adaptability provided by the implemented Delbruck circuit (figure 2b). Thus, it can certainly be claimed that the sensor is fully functional to extract optic flow at those illumination conditions, too. However, further work is needed to test alternative conditions, e.g. an environment where features yield lower contrast. Furthermore, the processing method used requires a computational power that is inadequate to be used in small-size embedded real-time systems. Future work will investigate algorithms that can be implemented in small ASICs beside the elementary eye and compute motion for real-time tasks.
5. A flexible array of artificial elementary eyes
Many functionally independent artificial elementary eyes could be arranged in morphologically different configurations to span large fields of view. For example, here we show a linear configuration of eyes for motion extraction over large fields of view on the horizontal plane. The device consists of a flexible PCB with two linear arrays of 34 artificial eyes and a microcontroller (Microchip® dsPIC33FJ128GP802, Chandler, AZ, USA) to collect and process signals (figure 5a). In this case, the fabrication procedure is identical to that described above, with the exception that cutting of the individual elementary eyes does not impact the flexible PCB, which includes the electric lines to carry the signals to the microcontroller. The flexible array can undergo reversible mechanical bending down to a radius of curvature of approximately 6 mm. Mechanical flexibility allows the sensor array to span a variable field of view, whose range is limited by the mechanical properties of the PCB.
Figure 5. (a) A flexible array of 68 artificial elementary eyes and microcontroller for signal extraction. The inset shows a microscopy image of three artificial elementary eyes of the flexible array. (b) Top view of the sensor array attached to the characterization set-up robot. The red lines indicate the elementary eye whose signals were recorded during the experiment. (c) Mean optic flow vectors resulted from a straight and centred trajectory of the robot along a 0.9 m long and 0.2 m wide straight corridor (electronic supplementary material, figure S2) at constant speed of 8 cm s−1.
The eye array was attached to the front side of a cylindrical wheeled robot, yielding a field of view of 135° along the horizontal direction and of 5.2° along the vertical direction (figure 5b). Visual signals were acquired from a subset of 11 elementary eyes uniformly distributed along the sensor while the robot moved on a straight trajectory at 8 cm s−1 in the centre of a corridor with textured walls (electronic supplementary material figure S2). Optic flow measured offline from the selected eyes sampled at 130 frames s−1 (figure 5c) shows that optic flow vectors from eyes closer to the extremities of the array display an angular orientation parallel to the ground and larger intensities, as predicted by optic flow patterns in translation [24].
6. Discussion
Despite its microscopic size and narrow field of view, the artificial elementary eye described here can extract local optic flow vectors at high speed, different lighting conditions and at low power. Further miniaturization of the eye could be possible. For example, the sensor footprint could be reduced by rearranging the layout of the electronics, which, in the prototype presented here, has been optimized for a linear layout of multiple eyes. Also, the area of the photodetectors could be reduced, which would also result in a lower eye height in order to keep the match between the projected light area and that of the photodetector footprint, albeit at the cost of lower sensitivity in dark environments. Finally, high-end CMOS fabrication processes could result in further size reduction of the embedded microelectronic circuitry and components, and the relatively large wire bond pads could be replaced by through-silicon-vias or flip-chip bonding.
The visual capabilities of the artificial elementary eye could be used for autonomous flight of microrobots [6], provide visual odometry to medical endoscopes, estimate incoming collisions in wearable alert systems, and many other applications that require fast motion detection in a small and lightweight package [26]. Artificial elementary eyes could also be embedded on the surface of soft and flexible robots with minimal impact on their mechanical properties, or arranged in flexible layers that can be adapted to arbitrary shapes, as shown with the linear array presented here, or embedded in smart clothing. Finally, artificial elementary eyes could also be aggregated on curved rigid surfaces to form artificial compound eyes with variable spatial density as required by the environment and motion of the agent, mimicking the diversity and evolution of biological compound eyes.
Authors' contributions
R.P.C., S.V., R.L., H.A.M. and D.F. conceived, designed and supervised the project. R.P.C. and R.J. set up and performed experiments. M.K.D. and R.L. built prototypes. R.P.C., S.V., H.A.M. and D.F. analysed data. R.P.C. and D.F. wrote the paper with assistance from S.V., R.L. and H.A.M.
Competing interest
We declare we have no competing interests.
Funding
We acknowledge financial support of European project FET-Open grant no. 237940 and of the Swiss National Centre of Competence in Research Robotics (NCCR).
Acknowledgements
We thank Wolfgang Buß, Felix Kraze and Julien Diperi for the realization of various assembly tasks; Marc Boyron, Grégoire Heitz and Géraud L'Eplattenier for electronic design and fabrication and Bryan Schubert for proofreading.
Footnotes
References
- 1
- 2
Warrant EJ, Kelber A, Wallén R, Wcislo WT . 2006Ocellar optics in nocturnal and diurnal bees and wasps. Arthropod. Struct. Dev. 35, 293–305. (doi:10.1016/j.asd.2006.08.012) Crossref, PubMed, Google Scholar - 3
Parsons MM, Krapp HG, Laughlin SB . 2006A motion-sensitive neurone responds to signals from the two visual systems of the blowfly, the compound eyes and ocelli. J. Exp. Biol. 209, 4464–4474. (doi:10.1242/jeb.02560) Crossref, PubMed, ISI, Google Scholar - 4
Schuppe H, Hengstenberg R . 1993Optical properties of the ocelli of Calliphora erythrocephala and their role in the dorsal light response. J. Comp. Physiol. A 173, 143–149. (doi:10.1007/bf00192973) Crossref, Google Scholar - 5
Berry RP, Stange G, Warrant EJ . 2007Form vision in the insect dorsal ocelli: an anatomical and optical analysis of the dragonfly median ocellus. Vision Res. 47, 1394–1409. (doi:10.1016/j.visres.2007.01.019) Crossref, PubMed, Google Scholar - 6
Ma KY, Chirarattananon P, Fuller SB, Wood RJ . 2013Controlled flight of a biologically inspired, insect-scale robot. Science 340, 603–607. (doi:10.1126/science.1231806) Crossref, PubMed, ISI, Google Scholar - 7
Gremillion G, Humbert JS, Krapp H . 2014Bio-inspired modeling and implementation of the ocelli visual system of flying insects. Biol. Cybern. 105, 223–239. (doi:10.1007/s00422-014-0610-x) Google Scholar - 8
Expert F, Ruffier F . 2015Flying over uneven moving terrain based on optic-flow cues without any need for reference frames or accelerometers. Bioinspiration Biomimetics 10, 026003. (doi:10.1088/1748-3182/10/2/026003) Crossref, PubMed, Google Scholar - 9
Floreano D 2013Miniature curved artificial compound eyes. Proc. Natl Acad. Sci. USA 110, 9267–9272. (doi:10.1073/pnas.1219068110) Crossref, PubMed, ISI, Google Scholar - 10
Song YM 2013Digital cameras with designs inspired by the arthropod eye. Nature 497, 95–99. (doi:10.1038/nature12083) Crossref, PubMed, ISI, Google Scholar - 11
Fuller SB, Karpelson M, Censi A, Ma KY, Wood RJ . 2014Controlling free flight of a robotic fly using an onboard vision sensor inspired by insect ocelli. J. R. Soc. Interface 11, 20140281. (doi:10.1098/rsif.2014.0281) Link, ISI, Google Scholar - 12
Duhamel PEJ, Perez-Arancibia NO, Barrows GL, Wood RJ . 2013Biologically inspired optical-flow sensing for altitude control of flapping-wing microrobots. IEEE-ASME Trans. Mechatron. 18, 556–568. (doi:10.1109/tmech.2012.2225635) Crossref, Google Scholar - 13
Reichardt W . 1961Autocorrelation, a principle for the evaluation of sensory information by the central nervous system. In Sensory communication (ed.Rosenblith WA ), pp. 303–317. New York, NY: Wiley. Google Scholar - 14
Franceschini N, Riehle A, Le Nestour A . 1989Directionally selective motion detection by insect neurons. In Facets of vision (edsStavenga DG, Hardie RC ), pp. 360–390. Berlin, Germany: Springer. Crossref, Google Scholar - 15
van de Grind WA, Koenderink JJ, van Doorn AJ . 1986The distribution of human motion detector properties in the monocular visual field. Vision Res. 26, 797–810. (doi:10.1016/0042-6989(86)90095-7) Crossref, PubMed, ISI, Google Scholar - 16
Maisak MS 2013A directional tuning map of Drosophila elementary motion detectors. Nature 500, 212–216. (doi:10.1038/nature12320) Crossref, PubMed, Google Scholar - 17
Land MF . 1997Visual acuity in insects. Annu. Rev. Entomol. 42, 147–177. (doi:10.1146/annurev.ento.42.1.147) Crossref, PubMed, ISI, Google Scholar - 18
Götz KG . 1965Die optischen Übertragungseigenschaften der Komplexaugen von Drosophila. Biol. Cybern. 2, 215–221. (doi:10.1007/bf00306417) Google Scholar - 19
Franceschini N, Kirschfeld K . 1971Les phénomènes de pseudopupille dans l'œil composé de Drosophila. Kybernetik 9, 159–182. (doi:10.1007/bf02215177) Crossref, PubMed, Google Scholar - 20
Delbruck T, Mead CA . 1994Adaptive photoreceptor with wide dynamic range. InIEEE Int. Symp. on Circuits and Systems ISCAS '94 ,London, UK , vol. 334, pp. 339–342. Crossref, Google Scholar - 21
Viollet S 2014Hardware architecture and cutting-edge assembly process of a tiny curved compound eye. Sensors 14, 21 702–21 721. (doi:10.3390/s141121702) Crossref, Google Scholar - 22
Matić T, Laughlin SB . 1981Changes in the intensity-response function of an insect's photoreceptors due to light adaptation. J. Comp. Physiol. 145, 169–177. (doi:10.1007/bf00605031) Crossref, Google Scholar - 23
Normann RA, Perlman I . 1979The effects of background illumination on the photoresponses of red and green cones. J. Physiol. 286, 491–507. (doi:10.1113/jphysiol.1979.sp012633) Crossref, PubMed, Google Scholar - 24
Koenderink JJ, Doorn AJ . 1987Facts on optic flow. Biol. Cybern. 56, 247–254. (doi:10.1007/bf00365219) Crossref, PubMed, ISI, Google Scholar - 25
Expert F, Viollet S, Ruffier F . 2011Outdoor field performances of insect-based visual motion sensors. J. Field Rob. 28, 529–541. (doi:10.1002/rob.20398) Crossref, Google Scholar - 26
Baird E, Boeddeker N, Ibbotson MR, Srinivasan MV . 2013A universal strategy for visually guided landing. Proc. Natl Acad. Sci. USA 110, 18 686–18 691. (doi:10.1073/pnas.1314311110) Crossref, ISI, Google Scholar


