Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood ‘super-resolution’ method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue ‘Super-resolution structured illumination microscopy (part 2)’.

JDM, 0000-0001-9260-3156 Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work.
This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.
2022 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/ by/4.0/, which permits unrestricted use, provided the original author and source are credited.

Introduction
Structured illumination microscopy (SIM) is a method in fluorescence microscopy which is capable of providing resolutions better than those that can be achieved using uniform illumination in epifluorescence wide-field microscopy. While first demonstrated, at the turn of the millennium, to enhance only lateral resolution [1][2][3][4], subsequent developments allow for the enhancement of axial resolution as well [5,6]. SIM has become a popular technique, particularly for live cell imaging [7][8][9], when better resolution is required as, unlike superresolution methods such as single molecule localization microscopy (SMLM) [10][11][12][13][14] and stimulated emission depletion microscopy (STED) [15][16][17], it does not require high laser powers nor special fluorophores. However, as will be discussed further later, in its normal form it only provides a roughly twofold enhancement of resolution.
It should be noted that, in 1997, an alternative form of structured illumination microscopy was presented by Neil et al. [18]. Here the goal was not to improve resolution, but to add optical sectioning to wide-field microscopy. This form of structured illumination has been commercially successful, with further developments of the technology available from Aurox (Clarity), Andor (Revolution DSD2) and Zeiss (Apotome). For the purposes of this article, SIM will be taken to correspond only to those methods designed to improve resolution (often referred to as SR-SIM, as opposed to the OS-SIM of Neil et al.).
In this article, we aim to provide a general overview of SIM and answer the 13 questions recently proposed by Prakash et al. [19] ( § §1-13), along with three more of our own invention ( § §14- 16). A similar attempt at answering the 13 questions of Prakash et al. has recently been presented by Heintzmann [20]. For some questions, the answers given by Heintzmann are significantly different from those presented here: where this is the case, this will be highlighted in later sections. In §2, we consider the so-called 'diffraction limit' and show that SIM as described here does not break it, even though it provides resolutions beyond those achievable with conventional wide-field imaging. Following this, in §3, we consider whether an impressive demonstration of SIM combined with total internal reflection fluorescence (TIRF) microscopy, which achieves a resolution of 84 nm, breaks the diffraction limit. We then look at the conditions required for SIM to break the diffraction limit in § §4 and 5.
In the second part of the manuscript, we show that, despite their prevalence, Fourier methods are not necessarily required to reconstruct images ( §6) and study the challenges of using SIM in thick samples ( § §7 and 8). We then argue that image scanning microscopy (ISM), while experimentally a different method, is a form of SIM and show how arbitrary illumination patterns can be used ( §9). Following this, we study the effects of illumination sparsity in §10. We note that SIM cannot be used to improve the resolution of transmission microscopy ( §11) and propose a careful approach to the use of machine learning with super-resolution ( §12).
In the last part of the manuscript, we discuss the practicalities of building a SIM system ( §13) and consider how to combine SIM with SMLM ( §14). We then investigate how the axial resolution can be improved beyond a twofold limit without a nonlinear response ( §15), how to combine SIM with light sheet microscopy ( §16) and how quickly SIM images can be acquired ( §17). Finally, we use a simple bibliometric analysis to track interest in SIM since its inception and conclude with an outlook on future developments ( §18). distinct in naked-eye astronomical observations [21,22]. These criteria are, however, heuristic and have no fundamental physical basis. Ernst Abbe was the first to propose a physically motivated limit to the resolution of a light microscope by studying diffraction from a grating [23]. He considered using a microscope to image a grating using transmitted light and noted that, in order for the image to have any periodicity, at least two of the diffracted orders from the grating must be captured by the objective. For head-on illumination, i.e. normal to the grating, he noted that three orders (the −1, 0 and +1 orders) were captured, giving a minimum grating periodicity of: where λ 0 is the vacuum wavelength, n the refractive index of the sample and α the semiaperture angle (i.e. the maximum angle from the objective's optical axis from which orders can be collected). For convenience, the product n sin α is known as the numerical aperture (NA). Referring to figure 1a (top row), we see that these three orders are located symmetrically about the centre of the back focal plane. By tilting the illumination, we can shift the location of these orders laterally. Given that we only need two orders, not three, for a grating-like structure to be seen, we can imagine that by tilting the illumination such that the zeroth order is right at the edge of the back focal plane, we could increase the order spacing by a factor of two and hence half the grating period (figure 1a, bottom row). This leads to the Abbe equation for oblique illumination: What about fluorescence imaging, where light from the sample is emitted in all directions? Abbe never studied the case of fluorescence imaging, but it turns out that the right-hand side of Abbe's equation for oblique illumination is correct for this case as well: . More specifically, this is the lateral resolution, i.e. that within the focal plane. Perpendicular to the focal plane, the axial resolution is given by . (2.4) Why are these equations correct? To explain this, we will follow an argument similar to that of Gustafsson [24], which in turn builds on arguments by McCutchen [25] and Ewald [26]. First, we must note that we will use the scalar approximation of the electric field, although the argument can be generalized by considering each component of the vectorial electric field separately. We will also work in the far-field, as near-field components decay exponentially quickly and will not be detected by a normal microscope. In addition, fluorescence is an incoherent process, with emission from a source bearing no phase relationship to the stimulating light or emission from another source (we ignore the small probability of stimulated emission). As such, we can limit our consideration to just one source, as the overall response from the sample is the incoherent sum of the responses from all the sources present.
Consider the amplitude of the electric field, E(r), produced by a source radiating monochromatically. The source produces a spherical wave which, in the far field, can be decomposed, via a Fourier transform, into a sum of plane waves with different directions. All these plane waves will have the same wavelength, λ, and so all the wavevectors, k, will have the same length. Hence, as there are no preferred directions and all wavevectors have the same length, the Fourier transform of E(r),Ẽ(k) (we will use the notation that the Fourier transform of X(r) is written asX(k) throughout), will have the form of a spherical shell, centered on the origin and with radius 1/λ (figure 1b). This detected far field is very much not the same as the original emitted field, as the evanescent components have been lost. The consequences of this can most clearly be seen by noting that a true spherical wave would decay smoothly with increasing distance from the origin, while this field has local minima and maxima all over space.
However, our image is not given by the amplitude of the electric field, but the intensity. This can be found by multiplying the amplitude by its complex conjugate, giving I = EE * . This means that in the Fourier domain, we haveĨ (k) =Ẽ(k) * Ẽ * (−k), (2.5) where * denotes convolution. Hence,Ĩ(k) is the autocorrelation ofẼ(k) and so will have the form of a ball with twice the radius of the sphere ofẼ(k), 2/λ (figure 1c). This is interesting, as it shows that there is a limit to the information carried by the far field, irrespective of the objective lens used: only spatial frequencies up to k c = 2/λ are present. Converting back to real space, this critical spatial frequency corresponds to a spatial scale of λ/2. If we consider that our source may be located in a medium of refractive index n, then we can convert λ to a vacuum wavelength, λ 0 = nλ, and so obtain the far-field limit: Now, let us look at what happens if we use an objective lens to capture the light from our source. The objective can only capture wavevectors over a certain angular range, meaning that the Fourier domain electric field amplitude distribution has the form of a spherical cap, rather than a full sphere (figure 1d). As such, the autocorrelation of this has the form of a toroidal solid, shown in cross-section in figure 1e.
We can identify that the maximum lateral spatial frequency is given by k l = 2n sin α/λ 0 , while the maximum axial spatial frequency is given by k a = n(1 − cos α)/λ 0 , where α is the semiaperture angle. Converting to real space, this gives: , (2.8) as was quoted previously.
Considering not just a single source, but a distribution of sources emitting light given by a distribution E(r), the toroidal solid formed by autocorrelating the spherical cap defines a transfer function,H(k). Given this, the image spectrum,D(k), of the emission spectrumẼ(k) is given bỹ where × denotes point-wise multiplication. In real space where H(r) is known as the point spread function. Equation (2.9) makes clear that the image spectrum only contains spatial frequencies for which H(k) is non-zero. As the form ofH(k) is based only on the phenomenon of far-field diffraction, this defines a true diffraction limit based on the spatial frequency bandwidth ofH(k). Now we are in a position to consider the mechanism of structured illumination and see how it circumvents this limit. For simplicity, we will restrict our consideration to one dimension and replace r with x.
The fundamental basis of SIM is the realization that, in our image formation equation, equation (2.10), we are not truly interested in the emission distribution function, E(x), but instead the distribution of sources S(x). These distributions are related by E(x) = S(x) × I(x), where I(x) is the illumination distribution. In the case of uniform illumination, I merely acts as a scalar multiplier of S(x). However, for non-uniform illumination we have an updated image formation equation: This suggests that, while the maximum spatial frequency that can be recorded is limited by the support ofH(k), a careful choice ofĨ(k) may let us shift the sample spectrum,S(k), around via the convolution such that other spatial frequencies are transmitted. SIM, in its simplest variant, achieves this by using illumination with two coherent beams, providing a pattern of the form I(x) = 1 + cos(px + φ), (2.13) or, in the Fourier domain,Ĩ where δ denotes the Dirac delta function. Hence, for a given phase, φ m , we havẽ This shows that we have the usual wide-field component,S(k)H(k), overlaid with two shifted copies of the sample spectrum. This alone is insufficient to 'see'S(k ± p), but by recording multiple images with different phases of the illumination pattern, these components can be linearly separated and used to reconstruct an image with improved resolution. As there are three components to separate, at least three phase shifts are required. For the purposes of this article, we shall not discuss the details of the image reconstruction routine, nor how SIM can be used to enhance the resolution in all directions (i.e. both laterally and axially). Instead, the interested reader is directed to the excellent article by Gustafsson et al., which provides detail on both these issues [5]. Further information on the individual steps of reconstruction is presented by Lahrberg et al. [27] (pattern estimation), Wicker [ reconstruction routine, as well as open-source software, is presented by Lal et al. [30]. ImageJcompatible software for checking the quality of raw and reconstructed data are provided by SIMcheck [31] and for reconstruction by fairSIM [32].
It is clear from equation (2.15) that higher resolutions are achieved with finer illumination periods. If we illuminate the sample with the same objective lens that we use to capture fluorescence, then the finest illumination we can create is Combining this with the emission diffraction limit, we see that the total resolution is given by and so we have where β = λ emission /λ illumination . Hence, for the case where λ emission = λ illumination , we see that SIM provides double the resolution of uniform (epifluorescence) illumination. While this is beyond the resolution of diffraction-limited imaging with uniform illumination, the resolution is still limited, in some sense, by diffraction. In order to not be diffraction-limited, we would need to be able to create patterns with arbitrarily fine periods. Extensions of SIM that provide this will be discussed in §4. These methods, theoretically, create images that do not have a spatial band-limit. To distinguish SIM, and other techniques, that provide improved resolution but are limited by diffraction in both illumination and emission, rather than just in emission, the term 'extended resolution' has been proposed [33][34][35][36].
This allows us to classify methods into four distinct groups based on their theoretical resolving power: (i) Standard resolution-as achieved by a normal wide-field epifluorescence microscope. (ii) Extended resolution-beyond standard resolution but still limited by the combination of the illumination and emission diffraction limits, as achieved by SIM and ISM (also referred to by some authors as restricted super-resolution). (iii) Enhanced resolution-beyond extended resolution, including methods with a finite resolution limit, such as photoactivated nonlinear SIM. (iv) Super resolution-the subset of enhanced resolution methods not limited by diffraction and theoretically capable of infinite resolution, discussed further in §4, such as stimulated emission depletion microscopy and saturated SIM.
Note that some authors, including Heintzmann in his answers to these questions [20], would classify all methods capable of imaging beyond the extended resolution limit as super-resolution methods, but this is not a sufficient condition for 'breaking' the diffraction barrier [37]. Furthermore, it is worth highlighting the difference between 'imaging at the diffraction limit', i.e. with standard resolution, and 'diffraction-limited resolution', which may be at any of the levels apart from diffraction-unlimited super resolution.
3. Should high-NA TIRF-SIM, which can achieve lateral resolutions down to 84 nm, be considered as diffraction limited?
Total internal reflection fluorescence (TIRF) microscopy exploits the nature of total internal reflection to produce a short-range, evanescent illuminating field tethered to a refractive index boundary. Given two materials with different refractive indices, n 1 > n 2 , Snell's Law states that light incident on the boundary, from the n 1 side, at angles equal to or greater than arcsin (n 2 /n 1 ) cannot be transmitted and is instead reflected. However, Maxwell's equations demand continuity at the boundary and so a rapidly decaying evanescent field is created on the n 2 side. For light incident at an angle α, the exponential decay length of the evanescent field is given by and is typically of the order of 100 nm. While this ensures that only a thin section of the sample is illuminated, it does not enhance axial resolution as such as the illumination cannot be scanned over the sample. TIRF-SIM uses two such high-angle beams to (i) reduce background fluorescence, (ii) provide finer patterns than can be created directly in the n 2 medium [38].
When imaging a watery sample, for which n = 1.33, the maximum possible numerical aperture of a water-immersion objective is n (i.e. sin α = 1), while oil immersion objectives with numerical apertures of 1.4, 1.49, 1.5 and even 1.7 are available. These objectives can form a pattern right at the coverslip boundary at the limit of their numerical apertures and have the near-field be patterned with the same spatial period, even though there is no way of creating such a fine pattern in water using far-field illumination.
Li et al. used such a 1.7-NA objective with between 30 and 100 W cm −2 of 488 nm illumination at 1.58 NA and approximately 510 nm detection and claim a resolution of 84 nm [36]. Using a modified version of equation (2.17), where we allow the numerical apertures of illumination and detection to be different, we can back-calculate the effective detection NA for their claim to be 1.38, consistent with the high end of the range of reported cytosolic refractive indices [39]. The corresponding non-SIM TIRF resolution for this case would then be 2.2× worse, at 185 nm, while the maximum possible far-field SIM resolution would be 90 nm. Nevertheless, this 84 nm is still diffraction-limited, as the period of the pattern created at the boundary is still limited by diffraction on the n 1 side. It is interesting to note that the authors rejected the possibility of collecting super-critical fluorescence (i.e. near-field emission that is converted to propagating light by interacting with the boundary) to further enhance their resolution calculation (i.e. the detection NA is limited to the refractive index of the sample), despite other measurements with the same objective showing that significant super-critical light can be present [40].
It is particularly instructive to compare this answer (yes) with the companion given by Heintzmann: 'a clear "no"' [20]. For Heintzmann, taking equation (2.17), all forms of SIM break the emission Abbe limit and hence break the diffraction limit. This equation can be used to define a new limit, that of 'restricted super resolution', introduced by Sheppard [41]. For TIRF-SIM, the structured illumination pattern has a finer frequency than that permitted by the refractive index of the sample, and so Heintzmann says that TIRF-SIM breaks this 'restricted super resolution' criterion. Hence, the suggestion is that TIRF-SIM is not diffraction-limited even under the considerations of this more strict approach.
However, the fact remains that the fineness of the illumination pattern is limited by diffraction, just in a medium of refractive index n 1 rather than n 2 . Even in the situation where the sample was mounted on a diamond coverslip (ignoring the deleterious effects of the strong dispersion in such a material) and we had an objective lens with numerical aperture matching that of the refractive index of the material, 2.4, TIRF-SIM of the same sample as Li et al. used could not produce a resolution better than

Can nonlinear SIM become broadly applicable and live-cell compatible?
Nonlinear SIM breaks the assumption that the fluorescent response from the sample is linearly related to the intensity of the illuminating light (E(x) = S(x) × I(x)). The nonlinearity adds extra harmonics (i.e. spatial frequency components above that of the linear case) to the effective illumination pattern, further extending the resolution. A patent application (DE19908883A1) describing this idea was submitted by Heintzmann and Cremer in March 1999, barely two months after the publication of the first SIM paper. The simplest case to consider is a normal SIM system in which a further sinusoidal pattern, R(x) = 1 + cos(px + φ), is used to control the fluorescence response, e.g. by photoswitching a fraction of the molecules between dark and light states. Now, the camera data acquired have the form: and in the Fourier domain:D Assuming that I(x) and R(x) have the same period, p, we now have a total of five delta functions, at k ∈ {−2p, −p, 0, p, 2p} (see figure 2b middle row). By phase-shifting both I(x) and R(x) simultaneously, we can unmix these five components with only five images, producing a reconstruction with sample information at three times the conventional limit.
Despite the further resolution increase, even this approach is diffraction-limited, as I(x) and R(x) are both individually diffraction-limited. Hence, the spatial frequencies available through this method are limited to a certain band. In effect, all we have done is taken the normal SIM equation (equation (2.11)) and replaced I with I eff = I 2 . We could imagine hypothetically extending the scheme even further, with yet another sinusoidal pattern controlling the response of the sample to the light which controls the fluorescence response, but this would correspond to I eff = I 3 and so, while providing higher resolution (to four times the usual limit), would still be diffraction-limited. Hence, while these methods would provide enhanced resolution beyond extended resolution, they do not belong to the class of diffraction-unlimited super-resolution methods.
In order to achieve true super-resolution imaging with SIM, we need not just a nonlinear response, but a non-polynomial one. This is because a polynomial response can be considered as a (scaled, repeated) convolution of the illumination spectrum with itself and so, as we start with a finite number of harmonics, we must end with a larger, but still finite, number. By contrast, a non-polynomial response would provide an infinite number of harmonics and hence cannot be said to be limited by diffraction (see figure 2a,b bottom rows).
To recap, a uniform illumination pattern provides standard resolution at the diffraction limit. Patterned illumination with a linear fluorescence response to illumination light provides extended resolution up to twice the diffraction limit (assuming excitation and emission light have the same wavelength). Replacing the linear response with a nonlinear, polynomial response provides further enhanced resolution, albeit one that is still bandlimited. Finally, implementing a nonpolynomial response allows for diffraction-unlimited super-resolution imaging.
The first non-polynomial SIM method to be proposed and demonstrated involves saturating the fluorescence, such that a linear increase in illumination intensity results in a sub-linear increase in fluorescence [42] (compare the top and bottom rows of figure 2a). Saturation occurs when a significant fraction of the illuminated molecules are in the excited state when further excitation photons are incident, as there are insufficient ground-state molecules that can be excited by the photon flux. So far, the only demonstration of saturated SIM was provided by Gustafsson in 2005 [43], which showed the ability to resolve a close-packed monolayer of 50 nm-diameter fluorescent beads. Here, pulsed illumination was used to provide an The pattern period and spatial frequency are denoted by x p and k p , respectively. Note that, in addition to containing an extra harmonic at k = ±2k p , the quadratic case has a stronger component at k = ±k p than in the linear case. Furthermore, while harmonics beyond the plot range are present in the saturated fluorescence case, their relative strengths compared to the DC peak at k = 0 are so weak that the harmonic at k = ±k p is already barely visible. The saturated fluorescence case was calculated assuming a mean excitation irradiance of one photon per cross-section per lifetime and continuous wave excitation.
where σ is the fluorescence excitation cross-section of the fluorophore used, k f = 1/τ (where τ is the fluorescence lifetime) and t p is the pulse duration. While, theoretically, an infinite number of harmonics were present, the signal-to-noise ratio limited the number of 'usable' harmonics to three, i.e. a fivefold resolution enhancement, although five harmonics were present in calibration data. This is a good illustration of the point that, while super-resolution methods may theoretically be capable of providing superior resolution to enhanced-resolution methods, in practice the distinction is often irrelevant. Given that the average illumination irradiance was only 35 mW cm −2 , one may wonder why this method has never been demonstrated on biological samples. This is because considering only the average irradiance is misleading, as the illumination was pulsed. Averaging instead over the pulse duration gives a peak irradiance of 10 MW cm −2 , comparable to the levels used for multiphoton microscopy. Photodamage mechanisms are known to be nonlinear with respect to light intensity, such that a doubling in intensity produces more than double the damage. Nevertheless, it is not clear that this irradiance is necessarily too damaging for imaging biological samples, especially as the saturation irradiance of EGFP has been measured to be similar, at 11 kW cm −2 [44], and multiphoton microscopy has successfully been used to image many varied biological samples. While multiphoton microscopy uses redder wavelengths of illumination, which is suggested to contribute less to photodamage, standard use of confocal fluorescence microscopy also often operates at illumination irradiances close to the saturation irradiances of similar fluorophores.
Another non-polynomial SIM approach, published in 2012 by Rego et al., involved saturated depletion of fluorescence, in which the photoswitchable nature of the fluorescent protein Dronpa was used to confine fluorescence to thin bands [45]. This followed a 2008 demonstration of the same mechanism by Hirvonen et al., although here the authors were unable to successfully reconstruct a high-resolution image [46]. The idea of saturating the depletion of fluorescence is one example of the RESOLFT concept [37], which was demonstrated in a parallelized, wide-field manner in 2007 [47]. In the work of Rego et al., images with resolutions better than 50 nm were produced despite the required irradiances being of the order of only 10 W cm −2 . Unfortunately, only fixed cells were imaged as one reconstructed frame required 945 s of imaging time.
In order to develop a live-cell compatible method, Li et al. abandoned the use of nonpolynomial SIM and instead focused on developing a technique that would work without the need for saturation [36]. To this end, they used the photoswitchable fluorescent protein Skylan-NS for patterned activation, rather than depletion. This enabled the timelapse imaging of live cells at resolutions better than the 84 nm of their high-NA linear SIM approach (see §3), producing a movie of 30 frames at 65 nm resolution. Furthermore, by increasing the activation irradiance to 490 W cm −2 , they began to saturate the transition, creating a non-polynomial method with an extra harmonic above the noise floor. This increased the resolution to 45 nm, but only provided 12 frames at useful signal-to-noise ratios.
Hence, both nonlinear and non-polynomial SIM have been demonstrated to be live-cell compatible. However, while linear SIM can image live samples for hundreds of time points, nonlinear approaches are currently limited to a few tens of time points. In addition, while nonpolynomial methods theoretically provide unlimited resolution, signal-to-noise concerns show that they are, in practice, less well-suited to extending resolution than carefully chosen nonlinear methods such as the patterned activation of Li et al. As an example of this, compare the 'quadratic' and 'saturated' plots in figure 2b. While the saturated case has more harmonics in total, the strengths of those at k p and 2k p are much stronger in the quadratic case. Given a finite photon budget, more photons must be spent to raise these harmonics above the spectrally flat noise floor using saturation. This problem is exacerbated at higher illumination powers as, while the strength of higher harmonics increase, the strength of the first harmonic decreases (see electronic supplementary material, video S1).
Numerical simulations suggest that saturation using continuous wave illumination, for which an illumination of I(x) is mapped to an effective excitation of I/(I + 1/τ ), produces harmonics that decay geometrically beyond the first harmonic, with the strength of the first harmonic decaying with increased saturation strength. For pulsed illumination, a similar pattern of harmonics is seen, with a short pulse length mimicking the effect of a higher intensity. For polynomial illumination response with N harmonics (i.e. 2N + 1 peaks in the spectrum) and perfect modulation contrast, the strength of the nth harmonic is given by where s = 1 would correspond to a harmonic as strong as the DC peak (n = 0). These strengths can be read off from every other row in Pascal's triangle (dividing by the central element), with uniform illumination corresponding to the first row.

Do you need 'switching' of states for nonlinear super-resolution imaging?
So far, all of our discussions of nonlinear SIM and super-resolution imaging have used continuous models. Despite this, we know that light is quantized into photons and fluorescent samples are quantized into fluorophores. As such, all the approaches discussed in §4 feature 'switching' of states, but this is also true of normal fluorescence imaging. Hence, switching is not a sufficient condition for nonlinear super-resolution microscopy. While switching of states can provide useful nonlinearities, it alone is not sufficient to encompass all possible nonlinearities (e.g. Rabi oscillations [48]), merely those that have been successfully demonstrated to work with SIM so far. Hence, switching is theoretically not a necessary condition for nonlinear super resolution, but has been used in all methods implemented so far. For more details on alternative super-resolution mechanisms, see Heintzmann's answer in reference [20].
6. Do high-quality SIM images require reconstruction in Fourier space?
In general, SIM reconstructions can be split into two separate steps: (i) estimating the parameters of the pattern of illumination used (period, orientation and phase) and (ii) using these parameters to separate and recombine information contained within the raw images. In many implementations, both of these steps have been conducted in the Fourier domain, although there is no inherent reason why they must be. Pattern estimation can potentially be skipped if the system is well calibrated and the sample does not distort the illumination pattern. If the pattern parameters are unknown, they are typically acquired via cross-correlating images with different phases. Assuming the pattern parameters are well known, either via calibration or through a pattern estimation routine that processes the raw data, a typical reconstruction would separate the different information components, shift them to their proper locations in Fourier space, and recombine them to form an image.
Separating the information components is typically thought of as solving the Fourier space matrix equation ⎡ ⎢ ⎣D where M, the mixing matrix, has been calculated for the case of equal 2π/3 phase steps with no global phase offset, for simplicity. However, this mixing matrix acts pixel-wise and so there is no reason to not consider the real space equivalent instead: This particular mixing matrix can be seen as a discrete Fourier transform along the phase direction, but this is not generally true and is not what is meant by a reconstruction in Fourier space (as this operates along the spatial directions material, video S2). It should, however, be noted that in many reconstructions a Fourier transform and inverse transform of a padded spectrum are used to artificially reduce the pixel size before shifting, as the raw data are not Nyquist-sampled for the higher resolution reconstruction. Staying in real space either requires oversampled raw data or an alternative upscaling technique.
Given shifted components, all that remains is to combine them to form an SIM image. This can be achieved with a direct sum, but most reconstructions choose to weight different parts of the separated components differently depending on their exact location in Fourier space, in an attempt to counteract the effects of the OTF.
As an alternative to this view of separating and recombining components, we can think of the process of structured illumination imaging as akin to amplitude modulation (AM), as used in early radios. Here, the carrier signal is our illumination pattern, the sample structure is our message signal and the recorded data is the product of these two, as is the AM signal. To demodulate an AM signal and recover the message signal, a radio can use a 'product detector', which heterodynes (i.e. multiplies) the AM signal with a signal from a local oscillator designed to match the carrier signal exactly. This suggests that a similar approach might be applicable to SIM data.
Consider a two-beam SIM system where, for simplicity, we will only consider one pattern orientation and perfect modulation contrast. In the usual way, the data acquired, D m (x), where m is used as a phase index, are equal to the product of the illumination and sample, convolved with the PSF, H(x): As usual, in the Fourier domain: Let us now consider multiplying our AM signals, D m (x), with a 'local oscillator', i.e. another sinusoidal pattern with the same phase and period: In the Fourier domain, our local oscillator takes the form: Now, let us heterodyne our signals: such that in the Fourier domain we have: Calculating this gives: Hence, the reconstructed image, obtained by multiplying each raw image with our local oscillator and summing all the images, has the same form as a traditional SIM reconstruction if we set α = β = 1. This form also makes clear why a SIM acquisition with only two phases does not reconstruct properly: in this case m e iφ m = 0 as desired, but m e iφ m = 2 and so we are left with terms of the formS(k ± 2p)H(k ± p) that we do not want. Figure 3 shows an example of this reconstruction process for simulated data. Here, the ground truth sample (figure 3a) was engineered to be periodic, to avoid 'cross' artefacts in the spectra ( figure 3d-f ). In addition, its form makes the Moiré effect of the fringe-like illumination particularly visible (figure 3h).
Considering a three-beam system, for which this approach provides and so we see that we should set α = β = γ = 1. Indeed, for n beams, the correctM m (k) is one which is spectrally flat, i.e. all delta functions have equal amplitude. This reconstruction approach will be revisited in §9 to consider the case of non-sinusoidal illumination. Similar real-space reconstruction approaches were also considered by Cragg & So early on in the development of SIM [4,49]. Intriguingly, the idea of using a patterned illumination and a local oscillator to enhance the resolution of an imaging system predates the idea of structured illumination microscopy by more than 30 years. In a series of papers, beginning in 1966, Lukosz investigated the possibility of enhancing the resolution of an imaging system by inserting opaque masks into object and image space [50][51][52]. While the details of the method are quite different to the modern idea of structured illumination microscopy, the underlying idea and the realization that an increase in spatial bandwidth must be accompanied by a decrease in temporal bandwidth were already present.

Can SIM be used for deep tissue imaging?
Using SIM for deep tissue imaging is complicated by two issues: (i) sample-induced aberrations and (ii) insufficient stripe contrast caused by out-of-focus and scattered light. We will restrict the discussion here to that of sample-induced aberrations and discuss (iii) in §8. In larger samples, it is common for the local refractive index to not be uniform over the entire sample. This means that light that arrives at or departs from a given location in the sample experiences a path-dependent phase shift with respect to light travelling in a uniform refractive index. These unanticipated phase shifts mean that the light is not properly focused by the microscope system. This is true for both illumination light and fluorescence emission.
Fortunately, it has been shown by Arigovindan et al. that aberrations predominantly affect the fluorescence emission light, as the effect of phase shifts on the illumination pattern can be taken into account in the reconstruction routine [53]. However, this assumes that the variation across the field-of-illumination is small (i.e. the 'isoplanatic patch' is large), otherwise the image must be split into blocks over which the pattern parameters are fixed. Correcting for aberrations in the fluorescence emission can be achieved using an 'adaptive optic', typically a deformable mirror, which applies a complementary phase shift to each part of the wavefront to cancel the aberrations. Usefully, this cannot only be used to compensate for sample-induced aberrations, but   any residual aberrations from the microscope itself. However, as the same correction is applied to every point in the field-of-view there is once again a requirement that the variation across the field is small. This is a limitation of all wide-field adaptive optics systems and is in contrast to pointscanning systems, for which the aberration correction can be updated for every scan position, if necessary [54]. Such adaptive optics SIM systems have been constructed by a number of groups, with most differences being in the way an appropriate wavefront correction is obtained. Using a multiphoton guide star allows for direct wavefront sensing, in which the wavefront is measured using a Shack-Hartmann sensor, but typically requires an expensive femtosecond laser which may cause excess sample damage [55,56]. By contrast, sensorless schemes assume that the wavefront can be decomposed into a sum of orthogonal modes, and simply see which combination of modes maximizes an image quality metric. Recently, Žurauskas et al. exploited the nature of the structured illumination itself to further improve the performance of such a sensorless scheme [57]. These approaches have the benefit of not requiring a guide star, but mean that a large number of images must be acquired to obtain a good correction (at best 2N + 1 for N modes) [54].
8. How can the fundamental limitation of SIM, i.e. generating sufficient stripe contrast in densely labelled and/or extended biological structures due to out-of-focus light, be addressed?
Even for situations in which the adaptive optics schemes discussed in §7 are not required, obtaining high-quality SIM reconstructions can be challenging in thicker samples due to the out-of-focus light contaminating the image and hence reducing stripe contrast. For single-cell imaging, the out-of-focus light produced by 2-beam or 3-beam SIM is normally insufficient to preclude high-quality reconstructions. However, for tissue imaging, background fluorescence from scattered light and out-of-focus regions of the sample can quickly degrade imaging performance.
In general, there are two approaches to solve this problem: (i) avoid producing out-of-focus light in the first place; (ii) somehow remove the out-of-focus light before it hits the detector. In conventional fluorescence microscopy, (i) is the approach taken by multiphoton and light sheet microscopy, while (ii) is that taken by confocal microscopy. All three approaches can usefully be combined with SIM.
In 2012, Andresen & Pollok et al. created a sinusoidal multiphoton SIM pattern by scanning an array of 32 beamlets perpendicular to the beamlet line, with phase-stepping and reconstruction in the usual manner [58]. However, as each beamlet is incoherent with the others, the fine fringe patterns necessary for a high-resolution enhancement would have too low a modulation contrast in this scheme, precluding a full doubling of resolution. Later, various authors combined multiphoton excitation with some form of image scanning microscopy (discussed further in §9), which uses a focused spot for excitation and a different reconstruction scheme [59]. This has also been combined with adaptive optics to improve the imaging performance at depth further still [60].
Initially, light sheet microscopy was first combined with incoherent OS-SIM to enhance optical sectioning, rather than resolution [61]. Soon after, Planchon et al. used an incoherent superpositions of Bessel beams to provide enhanced resolution as well [62]. However, this still used the reconstruction approach of OS-SIM, resulting in a spatially varying point spread function and a concomitant loss of ability to describe the image formation process linearly. Gao et al. developed this further, using the traditional reconstruction approach of SR-SIM to enhance effective resolution further still and maintain linear image formation [63]. Here, the authors showed that this Bessel light sheet structured illumination approach provided clearer images of live LLC-PK1 cells in anaphase than wide-field SIM, despite the superior theoretical resolution of the latter. In the final development of Bessel light sheet SIM, Chen et al. [ improved modulation contrast and a higher duty-cycle. However, one drawback of all these approaches is that only one pattern orientation is used, resulting in an anisotropic resolution enhancement. This is further discussed in §16.
The prototypical example of combining SIM with confocal microscopy is image scanning microscopy, discussed in §9. One of the main drawbacks of this approach is the need to acquire an entire image for each scan position of the confocal spot, meaning that hundreds or thousands of images must be recorded to reconstruct one enhanced resolution image. As such, multifocal structured illumination microscopy (MSIM) was introduced, using multiple spots of illumination in parallel [65]. Following this, it was realized independently by multiple research groups that the reconstruction procedure was amenable to an all-optical implementation [66][67][68]. A further approach, using a 32-segment confocal detector and multichannel photomultiplier tube has also been introduced in Zeiss' Airyscan product [69].
In all these approaches, the use of a confocal pinhole (either physical or digital) introduces optical sectioning, making these techniques less sensitive to out-of-focus light than traditional SIM. However, the effective resolution and contrast in non-scattering samples is lower, due to the less effective way in which the illumination brings higher spatial frequencies into the passband. Compared with traditional confocals, apart from the need for smaller pixel pitches to satisfy the Nyquist-Shannon sampling criterion, there is no drawback to using these SIM-like approaches, with benefits not only in resolution but also in signal-to-noise ratio, due to the superconcentration of light [70,71].

Should image scanning microscopy be considered a form of SIM and what forms of structured illumination could be used other than stripes?
Image scanning microscopy (ISM), first proposed by Sheppard in 1988 [72] and first demonstrated by Müller and Enderlein in 2010 [73], is experimentally quite different from SIM. In ISM, a tightly focused illumination spot is scanned over the sample, with a camera image of the emission recorded at every scan position. For each position in the spot image, the sample location most likely to have produced that intensity is not, as one might assume, at the location corresponding to the image position, but instead is located β/2 between the illumination scan position and image position, where β = λ emission λ illumination is the Stokes' shift factor (i.e. for no Stokes' shift, the most-likely location is halfway between). In ISM processing, every pixel in each image is relocated in this way, with the corresponding intensity being added to an upsampled pixel grid. At first, this seems very different from the SIM discussed so far. However, the idea that non-uniform illumination can provide higher resolution information is still the same, albeit with a very different form of structured illumination.
Indeed, the real-space reconstruction approach introduced in §6 can be extended to process ISM data as well, albeit not in a way that uses photon reassignment. For the purposes of visualization and computational speed, figure 4 uses multiple focused illumination spots in parallel, as produced by MSIM [65]. As before, each raw image (figure 4h) is multiplied by a pattern that mimics the spatial period and phase of the illumination, but is spectrally flat. These multiplied images (figure 4i) are then summed to produce the final reconstruction (figure 4c). As shown by a comparison of the spectra in figure 4e,f, the processed image contains information at higher spatial frequencies than in the simple sum of the raw data.
Furthermore, the real-space reconstruction approach can be modified to handle quasi-arbitrary illumination patterns. As an example of this, figure 5 shows a workflow for using a projection of the text 'SIM' as an illumination pattern. Electronic supplementary material, video S3 shows the steps in the acquisition and reconstruction as data is collected. While not necessarily spectrally optimal, a comparison of the spectra of the raw sum and remultiplied sum, along with linearly deconvolved images, shows that higher resolution information has successfully been extracted. An alternative approach in which the illumination pattern is not known, called 'blind SIM', has also been investigated [74][75][76][77].   (a) ground truth wide-field reconstruction ground truth spectrum wide-field spectrum reconstruction spectrum wide-field deconvolved single frame homodyned single frame reconstruction deconvolved single frame spectrum homodyned frame spectrum  As discussed in §8, SIM requires the illumination pattern to have good contrast in order to work well. For thicker samples, where much out-of-focus light contaminates the image of the plane of interest, sparse illumination is helpful. However, for maximum performance at high spatial frequencies, a dense illumination is required. In addition, a greater number of phase shifts must be used. This can be understood by considering real space and the Fourier domain as reciprocal spaces. Hence, sparse illumination results in a dense Fourier spectrum and vice versa. A dense Fourier spectrum means that more components must be unmixed, and so a greater number of phase shifts must be used.
Furthermore, dense illumination is more readily combined with total internal reflection fluorescence microscopy. To see this, consider an MSIM-like illumination pattern, where the back focal plane of the objective has the illumination shown in figure 6a, with circular polarization. This produces an illumination in the focal plane as shown in figure 6b. If we wished for the illumination to be confined to the TIRF-regime, then we must ensure that only the outer annulus of the back focal plane is illuminated, as shown in figure 6c. This results in the illumination shown in figure 6d. Here, the interference in the pattern causes unwanted illumination, leading to excess photodamage and noise.
Recently, Ingerman et al. considered a related question specifically for dense illuminations: is it better to use two-dimensional patterns or sequentially rotated one-dimensional patterns for linear and nonlinear SIM [78]? Perhaps surprisingly, they found that two-dimensional patterns were better for high resolutions in nonlinear SIM while one-dimensional patterns were superior for linear SIM, although the performance advantage of two-dimensional patterns disappeared when non-photoswitchable background was introduced. In addition, for the nonlinear case, they found that two-dimensional patterns require almost twofold more exposure to off-switching light as one-dimensional patterns, for the same resolution enhancement. This suggests that one-dimensional methods are superior if photodamage from off-switching light is present.
These considerations began more than a decade before appearing in print, but publication was delayed by the deteriorating health and untimely death of Mats Gustafsson

11
. Can SIM be used to improve the resolution of (Rayleigh scattering) transmission microscopy?
No. As noted in §2, the resolution limit for oblique illumination is already twice as good as that for head-on illumination. A structured illumination set-up can provide such oblique illumination, but there is no further resolution to be gained. For further information, see the excellent article by Wicker & Heintzmann [79].
12. Can we generate 'true' super-resolution images from simple instruments enhanced with machine-learning-based algorithms?
Machine learning has proven to be a powerful tool for diverse image processing applications, such as image segmentation, denoising, deconvolution, etc. [80][81][82][83]. However, care should be taken to distinguish between approaches that try to extract information from an image and those that try to add information to an image. With good training, it has been possible to develop multiple deep-learning approaches that convert an image, or set of images (such as an SIM acquisition sequence), into a higher resolution image [84][85][86][87]. While processing a set of images may fall into the 'extract' category, processing a single image in such a way must necessarily fall into the 'add' category. As such, machine-learning will never be able to truly replace, rather than augment, a super-resolution technique. Nevertheless, such single-image approaches may be able to improve the effective resolution in a manner similar to deconvolution.

Can research-grade super-resolution (SIM) microscopes be built cost-efficiently?
All structured illumination microscopes are relatively cheap-the expense lies in the time required to make them work well! A basic system can be constructed using a rotation mount, piezo translation mount, diffraction grating, polarizer and a standard laser, plus some lenses and mirrors (as in the very first SIM publications [1,2,5]). However, such a system will be too slow for live-cell imaging and can severely limit the throughput of fixed-sample imaging [8,9]. Constructing an SIM system fast enough for live-cell imaging requires a way of generating the sinusoidal illumination pattern without physical rotation of a grating. In general, two methods are common: spatial light modulators (SLMs) and interferometric systems. Spatial light modulators can, to a first approximation, be thought of as reprogrammable diffraction gratings and used in a similar way. However, instead of shifting and rotating the SLM, the pattern displayed by the SLM is instead updated to match. This arrangement maintains the common-path layout of grating-based systems, minimizing the sensitivity to air currents and temperature fluctuations. However, fast-switching ferroelectric SLMs have a very low diffraction efficiency (often less than 5%) and so require powerful laser sources to be used. Fast-switching nematic SLMs, with much higher diffraction efficiencies, are now commercially available, albeit at high cost and not yet capable of the frame rates of ferroelectric SLMs.
Interferometric systems normally feature some kind of Michelson/Twyman-Green or Mach-Zehnder interferometer, where pattern orientation is changed by changing optical paths and phase steps are accomplished by changing the length of one arm of the interferometer [88,89]. These approaches are much more photon-efficient than SLM-based systems and, as the pattern period changes with wavelength, rather than being fixed, can be used to ensure the same effective illumination NA over all wavelengths in use. This is the approach taken by the commercial OMX Blaze system [90]. However, such interferometers are extremely sensitive to air currents and temperature fluctuations, as any phase shift in one arm of the interferometer directly couples to a phase shift in the illumination pattern. In an attempt to circumvent this issue, fibre-based interferometeric SIM systems have recently been developed [91,92]. Overall, SLM-based systems are by far the most common, in both research-grade and commercial instruments. In particular, Forth Dimension Displays have cemented themselves as the preferred supplier of ferroelectric SLMs for the SIM community, due to their high pixel count and rapid switching speed. Nevertheless, successful operation of such an SLM is tricky for a newcomer, due to effects from the binary nature of the pixels, interpixel vias, duty-cycle requirements and low photon-efficiency. As such, there has recently been much interest in the development of systems that use a different type of SLM, namely digital micromirror devices (DMDs) [93,94]. Found in many projectors, these devices rely on tilting tiny micromirrors to direct light in one direction or another, acting as a binary amplitude display. While these provide high photon-efficiencies, their tilted-mirror nature means that they act akin to blazed gratings, with a strongly wavelength-dependent distribution of power into different diffraction orders. As such, their use is currently restricted to those laboratories interested in researching how best to make use of DMDs for SIM, rather than following the tried-and-tested approach of ferroelectric SLMs and powerful lasers.
14. How can information about single-molecule detection be best combined with the knowledge of the illumination structure?
This is an area of ongoing research, prompted by the development of MINFLUX [95]. In MINFLUX, an intensity minimum at the centre of a doughnut beam is used to localize single fluorophores by iteratively updating the doughnut position and minimizing the photon flux. While this approach can provide excellent localization precisions (less than 1 nm), it is inherently slow as it only probes one molecule at a time. Since the initial publication of the MINFLUX method, there have been a number of publications which combine full-field fringe-like structured illumination with the MINFLUX concept to provide high-precision localizations at faster rates [96][97][98][99]. Most recently, Schmidt et al. have shown that these methods only use half the localization information encoded by the structured illumination and demonstrate a further method that, unlike previous approaches, achieves the Cramér-Rao lower bound [100].

Can the axial resolution of an SIM microscope be enhanced beyond a twofold improvement?
Enhancing the axial resolution of an SIM system requires accessing the sample from the other side, either to introduce extra wavevectors for illumination, or to capture extra wavevectors of fluorescence. Perhaps the most impressive SIM microscope yet built, the I 5 S of Shao et al., does both [101]. By using a second opposing high-numerical-aperture objective, a further three beams of illumination are introduced. The same objective is also used to capture fluorescence, with the two objectives being aligned to such a precision that the fluorescence captured by both can be made to interfere via a beamsplitter, despite the approximately 1 µm coherence length of the light. This provides an almost-isotropic, better-than-100 nm resolution in all three dimensions, providing volumes of objects with no preferred viewing direction. Despite this impressive performance, I 5 S is not widely used, due to the challenging alignment and stability requirements for its operation. In addition, the very short coherence length of fluorescence emission (approx. 1 µm) means that sample-induced phase differences restrict the technique to imaging thin single cells. Recently, Manton et al. proposed that a lowernumerical-aperture, long-working-distance dipping objective could be used to introduce a single counterpropagating beam and provide much of the axial resolution enhancement of I 5 S in a much simpler set-up [102]. While such a system is theoretically capable of providing three-dimensional resolutions better than 125 nm, the transfer function at high spatial frequencies is lower than that of an I 5 S microscope without interferometric detection. As such, we expect that six-beam structured illumination provided by two opposing high-numerical-aperture objectives will be the future method of choice for extended-resolution imaging when isotropic resolution is key.

16.
Can light sheet microscopy and SIM, with multiple pattern orientations, be properly combined?
As noted in §8, light sheet and SIM appear to be a perfect match: light sheet microscopy is limited in resolution due to the relatively low numerical aperture objectives used, but generates very little out-of-focus light-the bane of SIM. However, traditional light sheet SIM (LSFM-SIM) suffers from the relatively coarse illumination pattern that can be provided by the excitation objective in LSFM systems and the fact that only one orientation of the pattern is present. As such, while improvements in axial resolution can be made, very little useful enhancement of lateral resolution is possible.
In an attempt to counter these issues, Manton & Rees proposed an LSFM-SIM system featuring three mutually orthogonal objectives [103]. However, the use of 0.8 NA objectives, necessary for reasons of steric hindrance, meant that the maximum possible lateral resolution would still only be around 230 nm (i.e. approximately the theoretical resolution of a 1.1 NA water-dipping objective as used in some other light sheet microscope designs). While they also suggested the use of interferometric excitation through two objectives to improve the resolution to 120 nm, the experimental complexity of such a system means that it is unlikely to ever be realized in practice.
In 2017, Chang et al. demonstrated a three-objective system featuring interferometric excitation through two objectives and counterpropagating light sheets [104]. While the use of counterpropagating sheets produces a fine fringe period, issues with steric hindrance limited the orientations over which patterns could be produced, precluding a reconstruction with isotropic resolution. In addition, the large distance between arms of the interferometer mean that the system is sensitive to temperature-induced phase shifts, as discussed in §13, and sample mounting is made more difficult.
Furthermore, there is a theoretical issue with using interferometric excitation through multiple objectives that has so far been unaddressed. In the traditional SIM approach, we consider the illumination to be described by a discrete sum of purely harmonic functions (i.e. functions which consist of a single spatial frequency). As shown in figure 7a, this condition is satisfied for lattice light sheet microscopy [64] as the pupil pattern consists solely of vertical lines, which project orthographically onto the Ewald sphere (i.e. they still look like straight lines when viewed from above, with the separation between lines being constant-the distribution is a discrete sum of harmonic functions along the pattern direction). However, now consider a system in which a single vertical line in the pupil is used to produce a light sheet with one objective, with another identical pupil and objective opposing (figure 7b). Now, when the Ewald sphere is viewed from above, the distribution is not a discrete sum of harmonic functions along the pattern direction. While the limitations on numerical aperture provided by use of a single objective can be circumvented by using C-shaped illumination profiles in ancillary objective pupils (figure 7c), it appears that the use of counterpropagating sheets is not allowed, at least not by standard reconstruction approaches.

How quickly can SIM image a sample?
Fundamentally, the speed of SIM is restricted by the requirement that enough photons are detected for the signal-to-noise ratio to permit a high-quality reconstruction. So far, the fastest SIM acquisitions demonstrated were acquired at 798 Hz and 714 Hz, corresponding to image frame rates of 88.7 Hz and 79 Hz [105,106]. Given that even a single fluorophore can emit at rates in excess of 100 million photons per second, it appears that current speed limits are more practical than fundamental.
In an attempt to reduce the acquisition time for volumetric data by reducing the number of stage movements that must be completed, SIM has been combined with multifocal microscopy [107]. Here, images from multiple depths within the sample are recorded at the same time. Here, the components of the Fourier intensity distribution are stretched out along the pattern direction, precluding conventional reconstructions in which harmonic components are assumed. (c) A potential method for producing lattice-like illumination patterns at spatial frequencies beyond that achievable with one objective. Here, C-shaped illumination profiles in the auxilliary objectives are used to introduce the off-centre Ewald sphere components, ensuring a harmonic Fourier intensity distribution. (Online version in colour.) scheme, the limited speed of the commercial SIM system used prevented fast acquisitions [108]. A more recent demonstration, in 2020, of high-speed acquisition used an aberration-inducing multifocus prism, limiting the depth over which images could be acquired in parallel [109].
While these experiments demonstrate an impressive effort in pushing hardware to its limits, other effort has been expended on improving the reconstruction process. To enable live visualization of a sample with extended resolution, Markwirth et al. developed a software system that can provide GPU-accelerated reconstructions in almost real-time [110]. Many groups, along with Zeiss in their Elyra 7, have adopted 'interleaved reconstruction', in which rolling sets of three orientations are used to produce reconstructions, rather than only producing one image from each group of three-orientation data [111]. While this produces smoother videos, it is a form of interpolation and does not truly increase speed [112].
A number of attempts have been made at reducing the number of raw images required for a successful reconstruction, down from nine. Heintzmann was the first to propose that a smaller set of images could be sufficient, identifying that using as few as four images might be possible [113]. Lal et al. have proposed such a reconstruction using just four images, but the illumination patterns used do not average out to a uniform intensity [114]. This means that, for time-lapse imaging, the illumination pattern would 'burn' into the sample via spatially dependent photobleaching. Orieux et al. have developed a Bayesian approach to SIM reconstruction and used this to develop a method that uses four images and has no specific constraints on the illumination [115]. Furthermore, Ströhl and Kaminski investigated the possibility of a further reduction to just three images and developed a maximum-likelihood approach to deal with the resulting underdetermined system [116].  Gustafsson [2] Gustafsson [43] Heintzmann and Cremer [1] Heintzmann et al. [42] Kner et al. [8] Li et al. [36] Rego et al. [45] Schermelleh et al. [117] Shao et al. [9] (b) Gustafsson [2] Gustafsson [43] Heintzmann & Cremer [1] Heintzmann et al. [42] Kner et al. [8] Li et al. [36] Rego et al. [45] Schermelleh et al. [117] Shao et al. [9]  new toy' that must be mentioned in the title or abstract. Nowadays, it suffices to just mention SIM in the main text alongside well-respected techniques such as confocal microscopy.

Outlook
Perhaps the most important development in the past few years is that of all-optical versions of ISM, providing extended-resolution imaging at high frame rates and without the need for computational reconstruction [66][67][68]118]. These instruments present the user with a system that effectively works as a familiar confocal or spinning disc microscope, albeit one with double the resolving power. Furthermore, by avoiding computational reconstruction, these approaches avoid the reconstruction artefacts that can often plague SIM [119]. However, in the absence of background fluorescence, at high spatial frequencies traditional SIM still provides superior contrast and is more compatible with TIRF imaging. As such, further investigation into which techniques provide better performance in different samples and imaging scenarios will be welcome, as would an all-optical SIM reconstruction approach.
For longer-term time-lapse imaging, light sheet fluorescence microscopy (LSFM) is now the technique of choice for minimal sample damage where fluorescence contrast is required. While combinations of LSFM and SIM have already been demonstrated (see § §8 and 16), an instrument that achieves maximal performance of both components has yet to be demonstrated.
Further enhancements to the capabilities of SIM for time-lapse imaging are being provided by enhanced reconstruction routines that use various forms of regularization to successfully reconstruct images with lower signal levels [120,121]. In combination with brighter fluorophores, further development of these techniques and other denoising procedures will help provide gentle, extended-resolution imaging of proteins of interest at endogenous expression levels.
As well as enhancing the resolution of images and movies, SIM has also been demonstrated to enhance the resolution of biophysical microscopy methods. While fluorescence lifetime imaging microscopy (FLIM) has been combined with OS-SIM [122] and used correctively with SR-SIM [123], so far no true combination of FLIM and SIM has been achieved. In 2019, Zhanghao et al. reported their method of polarized structured illumination microscopy (pSIM), providing extended-resolution imaging of dipoles in biopolymers such as cytoskeletal networks and λ-DNA [124]. Most recently, in 2019 Colin-York et al. have combined traction force microscopy (TFM) with 3D-SIM [125], while in 2021 Barbieri et al. combined TFM with TIRF-SIM [126].
Altogether, it seems that there is still much room for further work in SIM, both in terms of technique development and applications. It will be interesting to survey the field again in 20 years' time and see how many of the outstanding challenges have been solved and which unanticipated developments arose. If the past 20 years are any guide, the answers will be surprising.
Data accessibility. This article has no additional data. Authors' contributions. J.M.: investigation, software, visualization, writing original draft, writing review and editing.
Competing interests. I declare I have no competing interests. Funding. No funding has been received for this article.