Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
You have accessDiscussion

Answers to fundamental questions in superresolution microscopy

Rainer Heintzmann

Rainer Heintzmann

Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany

Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich-Schiller-University, Helmholtzweg 4, Jena, Germany

[email protected]

Google Scholar

Find this author on PubMed

    Abstract

    This article presents answers to the questions on superresolution and structured illumination microscopy (SIM) as raised in the editorial of this collection of articles (https://doi.org/10.1098/rsta.2020.0143). These answers are based on my personal views on superresolution in light microscopy, supported by reasoning. Discussed are the definition of superresolution, Abbe's resolution limit and the classification of superresolution methods into nonlinear-, prior knowledge- and near-field-based superresolution. A further focus is put on the capabilities and technical aspects of present and future SIM methods.

    This article is part of the Theo Murphy meeting issue ‘Super-resolution structured illumination microscopy (part 1)’.

    1. What is superresolution microscopy?

    ‘Superresolution’ naturally means to achieve resolution beyond what is achieved by ‘ordinary methods’. However, such a broad definition is in my view too unspecific and therefore I prefer to refer to superresolution imaging as methods that surpass Abbe's resolution limit. But even this definition is not very well defined, as currently superresolution methods are mostly relating to fluorescence microscopy and Ernst Abbe did not consider or write about fluorescence imaging when he specified his famous equation about resolution in optical systems [1]. For an in-depth review of superresolution, see [2]. Abbe's often-cited resolution limit is

    d=λ2nsin(α)=λ2 NA, 1.1
    with λ/n being the wavelength at the sample, λ the corresponding wavelength in vacuum, n the refractive index and α the half-opening angle of the objective (related via the Abbe sine condition to the objective pupil radius) and the numerical aperture being defined as NA = n sin(α). Abbe referred with this equation to the minimum pitch d of a periodic (grating-like) structure, which is still present in its image obtained under maximally oblique illumination such that the direct transmission is focusing at the edge of the objective pupil. Equation (1.1) referring to maximally oblique illumination is sometimes forgotten, causing common misunderstandings [3]. In his publication [1], Abbe explicitly states that the correct equation for a head-on illumination is given as
    d=λnsin(α)=λNA.

    For his experimental observations, he used the periodic structures of diatoms as a sample, as described in his publication [1]. For completeness, the corresponding resolution limit for any combination of illumination (NAillu) and detection (NAdet) apertures in imaging of elastically scattered light is described by

    d=λNAillu+NAdet, 1.2
    which holds both for oblique coherent illumination at the border of the illumination NA as well as incoherent illumination up to this numerical aperture. Note that most microscopy users use full-aperture incoherent illumination via a condenser, which leads to a massive loss in contrast at medium to high spatial frequencies. Only very careful observation shows that by increasing the illumination NA the resolution limit is in fact increased, despite the apparent loss in contrast. Abbe's oblique illumination geometry was in this respect very helpful to maintain contrast. In fact, the same lateral resolution limit (equation (1.2)) also applies to confocal microscopy imaging scattering samples. An especially interesting case concerns the case NAillu > NAdet, with the inner part below NAdet of the illumination blocked, referring to dark-field microscopy. A single grating with a spatial frequency, just below the limit as stated in eqn. 1.2, will at best cause a homogeneous image. Contradicting equation (1.2), this grating is, by itself, clearly not resolved and merely its presence is being detected, albeit being even significantly coarser than Abbe's limit. Ultimately this contradiction to the resolution prediction stems from the fact that we have no access to the phase of the light in the detector plane. If the phase of this wave on the detector plane (and the illumination wave) is resolved, the orientation and pitch of the grating can be calculated. Interestingly, more complex samples or more complicated coherent illumination or detection schemes such as ptychography, structured illumination or using holographic/phase-sensitive detection/wavefront sensors [4], provide multiple mutually interfering waves to the detector plane, which can enable us to ultimately obtain the wanted phase information with other waves serving as reference waves and therefore equation (1.2) can also be regarded as the resolution limit for NAillu > NAdet. Yet for imaging modalities that just directly observe the obtained image one may want to replace NAillu with min(NAdet, NAillu) in equation (1.2) as a practical limit for the imaging of elastically scattered light. It has been suggested [5] to define the ‘n-line resolution’ as ‘the minimum spacing between successive elements of an infinite array of line objects such that the lowest spatial frequency component other than the constant term appears in the image’. However, I prefer equation (1.2) over this n-line limit for ‘direct inspection’, as the n-line resolution may obscure the fact that a grating outside this limit may still be resolved if the object possesses multiple features, some of which provide the wanted reference wave at least for computational reconstruction, or if a separate reference path from the light-source is present. One just has to remember that equation (1.2) refers to which information can in principle be obtained, meaning with the appropriate method, that may require a separate (holographic) reference beam path from the illumination source.

    Abbe's equations are, in contrast to many other definitions of resolution limits such as the Full Width at Half Maximum (FWHM) of the PSF, the Rayleigh or the Sparrow limit, see also [2], concerned with a simple question requiring the answer to be ‘yes’ or ‘no’: Can a structure with periodicity d be observed in its image? Abbe realized that for imaging of elastically scattered light, the precondition to observe such a structure is the presence of at least two mutually interfering diffraction orders (here 0th and 1st) in the light contributing to the image by mutual interference. Very notably, his equations do not predict quantifications of object features such as the FWHM or the contrast in the image. A periodic structure is either within the limit (yes = can be detected) or not within the limit (no = not detectable). Yet his equations are often misinterpreted and wrong superresolution claims made. If we consider a grating at a pitch slightly above the Abbe limit (i.e. of pitch d) observed under oblique illumination as described by Abbe, the image of elastically scattered light can still have close to 100% contrast, meaning that the FWHM measured on the image of any bar of the grating at 100% contrast is d/2=λ/(4 NA). As this example illustrates, observing a feature in the image and obtaining a FWHM measurement with a value below λ/(2 NA) cannot be called ‘superresolution’ as it describes nothing more than the imaging situation that Abbe treated in his text [1]. How about showing a reduced FWHM of the point-spread function (PSF) as a proof of superresolution beyond Abbe? Even though the PSF is a good way of describing the imaging performance of a system, especially if it is linear shift-invariant, its FWHM can be too easily tampered with, while not enlarging the support of the corresponding optical transfer function (OTF). If you, for example, modify the detection pupil of your microscope by letting only two point-like holes near its edge transmit light, your PSF will become a cos2 function with an FWHM of λ/(4 NA) exhibiting side-lobes as high as the main lobe. Is this a superresolution method? In my definition it is clearly not, as Abbe's frequency limit is not improved and apart from one specific pattern pitch and orientation, nothing else can be sensibly imaged with such a PSF. Another fairly common mistake is to claim beating Abbe's limit by wrongly assuming that d describes the gap between adjacent bars of the grating rather than the full grating pitch.

    As a final note on Abbe's limit for transmission/reflection imaging, one may wonder if this one value as stated by equation (1.2) sufficiently describes the ‘yes’ or ‘no’ imaging abilities of a system. This is, of course, not the case. An imaging system, even while transferring high spatial frequencies well, may be incapable of transferring low spatial frequencies, as is the case for reflection confocal microscopy or optical coherence tomography. A system with axial spatial resolution may nevertheless be incapable of transferring any axial spatial frequency (kz > 0). This is the case for homogeneously illuminated widefield fluorescence imaging, which is called the ‘missing cone’ problem in the OTF. However, since Abbe's limit precisely describes the lateral frequency limit, I would regard the non-zero region of the support of the OTF to constitute its multi-dimensional generalization. This generalized Abbe's limit is a complete description about the object information transferrable by an optical system.

    Why is it that Abbe's limit, originally concocted for brightfield microscopy, is so often quoted in the context of fluorescence imaging? In my view, which may well be historically incorrect, the reason is that the same two rays residing at opposing edges of the pupil, forming the image during brightfield imaging of a grating, also interfere to contribute to the image of a point-like fluorescence emitter (PSF). As the incoherent OTF is the Fourier-transformation of the PSF, this interference also contributes the highest spatial frequency to the OTF. Thus Abbe's oblique imaging equation (equation (1.1)) maintains its meaning also for fluorescence imaging, by describing the finest periodic (here fluorescent) structure with pitch d that can be transferred by the microscope to the image, irrespective of the actually achieved contrast (the transfer strength of the OTF).

    The photo-physics of fluorescence destroys any phase relationship the emitted light may have with respect to the excitation light. This ‘loss of phase’ in fluorescence imaging is a blessing, as it leads to two important advantages:

    (A) Multiple incoherent fluorescence emitters can be assumed to emit, neglecting stimulated emission induced by adjacent emitters, at mutually random phase. This leads to the detection being described by the superposition of their image intensity rather than their amplitudes. With the additional assumption of shift-invariant imaging, the intensity superposition makes the description of a fluorescence image particularly simple. The convolution of the sample with a single intensity PSF encompasses the whole incoherent imaging process. This linearity between fluorophore concentration and detected image is very different from any imaging of elastically scattered light no matter how incoherent any far-field illumination of the sample may be in brightfield microscopy. Two equally strong adjacent point objects at a distance well below the wavelength will cause four times as much elastically scattered intensity. This fundamentally inherent nonlinearity of scattered intensity with respect to the local concentration of scattering molecules makes partially coherent imaging difficult to describe and acquired data difficult to interpret. Luckily, fluorescence imaging remains simple in its description in this respect.

    (B) This ‘loss of phase’ in the fluorescence photo-physics, as opposed to elastic scattering, is also the very reason that we obtain another independent handle by using the local illumination intensity structure (structured illumination) to ‘probe’ the sample by illumination intensity. This loss of phase is the reason that fluorescence structured illumination microscopy (SIM) is able to shift out-of-band information [6] into the detection passband.

    In imaging of elastically scattered light, the scattered amplitude, when illuminating with the superposition of mutually coherent waves, remains a coherent superposition of the imaged scattered amplitude (electric field) as obtained under individual illumination with each of these waves [3]. Therefore, even though structuring the illumination intensity may help with solving a phase problem in coherent imaging, it is fundamentally not suited to improve on the resolution limit in Abbe's sense. Separately imaging the sample under each of the illumination waves contains fundamentally the same information. The only possible advantage of simultaneously illuminating with multiple coherent waves is to provide potentially useful reference waves helpful for determining phase information. This may be especially worthy for dark-field situations. In other words: fluorophore excitation and emission amplitudes are not just linearly related quantities due to the phase-decoupled emission intensity. This independence allows for the circumvention of Abbe's limit and for remaining linear with respect to concentration and intensity. A new limit obtained from the linear combination of excitation and emission linked by fluorescence (or also Raman or Compton scattering), which can be called the excitation/emission Abbe limit will be discussed in the next two questions below.

    (a) Should diffraction-limited linear SIM be classified as ‘superresolution’?

    In my opinion: yes, it can be called superresolution as exploiting structured excitation and imaging emission circumvents the emission-only limit corresponding to Abbe's equation. Naturally, the same applies to confocal microscopy, which can be seen as a form of structured illumination but is not as efficient in transferring high spatial frequencies. Yet both of these linear methods stay within the excitation/emission Abbe limit and therefore belong to the class of restricted superresolution schemes [7].

    2. Should high-NA TIRF-SIM, which can achieve lateral resolution down to 84 nm, be considered as diffraction limited?

    If we assume that illumination structures are, similarly to the imaging of the fluorescence emission, limited by the wavelength λillu and numerical aperture of the illumination NAillu, we obtain a combined excitation/emission Abbe limit (termed ‘restricted superresolution’ in [7]) for any form of linear far-field structured illumination:

    d=12NAillu/λillu+2 NAdet/λdet, 2.1
    where the wavelength λdet and numerical aperture NAdet apply to the detection configuration. Note that we simply need to add the spatial frequency limits of the corresponding OTFs for illumination (1/dillu) and detection (1/ddet). If we neglect the Stokes shift (λ = λdet = λillu) and assume the identical objective for detection and illumination (NA = NAdet = NAillu), we can approximate the limit for linear structured illumination as
    dλ4 NA. 2.2

    There are currently three known ways [6] to surpass this combined excitation/emission Abbe limit (equation (2.1)) for fluorescence microscopy:

    (A) A nonlinear or multi-linear sample response violating the assumption of the detected signal contributing to the image being linearly related to a single excitation intensity structure [8,9]. Nonlinear structured illumination (NL-SIM, SPEM) and stimulated emission depletion (STED) microscopy are typical examples of this class. In a broader sense, quantum-physical approaches [10] typically also belong to this group, as multiple photons interact with the sample and the detected photons can be identified as ‘belonging to the same source’ via their anti-bunching statistics or in other approaches photons as ‘belonging together’ by sharing common location information via entanglement. This information then indirectly leads to the ability to introduce the nonlinearity retrospectively. One may also argue that these approaches belong to class B below, as the known information about the common origin (anti-bunching) or common target (entanglement) of interaction is exploited.

    (B) Exploiting prior knowledge about the sample and its properties, e.g. that it consists of a distinct set of single molecules that maintain their ability to emit over several emission cycles as in single-molecule localization microscopy (SMLM, MINFLUX). Switching of states is an attractive way to prevent adjacent individual emitters from simultaneously being present in the image, but the actual resolution gain is achieved by the improved localization precision stemming from the emission of (or, in MINFLUX, interaction with) multiple photons that all can be attributed to the same emitter by prior information. Resolution gains as obtained by model-based computational reconstruction approaches can also be summarized under this heading. Many deconvolution and machine learning approaches use the prior knowledge that emitted (fluorescence) intensity cannot be negative, which leads to a sample-dependent resolution gain [11]. Other examples that exploit prior knowledge are imaging samples of known limited spatial extent with a PSF possessing superoscillations (‘Toraldo-filters’, [12,13]).

    (C) Exploiting near-field excitation or emission. Non-propagating (evanescent) fields being structured beyond the frequency limit (λ/n) do not immediately vanish but are rapidly exponentially damped. Interacting fluorophores, which reside in such axially rapidly decaying but spatially varying intensity structures (in total internal reflection, TIR, configuration under SIM illumination or near a fibre tip as in near-field microscopy) can respond via fluorescence. Thus superresolution beyond the excitation/emission Abbe limit can be obtained even with linear emission response if total internal reflection fluorescence (TIRF) is combined with structured illumination.

    An intriguing case that claims superresolution and, at first glance, does not seem to fall in any of the above classes is Maxwell's fisheye lens [14], related to the ‘Lüneburg lens’ ([15], Luneburg's name being an English transcription of the German town ‘Lüneburg’). A point emitter embedded at a well-defined position in a well-designed inhomogeneous refractive index structure can be shown to cause a super-resolved ‘image’ at the opposite side of the ‘fisheye-lens’. Yet a close inspection reveals that the precondition for superresolution is a drain being precisely placed at the image position of the source. This near-field point absorber is then the cause for what seems to be an ‘image’ of the sample at theoretically infinite resolution.

    Referring to the question in the heading, high-NA TIRF-SIM exploits method C above. The periodic excitation structure in the sample but near the interface to the coverslip (CS) in TIR configuration can maintain its original periodicity down to dillu = λ/(2 NACS) as realized by far-field interference in the CS. This small illumination periodicity dillu clearly surpasses Abbe's illumination limit as set for the sample medium, even if we assume the NA of the sample to be the maximally possible (NA = nsample). Similar arguments can be made for supercritical emission. Emitters adjacent to the TIR interface can also emit into angles beyond the critical angle (supercritical angle fluorescence, SAF). This has two effects: the high angles, if transferred by the objective free of aberrations, can narrow the detection PSF and increase the Strehl ratio, which leads to a higher transfer strength for high spatial frequencies, even beyond the Abbe detection limit of the emitted light for the embedding medium. In addition, SAF enhances the radiative rate of the S1 to S0 transition in the fluorophore and therefore, in competition with other non-radiative rates, increases the quantum efficiency, brightness and resistance to photobleaching. TIR illumination and SAF detection are both very effective in the suppression of out-of-focus light, which further enhances the signal-to-noise ratio (SNR) for the in-focus light. TIRF is therefore highly beneficial to the performance of SIM. Yet, a slight complication is that the high spatial frequency of the illumination structure requires peak-finding algorithms, which work on the cross- or auto-correlation of the components to separate, but this is helped by the superior SNR and the two-dimensional nature of the data.

    To answer the question, structured illumination in TIR geometry surpasses Abbe's diffraction limit in two respects. It surpasses the Abbe limit of fluorescence emission imaging by using structured illumination (see above) and it also surpasses the combined excitation/emission Abbe limit (equation (2.1)) of the sample surroundings by exploiting the evanescent field, a near-field effect.

    In addition to exciting via the evanescent field, achieving 84 nm resolution, Li et al. [16] also exploited a nonlinear sample response to achieve 60 nm resolution by causing an illumination intensity structure via photo-switching to influence the sample emission in a nonlinear way, as also exploited in RESOLFT [17,18], anticipated in [8]. Considering all of these arguments, TIRF-SIM is clearly not diffraction-limited, not even in the extended excitation/emission limit sense, which leads to the answer to the question ‘Should High-NA TIRF-SIM, which can achieve lateral resolution down to 84 nm, be considered as diffraction limited?’ being a clear ‘no’.

    3. Can nonlinear SIM become broadly applicable and live-cell compatible?

    Live-cell compatibility of NL-SIM was demonstrated in [16]. This is not to say that live-cell compatibility cannot be further improved. Fluorescent dyes with less photobleaching, less phototoxicity, high quantum efficiency and faster switching are desirable as well as non-toxic orthogonal labelling methods requiring only a small molecular weight. Technical advances will encompass reconstruction methods, which smoothly transit between the single molecule and the continuous SIM reconstruction, exploiting the benefits of each regime. As for the broad applicability of NL-SIM, I think that this is possible, but currently difficult to achieve in practice and difficult to predict when broad applicability will be achieved. A key ingredient would be the stability of the fluorophores.

    4. Do you need ’switching’ of states for nonlinear superresolution (NL-SIM)?

    In my opinion, the answer is ‘no’ [6], even though it is sometimes claimed that state switching constitutes the common basis of superresolution imaging [19]. Note that the question in the heading is different from the question ‘Is there any state switching occurring in fluorescence nonlinear superresolution?’, which I would answer with ‘yes’, if you consider the ground and excited states of molecules and the excitation and emission transitions being ‘switching of states’. However, in this sense any fluorescence imaging can be considered to be ‘state switching’, which therefore does not constitute sufficient reason to regard ‘state switching’ as a requirement for superresolution. NL-SIM, presented as ‘Saturated Patterned Excitation Microscopy’ in [9], only relies on the same two states (S0 and S1) as used in every fluorescence event in conventional fluorescence microscopy, yet nonlinear superresolution beyond the excitation/emission limit is achieved. This means that we have to answer the question ‘Is state-switching a sufficient condition for super-resolution’ with ‘no’. How about state switching being a necessary condition for super-resolution? It is true that many fluorescence-based superresolution schemes currently use light to play with the photo-physics of the molecule. Structured illumination light, which somehow influences the system under study in such a way that we can detect this influence, is naturally suited to ‘probe’ the sample to achieve superresolution. But is this light fundamentally required to directly drive a switching transition such that switching can be identified as the basis for superresolution? I think not. Any influence detectable by a modification in the image can do. For example, magnetic fields can influence the detected fluorescence signal via the Zeeman Effect. Using strong local magnetic field gradients, fluorescence-detected superresolution is possible down to the nanometre scale [20,21]. Even when limiting the influence on the detected image to another pattern formed by light, this light can cause the modulation of the emission by many effects (change in local temperature, Rabi oscillations, optically induced Kerr-effect), only some of which can be directly related to the action of a state-switch.

    Superresolution approaches can be grouped into continuous and single-molecule methods (e.g. SMLM). Abbe's model of imaging does not require, and almost precludes, the assumption of a discrete (molecular) nature of the sample. It also does not require any ‘switching of states’. Nonlinear methods such as NL-SIM or STED yield a super-resolved image surpassing the combined excitation/emission Abbe limit. These and other methods are perfectly well described by assuming a nonlinear, or ‘multi-linear’ (meaning multiple patterns influence the emission in a linear way) dependence of the detected light on the local excitation intensity. All involved quantities such as local concentration and local excitation and emission intensity can conceptually all be continuous quantities and do not require the assumption of state switching in the model describing the image formation. The fact that the sample does indeed consist of molecules that each are in well-defined states at any moment in time is currently more of a problem to these ‘continuous’ methods than a gain. In NL-SIM and RESOLFT, the switching statistics causes unwanted super-Poissonian noise, meaning that multiple photons are typically detected from the same state of the molecule. Research towards exploiting the single-molecular discrete switching nature has begun with methods such as MINFLUX, ROSE, ModLoc, SIMPLE and SIMFLUX [22,23], but will doubtlessly continue towards nonlinear SIM or STED approaches.

    Violating the assumption that a molecule is always in a well-defined state (as in ‘state switching’) can be exploited: as described in [8], Rabi oscillations between molecular states can be used to cause the wanted nonlinear relationship between excitation and emission. There is clearly no ‘switching of states’ involved in such a situation, which clarifies why in my view ‘state switching’ does not constitute a common basis or prerequisite for superresolution.

    Yet, the fact that fluorescence comes in discrete quantities, molecules and molecular states and switching between them, is something that is clearly exploited in single-molecule localization methods (SMLM). But in SMLM the purpose of switching is separation, to be able to assign many emitted photons to the same entity, i.e. to separate individual emitters [24]. In SMLM and MINFLUX, the superresolution comes from the fact that multiple detection events (or also ‘not-detection’ events in the case of MINFLUX) can be assigned to the same entity (molecule) with (hopefully) still the same position. The superresolution stems from the multiple photons being observed (or not observed), and the prior knowledge about the single-molecule condition. The state switching is used here to separate emitters, but the resolution increase is achieved by the prior knowledge about multiple measurements being performed on the same emitter and jointly evaluating all the detected (or non-detected) photons.

    5. How can information about single-molecule detection be best combined with the knowledge of the illumination structure?

    This is a question of ongoing research. The future will provide more answers to this question. In my view, we will see additional methods based on the interplay between the prior knowledge of the sample, previously detected photons and the history of illumination structures. This is the Bayesian way to approach the problem: Every new illumination structure asks new ‘questions’ about the sample structure and the answer of the sample (when and where photons were detected or not detected) defines the best next question (illumination structure) to ‘ask’ to the sample.

    Models that attempt to estimate the sample structure need to consider its discrete molecular nature, knowledge about molecular states, and the history of illumination structures (questions) and detection events (answers). Such models can also include other general assumptions (only positive emissions, sample sparsity, thickness, etc.).

    6. Do high-quality SIM images require reconstruction in Fourier-space?

    No. Any linear method can always be written down in matrix form, which does not fundamentally require a Fourier-transformation. Yet fast Fourier-transformations are sometimes useful, as this can speed up the computation. This is the only justification for the need for Fourier-space. From early on, SIM data have successfully been reconstructed fully in real-space [25], even though the SNR was not necessarily as good as achieved by weighted averaging in Fourier-space.

    7. Can SIM be used for deep tissue imaging?

    Yes. Depending on the sample structure and the relative amount of staining, a problem can be a too large amount of out-of-focus light, which contributes, even if computationally removed to the noise in the reconstructed image. Sparse illumination via two-dimensional [26], line-like [27] or ROI-based SIM patterns [28] help to reduce the detrimental influence of out-of-focus light.

    8. How can the fundamental limitation of SIM, i.e. generating sufficient stripe contrast in densely labelled and/or extended biological structures due to out-of-focus light, be addressed?

    The answer to this question can be separated into two tasks. (A) Generating a SIM illumination of high contrast deep in a sample. For samples of only a few cell layers thick, this should not be a problem for a two- or three-beam SIM set-up. The sample will create unwanted local changes to the illumination phases of SIM beams, however, this will only lead to a local distortion of the SIM grating. Even the relative shifts, and therefore the component separation task, still works under such conditions. Sufficiently ‘smart’ image processing should be capable of reconstructing high resolution images. Yet, to probe really deep into tissue is a different issue and one would need adaptive optics approaches with thousands of degrees of freedom, that are theoretically capable of generating an aberration-free illumination structure even deep inside the tissue. This is already very hard to achieve in practice, but the next problem is to detect the image of the emitted light emerging from deep within the tissue. The required polychromatic and multi-conjugate adaptive optics to achieve this is currently beyond reach. (B) Yet a much simpler practical problem is that samples are typically labelled throughout and therefore the massive amount of detected out-of-focus light ruins the SNR in the reconstruction. This can partly be ameliorated by choosing a sparse and/or region-of-interest-illumination [2628]. Another promising option, albeit technically more difficult, is to combine light-sheet or oblique-plane illumination with SIM [29].

    9. Should image scanning microscopy be considered a form of SIM and what forms of structured illumination could be used other than stripes?

    If you call all methods that exploit a patterning of illumination ‘structured illumination’, than the answer is ‘yes’ as this description also applies to confocal microscopy illuminating with a single focused spot. However, commonly microscopy with focused illumination is not regarded to be ‘SIM’, which typically relies on generating illumination structures relatively close to the diffraction limit at very high contrast. In this sense, ISM is not a form of SIM, as the illumination structure focused to a spot contains high spatial frequencies near the Abbe limit of illumination only weakly. The overall performance strength of ISM is therefore rather in the mid-frequency regime compared to SIM concentrating on the high-frequency range.

    As for other forms of illumination patterns, two-dimensional illumination patterns can in some situations be beneficial to one-dimensional patterns [26,30,31]. However, in most cases the SNR is slightly advantageous for one-dimensional excitation patterns [31]. One practical advantage of two-dimensional patterns is the avoidance of the need for pattern rotation, which increases the speed and stability of mechanically shifted diffraction gratings, creating the periodic illumination structure. Especially noteworthy are also hexagonal arrangements of patterns [32].

    10. Can SIM be used to improve the resolution of (Rayleigh scattering) transmission microscopy?

    No, see [3]. See the answer to question 1 above for more detail.

    11. How does sparse illumination compare to dense illumination in linear and nonlinear SIM?

    Sparse illumination enables superior signal-to-noise performance for thick and densely labelled specimens [2628]. However, the drawback is a reduced performance for in-focus signals at high spatial frequencies as well as the need to acquire more images for a single slice reconstruction.

    12. Can we generate ‘true’ superresolution images from simple instruments enhanced with machine learning-based algorithms?

    Yes. If ‘true’ superresolution is defined as above, i.e. surpassing the emission-only Abbe limit. In combination with machine learning (exploiting ‘prior knowledge’), stunning resolution and SNR gains can be achieved even processing standard widefield fluorescence images. Spatial light modulator-based illumination strategies are simple to adapt to existing set-ups, which can then enhance the resolution further. Yet there is also a significant danger of basing the resolution gain on machine learning alone: machine learning is geared towards producing results similar to what it has been trained on. Training data can be based on separate high-quality input or it can be driven by the same dataset, which is to be evaluated, in which case machine learning looks for common local patterns [33]. Both of these aspects provide the danger that rare patterns may be missed or misinterpreted by machine learning. DeepDream networks nicely illustrate this basic fear by their spooky results.

    13. Can research-grade super-resolution (SIM) microscopes be built cost-efficiently?

    Inexpensive/simple SIM as well as inexpensive ISM have recently been demonstrated in our laboratory [34]. Consumer-devices such as DMDs are a big help to this aim. Yet a problem may be to achieve a performance close to the theoretical limit since the tolerances and stability are increasingly more difficult to realize close to the combined excitation/emission Abbe limit.

    Data accessibility

    This article does not contain any additional data.

    Competing interests

    We declare we have no competing interests.

    Funding

    R.H. acknowledges support by the Collaborative Research Center SFB 1278 (PolyTarget, project C04) funded by the Deutsche Forschungsgemeinschaft.

    Footnotes

    One contribution of 11 to a Theo Murphy meeting issue ‘Super-resolution structured illumination microscopy (part 1)’.

    Published by the Royal Society. All rights reserved.