7  Adding AI to Hardware

An Introduction to Smart Microscopy

Authors

Affiliations

Victoria Augoustides

Biochemistry and Biophysics Department, University of North Carolina at Chapel Hill

Giorgio Tortarolo

Institute of Physics, École Polytechnique Fédérale de Lausanne

Alara Kiris

Institute of Physics, École Polytechnique Fédérale de Lausanne

Suliana Manley

Institute of Physics, École Polytechnique Fédérale de Lausanne

Wesley Legant

Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill

Victoria Augoustides, Giorgio Tortarolo, and Alara Kiris contributed equally to this chapter. Suliana Manley and Wesley Legant are co-corresponding authors.

7.1 Imaging the Events of Biological Systems

Microscopy experiments answer biological questions by measuring structures and dynamic processes in living systems. To collect useful data, the microscopist must decide where, when, how fast, and how long to image based on features and processes in the specimen. Traditionally, the microscopist identifies regions of interest (ROIs), monitors sample behavior, and adjusts acquisition settings using domain expertise and visual inspection. While effective for small-scale experiments, this manual workflow limits throughput, introduces operator-dependent variability, and can miss rare or rapidly occurring biological phenomena.

These challenges motivate the development of smart microscopy systems that can automatically recognize biologically relevant features and respond in real time. Automated decision-making can increase acquisition speed, improve reproducibility, and enable experiments that would be impractical for a human operator to perform manually. Recent advances in machine learning have substantially expanded the range of biological signatures that can be recognized during image acquisition, bringing microscopy closer to a paradigm in which imaging systems actively direct an experiment’s trajectory rather than simply record data.

In this chapter, we introduce the concepts, technologies, and challenges involved in developing AI to augment microscopy experiments in real-time. Our goals are to:

• Define biological objects and events as targets for image acquisition and analysis
• Describe the evolution of automated object detection from classical image processing to modern machine learning
• Explain the unique data, labeling, and modeling considerations associated with event detection and real-time performance
• Evaluate the opportunities, limitations, and reliability of AI-driven microscopy workflows

7.2 Framing the Event Detection Landscape

For an imaging system to recognize and respond to biological activity, it must first distinguish between objects and events:

• An object is an identifiable feature located within a two-dimensional image or three-dimensional volume, such as a cell, organelle, or protein assembly. Objects are primarily defined by spatial characteristics including size, shape, intensity, and location.
• An event, by contrast, represents a change or transition that unfolds over time with distinct start and end points. Examples include cell division, mitochondrial fission, receptor clustering, or transient calcium signaling.

Whereas object detection relies primarily on spatial information, event detection requires both spatial and temporal information because the defining characteristic of an event is its evolution over time (Figure 7.1). This distinction has important consequences for image acquisition. To detect an object, the microscope must resolve the object’s relevant spatial features. To detect an event, the microscope must additionally sample the event at sufficient temporal resolution and for sufficient duration to capture its dynamics.

Figure 7.1: Objects versus events, illustrated with mitochondria. Left: objects, individual mitochondria, are identified and color-coded. Right: an event, fission of the cyan mitochondrion. Arrowheads (3 s) mark the constriction preceding scission; the dashed box (4 s) highlights the event site. Capturing objects requires sufficient spatial resolution; capturing events additionally requires sufficient temporal resolution and duration to resolve their dynamics. Scale bar, 2 µm. Figure credit: raw image acquired by Tatjana Kleele.

The spatial scales relevant to event detection span many orders of magnitude, ranging from millimeter-scale tissue remodeling, through tens-of-micrometer whole-cell movements, down to nanometer-scale reorganization of protein complexes. Temporal scales are similarly diverse, extending from days for embryonic development, to hours for processes such as cell division and differentiation, and down to milliseconds for rapid phenomena such as calcium transients, ion-channel activity, and single-molecule receptor binding. Importantly, many biological events encompass multiple spatial and temporal scales. For example, molecular-scale signaling events may occur rapidly and subsequently give rise to slower changes in cell morphology (Figure 7.2). Such complexity underscores the advantages of tailoring image acquisition parameters to the biological process under investigation, a central objective of smart microscopy.

Figure 7.2: Spatial and temporal scales of biological events. Processes are positioned by characteristic feature size (nm–mm) and timescale (µs–days), spanning ion-channel opening, endocytosis, cell division, apoptosis, and zebrafish development. Created with BioRender. Figure credit: inspired by Fig. 2 of Daetwyler & Fiolka¹.

Once the relevant spatiotemporal scales of an event have been identified, an appropriate microscopy modality can be selected. Different imaging methods provide distinct trade-offs between spatial resolution, temporal resolution, imaging depth, and phototoxicity. Widefield, spinning-disk confocal, and light-sheet microscopy generally offer high temporal resolution suitable for capturing fast dynamics, while providing varying levels of volumetric imaging capability and moderate spatial resolution. In contrast, super-resolution techniques such as stimulated emission depletion (STED) microscopy achieve exceptional spatial detail, often at the expense of acquisition speed and increased phototoxicity². Label-free imaging approaches, although not discussed in detail here, provide complementary contrast based on intrinsic sample properties while minimizing photobleaching and phototoxicity³.

Regardless of the imaging modality used, biological events must produce detectable and consistent changes within the acquired data. Some events manifest as morphological transformations, including cell migration, membrane remodeling, organelle reorganization, or changes in protein-complex architecture. Other events are more readily observed using fluorescent biosensors that report biochemical states such as enzyme activity, ion concentrations, metabolite levels, or membrane potential. These processes can be measured through image-derived signals including changes in fluorescence intensity or fluorescence lifetime. In each case, the measured signal serves as a proxy that provides an indirect but interpretable readout of the underlying biological event.

By selecting imaging strategies that align with the spatial and temporal scales of interest, researchers can ensure that the essential features of biological objects and events are faithfully represented in the acquired data. The next challenge is to determine when and where specific events occur. In the following sections, we discuss how this challenge is addressed through computational approaches to object and event detection.

7.3 Foundations of Automated Object Detection in Smart Microscopy

Before modern deep-learning methods, object detection in microscopy was typically performed using a combination of image-processing and machine-learning operations. Although the specific algorithms varied, most approaches followed a common workflow: first, objects were isolated from the background⁴, second, quantitative features were extracted from each object, and third, these features were used to classify objects into biologically meaningful categories^5,6.

The goal of an object detection algorithm is to identify objects within an image and describe their spatial location and extent. Figure 7.3 A illustrates representative biological objects that vary widely in their spatial distribution, size, and morphology, including nuclei, cytoplasmic RNA and nucleoli, mitochondria, and other cellular structures. Object detection pipelines generally characterize each detected object by reporting:

1. the centroid coordinates of each object,
2. the coordinates of a bounding region, and/or
3. a class label describing the object’s identity⁷.

The first step in many classical detection pipelines is image segmentation, in which pixels are separated into objects and background (Figure 7.3 B). A common approach is thresholding, in which pixels above a certain intensity are assigned to the foreground and all others to the background, producing a binary mask. Intensity alone is rarely sufficient to cleanly separate objects, so additional morphological operations (e.g., erosion, dilation, or watershed) can then be applied to refine object boundaries. These operations, among others, can help distinguish neighboring objects that may otherwise appear to be merged. For an extended discussion of object segmentation, please see Chapter 9.

Once objects have been segmented, quantitative measurements can be extracted from the individual objects in the masked raw image or from the mask itself, often using tools such as CellProfiler⁵. Common features include intensity, area, perimeter, shape descriptors, texture measurements, and spatial relationships to surrounding structures⁸. Together, these measurements are aggregated into a feature vector that provides a numerical description of each object (Figure 7.3 B).

Figure 7.3: The process of creating a feature vector from image data. (A) Representative fluorescence images are acquired illustrating five different cellular compartments and structures: nucleus (DNA), cytoplasmic RNA/nucleoli (RNA), endoplasmic reticulum (ER), mitochondria, and the actin, Golgi apparatus, and plasma membrane (AGP). (B) Following image acquisition, individual cells and nuclei are segmented to define cellular boundaries, enabling extraction of quantitative features of the objects in (A). These measurements are assembled into high-dimensional morphological profiles, which can be processed using dimensionality reduction and clustering methods to compare cellular responses across experimental conditions and support downstream biological inference. Reproduced from Fig. 1 of Rohban, et al.⁹ under a CC BY license.

The final stage is classification, in which feature vectors are assigned to biologically meaningful categories^10–12. In simple applications, classification can be performed using manually defined rules. For example, objects exceeding a particular size threshold might be classified differently from smaller objects. However, manually setting cutoffs for each feature becomes increasingly difficult as the number of measured features grows. To address this challenge, machine-learning methods were adopted to learn classification rules directly from data. Algorithms, such as perceptrons¹³, support vector machines (SVMs)¹⁴, and other classifiers¹⁵, learn decision boundaries that separate groups of feature vectors within a multidimensional feature space. For example, a classifier may learn to distinguish normal cells from cancer cells based on combinations of morphological and intensity-based measurements in feature vectors (Figure 7.4). Compared with manually-specified rules, these learned decision boundaries can more effectively exploit high-dimensional feature sets and often generalize better to new data¹⁶.

Figure 7.4: Schematic of supervised learning with a classifier As additional labeled examples of normal (blue) and cancer (orange) cells are introduced during new training iterations, the classifier adjusts its decision boundary to improve class separation. Inspired by Rosenblatt¹³. Cell illustrations from NIAID NIH BioArt Source (items 358, 508, 376, and 82 at https://bioart.niaid.nih.gov/bioart/)

The incorporation of machine learning into classical object detection pipelines marked an important transition from automated image processing to automated decision-making. The initial use for machine-learning classifiers was primarily to categorize objects during post-acquisition analysis^17–19. As computational capabilities advanced, these same classification approaches began to inform microscope control in real time, enabling imaging systems to react to the objects they detected^20–22.

CellProfiler Analyst (Case Study 7.1) exemplifies the first stage of this progression. The platform automated segmentation, feature extraction, and classification, allowing large populations of cells to be quantitatively characterized using high-dimensional cytoprofiles. Machine learning improved the accuracy and scalability of object classification, but the resulting decisions remained confined to downstream data analysis after images had been acquired.

MicroPilot (Case Study 7.2) illustrates the next stage, in which object detection was integrated directly into the acquisition process²³. Rather than simply classifying objects after imaging, MicroPilot used detection results to guide data acquisition in real time. In a benchmark study, the system autonomously identified and imaged 232 mitotic cells during four nights of unattended operation. Achieving a comparable dataset through manual screening would have required more than a month of effort by an experienced microscopist. Case Study 7.2 demonstrated how automated object detection could move beyond analysis to actively control image acquisition, a key step toward modern smart microscopy.

A common feature of both case studies, and of most object detection approaches of the time, was that machine-learning algorithms operated and learned from feature vectors rather than image pixels. Consequently, their performance depended on the quality of the upstream segmentation and feature-extraction steps, processes that may discard biologically relevant information present in the native image structure. Furthermore, object localization and classification were treated as separate stages, reinforcing a distinction between identifying where an object is and determining what it is.

Advances in model architectures and computational power eventually enabled end-to-end learning, in which object localization and classification could be optimized jointly²⁴. By learning directly from image data, these approaches allowed spatial and semantic information to be captured simultaneously. Combined with the emergence of larger imaging datasets, these developments greatly expanded the capabilities of object detection and laid the foundation for modern deep-learning approaches, discussed in the next section.

Case Study 7.1: Cell Profiler Analyst

Semi-Automation using ML for Post-Acquisition “Cytoprofiling”

CellProfiler⁵ is a post-acquisition software for automated image segmentation, feature extraction, and downstream analysis of biological images. While there are many different versions of CellProfiler, this study presented an example of a semi-automated machine learning approach to object detection using CellProfiler Analyst^17,19,25. Feature vectors, here called cytoprofiles, were automatically extracted from raw image data during the detection step (Figure 7.5 A). Next, a human identified phenotypes of interest from the data that the software used to create classification rules for scoring cytoprofiles. To automate classification, the user trained a supervised machine learning model (specifically a boosting algorithm applied to an ensemble of shallow decision trees²⁶) for each imaging experiment (Figure 7.5 B). Once trained, the model could be applied to feature vectors from new experiments to score wells in high throughput experiments (Figure 7.5 C). Using this approach, the authors processed 40,000 fluorescence images capturing 8.3 million cells from which they detected and classified 14 distinct phenotypes, such as diverse distributions of actin and cell cycle states (Figure 7.5 D).

Figure 7.5: Overview of the semi-automated CellProfiler Analyst workflow for high-content image analysis (A) Following automated image segmentation and feature extraction in CellProfiler, each detected cell is represented by a high-dimensional feature vector, a cytoprofile. (B) A user interactively labels example cells, which are used to train a supervised machine learning classifier to recognize phenotypes of interest. (C) The trained classifier is then applied to score cells across large imaging datasets. (D) Representative examples of automatically classified phenotypes, including actin organization and mitotic cell-cycle states. Reproduced from Jones, et al.²⁵ with permission from PNAS: Panels A–C are from Fig. 1, and panel D is from Figs. 3 and 4.

Case Study 7.2: MicroPilot

Integration of Object Detection with Automated Assays

Rather than analyzing data after imaging was complete, MicroPilot used ML models to classify structures from low-resolution preview scans in real time (Figure 7.6 A). First, the low-resolution preview scan was segmented, and a feature vector was generated (Figure 7.6 B). Once a structure of interest was classified by the ML model, the software automatically triggered a predefined imaging assay (Figure 7.6 B). This could range from acquiring high-resolution time-lapse videos to initiating complex laser-based perturbations, such as photobleaching or protein ablation, all without user intervention.

Figure 7.6: Machine learning-guided autonomous microscopy using MicroPilot (A) Overview of the MicroPilot workflow: Low-resolution prescans are automatically analyzed to detect and classify cells of interest. When a target phenotype is identified, the software autonomously triggers a predefined imaging assay, such as high-resolution imaging or laser-based perturbation, before returning to the prescreening loop. (B) Left: A low-resolution preview image is segmented to identify individual cells. Middle: Quantitative morphological features are extracted from each segmented object. Right: A support vector machine (SVM) classifier assigns each cell to a phenotype class. These classifications are then used to make real-time decisions that control subsequent microscope acquisition. Reproduced from Fig. 1 of Conrad et al.²³ reproduced with permission from SNCSC.

7.4 Deep Learning for Object and Event Detection

The transition toward end-to-end learning began with networks capable of learning features directly from images, starting with convolutional neural networks (CNNs) and more recently featuring transformer architectures. Although CNNs were originally developed in the 1980s, their widespread adoption was catalyzed by the ImageNet Large Scale Visual Recognition Challenge in 2010, which introduced a large, standardized dataset and robust evaluation metrics^27,28.

Early datasets like ImageNet typically included a single centered object per image, and over time, they evolved to include multiple objects per image. Therefore, models such as AlexNet²⁹ and ResNet³⁰ were first only applied to single-object classification and localization tasks (Figure 7.7 A). Later, these frameworks were extended to classify and localize many objects to perform object detection.

Figure 7.7: Progression of object detection architectures (A) Single object tasks such as image classification and localization. (B) Multi-object, multi-step detection pipelines, including sliding-window and region-proposal approaches, which separate object localization from classification. (C) Multi-object, single-shot detectors that jointly predict object locations, confidence scores, and class labels in a single end-to-end model. Figure credit: Panels A-B images sourced from Pexels under CC0; Panel C was reprinted from Fig. 2 of³¹ with permission from IEEE Proceedings.

7.4.1 Evolution of Object Detection

Multi-object, multi-step detection comprises three key steps: candidate region proposal, classification, and refining classified regions. The simplest way to profile an image for candidates is to iteratively scan the image with a sliding window^32–35, and then evaluate each window with a classifier (Figure 7.7 B). A more efficient approach came from the R-CNN series of models (2014-2017) that instead proposed regions in an image for classification (Figure 7.7 B)^24,36,37.

Sliding window and region-based models were succeeded by single-shot detection approaches, where detection and classification occur together in a single forward pass (Figure 7.7 C). Some of the most widely used algorithms are the You Only Look Once (YOLO) models, which treat object localization and classification as an optimization problem across a fixed grid^31,38–41. Although YOLO performs classification and localization jointly, the rate limiting steps of YOLO models occur in post-processing the proposed bounding boxes⁴².

These CNN-based models are well-suited for detecting small, well-separated features, such as puncta, nuclei, or other localized signals in biological images. However, CNNs can struggle to capture broader spatial relationships such as multicellular phenotypes within a tissue. Transformer-based models⁴³, including natural language processing models, process entire inputs at once via self-attention. For images, the Vision Transformer (ViT)⁴⁴, can process an entire image at once unlike CNNs. Similarly to extending early classification networks, detection models like DEtection TRansformer (DETR)⁴⁵ and RT-DETR⁴⁶ adapted the transformer architecture for object detection allowing these models to infer complex spatial relationships without relying on region-proposals, grids, or post-processing steps.

NoteTransformers

Due to the complexity of transformers, we direct the interested reader to the following video resources:

• Transformer Neural Networks, ChatGPT’s foundation, Clearly Explained!!!
• Vision Transformers - The big picture of how and why it works so well

7.4.2 Extending Object Detection to Events

Most object detection models, whether region-based, one-shot, or transformer-based, are designed to operate on individual two-dimensional (2D) images, treating each frame independently. In contrast, events unfold over time and require models that can reason across input sequences of images, such as a 2D image + time (2D+t) stack, rather than individual frames. Broadly, there are several strategies for detecting events from sequential image formats:

1. Decoupled Object Detection and Temporal Linking: Apply an ML object detector to each frame of a time stack independently, then use post-processing or a feedback loop to link detections over time into events⁴⁷.
2. Sequential Detection and Temporal ML Model: Detect objects in each frame and incorporate a sequence model (such as a recurrent neural network or long short-term memory ) to capture temporal dependencies and classify dynamic transitions^48,49.
3. Simultaneous Spatiotemporal Detection: Feed multiple frames of a 2D+t series into the model simultaneously allowing the network to directly output temporally aware predictions such as 2D+t bounding boxes or event labels⁵⁰.

It can be challenging to adapt ML approaches that are optimized around natural image data which is fundamentally different from microscopy images⁵¹. Biological signals, especially fluorescent signals, are often very sparse against a homogeneous background. Additionally, biological processes are often rare, asynchronous, and non-repetitive, unlike typical videos such as cars moving through traffic. In the next section, we delineate considerations for implementing event detection from other ML-based image algorithms in microscopy.

7.5 Building Object and Event Detection Models

7.5.1 Ground Truth and Labeling

The performance of a machine-learning model depends fundamentally on the quality of the examples used during training. Before a model can learn to detect biological objects or events, the experimental objective must be translated into a set of labeled examples, commonly referred to as ground truth. These annotations define the outputs that the model should predict from the input imaging data, such as the location of an object, its identity, the occurrence of an event, or the timing of a biological transition. For more information on designing model outputs, see Chapter 2 and Chapter 4.

In object detection tasks, labels typically describe the spatial position and identity of structures of interest. For example, annotations may indicate the location of a centrosome, or a cell or organelle boundary. Event detection introduces an additional temporal dimension: annotations must specify when an event begins and ends in addition to identifying what and where the object is. For example, labeling mitochondrial fission requires determining the precise frame at which a single mitochondrion divides into two daughter structures (Figure 7.1)^20,49.

Creating ground-truth annotations is often the most time-consuming stage of model development. High-quality datasets may require hundreds or thousands of manually annotated examples, and it’s important that within that volume, there is sufficient phenotypic diversity. For object detection, the annotator needs to capture diverse object appearances and experimental conditions. In contrast, event detection is often limited by the scarcity of events themselves and so the challenge is frequently obtaining sufficient examples of rare events. Biological imaging further complicates creating labeled data because annotations frequently require unbiased expert interpretation. Annotators must distinguish signal from noise, recognize subtle morphological features, and apply consistent criteria across diverse experimental conditions.

Furthermore, biological ground truth is rarely absolute and variability in the ground truth is multifaceted. Object boundaries may be diffuse, and event definitions may vary between experts. Differences in imaging conditions, sample preparation, fluorophore expression, and biological variability can further reduce annotation consistency. As a result, training datasets often contain a degree of uncertainty that ultimately influences both model performance and evaluation.

7.5.2 Unique Challenges of Event Detection

Many of these general challenges are amplified in event detection. Because events unfold over time, annotators must identify not only whether an event occurred, but also its precise temporal boundaries. This requirement introduces ambiguity when biological transitions are gradual or poorly defined. For example, in mitochondrial fission, multiple experts may disagree on the exact timestep when constriction becomes visible or when membrane scission occurs. Temporal sampling also plays a critical role; if imaging is performed too slowly, key transitions may be missed entirely.

Event detection datasets can exhibit severe class imbalance because biologically relevant events may occupy only a small fraction of the recorded frames. In addition, event annotations frequently depend on information distributed across multiple imaging channels or modalities, requiring annotators to interpret spatial, temporal, and molecular context simultaneously. These challenges make event labeling substantially more demanding than object labeling and often motivate the use of interactive annotation tools^5,6,52,53, weak supervision⁵⁴, or human-in-the-loop workflows⁵⁵ to improve efficiency and consistency.

7.5.3 Training Models

Once a ground-truth dataset has been established, the next step is to train a model that can accurately predict object locations, identities, or event occurrences (see also Chapter 4 and Chapter 9). Training proceeds as an iterative loop: the model makes a prediction on the image, compares that prediction to the corresponding ground truth annotation and adjusts its internal parameters to minimize the discrepancy between them. These updates are performed through backpropagation⁵⁶, an algorithm that calculates how much each individual parameter of the model contributes to the prediction error and updates the value(s) accordingly. Model optimization is guided by one or more loss functions, which quantify prediction errors in tasks such as object localization, classification, or object presence (Table 7.1).

To evaluate how well the model generalizes beyond its training data, the ground truth dataset pool is typically divided into separate training, validation, and test sets. The training set is used to optimize the model parameters, while the validation set is used periodically after each pass through the training dataset to assess performance on unseen data. Validation results can help guide decisions such as when to stop training or how to adjust model settings between training runs. Because these decisions are influenced by the validation dataset’s performance, a third, independent test set is reserved for a final evaluation. Model performance is then evaluated using metrics such as precision and recall that measure how well the predictions match ground-truth labels on the test data¹⁶.

Table 7.1: Examples of loss functions in object detection models

Type	Description	Examples
Localization Losses	Measure how close the predicted bounding box is to the ground truth box	L1 Loss L2 Loss IoU Loss57,58
Classification Losses	Classify each detected object into the correct class	Cross Entropy Loss Focal Loss59
Objectness Losses	Predict whether an object exists in a box	Binary Cross Entropy Focal Binary Cross Entropy

Model development typically proceeds in two phases: exploration and optimization. During the exploration phase, the goal is to determine whether the model can learn to solve the biological problem using the available data. Researchers often begin with a small representative dataset and a well-established architecture, such as U-Net⁶⁰, Mask R-CNN⁶¹, or YOLO³¹. Short training runs on limited datasets can reveal whether the model converges, produces biologically meaningful predictions, or exhibits obvious signs of overfitting, where the model fits the training data but is unable to generalize to an unseen dataset. These early experiments help identify potential obstacles before substantial effort is invested in optimization.

7.5.4 Refining and Optimizing Models

Once feasibility has been established, attention shifts toward improving model performance and robustness. In many cases, the most effective strategy is to expand and improve the annotated dataset, as model performance is fundamentally constrained by the quality and diversity of the training examples. Additional annotations, refined labels, and data collected under different experimental conditions can all improve generalization. When annotated data are scarce, simulated datasets, data augmentation, or pre-trained models may also be used to bootstrap learning and reduce annotation requirements.

As model development progresses, increasingly sophisticated architectures may be introduced. Three-dimensional convolutional networks can better exploit volumetric image data, while recurrent or transformer-based architectures can incorporate temporal information required for event detection. These approaches enable models to capture complex spatial and temporal relationships that are difficult to represent using classical object detection pipelines.

Model optimization is often constrained by computational resources. Training deep-learning models on high-resolution, volumetric, or time-lapse microscopy datasets can require hours or days of computation, limiting the speed at which new hypotheses can be tested. Consequently, model development is typically an iterative process in which improvements to data quality, model architecture, and training strategy are evaluated over multiple training cycles. As datasets and models increase in scale, researchers may transition from single-workstation training to high-performance computing or cloud-based resources capable of supporting large-scale experiments.

Ultimately, successful model development depends on balancing biological relevance, annotation quality, model complexity, and computational cost. These considerations become particularly important for event detection, where temporal dynamics, class imbalance, and annotation uncertainty introduce additional challenges beyond those encountered in conventional object detection tasks.

7.6 Real-Time Event Detection in Smart Microscopy

In smart microscopy workflows, event detection can be performed either after image acquisition or during image acquisition. These two approaches, often referred to as a-posteriori and real-time analysis respectively, differ not only in when inference occurs but also in how detection influences the experiment itself.

In a-posteriori analysis, events are identified after the complete dataset has been acquired and stored. This approach allows computationally intensive algorithms to be applied without constraints imposed by the instrument’s acquisition speed and is well suited for retrospective analysis, hypothesis generation, and machine learning model development. However, because detection occurs after the experiment is complete, the results cannot influence image acquisition or experimental conditions.

Real-time analysis, in contrast, processes image data as it is acquired. Rather than serving solely as a downstream analysis tool, event detection becomes part of the imaging workflow itself. This approach is particularly valuable when phototoxicity, storage requirements, or instrument throughput limit the feasibility of continuous high-resolution imaging. Real-time inference has strict time constraints and therefore requires efficient algorithms capable of producing reliable predictions from incomplete and continuously evolving datasets.

The principal advantage of real-time event detection is that it enables adaptive microscopy. By identifying biologically relevant events as they occur, imaging parameters can be adjusted dynamically to focus acquisition effort on the most informative regions, objects, or time periods. This strategy reduces unnecessary sample exposure, minimizes photobleaching and photodamage, and decreases the collection of uninformative data. As a result, experiments can often be performed for longer durations while generating smaller, more information-rich datasets.

Hybrid event-driven acquisition (Case Study 7.3) illustrates the benefits of adaptive microscopy for capturing rare and transient events while minimizing unnecessary fluorescence exposure⁴⁹. This framework monitored samples using a low-phototoxicity imaging modality and selectively triggered fluorescence imaging only when an event was detected, substantially increasing the number of cells that survived long-term imaging compared with continuous fluorescence acquisition.

Real-time detection also enables closed-loop experimentation. Detected events can trigger downstream actions such as object tracking, high-resolution imaging, experimental perturbations, or automated classification. More broadly, the microscope can use incoming observations to guide future acquisition decisions, transforming image acquisition from a passive recording process into an active, information-driven measurement strategy. This capability is central to the concept of smart microscopy.

Real-time event detection remains challenging because models must balance detection accuracy against computational latency, often while operating under limited hardware resources and in the presence of noisy biological data. Strategies that perform well for one imaging modality or biological system may not generalize readily to others, making robust deployment difficult. Furthermore, uncertainty estimates are rarely incorporated into acquisition decisions, despite their potential value for reducing false positives and missed events. Addressing these challenges will require continued advances in machine learning, microscope control software, and hardware–software integration.

Automated Early Generalizing Protein Aggregation ONset (Case Study 7.4) addresses two important barriers to broader adoption of real-time smart microscopy: computational efficiency and generalizability. The authors showed that deep learning models could maintain high event detection accuracy using substantially fewer temporal and spatial inputs, making real-time inference feasible on more modest computational hardware. They also demonstrated that the framework generalized across different microscope platforms and imaging modalities without transfer learning, illustrating how event-driven microscopy can become more accessible beyond highly specialized experimental systems.

Together, these developments and challenges illustrate the evolution of event detection from a retrospective analytical tool to a mechanism for real-time experimental control. By enabling microscopes to respond autonomously to biological processes as they unfold, event-driven microscopy represents a major step toward intelligent and adaptive imaging systems.

Case Study 7.3: Hybrid Event Driven Acquisition (EDA)

Smart hybrid microscopy for cell-friendly detection of rare events

Hybrid-EDA⁴⁹ is a real-time event detection and acquisition tool that addresses a fundamental trade-off in fluorescence microscopy: obtaining sufficient signal while minimizing photodamage caused by prolonged accumulation. This framework separates surveillance from event capture by monitoring the sample continuously under phase-contrast, a label-free imaging modality with minimal phototoxicity. When a neural network detects an event from subtle structural cues in the phase-contrast images, fluorescence imaging is triggered (Figure 7.8 A). This approach dramatically reduces phototoxic burden relative to continuous fluorescence surveillance (Figure 7.8 B).

The authors demonstrate this strategy by detecting mitochondrial division and organelle contacts. Mitochondrial division presents a detection challenge because phase contrast is a label-free modality that captures the whole cellular landscape alongside mitochondria, so the subtle structural cues preceding a fission event can easily be lost among unrelated features. To address this, the authors equipped a U-Net with temporal memory across frames, allowing it to track the evolution of a candidate site rather than evaluating each frame in isolation. Hybrid-EDA captured fast, fragile physiological signals that continuous fluorescence imaging would struggle to resolve without accelerating photodamage. As one example, membrane potential dynamics were visualized at a site of mitochondrial constriction using the potentiometric dye TMRE (Figure 7.9).

Figure 7.8: Hybrid-EDA balances phototoxicity and event capture in live-cell imaging. (A) Logic of hybrid-EDA: a neural network continuously evaluates phase-contrast frames and triggers a switch to combined phase/fluorescence imaging only when it detects an event, then reverts to phase-only monitoring. (B) Cell viability over ~30 minutes under continuous fluorescence, phase-contrast, and no illumination. Fluorescence causes rapid loss of viability, while phase-contrast remains comparable to the unilluminated control. Taken from Fig. 1 of Stepp, et al.⁴⁹ under a CC BY license.

Figure 7.9: Correlative imaging reveals a localized membrane potential drop at a mitochondrial constriction Top: phase-contrast time series showing a constriction site (arrowhead) that precedes division. Middle: neural network event score, which exceeds threshold at the constriction and triggers fluorescence acquisition (active period, magenta). Bottom: TMRE fluorescence over the same period, showing a transient, spatially confined loss of membrane potential (outlined in cyan) that remains restricted to one side of the constriction before recovering. Reproduced from Fig. 3 of Stepp, et al.⁴⁹ under a CC BY license.

Case Study 7.4: Automated Early Generalizing Protein Aggregation ONset (AEGON)

Performing longitudinal protein aggregation detection with AEGON

The AEGON framework⁶² uses fast fluorescence imaging to identify candidate biological events for a slower and more experimentally demanding Brillouin microscopy assay, a scattering-based technique for measuring the mechanical properties of biological specimens⁶³. The authors demonstrate the framework by identifying the onset of Httex1 protein aggregation in a well-characterized cellular model of Huntington’s disease. Early detection of these events is a challenge for conventional Brillouin microscopy due to its limited throughput. First, the microscope rapidly acquires fluorescence images across a predefined grid of stage positions (Figure 7.10 A), and a deep learning model classifies each image as either an aggregation event or a non-event (Figure 7.10 B). Locations predicted to contain an event are then revisited using optimized Brillouin acquisition settings for longitudinal monitoring (Figure 7.10 C).

A major contribution of this work was demonstrating that accurate real-time event detection could be achieved with substantially less input data, reducing the computational demands of closed-loop microscopy. First, the authors trained models on 4D image stacks and systematically reduced the dimensionality of input data by testing fewer timepoints (Figure 7.11 A). Next, they reduced the number of z-planes input to the ViViT model (Figure 7.11 B). With only a single time point and single z plane, it maintained 86% accuracy as compared to the 91% accuracy it achieved when provided with a single time point and eight z-planes. These results show that high-performance event detection can be maintained under a lower computational budget. The optimized model was then used for real-time event detection of fluorescently tagged Httex1 aggregation (Figure 7.11 C). This adaptive strategy enabled longitudinal monitoring of Httex1 aggregation using brightfield and Brillouin microscopy triggered after the initial detection event (Figure 7.11 D).

Figure 7.10: Self-driving microscopy workflow for Automated Early Generalizing Protein Aggregation ONset (AEGON) detection (A) The microscope performs low-cost imaging while scanning a spatial grid of stage positions. (B) Each acquired image is analyzed in real time by a previously trained neural network and classified as either an aggregation onset event or a non-event. Candidate event locations are added to a list and immediately trigger optimized acquisition. (C) The microscope automatically revisits detected events using optimized imaging parameters, including imaging modality, exposure time, laser power, z-stack settings, and timelapse frequency, to capture protein aggregation dynamics. Reproduced from Fig. 1 of Ibrahim, et. al⁶² under a CC BY license.

Figure 7.11: Reducing the computational requirements of AEGON while preserving real-time event detection performance. (A) Classification accuracy of the two deep learning architectures as the number of temporal frames used for training and prediction is varied (B) Classification accuracy of the ViViT model using a single time point with either one or eight z-planes (C) Illustration of the AEGON framework using real-time event detection to trigger optimized microscope acquisition. (D) Representative longitudinal imaging experiment. Following event detection (red arrow), the microscope automatically switches to optimized acquisition settings to monitor protein aggregation dynamics using fluorescence, brightfield, and Brillouin microscopy over time (yellow and black arrows). Reproduced from Fig. 3 of Ibrahim, et. al⁶² under a CC BY license.

7.7 Challenges and Future Directions in Smart Microscopy

Despite its promise, smart microscopy remains subject to important limitations. The performance of object and event detection models depends strongly on the quality and representativeness of their training data. Models may fail when presented with unfamiliar biological conditions, imaging modalities, or rare phenotypes, and biases in annotated datasets can influence both detection outcomes and subsequent acquisition decisions. Furthermore, adaptive imaging strategies can alter experimental conditions themselves, for example, by exposing different samples to different cumulative light doses, requiring careful experimental controls and validation.

Practical challenges also remain. Event detection often requires large, expertly annotated datasets. Integrating machine-learning models with microscope hardware and control software introduces additional complexity, while increasingly capable microscopes generate data volumes that can be difficult to store, process, and interpret.

At the same time, the opportunities afforded by smart microscopy are substantial. By coupling biological event recognition directly to image acquisition, microscopes can focus on the most informative regions, objects, and time periods while reducing data collection. Advanced instruments already combine multiple imaging modalities, adaptive acquisition strategies, or increasingly generalizable machine-learning models to detect a broader range of biological phenomena^{1,20,47,49,62,64}. More broadly, smart microscopy represents a shift from passive image collection toward active, information-driven experimentation, in which microscopes not only record biological processes but also respond to them in real time.

1.

Daetwyler, S. & Fiolka, R. P. Light-sheets and smart microscopy, an exciting future is dawning. Communications Biology 6, (2023).

2.

Wu, Y. & Shroff, H. Multiscale fluorescence imaging of living samples. Histochemistry and Cell Biology 158, 301–323 (2022).

3.

Shaked, N. T., Boppart, S. A., Wang, L. V. & Popp, J. Label-free biomedical optical imaging. Nature Photonics 17, 1031–1041 (2023).

4.

Meijering, E. Cell segmentation: 50 years down the road [life sciences]. IEEE Signal Processing Magazine 29, 140–145 (2012).

5.

Carpenter, A. E. et al. CellProfiler: Image analysis software for identifying and quantifying cell phenotypes. Genome Biology 7, (2006).

6.

Berg, S. et al. Ilastik: Interactive machine learning for (bio)image analysis. Nat. Methods 16, 1226–1232 (2019).

7.

Neha, F., Bhati, D., Shukla, D. K. & Amiruzzaman, M. From classical techniques to convolution-based models: A review of object detection algorithms. (2024).

8.

Huang, K. & Murphy, R. F. From quantitative microscopy to automated image understanding. Journal of Biomedical Optics 9, 893 (2004).

9.

Rohban, M. H. et al. Systematic morphological profiling of human gene and allele function via cell painting. eLife 6, (2017).

10.

Boland, M. V. & Murphy, R. F. A neural network classifier capable of recognizing the patterns of all major subcellular structures in fluorescence microscope images of HeLa cells. Bioinformatics 17, 1213–1223 (2001).

11.

Murphy, R. F., Velliste, M. & Porreca, G. Robust classification of subcellular location patterns in fluorescence microscope images. in Proceedings of the 12th IEEE workshop on neural networks for signal processing 67–76 (2022). doi:10.1109/nnsp.2002.1030018.

12.

Chen, X., Velliste, M. & Murphy, R. F. Automated interpretation of subcellular patterns in fluorescence microscope images for location proteomics. Cytometry Part A 69A, 631–640 (2006).

13.

Rosenblatt, F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408 (1958).

14.

Cortes, C. & Vapnik, V. Support-vector networks. Machine Learning 20, 273–297 (1995).

15.

Yuan, G.-X., Ho, C.-H. & Lin, C.-J. Recent advances of large-scale linear classification. Proceedings of the IEEE 100, 2584–2603 (2012).

16.

Bishop, C. Pattern Recognition and Machine Learning. (Springer, 2006).

17.

Jones, T. R. et al. CellProfiler analyst: Data exploration and analysis software for complex image-based screens. BMC Bioinformatics 9, (2008).

18.

Shariff, A., Kangas, J., Coelho, L. P., Quinn, S. & Murphy, R. F. Automated image analysis for high-content screening and analysis. SLAS Discovery 15, 726–734 (2010).

19.

Dao, D. et al. CellProfiler analyst: Interactive data exploration, analysis and classification of large biological image sets. Bioinformatics 32, 3210–3212 (2016).

20.

Mahecic, D. et al. Event-driven acquisition for content-enriched microscopy. Nature Methods 19, 1262–1267 (2022).

21.

Alvelid, J., Damenti, M., Sgattoni, C. & Testa, I. Event-triggered STED imaging. Nature Methods 19, 1268–1275 (2022).

22.

Morgado, L., Gómez-de-Mariscal, E., Heil, H. S. & Henriques, R. The rise of data-driven microscopy powered by machine learning. Journal of Microscopy 295, 85–92 (2024).

23.

Conrad, C. et al. Micropilot: Automation of fluorescence microscopy–based imaging for systems biology. Nature Methods 8, 246–249 (2011).

24.

Girshick, R., Donahue, J., Darrell, T. & Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. in 2014 IEEE conference on computer vision and pattern recognition 580–587 (2014). doi:10.1109/cvpr.2014.81.

25.

Jones, T. R. et al. Scoring diverse cellular morphologies in image-based screens with iterative feedback and machine learning. Proceedings of the National Academy of Sciences 106, 1826–1831 (2009).

26.

Friedman, J., Hastie, T. & Tibshirani, R. Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics 28, (2000).

27.

Deng, J. et al. ImageNet: A large-scale hierarchical image database. in 2009 IEEE conference on computer vision and pattern recognition 248–255 (2009). doi:10.1109/cvpr.2009.5206848.

28.

Russakovsky, O. et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 211–252 (2015).

29.

Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. in Advances in Neural Information Processing Systems vol. 25 (Curran Associates, Inc., 2012).

30.

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. in 2016 IEEE conference on computer vision and pattern recognition (CVPR) 770–778 (2016). doi:10.1109/cvpr.2016.90.

31.

Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: Unified, real-time object detection. in 2016 IEEE conference on computer vision and pattern recognition (CVPR) 779–788 (2016). doi:10.1109/cvpr.2016.91.

32.

Sung, K.-K. & Poggio, T. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 39–51 (1998).

33.

Rowley, H. A., Baluja, S. & Kanade, T. Neural network-based face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 23–38 (1998).

34.

Viola, P. & Jones, M. Rapid object detection using a boosted cascade of simple features. in Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 vol. 1 I-511-I-518.

35.

Cireşan, D. C., Giusti, A., Gambardella, L. M. & Schmidhuber, J. Mitosis detection in breast cancer histology images with deep neural networks. in Lecture notes in computer science 411–418 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2013). doi:10.1007/978-3-642-40763-5_51.

36.

Girshick, R. Fast R-CNN. in 2015 IEEE international conference on computer vision (ICCV) 1440–1448 (2015). doi:10.1109/iccv.2015.169.

37.

Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 1137–1149 (2017).

38.

Redmon, J. & Farhadi, A. YOLOv3: An incremental improvement. (2018).

39.

Bochkovskiy, A., Wang, C.-Y. & Liao, H.-Y. M. YOLOv4: Optimal speed and accuracy of object detection. (2020).

40.

Tian, Y., Ye, Q. & Doermann, D. YOLOv12: Attention-centric real-time object detectors. (2025).

41.

Jocher, G. et al. Ultralytics YOLO26: Unified real-time end-to-end vision models. (2026).

42.

Wang, A. et al. YOLOv10: Real-time end-to-end object detection. in Advances in neural information processing systems 37 107984–108011 (2024). doi:10.52202/079017-3429.

43.

Vaswani, A. et al. Attention is All you Need. in Advances in Neural Information Processing Systems vol. 30 (Curran Associates, Inc., 2017).

44.

Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. in International conference on learning representations (ICLR) (2021).

45.

Carion, N. et al. End-to-end object detection with transformers. in Lecture notes in computer science 213–229 (Springer International Publishing, Cham, 2020). doi:10.1007/978-3-030-58452-8_13.

46.

Zhao, Y. et al. DETRs beat YOLOs on real-time object detection. in 2024 IEEE/CVF conference on computer vision and pattern recognition (CVPR) 16965–16974 (2024). doi:10.1109/cvpr52733.2024.01605.

47.

Shi, Y. et al. Smart lattice light-sheet microscopy for imaging rare and complex cellular events. Nature Methods 21, 301–310 (2024).

48.

Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Computation 9, 1735–1780 (1997).

49.

Stepp, W. L. et al. Smart hybrid microscopy for cell-friendly detection of rare events. Nature Communications 17, (2026).

50.

Nie, W.-Z., Li, W.-H., Liu, A.-A., Hao, T. & Su, Y.-T. 3D convolutional networks-based mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations. in 2016 IEEE conference on computer vision and pattern recognition workshops (CVPRW) 1359–1366 (2016). doi:10.1109/cvprw.2016.171.

51.

Meijering, E. A bird’s-eye view of deep learning in bioimage analysis. Computational and Structural Biotechnology Journal 18, 2312–2325 (2020).

52.

Sofroniew, N. et al. Napari: A multi-dimensional image viewer for python. Zenodo https://doi.org/10.5281/zenodo.21361474 (2026).

53.

Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).

54.

Zhou, Z.-H. A brief introduction to weakly supervised learning. National Science Review 5, 44–53 (2018).

55.

Griebel, M. et al. Deep learning-enabled segmentation of ambiguous bioimages with deepflash2. Nature Communications 14, (2023).

56.

Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).

57.

Yu, J., Jiang, Y., Wang, Z., Cao, Z. & Huang, T. UnitBox: An advanced object detection network. in Proceedings of the 24th ACM international conference on multimedia 516–520 (2016). doi:10.1145/2964284.2967274.

58.

Rezatofighi, H. et al. Generalized intersection over union: A metric and a loss for bounding box regression. in 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR) 658–666 (2019). doi:10.1109/cvpr.2019.00075.

59.

Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 318–327 (2020).

60.

Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional networks for biomedical image segmentation. in Medical image computing and computer-assisted intervention – MICCAI 2015 (eds. Navab, N., Hornegger, J., Wells, W. M. & Frangi, A. F.) 234–241 (Springer International Publishing, Cham, 2015). doi:10.1007/978-3-319-24574-4_28.

61.

He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. in 2017 IEEE international conference on computer vision (ICCV) 2980–2988 (2017). doi:10.1109/iccv.2017.322.

62.

Ibrahim, K. A. et al. Self-driving microscopy detects the onset of protein aggregation and enables intelligent brillouin imaging. Nature Communications 16, (2025).

63.

Hockenberry, M. A. & Legant, W. R. Cells in the mechanical spotlight. Biophysical Journal 121, 3571–3572 (2022).

64.

Daetwyler, S. et al. Imaging of cellular dynamics from a whole organism to subcellular scale with self-driving, multiscale microscopy. Nature Methods 22, 569–578 (2025).