2  AI in a Nutshell

An Introduction to Artificial Intelligence

Author

Affiliation

Stephan Saalfeld

HHMI Janelia Research Campus

Empirical science relies on the analysis and interpretation of large amounts of diverse raw data. In the life sciences, microscopy has become one of the largest sources of such data. A single light-sheet experiment can produce terabytes of volumetric time-lapse data; high-content screening campaigns image millions of cells across thousands of conditions; electron microscopy reconstructs tissue at nanometer-scale resolution across volumes approaching a cubic millimeter, yielding more than a petabyte of images from a single sample^1,2.

A human expert might trace a few hundred neurons or label several thousand cells, but biological questions increasingly demand analysis that is both orders of magnitude larger and more complete. Even where each individual decision is easy, the sheer number of decisions puts manual analysis out of reach.

Artificial Intelligence (AI) methods promise to address this challenge, and they offer three distinct things. The first is speed: where a trained annotator might outline the cells in a hundred images per day, a well-designed AI model does the same in seconds, and many copies of it can run in parallel. The second is consistency: human observers vary in their judgments and tire over long sessions, whereas a model applies the same criteria uniformly. The third is sensitivity to structure that escapes human perception. Given only transmitted-light images, for instance, a model can predict where fluorescently labeled organelles would appear, reading information from an unstained sample as if it had been stained³.

But uniform is not the same as correct. Every AI method rests on two things: a computational structure that somebody designed and a set of examples it learned from. Choices on either side shape what the system does, and mistakes produce failures that are often distinctly non-human: systematic rather than sporadic, confident rather than hesitant, and capable of misleading a scientist in ways no tiring annotator would.

This chapter builds the vocabulary needed to make informed decisions: what terms like machine learning, deep learning, and supervised actually mean, what each family of methods asks of the user in return for what it does, and which failure modes each is prone to.

Two tasks recur here and throughout the book, because a great many microscopy applications reduce to one or the other: restoring images that the microscope could not record cleanly (Chapter 6), and segmenting them into the objects you want to measure (Chapter 9).

2.1 What is Artificial Intelligence?

The term Artificial Intelligence has been attached to systems as different as a chess engine, a spam filter, and a chatbot. For someone choosing a method, the most useful thing that separates them is where the division of labor falls: which parts of the solution a person designs, and which parts the machine derives from data. In the seventy years since the term was coined⁴, that boundary has moved steadily in one direction—away from the designer and into the learned system. The three families below are stops along that path:

1. Rule-based systems: you design everything. The quantities to measure, the algorithm that acts on them, and every parameter value are chosen by you, and you adjust them for a new sample. This class includes conversational systems like ELIZA⁵, expert systems for medical diagnosis like MYCIN⁶, web search ranked by PageRank⁷, the behavior of non-player characters in video games, image features designed to match views of the same scene across viewpoint changes⁸, and the thresholding, morphological operations, and watershed transforms that carried quantitative microscopy for decades⁹. Nothing about such a system is mysterious: every decision follows from a known rule, so a failure can be traced to the rule that produced it and repaired by editing that rule. The price is that an expert has to anticipate every scenario in advance, as well as design and test the rules to address each and every scenario.

2. Machine learning: you design the structure, and examples fix the parameters. Rather than defining everything, you specify a family of functions with free parameters and let an optimization procedure choose the parameters that reproduce the expected outcome on a set of examples. Classic methods learn only the decision, on top of hand-designed features. Support vector machines¹⁰, for example, learn to sort cells into classes from their measured size, shape, and brightness. Boosting trains a sequence of simple decision trees, each one correcting the errors its predecessors left behind^11,12. This boosting technique can be used to score cell phenotypes in CellProfiler Analyst (Tip 7.1) and, when paired with box filters that can be computed extremely fast, can make face detection real-time¹³. Random forests instead train many trees independently, each on a random subset of the examples, and let them vote¹⁴. They drive interactive pixel classification in tools like ilastik¹⁵ (Chapter 8). Deep learning pushes the boundary one level deeper: the features themselves are learned, together with the decision that uses them (Section 2.1.1). The principle was demonstrated on handwritten digits in the 1990s¹⁶, but only in 2012, when labeled data and GPUs caught up, did it decisively outperform hand-designed pipelines on large-scale image recognition¹⁷.

   What stays on the designer’s side is the architecture, the training objective, and the examples—annotated by hand, produced by a second measurement, as when one imaging modality is used to predict another³, or supplied by the data itself, as when a model learns to reconstruct image regions that were masked out¹⁸. Interpretability erodes along the same step. As long as the features are engineered, you can easily ask which measurement drove a call, because the measurements have names: area, eccentricity, mean intensity in the second channel. Once the features are learned, they do not trivially decompose into something a person would recognize. In practice, you find out what such a model does by watching what it does, although a growing body of interpretability research recovers a good deal of its reasoning (Section 2.4.3).

3. Foundation models: you use or adapt what somebody else has learned. These are machine learning models too; what sets them apart is scale and how you use them. A foundation model is pre-trained on an enormous collection of images or text, usually for a task that needs no annotation because the data supplies its own answers: predict the next word of a sentence, or fill in a masked-out patch of an image¹⁸. To do this well, the model has to learn how such data are structured in general. What it retains is a general-purpose representation, not the answer to any particular question, and with enough data this representation even replaces assumptions that designers used to build into the architecture¹⁹. Adapting the model to a real task reuses all of this instead of rebuilding it, whether by fine-tuning on a modest set of labeled examples or by training it to follow instructions²⁰. That is why a problem that would otherwise demand tens of thousands of annotations can often be solved with a few hundred. A single pre-training run serves many different tasks. Segment Anything (SAM), for example, outlines structures it was never explicitly taught: objects you point to with a few clicks or a box²¹, mark once in a video to have them followed through every frame²², or simply name in a short phrase²³.

   The catch is that a model can only absorb the structure of the data it was trained on. Because those data are typically vast, proprietary, or both, its biases and gaps are inherited unseen. This is what can make foundation models wrong in ways that are hard to anticipate. How these models are pre-trained, adapted, and potentially misapplied is the subject of Section 2.2.2 and Chapter 3.

The three examples we started with have each moved along this path. Chess engines went from the hand-written evaluation rules of Deep Blue to the learned position estimates of today’s far stronger engines. Chatbots went from the scripted patterns of ELIZA to large language models. Spam filters went from keyword rules to learned word statistics and, increasingly, to large language models as well. The further along this path, the less of the solution you have to build—and the less of it you can inspect (Table 2.1).

Table 2.1: From left to right, more of the solution is derived from data, and it gets harder to find out why a system did what it did.

	Rule-based	Machine learning	Foundation models
You supply	features, algorithm, parameter values	model architecture, training objective, many examples; for classic methods also the features	a prompt, or a small set of examples for fine-tuning
Derived from data	nothing	the parameters; with deep learning, the features as well	the representation: what images or texts are like in general
Diagnosing a failure	find the rule that failed	test the model, check the training examples	observe its behavior; the pre-training data are usually out of reach

The table shows a trend, not a law: a thousand-tree ensemble is no more readable than a small network. What changes reliably is how much of a system’s behavior you can anticipate without testing it. You know in advance that a threshold will break when the staining gets dimmer; a model that needed fewer examples gives no such warning. So prefer the earliest stop that solves your problem. Past that stop, trust has to be earned empirically, on ground truth the model never saw (Chapter 10). Much of what follows is about telling “works” from “mostly works”.

2.1.1 How deep learning learns

A deep learning model is one large function that turns an image into the desired output: a class, a mask, a restored image. It is built by chaining many simple steps: weighted sums of pixel values, small filters slid across the image, and simple operations such as setting negative values to zero. The weights in these steps are the model’s free parameters, often millions of them, and they start out random.

Training compares the model’s output with the expected one. A loss function turns the difference into a single number, the error. Every step of the model is differentiable, so for each parameter one can compute how a tiny change would change the error. The chain rule of calculus, applied backwards from the error through all steps (backpropagation), does this for all parameters at once, and gradient descent nudges each one a little in the direction that lowers the error. Repeated over many examples until the error stops falling, this process turns random parameters into a working model¹⁶. Because the whole chain from pixels to output is trained together, the model learns its own features rather than relying on measurements a person designed.

Such models are called artificial neural networks by historical analogy: many simple units, each adding up weighted inputs and passing the result through a simple threshold-like response, wired into a network whose collective behavior is far richer than any unit’s. “Deep” refers to the many layers of such units stacked on top of each other.

Differentiability is the only hard constraint, which leaves great freedom in how the steps are arranged. These architecture choices are where knowledge about the data re-enters a system that would otherwise have to learn everything from examples. Convolutional neural networks slide the same small filters across the whole image, building in the knowledge that a nucleus looks the same wherever it appears. Transformer architectures, originally developed for language, compare every part of the input with every other part, which helps when distant regions belong together—across a volume, or across the frames of a time-lapse. Chapter 4 explains architectures, loss functions, and the optimization procedures in detail.

The same freedom lets known physics be written into a model instead of learned from examples. If the point spread function of the microscope has been measured, the network does not have to rediscover how the microscope blurs: the known blur can be built into the architecture or the loss, and learning is spent only on what the optics do not explain^24,25. Such physics-informed models often need fewer training images, can be trained on images simulated from the optics, and produce intermediate results with a physical meaning, such as an estimate of the unblurred specimen, which makes them easier to check. Chapter 6 develops them for deconvolution and aberration correction.

2.2 Learning from examples

The promise of machine learning is that it can infer the rules from examples. Two conditions have to hold:

1. The model must be able to represent and express the rules.
2. The examples must be enough to pin them down.

Both fail quietly. A model that sees image patches too small to contain a whole cell cannot learn anything that depends on cell shape or size, however many patches it is shown. A model shown only healthy cells cannot learn to recognize a diseased one. And the more a system has to discover for itself, the more examples it needs: learning a decision on top of a few well-chosen measurements takes fewer cells than learning what to measure in the first place.

The approaches below differ in where the training signal comes from: a person who annotated the data, the structure of the data itself, or the consequences of an action. This determines how much annotation you have to provide and what kind of answer you get back, but not, by itself, how accurate that answer is. It is also a separate question from the division of labor in Section 2.1: a deep learning model can be trained with labels or without them, and clustering, the classic method that needs no labels at all, works on hand-designed measurements as well as on learned ones.

2.2.1 Supervised learning: the answer is given

In supervised learning, every training image comes paired with the correct answer, the target, and the model learns to produce that answer from the image. The recipe does not care what the answer is: the same machinery classifies cells, outlines them, or predicts one channel from another. What changes is the form of the target, and in microscopy image analysis, that form varies a lot:

• A category for an entire image or for one cell: treated or control, mitotic or interphase.
• A position: a point per cell, or a box around it, saying where things are without saying where they end.
• A class for every pixel—nucleus, cytoplasm, background—which is what semantic segmentation asks for²⁶.
• An identity for every object, so that two touching nuclei are two things rather than one region. This is instance segmentation, and it is what counting and per-cell measurements need.
• Another image: a low-noise acquisition of the same field as the target for a noisy one²⁷, or a fluorescence channel as the target for the transmitted-light image it is predicted from³.

The last case is still supervision, even though nobody annotated anything: the target came from a second measurement (Section 2.3.2).

The form of the target is the form of the answer you get back: a model trained on one class per image will never hand you cell boundaries, however many images it has seen. The target does not dictate how the model gets there, though. Instance segmentation in particular is rarely predicted directly: most methods predict something for every pixel, such as cell boundaries or the direction to each cell’s center, and assemble objects from it in a second step (Section 2.3.1).

Supervision has the virtue of directness. What you want the model to do is what the loss function measures, so accuracy can be quantified on labeled examples the model never saw, and a failure can at least be named: these nuclei were merged, those were missed. None of the label-free methods below offer that.

The cost is that the targets have to be made, usually by hand, and that they are not merely training material: they are the definition of the task. Systematic annotator habits become model behavior: if your annotators consistently skip dim nuclei at the edge of the field, so will the model, silently and everywhere. Where two experts disagree, the model learns the average of a disagreement rather than a fact, which sets a ceiling on any accuracy you can later claim. Curating such a set is the subject of Chapter 5.

How much annotation is involved varies over orders of magnitude, and two rules of thumb cover most of it. Adapting a pre-trained model to your data takes far fewer labels than training one from scratch (Chapter 9). And label shapes differ enormously in price: one class per image or one click per cell is quick; a dense mask of a crowded field is the most expensive annotation in routine use. When a count is all you need, buy the label shape that gives you counts.

2.2.2 Learning without labels

Two families of methods need no annotation at all. Without a person supplying the right answer, both scale to as much data as a microscope can generate, but neither can tell you whether what it found means anything biologically.

Unsupervised learning is the older and stricter of the two: it receives the images, or measurements derived from them, and nothing else. It optimizes a property of the data itself rather than agreement with a target: how compactly the data can be grouped, or how well a compressed version reproduces the original. Dimensionality reduction condenses the hundreds of numbers measured per cell into two or three, so that a whole screen fits into a scatter plot²⁸. Clustering sorts the cells into groups without being told what the groups mean. Together they are the standard tools of morphological profiling, which measures many features on every cell in every well and looks for structure in the result²⁹. Compounds with indistinguishable profiles then become candidates for a shared mechanism.

The limitation is easy to state and easy to forget: the algorithm separates points in the feature space it was given, and nothing guarantees that this space aligns with a biological category. Change the features and the groups change with them. The assumptions have moved from the labels into the features. A cluster is a hypothesis about your data; it becomes a finding only when independent evidence confirms it.

Self-supervised learning goes one step further: it turns unlabeled data into a supervised problem. The usual recipe is to withhold part of the data and train the model to restore it, for example masked patches of an image¹⁸ or masked words in a sentence³⁰. Hiding single pixels of a noisy image and predicting them from their neighbors trains a denoiser that has never seen a clean image³¹. Movies are so redundant that a model still learns well when it has to fill in 90% or more of each clip³².

The task sounds circular, but it cannot be solved by simple statistics. Filling a large hole convincingly requires some idea of what put the pixels there: that a membrane continues where it is hidden, that nuclei have characteristic shapes and textures, that illumination varies smoothly across the field. A network trained this way at sufficient scale comes to represent much of the structure that generated its images, without ever being told what that structure is.

A self-supervised denoiser is trained for its output: the restored pixel is the product, and the representation is only a means to it (Chapter 6). In pre-training it is the other way around. The reconstruction is discarded, and the representation is what you keep. This also separates self-supervision from clustering: clustering hands you groups to interpret, whereas a representation is not yet an answer to anything. A second stage puts it to work. Fine-tuning on a few hundred annotated images can turn a generic visual representation into a segmentation model for your specimen, and semi-supervised training adds a large pool of unlabeled images to a small labeled set. Neither repeats the expensive pre-training. On ImageNet, a benchmark of more than a million labeled photographs, pre-training without labels and fine-tuning with 1% of them comes close to a network trained on all labels. That is a tenfold gain in label efficiency over earlier approaches³³.

Not every general-purpose model got there without labels. The original Segment Anything model was trained on roughly a billion masks in eleven million images, masks that annotators produced with the help of the model itself. A dedicated component reads the clicks, boxes, or rough masks you give it²¹. The model segments structures it was never explicitly taught, but only in the ways its designers built in. Either way, the training data sets the limit: a model absorbs the structure of the images it was shown, with their biases and artifacts, not the structure of the world. Versions fine-tuned on microscopy data exist^34,35, but whether they help on your images has to be tested. Chapter 9 covers how to fine-tune, and Chapter 10 how to tell whether it worked.

TipOther recipes for self-supervision

Filling in masked-out data are only the most direct recipe. A second family learns embeddings that are insensitive to irrelevant variation but expressive enough to distinguishing meaningful classes. A contrastive loss pulls together the embeddings of two altered versions of the same image and pushes apart those of different images³⁶; other methods only ask the network to agree with a slowly updated copy of itself³⁷. Text models learn embeddings too, so that passages about the same topic land close together^30,38.

The result can be more structured than the training task suggests: the image regions such a network attends to trace the outlines of objects, although nothing in the training asked for a segmentation³⁷. Trained on large, carefully curated image collections, these networks learn features general enough that many tasks need only a simple classifier on top^39,40.

A third recipe hides part of the input, as in reconstruction, but asks the network to predict the representation of the hidden part rather than its pixels, because fine detail is largely unpredictable and not worth reproducing⁴¹.

Recipes that only reward agreement share a trap: describing every image the same way satisfies them perfectly and learns nothing. Most methods guard against this collapse with an assortment of engineering tricks; recent work replaces them with the single requirement that the embeddings be spread out evenly in all directions⁴².

NoteClustering or segmentation? Let the question decide

Take one compound screen: a few thousand fields, three channels, a few hundred conditions. Two questions a biologist might reasonably ask of it need entirely different methods.

How many cells are dying in each condition? The category already exists—you can say what a dying cell looks like—and the answer is a number per condition, so every cell has to be found and judged: supervised detection or segmentation, then supervised classification. You provide annotations, and what you get back is a measurement, a count you can put an error bar on, because annotated images the model never saw tell you how many cells it missed or invented.

Are there phenotypes in this screen I have not thought of? Here there is nothing to supervise, because naming the classes is the result you are after. So you measure many features per cell and group the cells, or whole conditions, whose measurements look alike. Without annotation, you get a hypothesis: a set of conditions that behave similarly, which needs an independent experiment before it counts as a finding.

The rule generalizes. Supervised methods answer questions you can already phrase; label-free methods help you find the question. Many projects run the second kind of analysis once, to decide what is worth counting, and the first kind forever after.

The two also combine more often than they compete: the features are usually measured on cells that a segmentation model found first, so an exploratory analysis inherits every bias of that model—a reason to check the segmentation even when the result is only a hypothesis.

2.2.3 Learning from consequences: reinforcement learning

A third approach needs no examples at all. In reinforcement learning, an agent acts, the environment responds, and a reward scores how well that went; the agent adjusts its behavior to collect more reward, generating its own training data as it goes⁴³. The reward can be sparse and arrive late—a game won twenty moves after the decisive one—which is what makes the approach both powerful and expensive.

Image analysis gives it little to work with, because there is no action to take on an image that has already been recorded: the pixels do not respond. Microscopy does offer one natural fit, and it is at the instrument. In closed-loop acquisition the model chooses where to look next, at what magnification, and for how long, and those choices have consequences that can be scored—an event captured or missed, a sample bleached or spared. Chapter 7 takes up event-driven imaging.

The term will also be familiar from large language models, which use reinforcement learning from human feedback. This is the stage in which a language model is tuned toward responses that people rated highly²⁰. The mechanics are the same, with a learned stand-in for human preference supplying the reward (Chapter 3).

2.3 The Tasks

The approaches above sort methods by where their training signal comes from; the tasks below sort them by what comes out.

2.3.1 Segmentation: turning pixels into objects

Segmentation converts an image into a list of things you can measure. Counts, sizes, per-cell intensities, colocalization, morphology—almost every quantitative claim made from a microscopy image passes through it, which is why the rest of this book keeps returning to it.

The literature uses three terms, not always consistently. Semantic segmentation assigns each pixel a class without separating objects of the same class, instance segmentation gives each object its own identity (Section 2.2.1), and panoptic segmentation does both at once. Tip 8.2 works the distinction through on a real image.

What makes an annotation expensive is not the number of pixels but how precisely the shape has to be drawn. Semantic segmentation is forgiving: a pixel classifier can be trained from a few brush strokes per class on a handful of images, as interactive tools such as ilastik have done since before deep learning¹⁵. Instance segmentation is not, because every object needs its own identity, and most methods need each boundary drawn accurately—slow, careful work rather than a lot of clicking.

How much of that care is genuinely necessary depends on the method rather than the task. A network that predicts an embedding per pixel needs the pixels of one object to agree, not its boundary to be right, so in principle sparse labels should suffice, even though published implementations often use dense ones^44,45. A network that only has to place a box needs no outlines at all, which is why annotating for counting is cheap. What cannot be negotiated is the number of dimensions: a 3D annotation has to stay consistent from section to section, so it is one decision carried through a volume rather than one drawing per slice, and adding time compounds it again.

How a segmentation is produced is rarely what the label suggests. Object identity has no canonical output: which integer a given cell receives is arbitrary, and swapping two of them changes nothing about the answer, so there is nothing stable for a per-pixel classifier to predict. Almost every method therefore predicts a quantity that is well defined per pixel and assembles objects from it in a second step.

TipA deeper understanding of what segmentation methods actually predict

Methods differ mainly in what the network is asked to output per pixel, and each choice implies its own way of recovering objects.

Boundaries and affinities. The network predicts, for each pair of neighboring pixels, whether they belong to the same object⁴⁶. Objects are then grown by watershed, or by agglomerative merging. Including pairs that are farther apart than one pixel helps detect thin gaps and provides evidence that two pixels belong to different objects; the mutex watershed uses that evidence to separate objects without seed points or a threshold^47,48.

A description of the shape the pixel belongs to. StarDist predicts the distance from each pixel to the object’s edge in a fixed set of directions and keeps the best non-overlapping polygons⁴⁹. Local shape descriptors predict summary statistics of the surrounding object—its size, elongation, and offset to its center—as an additional training target that markedly improves the affinities predicted alongside⁵⁰. PatchPerPix goes furthest and predicts an entire small binary mask of the object around each pixel, then assembles objects from the patches that agree⁵¹.

A direction toward the object’s center. Cellpose predicts, for every pixel, the direction toward the center of its cell. Following these directions makes the pixels of one cell converge on a single point, and those that arrive together become an object⁵². Nothing here assumes a shape, which is why it copes with irregular and elongated cells.

An embedding per pixel. The network learns a vector that should be identical for pixels of the same object and different across objects, and the objects fall out by clustering^44,45.

One object at a time. Flood-filling networks start from a seed and iteratively extend a single mask until it stops growing, then move on to the next seed⁵³. This sidesteps the identity problem rather than solving it, and it has proven itself where objects are enormous and branching, as in reconstructing neurons from electron microscopy.

The grouping step deserves as much attention as the network. It is where most false splits and false merges are made, and where most of the parameters a user still has to set actually live. It is also the knob that trades the two errors against each other, and which one you want depends on the question. A false split, or one cell cut in two, is usually easier to see and quicker to repair than a false merge, especially in objects with complicated shapes. Moreover, a merge quietly averages two cells into one measurement.

Chapter 4 covers the architectures behind these predictions, Chapter 8 how to find a tool that already makes them, Chapter 9 how to train or fine-tune one on your own data, and Chapter 10 how to tell whether it worked.

2.3.2 Restoration: recovering what the microscope could not record

Every image is a degraded observation. The microscope blurs the specimen by its point spread function, the detector records a finite number of photons and so returns a noisy count, and the pixel grid is often coarser than the finest detail you would like to resolve. Restoration estimates the underlying signal from what was recorded. This is an ill-posed inverse problem, because many different specimens are consistent with the same recorded image.

Supervised restoration fits the template of Section 2.2.1 exactly, with an image as the target instead of a label. The kind of target image defines the task: a high-signal acquisition of the same field for denoising, a sharper one for deconvolution and aberration correction, a higher-resolution or differently acquired one for resolution enhancement. Chapter 6 develops all of these. Restoration can also use a different channel entirely as a target for modality translation, where a fluorescence signal is predicted from transmitted light³.

The reason to care is not that the images look better. Noise, blur, and limited resolution are all paid for in photons, and photons are what a living sample cannot spare. A denoiser that recovers a usable image from a tenth of the illumination converts directly into a tenth of the phototoxicity, or ten times the frame rate, or a time-lapse that runs ten times longer before the specimen gives out. Restoration is an acquisition technique that happens to run after acquisition.

One caveat matters enormously: restoration is not measurement. Wherever the recorded data carry too little information, the estimate comes from the training set rather than from the specimen. Section 2.3.4 names the mechanism and Chapter 6 collects the consequences.

Denoising often does not need a clean reference: self-supervision (Section 2.2.2) can train a denoiser on noisy images alone.

TipDenoising without a clean image

Deep-learning-based denoising methods rest on one asymmetry: the signal in an image is predictable from its surroundings, and the noise, by definition, is not. Put a network in a position where it can only reproduce the predictable part, and it denoises.

The direct approach needs a pair of images of the same field, one noisy and one with high signal, and learns the map between them²⁷. Acquiring that pair is easy on a fixed sample and often impossible on a living one, which moves and changes between exposures.

Noise2Noise drops the need for a high-signal image. Two independent noisy exposures of the same field are enough, because the network cannot predict the particular noise in the target from the noise in the input; the best it can do is the signal the two share⁵⁴.

Noise2Void drops the need for a pair. A single noisy image suffices: hide one pixel, ask the network to predict it from its neighbors, repeat³¹. The network never sees the value it is trying to reproduce, so it cannot copy the noise. What it does require is that the noise be independent between pixels—true of photon shot noise, false of line artifacts such as scanning stripes, where the fix is to hide whole lines instead of single pixels⁵⁵.

A noisy image is consistent with many clean ones, and a network trained to minimize the average squared error returns their blend. Diversity denoising instead learns the distribution of clean images consistent with the input and draws samples from it, making the ambiguity visible rather than smoothing it away⁵⁶. It is a generative model in the sense of Section 2.3.4.

Similar networks can also undo a deliberate mix. If two labeled structures are imaged into a single channel, a network trained on separately acquired examples of each can pull them apart again, so fewer channels have to be acquired in the first place⁵⁷.

One assumption is shared by all of them: whatever cannot be predicted is treated as noise. Structure that is genuinely rare and genuinely unpredictable is therefore exactly what they are most likely to erase.

Chapter 6 discusses these approaches in more detail.

2.3.3 Classification, detection, and tracking

Classification attaches one label to an image, or to an object already found in it: cell type, cell cycle stage, treated or control, healthy or diseased. Image-level labels are the cheapest annotation there is, which makes classification a quick way to find out whether a signal is present in your data at all.

Detection locates each object by a point or a box, but not its exact outline. That annotation is dramatically simpler than a detailed boundary mask, and the output is sufficient whenever the answer is a count, a position, or a trigger to act on (Chapter 7).

Tracking links objects across frames, which is where much of the interesting biology is—division, migration, differentiation, events that a single timepoint cannot show. It combines two problems: finding the objects in each frame, which is segmentation or detection as above, and association, deciding which object in this frame is which object in the next.

Cell division makes it harder than plain matching, because one object legitimately becomes two, so the answer is a lineage tree rather than a set of paths—and a dividing cell rounds up and changes appearance precisely when the association is most delicate. Association errors also compound, since a single swap corrupts every measurement downstream of it in that lineage. A method can therefore score well on the fraction of correct links and still recover almost no complete tracks⁵⁸, which is why tracking has no single accuracy number (Chapter 10).

TipHow tracking is actually done

Most trackers in use separate the two problems: segment or detect each frame independently, then link. The separation is convenient, because either part can be replaced independently. It is also the structure that most standard benchmarks assume⁵⁸.

The classical linker has not been displaced. Linear assignment solves frame-to-frame matching as an optimal assignment problem, then makes a second global pass that stitches track fragments together and repairs gaps, merges, and splits⁵⁹. This is the main engine inside TrackMate, the tool most microscopists will actually open, which now accepts modern segmentation models as its detector⁶⁰. An important finding from a decade of benchmarking is that segmentation quality dominates: a simple linker on good detections beats a sophisticated linker on poor ones⁵⁸.

Deep learning has changed the linker in three ways. It can learn the association itself, with a transformer scoring candidate matches from position and appearance across a window of frames, which works across bacteria, cell culture, and particles without per-dataset tuning⁶¹. It can fold detection and linking into one network, predicting for each pixel the offset to where its object was in the previous frame⁶². Or it can supply many candidate segmentations, from several models and parameter settings, and leave it to temporal consistency to choose among them, which is what makes terabyte-scale embryo recordings tractable⁶³. Video foundation models are an obvious candidate as well: SAM 2 carries a mask through a video without any retraining²², and microscopy time-lapses are videos too.

What remains hard is well documented. Pre-trained trackers generalize poorly to morphologies and modalities they have not seen; densely annotated, fully tracked datasets are scarce, which limits what more supervision can fix; and recovery of complete lineages lags well behind the local link-accuracy scores that are usually reported⁵⁸.

Rare-event detection uses these tasks to find the one cell in ten thousand that does something interesting. A model can search vast amounts of recorded data for such events, and if it runs fast enough, it can steer the microscope during acquisition: record the event in detail while it happens, and spare the sample and the storage during the hours in which nothing happens (Chapter 7). The two kinds of error weigh very differently. A false negative is a lost event, and when events are this rare, every one counts. A false positive costs less, but even a low rate adds up: one false alarm per thousand cells buries each real event under ten false ones. That is acceptable, because the detector does not have to be the last word. Tuned to miss as little as possible, it shrinks the pile enough that a slower, more accurate method, or a person, can sort through what remains.

2.3.4 Generative models: sampling instead of averaging

A model trained to minimize the average squared error returns the mean of all outcomes consistent with its input. Where the input is ambiguous, that mean is a blend, which is why strongly denoised images can look slightly waxy. A generative model is instead trained to draw samples from the distribution of possible outcomes. Each sample carries the texture and edge contrast of real data, and running the model twice on the same input yields two different answers. No single sample represents the whole distribution, though: it is one plausible answer, not the best estimate.

Four families are in current use. Variational autoencoders compress an image into a small code and decode it back; because the code is a distribution rather than a single point, sampling it yields different plausible reconstructions⁶⁴. Adversarial training pits the generator against a second network that tries to tell generated images from real ones⁶⁵. Diffusion models learn to reverse a gradual corruption by noise, and generate by denoising pure noise in many small steps⁶⁶. Flow matching learns a more direct path from noise to data, which takes fewer steps⁶⁷.

The risk follows from what sampling means: a generative model is built to produce output indistinguishable from real data, whether or not the input supports the particular detail it draws—a vesicle that is sharp, plausible, correctly textured, and never measured. This inverts the usual relation between image quality and trust: the restored image is more convincing than the raw data, so judging it by eye selects for exactly the failures you need to catch⁶⁸. A structure that appears only after restoration must never be quantified.

2.4 Why It Fails, and How you find out

2.4.1 Overfitting and the discipline of splitting data

It is easy to say what kind of label, field of view, or resolution a task requires, and much harder to know whether a set of examples is sufficient, because nobody knows in advance which rules the model has to infer. Worse, the model may find a shortcut that works perfectly on the training examples and fails on the intended task. The only way to find out is to measure performance on a separate test dataset that was never used for training.

There is a second problem: gradient descent keeps driving the training loss lower, eventually by memorizing the training examples rather than learning rules that generalize. Because no training set fully represents the real world, this overfitting sets in if training runs long enough and the model sees each example too often. So when do you stop?

The answer is a validation dataset: examples withheld from training but used to monitor performance along the way. When performance on them stops improving, training stops.

The validation set must stay separate from the test set. Validation examples are not used for gradient descent, but they still shape the final model: you pick the training iteration, architecture, or hyperparameters that do best on them, so the model can overfit to the validation set too. Only the test set, used once at the very end, gives an unbiased estimate of performance on new data from the same source.

Both splits only work if the withheld examples are genuinely separate. Neighboring patches from the same stack are nearly identical, so splitting patches at random scatters near-copies across all three sets, and the score that comes back measures memory rather than performance. Chapter 5 covers how to build and split a dataset in practice, Chapter 10 what to measure on it.

2.4.2 Domain shift and shortcut learning

A model can pass both checks and still do worse on images acquired a few months later. Training and test data came from the same experiment; the new images did not. This is domain shift. The causes are mundane: a different objective, a realigned laser, a new batch of mounting medium, the same cell line at a higher passage number. The result is rarely outright failure but a slow degradation: slightly worse masks, a few more missed cells, a shifted effect size.

Microscopy is more exposed to this than most of computer vision. A model trained on hundreds of millions of photographs has already seen most of the variation it will meet. A microscopy model often knows only one laboratory’s protocol on one instrument, and that protocol changes routinely, sometimes without anyone noticing.

The same narrowness allows a second failure. One experiment’s images cannot separate the regularities of the biology from those of the protocol. If something incidental correlates with the label, gradient descent will use it, because it optimizes the loss and not the biology. This is shortcut learning⁶⁹. The outer wells of a plate lose more medium to evaporation, and their cells look different⁷⁰, so a classifier whose controls sit in column 1 may be reading well position rather than phenotype. Shortcuts need not come from the sample at all. Training crops that are mostly background teach a segmentation model to predict background everywhere, which minimizes the loss and finds nothing (Section 8.6.2).

Neither failure shows up as a poor validation score, because the validation data share the protocol, and therefore the confounds, of the training data. For example, in one case, it was easier to train a model to identify the hospital and department that acquired an image than one that would successfully classify pneumonia in images from other hospitals⁷¹.

Split at the level of whatever will vary in practice: whole plates, whole sessions, whole cell lines, not random crops from a common pool. That does not prevent domain shift, but it makes it visible before you rely on the model. The rest belongs elsewhere: a training set that spans the variation you expect, and augmentation to extend it (Chapter 5); fine-tuning when a published model does not transfer (Chapter 9); and appropriate measures of success (Chapter 10). A reported accuracy describes the data it was measured on, and says little about yours⁷².

2.4.3 The black box: a correct answer is not an explanation

Nothing inside a trained network is hidden: every weight and every intermediate value can be printed. But the resulting features do not trivially decompose into interpretable quantities (Section 2.1). That is what the black box is: not secrecy, but the absence of any description of the model short enough to reason with.

This has two important consequences. The first is that behavior cannot be predicted by inspection. For a rule-based pipeline, you can read off where it will fail, because the rules are written down; for a network, the failures of Section 2.4.1 and Section 2.4.2 can only be found by measurement. The second is that a correct prediction carries no reason with it. A classifier may separate treated cells from controls on every plate for something other than the phenotype, and no held-out test will reveal it, because the prediction is not wrong. That the network distinguishes the conditions is a measurement; that the treatment changes mitochondrial morphology is a claim the network never made.

The confidence score the model reports is not a check on this. It comes out of the same computation as the prediction, and current networks routinely report confidence above their measured accuracy⁷³, and most severely under domain shift, exactly when a warning would be most useful⁷⁴. Whether a given score can be trusted is itself measurable (Chapter 10).

What remains is evaluation on data like yours, and interpretability methods that have themselves been tested. A common tool is an attribution map, a per-pixel image of how much each part of the input contributed to the output; several widely used variants produce nearly the same map after the network’s weights are randomized, so they were never reporting on the trained model at all⁷⁵. Methods that pass such checks can deliver. A network trained on electron micrographs identifies which of six neurotransmitters a Drosophila synapse releases in 87% of cases. Ordinary attribution maps did not reveal what it relied on, but comparing each synapse with a generated counterfactual, the same synapse made to look like another transmitter type, isolated ultrastructural differences a person could name and nobody had described before⁷⁶.

A network nobody can explain can still be a good instrument, provided it has been characterized like one.

2.5 What comes next

A model can be put to two uses. In measurement, you already know what to count and the model counts it. It is part of the instrument, and its output is a number in your results⁷⁷. In exploration, you do not yet know what to look for, and the output is a suggestion about what to measure next. The two are not separate methods: the same segmentation model can count the phenotype you asked for and then reveal one you did not. What matters is saying which you were doing.

Using a model as an instrument means characterizing it like one: an accuracy, the metric it was computed with, and the data it was measured on. The last is the easiest to leave out and the most important (Section 2.4.2). The model, its version, whether you fine-tuned it and on what, and the score on held-out data belong in the methods, as specifically as an objective or an exposure time.

The rest of the book takes up what this chapter has only named. Chapter 3 covers the models whose output is language, Chapter 4 architectures and loss functions, and Chapter 5 the examples everything depends on. Chapter 6 treats restoration in depth, and Chapter 7 models that act on the specimen instead of reporting on it. Chapter 8 helps you find an existing tool, Chapter 9 shows how to train or fine-tune one, Chapter 10 how to measure whether it worked, and Chapter 11 where exploration is already producing new biology.

AI Disclosure

AI language models and agents (Anthropic’s Claude Code using Sonnet 5.5, Opus 4.6 and 5.5, and Fable 5.1) were used for the preparation of this chapter. They were used to brainstorm, discuss, draft, and edit prose, to check consistency across sections, glossaries, and other chapters. All content was reviewed, revised, and verified by the author, who takes full responsibility for the accuracy and scientific integrity of the final text. Selection and curation of references was done by the author.

1.

Shapson-Coe, A. et al. A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution. Science 384, eadk4858 (2024).

2.

The MICrONS Consortium et al. Functional connectomics spanning multiple areas of mouse visual cortex. Nature 640, 435–447 (2025).

3.

Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nature Methods 15, 917–920 (2018).

4.

McCarthy, J., Minsky, M. L., Rochester, N. & Shannon, C. E. A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955. AI Magazine 27, 12–14 (2006).

5.

Weizenbaum, J. ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM 9, 36–45 (1966).

6.

Shortliffe, E. H. & Buchanan, B. G. A model of inexact reasoning in medicine. Mathematical Biosciences 23, 351–379 (1975).

7.

Brin, S. & Page, L. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30, 107–117 (1998).

8.

Lowe, D. G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91–110 (2004).

9.

Gonzalez, R. C. & Woods, R. E. Digital Image Processing. (Prentice Hall, 2008).

10.

Cortes, C. & Vapnik, V. Support-vector networks. Machine Learning 20, 273–297 (1995).

11.

Schapire, R. E. The strength of weak learnability. Machine Learning 5, 197–227 (1990).

12.

Friedman, J., Hastie, T. & Tibshirani, R. Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics 28, (2000).

13.

Viola, P. & Jones, M. Rapid object detection using a boosted cascade of simple features. in Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 vol. 1 I-511-I-518 (2001).

14.

Breiman, L. Random forests. Machine Learning 45, 5–32 (2001).

15.

Berg, S. et al. Ilastik: Interactive machine learning for (bio)image analysis. Nature Methods 16, 1226–1232 (2019).

16.

LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 2278–2324 (1998).

17.

Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. in Advances in Neural Information Processing Systems vol. 25 (Curran Associates, Inc., 2012).

18.

He, K. et al. Masked autoencoders are scalable vision learners. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition 16000–16009 (2022). doi:10.1109/CVPR52688.2022.01553.

19.

Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. in International conference on learning representations (ICLR) (2021). doi:10.48550/arXiv.2010.11929.

20.

Ouyang, L. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022).

21.

Kirillov, A. et al. Segment anything. in 2023 IEEE/CVF international conference on computer vision (ICCV) 3992–4003 (2023). doi:10.1109/ICCV51070.2023.00371.

22.

Ravi, N. et al. SAM 2: Segment anything in images and videos. in International conference on learning representations (ICLR) (2025). doi:10.48550/arXiv.2408.00714.

23.

Carion, N. et al. SAM 3: Segment anything with concepts. in International conference on learning representations (ICLR) (2026). doi:10.48550/arXiv.2511.16719.

24.

Li, Y. et al. Incorporating the image formation process into deep learning improves network performance. Nature Methods 19, 1427–1437 (2022).

25.

Yanny, K., Monakhova, K., Shuai, R. W. & Waller, L. Deep learning for fast spatially varying deconvolution. Optica 9, 96–99 (2022).

26.

Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional networks for biomedical image segmentation. in Medical image computing and computer-assisted intervention – MICCAI 2015 (eds. Navab, N., Hornegger, J., Wells, W. M. & Frangi, A. F.) 234–241 (Springer International Publishing, Cham, 2015). doi:10.1007/978-3-319-24574-4_28.

27.

Weigert, M. et al. Content-aware image restoration: Pushing the limits of fluorescence microscopy. Nature Methods 15, 1090–1097 (2018).

28.

McInnes, L., Healy, J. & Melville, J. UMAP: Uniform manifold approximation and projection for dimension reduction. (2018) doi:10.48550/arXiv.1802.03426.

29.

Bray, M.-A. et al. Cell Painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nature Protocols 11, 1757–1774 (2016).

30.

Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) 4171–4186 (Association for Computational Linguistics, Minneapolis, Minnesota, 2019). doi:10.18653/v1/N19-1423.

31.

Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void - learning denoising from single noisy images. in 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR) 2124–2132 (2019). doi:10.1109/CVPR.2019.00223.

32.

Tong, Z., Song, Y., Wang, J. & Wang, L. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. in Proceedings of the 36th International Conference on Neural Information Processing Systems 10078–10093 (Curran Associates Inc., Red Hook, NY, USA, 2022).

33.

Chen, T., Kornblith, S., Swersky, K., Norouzi, M. & Hinton, G. Big self-supervised models are strong semi-supervised learners. in Proceedings of the 34th International Conference on Neural Information Processing Systems 22243–22255 (Curran Associates Inc., Red Hook, NY, USA, 2020).

34.

Archit, A. et al. Segment anything for microscopy. Nature Methods 22, 579–591 (2025).

35.

Pachitariu, M., Rariden, M. & Stringer, C. Cellpose-SAM: Superhuman generalization for cellular segmentation. bioRxiv (2025) doi:10.1101/2025.04.28.651001.

36.

Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. in Proceedings of the 37th International Conference on Machine Learning vol. 119 1597–1607 (JMLR.org, 2020).

37.

Caron, M. et al. Emerging properties in self-supervised vision transformers. in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 9630–9640 (2021). doi:10.1109/ICCV48922.2021.00951.

38.

Zhang, Y. et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. (2025) doi:10.48550/arXiv.2506.05176.

39.

Oquab, M. et al. DINOv2: Learning robust visual features without supervision. (2024).

40.

Siméoni, O. et al. DINOv3. (2025) doi:10.48550/arXiv.2508.10104.

41.

Bardes, A. et al. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research (2024).

42.

Balestriero, R. & LeCun, Y. LeJEPA: Provable and scalable self-supervised learning without the heuristics. (2025) doi:10.48550/arXiv.2511.08544.

43.

Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).

44.

Lalit, M., Tomancak, P. & Jug, F. EmbedSeg: Embedding-based instance segmentation for biomedical microscopy data. Medical Image Analysis 81, 102523 (2022).

45.

Goldsborough, T. et al. InstanSeg: An embedding-based instance segmentation algorithm optimized for accurate, efficient and portable cell segmentation. arXiv [cs.CV] (2024).

46.

Turaga, S. C. et al. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation 22, 511–538 (2010).

47.

Funke, J. et al. Large scale image segmentation with structured loss based deep learning for connectome reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 1669–1680 (2019).

48.

Wolf, S. et al. The mutex watershed and its objective: Efficient, parameter-free graph partitioning. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 3724–3738 (2021).

49.

Schmidt, U., Weigert, M., Broaddus, C. & Myers, G. Cell detection with star-convex polygons. in Medical image computing and computer assisted intervention - MICCAI 2018 (eds. Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) 265–273 (Springer International Publishing, Cham, 2018). doi:10.1007/978-3-030-00934-2_30.

50.

Sheridan, A. et al. Local shape descriptors for neuron segmentation. Nature Methods 20, 295–303 (2023).

51.

Mais, L., Hirsch, P. & Kainmueller, D. PatchPerPix for instance segmentation. in Computer vision – ECCV 2020 288–304 (Springer International Publishing, Cham, 2020). doi:10.1007/978-3-030-58595-2_18.

52.

Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: A generalist algorithm for cellular segmentation. Nature Methods 18, 100–106 (2021).

53.

Januszewski, M. et al. High-precision automated reconstruction of neurons with flood-filling networks. Nature Methods 15, 605–610 (2018).

54.

Lehtinen, J. et al. Noise2Noise: Learning image restoration without clean data. in Proceedings of the 35th international conference on machine learning (eds. Dy, J. & Krause, A.) vol. 80 2965–2974 (PMLR, 2018).

55.

Broaddus, C., Krull, A., Weigert, M., Schmidt, U. & Myers, G. Removing structured noise with self-supervised blind-spot networks. in 2020 IEEE 17th international symposium on biomedical imaging (ISBI) 159–163 (2020). doi:10.1109/ISBI45749.2020.9098336.

56.

Prakash, M., Krull, A. & Jug, F. Fully unsupervised diversity denoising with convolutional variational autoencoders. in International Conference on Learning Representations (ICLR) (2021). doi:10.48550/arXiv.2006.06072.

57.

Ashesh, A. et al. Micro𝕊plit: Semantic unmixing of fluorescent microscopy data. Nature Methods 23, 1047–1057 (2026).

58.

Maška, M. et al. The cell tracking challenge: 10 years of objective benchmarking. Nature Methods 20, 1010–1020 (2023).

59.

Jaqaman, K. et al. Robust single-particle tracking in live-cell time-lapse sequences. Nature Methods 5, 695–702 (2008).

60.

Ershov, D. et al. TrackMate 7: Integrating state-of-the-art segmentation algorithms into tracking pipelines. Nature Methods 19, 829–832 (2022).

61.

Gallusser, B. & Weigert, M. Trackastra: Transformer-based cell tracking for live-cell microscopy. in Computer vision – ECCV 2024 (Springer Nature Switzerland, Cham, 2024). doi:10.1007/978-3-031-73116-7_27.

62.

Löffler, K. & Mikut, R. EmbedTrack – simultaneous cell segmentation and tracking through learning offsets and clustering bandwidths. IEEE Access 10, 77147–77157 (2022).

63.

Bragantini, J. et al. Ultrack: Pushing the limits of cell tracking across biological scales. Nature Methods 22, 2423–2436 (2025).

64.

Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. in International Conference on Learning Representations (ICLR) (2014). doi:10.48550/arXiv.1312.6114.

65.

Goodfellow, I. J. et al. Generative adversarial networks. in Advances in Neural Information Processing Systems vol. 27 (2014).

66.

Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. in Advances in Neural Information Processing Systems vol. 33 (2020).

67.

Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M. & Le, M. Flow matching for generative modeling. in International Conference on Learning Representations (ICLR) (2023). doi:10.48550/arXiv.2210.02747.

68.

Lee, R. M., Eisenman, L. R., Khuon, S., Aaron, J. S. & Chew, T.-L. Believing is seeing – the deceptive influence of bias in quantitative microscopy. Journal of Cell Science 137, jcs261567 (2024).

69.

Geirhos, R. et al. Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 665–673 (2020).

70.

Caicedo, J. C. et al. Data-analysis strategies for image-based cell profiling. Nature Methods 14, 849–863 (2017).

71.

Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine 15, e1002683 (2018).

72.

Laine, R. F., Arganda-Carreras, I., Henriques, R. & Jacquemet, G. Avoiding a replication crisis in deep-learning-based bioimage analysis. Nature Methods 18, 1136–1144 (2021).

73.

Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. in Proceedings of the 34th international conference on machine learning vol. 70 1321–1330 (2017).

74.

Ovadia, Y. et al. Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. in Advances in Neural Information Processing Systems vol. 32 13969–13980 (2019).

75.

Adebayo, J. et al. Sanity checks for saliency maps. in Advances in Neural Information Processing Systems vol. 31 (2018).

76.

Eckstein, N. et al. Neurotransmitter classification from electron microscopy images at synaptic sites in Drosophila melanogaster. Cell 187, 2574–2594.e23 (2024).

77.

Wait, E. C., Reiche, M. A. & Chew, T.-L. Hypothesis-driven quantitative fluorescence microscopy – the importance of reverse-thinking in experimental design. Journal of Cell Science 133, jcs250027 (2020).