Glossary

A-posteriori Analysis

An approach in which images are processed after the complete dataset has been acquired and stored.

AI Agent

AI agents extend models into systems that can take actions (running code, calling APIs, controlling hardware), observe the results, and adapt their approach iteratively. At its core, an AI agent is a system built around a language model that operates in a loop: it observes the current state (an image, a dataset, an error message), reasons about what to do next, acts by invoking tools or executing code, and then observes the outcome — repeating until the task is complete. This observe-reason-act cycle is what distinguishes an agent from a chatbot.

Aberration

A distortion of the light’s wavefront caused by imperfections in the optical path or by refractive-index variations in the sample. It spreads the light and makes images blurrier than the ideal diffraction-limited image.

Activation Function

An activation function is a mathematical formula that calculates the output of a node. These functions are nonlinear, which allows the network to solve nontrivial problems. They are so named because the output of the function decides to what extent each node in the network is “activated,” an analogy to the way in which biological neurons can be activated.

Activations

The values of elements in neural network units, often specifically the values of the output of an activation function.

Adaptive Optics

Technology that senses distortions in the wavefront of light and cancels them, thereby suppressing optical aberrations to enhance image clarity.

Agentic Workflow

Agentic workflows are hand-designed sequences of tool calls with human-specified decision points. A developer would define: “first call the segmentation tool, then if the count is above 100 switch to algorithm B.” These workflows are effective but rigid: every decision point must be anticipated in advance.

Artificial Intelligence

Artificial Intelligence (AI) refers to a computational system that is capable of performing a task that would normally require human intelligence.

Attribution Map

A per-pixel image of how much each part of the input contributed to a model’s output. Some widely used variants are insensitive to the trained model and should be checked before they are trusted.

Autoencoder

A deep learning architecture used to learn efficient coding of unlabeled data. An autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation. It is an unsupervised learning version of the more general encoder-decoder architecture.

Auxiliary Variables

Auxiliary variables are variables added to a linear programming problem to simplify mathematics, improve precision, or handle missing data. An example of an auxiliary variable is distance to the boundary. These or other shape descriptors can be used as part of of a loss function.

Average Precision

Average Precision (AP) summarizes the precision-recall trade-off across confidence thresholds. A predicted instance counts as a true positive only if its IoU with a ground-truth instance exceeds a threshold \(x\):

\[\text{AP}_x = \sum_{k} (R_k - R_{k-1})\, P_k\]

where \(P_k\) and \(R_k\) are precision and recall at the \(k\)-th confidence threshold after matching predictions to ground-truth at IoU \(\geq x\). Higher \(x\) demands more precise localization.

Backpropagation

The method that computes, for every parameter of a neural network, how a small change in that parameter would change the error measured by the loss function. It applies the chain rule of calculus backwards, from the error, through all layers of the network. Backpropagation itself does not change the network; gradient descent uses the values it computes to adjust the parameters so that the error decreases.

Balanced Dataset

In a balanced dataset, each class or type of data is equally represented in the training data. In an unbalanced dataset, the model may become biased towards the classes that have more examples.

Batch

A small group of data that is processed together at the same time. For example, when training a machine learning model, a batch is a group of data that is given to the model for learning. Batches are commonly used to make the processes more efficient.

Batch Effects

Systematic technical differences between groups of samples that arise from preparation, handling, acquisition, or processing, rather than from the biological variable of interest.

Bayesian Optimization

A strategy that allows the optimization of black-box functions such as deep neural networks. It creates a surrogate model, which is a probabilistic representation of the objective function, using only a few example points.

Bias Vector

The bias vector is a set of values (one for each node in the layer) that is added to the weighted input of the layer before the activation function is calculated. Note that the use of “bias” here is mathematical and not related to any potential biases in the model’s decision making.

Binary Cross-Entropy

Binary cross-entropy (BCE) penalizes the model based on how confident it is in the wrong answer. BCE requires the output to be interpretable as a probability — it asks “is this pixel foreground or background?” and treats all wrongly classified pixels the same way, whether they are at a cell boundary, inside a hole, or belong to an entirely missing cell. For ground truth label \(y_i \in \{0,1\}\) and predicted probability \(\hat{y}_i \in [0,1]\): \[\text{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \bigl[ y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i) \bigr]\]

Binary Segmentation

A type of image segmentation where each pixel is classified into one of two categories — typically “foreground” (e.g., cell) or “background.” The output is a binary mask distinguishing objects (set to a value of 1) from their background (0).

Boosting Algorithm

An ensemble method that builds a strong classifier from many simple ones, typically small decision trees, trained one after another so that each corrects the errors its predecessors left behind.

Bounding Region

A geometric shape, typically a box, drawn around an object in an image to indicate its spatial extent.

CARE (Content-aware Image Restoration)

A deep learning-based method for image restoration that leverages content-specific features to enhance degraded images. See https://github.com/CSBDeep/CSBDeep for more information.

Capacity

A model’s capacity reflects how much information a network can store. It corresponds to a network’s ability to model an arbitrary function.

Centroid Coordinate

An object’s geometric center point, used as a simple way to describe its location in an image.

Channel Attention

Networks that use channel attention give individual weights to each channel in an input. This allows networks to learn the importance of individual channels.

Channels

In natural color images, there are often three channels (red, green, and blue). In microscopy images, the number of channels can refer to the number of fluorescent labels or image settings used while imaging. The term is used similarly in neural networks, where channels refers to the number of layers in an image. These channels can come from convolutional operations, leading to images with hundreds of channels corresponding to each filter used.

Class

A class describes the group or category an image, pixel, or object falls into. This includes distinguishing dataset types such as “control” or “mutant,” as well as distinguishing “foreground” and “background” pixels.

Class Label

A descriptive name selected from a predefined list of categories, assigned to a detected object or event to describe its identity.

Classification

The process of assigning a data point to one of a set of predefined categories, based on its measured features.

Clustering

Clustering groups unlabeled points within a dataset based on similarities or patterns in the data.

Coding Agent

Coding agents are AI systems that operate directly in development environments with access to terminals, file systems, and web browsers. These agents go well beyond earlier LLM assistants that could only suggest code snippets.

Computer Vision

A field of computer science wherein computers extract information from images. It often involves object detection within images and can involve classification of the images and/or objects.

Computing Cluster

A computing cluster connects individual computers (nodes) into a fast network that allows them to act as a more powerful computer.

Confidence Score

A continuous value, often between 0 and 1, that reflects how certain a model is about a given prediction. Unlike a distance transform, which encodes geometric information such as position within an object, a confidence score expresses the model’s belief that a prediction is correct — for example, that a pixel belongs to the foreground class or that a detected object was correctly identified during object detection. Confidence scores are useful for filtering predictions, such as discarding low-confidence detections, but a high confidence score does not guarantee a correct prediction, particularly when a model is poorly calibrated.

Context Window

A context window is the size of the largest input that a model can consider at once. Making the context window larger allows the model to incorporate more information into the output.

Contrastive Loss

A contrastive loss function attempts to minimize the distance between input pairs that are similar while simultaneously maximizing the distance between input pairs that are dissimilar.

Convolution

A mathematical process where a kernel (small matrix) slides over input data (e.g., images) to compute feature maps, highlighting patterns like edges or textures.

Convolutional Neural Networks (CNNs)

A deep learning architecture that applies convolutions to automatically learn features from images for computer vision tasks like classification and detection.

Coordinate transformation

A mathematical process for expressing the position of a point from one coordinate system to another. For example, knowing how many pixels were removed from a cropped image allows for calculating the location of a cropped pixel in the original image.

Cytoprofile

A numerical feature vector describing a biological cell’s phenotype, derived from measurements of shape, intensity, and texture extracted from microscopy images.

Data Augmentation

A strategy to artificially increase the diversity of a dataset prior to training by applying transformations such as rotation, flipping, or brightness adjustment. It helps improve model robustness and generalisation.

Data Leakage

Information from validation or test examples that influences model training or selection, often because related samples, duplicates, or preprocessing statistics cross dataset splits.

Data Loader

The data loader is a software component that connects the raw training data files with the training pipeline. It can manage tasks such as batching and efficiently feed samples to the model.

Data Provenance

The record of how each file in a dataset came to be and what produced it, forming a graph in which every file is a node and every processing step an edge back to the files it was derived from. Acquisition is the first node, holding the specimen, instrument, and settings that produced the raw image. Later nodes cover processing steps with their order and parameters, targets with the annotator, model, or script that generated them and the review they received, and the assignment of examples to splits. Provenance is what allows a result to be traced back to the raw data, corrected, and reproduced.

Decision Tree

Decision trees are flow-chart-like models that make simple yes/no decisions in a series of conditional steps. The first root node splits the entire dataset into two groups, after which internal nodes and branches continue to split the data through conditional questions, finally terminating in a leaf node that provides the answer, class, or prediction.

Deconvolution

A mathematical process to partially reverse the blurring effect caused by the microscope’s PSF, increasing contrast and resolution over the raw image data if performed carefully.

Deep Learning

Deep learning is a type of machine learning that uses multi-layer neural networks to perform tasks. The term “deep” refers to the use of many layers in the network.

Denoising

Denoising is a process in which unwanted noise or distortions are removed from an image or dataset. The goal of denoising is to preserve the underlying signal while doing so.

Detection

Detection finds where the objects are without defining their full shape. This requires simpler annotations (e.g., a point or box) than full segmentation.

Differentiable Function

A differentiable function is a function whose derivative exists at every point within the function’s domain.

Dimensionality Reduction

Taking a dataset that is in a high-dimensional space and reducing it to a lower-dimensional representation that tries to retain as much of the important information and underlying properties of the high-dimensional data as possible.

Distance Transform

A transformation that assigns each foreground pixel a value representing its distance to the nearest background pixel or object boundary, producing an image that peaks at object centers and falls to zero at their edges. In instance segmentation, distance transforms are commonly used to help separate touching or overlapping objects — for example, as a target the model learns to regress during training, or as an input to watershed-based post-processing that splits a binary mask into individual instances.

Domain Randomization

Using simulations or synthetic training data, domain randomization applies random and exaggerated variations to background, lighting, shapes, or textures in the synthetic dataset. This strategy helps the model learn domain-invariant features and is usually used for pretraining a neural network or to enable simulation-to-real transfer.

Domain Shift

Domain shift occurs when a model is trained and tested on one type of data, but later applied to a related but different data distribution. In microscopy, an example of this is training a model on data from one microscope, but then applying it to data acquired using a different system or technique.

Downsampling

Downsampling is a technique to reduce the sampling of input data. For an image, this involves descreasing the number of pixels, and thus decreasing the resolution of the image.

Effect Size

How “strong” a phenotype is, or how mathematically possible it is to distinguish a given population from the control population.

Embedding Space

The space of lists of numbers (embeddings) that a model uses to describe its inputs. Also known as latent space or latent feature space.

Encoder-Decoder

An encoder-decoder architecture refers to a network with two parts. The “encoder” component of the network is a series of layers that take the input and map it to an abstract representation. The second “decoder” component of the network is trained to recreate input data from the abstract representation. This type of architecture learns an efficient way to represent the input data and is often used for dimensionality reduction.

End-to-end Learning

A deep learning approach in which all stages of a task (e.g., locating and identifying objects) are optimized jointly within a single model, rather than being handled as separate, independent steps.

Ensemble

A machine-learning approach that combines predictions from multiple individual models to produce a single result that can be more robust than any single model alone.

Epoch

One complete pass through the entire training dataset during the training process.

Event

A set of spatial features that change or transition over time, temporally defined by a beginning and an end.

Event Detection

The algorithmic identification of temporal changes or transitions in a system, requiring analysis of data across time.

F1 Score

A classification metric that gives the harmonic mean of precision and recall. This metric was originally designed for binary classification but can be adapted to multiclass classification by calculating the F1 score per class. \[ \text{F1} = \frac{2\,\text{TP}}{2\,\text{TP}+\text{FP}+\text{FN}}\] This metric is also known as the DICE coefficient.

False Negatives

In a scenario where you have two classes “positive” and “negative”, you try to predict cases as one of those classes. False negatives are the cases that you incorrectly predicted as negative and were really positive. For example, a cell that was missed by a segmentation approach is a false negative: the cell is there (positive), but the segmentation is negative.

False Positives

In a scenario where you have two classes “positive” and “negative”, you try to predict cases as one of those classes. False positives are the cases that you incorrectly predicted as positive and were really negative. For example, a cell that appears in the segmentation (positive) but does not actually exist in the image is a false positive.

Feature vector

A set of numerical measurements (such as size, shape, intensity, or texture) that together provide a quantitative description of an object.

Fiji

An image processing platform that comes bundled with many plugins for scientific image analysis. See https://imagej.net/software/fiji/ for more information.

Fine-tuning

Fine-tuning takes a pre-trained model and adapts it to a new task by providing task-specific training data. In classic fine-tuning, all layers of the pre-trained model are retrained, with their weights initialised from the original training. Thus, you continue training the full model using the new data. This approach allows the model to adjust more comprehensively to new data while still preserving the advantages of pre-learned features.

Forward Pass

The process by which an input is fed through each layer of a neural network, in a sequential manner. The result of the final layer is the output which can be compared to ground truth.

Foundation Model

A foundation model is a model that was trained on a very large and very diverse dataset. Because of this diversity in training data, foundation models can be used for many downstream tasks or can be fine-tuned for specific purposes.

Frequency Domain

The representation of an image as a function of spatial frequency, obtained by transforming an image from the spatial domain using a Fourier transform.

Function Calling

Function calling is also called tool use. This is the ability for LLMs to invoke external tools, APIs, and software through structured interfaces rather than generating plain text.

Gaussian Process

A common surrogate model for optimization strategies such as Bayesian Optimization. A Gaussian Process is a non-parametric model that represents a probability distribution over possible functions. In the hyperparameter search scenario, the Gaussian Process models the probability of getting an objective function value based on some hyperparameters.

General Agency

General agency gives AI agents access to broad capabilities — a terminal, a file system, APIs, analysis libraries — and lets them determine the right sequence of actions for each task. Rather than pre-scripting workflows, the agent receives a goal (“segment and count the nuclei in this dataset”) and works out what tools to use, in what order, handling errors as they arise.

Generative Adversarial Networks (GANs)

A deep learning architecture where two neural networks, a generator and a discriminator, are trained in an adversarial process, enabling the generator to create synthetic data, such as realistic images, by learning to deceive the discriminator.

Generative Model

A model trained to draw samples from the distribution of possible outputs, rather than to return their mean. Running it twice on the same input can give two different answers. Each sample has the texture and contrast of real data, unlike the smooth mean, but no single sample represents the whole distribution. Variational autoencoders, generative adversarial networks, diffusion models, and flow matching are common examples.

Genetic Algorithms

An optimisation method inspired by the principles of natural selection and genetics. It starts with a population of solutions. These solutions are combined through a process called crossover to produce new solutions (offspring). During this process, random changes or mutations may occur to introduce diversity. After crossover and mutation, a selection step chooses the best solutions from both the parent and offspring populations to form the next generation. This cycle repeats for a set number of generations or until a predefined goal or stopping criterion is met.

Gradient Descent

Gradient descent computes the gradient of the loss with respect to each weight. Moving the weights in the negative direction of the gradient reduces the loss for the given images or data points over which the loss is computed.

Ground Truth

Accurate data against which a model can be evaluated. Ground truth data is often manually annotated. The data type itself will vary depending on the task and evaluation. For example, instance segmentation may be compared to ground truth object counts or masks.

Hallucinations

Outputs from a model that do not have a basis in the input data and may contain false or misleading information.

Harmonic Mean

A type of average of a set of numbers, calculated as the number of values divided by the sum of their reciprocals. Unlike the arithmetic mean, the harmonic mean is closer to the smaller of the values being averaged, so it penalizes cases where one value is much lower than the other. This makes it useful for combining two metrics, such as precision and recall, into a single score that only looks good when both metrics are good.

Hausdorff Distance

The Hausdorff distance measures how far apart two sets of boundary points are at their worst-case point. For boundary point sets \(A\) and \(B\):

\[d_H(A, B) = \max\!\left(\sup_{a \in A} \inf_{b \in B} d(a,b),\; \sup_{b \in B} \inf_{a \in A} d(a,b)\right)\]

A small Hausdorff distance means every point on one boundary has a close counterpart on the other.

Human-in-the-loop

Human-in-the-loop workflows use model predictions that are manually corrected for retraining. This iterative approach can be used to supervise the model output and improve accuracy.

Hyperparameters

The options you choose when training a machine learning model that affect the training process or the architecture of the model (e.g., learning rate, batch size, number of layers, training loss) are called hyperparameters. This term is used to differentiate them from the parameters (also known as weights) of the machine learning model.

Image Classification

A computer vision task where each image is associated with one class and the goal of this task is to correctly predict that class.

Image Restoration

The process of recovering clear, high-quality images from degraded raw data contaminated by blur, noise, or other distortions.

Image Translation

Image-to-image translation (or style transfer) is a generative AI technique which transforms an input image into the style of another image. In the microscopy context, this could manifest as a network which transforms an image of one modality (e.g., brightfield) into a different modality (e.g., DAPI staining of nuclei) in the same field of view.

Inference

After a model has been trained, it can be applied to new, unseen data. The process of applying the model to this new data is known as inference.

Instance Segmentation

A segmentation task that not only separates objects from the background but also distinguishes between individual objects of the same type (e.g., separating touching cells one by one).

IoU

IoU stands for “Intersection over Union”. It is a segmentation metric that calculates the area of overlap between two segmentation masks divided by the area of their union.

Kernel

Convolutional kernels are also known as filters. They are small matrices that define the function for taking input pixels and creating an output image. They are often used for tasks like blurring or edge detection.

Large Language Model (LLM)

A foundation model for text, pre-trained on a vast amount of text to predict the next word, and usually further trained to follow instructions. Users interact with LLMs in natural language to accomplish a variety of tasks.

Learning Rate

The amount weights are updated at every iteration during gradient descent. Metaphorically, this represents how fast the network “learns” from the current state at each iteration.

Learning Rate Schedule

A programmatic change in the learning rate over the course of an entire training run. This often involves reducing the learning rate near the end of training to enable smaller steps down the gradient, fine-tuning the weights near the loss minimum.

Linear Layer

Linear layers are also known as fully-connected layers or dense layers. They are a set of input nodes (or neurons) that are each connected to every output node of the layer.

Localization

The task of determining the spatial position of an object or event within an image.

Logprobs

Logprobs is short for log probabilities. Taking the logarithm of a probability has practical computational advantages. In the AI context, logprobs refers to the probability of each token, given the previous token, and can be used as a measure of model confidence.

Long-short Term Memory (LSTM)

A type of recurrent neural network designed to retain information over longer sequences, addressing limitations of standard RNNs in capturing long-range temporal dependencies.

Loss Function

A loss function (or cost function) is a formula for quantifying how much a model’s prediction deviates from the actual ground truth value. The loss function returns a single scalar number to quantify the loss. As training relies on gradient descent, the loss must be differentiable; the algorithm needs the slope of the loss with respect to every parameter to know which direction to step.

Machine Learning

A type of artificial intelligence in which a person designs a family of functions with free parameters, and an optimization procedure fits the parameters to a set of examples. The rules are learned from data rather than written down by hand, and the goal is that they generalize to new data.

Manual Annotation

The process of manually labeling specific structures or objects in an image using drawing tools. Typically done in software like Fiji or napari, this step is essential for creating ground truth data to train or evaluate machine learning models.

Mean Squared Error

Mean-squared error (MSE) penalizes the model proportionally to the squared difference between its prediction and the ground truth: \[\text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2\] MSE measures how far the prediction is from the target value, and does not require the output to be a probability — it works equally well when the model produces a continuous map such as a distance transform or a confidence score. Because it sums squared errors over all pixels, it is sensitive to the total extent of the error: predicting an entire missing cell as background accumulates far more MSE than a few wrongly-classified boundary pixels.

Metadata

Any data that provides additional information about other data. In bioimaging, examples include information about sample preparation, the imaging instrument, and image acquisition parameters.

Model Collapse

Model collapse occurs when performance degrades due to training on uncurated or synthetic data, especially training on data from another model (including the same model at prior training steps).

Model Context Protocol (MCP)

The Model Context Protocol (MCP) is a standardized way for AI systems to connect to external tools and data sources. MCP gives any AI model a universal way to discover and use any compatible tool — analogous to how USB standardized device connectivity. For microscopy, MCP means that a single agent could potentially access a microscope control system, an image analysis library, a literature database, and a model repository through the same protocol.

Momentum

In the context of learning rate, momentum speeds up learning when the gradient is consistent over multiple iterations, which allows the training to avoid local minima.

Multilayer Perceptron

A sequence of fully-connected layers that are applied in series, each on the output of the previous one, is called a multilayer perceptron (MLP). This is the simplest example of a deep neural network.

N2N (Noise2Noise)

A denoising method that trains a neural network on pairs of independently noisy images of the same scene, one as input and the other as target. It needs no clean reference image, but it needs the pairs. See https://github.com/NVlabs/noise2noise for more information.

N2S (Noise2Self)

A self-supervised denoising method that trains a neural network assuming statistically independent noise across the image, requiring only single noisy images without paired clean data. See https://github.com/czbiohub-sf/noise2self for more information.

N2V (Noise2Void)

A self-supervised denoising method that trains a neural network to predict pixel values from noisy images by masking input pixels, requiring only single noisy images without paired clean data. See https://github.com/juglab/n2v for more information.

NaN

Not-a-number (NaN) values are used where an operation results in something that is not a real number (such as dividing zero by zero).

Natural Image

Natural images are images of the environment. This most commonly refers to color photographs of people, nature, objects, etc. Many machine learning researchers focus on natural images for tasks like self-driving cars, but microscopy images pose additional challenges for AI.

Network Architecture

The architecture of a machine learning model refers to the design and structure of the model, including choices in how individual components (such as preprocessing or feature extraction) are connected to each other.

Neural Network

A function built from many simple units, each adding up weighted inputs and passing the result through a simple nonlinear response, arranged in layers. The weights are the parameters learned during training. The name is a historical analogy to networks of biological neurons.

Neural Network Unit

A neural unit is the processing element of a neural network. It also sometimes called a node or artificial neuron.

Nonlinear

If a system or function is non-linear, changes in its output are not proportional to the changes that are made to the input. Non-linear functions are incorporated into machine learning networks to allow for the solution of complicated problems because it allows for non-linear decision boundaries.

Nonlinear Problem

A mathematical problem where the governing equations or operations are nonlinear, meaning outputs are not linearly proportional to inputs.

Normalization

Data normalization is a pre-processing step to prepare input images (or features) such that their values are in a common range. Values are often rescaled to [0,1] or [-1, 1] to avoid models learning about absolute values of intensity rather than image features.

Nyquist Sampling

The Nyquist sampling rule states that recovering a continous signal requires sampling it at twice its highest frequency to accurately reconstruct it. In microscopy, Nyquist sampling thus requires that pixel sizes should be smaller than half the relevant optical resolution.

OME-Zarr

OME-Zarr is a next-generation file format (NGFF) developed by the Open Microscopy Environment. The Zarr format stores images in smaller chunked image arrays, which allows for efficient access, storage, and processing of large datasets, while OME provides standards for accessing microscopy metadata.

Object

A discrete spatial feature within an image, spatially defined by properties such as size, shape, intensity, and location.

Object Detection

A computer vision task that identifies and locates individual objects within an image, typically by drawing bounding boxes around them. It provides both the category (what) and position (where) of each object.

Object Storage

Object stoarge is a computer storage architecture that views data as blobs or objects, rather than a traditional file system file hierarchy. This has advanatges for indexing and metadata management.

Optimizer

An optimizer is a function that is used to adjust model parameters during training. By adjusting weights and biases, the optimizer attempts to minimize the loss function and thus make the model more accurate.

Overfitting

When a model fits its training data so closely that it memorizes noise and incidental details, and no longer generalizes well to new data.

Padding

Padding involves adding extra layers of pixels around an image to avoid edge effects. Padding can be performed by simply setting the extra pixels to zero, or it can take into account the image itself (such as in mirror padding or replicate padding).

Panoptic Segmentation

A computer vision technique that is a combination of semantic segmentation and instance segmentation. It separates an image into regions while also detecting individual object instances within those regions.

Patches

In image processing, a patch refers to a small region or subset of pixels that are extracted from the larger image.

Perceptron

Perceptrons are a specific type of an artificial neuron or node. They perform a calculation that is a linear weighted sum of the inputs to the node followed by a nonlinear activation function. They are used for binary classification, and can be combined to create more complex network architectures.

Perceptual Loss

Instead of comparing pixel intensity, perceptual loss functions compare high-level features — such as those defined in the embedding space — of images between model outputs and ground truth. This allows models to produce outputs that match the high-level structure and texture of the ground truth that human perception encodes.

Pixel Classifiers

Machine learning models that classify each pixel in an image based on features such as intensity, texture, or local neighborhood. Commonly used in traditional workflows for segmentation or classification tasks.

Point Spread Function (PSF)

A mathematical function that describes how an imaging system blurs a point source.

Pooling Layer

A pooling layer aggregates information from many vectors into fewer vectors, i.e., it downsamples the image. The pooling layer does this using sliding windows across the image of maximum or average operations. This removes redundant information and reduces the size of the image for the following calculations.

Pre-training

The first and most expensive stage of training, in which a model learns a general-purpose representation from a large dataset, usually without labels by self-supervised learning. The pre-trained model is then adapted to a specific task with far fewer labeled examples, by fine-tuning or semi-supervised learning. Foundation models are pre-trained models.

Precision

A classification metric that measures the proportion of predicted positives that are correct: true positives divided by the sum of true positives and false positives. Precision answers “of everything the model called positive, how much actually was?” A model can achieve high precision by only making positive predictions it is very confident about, at the cost of missing some true positives (i.e., lower recall). \[ \text{Precision} = \frac{\text{TP}}{\text{TP}+\text{FP}}\]

Probability Vector

A probability vector is composed of values that indicate the probability of a particular variable (e.g., the probability that an image is a dog or cat). The values of a probability vector sum to one.

Pseudo-label

Unlike a manually annotated label, a pseudo-label is created by a model. In semi-supervised learning, this allows a model to learn from a few manually annotated labels and large amount of unsupervised data. However, pseudo-labels can be biased by mistakes in the model and should validated in the same manner as manual annotations.

Quality Control Metric

Any metric that can be used to evaluate quality. It will vary depending on the task and data type. It can be binary (e.g., an image doesn’t have debris) or continuous (e.g., annotated object centroids are within 5 pixels of the ground truth centroids).

RCAN (Residual Channel Attention Network)

A deep learning-based method using residual learning and channel attention to improve image restoration tasks. See https://github.com/AiviaCommunity/3D-RCAN for more information.

Random Forest

An ensemble of decision trees, each trained on a random subset of the examples and features, whose votes are combined into a single prediction. Widely used for interactive pixel classification, e.g., in ilastik and Labkit.

ReLU

An activation function common in deep learning that outputs the input directly if it is positive, and outputs zero otherwise. That is, it is the ramp function where it equals x if x>0 and 0 if x<=0. This characteristic helps introduce non-linearity into the model and mitigate the vanishing gradient problem. ReLU stands for rectified linear unit.

Real-time Analysis

An approach in which images are processed as the data is acquired. This is in contrast to a-posterior analysis, where the data is acquired before processing.

Recall

A classification metric that measures the proportion of actual positives that were correctly identified: true positives divided by the sum of true positives and false negatives. Recall answers “of everything that was actually positive, how much did the model find?” A model can achieve high recall by predicting positive more liberally, at the cost of more false positives (i.e., lower precision). Also known as sensitivity or the true positive rate. \[ \text{Recall} = \frac{\text{TP}}{\text{TP}+\text{FN}}\]

Receptive Field

The receptive field size of a convolutional layer is the total spatial extent of pixels that influence the activations of the layer. The receptive field of the entire network describes the size of the region in the input image that influences the output image.

Recurrent Neural Networks (RNNs)

Neural network architectures designed to process sequential data by maintaining an internal memory of previous inputs, used to model temporal dependencies.

Region of Interest (ROI)

A defined area within an image or specimen selected for closer analysis, imaging, or targeted intervention.

Regularization

Regularization adds a penalty term to the loss function to prevent overfitting. This term discourages overly complex models.

Reinforcement Learning

In reinforcement learning, AI agents learn how to make decisions by acting and receiving rewards or penalties for the consequences, adjusting their behavior to collect more reward. No examples of correct behavior are needed.

Representation

The internal description of an input that a network computes on the way to its output: a list of numbers that summarizes what the network has learned to consider relevant about an image or a text. A good representation can be reused for tasks the network was not trained on, which is what makes foundation models useful.

Representation Collapse

A failure of self-supervised learning in which the network describes every input with the same, or nearly the same, embedding. This trivially satisfies objectives that only reward agreement between views, but the network learns nothing. Not to be confused with model collapse.

ResNet

A residual neural network (ResNet) is a deep learning architecture in which skip connections between layers are used to learn residual mappings instead of direct transformations, increasing training efficacy and deeper networks.

Resampling

In the context of CNNs, resampling describes changing the spatial dimensions of the image or features maps through either downsampling or upsampling. For example, an input 2D image could be resampled to be half the size in x and y.

Residual Connection

A specific instance of a skip connection, in which the input of a transformation block is added to the output.

Residual Learning

In conventional architectures, the models ideally learn to map inputs \(x\) to the optimized output \(O\). In residual learning, the implementation of residual connections adds \(x\) to the output. As a result, the network output now becomes \(O - x\). In words, the network learns the difference between the optimial output and the input. This can minimize the effect of large gradients, improve performance, and accelerate training.

Resolution Enhancement

Deep learning methods that attempt to predict higher resolution images given lower resolution input data.

Retrieval-Augmented Generation

A technique that allows LLMs to incorporate new supplmental information from outside the training set. This increases the flexibility of LLMs to respond to domain-specific queries.

Segmentation Mask

A segmentation mask is an image where each pixel is given a corresponding class. For example, this could be a black and white image where cells are white and the background is black.

Self-attention

Attention allows a model to determine the importance of each component of an input sequence. Self-attention is a type of attention mechanism that is commonly used in transformer architectures.

Self-supervised learning

A form of machine learning that turns unlabeled data into a supervised problem, typically by hiding part of the data and training the model to fill it in. The model either is used directly, as in self-supervised denoising, or provides a representation that is later adapted to other tasks.

Semantic Segmentation

A form of segmentation where each pixel in an image is assigned to a class (e.g., nucleus, cytoplasm, background), but it does not distinguish between separate instances of the same class.

Semi-Supervised Learning

Training on a small amount of labeled data combined with a large amount of unlabeled data, for example by letting the model label the unlabeled examples itself (pseudo-labels) and learning from those as well. It often starts from a model after pre-training without labels. Not to be confused with weak supervision, which trains on coarse or noisy labels.

Shallow Decision Trees

Simple, low-complexity decision tree models that make classification decisions using only a few splits, often used as building blocks within ensemble methods.

Shape Descriptors

A set of quantitative measurements that characterize the geometric form of an object, such as circularity, elongation, or perimeter.

Shortcut Learning

Shortcut learning occurs when a network learns about superficial or trivial features in the data rather than learning the important underlying concepts. For example, a classifier whose control wells all sit at the edge of a plate may learn to recognize the edge position rather than the phenotype, and networks trained on chest radiographs can pick up which hospital acquired an image.

Sigmoid Function

An activation function common in deep learning that non-linearly maps real inputs to outputs between 0 and 1, being most sensitive to changes in inputs around zero and increasingly compressing extreme positive or negative inputs as they approach 1 or 0 respectively; this characteristic enables it to model probabilities for binary classification and introduce smooth non-linearity.

Signal-to-noise Ratio

The signal-to-noise ratio (SNR) measures the level of the desired signal to the unwanted noise or background. In fluorescence microscopy, this often relates the intensity of the labeled object to the intensity of the image background.

Silent Error

A silent error occurs when a model outputs the wrong answer or otherwise fails without generating any warning or message.

Single-shot Detection

An object detection approach in which localization and classification are performed simultaneously in a single pass through the model, rather than through separate sequential stages.

Skills Atrophy

Skills atrophy is the decrease or loss of human expertise due to underuse. In this context, skills atrophy could be caused by an over-reliance on AI tools. This gradual erosion of the community’s ability would decrease the field’s ability to evaluate AI outputs.

Skip Connection

Skip connections are also sometimes known as residual connections. These connections are made when the output of a layer of the model is added to or concatenated with a later layer in the network while bypassing any layers that may be in between. For example, the layers in a U-Net architecture connect across image resolutions.

Smart Microscopy

An imaging approach in which acquisition parameters are automatically adjusted based on features or events recognized in the data, rather than relying solely on manual operator control.

Spatial Domain

The representation of an image as a function of spatial coordinates.

Star-convex Polygon

A geometric shape used in segmentation algorithms like StarDist. Imagine drawing straight lines (rays) from the centre of an object out toward its edges — if you can see the edge from the centre in all directions, the object is considered star-convex. This method works well for blob-like structures such as nuclei, because their general shape can be captured by measuring how far each ray travels from the centre to the boundary.

Stochastic Gradient Descent (SGD)

As most modern problems require large training datasets to effectively capture underlying signals, attempting to calculate the gradient over the entire dataset is often computationally prohibitive. This is overcome by batching the training data into smaller subsets. This naturally injects some stochasticty (randomness) into the gradient descent procedure based on how the batches are randomly determined.

Stride

When performing a convolution or cross-correlation, the kernel (aka the filter) moves across the entire image. When the stride is 1, this corresponds to moving the filter across every pixel in the image. For larger values of the stride, the kernel is moved more than one element at a time. For example, a stride of 2 would skip every other pixel in the image.

Structural Similarity Index (SSIM)

The structural similarity index measure (SSIM) is a metric which measures the similarity between two images. It attempts to improve upon MSE by taking image texture into account. It is calculated using the following formula.

\[\text{SSIM} ( x , y ) = \frac{( 2 \mu_{x} \mu_{y} + c_{1}) ( 2 \sigma_{xy} + c_{2})}{( \mu^{2}_{x} + \mu^{2}_{y} + c_{1} ) ( \sigma^{2}_{x} + \sigma^{2}_{y} + c_{2} ) }\]

where \(x , y\) are two images, \(\mu\) is the average intensity, \(\sigma\) is the variance of the intensity, and \(c_{1} , c_{2}\) are variables determined by the dynamic range of the image (\(c_{i} = ( k_{i} L )^{2}\) for \(k_{1} = 0.01 , k_{2} = 0.03\)).

Supervised Learning

A form of machine learning in which every training input comes paired with the correct answer (the target), and the model learns to produce that answer from the input. The target can be a class, a position, a mask, or another image.

Support Vector Machine (SVM)

A supervised machine-learning algorithm that learns the decision boundary that best separates data points of different classes by maximizing the margin between them.

Tensor

Tensors are multi-dimensional matrices that are used in many machine learning contexts as a way of organizing data.

Test Data

Data held out from both the training data and validation data, used only after training is complete to give an unbiased estimate of how a model will perform on new, unseen data. Because it plays no role in training or hyperparameter tuning, test data should only be used once a model is finalized.

Texture

A quantitative descriptor of the spatial pattern of intensity variation within a region of interest, capturing properties like smoothness or granularity.

Tokens

Tokens are units of data processed by an AI model. For an LLM, these tokens are chunks that may be whole words or subword pieces. For instance, “microscope” might be a single token, while “fluorescence” might be split into “fluor” + “escence.”

Tool Definitions

Tool definitions are structured descriptions of available functions, their parameters, and expected inputs. When a user makes a request, the model reasons about which tools are needed, generates structured calls with correct parameters, receives results, and decides whether to take further action or report findings.

Tracking

Tracking links objects over time. It requires both finding objects and determining how they are associated over time.

Training Data

Data used to teach a machine learning model to perform a task, such as classification, segmentation, restoration, or generation. Depending on the method, training data may be paired with target labels or ground truth (as in supervised learning) or used on its own without labels (as in self-supervised or unsupervised learning).

Training Target

A target is the feature or outcome that a model should output. This could be a label for an image, a segmentation, or even another image. In supervised learning, the model is trained on input images or features paired with a corresponding target.

Transfer Learning

A deep learning technique that reuses a model pre-trained on one task as the starting point for a new, related task, leveraging its learned knowledge to improve performance or reduce training requirements. In practice, part of a pretrained neural network (usually the initial layers, responsible for feature extraction) is frozen and reused in a new model. These frozen layers, with the knowledge from a previous dataset, are combined with untrained layers tailored for a specific bioimaging task. During training, only the new layers will be updated, allowing the model to adapt to the new task with limited data.

Transformer Models

A deep learning architecture, originally developed for language, in which every part of the input is compared with every other part (attention), so that distant regions can inform each other. A vision transformer (ViT) cuts an image into small patches and treats them like the words of a sentence.

Transposed Convolution

As in a normal convolution, a transposed convolution involves an image and a kernel. However, in this case, the image slides over the kernel (as opposed to the other way around). This naturally results in a spatially upsampled output, relative to the input. The stride and padding determines the upsampling factor. In particular, if the input is of size \((I_h , I_w)\), then a transposed convolution with kernel size \((k_h , k_w)\), padding size \((p_h , p_w)\), and stride \((s_h , s_w)\) will result in an output of size \((O_h , O_w)\): \[ \begin{aligned} O_h &= (I_h - 1) s_h + k_h - 2 p_h \\ O_w &= (I_w - 1) s_w + k_w - 2 p_w \end{aligned} \] This is often used in CNN architectures such as the U-Net to produce outputs at the correct spatial scale. See this example for more information.

True Negatives

In a scenario where you have two classes “positive” and “negative”, you try to predict cases as one of those classes. True negatives are the cases that you predicted as negative and were really negative. For example, if the network correctly lacks a segmentation in an empty/background region of an image, the segmentation is correctly negative for the negative region of the image.

True Positives

In a scenario where you have two classes “positive” and “negative”, you try to predict cases as one of those classes. True positives are the cases that you predicted as positive and were really positive. For example, if the network correctly segments a cell in the image, the segmentation is positive for the positive region of the image.

U-Net

A U-Net is a type of convolutional neural network that was specifically developed for the task of image segmentation. It has since been used for a variety of tasks. The model is so named because downsampling layers are followed by upsampling layers, which are each connected by skip connections, and together this gives diagrams of the network a U shape.

UUID

A Universally Unique Identifier (UUID) is a number used to identify a file or information. When generated using UUID standards, these 128-bit numbers are unlikely to be duplicated even across independent users and thus can be used without worrying about confusion with other files.

Unsupervised Learning

Unsupervised learning is a type of machine learning in which the machine works only with the raw, unlabeled data. It uses algorithms to find structure or patterns in the data without user guidance or predetermined categories.

Upsampling

Upsampling is a technique to increase the sampling of input data. For an image, this involves increasing the number of pixels and interpolating values for the newly sampled pixels.

Validation Data

Data set aside from the training data and used to evaluate a model’s performance during training, without being used to update the model’s weights. It is typically used to tune hyperparameters and monitor for overfitting, for example by deciding when to stop training.

Virtual Machine

On a physical computer, you install an operating system (e.g., Windows or Ubuntu) that you interact with. A virtual machine is a program that simulates a complete computer with its own operating system. This lets you run a “computer inside your computer” (e.g., using Linux inside Windows or the other way around). As this simulated computer is separate from your physical one, it adds an extra layer of security, because unless the user specifically allows it, the virtual machine cannot access or connect to your real computer.

Vision-Language Model (VLM)

A Vision-language Model (VLM) is a type of model that can process both images and text with a single architecture. Because they can accept both natural language and visual inputs, they can include context in their analysis.

Visual Question Answering

Visual Question Answering (VQA) is when a model is asked an open-ended question based on an image input. The result is a natural language response to a natural language question associated with an image.

Weakly Supervised Learning

Weakly supervised learning takes partial, noisy, or imprecise labels to guide training. These labels are often combined with unlabeled data which the model learns from in an unsupervised manner. This can significantly reduce the annotation time and speed up deployment of a model.

Wiener Filter

Wiener filters are used to estimate an unknown underlying signal from an observed noisy response. For an example use-case of this filter in image processing, see this MATLAB implementation.

Zernike Modes

A set of orthogonal polynomials used to describe and correct wavefront aberrations in optical systems.