Computer Vision

Computer vision turns pixels into useful information. A computer vision system can assign a label to an image, locate objects, mark each pixel by category, track motion, or estimate structure. The output supports another decision or action.

Use this mental model:

scene → sensor → pixels → preprocessing → model → prediction → product decision

The model is only one part of the system. Lighting, camera position, labels, preprocessing, decision thresholds, and the people affected by the result can matter as much as the model architecture.

What an image becomes

A digital image is an array of numeric pixel values. A color image often has separate channels for red, green, and blue. A video adds a time dimension. Frameworks represent these arrays as tensors so models can process batches of images efficiently.

Pixels do not carry meaning by themselves. The same object can occupy different positions, scales, and orientations. Its appearance changes with lighting, background, motion blur, occlusion, lens behavior, and sensor noise. A useful system must handle the variation expected in its operating environment.

Preprocessing puts inputs into the form a model expects. Common operations include resizing, cropping, color conversion, and normalization. Training-time augmentation creates altered examples, such as crops or flips, to expose the model to relevant variation. An augmentation is valid only when it preserves the intended label. A horizontal flip may be sensible for a flower, but wrong for text or a directional traffic sign.

Choose the task before the model

The required output defines the task:

Task	Output	Representative use
Image classification	One or more labels for an image	Sort product photos
Object detection	A class and bounding box for each object	Count vehicles
Semantic segmentation	A class for each pixel	Mark drivable road area
Instance segmentation	A separate pixel mask for each object	Measure individual cells
Keypoint estimation	Named landmark coordinates	Estimate body pose
Tracking	Object identities across frames	Follow items on a conveyor
Optical character recognition	Text recovered from an image	Read a document scan
Depth or geometry estimation	Distance, pose, or 3D structure	Support robotic navigation

Do not start with a fashionable architecture. Start with the decision the system must support and the output needed to make that decision. Classification cannot tell you where an object is. Detection gives coarse location but not an exact boundary. Segmentation adds spatial detail at a higher annotation and compute cost.

Two families of methods

Classical computer vision uses explicit image operations and hand-designed features. Examples include thresholding, filtering, edge detection, geometric transforms, feature matching, and camera calibration. These methods remain useful when the visual conditions and rules are stable, data is scarce, or you need interpretable geometry.

Learned computer vision fits model parameters from examples. Convolutional neural networks learn spatial feature hierarchies. Vision transformers process image patches with attention. Modern systems often combine learned models with classical operations before or after inference.

The choice is not ideological. Use the smallest method that meets the product requirement. A fixed threshold may beat a trained model for a controlled inspection task. A pretrained learned model may be a better starting point when appearance varies widely.

The data defines the problem

A dataset contains images and ground truth. Ground truth is the reference label, box, mask, landmark, or other target used for training and evaluation. Label rules must match the product question. If two annotators interpret a class differently, the model receives a contradictory target.

Split data into separate training, validation, and test sets. Train model parameters on the training set. Use the validation set for choices such as architecture, thresholds, and hyperparameters. Use the test set for a final estimate after those choices are fixed.

Random image splitting can leak near-duplicates across sets. Frames from one video, photos of the same physical item, or images from one patient may be strongly related. Group related examples before splitting so the test set represents genuinely unseen cases.

Inspect class balance and important data slices. Overall performance can hide failure for a rare object, camera type, weather condition, skin tone, location, or lighting regime. Your evaluation set should represent the conditions and groups that matter in use.

How models learn visual features

A convolution applies a small learned filter across an image. Early layers often respond to local patterns. Later layers combine those responses into features useful for the training objective. Pooling or strided operations can reduce spatial resolution while expanding the represented context.

A vision transformer divides an image into patches, embeds them, and uses attention to relate information across the image. Convolutional networks and vision transformers are model families, not task definitions. Either family can support several tasks when paired with the right output head and training objective.

Training compares predictions with ground truth through a loss function. Optimization adjusts model parameters to reduce that loss. A lower training loss does not prove useful behavior. The real question is whether the model generalizes to unseen, representative data.

Start from pretrained features

Training a large vision model from random initialization can require substantial labeled data and compute. Transfer learning starts from a model pretrained on another dataset. You can use it as a fixed feature extractor or fine-tune some or all of its parameters for the new task.

Pretraining is a starting point, not a guarantee. The source data, license, input format, and learned categories may not match your domain. Validate the resulting system on data from the intended environment.

Measure the error that matters

Accuracy is the fraction of correct predictions, but it can mislead when classes are imbalanced. Precision asks how many predicted positives are correct. Recall asks how many actual positives the model finds. A confusion matrix shows which classes the model confuses.

Detection and segmentation need spatial measures. Intersection over union compares the overlap of a predicted region and its ground-truth region. Detection benchmarks often summarize precision and recall across confidence and overlap thresholds. The exact metric definition and threshold must be recorded with the score.

The product cost determines the metric. A missed defect and a false alarm may have very different consequences. Select thresholds on validation data, then report performance at the chosen operating point. Include latency, memory, throughput, energy use, and model size when deployment constraints matter.

Deployment changes the test

An offline notebook receives a prepared image. A production system receives camera streams, damaged files, unexpected resolutions, empty scenes, new devices, and changing environments. The input pipeline must apply the same transformations used during evaluation.

Monitor input schemas, data distributions, latency, failures, and quality signals. Compare important slices, not only one aggregate score. Keep model, code, data, and configuration versions together so you can reproduce a result and roll back a release.

Human oversight needs a defined role. A person cannot correct a system if the interface hides uncertainty or if automation acts before review. High-impact uses require clear responsibility, documented limitations, privacy controls, and a safe fallback.

Limits and responsible use

A model learns statistical patterns in its data. It does not understand a scene as a person does, and a confidence score is not a guarantee. Performance can fall under distribution shift, occlusion, adversarial manipulation, novel objects, or sensor changes.

Images can reveal identity, location, health, behavior, and other sensitive information. Minimize collection, control access, define retention, and evaluate legal and organizational requirements before deployment. Measure performance for affected groups and contexts. NIST frames this work as continuous governance, mapping, measurement, and management across the AI lifecycle.

Computer vision is a poor choice when a direct sensor, barcode, structured form, physical control, or simple image rule solves the problem more reliably. It is also a poor choice when you cannot obtain representative data or define an acceptable error policy.

A path from foundation to competence

Learn image arrays, color channels, resizing, filtering, and geometric transforms.
Practice matching product questions to classification, detection, segmentation, tracking, and geometry tasks.
Build a small baseline and inspect its errors before increasing model complexity.
Learn convolutional networks, vision transformers, loss functions, and transfer learning.
Design grouped data splits, slice-based evaluation, and task-appropriate metrics.
Study deployment optimization, monitoring, privacy, security, and human oversight.
Specialize in a domain such as robotics, medical imaging, remote sensing, manufacturing, or document analysis.