Computer Vision
Computer vision builds systems that extract structured information from images and video: detecting objects, recognizing faces, segmenting scenes, reading text, and interpreting spatial relationships. It combines image processing, machine learning, and geometry to give machines visual perception.
itArtificial intelligence and machine learning | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Computer Vision
Computer vision is turning light into decisions. A camera captures a scene. Pixels arrive as numbers. Somewhere downstream, a label, a bounding box, or a pixel mask appears, and something acts on it. The gap between "pixels arrive" and "something acts on it" is where the interesting work lives.
Before you reach for a neural network, figure out what decision the system must support. That decision determines the task. A single label per image is classification. A label with a location is detection. A class per pixel is segmentation. The output shape dictates annotation cost, model choice, metric, and what "good enough" even means. Pick the least detailed output that supports the action. A lot of computer vision projects fail because someone started with an architecture instead of a question.
Data is the part that nobody finds exciting and everybody underestimates. Related images must stay in the same split — consecutive video frames from the same camera are near-duplicates, and if they leak across train and test, your metrics will lie to you with a straight face. Label quality dominates model quality: teams have doubled mAP by fixing annotations without touching the model. The accuracy number is hiding things. A model at 95% overall accuracy can have zero recall on a rare class that matters more than the common ones.
A pretrained model is a head start, not a finish line. Transfer learning — taking a model trained elsewhere and adapting it to your data — is where most practical work begins. The model's source data, input format, and license may not match your domain. Validate on data from the intended environment, not a convenient proxy.
When a model runs on a server, it gets clean inputs. When it runs in the real world, it gets fog, motion blur, a new camera angle, and empty frames. Deployment is a different environment. Monitor input distributions, not just predictions. Watch confidence scores spread — it happens weeks before accuracy visibly drops.
The pipeline is the system. The model is one stage. Define the decision, choose the output, build a baseline, inspect the data, and measure the failure that actually matters. The Glossary tab in the intro names the terms. The Cheatsheet has the quick reference. The Reference tab points to primary documentation for going deeper.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://docs.opencv.org/master/d6/d00/tutorial_py_root.html
Supports
- Images as arrays with pixel and channel operations
- Geometric transforms, filtering, feature detection, video analysis, camera calibration, and classical object detection
- The role of explicit image processing alongside learned models
- https://www.tensorflow.org/tutorials/images/classification
Supports
- The image classification workflow from data loading through training, testing, and deployment conversion
- Training and validation datasets, overfitting, data augmentation, and dropout
- Image resizing, batching, and model input preparation
- https://www.tensorflow.org/tutorials/images/segmentation
Supports
- Semantic segmentation as per-pixel classification
- Image and mask preprocessing, matching augmentation, class imbalance, and encoder-decoder models
- Transfer of a pretrained image encoder into a segmentation model
- https://docs.pytorch.org/tutorials/beginner/transfer_learning_tutorial
Supports
- Fine-tuning a pretrained convolutional network
- Using a pretrained network as a fixed feature extractor
- Separate training and validation phases and evaluation on custom images
- https://docs.pytorch.org/tutorials/intermediate/torchvision_tutorial.html
Supports
- Object detection as prediction of bounding boxes and class scores
- Instance segmentation as an additional mask prediction for each object
- Custom dataset targets, transfer learning, and adaptation of a pretrained Mask R-CNN model
- https://arxiv.org/abs/2010.11929
Supports
- A transformer applied to sequences of image patches
- Large-scale pretraining followed by transfer to image recognition benchmarks
- Vision Transformer as an alternative to convolutional architectures for image classification
- https://scikit-learn.org/stable/modules/model_evaluation.html
Supports
- Confusion matrices and classification reports
- Precision as control of false positives and recall as coverage of actual positives
- F-measures, multiclass averaging, and threshold-sensitive evaluation
- https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
Supports
- Evaluation folds with non-overlapping groups
- Keeping every defined group in only one test fold
- Group-aware splitting for related samples
- https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets
Supports
- Separate training, validation, and test sets
- Validation during development and test data for final evaluation
- Evaluation on unseen data after model choices are made
- https://developers.google.com/machine-learning/crash-course/production-ml-systems/monitoring
Supports
- Input schemas, feature checks, important data slices, and real-world metrics
- Training-serving skew, model age, reproducible data generation, and production quality monitoring
- Version tracking, latency, memory, throughput, and live model evaluation
- https://developers.google.com/machine-learning/managing-ml-projects/production
Supports
- Compute and quota planning for training and serving
- Logging, monitoring, alerting, deployment approval, staged rollout, and rollback
- Data drift, prediction drift, quality degradation, latency, and outages
- https://airc.nist.gov/airmf-resources/airmf/
Supports
- AI risk management across design, development, use, and evaluation
- Trustworthiness considerations including validity, safety, security, transparency, privacy, and fairness
- Govern, Map, Measure, and Manage functions across the AI lifecycle
- https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
Supports
- Continuous risk management and documented measurement of fairness and bias
- Governance, context mapping, measurement, impact assessment, and management of deployed AI risk
- https://www.cs.ubc.ca/~lowe/papers/iccv99.pdf
Supports
- Scale-invariant feature transforms for image matching and object recognition
- Classical local descriptor that remained the default for over a decade
- https://ieeexplore.ieee.org/document/990517
Supports
- Real-time face detection using Haar-like features and AdaBoost cascades
- First practical object detector widely deployed in cameras and phones
- https://image-net.org/challenges/LSVRC/2010/
Supports
- Large-scale visual recognition benchmark with 1000 categories and 1.2M training images
- The challenge that catalyzed the deep learning revolution in vision
- https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks
Supports
- GPU-trained CNN reducing ImageNet top-5 error from 26.2% to 15.3%
- The result widely credited with launching deep learning in computer vision
- https://arxiv.org/abs/1409.4842
Supports
- Inception module with parallel convolutions at multiple scales
- VGGNet showing deep stacking of simple 3x3 convolutions improves accuracy
- https://papers.nips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
Supports
- Region Proposal Networks sharing convolutional features with the detector
- Near real-time two-stage detection replacing Selective Search
- https://arxiv.org/abs/1512.03385
Supports
- Skip connections enabling 152-layer deep networks
- Residual learning framework becoming the backbone for subsequent vision architectures
- https://openaccess.thecvf.com/content_cvpr_2016/html/Redmon_You_Only_Look_CVPR_2016_paper.html
Supports
- Single-pass detection recast as regression from pixels to boxes
- Real-time detection at 45 fps enabling edge deployment
- https://arxiv.org/pdf/1703.06870v3
Supports
- Instance segmentation via parallel mask prediction and RoIAlign
- Unified framework for detection, segmentation, and keypoint tasks
- https://arxiv.org/abs/2104.14294
Supports
- Self-supervised ViT features encoding segmentation boundaries without labels
- k-NN classification reaching 80.1% top-1 on ImageNet
- https://arxiv.org/abs/2304.02643
Supports
- Promptable segmentation foundation model trained on 1.1B masks
- Zero-shot transfer to new distributions via point, box, or text prompts
- https://arxiv.org/abs/2304.07193
Supports
- 1B-parameter self-supervised ViT with general-purpose frozen features
- Features rivaling weakly-supervised models on classification and segmentation
- https://pmc.ncbi.nlm.nih.gov/articles/PMC8321059/
Supports
- Near-duplicate test images inflating reported accuracy by 9-14%
- Models memorizing augmentation-applied variants rather than generalizing
- https://doi.org/10.1109/access.2024.3383047
Supports
- Random video frame splitting inflating mAP by up to 40%
- Models learning background and camera cues instead of object properties
- https://doi.org/10.48550/arxiv.2304.02296
Supports
- 90% of training images being exact or augmented duplicates
- Deduplication reducing training set to 15k unique images from 280k
- https://repository.tudelft.nl/file/File_f0ec0d6e-22fd-4c8a-bd66-f80882098854
Supports
- 10% wrong class labels causing 34% mAP drop in YOLOv8
- Classification label sensitivity exceeding bounding box inaccuracy tolerance
- https://link.springer.com/article/10.1007/s44291-026-00244-5
Supports
- Annotation cleaning improving mAP from 0.13 to 0.56 without model change
- Label quality dominating architecture choice under noisy supervision
- https://academy.ultralytics.com/courses/yolo-in-production/cost-and-latency-tuning
Supports
- Order of optimization leverage: smaller model then smaller image then FP16 then INT8
- FP16 providing 1.5-2x speedup for negligible accuracy loss
- https://dev.to/multigrid/on-device-vision-detection-ocr-and-segmentation-within-a-frame-budget-423d
Supports
- Resolution cost being quadratic not linear in tensor size and latency
- Smaller detector on region of interest outperforming larger detector on full frame
- https://ifactoryapp.com/ai-vision-camera/ai-vision-model-drift-monitoring-production
Supports
- Confidence distribution spreading as leading indicator of data drift
- Silent degradation persisting weeks before aggregate accuracy moves
- https://aiandml.net/blog/class-imbalance-computer-vision-datasets/
Supports
- Rare class improving from 0.22 to 0.71 mAP after targeted annotation
- Aggregate metrics hiding 15-49 point gains needed on tail classes