Vision Models in Deep Learning: How AI Learned to See

Your phone unlocks when it sees your face. A car reads a speed-limit sign before you do. A parking gate opens because a camera read your plate, and a skincare store suggests a serum after looking at one selfie. Each of these is a vision model at work: a piece of deep learning software that looks at an image and decides what is in it. Fifteen years ago, none of this worked reliably. Today it is ordinary, and the story of how it got there is one of the clearest examples of what deep learning can do.

This guide is written for three kinds of reader: the curious, who want to know how a computer can “see”; AI researchers, who want the milestones and numbers in one place; and founders, who want to know what it takes to put a vision model into a product people trust. We cover what vision models are, how the main families work, the data that made them possible, where they are used, where they go wrong, and what we have learned building one ourselves.

What is a vision model?

A vision model is a neural network trained to understand images or video. It takes pixels in and returns something useful out: a label (“cat”), a box around an object, an exact outline of it, a description in words, or a score such as “73% likely to be a stop sign”. Computer vision is the field; vision models are the learned systems that now do most of its work.

To a computer, a photo is just a grid of numbers. A square 1080 × 1080 photo holds 1,166,400 pixels, and each pixel has a red, a green and a blue value, so the model receives 3,499,200 numbers. Nothing in those numbers says “face” or “road”. A vision model learns, from many examples, which patterns of numbers tend to mean which things. That learned mapping, from raw numbers to meaning, is the whole trick.

Computer vision models do several different jobs, and it helps to name them, because the right model depends on the job:

• Image classification: what is in this picture? One label, or a ranked list of labels.
• Object detection: where is each object? A box and a label for every car, person or product in view.
• Segmentation: exactly which pixels belong to each object, for example the outline of a tumour or the area of one cheek.
• Keypoints and pose: where are the joints of a body, or the corners of an eye and mouth?
• Text in images (OCR): read the characters on a document, a sign or a licence plate.
• Image and text together: find the photo that matches a sentence, describe a photo in words, or answer a question about it.

View full size

How does a vision model see an image?

The first successful idea came from biology. In 1959 David Hubel and Torsten Wiesel recorded single neurons in a cat’s visual cortex and found cells that fired for edges at particular angles, and other cells that combined them into more complex shapes. The work later earned them a Nobel Prize, and it suggested a recipe: detect simple patterns first, then combine them into bigger ones.

Kunihiko Fukushima turned that recipe into the Neocognitron in 1980, and in 1998 Yann LeCun and colleagues published LeNet-5, a convolutional neural network (CNN) that read handwritten digits well enough to be used commercially for reading cheques. A CNN slides small filters, often 3 × 3 pixels, across the whole image. Early layers learn to respond to edges and colour changes, middle layers to textures and parts, and the last layers to whole objects. The same filter is reused at every position, which is why a CNN can recognise a cat in any corner of the photo.

In 2020 a second idea arrived from language models. A vision transformer (ViT) cuts the image into small square patches, treats each patch like a word in a sentence, and uses self-attention, so that every patch can weigh every other patch from the very first layer. The diagram below shows the difference.

View full size

The ImageNet moment: when deep learning won

Ideas need data. In 2009 Fei-Fei Li’s team released ImageNet, a database that grew to more than 14 million labelled images. Its yearly challenge, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), asked models to sort about 1.28 million training photos into 1,000 categories and then label new ones. The headline score was top-5 error: how often the right answer was missing from the model’s five best guesses.

In 2012 a CNN called AlexNet, built by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, won with a top-5 error of 15.3%. The next best entry scored 26.2%. AlexNet had about 60 million parameters and was trained on two consumer graphics cards, and that result is widely seen as the moment deep learning went mainstream. Over the next three years, networks became much deeper. In 2015 ResNet, from Kaiming He and colleagues at Microsoft Research, used 152 layers joined by “skip connections” and reached 3.57%.

For comparison, when Andrej Karpathy labelled 1,500 ImageNet test images himself after practising, his top-5 error was 5.1%. That is the dashed line in the chart below. Human accuracy here is a narrow measure, since many errors came from telling apart 120 similar dog breeds, but the trend was impossible to ignore.

View full size

The challenge ended in 2017, with the winning error at 2.25%. ImageNet is still used to compare models, but the frontier has since moved to harder questions: recognising things the model was never trained on, understanding video, and connecting images with language.

CNN vs vision transformer: what is the difference?

The 2020 paper that introduced the vision transformer was titled “An Image is Worth 16x16 Words”, because it cut each image into patches of 16 × 16 pixels. Trained on a mid-sized dataset, the ViT did worse than strong CNNs. Pre-trained on Google’s internal JFT-300M dataset of 300 million images, its largest version reached 88.55% top-1 accuracy on ImageNet. The authors summed it up in one line: large-scale training trumps inductive bias. In other words, if you have enough data, you do not need to build assumptions about images into the model.

CNNs did not disappear. In 2022 researchers at Meta modernised the classic design in “A ConvNet for the 2020s”, and their ConvNeXt reached 87.8% top-1, showing that much of the transformer’s lead came from better training recipes rather than attention itself. Hybrids such as the Swin Transformer (2021) borrowed the CNN habit of looking at local windows first. Today the two families are less rivals than tools for different jobs.

How do vision models learn without labels?

Modern vision models learn from images paired with captions, or from images alone, instead of only from photos labelled by hand. That was the second big shift: ImageNet needed people to label every photo, and labelling does not scale to billions of images, so researchers found other teachers.

In 2021 OpenAI’s CLIP learned from 400 million images paired with the captions people had written for them on the web. Instead of learning fixed categories, it learned which caption goes with which image. That let it recognise new categories from a written description alone. Without using any of ImageNet’s 1.28 million training images, CLIP’s best version scored 76.2% on ImageNet “zero-shot”, matching the original ResNet-50, which had been trained on all of them. In 2022 the open LAION-5B project released 5.85 billion image-text pairs, so anyone could train CLIP-style models.

Meta’s DINO line went further and dropped labels and captions entirely. These self-supervised models learn by comparing different views of the same image. DINOv2 (2023) trained on a curated set of 142 million images. DINOv3, released in August 2025, scaled to a 7-billion-parameter model trained on 1.7 billion images, and Meta distilled it into smaller vision transformer and ConvNeXt versions that run on ordinary hardware. Meta reports that it is the first self-supervised model to beat weakly supervised models across a broad set of tasks.

View full size

This is why most teams no longer train vision models from scratch. They start from a “foundation” backbone such as CLIP or DINO that already understands images in general, then fine-tune it on a much smaller set of examples from their own problem.

What are object detection and segmentation models?

Object detection models find each object in an image and draw a labelled box around it; segmentation models go further and mark exactly which pixels belong to it. Knowing that a photo contains a car is not enough for a self-driving system or a checkout camera. They need to know where. In 2016 YOLO (“You Only Look Once”) showed that a single network could find and label every object in one pass, fast enough for live video. In 2015 U-Net, built for microscope images, became the standard design for segmentation, marking exactly which pixels belong to each object, and it is still used widely in medical imaging.

In April 2023 Meta released the Segment Anything Model (SAM) with SA-1B, a dataset of 11 million images and 1.1 billion masks, 99.1% of them generated automatically by the model itself. SAM can outline almost any object after a single click or a box. SAM 2 (2024) extended this to video. SAM 3, released on 19 November 2025 with open code and weights, takes a short text phrase or an example image, such as “yellow school bus”, then finds, outlines and tracks every matching object in an image or video. Detection, segmentation and tracking, once three separate models, are merging into one.

View full size

What are vision-language models (VLMs)?

A vision-language model (VLM) joins a vision encoder to a large language model, so it can describe an image or answer questions about it in words. The vision encoder, often CLIP-style, turns the image into a sequence of tokens. A small connector maps those tokens into the language model’s space, and the language model then reasons about the picture in words. DeepMind’s Flamingo (2022) and the open LLaVA (2023) showed the approach; GPT-4V brought it to millions of users in 2023. Today the leading assistants from Google, OpenAI and Anthropic all read images natively, and open-weight families such as Qwen-VL can run on a company’s own servers.

Vision-language models (VLMs) are remarkable generalists. You can ask one to read a chart, explain a meme or check a form for missing fields. They are still weaker than specialist models at exact counting, fine spatial detail and measurement, and like all language models they can state something wrong with confidence. For a narrow, high-volume job, such as reading licence plates or scoring visible skin concerns, a specialised model with a clear confidence score is usually cheaper, faster and easier to test.

Where are vision models used today?

Vision models now run in hospitals, shops, factories, cars, farms and on almost every phone. These are the most common uses:

• Healthcare: in 2017 a Stanford team trained a CNN on 129,450 clinical images covering 2,032 skin diseases, and it matched 21 board-certified dermatologists on the key classification tasks. Radiology, pathology and eye screening use similar models today, usually as a second reader for a clinician.
• Retail and beauty: visual search, virtual try-on, shelf monitoring, and selfie-based skin analysis that recommends products.
• Mobility and parking: reading licence plates, counting vehicles and spotting free bays. Our guide to how licence plate recognition works walks through one of these systems step by step.
• Manufacturing: finding scratches, cracks and missing parts on a production line faster than a person can.
• Agriculture: spotting crop disease, weeds and ripeness from drone and phone images.
• Documents: reading invoices, IDs and forms, often with a vision-language model on top to pull out the fields that matter.
• Accessibility: describing photos and scenes aloud for people who are blind or have low vision.

If you want to see how one of these works end to end, read how licence plate recognition works, or look at our License Plate Recognition API.

Why skin tone and lighting decide whether a vision model is fair

View full size

A vision model can only be as good as the images it learned from. That matters most when the images are of people. Dermatology uses the Fitzpatrick scale, created in 1975 to describe how skin responds to sun, with six types from I (lightest) to VI (darkest). In 2021 researchers at MIT and partners released Fitzpatrick 17k, a public set of 16,577 clinical images across 114 skin conditions, labelled with skin type. The imbalance was stark: 7,755 images of types I–II against 2,168 of types V–VI. Models trained on it performed best on skin types that resembled their training data.

View full size

The industry has started to respond. In May 2022 Google released the Monk Skin Tone Scale, a 10-shade scale developed with Harvard sociologist Ellis Monk, to test its own products for bias, and published it under an open licence. The lesson for anyone building a vision model is simple: report accuracy per group, not just one average, and collect data that looks like your real users, in their real lighting.

How does AI skin analysis work in a real store?

AI skin analysis uses a vision model to score visible skin concerns in a selfie and match them to suitable products. Everything above came together for us when a D2C skincare brand in India asked for a way to recommend products from a single selfie on its Shopify Plus store. Most skincare stores show every visitor the same product grid, and quizzes ask shoppers to judge their own skin, often in poor light. The brand wanted to connect what a customer’s skin actually looks like with what was in stock.

The deep learning model was only one of five steps. A guided selfie screen helps with framing and lighting. A quality check rejects blurred, dark or badly angled photos before any analysis. The face is split into regions: forehead, cheeks, under-eyes, nose and chin. The model scores visible concerns such as dryness, redness, fine lines, dark spots and uneven tone, each with a confidence level. Finally, rules map each confident finding to in-stock products in the brand’s own catalog. When the model is unsure, it asks for a retake instead of guessing.

View full size

We tested fairness the way the research says to. On a held-out set of 3,500 real-world selfies, concern detection reached 91% precision, and accuracy varied by no more than 4 points across Fitzpatrick skin types I to VI. Privacy was a design rule: each selfie is analysed in memory and deleted as soon as results are shown, and it is never used for training without the shopper’s explicit consent.

The business results, measured from 1 March to 31 May 2026 with a 50/50 A/B holdout across 48,300 scans: 74% of shoppers who started the scan finished it, against 42% for the old quiz. Shoppers who used it added recommended products to their cart 20% more often (3.0% to 3.6%) and spent 18% more per order (₹1,240 to ₹1,465). The full story, including the architecture, is in our AI skin analysis case study.

How do you build and ship a vision model?

To ship a vision model, start from a pre-trained foundation model, fine-tune it on images that match real use, and wrap it in quality checks, confidence thresholds and testing for every group of users. For founders and product teams, the hard part is rarely choosing an architecture; it is everything around the model. This is the checklist we use:

• Start from a foundation model. Fine-tune a pre-trained backbone such as DINOv3 or a CLIP-style encoder instead of training from scratch. You will need far less data and time.
• Collect images that look like real use. Bathroom mirrors, night shifts, rain on a lens. A model trained on studio photos will fail on the first real one.
• Check image quality before the model runs. Rejecting a blurred or dark photo is better than analysing it badly.
• Use confidence, not just answers. Set a threshold. Below it, ask for a retake or send the case to a person.
• Test every group you serve. Report accuracy by skin tone, lighting, device and region, not just one average.
• Decide where it runs. On the phone for speed and privacy, or in the cloud for bigger models. Both are valid; the use case decides.
• Treat images as personal data. Keep them only as long as the job needs, and say clearly what you capture and why.
• Secure it. Images can carry hidden instructions for vision-language models and adversarial patterns for classifiers. Test for both before launch.
• Watch it after launch. New phones, new lighting and new users shift the data. Monitor accuracy and retrain on fresh, consented examples.

If your product puts a vision-language model in front of users, our AI security assessment tests for exactly these attacks.

What is the future of vision models?

Vision models are heading towards one model for many tasks, a deeper grasp of video and 3D, smaller versions that run on devices, and stricter measurement. First, unification: models such as SAM 3 already detect, outline and track in one network, and vision-language models are absorbing more specialist tasks. Second, video and 3D: understanding how things move and occupy space, which robots and cars need, is where much of the research effort is going. Third, smaller and local: distilled models like the small DINOv3 versions bring strong vision to phones and cameras without sending images to a server. Fourth, measurement: as models get better at averages, the questions that matter are how they do on the hardest 5% of images and on every group of people they serve.

Vision models went from a research curiosity to everyday infrastructure in little more than a decade. The models are now widely available; the advantage lies in the data you collect, the checks you build around the model, and the care you take with the people in the pictures. If you are planning a product that needs to see, talk to our team.