Data annotation

Data annotation is the process within a dataset of adding relevant metadata labels or tags to enable machines to interpret the data in line with its intended use. Data is a fundamental component in the development of artificial intelligence (AI). Data annotation allows AI models to interpret data. For example, it may inform the model that a particular set of pixels is a picture of a bicycle, or that a particular structure of sentence should be interpreted as a mathematical formula. Training AI models, particularly in computer vision and natural language processing, requires large volumes of annotated data. Annotation choices determine how machine learning algorithms recognize patterns and also drive the predictions they make.

The dataset can take various forms, including images, text, audio files and video footage. Data annotation labels can be human-generated, system-generated, or a mix of both.

The availability of large-scale annotated datasets has been a major factor in the development of modern artificial intelligence systems; particularly deep learning models that require vast quantities of labelled examples.

Applications

Data annotation is used in almost all fields, including law, scientific research, healthcare, autonomous vehicles, retail, security, and entertainment. By accurately labelling data, machine learning models can perform complex tasks such as object detection, sentiment analysis, and speech recognition with greater precision.

Agentic AI, in which AI models are able to set goals, plan steps and use tools independently, has greatly increased the need for annotation of data related to complex tasks. Early growth was driven by computer vision applications, particularly in autonomous driving, surveillance, industrial automation, and robotics. These systems require large volumes of accurately labelled image and video data, often including pixel-level segmentation and temporal tracking across frames.. More recently, the development of large language models has significantly increased demand for high-quality human feedback data. This includes preference rankings, factuality assessments, and safety evaluations used in reinforcement learning from human feedback (RLHF). As a result, data annotation has shifted from primarily large-scale labelling to tasks requiring expert evaluation.

This growing demand has led to the emergence of specialized sectors and platforms dedicated to AI training and human-in-the-loop workflows, which often utilize Reinforcement Learning from Human Feedback (RLHF) to refine model behavior. A market has developed whereby highly-qualified mathematicians, scientists, lawyers and other professionals are employed to develop AI learning tasks, often as an adjunct to their main occupation. A 2026 global labor market analysis by Randstad Digital found that "AI trainer" and "data annotation" job postings increased by 281% between 2021 and 2026, making it the fastest-growing standalone technology role as industries shift focus toward human oversight, model safety, and system optimization.

Annotation Types

Data annotation varies significantly depending on the type of data and the intended machine learning application.

Image and video annotation

Image classification

Image classification, also known as image categorization, involves assigning predefined labels to images. Machine learning algorithms trained on classified images can later recognize objects and differentiate between categories. For instance, an AI model trained to recognize furniture styles can distinguish between Georgian and Rococo armchairs.

Semantic segmentation

Semantic segmentation assigns each pixel in an image to a specific class, such as trees, vehicles, humans, or buildings. This type of annotation enables machine learning models to differentiate objects by grouping similar pixels, allowing for a detailed understanding of an image.

Bounding boxes

Bounding box annotation involves drawing rectangular boxes around objects in an image. This technique is commonly used in autonomous driving, security surveillance, and retail analytics to detect and classify objects such as pedestrians, vehicles, and products on store shelves.

3D cuboids

3D cuboid annotation enhances traditional bounding boxes by adding depth, enabling models to predict an object's spatial orientation, movement, and size. This method is particularly useful for autonomous vehicles and robotics, where understanding object dimensions and depth is critical.

Polygonal annotation

For objects with irregular shapes, such as curved or multi-sided items, polygonal annotation provides more precise labeling than bounding boxes. This technique is often used in applications that require detailed object recognition, such as medical imaging or aerial mapping.

Keypoint annotation

Keypoint annotation marks specific points on an object, such as facial landmarks or body joints, to enable tracking and motion analysis. This method is widely used in facial recognition, emotion detection, sports analytics, and augmented reality applications.

In computer vision, annotation commonly includes image classification, object detection, semantic segmentation, instance segmentation, and keypoint labelling.. Object detection tasks typically involve drawing bounding boxes around objects of interest, while segmentation tasks require pixel-level labelling to delineate object boundaries. Video annotation extends these tasks temporally, requiring tracking of objects across frames and maintaining consistent identities over time, which is particularly important in autonomous driving systems and robotics applications.

Text annotation

Natural language processing (NLP) tasks involve a range of annotation types including text classification, named entity recognition, sentiment analysis, part-of-speech tagging, and relation extraction. In modern large language model development, text annotation also includes ranking model outputs, evaluating factual accuracy, and assessing qualities such as helpfulness, coherence, and safety.

Audio and speech annotation

Speech annotation involves transcription of spoken language, segmentation of audio streams, together with labelling of acoustic features such as emotion or intent. These annotations support applications including automatic speech recognition, voice assistants, and real-time translation systems.

Human annotation

Human annotation is typically carried out using web-based platforms that present tasks to annotators in a structured format and capture their responses in a standardised way. These platforms may include built-in validation rules, annotation guidelines, and interface tools such as bounding box selectors, text highlighters, or audio playback controls. In many systems, annotation tasks are distributed across large workforces, often geographically dispersed, enabling parallel processing of large datasets.

History and Development

While early data annotation providers focused primarily on large-scale human labelling, the sector has increasingly diversified, driven particularly by recent agentic AI models.

Crowdsourcing and move to early workforce platforms

Crowdsourcing, in which large numbers of distributed workers completed small labelling tasks via online platforms, was one of the earliest examples of data annotation. Amazon’s Mechanical Turk, launched in 2005, enabled researchers and companies to access low-cost human labour for dataset creation and validation tasks. Appen, founded in 1996, became a major provider of annotated datasets for machine learning, supporting applications in speech recognition and computer vision. CloudFactory, founded in 2010, developed its own workforce, combining cloud-based task distribution in a way that was aimed at providing long-term employment for workers, particularly in Nepal and Kenya.

Diversification of offerings

As demand for labelled data increased and the data annotation market matured, companies offering different approaches to data annotation management emerged. Companies such as Labelbox, SuperAnnotate, and Encord. worked to develop annotation software platforms that enable organisations to manage their own labelling workflows. These systems typically provide tools for dataset management, collaborative labelling, quality control, and integration with machine learning pipelines. Conversely, managed annotation and data providers such as Scale AI and Surge AI continued to provide high-quality annotated datasets for autonomous vehicles, defence applications, and large language model training which they developed themselves using an increasingly skilled workforce.

Expert networks and specialised evaluation providers

Across the industry, a notable trend is the shift from large-scale commodity labelling toward high-value expert annotation and evaluation. While traditional image and text labelling remains important for many applications, the development of large language models has increased demand for nuanced human feedback, including preference ranking, safety evaluation, and reasoning assessment. together with expert-driven annotation and evaluation, particularly for tasks requiring professional judgement in fields such as law, medicine, software engineering, and scientific reasoning. Turing, founded in 2018, initially operated as a global talent platform for software engineers but has since expanded into providing expert contributors for AI training and evaluation. Mercor, founded in 2023, operates as an AI-enabled talent matching platform connecting subject-matter experts—including engineers, physicians, and lawyers—with AI companies requiring high-quality evaluation data for model training and assessment. Deccan focuses on sourcing technical experts in India for tasks such as generating expert feedback, running evaluations, and building reinforcement learning, reflecting the broader shift toward specialised human feedback in foundation model development. Poindexter Labs operates in a similar space, providing expert-led evaluation and annotation services for AI systems requiring high-precision human judgement, particularly in enterprise and research contexts.

See also

• Reinforcement learning from human feedback
• Labeled data
• Humans in the Loop (film)