Image Classification by Accessible Purpose

⚠️ Not currently in production. This checkpoint is not part of the live product right now: the image classification models it relied on were discontinued. Integration is planned for an upcoming release.

This document explains the logic followed by the image classification test and the prompt used for AI-based classification.

Table of Contents

1. Overview
2. Test Flow
3. The Classification Prompt
4. Layered Analysis Methodology
5. Classification Types
6. Critical Rules
7. Alternative Text Quality Analysis

---

Overview

The image classification system uses a combination of AI analysis and heuristic rules to classify images on a web page according to their accessible purpose, following WCAG 2.1 guidelines and W3C WAI tutorials. The system classifies images into three categories:

• ACTIVE: Functional images that initiate actions or act as interactive controls
• INFORMATIVE: Images that convey essential information
• DECORATIVE: Purely ornamental or redundant images

---

Test Flow

1. Image Element Extraction

The test begins by extracting all image elements from the web page from two sources:

1.1 Accessibility Tree (AXTree)

• The complete accessibility tree of the page is obtained
• Nodes with roles are searched: image, img, graphics-symbol
• Elements with tags are identified: img, svg
• Ignored nodes with reason emptyAlt are detected (likely decorative images)
• Key attributes are extracted: alt, aria-hidden, role, name

1.2 DOM (Document Object Model)

• Additional elements not captured by the AXTree are searched
• <img>, <svg>, and <input type="image"> elements are identified
• Elements with role="img" are detected
• Attributes are extracted directly from the DOM

2. Context Enrichment

For each image element found, it is enriched with contextual information:

• elementOuterHTML: The complete HTML of the element (maximum 250 characters)
• parentOuterHTML: The HTML of the parent element (if available)
• containerOuterHTML: The HTML of the container (as an alternative to the parent)
• interactiveAncestor: Information about the nearest interactive ancestor (if it exists)
  • tagName: Ancestor tag
  • role: Ancestor role
  • accessibleName: Accessible name of the ancestor
  • ariaLabel: ARIA label of the ancestor
  • textContent: Text content of the ancestor
  • outerHTML: HTML of the ancestor

3. Screenshot Capture

• A full-page screenshot is captured
• It is converted to PDF format for AI analysis
• The PDF contains the complete visual representation of the page

4. Batch Processing

Elements are processed in batches of 50 elements to optimize token usage and improve efficiency:

batch_size = 50
for batch_idx in range(0, total_elements, batch_size):
    batch_elements = enriched_elements[batch_idx : batch_idx + batch_size]
    # Process batch...

5. AI Analysis

For each batch:

1. Prompt Creation: A detailed prompt is generated with all contextual information from the batch elements
2. Visual Analysis: The AI analyzes the PDF (screenshot) together with the prompt
3. Classification: The AI classifies each image as ACTIVE, INFORMATIVE, or DECORATIVE
4. Quality Evaluation: The AI evaluates the quality of the alternative text by comparing it with the visual content

6. Results Processing

After receiving the AI results, the system:

1. Parses the Response: Extracts classification results from the JSON returned by the AI
2. Applies Priority Rules: Overrides the AI classification according to critical rules:
   • If aria-hidden="true" or role="presentation" → DECORATIVE (absolute priority)
   • If input[type="image"] → ACTIVE (always active, unless it has aria-hidden="true")
3. Resolves Alternative Text: Obtains alternative text from multiple sources (AXTree, DOM, attributes)
4. Generates Reports: Creates accessibility reports for each classified image

7. Report Generation

For each classified image, a report is generated that includes:

• Final classification (ACTIVE, INFORMATIVE, DECORATIVE)
• Confidence level (high, medium, low)
• Classification reasoning
• Alternative text quality (good, needs_improvement, missing)
• Accessibility score (0-10)
• Specific recommendations
• Metadata (backendId, uniqueId, source, etc.)

---

The Classification Prompt

The prompt is a detailed instruction sent to the AI to guide the classification process. It is designed to ensure 99.9% accuracy by following a strict analysis order.

Prompt Structure

The prompt contains the following main sections:

1. Main Instruction: Defines the expert's role and objective
2. Critical Rules: Strict order of application for maximum accuracy
3. Layered Analysis Methodology: Mandatory step-by-step process
4. Classification Types: Detailed definitions of each category
5. Contextual Examples: Common use cases
6. Alternative Text Quality Analysis: Evaluation criteria
7. Special Cases: Handling of edge cases
8. Output Format: Required JSON structure

Prompt Content

Main Instruction

You are a web accessibility expert classifying images on a webpage based on WCAG 2.1
and W3C WAI tutorials. Analyze the provided PDF document which contains a full page
screenshot of the webpage. Review the PDF document carefully to examine the visual
content and context of each image, then classify each image as ACTIVE, INFORMATIVE,
or DECORATIVE.

Critical Rules (Strict Order)

The prompt establishes 4 critical rules that must be applied in strict order:

1. ARIA-HIDDEN Precedence: If aria-hidden="true" is present in ANY layer → DECORATIVE (overrides everything)
2. Special INPUT TYPE="IMAGE" Rule: <input type="image"> elements are ALWAYS ACTIVE (unless they have aria-hidden="true")
3. Decorative Markers: If ariaHidden=true, rolePresentation=true, or altText="" → DECORATIVE
4. Fundamental Principle: Classify based on VISUAL CONTENT + CONTEXT, not just alternative text

---

Layered Analysis Methodology

The prompt defines a 4-layer analysis methodology that must be strictly followed:

Layer 1: elementOuterHTML Analysis

• Parse the elementOuterHTML string (if available)
• Extract attributes:
  • aria-hidden → if "true", classify as DECORATIVE (STOP)
  • For input elements: type → if "image", classify as ACTIVE immediately
  • alt → check if empty or not
  • role → check for presentation role
  • Element structure → determine if interactive by nature

Layer 2: parentOuterHTML/containerOuterHTML Analysis

• If Layer 1 did not provide complete information:
  • Parse parentOuterHTML (preferred) or containerOuterHTML (alternative)
  • Find the element within the parent/container HTML using uniqueId or data-unique-id
  • Extract the same attributes as in Layer 1
  • Check surrounding context (sibling elements, parent structure)

Layer 3: Interactive Ancestor Analysis

• If interactiveAncestor is present:
  • Parse interactiveAncestor.outerHTML to understand the interactive context
  • Determine if the image is the sole content or supplementary content
  • Evaluate whether the image drives interactivity (→ ACTIVE) or is decorative (→ DECORATIVE)

Layer 4: Visual and Contextual Analysis

• Examine the PDF document for the visual content of each image
• Review the screenshot within the PDF to understand the visual context
• Check nearby text in the HTML context and in the PDF document
• Evaluate whether the text completely describes the image (→ DECORATIVE) or omits key information (→ INFORMATIVE)

Analysis Steps (Follow Strictly)

1. Layered OuterHTML Review:

   • LAYER 1: Parse elementOuterHTML for aria-hidden="true" (→ DECORATIVE). For tagName "input" or if the element contains <input, extract the type attribute — if type="image" → ACTIVE (stop here, alt attribute not required).
   • LAYER 2: If Layer 1 is incomplete, parse parentOuterHTML or containerOuterHTML to locate the actual input element. Search for <input tags and check type="image" → ACTIVE (stop here, alt attribute not required).

2. Accessible Name Verification (CRITICAL FOR INFORMATIVE):

   • Check accessible name: alt attribute (non-empty), aria-label, aria-labelledby, title attribute, or accessible name field.
   • If NO accessible name is found → CANNOT be INFORMATIVE; classify as DECORATIVE (unless ACTIVE due to special rules).

3. Tag/Context Verification: Confirm special rules (e.g., input[type="image"] → ACTIVE, regardless of alt attribute).

4. Structure Evaluation: Is the image the sole interactive content? (→ ACTIVE) Or is it supplementary/redundant? (→ DECORATIVE/INFORMATIVE)

5. Text Context: Does nearby text completely describe the image? (→ DECORATIVE) Or does it omit key information? (→ INFORMATIVE, but ONLY if it has an accessible name from step 2).

6. Visual Review: Review the PDF document to examine each image visually. Decorative shapes/patterns (→ DECORATIVE); significant visuals (→ INFORMATIVE, but ONLY if it has an accessible name from step 2).

7. Alternative Text Quality Evaluation:

   • Compare the visual content of the image (from the PDF document) with the alternative text or accessible name.
   • Determine whether the alternative text accurately describes what is shown in the image.
   • Consider context: decorative images should have empty alternative text, functional images need purpose description, informative images need alternative text that conveys essential information.

8. Markers/Verification: Cross-check metadata, but prioritize layered outerHTML analysis.

---

Classification Types

1. ACTIVE (Functional)

ACTIVE images are those that initiate an action, act as a hyperlink, control the interface, or modify the understanding of an interaction.

Prerequisite: To be classified as ACTIVE, the element (img, svg) MUST have an accessible name (alt, aria-label, aria-labelledby, or title). If it lacks this, it falls to DECORATIVE immediately.

ACTIVE Image Categories:

• Sole Interactive Content: The image is the only content within a link (<a>) or button (<button>). Examples: A magnifying glass icon for "Search", a printer icon, or a company logo linking to the homepage.
• Functional Modifiers: Images within links/buttons that appear alongside text BUT provide essential context about the behavior or format of the interaction that the text does NOT explicitly establish.
  • Examples: An icon indicating "opens in new window", "external link", "download", or file type (e.g., a PDF icon next to "Annual Report").
• Input Controls: Images used as form inputs (e.g., <input type="image">).
• Status Indicators: Icons representing a state not described by text (e.g., a battery level icon, a star icon for "favorite").

Special Rule: <input type="image"> elements are ALWAYS ACTIVE because they function as submit buttons, regardless of whether they have an alt attribute or not. This is a W3C/WCAG requirement.

2. INFORMATIVE

INFORMATIVE images convey a simple concept, emotion, or information that is NOT fully conveyed by adjacent text.

Critical Requirement: An image is INFORMATIVE ONLY if it BOTH (1) adds distinct value/meaning AND (2) has an accessible name (alternative text, aria-label, title). If an image conveys meaning visually but lacks an accessible name, classify as DECORATIVE (treat as a technical failure where the image becomes invisible to assistive technologies).

INFORMATIVE Image Patterns:

• Distinguishes elements (phone icon vs. fax icon) WITH accessible name.
• Complements text (undescribed photo) WITH accessible name.
• Shows actions (diagrams) WITH accessible name.
• Indicates type (PDF icon if text omits "(PDF)") WITH accessible name.

Examples:

• Chart with data + alternative text
• Logo WITH alternative text/aria-label
• PDF icon + "Download report" + alternative text
• Status icons WITH accessible names

NOT INFORMATIVE:

• Logos without alternative text/aria-label (→ DECORATIVE)
• SVGs in labeled links that add no information (→ DECORATIVE)
• Images without an accessible name that add no value beyond context (→ DECORATIVE)

3. DECORATIVE

DECORATIVE images are purely ornamental or redundant; removing them would not affect the user's understanding of the content.

DECORATIVE Image Patterns:

• Embellishes (borders, patterns).
• Text fully covers it (photo + full description).
• Repetitive styling (list arrows).
• Supplementary in labeled controls.
• Lacks an accessible name (no alternative text, no aria-label, no title) and adds no value beyond context (e.g., SVG in a labeled link, logo without alternative text).
• Adds no information beyond what is already conveyed by nearby text/structure.

Examples:

• Separators
• Gradients
• Icon + "Download (PDF)"
• Elements with aria-hidden
• Logos without alternative text/aria-label
• SVGs in labeled links/buttons that add no information
• Images without an accessible name that are redundant to context

---

Critical Rules

Rule 1: ARIA-HIDDEN Precedence

Absolute Priority: Perform LAYERED ANALYSIS — scanning ALL available outerHTML in order: first elementOuterHTML, then parentOuterHTML or containerOuterHTML, and finally interactiveAncestor.outerHTML if available. Search for aria-hidden="true" in ANY layer. If present ANYWHERE, classify as DECORATIVE immediately — this overrides everything else.

Rule 2: INPUT TYPE="IMAGE" (ALWAYS ACTIVE)

Layered Analysis to Detect input[type="image"]:

• LAYER 1 - elementOuterHTML: If tagName is "input" OR if elementOuterHTML contains an <input tag, parse the HTML string to extract:
  • Search for type="image" or type='image' (case-insensitive)
  • If type="image" is found, classify as ACTIVE immediately — regardless of other factors (including presence or content of the alt attribute).
• LAYER 2 - parentOuterHTML/containerOuterHTML: If Layer 1 did not find the input element (e.g., elementOuterHTML may be missing or incomplete), parse parentOuterHTML or containerOuterHTML to locate the actual <input> element. Search for:
  • <input tags within the parent/container HTML
  • Check each input for type="image" or type='image' (case-insensitive, attribute order does not matter)
  • If found with type="image", classify as ACTIVE immediately.

Examples that MUST be detected as ACTIVE:

• <input type="image" src="search.png"> → ACTIVE (even without alt)
• <input type="image" src="search.png" alt="Search"> → ACTIVE
• <input name="submit" src="..." type="image"> → ACTIVE (attribute order does not matter, even without alt)
• <input name="submit" src="/path/to/image.png" type="image" style="..."> → ACTIVE (even with other attributes, even without alt)

CRITICAL: input[type="image"] elements are ALWAYS ACTIVE because they function as submit buttons, regardless of whether they have an alt attribute or not. This is a W3C/WCAG requirement.

Rule 3: Decorative Markers

If ariaHidden=true, rolePresentation=true, or altText="" (empty string, verified in outerHTML), classify as DECORATIVE.

Rule 4: Fundamental Principle

For non-special cases, classify based on VISUAL CONTENT + CONTEXT (nearby text/structure). Do NOT rely solely on alternative text — many decorative images have incorrect alternative text that should be removed.

---

Alternative Text Quality Analysis

The prompt includes detailed instructions for evaluating alternative text quality by comparing it with the visual content of the image.

DESCRIBES_IMAGE Field (True/False)

• True: The alternative text or accessible name accurately describes the essential visual content or function of the image
• False: The alternative text is missing, empty (when it should not be), inaccurate, or does not convey the essential information

ALT_TEXT_QUALITY_EXPLANATION Field

Provide a detailed explanation covering:

1. What the image shows visually (based on reviewing the PDF document)
2. What the alternative text says (from altText, name, or accessible name fields)
3. Whether they match and why
4. Context considerations (decorative images should have empty alternative text, functional images need purpose description, informative images need alternative text that conveys essential information)

Quality Evaluation Guidelines

• DECORATIVE images: Should have empty alternative text (alt="") or aria-hidden="true". If they have descriptive alternative text, set describes_image: false and explain that decorative images should not have alternative text.
• ACTIVE images: Alternative text should describe the function/action, not necessarily the visual appearance. E.g., "Search" not "Magnifying glass icon".
• INFORMATIVE images: Alternative text should convey the essential information shown in the image. E.g., "Sales increased 25% from Q1 to Q2" not just "Bar chart".
• Missing alternative text: Always set describes_image: false when alternative text is missing for ACTIVE or INFORMATIVE images.
• Generic alternative text: Alternative text like "image", "photo", "icon" without specific description should be describes_image: false.

---

Contextual Examples

The prompt includes numerous examples to guide classification:

Classification Examples

• PDF icon + "Download report" + alt text → INFORMATIVE (adds format, has accessible name)
• PDF icon + "Download report" WITHOUT alt text/aria-label → DECORATIVE (lacks accessible name, cannot be informative)
• PDF icon + "Download report (PDF)" → DECORATIVE (redundant)
• <button><img alt="Save"></button> → ACTIVE (sole content)
• <button>Save <img alt=""></button> → DECORATIVE (text is sufficient)
• <input type="image" src="search.png"> → ACTIVE (special rule, Layer 1 detection, even without alt)
• <input type="image" alt="Search"> → ACTIVE (special rule, Layer 1 detection)
• <a><img aria-hidden="true"></a> → DECORATIVE (hidden overrides)
• <a href="/home">Home <svg></svg></a> (SVG without alt/aria-label) → DECORATIVE (no accessible name, adds no information beyond link text)
• Logo without alt text/aria-label → DECORATIVE (lacks accessible name, cannot be informative)
• Logo WITH alt text/aria-label → INFORMATIVE (if it adds value) or DECORATIVE (if redundant)
• Photo of sleeping dog + "A dog sleeps peacefully" → DECORATIVE (described)
• Product photo in review + alt text → INFORMATIVE (shows appearance, has accessible name)
• Product photo in review WITHOUT alt text → DECORATIVE (lacks accessible name, cannot be informative)

---

Special Cases

The prompt includes detailed handling of special cases:

• SVGs with aria-hidden="true" → DECORATIVE (even in links/buttons)
• SVGs without accessible name (no alt, no aria-label, no title) in labeled links/buttons → DECORATIVE (cannot be informative without accessible name, and if it adds no information beyond the label, it is decorative)
• Empty alt in links → DECORATIVE (not ACTIVE, and cannot be informative without accessible name)
• Logos without alt text/aria-label/title → DECORATIVE (lacks accessible name, cannot be informative)
• Images without accessible name that add no information beyond context → DECORATIVE (e.g., SVG in a link that already has a text label)
• Multiple images in a control: Classify each independently
• Incorrect alt on decorative → Still DECORATIVE (recommend removal)
• CRITICAL: INFORMATIVE REQUIRES ACCESSIBLE NAME: An image cannot be INFORMATIVE if it lacks an accessible name (alternative text, aria-label, title, etc.), even if it appears to have content value. Classify as DECORATIVE.

---

Output Format

The prompt specifies a strict JSON format for the response:

{
    "images": [
        {
            "index": 1,
            "backendId": "backend_id_if_available_or_null",
            "uniqueId": "unique_id_if_available_or_null",
            "classification": "active|informative|decorative",
            "confidence": "high|medium|low",
            "reasoning": "Concise explanation (1-2 sentences), citing steps/rules used (e.g., 'Rule 2: input type=\"image\" is always ACTIVE')",
            "describes_image": true|false,
            "alt_text_quality_explanation": "Detailed explanation of whether the alternative text or accessible name accurately describes the image content and why",
            "alt_text_quality": "good|needs_improvement|missing",
            "accessibility_score": 0-10,
            "recommendations": ["Bullet 1", "Bullet 2"]  // Empty array if none
        },
        ...
    ],
    "summary": {
        "total_images": N,
        "active_count": 0,
        "informative_count": 0,
        "decorative_count": 0,
        "average_accessibility_score": 0.0
    }
}

---

Priority Data Sources

The prompt establishes a priority order for data sources:

1. OuterHTML (HIGHEST PRIORITY): ALWAYS parse available outerHTML in layer order — it shows actual HTML attributes/structure. Extract attributes like type, alt, aria-hidden directly from the HTML string.

   • For input elements: Search for type="image" in BOTH elementOuterHTML AND parentOuterHTML/containerOuterHTML to trigger ACTIVE (alt attribute not required).
   • Reveals whether the image is sole content (ACTIVE), redundant (DECORATIVE), or additive (INFORMATIVE).
   • Verify attributes directly (e.g., aria-hidden, alt, role).

2. Interactive Ancestor: If present, use to evaluate whether the image drives interactivity (e.g., sole content of a button → ACTIVE).

3. Parent/Container HTML: For non-interactive context; evaluate relationship with nearby text/elements.

4. Visual Content: Examine the PDF document and the screenshot within it — decorative (shapes/patterns) vs. informative (objects/data). Review the PDF carefully to see the visual representation of each image.

5. Contextual Text: Nearby text in the PDF document/screenshot/outerHTML; if it covers image information, → DECORATIVE.

---

Summary

The image classification system uses a multi-layer approach that combines:

1. Comprehensive extraction of image elements from AXTree and DOM
2. Contextual enrichment with HTML information and interactive ancestors
3. Visual analysis via AI with a detailed and structured prompt
4. Application of priority rules to override classifications when necessary
5. Quality evaluation of alternative text by comparing it with visual content
6. Detailed report generation with specific recommendations

The prompt is designed to guide the AI through a systematic and rigorous process that ensures high accuracy in classification, following web accessibility best practices according to WCAG 2.1 and W3C WAI tutorials.