Analysis by Claude

1. validation.yaml (reporting.yaml)

Status: ✅ ACTIVELY USED - CORE INFRASTRUCTURE

Current Usage

In linkml codebase:

• Core to validation framework at linkml/validator/
• Used in 8+ files:
  • validator/validator.py
  • validator/report.py (defines Pydantic equivalents)
  • validator/plugins/jsonschema_validation_plugin.py
  • validator/plugins/shacl_validation_plugin.py
  • validator/plugins/pydantic_validation_plugin.py
  • validator/plugins/recommended_slots_plugin.py
  • validator/__init__.py

In linkml-runtime:

• Present at linkml_runtime/linkml_model/model/schema/validation.yaml
• Generated Python model at linkml/reporting/model.py

Key Classes:

• ValidationReport - Container for validation results
• ValidationResult - Individual validation finding
• Severity enum: FATAL, ERROR, WARNING, INFO

Why Not in meta.yaml?

Intentional separation - Validation is a runtime concern, not a metamodel concern. The metamodel defines what schemas look like; validation.yaml defines what validation reports look like.

Essential infrastructure. Do not remove or modify.

2. array.yaml

Status: ✅ COMPLEMENTARY TO meta.yaml

Critical Discovery: Two Different Array Approaches

array.yaml provides:

• Class-based approach: NDArray, DataArray classes
• Used when you need arrays with additional metadata (units, conversion factors, etc.)
• Import via: implements: [linkml:lib/arrays#NDArray]
• Example use case: NWB (Neurodata Without Borders) needs arrays with precision, units, conversion metadata

meta.yaml's array_expression provides:

• Slot-level approach: Define array shapes inline
• Used when you just need dimensions specification
• Import via: Direct use of array: slot property
• Introduced in LinkML 1.7.0 (February 2024)

Key Difference

# Using array.yaml (class-based, with metadata)
classes:
  TimeSeries:
    attributes:
      data:
        range: NDArray
        # NDArray class has elements, units, conversion, etc.

# Using meta.yaml's array_expression (inline dimensions)
classes:
  ImageData:
    attributes:
      pixels:
        range: integer
        array:
          dimensions:
            - alias: width
            - alias: height

GitHub Evidence

Active Development:

• Issue Bad link in main page markdown document #199 (linkml-model): Enable any_of with ArrayExpressions - RESOLVED
• PR removed \n Add enums to markdowngen #120 #200 (linkml-model): Moved array to slot_expression mixin - MERGED Oct 2024
• Issue Add support for linkml:array in pydanticgen #1888 (linkml): Add array support to pydanticgen - CLOSED (implemented in 1.7.0)
• Issue decide on strategy for serializing NDArrays as RDF and validating using shacl/shex #1890 (linkml): NDArray RDF serialization strategy - OPEN (active discussion)
• Issue array: null for any shaped arrays not allowed #2406 (linkml): Support array: null for any-shaped arrays - OPEN
• PR Generate basic Pydantic model for data arrays #1397 (linkml): Generate Pydantic models for data arrays - MERGED
• Working group: @linkml/ndarray-wg actively developing

Test Coverage:

• tests/test_compliance/test_array_compliance.py - Comprehensive array validation tests
• Tests across pydanticgen, jsonschemagen, etc.

Why Two Approaches Coexist

1. Different use cases: Simple dimensions vs. rich metadata
2. Different communities: General LinkML users vs. scientific data (NWB, neuroimaging)
3. Historical: array.yaml predates array_expression
4. Intentional design: Both are maintained by design

3. datasets.yaml

Status: ⚠️ DORMANT WITH KNOWN BUGS

Purpose

Multi-standard metadata model mapping to:

• Frictionless Data Package/Resource
• W3C DCAT (Data Catalog Vocabulary)
• W3C VOID (Vocabulary of Interlinked Datasets)
• Schema.org (CreativeWork, DataDownload)
• CSVW (CSV on the Web)

Timeline

• Created: December 2021 (commit a14b757b)
• Total commits: 11 over 3 years
• Last update: February 2024 (maintenance only)
• No active development since Feb 2024

Known Issues

CRITICAL BUG - Issue #1763 (OPEN since Dec 2023):

• NameError: name 'PermissibleValue' is not defined when importing
• Generated Python code has import errors
• Projects cannot reliably import DataPackage/DataResource classes
• Workaround: Projects copy relevant parts instead

Usage Evidence

Minimal adoption:

• Only 1 external project found attempting to use it: bridge2ai/data-sheets-schema (hit the import bug)
• No test files in linkml repo
• No examples in linkml-model
• No documentation

In linkml-runtime:

• Schema file present but unused
• Available via linkml_runtime.linkml_model.datasets module

Related Discussions

• Issue Add a utility to validate and transform all registered example or data files in a repo #501 (2021): Utility to validate data files - discussed datasets concept, deferred
• Issue Add frictionless schema importer/exporter #861 (2022): Frictionless schema importer/exporter - closed without completion

Why Not Used?

1. Import bug prevents adoption (Issue Tests break due to NameError: name 'PermissibleValue' is not defined when importing linkml:datasets #1763)
2. Projects use Frictionless directly instead
3. No clear advantage over existing standards
4. Lacks documentation and examples

Recommendation: KEEP but Fix or Deprecate Formally

Options:

1. Fix Issue Tests break due to NameError: name 'PermissibleValue' is not defined when importing linkml:datasets #1763 and create examples/documentation → Make it actually usable
2. Formally deprecate with migration guide to Frictionless Data
3. Move to examples/ as reference implementation, not production schema

Do NOT silently remove - it represents legitimate W3C standards mappings. Either fix or deprecate explicitly.

4. extended_types.yaml

Status: ❌ ORPHANED - ZERO USAGE

Timeline

• Created: PR Add generation of __hash__ to pythongen #185 (Feb 8, 2024)
• Merged: April 12, 2024
• Origin: Issue Add alternative numeric type system #1871 - Request from ndarray working group
• NOT in linkml-runtime: Confirmed absent
• Zero code usage: No imports found anywhere

What It Provides

NumPy-style extended numeric types:

• Signed integers: int8, int16, int32, int64
• Unsigned integers: uint8, uint16, uint32, uint64
• Floating point: float16, float32, float64
• Union type: any_number
• Unconstrained: Any

All with NumPy documentation refs and value constraints.

Why It Exists But Isn't Used

1. Never integrated into linkml-runtime - exists only in linkml-model
2. No generator support - would need pydanticgen, jsonschemagen updates
3. Unclear adoption path - how would users actually use these types?
4. Minimal discussion - PR had almost no comments beyond bool inclusion debate

No Test Coverage

• No test files reference extended_types
• No examples demonstrate usage
• No documentation mentions it

Recommendation: REMOVE OR INTEGRATE PROPERLY

This schema is aspirational but abandoned. Options:

Option 1: Remove entirely

• Minimal value currently
• Zero usage detected
• Not in runtime
• Creates maintenance burden

Option 2: Integrate properly (significant work)

• Add to linkml-runtime
• Update all generators (pydantic, jsonschema, etc.)
• Create documentation and examples
• Announce the feature

Option 3: Keep as experimental (status quo)

• Document as experimental/unstable
• Low maintenance cost (single file)
• Available for future use

My recommendation: REMOVE unless there's commitment to Option 2 integration work. An unused, undocumented feature that's not in the runtime provides no value and creates confusion.

Comparison: Why Some Schemas Aren't in meta.yaml

The Pattern

Schemas NOT imported into meta.yaml fall into categories:

1. Runtime concerns (validation) - Used at validation time, not schema definition time
2. Optional rich features (array.yaml) - Available but not required
3. Standards mappings (datasets) - Bridge to external ecosystems
4. Experimental/abandoned (extended_types) - Created but never finished

Recommendations for GitHub Discussion

Immediate Actions

1. validation.yaml: Document why it's separate from meta.yaml (runtime vs metamodel)
2. array.yaml: Document relationship with meta.yaml's array_expression, provide usage guide
3. datasets.yaml: Either fix Issue Tests break due to NameError: name 'PermissibleValue' is not defined when importing linkml:datasets #1763 + add docs, or formally deprecate
4. extended_types.yaml: Remove or commit to full integration

Documentation Needed

Create linkml-model/README.md explaining:

• Purpose of each schema file
• Which are core (in meta.yaml) vs optional
• How to use optional schemas
• Relationship between array.yaml and array_expression

Questions for Maintainers

1. Is there commitment to fix datasets.yaml Issue Tests break due to NameError: name 'PermissibleValue' is not defined when importing linkml:datasets #1763?
2. Should extended_types be integrated into runtime or removed?
3. Should array.yaml usage be documented prominently?
4. Should validation.yaml be documented as the standard validation report format?

I initially thought you triggered Claude to answer this in GH but it seems like you must have asked Claude separately and then pasted its answer here?

so should i move this to issues?