Adding metadata to document chunks, following guidance from docling-haystack package.
Reference code here.
Note: we cannot integrate the package as-is because it depends on docling = "^2.9.0" while we are forced to docling>=2.4.2,<=2.8.3 from instructlab-sdg.

Metadata fields:
All the DocMeta fields apart from: schema_name, version and doc_items

Issue resolved by this Pull Request:
Closes #3192

Verifying the generated schema:
Sequence of commands to validate the schema of the default in-memory store:

ilab rag ingest --input-dir _YOUR_DOCS_DIR_
cat $(ilab config show -k rag.document_store.uri) | jq > embeddings.json
jq -r 'paths | map(if type=="number" then "[*]" else tostring end) | join(".")' embeddings.json | sort -u

Sample output (edited to show only the relevant fields):

documents.[*].content
documents.[*].embedding
documents.[*].embedding.[*]
documents.[*].id
documents.[*].meta
documents.[*].meta.headings
documents.[*].meta.headings.[*]
documents.[*].meta.origin
documents.[*].meta.origin.binary_hash
documents.[*].meta.origin.filename
documents.[*].meta.origin.mimetype
documents.[*].score
documents.[*].sparse_embedding

And a snippet of a chunk metafdata from the JSON document:

...
  "documents": [
    {
      "id": "794e8193117c68bb07ad5c58ef55f6111ecb7d8e449ec3f968769ebbfb4b5321",
      "content": "InstructLab 🐶 (ilab)\n❓ What is InstructLab Core\ngraph TD;\n download-->chat\n chat[Chat with the LLM]-->add\n add[Add new knowledge<br/>or skill to taxonomy]-->generate[generate new<br/>synthetic training data]\n generate-->train\n train[Re-train]-->|Chat with<br/>the re-trained LLM<br/>to see the results|chat\nFor an overview of the full workflow, see the workflow diagram.\n[!IMPORTANT]",
      "dataframe": null,
      "blob": null,
      "meta": {
        "headings": [
          "InstructLab 🐶 (ilab)",
          "❓ What is InstructLab Core"
        ],
        "origin": {
          "mimetype": "text/markdown",
          "binary_hash": 10151905962323472718,
          "filename": "README.md"
        }
      },
      "score": null,
      "embedding": [
        -0.04702334851026535,
    ...

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the
  conventional commits.
• Changelog updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Functional tests have been added, if necessary.
• E2E Workflow tests have been added, if necessary.