In order to reduce the need for the hardcoded chat templates during data processing,
the APIs related to the tokenizer need to be modified. These changes will take place in
the following training PR: instructlab/training#428

To make the transition smooth, we will follow this process:

1. Cap the version of instructlab-training to be v0.7 and not v0.7.1
2. Merge PR Updates data processing logic to remove dependency on hardcoded chat templates training#428
3. Release a new version of the training library
4. Update the tokenizer logic usage within the instructlab/instructlab repo (

   instructlab/src/instructlab/model/full_train.py

   Line 54 in 7841ce9

   CHAT_TEMPLATE, SPECIAL_TOKENS = utils.retrieve_chat_template(

   ) and upgrade the version of instructlab-training

Signed-off-by: Oleg Silkin [email protected]

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the
  conventional commits.
• Changelog updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Functional tests have been added, if necessary.
• E2E Workflow tests have been added, if necessary.