Retraining script for spacy #52
Conversation
brabbit61
commented
Jun 21, 2023
- Added documentation to train and retrain spacy models
- Added scripts for the above tasks
tieandrews
left a comment
There was a problem hiding this comment.
Looking good just need to rearrange some things and reuse our existing code for parts. Nice work.
| if os.path.exists(val_path): | ||
| shutil.rmtree(val_path) | ||
| logger.info(f"The folder '{val_path}' has been deleted.") | ||
| if os.path.exists(test_path): |
There was a problem hiding this comment.
issue: this is duplicating the process done in the labelling_data_split.py file, instead run that script first in bash with these args then make this function run with those processed folders, that way we remove code duplication and make the parts of the process more re-usable.
| test_gdd_ids | ||
| ) | ||
|
|
||
| def get_article_gdd_ids(labelled_file_path: str): |
There was a problem hiding this comment.
Delete duplicate function, see comment above
|
|
||
| # TODO: Else If the data_path consists of parquet files, load JSON files from all parquet files in the directory | ||
|
|
||
| def split_train_val_test( |
There was a problem hiding this comment.
delete duplicate function, see comment above
|
|
||
| rm -f spacy_transformer_$VERSION.cfg | ||
|
|
||
| python3 spacy_preprocess.py \ |
There was a problem hiding this comment.
See comments on this file below, should remove duplicate code, use labelling_data_split.py here then call pre-processing