Convert FLEx data to CLDF-ready CSV.

Many descriptive linguists have annotated language data in a FLEx (SIL's Fieldworks Lexical Explorer) database, which provides perhaps the most popular and accessible assisted segmentation and annotation workflow. However, a reasonably complete data export is only available in XML, which is not human-friendly, and is not readily converted to other data. A data format growing in popularity is the CLDF standard, a table-based approach with human-readable datasets, designed to be used in CLLD apps and easily processable by any software that can read CSV files, including R, pandas or spreadsheet applications. The goal of cldflex is to convert lexicon and corpus data stored in FLEx to CSV tables, primarily for use in CLDF datasets.

cldflex is available on PyPI:

pip install cldflex

At the moment, there are three commands: cldflex corpus for .flextext files; cldflex dictionary and cldflex wordlist for .lift files. All commands create a number of CSV files. One can either use cldfbench to create one's own CLDF datasets from these files, or add the --cldf argument to create a simple CLDF dataset. Project-specific configuration can be passed by --conf your/config.yaml, or creating a file cldflex.yaml

Basic usage:

cldflex corpus texts.flextext

Connect the corpus with the lexicon:

cldflex corpus texts.flextext --lexicon lexicon.lift

Create a CLDF dataset:

cldflex corpus texts.flextext --lexicon lexicon.lift --cldf

Extract morphemes, morphs, and entries from lexicon.lift:

cldflex dictionary lexicon.lift

Create a CLDF dataset with a Dictionary module:

cldflex dictionary lexicon.lift --cldf

Create a CLDF dataset with a Wordlist module:

cldflex wordlist lexicon.lift --cldf

The functions corresponding to the commands above are cldflex.corpus.convert() and cldflex.lift2csv.convert().

There is no default configuration. Rather, cldflex will guess values for most of the parameters below and tell you what it's doing. It is suggested to start out configuration-free until something goes wrong or you want to change something. Create a YAML file for CLI usage, pass a dict to the convert methods.

• obj_lg: the object language
• gloss_lg: the language used for glossing / translation
• msa_lg: the language used for storing POS information
• lang_id: the value to be used in the created tables
• glottocode: used to look up language metadata from glottolog
• csv_cell_separator: if there are multiple values in a cell (allomorphs, polysemy...), they are by default separated by "; "
• form_slices: set to false if you don't want form slices connecting morphs and word forms
• mappings: a dictionary specifying name changes of columns in the created CSV files