Links to publicly available russian corpora + code for loading and parsing. 20+ datasets, 350Gb+ of text.

For example lets use dump of lenta.ru by @yutkin. Manually download the archive (link in the Reference section):

wget https://github.com/yutkin/Lenta.Ru-News-Dataset/releases/download/v1.0/lenta-ru-news.csv.gz

Use corus to load the data:

>>> from corus import load_lenta

>>> path = 'lenta-ru-news.csv.gz'
>>> records = load_lenta(path)
>>> next(records)

LentaRecord(
    url='https://lenta.ru/news/2018/12/14/cancer/',
    title='Названы регионы России с\xa0самой высокой смертностью от\xa0рака',
    text='Вице-премьер по социальным вопросам Татьяна Голикова рассказала, в каких регионах России зафиксирована наиболее высокая смертность от рака, сооб...',
    topic='Россия',
    tags='Общество'
)

Iterate over texts:

>>> records = load_lenta(path)
>>> for record in records:
...     text = record.text
...     ...

For links to other datasets and their loaders see the Reference section.

corus supports Python 2.7+, 3.4+ и PyPy 2, 3.

$ pip install corus

MIT

• Chat — https://telegram.me/natural_language_processing
• Issues — https://github.com/natasha/corus/issues

Tests:

make test

Add new source:

1. Implement corus/sources/<source>.py
2. Add import into corus/sources/__init__.py
3. Add meta into corus/source/meta.py
4. Add example into docs.ipynb (check meta table is correct)
5. Run tests (readme is updated)

Package:

make version
git push
git push --tags

make clean wheel upload