lexical-analysis
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 625 public repositories matching this topic...
百度NLP：分词，词性标注，命名实体识别，词重要性
-
Updated
May 25, 2021 - C++
xmnlp：提供中文分词, 词性标注, 命名体识别，情感分析，文本纠错，文本转拼音，文本摘要，偏旁部首，句子表征及文本相似度计算等功能
-
Updated
Nov 12, 2022 - Python
DFA regular expression library & friends
-
Updated
Aug 29, 2025 - C
INTERSPEECH 2023-2024 Papers: A complete collection of influential and exciting research papers from the INTERSPEECH 2023-24 conference. Explore the latest advances in speech and language processing. Code included. Star the repository to support the advancement of speech technology!
-
Updated
Dec 25, 2024
Chinese text segmentation with R. R语言中文分词 （文档已更新 🎉 ：https://qinwenfeng.com/jiebaR/ )
-
Updated
Jul 13, 2020 - C++
LLVM Tutorial: Kaleidoscope (Implementing a Language with LLVM)
-
Updated
Dec 29, 2022 - C++
Allocators, I/O streams, math, geometry, image and audio processing for D
-
Updated
Aug 30, 2025 - D
Implementing a complete Compiler for a simple C-like language using the C-tools Flex and Bison
-
Updated
Feb 8, 2023 - C
OysterKit is a framework that provides a native Swift scanning, lexical analysis, and parsing capabilities. In addition it provides a language that can be used to rapidly define the rules used by OysterKit called STLR
-
Updated
Aug 15, 2019 - Swift
[EMNLP 2021] LingFeat - A Comprehensive Linguistic Features Extraction ToolKit for Readability Assessment
-
Updated
Mar 7, 2023 - Python
😸 💬 A module to compute textual lexical richness (aka lexical diversity).
-
Updated
Aug 27, 2023 - Python
clex is a simple lexer generator
-
Updated
Jul 19, 2025 - C
A procedural programming language built in Rust which compiles to QBE
-
Updated
Sep 2, 2025 - Rust
Modular static malicious JavaScript detection system
-
Updated
Jan 18, 2021 - Python
A Lexical Analyzer Generator
-
Updated
Dec 15, 2024 - Common Lisp
A simple compiler written from scratch in C++ for an undergraduate course in program translation.
-
Updated
Apr 29, 2018 - C++
These are my programs for compiler design lab work in my sixth semester
-
Updated
May 25, 2022 - Python
Platform for parser generators and other grammarware research and development. GLL, RNGLR, graph parsing algorithms, and many others are included.
-
Updated
Jul 25, 2019 - F#
- Followers
- 11k followers
- Website
- github.com/topics/parsing
- Wikipedia
- Wikipedia