Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Refactor Tokenization #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
alvinwan merged 32 commits into from
Jul 19, 2020
Merged

Refactor Tokenization #102

alvinwan merged 32 commits into from
Jul 19, 2020

Conversation

@alvinwan
Copy link
Owner

@alvinwan alvinwan commented Jul 19, 2020

Major refactoring for tokenization, as suggested in #75, into 2 steps:

  1. Each char first categorized as a catcode
  2. Catcodes are then used for tokenization, grouping catcodes to make "token codes".

Token codes are then use for parsing. This strict abstraction makes parser refactoring and advanced features possible (e.g., fault tolerance, macros)

alvinwan added 30 commits July 15, 2020 17:03
@alvinwan
refactor tokenize out
8b2de72
@alvinwan
autopep8 + pycodestyle + manual fixes
92a8469
@alvinwan
renamed TokenWithPosition -> Token
333d39f
@alvinwan
rname file
f61d2c1
@alvinwan
rm some * mipo,rts
fb0c474
@alvinwan
defined category codes
f89edcd
@alvinwan
added category coddes
929f88e
@alvinwan
added category coddes
a3a0935
@alvinwan
replaced with cat coddes where possible
ac9d562
@alvinwan
addditional math env classes
b73a396
@alvinwan
use constants
b465954
@alvinwan
addded caetgory to token class + to_buffer can convert input and outp…
b1d0801 
…ut to bbuffer
@alvinwan
categorize each char into catcode
0866c34
@alvinwan
rename command to category
7410035
@alvinwan
add category to all token modifications
554a85e
@alvinwan
base new-style tokenizers on cat codes
5a8508b
@alvinwan
set match switch grgouped cat
8a1587d
@alvinwan
group asymmetric math switches
8814b8c
@alvinwan
mostly removed most hard-coded catcodes
86e066d
@alvinwan
filled in 'other' categgory
4bf73a9
@alvinwan
replaced catch alls with ccategory coes
6c4ecd9
@alvinwan
parsing comman name separate from escacpe (testts dont pass)
c0a50d6
@alvinwan
tmp env stopgap + tokeninzer gets access to prev toknized result (not…
9dd342d 
… just prev char)
@alvinwan
new-style line break and invali tokenn parsing
a07f8e7
@alvinwan
uneeded None chheck, read_text to_bufffer
24edbd6
@alvinwan
attach token code to all tokens
23931df
@alvinwan
new category
70bd5f9
@alvinwan
use token codes in reader, catcodes in tokenizer for arg delims
abc163c
@alvinwan
fix spaec for test
d793960
@alvinwan
fixed py3.4 compat, thank goodness for #88
15eb854
@coveralls
Copy link

coveralls commented Jul 19, 2020

Coverage Status

Coverage remained the same at 100.0% when pulling eb2e563 on tokenize_refactor into edc1194 on master.

1 similar comment
@coveralls
Copy link

coveralls commented Jul 19, 2020

Coverage Status

Coverage remained the same at 100.0% when pulling eb2e563 on tokenize_refactor into edc1194 on master.

@alvinwan alvinwan merged commit 505ecb9 into master Jul 19, 2020
@alvinwan alvinwan deleted the tokenize_refactor branch July 19, 2020 01:43
@alvinwan alvinwan mentioned this pull request Jul 20, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alvinwan @coveralls