Use py scadparser #170
Use py scadparser #170
Conversation
(and added missing newline before EOF)
- parse_scad_callables now receives a filename -> write test_code to tempfile - since the code get written to a file string escapes need to be double escaped ;) - corrected syntax in var_with_functions parameters
- I hope this works for everybody else, but I think it should
|
2 out of the 3 new commits should fix the issues you mentioned in #166. The third commit made little changes to run_all_test.sh to make it work on my system and I assume it should not break it for anyone else (working on linux!) and hopefully makes it work on more systems. if you don't like it we can revert it and I keep it as a local copy. |
|
This looks good. Couple questions:
|
|
One other thing: Do you know what the deal is with this parser output?: In [2]: MCAD = import_scad('MCAD')
Illegal character (31) """
31:1132 < - <
syntex error
Illegal character (31) "'"
Illegal character (31) "'"
Illegal character (32) "'"
Illegal character (32) "'"
Illegal character (33) "'"
Illegal character (33) "'"
Illegal character (33) """
36:1305 } - }
syntex error
... and similar....The MCAD import all works, but I'm not sure what the parser is doing with these bits. Do you think it's a problem with the grammar? |
hmmmmm this should not happen..... I'll check and think about it..... How do you execute your code? (what is I've no clue what this is about right now. I'll try to improve the error "logging" to include the filename and we should try to track it down. When does it occur? Only with MCAD? |
|
While trying to redo what you've done I by accident stumbled over this: I think this is another bug, I assume the filenames (-> module names) don't get checked whether they are valid python modifiers..... |
|
I can't reproduce your issue.... I can import MCAD without any of these messages and they should not occur! It means that the lexer does not now certain characters and the parser gets confused because the whole gramar does not make any sense with missing tokens. |
I'm absolutely not into packaging, I've no clue what you're talking about ;)
ply (bison) are somehow table driven. They "create" a certain table from the grammar and use that table to parse stuff. My first thought was to .gitignore it and let every installation generate it's own parsetab.py on the first execution, BUT this does not worked if installed in an read only location (e.g. system-wide). So we should put it under version control and ship it with the package. from ply.docs
I guess we can .gitignore it
I don't get this.... to me it seems to be the same and should be fixed by the branch you mentioned. Btw: I thought about adding a |
|
I think we should both add the parser output to source control, AND add it to .gitignore; otherwise the time stamp change on first run will appear in I wonder if the MCAD imports are some kind of case-sensitivity issue? I’m running ion MacOS which has less strict (and less logical) case treatment than Linux. I’m away from a computer again for a couple days, but will poke at that some more when I’m back The issue with the missing initial digit handling is a GitHub wrinkle; I pushed my commits to a branch in the SolidCode/SolidPython repo rather than your PR branch. Looks like it’s working OK. I’ll grab that and the filenames bug you found. We’ll get there! Thanks again for your work on this and your good instincts |
I don't think so and I don't think it's a good idea ;) Furthermore I would be afraid this could become a commit fall trap. If you change the grammar (bugfixes) the resulting generated tables will not show up in git and you're very likely to forget to commit them....
If that's the output there's something not working completely off and if wonder how it can "still work". Does MacOS use different line endings or tabs or something like that? I wonder if our MCAD/...* file(s) that cause the problem are the same. |
- could this maybe fix the "MacOS mcad issue" from SolidCode#170
|
Could you try to reproduce the error with the two patches applied and if they still occur send me the error log and at least the first file mentioned in it |
|
I just tried to use BOSL2 and it throws syntax errors. I started extending the grammar (bosl2 uses some language features the parser (and me) hasn't seen yet). Therefor I think you can wait until I'll extended the grammar (everything that bosl2 comes up with) and we'll give your issue another try afterwards because the bosl2 changes might include a fix for your issue. |
- [un]subbed_keyword should now handle $parameters - it should be possible to remove all $fn <-> segments code from everywhere and let [un]subbed_keyword do the work
|
Give it a try....... It can now import bosl2 and is based on the parser.y from openscad itself (could have done that earlier, but.....that would have been to easy....) I merged openscad_identifiers_starting_with_digits into this branch so #171 would be obsolete if you merge this branch. (it was just annoying to have already fixed bugs showing up while debugging and merging this....) ca012e5 also contains code that [un]subbed_keyword should now be able to handle all the $fn <-> segments mapping stuff. I think we can delete all the other bits of code that (now >seems<) to handle it. This was necessary, because py_scadparser now handles $fn parameters correctly and thus solidpython must handle it in a general manner (e.g. bosl2 contains custom $parameters). Offtopic https://github.com/revarbat/BOSL2/wiki part-hole stuff: https://github.com/revarbat/BOSL2/wiki/attachments.scad#module-diff |
|
OK, right on. I just pulled the top of your branch tree, and I'm liking where things are at. I've merged this to master, so we may be able close this issue and a couple others. There are a couple details I still want to look at, but let's put them into new issues as needed.
|
|
Hmm.. Just looked at PR #173, which looks like it's a better solution for python-illegal filenames than the little hack I put into master. I may look to merge that in instead of my fix. |
After having a look at BOSL2 I wonder if MCAD becomes obsolete....
-> #176 should solve this. A single resolve_scad_filename function that puts the "solidpython version" behind the user space version.... but I would probably put it before a system wide version, e.g. the order the resolve filenames could be:
I think that would be the way I would do it. But then I would not import those libs on a regular startup. Don't include them in init or similar, because importing BOSL2 -- at least on my system -- takes 2secs. I would do something like creating a sub package for -- each? -- library. -->
Actually I don't know of any special treatment for $tags is not OpenSCAD, it's from BOSL2. You can "tag" -- I guess -- any object in the tree with a certain tag. Afterwards you can do something like
hmmmm I don't really like it if it's not really necessary. Did you try to add parsetab.py to git (and not to .gitignore) and parse.out to .gitignore? I didn't try it but I guess it'll work. |
Another solution could be to keep import_scad exactly the way it is right now (searches relative paths, ~/.local/... and I would add /usr/share/openSCAD/libraries) and then add solid.bosl2 as a package that -- somehow -- imports the version installed with solidpython (for example resolving the absolute path of it and than import it with an absolute path). |
|
I just ran some tests on the non-cached parser version. That's a big time sink (about 5x time!) , so I'll definitely be switching to write |
|
re: https://docs.python.org/3/tutorial/classes.html#private-variables |
|
I guess: If you take a look at parsetab.py line 9 it says: _lr_signature = a string that seems to be or similar. I assume whenever ply is called it will extract exactly that signature from the source and compare it. Only if it changed it'll regenerate parsetab.py. That would mean it is completely independent of the file timestamps and even of the file contents (as long as the signature didn't change) and it will only generate the tables if it is needed. And if that's the case it's absolutely right that git shows the parsetab.py as changed (because it needs to be update in the repo). Therefor there should be no issues (at all?). |
|
Agreed. I've added parsetab.py in my local version, will push to master when I've kicked the tires a little bit. I didn't add the debug argument back in (which causes parser.out to be written). Do you see any reason why that file should be there? |
|
https://www.dabeaz.com/ply/ply.html#ply_nn49 Packaging advice from the ply docs especially about parsetab.py (but it unfortunately doesn't go into the details) |
|
from the docs:
No, seems to be pure debugging. Get rid of it ;) |
|
There's one more issue concerning this merge. Since 8 hours ago py_scadparser/scad_parser.py is based on openscad/parser.y. What I actually did was to grab that file. Try to make it run with pybison (what didn't work out of the box) but it generated a python version of it and I grabed that and mouldet it into a ply version that's now scad_parser.py. In other words the grammar definition was extracted from parser.y. I'm not into all this packaging and licensing stuff, but if I understand it correctly the result of that is, that scad_parser.py is technically a (heavy) modified version of openscad/parser.y, right? And if I understand the GPLv2 correctly that means scad_parser.py also has to be under GPLv2. I changed the LICENSE file in the py_scadparser repo and I think we should do the same in SolidPython/solid/py_scadparser. Sorry for that. |
|
Oh man, I also try to avoid having to think about licenses at all. They're a real drag. I think however, that if you're not committing The current SolidPython license is LGPL 2.1, since, as I recall, it's minimally restrictive. I just want to share cool things and not worry about those details (and some GNU troll may show up shortly and write me a novel about how this is exactly why I should care about their troll-y legal stuff and also would I maybe like some of their Kool Aid. Thank you, I would not). So anyway, I'm emphatically opposed to any kind of viral licensing. My idea of free to share means free to share, change, steal, relabel, whatever. If you look into this some more and you find to the best of your understanding that there's no way to avoid keeping the code you added this morning without a license change, we'll back it out. |
|
You wrote the first version of the OpenSCAD grammar by hand though, right? How difficult would it be to just eyeball the diffs between your original version and the parser.y version and correct any mistakes you made? Seems like you couldn't run into any license issues that way |
|
Aaeeh, yeah I might do something like that, but my motivation to do
it.............
|
|
I hear you man; not super fun. Thanks for all your work on this; I think you have better instincts about these things than I do. I'm going to roll master back to where it was before until we've resolved license issues. |
|
There's something in the pipeline that looks pretty ok (not as nice as the other one but ok) but the polishing is still missing... |
Take a look at: 0fda2d2 (extend import_scad to be able to pass a destination namespace as parameter) The result is: import solid.libs.bosl2
solid.libs.bosl2.linear_bearings.linear_bearing_housing()works just fine |
|
Let's move the BOSL2 issues to a separate PR and work there. I'm trying to figure out how best to incorporate the BOSL2 code into SolidPython's. Should we inline the code? Use a git submodule that could be updated (I think so? Submodules don't autopopulate though, which is often a hassle) to match the current versions? Anyway, let's continue in another thread |
|
Hey @jeff-dh - I wonder if you've got any ideas about very long parse times for SolidPython-generated SCAD files. I ran across this while running unit tests. If you you run from solid import import_scad
maze = import_scad('/$PATH_TO/SolidPython/solid/examples/Compiled_examples/mazebox.scad')It will take a surprising (10+ seconds) amount of time to parse. That's not ridiculous, since one line of that file contains 1.5M characters. That isn't a horrible outcome, and the times when you're recursively importing OpenSCAD-generated |
|
Aaand, one more question. Looks like those backslashes ( |
|
aah, r"..." is a raw string? I though it would be a shortcut for regex... |
|
They get used for regexes a bunch so that characters' special meanings can be used for regex purposes, not for Python's normal escape purposes. Like single quotes in Bash as opposed to double quotes? Anyway, it's not super important but seems like one last wrinkle to smooth out. What could go wrong, right? |
|
Not sure whether I understand you correctly in any single point, but I'll give you a few thoughts about the whole topic. I don't think you should include such a scad file ;) If you do something like this it seems to me like a design error (in the openscad world). If I create an OpenSCAD module that provides a public interface it should not contain enormous amounts of data......... I would suggest to extract and separate them. The fix is pretty simple: Don't do it ;) BUT: Yeah I know that the parser is not very fast. I think this is because its a pure python parser. I'm pretty sure regular bison & flex are gonna be probably at least 10 times faster, but well, this is the price we pay for all the other nice features of python. I spent half an hour looking into some other python parsers which are running on a c-backend but couldn't find an easy to do alternative. Anyway: My first try to write a scad parser was exactly based on the idea you mentioned. Only parse the lines starting with the module or function keyword but I couldn't make it work in bison to skip the whole body (because you do need context to recognize the corresponding closing brace, you can't do it in the lexer). This might be possible somehow but I couldn't figure out how to do it. I don't have a good solution to improved the parser speed itself, BUT I had the same issue while working on bosl2 (which consists of ~30 scad files) and it took 2secs to import bosl2 in every single test cycle and found a "funny" solution for it: the pickle-cache. What it actually does it is caches the output of the parser to disk and if it is ask to parse a file it first looks into the pickle-cache-directory whether there's a valid cached version of the parsed-output. If so read and use it. This increases the speed importing bosl2 by a factor of ~15 and thus was a very easy solution and is good enough for me. It basically only parses each (at least library files) once, so the first time you -- for example -- import bosl2 it is "slow" (2secs) but thereafter its fast (100ms). https://github.com/jeff-dh/SolidPython/blob/exp_solid/solid/core/parse_scad.py That's the parse_scad module including the pickle-cache used in exp_solid take a look at it (or try to plug it into master, might work) |
It seems like that's the solution: ply docs:
python re docs:
|
|
Right on. I think your "Don't do that" fix is the right one. The only reason I ran into it at all was that I had a unit test that was recursively importing the whole I'll just let this one go. The pickle solution you've got going there looks nice and super fast for anybody using a lot of imported SCAD code. Once I've caught up to your other changes, maybe we can fold that into master. |
This branch uses a py_scadparser a python scad parser based on ply to parse open scad code.