LtacProf (Coq v8.5) #150
LtacProf (Coq v8.5) #150
Conversation
Patch by Paul A. Steckler Example of the bug: ``` $ cat foo.v Fixpoint fact n := match n with 0 => 1 | S n' => n * fact n' end. Ltac a := idtac; let x := fresh in set (x := fact 7). Definition foo := Eval compute in fact 7. Goal fact 7 * 10 = foo * 10. Proof. unfold foo. Start Profiling. Time a. Show Profile. Time a. Show Profile. Time a. Show Profile. Time a. Show Profile. Abort. In 8.4pl6 + ltacprof: $ coqc foo.v Finished transaction in 0. secs (0.144u,0.s) total time: 0.144s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 1 0.144s ─a ------------------------------------- 0.0% 100.0% 1 0.144s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 1 0.144s └set (x := fact 7) --------------------- 100.0% 100.0% 1 0.144s Finished transaction in 0. secs (0.128u,0.s) total time: 0.272s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 2 0.144s ─a ------------------------------------- 0.0% 100.0% 2 0.144s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 2 0.144s └set (x := fact 7) --------------------- 100.0% 100.0% 2 0.144s Finished transaction in 0. secs (0.132u,0.s) total time: 0.404s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 3 0.144s ─a ------------------------------------- 0.0% 100.0% 3 0.144s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 3 0.144s └set (x := fact 7) --------------------- 100.0% 100.0% 3 0.144s Finished transaction in 0. secs (0.128u,0.s) total time: 0.532s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 4 0.144s ─a ------------------------------------- 0.0% 100.0% 4 0.144s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 4 0.144s └set (x := fact 7) --------------------- 100.0% 100.0% 4 0.144s In v8.5+ltacprof (note: I've just rebased against upstreams v8.5 branch and pushed, but I don't think it changes anything) (after trimming*): Finished transaction in 0.323 secs (0.312u,0.011s) (successful) total time: 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 1 0.324s ─a ------------------------------------- 0.0% 100.0% 1 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 1 0.324s └set (x := fact 7) --------------------- 100.0% 100.0% 1 0.324s total time: 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 100.0% 1 0.324s ─a ------------------------------------- 0.0% 100.0% 1 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 100.0% 1 0.324s └set (x := fact 7) --------------------- 100.0% 100.0% 1 0.324s Finished transaction in 0.292 secs (0.291u,0.s) (successful) total time: 0.616s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 52.6% 2 0.324s ─a ------------------------------------- 0.0% 52.6% 2 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 52.6% 2 0.324s └set (x := fact 7) --------------------- 100.0% 52.6% 2 0.324s total time: 0.616s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 52.6% 2 0.324s ─a ------------------------------------- 0.0% 52.6% 2 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 52.6% 2 0.324s └set (x := fact 7) --------------------- 100.0% 52.6% 2 0.324s Finished transaction in 0.289 secs (0.284u,0.004s) (successful) total time: 0.904s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 35.8% 3 0.324s ─a ------------------------------------- 0.0% 35.8% 3 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 35.8% 3 0.324s └set (x := fact 7) --------------------- 100.0% 35.8% 3 0.324s total time: 0.904s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 35.8% 3 0.324s ─a ------------------------------------- 0.0% 35.8% 3 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 35.8% 3 0.324s └set (x := fact 7) --------------------- 100.0% 35.8% 3 0.324s Finished transaction in 0.293 secs (0.288u,0.008s) (successful) total time: 1.200s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 27.0% 4 0.324s ─a ------------------------------------- 0.0% 27.0% 4 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 27.0% 4 0.324s └set (x := fact 7) --------------------- 100.0% 27.0% 4 0.324s total time: 1.200s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─set (x := fact 7) --------------------- 100.0% 27.0% 4 0.324s ─a ------------------------------------- 0.0% 27.0% 4 0.324s tactic self total calls max ────────────────────────────────────────┴──────┴──────┴───────┴─────────┘ ─a ------------------------------------- 0.0% 27.0% 4 0.324s └set (x := fact 7) --------------------- 100.0% 27.0% 4 0.324s Note that the highest number in the total column gets lower after every successive call to a tactic. ```
This does the equivalent of inserting `Start Profiling` at the top of each file and `Show Profile` at the bottom.
|
This looks OK to me implementationwise. What is the average overhead due to this patch when not using the debugging feature? Is it observable? |
|
@mattam82 why shouldn't this go into a pl? (What's the boundary between acceptable for pl vs not? My understating was that pls can contain changes which do one of:
I think LtacProf basically fits into this last category, since none of the commands are meant to be used in non-testing settings.) @ppedrot I haven't tested, but I seem to recall @tebbi mentioning a 1%-2% slowdown. I'll plan to give you data on HoTT and the standard library later today. |
|
The overhead on the standard library seems to be about 2.5% (about 4% on the overall build time of Coq): Build time of successive builds of Coq (4c078b0) without LtacProf: with LtacProf: The command was: make clean && git clean -xfd && ./configure -local && /usr/bin/time -f "all coq (real: %e, user: %U, sys: %S, mem: %M ko)" make STDTIME='/usr/bin/time -f "$* (real: %e, user: %U, sys: %S, mem: %M ko)"' TIMED=1 -j4 2>&1The only file that changed by more than 0.6s of user time was I suspect that most of this overhead comes from either moving more things into the tactic monad, or stringifying the names of tactics even when not profiling. I'm currently testing a change that makes it so we only stringify tactic names when profiling, and after that I'll make a test case that stresses the calling of multiple tactics, and perhaps profile some chunk of bedrock. |
|
The lazy stringification patch brings the overhead down to 1%, and pushes the overall compilation time down to The log for the standard library is: |
This brings the overhead on the standard library when not profiling down from 2.5% to 1%.
|
There is a bug in LtacProf that I'm not sure how to fix, though I'm not sure that this should prevent it from being merged: LtacProf can't handle backtracking into the middle of tactics with multiple successes, because it doesn't know how to reconstruct the trace tree. Here is a small example that gives rise to the anomaly: Start Profiling.
Ltac multi := (idtac + idtac).
Goal True.
try (multi; fail). (* Anomaly: Uncaught exception Failure("hd"). Please report. *)
Admitted.Certainly, we could make the usage more friendly by catching this exception and printing a warning instead (what's the appropriate way to print a warning in Coq?). I think for trunk, there should be a way to hook into the re-entering of a tactic with multiple successes, so that we can restart the timers appropriately. ( @aspiwack @ppedrot @tebbi, thoughts on this?) |
|
@JasonGross msg_warning would be the way to print a warning |
|
@mattam82 Thanks! I've turned the anomaly into a warning. |
|
What's blocking this? @mattam82, any comments on my point in #150 (comment)? |
|
On a metalevel note, I am unsure that adding features after a release is a good development practice, but I am not too fond of outrageously strict rules either. Notwithstanding this point, I think we should indeed include it. What is the opinion of @maximedenes, all hail to our Great Good Development Practice Enforcer? |
|
@ppedrot, a good point once made by @silene is that adding a feature in plX+1 is bad if then some code working with plX+1 does not work with plX. Eg. if we add a compilation flag in 8.5pl2, then coqc from8.5pl1 would complain (no way way to make a Makefile that works with all pl releases). Said that, I don't know if LtacProf classifies: Start Profiling seems a debugging command to me. I mean, not something ending up in a standard file. |
|
I agree with the point about not adding new features in the middle of pls in general, but I think adding debugging features (and flags like -time, -profileltac, Start Profiling, etc) should be fine. @mattam82 what do you mean by specializing do_profile? Do you mean storing the function in a ref cell and changing which function is stored when you change the profiling behavior? |
|
No, I mean doing the condition on profiling earlier, avoiding a bind and finally |
|
|
@gares can I just override the |
|
As a general rule of thumb, I think you should go for the immutable structure instead. That removes a lot of potential interaction bugs... |
|
@maximedenes on the Bedrock src target (with a few admits, made with -j5), the stats are:
Of the user time reported by the profiler, 2302.384s of it was spent in tactic-land. This is well under 0.05% overhead when the profiler is off, and the overhead when it's turned on seems to be about 2% on tactics. |
This is suboptimal, because mutation leaves room for subtle bugs, but rewriting @tebbi's code to be functional was a pain, and not something I could figure out how to do easily. I'm working under the assumption that there is no sharing in a single treenode, which I'm not completely sure is valid. That said, a few simple tests seem to indicate that this works as expected.
|
@ppedrot @gares @herbelin I've pushed a commit synchronizing the profiling data with the document state. I spent the afternoon and evening trying to recode it as a functional data structure, and did not succeed (@tebbi's mutation-using code is too tricky for me), so I just provided a deepcopy freeze method, which seems to work in the basic tests I did. Eventually we'll want to reimplement this as a plugin on top of a nonlogical extension to the tactic monad that supports |
|
To be clear, it's now the case that |
|
There's now Current status of this PR, as I understand it: waiting for general feedback from @aspiwack , an ack from @gares / @ppedrot on the freeze function I wrote, comments from @maximedenes on if the profiling of bedrock src is satisfactory, and a response from @herbelin on whether I should stick with |
test-suite/success/ltacprof.v
Outdated
| Set Ltac Profiling. | ||
| Ltac multi := (idtac + idtac). | ||
| Goal True. | ||
| try (multi; fail). (* Anomaly: Uncaught exception Failure("hd"). Please report. *) |
There was a problem hiding this comment.
Is this comment up to date? I thought this had been changed to a warning.
There was a problem hiding this comment.
I've taken to annotating test suite lines with the error messages they're supposed to prevent, to make it easier to debug their purpose if they break as Coq changes. I'll add a "used to be: " to the front of the comment.
|
Sorry for being late to the party. TL;DR: I'm fine with this. I like that it is a fairly non-intrusive change (on that subject, @JasonGross: if you use emacs, I would recommend using ws-butler (kindly available on melpa) rather than I'm less fond of the fact that this profiler interacts purely with the Ltac layer. But as we had seen in the last coding sprint, it may not be obvious how to plug this sort of things directly within the tactic monad (@JasonGross had troubles trying to implement a tracing mechanism). So I'm ok with settling for a pragmatic solution. I must say I am a bit concerned about the issue with backtracking (it means that if backtracking is used, even a little, in commonly used tactics, profiling may start to be mostly useless). It's probably not an issue for 8.5, but for subsequent versions of Coq, it may be useful to identify what is required for profiling to handle backtracking. |
|
Thanks for the ws-butler suggestion. Plugging the profiler into the tactic monad will eventually fix the issue with backtracking, at the cost of additional not-yet--determined overhead. The trace info calculation is already working, all that remains there is figuring out the right interface for plugins. But there's extra overhead there, especially when profiling (we have to compare call stacks on every call) because there's no naive way to be notified when backtracking has happened, at least none that I've seen. So I'd rather get this merged, because it's noninvasive and low overhead and works, and update it when I next have time to work on tracing, which is rather invasive. If we don't plug the profiler into the monad, the minimum requirement for handling backtracking is some way to be notified of the diff between the call stack before backtracking and the call stack after back tacking. Note that the call_trace argument actually contains only a tiny segment of the call stack, and so can't be used for this purpose. |
You are very welcome :-)
As said above, I do agree.
I'm afraid I don't really understand what you mean. Can you elaborate a bit, on how this would be used? |
The profiler maintains it's own copy of the current call stack. Whenever you enter a new tactic, you push that call on the stack; whenever you leave a tactic, you "close" the most recent stack entry, update the children of the entry above it (because this tactic you just closed is a child, and it ran for some time), and subtract off the time spent in the child from the local time that was spent in the parent itself. The code assumes the invariant that if you wrap each tactic
Great! Current status of this PR, as I understand it: waiting for an ack from @gares / @ppedrot on the freeze function, comments from @maximedenes on if the profiling of bedrock src is satisfactory, and a response from @herbelin on whether I should stick with |
|
No special comments here, I'm OK with it. |
I see. I believe the only sane way to do that would be to keep a call stack in the monad (I believe the mechanism for Info can easily be extended for call stack). |
|
@aspiwack are you thinking of something like what we worked on for debug traces, or something else? |
|
What's the expected timeline on this? / Who's the one to give the final green-light and merge this? |
Yes. Though I assume that a call stack will be more like the Info trace (which is easy) than a debug trace (which is hard).
I'm guessing @ppedrot or @mattam82 are best placed for that. |
rocq-prover#150 (comment) ```bash git grep --name-only profileltac | xargs sed s'/profileltac/profile-ltac/g' -i ```
This is rocq-prover/rocq#150, LtacProf can't handle backtracking into multi-success tactics.
This is rocq-prover/rocq#150, LtacProf can't handle backtracking into multi-success tactics.
|
Closing; it's been merged in trunk, and will not be merged in 8.5. |
150: Add better names for `Sort`s; introduce notation. r=Janno a=Janno This commit gives longer names to the constructors of `Sort`: `Type_sort` and `Prop_sort`. To avoid having to type those names, the commit also introduces notation `Typeₛ` and `Propₛ`. On top of that, I also added some more speculative notations: `funₛ, λₛ, forallₛ, ∀ₛ, ->ₛ`. These are not used yet explicitly yet but I think they improve readability in functions that use `Fun` and `ForAll`. Note that this commit disables the `unrecognized_unicode` warning. Hopefully this will not be necessary in the future once rocq-prover#9500 is resolved. Co-authored-by: Jan-Oliver Kaiser <[email protected]>
Publish the 8.11.2 release.
With the help of Paul Steckler (and @aspiwack and others at the Coq Coding Sprint), @tebbi's LtacProf now works with Coq 8.5. The
append_end_library_hookshould possibly be done differently, but I'm not sure how it should be done. I think this is reasonable to merge into 8.5 because it shouldn't change the behavior of any existing scripts or code, and the features it adds are used primarily for debugging, and shouldn't show up in production code (so code that works with this patch shouldn't stop working in older versions). And having a profiler for Ltac is really, really useful.I can squash these commits and/or remove the trailing-spaces-stripping if desired.
(Paul is currently working on porting this to trunk.)