Support 5.3 statmemprof #22
Conversation
10fd2eb to
b261709
Compare
|
Removing the use of Obj.magic ends up being rather awkward, leaving it as is. |
b261709 to
26366bc
Compare
|
@stedolan Does it make sense to merge this? |
|
I tried this (with the 5.2 backport): Running it on my program then failed when using 7 worker domains: (code is at https://github.com/talex5/solver-service/tree/memtrace) It did record a trace successfully with 3 domains. But then, the viewer wouldn't read it: Is there something else I need to do to make this work? |
|
I also got the same assert failure when trying to view a trace created using this fork. I created an issue at janestreet/memtrace_viewer#14. It’s unclear to me whether the problem lies in this PR or in memtrace_viewer. |
|
I can reproduce both failures using OCaml 5.2.0 with This one for captures that errors when running on multiple domains: And the memtrace-viewer error: |
|
TL;DR Recording single domain traces seems fine, handling multiple domains requires a new trace file version. Reporting what I've found so far in long form. The error recording a trace file in (* fib_par.ml *)
let num_domains = try int_of_string Sys.argv.(1) with _ -> 1
let n = try int_of_string Sys.argv.(2) with _ -> 1
(* Sequential Fibonacci *)
let rec fib n =
if n < 2 then 1 else fib (n - 1) + fib (n - 2)
module T = Domainslib.Task
let rec fib_par pool n =
let _ = Buffer.create 10000 in
if n > 20 then begin
let a = T.async pool (fun _ -> fib_par pool (n-1)) in
let b = T.async pool (fun _ -> fib_par pool (n-2)) in
T.await pool a + T.await pool b
end else
(* Call sequential Fibonacci if the available work is small *)
fib n
let main () =
Memtrace.trace_if_requested ~context:"fib" ();
let pool = T.setup_pool ~num_domains:(num_domains - 1) () in
let res = T.run pool (fun _ -> fib_par pool n) in
T.teardown_pool pool;
Printf.printf "fib(%d) = %d\n" n res
let _ = main ()I can generate traces and read them successfully, but I think that is just good luck. Limiting the trace capture to a single domain usually yields a valid trace file that can be opened with The second error in |
There was indeed a bug (fixed by #23) involving the MTF cache, and having looked at @OlivierNicole's trace I think it was at play there. @stedolan will know more than me, but I imagine for multi-domain support we'll just need a proper mutex instead of this |
|
Thanks @lukemaurer I'll pick this back up and add a proper mutex (rather than using kcas) around things. |
20f586e to
3789a1f
Compare
|
3789a1f puts a single Mutex around writing the CTF file and removes the two mutable fields Running @talex5 's example above from is working. Note, since this is using a Mutex there might be more blocking occurring than you would want. The upside is it'll be consistent across 4.14 and 5.3. |
|
is there any chance for this PR to get merged and released before the final release of OCaml 5.3 planned for mid-december? It would be a great publicity for both 5.3 and memtrace if users could use them during the holiday season and be encouraged by the added value of both projects towards production grade software. |
|
ping |
|
Would appreciate if this could be merged, too. |
README.md
Outdated
| These instructions are for using statmemprof with OCaml 5.3.0+trunk | ||
|
|
||
| ``` shell | ||
| # Setup a new Blank switch | ||
| opam switch create 5.3.0~alpha1 --no-install | ||
| eval $(opam env --switch=5.3.0~alpha1 --set-switch) |
There was a problem hiding this comment.
| These instructions are for using statmemprof with OCaml 5.3.0+trunk | |
| ``` shell | |
| # Setup a new Blank switch | |
| opam switch create 5.3.0~alpha1 --no-install | |
| eval $(opam env --switch=5.3.0~alpha1 --set-switch) | |
| These instructions are for using statmemprof with OCaml >= 5.3 | |
| ``` shell | |
| # Setup a new Blank switch | |
| opam switch create ocaml --packages 'ocaml>=5.3' --no-install | |
| eval $(opam env --switch=ocaml --set-switch) |
There was a problem hiding this comment.
Thanks @MisterDA
I don't expect these instructions to be included, the intention was to show how to test this PR.
Now that 5.3 is out it should be:
$ opam switch create 5.3.0-statmemprof 5.3.0
$ eval $(opam env --switch=5.3.0-statmemprof--set-switch)
$ opam pin memtrace.dev 'https://github.com/tmcgilchrist/memtrace.git#5_1_statmemprof'
once this gets released then a simple opam install memtrace will be enough.
| (description "Generates compact traces of a program's memory use.") | ||
| (depends | ||
| (ocaml (>= 4.11.0)))) | ||
| (ocaml (or (>= 5.3.0) (and (>= 4.11.0) (< 5.0)))))) |
There was a problem hiding this comment.
Could this consider the ocaml-variants 5.2.0+statmemprof?
There was a problem hiding this comment.
I was treating that variant as an alpha release until 5.3.0 was available. I don't see much appeal in supporting that now that 5.3 is available.
|
I've happily been able to use memtrace on 5.3 with this PR. Thanks! |
src/memprof_tracer.ml
Outdated
| let n = Atomic.get bytes_before_ext_sample - bytes in | ||
| Atomic.set bytes_before_ext_sample n; |
There was a problem hiding this comment.
Here it seems more appropriate to do:
Atomic.fetch_and_add bytes_before_ext_sample (- bytes)Unless if I'm missing some detail on how ext_alloc can be used.
There was a problem hiding this comment.
I don't have much insight into how ext_alloc is being used by JaneStreet but changing this to Atomic.fetch_and_add seems safer.
src/memprof_tracer.ml
Outdated
| Atomic.set bytes_before_ext_sample | ||
| (Atomic.get bytes_before_ext_sample + draw_sampler_bytes s); |
There was a problem hiding this comment.
Here it seems safe to do Atomic.get, set but still Atomic.fetch_and_add is probably more efficient?
There was a problem hiding this comment.
Maybe since bytes_before_ext_sample can be mutated outside of the critical section (via ext_alloc) you actually should use fetch_and_add here too.
3789a1f to
1de28f8
Compare
| let samples = Atomic.make 0 in | ||
| while Atomic.get bytes_before_ext_sample <= 0 do | ||
| ignore (Atomic.fetch_and_add bytes_before_ext_sample (draw_sampler_bytes s)); | ||
| Atomic.incr samples | ||
| done; | ||
| assert (!samples > 0); | ||
| assert (Atomic.get samples > 0); | ||
| let callstack = Printexc.get_callstack max_int in | ||
| Some (Trace.Writer.put_alloc_with_raw_backtrace s.trace | ||
| (Trace.Timestamp.now ()) | ||
| ~length:size_words | ||
| ~nsamples:!samples | ||
| ~nsamples:(Atomic.get samples) |
There was a problem hiding this comment.
Seems that samples can in fact be a ref as it was originally, because it's only used locally.
8ad6a10 to
f90106c
Compare
Signed-off-by: Tim McGilchrist <[email protected]>
Signed-off-by: Tim McGilchrist <[email protected]>
Signed-off-by: Tim McGilchrist <[email protected]>
Signed-off-by: Tim McGilchrist <[email protected]>
f90106c to
e315e21
Compare
This PR makes the minimal amount of changes to support ocaml/ocaml#12923
I want to fix the
raw_backtracecode to use the new functions in ocaml/ocaml#9663