Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@frej
Copy link
Contributor

@frej frej commented Jun 27, 2023

This is a set of patches which fixes a number of errors in the compiler's alias analysis pass which have been there since the code was first merged. The final commit is unfortunately large, but I have been unable to create a smaller patch, which doesn't disable previously working and correct test cases.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2023

CT Test Results

       2 files     296 suites   14m 36s ⏱️
   778 tests    776 ✔️ 2 💤 0
4 921 runs  4 919 ✔️ 2 💤 0

Results for commit c389665.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@jhogberg jhogberg added the team:VM Assigned to OTP team VM label Jun 27, 2023
@frej frej force-pushed the frej/alias-fixes branch from 1ed37c6 to 5175353 Compare June 28, 2023 08:02
@frej
Copy link
Contributor Author

frej commented Jun 28, 2023

Tweak commit subject for 5175353.

@jhogberg jhogberg added the testing currently being tested, tag is used by OTP internal CI label Jun 29, 2023
@RobinMorisset
Copy link
Contributor

Thanks for this PR, but it introduces a compiler crash on:

f() ->
    erlang:element(ok, length(erlang:pre_loaded())).

erlc crashes with

Sub pass ssa_opt_alias
/home/rmorisset/minimized/test7984.erl: internal error in pass beam_ssa_opt:
exception error: no case clause matching #{{status,{b_var,0}} => aliased,
  {status,{b_var,1}} => unique,
  {status,{b_var,2}} => unique,
  {status,{b_var,{'@ssa_bool',4}}} => unique,
  {status,{b_var,{'@ssa_bool',6}}} => unique}
  in function  beam_ssa_alias:aa_get_element_extraction_status/2 (beam_ssa_alias.erl, line 707)
  in call from beam_ssa_alias:aa_update_annotation/3 (beam_ssa_alias.erl, line 845)
  in call from beam_ssa_alias:'-aa_update_annotation_block/3-lc$^0/1-0-'/3 (beam_ssa_alias.erl, line 836)
  in call from beam_ssa_alias:aa_update_annotation_block/3 (beam_ssa_alias.erl, line 836)
  in call from beam_ssa_alias:aa_update_annotation_blocks/4 (beam_ssa_alias.erl, line 830)
  in call from beam_ssa_alias:aa_update_fun_annotation/3 (beam_ssa_alias.erl, line 815)
  in call from beam_ssa_alias:'-aa_update_annotations/2-anonymous-1-'/3 (beam_ssa_alias.erl, line 807)
  in call from lists:foldl/3 (lists.erl, line 1594)

@frej
Copy link
Contributor Author

frej commented Jul 4, 2023

The bug found by @RobinMorisset should now be fixed.

@frej
Copy link
Contributor Author

frej commented Jul 4, 2023

The build failures are due to write /dev/stdout: no space left on device, so I'd say that this PR still qualifies for label:testing.

@RobinMorisset
Copy link
Contributor

Thanks for the quick fix!
Erlfuzz found another failing test case overnight:

f() ->
    [ 0 || _V1 <- erlang:memory(),
        { maybe
            error ?= _V1,
            ok
        end,
        maybe 
            {<<_>>} ?= _V1,
            ok
        end } ].

Without this PR it compiles fine, but with it erlc fails with:

Sub pass ssa_opt_alias
/home/rmorisset/minimized/test282060.erl: internal error in pass beam_ssa_opt:
exception error: no match of right hand side value true
  in function  beam_ssa_alias:aa_tuple_extraction/4 (beam_ssa_alias.erl, line 1176)
  in call from beam_ssa_alias:aa_is/4 (beam_ssa_alias.erl, line 456)
  in call from beam_ssa_alias:aa_blocks/3 (beam_ssa_alias.erl, line 392)
  in call from beam_ssa_alias:aa_fun/3 (beam_ssa_alias.erl, line 372)
  in call from beam_ssa_alias:aa_fixpoint/6 (beam_ssa_alias.erl, line 340)
  in call from compile:run_sub_passes_1/3 (compile.erl, line 424)
  in call from beam_ssa_opt:run_phases/3 (beam_ssa_opt.erl, line 75)
  in call from beam_ssa_opt:module/2 (beam_ssa_opt.erl, line 71)

@frej
Copy link
Contributor Author

frej commented Jul 5, 2023

Another day, another fix :)

@RobinMorisset, how much CPU-time was required to find the latest crash? I'm starting to think I should set up my own fuzzing instance. Is it worthwhile to run on a single PC overnight, or is a cluster/cloud required to get results?

@RobinMorisset
Copy link
Contributor

It took a few hours on a moderately beefy server (on par with a high-end gaming PC I'd say), so it would probably have taken a day or two of running on the average laptop.
I think that the main challenge in running erlfuzz is filtering out the duplicates of #7342 (although it got way better since some recent fixes were merged).

@frej
Copy link
Contributor Author

frej commented Jul 6, 2023

I think that the main challenge in running erlfuzz is filtering out the duplicates of #7342

Then I think I'll leave the fuzzing to you :)

@RobinMorisset
Copy link
Contributor

The fuzzer did not find any new bug on this PR for a couple of days, so it looks good to me.

frej added 6 commits July 25, 2023 13:02
When a value is extracted from a map and the map dies, the extracted
value should inherit the status of the map. If the map does not die,
both the map and the extracted value should be aliased.

This patch corrects an error in the alias analysis pass which failed
to propagate a status of 'aliased' from the map to the extracted
value. It also unifies the handling of the BIF `map_get` and the
instruction `get_map_element`. The patch also adds test-cases to check
that both the BIF and the instruction behaves the same.
When the BIF `element` was used to extract an element from a tuple,
the result is wrong and does not not match the result of using the
`get_tuple_element` instruction. This patch corrects the error by
using the same code as for the instruction to handle the BIF.
When the BIFs `hd/1` and `tl/1` are used to extract an element from a
pair, the results are wrong and do not match the result of using the
`get_hd` and `get_tl` instructions. This patch corrects the error by
using the same code as for instructions to handle the BIFs.
When elements are extracted from pairs and tuples, the extracted
element forms an alias to the element in the original aggregate. But
as long as the reference to the original is only used to extract other
elements and the original reference is not use for anything else, it
does not lead to any aliasing that is an issue for destructive updates.

This patch enhances the alias analysis to account for extracted
elements. This is done by extending `aa_get_status/2` to consider a
reference from which elements has been extracted as aliased. Likewise
`aa_set_status/3` now updates the status of all extracted elements if
the original reference becomes aliased. Correspondingly
`aa_tuple_extraction/2` and `aa_pair_extraction/4` are modified to
disregard extracted elements as long as the element being extracted
has not already been extracted.

As this enhancement will let the private-append pass explore new
execution paths which were ignored previously due to aliasing, the
value tracking is updated to handle mismatched element types.
Information about the aliasing state of call arguments used to be
accumulated in the local, per function, alias status map and then
merged into the global #aas{} when the function was done.

This patch changes the handling of the aliasing state of call
arguments to update the `#aas{}` directly. The change requires the
conditions for determining when a single iteration has led to changes,
to also include `#aas.call_args`.

Functionally this patch leads to no changes in the output of the
compiler. It is the first step in switching the alias analysis from a
per-function unified alias status map to a representation that can
account for different execution paths within the function.
The alias analysis pass uses an algorithm inspired Kotzmann and
Mössenböck's 2005 algorithm for Escape Analysis. Unfortunately the
algorithm completely fails to identify some cases of aliasing due to
they way it tracks aliasing and updates annotations.

The alias analysis is performed by traversing the functions in the
module and their code. For each operation the uniqueness and alias
status are updated and annotations are updated. The unique/aliased
status is maintained in a map which maps a variable to a either a
status or another variable. Thus constructing equivalent sets in the
same way as Kotzmann and Mössenböck. At call sites aliasing
information is fed back to the callee and the alias analysis is repeated
for the callee. For loops this works correctly, but for non-loops this
can lead to bad aliased/unique verdicts. Consider the following
example:

1: tuple_element_aliasing() ->
2:    T = make_empty_binary_tuple(),
3:    X = element(1, T),
4:    Z = <<X/binary,1:8>>,
5:    {Z, T}.

6: make_empty_binary_tuple() ->
7:    {<<>>, <<>>}.

At line 4, the algorithm will mark X as unique as there is no feedback
from line 5 where a reference to the tuple from which X is extracted
is captured in the newly constructed tuple.

The obvious solution of waiting to update alias/unique annotations
until all basic blocks of a function have been processed, avoids the
error described above, but also disables many private-append
optimizations where the broken algorithm was correct. The main reason
for this, is that the algorithm does not consider flow-control
relationships between blocks in a function, to it, all blocks are
executed even if flow control guarantees that only one of two blocks
will ever be executed.

This patch greatly strengthens the analysis done by aliasing analysis
by accounting for flow-control within functions. It also changes the
data structure used to describe the alias status of variables in order
to support incorporating flow-control information in the per function
analysis.

In short, this patch does two things, neither of which can be done in
isolation without breaking the test suite:

1) The data structure used to describe the alias status of variables,
   called a Sharing State, is changed to track three aspects:

    - A variable is either unique, aliased.

    - A unique variable can be derived, that is extracted from a term
      or constructed from other variables.

    - The relationship between the source and a derived value is
      tracked so that it is possible to find the parent of a derived
      value and likewise find all values that are derived from a
      value.

2) The basic blocks of a function are now traversed reverse post
   order. When the end of a block is reached, the current sharing
   state is propagated to the block's successors by merging the
   sharing state of all the successor block's predecessors' sharing
   states.

   When all blocks have been visited and the sharing states at the end
   of each block are known, information about aliased variables are
   propagated along the edges of the execution graph during a post
   order traversal of the basic blocks.

compiler: Change data structure describing the alias status of variables

Change the data structure describing the alias status of
variables. Instead of a map, where different keys represented
different aspects of a variable, this patch maps a variable to a
single record describing all aspects of the variable. The change
greatly speeds up the alias analysis by reducing the number of visited
entries when merging two alias states while also reducing the
complexity of the merging logic.
@frej frej force-pushed the frej/alias-fixes branch from 9b1bf68 to c389665 Compare July 25, 2023 12:08
@frej
Copy link
Contributor Author

frej commented Jul 25, 2023

Refreshed the patch onto today's maint. It includes a complete rewrite of the top "compiler: Correct errors in alias analysis pass" commit. The new version has better performance, is easier to understand and comes with asserts checking its invariants. In short, it is what the first version should have been :)

@jhogberg jhogberg merged commit 6022f86 into erlang:maint Jul 31, 2023
@frej frej deleted the frej/alias-fixes branch July 31, 2023 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team:VM Assigned to OTP team VM testing currently being tested, tag is used by OTP internal CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants