Implemented ASR checks for function dependencies #2167

arteevraina · 2023-07-17T12:22:30Z

Towards: #2006

arteevraina · 2023-07-17T12:23:23Z

src/libasr/asr_verify.cpp

@@ -440,6 +440,20 @@ class VerifyVisitor : public BaseWalkVisitor<VerifyVisitor>
        verify_unique_dependencies(x.m_dependencies, x.n_dependencies,
                                   x.m_name, x.base.base.loc);

+        // Dependencies of the function should be from same symbol table.


@certik Do you think this logic is on the right track for checking the that dependencies & function should be from same symbol table ?

certik · 2023-07-20T00:30:36Z

src/libasr/asr_verify.cpp

+
+            require(dep_sym != nullptr,
+                    "Dependency " + found_dep +  " does not belong to same parent symbol table of " + std::string(x.m_name));
+        }


I am trying to understand the logic here.

Are we ensuring that all dependencies are from the same parent symbol table?

It seems this logic is testing that all the symbols (except ExternalSymbol) are NOT from the parent symbol table. Tests pass, so I guess this is the correct logic, but I don't understand it. Once I understand it, let's document this as a comment.

src/libasr/asr_verify.cpp

arteevraina · 2023-07-23T17:14:32Z

ASR verify fails in nested_vars pass when trying to generate the LLVM. But, run perfectly fine when trying to generate only ASR using --show-asr flag.

Here is the stacktrace:

ASR Pass starts: 'nested_vars'
ASR verify pass error: ASR verify: Dependency _lpython_floordiv@__lpython_overloaded_6___lpython_floordiv was not found in the parent symbol table main
Internal Compiler Error: Unhandled exception
Traceback (most recent call last):
  File "/Users/arteevraina/GSI/lpython/src/bin/lpython.cpp", line 1872
    err = compile_python_to_object_file(arg_file, tmp_o, runtime_library_dir,
  File "/Users/arteevraina/GSI/lpython/src/bin/lpython.cpp", line 827
    res = fe.get_llvm3(*asr, pass_manager, diagnostics, infile);
  File "/Users/arteevraina/GSI/lpython/src/lpython/python_evaluator.cpp", line 58
    run_fn, infile);
  File "/Users/arteevraina/GSI/lpython/src/libasr/codegen/asr_to_llvm.cpp", line 8227
    pass_manager.apply_passes(al, &asr, pass_options, diagnostics);
  File "/Users/arteevraina/GSI/lpython/src/libasr/pass/pass_manager.h", line 285
    apply_passes(al, asr, _passes, pass_options, diagnostics);
  File "/Users/arteevraina/GSI/lpython/src/libasr/pass/pass_manager.h", line 156
    throw LCompilersException("Verify failed in the pass: "
LCompilersException: Verify failed in the pass: nested_vars

cc: @certik @czgdp1807

…symtab

czgdp1807

I think these changes should be simultaneously applied to LFortran as well. And then both the PRs should be merged together.

arteevraina · 2023-07-28T06:36:16Z

I think these changes should be simultaneously applied to LFortran as well. And then both the PRs should be merged together.

Sure. I would be happy to do that.

I think there are still some things left to correct in this PR.

As, few tests like https://github.com/lcompilers/lpython/blob/v0.19.0/integration_tests/callback_01.py are still having dependencies that are not from same parent symbol table. As, in this one, g depends on func which is not in the same parent symtab as of g.

Can this be due to some of the checks that I implemented in this PR actually fail or am I missing some places to implement the same checks ? @czgdp1807

(lp) arteevraina@Arteevs-MacBook-Air lpython % src/bin/lpython ./integration_tests/callback_01.py --show-asr
(TranslationUnit
    (SymbolTable
        1
        {
            __main__:
                (Module
                    (SymbolTable
                        2
                        {
                            __main____global_statements:
                                (Function
                                    (SymbolTable
                                        9
                                        {
                                            
                                        })
                                    __main____global_statements
                                    (FunctionType
                                        []
                                        ()
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    [check]
                                    []
                                    [(SubroutineCall
                                        2 check
                                        ()
                                        []
                                        ()
                                    )]
                                    ()
                                    Public
                                    .false.
                                    .false.
                                    ()
                                ),
                            check:
                                (Function
                                    (SymbolTable
                                        8
                                        {
                                            
                                        })
                                    check
                                    (FunctionType
                                        []
                                        ()
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    [g]
                                    []
                                    [(Assert
                                        (IntegerCompare
                                            (FunctionCall
                                                2 g
                                                ()
                                                [((Var 2 f))
                                                ((IntegerConstant 10 (Integer 4)))]
                                                (Integer 4)
                                                ()
                                                ()
                                            )
                                            Eq
                                            (IntegerConstant 11 (Integer 4))
                                            (Logical 4)
                                            ()
                                        )
                                        ()
                                    )
                                    (Assert
                                        (IntegerCompare
                                            (FunctionCall
                                                2 g
                                                ()
                                                [((Var 2 f2))
                                                ((IntegerConstant 20 (Integer 4)))]
                                                (Integer 4)
                                                ()
                                                ()
                                            )
                                            Eq
                                            (IntegerConstant 30 (Integer 4))
                                            (Logical 4)
                                            ()
                                        )
                                        ()
                                    )
                                    (Assert
                                        (IntegerCompare
                                            (FunctionCall
                                                2 g
                                                ()
                                                [((Var 2 f3))
                                                ((IntegerConstant 5 (Integer 4)))]
                                                (Integer 4)
                                                ()
                                                ()
                                            )
                                            Eq
                                            (IntegerConstant 21 (Integer 4))
                                            (Logical 4)
                                            ()
                                        )
                                        ()
                                    )]
                                    ()
                                    Public
                                    .false.
                                    .false.
                                    ()
                                ),
                            f:
                                (Function
                                    (SymbolTable
                                        3
                                        {
                                            _lpython_return_variable:
                                                (Variable
                                                    3
                                                    _lpython_return_variable
                                                    []
                                                    ReturnVar
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                ),
                                            x:
                                                (Variable
                                                    3
                                                    x
                                                    []
                                                    In
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                )
                                        })
                                    f
                                    (FunctionType
                                        [(Integer 4)]
                                        (Integer 4)
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    []
                                    [(Var 3 x)]
                                    [(=
                                        (Var 3 _lpython_return_variable)
                                        (IntegerBinOp
                                            (Var 3 x)
                                            Add
                                            (IntegerConstant 1 (Integer 4))
                                            (Integer 4)
                                            ()
                                        )
                                        ()
                                    )
                                    (Return)]
                                    (Var 3 _lpython_return_variable)
                                    Public
                                    .false.
                                    .false.
                                    ()
                                ),
                            f2:
                                (Function
                                    (SymbolTable
                                        4
                                        {
                                            _lpython_return_variable:
                                                (Variable
                                                    4
                                                    _lpython_return_variable
                                                    []
                                                    ReturnVar
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                ),
                                            x:
                                                (Variable
                                                    4
                                                    x
                                                    []
                                                    In
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                )
                                        })
                                    f2
                                    (FunctionType
                                        [(Integer 4)]
                                        (Integer 4)
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    []
                                    [(Var 4 x)]
                                    [(=
                                        (Var 4 _lpython_return_variable)
                                        (IntegerBinOp
                                            (Var 4 x)
                                            Add
                                            (IntegerConstant 10 (Integer 4))
                                            (Integer 4)
                                            ()
                                        )
                                        ()
                                    )
                                    (Return)]
                                    (Var 4 _lpython_return_variable)
                                    Public
                                    .false.
                                    .false.
                                    ()
                                ),
                            f3:
                                (Function
                                    (SymbolTable
                                        5
                                        {
                                            _lpython_return_variable:
                                                (Variable
                                                    5
                                                    _lpython_return_variable
                                                    []
                                                    ReturnVar
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                ),
                                            x:
                                                (Variable
                                                    5
                                                    x
                                                    []
                                                    In
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                )
                                        })
                                    f3
                                    (FunctionType
                                        [(Integer 4)]
                                        (Integer 4)
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    [f
                                    f2]
                                    [(Var 5 x)]
                                    [(=
                                        (Var 5 _lpython_return_variable)
                                        (IntegerBinOp
                                            (FunctionCall
                                                2 f
                                                ()
                                                [((Var 5 x))]
                                                (Integer 4)
                                                ()
                                                ()
                                            )
                                            Add
                                            (FunctionCall
                                                2 f2
                                                ()
                                                [((Var 5 x))]
                                                (Integer 4)
                                                ()
                                                ()
                                            )
                                            (Integer 4)
                                            ()
                                        )
                                        ()
                                    )
                                    (Return)]
                                    (Var 5 _lpython_return_variable)
                                    Public
                                    .false.
                                    .false.
                                    ()
                                ),
                            g:
                                (Function
                                    (SymbolTable
                                        6
                                        {
                                            _lpython_return_variable:
                                                (Variable
                                                    6
                                                    _lpython_return_variable
                                                    []
                                                    ReturnVar
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                ),
                                            arg:
                                                (Variable
                                                    6
                                                    arg
                                                    []
                                                    In
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                ),
                                            func:
                                                (Function
                                                    (SymbolTable
                                                        7
                                                        {
                                                            func_arg_0:
                                                                (Variable
                                                                    7
                                                                    func_arg_0
                                                                    []
                                                                    Unspecified
                                                                    ()
                                                                    ()
                                                                    Default
                                                                    (Integer 4)
                                                                    ()
                                                                    Source
                                                                    Public
                                                                    Required
                                                                    .false.
                                                                ),
                                                            func_return_var_name:
                                                                (Variable
                                                                    7
                                                                    func_return_var_name
                                                                    []
                                                                    ReturnVar
                                                                    ()
                                                                    ()
                                                                    Default
                                                                    (Integer 4)
                                                                    ()
                                                                    Source
                                                                    Public
                                                                    Required
                                                                    .false.
                                                                )
                                                        })
                                                    func
                                                    (FunctionType
                                                        [(Integer 4)]
                                                        (Integer 4)
                                                        BindC
                                                        Interface
                                                        ()
                                                        .false.
                                                        .false.
                                                        .false.
                                                        .false.
                                                        .false.
                                                        []
                                                        []
                                                        .false.
                                                    )
                                                    []
                                                    [(Var 7 func_arg_0)]
                                                    []
                                                    (Var 7 func_return_var_name)
                                                    Public
                                                    .false.
                                                    .false.
                                                    ()
                                                ),
                                            ret:
                                                (Variable
                                                    6
                                                    ret
                                                    []
                                                    Local
                                                    ()
                                                    ()
                                                    Default
                                                    (Integer 4)
                                                    ()
                                                    Source
                                                    Public
                                                    Required
                                                    .false.
                                                )
                                        })
                                    g
                                    (FunctionType
                                        [(FunctionType
                                            [(Integer 4)]
                                            (Integer 4)
                                            BindC
                                            Interface
                                            ()
                                            .false.
                                            .false.
                                            .false.
                                            .false.
                                            .false.
                                            []
                                            []
                                            .false.
                                        )
                                        (Integer 4)]
                                        (Integer 4)
                                        Source
                                        Implementation
                                        ()
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        .false.
                                        []
                                        []
                                        .false.
                                    )
                                    [func]
                                    [(Var 6 func)
                                    (Var 6 arg)]
                                    [(=
                                        (Var 6 ret)
                                        (FunctionCall
                                            6 func
                                            ()
                                            [((Var 6 arg))]
                                            (Integer 4)
                                            ()
                                            ()
                                        )
                                        ()
                                    )
                                    (=
                                        (Var 6 _lpython_return_variable)
                                        (Var 6 ret)
                                        ()
                                    )
                                    (Return)]
                                    (Var 6 _lpython_return_variable)
                                    Public
                                    .false.
                                    .false.
                                    ()
                                )
                        })
                    __main__
                    []
                    .false.
                    .false.
                ),
            main_program:
                (Program
                    (SymbolTable
                        10
                        {
                            __main____global_statements:
                                (ExternalSymbol
                                    10
                                    __main____global_statements
                                    2 __main____global_statements
                                    __main__
                                    []
                                    __main____global_statements
                                    Public
                                )
                        })
                    main_program
                    [__main__]
                    [(SubroutineCall
                        10 __main____global_statements
                        2 __main____global_statements
                        []
                        ()
                    )]
                )
        })
    []
)

czgdp1807 · 2023-07-28T09:20:43Z

Now there are two things that can be done,

func shouldn't be present inside the dependency list of g.
Define func outside the symbol table of g.

Second option makes more sense to me. Defining func inside g doesn't make sense because the actual function callbacks which are passed to g calls are defined outside g. So placement of func inside g's symbol table is problematic to me. Also, in the backends, for defining g, first func should be declared, and that is not possible if you put it inside g's symbol table.

czgdp1807 · 2023-08-03T04:56:09Z

@arteevraina Will you be working on this? Please let us know the status. I am marking it as draft for now. However whenever you are done, please mark it ready for review. Thanks for your work.

arteevraina · 2023-08-03T04:59:14Z

@arteevraina Will you be working on this? Please let us know the status. I am marking it as draft for now. However whenever you are done, please mark it ready for review. Thanks for your work.

Yes, I will be continuing my work on this. I was away for few days due to some work but now I will continue working on this.

src/libasr/pass/pass_utils.h

… current scope

src/libasr/pass/pass_utils.h

certik · 2023-08-30T04:06:27Z

I am still reviewing this. Before we can merge, can you please also send exactly the same PR against LFortran? We have to see that it passes all the tests there as well. @czgdp1807 do you see any issues with this PR?

arteevraina · 2023-08-30T04:17:38Z

I am still reviewing this. Before we can merge, can you please also send exactly the same PR against LFortran? We have to see that it passes all the tests there as well. @czgdp1807 do you see any issues with this PR?

Sure @certik , I will do that meanwhile.

src/libasr/pass/pass_utils.h

arteevraina · 2023-09-02T13:15:59Z

@czgdp1807 I have resolved all of the comments on this PR.

certik · 2023-09-02T13:52:09Z

@arteevraina can you please submit a PR against LFortran as well? Let's see if things work there.

arteevraina · 2023-09-02T13:53:24Z

@arteevraina can you please submit a PR against LFortran as well? Let's see if things work there.

Sure @certik , I am on it.

arteevraina · 2023-09-02T15:16:55Z

Regarding the lfortran PR, I replicated the changes in the lfortran for libasr folder as it is and there were some dependency checks required in ast_common_visitor class as it is unique to lfortran.

After applying changes, I got some failures in integration tests, need to debug that. Will create the PR after fixing the tests locally.

certik · 2023-09-02T17:13:47Z

Perfect! You can submit a PR to LFortran even if it doesn't work yet. I expected there would be some fixes needed, so we can just work on it iteratively.

arteevraina · 2023-09-03T15:46:58Z

PR -> lfortran/lfortran#2327

A total of 15 test cases are failing in lfortran in this PR. I am working to resolve the issues with them one by one. @certik

certik · 2023-10-24T11:33:11Z

@czgdp1807 can you please review this?

ubaidsk · 2023-10-24T14:50:07Z

tests/reference/asr-array_01_decl-39cf894.stderr

+warning: The module 'numpy' located in $DIR/src/bin/../runtime/lpython_intrinsic_numpy.py cannot be loaded
+ --> tests/../integration_tests/array_01_decl.py:2:1
+  |
+2 | from numpy import empty, int16, int32, int64, float32, float64, complex64, complex128
+  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ imported here
+


It seems there are several errors like above that got committed in this PR. Are these reference errors expected or is it a bug?

ubaidsk · 2023-10-24T14:51:24Z

src/libasr/asr_verify.cpp

+            ASR::symbol_t* dep_sym = x_symtab->resolve_symbol(found_dep);
+
+            require(dep_sym != nullptr,
+                            "Dependency " + found_dep +  " is inside symbol table " + std::string(x.m_name));


I think this should be "Dependency " + found_dep + " is not* inside symbol table "

correct, "not inside".

Yes, you both are right.

I assumed the resolve_symbol checks in parent scope only and not in the current_scope. Now that i checked it's documentation, it checks both in current & parent scopes.

The check should will be updated like this :
SymbolTable *x_parent_symtab = x.m_symtab->parent; instead of SymbolTable* x_symtab = x.m_symtab;
and then "inside" phrase will make sense here.

This was committed by a mistake in lcompilers#2167.

arteevraina commented Jul 17, 2023

View reviewed changes

arteevraina added 8 commits July 19, 2023 22:57

fix: check dependencies are from same symbol table

049fbc6

fix: remove print statement

fc86d98

fix: update verify logic

42d5146

fix: update logic for verification of dependencies

4805b69

refactor: remove cout

71bee66

fix: get the parent symtab of x

70f2f9c

fix: skip the symbols which are nullptr or of type External Symbol

a2e802b

fix: update cmakelists

8d8bbc7

arteevraina force-pushed the checks-for-same-symtab-dependencies branch from bead745 to 8d8bbc7 Compare July 19, 2023 17:28

arteevraina marked this pull request as ready for review July 19, 2023 17:55

certik reviewed Jul 20, 2023

View reviewed changes

czgdp1807 reviewed Jul 20, 2023

View reviewed changes

src/libasr/asr_verify.cpp Outdated Show resolved Hide resolved

fix: check while adding dependencies to make sure they are from same …

c1116e3

…symtab

czgdp1807 reviewed Jul 28, 2023

View reviewed changes

czgdp1807 marked this pull request as draft August 3, 2023 04:56

arteevraina and others added 4 commits August 11, 2023 21:36

fix: asr verify passes for callback_01 test

b531009

fix: tests relating to external symbols and function dependencies

2c436ba

fix: condition for checking intrinsic function

6fc8a6d

Allow global scope dependencies as well

f1d77ae

arteevraina commented Aug 21, 2023

View reviewed changes

src/libasr/pass/pass_utils.h Outdated Show resolved Hide resolved

czgdp1807 added 2 commits August 25, 2023 10:37

Merge main

f94840f

Robust check for making sure that dependency isn't defined inside the…

47d0f5d

… current scope

arteevraina commented Aug 25, 2023

View reviewed changes

src/libasr/pass/pass_utils.h Outdated Show resolved Hide resolved

fix: add robust checking using resolve symbol when symbol is nested

456071b

czgdp1807 requested changes Aug 31, 2023

View reviewed changes

src/libasr/pass/pass_utils.h Show resolved Hide resolved

src/libasr/pass/pass_utils.h Outdated Show resolved Hide resolved

refactor: function call & subroutine call dependencies

f1e1c2f

certik requested a review from czgdp1807 September 2, 2023 13:51

arteevraina mentioned this pull request Sep 3, 2023

feat: init checks for dependency from current scope implemented lfortran/lfortran#2327

Merged

arteevraina added 8 commits October 4, 2023 16:08

feat: sync libasr

261c187

fix: update branch

6f7c8fa

dev: udpate branch

ccaa3ec

fix: updated all references

f68f71a

dev: sync libasr with lfortran

0c7e8b5

test: update references

74e4251

tests: update references

1801bb5

fix: set asr_owner when visiting For

895f411

czgdp1807 approved these changes Oct 24, 2023

View reviewed changes

certik merged commit 800c657 into lcompilers:main Oct 24, 2023

ubaidsk reviewed Oct 24, 2023

View reviewed changes

certik added a commit to certik/lpython that referenced this pull request Oct 24, 2023

Remove spurious reference results

bcdb732

This was committed by a mistake in lcompilers#2167.

certik mentioned this pull request Oct 24, 2023

Remove spurious reference results #2391

Merged

ubaidsk mentioned this pull request Oct 24, 2023

Incorrect Error message in ASRVerfiy #2392

Closed

Agent-Hellboy pushed a commit to Agent-Hellboy/lpython that referenced this pull request Mar 5, 2024

Implemented ASR checks for function dependencies (lcompilers#2167)

293e12b

Agent-Hellboy pushed a commit to Agent-Hellboy/lpython that referenced this pull request Mar 5, 2024

Remove spurious reference results

7cd0165

This was committed by a mistake in lcompilers#2167.

Implemented ASR checks for function dependencies #2167

Implemented ASR checks for function dependencies #2167

Uh oh!

Conversation

arteevraina commented Jul 17, 2023 • edited by certik Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arteevraina Jul 17, 2023

Choose a reason for hiding this comment

Uh oh!

certik Jul 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arteevraina commented Jul 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

czgdp1807 left a comment

Choose a reason for hiding this comment

Uh oh!

arteevraina commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

czgdp1807 commented Jul 28, 2023

Uh oh!

czgdp1807 commented Aug 3, 2023

Uh oh!

arteevraina commented Aug 3, 2023

Uh oh!

Uh oh!

Uh oh!

certik commented Aug 30, 2023

Uh oh!

arteevraina commented Aug 30, 2023

Uh oh!

Uh oh!

Uh oh!

arteevraina commented Sep 2, 2023

Uh oh!

certik commented Sep 2, 2023

Uh oh!

arteevraina commented Sep 2, 2023

Uh oh!

arteevraina commented Sep 2, 2023

Uh oh!

certik commented Sep 2, 2023

Uh oh!

arteevraina commented Sep 3, 2023

Uh oh!

certik commented Oct 24, 2023

Uh oh!

ubaidsk Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

ubaidsk Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

certik Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

arteevraina Oct 25, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arteevraina commented Jul 17, 2023 •

edited by certik

Loading

arteevraina commented Jul 23, 2023 •

edited

Loading

arteevraina commented Jul 28, 2023 •

edited

Loading