Allow using a fresh solver in each engine #54
Conversation
… on term translation
There was a problem hiding this comment.
I completed my pass over the changes and left some comments.
@makaimann , checking equivalence between an original and copied transition systems using a solver is a great idea and very useful for testing. I was wondering whether we could run a test like the one implemented in file tests/test_btor2_ts_copy_equal.cpp on the cluster on all the benchmarks we have, using a large time out.
| bool no_next(const smt::Term & term) const; | ||
|
|
||
| // term building functionality -- forwards to the underlying SmtSolver | ||
| // assumes all Terms/Sorts belong to solver_ |
There was a problem hiding this comment.
Do we check whether the terms/sorts that are forwarded belong to solver_, i.e., is there some assertion checking on the smt-switch level? Or would Pono crash in that case? And would it be possible to check that on the Pono level also?
There was a problem hiding this comment.
We don't check this in smt-switch yet, but that's something I've been wondering if we should check. It might be nice to discuss this further. There's not an easy way to do this at the Pono level either. Terms/Sorts currently don't have a reference to the solver. So we could either add a reference to the solver in smt-switch or have a special function that needs to be implemented which allows you to check.
| { | ||
| function<Term(const Term &)> transfer; | ||
| function<Term(const Term &, SortKind)> transfer_as; | ||
| if (other_ts.solver() == tt.get_solver()) { |
There was a problem hiding this comment.
A general question: if I am not mistaken, then operator == applied to shared_ptr of two solvers checks for equivalence of the object pointers stored in the two shared_ptrs. Suppose other_ts.solver() and 'tt.get_solver()' are both Boolector solvers but different solver objects. In that case, the comparison by == would return false and we enter the "then"-branch, where terms are transferred (Boolector terms to Boolector terms). Is that the intended behavior?
There was a problem hiding this comment.
Yep, that's the intended behavior! Underlying solvers aren't expected to be able to share terms between different instances, even if they're the same type of solver.
| smt::SmtSolver solver_; | ||
| smt::TermTranslator to_prover_solver_; | ||
| const Property property_; | ||
| const TransitionSystem & |
There was a problem hiding this comment.
Would it also be possible to just access that transition system by accessing it via property_?
There was a problem hiding this comment.
Yeah definitely, this is just for convenience. We could remove it if you prefer?
|
|
||
| // getters | ||
| smt::SmtSolver & solver() { return solver_; }; | ||
| const smt::SmtSolver & solver() const { return solver_; }; |
There was a problem hiding this comment.
The const at the beginning of the line is necessary for the copy constructor, I guess?
There was a problem hiding this comment.
Yeah that's exactly right.
|
All right, I addressed all the comments by making changes or responding I think! There are a few things that aren't really resolved, but I'm thinking they could be addressed in a separate PR/discussion. In particular, I think it would be good to discuss how to best check that terms/sorts match the solver. The debug versions of the jobs also finished, and the results are here. Looks good to me. |
|
Hey @lonsing and @ahmed-irfan -- I just updated this branch to keep it inline with master. Sorry I forgot -- where were we on this again? Do you think it's ready to merge or should we make other changes? |
|
Hi @makaimann, this looks good to me! Some points that we wanted to discuss after merging:
|
|
Great, thanks! Just updated it for changes in master. Everything passed so I'll merge it now. |
This PR adds features to support checking multiple properties on the same transition system. Instead of using the same solver everywhere, it adds a TermTranslator that moves terms to a fresh solver. This was implemented as a type of specialized copy constructor in TransitionSystem and Property. For this change to work, some classes that did not previously "own" a solver now need to own it (i.e. use a shared_ptr directly instead of a reference to a shared_ptr). It took me a little while to get this right, especially with const-ness. The default solver for a TransitionSystem is CVC4 because it does not rewrite on the fly and supports the most theories. This also adds a new test that encodes several btor2 examples, copies them between TransitionSystems with different SmtSolvers and tests that the resulting TransitionSystems are equivalent. Unfortunately, we cannot rely on structural/syntactic equivalence because the TermTranslator object has to do some casting to handle sort-aliasing when transferring to/from Boolector. Thus, it uses a solver to check that the system is equivalent after being copied to a different solver and then copied back to the original. For the largest system (ridecore.btor) it only does this check with Boolector because the other solvers take way too long. If it ends up being too slow, we can disable that test for ride core. Finally, the binary is essentially unchanged at this time -- it uses the same solver to construct the transition system and solve to avoid the copying step. In the future, we can make a PR where the binary uses a fresh solver as well (so it can solve multiple properties).
This PR adds features to support checking multiple properties on the same transition system. Instead of using the same solver everywhere, it adds a TermTranslator that moves terms to a fresh solver. This was implemented as a type of specialized copy constructor in
TransitionSystemandProperty. For this change to work, some classes that did not previously "own" a solver now need to own it (i.e. use a shared_ptr directly instead of a reference to a shared_ptr). It took me a little while to get this right, especially withconst-ness. The default solver for aTransitionSystemis CVC4 because it does not rewrite on the fly and supports the most theories.I added a new test that encodes several
btor2examples, copies them betweenTransitionSystemswith differentSmtSolversand tests that the resultingTransitionSystemsare equivalent. Unfortunately, we cannot rely on structural/syntactic equivalence because theTermTranslatorobject has to do some casting to handle sort-aliasing when transferring to/from Boolector. Thus, I actually use a solver to check that the system is equivalent after being copied to a different solver and then copied back to the original. For the largest system (ridecore.btor) I only do this check with Boolector because the other solvers take way too long. If you think the tests are still too slow, I can remove that check.Finally, the binary is essentially unchanged at this time because I left the solver instantiation the same. In the future, we can make a PR where the binary uses a fresh solver as well (so it can solve multiple properties). However, I would like to test this on the cluster first. I'll create a temporary version that uses a fresh solver in this PR and run it on benchmarks on the cluster to look for bugs and compare runtime. This might take a little while to complete so I'll mark this do-not-merge in the meantime, but you can start reviewing whenever you have time. Thanks!