compilation time #1912

sreenivasan-eaglegenomics · 2022-10-15T18:58:19Z

sreenivasan-eaglegenomics
Oct 15, 2022

Hi Team,

Is there a way to reduce the compilation time if the constraints consists of a matrix of large size say in 10,000.

GillesFa · 2023-05-19T01:26:07Z

GillesFa
May 19, 2023

Hi Guys,
We have got a similar issue with compilation time taking too long. This is CVXpy console prints:

===============================================================================
CVXPY
v1.3.1

(CVXPY) May 19 10:56:26 AM: Your problem has 103920 variables, 196165 constraints, and 0 parameters.
(CVXPY) May 19 10:56:56 AM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) May 19 10:56:56 AM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) May 19 10:56:56 AM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.

                              Compilation

(CVXPY) May 19 10:57:09 AM: Compiling problem (target solver=GUROBI).
(CVXPY) May 19 10:57:09 AM: Reduction chain: EvalParams -> CvxAttr2Constr -> Qp2SymbolicQp -> QpMatrixStuffing -> GUROBI
(CVXPY) May 19 10:57:09 AM: Applying reduction EvalParams
(CVXPY) May 19 10:57:16 AM: Applying reduction CvxAttr2Constr
(CVXPY) May 19 10:58:10 AM: Applying reduction Qp2SymbolicQp
(CVXPY) May 19 10:59:28 AM: Applying reduction QpMatrixStuffing
(CVXPY) May 19 11:19:54 AM: Applying reduction GUROBI
(CVXPY) May 19 11:19:56 AM: Finished problem compilation (took 1.380e+03 seconds).

Most of the time is taken by CvxAttr2Constr, Qp2SymbolicQp and especially QpMatrixStuffing. We tried several options (ignore_dpp = True, canon_backend=cp.SCIPY_CANON_BACKEND) but the compilation time does not change.

Please could you advise what we can do to speed up the compilation time or if you required additional information.

Thanks
Gilles

2 replies

SteveDiamond May 19, 2023
Maintainer

Can you share your problem with us?

anttimc May 23, 2023

I had a similar issue and profiled it a bit.

This call seems to be the bottleneck

cvxpy/cvxpy/cvxcore/python/cvxcore.py

Line 1006 in 2a135d2

return _cvxcore.build_matrix(*args)

QpMatrixStuffing part takes a long time in this matrix building phase.

GillesFa · 2023-05-19T03:46:19Z

GillesFa
May 19, 2023

Hi Steven, Do you mean sharing our python code? Or a dump of the constraints and objective will be sufficient? I will need to ask to the Uni if it is ok. Thanks Gilles Faure | Software Engineer Rio Tinto Centre for Mine Automation Sydney Robotics Hub J18 8 Little Queen Street, Chippendale, 2008 NSW www.acfr.usyd.edu.au<http://www.acfr.usyd.edu.au> From: Steven Diamond ***@***.***> Sent: Friday, 19 May 2023 1:26 PM To: cvxpy/cvxpy ***@***.***> Cc: Gilles Faure ***@***.***>; Comment ***@***.***> Subject: Re: [cvxpy/cvxpy] compilation time (Discussion #1912) Can you share your problem with us? — Reply to this email directly, view it on GitHub<#1912 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZXIO2UMM6AXAZKTYK2LUVDXG3R4BANCNFSM6AAAAAARGAUZ54>. You are receiving this because you commented.Message ID: ***@***.******@***.***>>

0 replies

GillesFa · 2023-05-23T00:15:26Z

GillesFa
May 23, 2023

Hi,
I created a minimal script.
Run the script from the folder "\MAST" as follow: python CVXPY_main.py
Let me know if you need something else
thanks
Gilles
for_cvxpy.zip

6 replies

Transurgeon Jun 1, 2023
Collaborator

@GillesFa Is there anyway to condense your example? It is quite difficult to analyse with the abundance of variables and for loops. A minimal example should suffice.. ie. a problem that suffers the same compilation time slowdowns you mention but maybe not on such a large scale.

chamanbaz Jun 4, 2023

@Transurgeon I am Gilles coleague. The model is an MILP developed for unit commitment. The performance is indeed slow for the example sent by Gilles, as clear from the logs, most of the time is spent in CvxAttr2Constr, Qp2SymbolicQp and especially QpMatrixStuffing. Do you reformulate the problem as QP even if it is LP? At least from the names, it seems to me that the problem is treated as a QP.
I am not sure what you mean by condensing the example. Do you mean removing constraints from the model, or do you mean generating the model for a simpler network configuration that results in a smaller number of constraints and decision variables? This is a pretty standard unit commitment model. It was initially coded in Matlab and AMPL and we just ported it to Python. There is no computational issue with the Matlab version and both Matlab and CVXPY converge to the same solution (both are using Gurobi as the solver).

SteveDiamond Jun 4, 2023
Maintainer

LPs are normally canonicalized through the conic pathway, but for Gurobi the QP pathway/interface is used under the advice of the Gurubi devs. One major difference between cvxpy and AMPL is in cvxpy you are supposed to vectorize your code. GPT-4 can do this for you.

chamanbaz Jun 4, 2023

@SteveDiamond - Thanks for your reply. Then, does it mean the long compilation time is the expected behaviour for non-vectorised models and the only way to reduce the time is to vectorise the MILP model?

SteveDiamond Jun 5, 2023
Maintainer

In the long run both vectorized and non-vectorized models will be fast to compile, but for the moment vectorization will be dramatically faster.

phschiele · 2023-06-08T19:22:43Z

phschiele
Jun 8, 2023
Maintainer

@mrchamanbaz @GillesFa Just following up here, we are working on speeding up the compilation over the summer. We can test the code you provided to see how much faster it got.
Additionally, we also collect problems that we continuously monitor, but for these, we generally look for problems that have a very short description length.

1 reply

chamanbaz Jun 15, 2023

@phschiele Thanks for following up.
The model that we sent you is pretty standard. If I reduce it by removing constraints, it will be incomplete and I am not sure if we get the same compilation problem. What we can do is to create the model for a smaller network; meaning that instead of e.g. X nodes in the network there are Y<<X nodes.

nlaws-camus · 2024-12-19T20:18:36Z

nlaws-camus
Dec 19, 2024

I have the same problem with the compilation time in QpMatrixStuffing, which takes 593 seconds -- while Gurobi takes 5 seconds to solve the problem (which I cannot share unfortunately). It's not a very big problem and is an LP.

===============================================================================
                                     CVXPY                                     
                                     v1.6.0                                    
===============================================================================
(CVXPY) Dec 19 08:04:30 PM: Your problem has 6424 variables, 7477 constraints, and 0 parameters.
(CVXPY) Dec 19 08:04:30 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Dec 19 08:04:30 PM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Dec 19 08:04:30 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Dec 19 08:04:30 PM: Your problem is compiled with the CPP canonicalization backend.
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
(CVXPY) Dec 19 08:04:30 PM: Compiling problem (target solver=GUROBI).
(CVXPY) Dec 19 08:04:30 PM: Reduction chain: Complex2Real -> FlipObjective -> CvxAttr2Constr -> Qp2SymbolicQp -> QpMatrixStuffing -> GUROBI
(CVXPY) Dec 19 08:04:30 PM: Applying reduction Complex2Real
(CVXPY) Dec 19 08:04:31 PM: Applying reduction FlipObjective
(CVXPY) Dec 19 08:04:31 PM: Applying reduction CvxAttr2Constr
(CVXPY) Dec 19 08:04:31 PM: Applying reduction Qp2SymbolicQp
(CVXPY) Dec 19 08:04:32 PM: Applying reduction QpMatrixStuffing
(CVXPY) Dec 19 08:14:22 PM: Applying reduction GUROBI
(CVXPY) Dec 19 08:14:23 PM: Finished problem compilation (took 5.934e+02 seconds).
-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Dec 19 08:14:23 PM: Invoking solver GUROBI  to obtain a solution.

Set parameter OutputFlag to value 1
Set parameter QCPDual to value 1
Gurobi Optimizer version 12.0.0 build v12.0.0rc1 (linux64 - "Ubuntu 22.04.5 LTS")

CPU model: AMD EPYC 7B12, instruction set [SSE2|AVX|AVX2]
Thread count: 4 physical cores, 8 logical processors, using up to 8 threads

Non-default parameters:
QCPDual  1

Optimize a model with 10688 rows, 9636 columns and 20644580 nonzeros
Model fingerprint: 0x13508622
Coefficient statistics:
  Matrix range     [2e-12, 1e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [0e+00, 0e+00]
  RHS range        [8e-01, 3e+03]
Presolve removed 10688 rows and 9636 columns
Presolve time: 2.87s
Presolve: All rows and columns removed
Iteration    Objective       Primal Inf.    Dual Inf.      Time
       0   -8.8537809e+03   0.000000e+00   0.000000e+00      5s
Solved in 0 iterations and 5.05 seconds (2.96 work units)

4 replies

nlaws-camus Dec 19, 2024

Having the same issue with HiGHS FWIW. Is there an LP solver option that does not use the QpMatrixStuffing?

SteveDiamond Dec 19, 2024
Maintainer

Mosek uses the conic interface. If your problem isn't vectorized the conic interface may be slow as well though.

rileyjmurray Dec 19, 2024
Maintainer

Nick, something is strange, and possibly wrong, with your problem. Note
the text here:

Optimize a model with 10688 rows, 9636 columns and 20644580 nonzeros
Model fingerprint: 0x13508622
Coefficient statistics:
  Matrix range     [2e-12, 1e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [0e+00, 0e+00]
  RHS range        [8e-01, 3e+03]
Presolve removed 10688 rows and 9636 columns

Gurobi is presolving away the entire problem.

nlaws-camus Dec 30, 2024

@rileyjmurray Good eye. I since fixed the issue in the model (some variables weren't getting defined). But the problem of ~10 minutes in QpMatrixStuffing persists.

What do you mean by vectorized? Like constraints are defined using linear algebra instead of for loops for each index?

nlaws-camus · 2024-12-30T22:43:26Z

nlaws-camus
Dec 30, 2024

I'm working on parameterizing the problem (and meeting DPP requirements) so that I will hopefully only have to pay the build cost once. Here is the solve log with Clarabel, note that about 10 minutes is spent in ConeMatrixStuffing. All but 3 of the 17106 constraints are vectorized (if I understand the term correctly). Can just 3 constraints lead to the time spent in matrix stuffing?

===============================================================================
                                     CVXPY                                     
                                     v1.6.0                                    
===============================================================================
(CVXPY) Dec 30 10:10:59 PM: Your problem has 6424 variables, 17104 constraints, and 10326580 parameters.
(CVXPY) Dec 30 10:10:59 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Dec 30 10:10:59 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Dec 30 10:10:59 PM: Your problem is compiled with the CPP canonicalization backend.
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
(CVXPY) Dec 30 10:10:59 PM: Compiling problem (target solver=CLARABEL).
(CVXPY) Dec 30 10:10:59 PM: Reduction chain: EvalParams -> Complex2Real -> FlipObjective -> Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> CLARABEL
(CVXPY) Dec 30 10:10:59 PM: Applying reduction EvalParams
(CVXPY) Dec 30 10:10:59 PM: Applying reduction Complex2Real
/home/builder/.cache/bazel/_bazel_builder/99bd544777c52d5a0d8a66ea3b05626a/execroot/ritta/bazel-out/k8-dbg/bin/oe/opf/notebook/notebook.runfiles/common_deps_cvxpy/site-packages/cvxpy/reductions/solvers/solving_chain.py:245: UserWarning:

You are solving a parameterized problem that is not DPP. Because the problem is not DPP, subsequent solves will not be faster than the first one. For more information, see the documentation on Disciplined Parametrized Programming, at https://www.cvxpy.org/tutorial/dpp/index.html

(CVXPY) Dec 30 10:11:00 PM: Applying reduction FlipObjective
(CVXPY) Dec 30 10:11:00 PM: Applying reduction Dcp2Cone
(CVXPY) Dec 30 10:11:00 PM: Applying reduction CvxAttr2Constr
(CVXPY) Dec 30 10:11:00 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Dec 30 10:20:56 PM: Applying reduction CLARABEL
(CVXPY) Dec 30 10:21:01 PM: Finished problem compilation (took 6.016e+02 seconds).
-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Dec 30 10:21:01 PM: Invoking solver CLARABEL  to obtain a solution.
-------------------------------------------------------------
           Clarabel.rs v0.9.0  -  Clever Acronym                

                   (c) Paul Goulart                          
                University of Oxford, 2022                   
-------------------------------------------------------------

problem:
  variables     = 9636
  constraints   = 17106
  nnz(P)        = 0
  nnz(A)        = 20650998
  cones (total) = 2
    :        Zero = 1,  numel = 3216
    : Nonnegative = 1,  numel = 13890

settings:
  linear algebra: direct / qdldl, precision: 64 bit
  max iter = 200, time limit = Inf,  max step = 0.990
  tol_feas = 1.0e-8, tol_gap_abs = 1.0e-8, tol_gap_rel = 1.0e-8,
  static reg : on, ϵ1 = 1.0e-8, ϵ2 = 4.9e-32
  dynamic reg: on, ϵ = 1.0e-13, δ = 2.0e-7
  iter refine: on, reltol = 1.0e-13, abstol = 1.0e-12,
               max iter = 10, stop ratio = 5.0
  equilibrate: on, min_scale = 1.0e-4, max_scale = 1.0e4
               max iter = 10

iter    pcost        dcost       gap       pres      dres      k/t        μ       step      
---------------------------------------------------------------------------------------------
  0  -4.5019e+03  -1.8080e+04  3.02e+00  1.35e-02  2.64e-16  1.00e+00  2.57e+00   ------   
  1  -6.5659e+03  -9.1652e+03  3.96e-01  2.21e-03  3.62e-17  2.23e-01  4.89e-01  8.23e-01  
  2  -8.9678e+03  -9.0542e+03  9.63e-03  5.96e-05  3.22e-18  1.56e-02  1.65e-02  9.90e-01  
  3  -8.9820e+03  -9.0306e+03  5.41e-03  3.35e-05  1.32e-16  1.05e-02  8.68e-03  5.83e-01  
  4  -8.8622e+03  -9.0399e+03  2.00e-02  1.24e-04  2.89e-16  6.58e-02  8.66e-03  1.24e-01  
  5  -8.8610e+03  -8.8844e+03  2.65e-03  1.64e-05  2.78e-16  8.66e-03  1.16e-03  8.66e-01  
  6  -8.8540e+03  -8.8543e+03  3.47e-05  2.14e-07  1.73e-15  1.14e-04  1.54e-05  9.87e-01  
  7  -8.8538e+03  -8.8538e+03  3.47e-07  2.14e-09  1.33e-16  1.14e-06  1.54e-07  9.90e-01  
  8  -8.8538e+03  -8.8538e+03  3.47e-09  2.14e-11  1.32e-16  1.14e-08  1.54e-09  9.90e-01  
---------------------------------------------------------------------------------------------
Terminated with status = Solved
solve time = 516.657558191s
-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Dec 30 10:29:42 PM: Problem status: optimal
(CVXPY) Dec 30 10:29:42 PM: Optimal value: 8.854e+03
(CVXPY) Dec 30 10:29:42 PM: Compilation took 6.016e+02 seconds
(CVXPY) Dec 30 10:29:42 PM: Solver (including time spent in interface) took 5.215e+02 seconds

9 replies

nlaws-camus Jan 2, 2025

Even if I could I don't think that it would help (because the cvxpy part is only a small part of a large infrastructure so you wouldn't be able to construct the model). Can cvxpy export a .lp file or the like that you can use?

SteveDiamond Jan 2, 2025
Maintainer

Unfortunately an .lp file is not enough to debug effectively.

nlaws-camus Jan 2, 2025

I started hacking together an MWE to send your way and I did not bother to include the cp.Parameters. Lo and behold huge savings in build time (down to 30 seconds from 10 minutes):

===============================================================================
                                     CVXPY                                     
                                     v1.6.0                                    
===============================================================================
(CVXPY) Jan 02 10:43:31 PM: Your problem has 6424 variables, 22486 constraints, and 0 parameters.
(CVXPY) Jan 02 10:43:31 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Jan 02 10:43:31 PM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Jan 02 10:43:31 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Jan 02 10:43:31 PM: Your problem is compiled with the CPP canonicalization backend.
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
(CVXPY) Jan 02 10:43:31 PM: Compiling problem (target solver=CLARABEL).
(CVXPY) Jan 02 10:43:31 PM: Reduction chain: Complex2Real -> FlipObjective -> Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> CLARABEL
(CVXPY) Jan 02 10:43:31 PM: Applying reduction Complex2Real
(CVXPY) Jan 02 10:43:31 PM: Applying reduction FlipObjective
(CVXPY) Jan 02 10:43:31 PM: Applying reduction Dcp2Cone
(CVXPY) Jan 02 10:43:31 PM: Applying reduction CvxAttr2Constr
(CVXPY) Jan 02 10:43:31 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Jan 02 10:43:54 PM: Applying reduction CLARABEL
(CVXPY) Jan 02 10:44:02 PM: Finished problem compilation (took 3.111e+01 seconds).

So it does indeed appear to be an issue with cp.Parameters. I think that all you have to do to recreate the issue is make a model where all the coefficients are cp.Parameters.

SteveDiamond Jan 2, 2025
Maintainer

That's odd, normally if you add ignore_dpp=True it's equivalent to replacing parameters with constants. Maybe it's related to using complex parameters? Not an issue we've seen before.

nlaws-camus Jan 4, 2025

I've yet to try adding parameters back in, but I have been able to get the build time down to 5 seconds by splitting the complex variable into real and imaginary variables using the sparsity argument in cp.Variable. Not sure which part gave the big improvement but it seems that I was wrong about parameters being the culprit.

compilation time #1912

Uh oh!

Replies: 6 comments · 22 replies

Uh oh!

=============================================================================== CVXPY v1.3.1

Uh oh!

SteveDiamond May 19, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Transurgeon Jun 1, 2023 Collaborator

Uh oh!

Uh oh!

SteveDiamond Jun 4, 2023 Maintainer

Uh oh!

Uh oh!

SteveDiamond Jun 5, 2023 Maintainer

Uh oh!

phschiele Jun 8, 2023 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveDiamond Dec 19, 2024 Maintainer

Uh oh!

rileyjmurray Dec 19, 2024 Maintainer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SteveDiamond Jan 2, 2025 Maintainer

Uh oh!

Uh oh!

SteveDiamond Jan 2, 2025 Maintainer

Uh oh!

Replies: 6 comments 22 replies

===============================================================================
CVXPY
v1.3.1

SteveDiamond May 19, 2023
Maintainer

Transurgeon Jun 1, 2023
Collaborator

SteveDiamond Jun 4, 2023
Maintainer

SteveDiamond Jun 5, 2023
Maintainer

phschiele
Jun 8, 2023
Maintainer

SteveDiamond Dec 19, 2024
Maintainer

rileyjmurray Dec 19, 2024
Maintainer

SteveDiamond Jan 2, 2025
Maintainer

SteveDiamond Jan 2, 2025
Maintainer