Maximizing Profit Subject to a CVaR Constraint? #2830

kmcentush-eq · 2025-06-13T02:10:27Z

kmcentush-eq
Jun 13, 2025

UPDATE: This has been solved as part of #2836

Per the title, I'm trying to maximize profit of a portfolio subject to not exceeding a specified CVaR. I'm trying the code below but getting an error when the solution is being inverted. I imagine this is a result of what I'm trying to do, so any advice is appreciated!

Code that Errors

import numpy as np
import cvxpy as cp

# Generate data
num_stocks = 5
num_samples = 25
pnl_samples = np.random.uniform(low=0.0, high=1.0, size=(num_samples, num_stocks))
pnl_expected = pnl_samples.mean(axis=0)

# Prepare to solve
quantile = 0.05
w = cp.Variable(num_stocks, nonneg=True)
cvar = cp.cvar(pnl_samples @ w, 1 - quantile)
pnl = w @ pnl_expected

# Solve
objective = cp.Maximize(pnl)
constraints = [cvar <= 0.5]
problem = cp.Problem(objective, constraints)
problem.solve(
    solver=cp.HIGHS,
    verbose=True,
)

Error Traceback

File <command-6934689331428126>, line 19
     17 constraints = [cvar <= 0.5]
     18 problem = cp.Problem(objective, constraints)
---> 19 problem.solve(
     20     solver=cp.HIGHS,
     21     verbose=True,
     22     log_to_console=True,
     23     output_flag=True,
     24 )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/cvxpy/problems/problem.py:600, in Problem.solve(self, *args, **kwargs)
    597         raise ValueError(
    598             "Cannot specify both 'solver' and 'solver_path'. Please choose one.")
    599     return self._solve_solver_path(solve_func,solver_path, args, kwargs)
--> 600 return solve_func(self, *args, **kwargs)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/cvxpy/problems/problem.py:1193, in Problem._solve(self, solver, warm_start, verbose, gp, qcp, requires_grad, enforce_dpp, ignore_dpp, canon_backend, **kwargs)
   1191 end = time.time()
   1192 self._solve_time = end - start
-> 1193 self.unpack_results(solution, solving_chain, inverse_data)
   1194 if verbose:
   1195     print(_FOOTER)
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/cvxpy/problems/problem.py:1508, in Problem.unpack_results(self, solution, chain, inverse_data)
   1486 def unpack_results(self, solution, chain: SolvingChain, inverse_data) -> None:
   1487     """Updates the problem state given the solver results.
   1488 
   1489     Updates problem.status, problem.value and value of
   (...)
   1505         If the solver failed
   1506     """
-> 1508     solution = chain.invert(solution, inverse_data)
   1509     if solution.status in s.INACCURATE:
   1510         warnings.warn(
   1511             "Solution may be inaccurate. Try another solver, "
   1512             "adjusting the solver settings, or solve with "
   1513             "verbose=True for more information."
   1514         )
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/cvxpy/reductions/chain.py:84, in Chain.invert(self, solution, inverse_data)
     81 """Returns a solution to the original problem given the inverse_data.
     82 """
     83 for r, inv in reversed(list(zip(self.reductions, inverse_data))):
---> 84     solution = r.invert(solution, inv)
     85 return solution
File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/cvxpy/reductions/dcp2cone/cone_matrix_stuffing.py:439, in ConeMatrixStuffing.invert(self, solution, inverse_data)
    436             dual_vars[old_con] = solution.dual_vars[new_con]
    437         else:
    438             dual_vars[old_con] = np.reshape(
--> 439                 solution.dual_vars[new_con],
    440                 shape,
    441                 order='F'
    442             )
    444 return Solution(solution.status, opt_val, primal_vars, dual_vars,
    445                 solution.attr)

Edit: For those finding this in the future and looking for a working example: technically the snippet above doesn't have the right sign convention to properly limit CVaR if the pnl variables are representing profit as positive. See my comment below with the histogram for working code with proper sign convention.

Answered by kmcentush-eq

Jun 13, 2025

Update: HiGHs should work for all versions that include #2836

Swapping the solver from cp.HIGHS to cp.SCS seemed to fix my problem, but now I'm curious why HiGHS is having a problem inverting. The printout suggests it finds the same optimum as SCS, so my naive guess is that it isn't related to the type of problem but something else instead.

My actual problem is larger/more complex than the example shared in this discussion and SCS is infeasibly slow to solve compared to HiGHS, so any advice is appreciated!

Edit: Using cp.SCIPY also works, and seems to use HiGHS under the hood without any inversion error.

HiGHS Output (pre-error)

(CVXPY) Jun 13 02:10:05 AM: Your problem has 5 variables, 1 …

View full answer

kmcentush-eq · 2025-06-13T02:11:55Z

kmcentush-eq
Jun 13, 2025
Author

Update: HiGHs should work for all versions that include #2836

Swapping the solver from cp.HIGHS to cp.SCS seemed to fix my problem, but now I'm curious why HiGHS is having a problem inverting. The printout suggests it finds the same optimum as SCS, so my naive guess is that it isn't related to the type of problem but something else instead.

My actual problem is larger/more complex than the example shared in this discussion and SCS is infeasibly slow to solve compared to HiGHS, so any advice is appreciated!

Edit: Using cp.SCIPY also works, and seems to use HiGHS under the hood without any inversion error.

HiGHS Output (pre-error)

(CVXPY) Jun 13 02:10:05 AM: Your problem has 5 variables, 1 constraints, and 0 parameters.
(CVXPY) Jun 13 02:10:05 AM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Jun 13 02:10:05 AM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Jun 13 02:10:05 AM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Jun 13 02:10:05 AM: Your problem is compiled with the CPP canonicalization backend.
(CVXPY) Jun 13 02:10:05 AM: Compiling problem (target solver=HIGHS).
(CVXPY) Jun 13 02:10:05 AM: Reduction chain: FlipObjective -> Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> HIGHS
(CVXPY) Jun 13 02:10:05 AM: Applying reduction FlipObjective
(CVXPY) Jun 13 02:10:05 AM: Applying reduction Dcp2Cone
(CVXPY) Jun 13 02:10:05 AM: Applying reduction CvxAttr2Constr
(CVXPY) Jun 13 02:10:05 AM: Applying reduction ConeMatrixStuffing
(CVXPY) Jun 13 02:10:05 AM: Applying reduction HIGHS
(CVXPY) Jun 13 02:10:05 AM: Finished problem compilation (took 7.750e-03 seconds).
(CVXPY) Jun 13 02:10:05 AM: Invoking solver HIGHS  to obtain a solution.
===============================================================================
                                     CVXPY                                     
                                     v1.6.6                                    
===============================================================================
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
Running HiGHS 1.11.0 (git hash: 364c83a): Copyright (c) 2025 HiGHS under MIT licence terms
LP   has 81 rows; 32 cols; 407 nonzeros
Coefficient ranges:
  Matrix [2e-03, 1e+00]
  Cost   [4e-01, 6e-01]
  Bound  [0e+00, 0e+00]
  RHS    [5e-01, 5e-01]
Presolving model
51 rows, 32 cols, 377 nonzeros  0s
51 rows, 32 cols, 377 nonzeros  0s
Presolve : Reductions: rows 51(-30); columns 32(-0); elements 377(-30)
Solving the presolved LP
Using EKK dual simplex solver - serial
  Iteration        Objective     Infeasibilities num(sum)
          0    -3.6823900741e+01 Ph1: 51(36565.5); Du: 5(36.8239) 0s
         50    -4.9261972704e-01 Pr: 0(0) 0s
Solving the original LP from the solution after postsolve
Model status        : Optimal
Simplex   iterations: 50
Objective value     : -4.9261972704e-01
P-D objective error :  2.7961942383e-17
HiGHS run time      :          0.00

SCS Output

(CVXPY) Jun 13 02:16:25 AM: Your problem has 5 variables, 1 constraints, and 0 parameters.
(CVXPY) Jun 13 02:16:25 AM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Jun 13 02:16:25 AM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Jun 13 02:16:25 AM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Jun 13 02:16:25 AM: Your problem is compiled with the CPP canonicalization backend.
(CVXPY) Jun 13 02:16:25 AM: Compiling problem (target solver=SCS).
(CVXPY) Jun 13 02:16:25 AM: Reduction chain: FlipObjective -> Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> SCS
(CVXPY) Jun 13 02:16:25 AM: Applying reduction FlipObjective
(CVXPY) Jun 13 02:16:25 AM: Applying reduction Dcp2Cone
(CVXPY) Jun 13 02:16:25 AM: Applying reduction CvxAttr2Constr
(CVXPY) Jun 13 02:16:25 AM: Applying reduction ConeMatrixStuffing
(CVXPY) Jun 13 02:16:25 AM: Applying reduction SCS
(CVXPY) Jun 13 02:16:25 AM: Finished problem compilation (took 9.133e-03 seconds).
(CVXPY) Jun 13 02:16:25 AM: Invoking solver SCS  to obtain a solution.
(CVXPY) Jun 13 02:16:25 AM: Problem status: optimal
(CVXPY) Jun 13 02:16:25 AM: Optimal value: 4.923e-01
(CVXPY) Jun 13 02:16:25 AM: Compilation took 9.133e-03 seconds
(CVXPY) Jun 13 02:16:25 AM: Solver (including time spent in interface) took 2.120e-03 seconds
===============================================================================
                                     CVXPY                                     
                                     v1.6.6                                    
===============================================================================
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
------------------------------------------------------------------
	       SCS v3.2.7 - Splitting Conic Solver
	(c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem:  variables n: 32, constraints m: 81
cones: 	  l: linear vars: 81
settings: eps_abs: 1.0e-05, eps_rel: 1.0e-05, eps_infeas: 1.0e-07
	  alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
	  max_iters: 100000, normalize: 1, rho_x: 1.00e-06
	  acceleration_lookback: 10, acceleration_interval: 10
lin-sys:  sparse-direct-amd-qdldl
	  nnz(A): 407, nnz(P): 0
------------------------------------------------------------------
 iter | pri res | dua res |   gap   |   obj   |  scale  | time (s)
------------------------------------------------------------------
     0| 7.96e+00  4.58e-01  8.74e+00 -4.76e+00  1.00e-01  4.65e-04 
   250| 1.52e+00  1.85e-01  5.47e-05 -4.92e-01  1.00e-01  1.05e-03 
   500| 2.85e-05  4.35e-05  1.65e-05 -4.92e-01  1.20e+00  1.63e-03 
   575| 3.53e-06  1.36e-05  4.98e-06 -4.92e-01  1.20e+00  1.80e-03 
------------------------------------------------------------------
status:  solved
timings: total: 1.80e-03s = setup: 2.78e-04s + solve: 1.53e-03s
	 lin-sys: 9.30e-04s, cones: 9.21e-05s, accel: 8.72e-05s
------------------------------------------------------------------
objective = -0.492317
------------------------------------------------------------------
-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------

SCIPY Output

===============================================================================
                                     CVXPY                                     
                                     v1.6.5                                    
===============================================================================
(CVXPY) Jun 12 09:07:30 PM: Your problem has 57100 variables, 168010 constraints, and 0 parameters.
(CVXPY) Jun 12 09:07:30 PM: It is compliant with the following grammars: DCP, DQCP
(CVXPY) Jun 12 09:07:30 PM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
(CVXPY) Jun 12 09:07:30 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
(CVXPY) Jun 12 09:07:30 PM: Your problem is compiled with the CPP canonicalization backend.
-------------------------------------------------------------------------------
                                  Compilation                                  
-------------------------------------------------------------------------------
(CVXPY) Jun 12 09:07:30 PM: Compiling problem (target solver=SCIPY).
(CVXPY) Jun 12 09:07:30 PM: Reduction chain: FlipObjective -> Dcp2Cone -> CvxAttr2Constr -> ConeMatrixStuffing -> SCIPY
(CVXPY) Jun 12 09:07:30 PM: Applying reduction FlipObjective
(CVXPY) Jun 12 09:07:30 PM: Applying reduction Dcp2Cone
(CVXPY) Jun 12 09:07:30 PM: Applying reduction CvxAttr2Constr
(CVXPY) Jun 12 09:07:30 PM: Applying reduction ConeMatrixStuffing
(CVXPY) Jun 12 09:07:36 PM: Applying reduction SCIPY
(CVXPY) Jun 12 09:07:38 PM: Finished problem compilation (took 8.379e+00 seconds).
-------------------------------------------------------------------------------
                                Numerical solver                               
-------------------------------------------------------------------------------
(CVXPY) Jun 12 09:07:38 PM: Invoking solver SCIPY  to obtain a solution.
Solver terminated with message: Optimization terminated successfully. (HiGHS Status 7: Optimal)
-------------------------------------------------------------------------------
                                    Summary                                    
-------------------------------------------------------------------------------
(CVXPY) Jun 12 09:07:52 PM: Problem status: optimal
(CVXPY) Jun 12 09:07:52 PM: Optimal value: 4.236e+04
(CVXPY) Jun 12 09:07:52 PM: Compilation took 8.379e+00 seconds
(CVXPY) Jun 12 09:07:52 PM: Solver (including time spent in interface) took 1.400e+01 seconds

2 replies

Transurgeon Jun 13, 2025
Collaborator

HiGHS Output (pre-error)

Can you explain what you mean by this? I don't understand why you are able to run successfully with HiGHS.
Based on your detective work, I suppose the issues might be somewhere in the HiGHS solver interface.

kmcentush-eq Jun 13, 2025
Author

So the detailed log output (above) indicates that the HiGHS solver finds an optimum, but then the error is thrown within CVXPY when backing that out to Python (see traceback in original post). The SCIPY interface also uses HiGHS under the hood, but does not have this problem.

The example snippet in the original post illustrates the "successful" solve but KeyError when coming back to Python.

I don't understand why you are able to run successfully with HiGHS.

Does the solver supposedly not support this kind of problem?

kmcentush-eq · 2025-06-13T05:29:49Z

kmcentush-eq
Jun 13, 2025
Author

Final question: I'm still a little confused on how the CVXPY calculated value connects to the actual value, but I think I hacked my way to it below. Ultimately, this code returns the positively signed version of the actual mean CVaR (where negative sign convention = loss). Is this the correct way to think about it?

Future edit: I've convinced myself this is correct. Thanks again all of the work on this library!

Code

# Generate samples
# By convention a positive sign means profit
quantile = 0.05
np.random.seed(0)
samples = np.random.normal(loc=0.25, scale=0.2, size=2500)

# Calculate CVaR manually
var = np.quantile(samples, quantile)  # note syntax is just `quantile`
tail_losses = samples[samples <= var]
cvar = np.mean(tail_losses)
print(var, cvar)

# Calculate CVaR via CVXPY
# We want to calculate CVaR on losses, so note the syntax below
x = cp.Variable(samples.shape)
x.value = samples
cp_cvar = cp.cvar(-x, 1 - quantile).value  # note the change in syntax from `x` to `-x` and `quantile` to `1 - quantile`
print(cp_cvar)

# Plot
fig = px.histogram(x=samples)
fig.add_vline(var, line_dash="dash", annotation_text="Actual VaR")
fig.add_vline(cvar, line_dash="dash", annotation_text="Actual CVaR")
fig.add_vline(cp_cvar, line_dash="dash", annotation_text="CVXPY CVaR")
fig.add_vline(-cp_cvar, line_dash="dash", annotation_text="CVXPY -CVaR", annotation_position="top left")
fig.show()

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Maximizing Profit Subject to a CVaR Constraint? #2830

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Maximizing Profit Subject to a CVaR Constraint? #2830

Uh oh!

Uh oh!

kmcentush-eq Jun 13, 2025

Replies: 2 comments · 2 replies

Uh oh!

Uh oh!

kmcentush-eq Jun 13, 2025 Author

Uh oh!

Transurgeon Jun 13, 2025 Collaborator

Uh oh!

kmcentush-eq Jun 13, 2025 Author

Uh oh!

Uh oh!

kmcentush-eq Jun 13, 2025 Author

kmcentush-eq
Jun 13, 2025

Replies: 2 comments 2 replies

kmcentush-eq
Jun 13, 2025
Author

Transurgeon Jun 13, 2025
Collaborator

kmcentush-eq Jun 13, 2025
Author

kmcentush-eq
Jun 13, 2025
Author