Everipedia is now IQ.wiki - Join the IQ Brainlist and our Discord for early access to editing on the new platform and to participate in the beta testing.

Duality (optimization)

In mathematical optimization theory, duality or the duality principle is the principle that optimization problems may be viewed from either of two perspectives, the primal problem or the dual problem. The solution to the dual problem provides a lower bound to the solution of the primal (minimization) problem.[1] However in general the optimal values of the primal and dual problems need not be equal. Their difference is called the duality gap. For convex optimization problems, the duality gap is zero under a constraint qualification condition.

Dual problem

Usually the term "dual problem" refers to the Lagrangian dual problem but other dual problems are used – for example, the Wolfe dual problem and the Fenchel dual problem. The Lagrangian dual problem is obtained by forming the Lagrangian of a minimization problem by using nonnegative Lagrange multipliers to add the constraints to the objective function, and then solving for the primal variable values that minimize the original objective function. This solution gives the primal variables as functions of the Lagrange multipliers, which are called dual variables, so that the new problem is to maximize the objective function with respect to the dual variables under the derived constraints on the dual variables (including at least the nonnegativity constraints).

In general given twodual pairsofseparatedlocally convex spacesandand the function, we can define the primal problem as findingsuch thatIn other words, ifexists,is theminimumof the functionand theinfimum(greatest lower bound) of the function is attained.
If there are constraint conditions, these can be built into the functionby lettingwhereis a suitable function onthat has a minimum 0 on the constraints, and for which one can prove that. The latter condition is trivially, but not always conveniently, satisfied for thecharacteristic function(i.e.forsatisfying the constraints andotherwise). Then extendto aperturbation functionsuch that.[2]

The duality gap is the difference of the right and left hand sides of the inequality

whereis theconvex conjugatein both variables anddenotes thesupremum(least upper bound).[2][3][4]

Duality gap

The duality gap is the difference between the values of any primal solutions and any dual solutions. Ifis the optimal dual value andis the optimal primal value, then the duality gap is equal to. This value is always greater than or equal to 0. The duality gap is zero if and only ifstrong dualityholds. Otherwise the gap is strictly positive andweak dualityholds.[5]

In computational optimization, another "duality gap" is often reported, which is the difference in value between any dual solution and the value of a feasible but suboptimal iterate for the primal problem. This alternative "duality gap" quantifies the discrepancy between the value of a current feasible but suboptimal iterate for the primal problem and the value of the dual problem; the value of the dual problem is, under regularity conditions, equal to the value of the convex relaxation of the primal problem: The convex relaxation is the problem arising replacing a non-convex feasible set with its closed convex hull and with replacing a non-convex function with its convex closure, that is the function that has the epigraph that is the closed convex hull of the original primal objective function.[6] [7][8][9] [10] [11] [12] [13] [14] [15][16]

Linear case

Linear programming problems are optimization problems in which the objective function and the constraints are all linear. In the primal problem, the objective function is a linear combination of n variables. There are m constraints, each of which places an upper bound on a linear combination of the n variables. The goal is to maximize the value of the objective function subject to the constraints. A solution is a vector (a list) of n values that achieves the maximum value for the objective function.

In the dual problem, the objective function is a linear combination of the m values that are the limits in the m constraints from the primal problem. There are n dual constraints, each of which places a lower bound on a linear combination of m dual variables.

Relationship between the primal problem and the dual problem

In the linear case, in the primal problem, from each sub-optimal point that satisfies all the constraints, there is a direction or subspace of directions to move that increases the objective function. Moving in any such direction is said to remove slack between the candidate solution and one or more constraints. An infeasible value of the candidate solution is one that exceeds one or more of the constraints.

In the dual problem, the dual vector multiplies the constraints that determine the positions of the constraints in the primal. Varying the dual vector in the dual problem is equivalent to revising the upper bounds in the primal problem. The lowest upper bound is sought. That is, the dual vector is minimized in order to remove slack between the candidate positions of the constraints and the actual optimum. An infeasible value of the dual vector is one that is too low. It sets the candidate positions of one or more of the constraints in a position that excludes the actual optimum.

This intuition is made formal by the equations in Linear programming: Duality.

Nonlinear case

In nonlinear programming, the constraints are not necessarily linear. Nonetheless, many of the same principles apply.

To ensure that the global maximum of a non-linear problem can be identified easily, the problem formulation often requires that the functions be convex and have compact lower level sets.

This is the significance of the Karush–Kuhn–Tucker conditions. They provide necessary conditions for identifying local optima of non-linear programming problems. There are additional conditions (constraint qualifications) that are necessary so that it will be possible to define the direction to an optimal solution. An optimal solution is one that is a local optimum, but possibly not a global optimum.

The strong Lagrangian principle: Lagrange duality

Given a nonlinear programming problem in standard form

with the domainhaving non-empty interior, the Lagrangian functionis defined as
The vectorsandare called the dual variables or Lagrange multiplier vectors associated with the problem. The Lagrange dual functionis defined as
The dual function g is concave, even when the initial problem is not convex, because it is a point-wise infimum of affine functions. The dual function yields lower bounds on the optimal valueof the initial problem; for anyand anywe have.
If aconstraint qualificationsuch asSlater's conditionholds and the original problem is convex, then we havestrong duality, i.e..

Convex problems

For a convex minimization problem with inequality constraints,

the Lagrangian dual problem is

where the objective function is the Lagrange dual function. Provided that the functionsandare continuously differentiable, the infimum occurs where the gradient is equal to zero. The problem
is called the Wolfe dual problem. This problem may be difficult to deal with computationally, because the objective function is not concave in the joint variables. Also, the equality constraintis nonlinear in general, so the Wolfe dual problem is typically a nonconvex optimization problem. In any case,weak dualityholds.[17]

History

According to George Dantzig, the duality theorem for linear optimization was conjectured by John von Neumann immediately after Dantzig presented the linear programming problem. Von Neumann noted that he was using information from his game theory, and conjectured that two person zero sum matrix game was equivalent to linear programming. Rigorous proofs were first published in 1948 by Albert W. Tucker and his group. (Dantzig's foreword to Nering and Tucker, 1993)

• Duality

• Relaxation (approximation)

References

[1]
Citation Linkwww.stanford.eduBoyd, Stephen P.; Vandenberghe, Lieven (2004). Convex Optimization (pdf). Cambridge University Press. p. 216. ISBN 978-0-521-83378-3. Retrieved October 15, 2011.
Oct 1, 2019, 4:59 AM
[2]
Oct 1, 2019, 4:59 AM
[3]
Citation Linkopenlibrary.orgCsetnek, Ernö Robert (2010). Overcoming the failure of the classical generalized interior-point regularity conditions in convex optimization. Applications of the duality theory to enlargements of maximal monotone operators. Logos Verlag Berlin GmbH. ISBN 978-3-8325-2503-3.
Oct 1, 2019, 4:59 AM
[4]
Citation Link//www.ams.org/mathscinet-getitem?mr=1921556Zălinescu, Constantin (2002). Convex analysis in general vector spaces. River Edge, NJ: World Scientific Publishing Co., Inc. pp. 106–113. ISBN 981-238-067-1. MR 1921556.
Oct 1, 2019, 4:59 AM
[5]
Citation Linkopenlibrary.orgBorwein, Jonathan; Zhu, Qiji (2005). Techniques of Variational Analysis. Springer. ISBN 978-1-4419-2026-3.
Oct 1, 2019, 4:59 AM
[6]
Citation Linkopenlibrary.orgAhuja, Ravindra K.; Magnanti, Thomas L.; Orlin, James B. (1993). Network Flows: Theory, Algorithms and Applications. Prentice Hall. ISBN 0-13-617549-X.
Oct 1, 2019, 4:59 AM
[7]
Citation Linkopenlibrary.orgBertsekas, Dimitri; Nedic, Angelia; Ozdaglar, Asuman (2003). Convex Analysis and Optimization. Athena Scientific. ISBN 1-886529-45-0.
Oct 1, 2019, 4:59 AM
[8]
Citation Linkopenlibrary.orgBertsekas, Dimitri P. (1999). Nonlinear Programming (2nd ed.). Athena Scientific. ISBN 1-886529-00-0.
Oct 1, 2019, 4:59 AM
[9]
Citation Linkopenlibrary.orgBertsekas, Dimitri P. (2009). Convex Optimization Theory. Athena Scientific. ISBN 978-1-886529-31-1.
Oct 1, 2019, 4:59 AM
[10]
Citation Link//www.ams.org/mathscinet-getitem?mr=2265882Bonnans, J. Frédéric; Gilbert, J. Charles; Lemaréchal, Claude; Sagastizábal, Claudia A. (2006). Numerical optimization: Theoretical and practical aspects. Universitext (Second revised ed. of translation of 1997 French ed.). Berlin: Springer-Verlag. pp. xiv+490. doi:10.1007/978-3-540-35447-5. ISBN 3-540-35445-X. MR 2265882.
Oct 1, 2019, 4:59 AM
[11]
Citation Link//www.ams.org/mathscinet-getitem?mr=1261420Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). Convex analysis and minimization algorithms, Volume I: Fundamentals. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. 305. Berlin: Springer-Verlag. pp. xviii+417. ISBN 3-540-56850-6. MR 1261420.
Oct 1, 2019, 4:59 AM
[12]
Citation Link//www.ams.org/mathscinet-getitem?mr=1295240Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). "14 Duality for Practitioners". Convex analysis and minimization algorithms, Volume II: Advanced theory and bundle methods. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. 306. Berlin: Springer-Verlag. pp. xviii+346. ISBN 3-540-56852-2. MR 1295240.
Oct 1, 2019, 4:59 AM
[13]
Citation Link//www.ams.org/mathscinet-getitem?mr=1888251Lasdon, Leon S. (2002) [Reprint of the 1970 Macmillan]. Optimization theory for large systems. Mineola, New York: Dover Publications, Inc. pp. xiii+523. ISBN 978-0-486-41999-2. MR 1888251.
Oct 1, 2019, 4:59 AM
[14]
Citation Link//www.ams.org/mathscinet-getitem?mr=1900016Lemaréchal, Claude (2001). "Lagrangian relaxation". In Jünger, Michael; Naddef, Denis (eds.). Computational combinatorial optimization: Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000. Lecture Notes in Computer Science (LNCS). 2241. Berlin: Springer-Verlag. pp. 112–156. doi:10.1007/3-540-45586-8_4. ISBN 3-540-42877-1. MR 1900016.
Oct 1, 2019, 4:59 AM
[15]
Citation Link//www.ams.org/mathscinet-getitem?mr=0868279Minoux, Michel (1986). Mathematical programming: Theory and algorithms. Egon Balas (forward); Steven Vajda (trans) from the (1983 Paris: Dunod) French. Chichester: A Wiley-Interscience Publication. John Wiley & Sons, Ltd. pp. xxviii+489. ISBN 0-471-90170-9. MR 0868279. (2008 Second ed., in French: Programmation mathématique : Théorie et algorithmes, Éditions Tec & Doc, Paris, 2008. xxx+711 pp. ).
Oct 1, 2019, 4:59 AM
[16]
Citation Link//www.ams.org/mathscinet-getitem?mr=0544669Shapiro, Jeremy F. (1979). Mathematical programming: Structures and algorithms. New York: Wiley-Interscience [John Wiley & Sons]. pp. xvi+388. ISBN 0-471-77886-9. MR 0544669.
Oct 1, 2019, 4:59 AM
[17]
Citation Link//www.jstor.org/stable/2028848Geoffrion, Arthur M. (1971). "Duality in Nonlinear Programming: A Simplified Applications-Oriented Development". SIAM Review. 13 (1): 1–37. doi:10.1137/1013001. JSTOR 2028848.
Oct 1, 2019, 4:59 AM
[18]
Citation Linkwww.springer.comNumerical optimization: Theoretical and practical aspects
Oct 1, 2019, 4:59 AM
[19]