Concept
The mathematics of minimizing or maximizing a stated objective subject to constraints. An optimum exists only after “better” and “feasible” are specified; local versus global, exact versus approximate, computable versus socially desirable remain distinct.
Understand it in one breath
Find a choice that minimizes or maximizes an objective subject to constraints. Linear programming, gradient methods, and stochastic search provide different guarantees for different structures. In a convex problem every local minimum is global; in a nonconvex problem such as neural-network training, a zero gradient alone is no certificate of optimality. Logistics, design, statistics, and learning supply many applications.
At a glance
Key formula
Key moments
Fermat — extrema before differential notation
Fermat’s adequality compared nearby expressions to solve many tangent and extremum problems. It did not state the modern gradient-equals-zero rule or supply a sufficient condition covering boundaries and nondifferentiable points.
Euler — choosing an entire function
Methodus inveniendi systematized extrema of quantities such as length and time attached to curves. Lausanne marks publication, and Lagrange and others later developed the formalism.
Cauchy — descending by repeated negative gradients
Cauchy proposed a steepest-descent procedure for simultaneous equations. Step size, initialization, and landscape determine whether it converges, stalls, or diverges; global optimality is not automatic.
Karush and Kantorovich — constraints and resource allocation
Karush studied conditions for inequality constraints while Kantorovich modeled linear production planning in a different institution. Their work connects to KKT and simplex without becoming one person’s or one city’s invention.
Robbins and Monro — approximation through noise
Noisy observations and properly diminishing steps could estimate a target without exact function values. This is an important ancestor of stochastic gradient methods, not an identical modern mini-batch algorithm.
AlexNet — combining data, GPUs, and loss optimization
Large image data, convolutional networks, GPUs, backpropagation, and stochastic optimization sharply improved benchmark performance. Minimizing that loss did not automatically optimize intelligence, truth, or fairness.
Modern applications
Neural-network training, airline scheduling, logistics, design, energy systems, portfolios, and game AI. Results remain conditional on objectives, constraints, data, uncertainty, algorithms, and stopping rules; omitted costs and fairness do not optimize themselves.
Beyond MathVoyage
Loading…