Without using any calculus:
If x + y² = 5, then y² = 5 - x, so that
b = x y² = x (5 - x) = 5 x - x²
Complete the square to get
b = 25/4 - (x - 5/2)²
Then b gets a maximum value of 25/4 when x = 5/2.
Simple calculus method:
With b = 5 x - x², differentiate b with respect to x and set the derivative equal to 0 to solve for critical points:
db/dx = 5 - 2 x = 0 ==> x = 5/2
Take the second derivative to determine whether b is concave or convex:
d²b/dx² = -2
The second derivative is negative, which means b is concave with respect to x, and this indicates x = 5/2 is the site of a local maximum, which is b (5/2) = 25/4.
More complicated calculus method:
Lagrange multipliers. The Lagrangian is
L (x, y, λ) = x y² - λ (x + y² - 5)
where we assume λ ≠ 0.
Take its derivatives:
dL/dx = y² - λ
dL/dy = 2 x y - 2 λ y
dL/dλ = x + y² - 5
Set each derivative equal to 0 to find the critical points. The first equation tells us y² = λ, and the second equation tells us x = λ, so
x + y² - 5 = 2 λ - 5 = 0
Solving this gives λ = x = y² = 5/2, so that b has an extreme value of
b = (5/2) (5/2) = 25/4
Deciding whether this is a maximum or minimum comes down to computing the Hessian determinant at the critical value. The Hessian is
with determinant |H (x, y)| = -4 y². At the critical point, the determinant is -10 < 0, which indicates a maximum value of 25/4.
