Identities for Gradient Computation

Gradient computation is the process of calculating the gradient (or vector of partial derivatives) of a function with respect to its variables. In mathematical optimization and machine learning, this often involves finding how a change in each input variable affects the output of the function. For any function f the gradient of f, denoted as ∇f or grad f.

For a function f(x₁, x₂, . . . , x_n), the gradient is represented as a vector of partial derivatives which is given by :

∇f = (​∂f/∂x_1​, ​∂f/∂x₂​, . . . , ​∂f/∂x_n​ )

Where:

• f: A scalar function of multiple variables f(x₁, x₂, . . . , x_n).
• ∇f: The gradient of the function f, which is a vector of partial derivatives.
• ∂f/∂x_i: The partial derivative of f with respect to x_i​. It measures how f changes as x_i​ changes while keeping all other variables constant.
• x₁, x₂, . . . , x_n: The independent variables of the function f.

Identities For Gradient Computing

Here are some useful identities for Gradient Computing:

Gradient of a Sum

∇(f(x) + g(x)) = ∇f(x) + ∇g(x)

The gradient of a sum is the sum of the gradients.

Gradient of a Scalar Multiple

∇(c ⋅ f(x)) = c⋅∇f(x)

Multiplying a function by a constant scales its gradient by the same constant. Where c is a constant.

Gradient of a Product (Product Rule)

∇(f(x) ⋅ g(x)) = f(x)∇g(x) + g(x)∇f(x)

The gradient of a product involves both functions and their gradients.

Gradient of a Quotient

∇(f(x)/g(x)​) = g(x)∇f(x) − f(x)∇g(x)/[g(x)]²​

Assuming g(x) ≠ 0, for a quotient, the gradient uses the difference of gradients over the square of the denominator.

Gradient of a Composition (Chain Rule)

If z = f(g(x)), then:

∇z = df/dg ⋅ ∇g(x)

For functions composed of multiple variables, the gradient of a composition is the derivative of the outer function times the gradient of the inner function.

Gradient of a Dot Product (Vector-Valued Functions)

For a ⋅ x, where a is a constant vector:

∇(a ⋅ x) = a

The gradient of a dot product with a constant vector is that vector.

Gradient of a Norm (Vector-Valued Function)

For ∥x∥², where x is a vector:

∇(∥x∥²) = 2x

The gradient of the squared norm of a vector is twice the vector.

Gradient of Logarithm

∇(log⁡ f(x)) = 1/f(x) × (∇f(x))

The gradient of the logarithm of a function is given by the product of the reciprocal of the function and its gradient.

Note: These identities help simplify and calculate gradients in various scenarios.

Read More,

• Vector Calculus
• Divergence and Curl in Mathematics
• Divergence Theorem