Recursive Least Square Algorithm

The Recursive Least Squares (RLS) algorithm is used in fields like signal processing, adaptive control and system identification. It is an extension of Least Squares method which is designed to continuously update its parameter estimates as new data arrives. This makes RLS particularly useful in real-time applications where the model must adapt to changing conditions. It is a method that helps in estimating the parameters of a model by minimizing the difference between the predicted and actual values.

Lets understand the mathematical formulation,

Consider a linear model:

y_k = \theta^\top x_k + \epsilon_k

Where,

• y_k is the scaler output at time k,
• x_k is the vector output of input variables,
• \theta is the parameter vector to be estimated,
• \epsilon_k is the noise term.

The objective is to minimize the cost function:

J(\theta) = \sum_{i=1}^{k} \lambda^{k-i} (y_i - \theta^\top x_i)^2

Where \lambda is the forgetting factor, determining the weight given to past observations.

The recursive update equations for \theta_k are:

\theta_k = \theta_{k-1} + K_k (y_k - \theta_{k-1}^\top x_k)

K_k = \frac{P_{k-1} x_k}{\lambda + x_k^\top P_{k-1} x_k}

P_k = \frac{1}{\lambda} \left( P_{k-1} - K_k x_k^\top P_{k-1} \right)

Where,

• P_k is the error covariance matrix,
• K_k is the Kalman gain vector.

The RLS algorithm can be derived by minimizing the weighted least squares cost function using the Kalman filter framework. This derivation provides insights into the algorithm's structure and its connection to optimal estimation theory.

The plot in figure 2 tracks how two estimated parameters \theta_0 and \theta_1 evolve over 100 iterations of RLS.

Versions of RLS Algorithm

Various versions of RLS have been developed over time so as to improve performance or address specific problems:

• Fast RLS: This is a quicker version of RLS that reduces the computation needed for large datasets. It has a time complexity of O(n).
• Square-Root RLS: This version uses a different technique to keep the calculations more stable, especially when dealing with small numbers or large systems.
• Filtered-X RLS: This is used in noise control applications, where the goal is to remove unwanted noise from a signal.

Application

RLS is used in a variety of fields where the need to continuously adjust parameters based on new data is needed:

• System Identification: In systems where we don't know the model exactly, RLS helps us figure out the model parameters by comparing predictions with real data.
• Adaptive Control: RLS can adjust the settings of controllers in systems like robots or drones to ensure they adapt to changing conditions.
• Signal Processing: It’s used in noise reduction or signal enhancement, like improving the quality of a phone call.
• Echo Cancellation: RLS can help remove echoes in communication systems, improving sound clarity.

Advantages of RLS Algorithm

There are various advantages of using RLS Algorithm,

• Faster Convergence: RLS finds the right solution faster than other algorithms, especially when data is correlated.
• Real-Time Processing: It’s perfect for applications where data is constantly coming in, like speech recognition or live data analysis.
• Flexibility: It works well for systems that change over time, as it can adapt to new information quickly.

Limitations of RLS Algorithm

• High Computation: RLS can be computationally expensive, especially if there are a lot of data points.
• Numerical Instability: If the data isn't well-conditioned (i.e., it contains small or large numbers that don't balance well), RLS might give inaccurate results.
• Memory Usage: Since it tracks the error covariance matrix, RLS can use a lot of memory in larger systems.