Efficient Multi-Step Forecasting with Echo State Networks: A Reservoir-Based Perspective

Creating a Multi-Step Forecasting Machine Learning Model

This article presents a machine learning model known as the Echo State Networks (ESN) algorithm. It explains the intuition and how the model is structured, then it creates a multi-step forecasting model in order to predict using the model’s inputs.

Multi-step forecasting is a technique to project multiple values to the future, thus creating a trajectory. It uses previous forecasts as inputs to forecast new values. Naturally, this comes with a great bias, but it remains interesting to see it in action.

---

Intuition of the Model

An ESN is a type of recurrent neural network (RNN) designed for efficient training and fast computation, especially in time-series tasks like prediction, classification, or control.

Unlike traditional RNNs, only the output weights are trained. The rest of the network (the recurrent part, called the reservoir) remains fixed after initialization.

An ESN consists of:

• Input layer: It connects the input vector to the reservoir. It can be scaled and randomly weighted.

• Reservoir: A large, sparsely connected pool of neurons with randomly initialized weights. It has nonlinear dynamics and captures temporal patterns via its internal echoes of past inputs.

• Output layer: A readout layer. Trained via simple regression (e.g., linear regression or ridge regression) to map reservoir states to target outputs.

The echo state property means that the internal state of the reservoir eventually becomes a function of the input history, not the initial conditions. This ensures the reservoir doesn't blow up or get stuck.

For this to happen, the spectral radius (largest absolute eigenvalue) of the reservoir weight matrix must be less than or around 1.

The way an ESN works is to randomly create reservoir with fixed internal weights (sparse, scaled), then randomly assign input weights. After that, it chooses spectral radius carefully to ensure stability.

It feeds input into network and records reservoir states at each time step. Finally, it uses recorded reservoir states and known outputs to solve a linear regression problem. For new input, it updates reservoir state and compute output using trained weights.

I recommend reading Jaeger, H. (2001). “The ‘Echo State’ Approach to Analysing and Training Recurrent Neural Networks” – foundational paper.

It's powerful for nonlinear, chaotic time series like the Mackey-Glass system.

The Mackey-Glass system is a classic time-delay differential equation used to model nonlinear, chaotic dynamics. It was originally proposed to simulate physiological processes, like blood production, but it's now widely used as a benchmark in time series prediction and nonlinear systems analysis. We will use it to create a time series.

Do you want to master Deep Learning techniques tailored for time series, trading, and market analysis🔥? My book breaks it all down from basic machine learning to complex multi-period LSTM forecasting while going through concepts such as fractional differentiation and forecasting thresholds. Get your copy here 📖!

---

Creating a Multi-Step Forecasting Algorithm

The plan of attack will be as follows:

1. Generate synthetic time series (Mackey-Glass).

2. Define a simple ESN.

3. Train it on past values to predict the next one.

4. Plot the prediction.

Use the following code to implement the experiment:

import numpy as np
import matplotlib.pyplot as plt
from ESN import ESN

# Mackey-Glass generator (same as before)
def mackey_glass(length=2000, tau=5, delta_t=1.0, beta=0.2, gamma=0.1, n=10):
    history = [1.2] * (tau + 1)
    for t in range(length):
        xtau = history[-tau] if len(history) > tau else 0
        x = history[-1]
        x_dot = beta * xtau / (1 + xtau ** n) - gamma * x
        history.append(x + delta_t * x_dot)
    return np.array(history[tau+1:])

# Generate data
data = mackey_glass()
data = (data - np.min(data)) / (np.max(data) - np.min(data))

# Split
trainlen = 500
testlen = 250
X_train = data[:trainlen]
Y_train = data[1:trainlen + 1]
X_test = data[trainlen:trainlen + testlen]

# Train ESN
esn = ESN(n_inputs=1, n_outputs=1, n_reservoir=1000, sparsity=0.1, spectral_radius=1.25, random_state=123)
esn.fit(X_train.reshape(-1, 1), Y_train.reshape(-1, 1))

# Recursive Multi-step Forecast
n_steps = 50  # how far to predict ahead
init_input = X_test[0]
predictions = []

for _ in range(n_steps):
    pred = esn.predict(np.array(init_input).reshape(1, 1))
    predictions.append(pred.item())  # get scalar
    init_input = pred  # feed back the prediction

# Ground truth for comparison
ground_truth = X_test[1:n_steps + 1]

# Plot
plt.figure(figsize=(10, 5))
plt.plot(range(n_steps), ground_truth, label="True Future")
plt.plot(range(n_steps), predictions, label="Multi-step Forecast")
plt.xlabel("Time steps into the future")
plt.ylabel("Normalized Value")
plt.legend()
plt.title("ESN Recursive Multi-step Forecasting")
plt.show()

The following is the result of the code:

It seems that the ESN needs more tuning in order to improve its forecasts. Hyperparameter turning is one of ML’s hidden arts. For now, it seems to just converge on the downside.

Here are a few limitations to consider for the ESN:

• Design choices are critical (e.g., reservoir size, spectral radius, input scaling).

• Reservoir is not trained, so representational capacity may be suboptimal.

• Hard to scale to very complex tasks (e.g., long-term dependencies or high-dimensional outputs).

• No backpropagation through time (which is both a pro and con).

Every week, I analyze positioning, sentiment, and market structure. Curious what hedge funds, retail, and smart money are doing each week? Then join hundreds of readers here in the Weekly Market Sentiment Report 📜 and stay ahead of the game through chart forecasts, sentiment analysis, volatility diagnosis, and seasonality charts.

Free trial available🆓