15 mins read

Dimensionality Reduction

Dimensionality reduction is a technique used to represent a dataset with fewer features (i.e., dimensions) while retaining its essential properties (Den, 2022). This process involves removing irrelevant, redundant, or noisy data to create a more concise model with fewer variables.

Dimensionality reduction covers various feature selection and data compression methods employed during preprocessing. Despite differing operational approaches, these methods all aim to transform high-dimensional spaces into lower-dimensional ones through variable extraction or combination, thereby simplifying the dataset while preserving its meaningful information.

Machine learning is affected by the curse of dimensionality, which describes how increasing the number of input variables (dimensions) in a model can reduce its generalizability. As dimensions increase, the feature space becomes more extensive, but if the number of data points stays the same, the data becomes sparse, leaving most of the space empty.

This sparsity makes it harder for models to identify patterns, leading to less effective predictions. To explain patterns in sparse data, models may overfit on training data, reducing their ability to generalize. Additionally, high dimensionality can cause multicollinearity, where variables are redundant or correlated, making the model harder to interpret.

To combat this, collecting more data can reduce sparsity, but the amount needed increases exponentially with the number of dimensions, which is often impractical. Therefore, dimensionality reduction techniques are essential to improve data analysis by reducing the number of features while preserving important information.

Dimensionality reduction is crucial for several reasons:

Visualization: High-dimensional data is difficult to visualize. By reducing dimensionality, we can create scatter plots or other visualizations, making it easier to gain insights and understand patterns within the data.

Computational Efficiency: Machine learning algorithms often perform poorly with a high number of features due to the curse of dimensionality. Reducing dimensionality decreases computational complexity and accelerates the training process, making algorithms more efficient.

Noise Reduction: High-dimensional data typically contains noise or irrelevant features. Dimensionality reduction helps filter out these noisy variables, allowing the model to focus on the most meaningful features, thereby improving model performance and accuracy.

Interpretability: Reduced dimensionality enhances the interpretability of the data. It simplifies the relationships between variables, making it easier for analysts to understand and communicate the underlying patterns and insights derived from the data.

Origin & History

The evolution of dimensionality reduction has been driven by the exponential growth of data in various fields, such as biology, image processing, and text analysis. The increasing complexity and volume of data necessitated the development of methods to efficiently reduce dimensions while maintaining the integrity of the original information.

I. Early Concepts

The roots of dimensionality reduction can be traced back to statistical analysis techniques developed in the early 20th century, a period that witnessed the emergence of foundational methods for managing and interpreting complex datasets. One of the earliest and most influential techniques in this field is Principal Component Analysis (PCA). Introduced in the early 1900s, PCA was grounded in the pioneering work of (Karl Pearson, 1901), who laid its mathematical foundation in 1901.

Pearson’s objective with PCA was to address the challenge of high-dimensional data by reducing its dimensionality. The method transforms a set of correlated variables into a set of linearly uncorrelated variables, known as principal components. These principal components are ordered by the amount of variance they capture from the original data, with the first few components retaining the most significant patterns. This transformation simplifies the dataset and preserves its essential structure, making it easier to analyze and visualize.

PCA’s ability to reduce complexity while maintaining critical information made it a cornerstone technique in multivariate analysis. Its applications have since expanded across various fields, including economics, biology, and engineering, where it helps uncover underlying patterns, compress data, and improve the efficiency of other analytical processes. The development of PCA marked a significant milestone in the history of dimensionality reduction, setting the stage for subsequent advancements in the field.

II. Advancements in the 20th Century

Throughout the 20th century, various statistical techniques evolved that significantly contributed to the field of dimensionality reduction. Each of these methods brought unique approaches to simplifying high-dimensional data while preserving its essential characteristics and uncovering underlying structures.

For example, Factor Analysis emerged in the 1930s and 1940s as a method to identify underlying relationships between variables. This technique works by reducing the number of observed variables into fewer latent factors, which represent the underlying dimensions of the data. Factor Analysis has been widely used in fields such as psychology and social sciences to identify and interpret hidden constructs within complex datasets.

Linear Discriminant Analysis (LDA), introduced by Ronald A. Fisher in 1936, was one of the earliest techniques to address dimensionality reduction while maintaining class separability. LDA aims to project data onto a lower-dimensional space so that the separation between different classes is maximized. This method has been particularly influential in pattern recognition and classification tasks, enhancing model performance by focusing on the most discriminative features.

The 1980s and 1990s saw significant advancements in dimensionality reduction techniques, with methods such as Singular Value Decomposition (SVD) and Independent Component Analysis (ICA) gaining prominence. SVD, a mathematical technique that decomposes a matrix into its constituent components, has become a fundamental tool in linear algebra and data processing. It is widely used in signal processing, image compression, and recommendation systems.

Independent Component Analysis (ICA), developed in the 1990s, offers a way to decompose a multivariate signal into additive, independent non-Gaussian components. This method has been particularly useful in fields such as neuroscience for separating mixed signals, like brain activity patterns captured through EEG or fMRI.

These advancements expanded the toolkit available for handling high-dimensional data, providing researchers and data scientists with powerful methods to decompose and analyze complex datasets. Each technique contributed to the evolution of dimensionality reduction, enhancing our ability to manage and interpret large-scale data across various scientific and engineering domains.

III. Emergence of Nonlinear Techniques

As data analysis needs grew more complex, especially with the advent of computer technology, researchers began exploring nonlinear dimensionality reduction techniques to better handle and interpret the increasingly vast and intricate datasets. Traditional linear methods like PCA were often insufficient for capturing the underlying structures in high-dimensional data, leading to the development of more advanced techniques.

Significant progress was made in this area in the late 1990s and early 2000s. One of the pioneering methods developed during this period was “Locally Linear Embedding (LLE)” by Sam T. Roweis and Lawrence K. Saul in 2000. LLE is designed to preserve the local relationships between data points by mapping high-dimensional data into a lower-dimensional space. This technique helps visualize and understand complex structures within the data, making it easier to identify patterns and clusters that are not evident in the original high-dimensional form.

Another groundbreaking method, “T-distributed Stochastic Neighbor Embedding (t-SNE),” was introduced by Laurens van der Maaten and Geoffrey Hinton in 2008. t-SNE focuses on minimizing the divergence between probability distributions representing pairwise similarities in high-dimensional and low-dimensional spaces. This approach allows for effective visualization of data, particularly useful for exploring and visualizing high-dimensional datasets in a way that preserves local structures and reveals clusters in the data.

These nonlinear dimensionality reduction techniques have become essential tools in modern data analysis, enabling researchers and data scientists to visualize and interpret high-dimensional data more effectively, leading to better insights and more informed decision-making.

IV. Integration with Machine Learning

The rise of machine learning in the 2000s further popularized dimensionality reduction techniques. As datasets became larger and more complex, dimensionality reduction became essential for improving the efficiency and performance of machine learning algorithms. Techniques like Linear Discriminant Analysis (LDA) were adapted to not only reduce dimensions but also enhance class separability in classification tasks.

Some common techniques for integrating dimensionality reduction with machine learning include:

Principal Component Analysis (PCA): PCA is a linear dimensionality reduction technique that projects the data onto a lower-dimensional space while preserving the maximum variance. It is often used as a preprocessing step for machine learning models.

Linear Discriminant Analysis (LDA): LDA is a supervised dimensionality reduction technique that finds the linear combination of features that best separates different classes. It is commonly used for classification tasks.

Autoencoders: Autoencoders are a type of neural network that learns a compressed representation of the input data. They can be used for dimensionality reduction and are often integrated with other machine learning models.

t-SNE: t-SNE is a non-linear dimensionality reduction technique that is particularly useful for visualizing high-dimensional data. It can be used in conjunction with machine learning models for exploratory data analysis.

Feature Selection in Dimensionality Reduction

Dimensionality reduction can be broadly categorized into two main approaches: feature selection and feature extraction. Both approaches aim to reduce the number of features in a dataset while retaining its essential characteristics, but they do so in different ways.

I. Feature Selection

Feature selection involves selecting a subset of the original features that are most relevant to the problem at hand. The goal is to reduce the dimensionality of the dataset while retaining the most important features. By focusing on the most relevant features, feature selection helps simplify the model, reduce overfitting, and improve model interpretability. There are several methods for feature selection:

Filter Methods

Filter methods evaluate the relevance of each feature by examining its intrinsic properties and its statistical relationship with the target variable. These methods operate independently of any specific machine learning algorithm, making them computationally efficient and easy to implement. Common filter methods include:

Correlation Coefficients: Measures the linear relationship between each feature and the target variable, selecting features with the highest correlations.

Chi-Square Tests: Evaluate the independence of categorical features from the target variable, selecting features with the most significant relationships.

Mutual Information: Assesses the amount of information shared between each feature and the target variable, selecting features that contribute the most to predicting the target.

Wrapper Methods

Wrapper methods evaluate subsets of features by training and evaluating a model on each subset. These methods use performance metrics, such as accuracy, precision, recall, or F1-score, to determine the best subset of features. While potentially more accurate than filter methods, wrapper methods are more computationally expensive due to the repeated training and evaluation of the model. Common wrapper methods include:

Recursive Feature Elimination (RFE): Iteratively removes the least important features based on model performance, refining the feature subset until the optimal number is reached.

Forward Selection: Starts with an empty feature set and adds features one by one, evaluating the model’s performance at each step until the addition of further features no longer improves performance.

Backward Elimination: Starts with all features and removes them one by one, evaluating the model’s performance at each step until the removal of further features no longer improves performance.

Embedded Methods

Embedded methods integrate feature selection into the model training process. These methods select relevant features by applying a penalty to the coefficients of less important features during model training, balancing computational efficiency and model performance. Common embedded methods include:

Lasso Regression (L1 Regularization): Adds a penalty equivalent to the absolute value of the magnitude of coefficients, shrinking some coefficients to zero and effectively performing feature selection.

Ridge Regression (L2 Regularization): Adds a penalty equivalent to the square of the magnitude of coefficients, shrinking coefficients but generally retaining all features, which helps in reducing multicollinearity.

Elastic Net: Combines both L1 and L2 regularization penalties, selecting features while also maintaining some degree of shrinkage to handle correlated predictors.

II. Feature Projection

Feature projection, also known as feature extraction, is a technique used to transform data from a high-dimensional space to a lower-dimensional space. This transformation aims to retain the essential features of the data while reducing its complexity. Feature projection can be achieved through both linear and non-linear methods.

Linear Transformations

Principal Component Analysis (PCA): PCA is a widely used linear technique that transforms the original features into a new set of uncorrelated features called principal components. These components are ordered by the amount of variance they capture from the data, with the first few components retaining most of the data’s variability. PCA is particularly useful for simplifying data and reducing dimensionality while preserving as much information as possible.

Linear Discriminant Analysis (LDA): LDA is another linear technique primarily used for classification tasks. It transforms the data by finding the linear combinations of features that best separate different classes. Unlike PCA, which focuses on maximizing variance, LDA aims to maximize the separability between classes. This makes LDA especially useful for tasks where class discrimination is crucial.

Non-Linear Transformations

T-Distributed Stochastic Neighbor Embedding (t-SNE): t-SNE is a popular non-linear dimensionality reduction technique designed for data visualization. It transforms high-dimensional data into a lower-dimensional space (typically 2 or 3 dimensions) while preserving the local structure of the data. t-SNE calculates the probability that pairs of points are similar in high-dimensional space and then tries to preserve these probabilities in the lower-dimensional space. This method is highly effective for visualizing clusters and patterns in complex datasets.

Methods of Dimensionality Reduction

I. Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is one of the most widely used dimensionality reduction methods. It is a type of feature extraction that combines and transforms the original features of a dataset to create new features, known as principal components. PCA identifies a subset of variables that capture most of the variance in the original dataset and projects the data onto a new space defined by these principal components (Jolliffe, 2002).

For example, consider a dataset about snakes with five variables: body length (X1), body diameter at the widest point (X2), fang length (X3), weight (X4), and age (X5). Some of these features, like body length, diameter, and weight, may be correlated. This redundancy can lead to overfitting and decreased generalizability of the model.

PCA addresses this by calculating a new variable (PC1) that combines multiple original variables and maximizes the variance in the data. By doing this, PCA reduces the number of variables in the model, making it more efficient and less prone to overfitting.

In this example, PCA could reduce the five original variables to a smaller set of new variables (principal components). For instance, starting with five dimensions, the reduced model might have anywhere from one to four dimensions. The data is then mapped onto this new, lower-dimensional space.

The new principal component (PC1) is not one of the original variables but a combined feature created through a linear transformation of the original data’s covariance matrix. Specifically, PC1 is the eigenvector corresponding to the largest eigenvalue in the covariance matrix. Additional principal components can be created by finding eigenvectors corresponding to the next largest eigenvalues (PC2, PC3, etc.), each capturing progressively less variance.

II. Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis (LDA) is a dimensionality reduction technique similar to Principal Component Analysis (PCA). It projects data onto a new, lower-dimensional space derived from the original features. However, LDA differs from PCA by focusing on retaining the classification labels in the dataset. While PCA aims to maximize data variance in the new components, LDA aims to maximize class differences.

LDA works through the following steps:

Calculate the Scatter Matrix: Unlike PCA, which uses the covariance matrix, LDA uses the scatter matrix to measure the spread of different classes in the data.

Compute Linear Combinations: LDA finds linear combinations of the original features that correspond to the largest eigenvalues from the scatter matrix.

Maximize Class Separation: The goal is to maximize the difference between classes (interclass difference) while minimizing the difference within each class (intraclass difference).

III. t-Distributed Stochastic Neighbor Embedding (t-SNE)

LDA (Linear Discriminant Analysis) and PCA (Principal Component Analysis) are linear dimensionality reduction algorithms. In contrast, t-distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear dimensionality reduction technique, also known as manifold learning.

LDA and PCA aim to preserve the overall variance in the data, focusing on retaining distances between dissimilar data points in their lower-dimensional representations. They can reduce the dataset to any number of dimensions as long as it’s fewer than the original number of features.

On the other hand, t-SNE aims to preserve the local structure of the data, meaning it maintains the relationships between closely situated points in the high-dimensional space. Unlike LDA and PCA, which can produce models with more than three dimensions, t-SNE visualizes all datasets in either two or three dimensions.

t-SNE does not use traditional data matrices. Instead, it utilizes a Gaussian kernel to calculate the pairwise similarity of data points. Points that are close to each other in the original dataset have a higher probability of being close in the lower-dimensional space. t-SNE then maps all the data points onto a two- or three-dimensional space, attempting to preserve the local structure and relationships between data pairs.

Advantages of Dimensionality Reduction

Dimensionality reduction offers several benefits, including data compression, which reduces storage space and computational efficiency by decreasing processing time. It eliminates redundant features, improves dataset quality, and enables the use of machine learning algorithms that struggle with high-dimensional data. Additionally, it addresses multicollinearity, enhancing model stability and filtering out noise, leading to more accurate and reliable predictive models.

I. Data Compression

Dimensionality reduction compresses data by transforming it into a lower-dimensional space, significantly reducing storage space requirements. This is especially beneficial for large datasets, where storage costs and data management can become substantial. Efficient data compression ensures that essential information is preserved while reducing the overall dataset size.

II. Computation Efficiency

Reducing the number of features in a dataset decreases computational complexity, resulting in faster processing times. This efficiency gain allows for quicker data analysis, training of machine learning models, and overall data processing. With fewer dimensions, algorithms can run more swiftly, enabling faster insights and decisions.

III. Removal of Redundant Features

Dimensionality reduction identifies and removes redundant features that do not contribute significant value to the dataset. For example, if measurements are stored in both meters and inches, one can be eliminated. This streamlines the dataset, making it cleaner and more manageable, and enhances the model’s performance by focusing on the most relevant features.

IV. Algorithm Compatibility

Many machine learning algorithms perform poorly with high-dimensional data due to increased complexity. By reducing dimensions, these algorithms become more feasible and effective. Dimensionality reduction allows the use of simpler, more efficient algorithms that would otherwise be impractical with a large number of features, improving model accuracy and performance.

V. Handling Multicollinearity

Multicollinearity occurs when features in a dataset are highly correlated, leading to unreliable model estimates. Dimensionality reduction mitigates this issue by removing correlated and redundant features, thus enhancing the model’s predictive power and stability. This ensures that the model’s predictions are more accurate and interpretable.

VI. Noise Reduction

High-dimensional data often contains noise or irrelevant features that can obscure important patterns. Dimensionality reduction filters out these noisy variables, improving the signal-to-noise ratio. This results in better model performance, as the models can focus on the most significant features, leading to more accurate and reliable predictive modeling.

Disadvantages of Dimensionality Reduction

Dimensionality reduction has drawbacks, such as potential data loss, which can reduce model accuracy. Techniques like PCA may only capture linear relationships, inadequately representing complex datasets. Relying on mean and covariance can be insufficient for some data structures, and determining the optimal number of components is challenging, often requiring heuristic methods.

I. Data Loss

Dimensionality reduction techniques, such as PCA, reduce the number of features by combining or discarding some information. This process inevitably leads to the loss of some data, which may affect the model’s ability to make accurate predictions or identify nuanced patterns within the dataset.

II. Linear Correlations with PCA:

PCA is designed to capture linear relationships between variables. However, many real-world datasets exhibit non-linear relationships, making PCA less effective. For these datasets, PCA might not capture the essential characteristics, leading to suboptimal model performance.

III. Inadequacy of Mean & Covariance

PCA uses mean and covariance to capture the structure of the data. In cases where the data distribution is complex and cannot be adequately described by these measures alone, PCA may fail. This limitation can result in a loss of important information and a reduction in the effectiveness of the dimensionality reduction.

IV. Determining Principal Components

Selecting the appropriate number of principal components to retain is not straightforward. While there are rules of thumb and techniques to estimate the number, these are not foolproof. Retaining too few components may lead to significant information loss, while retaining too many may not simplify the dataset as intended, reducing the benefits of dimensionality reduction.

Dimensionality Reduction in Trading

Dimensionality reduction is a vital tool in the arsenal of quantitative traders and financial analysts. It simplifies complex datasets, enhances computational efficiency, improves the accuracy of predictive models, and provides clearer insights through visualization. By applying techniques such as PCA, LDA, t-SNE, FRPCA, and KPCA, traders can gain a deeper understanding of market dynamics and make more informed, data-driven trading decisions.

Case Study I

In the study “DNN models based on dimensionality reduction for stock trading” by Dongdong Lv, Dong Wang, Meizi Li, and Yang Xiang, the effectiveness of dimensionality reduction operations (DRO) in enhancing the performance of deep neural network (DNN) algorithms for stock trading was analyzed. The research focused on large-scale stock datasets from the American and Chinese markets. The primary aim was to determine whether dimensionality reduction could improve trading performance when using DNNs to generate trading signals.

The study utilized four common dimensionality reduction techniques: Principal Component Analysis (PCA), Least Absolute Shrinkage and Selection Operator (LASSO), Classification and Regression Trees (CART), and Autoencoder (AE). These techniques were applied to preprocess the original high-dimensional feature sets. The new, reduced-dimensional feature sets were then used as inputs for six popular DNN algorithms: Multilayer Perceptron (MLP), Deep Belief Network (DBN), Stacked Autoencoders (SAE), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU).

The study conducted extensive back-testing using daily trading signals generated by these DNN models. The trading performance was evaluated using several indicators: Winning Ratio (WR), Annualized Return Rate (ARR), Annualized Sharpe Ratio (ASR), Maximum Drawdown (MDD), and Average Training Time (ATT).

Key Findings

LASSO’s Performance: The study found that LASSO significantly improved the performance of RNN, GRU, and LSTM models. For example, in the S&P 500 index component stocks (SPICS), LASSO enhanced the ASR of RNN by 63.49% and reduced its MDD by 62.68%. In the CSI 300 index component stocks (CSICS), LASSO increased the ASR of GRU by 13.06% and decreased its MDD by 11.70%. This improvement is attributed to LASSO’s ability to effectively select relevant features by forcing the coefficients of less important variables to zero, leveraging the label information for better prediction.

Effectiveness of PCA, CART, and AE: PCA and CART did not significantly improve the performance of most DNN models. In some cases, they even degraded performance, potentially due to information loss (PCA) or overfitting (CART). AE’s impact on trading performance was generally neutral, maintaining most of the information on the original features but not significantly enhancing performance.

Model-Specific Insights: The MLP, DBN, and SAE models showed no significant performance improvement with any of the DRO methods. This lack of improvement may be due to these models’ inherent ability to handle high-dimensional data effectively or the nature of the features used.

Execution Speed: The study noted that while some DRO methods, particularly CART, could accelerate the execution speed of DNN algorithms, none of the DRO methods significantly enhanced the average training time (ATT) for any of the DNN models.

Conclusion

The research concluded that while certain dimensionality reduction techniques like LASSO can significantly enhance the trading performance of specific DNN models (notably RNN, GRU, and LSTM), other methods such as PCA, CART, and AE might not have the same effect and can sometimes negatively impact performance. This highlights the importance of carefully selecting the appropriate dimensionality reduction method based on the specific DNN model and application context in stock trading.

Case Study II

Forecasting stock market movements is a complex task influenced by numerous interrelated factors, including economic variables, industry-specific metrics, company-specific information, psychological factors, and political events. The study by Xiao Zhonga and David Enke, “Forecasting daily stock market return using dimensionality reduction,” forecasts the daily direction of the S&P 500 Index ETF (SPY) using 60 financial and economic features through three dimensionality reduction techniques: Principal Component Analysis (PCA), Fuzzy Robust Principal Component Analysis (FRPCA), and Kernel-based Principal Component Analysis (KPCA). The data spans 2518 trading days from June 1, 2003, to May 31, 2013, and includes various financial and economic indicators.

Dimensionality Reduction Techniques

Principal Component Analysis (PCA): PCA transforms the data into a set of linearly uncorrelated variables known as principal components. The study found that 17 principal components could explain almost 100% of the data variance. Eleven principal components could explain 99% of the data variance, emphasizing the importance of exchange rates, SPY returns, and returns of major indices.

Fuzzy Robust Principal Component Analysis (FRPCA): FRPCA, developed by Yang and Wang (1999), enhances PCA’s robustness by handling outliers more effectively. The study applied the New Nonlinear FRPCA3 algorithm to the cleaned data to generate principal components used for classification.

Kernel-based Principal Component Analysis (KPCA): KPCA uses kernel methods to project data into a high-dimensional space, making it easier to find linear separations. The polynomial kernel was used in this study, with parameters chosen to balance computational cost and accuracy.

Data Analysis and Model Development

The ANN classifier was employed to predict the daily direction of SPY using the transformed datasets from the three-dimensionality reduction techniques. The classifier’s performance was evaluated using the Mean Squared Error (MSE) and confusion matrix. The study used a three-layer feed-forward neural network with a scaled conjugate gradient backpropagation algorithm optimized through trial and error.

Results

The performance of the ANN classifiers was compared across the dimensionality reduction methods. The key findings include:

PCA combined with ANN provided slightly higher classification accuracy than FRPCA and KPCA.
Trading strategies based on PCA and ANN predictions yielded significantly higher risk-adjusted profits than benchmarks and were slightly better than strategies based on FRPCA and KPCA models.

Conclusion

The study demonstrated that combining dimensionality reduction techniques with ANN classifiers can enhance the accuracy of daily stock market return forecasts. Among the techniques tested, PCA showed the most promise in terms of classification accuracy and trading strategy performance. This research underscores the importance of selecting appropriate dimensionality reduction methods to improve the efficiency and accuracy of financial time series forecasting.

The Bottom Line

Dimensionality reduction is a critical technique in data science and machine learning, particularly in the field of trading. Transforming high-dimensional data into a more manageable form enhances computational efficiency, improves model performance, and facilitates better data visualization. Techniques such as PCA, LDA, t-SNE, FRPCA, and KPCA play pivotal roles in filtering out noise, reducing redundancy, and highlighting the most significant features. In the context of trading, dimensionality reduction is particularly valuable for managing large volumes of financial data, reducing noise, and focusing on the most impactful variables.