13 mins read

Logistic Regression

Logistic regression is a supervised machine learning algorithm used for analyzing datasets in which the outcome variable is categorical. It is effective in handling dichotomous outcomes, meaning those that have two possible outcomes, such as yes/no or true/false scenarios. The algorithm examines the relationship between one or more independent variables and classifies data into discrete categories. It’s widely used in predictive modeling to estimate the probability of an instance belonging to a specific category.

Logistic Regression was developed as an alternative to overcome the limitations of ordinary least squares (OLS) regression due to several key reasons discussed below.

Binary and Categorical Outcomes: OLS is designed for continuous outcomes, meaning it can predict a range of values (like 0.5, 1.5, etc.). However, many practical problems involve binary (0 or 1) or categorical outcomes (e.g., pass/fail, buy/not buy). Logistic regression is specifically designed for binary outcomes, providing probabilities that are bound between 0 and 1, which aligns with the nature of binary classification problems.

Non-linearity of the Response Function: OLS assumes a linear relationship between the independent variables and the dependent variable. However, in many real-world scenarios, the relationship between predictors and a binary outcome is not linear. On the other side, logistic regression uses the logistic (sigmoid) function to model the probability of the binary outcome. This function is inherently nonlinear and provides a more appropriate model for binary data.

Probabilistic Interpretation: OLS does not directly provide probabilities for classification problems. It predicts values that may not be valid probabilities (i.e., less than 0 or greater than 1). However, logistic regression outputs probabilities, offering a natural and interpretable way to express the likelihood of an outcome. This probabilistic output is crucial for many applications, such as risk assessment and decision-making processes.

Heteroscedasticity: OLS assumes homoscedasticity, meaning the variance of the errors is constant across all levels of the independent variables. In classification problems, this assumption is often violated because the variance of the errors can change with the level of the predictors. However, logistic regression does not assume homoscedasticity and is thus more robust in cases where the variance of the errors varies with the predictors.

Boundary Constraints: Predictions made by OLS can fall outside the 0 and 1 range, which is problematic for binary outcomes. However, logistic regression naturally constrains the predicted probabilities within the [0, 1] interval, ensuring meaningful and interpretable results.

Logistic regression uses the sigmoid function to map predictions to probabilities. This S-shaped curve converts any real value to a range between 0 and 1. If the sigmoid function’s output (estimated probability) exceeds a predefined threshold, the model predicts the instance belongs to that class; otherwise, it predicts it does not. For example, if the threshold is 0.5, outputs above 0.5 are classified as 1, and below 0.5 as 0. An output of 0.65 implies a 65% chance of the event occurring, like a coin toss.

The sigmoid function, also known as the activation function in logistic regression, is defined as:

e is the base of the natural logarithm, approximately equal to 2.71828. It’s a mathematical constant.

x is the input variable, which can be any real number. It represents the value for which the sigmoid function is being calculated.

The following equation defines logistic regression:

x is the input variable

y represents the predicted probability of the event occurring.

b0 is the intercept term.

b1 is the coefficient for the predictor variable X

e is the base of the natural logarithm, approximately equal to 2.71828.

This equation is similar to linear regression, where input values are combined linearly using weights or coefficients to predict an output value. However, unlike linear regression, which predicts a continuous numeric value, logistic regression models a binary output value (0 or 1).

Origin & History

The development of logistic regression has been shaped by significant contributions from various researchers over the past century. From Verhulst’s initial discovery of the logistic function to Berkson’s promotion of the logit model, and Cox’s introduction of the multinomial logit model, each advancement has built on the previous work, leading to the robust and widely used logistic regression we have today.

I. Logistic Function

The origins of logistic regression can be traced back to the early 19th century, specifically to the work of Pierre François Verhulst, a Belgian mathematician. In 1838, Verhulst introduced the logistic function under the name- “Correspondance mathématique et physique” and described the self-limiting growth of a biological population.

This function was characterized by an S-shaped curve, or sigmoid curve, which was crucial for understanding population growth processes. The logistic function modeled how populations grow rapidly at first but then slow down as they approach a carrying capacity, limited by resources and environmental factors (Verhulst, 1838).

In 1845, Verhulst published a more detailed version of logistic function: “Recherches mathématiques sur la loi d’accroissement de la population” further cementing its significance in modeling population dynamics.

II. Probit Model

Following Verhulst’s initial discovery of the logistic function, two significant advancements were the development of the probit model by Chester Ittner Bliss in 1934 and the introduction of maximum likelihood estimation (MLE) by Ronald Fisher in 1935.

The probit model was introduced as a method for analyzing binary response data. The term “probit” is derived from “probability unit,” and the model was developed as a way to handle situations where the response variable is binary, taking on one of two possible values (e.g., success/failure, yes/no).

Bliss’s application of the probit model allowed researchers to analyze dose-response relationships and understand the effects of various doses on outcomes, making it a powerful tool in biological and medical research.

III. Logit Model

In 1943, Wilson and Worcester applied the logistic model to bioassay, marking the first known use of the logistic model in this field. Bioassay involves measuring the effects of a substance on living organisms and often requires the analysis of binary outcomes, such as whether an organism responds to a treatment or not. The logistic model, with its ability to model binary data, was well-suited for this purpose and represented a significant advancement in bioassay analysis.

Joseph Berkson played a significant role in advancing the logistic model. He coined the term “logit,” drawing a parallel to the already established “probit” terminology. Despite initial resistance and the perception that the logit model was inferior to the probit model, the future eventually favored the widespread acceptance of the logit model.

Over time, the logit model gained acceptance due to its computational simplicity and practical advantages. Unlike the probit model, which uses the cumulative distribution function of the normal distribution, the logit model uses the logistic function, making it easier to compute and interpret. As statistical computing advanced, the logit model’s advantages became more apparent, leading to its widespread adoption in various fields.

IV. Multinomial Logit Model

In 1966, David Cox introduced the multinomial logit model, which extended the binary logistic regression to handle multiple categories. The multinomial logit model allowed for the analysis of outcomes with more than two categories, significantly broadening the scope of logistic regression applications. This development was a crucial step forward, enabling more complex modeling of categorical data.

V. Widespread Adoption

By the early 1980s, logistic regression had become widely accepted in statistical analysis across various fields. Its ability to model dichotomous outcomes effectively made it a robust alternative to OLS (Ordinary Least Squares) regression. Researchers appreciated its flexibility in handling multiple predictors and adjusting for confounders, which enhanced the accuracy and interpretability of their models. Logistic regression’s versatility led to its adoption in diverse disciplines, including medicine, social sciences, and economics, where it was used for tasks such as predicting disease outcomes, understanding behavioral responses, and credit scoring (Cox, 1983).

Construction of Logistic Regression Model

By following the steps outlined below, you can build a robust logistic regression model to predict binary outcomes, such as the likelihood of heart disease, based on various predictors. This approach ensures the model is well-prepared and accurately reflects the relationships within the data.

I. Import the Required Libraries

To begin constructing our logistic regression model, we need to import the necessary libraries. For this example, we will be using Pandas and NumPy to handle our data and scikit-learn for model construction and evaluation. Pandas is used to access and manipulate the dataset, while the read_csv function reads the data from a CSV file into a DataFrame. The head() function displays the first few records of the DataFrame to give a glimpse of the data.

import pandas as pd

import numpy as np

# Load the dataset
df = pd.read_csv("framingham_heart_disease.csv")
df.head()

II. Clean the Dataset

Data cleaning involves detecting and correcting errors or inconsistencies in the dataset. This includes handling missing values and dropping irrelevant features. Here, we drop columns that are not useful for our analysis (currentSmoker and education), fill in missing values for cigsPerDay with the mean value, and remove any remaining rows with missing values.

# Handling missing data
series = pd.isnull(df['cigsPerDay'])

# Dropping unwanted columns
data = df.drop(['currentSmoker', 'education'], axis='columns')

# Fill missing values
cigs = data['cigsPerDay']
cig = cigs.mean()
integer_value = np.floor(cig)
cigs.fillna(integer_value, inplace=True)
data.dropna(axis=0, inplace=True)

III. Analyze the Data Set

Once the data is clean, we can analyze it by creating separate DataFrames for different risk groups and performing feature engineering. We also need to scale the data to ensure the logistic regression model performs accurately. By separating the data into high and low-risk groups, we can better understand the distribution of features. Dropping features with similar values across outcome categories helps to refine the model.

# Analyzing the Dataset
Heart_Attack = data[data.TenYearCHD == 1]

No_Heart_Attack = data[data.TenYearCHD == 0]
final = data.drop(['diaBP', 'BMI', 'heartRate'], axis='columns')

IV. Prepare the Model

Next, we split the dataset into training and testing sets. This allows us to train the model on one subset of data and evaluate its performance on another.

rom sklearn.model_selection import train_test_split

# Split the data
X = final.drop('TenYearCHD', axis='columns')
y = final['TenYearCHD']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=99)

V. Train & Evaluate the Model

After preparing the data, we can initialize the logistic regression model, train it on the training set, and evaluate its performance. The logistic regression model is trained using the fit method, and its performance is evaluated using metrics such as accuracy, confusion matrix, classification report, and ROC AUC score. These metrics help to summarize the model’s predictive power and reliability.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve

# Train the model
model = LogisticRegression()
model.fit(X_train, y_train)

# Evaluate the model
accuracy = model.score(X_test, y_test)
y_pred = model.predict(X_test)
conf_matrix = confusion_matrix(y_test, y_pred)
class_report = classification_report(y_test, y_pred)
y_pred_prob = model.predict_proba(X_test)[:, 1]
roc_auc = roc_auc_score(y_test, y_pred_prob)

print(f"Model Accuracy: {accuracy * 100:.2f}%")
print("Confusion Matrix:\n", conf_matrix)
print("\nClassification Report:\n", class_report)
print(f"ROC AUC Score: {roc_auc:.2f}")

Types of Logistic Regression

There are three types of logistic regression models, which are defined based on categorical responses.

I. Binary Logistic Regression

In binary logistic regression, the response or dependent variable is dichotomous, meaning it has only two possible outcomes (e.g., 0 or 1). Common examples include predicting whether an email is spam or not spam or whether a tumor is malignant or not malignant. This approach is the most commonly used form of logistic regression and is widely recognized as one of the most prevalent classifiers for binary classification tasks.

II. Multinomial Logistic Regression

In the Multinomial Logistic Regression model, the dependent variable has three or more possible outcomes, with no specified order among them. For example, movie studios may want to predict which genre of film a moviegoer is likely to see to market films more effectively. A multinomial logistic regression model can help determine the influence of a person’s age, gender, and dating status on their film preferences. The studio can then target an advertising campaign for a specific movie towards the group of people most likely to watch it.

III. Ordinal Logistic Regression

Ordinal logistic regression is used when the response variable has three or more possible outcomes in a natural order. Unlike multinomial logistic regression, which deals with unordered categories, ordinal logistic regression accounts for the inherent ranking among the outcomes. For example, consider a grading scale from A to F. Each grade represents a distinct level of performance, with A being the highest and F the lowest. Similarly, a rating scale from 1 to 5 reflects different satisfaction levels, with 1 being very dissatisfied and 5 being very satisfied.

Advantages of Logistic Regression

Logistic regression offers a range of advantages that make it a popular choice for binary classification problems. Its simplicity, transparency, computational efficiency, extensive libraries, and ability to provide probabilistic predictions make it valuable across various domains.

I. Simplicity

Logistic Regression is straightforward and easy to understand. It’s an extension of Linear Regression, which is one of the most basic predictive algorithms. Linear Regression predicts a continuous outcome (like predicting someone’s height based on their age). Logistic Regression, on the other hand, predicts a binary outcome (like predicting whether an email is spam or not). Moreover, logistic regression involves fewer hyperparameters compared to more complex models like neural networks or ensemble methods. This simplicity reduces the complexity of the implementation process, as there is less need for extensive hyperparameter tuning. For users new to machine learning, fewer hyperparameters mean fewer potential points of failure and confusion.

II. Transparency

Logistic Regression is transparent because it lets you view how each input (feature) contributes to the final prediction. For example, in predicting if an email is spam, you can see how much the presence of certain words influences the outcome. Moreover, the model assigns a weight to each input feature, which tells us how important that feature is in making the prediction. If a certain word significantly increases the chance of an email being spam, the model will clearly reflect this.

III. Extensive Libraries

There are many well-maintained libraries and frameworks {such as scikit-learn in Python, Statsmodels (Python), glm (R), and caret (R)} that provide built-in functions to implement logistic regression. These libraries are collections of pre-written code that make it easy to implement Logistic Regression without needing to write all the complex math and algorithms yourself, enabling users to focus on more critical aspects of their work, such as feature engineering, data preprocessing, and model evaluation.

IV. Versatility

Logistic Regression’s ability to handle both categorical and continuous variables makes it a versatile and practical algorithm. It allows you to create a more complete and accurate model by incorporating all relevant information, whether categorical or continuous. For example, in medical studies, predicting whether a patient has a disease might involve categorical variables (like gender and smoking status) and continuous variables (like age and blood pressure).

V. Multiple Variable

Logistic Regression can look at several factors (variables) at the same time to understand how they all contribute to the outcome you’re trying to predict. This is important because real-world scenarios usually involve many factors that interact with each other. Instead of looking at each factor separately, you can see how they all work together to influence the outcome. This gives a more comprehensive understanding.

VI. Computational Efficiency

Logistic regression models a linear relationship between the features and the log-odds of the target variable. This simplicity means that the computational complexity is relatively low compared to more complex models like neural networks. Moreover, one of the most commonly used optimization algorithms for logistic regression is gradient descent, which iteratively updates the model parameters. The number of computations per iteration is linear to the number of features and observations, making each step computationally inexpensive.

Disadvantages of Logistic Regression

While logistic regression has several advantages, it also comes with its share of disadvantages. Understanding these limitations is crucial for making informed decisions about when and how to use logistic regression.

I. Assumption of Linearity

Logistic Regression assumes a linear relationship between the independent variables (predictors) and the log odds of the dependent variable (outcome). This means that the model presumes that each predictor has a linear and additive effect on the outcome in the log-odds space. However, this assumption is often unrealistic in real-world scenarios where relationships between variables are usually more complex and non-linear. For example, in predicting disease risk, the relationship between age and risk might be non-linear, where risk increases rapidly after a certain age rather than steadily over time. Logistic Regression would struggle to model this correctly.

II. Sensitivity to Outliers

Logistic Regression is sensitive to outliers, meaning that unusual or extreme values in the data can disproportionately affect the model’s performance. Outliers are data points that differ significantly from the majority of the data. This model estimates the parameters (coefficients) of the model based on the input data. Outliers can heavily influence these estimates, leading to skewed results. For example, if you are predicting the likelihood of a disease based on age and one of the patients has an age far outside the typical range (e.g., a newborn in a dataset of adults), this outlier can significantly affect the model’s coefficients for age.

III. Inability to Capture Complex Patterns

Logistic Regression is a relatively simple algorithm that works well for problems with linear relationships. However, it struggles with complex patterns and interactions in the data. More powerful and complex algorithms like neural networks can model these intricate relationships much more effectively, often leading to significantly better performance. For example in image recognition, the relationship between pixel values and the object being identified is highly non-linear. Logistic Regression would fail to accurately classify images because it cannot model these complex patterns.

IV. P > N Problem

The “p > n” problem, where p represents the number of features (predictors) and n represents the number of observations (samples), arises when the dataset has more features than observations. In this case, Logistic Regression should not be used, otherwise, it may lead to overfitting. Logistic regression fails in the “p > n” scenario because there are more parameters to estimate than there are data points, leading to an overfitted model that captures noise rather than the true signal. Furthermore, when p > n, logistic regression coefficients can become very large and unstable. Small changes in the data can lead to significant changes in the estimated coefficients, making the model unreliable and difficult to interpret.

Logistic Regression in Trading

Logistic regression is a statistical method widely used for binary classification problems. In trading, logistic regression can be employed to predict the probability of a specific event, such as the direction of a stock price movement (up or down) in the next trading period. This method can be particularly useful for developing trading strategies based on probabilistic forecasts.

The paper by Jibing Gong and Shengtao Sun explains how the Logistic Regression Model is effective and efficient in predicting stock price trends- (Gong & Sun, 2009). Data from the RESSET Financial Research Database was used, specifically the stock integrated index data from 2005 to 2007. The model was trained using data from the full year of 2005, with iteration operations repeated until the regression coefficients were generated.

The financial data from 2006 was used to determine the final regression coefficients from nine candidate groups. This selection process involved adjusting the threshold to 0.5 for better price trend prediction. The model was built using the selected optimizing regression coefficients and was tested with the stock price trend data of 2007. The model’s input was the average value of all feature index variables for the current month, and the output was the probability representing the stock price trend.

The experiments showed that the prediction accuracy of the proposed model reached at least 83%, which was found to be comparable to or better than other methods such as the Radial Basis Function Artificial Neural Network (RBF-ANN) model. The proposed model was noted for its lower complexity and higher efficiency.

Another study by Bahtiar Jamili Zaini, Rosnalini Mansor, Norhayati Yusof, and Beh Hui Sang (Zaini, Mansor, Yusof, & Sang 2019) focuses on predicting stock market movement using logistic regression based on technical analysis indicators. This study utilizes daily stock data from one company listed in Bursa Malaysia (formerly Kuala Lumpur Stock Exchange) over eight months to build and validate a logistic regression model. In this study, seven technical indicators were used as predictor variables in the logistic regression model:

Moving Average Convergence Divergence (MACD): A leading indicator that suggests a buy signal when the MACD line crosses above the signal line or the zero line, and a sell signal when the MACD line crosses below these lines.
Relative Strength Index (RSI): Measures the speed and change of price movements. RSI values range between 0 and 100, with values above 70 considered overbought and below 30 considered oversold.
Stochastic Oscillator (SO): Indicates the current price relative to its price range over a set number of periods. SO values above 80 are considered overbought, and values below 20 are considered oversold.
Moving Average (MA): The average price of a stock over a specified number of periods. It helps identify trend directions and support/resistance levels.
Exponential Moving Average (EMA): Similar to MA but gives more weight to recent prices to reduce lag.
Rate of Change (ROC): Measures the percentage change in stock price over a given period. ROC values greater than zero indicate an uptrend, while values less than zero indicate a downtrend.
Volume Trading (VT): Indicates the amount of a financial instrument traded over a specific period. High volume suggests high interest at the current price, while low volume can signal potential price changes.

The logistic regression model used in this study predicts the probability of stock movement, classified as either an uptrend or downtrend. The initial logistic regression model included all seven technical indicators. The significance of each predictor was assessed using the Wald statistic.

Insignificant predictors were removed, resulting in a final model with four significant indicators: MACD, RSI, SO, and ROC. The model demonstrated good predictive accuracy with an in-sample classification rate of 86% and an out-of-sample rate of 71.43%.

This logistic regression approach provides a valuable tool for investors to make informed decisions based on technical analysis indicators. The study suggests that future work could explore other methods such as fuzzy logic, discriminant analysis, and multiple regression to further improve forecasting accuracy.

The Bottom Line

Logistic regression is a powerful supervised machine learning algorithm that addresses the limitations of Ordinary Least Squares (OLS) regression when dealing with binary or categorical outcome variables. By leveraging the logistic (sigmoid) function, it effectively models the probability of a binary event, ensuring predicted probabilities are bound between 0 and 1.

This makes it particularly suitable for scenarios requiring a clear yes/no or true/false decision, such as medical diagnoses, financial predictions, and policy adherence. Despite its simplicity and computational efficiency, logistic regression provides robust probabilistic interpretations and handles both categorical and continuous variables, making it a versatile and essential tool in various fields of research and industry.