EDA

-

 EDA 


1. Descriptive Statistics

  • Mean, Median, Mode: Central tendency measures.

  • Variance, Standard Deviation: Spread of the data.

  • Range, Interquartile Range (IQR): Range and spread of data excluding outliers.

2. Data Visualization

  • Histograms: Distribute data across bins to show frequency.

  • Box Plots: Display distribution, central value, and variability.

  • Scatter Plots: Show relationships between two variables.

  • Heatmaps: Visualize correlations between variables.

3. Missing Value Analysis

  • Identification: Detect missing values in the dataset.

  • Imputation: Handle missing values using mean, median, mode, or more advanced techniques.

4. Outlier Detection

  • Box Plots: Identify outliers using IQR.

  • Z-Score: Determine if a data point is significantly different from the mean.

5. Correlation Analysis

  • Correlation Matrix: Check the strength and direction of relationships between pairs of variables.

  • Heatmaps: Visualize correlations.

6. Distribution Analysis

  • Normality Tests: Check if data follows a normal distribution.

  • Skewness and Kurtosis: Measure asymmetry and peakedness of distribution.

7. Feature Engineering

  • Transformation: Apply log, square root, or other transformations to stabilize variance.

  • Encoding: Convert categorical variables into numerical format.



8. Univariate Analysis

  • Objective: Understand the distribution and characteristics of each variable individually.

  • Techniques: Histograms, box plots, density plots.

9. Bivariate Analysis

  • Objective: Explore relationships between two variables.

  • Techniques: Scatter plots, correlation coefficients, and cross-tabulations.

10. Multivariate Analysis

  • Objective: Analyze more than two variables simultaneously to understand complex relationships.

  • Techniques: Pair plots, correlation matrices, and 3D scatter plots.

11. Time Series Analysis

  • Objective: Explore temporal patterns and trends in time series data.

  • Techniques: Line plots, moving averages, seasonal decomposition.

12. Dimensionality Reduction

  • Objective: Reduce the number of variables while preserving as much information as possible.

  • Techniques: Principal Component Analysis (PCA), t-Distributed Stochastic Neighbor Embedding (t-SNE).

13. Handling Imbalanced Data

  • Objective: Address datasets where some classes are significantly underrepresented.

  • Techniques: Resampling methods (oversampling and undersampling), Synthetic Minority Over-sampling Technique (SMOTE).

14. Feature Interactions

  • Objective: Explore how the interaction between features impacts the target variable.

  • Techniques: Interaction plots, creating interaction terms in regression models.

15. Automated EDA Tools

  • Tools: Libraries like Pandas Profiling, Sweetviz, and DataPrep can automate parts of your EDA, generating detailed reports on your data.


16. Data Transformation

  • Objective: Improve the performance of the model by transforming variables.

  • Techniques: Log transformation, square root transformation, and Box-Cox transformation.

17. Seasonal Decomposition of Time Series (STL)

  • Objective: Decompose a time series into seasonal, trend, and residual components.

  • Use Case: Identify underlying patterns in time series data.

18. Interaction Terms

  • Objective: Identify and include interaction terms in your model to capture the combined effect of variables.

  • Example: In regression, you might include X1×X2 if you suspect that the interaction between X1 and X2 affects the dependent variable.

19. Pair Plot (Scatterplot Matrix)

  • Objective: Visualize relationships between all pairs of variables.

  • Tool: Seaborn’s pairplot() function.

  • Benefit: Quickly identify linear and non-linear relationships.

20. Box-Cox Transformation

  • Objective: Transform non-normally distributed data into a normal distribution.

  • Application: Useful before applying machine learning algorithms that assume normality.

21. Principal Component Analysis (PCA)

  • Objective: Reduce the dimensionality of the data while retaining most of the variance.

  • Benefit: Simplifies the model, reduces computation time, and mitigates multicollinearity.

22. Clustering for Exploration

  • Objective: Group similar data points together to identify underlying structures.

  • Techniques: K-means, hierarchical clustering.

  • Application: Useful for segmenting data before further analysis.


23. Advanced Statistical Tests

  • Objective: Validate assumptions and test hypotheses.

  • Techniques: Chi-square test, ANOVA, t-tests, Mann-Whitney U test.





GRAPHICAL REPRESENTATION

1. Histograms

  • Purpose: Show the distribution of a single variable.

  • Use Case: Understand the spread and central tendency of your data.

  • Example: Visualizing the distribution of house prices.

2. Box Plots (Whisker Plots)

  • Purpose: Display the distribution of data based on a five-number summary (minimum, first quartile, median, third quartile, and maximum).

  • Use Case: Identify outliers and understand the spread and skewness of the data.

  • Example: Comparing salaries across different job roles.

3. Scatter Plots

  • Purpose: Show the relationship between two numerical variables.

  • Use Case: Identify correlations or patterns.

  • Example: Plotting house prices against square footage.

4. Pair Plots (Scatterplot Matrix)

  • Purpose: Visualize pairwise relationships in a dataset.

  • Use Case: Identify relationships and correlations among multiple variables.

  • Example: Examining relationships between height, weight, and age.

5. Heatmaps

  • Purpose: Visualize the correlation matrix.

  • Use Case: Identify correlations between multiple variables.

  • Example: Correlation between exam scores in different subjects.

6. Bar Charts

  • Purpose: Compare categorical data.

  • Use Case: Show the frequency or count of categories.

  • Example: Number of sales per product category.

7. Line Plots

  • Purpose: Display data points over time.

  • Use Case: Identify trends and patterns in time series data.

  • Example: Stock prices over the last year.

8. Violin Plots

  • Purpose: Combine the benefits of box plots and density plots.

  • Use Case: Visualize the distribution of the data across different categories.

  • Example: Distribution of exam scores across different classes.

9. Density Plots

  • Purpose: Show the distribution of a continuous variable.

  • Use Case: Similar to histograms but provide a smoother estimate of the distribution.

  • Example: Density plot of daily temperature readings.

10. Pairwise Correlation Plots

  • Purpose: Visualize correlations between all pairs of variables.

  • Use Case: Quickly identify the strength and direction of relationships.

  • Example: Correlations between different financial indicators.


11. Treemaps

  • Purpose: Visualize hierarchical data as nested rectangles.

  • Use Case: Represent parts-to-whole relationships.

  • Example: Display the market share of different tech companies.


12. Pair Grid

  • Purpose: Build a grid of scatter plots for every pair of variables in the dataset.

  • Use Case: Explore relationships between multiple pairs of variables.

  • Example: Visualize relationships in the Boston Housing dataset.


13. Joint Plots

  • Purpose: Combine scatter plots and histograms/density plots to visualize the relationship between two variables, along with their distributions.

  • Use Case: Detect correlation and distribution patterns simultaneously.

  • Example: Visualize the relationship and individual distributions of weight and height.

14. Count Plots

  • Purpose: Display the frequency count of categories.

  • Use Case: Similar to bar plots, but more focused on counts.

  • Example: Count of passengers in each class on the Titanic.


15. Treemaps

  • Purpose: Display hierarchical data using nested rectangles.

  • Use Case: Visualize proportions within hierarchical data.

  • Example: Market share of different companies in a sector.


16. Fourier Transforms

  • Purpose: Transform a time series from the time domain to the frequency domain.

  • Use Case: Analyze the frequency components of time series data.

  • Example: Frequency analysis of stock market data.






Comments

Post a Comment

Popular posts from this blog

Resume Work and Project Details

-
 Conceptual / MMM Basics Q1. What is Media Mix Modeling (MMM)? MMM quantifies how different marketing channels and external factors drive business outcomes such as sales or conversions, using regression-based or Bayesian models on historical time-series data. Unlike attribution models, MMM doesn’t require user-level data and works even with aggregated spend data. Q2. What problem does MMM solve? It helps marketers measure ROI by channel and optimize future spend allocation while accounting for carryover, seasonality, and external effects. Q3. What is the dependent variable and what are the independent variables in your models? Dependent variable: Sales or revenue. Independent variables: Marketing touchpoints (TV, Digital, Print, Radio) and control variables (Price, Competitor, Macro factors, Seasonality). Q4. How is MMM different from digital attribution models? Attribution models rely on user-level tracking (cookies, clicks). MMM uses aggregate, privacy-compliant data and...
Read more

Time Series and MMM basics

-
🕒 1. What is Time Series Data? Definition: Time series data is a sequence of observations recorded at specific time intervals (daily, weekly, monthly, etc.). Each data point has two key components: Time (t): a timestamp (e.g., date, hour, quarter) Value (y): a measurement (e.g., sales, ad spend, temperature) In Media Mix Modeling: time series data might include weekly sales, ad impressions, or marketing spend across channels. Example: Week TV Spend Online Spend Sales 1 1000 500 15000 2 1200 600 16200 3 900 550 14800 4 1100 700 17000 Here, Sales is the time series variable we want to analyze or forecast. 📊 2. What is Time Series Analysis? Definition: Time series analysis is the study of data points collected over time to identify patterns , trends , seasonality , and dependencies —and to model or forecast future behavior. In MMM, this helps us understand how marketing efforts (TV, digital, print) influence sales over time and fore...
Read more

LINEAR REGRESSION

-
Image
Machine Learning Machine Learning (ML) is a branch of artificial intelligence that focuses on building systems that learn from and make decisions based on data. There are several types of ML, but let’s focus on the two main categories:  Supervised  and  Unsupervised Learning . Supervised Learning Definition : In supervised learning, the algorithm is trained on labeled data. This means the input data is paired with the correct output, and the goal is to learn a mapping from inputs to outputs. Examples : Classification : Predicting whether an email is spam or not spam. Regress...
Read more