Week 02: Data Manipulation and Cleaning with Pandas

Objective: Teach participants to clean and prepare data for analysis, handling common issues in business datasets.

• Importing and exporting data (CSV, Excel, JSON)
• Data manipulation with Pandas: selecting, filtering, merging, and aggregating data
• Data cleaning: handling missing values, duplicate data, outliers, and data types
• Exploratory Data Analysis (EDA) techniques for business insights
• Hands-on exercise: Cleaning and preparing a business dataset
• Mini-project: Using Pandas to prepare a dataset for analysis (e.g., sales or financial data)

Detail overview of Data

Category of data

1. Primary Data

Primary data is original and collected firsthand by the researcher for a specific purpose. It is typically gathered through methods like surveys, experiments, or direct observation. Primary data can further be divided into:

• Interactive: Data collected through direct interaction with subjects, like interviews, surveys, or questionnaires. Example: Conducting a survey to understand consumer preferences.
• Observational: Data collected by observing subjects without direct interaction, typically in a natural setting. Example: Observing customer behavior in a retail store.
• Simulation: Data generated by simulating a real-world process or system under controlled conditions. Example: Running a simulation to understand the potential spread of a virus in a population.

2. Secondary Data

Secondary data is pre-existing data that was collected by someone else, usually for a different purpose. It can be accessed through sources like research publications, databases, and historical records. Secondary data can be categorized as:

• Cross-Sectional: Data collected at a single point in time, providing a snapshot of a particular phenomenon. Example: Census data that captures the demographic distribution at one specific time.
• Time Series: Data collected over a period of time, allowing analysis of trends and changes. Example: Monthly unemployment rates collected over several years to study economic trends.
• Panel: A type of data that involves multiple observations over time for the same subjects, combining elements of cross-sectional and time-series data. Example: Tracking the income levels of the same group of individuals over several years.

Each category serves different research purposes, and the choice depends on the nature of the study, the research question, and the availability of resources.

Data Types in Statistics

1. Quantitative Data:

This type of data can be measured numerically and is often divided into Continuous and Discrete types.

• Continuous Data: This type of data has a continuous range of values, like length, weight, or temperature. It can be measured in fractions or decimals.
• Discrete Data: This type of data includes specific values only, such as the number of students in a class or the count of coins.

Qualitative Data:

This data cannot be measured numerically and is instead categorized based on attributes or qualities.

• Nominal Data: In this type, data is categorized by names or labels, without any inherent order, such as gender (male, female) or color (red, blue).
• Ordinal Data: This type has a clear order or ranking among categories, but the intervals between them cannot be measured, such as rankings like "Good," "Very Good," "Excellent."

Design Data (Social Sciences) vs Organic Data (Computer Sciences)

In social sciences, "design data" refers to structured data collected through planned research designs, such as surveys, experiments, and observational studies. This data is purposefully gathered to answer specific questions or test hypotheses within a controlled environment, allowing researchers to analyze behavior, attitudes, or interactions systematically.

In computer science, "organic data" refers to naturally occurring data, often unstructured, that is generated without a specific research design in mind. This data comes from real-world sources like social media posts, sensor outputs, transaction logs, and web activity. Organic data is typically analyzed to identify patterns, trends, and insights in a way that reflects real-world, spontaneous behavior rather than controlled experimental settings.

Population and Sample in Data Analysis

• Population: Refers to the complete group of individuals or items sharing a specific characteristic and targeted in a study. Researchers define the population based on study goals. Example: Studying the eating habits of college students in Pakistan would have all Pakistani college students as the population.
• Sample: A subset of the population chosen for analysis, often due to practical constraints (time, cost, etc.). Ideally, the sample is representative of the population to generalize findings. Example: Selecting 500 students from different universities across Pakistan as a sample to represent all college students.

- Types of Sampling:

1. Probability Sampling: Each population member has a known, non-zero chance of selection. This method is randomized and aims to create a representative sample, enhancing result generalization.
   • Types: Simple Random, Systematic, Stratified, and Cluster Sampling.
2. Non-Probability Sampling: Selection chances are not known for all members, making it less representative and potentially introducing bias. It’s often faster and cheaper but limits generalizability.
   • Types: Convenience, Purposive (Judgmental), Quota, and Snowball Sampling.

Sample Size Determination Using Krejcie and Morgan Table

• Source for detail:

Sample Size Determination Using Krejcie and Morgan Table

Pandas

• Pandas, named after "panel data" from econometrics, was created by Wes McKinney to simplify handling multi-dimensional, labeled, and time-series data in Python. It emerged as a tool to streamline complex data manipulation tasks, making them accessible with just a few lines of code.
• The playful panda bear reference added a friendly touch, making pandas not only technically powerful but also memorable and approachable in the Python ecosystem. Today, it’s an essential library for data manipulation, beloved by data scientists and analysts alike.

Probability and Statistis for Computer Science"

Its Cover several key concepts

• Probability Basics: Fundamental principles of probability, including definitions, types of probabilities, and basic operations like union and intersection of events.
• Random Variables and Distributions: Explanation of random variables, both discrete and continuous, and how probability distributions apply to each. It also includes types of distributions like binomial, Poisson, and normal.
• Central Tendency and Dispersion: Measures like mean, median, mode, variance, and standard deviation to describe the central point and spread of data. Includes tools like quartiles and boxplots.
• Entropy and Information Theory: Introduction to entropy as a measure of uncertainty or disorder in data, as well as its applications in machine learning and communication systems.
• Data Visualization: Techniques for visualizing data through graphs such as histograms, contour plots, and scatter plots to better understand data distribution and relationships.
• Correlation and Dependence: Concepts of correlation, dependence, and methods to analyze the relationship between two or more variables, including covariance and correlation coefficients.
• Statistical Hypothesis Testing: Introduction to hypothesis testing for statistical inference, including p-values and confidence intervals, commonly used to validate or refute assumptions about data.
• Regression Analysis: Basics of linear regression, modeling relationships between variables, and evaluating the fit of these models.
• Time Series Analysis: Methods for analyzing data points collected or sequenced over time, including trends, seasonality, and smoothing techniques.
• Applications in Machine Learning: Use of statistical concepts, especially entropy and data distributions, in machine learning to improve model training, understanding, and evaluation.

Sources of Secondary Data:

Mathematical and Statistics Techniques for Applied Data Analysis

Software of Data Analysis

Greek Alphabet and It's Pronunciation

(for integer n