Week 03: Overview of Python NumPy

Objective: Introduce the essential functionality of NumPy for efficient numerical computations in Python. Cover the creation, manipulation, and analysis of arrays, along with basic statistical operations relevant for data science and engineering applications.

• Introduction to NumPy and Creating and Manipulating Arrays Overview of NumPy’s role in Python for efficient numerical and scientific computing. Discuss its benefits for handling large datasets and performing complex calculations. Learn to create arrays from lists, ranges, and functions like np.zeros and np.ones. Covers reshaping, changing dimensions, and understanding array structures.
• Array Operations and Math Functions and Indexing, Slicing, and Filtering Perform element-wise operations such as addition and multiplication. Explore aggregate functions like np.sum and np.mean for data summarization. Access specific elements or subsets using indexing and slicing. Apply Boolean indexing to filter data based on conditions.
• Random Number Generation and Statistical Analysis with NumPy Generate random numbers and arrays for simulations using np.random functions. Useful for creating test data and sampling. Use statistical functions to calculate mean, median, variance, and percentiles. Analyze data distributions and trends within datasets.

Introduction to NumPy

NumPy (Numerical Python) is a powerful Python library for numerical and scientific computing. Built on top of C, it allows for fast operations on arrays and matrices and provides an extensive collection of mathematical functions. NumPy is essential for handling large datasets and performing complex calculations efficiently, making it a cornerstone for data science, machine learning, and scientific computing.

Core Concepts of NumPy

1. Getting Started with NumPy

• Installation (pip install numpy) and importing (import numpy as np)
• Overview of NumPy’s core functionality and differences from Python lists
• Advantages: speed, memory efficiency, and broadcasting

2. NumPy Arrays

• Creating Arrays: Arrays can be created from lists, tuples, or built-in functions.
  • np.array(): Basic array creation from a list or tuple
  • np.zeros(), np.ones(), np.arange(), and np.linspace() for creating arrays with specific patterns
• Array Dimensions: Understanding 1D, 2D, and ND arrays, with examples
• Array Indexing and Slicing: Accessing elements, slicing arrays for subsets, and using Boolean indexing for filtering
• Array Shape and Reshape: Modifying the dimensions of an array (reshape, flatten, ravel)

3. Array Operations and Manipulations

• Basic Math Operations: Element-wise addition, subtraction, multiplication, and division
• Matrix Operations: Dot product, cross-product, and matrix multiplication using np.dot()
• Aggregations: Common functions like np.sum(), np.mean(), np.min(), np.max(), and np.std() for data aggregation
• Broadcasting: How NumPy handles operations on arrays of different shapes, examples of element-wise operations

4. Data Types and Conversions

• Understanding and specifying data types with dtype
• Type conversions with .astype() method for changing data types (e.g., integer to float)

5 Random Number Generation

• Generating Random Arrays: Using np.random for generating random numbers, such as np.random.rand(), np.random.randint(), and np.random.normal()
• Setting random seed with np.random.seed() for reproducible results

6. Statistical Functions and Analysis

• Common statistical functions (mean, median, variance, std, percentile)
• Practical exercises to analyze sample datasets

1. Getting Started with NumPy

To effectively utilize the NumPy library, we need to start with the basics, from installation and setup to understanding its core functionality. NumPy stands out as a powerful library because it enables efficient numerical computations on large datasets like enriched_sales_data.xlsx.

1.1 Installation and Importing NumPy

To use NumPy, you’ll need to have it installed in your Python environment. Install it via pip and import it for use:

pyhton
pip install numpy

Once installed, you can import it in your code:

import numpy as np

1.2 Overview of NumPy’s Core Functionality

NumPy is built around a key data structure called the array. This array is different from Python lists in several ways:

• Fixed Size: NumPy arrays are of fixed size, meaning their size cannot change after creation. This differs from Python lists, which are dynamic.
• Type Homogeneity: Each element in a NumPy array must be of the same data type (e.g., all integers or all floats). This uniformity leads to optimized performance and memory usage.
• Fast Operations: NumPy operations are implemented in C, making them faster than similar operations on Python lists.

1.3 Differences Between NumPy Arrays and Python Lists

Consider a small subset of enriched_sales_data.xlsx for a practical illustration. Let’s say we want to store and analyze the Units_Sold column for product sales.

Example Using Python Lists:

# Python list
units_sold = [45, 65, 30, 10, 90, 50]

# Adding 5 to each unit sold using a loop
updated_units = [unit + 5 for unit in units_sold]
print(updated_units)  # Output: [50, 70, 35, 15, 95, 55]

Example Using NumPy Arrays:

# NumPy array
units_sold_np = np.array([45, 65, 30, 10, 90, 50])

# Adding 5 to each unit sold using broadcasting
updated_units_np = units_sold_np + 5
print(updated_units_np)  # Output: [50 70 35 15 95 55]

Here, you can see that NumPy’s broadcasting feature (explained in detail below) allows us to perform operations without explicitly looping over elements, making the code cleaner and faster.

1.4 Advantages of Using NumPy

• Speed: NumPy is highly optimized for numerical calculations. For instance, processing the Units_Sold column with millions of entries would be significantly faster in NumPy than with regular Python lists.

Example: Summing Units Sold

# Generating large random data for simulation
units_sold_np_large = np.random.randint(1, 100, size=1000000)  # 1 million entries

# Summing all units
total_units = np.sum(units_sold_np_large)
print(total_units)

NumPy’s internal optimizations allow this to be done in milliseconds, whereas Python lists would take much longer.

• Memory Efficiency: NumPy arrays are more memory-efficient than Python lists because they store data in a contiguous block of memory. This layout allows for faster access and reduced memory usage, particularly beneficial when dealing with extensive datasets like enriched_sales_data.xlsx.
• Broadcasting: One of NumPy’s standout features is broadcasting. Broadcasting enables NumPy to handle operations on arrays of different shapes. If we want to add a constant to every element in an array or perform arithmetic operations across arrays of different sizes, broadcasting does it effortlessly. Video Array Broadcasting Example: Applying Discounts Let’s say we want to apply a 10% discount across all entries in the Unit_Price column:

# Original unit prices
unit_prices = np.array([150, 200, 300, 250, 180, 120])

# Apply a 10% discount
discounted_prices = unit_prices * 0.9
print(discounted_prices)  # Output: [135. 180. 270. 225. 162. 108.]

• Convenience with Multidimensional Data: While Python lists require nested structures to handle multi-dimensional data (like rows and columns in a table), NumPy can handle multi-dimensional arrays seamlessly. For instance, we can load multiple columns like Units_Sold, Unit_Price, and Sales_Amount from enriched_sales_data.xlsx as a 2D array and perform operations across the rows and columns easily.

# Sample data for demonstration
    data = np.array([
        [45, 150, 6750],   # Units_Sold, Unit_Price, Sales_Amount
        [60, 100, 6000],
        [75, 200, 15000]
    ])

    # Calculate total units sold, mean unit price, and total sales
    total_units = np.sum(data[:, 0])
    mean_unit_price = np.mean(data[:, 1])
    total_sales = np.sum(data[:, 2])

    print("Total Units Sold:", total_units)
    print("Mean Unit Price:", mean_unit_price)
    print("Total Sales:", total_sales)

Summary

Using the enriched_sales_data.xlsx dataset with NumPy will empower us to perform these kinds of efficient, large-scale data operations throughout the course. This foundational understanding of NumPy’s speed, memory efficiency, and broadcasting capabilities will be essential as we explore data manipulation and analysis techniques in upcoming weeks.

pip install numpy

Requirement already satisfied: numpy in c:\users\mohsin\anaconda3\lib\site-packages (1.26.4)
Note: you may need to restart the kernel to use updated packages.

import numpy as np

#Example Using Python Lists:
print ('# Python list')
units_sold = [45, 65, 30, 10, 90, 50]
 
print ('# Adding 5 to each unit sold using a loop')
updated_units = [unit + 5 for unit in units_sold]
print(updated_units)  # Output: [50, 70, 35, 15, 95, 55]

#Example Using NumPy Arrays:

print ('# NumPy array')
units_sold_np = np.array([45, 65, 30, 10, 90, 50])

print ('# Adding 5 to each unit sold using broadcasting')
updated_units_np = units_sold_np + 5
print(updated_units_np)  # Output: [50 70 35 15 95 55]

# Python list
# Adding 5 to each unit sold using a loop
[50, 70, 35, 15, 95, 55]
# NumPy array
# Adding 5 to each unit sold using broadcasting
[50 70 35 15 95 55]

print('\n\n# Generating large random data for simulation')
units_sold_np_large = np.random.randint(1, 100, size=1000000)  # 1 million entries
print('# Summing all units')
total_units = np.sum(units_sold_np_large)
print(total_units)

# Generating large random data for simulation
# Summing all units
49973793

print ('# Original unit prices')
unit_prices = np.array([150, 200, 300, 250, 180, 120])

print('# Apply a 10% discount')
discounted_prices = unit_prices * 0.9
print(discounted_prices)  # Output: [135. 180. 270. 225. 162. 108.]

# Original unit prices
# Apply a 10% discount
[135. 180. 270. 225. 162. 108.]

print ('# Sample data for demonstration')
data = np.array([
    [45, 150, 6750],   # Units_Sold, Unit_Price, Sales_Amount
    [60, 100, 6000],
    [75, 200, 15000]
])

print ('\n\n# Calculate total units sold, mean unit price, and total sales')
total_units = np.sum(data[:, 0])
mean_unit_price = np.mean(data[:, 1])
total_sales = np.sum(data[:, 2])

print("Total Units Sold:", total_units)
print("Mean Unit Price:", mean_unit_price)
print("Total Sales:", total_sales)

# Sample data for demonstration


# Calculate total units sold, mean unit price, and total sales
Total Units Sold: 180
Mean Unit Price: 150.0
Total Sales: 27750

NumPy Methods and Usage in Python

2. NumPy Arrays

3. Array Operations and Manipulations

4. Data Types and Conversions

5. Random Number Generation

6. Statistical Functions and Analysis