Python for Data Analysis Basics

Introduction

Python, a popular high-level programming language, provides extensive libraries and modules that can efficiently handle and analyze large sets of data. In this guide, we will introduce data analysis using Python, focusing on the pandas library, and discuss data structures, data imports/exports, data cleaning, and basic statistical operations.

Setup and installation

Before starting, ensure you have Python installed on your machine. In addition to this, we'll also use pandas, numpy and matplotlib libraries. You can install it via pip:

pip install pandas numpy matplotlib

Data Structures

Series

A Series is a one-dimensional array-like object that can hold any data type.

import pandas as pd

series = pd.Series([1,3,5,np.nan,6,8])
print(series)

DataFrame

A DataFrame is a two-dimensional table of data with rows and columns.

data = {
    'year': [2010, 2011, 2012, 2011, 2012, 2010, 2011, 2012],
    'team': ['Bears', 'Bears', 'Bears', 'Packers', 'Packers', 'Lions', 'Lions', 'Lions'],
    'wins': [11, 8, 10, 15, 11, 6, 10, 4],
    'losses': [5, 8, 6, 1, 5, 10, 6, 12]
}
football = pd.DataFrame(data)
print(football)

Exploring Data

DataFrame provides several methods to explore the data.

• head(n): displays the first n rows.
• describe(): provides summary statistics.

print(football.head())
print(football.describe())

Data Cleaning

Pandas provides several methods to clean the data.

• dropna(): removes missing values.
• replace(from, to): replaces a value with another.

data = pd.DataFrame({'k1': ['one'] * 3 + ['two'] * 4, 'k2': [1, 1, 2, 3, 3, 4, 4]})
print(data.replace(1, 100))
print(data.drop_duplicates())

Basic Statistics

Pandas provide methods to perform basic statistical operations.

• mean(), median(), mode(): calculates the mean, median, and mode.
• corr(): computes pairwise correlation of columns.

print(football['wins'].mean())
print(football['wins'].mode())
print(football.corr())

Data Import and Export

You can read data from a CSV file using read_csv(). Let's assume you have a file data.csv.

data = pd.read_csv('data.csv')

Similarly, you can write data to a CSV file using to_csv().

data.to_csv('data.csv')

And that covers the basics of data analysis with Python. As you continue your learning journey, consider exploring further functionalities and libraries, such as numpy, matplotlib, and scikit-learn for more complex analysis and data visualizations.

Advanced Techniques in Python for Data Analysis

In this guide, we'll dive into advanced techniques for data analysis using Python, particularly focusing on libraries like Pandas, NumPy, SciPy, and Matplotlib, as well as data cleaning, visualization and using machine learning algorithms for in-depth analysis.

Assuming that you are already skilled with basic Python programming and drawing insights from data, this guide will up your data analysis game. Let's get started!

Advanced Pandas Techniques

Handling Missing Data

Missing data is common in real-life data sets. Pandas offers powerful mechanisms to handle them elegantly. We use replace(), fill(), dropna() methods to handle missing data.

import pandas as pd
import numpy as np

# Creating a sample dataframe
df = pd.DataFrame({'A': [1, 2, np.nan], 'B': [5, np.nan, np.nan], 'C': [1, 2, 3]})
print("Original Dataframe:\n", df)

# Using fillna()
df.fillna(value='Fill Value')  # Fills missing values with 'Fill Value'

# Using dropna()
df.dropna()  # It drops the rows where at least one element is missing.

GroupBy

The groupby() function is used to split the data into groups based on some criteria. This accompanies aggregation functions like sum(), mean() etc.

data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],
       'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],
       'Sales':[200,120,340,124,243,350]}

df = pd.DataFrame(data)
comp_group = df.groupby('Company')
print(comp_group.mean())  # Gives mean sales of each company

Advanced Matplotlib Techniques

Matplotlib is a Python 2D plotting library. It is used to generate plots, histograms, power spectra, bar charts, error charts, scatterplots, etc.

import matplotlib.pyplot as plt

# Creating figure
fig = plt.figure()

# Add set of axes to figure
axes = fig.add_axes([0.1, 0.1, 0.8, 0.8])

# Plot on that set of axes
axes.plot(x, y, 'b')
axes.set_xlabel('Set X Label')
axes.set_ylabel('Set y Label')
axes.set_title('Set Title')

Data Normalization with Scikit

Normalization is a statistical method that helps mathematical-based algorithms interpret features with different magnitudes and distributions equally.

from sklearn import preprocessing

# Creating sample data
data = {'score': [234,24,14,27,-74,46,73,-18,59,160]}
df = pd.DataFrame(data)

# Create scaler
scaler = preprocessing.MinMaxScaler()

# Transforming data
df_normalized = pd.DataFrame(scaler.fit_transform(df), columns=df.columns)

Machine Learning with Scikit-learn

Scikit-learn provides simple and efficient tools for machine learning and statistical modeling including classification, regression, clustering and dimensionality reduction. Here, we will run a simple linear regression model.

from sklearn.linear_model import LinearRegression

# Initializing the model
linear_model = LinearRegression()

# Train the model
linear_model.fit(X_train, Y_train)

# Test the model
predictions = linear_model.predict(X_test)

All the code snippets shown above are bare minimum samples and real-life data cases may require adjustments. These advanced data analysis techniques, once mastered, can help you to deal with complex data-driven problems effectively.

Practical Data Analysis: Use-Cases and Projects

This section provides a practical implementation of data analysis projects using Python. It covers three main sections:

1. Customer Segmentation
2. Sales Forecasting
3. Social Media Sentiment Analysis

1. Customer Segmentation (Clustering)

Customer segmentation groups potential customers into segments based on similar characteristics. Here, we will use the KMeans clustering algorithm.

Step 1. Data Pre-processing

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

# Assume you've read data into a pandas DataFrame df
df.dropna(inplace=True) # Remove NaN values

# Let's scale our data
scaler = StandardScaler()
df_scaled = scaler.fit_transform(df.iloc[:, 1:]) #assuming the first column is non-numeric

Step 2. KMeans Clustering

kmeans = KMeans(n_clusters=4, random_state=0).fit(df_scaled)
labels = kmeans.labels_

Step 3. Post Cluster Analysis

df['labels'] = labels
grouped = df.groupby('labels').mean()

In this step, you can examine the 'grouped' DataFrame to understand the common characteristics of the different groups (clusters).

2. Sales Forecasting (Time Series Analysis)

Sales forecasting is about predicting future sales. Various methods can be used, and here, we use an Auto Regressive Integrated Moving Average (ARIMA) model.

Step 1. Data Preprocessing

from statsmodels.tsa.arima_model import ARIMA

# Again assume that the time series data is in a pandas DataFrame df.
df.dropna(inplace=True) # Remove NaN values
df.index = pd.to_datetime(df.index) # Confirm the index is dateTime

Step 2. Train the ARIMA model

model = ARIMA(df, order=(5, 1, 0)) # You might need to tune these parameters
model_fit = model.fit(disp=0)

Step 3. Prediction

forecast = model_fit.forecast(steps=10) # Predict the next 10 steps

3. Social Media Sentiment Analysis (Natural Language Processing)

This involves determining the sentiment of posts on social media (e.g., tweets).

Step 1. Data Preprocessing

import nltk
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer

nltk.download('stopwords')
stopwords_list = stopwords.words('english')

# Assume we've read a set of tweets into a pandas DataFrame df.
df['text'] = df['text'].str.lower() # Convert text to lowercase.
df['text'] = df['text'].str.replace('[^\w\s]','') # Remove punctuation.
df['text'] = df['text'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stopwords_list)])) 

Step 2. Convert Text to Vectors

vectorizer = CountVectorizer()
features = vectorizer.fit_transform(df['text'])

Step 3. Train a Model for Sentiment Analysis

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(features, df['label'], test_size=0.2, random_state=0)
model = LogisticRegression()
model.fit(X_train, y_train)

Step 4. Make Predictions

predictions = model.predict(X_test)

In this part, you are now able to see predicted sentiments for your test set.

Please note, in all the above methods, you can further improve your models by tuning hyperparameters and using techniques such as cross-validation, grid search, and ensemble methods.