Introduction to R and RStudio: Practical Implementation

Welcome to Unit 1 of our in-depth course on advanced analytics using R. This unit will provide you with a comprehensive introduction to the R programming language and its development environment, RStudio.

Setting Up R and RStudio

Step 1: Installing R

1. Download R:

   • Go to the CRAN R Project website.
   • Select your operating system (Windows, macOS, or Linux).
   • Follow the instructions to download the appropriate R installer.

2. Install R:

   • Open the downloaded installer file.
   • Follow the on-screen instructions to complete the installation process.

Step 2: Installing RStudio

1. Download RStudio:

   • Visit the RStudio website.
   • Choose the free version of RStudio Desktop.
   • Download the installer for your operating system.

2. Install RStudio:

   • Open the RStudio installer file you downloaded.
   • Follow the installation wizard to install RStudio on your computer.

Step 3: Running R and RStudio

1. Launching RStudio:

   • After installing, open RStudio.
   • You should see the RStudio interface, which is divided into four main panels: Source, Console, Environment/History, and Files/Plots/Packages/Help.

2. Running Your First R Code:

   • In the Console panel (bottom left), type the following R code:

print("Hello, World!")

• Press Enter to execute the code. You should see the output:

[1] "Hello, World!"

Basic R Commands and Operations

Arithmetic Operations

# Addition
2 + 3

# Subtraction
5 - 2

# Multiplication
4 * 3

# Division
10 / 2

Variable Assignment

# Assigning values to variables
x <- 10
y <- 5

# Printing variables
print(x)
print(y)

Basic Data Structures

Vectors

# Creating a numeric vector
num_vector <- c(1, 2, 3, 4, 5)
print(num_vector)

# Creating a character vector
char_vector <- c("a", "b", "c")
print(char_vector)

Matrices

# Creating a matrix
matrix_example <- matrix(1:9, nrow = 3)
print(matrix_example)

Data Frames

# Creating a data frame
df <- data.frame(
  id = c(1, 2, 3),
  name = c("John", "Doe", "Smith"),
  age = c(28, 22, 35)
)
print(df)

Exploring RStudio Environment

Working with Scripts

1. Creating a New Script:

   • Click on File > New File > R Script.
   • This opens a new script file in the Source panel (top left).

2. Writing and Running Code in Script:

   • Write the following code in the script:

# Sample R script
message <- "Hello, R and RStudio!"
print(message)

• Save the script with a .R extension.
• Highlight the code and click Run or press Ctrl+Enter (Cmd+Enter on macOS) to execute the code.

Using RStudio Help

1. Accessing Help:
   • Use the Help tab in the bottom right panel.
   • You can search for help on functions by typing ?function_name in the console. For example:

?print

Conclusion

Congratulations! You have successfully set up R and RStudio, run basic R commands, and explored the RStudio environment. This foundational knowledge will serve as a stepping stone for more advanced analytics capabilities in subsequent units. Stay tuned for the next unit where we will delve deeper into data manipulation and visualization using R.

Part 2: Data Manipulation and Cleaning with dplyr and tidyr

Here we will demonstrate how to use the dplyr and tidyr libraries in R to manipulate and clean your data. We assume that you are already familiar with the basics of R and RStudio from Part 1 of your course.

Loading Libraries

First, ensure you have the dplyr and tidyr packages loaded:

library(dplyr)
library(tidyr)

Sample Data

We will use a sample dataset for our examples. Consider the following data frame df:

df <- data.frame(
  ID = 1:5,
  Name = c("Alice", "Bob", "Cathy", "Dan", "Eva"),
  Age = c(24, 27, NA, 22, 29),
  Score1 = c(NA, 88, 92, 85, 90),
  Score2 = c(78, NA, 85, 88, 84)
)

Data Manipulation Using dplyr

1. Filtering Rows: Select rows where Age is not NA:

   df_filtered <- df %>%
     filter(!is.na(Age))

2. Selecting Columns: Select specific columns:

   df_selected <- df %>%
     select(ID, Name, Age)

3. Mutating Columns: Add a new column that is a transformation of existing columns:

   df_mutated <- df %>%
     mutate(TotalScore = Score1 + Score2)

4. Summarizing Data: Calculate summary statistics:

   df_summary <- df %>%
     summarise(
       MeanScore1 = mean(Score1, na.rm = TRUE),
       MeanScore2 = mean(Score2, na.rm = TRUE)
     )

5. Grouping Data: Perform operations by group:

   df_group_summary <- df %>%
     group_by(Age) %>%
     summarise(
       MeanScore1 = mean(Score1, na.rm = TRUE),
       MeanScore2 = mean(Score2, na.rm = TRUE)
     )

Data Cleaning Using tidyr

1. Replacing NAs: Replace NAs with a specified value:

   df_replaced <- df %>%
     replace_na(list(Age = 0, Score1 = 0, Score2 = 0))

2. Tidying Data: Gathering multiple columns into key-value pairs:

   df_long <- df %>%
     gather(key = "ScoreType", value = "Score", Score1, Score2)

3. Spreading Data: Spreading key-value pairs back into multiple columns:

   df_wide <- df_long %>%
     spread(key = "ScoreType", value = "Score")

4. Separating Columns: Split a column into multiple columns:

   df_separated <- df %>%
     separate(Name, into = c("FirstName", "LastName"), sep = " ")

5. Uniting Columns: Combine multiple columns into one:

   df_united <- df %>%
     unite("FullName", FirstName, LastName, sep = " ")

By utilizing dplyr and tidyr, you can seamlessly manage and clean your data in R for robust data analytics and insights.

Exploratory Data Analysis with ggplot2

In this section, we will perform an exploratory data analysis (EDA) using the ggplot2 package in R. ggplot2 is a powerful and flexible tool for creating graphics and visualizations.

The dataset we will use for this demonstration is the mtcars dataset, which is readily available in R.

1. Load Required Libraries and Dataset

library(ggplot2)
data(mtcars)

2. Summary of the Dataset

summary(mtcars)

Review the summary to understand the central tendency, distributions, and the structure of the dataset.

3. Pair Plot: Visualizing Relationships

One key step in EDA is to visualize the relationship between variables. We will create a pair plot using ggpairs from the GGally package.

library(GGally)
ggpairs(mtcars)

4. Distribution of a Single Variable: Histogram

Visualize the distribution of a single variable, such as mpg (miles per gallon).

ggplot(mtcars, aes(x = mpg)) +
  geom_histogram(binwidth = 2, fill = "blue", color = "black") +
  labs(title = "Distribution of Miles Per Gallon (mpg)")

5. Boxplot: Distribution by Group

Boxplots can be used to compare distributions across groups. For instance, comparing mpg by the number of cylinders.

ggplot(mtcars, aes(x = as.factor(cyl), y = mpg)) +
  geom_boxplot(fill = "blue", color = "black") +
  labs(title = "MPG by Number of Cylinders", x = "Number of Cylinders", y = "Miles Per Gallon")

6. Scatter Plot: Relationship Between Two Continuous Variables

Scatter plots are useful to explore relationships between two continuous variables. We will examine the relationship between mpg and hp (horsepower).

ggplot(mtcars, aes(x = hp, y = mpg)) +
  geom_point(color = "blue") +
  labs(title = "MPG vs Horsepower", x = "Horsepower", y = "Miles Per Gallon")

7. Adding a Smoothing Line

To better visualize trends in scatter plots, you can add a smoothing line using geom_smooth().

ggplot(mtcars, aes(x = hp, y = mpg)) +
  geom_point(color = "blue") +
  geom_smooth(method = "lm", color = "red") +
  labs(title = "MPG vs Horsepower with Smoothing Line", x = "Horsepower", y = "Miles Per Gallon")

8. Bar Plot: Categorical Data

Bar plots are useful for visualizing categorical data. For instance, the count of cars for each number of cylinders.

ggplot(mtcars, aes(x = as.factor(cyl))) +
  geom_bar(fill = "blue", color = "black") +
  labs(title = "Count of Cars by Cylinder", x = "Number of Cylinders", y = "Count")

9. Faceting: Multi-Panel Plots

Faceting allows the creation of multi-panel plots based on the values of a categorical variable. Here, we facet the scatter plot of hp vs mpg by the number of cylinders.

ggplot(mtcars, aes(x = hp, y = mpg)) +
  geom_point(color = "blue") +
  facet_wrap(~ cyl) +
  labs(title = "MPG vs Horsepower Faceted by Cylinders", x = "Horsepower", y = "Miles Per Gallon")

Conclusion

This section provided a practical guide to performing EDA using ggplot2. You visualized the dataset's structure and relationships between variables using different types of plots. This analysis is crucial for understanding your data and preparing for further statistical analysis or modeling.

Statistical Modeling and Hypothesis Testing

Overview

In this section, we will cover the practical implementation of statistical modeling and hypothesis testing using R. This includes setting up a linear regression model, performing t-tests, and conducting ANOVA.

Linear Regression

Fit a Linear Regression Model

# Load necessary libraries
library(tidyverse)

# Assume data is already loaded in a DataFrame named `data`
# Fit a linear regression model
model <- lm(response_variable ~ predictor_variable, data = data)

# Display the summary of the model
summary(model)

Hypothesis Testing

t-Test

# Perform a two-sample t-test
t_test_result <- t.test(data$group1, data$group2)

# Display the results
print(t_test_result)

ANOVA

# Fit an ANOVA model
anova_model <- aov(response_variable ~ factor_variable, data = data)

# Display the summary of the ANOVA model
summary(anova_model)

Practical Example

Data Preparation

Suppose we have a dataset named mtcars and we want to explore the relationship between mpg (miles per gallon) and wt (weight of the car).

# Load the dataset
data(mtcars)

# Fit a linear regression model
model_mtcars <- lm(mpg ~ wt, data = mtcars)

# Display the summary of the model
summary(model_mtcars)

# Perform a t-test comparing mpg of two different groups of cars
# Let's assume we want to compare cars with different numbers of cylinders (4 vs 6)
t_test_mtcars <- t.test(mtcars$mpg[mtcars$cyl == 4], mtcars$mpg[mtcars$cyl == 6])

# Display the t-test results
print(t_test_mtcars)

# Perform ANOVA to see the effect of the number of cylinders on mpg
anova_mtcars <- aov(mpg ~ factor(cyl), data = mtcars)

# Display the ANOVA results
summary(anova_mtcars)

Conclusion

In this section, we have implemented linear regression, t-tests, and ANOVA using the lm, t.test, and aov functions in R. Each of these methods is essential for performing statistical modeling and hypothesis testing, which are critical components of data analysis.

Unit 5: Machine Learning with caret and RandomForest in R

This unit focuses on implementing machine learning algorithms using the caret package, specifically highlighting the Random Forest algorithm in R.

Step-by-Step Implementation

1. Loading the Required Libraries

library(caret)
library(randomForest)

2. Preparing the Data

To illustrate the process, we will use the iris dataset.

data(iris)

# Splitting the dataset into training and testing sets
set.seed(123)  # For reproducibility
trainIndex <- createDataPartition(iris$Species, p = 0.8, 
                                  list = FALSE, 
                                  times = 1)
trainData <- iris[ trainIndex,]
testData  <- iris[-trainIndex,]

3. Training the Random Forest Model

# Define training control
trainControl <- trainControl(method = "cv", number = 5)

# Train the model
rfModel <- train(Species ~ ., data = trainData, method = "rf",
                 trControl = trainControl)

4. Evaluating the Model

Predictions on Test Data

predictions <- predict(rfModel, testData)

Confusion Matrix

confMat <- confusionMatrix(predictions, testData$Species)
print(confMat)

5. Tuning the Random Forest Model

To tune hyperparameters, one might adjust the mtry parameter – the number of features sampled at each split.

# Grid of tuning parameters
tuneGrid <- expand.grid(.mtry = c(1, 2, 3, 4))

# Train the model with tuning
rfModelTuned <- train(Species ~ ., data = trainData, method = "rf",
                      trControl = trainControl, 
                      tuneGrid = tuneGrid)

Evaluating the Tuned Model

predictionsTuned <- predict(rfModelTuned, testData)
confMatTuned <- confusionMatrix(predictionsTuned, testData$Species)
print(confMatTuned)

6. Feature Importance

Random Forest allows for assessing the importance of each feature.

# Extracting feature importance
importance <- varImp(rfModel)
print(importance)

7. Visualizing the Results

To visualize the performance and feature importance of the model:

Visualization of Feature Importance

plot(importance)

ROC Curve (if applicable for binary classification)

if(nlevels(trainData$Species) == 2) {
  library(pROC)
  rfProbs <- predict(rfModel, testData, type = "prob")
  rocCurve <- roc(response = testData$Species, predictor = rfProbs[,2])
  plot(rocCurve)
}

This completes the implementation of training and evaluating a Random Forest model using the caret package in R.

Time-Series Analysis and Forecasting with prophet

Introduction

In this segment of our advanced analytics course, we'll leverage the prophet package to perform time-series analysis and forecasting in R. The prophet package, developed by Facebook, is widely used for analyzing time-series data and producing high-quality forecasts.

Practical Implementation

1. Load Necessary Libraries

Start by loading the prophet package and other necessary libraries for data manipulation and visualization.

library(prophet)
library(dplyr)
library(ggplot2)

2. Prepare the Data

Ensure your data is in a data frame with two columns: ds (date) and y (value to forecast). Here is a sample dataset preparation:

# Sample time-series data
date_seq <- seq(as.Date("2020-01-01"), by = "month", length.out = 24)
values <- round(runif(24, min = 100, max = 500))

# Creating a dataframe
df <- data.frame(ds = date_seq, y = values)

3. Fit the Prophet Model

Fit the prophet model on the prepared data.

model <- prophet(df)

4. Make a Future Dataframe

Generate a dataframe for the future dates you want predictions for. Extend the dataframe by the number of periods you need.

future <- make_future_dataframe(model, periods = 12, freq = 'month')

5. Predict Future Values

Use the predict function to forecast future values.

forecast <- predict(model, future)

6. Visualize the Forecast

Plot the results using Prophet's built-in plotting functions.

plot(model, forecast) + 
  labs(title = "Forecasted Time-Series Data")

For more detailed insights, you can plot the forecast components:

prophet_plot_components(model, forecast)

7. Evaluate the Model

To evaluate the model, you could use cross-validation functions provided within prophet. For instance, use performance_metrics and plot_cross_validation_metric.

First, create a cross-validation object:

df_cv <- cross_validation(model, initial = 365, period = 180, horizon = 365, units = 'days')

Then, calculate performance metrics:

df_p <- performance_metrics(df_cv)
print(df_p)

Lastly, plot the cross-validation metrics:

plot_cross_validation_metric(df_cv, metric = 'rmse')

Complete Example

Below is a complete example integrating all steps:

# Load Libraries
library(prophet)
library(dplyr)
library(ggplot2)

# Prepare Data
date_seq <- seq(as.Date("2020-01-01"), by = "month", length.out = 24)
values <- round(runif(24, min = 100, max = 500))
df <- data.frame(ds = date_seq, y = values)

# Fit Model
model <- prophet(df)

# Create Future Dataframe
future <- make_future_dataframe(model, periods = 12, freq = 'month')

# Predict Future Values
forecast <- predict(model, future)

# Visualize Forecast
plot(model, forecast) + labs(title = "Forecasted Time-Series Data")
prophet_plot_components(model, forecast)

# Cross-Validation
df_cv <- cross_validation(model, initial = 365, period = 180, horizon = 365, units = 'days')
df_p <- performance_metrics(df_cv)
print(df_p)
plot_cross_validation_metric(df_cv, metric = 'rmse')

With this comprehensive implementation, you should be able to perform robust time-series forecasting using prophet in R.

Text Mining and Sentiment Analysis with tm and tidytext

This part of your project focuses on leveraging the tm and tidytext packages in R for text mining and performing sentiment analysis on textual data.

Text Mining with the tm Package

1. Load necessary libraries:

   library(tm)
   library(tidytext)
   library(dplyr)

2. Create a Corpus:

   documents <- c("Text mining is a technique to transform text into data for analysis.",
                  "Sentiment analysis is widely used to understand the emotions conveyed in text.",
                  "R provides powerful tools for text analysis.")
   corpus <- Corpus(VectorSource(documents))

3. Preprocess the Text:

   corpus <- tm_map(corpus, content_transformer(tolower))
   corpus <- tm_map(corpus, removePunctuation)
   corpus <- tm_map(corpus, removeNumbers)
   corpus <- tm_map(corpus, removeWords, stopwords("en"))
   corpus <- tm_map(corpus, stripWhitespace)

4. Create a Document-Term Matrix:

   dtm <- DocumentTermMatrix(corpus)

5. Convert to DataFrame:

   dtm_matrix <- as.matrix(dtm)
   dtm_df <- as.data.frame(dtm_matrix)

Sentiment Analysis with the tidytext Package

1. Convert Corpus to Tidy Format:

   tidy_corpus <- tidy(corpus)

2. Tokenize Text:

   tokens <- tidy_corpus %>%
     unnest_tokens(word, text)

3. Remove Stop Words (if not done during preprocessing):

   tokens <- tokens %>%
     anti_join(stop_words)

4. Perform Sentiment Analysis:

   • Using the Bing Lexicon:

     bing_sentiments <- get_sentiments("bing")
     sentiment_analysis_bing <- tokens %>%
       inner_join(bing_sentiments, by = "word") %>%
       count(sentiment, sort = TRUE)

   • Using the NRC Lexicon:

     nrc_sentiments <- get_sentiments("nrc")
     sentiment_analysis_nrc <- tokens %>%
       inner_join(nrc_sentiments, by = "word") %>%
       count(sentiment, sort = TRUE)

5. Visualize Sentiment:

   library(ggplot2)
   
   ggplot(sentiment_analysis_bing, aes(x = sentiment, y = n, fill = sentiment)) +
     geom_bar(stat = "identity") +
     theme_minimal() +
     labs(title = "Sentiment Analysis with Bing Lexicon",
          x = "Sentiment",
          y = "Frequency")

   ggplot(sentiment_analysis_nrc, aes(x = reorder(sentiment, n), y = n, fill = sentiment)) +
     geom_bar(stat = "identity") +
     theme_minimal() +
     labs(title = "Sentiment Analysis with NRC Lexicon",
          x = "Sentiment",
          y = "Frequency") +
     coord_flip()

This provides a complete implementation pipeline for text mining and sentiment analysis using tm and tidytext in R. The steps include text preprocessing, tokenization, and visualizing the sentiments extracted from the text.

Interactive Data Visualization with Shiny

This section will guide you through creating an interactive data visualization application using the Shiny package in R. We'll build a Shiny app that enables users to visualize and interact with a dataset of their choice.

Prerequisites

Ensure you have the shiny package installed. You can install it using:

install.packages("shiny")

Step-by-Step Implementation

Step 1: Load Required Libraries

library(shiny)
library(ggplot2)  # In case we need advanced plotting

Step 2: Define the UI

Create the user interface (UI) for your Shiny app that includes a selection input and a plot output.

ui <- fluidPage(
  titlePanel("Interactive Data Visualization with Shiny"),
  
  sidebarLayout(
    sidebarPanel(
      fileInput("file1", "Choose CSV File", accept = ".csv"),
      uiOutput("varSelectUI")
    ),
    
    mainPanel(
      plotOutput("distPlot")
    )
  )
)

Step 3: Define the Server Logic

Create server logic to read the input dataset, generate the UI dynamically, and render the interactive plot.

server <- function(input, output, session) {
  
  # Reactive function to read the uploaded dataset
  dataset <- reactive({
    req(input$file1)
    file <- input$file1$datapath
    read.csv(file, stringsAsFactors = FALSE)
  })
  
  # Dynamically generate UI for variable selection based on the dataset
  output$varSelectUI <- renderUI({
    req(dataset())
    df <- dataset()
    selectInput("var", "Choose a Variable to Plot", choices = names(df))
  })
  
  # Render the plot based on the selected variable
  output$distPlot <- renderPlot({
    req(input$var)
    df <- dataset()
    selected_var <- input$var
    
    ggplot(df, aes_string(selected_var)) +
      geom_histogram(binwidth = 10, fill = "blue", color = "black") +
      labs(title = paste("Distribution of", selected_var), x = selected_var, y = "Frequency")
  })
}

Step 4: Run the App

Combine the UI and server to run the Shiny app.

shinyApp(ui = ui, server = server)

Complete Code

library(shiny)
library(ggplot2)

ui <- fluidPage(
  titlePanel("Interactive Data Visualization with Shiny"),
  
  sidebarLayout(
    sidebarPanel(
      fileInput("file1", "Choose CSV File", accept = ".csv"),
      uiOutput("varSelectUI")
    ),
    
    mainPanel(
      plotOutput("distPlot")
    )
  )
)

server <- function(input, output, session) {
  
  dataset <- reactive({
    req(input$file1)
    file <- input$file1$datapath
    read.csv(file, stringsAsFactors = FALSE)
  })
  
  output$varSelectUI <- renderUI({
    req(dataset())
    df <- dataset()
    selectInput("var", "Choose a Variable to Plot", choices = names(df))
  })
  
  output$distPlot <- renderPlot({
    req(input$var)
    df <- dataset()
    selected_var <- input$var
    
    ggplot(df, aes_string(selected_var)) +
      geom_histogram(binwidth = 10, fill = "blue", color = "black") +
      labs(title = paste("Distribution of", selected_var), x = selected_var, y = "Frequency")
  })
}

shinyApp(ui = ui, server = server)

In this code, users can upload a CSV file, select a variable from the dataset, and visualize its distribution interactively. This demonstrates the core interactivity and flexibility of Shiny applications in R.