```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

## Goals of this lesson

Students will learn:

1. How to check for problematic data and address issues.
2. How to filter, rearrange and shape data in preparation for analysis. 

## Loading packages

```{r, message = FALSE, warning = FALSE}

library(tidyverse)
library(psych)

```

## Loading your data

As you covered in the last lesson, we can easily load data from existing files (e.g., csv, txt) into our R environment. We'll start by loading the same dataframe (Study1.csv) used in last week's lesson.

```{r}


# take another peek to remind yourself what's in there

```

We're also going to follow the same initial data cleaning steps as we did before.

```{r}

# first, rename the "Personality" column to something more meaningful (Neuroticism)


# then rename the timepoint columns


# and rearrange the order of the columns (based on the order they're passed into select()


# and replace the missing age cell (here we're assuming we KNOW the correct value)

```

### Check for correct values

Now that we've addressed missing values, we want to make sure the rest of our variables look correct. To look at some of the other variables, let's use the `table()` function. This works well for factors or variables with only a few different values. Our condition and sex variables are good here.

Use table function. 
```{r}

# make tables of categorical variables

# you could also look at a table of ages

# we see that there are 3 values that are "Femle" - this is incorrect!

```

When we look at the table for the Sex variable, we see another data entry problem. We have a third category that should really be another case of "Female". Here we are going to use the function `recode` from the `dplyr` package. 

```{r}

# replace bad category label

# in the recode() function, you should first put the value that we want to change,
# and then we put the value that we want the variable to be (e.g., Femle = "Female")


# all better!

```

Now let's look at the continuous variables. You can also look at these with the table function, but sometimes it's easier to visualize. The `hist()` function, which creates histograms, is good here.

```{r}

# create histograms to see data distribution

# make a histogram of the data variable

# visualizations are one of the most valuable ways to understand your data

```

Looks like we have a potential outlier on the neuroticism score. This could be a data entry error, but it could also be a real value that just happens to be really low. This is why data inspection is so important for later analysis -- now you know that value is there, it's up to you to decide how to deal with it.

#### Get a table of descriptive stastistics for variables in your dataset.

The "psych" package has a couple of useful functions for providing a table of a range of descriptive statistics, including:
1. describe() provides a table with the item name, item number, nvalid, mean, sd, median, mad, min, max, skew, kurtosis, se.
2. describe.by() for generating descriptive statistics by a group variable.

It is important to check each continuous variables skew and kurtosis to determine whether they need to be transformed (for example, outcomes variables in linear regression need to be normally distributed). 
Skew values > | 2 | are considered significantly skewed.
Kurtosis values > | 10 | are considered significantly not normal (flat top). 

Obtain descriptive statistics for your dataset.

```{r}

```

Obtain descriptive statistics for your dataset grouped by Sex.

```{r}


```

### Arranging data

Sometimes it is helpful for you to arrange the observations (rows) in your dataset by values of a variable. 
Arrange() sorts your observations in a dataset in ascending or descending order.

```{r}

# arrange the dataset based on sorting the Condition variable in ascending order.

# use desc() to sort in descending order.

```

### Filtering data 

Let's say we have decided a priori to exclude outliers 3 SDs above or below the mean. We will use the filter() function in `tidyverse` to clean the dataset.

```{r}

# first, mark upper and lower boundaries of data
# anything above upper or below lower will be excluded as outlier.

# compute the upper bound (and store it in a variable)

# compute the lower bound (and store it in a variable)

# only keep values within this range! 

  # read as: "Neuroticism is greater than 'lower' AND neuroticism is less than 'upper'."
  # you can apply as many filter functions as we want to
  # you can also use the | or & function within filter as well, and all other arithmetic operators

# check that we excluded the 1 outlier


```

For filtering, you might want to filter out the NAs here. If so, you can use the command: 

filter(!is.na(variable_name) within dplyr. This will take out all the NAs for ONE specific variable, to use for multiple: 
filter(!is.na(variable_1), !is.na(variable_2), !is.na(variable_3) ... etc.

## Getting ready for analysis

Now that we've gone through and cleaned up the problems, you can think ahead to how you'll want to use this data. 

### Recoding variables

Sometimes we want to treat categorical variables as factors, but sometimes we want to pretend they're numeric (as in a regression, when binary variables can be coded as 0 and 1). Right now, Condition is coded as a binary numeric variable, but that's not very informative, so you'd rather have the values be descriptive. Here, the function `recode()` from the `dplyr` package is really useful again, but we're going to use a different version that will also turn the variable from numeric or factor data. We'll also create a second variable instead of overwriting Condition.

```{r}

# transform into factor with labels using mutate () and recode_factor() 


# these line create new column ConditionF - specifies what we are calling the new variable.

# check that your variable is now recoded as you'd like! 

# REMINDER: all changes we're making to our data are only within the R workspace; the original .csv file doesn't change

```


```{r}

# we can also create new transformations of existing variables

# imagine you want to turn Neuroticism, which is continuous, into a categorical variable (e.g., high vs. low)
# one common way to do something like this is with a median split

# first, we'll need to get the median

# recode a continuous variable into a median-split factor using mutate() and case_when()


# you can also do the same thing using mutate() and ifelse()

```

### Calculating new variables

You may also want to recalculate or rescale some variables. For example, we can turn Neuroticism into a Z score, or calculate an average response across the four time points. Here we are using `mutate()`, a dplyr function used for adding new variables to a data frame. Mutate is nice because it makes use of other functions, such as `scale()` to make Z scores or `rowMeans()` to calculate means.

```{r}

# calculate mean centered Z scores using mutate()  and scale() 


         # create a new variable, Neuroticismz, using the scale function by meaning centering it and set the SD to be equal to 1
         # scale creates a matrix variable with width 1 — so we need to convert this back to a numeric variable

         # round the resulting z-scores to two decimal points


# mean should now be (very close to) 0

# calculate means across multiple columns using rowMeans

# this line computes the mean score across all of the days
# rowMeans() calculates the mean of the rows
# note that cbind (which stands for column bind) is necessary here because rowMeans expects a Matrix


# calculate means across multiple columns using arithmetic operators
# group by ID to calculate a mean for each row of the dataframe (i.e. participant).
# always ungroup


```

### Combining data from multiple sources

Sometimes, data might be spread across multiple spreadsheets, and you'll want to combine those for your analysis. For example, maybe this study had a follow-up survey on Day 30. Scores from that survey were entered into another spreadsheet, which only has the subject ID and that score. We want to include that score into our data.

Here I'm using the mutating join functions in tidyverse. Mutating joins combine variables from the two data.frames:

x = first dataset you enter; y = second dataset you enter

1. inner_join() - return all rows from x where there are matching values in y, and all columns from x and y. If there are multiple matches between x and y, all combination of the matches are returned.

2. left_join() return all rows from x, and all columns from x and y. Rows in x with no match in y will have NA values in the new columns. If there are multiple matches between x and y, all combinations of the matches are returned.

3. right_join() return all rows from y, and all columns from x and y. Rows in y with no match in x will have NA values in the new columns. If there are multiple matches between x and y, all combinations of the matches are returned.

4. full_join() return all rows and all columns from both x and y. Where there are not matching values, returns NA for the one missing.

```{r}

# first, let's load some new data

# merge the two dataframes
# to make sure the rows match up, we use the 'by' argument to specify that IDs should match
# that way even if the data is in a different order you will get scores matched togther correctly

  
# note that mydata has one less row than mydata_followup because we removed that outlier
# by default, left_join will use only cases of the 'by' variable that exist in both dataframes,
# so it drops that one row from mydata_followup when merging

# full_join, on the other hand, will include all cases from both mydata and mydata_followup
# if they are present in either dataframe

```

### Shaping data

Finally, you may want to change the layout of your data. Right now, our data frame is in "wide" format, which means that each row is a subject, and each observation gets its own column. For some analyses, you'll need to use "long" format, where each row is an observation, and columns specify things like time and ID to differentiate the observations. There are lots of packages that can handle data reshaping, but I'll show the `pivot_longer()` and `pivot_wider()` functions from `tidyr`.

Two main data formats
1) WIDE format - each row of our dataframe is one subject.
2) LONG format - if we have subjects with multiple observations per subject; each row is one observation (rather than each row being for one subject). Will have more rows than a long dataframe.

```{r}

# our goal here is to make a separate row for each timepoint in the dataframe
# to do this, we need two kinds of new columns:
# - first, the "key" column will identify which observation (i.e., which day) the row corresponds to
# - second, the "value" column will contain the actual values/measurements (i.e., the score for each day)

# in theory, you can name these whatever you want, but to keep everyone on the same page,
# we'll name the key "Time" and the value "Score"

 # selects the columns we're bringing into long format
 # the "key" column
 # the "value" column
  
# another way to write this


# pivot_wider lets you go back the other direction; the result should be identical to the original mydata 
# the function takes columns that are multiple observations (Time) and spreads back the values (Score)
# to their own columns so that there is only one row per subject

  
```

## Saving your work

Once you've created a data cleaning script like this one, you'll have a record of all the edits you've made on the raw data, and you can recreate your cleaned data just by running the script again. However, it's often easier to save your cleaned data as its own file **(never overwrite the raw data)**, so when you come back to do analysis you don't have to bother with all the cleaning steps. 

You can always save data frames as a .csv for easy sharing and viewing outside of R.

```{r, eval = FALSE}

# write data to a .csv

# cleaned data only exists in R, so we need to save this as a (SEPARATE!) .csv file 
# again, it's good practice to NEVER overwrite your original data file — only create new copies

```

However, you can also save in an R format that lets you save multiple variables/objects in the same file. For example, you might want to have a long and wide format, or one dataframe with all the data and one with just subject information. Saving as a .rda file allows you to save multiple objects at once for easy loading into R. You can also have the outputs of statistical models saved in these, along with their data.

```{r, eval = FALSE}

# save a .rda (which can store multiple objects/variables, but can only be read by R)

# add as many objects/variables as you want, separated by commas

```

## Data Challenge!

Go to the file challenge_nocode.Rmd file (download from the SIPPS website) for your data challenge. This challenge will include skills from this tutorial, but it will also ask you to explore new functions we didn't cover. Remember to use the help command in R (?<function name>) and google as necessary. You'll also be using three files from "Study 2", so make sure they're in your folder: README_Study2.txt, Study2_Subjects.csv, and Study2_Trials.csv.

