R Tutorials Top Banner

DATA FRAMES


Preamble

There is plenty to say about data frames because they are the primary data structure in R. Some of what follows is essential knowledge. Some of it will be satisfactorily learned for now if you remember that "R can do that." I will try to point out which parts are which. Set aside some time. This is a long one!


Definition and Examples (essential)

A data frame is a table, or two-dimensional array-like structure, in which each column contains measurements on one variable, and each row contains one case. As we shall see, a "case" is not necessarily the same as an experimental subject or unit, although they are often the same. Technically, in R a data frame is a list of column vectors, although there is only one reason why you might need to remember such an arcane thing. Unlike an array, the data you store in the columns of a data frame can be of various types. I.e., one column might be a numerical variable, another might be a factor, and a third might be a character variable. All columns have to be the same length (contain the same number of data items).

Let's say we've collected data on one response variable or DV from 15 subjects, who were divided into three experimental groups called control ("contr"), treatment one ("treat1"), and treatment two ("treat2"). We might be tempted to table the data as follows...

contr     treat1    treat2
---------------------------
  22        32        30
  18        35        28
  25        30        25
  25        42        22
  20        31        33
---------------------------
While this is a perfectly acceptable table, it is NOT a data frame, because values on our one response variable have been divided into three columns (and so have values on the grouping or independent variable). A data frame has the name of the variable at the top of the column, and values of that variable in the column under the variable name. So the data above should be tabled as follows...
scores     group
----------------
  22       contr
  18       contr
  25       contr
  25       contr
  20       contr
  32      treat1
  35      treat1
  30      treat1
  42      treat1
  31      treat1
  30      treat2
  28      treat2
  25      treat2
  22      treat2
  33      treat2
----------------
This is a proper data frame (and leave out the dashed lines, although in actual fact R could read this table just as you see it here). It does not matter what order you type the columns in, as long as each column contains values of one variable, and every recorded value of that variable is in that column.

In a previous tutorial we used the data object "women" as an example of a data frame...

> women
   height weight
1      58    115
2      59    117
3      60    120
4      61    123
5      62    126
6      63    129
7      64    132
8      65    135
9      66    139
10     67    142
11     68    146
12     69    150
13     70    154
14     71    159
15     72    164
In this data frame we have two numerical variables and no real explanatory variables (IVs) or response variables (DVs). Notice when R prints out a data frame, it numbers the rows. These numbers are for convenience only and are not part of the data frame, and I'll have much more to say about them shortly.

We can refer to any value, or subset of values, in this data frame using the already familiar notation...

> women[12,2]                          # row 12, column 2; note: square brackets
[1] 150
> women[8,]                            # row 8, all columns
  height weight
8     65    135
> women[1:5,]                          # rows 1 to 5, all columns
  height weight
1     58    115
2     59    117
3     60    120
4     61    123
5     62    126
> women[,2]                            # all rows, column 2
 [1] 115 117 120 123 126 129 132 135 139 142 146 150 154 159 164
> women[c(1,3,7,13),]                  # rows 1, 3, 7, and 13, all columns
   height weight
1      58    115
3      60    120
7      64    132
13     70    154
> women[c(1,3,7,13),1]                 # rows 1, 3, 7, and 13, column 1
[1] 58 60 64 70
Here's the catch. Those index numbers do NOT necessarily correspond to the numbers you see printed out with the data frame. This can be confusing at first, and it is something you need to keep in mind. I will explain in a moment.

Another built-in data object that is a data frame is "warpbreaks". This data frame contains 54 cases, so I will print out only every third one. I do this with the sequence function, since this function creates a vector just as the c( ) function did in the above examples...

> warpbreaks[seq(1,54,3),]
   breaks wool tension
1      26    A       L
4      25    A       L
7      51    A       L
10     18    A       M
13     17    A       M
16     35    A       M
19     36    A       H
22     18    A       H
25     28    A       H
28     27    B       L
31     19    B       L
34     41    B       L
37     42    B       M
40     16    B       M
43     21    B       M
46     20    B       H
49     17    B       H
52     15    B       H
In this data frame we have one numerical variable (number of breaks), and two categorical variables (type of wool and tension on the wool). We don't have to look at the data frame itself to get this information. We can also use the str( ) function, which displays a breakdown of the structure of a data frame...
> str(warpbreaks)
'data.frame':   54 obs. of  3 variables:
 $ breaks : num  26 30 54 25 70 52 51 26 67 18 ...
 $ wool   : Factor w/ 2 levels "A","B": 1 1 1 1 1 1 1 1 1 1 ...
 $ tension: Factor w/ 3 levels "L","M","H": 1 1 1 1 1 1 1 1 1 2 ...

Another example is the data object "sleep"...

> sleep
   extra group
1    0.7     1
2   -1.6     1
3   -0.2     1
4   -1.2     1
5   -0.1     1
6    3.4     1
7    3.7     1
8    0.8     1
9    0.0     1
10   2.0     1
11   1.9     2
12   0.8     2
13   1.1     2
14   0.1     2
15  -0.1     2
16   4.4     2
17   5.5     2
18   1.6     2
19   4.6     2
20   3.4     2
Here we have two variables, the change in sleep time a subject got ("extra"), and what drug the subject received ("group"). In this case, the first variable (the dependent variable, DV, response variable, etc.) is numerical and the second (the independent variable, IV, explanatory variable, grouping variable, etc.) is categorical, even though the categorical variable is coded as a number. Once again, it does not matter in what order the columns occur. Put the IV in the first column and the DV in the second column if you want.

However, if categorical variables are coded as numbers (a common practice), R will not know this until you tell it...

> str(sleep)
'data.frame':   20 obs. of  2 variables:
 $ extra: num  0.7 -1.6 -0.2 -1.2 -0.1 3.4 3.7 0.8 0 2 ...
 $ group: Factor w/ 2 levels "1","2": 1 1 1 1 1 1 1 1 1 1 ...
In this case, the fact that "group" is a factor is stored internally in the data frame, but that will not always be the case. So it's worth taking a look to make sure things you intend to be factors are being interpreted as factors by R. You can do this with str( ), but you can also do it with summary( ), because numerical variables and factors are summarized differently...
> summary(sleep)
     extra        group 
 Min.   :-1.600   1:10  
 1st Qu.:-0.025   2:10  
 Median : 0.950         
 Mean   : 1.540         
 3rd Qu.: 3.400         
 Max.   : 5.500
Notice that numerical variables (extra) are summarized with numerical summary statistics, while factors are summarized with a frequency table. In these data, there are 10 subjects in group 1 and 10 subjects in group 2.


An Ambiguous Case (essential)

Entering data into a data frame sometimes involves making a tough decision as to what your variables are. The following example is from a built-in data object called "anorexia". This data set is not in the libraries that are loaded by default when R starts, so to see it, the first thing we need to do is attach the correct library to the search path. Let's see how that works...

> search()
 [1] ".GlobalEnv"        "tools:RGUI"        "package:stats"    
 [4] "package:graphics"  "package:grDevices" "package:utils"    
 [7] "package:datasets"  "package:methods"   "Autoloads"        
[10] "package:base"
This is the default search path, the one you have right after R starts. (It will be a little different in different operating systems.) We want to see an object in the MASS library (or package), which is not currently in the search path. So to get it into the search path, do this...
> library(MASS)
> search()
 [1] ".GlobalEnv"        "package:MASS"      "tools:RGUI"       
 [4] "package:stats"     "package:graphics"  "package:grDevices"
 [7] "package:utils"     "package:datasets"  "package:methods"  
[10] "Autoloads"         "package:base"
Notice we have added "package:MASS" to the search path in position 2. This means if we request an R object, R will look first in the global environment (the workspace), and if the object is not found there, R will look next in MASS, then in RGUI, then in stats, and so on, until the object either is found or R runs out of places to look for it. The "anorexia" data frame is 72 cases long, so to conserve space we will look at only every fifth row of it...
> anorexia[seq(1,72,5),]
   Treat Prewt Postwt
1   Cont  80.7   80.2
6   Cont  88.3   78.1
11  Cont  77.6   77.4
16  Cont  77.3   77.3
21  Cont  85.5   88.3
26  Cont  89.0   78.8
31   CBT  79.9   76.4
36   CBT  80.5   82.1
41   CBT  70.0   90.9
46   CBT  84.2   83.9
51   CBT  83.3   85.2
56    FT  83.8   95.2
61    FT  79.6   76.7
66    FT  81.6   77.8
71    FT  86.0   91.7
The data frame contains data from women who underwent treatment for anorexia. In the first column we have the treatment variable ("Treat"). The second column contains the pretreatment body weight in pounds ("Prewt"). The third column contains the posttreatment body weight in pounds ("Postwt"). So where is the ambiguity?

Here's the awkward question. In our analysis of these data, do we wish to treat weight as two variables (pre and post) each measured once on each subject, or as one variable (weight) measured twice on each subject? The data frame is currently arranged as if the plan was for an analysis of covariance, with "Postwt" being the response, "Treat" the explanatory variable, and "Prewt" the covariate. Prewt and Postwt are treated as two variables.

If the plan was for a repeated measures ANOVA, then the data frame is wrong, because in this case, "weight" is ONE variable measured twice ("pre" and "post") on each woman. In this analysis, we would also need to add a "subject" variable to the data frame as well, since each subject would have two lines, a "pre" line and a "post" line.

It's not a disaster. The data frame is easy enough to rearrange on the fly, and we will do so below.

By the way, this is how you get the MASS package out of the search path if you no longer need it...

> detach("package:MASS")


Creating a Data Frame in R (essential)

The easiest way--and the usual way--of getting a data frame into the R workspace is to read it in from a file. We will do that in the next tutorial. Sometimes it becomes necessary to create one at the console, however. Here are the steps involved:

  • Type each variable into a vector.
  • Use the data.frame( ) function to create a data frame from the vectors.

You may remember these data from the "Objects" tutorial...

name     age  hgt  wgt  race year   SAT 
Bob       21   70  180  Cauc   Jr  1080
Fred      18   67  156 Af.Am   Fr  1210
Barb      18   64  128 Af.Am   Fr   840
Sue       24   66  118  Cauc   Sr  1340
Jeff      20   72  202 Asian   So   880
Let's make a data frame of this...
> ls()                            # A clean workspace is a good start!
character(0)
> name = scan(what="character")
1: Bob Fred Barb Sue Jeff         # Remember: press Enter twice to end data entry.
6: 
Read 5 items
> age = scan()
1: 21 18 18 24 20
6: 
Read 5 items
> hgt = scan()
1: 70 67 64 66 72
6: 
Read 5 items
> wgt = scan()
1: 180 156 128 1118 202
6: 
Read 5 items
> race = scan(what="character")
1: Cauc Af.Am Af.Am Cauc Asian
6: 
Read 5 items
> year = scan(what="character")
1: Jr Fr Fr Sr So
6: 
Read 5 items
> SAT = scan()
1: 1080 1210 840 1340 880
6: 
Read 5 items
> my.data = data.frame(name, age, hgt, wgt, race, year, SAT)
> my.data
  name age hgt  wgt  race year  SAT
1  Bob  21  70  180  Cauc   Jr 1080
2 Fred  18  67  156 Af.Am   Fr 1210
3 Barb  18  64  128 Af.Am   Fr  840
4  Sue  24  66 1118  Cauc   Sr 1340
5 Jeff  20  72  202 Asian   So  880
Tah dah! It's as simple as that. You wouldn't want to have to do that with a large data set, however, and that's why we'll learn how to read them in from a file in the next tutorial. DON'T clean up your workspace. We will carry this example over into the next section.


Accessing Information Inside a Data Frame (essential)

First, let's look at a few functions that allow us to get general information about a data frame...

> dim(my.data)               # Get size in rows by columns.
[1] 5 7
> names(my.data)             # Get the names of variables in the data frame.
[1] "name" "age"  "hgt"  "wgt"  "race" "year" "SAT" 
> str(my.data)               # See the internal structure of the data frame.
'data.frame':   5 obs. of  7 variables:
 $ name: Factor w/ 5 levels "Barb","Bob","Fred",..: 2 3 1 5 4
 $ age : num  21 18 18 24 20
 $ hgt : num  70 67 64 66 72
 $ wgt : num  180 156 128 1118 202
 $ race: Factor w/ 3 levels "Af.Am","Asian",..: 3 1 1 3 2
 $ year: Factor w/ 4 levels "Fr","Jr","So",..: 2 1 1 4 3
 $ SAT : num  1080 1210 840 1340 880
These are self-explanatory, with the exception of str( ). First, notice that our character variables were entered into the data frame as factors. This is standard in R, but it may not be what you want. Second, notice on the lines giving info about factors that there are strange numbers at the ends of those lines. You don't have to worry about these. What R is telling you is that factors are coded internally in R as numbers. R will keep it all straight for you, so don't sweat the details.

The summary( ) function is also useful here...

> summary(my.data)
   name        age            hgt            wgt            race   year  
 Barb:1   Min.   :18.0   Min.   :64.0   Min.   : 128.0   Af.Am:2   Fr:2  
 Bob :1   1st Qu.:18.0   1st Qu.:66.0   1st Qu.: 156.0   Asian:1   Jr:1  
 Fred:1   Median :20.0   Median :67.0   Median : 180.0   Cauc :2   So:1  
 Jeff:1   Mean   :20.2   Mean   :67.8   Mean   : 356.8             Sr:1  
 Sue :1   3rd Qu.:21.0   3rd Qu.:70.0   3rd Qu.: 202.0                   
          Max.   :24.0   Max.   :72.0   Max.   :1118.0                   
      SAT      
 Min.   : 840  
 1st Qu.: 880  
 Median :1080  
 Mean   :1070  
 3rd Qu.:1210  
 Max.   :1340
Or at least that would be useful if the data frame were larger!

There are four ways to get at the data inside a data frame, and this is NOT one of them...

> SAT
[1] 1080 1210  840 1340  880
That only seemed to work, because remember when you created the data frame, you started by putting a vector called "SAT" into the workspace. THAT'S what you're seeing now! You are not seeing the SAT variable from inside the data frame.

Let's erase all those vectors EXCEPT "age", which we will keep to illustrate something that you will need to remember about R...

> ls()
[1] "age"     "hgt"     "my.data" "name"    "race"    "SAT"     "wgt"    
[8] "year"   
> rm(hgt, name, race, SAT, wgt, year)  ### Don't erase my.data!
> ls()
[1] "age"     "my.data"
Now if we try to see SAT as we did above...
> SAT
Error: object 'SAT' not found
...we get an error. R will not look inside data frames for variables unless you tell it to. Here are the four ways to do that...

  • by using $
  • by using with( )
  • by using data=
  • by using attach( )

A data frame is a list of column vectors. We can extract items from inside it by using the usual list indexing device, $. To do this, type the name of the data frame, a dollar sign, and the name of the variable you want to work with...

> my.data$SAT
[1] 1080 1210  840 1340  880
> mean(my.data$SAT)
[1] 1070
If that dollar sign stuff gets hard to read, you can put spaces around the $ to make the command line easier to read...
> mean(my.data $ SAT)
[1] 1070
This can certainly be a nuisance, because it will mean that in some commands you have to type the data frame name multiple times. An example is the command that calculates a correlation...
> cor(my.data$hgt, my.data$wgt)
[1] -0.2531835
In this case, you can use the with( ) function to tell R where to get the data from...
> with(my.data, cor(hgt, wgt))
[1] -0.2531835
It doesn't save much typing in this example, but there are cases where that will save a LOT of typing! Notice the syntax of this function. You type the name of the data frame first, followed by a comma, followed by the function you want to execute, then you close the parentheses on with( ).

As we will learn later, some functions, especially significance tests, take what's called a formula interface. When that's the case, there is always a data= option to specify the name of the data frame where the variables are to be found. I'll just show you an example for now. We'll have plenty of time to examine the formula interface later...

> cor.test( ~ hgt + wgt, data=my.data)

        Pearson's product-moment correlation

data:  hgt and wgt 
t = -0.4533, df = 3, p-value = 0.6811
alternative hypothesis: true correlation is not equal to 0 
95 percent confidence interval:
 -0.9281289  0.8100218 
sample estimates:
       cor 
-0.2531835

Finally, there is the dreaded attach( ) function. This attaches the data frame to your search path (in position 2) so that R will know to look there for data objects that are referenced by name. Some people use this device routinely when working with data frames, but it can cause problems, and we are about to see one...

> attach(my.data)

        The following object(s) are masked _by_ .GlobalEnv :

         age
Say what? When an object is masked (or shadowed) by the global environment, that means there is a data object in the workspace that has this name AND there is a variable inside the data frame that has this name. I can now ask for any variable inside the data frame EXCEPT age...
> SAT
[1] 1080 1210  840 1340  880
> mean(SAT)
[1] 1070
> table(year)
year
Fr Jr So Sr 
 2  1  1  1 
> age
[1] 21 18 18 24 20
You might think you are seeing my.data$age here, but YOU ARE NOT! You're seeing "age" from the workspace. In this case they're the same, but that won't always be true...
> age = 112
> age
[1] 112
The assignment changed the value of "age" in the workspace, but not in the data frame...
> my.data$age
[1] 21 18 18 24 20
If we remove age from the workspace, R will then search inside the data frame for it...
> rm(age)
> age
[1] 21 18 18 24 20
The lesson is, when you get one of these masking (or shadowing) conflicts, WATCH OUT! Be extra careful to know which version of the variable you're working with. This has tripped up many an R user, including me. This is why you want to keep your workspace as clean as possible. The best strategy here is to remove the masking variable from the workspace. If you want to keep it, at least rename it and then remove the conflicting version from the workspace. You'll eventually be sorry if you don't!

One more lesson...

> detach(my.data)
When you're done with an attached data frame, ALWAYS detach it. This will remove it from the search path so that R will no longer look inside it for variables. You'll have to go back to using $ to reference variables inside the data frame after it is detached. This isn't necessary if you're going to quit your R session right away. Quitting detaches everything that was attached. But if you're going to continue working, detach data frames you no longer need. Otherwise, your search path will get messy, and you'll get more and more masking conflicts as other objects are attached.

DON'T erase my.data. We still need it.


Data Frame Indexing and Row Names (critical)

This will cost you BIGTIME eventually if you don't pay close attention!

> ls()                            # Still there?
[1] "my.data"
> my.data
  name age hgt  wgt  race year  SAT
1  Bob  21  70  180  Cauc   Jr 1080
2 Fred  18  67  156 Af.Am   Fr 1210
3 Barb  18  64  128 Af.Am   Fr  840
4  Sue  24  66 1118  Cauc   Sr 1340
5 Jeff  20  72  202 Asian   So  880
Let's talk about those line numbers at the leftmost verge of the printed data frame. THEY ARE NOT NUMBERS. Let me repeat that. THEY ARE NOT NUMBERS. They are row names. So the rows and columns of this data frame are NAMED as follows:
> dimnames(my.data)
[[1]]
[1] "1" "2" "3" "4" "5"

[[2]]
[1] "name" "age"  "hgt"  "wgt"  "race" "year" "SAT"
What's the big deal?

Look at the printed data frame. Suppose we wanted to extract Barb's weight. That's the value in row 3 and column 4, so we could get it this way...

> my.data[3,4]                    # Remember to use square brackets for indexing.
[1] 128
"Yeah, so?" We could also get it this way...
> my.data[3,"wgt"]
[1] 128
...and this way...
> my.data["3","wgt"]
[1] 128
Those last two ways seem to be the same, BUT THEY ARE NOT!!!

Let's sort the data frame using the age variable. Sorting a data frame is done using the order( ) function. Remember how it worked when we sorted a vector? If a call to the order( ) function is put in place of the row index the data frame will be sorted on whatever variable is specified inside that function. You will have to use the full name of the variable; i.e., you will have to use the $ notation. (Why?) Otherwise, R will be looking in your workspace for a variable called "age", not finding it, and giving a "not found" error. It happens to me a lot, so you might as well just get used to it!

> my.data[order(my.data$age),]
  name age hgt  wgt  race year  SAT
2 Fred  18  67  156 Af.Am   Fr 1210
3 Barb  18  64  128 Af.Am   Fr  840
5 Jeff  20  72  202 Asian   So  880
1  Bob  21  70  180  Cauc   Jr 1080
4  Sue  24  66 1118  Cauc   Sr 1340
Observe the row names! They have also sorted, haven't they? Let's save this into a new data object so we can play with it a bit...
> my.data[order(my.data$age),] -> my.data.sorted      # Did you remember up arrow?
> my.data.sorted
  name age hgt  wgt  race year  SAT
2 Fred  18  67  156 Af.Am   Fr 1210
3 Barb  18  64  128 Af.Am   Fr  840
5 Jeff  20  72  202 Asian   So  880
1  Bob  21  70  180  Cauc   Jr 1080
4  Sue  24  66 1118  Cauc   Sr 1340
Now let's try to extract Barb's weight from this new data frame...
> my.data.sorted[3,4]                  ### Wrong!
[1] 202
> my.data.sorted[3,"wgt"]              ### Also wrong!
[1] 202
> my.data.sorted["3","wgt"]            ### Correct!
[1] 128
> my.data.sorted[2,4]                  ### Also correct!
[1] 128
Confused yet?

Here's what you have to remember. Those numbers that often print out on the left side of a data frame ARE NOT NUMBERS. They're row names. So data frames have both row and column names, whether you like it or not! The point becomes clearer when we give the rows actual names. Let's erase the names from my.data and then re-enter them as row names...

> rm(my.data.sorted)                   # Get rid of that first.
> my.data$name <- NULL                 # This is how you erase a variable.
> my.data                              # See?
  age hgt  wgt  race year  SAT
1  21  70  180  Cauc   Jr 1080
2  18  67  156 Af.Am   Fr 1210
3  18  64  128 Af.Am   Fr  840
4  24  66 1118  Cauc   Sr 1340
5  20  72  202 Asian   So  880
> rownames(my.data) <- c("Bob","Fred","Barb","Sue","Jeff")
> my.data
     age hgt  wgt  race year  SAT
Bob   21  70  180  Cauc   Jr 1080
Fred  18  67  156 Af.Am   Fr 1210
Barb  18  64  128 Af.Am   Fr  840
Sue   24  66 1118  Cauc   Sr 1340
Jeff  20  72  202 Asian   So  880
> my.data["Barb", "wgt"]               # Makes getting Barb's weight a lot easier!
[1] 128
Notice the numbers are gone now because we have actual row names. And OF COURSE they sort with the rest of the data frame...
> my.data[order(my.data$age),]
     age hgt  wgt  race year  SAT
Fred  18  67  156 Af.Am   Fr 1210
Barb  18  64  128 Af.Am   Fr  840
Jeff  20  72  202 Asian   So  880
Bob   21  70  180  Cauc   Jr 1080
Sue   24  66 1118  Cauc   Sr 1340
It would be absolutely silly if they didn't! Just remember: Data frames ALWAYS have row names. Sometimes those row names just happen to look like numbers. It's the row names that print out to your console when you ask to see the data frame, or any part of it, and NOT the index numbers.

Don't remove my.data yet. We still need it.


Modifying a Data Frame (not so essential just now)

Rule number one with a bullet:

  • NEVER MODIFY AN ATTACHED DATA FRAME!

While this isn't strictly against the law, it's a bad idea and can get very confusing as to exactly what it is you've modified. I could try to explain it, but I'm not sure I understand it myself. So just don't do it!

The time will come when you want to change a data frame in some way. Here are some examples of how to do that. You may have noticed that Sue (in the my.data data frame) is a wee bit on the chunky side. This was an innocent mistake. I really didn't do that on purpose. How do we fix it? The value was supposed to be 118, but let's change it to 135 just for kicks...

> ls()                                 # Still there?
[1] "my.data"
> my.data
     age hgt  wgt  race year  SAT
Bob   21  70  180  Cauc   Jr 1080
Fred  18  67  156 Af.Am   Fr 1210
Barb  18  64  128 Af.Am   Fr  840
Sue   24  66 1118  Cauc   Sr 1340
Jeff  20  72  202 Asian   So  880
> my.data["Sue","wgt"] <- 135
> my.data
     age hgt wgt  race year  SAT
Bob   21  70 180  Cauc   Jr 1080
Fred  18  67 156 Af.Am   Fr 1210
Barb  18  64 128 Af.Am   Fr  840
Sue   24  66 135  Cauc   Sr 1340
Jeff  20  72 202 Asian   So  880
That's all there is to it. Use any kind of indexing you like. Let's use numerical indexing to give Sue her correct weight...
> my.data[4,3] <- 118
> my.data
     age hgt wgt  race year  SAT
Bob   21  70 180  Cauc   Jr 1080
Fred  18  67 156 Af.Am   Fr 1210
Barb  18  64 128 Af.Am   Fr  840
Sue   24  66 118  Cauc   Sr 1340
Jeff  20  72 202 Asian   So  880
Just remember that "wgt" is now in column 3, since the row names don't count as a column.

I have to warn you about modifying data frames. It's always a good idea to make a backup copy in the workspace first. Because there are some commands that modify data frames that, if they go wrong, can really screw things up! But let's live dangerously. Suppose we wanted "wgt" to be in kilograms instead of pounds. Easy enough...

> my.data$wgt / 2.2
[1] 81.81818 70.90909 58.18182 53.63636 91.81818
> my.data                                # Nothing has changed yet. Why not?
     age hgt wgt  race year  SAT
Bob   21  70 180  Cauc   Jr 1080
Fred  18  67 156 Af.Am   Fr 1210
Barb  18  64 128 Af.Am   Fr  840
Sue   24  66 118  Cauc   Sr 1340
Jeff  20  72 202 Asian   So  880
> my.data$wgt / 2.2 -> my.data$wgt       # Aha! It has to be stored back into my.data.
> my.data
     age hgt      wgt  race year  SAT
Bob   21  70 81.81818  Cauc   Jr 1080
Fred  18  67 70.90909 Af.Am   Fr 1210
Barb  18  64 58.18182 Af.Am   Fr  840
Sue   24  66 53.63636  Cauc   Sr 1340
Jeff  20  72 91.81818 Asian   So  880
> round(my.data$wgt, 1) -> my.data$wgt   # A little rounding for good measure.
> my.data
     age hgt  wgt  race year  SAT
Bob   21  70 81.8  Cauc   Jr 1080
Fred  18  67 70.9 Af.Am   Fr 1210
Barb  18  64 58.2 Af.Am   Fr  840
Sue   24  66 53.6  Cauc   Sr 1340
Jeff  20  72 91.8 Asian   So  880
Now that we've rounded them off, we've lost the original weight data in pounds...
> my.data$wgt*2.2
[1] 179.96 155.98 128.04 117.92 201.96
We could have avoided this by making a backup copy of my.data first, or by putting the new weight in kilograms into a new column in the data frame.

Let's see how to create a new column in the data frame...

> my.data$IQ = c(115, 122, 100, 144, 96)
> my.data
     age hgt  wgt  race year  SAT  IQ
Bob   21  70 81.8  Cauc   Jr 1080 115
Fred  18  67 70.9 Af.Am   Fr 1210 122
Barb  18  64 58.2 Af.Am   Fr  840 100
Sue   24  66 53.6  Cauc   Sr 1340 144
Jeff  20  72 91.8 Asian   So  880  96
Just name it and assign values to the name in a vector. The new vector has to be the same length as the other variables already in the data frame.

You can clean up now. We're done with this data frame.


Missing Values (kinda important, so listen up!)

Do this...

> library(MASS)
> data(Cars93)
> attach(Cars93)
> str(Cars93)                     # Output not shown.
This is a data frame with 93 observations on 27 variables. You can see what the variables represent by looking at the help page for this data set: ?Cars93. We're interested in the variable "Luggage.room" in particular, which is the trunk space in cubic feet, to the nearest cubic foot...
> summary(Luggage.room)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
   6.00   12.00   14.00   13.89   15.00   22.00   11.00
This is a numerical variable, so we get the summary we are accustomed to by now. But what are those NAs? Whether we like it or not, data sets often have missing values, and we need to know how to deal with them. R's standard code for missing values is "NA", for "not available". The number associated with NA is a frequency. There are 11 cases in this data frame in which "Luggage.room" is a missing value. If you looked at the help page, you know why.

Some functions fail to work when there are missing values, but this can (almost always) be fixed with a simple option...

> mean(Luggage.room)
[1] NA
> mean(Luggage.room, na.rm=TRUE)
[1] 13.89024
> mean(Luggage.room, na.rm=T)
[1] 13.89024
There is no mean when some of the values are missing, so the "na.rm" option removes them when set to TRUE (must be all caps, but the shorter form T also works provided you haven't assigned another value to it). If you want to clean the data set by removing casewise all cases with missing values on any variable, use the na.omit( ) function...
> na.omit(Cars93)                 # Output not shown.
I will not reproduce the output here because it is extensive, but it is also instructive, so take a look at it. Scroll the console window backwards to see all of it. Of course, to use this cleaned data frame, you would have to assign it to a new data object.

The which( ) function does not work to identify which of the values are missing. Use is.na( ) instead...

> which(Luggage.room == NA)
integer(0)
> is.na(Luggage.room)
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[12] FALSE FALSE FALSE FALSE  TRUE  TRUE FALSE  TRUE FALSE FALSE FALSE
[23] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[34] FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[45] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[56]  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
[67] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[78] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
[89]  TRUE FALSE FALSE FALSE FALSE
> which(is.na(Luggage.room))
 [1] 16 17 19 26 36 56 57 66 70 87 89
Finally, some data sets come with other codes for missing values. 999 is a common missing value code, as are blank spaces. Blanks are a very bad idea. If you find a data set with blanks in it, it may have to be edited in a text editor or spreadsheet before the file can be read into R. It depends on how the file is formatted. In some cases, R will automatically assign NA to blank values, but in other cases it will not. Other missing value codes are not a problem, as they can be recoded...
> ifelse(is.na(Luggage.room), 999, Luggage.room) -> temp
> temp
 [1]  11  15  14  17  13  16  17  21  14  18  14  13  14  13  16 999 999
[18]  20 999  15  14  17  11  13  14 999  16  11  11  15  12  12  13  12
[35]  18 999  18  21  10  11   8  12  14  11  12   9  14  15  14   9  19
[52]  22  16  13  14 999 999  12  15   6  15  11  14  12  14 999  14  14
[69]  16 999  17   8  17  13  13  16  18  14  12  10  15  14  10  11  13
[86]  15 999  10 999  14  15  14  15
> # first we'll mess it up
> # and then we'll fix it
> ifelse(temp == 999, NA, temp) -> fixed
> fixed
 [1] 11 15 14 17 13 16 17 21 14 18 14 13 14 13 16 NA NA 20 NA 15 14 17 11
[24] 13 14 NA 16 11 11 15 12 12 13 12 18 NA 18 21 10 11  8 12 14 11 12  9
[47] 14 15 14  9 19 22 16 13 14 NA NA 12 15  6 15 11 14 12 14 NA 14 14 16
[70] NA 17  8 17 13 13 16 18 14 12 10 15 14 10 11 13 15 NA 10 NA 14 15 14
[93] 15
The ifelse( ) function is very handy for recoding a data vector, so let me take a moment to explain it. Inside the parentheses, the first thing you give is a test. In the second of these commands above, where we are going from the messed up variable back to "fixed", the test was "if any value of temp is equal to 999". Notice the double equals sign meaning "equal". (I still get this wrong a lot!) The second thing you give is how to recode those values, and finally you tell what to do with the values that don't pass the test. So the whole command reads like this: "If any value of temp is equal to 999, assign it the value NA, else assign it the value that is currently in temp."

In the first instance of the function, we had to use is.na, since nothing can really be "equal to" something that is not available! Try these, and say them in words as you're typing them...

> ifelse(fixed == 10, 0, 100)          # Output not shown.
> ifelse(fixed > 10, 100, 0)           # Output not shown.
> ifelse(fixed > 10, "big", "small")   # Output not shown.
If you stored that last one, it would create a character vector.

Don't forget to clean up your workspace and search path!!


Subsetting a Data Frame (optional)

We will use a data frame called USArrests for this exercise...

> data(USArrests)
> head(USArrests)
           Murder Assault UrbanPop Rape
Alabama      13.2     236       58 21.2
Alaska       10.0     263       48 44.5
Arizona       8.1     294       80 31.0
Arkansas      8.8     190       50 19.5
California    9.0     276       91 40.6
Colorado      7.9     204       78 38.7
Here is another useful function for looking at a data frame. The head( ) function shows the first six lines of data (cases) inside a data frame. There is also a tail( ) function that shows the last six lines, and the number of lines shown can be changed with an option (see the help pages).

In this case we have a data frame with row names set to state names and containing variables that give the crime rates (per 100,000 population) for Murder, Assault, and Rape, as well as the percentage of the population that lives in urban areas. These data are from 1973 so are not current.

Because state names are used as row names, to see the data for any state, all we have to be able to do is spell the name of the state...

> USArrests["Pennsylvania",]      # No column index, so all columns displayed.
             Murder Assault UrbanPop Rape
Pennsylvania    6.3     106       72 14.9
We do not have to figure out what the index number would be for that row. Thus, explicit row names can be very handy. To display the entire row of data for PA, we just left out the column index, but THE COMMA STILL HAS TO BE THERE! Otherwise, you are trying to index a two-dimensional data object using only one index, and R will tell you to knock it off!

Let's answer the following questions from these data...

  • Which state has the lowest murder rate?
  • Which states have murder rates less than 4.0?
  • Which states are in the top quartile for urban population?

> min(USArrests$Murder)                     # What is the minimum murder rate?
[1] 0.8
> which(USArrests$Murder == 0.8)            # Which line of the data is that?
[1] 34
> USArrests[34,]                            # Give me the data from that line.
             Murder Assault UrbanPop Rape
North Dakota    0.8      45       44  7.3
>
> which(USArrests$Murder < 4.0)             # Gives the result in a vector.
 [1]  7 12 15 19 23 29 34 39 41 44 45 49
> USArrests[which(USArrests$Murder < 4.0),] # Use that vector as an index.
              Murder Assault UrbanPop Rape
Connecticut      3.3     110       77 11.1
Idaho            2.6     120       54 14.2
Iowa             2.2      56       57 11.3
Maine            2.1      83       51  7.8
Minnesota        2.7      72       66 14.9
New Hampshire    2.1      57       56  9.5
North Dakota     0.8      45       44  7.3
Rhode Island     3.4     174       87  8.3
South Dakota     3.8      86       45 12.8
Utah             3.2     120       80 22.9
Vermont          2.2      48       32 11.2
Wisconsin        2.6      53       66 10.8
>
> summary(USArrests$UrbanPop)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  32.00   54.50   66.00   65.54   77.75   91.00 
> USArrests[which(USArrests$UrbanPop >= 77.75),]
              Murder Assault UrbanPop Rape
Arizona          8.1     294       80 31.0
California       9.0     276       91 40.6
Colorado         7.9     204       78 38.7
Florida         15.4     335       80 31.9
Hawaii           5.3      46       83 20.2
Illinois        10.4     249       83 24.0
Massachusetts    4.4     149       85 16.3
Nevada          12.2     252       81 46.0
New Jersey       7.4     159       89 18.8
New York        11.1     254       86 26.1
Rhode Island     3.4     174       87  8.3
Texas           12.7     201       80 25.5
Utah             3.2     120       80 22.9

Suppose we wanted to work with data only from these states. How can we extract them from the data frame and make a new data frame that contains only those states? I'm glad you asked...

> subset(USArrests, subset=(UrbanPop >=77.75)) -> high.urban
> high.urban
              Murder Assault UrbanPop Rape
Arizona          8.1     294       80 31.0
California       9.0     276       91 40.6
Colorado         7.9     204       78 38.7
Florida         15.4     335       80 31.9
Hawaii           5.3      46       83 20.2
Illinois        10.4     249       83 24.0
Massachusetts    4.4     149       85 16.3
Nevada          12.2     252       81 46.0
New Jersey       7.4     159       89 18.8
New York        11.1     254       86 26.1
Rhode Island     3.4     174       87  8.3
Texas           12.7     201       80 25.5
Utah             3.2     120       80 22.9
The subset( ) function does the trick. The syntax is a little squirrelly, so let me go through it. The first thing you give is the name of the data frame. That is followed by the subset= option. Then inside of parentheses (which actually aren't necessary) give the test that defines the subset. Store the output into a new data object so that you can then work with it. Functions that take a data= option can also take a subset option, so it's a useful thing to know.

You can clean up your workspace now.


Stacking and Unstacking (optional)

Suppose someone has retained your services as a data analyst and gives you his data (from an Excel file or something) in this format...

contr     treat1    treat2
---------------------------
  22        32        30
  18        35        28
  25        30        25
  25        42        22
  20        31        33
---------------------------
If you're working for free, you can yell at him and make him do it the right way, but if you're being paid, you probably really shouldn't. Here's how to deal with it. First, let's get these data into a "data frame" in this format, and I will leave out the command prompts so that you can just copy and paste these three lines directly into R...
### start copying here
wrong.data = data.frame(contr = c(22,18,25,25,20),
                        treat1 = c(32,35,30,42,31),
                        treat2 = c(30,28,25,22,33))
### stop copying here
> wrong.data
  contr treat1 treat2
1    22     32     30
2    18     35     28
3    25     30     25
4    25     42     22
5    20     31     33
Now do this...
> stack(wrong.data) -> correct.data
> correct.data
   values    ind
1      22  contr
2      18  contr
3      25  contr
4      25  contr
5      20  contr
6      32 treat1
7      35 treat1
8      30 treat1
9      42 treat1
10     31 treat1
11     30 treat2
12     28 treat2
13     25 treat2
14     22 treat2
15     33 treat2
And there you go. Now you have a proper data frame.

There is also an unstack( ) function that does the reverse of this, and it will work automatically on a data frame that has been created by stack( ), but otherwise is a little trickier to use. You probably won't have to use it much, so I'll refer you to the help page if you ever need it.

You can remove these data objects. We won't use them again.


Going From Wide to Long and Long to Wide (eventually you'll probably need to know this)

I mention this above under "An Ambiguous Case." There are two kinds of data frames in R, and in most statistical software: wide ones and long ones. Let's fetch the "anorexia" data again (and we'll do it without attaching the MASS package this time)...

> data(anorexia, package="MASS")
What we are about to do is a little confusing until you get some experience with it, so it will be necessary to be able to see what's happening. The anorexia data frame is too long to print to a single console screen with causing it to scroll, so I'm going to cut it down to only nine cases, three from each group. This will help us to see the difference between wide and long data frames without constantly scrolling the console window...
> anorexia[c(1,2,3,27,28,29,56,57,58),] -> anor
> anor
   Treat Prewt Postwt
1   Cont  80.7   80.2
2   Cont  89.4   80.1
3   Cont  91.8   86.4
27   CBT  80.5   82.2
28   CBT  84.9   85.6
29   CBT  81.5   81.4
56    FT  83.8   95.2
57    FT  83.3   94.3
58    FT  86.0   91.5
I also shortened up the name of our data frame, because we're going to be typing it a lot.

This is a wide data frame. It's wide because each line of the data frame contains information on ONE SUBJECT, even though that subject was measured multiple times (twice) on weight (Prewt, Postwt). So all the data for each subject goes on ONE LINE, even though we could interpret this as a repeated measures design, or longitudinal data.

In a long data frame, each value of weight would define a case. So each of these subjects would have two lines in such a data frame, one for the subject's Prewt, and one for her Postwt. A wide data frame would be used, for example, in analysis of covariance. A long data frame would be used in repeated measures analysis of variance. Do we have to retype the data frame to get from wide to long? Fortunately not! Because R has a function called reshape( ) which will do the work for us.

It is not an easy function to understand, however (and don't count on the help page being a whole lot of help!). So let me illustrate it, and then I will explain what's happening...

> reshape(data=anor, direction="long",
+         varying=c("Prewt","Postwt"), v.names="Weight",
+         idvar="subject", ids=row.names(anor),
+         timevar="PrePost", times=c("Prewt","Postwt")
+        ) -> anor.long
> anor.long
          Treat PrePost Weight subject
1.Prewt    Cont   Prewt   80.7       1
2.Prewt    Cont   Prewt   89.4       2
3.Prewt    Cont   Prewt   91.8       3
27.Prewt    CBT   Prewt   80.5      27
28.Prewt    CBT   Prewt   84.9      28
29.Prewt    CBT   Prewt   81.5      29
56.Prewt     FT   Prewt   83.8      56
57.Prewt     FT   Prewt   83.3      57
58.Prewt     FT   Prewt   86.0      58
1.Postwt   Cont  Postwt   80.2       1
2.Postwt   Cont  Postwt   80.1       2
3.Postwt   Cont  Postwt   86.4       3
27.Postwt   CBT  Postwt   82.2      27
28.Postwt   CBT  Postwt   85.6      28
29.Postwt   CBT  Postwt   81.4      29
56.Postwt    FT  Postwt   95.2      56
57.Postwt    FT  Postwt   94.3      57
58.Postwt    FT  Postwt   91.5      58
In this example, the first argument I gave to the reshape( ) function was the name of the data frame to be reshaped, and that was given in the data= option. Then I specified the direction= option as "long" so that the data frame would be convert TO a long format.

In the second line of this command, I specified varying= as a vector of variable names in anor that correspond to the repeated measures or longitudinal measures (the time-varying variables). These values will be given in one column in the new data frame, so I named that new column using the v.names= option.

A long data frame needs two things that a wide one does not have. One of those things is a column identifying the subject (case or experimental unit) from which the data in a row of the data frame come from. This is necessary because each subject will have multiple rows of data in a long data frame. So I used the idvar= option to specify the name of this new column that would identify the subjects. I then used ids= to specify how the subjects were to be named. I told it to use the row names from anor, which is a sensible thing to do.

The other thing a long format data frame needs that a wide one does not is a variable giving the condition (or time) in which the subject is being measured for this particular row of data. In the wide format, this information is in the column (variable) names, but that will no longer be true in the long format. We need to know which measure is Prewt and which measure is Postwt for each subject, since these will be on different rows of the data frame in long format. I named this new variable using the timevar= option, and I gave its possible values in a vector using the times= option. The order in which those values should be listed is the same as the order in which the corresponding columns occur in the wide data frame.

Finally, I closed the parentheses on the reshape( ) function and assigned the output to a new data object. Done!

This can also be made to work if you have more than one repeated measures variable, in which case all I can say is may the saints be with you!

If the data frame results from a reshape( ) command, then it can be converted back very simply. All you have to do is this...

> reshape(anor.long)
         Treat subject Prewt Postwt
1.Prewt   Cont       1  80.7   80.2
2.Prewt   Cont       2  89.4   80.1
3.Prewt   Cont       3  91.8   86.4
27.Prewt   CBT      27  80.5   82.2
28.Prewt   CBT      28  84.9   85.6
29.Prewt   CBT      29  81.5   81.4
56.Prewt    FT      56  83.8   95.2
57.Prewt    FT      57  83.3   94.3
58.Prewt    FT      58  86.0   91.5
The row names have gone a little screwy, but all the correct information is there. This isn't very useful actually, because we already have the data in wide format in the data frame anor, which we were smart enough not to overwrite. So let's see how to convert from long to wide the hard way.

First, we will get rid of those ridiculous row names...

> rownames(anor.long) <- as.character(1:18)      # Just do it!
> anor.long
   Treat PrePost Weight subject
1   Cont   Prewt   80.7       1
2   Cont   Prewt   89.4       2
3   Cont   Prewt   91.8       3
4    CBT   Prewt   80.5      27
5    CBT   Prewt   84.9      28
6    CBT   Prewt   81.5      29
7     FT   Prewt   83.8      56
8     FT   Prewt   83.3      57
9     FT   Prewt   86.0      58
10  Cont  Postwt   80.2       1
11  Cont  Postwt   80.1       2
12  Cont  Postwt   86.4       3
13   CBT  Postwt   82.2      27
14   CBT  Postwt   85.6      28
15   CBT  Postwt   81.4      29
16    FT  Postwt   95.2      56
17    FT  Postwt   94.3      57
18    FT  Postwt   91.5      58
And now for the reshaping. I won't bother storing it...
> reshape(data=anor.long, direction="wide",
+         v.names=c("Weight"),
+         idvar="subject",
+         timevar="PrePost"
+        )
  Treat subject Weight.Prewt Weight.Postwt
1  Cont       1         80.7          80.2
2  Cont       2         89.4          80.1
3  Cont       3         91.8          86.4
4   CBT      27         80.5          82.2
5   CBT      28         84.9          85.6
6   CBT      29         81.5          81.4
7    FT      56         83.8          95.2
8    FT      57         83.3          94.3
9    FT      58         86.0          91.5
We didn't quite recover the original table, but then we probably didn't really want to. The first two options name the data frame we are reshaping and tell the direction we are reshaping TO. The next option, v.names=, gives the name of the time-varying variable that will be split into two (or more) columns. The idvar= option gives the name of the variable that is the subject identifier. Finally, the timevar= option gives the name of the variable that contains the conditions under which the longitidinal information was collected; i.e., there were two weights, a Prewt and a Postwt. Notice these values were used to name the two new columns of Weight data. Want a pneumonic to help you remember all that? Yeah, me too!


revised 2010 Aug 1
Return to the Table of Contents