README.md

Getting and Cleaning Data - UCI HAR Dataset

1. The first step is to understand the project. We are given 10299 experiments consisting of 561 observations each. Each of the experiments has a subject (1-30) and an activity (1-6) being performed, and the 561 data points are the readings of accelerometers and gyroscopes in the cell phone the subject is wearing.

   This means: for each experiment we have a matrix (S, A) where S is subject and A is activity, only one row and column of this matrix is TRUE, all the rest are FALSE - the one that's true is row S and column A.

   We have 10299 of these matrices, you can envision them as the third dimension of the (S, A) table. So then we have an (S,A,E) table where E is the experiment number. Each entry in this table along the E axis is a pointer to the 561 observations for that experiment.

2. To make data access a little more efficient, we can change the TRUE/FALSE matrix entry in the (S,A) matrix to be a number, containing the pointer to the observations. So then, all entries in the (S,A) matrix are 0 except for row S and column A which contains a number (a pointer to the observations). This is probably the best representation of the dataaset and the most efficient way to access the data.

3. So now that we understand the experiment and the way we'd like to represent it, we can see what we're given in the raw dataset. We are told there are 30 subjects, they'd like to remain anonymous so we'll just call them 1-30. Then we are given a set of activities in the file activity_labels.txt. So those will be the titles for the (S,A) matrix, we'll have subjects 1-30 rows, and activities 1-6 columns (except labeled with the activity name instead of the number). (By the way the files are all uniquely named, so you can put them all in the same folder if you want, that way you can more conveniently read them into R).

4. Next we have 10299 of these (S,A) matrices, one for each experiment. Let's make a judgement call and organize the overall data frame as an (S,A,E) matrix instead of an (E,A,S) matrix, because a frequent use of this dataset will be the extraction of information per subject and per activity. So essentially the (S,A,E) table acts as an index allowing us to do efficient lookups. If we just want the experiment number we can use [,,E] and then we can see which subject it was and which activity it was.

5. Next, we have two sets of observations, called 'train' and 'test'. Train and test are groups of experiments. An experiment is either in the 'train' group or the 'test' group. As the data is given to us, we have 7352 experiments in the 'train' group and 2947 experiments in the 'test' group. They are conveniently separated for us in advance. So what we can do is have a "group" attribute attached to each entry in the (S,A) matrix. In other words, this experiment with this subject and this activity, was in the <> group.

6. Each experiment is a line item in the raw data tables (either test or train). There's really no need to have separate tables for test and train, for our purposes we'd like to combine the testing and training data into one gigantic table and simply label each entry with the group attribute. So now we can consider some different ways we could represent this in R. One way of doing this would be to use a list of experiments, where each list entry is itself a list, containing the (S, A) matrix, the group ID, and the pointer to the observations. This is easy to implement but not so great in terms of the ability to be able to conveniently index by subject or by activity. We shouldn't have to know the experiment numbers if we just want data per subject, so we'd have to search the list every time, which is inefficient.

7. A better organization is to have a 3-d array to represent our (S,A,E) table. We could use something like this: SAE <- array(1:x, dim=c(30,6,10299), where x is 30610299=1853820. Unfortunately the data in an array has to be of the same type, and we need various types. What we really want is a data frame where each row is an experiment, and the columns tell us the subject, activity, group, and observations. This puts all the data in one place, allows us to conveniently label everything, and allows efficient lookups and searches. The size of our dataframe will then become 10299 by (561 + 3), or 5,808,636. Not too bad, about 5 mB.

8. We'll assume that most people won't need or want to access the very rawest underlying data of individual x, y, and z coordinates for the gyros and accelerometers. Most people will trust the calculated values that are in the Observations columns of our final dataset. However we still need to make the rawest underlying data available to the people who want it or need it, and therefore we can simply index the underlying raw tables by experiment number, and once again we can combine the raw observations from the test and train files into one large data frame in R, with rows being the experiment numbers and columns being the actual x, y, and z data (each of which is a 128-element vector). In this dataset we'll start with the total acceleration, then the body acceleration which is just total acceleration - gravity, and finally the angular velocities from the gyroscope.

   We "could" put all of the raw data into the first table, instead of having two separate tables, but that would slow down queries and it would also give us a lot of unneeded "junk" in the R output if all we want is information about an experiment. We could also split up the first table into (S,A,E) and (E,O), but our assumption will be that most people who access this data will want to see some information about means or variances or other high-level details when they look up an experiment, so it's easiest just to show all the high level information for each experiment, that way our user won't have to issue joins every time they want to look at something.

9. The CodeBook.md file shows the sequence of massaging the original data into the desired form. The approach we take is to read everything into R data frames using read.table(), then label each data frame with consistent names so we can easily manipulate the data by row or by column, then slice and dice the source data to get it into the desired form. We will end up with two tables, cross referenced by experiment number. The first table will contain the data about each experiment and the processed observations for each. The second table will contain the raw data from the accelerometers and gyroscopes for each experiment. Thus the only time the end user will have to join tables is when they want to see how the processed observations were derived from the underlying raw gyro and accelerometer data.

10. All the scripts for performing each step listed in the Code Book are contained in the main directory. They are called 'script' followed by the step number, with .R extensions. For example the script for step 2 is 'step2.R'. Reasoning and further information for each step is contained in the CodeBook.md file.