Part X
Code Basics &
Data Manipulation with R
Literature:
Wickham & Grolemund; R for Data Science; ch. 3, 16
42
Code basics
Very basics:
• The simple calculator: 5 / 200 * 30
• Type pi and press enter
Functions:
• Functions are written as function( <value>)
• Call a build-in function: sin(pi/2) or log(2.718)
Objects:
• Create objects with x <- 1 and watch your global environment window
• Inspect an object by typing its name, e.g. x
• Create a vector with v <- c(1, 5)
• Create a vector with v2 <- c ('r', ‘f’)
• Inspect the vectors
Lectures in Data Mining Winter 2018 44
Code basics
• Create a vector with v2 <- c ('r', 'f’)
• Create a vector with v <- c(1, 5)
• Create a vector v <- seq(1, 5)
• Create a vector v2 <- seq(1, 10, length.out = 5)
• Inspect the vector
• Try (v <- seq(1, 10, length.out = 5)) …. What happens?
• Inspect the third element of vector v v[3]
Lectures in Data Mining Winter 2018 45
Code basics
- useful functions -
Compute a lagged version of a time series, shifting the time base back by
a given number of observations:
lag() and lead()
(x <- seq(1, 10))
lag(x)
Drawbacks:
• Need exactly one variable for sorting or sorted dataset.
• Does not support partitioning.
Lectures in Data Mining Winter 2018 46
Code basics
- remember tibbles -
tibble( seq(1, 10), seq(10, 1) )
tibble(a = seq(1,10), b = a*a, c = b - lag(b))
Cumulative and rolling aggregates: cumsum(), cumprod(), cummin(),
cummax() and cummean() RcppRoll-package for rolling windows.
tibble(a = sample(1:10, 20), b = cumsum(a))
Rankings: see min_rank(), dense_rank(), percent_rank()
Lectures in Data Mining Winter 2018 47
Code basics
- tibbles and cross table -
Columns in tables can be addressed by using <TABLENAME>$<COLUMN>
Try:
tibble(mpg$manufacturer, mpg$drv)
tibble(a = sample(1:10, 20), b = cumsum(a)) $b
What is a cross table?
A cross table is a two-way table consisting of columns and rows. It is also
known as a pivot table or a multi-dimensional table. Its greatest strength
is its ability to structure, summarize and display large amounts of data.
Cross tables can also be used to determine whether there is a relation
between the row variable and the column variable or not.
Lectures in Data Mining Winter 2018 48
Code basics
- cross table -
Table() uses the cross-classifying factors to build a contingency table of
the counts at each combination of factors.
Try:
table(mpg$manufacturer)
table(mpg$manufacturer, mpg$drv)
table(mpg$manufacturer, mpg$drv, mpg$cyl)
Exercise:
table(mpg$manufacturer, front = mpg$drv == 'f’ )
Try out and explain!
Lectures in Data Mining Winter 2018 49
Data manipulation in R
with sqldf (SELECT)
• Install the package and load the library with:
install.packages ("sqldf")
library(sqldf)
• Remember: Install a package once but load the library
each time you restart R.
• Test:
sqldf ( "select manufacturer, cyl, hwy from mpg limit 30")
• Library RODBC allow to connect an Oracle DB to R.
Lectures in Data Mining Winter 2018 50
The dplyr package
• Great for data exploration and transformation
• Intuitive to write and easy to read, especially when using the
“pipelining” syntax (covered below)
• Fast on data frames
dplyr functionality
• Five basic verbs: filter, select, arrange, mutate, summarize (plus
group_by)
• Can work with data stored in databases and data tables
• Joins: inner join, left join, semi-join, anti-join
• Window functions for calculating ranking, offsets, and more
Lectures in Data Mining Winter 2018 51
The dplyr package
- filter -
Similar syntax of all dplyr functions:
• First argument is data set
• Subsequent arguments describe what to do with the data set
• Functions return modified data sets
General syntax for filter():
filter( data = #DATA, <CONDITION>, <MORE_CONDITIONS>)
filter(mpg, cyl == 6)
filter(mpg, cyl ==6, drv == 'r')
Lectures in Data Mining Winter 2018 52
The dplyr package
- filter -
• Filter uses Conditions:
– Equals (==)
– Larger (>) and larger or equal (>=)
– Smaller (<) and smaller or equal (<=)
• NA argument always return NA (not FALSE and not TRUE)
– Try NA == NA
– Try is.na(NA)
• Conditions may contain uses logical operators and brakets
– And: & , Or: |, Not: !
– In : %in% <VECTOR>
• Try using condition & instead of ,
• Replace & by | (or) and look what happens.
Lectures in Data Mining Winter 2018 53
The dplyr package
- arrange -
• Arrange sorts the data set by one or more attributes.
• General syntax for arrange():
arrange( data = #DATA, <ATTRIBUTE>, <MORE_ATTRIBUTES>)
• Use functions desc(<ATTRIBUTE>) and asc(<ATTRIBUTE>) to define
the order.
arrange(mpg, hwy)
arrange(mpg, desc(hwy))
Lectures in Data Mining Winter 2018 54
The dplyr package
- select -
• Select allows to take subsets out of a data set by specifying the subset
by its attributes.
• General syntax of select():
select( data = #DATA, <ATTRIBUTE1>, <ATTRIBUTES2>, …)
• To find columns in tables with a lot of attributes use the functions
starts_with(), contains() or ends_with() to define the subset.
select(mpg, 1, 2, 9)
select(mpg, manufacturer, model, hwy)
select(mpg, "manufacturer", "model", "hwy")
select(mpg, starts_with("m"), hwy)
Lectures in Data Mining Winter 2018 55
The dplyr package
- select -
• Select allows to take subsets out of a data set by specifying the subset
by its attributes.
• General syntax of select():
select( data = #DATA, <ATTRIBUTE_SET> …)
• To find select several subsequent columns at one in a table use the
colon in the select function to define the subset.
select(mpg, 1 : 4)
select(mpg, manufacturer : year)
Lectures in Data Mining Winter 2018 56
The dplyr package
- select -
• Select allows to alias attributes by specifying new attribute names.
• General syntax of select():
select( data = #DATA, <NEW_NAME> = <ATTRIBUTE1>, …)
• To keep the remaining columns in tables use the function
everything().
• To remove a column you can use minus (– <ATTRIBUTE>)
Try the following statements and look what happens:
select(mpg, maker = manufacturer, model, hwy)
select(mpg, maker = manufacturer, everything())
select(mpg, maker = manufacturer, everything(), category = class)
select(mpg, maker = manufacturer, everything(), - cyl)
Lectures in Data Mining Winter 2018 57
The dplyr package
- select and rename-
• Rename also allows to alias attributes by specifying new attribute
names and keeps the remaining columns.
• General syntax of rename():
rename( data = #DATA, <NEW_NAME> = <ATTRIBUTE1>, …)
Try and look what happens:
rename(mpg, maker = manufacturer)
Lectures in Data Mining Winter 2018 58
The dplyr package
- mutate -
• Before we start let’s switch the example and prepare the data:
install.packages("nycflights13")
library(nycflights13)
flights
(flights_sml <- select(flights,
year:day,
ends_with("delay"),
distance, air_time
))
Lectures in Data Mining Winter 2018 59
The dplyr package
- mutate -
• Mutate adds attributes (or columns) that are functions of other
attributes at the end of the dataset.
• General syntax of mutate():
mutate( data = #DATA, <NEW_NAME> = <FUNCTION>, …)
Try:
mutate( flights_sml,
gain = arr_delay – dep_delay,
speed = distance / air_time * 60)
Arithmetic operations: +, -, *, /, ^
Try transmute with the same parameters and look what happens.
Lectures in Data Mining Winter 2018 60
The dplyr package
- mutate -
• Mutate allows to reuse variables:
mutate( flights_sml,
gain = arr_delay – dep_delay,
hours = air_time / 60,
gain_per_hour = gain/hours)
Lectures in Data Mining Winter 2018 61
The dplyr package
- mutate functions -
Modular arithmetic: %/% - integer division
%% - remainder
mutate( flights_sml, air_time,
air_hours = air_time %/% 60,
air_mins = air_time %% 60)
Lectures in Data Mining Winter 2018 62
The dplyr package
- Grouping and summarize -
• Summarize() collapses a data set to a single row.
• Summarize() can be paired with group_by(). Consequently, the
aggregation functions use the attributes of the grouped data set
(here: #DATA)
#NEW_DATA <- group_by( data = #DATA, <ATTRIBUTES>)
summarize( #NEW_DATA,
<NEW_NAME> = <AGGR_FUNCTION>, …)
by_dest <- group_by(flights, dest)
delay <- summarize( by_dest, count = n(),
dist = mean(distance, na.rm = TRUE),
delay = mean(arr_delay, na.rm = T))
Lectures in Data Mining Winter 2018 63
The dplyr package
- Grouping and summarize -
• Let’s visualize:
by_dest <- group_by(flights, dest)
delay <- summarize( by_dest, count = n(),
dist = mean(distance, na.rm = TRUE),
delay = mean(arr_delay, na.rm = T))
delay <- filter( delay, count > 20, dest != "HNL")
ggplot(data = delay, mapping = aes(x = dist, y = delay)) +
geom_point(aes(size = count), alpha = 1/2) +
geom_smooth(se = FALSE)
Lectures in Data Mining Winter 2018 64
The dplyr package
- Piping -
• Instead of creating data set it is possible to send data from one
command to the next by using %>%.
#NEW_DATA <-
<FUNCTION> (#DATA %>%, <ATTRIBUTES>) %>%
<NEXT_FUNCTION> (<ATTRIBUTES>)
Applied to our example:
delay <- group_by( flights, dest) %>%
summarize( count = n(),
dist = mean(distance, na.rm = T),
delay = mean(arr_delay, na.rm = T)) %>%
filter( count > 20, dest != "HNL")
Lectures in Data Mining Winter 2018 65
The dplyr package
- Piping -
• Alternatively, it is possible to put the filter from the summary to the
input data:
delay <- filter(flights, !is.na(arr_delay)) %>%
group_by(dest) %>%
summarize( count = n(),
dist = mean(distance, na.rm = T),
delay = mean(arr_delay, na.rm = T)) %>%
filter( count > 20, dest != "HNL")
• To count only non empty values use
count = sum(!is.na(arr_delay))
Lectures in Data Mining Winter 2018 66
The dplyr package
- Piping -
• To make adding commands less work, start with the data set and pipe
it to the first command by using %>%.
Applied to our example:
delay <- flights %>%
filter(flights, !is.na(arr_delay)) %>%
group_by( dest) %>%
summarize( count = n(),
dist = mean(distance),
delay = mean(arr_delay) %>%
filter( delay, count > 20, dest != "HNL")
Lectures in Data Mining Winter 2018 67
The dplyr package
- More functions -
• Some other functions:
– lag and lead return the predecessor or the successor of the dataset
flights %>%
group_by(month) %>%
summarise(flight_count = n()) %>%
arrange(month) %>%
mutate(change = flight_count - lag(flight_count))
– Function sample_n( #x) returns #x randomly sampled data objects from
the data set.
– Function sample_frac( #frac) returns #frac * number of row data objects
from the data set. Default option is “without replacing”.
flights %>% sample_n(10 )
flights %>% sample_frac( 0.01, replace=TRUE)
Lectures in Data Mining Winter 2018 68
Power Supplier
- Switch Process -
Energy Grid
New Supplier Current
Customer (you) Supplier
Request delivery and
Announce delivery Request
declare consumption deregistration
Answer dereg.
Deregistration Check
Request
Confirm
Confirm registration deregistration
Consumption correction or
confirmation of declaration
Confirm delivery
Lectures in Data Mining Winter 2018 69
Exercise or Homework
What data is available
• SWITCHING_REASON: The reason of the new contract: Move into a new home (MOVE_IN)
or change of supplier (SUPPLIER_SWITCH)
• DECLARATION_CONSUMP_CUST: The energy consumption the customer expects to use
and thus declares to the new supplier.
• DECLARATION_CONSUMP_GRID: The grid has its own estimation of the energy
consumption. If it (significantly) differs from the above value the supplier received it in the
delivery confirmation.
• CORRECTIVE_VALUE: Pre-calculated by your lecturer as “hypothetical” correction of
consumption if the grid is correct. (to make it easier for you)
• MONTH_BILLING: The month when the customer received its first bill from the new
supplier.
• DAYS_INVOICED: Number of days between delivery start and billing.
• CONSUMPTION_INVOICED: Consumption as invoiced in the first bill of the new supplier.
• READ_OUT_CAT: The first year consumption can have different sources. Depending on the
situation it is a value from meter reading or an estimation (see categories).
• BILLING_TYPE : Final if the customer quits or rotational
Lectures in Data Mining Winter 2018 70
Exercise or homework
• Import data sets
HH_SAMPLE_POWER_CONSUMPTION_GRID
_A.csv into R Studio
• Download the zip file from moodle
Topic “Visualisation” “Examples”
and extract to a local folder
• library(readr)
consumption <-
read_delim("C:/PATH_TO_FILE/HH_SAMPLE_POWER_CONSU
MPTION_GRID_A.csv", ";", na = "empty", trim_ws = TRUE)
Lectures in Data Mining Winter 2018 71
Exercise or Homework
Rename the dataset and the columns
• Create a new table or tibble with the name hh_spc:
– month_billing = MONTH_BILLING
– swt_reason = SWITCHING_REASON
– cust_decl_kwh = DECLARATION_CONSUMP_CUST
– grid_decl_kwh = DECLARATION_CONSUMP_CUST + CORRECTIVE_VALUE
– bill = BILLING_TYPE
– inv_kwh_365 = CONSUMPTION_INVOICED * 365 / DAYS_INVOICED
• Use round function to cut decimals
• Limit the selection to:
– inv_kwh_365 < 10000 consumption <-
read_delim("C:/PATH_TO_FILE/H
– cust_decl_kwh < 7500
H_SAMPLE_POWER_CONSUMPT
– cust_decl_kwh > 0 ION_GRID_A.csv", ";", na =
• Order by month_billing "empty", trim_ws = TRUE)
Lectures in Data Mining Winter 2018 72
Exercise or Homework
• Plot a sample of 400 of these point & including smoothing function
– Compare customers’ declared consumption and (invoiced) 365 day consumption
– Separate switching reason by color
– Use se = FALSE for the smoothing
• Create a boxplot with switching reason and 365 day consumption by
using a 10% sample.
(Hint: If it does not work change x and y attributes)
Lectures in Data Mining Winter 2018 73
