---
title: "Fehlende Werte - Missing Values"
output: html_document
---



### [Rmd](miss.Rmd)


Missing Data werden in R durch NA (not available) repräsentiert und haben eine eigene Funktionalität.
Normalerweise wird das Ergebnis einer Operation, in der NA vorkommen ebenfalls auf NA gesetzt.

Missing values are specified with NA, which is a logical vector of length 1. 
NA will always be coerced to the correct type if used inside c(), 
or you can create NAs of a specific type with NA_real_ (a double vector), NA_integer_ and NA_character_.

Viele Funktionen und Verfahren haben ein Flag (Aufrufparamter) für den Umgang mit NA (na.rm), das häufig als Voreinstellung auf TRUE gesetzt ist, nicht auf FALSE.
Dabei werden die Beobachtungen mit NA aus der jeweiligen Berechnung ausgeschlossen.
Dies entspricht dem Standard-Vorgehen vieler Statistikprogramme, führt aber u. U. zu unterschiedlichen Stichproben bei den Berechnungen.

Für logische Operationen gibt es den Befehl `is.na` bzw. `!is.na` (not is NA, logische Prüfung auf nicht-fehlend)


```{r}
# get a working base idea
ddf <- data.frame(x = c(1, 2, 3), y = c(0, 10, NA))
is.na(ddf)
!is.na(ddf)
```


Häufig ist die Anzahl von fehlenden Werten relevant.
Hier kann das Auszählen von `TRUE`-Werten beim Prüfen auf fehlende Werte helfen.
Alternativ können wir `which()` einsetzen

```{r}
# create sample dataframe
ddf <-data.frame(
  subj   = c(  1,   2,   3,   4,   5,   6,   7,   8,   9),
  uni    = c(  1,   1,   1,   2,   2,   2,   3,   3,   3),
  grade1 = c(1.0,  NA, 3.7, 1.3,  NA, 1.0, 3.3, 4.0,  NA),
  grade2 = c(4.0, 3.0, 1.3, 1.3, 1.0, 1.3, 2.7, 4.0, 3.3),
  grade3 = c(1.3,  NA, 2.7, 1.0, 1.3, 1.3, 2.3, 3.7, 3.0)
)
# we get a T/F vector
is.na(ddf)
length(is.na(ddf))
length(is.na(ddf)[is.na(ddf) == T])
# or using which()
which(is.na(ddf))
# and count it
length(which(is.na(ddf)))
# how about missings in a column
length(which(is.na(ddf$grade1)))
# or in a single row
length(which(is.na(ddf[2,])))
```


Welche Beobachtungen haben missing values in einem Data-Frame?

```{r}
# take care: variables in common have to be equal in both dataframes for all subjects
# example: gender of subj 2 is different in the two dataframes
# data of v1: three subjects, 3 vars
ddf.v1 <- data.frame(
  subj   = c(  1,   2,   3),
  gender = c('w', 'm', 'w'),
  age    = c( 20,  22,  27)
  )
# data of v2, more subjects, two more vars, subj 2 has a typo in variable gender
ddf.v2 <- data.frame(
  subj   = c(  1,   2,   3,   4,   5),
  weight = c( 67,  85,  78,  66,  72),
  gender = c('w', 'x', 'w', 'w', 'm'),
  height = c(172, 185, 180, 165, 177)
  )
  # take care: gender of ddf.v2 replaces gender of ddf.v1
# variables, the dataframes have in common, have to appear in parameter `by`
# there still are missing data, so the problem of what to do with incomplete data persists
# solution 1: get only complete subjects after merge
merge(ddf.v1, ddf.v2, by=c("subj", "gender"))
# solution 2: impute NA where data are missing by using flag `all`
merge(ddf.v1, ddf.v2, by=c("subj", "gender"), all=T)
ddf <- merge(ddf.v1, ddf.v2, by=c("subj", "gender"), all=T)

# where are missings in ddf? how many?
apply(ddf, 1, function(x) length(which(is.na(x))))

```


Alle Beobachtungen mit missing values aus Data-Frame ausschließen mit `cleaned <- na.omit(Data-Frame)`.

```{r}
# get a working base idea
ddf <- data.frame(x = c(1, 2, 3), y = c(0, 10, NA))
ddf
# na.omit() deletes!
na.omit(ddf)


# search for the first missing value
ddf = read.csv("http://www.bodowinter.com/tutorial/politeness_data.csv")
# first occurance of a missing in dataframe
which(is.na(ddf))
# number issued comes from iterating over subjects column after column
which(is.na(ddf)) %% length(ddf[,1])
# exclude all observations with missing data
ddf.cleaned <- na.omit(ddf)     #remove the cases with missing values
# n of excluded:
cat(nrow(ddf) - nrow(ddf.cleaned), ' observations excluded')


```




Alle Werte eines Dataframes, die negativ sind, auf NA setzen.

```{r}
ddf <- data.frame(x = c(1, 2, 3), y = c(0, 10, NA))

ddf <- data.frame(lapply(ddf, function(x) { x[x < 0] <- NA; x }))
```



Welche Beobachtungen haben missing values in einem Data-Frame?

```{r}
ddf = read.csv("http://www.bodowinter.com/tutorial/politeness_data.csv")
## todo
```


Ein paar Beispiele zur Verwendung von Hmisc::impute() 


```{r}
library(Hmisc)
DF <- data.frame(age = c(10, 20, NA, 40), sex = c('male','female'))

# impute with mean value

DF$imputed_age <- with(DF, impute(age, mean))

# impute with random value
DF$imputed_age2 <- with(DF, impute(age, 'random'))

# impute with the media
with(DF, impute(age, median))
# impute with the minimum
with(DF, impute(age, min))

# impute with the maximum
with(DF, impute(age, max))
```



`aregImpute()`

```{r}
require(Hmisc)
# read data
ddf <- read.delim("http://md.psych.bio.uni-goettingen.de/data/virt/v_bmi.txt")
# attach ddf for more comfort
attach(ddf)
# copy column c_phys_app
n_c_phys_app <- c_phys_app
# choose values to set to NA
to_erase <- sample(length(c_phys_app), 5)
# show observations to erase
to_erase
# set selected values to missing
n_c_phys_app[to_erase] <- NA
# show new var with erased data
n_c_phys_app
# compare with original
cbind(c_phys_app, n_c_phys_app)


ddf$n_c_phys_app
imputed_n_c_phys_app <- aregImpute(~ gender + height + weight + grade + c_phys_app + c_good_way + c_dress + c_bad_way + c_figure + filling_time, data=ddf, n.impute=10)
imputed_n_c_phys_app
ddf$n_c_phys_app
```


Aus Baron & Lee
[http://www.psych.upenn.edu/~baron/rpsych/rpsych.html]

7.15  Imputation of missing data

Schafer and Graham (2002) provide a good review of methods for dealing with missing data. 
R provides many of the methods that they discuss. 
One method is multiple imputation, which is found in the Hmisc package. 
Each missing datum is inferred repeated from different samples of other variables, and the repeated inferences are used to determine the error. 
It turns out that this method works best with the ols() function from the Design package rather than with (the otherwise equivalent) lm() function. 
Here is an example, using the data set t1.

```{r}
# todo:
#library(Hmisc)
#f <- aregImpute(~v1+v2+v3+v4, n.impute=20,
#     fweighted=.2, tlinear=T, data=t1)
#library(Design)
#fmp <- fit.mult.impute(v1~v2+v3, ols, f, data=t1)
#summary(fmp)
```

The first command (f) imputes missing values in all four variables, using the other three for each variable. 
The second command (fmp) estimates a regression model in which v1 is predicted from two of the remaining variables. 
A variable can be used (and should be used, if it is useful) for imputation even when it is not used in the model.





## Beispiele Übungen

Übungs-Datensatz

```{r}


#ddf <- read.delim("http://md.psych.bio.uni-goettingen.de/data/virt/v_bmi_miss.txt")



```




## Links und Referenzen


[http://www.ats.ucla.edu/stat/r/faq/missing.htm](http://www.ats.ucla.edu/stat/r/faq/missing.htm)


<!-- in text 
[Field (2012, p. 245)][field] **Kapitel 7: Regression**

Die Daten und das Beispiel entstammen dem Buch [Everitt (2010)][everitt]
-->

<!-- this below does not appear in html-page. It's used only to arm links and references. -->
[field]: http://www.uk.sagepub.com/dsur/main.htm "Field et al (2012). Discovering Statistics using R"

[everitt]: http://www.crcpress.com/product/isbn/9781439807699 "Brian S. Everitt (2010). Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences"

