---
title: "Logistic Regression"
output:
  html_document: default
  html_notebook: default
---

## Plots

### Logistic Function

\[ y = \frac{e^{x}}{1+ e^{x}} \]

```{r}
plot.function(function(x){exp(x)/(1+exp(x))},-10,10,ylab="Logistic")
```

### Logit Function

```{r}
plot.function(function(p){log(p/(1-p))},0,1,ylab="Logit", xlab='p(x)')
```

## Log Likelihood 

Plot log likelihood for estimating the probability of a (possibly biased) coin landing heads. The data is a number of heads `h` and the number of tails `t`. The likelihood function is:
\[ \mathcal{L}(p) = p^h \, (1-p)^t ~.\]
Some example of a likelihood functions:
```{r}
#pdf("/tmp/likelihood1.pdf",7,5)
h <- 1; t <- 1
plot.function(function(p){p^h * (1-p)^t},0,1,ylab = "Likelihood", xlab = "p")
#dev.off()
#pdf("/tmp/likelihood10.pdf",7,5)
h <- 10; t <- 10
plot.function(function(p){p^h * (1-p)^t},0,1,ylab = "Likelihood", xlab = "p")
#dev.off()
#pdf("/tmp/likelihood100.pdf",7,5)
h <- 100; t <- 100
plot.function(function(p){p^h * (1-p)^t},0,1,ylab = "Likelihood", xlab = "p")
#dev.off()
```


The likelihood function $\mathcal{L}$ function is usually difficult to maximize. The function is not *concave* and the derivative is complex. Instead, we can maximize **Log-likelihood** $\log\mathcal{L}$ which, in this case, is concave. The maximum of a log-likelihood  is the same as the maximum of likleihood:
\[ \arg\max_{p} \mathcal{L}(p) = \arg\max_{p} \log\mathcal{L}(p)  \]
This is because $\log$ is a **strictly increasing function** and the transformation using such function does not change the maximum value.

The plot of the log-likelihood function for an observation with the same number of heads and tails is as follows.
```{r}
#pdf("/tmp/loglikelihood.pdf",7,5)
h <- 10; t <- 10
plot.function(function(p){h*log(p) + t*log(1-p)},0,1,ylab = "Loglikelihood", xlab = "p")
#dev.off()
```

With mode heads than tails, the value of $p$ that maximizes the likelihoods becomes biased towards heads.
```{r}
#pdf("/tmp/loglikelihood_biased.pdf",7,5)
h <- 20; t <- 5
plot.function(function(p){h*log(p) + t*log(1-p)},0,1,ylab = "Loglikelihood", xlab = "p")
#dev.off()
```