Overview
We recall properties of random variable related to the expectation time, Poisson processes, and order statistics. They will be used as soon as we turn to estimates. The end of this part covers the influential Kolmogorov-Smirnov statistics.
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$
Let $T$ be the moment of the first success in a sequence of Bernoulli trials with the probability of success $p_{\delta}$, where the trials occur at time moments $\delta$, $2\delta$, $\ldots$. Then $\Prob{T > n\delta} = (1-p_{\delta})^n$. The expected number of trials before the first success is $1/p_{\delta}$ (why?). Therefore, the expected waiting time is $\delta/p_{\delta}$. The continuous case is obtained through the tendency of $\delta$ to zero. A reasonable assumption regarding other parameters is that the expected waiting time is conserved: $\delta/p_{\delta} \to a$. Then
$$ \Prob{T > n\delta} = e^{n\ln(1-p_{\delta})} \sim e^{-np_{\delta}} \to e^{\frac{-n\delta}{a}}. $$Eventually,
$$ \Prob{T > t} = e^{-t/a}. $$In other words, the random variable $T$ has the exponential distribution with the mean $a$.
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$
Let $U(t) = \Prob{T > t}$ be the complement distribution function. It may be interpreted as the probability at birth that the lifetime exceeds $t$. The conditional probability that the residual lifetime exceeds $t$, given that the lifetime is at least $s$, $\Prob{T > s+t | T > s} = U(s+t) / U(s)$. It coincides with the unconditional probability to live at least $t$ time units, if and only if
$$ U(s+t) = U(s)U(t). \quad\quad \textrm{(MARKOV)} $$Equation (MARKOV) holds for the exponential distribution.
Prove the inverse statement: if equation (MARKOV) holds then the distribution is exponential.
The lack of memory is known as the \emph{Markov} property of the distribution.
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$
Let $X$ and $Y$ be independent random variables. Their cumulative distribution functions are $F$ and $G$ and the probability densities are $f$ and $g$ respectively. Then we define $Z = X+Y$. Let $H$ be the cumulative distribution function function of $Z$. Then
\begin{equation*} H(s) = \Prob { X+Y < s } = \sum_y \Prob{Y\approx y \text{ and } X < s-y} = \int_{-\infty}^{\infty} F(s-y) g(y) dy. \end{equation*}Differentiating,
$$ h(s) = \int_{-\infty}^{\infty} f(s-y) g(y) dy. $$The probability density function $h(s)$ is obtained with the convolution of $f$ and $g$: $h = f*g$.
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$
Let $X_1$, $\ldots$, $X_n$ be independent random variables with the common exponential distribution. Put, $S_0 = 0$, \begin{equation*} S_n = X_1 + \ldots + X_n, \quad n = 1, 2, \ldots \end{equation*} We introduce a family of new random variables $N(t)$ as follows: $N(t)$ is the number of indices $k \ge 1$ such that $S_k \le t$. The event $\{N(t) = n\}$ occurs if and only if $S_n \le t$ and $S_{n+1} > t$. Therefore,
$$ \Prob{N(t) = n} = G_{n}(t) - G_{n+1}(t) = \frac{(at)^n}{n!}e^{-at}. $$(To prove, equate two events: $\{S_n \le t < S_{n+1} \,\,$ OR $\,\, S_{n+1} \le t\} = \{S_n \le t\}$). In words, the random variable $N(t)$ has a Poisson distribution with the expectation $at$. To repeat, $N(t)$ is the number of events' occurrence over the interval $[0, t]$, where the waiting time of events is given by the exponential distribution.
$\newcommand{\Prob}[1]{\mathsf{P}\{#1\}}$ $\newcommand{\Exp}{\mathbf{E}}$
Let $X_0$ be my waiting time (or financial loss) of some random event. The results of friends are $X_1$, $X_2$, $\ldots$, where $X_0$, $X_1$, $X_2$, $\ldots$, are random variables with a common distribution.
Measure of my bad luck is the time required to my friends to get the result that is worse than mine. Formally, let $N$ be the value of the first subscript $n$ such that $X_n > X_0$. Then the event $\{N > n-1\}$ means that $X_0$ is the maximal term among $X_0$, $X_1$, $X_2$, $\ldots$, $X_{n-1}$. For reason of symmetry, the probability of this event is $n^{-1}$: $\Prob{N > n-1} = 1/n$. As $\{N = n\} = \{N > n-1\} \setminus \{N > n\}$,
$$ \Prob{N = n} = \frac{1}{n} - \frac{1}{n+1} = \frac{1}{n(n+1)}, \quad \Exp N = +\infty. $$Let $X$ and $Y$ be two independent random variables with a common exponential distribution; $Z = Y/X$. We denote $R$ and $r$ the cdf and pdf of $Z$.
$$ R(t) = \int_0^{+\infty} \Prob{Y \le tx} ae^{-ax} dx = \int_0^{+\infty} \big(1-e^{-atx}\big) ae^{-ax} dx = \frac{t}{1+t}. $$$$ r(t) = \frac{1}{(1+t)^2}. $$Mind that $\Exp Z = +\infty$.
Task:
Create samples with the values taken from the observation of the random
variable $Z$. Find the mean of each sample. Sketch a graph that displays
the dependence of the mean on the volume of the sample. Do you observe
some regularities in the graph?
Notation: ordered tuple $(x_1, x_2, \ldots, x_n)$ is written as
$$ (x_{(1)}, x_{(2)}, \ldots, x_{(n)}), \quad\text{where}\quad x_{(1)} \le x_{(2)} \le \ldots \le x_{(n)}. $$Further,
$$ (X_{(1)}, X_{(2)}, \ldots, X_{(n)}), $$is the ordered values of the random variables $(X_1, X_2, \ldots, X_n)$. Let $X_1$, $\ldots$, $X_n$ be independent random variables with the common exponential distribution. Then the event $\{X_{(1)} > t\}$ means that all values are larger than $t$:
$$ \Prob{X_{(1)} > t} = (1-F(t))^n = e^{-nat}. $$At the epoch $X_{(1)}$, the waiting time for the next event is $X_{(2)}-X_{(1)}$ which is the time for the first event among {\color{red} $n-1$ events}. Because of the lack of memory this waiting time follows the same distribution but with $n-1$ random variables:
$$ \Prob{X_{(2)} - X_{(1)} > t} = (1-F(t))^{n-1} = e^{-(n-1)at}. $$Let $X_1$, $\ldots$, $X_n$ be independent random variables that are uniformly distributed on $(0, 1)$.
Verify that $\Exp(X_{(1)}) = 1/(n+1)$. This defines $1/(n+1)$ as a unit of measure. Let $n$ tend to $\infty$. Then
$$ \Prob{n X_{(1)} > t} = \left( 1-\frac{t}{n} \right)^n \to e^{-t}. $$In words, $X_{(1)}$ is exponentially distributed with expectation $n^{-1}$. Further exploiting the equation of the probability density of $X_{(k)}$, we get
$$ \Prob{n X_{(2)} > t} = \left( 1-\frac{t}{n} \right)^n +\binom{n}{1}\frac{t}{n} \left( 1-\frac{t}{n} \right)^{n-1} \to e^{-t} + te^{-t}. $$As $n\to\infty$, the distribution of $nX_{(k)}$ tends to the gamma distribution $G_k$. Then if $X_1$, $\ldots$, $X_n$ determines the partition of $[0,1]$ onto $n+1$ intervals, then $X_{(k)}$ is the sum of the length of the first $k$ intervals. Our computation shows that the length of the successive intervals of the partition behaves in the limit as if they were mutually independent exponentially distributed variables.
The definition yields that $F_n(x; a_1,\ldots,a_n)$ is the fraction of points $a_1$, $\ldots$, $a_n$ that are less than $x$.
We will assume today that $X_1$, $X_2$, $\ldots$, $X_n$ are mutually independent random variables with a common continuous distribution $F$. For fixed $x$ the number of variables $X_k$ with $X_k < x$ has a binomial distribution with the probability of success $p = F(x)$. Therefore, the random variable $\mathbf{F}_n(x)$ has a binomial distribution with possible values $0$, $1/n$, $\ldots$, $1$.
More interesting, is the deviation of the whole graph of $\mathbf{F}_n(x)$, which depends on the realization of random variables, from the graph of $F(x)$. Put,
$$ D_n = \sup_{x} |\mathbf{F}_n(x) - F(x)|. $$Let $X_1$, $\ldots$, $X_n$, $X'_1$, $\ldots$, $X'_n$, be $2n$ mutually independent random variables with the common continuous distribution $F$, and denote the empirical distributions $(X_1, \ldots, X_n)$ and $(X'_1, \ldots, X'_n)$ by $\mathbf{F}_n$ and $\mathbf{F}'_n$ respectively.
Put,
Summary
In this manual, we discussed