We play a game by rolling a die twice. Let @@@X@@@ be the value of the smaller number and let @@@Y@@@ be the value of the larger number. Let’s calculate the probability that the RV @@@(X,Y)@@@ is equal to @@@(i,j)@@@. This event is of course @@@\{(X,Y)=(i,j)\}@@@, but it’s also the intersection @@@\{X=i\}\cap \{Y=j\}@@@. Both mean the same. Let’s calculate: clearly, @@@P((X,Y)=(i,j))@@@ can be only positive if @@@i\le j@@@ and both @@@i,j \in \{1,\dots,6\}@@@. For such pairs of @@@(i,j)@@@ we need to consider two cases.

1. @@@i<j@@@ (e.g. @@@X=3@@@ and @@@Y=4@@@). This can be obtained by either rolling @@@i@@@ followed by @@@j@@@, or by rolling @@@j@@@ followed by @@@i@@@. Therefore the probability is @@@2*\frac{1}{36}=\frac{1}{18}@@@.
2. @@@i=j@@@ (e.g. @@@X=4@@@ and @@@Y=4@@@). This can be only obtained by rolling @@@i@@@, then again @@@i@@@, so the probability is @@@\frac{1}{36}@@@.

Summarizing,

$$ P((X,Y)=(i,j)) = \begin{cases} \frac{1}{18} & 1\le i < j \le 6 \\ \frac{1}{36} & i=j \in \{1,\dots,6\}\\ 0 & \mbox{otherwise.}\end{cases} $$

Let’s continue developing this example. We know what is the probability the vector attains any value, but what is the distribution of, say @@@X@@@? Clearly, the support of @@@X@@@ is @@@\{1,2,3,4,5,6\}@@@, so @@@X@@@ is a discrete RV, and to find its distribution, it is enough to find its PMF. Since we can split the event @@@\{X=i\}@@@ according to the values of @@@j@@@, the law of total probability gives us

$$P(X=i) = \sum_{j}P(X=i \mbox{ and }Y=j)=\sum_{j}P((X,Y)=(i,j)).$$

Recall that the probabilities in the sum on the right are only positive if @@@j\ge i@@@. The one corresponding to @@@j=i@@@ is @@@\frac{1}{36}@@@, while the remaining ones, if any, are equal to @@@\frac{1}{18}@@@. This gives

$$ P(X= i) = \begin{cases} \frac{1}{36}+(6-i) \frac{1}{18} & i\in \{1,\dots,6\} \\ 0 & \mbox{otherwise.} \end{cases} $$

So @@@X@@@ is the most likely to be @@@1@@@ and least likely to be @@@6@@@. The whole procedure makes sense, right? We will make this into a method very soon.

Let @@@(X,Y)@@@ be a random vector and suppose @@@F_{X,Y} (x,y) = xy^2@@@ for @@@0\le x,y \le 1@@@.

1. Extend @@@F_{X,Y}@@@ to the entire plane.
2. What is the CDF of @@@X@@@ and of @@@Y@@@?

3. What is the probability of @@@\frac 13< X \le \frac 12, Y > 1/3@@@? Express your answer in terms of the CDF.

4. We only define @@@F_{X,Y}@@@ on @@@[0,1]^2@@@. But @@@F_{X,Y}(1,1) = 1@@@, that is @@@P(X\le 1,Y\le 1)=1@@@ and since this the probability of the intersection of two events and therefore less than or equal to the probability of each one, and probabilities are never larger than @@@1@@@, so we have that @@@P(X\le 1) = P(Y\le 1) =1@@@. Because of this, we conclude that @@@F_{X,Y} (x,1) = P(X\le x, Y\le 1) = P(X\le x)@@@ (probability of event with another that has probability @@@1@@@ is always equal to the probability of the former), and similarly @@@F_{X,Y} (1,y) = P(Y\le y)@@@. The last two observations identify the CDFs of @@@X@@@ and @@@Y@@@, but they also allow us to conclude that @@@P(X\le 0) =P(Y\le 0)=0@@@. So now we know that the two RVs @@@X@@@ and @@@Y@@@ take values in @@@[0,1]@@@. We will use this to expand @@@F_{X,Y}@@@ to the plane. Clearly it has to be zero on the second, third and fourth quadrant because the evaluation of@@@F_{X,Y}@@@ at every point in each of these corresponds calculation of an event with zero probability (@@@X@@@ or @@@Y@@@ being negative). As for the first quadrant, all we need to know is that @@@P(X\le 1)=1@@@ and @@@P(Y\le 1)=1@@@, so the probability of @@@\{X\le x,Y\le y \}@@@ is the same as @@@\{X\le \min(x,1),Y \le \min(y,1)\}@@@, and this allows to conclude that

$$F_{X,Y}(x,y) P(X\le x ,Y\le y) = \begin{cases} \min (x,1)\min(y,1)^2 & 0 \le x,y\\ 0 & \mbox{otherwise}\end{cases}$$

The first line gives the first quadrant and the second gives all remaining ones.

1. If we let @@@y\to\infty@@@ in the above formula (or use the previously derived observation @@@P(X\le x) = F_{X,Y}(x,1)@@@), we conclude that @@@X\sim \mbox{U}[0,1]@@@, and repeating for @@@Y@@@ we have that the CDF of @@@Y@@@ is zero for negative numbers, @@@y^2@@@ for @@@y \in [0,1)@@@ and equal to @@@1@@@ for @@@y\ge 1@@@.
2. We show how to manipulate the expression to express it as differences/sums of the CDFs. We will be able to do it much faster after we learn about independence of RVs. This is

$$\begin{align*} P( \frac 13 <X\le \frac 12 , Y> \frac 13)& = P( \frac 13 < X \le \frac 12 ) - P( \frac 13 < X \le \frac 12, Y \le \frac 13) \\ & = F_X(\frac 12,1) - F_X(\frac 13,1) -( P( X\le \frac 12, Y \le \frac 13) - P( X \le \frac 13, Y \le \frac 13)) \\ & = F_X(\frac 12,1) - F_X(\frac 13,1) - F_{X,Y} (\frac 12,\frac 13) +F_{X,Y}(\frac 13,\frac 13). \end{align*}$$

• Let @@@(X,Y)@@@ be a random vector with independent marginals, @@@X\sim \mbox{Exp}(\lambda)@@@ and @@@Y\sim \mbox{Exp}(\mu)@@@. Show that @@@\min(X,Y)\sim \mbox{Exp}(\mu+\lambda)@@@.

Let @@@Z=\min(X,Y)@@@. Then for @@@z>0@@@:

$$\begin{align*} F_Z(z) &= P(Z\le z)\\ & =P( \min(X,Y) \le z) \\ & = 1-P(\min(X,Y) >z) \\ &= 1- P(X>z,Y>z)=1-P(X>z)P(Y>z) \\ & = 1-e^{-\lambda z} e^{-\mu z}\\ & =1-e^{-(\lambda+ \mu)z} \end{align*}$$

Also since @@@X,Y \ge 0@@@, it follows that @@@F_Z(z)=0@@@ for @@@z\le 0@@@. Therefore @@@Z@@@ has CDF of @@@\mbox{Exp}(\lambda+ \mu)@@@, that is @@@Z \sim \mbox{Exp}(\lambda + \mu)@@@.

• What if @@@X@@@ and @@@Y@@@ where not independent? Let’s consider the most extreme case where one is a function of the other. As before take @@@X\sim \mbox{Exp}(\lambda)@@@, and set @@@Y=\lambda/\mu X@@@. Observe that

$$P(Y>y) = P( X > \mu/\lambda y) = e^{-(\mu/\lambda) \lambda y} = e^{-\mu y}. $$

Therefore @@@Y\sim \mbox{Exp}(\mu)@@@. Clearly, @@@Y\le X@@@ if @@@\lambda \le \mu@@@ and otherwise @@@X<Y@@@. In other words, @@@\min (X,Y)@@@ is @@@Y@@@ when @@@\lambda \le \mu@@@ and @@@X@@@ when @@@\mu <\lambda@@@, and therefore the minimum is not @@@\mbox{Exp}(\lambda+ \mu)@@@-distributed.

The return rate for my portfolio for each of the four quarters of the year, expressed in percentage points, is a random variable @@@X_i,~i=1,\dots,4@@@. Assuming that the return rates are independent, find the expected annual return rate.

Well, after the first quarter I have @@@1+X_1/100@@@ of what I Started with, and after the second quarter I have @@@(1+X_1/100)(1+X_2/100)@@@ of what I started with, etc. Therefore after @@@4@@@ quarters I have @@@\prod_{i=1}^4 (1+X_i/100)@@@ of what I started with, so that my return rate is @@@R= 100(\prod_{i=1}^4 (1+X_i/100) -1) @@@ (multiplication by @@@100@@@ to express in percentage points). The expected return rate is therefore

$$\begin{align*} E[R ]&= E [100 \prod_{i=1}^4 (1+X_i/100) -1 ]\\ & = 100 E[ \prod_{i=1}^4 (1+X_i/100) -1 ] \\ & = 100 [E[ \prod_{i=1}^4 (1+X_i/100) ]-100 \\ & = 100 \left (\prod_{i=1}^4 E [1+X_i/100] - 1\right)\\ & = 100 \left ( \prod_{i=1}^4 (1+ E[X_i]/100)-1\right) \end{align*}$$

(the “@@@-1@@@” is outside the product)

The costs for the producers of a rock festival is approximately @@@\$ n e^{-n/100}@@@, where @@@n@@@ is the number of tickets sold. Suppose that the number of tickets sold is @@@N\sim \mbox{Bin}(1000,0.3)@@@. Find the expected cost per patron.

The cost per patron is @@@e^{-N/100}@@@. But since @@@N\sim \mbox{Bin}(1000,0.3)@@@, it is a sum of @@@1000@@@ independent @@@\mbox{Bern}(0.3)@@@, @@@X_1,\dots,X_{1000}@@@. Thus,

$$\begin{align*} E[e^{-N}]&=E[e^{-\sum_{j=1}^{1000} X_j/100}]= E[\prod_{j=1}^{1000} e^{-X_j/100}]\\ & =\prod_{j=1}^{1000} E[e^{-X_j/100}]\\ & = \prod_{j=1}^{1000} (e^{-1/100} 0.3+ 1* 0.7)=\$ 0.05. \end{align*}$$

Remember the negative binomial, @@@X_r\sim \mbox{NB}(r,p)@@@? We calculated its expectation earlier. Let’s redo it.

comment %} Example 8. endcomment %}

Recall that if @@@X_r@@@ is the number of trials until the @@@r@@@-th success in a sequence of independent trials with probability @@@1-p@@@ of success in each, then @@@X=X_r-r@@@, that is @@@X@@@ is the number of trials until the @@@r@@@-th success. Since the variance does not change when adding a constant, @@@\sigma^2_X = \sigma^2_{X_r}@@@. Also, observe that @@@X_r@@@ is a sum of @@@r@@@ independent @@@\mbox{Geom}(1-p)@@@. Indeed, the number of trials until the first success is @@@\mbox{Geom}(1-p)@@@, and the number of additional trials until the next success is also @@@\mbox{Geom}(1-p)@@@, independent of the former. More generally, the number of trials between successive successes are all independent @@@\mbox{Geom}(1-p)@@@, as can be shown by simple induction. The variance of @@@\mbox{Geom}(1-p)@@@ is @@@\frac{p}{(1-p )^2}@@@, as we previously calculated and therefore,

$$\sigma^2_X =\sigma^2_{X_r} =\frac{rp}{(1-p)^2}. $$

Toss a fair dice repeatedly until you see an even number. Let @@@X@@@ denote the number of tosses, and let @@@Y@@@ denote the even number observed. What is the PMF of the random vector @@@(X,Y)@@@?

The random vector @@@(X,Y)@@@ can take any value of the form @@@(n,e)@@@, where @@@n=1,2,\dots@@@ and @@@e=2,4,6@@@. The event @@@(X,Y)=(n,e)@@@ corresponds to all first @@@n-1@@@ tosses being odd, and @@@n@@@-th toss being @@@e@@@. Since tosses are independent, the probability is then equal to

$$p_{X,Y}(n,e) = (\frac{3}{6})^{n-1} \frac{1}{6} = \frac{1}{2^{n}}\frac{1}{3}.$$

Summarizing:

$$\begin{equation} \label{eq:until_even} p_{X,Y} (n,e) = \begin{cases} \frac{1}{3 \cdot 2^n} & n =1,2,\dots,~e=2,4,6 \\ 0 & \mbox{otherwise}\end{cases} \end{equation}$$

Let’s continue Example 7, where we roll a die repeatedly until we see an even number. The number of rolls is @@@X@@@ and the number appearing on the last roll is @@@Y@@@. What are the PMFs of @@@X@@@ and @@@Y@@@?

• Let’s start with @@@X@@@. If @@@n=1,2,\dots@@@, then

$$\begin{equation} \label{eq:wait_marginal} p_X(n) = \sum_e p_{X,Y}(n,e) \overset{\eqref{eq:until_even}}{=} 3 \times \frac{1}{3 \cdot 2^n}= \frac{1}{2^n}, \end{equation}$$

and for all other @@@n@@@, @@@p_X(n)=0@@@. This is the PMF of @@@\mbox{Geom}(\frac 12)@@@. Therefore @@@X\sim \mbox{Geom}(1/2)@@@.

• What about @@@Y@@@? For @@@e=2,4,6@@@, we have

$$\begin{equation} \label{eq:even_marginal} p_Y(e) = \sum_{n=1}^\infty p_{X,Y} (n,e) \overset{\eqref{eq:until_even}}{=} \sum_{n=1}^\infty \frac{1}{3 \cdot 2^n} = \frac{1}{3}, \end{equation}$$

while for all other @@@e@@@, @@@p_Y(e)=0@@@. Therefore @@@Y@@@ is uniform on @@@\{2,4,6\}@@@.

We have one fair @@@6@@@-face die and toss it twice. Let @@@X@@@ denote the number in the first toss and @@@Y@@@ the number in the second toss. Then @@@X@@@ and @@@Y@@@ are independent uniform on @@@\{1,\dots,6\}@@@.

As usual, without any assumptions, we take @@@P@@@ as the uniform distribution on the @@@36@@@ possible outcomes. Indeed, choose @@@i,j\in \{1,\dots 6\}@@@….

Let’s repeat the same experiment as in the last example, and let @@@S=X+Y@@@. Then @@@S@@@ and @@@Y@@@ are not independent. Why? @@@P(X=6)=\frac 16@@@, but @@@P(X=6|S=12) \ne P(X=6)@@@: conditioning the event @@@\{X=6\}@@@ on the event @@@\{S=12\}@@@ changes the probability of the former, hence the two events are not independent.

• All we need is one case to ruin independence.
• Note, for example that @@@P(S=7) = \frac{6}{36}@@@, and therefore

$$P(X=i,S=7) = P(X=i,Y=7-i) = \frac{1}{36} = P(X=i) P(S=7).$$

That is for every @@@i@@@, @@@\{X=i\}@@@ and @@@\{S=7\}@@@ are independent, although the random variables @@@X@@@ and @@@S@@@ are not independent.

Are the RVs @@@X@@@ and @@@Y@@@ from Example 7 independent? We only need to check for @@@n=1,2,\dots@@@ and @@@e=2,4,6@@@, the respective supports. This can be seen from our formula for the PMF \eqref{eq:until_even}, or from the formula for the marginals we obtained in Example 8, which we will now use to prove the independence. We have

$$p_{X,Y}(n,e)\overset{\eqref{eq:until_even}}{=} \frac{1}{3\cdot 2^n} \overset{\eqref{eq:wait_marginal},\eqref{eq:even_marginal}}{=} p_Y(e) p_X(n) ,$$

and therefore @@@X@@@ and @@@Y@@@ are independent. Indeed, waiting until an even number lands does not tell anything about which number it would be.

Toss a fair dice repeatedly. Let @@@X@@@ be number of tosses until the first “@@@1@@@” and let @@@Y@@@ denote the number of tosses until the first “@@@2@@@”. Find the PMF of @@@(X,Y)@@@, the distribution of each marginal and whether they are independent. To begin, it is extremely easy to identify the distributions of the marginals and we don’t need to compute the joint PMF first. The tosses are independent. In each toss, the probability it lands @@@1@@@ is @@@1/6@@@. Therefore @@@X@@@, the number of tosses until first @@@1@@@ is @@@\mbox{Geom}(1/6)@@@. Same with @@@Y@@@. That is,

$$p_Y(y) = p_X(y) = (\frac{5}{6})^{y-1} \frac{1}{6} = \frac{5^{y-1}}{6^y}.$$

To find the joint PMF, observe that the vector @@@(X,Y)@@@ only takes values of the form @@@(x,y)@@@ where @@@x\ne y@@@ and @@@x,y =1,2,\dots@@@. Let’s fix such pair @@@x,y@@@ and assume that @@@x<y@@@. The event @@@\{(X,Y)=(x,y)\}@@@ means that the first @@@x-1@@@ tosses are in @@@\{3,\dots,6\}@@@, the @@@x@@@-th toss is @@@1@@@, the tosses after the @@@x@@@-th and before the @@@y@@@-th are not @@@2@@@ (total of @@@y-x-1@@@ tosses), and the @@@y@@@-th toss is a @@@2@@@. By independence of the tosses, we have \begin{align} p_{X,Y} (x,y) & = (\frac{4}{6})^{x-1} \frac {1}{6} (\frac{5}{6})^{y-x-1} \frac{1}{6}\ & = \frac{4^{x-1}5^{y-x-1}}{6^y}=\frac {1}{25}(\frac{5}{6})^y(\frac{4}{5})^{x-1}. \end{align} Repeating the computation with in the case @@@y<x@@@ gives the same formula, with the roles of @@@x@@@ and @@@y@@@ changed in the RHS. For all other cases, including @@@x=y@@@, @@@p_{X,Y}(x,y)=0@@@. Are @@@X@@@ and @@@Y@@@ independent? Ask yourself the following: does knowing something on @@@X@@@ can change the distribution of @@@Y@@@? Yes. If @@@X=1@@@ then @@@Y@@@ cannot be @@@1@@@, although @@@p_Y(1)= \frac{1}{6}@@@. That is,

$$0= p_{X,Y}(1,1) \ne p_X(1) p_Y(1) = \frac {1}{36}.$$

Therefore \eqref{eq:disc_vec_ind} fails for one pair, and therefore the RVs @@@X@@@ and @@@Y@@@ are not independent.

Suppose that @@@X\sim \mbox{Geom}(p_1)@@@ and @@@Y\sim Geom(p_2)@@@ are independent.

1. For @@@k\in\Z_+@@@ find @@@P(X=Y+k)@@@.
2. Find @@@P(X>Y)@@@. Observe that

$$\begin{align*} P(X=Y+k) & = \sum_{y} P(X=Y+k,Y=y)\\ & = \sum_{y} P(X=y+k,Y=y) \\ & = \sum_{y} p_X (y+k)p_Y(Y=y)\\ & = \sum_{y=1}^\infty (1-p_1)^{y+k-1}p_1 (1-p_2)^{y-1}p_2\\ & = p_1 p_2 (1-p_1)^k \sum_{y=1}^\infty ((1-p_1)(1-p_2))^{y-1}\\ & = \frac{p_1p_2 (1-p_1)^k}{1-(1-p_1)(1-p_2)}. \end{align*}$$

We have used the formula for the geometric series to get the last line. We turn to b. We can do it by observing that

$$P(X>Y) = \sum_{k=1}^\infty P(X=Y+k),$$

and using the result from part a., or skipping part a, and proceed as follows:

$$\begin{align*} P(X>Y) & = \sum_{y} P(X>Y,Y=y) \\ & =\sum_{y} P(X>y,Y=y)\\ & = \sum_{y} P(X>y) p_Y(y)\\ & = \sum_{y=1}^\infty (1-p_1)^y (1-p_2)^{y-1}p_2\\ & = (1-p_1)p_2 \sum_{y=1}^\infty ((1-p_1)(1-p_2))^{y-1}\\ & = \frac{(1-p_1)p_2}{1-(1-p_1)(1-p_2)}. \end{align*}$$

Suppose that @@@p_{X,Y}(x,y)@@@ is equal to a constant @@@c@@@ on the triangular set @@@\{(x,y): x\in \{0,\dots,4\},y \in \{0,\dots,x\}\}@@@ and is zero otherwise. Are @@@X@@@ and @@@Y@@@ independent? Here’s an incorrect argument: the PMF is constant @@@c@@@, and therefore trivially a multiple of a function of @@@x@@@ times a function of @@@y@@@, both can be taken as constant functions. From our second critertion for independence, the RVs are then independent. The problem is that the PMF is not constant. It is constant on the prescribed set, but is zero otherwise. This is very important and completely violates the condition of independence. Note, for example, that the probability that @@@X = 4@@@ is

$$\sum_{y=0}^4 c = 5c,$$

and the probability that @@@Y=4@@@ is

$$\sum_{x=4}^4 c = c.$$

The probability of the intersection @@@\{X=4,Y=4\}@@@ is exactly @@@c@@@ (one point in our triangular set), so independence would imply @@@5c^2 = c@@@, which gives @@@c=0@@@ or @@@c=\frac{1}{5}@@@. The value @@@c=0@@@ is inadmissible, because a PMF has to sum to @@@1@@@. And the value @@@c=\frac 15@@@ has a similar fate, because of the very same reason: choosing it and summing over our triangular shape we get total probability exceeding @@@1@@@ (there are exactly @@@15@@@ points in our triangle).

Find @@@p_{X|Y}(x|y)@@@ for @@@X@@@ and @@@Y@@@ as in Example 12. First we need to fix @@@y@@@. All that’s left is to plug into the definition. When @@@x<y@@@ we have

$$\begin{equation} \label{eq:disc_cond} p_{X|Y}(x | y)= \frac{p_{X,Y}(x,y)}{p_Y(y)} = \frac{ 6^{-y} 4^{x-1}5^{y-x-1}}{6^{-y}5^{y-1}}=\frac{1}{4} (\frac{4}{5})^{x}=\frac{1}{5}(\frac{4}{5})^{x-1}. \end{equation}$$

For @@@x=y@@@, we have @@@ p_{X|Y}(x|y) = 0@@@ because although @@@p_Y(y)>0@@@, @@@p_{X,Y}(y,y)=0@@@. Finally, for @@@x>y@@@, we repeat the computation, in \eqref{eq:disc_cond} but switching the roles of @@@x@@@ and @@@y@@@ in the numerator. This gives

$$\begin{align*} p_{X|Y}(x | y) & = \frac{ 6^{-x} 4^{y-1}5^{x-y-1}}{6^{-y}5^{y-1}}\\ & =6^{y-x} 4^{y-1} 5^{x-2y}\\ & =\frac 14 (\frac{4}{5})^y(\frac{5}{6})^{x-y}\\ & = (\frac{4}{5})^{y-1}\frac 16 (\frac{5}{6})^{x-y-1} \end{align*}$$

Let’s summarize:

$$\begin{equation} \label{eq:disc_cond_sum} p_{X|Y}(x | y) = \begin{cases} \frac15 (\frac{4}{5})^{x-1} & x<y \\ 0 & x=y \\ (\frac{4}{5})^{y-1} \frac 16(\frac{5}{6})^{x-y-1}& x>y \end{cases} \end{equation}$$

Let’s try to intuitively understand this formula. Suppose we know that the first @@@2@@@ occurred at time @@@y@@@. This simply tells us that the first @@@y-1@@@ tosses are not @@@2@@@, the @@@y@@@-th toss is @@@2@@@, and that’s it. No other conditions. Armed with this, the (conditioned) probability that @@@X=x@@@ is obtained from the following argument:

• If @@@x<y@@@, then @@@x-1@@@ tosses where outcomes for each are any of the four @@@\{3,4,5,6\}@@@ out of the possible five, @@@\{1,3,4,5,6\}@@@, followed by the toss whose outcome is @@@1@@@, one out of five, giving the first line in \eqref{eq:disc_cond_sum}
• If @@@x=y@@@, since we can’t have two different numbers at the same toss, we obtain that the probability is zero.
• If @@@x>y@@@, we have @@@y-1@@@ first tosses where outcomes for each are in @@@\{3,4,5,6\}@@@, followed by @@@y@@@-th toss equal to @@@2@@@ (assumed, so probability of that is one), then followed by @@@x-y-1@@@ tosses, each in any of five @@@\{2,3,4,5,6\}@@@ out of the possible six, followed by last toss equal to @@@1@@@, one out of six, giving the last line in \eqref{eq:disc_cond_sum}.

The number of visits to an online store in a day is @@@N\sim \mbox{Pois}(\lambda)@@@ for some @@@\lambda@@@. Each visitor makes a purchase with probability @@@p@@@, independently of anything else. Find the distribution of the number of purchases, @@@X@@@ per day, and the joint distribution of @@@N@@@ and @@@N-X@@@. Are the latter independent? To find the distribution of @@@X@@@, observe that what we are told is the following: given @@@N=n@@@, the number of purchases @@@X@@@ is Binomial with parameters @@@n@@@ and @@@p@@@. Therefore,

$$p_{X|N}(x|n) = \binom{n}{x} p^x (1-p)^{n-x} \mbox{ with }, ~x \in \{0,\dots,n\}.$$

Let’s calculate:

$$\begin{align*} p_X(x) & = \sum_{n}p_{X|N}(x | N)p_N(n) \\ & =\sum_{n=x}^\infty \binom{n}{x} p^x (1-p)^{n-x} e^{-\lambda}\frac{ \lambda^n}{n!} \\ & = \frac{e^{-\lambda}(\lambda p )^x }{x!} \sum_{n=x}^\infty \frac{(1-p)^{n-x}\lambda^{n-x} }{(n-x)!} \\ & = \frac{1}{x!} e^{-\lambda} (\lambda p)^x e^{\lambda (1-p)} = e^{-\lambda p} \frac{ (\lambda p )^x}{x!}, \end{align*}$$

Hence @@@p_X(n) \sim \mbox{Pois}(\lambda p)@@@. This is interesting, right? Similarly, @@@N-X@@@ has distribution @@@\mbox{Pois}((1-p)\lambda)@@@(because we can repeat this calculation but now count those visitors which did not make a purchase, selecting now each with probability @@@1-p@@@). But that’s not all, yet. Let’s look at the joint PMF of @@@X@@@ and @@@N-X@@@.

$$\begin{align*} p_{X,N-X}(x,y) & = p_{X,N}(x,x+y)\\ & = p_{X|N}(x | x+y) p_{N}(x+y)\\ & = \binom{x+y}{x} p^x (1-p)^{y} e^{-\lambda} \frac{\lambda ^{x+y} }{(x+y)!}\\ & = \frac{e^{-\lambda p}(\lambda p)^x }{x!} \times \frac{e^{-\lambda (1-p)} (\lambda (1-p))^y }{y!}\\ & = p_X(x) p_{N-X} (y) \end{align*}$$

Wow! The number of customers who made a purchase and the number of customers who did not make a purchase are independent. This is one of the beautiful features of Poisson: if a Poisson RV represents a number of objects, and you mark each object with fixed probability, independently of the others, then the number of marked and number of unmarked are both Poisson, and are independent. The first property…

Let’s revisit the last derivation of the expectation of Geometric RV in the Example disc_exp. Let @@@X\sim \mbox{Geom}(p)@@@ for @@@p\in (0,1]@@@. We will argue that @@@E[X]=1/p@@@ by conditioning on @@@Y@@@, where @@@Y@@@ is the indicator that the first trial is successful. We know that @@@E[X | Y=1]@@@ is @@@1@@@. What about @@@E[X | Y=0]@@@? We have a failure in the first trial. Then we wait for the first success but now starting from the second trial. Therefore @@@E[X | Y=0]=E[1+X]@@@. Putting it all together, we have We have

$$ \begin{align*} E[X]&= E[X | Y=1]p_Y(1) + E[X | Y=0]p_Y(0)\\ & = 1 *p + E[ X | Y=0](1-p)\\ & = 1*p + E[1+X](1-p) \\ & = 1+(1-p) E[X], \end{align*} $$

from which it follows that @@@E[X]=1/p@@@.

Compute @@@E[X | Y=y]@@@ where @@@X@@@ and @@@Y@@@ are as in Example 12. Writing down the formula is pretty straightforward:

$$ E[X | Y=y] =\underset{(*)}{\underbrace{\sum_{x<y} x \frac{1}{5} (\frac{4}{5})^{x-1}}} + (\frac{4}{5})^{y-1} \underset{(**)}{\underbrace{\sum_{x>y} x \frac{1}{6} (\frac{5}{6})^{x-y-1}.}}$$

We need to compute its value. By changing indices in @@@(**)@@@ from @@@x@@@ to @@@k=x-y@@@, we have

$$ (**) = \sum_{k=1}^\infty (k+y) (\frac{1}{6})(\frac{5}{6})^{k-1}=6+y,$$

which holds because the expression in the middle is simply the expectation of @@@\mbox{Geom}(1/6)+y@@@. As for @@@(*)@@@, we will use differentiation under the summation sign:

$$ (*) =\frac 15 \frac{d}{d p } \sum_{x=1}^{y-1} p^x, \mbox{ with } p= 4/5.$$

Since @@@\sum_{x=1}^{y-1} p^x = \frac{1-p^y}{1-p}@@@, the derivative is equal to @@@-y p^{y-1} (1-p)^{-1} + (1-p^y)(1-p)^{-2} = (1-p)^{-2} (1-p^y - y(1-p)p^{y-1})@@@. In our case, @@@p=\frac{4}{5}@@@ and @@@ 1-p=\frac{1}{5}@@@, so we obtain

$$ \begin{align*} (*) &= 5 - 5 (\frac{4}{5})^y -y (\frac{4}{5})^{y-1}\\ & = 5 - (\frac{4}{5})^{y-1} (4+y) \end{align*} $$

All together,

$$ E[ X | Y=y ] = 5 - (\frac{4}{5})^{y-1} (4+y-y-6)= 5 +2 (\frac{4}{5})^{y-1}.$$

To finish, let’s compute @@@E[X]@@@. Recall that @@@X@@@ is the number of tosses until the first @@@1@@@. Therefore @@@X\sim \mbox{Geom}(1/6)@@@, and @@@E[X]=6@@@. We compute it again, using the identity \eqref{eq:disc_noncond_from_cond}. Applying the identity with @@@g(x,y) = x@@@ and recalling that @@@Y@@@ is also @@@\mbox{Geom}(1/6)@@@, we obtain

$$ \begin{align*} E[X] &= \sum_{y} E[X | y] p_Y(y) =\sum_{y=1}^\infty (5+ 2 (\frac{4}{5})^{y-1}) p_Y(y)\\ & = 5 + \frac{2}{6} \sum_{y=1}^\infty (\frac{4}{5} \times \frac{5}{6})^{y-1} \\ & = 5 + 1=6. \end{align*} $$

You’re tossing a fair dice. What is the expected number of tosses until you first see a @@@2@@@, conditioned on all tosses were even?

Easy. If I know all tosses were even, then the probability of seeing a @@@2@@@ each time is @@@1/3@@@. Therefore the number of tosses until the first @@@2@@@ is geometric with parameter @@@1/3@@@ and the answer is @@@3@@@.

'’Wrong!’’ What exactly is wrong? ‘‘Everything’’ (had to say it, but not really). Let me explain: in this problem we are

• not conditioning on “all tosses were even” because this is an event of probability zero. This is not even what we’re saying.
• we are conditioning on “all tosses before the first @@@2@@@ were even” or, equivalently, “@@@2@@@ is the first to appear among the numbers @@@1,2,3,5@@@”. Heuristically, by conditioning, we are restricting the discussion to @@@1/4@@@ of the sequences, and then each sequence attains @@@4@@@ times its original probability. So what is the probability that the first toss is a @@@2@@@? @@@4*1/6@@@. What is the probability that @@@2@@@ appears first in the second toss (remember that we do not allow @@@1,3,5@@@ to appear before @@@2@@@? @@@4* (2/6)*1/6@@@, more generally, what is the probability that @@@2@@@ will appear for the first time at the @@@n@@@-th toss? @@@4* (2/6)^{n-1}1/6=(1/3)^{n-1}2/3@@@. In other words, the distribution of the number of tosses until (including) the first @@@2@@@ is @@@\mbox{Geom}(2/3)@@@, and the expectation is @@@3/2@@@!

Let’s do it right now. Let @@@B@@@ be the event that all tosses before the first @@@2@@@ were even, and let @@@T@@@ be the number of the toss @@@2@@@ is first observed.

$$ P(T=n,B)= (\frac{2}{6})^{n-1}\times \frac{1}{6},$$

because tosses are independent, and the @@@n-1@@@ first tosses are each @@@4@@@ or @@@6@@@, while the @@@n@@@-th toss is @@@2@@@.

By the total probability formula,

$$ P(B) = \sum_{n=1}^\infty P(T=n,B) = \frac{1}{6}\sum_{n=1}^\infty (\frac{2}{6})^{n-1}= \frac{1}{6}\times \frac{1}{1-1/3} =\frac 14.$$

Summarizing,

$$ P(T=n | B) = \frac{P(T=n,B) }{P(B)}= \frac 23 (\frac{1}{3})^{n-1}.$$

In other words, the distribution of @@@T@@@ conditioned on @@@B@@@ is @@@\mbox{Geom}(2/3)@@@. Its expectation (or conditional expectation) is therefore @@@3/2@@@. That was cool, right? comment %} The probability of seeing a @@@2@@@ each time is not @@@1/3@@@, but twice. Why? Let’s view this differently. The conditioning is merely saying that @@@2@@@ will appear before @@@1,3,5@@@. Or: The first time a number other than @@@4@@@ or @@@6@@@ will appear, it will be a @@@2@@@. By conditioning, we are restricting our sample space to only those sequences corresponding to @@@2@@@ being the first to appear ‘'’among @@@1,2,3,5@@@’’’. This means we are looking at only a “@@@1/4@@@” of all sequences, and as a result, the probability of each remaining sequence is @@@4@@@ times its original probability. So what is the probability that the first toss is a @@@2@@@? @@@4*1/6@@@. What is the probability that @@@2@@@ appears first in the second toss (remember that we do not allow @@@1,3,5@@@ to appear before @@@2@@@? @@@4* (2/6)*1/6@@@, more generally, what is the probability that @@@2@@@ will appear for the first time at the @@@n@@@-th toss? @@@4* (2/6)^{n-1}1/6=(1/3)^{n-1}2/3@@@. In other words, the distribution of the number of tosses until (including) the first @@@2@@@ is @@@\mbox{Geom}(2/3)@@@, and the expectation is @@@3/2@@@! endcomment %}

Let @@@(X,Y)@@@ be a random vector with independent marginals, @@@X\sim \mbox{Pois}(\lambda)@@@ and @@@Y\sim\mbox{Pois}(\mu)@@@. We will show that

$$ \begin{equation}\label{eq:sum_poisson} Z=X+Y\sim\mbox{Pois}(\lambda+ \mu). \end{equation} $$

Before showing the claim, note that by induction \eqref{eq:sum_poisson} shows that if @@@X_1,\dots,X_n@@@ are independent Poisson then their sum is a Poisson RV with parameter equal to the sum of the parameters @@@X_1,\dots,X_n@@@. Now let’s derive \eqref{eq:sum_poisson}. We use the discrete convolution formula. This gives

$$ \begin{align*} p_Z(z) &= \sum_{x=0}^\infty p_X (x) p_Y(z-x)\\ & = \sum_{x=0}^{z} e^{-\lambda} \frac{\lambda^x}{x!} e^{-\mu} \frac{\mu^{z-x}}{(z-x)!}\\ & = \frac{e^{-(\lambda+ \mu)}}{z!} \sum_{x=0}^z \frac{z!} {x! (z-x)!}\lambda^x \mu^{z-x}\\ & = e^{-(\lambda+ \mu)} \frac{ (\lambda + \mu)^z}{z!}. \end{align*} $$

We have used the binomial formula to get the last equality.

Suppose that @@@X\sim \mbox{U}[a,b]@@@ and @@@Y\sim \mbox{U}[c,d]@@@ are independent. Find the joint CDF of the random vector @@@(X,Y)@@@ and determine if it has a density. If it does, find it.

To solve, observe that

$$F_{X,Y} (x,y) = P(X\le x,Y \le y) = P(X\le x) P(Y\le y),$$

with last equality due to independence. Thus,

$$F_{X,Y}(x,y) = F_X (x) F_Y(y) = \int_{-\infty}^x f_X(s) ds \int_{-\infty}^y f_Y(t) dt = \int_{-\infty}^y\int_{-\infty}^x f_X(s) f_Y(t) ds dt.$$

This proves that @@@(X,Y)@@@ has density @@@f_{X,Y}(x,y)@@@, given by

$$f_{X,Y}(x,y) = f_X(s) f_Y(y) = \begin{cases} \frac{1}{(d-c)(b-a)} & a < x < b, c < y< d \\ 0 & \mbox{otherwise}\end{cases}$$

Suppose that the random vector @@@(X,Y)@@@ has joint CDF

$$F_{X,Y}(x,y)=\begin{cases} e^{-x} & 0\le x<y \\ (1-e^{-y})+(1-e^{-y})(e^{-y}-e^{-x}) & 0\le y\le x\\ 0 & \mbox{ otherwise} \end{cases}$$

Does @@@(X,Y)@@@ have density?

To answer, let’s differentiate, except on the axes and the ray @@@\{(x,x):x\ge 0\}@@@, where we cannot differentiate. It should not be a problem because the Riemann integral of any integrable function over any line is zero anyway.

• At points @@@(x,y)@@@ outside the first quadrant, the CDF is identically zero, so @@@\frac{\partial^2 F_{X,Y}}{\partial x \partial y}(x,y)=0@@@.
• At all points @@@(x,y)@@@ where @@@0<x<y@@@, @@@\frac{\partial F_{X,Y}}{\partial y}=0@@@, so that the mixed derivative @@@\frac{\partial^2 F_{X,Y}}{\partial x \partial y}@@@ is zero.
• At all points @@@(x,y)@@@ where @@@0<y<x@@@, differentiating first with respect to @@@y@@@ and then with respect to @@@x@@@ gives

$$ \frac{\partial^2 F_{X,Y}}{\partial y \partial x}(x,y) = e^{-x}e^{-y}.$$

Summarizing, we have

$$ f(x,y) = \frac{\partial^2 F_{X,Y}(x,y)}{\partial x \partial y} = e^{-x} e^{-y}$$

if @@@y<x@@@, and elsewhere, except on the axes and the ray @@@\{(x,x):x\ge 0\}@@@, the mixed derivative is @@@0@@@. But note that if we did have a density, the values of the density along these linear piece are irrelevant because they have no affect on the integral (area of a linear segment is zero). So we’ve found a density, right? No. Let’s look at the integral of the function which is our candidate for density:

$$ \begin{align*} \iint f(x,y) dx dy &= \int_0^\infty \int_0^x f(x,y)dy dx\\ &=\int_0^\infty e^{-x} \int_0^x e^{-y} dy dx \\ = \int_0^\infty e^{-x} (1-e^{-x}) dx = 1 -\frac 12 = \frac 12. \end{align*} $$

Oops! Then there is no density here, and this is no mistake. This example is of a random vector which is an analog of mixed RVs. What is missing? Let’s observe the following (this is a solution to Exercise 11). If we had a density @@@f@@@, then

$$ P(X=Y) = \iint_{\{(s,s):s \in \R \}}f(s,t) dsdt = \int_{-\infty}^\infty \underset{=0}{\underbrace{\int_{t}^t f(s,t) ds}} dt =0.$$

In our case, @@@P(X=Y)@@@ is not zero. It is actually @@@1/2@@@. This is were all the missing probability left. We are not going to show it because we will let you show it in Problem 2, where you will see where this joint CDF comes from.

Let

$$f(x,y) = \begin{cases} 2e^{-(x+y)} & x>y>0 \\ 0 & \mbox{otherwise}\end{cases} $$

Show that @@@f@@@ is a density function and find the densities of each of the marginals.

Clearly @@@f(x,y)\ge0@@@. Also

$$\iint f(x,y) dx dy =\int_0^\infty \int_x^\infty e^{-(x+y)} dy=2\int_0^\infty e^{-x} e^{-x}= 1.$$

Now

$$f_X(x) = \int f(x,y) dy =2\int_0^x e^{-(x+y)}dy= 2(e^{-x}-e^{-2x}),$$

while

$$f_Y(y) = \int f(x,y) dx = 2\int_y^\infty e^{-(x+y)} dx =2e^{-2y}.$$

Clearly, @@@Y\sim \mbox{Exp}(2)@@@.

Disclosure: I love my Subaru.

Suppose that the first time a new Honda breaks is Exponential with expectation one year, and the first time a Subaru breaks is Exponential with expectation six months. I have two new Hondas and one new Subaru. Assuming times until either car breaks are independent, what is the distribution of the time the first car breaks? What is the probability my Subaru will be the first to break? Write @@@H_1,H_2,S@@@ for the time in years, the first, second Honda and Subaru break respectively. Then @@@H_1,H_2 \sim \mbox{Exp}(1)@@@, and @@@S\sim \mbox{Exp}(2)@@@, and the three RVs are independent. We first reduce the problem to two RVs, setting @@@H=\min (H_1,H_2)@@@. As @@@P(H > t) = P( H_1 > t,H_2>t) =P(H_1>t) P(H_2>t) = (e^{- t})^2=e^{-2t}@@@. We conclude that @@@H\sim\mbox{Exp}(2)@@@. We showed this in Example 3, but it won’t hurt repeating. Also, since @@@H@@@ is a function of @@@H_1@@@ and @@@H_2@@@, it is independent of @@@S@@@. Repeating this argument we can see that the time the first car will break, @@@\min(H,S)@@@ satisfies:

$$ P( \min (H,S)>t) = P(H>t,S>t) = e^{-2t} e^{-2t} = e^{-4t},$$

Therefore @@@\min (H,S) \sim\mbox{Exp}(4)@@@, and on the average, the first car will break down in 3 months (@@@1/4@@@ of a year). Now what is the probability it is the Subaru? Or, in other words, what is @@@P(S<H)@@@. Let’s calculate.

$$ \begin{align*} P(S<H) &= \int_0^\infty\int_s^\infty 2 e^{-2s} 2e^{-2t} dtds \\ &= \int_0^\infty 2e^{-4s}ds\\ &=\frac 12 \end{align*} $$

Suppose that @@@(X,Y)@@@ is a random vector with independent marginals, @@@X\sim \mbox{Exp}(\lambda)@@@ and @@@Y\sim \mbox{Exp}(\mu)@@@. Show that

$$ P(X<Y) = \frac{\lambda}{\lambda+\mu}.$$

We’ve seen in Example 3 that @@@\min (X,Y)\sim \mbox{Exp}(\lambda+\mu)@@@. What is the probability that @@@X<Y@@@? Remember that larger parameter means small expectation (steeper drop in density as a function of @@@X@@@), so larger parameter means higher probability of being smaller. Enough talking, integral-time:

$$ \begin{align*} P(X< Y) &= \iint_{\{(x,y)\in \R^2:x<y\}} f_X(x) f_Y(y) dydx \\ & = \int_0^\infty \int_x^\infty \lambda e^{-\lambda x} \mu e^{-\mu y}dy dx\\ & = \int_0^\infty \lambda e^{-(\lambda + \mu) x} dx \\ & = \frac{\lambda}{\lambda+ \mu}. \end{align*} $$

Let @@@(X,Y)@@@ be a random vector with independent @@@\mbox{U}[0,1]@@@ marginals. Compute @@@E[\max (X,Y)]@@@.

Recall that

$$\max(x,y) = \begin{cases} x & x \ge y \\ y & x< y. \end{cases}$$

Now

$$ \begin{align*} E[\max(X,Y)] &= \int_0^1 \int_0^1 \max (x,y) dx dy \\ & =\int_0^1 (\int_0^y y dx ) dy + \int_0^1 (\int_{y}^1 x dx) dy.\\ & = \int_0^1 y^2 dy + \int_0^1 \frac{ 1- y^2}{2}dy \\ & = \frac 13 + \frac 12 - \frac 16 \\ & = \frac {2}{3} \end{align*} $$

In the second line we split the inner integral first to values in @@@[0,y]@@@, where @@@y@@@ is the maximum, and then from @@@y@@@ to @@@1@@@ where @@@x@@@ is the maximum. Note that there is another way to do this.

$$ E[ \max(X,Y)] = \int_0^\infty P(\max (X,Y)>t) dt=\int_0^1 P(\max(X,Y)>t) dt.$$

Now @@@\{\max(X,Y)>t\}@@@ is the complement of @@@\{\max (X,Y)\le t\}=\{X\le t\} \cap \{Y\le t\}@@@. Therefore

$$ P( \max (X,Y) > t) = 1- P(\max (X,Y) \le t) =1 - P(X\le t) P(Y\le t) = 1-t^2,$$

where the second equality is due to independence. The integral now becomes

$$ E[\max(X,Y) ] = \int_0^1 1- t^2 dt = 1 - \frac 13 = \frac 23.$$

Note that the CDF of @@@\max(X,Y)@@@ is given by the formula

$$ \begin{equation} \label{eq:cdf_max_unif} F_{\max(X,Y)} (t) = t^2,~ 0\le t \le 1 \end{equation} $$

Let @@@(X,Y)@@@ be a point selected at random on the unit disk @@@D= \{(x,y)\in \R^2: x^2+y^2 <1\}@@@. Find the expected distance of the random variable @@@R@@@, where @@@R@@@ is the distance of @@@(X,Y)@@@ from the origin. The distance from the origin of a point @@@(x,y)@@@ is equal to @@@\sqrt{ x^2 + y^2}@@@. Therefore @@@R=\sqrt{X^2+Y^2}@@@. The joint density of @@@(X,Y)@@@ is equal to

$$ f_{X,Y} (x,y) = \frac{1}{\pi} {\bf 1}_D (x,y).$$

Therefore, the expectation is equal to

$$ E[ R ] = \frac{1}{\pi} \iint \sqrt{x^2+y^2}dxdy$$

We change to polar coordinates to calculate this integral. Recall that @@@\sqrt{ x^2+y^2}=r@@@. Therefore the RHS is equal to

$$ \frac{1}{\pi} \int_0^{2\pi} \int_0^1 r r dr d\theta,$$

where the extra @@@r@@@ is the Jacobian for this subtitution. It follows that

$$ E[ R ] = \frac{1}{\pi} \int_0^{2\pi} \frac 13 d\theta= \frac 23.$$

By the way, what is the CDF of @@@R@@@? Let’s do a quick calculation.

$$ P(R \le t) = P( X^2 +Y^2 \le t^2) = \frac{1}{\pi} \iint_{\{(x,y): x^2 + y^2 \le t^2 \}} dxdy = \frac{1}{\pi} \pi t^2 =t^2.$$

That is

$$ F_{R} (t) = t^2,~ t \in (0,1).$$

Compare this with the CDF of the maximum of two independent @@@\mbox{U}[0,1]@@@, \eqref{eq:cdf_max_unif}. They are the same! Quite surprising, right? Same distribution obtained from two completely different procedures. Also, note that @@@\sqrt{\mbox{U}[0,1]}@@@ also has the same distribution. Can you explain the connections?

Consider the random vector from Example 23. Find the density of @@@X@@@ conditioned on @@@Y=y@@@ and the conditional expectation @@@E[X|Y=y]@@@.

Since we have calculated @@@f_Y(y)= 2 e^{-2y}@@@, it follows that

$$f_{X | Y} (x | y) = \frac {f_{X,Y}(x,y)}{f_Y(y)} = \frac{2e^{-x-y}}{2e^{-2y}}=e^{y} e^{-x},~x>y.$$

Remember that here @@@y@@@ is frozen. It is easy to see that this is the distribution of @@@\mbox{Exp}(1)@@@, conditioned to be above @@@y@@@. Indeed, let @@@Z\sim\mbox{Exp}(1)@@@. Then for @@@x>y@@@,

$$P(Z>x | Z>y) = P(Z>x)/P(Z>y) = e^{-x} / e^{-y}.$$

Therefore

$$ P(Z\le x | Z>y) = 1- e^{-x} / e^{-y},$$

and after differentiation we observe that the density of @@@Z@@@ conditioned to be above @@@y@@@ is indeed @@@e^y e^{-x}@@@ for @@@x>y@@@.

Now let’s compute the conditional expectation.

$$\begin{align*} E[X | Y=y] &= \int x f_{X | Y}(x | y) dx\\  &  = \int_y^\infty xe^{-(x-y)}dx\\  & \underset{u=x-y} {=} \int_0^\infty (u+y)e^{-u} du\\  & = 1+y. \end{align*}$$

Can you give an intuitive explanation to this value?

The distribution of tree height in the Amazon rain forest (this is all made up!) is @@@Y\sim \mbox{Exp}(\lambda)@@@. Conditioned on @@@Y=y@@@, the radius of the root system is @@@X\sim \mbox{Exp}(y)@@@. What is the distribution of the radius of the root system?

Here we are given the fact that @@@f_Y(y) = \lambda e^{-\lambda y},~y>0@@@ and @@@f_{X | Y} (x | y) = y e^{-y x}@@@. Therefore, the joint density function satisfies

$$f_{X,Y} (x,y) = f_{X | Y} (x | y) f_Y(y) = y e^{-xy} \lambda e^{-\lambda y} = y \lambda e^{- (x + \lambda)y }.$$

Now

$$\begin{align*} f_X(x) &= \int f(x,y) dy  = \lambda \int_0^\infty y e^{- (x+\lambda)y }\\   &=\frac{ \lambda}{\lambda+ x}   \underset{=E[Exp(x+\lambda)]}{\underbrace{ (\lambda + x)\int_0^\infty y e^{-(x+\lambda)y} dy}}\\   & = \frac{\lambda}{(\lambda+ x)^2}   \end{align*}$$

Consider the random vector @@@(X,Y)@@@ from Example 29. Find @@@ f_{Y | X} (y | x)@@@.

We have

$$\begin{align*} f_{Y | X} (y | x) &= \frac{ f_{X | Y} (x | y) f_Y(y)}{f_X(x)} \\ & = \frac{y \lambda e^{- (x+ \lambda)y }} {\lambda / (\lambda+x)^2}  \\ & = (\lambda +x)^2 y e^{-(x+\lambda)y}. \end{align*}$$

As we will see in the first part of Example 33, this is the density of a sum of two independent @@@\mbox{Exp}(\lambda +x)@@@ RVs.

We will show that for @@@a,b\in {\mathbb Z}_+@@@,

$$\begin{equation} \label{eq:integral} \int_0^1 x^a (1-x)^b dx = \frac{a!b!}{(a+b+1)!} \end{equation}$$

How do we do that? One approach would be repeated integration by parts. We will offer a purely probabilistic method that won’t require calculation of any integrals. Let’s begin. Of course, there’s really no point continuing if @@@a=b=0@@@. It’s a triviality (don’t forget that @@@0!=1!=1@@@). Let’s assume then that @@@a+b=N\ge 1@@@. Now take @@@U,U_1,\dots,U_{N}@@@ IID @@@\mbox{U}[0,1]@@@.

The key to our solution is the answer to the following question:

'’What is the probability that @@@U@@@ is the @@@a+1@@@-th smallest among all @@@N+1@@@ RVs?’’

Call the event @@@A_a@@@ and denote its probability by @@@p_a@@@.

• First note that the question is well-defined. Indeed, the probability that any two of the @@@N+1@@@ RVs are equal is zero, because we have a continuous random vector. Therefore, with probability @@@1@@@ we can rank them with no ties (this would not be the case if we have IID which are discrete).
• Because the RVs are IID, the probability that @@@U@@@ happens to be the @@@a+1@@@-th is exactly the same as any of the @@@N+1@@@ RVs is the @@@a+1@@@-th. Since this gives us @@@N+1@@@ disjoint events whose union has probability @@@1@@@ (one of the RV’s must be ranked @@@a+1@@@-th),

$$\begin{equation} \label{eq:pa}  p_a = \frac{1}{N+1} \end{equation}$$

• The event @@@A_a@@@ can be decomposed as follows.
  • Let @@@{\cal C}@@@ be the set of combinations of @@@a@@@ elements from @@@\{1,\dots,N\}@@@. An element of @@@{\cal C}@@@ is a subset of @@@\{1,\dots,N\}@@@ consisting of exactly @@@a@@@ elements.
  • For each combination in @@@C\in {\cal C}@@@, let @@@A_C@@@ be the event @@@\{\max_{i\in C} (U_i)<U\}\cap \{\min_{i\in C} U_i >U\}@@@. That is, the elements of @@@C@@@ represent the indices of the @@@a@@@ RVs smaller than @@@U@@@ and the remaining represent the indices of those larger than @@@U@@@.
  • Then, up to an event with probability equal to zero, we have that @@@A_a = \cup_{C\in {\cal C}} A_C@@@, splitting the event @@@A@@@ according to the “identity” (=indices) of the remaining RVs are smaller than it.
  • Therefore

$$\begin{equation} \label{eq:summing_PAC} p_a = \sum_{C\in {\cal C}}P(A_C). \end{equation}$$

• Now calculate @@@P(A_C)@@@ for @@@C\in {\cal C}@@@. To do that, condition on @@@U=x@@@. Because of the independence, for every @@@C\in {\cal C}@@@ we have

$$ P( A_C | U= x) = P(\cap_{i\in C} \{U_i < x\} ) P( \cap_{i\not \in C} \{U_i > x\} ) = x^a (1-x)^{N-a}=x^a(1-x)^b,$$

and since @@@U@@@ is @@@U[0,1]@@@, we have

$$\begin{equation} \label{eq:individual_PAC} P(A_C) = \int_0^1 P(A_C | U=x )dx =  \int_0^1 x^a (1-x)^{b}dx. \end{equation}$$

• Since the number of elements in @@@{\cal C}@@@ is @@@\binom{a+b}{a}@@@, it follows that

$$ \frac{1}{a+b+1} \overset{\eqref{eq:pa}}{=} p_a \overset{\eqref{eq:summing_PAC},\eqref{eq:individual_PAC}}{=} \binom{a+b}{b} \int_0^1 x^a (1-x)^b dx.$$

This establishes \eqref{eq:integral}, without calculating a single integral. I’d also like you to take a second look at the integral and note that is equal to @@@E[U^a (1-U)^b]@@@.

Let @@@(X,Y)@@@ be a random vector with independent @@@\mbox{U}[0,1]@@@ marginals. Find the density of @@@Z=X+Y@@@.

By the convolution formula,

$$  f_Z (z) =\int f_X (x) f_Y(z-x) dx.$$

I did not write the limits because I want to discuss them. As @@@Z@@@ is the sum of two @@@\mbox{U}[0,1]@@@, its values can range from @@@0@@@ to @@@2@@@. Let’s first calculate the density when @@@z \in [0,1]@@@. In this case, since @@@f_Y@@@ is zero on negative numbers, we must have @@@z-x>0@@@ or @@@x<z@@@. Since @@@f_X@@@ and @@@f_Y@@@ are equal to @@@1@@@ on @@@[0,1]@@@ we have

$$ f_Z (z) = \int_0^z  dx =z=1-(1-z).$$

Let’s turn to the case @@@z\in [1,2]@@@. Here we need to integrate over a range of values for @@@x@@@ such that @@@z-x <1@@@, or @@@x> z-1@@@. That is

$$ f_Z (z) = \int_{z-1}^1 dx = 1-(z-1).$$

Combining both cases, we have just obtained the following formula

$$ f_Z (z) = \begin{cases} 1-|z-1| & z\in [0,2] \\  0 & \mbox{otherwise}\end{cases}$$

Draw this!

Suppose that @@@(X_1,X_2)@@@ is a random vector with independent marginals @@@X_1\sim \mbox{Exp}(\lambda_1)@@@ and @@@X_2\sim \mbox{Exp}(\lambda_2)@@@. Find the density of @@@Z= X_1+X_2@@@.

Let’s find the convolution of @@@f_{X_1}@@@ and @@@f_{X_2}@@@:

$$\begin{align*}  f_Z (z) &= \int f_{X_1} (x) f_{X_2} (z-x)dx  \\ & = \lambda_1 \lambda_2 \int_0^z  e^{-\lambda_1 x } e^{-\lambda_2(z-x)} dx \\ & = \lambda_1 \lambda_2 e^{-\lambda_2 z} \int_0^z e^{-(\lambda_1 -\lambda_2)x } dx. \end{align*}$$

Note that the integral is up to @@@z@@@ because the density @@@f_{X_2}@@@ is zero for negative numbers. To conclude we nee to consider two cases

1. @@@\lambda_1=\lambda_2@@@. Then

$$ f_Z(z) = \lambda_1^2 ze^{-\lambda_1 z}.$$

1. @@@\lambda_1 \ne \lambda_2@@@. Without loss of generality, we will assume @@@\lambda_1>\lambda_2@@@. Then

$$ f_Z (z) = \lambda_1 \lambda_2 \frac{ e^{-\lambda_2 z} - e^{-\lambda_1 z}}{ \lambda_1 - \lambda_2}.$$

Calculate the density of the sum of @@@n@@@ independent @@@\mbox{Exp}(\lambda)@@@ RVs. Observe that for @@@n=1@@@, the density is, of course, @@@\lambda e^{-\lambda x}@@@. As we saw in the first part of Example 33 for @@@n=2@@@, the density is @@@\lambda (\lambda x)e^{-\lambda x}@@@. You probably see where this is going to (more or less, there may be a normalizing constant to make the integral @@@1@@@). Let’s assume the density of @@@n-1@@@ is @@@f_{n-1}@@@. We need to find its convolution with the density of @@@\mbox{Exp}(\lambda)@@@:

$$f_{n} (x) = \int_0^x f_{n-1} (y) \lambda e^{-\lambda (x-y)} dy.$$

Let’s play smart and use some differential equations instead of guessing. Multiply both sides by @@@e^{\lambda x}@@@. What we get is

$$e^{\lambda x} f_{n} (x) = \lambda \int_0^x f_{n-1} (y)  e^{\lambda y } dy.$$

Or, if we define @@@g_n(x) = e^{\lambda x}f_n (x)@@@ (and same for @@@n-1@@@), we have shown that

$$ g_{n} (x) =\lambda \int_0^x  g_{n-1}(y)dy.$$

Since @@@g_1(x) = \lambda@@@, @@@g_2(x) = \lambda (\lambda x)@@@, @@@g_3 = \lambda \lambda^2 x^2/2@@@, and by induction,

$$g_n (x) = \lambda \frac{ (\lambda x)^{n-1}}{(n-1)!}.$$

Thus for all @@@n@@@,

$$f_n (x) = \lambda \frac{(\lambda x)^{n-1}}{(n-1)!} e^{-\lambda x} .$$

Let @@@(X,Y)@@@ be uniformly distributed on the unit disk @@@D=\{(x,y)\in \R^2:x^2+y^2<1\}@@@. That is

$$f_{X,Y} (x,y) = \begin{cases} \frac{1}{\pi} & x^2 + y^2 <1 \\ 0 & \mbox{otherwise.}\end{cases}$$

1. Are @@@X@@@ and @@@Y@@@ independent? Careful here. Can we write @@@f_{X,Y}(x,y)@@@ as a product of a function of @@@x@@@ and a function of @@@y@@@? The answer is negative, and the reason is that @@@D@@@ is a circle (not a rectangle with sides parallel to axes). Let’s give two arguments. The first will be based on joint densities. By symmetry if @@@f_{X,Y} (x,y) = f(x)g(y)@@@, then @@@g=f@@@. In particular, we have @@@f_{X,Y}(x,x) = f(x)^2@@@. This implies that @@@f(x) = \frac{1}{\sqrt{\pi}}@@@ for all @@@|x|<1@@@. This choice of @@@f@@@ is not a density function. Ok. I said I’ll give you another one. This is based on the definition of independent RVs. The probability that @@@X>3/4@@@ is positive and is equal to the probability that @@@Y>3/4@@@ (why?). However, the event @@@\{X>3/4,Y>3/4\}@@@ is contained in the event @@@\{X^2 + Y^2 > 2 \frac{9}{16}\}@@@. But @@@2 \frac{9}{16}>1@@@, and therefore the probability of the intersection is zero, as @@@(X,Y)@@@ is in the unit disk.
2. Write @@@X@@@ and @@@Y@@@ in polar coordinates:

$$X = R \cos \Theta,~Y = R \sin \Theta.$$

Find the joint distribution of @@@(R,\Theta)@@@. To solve we use the last result:

$$f_{R,\Theta}(r,\theta) = f_{X,Y}(r \cos \theta, r\sin \theta) |J(r,\theta)|.$$

The Jacobian in this case is equal to @@@r@@@. Therefore the joint density of @@@R@@@ and @@@\Theta@@@ is

$$f_{R,\Theta} (r , \theta) = \begin{cases} \frac{1}{\pi} r & r \in (0,1),~\theta \in [0,2\pi)\\ 0 &\mbox{ otherwise.}\end{cases}$$

Thus, @@@R@@@ and @@@\Theta@@@ are independent because the joint density is the product of the densities of the marginals. What are the distributions of each of the marginals?

Now @@@R@@@ has density

$$f_{R}(r) = \int_{0}^{2\pi} f_{R,\Theta} (r,\theta) d \theta = 2 r,~r \in (0,1),$$

or

$$F_R(r) = r^2,~ r \in (0,1).$$

But this is not news. We’ve already computed it before in Example 27. The marginal @@@\Theta@@@ is @@@\mbox{U}[0,2\pi]@@@ because the joint density is only a function of @@@r@@@ and after integrating @@@r@@@ out, we get a constant (independent of @@@\theta@@@).

The random vector @@@(U,V)@@@ has independent marginals with @@@U\sim \mbox{Exp}(7)@@@ and @@@V \sim \mbox{Exp}(14)@@@. Set @@@X = U+3V@@@ and @@@Y= 2U-V@@@. Find the joint density of @@@(X,Y)@@@.

The first thing we’d like to mention and which is extremely important is the range of @@@(X,Y)@@@. Yes, @@@(U,V)@@@ has a density on the first quadrant, but @@@(X,Y)@@@ is a transformation, so it may take values elsewhere. To deal with this, see where the boundaries are mapped. The half-line @@@v=0,u>0@@@ is mapped to @@@(u,2u),~u>0@@@, and the half-line @@@u=0,v>0@@@ is mapped to @@@(3v,-v),~v>0@@@, that is the half line with slope @@@-1/3@@@. Now look at any arbitrary point in the @@@(U,V)@@@ range, say @@@(1,1)@@@. As this is mapped to @@@(4,1)@@@, it follows that the range of @@@(X,Y)@@@ is the cone

$$C = \{(x,y): 0<x,~  -\frac x3< y<2x\}.$$

So our new density will be on this domain. The rest is algebra. Since

$$\begin{cases} x= f_1(u,v) = u +3v \\ y =f_2(u,v) =  2u -v \end{cases}$$

Solving, we have

$$ \begin{cases} u = g_1(x,y) = \frac{1}{7} (x+3y)\\ v= g_2(x,y) = \frac{1}{7} (2x-  y) \end{cases}$$

The Jacobian matrix for @@@T^{-1}@@@ is equal to

$$\left(  \begin{array}{cc} \frac 17&  \frac 37 \\ \frac{2}{7} & -\frac{1}{7}\end{array}\right)$$

Therefore

$$J_{T^{-1}}(x,y) = \frac{1}{49} \left( -1 -6 \right)=-\frac 17.$$

Thus,

$$ f_{X,Y}(x,y) = f_{U,V}(\frac{x+3y}{7},\frac{2x-y}{7})\left|\det(J_{T^{-1}})\right|= f_{U,V}(\frac{x+3y}{7},\frac{2x-y}{7})\frac{1}{7}$$

Since @@@U@@@ and @@@V@@@ are independent, @@@f_{U,V}(u,v) = f_U(u) f_V(v) = 7e^{-7u} \cdot 14e^{-14v}@@@ for @@@u,v>0@@@.

$$ f_{X,Y}(x,y) = \left(7 e^{-7 \frac{x+3y}{7}} \cdot 14 e^{-14 \frac{2x-y}{7}}\right) \frac{1}{7}= 14 e^{- (x+3y)} e^{-2(2x-  y)}\frac{1}{7}= 14 e^{-x-3y-4x+2y}\frac{1}{7}= 2 e^{-5x-y} \mbox{ on }C$$

and @@@f_{X,Y}(x,y)=0@@@ otherwise. Want to check?

(source) Two points are sampled uniformly and independently from the square @@@[0,1]\times [0,1]@@@. What is the probability that the center of the square is contained in the circle formed by the diameter of the two points?

It doesn’t really matter where the square is, so to exploit the symmetry of the problem, assume that the square is @@@[-1/2,1/2]\times [-1/2,1/2]@@@, and let @@@Z_1=(X_1,Y_1),Z_2=(X_2,Y_2)@@@ be the two points selected. Then @@@X_1,Y_1,X_2,Y_2@@@ are all independent @@@\mbox{U}[-1/2,1/2]@@@.

Observe that a circle intersects the origin if and only if the norm of its center is smaller than its radius.

The norm of the center squared is

$$\left\|\frac{Z_1+Z_2}{2}\right\|^2=\frac 14 \|Z_1\|^2 + \frac 14 \|Z_2\|^2 +\frac 12 Z_1 \cdot Z_2,$$

and the norm of the radius squared is

$$\left\|\frac{Z_1-Z_2}{2}\right\|^2=\frac 14 \|Z_1\|^2 + \frac 14 \|Z_2\|^2 - \frac 12 Z_1\cdot Z_2.$$

Therefore the origin is in the circle if and only if

$$\left\|\frac{Z_1+Z_2}{2}\right\|^2 < \left\|\frac{Z_1-Z_2}{2}\right\|^2$$

if and only if

$$\frac 14 \|Z_1\|^2 + \frac 14 \|Z_2\|^2 +\frac 12 Z_1 \cdot Z_2 < \frac 14 \|Z_1\|^2 + \frac 14 \|Z_2\|^2 - \frac 12 Z_1\cdot Z_2$$

if and only if @@@\frac 12 Z_1 \cdot Z_2 < - \frac 12 Z_1 \cdot Z_2@@@ if and only if @@@Z_1 \cdot Z_2>0@@@. But this RV is continuous and symmetric, therefore the probability is @@@1/2@@@.

Let @@@X_1,\dots,X_n@@@ be independent @@@\mbox{Exp}(\lambda)@@@. We already showed that the probability that any two of the variables are the same is zero. That is, with probability @@@1@@@, there is a unique well defined minimum, second smallest, etc. Write @@@X_{(1)}@@@ for the minimum of @@@X_1,\dots,X_n@@@, @@@X_{(2)}@@@ for the second smallest, etc. More precisely, once @@@X_{(j)}@@@ has been defined we continue inductively letting

$$ X_{(j+1)} = \{\min X_k: k=1,\dots,n \mbox{ and } X_k > X_{(j)}\}.$$

The “sorted” sequence @@@(X_{(1)},\dots,X_{(n)})@@@ is often called the \text{order statistics} of the sequence @@@(X_1,\dots,X_n)@@@. Note that the vector of order statistics is a (rather complex) transformation of our original vector:sorting is a deterministic function of the input.

What we will do is try to get some understanding of the order statistics. We already showed that @@@X_{(1)}@@@ is @@@\mbox{Exp}(n\lambda)@@@. In order to continue, we will not work directly with the order statistics but instead look at the sequence of induced increments. More precisely, let @@@Y_1 = X_{(1)}@@@ and continue inductively letting

$$ Y_{j+1} = X_{(j+1)} - X_{(j)},~j=1,\dots,n-1.$$

That is @@@Y_2@@@ is the difference between the second smallest and the smallest, @@@Y_3@@@ is the difference between the third smallest and the second smallest, etc. Note that the sequence @@@(Y_1,\dots,Y_n)@@@ is a transformation of the order statistics. A rather simple one, right? We will calculate the joint distribution of @@@(Y_1,\dots,Y_n)@@@ directly without any special formulas for transformations, starting from the joint distribution of @@@(X_1,\dots,X_n)@@@. Get ready for a fun ride!

Let’s write the joint density of @@@(X_1,\dots,X_n)@@@, conditioned on the event @@@A = \{X_1< X_2<\dots <X_n\}@@@. Note that the event @@@A@@@ corresponds to one of the @@@n!@@@ possible permutations of indices of @@@\{1,\dots,n\}@@@, each corresponding to a unique ordering of the @@@n@@@ RVs in increasing values. Since the union of all of the corresponding events has probability @@@1@@@ (all are different with probability @@@1@@@, and when all are different there are @@@n!@@@ ways to order them), it follows that @@@P(A) = 1/n!@@@. We therefore have

$$ f(x_1,\dots,x_n | A) = n! \prod_{j=1}^n (\lambda e^{-\lambda x_j} {\bf 1}_{\{x_j > x_{j-1}\}}),$$

with @@@x_0=0@@@ for convenience. The trick is to rewrite this, expressing it through the increments @@@x_j -x_{j-1}@@@. Let’s do it. Write @@@y_j = x_j-x_{j-1}@@@. Then for @@@j\ge 2@@@, @@@x_j = y_1 + \sum_{k=2}^j y_k@@@. Putting it all together we have that the joint density, expressed in terms of the variables @@@y_1,\dots,y_n@@@ is the function @@@g(y_1,\dots,y_n|A)@@@, which for @@@y_1,y_2,\dots,y_n>0@@@ is given by

$$\begin{align*} g(y_1,\dots,y_n|A) & =  f(x_1,\dots,x_n|A) = n! \lambda^n e^{-\lambda y_1}e^{-\lambda (y_1 + y_2)} e^{-\lambda (y_1 + y_2 + y_3)} \cdots e^{-\lambda (y_1+ \cdots + y_n)} \\ & = n! \lambda^n  e^{-\lambda n y_1}  e^{-\lambda (n-1) y_2} \cdots e^{-\lambda y_n}\\ & = \prod_{j=1}^n ((n+1-j) \lambda) e^{-(n+1-j) \lambda y_j}. \end{align*}$$

From their definitions, the variables @@@y_1,\dots,y_n@@@ represent differences between consecutive order statistics, and so we’re looking at the joint density of @@@Y_1,\dots,Y_n@@@ conditioned on the event @@@A@@@.

Since this particular form of density is invariant under the @@@n!@@@ different permutations, the joint density of @@@Y_1,\dots,Y_n@@@ remains the same when we do not condition on any particular permutation. In other words,

$$f_{Y_1,\dots,Y_n} = \prod_{j=1}^n ((n+1-j) \lambda) e^{-(n+1-j) \lambda y_j},~y_1,\dots,y_n >0.$$

That is, @@@(Y_1,\dots,Y_n)@@@ are independent exponential RVs with @@@Y_j \sim \mbox{Exp}((n+1-j) \lambda)@@@. Let’s frame this:

$$\begin{equation} \boxed {\label{eq:exp_order} Y_1=X_{(1)} \sim \mbox{Exp}(\lambda n),Y_2=X_{(2)}-X_{(1)} \sim \mbox{Exp}(\lambda (n-1)),\dots, Y_n=X_{(n)}-X_{(n-1)} \sim \mbox{Exp}(\lambda), }\end{equation}$$

and all are independent.