Intro

Random variables are neither random nor variables. They are simply real-valued functions on the sample space, or real-valued measurements: an assignment of a numerical value for each outcome in the sample space. Number of Heads in a sequence of tosses is a random variable, and so (a bit of stretch) is the price I’ll pay at checkout next time I put my car in the repair shop, the amount you’ll win next time you play the lottery, the time I’ll wait at the station for the bus home, etc. Remember: functions, numerical assignments to outcomes in the sample space.

Random Variables

Definition and First Examples

First an acronym: RV. So why do we even care about RVs? Let’s think of a few reasons. Numbers are much easier to manipulate than abstract objects suchex as sequences of Heads and Tails, etc. They also allow to concisely summarize the outcome (e.g. “@@@7@@@ Heads”, rather than listing the entire sequence). Want another one? RVs could be thought of as monetary value assigned to outcomes: if you play a game of chance, your net gain is a random variable.

'’Essentially’’, if @@@\Omega@@@ is your sample space and @@@f:\Omega\to [-\infty,\infty]@@@ is a function, then @@@f(\omega)@@@ is a random variable. I write ‘‘essentially’’, because this captures the essence, but ignores an important technical aspect. Here is the precise definition

Remember we worked really hard to understand what events are? The technical aspect about RVs is that we want any (reasonable) statement about them like “@@@X@@@ is less than or equal to @@@100@@@” or “@@@X@@@ is strictly between @@@2@@@ and @@@4@@@” or “@@@X@@@ is even”, etc… to be an event. Why? Because (eventually) we want to be able to associate probabilities to the numerical values: for example, we would like to know what is the probability that our plane will be no more than 20 minutes late (identify the random variable here), and we only assign probabilities to events, right?

Turns out that the condition listed in the definition, requiring only a relatively small statements to be events is all we need in order to make pretty complex statements. We’ve already been in a very similar situation when discussing @@@\sigma@@@-algebras.

The most trivial RV? A constant. Whatever the @@@\sigma@@@-algebra is any constant function is always a RV. But that’s boring. Let’s turn to more interesting examples.

Similarly to events, to define RVs all we need is a sample space and a @@@\sigma@@@-algebra (a pair known as a measurable space), not even a probability distribution. This is because RVs are deterministic real-valued functions on the sample space satisfying a consistency condition with the @@@\sigma@@@-algebra.

Remember we only require @@@\{X\le \alpha\}@@@ to be an event, where @@@\alpha@@@ ranges over all real numbers. Let’s see what this gives us, almost for free. It’s much more. Remember: complements, countable unions and countable intersections of events are all events. So:

• @@@\{X>\alpha\}@@@ is an event because it’s the complement of @@@\{X\le \alpha\}@@@, which, by definition, is an event.
• @@@\{\beta < X \le \alpha\}@@@ is an event, because it’s the intersection of the events @@@\{X\le \alpha\}@@@ (definition) and @@@\{X>\beta\}@@@ (what we just showed);
• @@@\{\beta < X < \alpha\}@@@ is an event: it’s the union of @@@\{\beta < X \le \alpha - \frac 1n\}@@@, @@@n=1,2,\dots@@@.
• @@@\{\beta \le X < \alpha\}@@@ is an event: it’s the intersection of @@@\{\beta -\frac 1n< X <\alpha\}@@@ over @@@n=1,2,\dots@@@.
• @@@\{\beta \le X \le \alpha\}@@@ is an event. Note the special case @@@\beta=\alpha@@@: @@@\{X=\alpha\}@@@ is an event too.
• The complement of any of the above is an event.
• Any countable union of any of the above is an event, and so are complements of such unions, unions of those complements and anything listed before, complements of those, etc…

I hope you got the gist of it by now: if @@@I@@@ is any interval (open/closed/half of each/finite/infinite), then @@@\{X\in I\}@@@ is an event, and since the collection of events is a @@@\sigma@@@-algebra, we have that @@@\{X\in A\}@@@ is an event whenever @@@A@@@ is a countable union of intervals, or complements of such sets, or intersections of such sets, or unions of such intersections, etc.

Upon a slightly closer inspection, we see that the requirement @@@\{X\le \alpha\}@@@ is an event implies that the collection of subsets of @@@\R@@@

is itself as @@@\sigma@@@-algebra on @@@\R@@@. In other words, whatever @@@(\Omega,{\cal F})@@@ were, a RV @@@X@@@ induces a @@@\sigma@@@-algebra on @@@\R@@@. This @@@\sigma@@@-algebra, includes all intervals as elements. Therefore it contains the Borel @@@\sigma@@@-algebra. We state this as a Corollary:

Indicators

Simply put, an indicator is any RV taking only the values @@@0@@@ and @@@1@@@ (possibly only one of them). Indicators are also commonly known as Bernoulli RVs, a well-deserved name as these are the building blocks of RVs, and Mr. Bernoulli did make some significant contributions to the area.

How can one obtain a RV taking only the values @@@0@@@ and @@@1@@@? Fix an event @@@A@@@, and let

That is: @@@X@@@ “indicates” whether @@@A@@@ has occurred or not, and is therefore referred to as the ‘'’indicator (function) of @@@A@@@’’’, denoted for brevity by @@@{\bf 1}_A@@@. Let’s take @@@\alpha \in \{-\infty\}\cup \R@@@ and consider the event @@@\{X \le \alpha\}@@@. Then

• It is @@@\emptyset@@@ when @@@\alpha <0@@@;
• It is @@@A^c@@@ if @@@\alpha \in [0,1)@@@; and
• It is @@@\Omega@@@ if @@@\alpha \ge 1@@@. As each of these are events, @@@X@@@ is an RV.

If @@@X@@@ is defined as the indicator of an event @@@A@@@, then it only takes the values @@@0@@@ and @@@1@@@. The converse is also true. I’m leaving this as an exercise.

Some algebraic operations on indicators correspond to set operations on the respective events. Suppose @@@X@@@ is the indicator of the event @@@A@@@ and @@@Y@@@ is the indicator of the event @@@B@@@. Then @@@XY@@@ is equal to @@@1@@@ on @@@A \cap B@@@ and is zero otherwise. Therefore, @@@XY@@@ is the indicator of @@@A\cap B@@@. Note that @@@XY@@@ is also equal to the minimum of @@@X@@@ and @@@Y@@@ (look at each of the values and take the lower), denoted by @@@\min(X,Y)@@@. Let’s look at other operations. Before doing that, let me make sure you are familiar with the maximum and minimum functions. We’ll use them a lot. The maximum (larger) of two numbers @@@a@@@ and @@@b@@@ is denoted by @@@\max(a,b)@@@, and the minimum (smaller) is denoted by @@@\min(a,b)@@@. For example @@@\max (7,10)=10,\min(7,10)=7,\max(5,5)=5@@@, etc…

Here’s a video explaining RVs and indicators

Ready?

One can combine indicators to obtain more elaborate random variables.

Feel free to skip this last example at first reading. I know Jakob Bernoulli looks intimidating. I promise I won’t tell him.

Operations with RVs

Let’s get into the business of generating new RVs from existing ones. We sampled a little of it, but we’re going much deeper now, but it will be short.

Let’s see what this tells us.

In a few words: pretty much every (reasonable) algebraic manipulation of random variables yields a random variable.

Let’s be a little more specific.

From the first part we see that things like @@@2X@@@, @@@X+1@@@, @@@e^X@@@ are all RVs. Similarly, if @@@X@@@ is any RV, then letting

is also a RV because it is a piecewise continuous function of @@@X@@@ (you can think of it as acceptance status as a function of SAT: @@@1@@@ for accepted, @@@1/2@@@ for waitlisted and @@@0@@@ for rejected).

From the second part, we see that @@@X\pm Y@@@ as well as @@@XY@@@ are RVs. Using this repeatedly with the first part, we see that objects like @@@\sin (X)+\cos (X-2Y)@@@ is a RV. By the way, you can iterate this to show that similar expressions involving three, four or any finite number of RVs is still a RV.

So what about the last part? What is even @@@\sup@@@? Definitely not something you take to the lake. It’s pronounced as “supremum” and is the equivalent of the notion of maximum, but when we have infinitely many numbers. Suppose you’re looking at an infinite sequence (of nmubers, of course). The supremum is defined as the smallest among all numbers bigger or equal to all elements in the sequence, and @@@+\infty@@@ if there are no such numbers. Let’s clarify:

• When the sequence has a largest element, this is the supremum. In this case it coincides with the maximum.
• When the sequence is something like @@@1-\frac 12,1-\frac 13,1-\frac 14,\dots@@@, then it does not have a largest element (a maximum). There are many numbers bigger than any number in the sequence: @@@2, 1.5, 1.001, 1.0001@@@. The smallest amongthem is @@@1@@@. That’s the supremum.
• When the sequence does not have any number bigger than all of its elements then the supremum is defined as @@@+\infty@@@. The sequence @@@1,2,3,\dots@@@ is such a sequence, right?

Now that we understand the notion, let’s continue with the main message. It is important if we want to take limits (not to “limit” ourselves to finite operations). If @@@X_1,X_2,\dots@@@ are nonnegative RVs, then by the second part, the partial sums @@@S_1=X_1,S_2=X_1+X_2,S_3= X_1+X_2+X_3@@@ are all RVs. Observe that the series @@@\sum_{j=1}^\infty X_j@@@ (which can also take the value @@@+\infty@@@), is simply @@@\sup_{n=1,2,\dots} S_n@@@, and is therefore a RV. This explains why the series in Example Example 8 from last section is a RV.

Distribution (Functions) of RVs

In this section we finally make the link between RVs and probability measures.

Suppose we’ve got ourselves an RV @@@X@@@ on a probability space @@@(\Omega, {\cal F},P)@@@. Note that we now include a probability measure. Since we’ve already observed that for any Borel set @@@B@@@, @@@\{X\in B\}@@@ is an event, namely an element of @@@{\cal F}@@@, it has an assigned @@@P@@@-probability, @@@P (X \in B)@@@. Thus, letting

we immediately see that @@@P_X@@@ is (drum roll) a probability measure on the measurable space @@@(\R,{\cal B})@@@, the real line equipped with the Borel @@@\sigma@@@-algebra @@@{\cal B}@@@. This probability measure is called ‘'’the distribution of @@@X@@@’’’.

Why is this important? No matter what @@@(\Omega,{\cal F})@@@ is, we are now dealing with probabilities on subsets of @@@\R@@@ (representing values @@@X@@@ takes), putting everything in one context of real numbers rather than obscure (or not, but completely general) sample spaces or @@@\sigma@@@-algebras.

We won’t work directly with distributions of RV, but we need to know that they give us the probabilities of all statement one can make about the RV. Instead of working with these measures, we will often work with something more pleasant:

That is, @@@F_X (x) = P_X ((-\infty,x])@@@, so at first, the CDF may seem as a snapshot or a shadow of the distribution of the RV. This is not really true:

What we mean by this is knowing the CDF of @@@X@@@ (which describes the values @@@P_X@@@ assigns to intervals of the form @@@(-\infty,x]@@@) allows to determine @@@P_X(B)@@@ for any @@@B@@@, at least theoretically. This makes life easier, right? Let’s focus on CDFs and their properties.

Here’s a video explaining the concept of the distribution of a RV, and the CDF.

We now list some important properties of CDFs.

The converse is also true.

We now illustrate how to use the CDF to find probability that @@@X@@@ is in any given interval, or any union of disjoint intervals. Let’s get rolling. For ease of notation, we write @@@F@@@ for the CDF of @@@X@@@, omitting the subscript. We assume @@@-\infty<\beta\le \alpha<\infty@@@

• @@@\eqto{P(X \le \alpha)}{F(\alpha)} = 1- P(X>\alpha)\quad \Rightarrow \quad P(X>\alpha) = 1-F(\alpha)@@@.
• @@@\eqto{P(X\le \alpha)}{F(\alpha)} = \eqto{P( X\le \beta)}{F(\beta)} + P( \beta <X \le \alpha)@@@. Therefore

• The CDF gives the probability of @@@X@@@ being less than or equal to @@@\beta@@@. What about the probability of being strictly less than @@@\beta@@@?

Before we answer, some additional notation. Remember that @@@F@@@ is nondecreasing? Then when we look at @@@F(x)@@@ as we increase @@@x@@@ to @@@\beta@@@ , the values are increasing to some limit, known as the left limit of @@@F@@@ at @@@\beta@@@ and denoted by @@@F(\beta-)@@@. This is not necessarily equal to @@@F(\beta)@@@. Why? Because we did not require @@@F@@@ to be continuous but only right-continuous. There may be a jump at @@@\beta@@@: look at the graph in Example Example 8. Well, the event @@@\{X< \beta\}@@@ is the union of the events @@@A_n= \{X \le \beta - 1/n\}@@@ as @@@n@@@ ranges over all @@@n\in\N@@@. Note that @@@A_{n}\subseteq A_{n+1}@@@, and therefore by continuity of probability measures,

In other words, @@@\eqto{P(X\le \beta -\frac 1n)}{F(\beta- \frac 1n)} \underset{n\to \infty} {\nearrow} P(X <\beta)@@@. Therefore

• @@@\eqto{P( X \le \alpha)}{F(\alpha)} =\eqto{ P(X<\beta )}{F(\beta-)} +P(\beta \le X \le \alpha)@@@. Therefore

• By letting @@@\alpha=\beta@@@ in last case, we have

• @@@\eqto{P(\beta < X \le \alpha)}{F(\alpha) - F(\beta)} = P(\beta < X < \alpha) + \eqto{P(X=\alpha)}{F(\alpha)-F(\alpha-)}.@@@ Therefore,

Here is a summary of everything we derived:

You can get other types of intervals from these above:

• @@@[\alpha,\beta)@@@ ? You can start from @@@(\alpha,\beta)@@@ and take its union with @@@[\alpha]@@@. This is a disjoint union, so probabilities add up:

• @@@[\alpha,\beta]@@@ ? We can repeat the process from the last example and add @@@P(X=\beta)@@@. This gives

• @@@(-\infty,\beta)@@@ ? You can start from @@@(-\infty, \beta]@@@ and subtract the probability of @@@X=\beta@@@. This gives you

Why did we give \eqref{eq:atom} a number? Since @@@F@@@ is nondecreasing, the only discontinuities it has are jumps: the limit from the left and right exist and differ (limit from the right equal to the value at the point due to right-continuity). This equation explains the origins of jumps: @@@F@@@ has a jump at @@@\alpha@@@ if and only if @@@P(X=\alpha)>0@@@, in which case we say that @@@X@@@ has an ‘'’atom’’’ at @@@\alpha@@@. We’ll get back to atoms soon.

After all these calculations let’s talk a little about theory (I know it may sound funny). If @@@X@@@ is an RV on one probability space and @@@Y@@@ an RV on another probability space, it does not make sense to compare the two: they are functions on different domains. However, the introduction of the probability distribution and the CDF for each strips the sample space dependence from the random variable, keeping only the numerical values attained and the associated probabilities. This leads to the following (almost trivial definition).

Equality in distribution is an Equivalence relation, an important notion in mathematics. If you know what an equivalence relation is, try to show this. Otherwise, think of an equivalence relation as a notion of similarity or as a “cheap equality”: we only consider equality of certain aspects or when viewed behind some filter. In our case @@@X@@@ and @@@Y@@@ are equal in distribution if their the probability of @@@\{X\in A\}@@@ and of @@@\{Y\in A\}@@@ is the same for all Borel sets @@@A@@@.

Three Distribution Types

CDFs allow to classify distributions of RVs into three types.

Though the three types describe the distribution of the RV rather than the RV itself, we will often refer to a RV as “continuous”, “discrete” or “mixed”, meaning that the underlying probability measure is such that the distribution of the RV is of the described type.

Discrete distributions

Basically, a distribution is discrete if the CDF only increases through jumps.

When @@@X@@@ is discrete, the finite or countable set @@@A@@@ satisfying @@@P(X=\alpha)>0@@@ for all @@@\alpha \in A@@@, is called the ‘'’support’’’ of @@@X@@@ (warning: in advanced math we use the same word for a slightly different object). We also define an associated function, the ‘'’probability mass function (PMF)’’’ of @@@X@@@, denoted by @@@p_X@@@ and given by

Of course, @@@p_X@@@ is a function from @@@\R@@@ to @@@[0,1]@@@ and it is nonzero only the support of @@@X@@@. It is not hard to see that the PMF determines the CDF:

In other words, to recover the CDF from the PMF, just add up all the nonzero values of the PMF at points @@@\alpha@@@ less than or equal to @@@x@@@. If there are none, @@@F_X(x)=0@@@.

The PMF is a very useful and usually easier object to work with than the CDF. Usually, we describe discrete RVs in terms of their PMFs:

Let @@@X@@@ be the number when rolling a standard and fair @@@6@@@-faced die. Then the PMF is pretty obvious: @@@p_X(1) = p_X(2) = \dots = p_X(6) = \frac 16@@@, while @@@p_X (x)=0@@@ for all other @@@x@@@. To recover the CDF, we need to do some summation, right? Here’s an illustration of the CDF. The default view gives you the CDF, but there are two alternate views that give you the PMF: one showing the summation process, and a second showing the PMF isolated from the CDF.

Let’s look at some more interesting examples.

Continuous distributions

Recall that the distribution of a RV @@@X@@@ is continuous when the corresponding CDF is continuous, or, equivalently when @@@P(X=x)=0@@@ for all @@@x@@@.

Let’s go one step further with continuity. Let’s make a defition

Here’s an immediate observation.

Of course, we bother to introduce CDFs with densities because not every continuous CDF has a density. Ever heard of the Cantor function? Take a look at the figure. This function is continuous, but not an integral of anything. It is differentiable on an open set which is dense in @@@[0,1]@@@ and where differentiable, the derivative is zero. All the increase comes from a small but uncountable set, known as the Cantor set. Very roughly roughly, this is a continuous distribution which is “as close as it gets” to being discrete.

Lucky for us, almost all continuous distributions we will consider will have a density.

Before more theory, let’s consider a concrete case.

With this extremely simple example in mind, suppose that @@@X@@@ has CDF @@@F_X@@@ and density @@@f_X@@@. Since @@@F_X@@@ is continuous for every interval @@@I@@@ with left endpoint @@@a@@@ and right endpoint @@@b@@@ (here of course @@@a\le b@@@, and the interval can be open/closed or neither), we have the equality

a Now let’s fix some @@@x_0 \in\R@@@. Assume that @@@f_X@@@ is continuous at @@@x_0@@@. Then the continuity of @@@X@@@ implies @@@P(X=x_0)=0@@@. Yet, taking @@@a=x_0-\epsilon,b=x_0+\epsilon@@@, for some small @@@\epsilon>0@@@, the euqation above gives

because the integral is over an interval so small that its value is essentially @@@f_X(x_0)@@@, the value of the function at the center of the interval, times the length of the interval, plus a smaller “error”. In other words, the probability that @@@X@@@ lies in the interval of length @@@2\epsilon@@@ centered around @@@x_0@@@ is roughly the length of the interval times the density at the center of the interval.

Bottom line(s):

• Since probability of attaining a specific value is zero, we look at intervals instead.
• We use lengths of intervals as reference for computing probabilities in intervals.
• The density at a given point describes the ratio between the probability of being in a small interval around that point and the length of that interval, as the length of the interval tends to zero.

Before we move on, observe that the notion of density is the continuous analog of the PMF, and in fact, some authors will refer to PMF as “density”. To see the analogy, review our definition of density \eqref{eq:primitive} and the the connection between PMF and CDF

Now given an RV, when does it have a density? We will discuss some pretty sophisticated methods of finding out, but at the momemt, let’s stick to basic calculus. From the Fundamental Theorem of Calculus, we learn the following:

Let’s try to digest this. Gimmie some RV @@@X@@@, let’s sat @@@X\sim \mbox{Bern}(0,1)@@@. Remember its CDF we plotted earlier? Well, it is flat, except for the jump at @@@0@@@ and at @@@1@@@. Therefore the derivative exists except at two points, and is equal to zero whereever it exists. The integral of the zero function is simply zero, so oops, this RV does not have a density.

On the other hand, if we look at the RV with CDF given in \eqref{eq:cdf_uni}, then again, its derivative exists everywhere expect for… the same towo points @@@0@@@ and @@@1@@@, and this derivative is given by the function \eqref{eq:density_uni}, which upon simple inspection, does integrate to @@@1@@@. Therefore this RV has a density.

Here’s an example that will show how far we can stretch this notion.

Mixed distributions

The next one is from Mix and Match Jokes.

Q: What do you get when you cross a karate expert with a pig?

A: A porkchop.

Mixed are, in essence, obtained through a similar procedure. Take a discrete CDF and continuous one, combine to get a mixed RV. We’ll get to the mathematical theorem, but first let’s return to our definition of a mixed CDF: not continuous (so has jumps), and not discrere (so not only supported on jumps). This basically gives us

How can you produce such mixed RVs? All you need is one example.

Approximation of one type by another

Suppose @@@X@@@ is a continuous or mixed RV. Then it’s pretty easy to approximate it with a discrete RV. Here’s how.

Recall the floor function, @@@x\to \lfloor x\rfloor@@@ which gives us the largest integer @@@\le x@@@. For example, @@@\lfloor 0.9\rfloor=0@@@, @@@\lfloor 3\rfloor =3@@@, @@@\lfloor -.01\rfloor =-1@@@. Observe that

Now let’s take a RV @@@X@@@. We will come up with a discrete RV close to @@@X@@@. How close? Fix any @@@\epsilon>0@@@. We will find a RV @@@Y_\epsilon@@@ such that @@@0 \le X - Y_\epsilon <\epsilon@@@. All that’s left is to define our approximation:

Then from \eqref{eq:floor} we see that @@@X -\epsilon < Y_\epsilon \le X@@@. Also, @@@Y_\epsilon@@@ is discrete: it is always an integer multiple of @@@\epsilon@@@.

Now we can also approximate any RV with a RV that has a density. We need more theory to make this seem as simple, and we provide the construcion later. In the meantime, we will show that any CDF ca be approximated through one with density.

Pick a CDF @@@F@@@, and fix an @@@\epsilon>0@@@. This, again, will serve as how close the newly minted CDF will be to @@@F@@@. We define

Since @@@F@@@ is nondecreasing, so is @@@F_\epsilon@@@, and since an integral is continuous as a function of the lower and upper limits, @@@F_\epsilon@@@ is continuous. Furthermore,

and since @@@F@@@ is right continuous, @@@\lim_{\epsilon\to 0} F_\epsilon (x) = F(x)@@@ for all @@@x@@@.

Finally, observe that

Therefore, according to our definition, @@@F_\epsilon(x)@@@ has density @@@\frac{F(x-\epsilon) - F(x)}{\epsilon}@@@.

Bottom line: we have found a CDF with density that converges to @@@F@@@. If we repeat the process again for @@@F_\epsilon@@@, we can obtain an approximation with a continuous density. Of course we can repeat again to get continuously differentiable density, etc. But there are simpler ways to get that in one operation. Again, when we learn more it will be much simpler.

• Intro
• Random Variables
  • Definition and First Examples
  • Indicators
  • Operations with RVs
• Distribution (Functions) of RVs
  • Three Distribution Types
  • Discrete distributions
  • Continuous distributions
  • Mixed distributions
  • Approximation of one type by another