1. Discrete systems and their probability behaviours

1.1. Basic Defination

• A discrete random variable(RV) $X$: take value in a finite set $x$
• The size of the set $x$: $\left|x\right|$ is the range of $X$
• The probability distribution of $X$ is denoted by a function: $P(·):x\rightarrow [0,1]$
• $\forall x\in X$, $P_X(x)$ denoteds the probability that $X$ takes on a specific value $x\in X$
• The Normalization condition: $\sum{}_{x \in X}P_X(x)=1$

<aside> 💡

Shorthand notion: $P_X=P_X(X=x)=P_X(x)$

</aside>

If $P$ is a distribution and $X$ is a random variable, we will write $X\sim P$ to indicate that the distribution of $X$ is $P$

Consider $X$, $Y$ are both RVs, sometimes we will write $X\sim Y$, to indicate that $X$, $Y$ have the same distribution.

<aside> 💡

Example:

Let the finite set $X=\{1,2,3,4,5,6\}$ correspond to the faces of a six-sides die.

If the die is fair, then $\forall x \in X,~P_X(x)=\frac{1}{6}$ ($P_X=\frac{1}{6}$)

The size of range of $X$ is $\left|x\right|=6$

</aside>

2. Joint and Marginal probability distributions

• The joint distribution (联合分布) of $X$, $Y$ is denoted by function: $P_{XY}(·,·):x\times y\rightarrow [0,1]$

• The marginal distribution (边缘分布/ 边际分布) of $X$: $P_X(x)=\sum{}{y\in Y}P{XY}(x,y)$

• Conditional Probability:

  • Definition: the probability that $X$ takes on the value $x$, conditional on the event that $Y$ takes on the value $y$.
  • Bayes’ theorem (贝叶斯定理): relates the conditional probability to the joint prob.
    • $P_{X|Y}(x|y)=\frac{P_{XY}(x,y)}{P_Y(y)}=P_X(x)\cdot \frac{P_{Y|X}(y|x)}{P_Y(y)}$, $P_Y(y)>0$
    • Ps: $P_{X|Y}\cdot P_Y=P_{Y|X}\cdot P_X$
    • Shorthand: $P_{X|Y}=P(X=x|Y=y)=P_{X|Y}(x|y)$

  <aside> 💡

  Example

  Let $Y\in y=\{"fait","unfair"\}$ refers to the choice of either fair die or an unfair die, chosen with equal probability:

  $P_Y(fair)=P_Y(unfair)=\frac{1}{2}$

  If $X$ denotes the fair or unfair, where the unfair die always rolls “6”, that is, $x=\{1,2,3,4,5,6\}$ with $P_X(6)=1,~P_X(x)=0(\forall x=1,2,3,4,5)$;

  For the fair die, $P_X(x)=\frac{1}{6},~\forall x=1,2,3,4,5,6.$

  • $P_{X|Y}(6|unfair)=1~~~~P_{X|Y}(x|unfair)=0(\forall x\neq 6)$
  • $P_{Y|X}(unfair|6)=\frac{P_{XY}(x,y)}{P_X(x)}=\frac{\frac{1}{2}\cdot 1}{\frac{1}{2}\cdot\frac{1}{6}+\frac{1}{2}}=\frac{6}{7}$
  • $P_{Y|X}(unfair|x)=\frac{0}{0}$ </aside>

• Correlation function

  • $X$, $Y$ are independent, if the joint probability of events $x$, $y$ is the same as the product of their marginal distribution.
  • $P_{XY}=P_X(x)\cdot P_Y(y)$

3. Correlation functions of multiple RVs