Artificial Intelligence. Machine Learning. Chapter 3 naive Bayes Algorithms.

I’m sure that if you read this article, you know or have learned about Bayes, an application of Bayesian ‘theorem. English mathematician ** Thomas Bayes *** first proposed the theorem in a paper published in 1763. Pushu Bayes is an application of this theorem.

Naive Bayes is quite different from other statistical inference methods. It is based on subjective judgment, meaning that you can estimate a value without objective evidence and then revise it based on actual results. Just because of its strong subjectivity, it has been criticized by many statisticians.

Naive Bayes requires a lot of computation, so for a long time in history, it was not widely used. It was only with the birth of the computer that it gained real importance. It has been found that many statistics cannot be judged objectively in advance, and the emergence of large data sets in the Internet age, coupled with high-speed computing power, has made it easy to verify these statistics and created conditions for the application of Bayesian inference, which is becoming increasingly powerful. Well, the author is limited in his ability to give an accurate concept of naive Bayes despite having consulted a large amount of literature. I can only quote the concept of naive Bayes given by Daniel here.

Concept: Naive Bayes is a classification method based on Bayes’ theorem and the assumption that feature conditions are independent. For a given training data set, the joint probability distribution of input/output is first learned based on the assumption of feature condition independence. Then based on this model, for a given input x x x, bayes’ theorem is used to find the output y y y with the highest posteriori probability.

[Note] The above is the concept of Bayesian in Statistical Learning Methods by Li Hang.

After reading the concepts above, I’m sure many readers may be a little confused about how naive Bayes should be used and how to classify them. In order to visually describe how naive Bayes works in depth, as in the previous chapter, we will introduce naive Bayes through an example.

The probability of male and female students wearing military uniform in a university is 2 to 1. Question: If you meet a person wearing military uniform at school, what is the probability of male or female? Ok, so let’s talk about prior probability, conditional probability, total probability formula and posterior probability.

Conditional probability is the core of the bayes’ theorem, the so-called “Conditional probability” (Conditional aim-listed probability), is refers to under the condition of the event B B B occurs, the probability of events A aa, with P (A ∣ B) P (A | B) P (A ∣ B).

FIG. 1 Venn diagram

According to Venn’s diagram, it can be clearly seen that in the case of event B B B, the probability of occurrence of event A A A is P(A∩B) P(A∩B) P(B).

As a result,

And by the same token,



So,



即



That’s the formula for conditional probability.

By deforming the conditional probability formula, the following form can be obtained:



We refer to P(A) P(A) P(A) P(A) as Prior probability, that is, our judgment of the probability of A, A and A before the occurrence of B events. P (A ∣ B) P (A | B) P (A ∣ B) referred to as the “Posterior probability” (Posterior aim-listed probability), namely in B B B after the incident, we review of aa A probability of occurrence. P/P (B ∣ A) (B) P (B | A)/P (B) P (B ∣)/P (B) known as the “likelihood function (Likelyhood),” this is an adjustment factor, makes the forecast probability closer to the true probability. Therefore, conditional probability can be understood as the following formula:

Posterior probability = prior probability x adjustment factor

We estimate a prior probability, and then add the results of the experiment to see whether the experiment enhances or weakens the prior probability to get a more realistic posterior probability.

Over here, if we have P of B given A divided by P of B gt; 1 P(B|A)/P(B)> One P of B given A divided by P of B is greater than one, which means the prior probabilities are increased, the probability of events A, A, A is greater; If the probability function is equal to 1, that means that events B, B, B are not helpful in determining the probability of events A, A, A; If the probability function is less than 1, that means that the prior probability is weakened, that event A, A, A is less likely.

Well, a more formal expression is as follows.

Conditional probabilityThis is the probability of the event X= X, X= X, X= X under the condition that the event Y= Y, Y= Y, Y= Y has already occurred. Conditional probability can be expressed as P (X = X ∣ Y = Y), P (X = X | Y = Y), P (X = X ∣ Y = Y) :. And the conditional probability calculation formula is:



The P (X = X ∣ Y = Y), P (X = X | Y = Y), P (X = X ∣ Y = Y) is the joint probability, the probability of two events occur together. And P(Y= Y) P(Y= Y) P(Y= Y) and P(X= X) P(X= X) P(X= X) are prior probabilities.

Let’s use examples to illustrate this. “The probability of male students wearing military uniforms in a certain school is 1/2, 1/2, 1/2”, that is to say, “under the premise of male students, the probability of wearing military uniforms is 1/2, 1/2, 1/2”, which is a conditional probability. P (X = 1 X ∣ Y = Y b o Y) = 1/2 P (X = x_1 | Y = y_ {boy}) = 1/2 P (X = x1 ∣ Y = yboy) = 1/2

Similarly “girls in uniform probability for the two thirds two-thirds two-thirds” as the conditional probability, namely, P (X = 1 X ∣ Y = Y g I r l) = 1/2 P (X = x_1 | Y = y_ {girl}) = 1/2 P (X = x1 ∣ Y = ygirl) = 1/2

A posterior probability is the probability that an event X= X, X= X, X= X has already occurred, and that the event occurred because of the event Y= Y, Y= Y, Y= Y. That is, in the above example, “what is the probability that a person is male or female, given that the person wears slippers?” Formal is: posterior probability P (Y = Y o Y b ∣ X = 1 P (Y = X y_ {boy} | X = x_1 P (Y = yboy ∣ X = x1

The calculation of posterior probability should be based on prior probability. A posteriori probability can be calculated by using a prior probability and likelihood function through The Bayesian formula.

Bayes formulaAs follows:



The P (Y = Y b o Y ∣ X = 1 P (Y = X y_ {boy} | X = x_1 P (Y = yboy ∣ X = x1 is demanding a posteriori probability, P (X = 1 X ∣ Y = Y b o Y P (X = x_1 | Y = y_ {boy} P (X = x1 ∣ Y = yboy conditional probability, P (Y = Y b o Y P (Y = y_ {boy} P (Y = yboy prior probability.

The bayesian algorithm uses the above information to solve the posterior probability and classifies according to the posterior probability value.

Using the above example to understand, the posterior probability is:

In other words, given the fact that a person is wearing a military uniform, the probability that the student is a boy is 3/5, 3/5, 3/5, and the probability that the student is a girl is 2/5, 2/5, 2/5. If the question is “decide whether the student is a boy or a girl”, the question is a classification problem. And since the posteriori probability calculated by Bayes’ formula is that there’s a higher probability of boys and then there’s a higher probability of girls

P (Y = Y b o Y = X one > P (Y = Y g I r l ∣ X = X 1 P (Y = y_ {boy} | X = x_1 & gt; P (Y = y_ {girl} | X = P (Y = yboy x_1 ∣ X = x1 > P (Y = ygirl ∣ X = x1

Then we can classify it as male (without actually solving the denominator when using naive Bayes for classification).

So far, we have used examples to illustrate the basic steps and simple principles of using naive Bayes for classification. We’re going to talk a little bit more about naive Bayes’ principle, and this is just a summary of what we’ve seen before, so don’t be intimidated by a lot of formulas. Okay, so let’s move on.

Reference: www.ruanyifeng.com/blog/2011/0…

When maximum likelihood estimation is used, the probability value to be estimated may be 0, which will affect the calculation result of posterior probability and result in classification deviation. In order to solve this problem, Bayesian estimation is adopted. The specific Bayes of conditional probability are shown as follows.



Where S, j, S_j, Sj are characteristics, and the number of values of X, j X_j, and Xj is h, H, h. In the formula λ > = 0 \lambda > =0 λ>=0, which is equivalent to giving a positive number λ &gt on the frequency of each value of the random variable; 0 \lambda > 0 λ>0, when λ=0 \lambda =0, is the maximum likelihood estimator. When lambda =1, its answer is Laplace smoothing. For any l=1,2,.. S j l=1,2… , S_j l = 1, 2,… ,Sj, k=1,2.. k, k=1,2… , K K = 1, 2,… , K

P (X j = a J L ∣ Y = C k) > 0 P(X_j=a_{jl}|Y=C_k)> 0 P (Xj = ajl ∣ Y = Ck) > 0

∑ L = 1 S j P (X j = a J L Y = C k) > 0 \sum_{l=1}^{S_j}P(X_j=a_{jl}|Y=C_k)> ∑ 0 l = 1 SJP (Xj = ajl ∣ Y = Ck) > 0

Again, the Bayesian estimate of prior probability is zero

The main principle of naive Bayes algorithm has been summarized, and the advantages and disadvantages of naive Bayes algorithm are summarized here. The main advantages of naive Bayes are:

1) Naive Bayes model originated from classical mathematical theory and has stable classification efficiency. 2) Good performance on small-scale data, able to handle multi-classification tasks, suitable for incremental training, especially when the amount of data exceeds the memory, we can conduct incremental training in batches. 3) It is not sensitive to missing data and the algorithm is relatively simple, which is often used for text classification.

Main disadvantages of naive Bayes:

1) Theoretically, naive Bayes model has the minimum error rate compared with other classification methods. However, this is not always the case in fact, because when the naive Bayes model gives the output category, it assumes that the attributes are independent from each other, which is often not valid in practical application. When the number of attributes is large or the correlation between attributes is large, the classification effect is not good. Naive Bayes has the best performance when the attribute correlation is small. For this, there are semi-naive Bayes and other algorithms that improve modestly by taking into account partial correlations. 2) It is necessary to know the prior probability, and the prior probability depends on the hypothesis in most cases. There can be many models of the hypothesis, so the prediction effect will be poor in some cases due to the prior model of the hypothesis. 3) Because we determine the posteriori probability through prior and data, there is a certain error rate in classification decisions. 4) Very sensitive to the expression of input data.

Reference: [1] Statistical Learning Methods, li Hang.

Video: www.youtube.com/watch?v=DNv…