This is the 18th day of my participation in the August Genwen Challenge.More challenges in August

Naive Bayes

Conditional probability hypothesis attributes of statistical attributes are independent of each other text classification – spam text filtering, sentiment prediction, recommendation system

The specific process

  1. Give training data
  2. The corresponding classification of data
  • Category probability and conditional probability

Bayesian principle, Bayesian classification and Naive Bayes

Bayes principle – the biggest concept – solves the problem of “reverse probability” in probability theory bayes classifier – Naive Bayes classifier designed on the basis of Bayes principle – the simplest Bayes classifier

Working principle of naive Bayes classification

Discrete data case

Continuous data case

  1. Let’s say they’re all normally distributed
  2. Find the mean and variance – get the probability density function – plug in the values, and figure out the value of the density function at some point
    • Into discrete data distribution, directly also use the formula

Plain Bayesian workflow

  1. Preparation stage

Determine characteristic attributes – divide each characteristic attribute appropriately – classify a part of data manually – form training samples

  1. The stage of training

Generate classifier – Calculate the frequency of occurrence of each category in the training sample and the conditional probability input of each feature attribute partition for each category: feature attribute, training sample output: classifier

  1. Application stage

Input: classifier, new data output: classification result of new data

O IF – the IDF

Creation of the TfidfVectorizer class

  1. Create TfidfVectorizer class
  2. Fit the list – FIT_transform – text matrix
  3. The output
    1. All non-repeating words – get_feature_names()
    2. Corresponding ID – vocabulary_
    3. TFDF value for each word – toarray()

Classify documents

  1. Preparation stage
    1. Document segmentation
      1. English – me
        1. Word_tokenize (text) – participle
        2. Pos_tag () – Marks the parts of speech
      2. Chinese – jieba
    2. Load stop word
    3. TfidfVectorizer class fit_transform fitting tF-IDF feature space features TF = TfidVectorizer(stop_words, max_df) features = tf.fit_transform(train_contents)
  2. Classification stage

    1. MultinomialNB CLF = + + + + + + + + + + + + + + MultionmialNB (alpha = 0.001). The fit (train_features train_labels)

    2. Classifiers are applied to test sets

    3. Calculate accuarcy_score from sklearn import metrics print metrics. Accuracy_score (test_labels, predicted_labels)