Data analysis – Missing value processing

import numpy as np
import pandas as pd
__author__='don't let'Copy the code

Create a tabular two-dimensional array
df=pd.DataFrame({'a': [34.6.20,np.nan,56].'b': ['juejin'.'number'.'one'.'good',np.nan]})Copy the code

The output is as follows:

Determine whether missing values exist in the data and filter non-missing values:

Check whether the two-dimensional array df is missing
print(df.notnull(),'\n')
Check if column A is missing by index
print(df['a'].notnull(),'\n')
Filter column A without missing value array
print(df[df['a'].notnull()])Copy the code

The output is as follows:

       a      b
0   True   True
1   True   True
2   True   True
3  False   True
4   True  False 

0     True
1     True
2     True
3    False
4     True
Name: a, dtype: bool 

      a       b
0  34.0  juejin
1   6.0  number
2  20.0     one
4  56.0     NaNCopy the code

Deletion of missing values

Filtering by notnull Boolean sequence values above is also a way to remove missing values

The deletion of the missing data needs to be handled according to the actual data situation and service situation. Sometimes all the missing data needs to be deleted, sometimes part of the missing data needs to be deleted, and sometimes only the specified missing data needs to be deleted.

Drop missing values: Dropna (Axis)

• The default parameter axis= 0 is used to delete row data, and when axis=1 is used to delete column data (but axis=1 is not usually selected; if it is 1, an entire variable is deleted directly).
• Passing thRESH =n preserves rows with at least n non-nan data

Create a tabular two-dimensional array
df2=pd.DataFrame([[1.2.3], ['juejin',np.nan,np.nan],['a'.'b',np.nan],[np.nan,np.nan,np.nan],['d'.'j'.'h']],
                 columns=list('ABC'))
print(df2,'\n')
Delete all rows with missing values
print(df2.dropna(),'\n')
Delete some rows with missing values and keep at least n non-nan rows (e.g., keep at least one non-nan row)
print(df2.dropna(thresh=1),'\n')
Delete all rows from A column with missing values, same as above Boolean sequence filter
print(df2[df2['A'].notnull()])Copy the code

The output is as follows:

        A    B    C
0       1    2    3
1  juejin  NaN  NaN
2       a    b  NaN
3     NaN  NaN  NaN
4       d    j    h 

   A  B  C
0  1  2  3
4  d  j  h 

        A    B    C
0       1    2    3
1  juejin  NaN  NaN
2       a    b  NaN
4       d    j    h 

        A    B    C
0       1    2    3
1  juejin  NaN  NaN
2       a    b  NaN
4       d    j    hCopy the code

Missing values fill/replace

Fillna (value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs)

• Value parameter: fill value
• Method parameters: pad/ffill → fill with previous data, backfill/bfill → fill with later data

Replace (to_replace=None, value=None, inplace=False, limit=None, regex=False, method=’pad’, axis=None)

• To_replacec parameter: the value to be replaced
• Value: Indicates the replacement value

The following is an example:

import copy
df3=pd.DataFrame([[1.2.3], ['juejin',np.nan,np.nan],['a'.'b',np.nan],['k',np.nan,np.nan],['d'.'j'.'h']],
                 columns=list('ABC'))
df4=copy.deepcopy(df3)
print(df3,'\n')
All missing values are filled with zeros
print(df3.fillna(0),'\n')
#method= 'pad' so that each missing value in column B is filled with the previous value of the missing value
df3['B'].fillna(method='pad',inplace=True)
print(df3,'\n')
# replace with
print(df4,'\n')
df4.replace(np.nan,'juejin',inplace = True)
print('Replace the missing value with juejin\n',df4)Copy the code

        A    B    C
0       1    2    3
1  juejin  NaN  NaN
2       a    b  NaN
3       k  NaN  NaN
4       d    j    h 

        A  B  C
0       1  2  3
1  juejin  0  0
2       a  b  0
3       k  0  0
4       d  j  h 

        A  B    C
0       1  2    3
1  juejin  2  NaN
2       a  b  NaN
3       k  b  NaN
4       d  j    h 

        A    B    C
0       1    2    3
1  juejin  NaN  NaN
2       a    b  NaN
3       k  NaN  NaN
4D j H replace the missing value with Juejin A B C0       1       2       3
1  juejin  juejin  juejin
2       a       b  juejin
3       k  juejin  juejin
4       d       j       hCopy the code

Missing value interpolation

The above mentioned missing value filling, but in the actual data processing process, missing value processing is not to randomly find all data filling, but to carry out interpolation filling for each local missing value.

Several typical missing value interpolation methods are selected here:

• Median/mode/mean interpolation
• Near value interpolation
• Lagrangian interpolation

Median/mode/mean interpolation