This is the 8th day of my participation in the First Challenge 2022. For details: First Challenge 2022.

Introduction to the

On April 15, 1912, the supposedly unsinkable Titanic sank due to a collision with an iceberg. Without adequate rescue equipment, 1,502 of the 2,224 passengers died. The accident happened, but can we detect any patterns in the historical data from the Titanic? In this article, you will learn how to use Pandas for data analysis.

Titanic passenger data

We have downloaded some Titanic passenger data from Kaggle, which includes the following fields:

The downloaded file is a CSV file. Let’s see how pandas can be used for data analysis.

Analyze the data using PANDAS

Importing dependency packages

This article mainly uses PANDAS and matplotlib, so the following general Settings need to be set up first:

from numpy.random import randn
import numpy as np
np.random.seed(123)
import os
import matplotlib.pyplot as plt
import pandas as pd
plt.rc('figure', figsize=(10, 6))
np.set_printoptions(precision=4)
pd.options.display.max_rows = 20
Copy the code

Read and analyze data

Pandas provides a read_csv method to read and convert CSV data to a DataFrame:

path = '.. /data/titanic.csv' df = pd.read_csv(path) dfCopy the code

Let’s look at the data we read:

418 rows × 11 columns

Basic statistics can be viewed by calling the Describe method of DF:

If you want to see the port where the passenger is logged in, you can choose:

df['Embarked'][:10]
Copy the code

0    Q
1    S
2    Q
3    S
4    S
5    S
6    Q
7    S
8    C
9    S
Name: Embarked, dtype: object
Copy the code

Value_counts counts them:

embark_counts=df['Embarked'].value_counts()
embark_counts[:10]
Copy the code

S    270
C    102
Q     46
Name: Embarked, dtype: int64
Copy the code

As can be seen from the results, 270 passengers have logged in from PORT S, 102 from port C and 46 from port Q.

Similarly, we can count the age information:

age_counts=df['Age'].value_counts()
age_counts.head(10)
Copy the code

The top 10 ages are as follows:

24.0    17
21.0    17
22.0    16
30.0    15
18.0    13
27.0    12
26.0    12
25.0    11
23.0    11
29.0    10
Name: Age, dtype: int64
Copy the code

Calculate the average age:

df['Age'].mean()
Copy the code

30.272590361445783
Copy the code

In fact, some data are age-free, and we can fill them in with averages:

clean_age1 = df['Age'].fillna(df['Age'].mean())
clean_age1.value_counts()
Copy the code

You can see that the average is 30.27 and the number is 86.

30.27259 86 24.00000 17 21.00000 17 22.00000 16 30.00000 15 18.00000 13 26.00000 12 27.00000 12 25.00000 11 23.00000 11 . 36.50000 1 40.50000 1 11.50000 1 34.00000 1 15.00000 1 7.00000 1 60.50000 1 26.50000 1 76.00000 1 34.50000 1 Name: Age, Length: 80, dtype: int64Copy the code

It may not be a good idea to use averages as ages. Another option is to discard averages:

clean_age2=df['Age'].dropna()
clean_age2
age_counts = clean_age2.value_counts()
ageset=age_counts.head(10)
ageset
Copy the code

24.0    17
21.0    17
22.0    16
30.0    15
18.0    13
27.0    12
26.0    12
25.0    11
23.0    11
29.0    10
Name: Age, dtype: int64
Copy the code

Graphical representation and matrix transformation

Graphics are very helpful for data analysis. We use bar charts to represent the ages of the top 10 obtained above:

import seaborn as sns
sns.barplot(x=ageset.index, y=ageset.values)
Copy the code

Now let’s do a complex matrix transformation. Let’s filter out the data where age and sex are null:

cframe=df[df.Age.notnull() & df.Sex.notnull()]
cframe
Copy the code

332 rows × 11 columns

Next use groupby to group age and sex:

by_sex_age = cframe.groupby(['Age', 'Sex'])
by_sex_age.size()
Copy the code

Age Sex 0.17 female 1 0.33 male 1 0.75 male 1 0.83 male 1 0.92 female 1 1.00 female 3 2.00 female 1 male 1 3.00 female 1 5.00 male 1.. 60.00 Female 3 60.50 male 1 61.00 male 2 62.00 male 1 63.00 Female 1 male 1 64.00 Female 2 male 1 67.00 male 1 76.00 female 1 Length: 115, dtype: int64Copy the code

Use unstack to convert Sex column data to rows:

79 rows × 2 columns

We add the number of people with the same age and use argsort to get the sorted index:

indexer = agg_counts.sum(1).argsort()
indexer.tail(10)
Copy the code

Age
58.0    37
59.0    31
60.0    29
60.5    32
61.0    34
62.0    22
63.0    38
64.0    27
67.0    26
76.0    30
dtype: int64
Copy the code

Extract the last 10, the largest 10, from agG_COUNTS:

count_subset = agg_counts.take(indexer.tail(10))
count_subset=count_subset.tail(10)
count_subset
Copy the code

The above operation can be simplified to the following code:

agg_counts.sum(1).nlargest(10)
Copy the code

Age
21.0    17.0
24.0    17.0
22.0    16.0
30.0    15.0
18.0    13.0
26.0    12.0
27.0    12.0
23.0    11.0
25.0    11.0
29.0    10.0
dtype: float64
Copy the code

Count_subset for stack operation, convenient for the following drawing:

stack_subset = count_subset.stack()
stack_subset
Copy the code

Age   Sex   
29.0  female     5.0
      male       5.0
25.0  female     1.0
      male      10.0
23.0  female     5.0
      male       6.0
26.0  female     4.0
      male       8.0
27.0  female     4.0
      male       8.0
18.0  female     7.0
      male       6.0
30.0  female     6.0
      male       9.0
22.0  female    10.0
      male       6.0
21.0  female     3.0
      male      14.0
24.0  female     5.0
      male      12.0
dtype: float64
Copy the code

stack_subset.name = 'total'
stack_subset = stack_subset.reset_index()
stack_subset
Copy the code

The diagram is as follows:

sns.barplot(x='total', y='Age', hue='Sex',  data=stack_subset)
Copy the code

Examples for this article can be found at: github.com/ddean2009/l…

This article is available at www.flydean.com/01-pandas-t…

The most popular interpretation, the most profound dry goods, the most concise tutorial, many you don’t

Welcome to pay attention to my public number: “procedures those things”, understand technology, more understand you!