Click “like” to see, form a habit, wechat public account search [JackCui-ai] to follow this programmer who loves to send technology dry goods. In this paper, making github.com/Jack-Cheris… Has been included, a line of large factory interview complete test sites, information and my series of articles.

One, foreword

Attitude estimation has been a hot research topic in recent years.

It is based on the picture, capture the motion posture of the human body, such as 2D posture estimation:

Another example is 3D pose estimation:

It looks fun, but what is it supposed to do?

Automatic driving, we should be familiar with, many companies research this direction.

On autopilot, human behavior recognition is used.

The camera can capture and track human movement changes, and judge human movement behaviors according to body movements or change angles, which can be used for behavior analysis demand scenarios such as unmanned vehicles, robots and video surveillance.

And the basis of all of this is the estimation of human posture.

Another example is avatars.

Through reality-driven, virtual images can simulate real body movements, and can be combined with 3D face effects, gesture recognition and other functions to make virtual images more flexible and vivid, which can be used in virtual IP driven, physical-driven games, remote teaching or broadcasting scenes.

Attitude estimation is undoubtedly of great application value.

But today, the algorithm we’re talking about is even cooler than the one I mentioned above!

It was FrankMocap.

Second, the FrankMocap

FrankMocap is a 3D human pose and shape estimation algorithm proposed by Hong Kong Chinese and Facebook AI Research Institute. **

Not only can we estimate the movement of the human body, but also the shape of the body and the movements of the hands can be calculated together.

The algorithm is powerful enough to simultaneously estimate 3D human and hand movements from monocular video, reaching 9.5 FPS on a GeForce RTX 2080 GPU.

The disadvantage is that the speed is a little slow, the amount of calculation is large, and it is difficult to achieve real-time performance.

However, with the rapid development of hardware, this is not a problem, “old Huang Dao method” is very powerful.

FrankMocap algorithm is in these days, just open source, have Python and deep learning foundation friends, not to be missed.

Project Address:

Github.com/facebookres…

The algorithm uses SMPL-X mannequin.

Given a color picture, two network modules are used to predict hand posture and human posture respectively.

Then the hand and body are combined together through the integration module to obtain the final 3D full-body model. The whole process is shown in the figure below:

In the specific implementation of each module, it involves specific mathematical formulas.

This article does not do the extension, interested partners, you can directly read the paper.

Address: arxiv.org/pdf/2008.08…

The paper records the formula of each module in detail.

How does the FrankMocap algorithm play?

Here is a detailed installation tutorial:

Github.com/facebookres…

Detectron2 for hand motion capture and Pytorch3D for pose rendering are needed in addition to the regular third-party libraries.

You are advised to use Anaconda. After configuring the environment, you can start to experience the configuration.

How to use it is simple:

For body pose capture only, you can run the following command:

# using a machine with a monitor to show output on screen
python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
# screenless mode (e.g., a remote server)
xvfb-run -a python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
Copy the code

Operating effect (screenshot example) :

For hand pose capture only, run the following command:

# using a machine with a monitor to show outputs on screen
python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output
# screenless mode  (e.g., a remote server)
xvfb-run -a python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output
Copy the code

Operating effect (screenshot example) :

For whole-body posture estimation, use the following instructions:

# using a machine with a monitor to show outputs on screen
python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
# screenless mode  (e.g., a remote server)
python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
Copy the code

Operating effect (screenshot example) :

Interested partners, try it!

Third, summary

3D human body motion capture, can do a lot of interesting things.

When you look at this technology, the first thing you think about is, what fun application can you make?

I’m Jack Cui, a programmer who likes to do technical stuff, and we’ll see you next time

This article will be updated continuously. You can find it on our wechat official account by searching [JackCui-ai]. GitHub github.com/Jack-Cheris… Welcome Star.