60 lines of code to crawl zhihugod reply

A previous article crawler climbed the god reply on zhihu, has been laughing party ~ published, caused a warm response. Many friends thought it was amazing, and privately asked Johnny how he did it. Some friends still didn’t believe it. In fact, it is very simple to get the reply of zhihugod. In this article, we will reveal the principle behind it.

What are the characteristics of zhihu god’s reply? So let’s take a look.

Do you see any patterns? Short and pithy? Agree with a lot? So we just have to crawl the answers that are more in agreement and less in words. Simple two steps can be achieved, the first step to crawl zhihu answers, the second part of the screening of answers. Isn’t it easy?

Climb to know hu answer

The first step is to get answers from Zhihu. There are so many answers on Zhihu that it will take a long time to crawl all the answers at once. We can choose several topics and crawl the content of these topics.

The following function is used to crawl the content of a given topic

def get_answers_by_page(topic_id, page_no):
    offset = page_no * 10
    url = <topic_url> # topic_URL is the url for this topic
    headers = {
        "User-Agent": "Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
    }
    r = requests.get(url, verify=False, headers=headers)
    content = r.content.decode("utf-8")
    data = json.loads(content)
    is_end = data["paging"] ["is_end"]
    items = data["data"]
    client = pymongo.MongoClient()
    db = client["zhihu"]
    if len(items) > 0:
        db.answers.insert_many(items)
        db.saved_topics.insert({"topic_id": topic_id, "page_no": page_no})
    return is_end

Copy the code

The get_answers_by_page function takes two arguments, the first being the id of the topic, and the second indicating the page to climb.

There are a few fields to be aware of in the climb down, highlighted in yellow in the figure below

These fields have the following meanings:

Question. Title – The title of the question
Content – The content of the answer
Voteup_count – Number of endorsements

These fields will be used in the next step of filtering responses.

Screening of answer

After climbing the data, let’s sift through the results.

We filtered the answers using the Aggregation Pipeline in MongoDB (see the Aggregation Pipeline Quick Reference for the MongoDB Aggregation Pipeline usage)

client = pymongo.MongoClient()
db = client["zhihu"]
items = db.answers.aggregate([
    {"$match": {"target.type": "answer"}},
    {"$match": {"target.voteup_count": {"$gte": 1000}}},
    {"$addFields": {"answer_len": {"$strLenCP": "$target.content"}}},
    {"$match": {"answer_len": {"$lte": 50}}},])Copy the code

The code above filters all responses that agree with more than 1,000 words and fewer than 50 words, and the result is a short, pithy-like response.

The above is the core code, the complete code has been uploaded to Github, you can reply “Zhihu God reply” in the background of [Python and Data Analysis] public account to obtain the address.

God replied

So we’re done with the code, so let’s run it. It happened to be programmer’s Day yesterday, so we sifted through some of the most amazing replies about programmers. The result is 75 funny jokes 😂

Author: Xia Ziyao

Link: What are the most common “lies” that code farmers tell?

Source: Zhihu

Q: What are the most common “lies” that code farmers tell? A: //TODO

Author: Anonymous user

Link: What’s it like to be green for 365 days on GitHub?

Source: Zhihu

Q: What’s it like to be green for 365 days on GitHub? A: I kept it green for more than 200 days, but I neglected my girlfriend and kept it green until now.

Author: Rebel

Link: How to refute the idea that “programmers are useless without computers”?

Source: Zhihu

Q: How to refute the idea that programmers are useless without computers? A: No, no, no. Many programmers are useless in front of computers.

Author: XLZD

Link: What if one day everyone spoke in a computer language?

Source: Zhihu

Q: What would happen if one day everyone spoke in a computer language? A: Hello, world. �d}� R�0:� V? .

Author: Zhihu user

Link: suddenly want to open a programmer theme restaurant, the name of the programmer’s dish, dish name is called the keywords in various languages, you show me, have a future?

Source: Zhihu

Q: SUDDENLY want to open a programmer theme restaurant, the name is called programmers dishes, dishes called keywords in various languages, please give me some advice, have a future? A: A big Hello World dish called “Braised Product Manager” is sure to fill the room

Author: Anonymous user

Link: How to translate the most basic programming term “bug”?

Source: Zhihu

Q: How to translate the most basic programming term “bug”? A: Crap, your program has screwed up again.

Author: neksusk

Link: What’s the fun of programming?

Source: Zhihu

Q: What’s fun about programming? A: Man’s sense of achievement comes from two things, creation and destruction.

Author: Michael Lingwei

Link: How to refute the idea that “programmers are useless without computers”?

Source: Zhihu

Q: How to refute the idea that programmers are useless without computers? A: Honestly, if you can talk to A woman like this, do you want to fuck her?

By Amb BAI

Link: As a programmer, what math pitfalls do you suffer when programming?

Source: Zhihu

Q: As a programmer, what math problems do you suffer from? A: An “apparently” pushed me all afternoon while reading the paper

Author: Zhihu user

Link: What do rich programmers have on their devices?

Source: Zhihu

Q: What are the devices of rich programmers? A: Girlfriend…

Author: Rabbit ii

Link: Who to worship for bug free code?

Source: Zhihu

Q: Which god do you worship to keep your code bug free? A: Worship Yongzheng, the eight elder brothers.

Author: Zhihu user

Is IT the only way for poor Chinese children to rise to the middle class?

Source: Zhihu

Q: Is IT the only way for poor Chinese children to rise to the middle class by going to a good university and studying IT? A: Yes, there are four ways to write code and finance in the code circle

Author: Cascade

Link: Why do programmers carry computer bags everywhere they go, even if there’s no computer in them?

Source: Zhihu

Q: Why do programmers like to carry computer bags wherever they go, even if there is no computer in them? A: Because they don’t have any other bags.

Author: SErHo

Talk is cheap. Show me the code.

Source: Zhihu

Q: Talk is cheap. Show me the code. A: Cut the bullshit and put the code in.

Author: An0th3r

Why are programmers’ girlfriends and wives so much more attractive than their husbands? Or is it that programmers are already a quality stock in the dating market?

Source: Zhihu

Q: Why do programmers’ girlfriends and wives tend to be much more attractive than their husbands? Or is it that programmers are already a quality stock in the dating market? A: programmer girl friend appearance level is high, I am of, because casually ask 10 programmer his girl friend is who, have 9 answer is new wall knot clothes

Author: fullsail

Link: Why do some people prefer to buy mechanical keyboards instead of applying facial masks?

Source: Zhihu

Q: Why do some people prefer to buy mechanical keyboards instead of applying facial masks to themselves? A: I don’t live with my face. My hard-earned cash. I can spend it any way I want.

Author: Su Yu

Link: programmer couple wedding ring engraved what word good?

Source: Zhihu

Q: What word is good for programmer couple’s wedding ring? A: 0 error 0 warning

By Surianne

Link: Do IT engineers feel uncomfortable being called “code farmers”?

Source: Zhihu

Q: Do IT engineers feel uncomfortable when they are called “code farmers”? A: We’re still human, and our products and designs are dogs…

Author: Bai Rubing

Link: Why would a salesman (30) invite me, a programmer (24), to a nearby Starbucks?

Source: Zhihu

Q: Why did a salesman (30 years old) invite me, a programmer (24 years old), to a nearby Starbucks? A: Based on my years of experience, he should have A great idea and then only need programmers to implement it

Author: With pleasure

Link: How to find a girlfriend who likes programmers?

Source: Zhihu

Q: How can I find a girlfriend who likes programmers? A: See fate, zhihu so many users, you pay attention to me is fate.

Author: Zhihu user

How does a programmer’s girlfriend celebrate a programmer’s boyfriend’s birthday?

Source: Zhihu

Q: How does a programmer’s girlfriend celebrate a programmer’s boyfriend’s birthday? A: Tell him the interface is ready.

Author: Programmer uncle

Link: How did you find a girlfriend while working as a programmer?

Source: Zhihu

Q: How did you find a girlfriend while working as a programmer? A: The subject as A programmer for so long, but also like girls has been commendable.

Author: some small super

Link: What does a programmer need to prepare for a career in grilling, and what are the advantages and disadvantages?

Source: Zhihu

Q: What does a programmer need to do to prepare for a career in grilling? What are the advantages and disadvantages? A: You see, you don’t even know the advantages and disadvantages of making your own barbecue, so you still need A product manager.

By Jack Ma

Link: What makes programmers tick?

Source: Zhihu

Q: What makes programmers tick? A: I walked by his computer and said, “Oh, you’re writing A bug again!”

Author: Yao Dong

Link: one of my teachers said that Java is for large software and C# is for small and medium software. Is it true?

Source: Zhihu

Q: one of my teachers said that Java is for large software and C# is for small and medium software. Is it true? A: Java has A talent for turning small and medium software into large ones. 27

Why are programmers paid so much in 2014? Source: Zhihu

Q: Why are programmers paid so much in 2014? A: The hourly wage is not very high

Author: Liao Xuefeng

Are most programmers complaining about low pay?

Source: Zhihu

Q: Are most programmers complaining about low pay? A: Who, who is complaining about the high salary?

By Ze Ran

Link: What if a bachelor app solves a technical problem and has no girl to show off or brag about?

Source: Zhihu

Q: What if a bachelor app solves a technical problem and has no girl to show off or brag about? A: Now you see why so many programmers write tech blogs.

Author: Zhihu user

Link: Do Chinese programmers prefer jackets, jeans and sneakers? If so, why the trend?

Source: Zhihu

Q: Do Chinese programmers prefer “jackets + jeans + sneakers”? If so, why the trend? A: Do you want to show the program?

Author: Anonymous user

Link: As an IT practitioner, what tools do you feel have greatly increased your productivity?

Source: Zhihu

Q: What tools do you feel have greatly increased your productivity as an IT practitioner? A: single

Akik: Warm wine

Link: Why do I think programmers seem mostly inarticulate?

Source: Zhihu

Q: Why do I think programmers seem mostly inarticulate? A: Think of it as our low eq, so that you are happy and we are happy.

Author: VCZH

In China, the oldest programmer is only about 40 years old. What can Chinese programmers do in the future?

Source: Zhihu

Q: In China, the oldest programmer is only about 40 years old. What can Chinese programmers do in the future? A: It’s the same principle as why no one born in the 1990s lived past 30

By Ze Ran

Link: How to reply to a text from a programmer: “Hello world”?

Source: Zhihu

Q: How do I reply to a text message from a programmer: “Hello world”? A: hello nerd.

Author: Li Jigang

Link: How to tell IT guy likes a girl?

Source: Zhihu

Q: How can you tell that an IT guy likes a girl? A: When he goes all out of his way to get close to you with the habit of being silent that he has already developed

Author: Rebel

Link: Why Shouldn’t programmers be able to fix computers?

Source: Zhihu

Q: Why shouldn’t programmers be able to fix computers? A: Does Fan Bingbing need to be able to fix the TV?

Author: Anonymous user

Link: my colleague said that his C++ level is the first in China, how to make him realize that he is not so good?

Source: Zhihu

Q: my colleague said that he was the best C++ player in China. How did he realize that he was not so good? A: to be honest, I’m not pretending to be A pusher either: my C++ level ranks the 0 in China.

Author: Liu Xiaotou

Link: Why do iPhone ICONS shake when deleting apps?

Source: Zhihu

Q: Why do all ICONS of iPhone delete apps have to be shaken? A: The third-party software is scary, and the software built in the system is dither

Author: AcFun

Link: Revolver loaded with one bullet, shot in the head reward 100,000 yuan, two shots 100 million, three shots 200 million, four shots 400 million, five shots 1.6 billion, is it worth it?

Source: Zhihu

Q: Revolver is loaded with one bullet, one shot at your head is worth 100,000 yuan, two shots are worth 100 million yuan, three shots are worth 200 million yuan, four shots are worth 400 million yuan, five shots are worth 1.6 million yuan. A: As long as you don’t hit the jugular, I tell you, I can hit our A listing ！！！！

Author: Zhang Xiaojian

Link: At the rate that iPhone processors are doubling every year, will they soon catch up with or even surpass desktop processors?

Source: Zhihu

Q: At the current rate of processor performance doubling every year, will iPhone processors soon catch up with and even surpass desktop processors? A: When I was young, I always thought that in two years I would be as old as my brother who was two years older than me.

Author: Xie Lei

Link: What is the least benefit of Zhihu for you?

Source: Zhihu

Q: What is the least benefit of Zhihu for you? A: Killing time without feeling guilty.

By Derek Chan

Link: What anti-human technological inventions or designs?

Source: Zhihu

Q: What are the anti-human technological inventions or designs? A: The computer is not connected to the Internet. After the diagnosis, it reminds me to connect to the Internet

Author: Wang Wei

Link: Why don’t designers want to be called Artists?

Source: Zhihu

Q: Why don’t designers want to be called artists? A: As long as the salary is high, you can call me auntie.

Author: Mr. Robot

Link: Why do people think netease cloud music is the conscience of the industry?

Source: Zhihu

Q: Why do people think netease cloud music is the conscience of the industry? A: One day it suddenly sent me A message that the lyrics I wanted were found

Author: Zhihu user

Link: Why aren’t there any drone self-destruct weapons? Have terrorists used it?

Source: Zhihu

Q: Why are there no drone self-destruct weapons? Have terrorists used it? A: You mean missiles?

Author: Anya

Link: If my thoughts are mine, then why can’t I control my negative emotions sometimes?

Source: Zhihu

Q: Since my thoughts are mine, why can’t I control my negative emotions sometimes? A: The operating system will not allow users to access, modify, or delete core system files, because this will damage the system and cause abnormal operation.

Author: isotone

Link: Although Lu Xun is very cattle, but in this world’s top ten writers is not the number?

Source: Zhihu

Q: Lu Xun is great, but is he just one of the world’s top ten writers? A: Why should writers pay for the ranks of illiterates?

Author: Miss a Miracle

Link: Which technologies have reached a bottleneck that hasn’t seen a major breakthrough in a long time?

Source: Zhihu

Q: What technologies have reached the bottleneck and haven’t had a major breakthrough in a long time? A: the boiling water

Author: VCZH

Link: how to view the preference of some people to download software to the official website?

Source: Zhihu

Q: How do you view the preference of some people to download software to the official website? A: Have you ever won baidu Family bucket?

Author: Zhihu user

Link: Why do so many people buy laptops to play games instead of more powerful desktops?

Source: Zhihu

Q: Why do many people buy laptops to play games instead of more powerful desktops? A: Because I can’t afford A house…

Author: DillyLiu

Link: How much did listening to headphones for the first time shock you?

Source: Zhihu

Q: How much did listening to headphones for the first time shock you? A: They don’t shock you the first time you listen to them, but when you switch back to regular headphones, they do

Author: Lin Canbin

Link: Is Chrome really a power hog?

Source: Zhihu

Q: Is Chrome really a battery hog? A: No. I’m using Chrome right now. As long as I’ve been using it, the battery of my laptop is still 50%

Author: Wang Wenhan

Link: What’s the experience like with Windows on a MacBook?

Source: Zhihu

Q: What is the experience of using Windows on a MacBook? A: It’s like suddenly you have A soft underbelly and you lose your armor.

Author: suchsu

Link: What would it be like to have an Apple product for all relevant products in your home?

Source: Zhihu

Q: What is it like to use Apple products for all related products in your home? A: There was A phone call and the whole family rang

Author: Mayonnaise destroys parking lots

Link: Why didn’t you buy the iPhone X?

Source: Zhihu

Q: Why didn’t you buy the iPhone X? A: The contradiction between the growing need for A better life and the reality of poverty

Author: Zhang Xin

Link: Why would someone pay thousands of yuan for an iPhone, but not tens of yuan for genuine iPhone software and games?

Source: Zhihu

Q: Why are some people willing to pay thousands of yuan for iPhone, but not tens of yuan for legitimate iPhone software and games? A: Because they can’t download the iPhone

Author: Bitter melon

Link: What App has an amazing name?

Source: Zhihu

Q: Is there an App with an amazing name? A: Water meter assistant… It’s for deliveries…

Author: Wang Sta

Link: Why do you buy a portable hard drive?

Source: Zhihu

Q: Why did you buy a portable hard drive? A: Even if the conditions are good, you should make your women comfortable

Author: Rebel

Link: How to Remotely Shut down a PC with an iPad?

Source: Zhihu

Q: How to use iPad remote control PC shutdown? A: Throw it at the PC power button

Author: Zhihu user

Link: What to say about Apple’s September 7, 2016 event?

Source: Zhihu

Q: How do you comment on apple’s launch event on September 7, 2016? A: I watched three launches for the new MacBook Pro in half A year…

By Shaun Chen

Link: How do I evaluate Internet Explorer?

Source: Zhihu

Q: How do you evaluate Internet Explorer? A: Download other browsers browser —– A year later —– IE8 below suck, do front-end want to cry rhythm.

Author: Zhihu user

Link: What if my parents tell me to save for a house, but I want to buy an Apple computer?

Source: Zhihu

Q: What if my parents tell me to save money for a house, but I want to buy an Apple computer? A: If you can really save 500,000 yuan for A house in 3 years, do you need 17,000 yuan to buy A computer, big brother?

Author: Warmth

Link: What are the spam apps?

Source: Zhihu

Q: What kind of junk phone apps are there? A: SMS blocker! Then it tells you it intercepted a text message. I’m sure 99% of you will click on it again to see the intercepted text messages!

By Simon Awen

Link: a complete PPT to do down, what is the most headache for you?

Source: Zhihu

Q: What is the biggest headache when you finish a complete PPT? A: How to hide one’s strength from the leader.

Author: Sun Jingdong

Link: What can Vim do that Emacs can’t?

Source: Zhihu

Q: What can Vim do that Emacs can’t? A: Help the poor children in Uganda…

Author: Wang Erxiao

Link: Why Do Apple Users choose Apple?

Source: Zhihu

Q: Why do Apple users choose Apple? A: Because people who don’t use Apple aren’t Apple users.

Author: Tang Qian

Link: What are the classic myths in the computer world?

Source: Zhihu

Q: What are some of the classic myths in the computer world? A: Windows is looking for A solution online.

Author: Feather Fan. 2010

Link: Will the wired mouse be replaced by the wireless one?

Source: Zhihu

Q: Will the wired mouse be replaced by the wireless mouse? A: I don’t think the wired mouse will be replaced in Internet cafes

Author: Up north

Link: What are the classic myths in the computer world?

Source: Zhihu

Q: What are some of the classic myths in the computer world? A: I have read and agree to the terms

Akik Sometimes died on the right

Link: What are some funny events in warcraft?

Source: Zhihu

Q: What are some funny events in warcraft? A: Hello, I think I’m the biggest joke in World of Warcraft…

Author: CCafe

Link: What do computer science students say?

Source: Zhihu

Q: What do computer science students say? A: It’s running fine on my computer

By Bell

Link: How to chat when you meet Jack Ma on the plane?

Source: Zhihu

Q: How do you chat when you meet Jack Ma on the plane? A: Hello Jack, my name is Jackson.

Author: Anonymous user

Link: How to understand Jack Ma said that eight years later, the house is like green onion?

Source: Zhihu

Q: How do you understand Jack Ma’s saying that after eight years, houses are like green Onions? A: Hurry up and buy spring Onions, the price of spring Onions is going up!!

Author: Colliot

Link: How to understand Jack Ma said that “killing the landlord does not mean you can get rich” this sentence?

Source: Zhihu

Q: What do you think of Jack Ma’s saying that “killing the landlord does not mean you can become rich”? A: He means “Don’t kill me”

Author: justjavac

Link: how to view Baidu in wei Zexi incident in the past and quietly the promise of rectification of the advertising prompt color tone?

Source: Zhihu

Q: What do you think of Baidu quietly dimming the color of the advertising prompts that promised rectification after the Wei Zexi incident? A: Please don’t black Baidu, I am doing front-end development, this is A long time, web CSS faded

60 lines of code to crawl zhihugod reply

Climb to know hu answer

Screening of answer

God replied

Related Posts

nodejs debugging

Can’t you log off AWS Cognito for Spring Security microservices?

Spring AOP source code analysis