This Friday, no beautiful sister took me out to play purr, bored at home to button the phone, found everyone in the synthesis of watermelon. As a young, do-nothing computer scientist (or maybe people’s artist) in the future, I would have disdainfully played such a time-wasting and boring game (it took me four hours to synthesize my first watermelon), but in order to engage with the masses, I decided to try writing a program to hang up and run.

After an afternoon of writing, I finally finished my first version of the Python watermelon. I’m still mentally retarded, but at least I beat 50% of the people and passed the Turing Test. In fact, I am afraid that I will write the final version in two days, everyone has played enough, the clown is me, in order to eat a bite of hot S ** T, I now send hahaha.

I’ll write it again next Friday if anyone plays it.

This article is divided into six parts (if I can make it through) :

  1. Image capture

  2. Fruit detection

  3. Fruit recognition

  4. Get rid of the noise

  5. Click on the control

  6. The game strategy

Don’t say a word, let’s go!

1. Image capture

As a computer program, the first step to synthesizing a large watermelon is to be able to see it

This step is super easy, just use The Windows GRAPHICAL interface Win32GUI that comes with Python. The most important step is to get the coordinates of the top, bottom, left and right vertices of the browser window, call FindWindow method to get the window object, and then pass the window object to GetWindowRect to get the vertex coordinates.

I was stuck for a long time because FindWindow needed to know the name of the window first. I tried “Edge” not right, “Composite watermelon” not right, and finally I tacitly opened Alt + del and found that it was called “mini Game: Synthetic watermelon – Personal – Microsoft\ U200B Edge “seems to say to me:” I don’t call hello, I call…”

Then use ImageGrab method of PIL library to capture the pixels in the screen window, and convert them into NUMpy array, then you can carry out subsequent processing.

2. Fruit testing

Once we have the image, the next step is to identify the location of the fruit, a natural thought would be to identify the circle, since the fruit is all round (thank you for not adding a banana. What? You said sugar cane is fruit…)

And then I want to tease you, what the hell is this thing down here? Is it two rows of teeth? Does anyone else find it as disgusting as I do? And it’s even worse when it’s greyscale…

Ok Based on this idea, we just need to find the circle in the picture!

The idea here is that the classic circle detection method that has been implemented in OpenCV is the Hough circle detection method (see clearly, not Hoffman, not Hoffman, not Hoffman). The basic idea is to find a few edges and draw circles as you go, and then see if the centers come together… I am not engaged in CV, if there is a mistake please big guys comment area correct.

There are two main difficulties in this step. The first is that the fruit is stacked on top of each other, so the two circles are easily connected and you can’t find the circle. I did this by setting up a 13 by 13 Gaussian filter, so it’s much better to blur the edges and then extract them.

The second difficulty is that there are three or four parameters to be adjusted for Cv2. HoughCircles. I am traversing the parameters through GridSearch to see the recognition effect of the selected last parameter, the old deep learning alchemist hahaha.

Finally we get the coordinates and radii of many circles.

3. Fruit recognition

The next step, once we’ve got a bunch of circles, is to figure out what kind of fruit each circle is. Then I wanted to interview the game’s developers. Why are oranges and lemons designed to be the same size? Lavoisier’s coffin was breaking down.

So I can’t determine which fruit it is by the radius of the circle (crying), and the result of hough circle test is also quite large, the radius of the same fruit may be different in two tests.

At this point, a smart guy would type import Pytorch to get ready for pattern recognition. What I chose to do is to calculate the average RGB value inside the circle and compare it with the radius to find the most similar value.

I feel a little sorry for my math teacher in junior high school. I forgot how to calculate the inner coordinates of a circle according to the center and radius of the circle.

Finally, a mapping from RGB mean + radius to fruit species was established:

Fruit_type = {' GRAPE: [133.68415638, 42.41563786, 112.84156379, 18], 'KIWI' : [132.0420593, 201.00264307, 64.44633418, 53], 'CHERRY' : [238.33213966, 39.17905103, 56.09982095, 28], "ORANGE" : [246.2261046, 129.05342651, 21.69071235, 37], 'LEMON' : [237.19926471, 216.40716912, 65.56176471, 43], 'TOMATO' : [238.18209682, 89.91402075, 95.00730902, 66], 'PEACH' : [235.99086897, 165.13803074, 97.37832902, 65], 'PINEAPPLE' : [248.87955751, 219.50704342, 88.25265164, 89], 'COCONUT: [230.9732507, 225.98286918, 211.98437795, 101],' WATERMELON ': [236.62388536, 100.24692594, 121.0164732, 143],}Copy the code

4. Eliminate noise

Just when I thought I could take the next step, I found a lot of problems, mainly with big pineapples and coconuts, I love coconut water, I hate coconuts.

The pineapple and coconut have two layers and contain two circles themselves, so the identification results often produce many overlapping circles, such as the following:

So I have to find a way to get rid of the circles that overlap, and the way I chose to do this is to compare all the centers in pairs, and if the sum of their distances is less than the sum of the radii of the two circles, then I’m in coincidence. So how do we figure out which circle is the right one? Keep comparing similarities… The ones that look like the standard ones are correct.

It’s unpretentious, but it does seem to work.

5. Click control

This is actually pretty easy, no hard points, right

Use Python’s built-in Win32api-mouse_event method to simulate mouse click events. The only caveat is that it forces control of your mouse… So if your hands aren’t fast enough, you might not be able to click the “x” in the top right corner to close the app, and watch your fruit pile up on top…

6. Game strategy

In fact, today I was mainly trying to find out how to recognize and control, the algorithm I thought was not realized, in order to eat a mouthful of hot S ** T, now it is only a retarded version.

The strategy is if you have the same, you go there, if you don’t, you go in the middle.

This is obviously a very bad strategy, because it’s going to be bigger and it’s going to be worse for the structure. If nothing is the same, falling in different places will also result in different overall structure… I think the first should be to avoid small on the big top, similar to 2048 try not to slide up the strategy, because if the big on the top of the small may never be synthesized.

How do you maintain a good structure? Do you design a multi-level cache like a computer architecture? Can an evaluation index be designed for each step and the algorithm be automatically optimized iteratively? Is the fruit order of each refresh the same, and is there a certain statistical law? If we knew in advance what fruit to refresh next, we could search using A* or some other heuristic. Can we even do it with AI methods like reinforcement learning?

There are a lot of interesting ideas to try, if the game is still played next Friday, I am willing to try some new ideas, the synthetic watermelon series will continue to update haha. But now, really, I’ve been doing it all day and all I can think about is mangosteen + mangosteen = cherry… One more look and I’m gonna throw up

7. About the author

I’d like to introduce Xiaoyu, a young, do-nothing computer PhD student and future people’s artist living in Qingdao. I spend most of my spare time reading books and doing Research. I will come to Zhihu to write articles when I have time. You can pay attention to my two columns: Xiaoyu girl machine learning notes and Xiaoyu girl algorithm notes, seriously write knowledge of the kind, with this article is not the same, serious, ha ha.

I was so sleepy that I couldn’t keep my eyes open, but I still finished writing… So, I’m going to play with two plates of big watermelons.

Well, that’s all for today’s sharing. If you’re interested in Python, join us.Python Learning communicationSkirt 】, receive free learning materials and source code.