This article was first published on wechat public number — interesting things in the world, handling reprint please indicate the source, otherwise will be held responsible for copyright. Wechat account: A1018998632, QQ group: 859640274

Recently, I have studied deep learning for nearly a month and a half, so I want to test my learning results. Just as the design is the realization of image processing APP, so the fast style transfer feedforward neural network through Tensorflow for Android was transplanted to the APP, as a filter fast style transfer effect is quite good, but the speed is a little slow. Deep learning on Android doesn’t support GPU yet.

1. About MyPhotoShop

This is an image processing APP that uses Opencv, deep learning, MVVM, Databinding, RxJava, various design patterns, etc. I will be writing a series of blog posts about this project step by step in the coming days. I hope you will pay more attention to it.

Effect of 1.

2. Project related

  • 1. Github address: MyPhotoShop image processing APP project address.
  • 2. Apk download: MyPhotoShop download address, APK is quite large, but most of the neural network model files, so it can not be reduced

3. The shortcomings

  • 1. No componentization
  • 2. No confusion
  • 3. Some places are not abstract enough

2. Basic concepts of deep learning and neural networks

1. What is deep learning

  • The first three concepts are progressive. To put it simply, deep learning is a kind of machine learning. Deep learning is the use of machines to learn a lot of data, and machine learning is a way to realize AI.
  • 2. There are two important things in deep learning: data and neural networks. There are two important processes in deep learning: training and testing
    • 1. Data and Network
      • 1. Data: Let’s imagine a simple picture classification scenario where we have 10,000 pictures that have been sorted by people, and each picture has a correct classification, such as cat, dog, etc.
      • 2. Network: We can imagine the neural network here as a function. Our input is a picture, and the output is the score of the picture under each category. It’s just an array of fractions.
    • 2. Training and testing:
      • 1. The training: We will photo collections of images in training time and input into the neural network, and then will get the pictures again and again under each category scores, every time we come up with a score after array we can calculate the current neural network loss value (the lower the current network, the higher the accuracy loss value), with the loss value, Our goal is to reduce the loss value. Those of you who know derivatives know that we can take the derivative of the lose-value function and get the gradient direction that makes the lose-value decrease, and feed that back into the neural network. This cycle again and again, so that the loss value to the minimum.
      • 2. Test: When we train the neural network to an optimal state, we can input the image we need to classify into the neural network and get the final result of the neural network to classify the image.
    • 3. Conclusion: How does deep learning work? We can see that our training data has been processed by human, so the process of deep learning is to solidify human processing into our neural network, and finally let the neural network replace the process of manual processing.
    • 4. The above is just an introduction to the basic process of deep learning. If you want to know more about it, you can read this blog post

2. What are neural networks

As we said in the last video, eventually the human process of processing data is solidified into a neural network through our training. I’m going to talk a little bit about the neural networks that WE talked about.

  • 1. Still in the simple picture classification scene:
    • 1. We assume that the size of the picture x is 100 * 100(we tiled the picture into a 1 * 10000 matrix), and there are 10 categories of the picture.
    • 2. Then a two-layer neural network is like this: y = x * w1 * w2(w1 is the matrix of 10000 * A, w2 is the matrix of A * 10), where y is the final fraction of a picture under each classification, and the multiplication in the formula is matrix multiplication.
    • 3. Of course, a neural network with more layers has more W, and a in W1 and W2 can be defined by itself.
  • Y = x * w1 * w2
    • 1. Research shows that when we look at the picture X, we first look at the outline of the picture. Here, the neurons in our brain that see the outline of the picture are equivalent to W1
    • 2. After looking at the outline, we will have a basic sense of the things in the picture and judge which category the picture belongs to. The category here is the result of X * w1
    • The result in 3.2 is fed to the next layer of neurons in the brain, which is equivalent to W2, and after w2 we produce a result which is y.
    • 4. Of course, humans have many more layers of neurons than that
  • 3. Training w1 y = x * * w2 in the process of comparing is equivalent to: we have a bunch of pictures to show a child what also don’t know, at first he is sure that the output result is wrong, but we just every time to correct his mistakes, he neurons in the brain (w) will constantly change and identification accuracy is improved.

3. Tensorflow in Android

This section will cover how to use a trained neural network in Android

1. Start

In this article, I will only take a demo as an example to explain, and the MyPhotoShop project mentioned above will be analyzed in another topic.

  • 1. Demo address: Github address
  • 2. Introduce Tensorflow: compile ‘org. Tensorflow: Tensorflow – android: +’

2. Concepts in Tensorflow

  • 1. Graph: We explained what a neural network looks like earlier. In Tensorflow, every neuron W of the neural network belongs to a node in the graph, and all nodes of the neural network form a directed acyclic graph, which is a part of the graph of Tensorflow. Of course, in the diagram of Tensorflow, in addition to the nodes of neural network, there are other auxiliary operations, such as image decoding, image coding, image pre-processing and so on. Here’s an example of a graph: Picture A — “decode picture produce B –” process B produce picture data matrix C (1 * 10000)– “C and W1 (10000 * x) matrix multiply produce D (1 * x)–” D and W2 (x * 10) matrix multiply produce E (1 * 10)– “select the classification with the largest median value of E, The neural network decides that picture A is one of these categories.
  • 2. Node: Each node is a part of the graph. Each node has input parameters, output parameters, specific operation functions (such as matrix multiplication), and possibly neuron value W.
  • 3. TensorFlowInferenceInterface: a Tensorflow training in context, different names in different languages. The interior contains all the instances needed for a training.

3. Demo code explanation

This demo only involves the use of Tensorflow neural network model in Android, and does not involve the training process. There are two reasons: 1. Mobile terminal is not suitable for training neural networks; 2.Tensorflow for Android has no training API.

  • 1. The neural network I use this time is a trained fast style migration network
  • 2. For the model, our input is: a picture is converted into a tensor of type float, size is (1 * 800 * 600 * 3), the input node name is PADsss :0, where the name is defined in the training process.
  • 3. Our output for this model is a float tensor of size (1 * 780 * 680 * 3). The name of the output node is squeezesss:0, which was also defined during training.

  • 4. Let’s look at the code and obtain the permission with RxPermission first. After obtaining the permission, write the pictures to be processed in assets to the SD card for use later and enter the make() method

  • 5. Read the image in 4 to the memory
  • 6. Taking ARGB as an example, we know that each pixel in Bitmap is stored in an int hexadecimal format, similar to FFFFFFFF, so every two bits is the value of a channel, with the upper limit of 256. So the next step is to convert the pixel values in the Bitmap into a float array of (800 * 600 * 3) size.
  • 7. Create a TensorFlowInferenceInterface object, the ginseng is the AssetManager and model files, here is said to neural network was set up in memory
  • 8. Print the name of each node
  • 9. Input the name of the input node, the data of the input node and the dimension of the data tensor into the neural network
  • 10. Run the neural network, input is the name of the output node
  • 11. The runtime of the neural network is blocked, so when it is running, it will fetch the data in a float array (780 * 580 * 3).
  • 12. Reintegrate the float array into the pixel values of the Bitmap and write them to the Bitmap.

4. Pay attention to the point

  • 1. Demo will run slowly. Please be patient
  • 2. The devices I run are: Mi Mix2 and Android8.0. Other devices may have problems, either very slow, or perhaps not supported by the CPU or system version.

4. To summarize

Running a trained neural network in Android is relatively easy, as long as you know the input and output, just like running a normal function. As for how to train a neural network, that is another story, can follow my I learn machine learning corpus! There will be continuous updates on my machine learning experience and experience.

No angst peddling, no clickbait. Share some interesting things about the world. Topics include but are not limited to: science fiction, science, technology, the Internet, programmers, computer programming. The following is my wechat public number: Interesting things in the world, dry goods waiting for you to see.