This article was originally published by AI Frontier.
PyTorch released a one-year team summary: Running resources reduced to 1/10, king of the pack


Nuclear Coke

“The PyTorch Team has released an annual inventory to mark The one-year anniversary of PyTorch’s release. As mentioned in Teacher Mu’s “Technical Analysis of Deep Learning Framework”, TensorFlow system has the highest integrity, but performance problems need to be solved. PyThorch is the NLP application of choice and the king in standalone scenarios, but the disadvantage is that it is difficult to support large-scale applications. We can take another look at the PyTorch team’s summary.”


The translation is as follows:

Today marks the one-year anniversary of the PyTorch project’s public release. It’s a crazy process — we want to build a flexible deep learning research platform. Over the past year, we have witnessed how actively PyTorch has been used, contributed and spread by a very distinguished group of participants — and we thank you for your kindness. Looking back, we wanted to summarize PyTorch’s year, including progress from the community, the latest news, and key points.

community

We have a strong community of researchers and engineers who love PyTorch deeply. The core team includes engineers and researchers from many countries, enterprises and universities. It is fair to say that without the positive contributions of everyone involved, we would never have been able to turn PyTorch from idea to reality.

Research papers, software packages and GitHub

Within days of the project’s release, users from the community were implementing research papers they were interested in in PyTorch and Posting code on GitHub. Open source code has become the main tool used by researchers today. Together, the participants created torchText, TorchVision, and Torchaudio software packages designed to democratize research in different areas of the PyTorch project.

The first PyTorch based community package comes from Brandon Amos, a package called Block that makes it easier to work with Block matrices. The Locus lab of Carnegie Mellon University subsequently released several PyTorch software packages and implementations closely related to its research results. The first research paper was codebunked by Sergey Zagoruyko and titled “Focus on the ‘Focus Mechanism'”.

Jun-yan Zhu, Taesung Park, Phillip Isola, Alyosha Efros and their team from the University of California, Berkeley, have released the highly acclaimed Cyc-GAN and Pix2PIx, which implements image-to-image conversion.

The researchers at HarvardNLP and Systran chose to develop and improve OpenNMT in PyTorch, inspired by a recreation of [Lua]Torch code written by Facebook’s Adam Lerer. Twitter’s MagicPony team also contributed by introducing its early super-resolution work into the PyTorch example.

Salesforce Research has released several packages, including its proud PyTorch-QNN, an RNN type optimized by CuDNN to achieve anywhere from 2x to 17x faster performance than standard LSTM. James Bradbury and his team have become one of the most active and engaging teams in PyTorch You.

“The @PyTorch-QrNN we released had anywhere from 2 to 17 times the speed performance of nvidia’s cuDNN LSTM. This level of speed is thanks to 50 lines of code implemented through CuPy.”

Researchers from Uber, Northeastern, and Stanford university have come together to form an active probabilistic programming community around Pyro and ProbTorch. The community has achieved a dynamic and rapid development, We met Fritz Obermeyer, Noah Goodman, Jan-Willem van de Meent, and Brooks at NIPS 2017’s First PyTorle-Probabilistic Programming conference Paige, Dustin Tran, and 22 other panellists discussed how to build a world Bayesian network.

Nvidia researchers published three sets of high-quality libraries to realize the results in the pix2PiX-HD, Sentiment Neuron and FlowNet2 papers. Their work on the scalability analysis of the different data parallelism models in PyTorch has contributed greatly to the community.

The Allen Institute for AI has released the AllenNLP, which includes several industry-leading natural language processing models — including reference implementations for standard natural language processing tasks and easy-to-use Web presentation materials.

In July 2017, we also welcomed our first Kaggle winning team, GRT123. They won a Lung cancer detection award at the DataScience Bowl 2017 and announced PyTorch’s implementation.

On the virtualization side, Tzu-Wei Huang developed a Tensorboard-PyTorch plug-in, while Facebook AI Research implemented PyTorch compatibility for its VisDOM virtualization package.

Finally, Facebook AI Research has released several projects, including ParlAI, fairseeq-py, VoiceLoop, and FaderNetworks, that aim to leverage cutting-edge models and interface datasets from multiple domains.

There is not enough space to list all the outstanding projects here, but those who are interested can check out the list: github.com/soumith?tab…

We would also like to thank our friends who have strongly supported the forum’s suggestions, especially PTRBLCK, JPEG729, QuantScientist, albanD, Tom and Chenyun TC. Thank you again for your invaluable support.

Digital indicators

Let’s learn more by the numbers:

  • On GitHub, there are 87,769 lines of Python code for importing Torch.
  • GitHub has 3,983 libraries with names or descriptions when referring to PyTorch projects.
  • PyTorch binary resources have been downloaded over 500,000 times, or to be more specific, 650,000 1,916 times.
  • 5400 users have posted 21,500 discussion articles on 5200 topics on our forum discuss.pytorch.org (discuss.pytorch.org/).
  • Since its release, PyTorch has been mentioned 131 times on Reddit/R/Machinelearning. Meanwhile, TensorFlow was mentioned 255 times.

The indicators

PyTorch is a research-centric framework. Therefore, one of the indicators we are looking for is the use of PyTorch by academics from machine learning research papers.

  • At the recent ICLR2018 conference, PyTorch was mentioned in 87 papers, TensorFlow in 228, Keras in 42, and Theano and Matlab in 32 papers each.
  • Arxiv.org mentions PyTorch 72 times a month, TensorFlow 273 times, Keras 100 times, Caffe 94 times and Theano 53 times.

Courses, tutorials and books

When we first launched PyTorch, we had good API documentation, but the tutorial material was limited to a few ipython notes — helpful, but not ideal.

Sasank Chilamkurthy took the job and transformed the content into the beautiful website it is today.

Sean Robertson and Justin Johnson have written a number of excellent new tutorials – natural language processing directions, for example. Yunjey Choi provides an amazing tutorial on how to implement most models in 30 lines or less. Every new tutorial can help users find their own way to learn better and faster. Goku Mohandas and Delip Rao are tweaking the code in their book to use the PyTorch project.

In addition, we found that PyTorch is a major teaching tool in many university machine learning courses, including Harvard University’s CS287. To further democratize learning, here we recommend three online courses closely related to PyTorch.

  • In September 2017, Jeremy and Rachel announced that future Fast. Ai courses would be designed almost entirely based on PyTorch.
  • Ritchie Ng, a researcher with close ties to the National University of Singapore and Tsinghua University, has launched a Udemy course called “Deep Learning with PyTorch”.
  • Sung Kim from The Hong Kong University of Science and Technology posted an online course on YouTube titled “Learn PyTorch from Scratch” for a general audience.

Engineering and technology

Over the past year, we have implemented a number of features, improved performance across the board, and fixed a number of bugs. A complete list of the work we did can be found in our release notes. The following are our core priorities for the past year:

High degree of ladder

With a series of papers published on gradient punishment and second-order gradient methods, this has become an important and highly concerned function. In August 2017, we implemented a generalized interface that can use the NTH derivative and gradually increase the coverage of the supported high ladder functions over time. As of press time, it has support for almost all operations.

Distributed PyTorch

In August 2017, we released a small distributed package that takes the popular MPI common approach. The software package has multiple backends, such as TCP, MMPI, Gloo, and NCCL2. It supports multiple CPU/GPU co-computing and use cases, and can integrate distributed technologies such as Infiniband and RoCE. Distribution is never easy, and there were some mistakes in our first iterations. But in subsequent releases, we improved the stability of the package and its performance improved.

Approached NumPy

One of the most pressing demands from users is that they want NumPy functionality that they are familiar with. Such features, such as convenient broadcasting and advanced indexing, can save users time and effort. We implemented these features, but started tweaking our API to get closer to NumPy. Over time, we hope to be able to provide a fully NumPy compliant API in due course.

Sparse tensor

In March 2017, we released a small package that supports sparse tensors. In May of that year, we released CUDA support for this sparse package. The software package is small in size and has limited functions. It is mainly used to implement sparse embedding and various sparse paradigms commonly used in deep learning. It is still small and needs to be extended – if you are interested in participating in the expansion of this package, please contact the special agent on the discussion board: discuss.pytorch.org/.

performance

The quest for performance is a never-ending struggle, especially for a dynamic framework like PyTorch that wants to maximize flexibility. Over the past year, we’ve improved performance from the core Tensor library to neural network algorithms, and we’ve also written faster micro-optimizations.

  • We’ve added special AVX and AVX2 inline functions for Tensor operations.
  • Write faster GPU cores for high frequency workloads, such as Concatenation and Softmax.
  • The code is rewritten for several neural network operators, especially nn.Embedding convolved with groups.

Total operating resource requirement reduced to 1/10

Since PyTorch is a set of dynamic graphics frameworks, we need to create a new graphic dynamically during each iteration of the training cycle. This means that the running resource overhead of the framework must be low, or the workload must be so large that the running overhead of the framework is almost negligible. In August 2017, DyNet’s authors (Graham Neubig and his team) reported that it could handle small NLP models much faster than PyTorch. It was an interesting challenge, but at the time we didn’t know the exact size of the models being trained. Over the next few months, we worked on a major rewrite inside PyTorch, tweaking every algorithm that took more than 10 microseconds to execute in about 1 microsecond.

ATen

While we were internally redesigning PyTorch, we also built the Aten C++ 11 library that is now responsible for supporting the full PyTorch back end. ATen has apis that mirror PyTorch’s PyThon API, making it a handy C++ library for tensor calculations. ATen can be built and used independently of PyTOrch.

Export models to production — ONNX support with JIT compiler

Listening to the community, one common request we received was to export the PyTorch model to another framework. Users can do quick research cycles in PyTorch and want to send work to large projects that require only C++ once it’s done.

With this in mind, we built a set of trackers for PyTorch that export the PyTorch model as an intermediate representation. Subsequent tracing can be used to run the existing PyTorch model more efficiently (by optimizing it) or to convert it to ONNX format to be sent to other frameworks, including Caffe2, MXNet and TensorFlow. Or send it directly to a hardware acceleration library like CoreML or TensorRT. Stay tuned to learn more about JIT compiler performance improvements in the coming year!

Original link:

Pytorch.org/2018/01/19/…

For more content, you can follow AI Front, ID: AI-front, reply “AI”, “TF”, “big Data” to get AI Front series PDF mini-book and skill Map.