0 x00 the

Since we mentioned the static architecture of the Autograd engine, this article begins with a dynamic look at how the engine works.

Other articles in this series are as follows:

Automatic Differentiation of Deep Learning Tools (1)

Automatic Differentiation of Deep Learning Tools (2)

Automatic differentiation of Deep Learning Tools (3) — Interpretation of examples

PyTorch implements forward propagation (1) — Base class (1)

PyTorch implements forward propagation (2) — Base class (2)

PyTorch how to implement forward propagation (3) – implementation

How to implement back propagation (1)—- call engine

Pytorch how to implement backward propagation (2)—- engine static structure

0x01 Previous review

Above, we analyzed the engine from the static point of view, and finally obtained a dynamic graph as follows, which will be further refined in this paper.

  • 1) The main thread adds a NodeTask to the CPU ReadyQueue.
  • 2) Worker thread 1 takes the NodeTask from CPU ReadyQueue and starts executing it.
  • 3) After worker thread 1 terminates, insert a NodeTask into one of the readyqueues of Device_readY_queues_.
  • 4) The worker thread corresponding to the ReadyQueue 2 fetched the NodeTask and started executing it.
                            +-------------------------+
                            | GraphTask               |
                            |                         |
                            |        cpu_ready_queue_ |
                            |            +            |
                            |            |            |
                            +-------------------------+
                                         |
+--------------+                         |                           +-----------------+
| Main Thread  |                         v                           | Worker Thread 1 |
|              |       1         +-------+---------+       2         |                 |
|              | push(NodeTask)  |                 |  pop(NodeTask)  |                 |
|          +-------------------> | CPU ReadyQueue  +----------------------->           |
|              |                 |                 |                 |                 |
|              |                 +-----------------+                 |                 |
|              |              +----------------------+               |                 |
|              |              | Device ReadyQueues   |               |                 |
|              |              |                      |               |                 |
|              |              |                      |    3          |                 |
|              |              |    +-------------+   | push(NodeTask)|                 |
|              |              |    | ReadyQueue 1| <-----------------------+           |
|              |              |    +------+------+   |               |                 |
|              |              |           |          |               |                 |
+--------------+              |           |          |               +-----------------+
                              |           +------------------+
                              |                      |       |       +-----------------+
+------------------------+    |          .           |       |       | Worker Thread 2 |
| Engine                 |    |          .           |       |       |                 |
|                        |    |          .           |       |       |                 |
|                        |    |                      |       |       |                 |
|  device_ready_queues_ +---> |    +-------------+   |       +------------->           |
|                        |    |    | ReadyQueue 2|   | pop(NodeTask) |                 |
|                        |    |    +-------------+   |     4         |                 |
+------------------------+    |                      |               |                 |
                              |                      |               |                 |
                              |    +-------------+   |               |                 |
                              |    | ReadyQueue 3|   |               |                 |
                              |    +-------------+   |               |                 |
                              |                      |               |                 |
                              +----------------------+               +-----------------+
Copy the code

0x02 Overall Engine Architecture

Let’s look at the overall architecture of the engine from a dynamic runtime perspective. The general logic of Engine::execute is as follows:

  • Start the engine.

    • Initialize the local ready_queue.
    • Build a GraphTask.
    • Build GraphRoot, which is the root node.
    • Calculate the minimum number of topologies.
    • Calculate the dependencies of each node. The purpose is to calculate the number of dependencies of all nodes.
    • If the output is not empty, the graph_task is initialized by calling graph_task->init_to_execute(*graph_root, outputs, accumulate_grad, min_topo_nr).
    • Configure the various inputs for the worker thread.
    • Start the worker thread.

  • Run the engine using execute_with_graph_task(graph_task, graph_root,…) Start the worker thread.

    • Each thread has a ReadyQueue, and the Root node is placed in the queue.

    • After initialization, the child thread calls thread_main(NULlptr) to work.

    • Evaluate_function (task) was repeatedly called in thread_main to calculate the gradient of each Node, and the next Edge was continuously searched by the next_edges until the gradient of all nodes was calculated, and finally the whole graph was calculated.

      • Perform the NodeTask calculation.
      • Iterate over all next_functions of the current Function and reduce their dependencies by one to see if they are ready.
      • If ready, the device of the InputBuffer determines which ReadyQueue to place it in.
      • If not, put it in not_ready in GraphTask.
      • If graph_task->outstanding_tasks <= 0 then exit the loop. That is, all nodes of GraphTask are executed.

  • The main process blocks and waits for graph_Task ->future_result_, the worker thread, to terminate.

The specific code is as follows:

Auto Engine::execute(const edge_list& roots, const variable_list& inputs, Bool accumulate_graph (); // Whether to construct a differential graph bool accumulate_grad. Outputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs < inputs > [](const std::string& msg) { return msg; }); // A fresh first time Engine::execute call should start on the CPU device, initialize // a new thread local ready queue on CPU or reuse the existing one (if there is one // allocated already, i.e. consecutive backward calls, re-entrant backward calls), // then memoize the local_ready_queue in GraphTask init_local_ready_queue(); Local ready_queue bool not_reentrant_backward_call = worker_device == NO_DEVICE; // Initialize local ready_queue bool not_reentrant_backward_call = worker_device == NO_DEVICE; Auto graph_task = STD ::make_shared<GraphTask>(/* keep_graph */ keep_graph, /* create_graph */ create_graph, /* depth */ not_reentrant_backward_call ? 0 : total_depth + 1, /* cpu_ready_queue */ local_ready_queue); // Build GraphRoot // If we receive a single root, skip creating extra root node bool skip_dummy_node = root.size () == 1; auto graph_root = skip_dummy_node ? roots.at(0).function : std::make_shared<GraphRoot>(roots, inputs); Outputs auto min_topo_nr = compute_min_topological_nr(outputs); Compute_dependencies (graph_root.get(), *graph_task, min_topo_nr); Outputs graph_task to initialize if (! outputs.empty()) { graph_task->init_to_execute(*graph_root, outputs, accumulate_grad, min_topo_nr); } Queue the root if (skip_dummy_node) {InputBuffer input_buffer(roots.at(0). Function ->num_inputs()); auto input = inputs.at(0); const auto input_stream = InputMetadata(input).stream(); const auto opt_next_stream = roots.at(0).function->stream(c10::DeviceType::CUDA); input_buffer.add(roots.at(0).input_nr, std::move(input), input_stream, opt_next_stream); // Start the worker process execute_with_graph_task(graph_task, graph_root, STD ::move(input_buffer)); } else {// Start the worker process execute_with_graph_task(graph_task, graph_root, InputBuffer(variable_list())); } // Avoid a refcount bump for the Future, since we check for refcount in // DistEngine (see TORCH_INTERNAL_ASSERT(futureGrads.use_count() == 1) // in Dist_engine.cpp). // The main process blocks waiting for graph_Task ->future_result_. auto& fut = graph_task->future_result_; fut->wait(); return fut->value().toTensorVector(); }Copy the code

Let’s follow the steps to analyze.

0x03 Start An Engine

The starting engine section includes:

  • Initialize the local ready_queue.
  • Build a GraphTask.
  • Build GraphRoot, which is the root node.
  • Calculate the minimum number of topologies.
  • Calculate the dependencies for each node.
  • If the output is not empty, the graph_task is initialized by calling graph_task->init_to_execute(*graph_root, outputs, accumulate_grad, min_topo_nr).
  • Configure the various inputs for the worker thread.

Let’s go through them one by one.

3.1 Initializing the Local Ready Queue

As mentioned earlier, each Autogard worker thread is associated with a ready queue that specifies the workflow to be executed by that thread, as defined below.

static thread_local std::shared_ptr<ReadyQueue> local_ready_queue = nullptr;
Copy the code

This shared_ptr is a thread_local pointer to each thread’s ready_queue, which should be initialized by a call to Engine::init_local_ready_queue() in each corresponding thread before execution.

In addition, GraphTask also has a CPU queue member variable cpu_readY_queue_, which is dedicated to handling back-propagation related CPU work.

The init_local_ready_queue code has two execution paths:

  • If ready_queue is not configured, Engine::execute is a new call and should be started on the CPU. So you need to initialize a new thread-local ready queue on the CPU or reuse an existing thread-local ready queue (such as reentrant backward propagation) and save it on the worker thread’s local_ready_queue. This is done with the following code, where local_ready_queue is the thread local variable for the main thread.
  • Worker thread execution path: The ready_queue parameter is configured and yesstd::thread t(&Engine::thread_init, this, i, device_ready_queues_[i], true)Local_ready_queue is the local variable of the worker thread.

Init_local_ready_queue takes no arguments and generates local_ready_queue for the worker thread.

Void Engine::init_local_ready_queue(STD ::shared_ptr<ReadyQueue> ready_queue) {if (ready_queue) {// Worker thread execution path // if ready_queue provided in the caller, use the caller's ready_queue to initialize local_ready_queue local_ready_queue = std::move(ready_queue); } else if (! Local_ready_queue){// Main thread execution path. // otherwise if local_ready_queue not allocated, allocate a new ready_queue local_ready_queue = std::make_shared<ReadyQueue>(); }}Copy the code

3.2 build GraphTask

The GraphTask is then built, passing in local_ready_queue for the main thread.

Auto graph_task = STD ::make_shared<GraphTask>(/* keep_graph */ keep_graph, /* create_graph */ create_graph, /* depth */ not_reentrant_backward_call ? 0 : total_depth + 1, /* cpu_ready_queue */ local_ready_queue);Copy the code

Within the constructor, local_ready_queue is assigned to the GraphTask member variable CPU_ready_queue_.

GraphTask(
    bool keep_graph,
    bool grad_mode,
    int reentrant_depth,
    std::shared_ptr<ReadyQueue> cpu_ready_queue,
    bool exit_on_error = false)
    : keep_graph_(keep_graph),
      grad_mode_(grad_mode),
      owner_(NO_DEVICE),
      reentrant_depth_(reentrant_depth),
      exit_on_error_(exit_on_error),
      cpu_ready_queue_(std::move(cpu_ready_queue)),
      future_result_(std::make_shared<at::ivalue::Future>(c10::ListType::create(c10::TensorType::get()))) {}
      
Copy the code

The internal queue of the GraphTask is generated, but the internal device_ready_queues_ is not yet generated:

+------------------------+                                     +-----------------------+
| GraphTask              |                                     | Main Thread           |
|                        |       |-----------------|           |                       |
|     cpu_ready_queue_+------->  | CPU ReadyQueue  | <-----------+ local_ready_queue   |
|                        |       +-----------------+           |                       |
|                        |                                     |                       |
+------------------------+                                     +-----------------------+
​
​
+------------------------+
| Engine                 |
|                        |
|                        |
|  device_ready_queues_  |
|                        |
|                        |
+------------------------+
Copy the code

3.3 Building a root Node

Next, build the root node. We have a question about building the root node: What if, in a forward calculation, there are multiple outputs? What if there are multiple input roots when you propagate backward? Here are the answers:

  • If there is only one root Node, skip creating other roots and return roots.at(0). Function as GraphRoot, which is a Node Node.

  • If there are multiple root input roots, a GraphRoot is constructed and used to drive backward propagation.

    • Take these roots as arguments and build a GraphRoot that acts as the real root node.
    • Root is the edge of Node. That is, transform the edge_list corresponding to these roots into the next_edges_ in Node, and the GraphRoot can be considered a virtual Root.

The specific code is as follows:

// If we receive a single root, skip creating extra root node bool skip_dummy_node = roots.size() == 1; auto graph_root = skip_dummy_node ? Roots.at (0). Function: // Single root, use STD ::make_shared<GraphRoot>(roots, inputs); // Multiple root input roots, construct a GraphRoot, use it to drive backward propagationCopy the code

Remember the definition of GraphRoot, and you can verify that.

using edge_list = std::vector<Edge>; struct TORCH_API GraphRoot : public Node { GraphRoot(edge_list functions, variable_list inputs) : Node(std::move(functions)), outputs(std::move(inputs)) { // Ensures calls to stream() on a GraphRoot instance reflect current stream(s) // on devices of root grad tensors at the time the instance is constructed. for (const auto& t : outputs) { add_input_metadata(t); } } variable_list apply(variable_list&& inputs) override { return outputs; Outputs variable_list; // This is the gradient};Copy the code

3.4 Calculating the minimum topology

The compute_min_topological_nr traverse side, find the minimum number of topology min_topo_nr. Min_topo_nr will be used next to calculate dependent functions.

inline static uint64_t compute_min_topological_nr(const edge_list& outputs) { // Computes the mininum topological number  among all the outputs if (outputs.empty()) { return 0; } auto min_topo_nr = std::numeric_limits<uint64_t>::max(); for (auto & output_edge : outputs) { auto topo_nr = output_edge.function.get()->topological_nr(); min_topo_nr = (min_topo_nr < topo_nr) ? min_topo_nr : topo_nr; } return min_topo_nr; }Copy the code

Topological_nr_ is the topological sequence number of a “node” and represents the length of the longest possible path from that node to any leaf node. If a node is a leaf node, that is, AccumulateGrad, topological_nr_ will be zero. Topological_nr_ is used to trim branches in the DAG during autograd discovery, and maintaining topology topological_nr_ helps us to complete the check at O(1) time when there is no directed path between two nodes.

Topological_nr_ has the following properties:

  • For each pair of nodes X, Y in G, if there is a directed path from X to Y, it means topo_nr(X) > topo_NR (Y). However, this is not the case, so we cannot prove the existence of a path from X to Y, only that it does not exist.
  • One of the assumptions we make when using topological_nr_ is that once a node is used, that is, it has a parent node, its own Topological_nr_ will not change. We have added some checks to the “has_parent_” field to enforce this.

You can also use comments in the code to verify.

// NOTE [ Topological Number ] // // topological_nr is used to prune branches in the DAG during autograd discovery as //  maintaining topological_nr helps us check in O(1) if there does NOT exist // a directed path between two nodes. // // The topological order number of this `Node` representing the length of the // longest possible path from this Node to any leaf node. If you are leaf node, // aka AccumulateGrad, this will be zero. This value has the property that // For every pair of nodes X, Y in G, existence of a directed path from X to Y // implies topo_nr(X) > topo_nr(Y). The converse is not true, however, so we // cannot prove existence of a path from X to Y, only non-existence. // // One assumption we make when using topo_nr is that once a node // has been used, i.e., has a parent node, its own topo_nr does not change // we have added some checks with the `has_parent_` field to enforce this. // // What NOT to do: // // 1) 2 -> 1 -> 0 In this diagram we label nodes with their topo_nr. // 2 -> 1 -> 0 We have two simple graphs that can each arise from // `t.exp().exp()`, for example. // 2) 2 -> 1 -> 0 // / // 2 -> 1 -> 0 We add 2 as a next edge to 1 even though 1 already // has a parent. // 3) 2 -> 1 -> 0 // / // 2 -> 3 -> 0 2 < 3, yet there exists a path from 2 to 3! // uint64_t topological_nr() const noexcept { has_parent_ = true; return topological_nr_; }Copy the code

3.5 Computing Dependencies

The GraphTask dependencies_ member variable is of the following type to determine whether the subsequent node is ready to execute:

std::unordered_map<Node*, int> dependencies_;
Copy the code

The number of keys in dependencies is the number of nodes in the differential graph.

The dependencies member is initialized in the compute_dependencies call, and dependencies[this_grad_fn] increments by one whenever one of the grad_fn functions appears once on someone else’s next_edges(). If dependencies[this_grad_fn] is greater than 0, “this_grad_fn” must wait for “this_grad_fn” to complete the calculation of “this_grad_fn”.

Compute_dependencies is a Dependencies_ for the GraphTask. The logic is as follows: starting from graph_root, the dependencies of each node in the differential graph are calculated, starting from the root node, by breadth-first algorithm. If a grad_fn function appears once on someone else’s next_edges(), then dependencies[grad_fn] increments by 1. The specific code is as follows:

auto Engine::compute_dependencies(Node* root, GraphTask& task, uint64_t min_topo_nr) -> void { // Computes the number of dependencies for each function which requires grad std::unordered_set<Node*> seen; std::vector<Node*> queue { root }; // Queue contains all nodes that will start propagating gradients. // We no longer have to expand functions that don't require grad. auto& dependencies = task.dependencies_; while (! queue.empty()) { auto fn = queue.back(); queue.pop_back(); if (fn->topological_nr() < min_topo_nr) { continue; } for (const auto& edge : fn->next_edges()) { if (auto next_ptr = edge.function.get()) { dependencies[next_ptr] += 1; const bool was_inserted = seen.insert(next_ptr).second; if (was_inserted) queue.push_back(next_ptr); }}}}Copy the code

Like our code

a = torch.tensor(2., requires_grad=True)
b = torch.tensor(6., requires_grad=True)
Q = 3*a**3 - b**2
​
Copy the code

The corresponding calculation diagram is

To get a dependency is:

> > < p class = "nb" > < p class = "nb" Dependencies [PowBackward0_2] = 1 #Copy the code

Execute if the dependencies of a node are 0.

3.6 Initializing GraphTask ExecInfo

If the outgoing edge is not empty, the graphTask.exec_info_ is initialized with a call to init_to_execute.

if (! outputs.empty()) { graph_task->init_to_execute(*graph_root, outputs, accumulate_grad, min_topo_nr); }Copy the code

Exec_info_ is used to configure ExecInfo for each Node of GraphTask, which is the execution information for that Node.

  • If exec_info_ is empty, the task is running in default mode, that is, all next_edges encountered need to be executed.
  • If exec_info_ is not empty, it indicates that only specific functions with an entry whose “has needed == True” will be executed.

Exec_info_ When is empty? When is it not empty?

  • When the graph is executed with.backward() and no input parameters are passed, exec_info is empty.
  • Exec_info_ is non-empty if only executed with.grad(), or when executed with.backward() and given input parameters.

Thus, exec and CAPtured_vars_ are backward() for grad() and specified parameters, marking which gradients need to be calculated in this case. In this case, only certain nodes need to be executed, and from those nodes, there is a path to outpus.

Init_to_execute fills exec_info with data. The purpose of init_to_execute is to set the member variable exec_info[node].needed_ = true for the node that should execute. Only certain nodes are executed, and these nodes have an output edge, the other end of which is in the “outputs”.

The main algorithm logic is that exec_info is filled with a recursive approach, but the actual code is iterative. In the iterative version, when you play with the current node, you update your parent node after all your child nodes have been updated.

void GraphTask::init_to_execute(Node& graph_root, const edge_list& outputs, bool accumulate_grad, uint64_t min_topo_nr) { // Populates exec_info so nodes that should be executed have `exec_info[node].needed_ = true` //  Only nodes that have a path to any edge in `outputs` should be executed. // The code below populates exec_info using recursion, but the actual code does this // iteratively. Refer to the numbering to see how the actual code corresponds. // A difference to note is that in the iterative version, when you are working with // the current Node, you are reponsible to update your parent's is_needed after all your // children have been updated. // // is_needed = {fn: True for fn in outputs} # (0) // seen = {} // def compute_is_needed(fn): // for next_edge in fn.next_edges: // child_fn = next_edge.fn // if child_fn in seen and is_needed[child_fn]: # (1) // is_needed[fn] = true // else: // seen.add(child_fn) // if compute_is_needed(child_fn): // is_needed[fn] = true # (2) // # (3) exit for-loop // return is_needed[fn] // compute_is_needed(graph_root) // // NB: you might be wondering why we don't populate `seen` with outputs. We cannot // because in the case where two outputs lie  on the same path, we still need to explore past // the first output or we would miss the nodes that are required to compute the second output. int output_idx = 0; for (auto & output_edge : outputs) { // (0) `is_needed` above corresponds to `exec_info_[fn].needed_` Node *output = output_edge.function.get(); auto & info = exec_info_[output]; if (accumulate_grad) { // if called through `.backward()` we directly set `needed_` for all the outputs to true info.needed_ = true; } else { // otherwise it is `.grad()` and we set exec_info[fn].captures_ instead // In terms of populating the rest of exec_info though, you can basically // think of this as the same as setting `needed_` is true directly. if (! info.captures_) { info.captures_ = make_unique<std::vector<ExecInfo::Capture>>(); Captures_ ->emplace_back(output_edge.input_nr, output_idx++); } } captured_vars_.resize(output_idx); struct Frame { Frame (Node *fn) : fn_(fn), next_next_fn_(0) {} Node *fn_; size_t next_next_fn_; Node* get_next_fn() { const auto & next = fn_->next_edges(); auto num_next = next.size(); while (next_next_fn_ < num_next) { auto fn = next[next_next_fn_++].function.get(); if (fn) return fn; } return nullptr; }}; auto nodeShouldExecute = [this](Node *fn) { auto it = exec_info_.find(fn); return it ! = exec_info_.end() && it->second.should_execute(); }; std::vector<Frame> stack; std::unordered_set<Node*> seen; stack.emplace_back(&graph_root); exec_info_.emplace(stack.back().fn_, ExecInfo()); Exec_info while (! stack.empty()) { auto &frame = stack.back(); const auto fn = frame.fn_; Node *child_fn = nullptr; while((child_fn = frame.get_next_fn()) && ! seen.emplace(child_fn).second) { // (1) next child exists AND has already been seen if (nodeShouldExecute(child_fn)) { exec_info_[fn].needed_ = true; } } if (child_fn) { // (2) next child exists but has not been seen if (child_fn->topological_nr() < min_topo_nr) { // child created before the first output means this child cannot have // an edge to output continue; } stack.emplace_back(child_fn); } else { // (3) no next child exists for `fn` means its `needed` has already been // finalized. pop stack and update parent stack.pop_back(); if (nodeShouldExecute(fn) && ! stack.empty()) { exec_info_[stack.back().fn_].needed_ = true; }}}}Copy the code

3.7 Configuring worker thread Input

Next in the main thread, InputMetadata is built by configuring the worker thread’s input.

// Queue the root if (skip_dummy_node) { CUDA queue InputBuffer input_buffer(roots.at(0). Function ->num_inputs()); auto input = inputs.at(0); Const auto input_stream = InputMetadata(input).stream(); const auto opt_next_stream = roots.at(0).function->stream(c10::DeviceType::CUDA); input_buffer.add(roots.at(0).input_nr, std::move(input), input_stream, opt_next_stream); execute_with_graph_task(graph_task, graph_root, std::move(input_buffer)); } else {// If it is a multi-input root node, the virtual root node was built earlier, CPU queue execute_with_graph_task(graph_task, graph_root, InputBuffer(variable_list())); }Copy the code

3.8 Starting

Execute_with_graph_task is then called. What execute_with_graph_task does is start various subsequent threads, which we’ll talk about later.

Here is to know, according to the current device to take different paths, the specific logic is as follows:

  • If worker_device == NO_DEVICE, this is the main thread.

    • Get the relevant queue using input_buffer.device(), which used the InputBuffer in the previous section. Cpu_ready_queue_ gets the graphTask. cpu_ready_queue_ gets the CPU, and the GPU queue gets the GPU queue, more on that later.
    • Insert NodeTask in queue.
    • Call thread_main to run GraphTask.

  • Otherwise, it is the main thread in the case of reentrant backpropagation, then:

    • Use graph_task-> OWner_ = worker_device to specify which device the current device is, GPU or CPU.

      • If the maximum recursion depth is reached, either the GPU thread or CPU thread is started with add_thread_pool_task.
      • Otherwise, run thread_main.

The specific code is as follows:

std::shared_ptr<at::ivalue::Future> Engine::execute_with_graph_task( const std::shared_ptr<GraphTask>& graph_task, std::shared_ptr<Node> graph_root, InputBuffer&& input_buffer) { initialize_device_threads_pool(); STD ::unique_lock< STD ::mutex> Lock (graph_task->mutex_); Input_buffer.device () ¶ auto queue = ready_queue(graph_task->cpu_ready_queue_, input_buffer.device()); // worker_device == NO_DEVICE it's a CPU thread and it's trying to drive the // autograd engine with corresponding GraphTask, and its NOT a re-entrant call if (worker_device == NO_DEVICE) { // We set the worker_device to CPU_DEVICE only if worker_device was previously // no_device.setting it to CPU afterwards allow us to detect whether this is // a re-entrant call or not. set_device(CPU_DEVICE); // set the graph_task owner to the current device graph_task->owner_ = worker_device; // Now that all the non-thread safe fields of the graph_task have been populated, // We can enqueue it. Queue ->push(NodeTask(graph_task, STD ::move(graph_root), STD ::move(input_buffer))); // The owning thread start to drive the engine execution for any CPU task that // was just pushed or will be added later  from other worker threads lock.unlock(); thread_main(graph_task); TORCH_INTERNAL_ASSERT(graph_task->future_result_->completed())); // reset the worker_device after the completion of the graph_task, this is so // that the initial state of the engine remains the same across every backward() // or grad() call, we don't need to reset local_ready_queue as we could possibly // reuse it for new backward calls. worker_device = NO_DEVICE; // If you get to this point, there must be no active working device, which means that all graphTasks are finished, or if not, reentrant, Else {// If worker_device is any devices (i.e. CPU, CUDA): this is a re-entrant // backward call from that device. graph_task->owner_ = worker_device; // Now that all the non-thread safe fields of the graph_task have been populated, // We can enqueue it. // Insert the first NodeTrask, Graph_root queue->push(NodeTask(graph_task, STD ::move(graph_root), STD ::move(input_buffer)); if (current_depth >= max_recursion_depth_) { // See Note [Reentrant backwards] // If reached the max depth, Switch to a different thread // reach the maximum reentrant depth, where a new thread add_thread_pool_task(graph_task) is started; Else {// Total depth needs to be updated only in this codepath, since it is // not used in the block above (when we call add_thread_pool_task). // In the codepath above, GraphTask.reentrant_depth_ is used to // bootstrap total_depth in the other thread. ++total_depth; // Get back to work while we wait for our new graph_task to // complete! ++current_depth; lock.unlock(); thread_main(graph_task); // Run on the main thread, which blocks above the queue --current_depth; --total_depth; // The graph task should have completed and the associated future should // be marked completed as well since 'thread_main' above is a call // blocking an autograd engine thread. TORCH_INTERNAL_ASSERT(graph_task->future_result_->completed()); // graph_task_exec_post_processing is done when the Future is marked as // completed in mark_as_completed_and_run_post_processing. return graph_task->future_result_; }Copy the code

3.9 Configuring the Device and ReadyQueue

In the previous section, the following code is used to configure the device.

set_device(CPU_DEVICE);
Copy the code

3.9.1 CPU_DEVICE

Impl ->setDevice = impl->setDevice

void set_device(int device) { // NB: We MUST NOT construct the guard for device CPU, // as in some settings we compile with cuda, but // have lazy stubs for CUDA functionality (so actually // attempting to setup a guard(CPU_DEVICE) will cause an // error, because it will still query cudaGetDevice). // // Don't use DeviceGuard here because its destructor may be called before  the // device is reset. This is fine because the device is thread local. if (device ! = CPU_DEVICE) { for (size_t i = 0; i < static_cast<size_t>(c10::DeviceType::COMPILE_TIME_MAX_DEVICE_TYPES); i++) { auto* impl = c10::impl::device_guard_impl_registry[i].load(); if (impl && device < impl->deviceCount()) { impl->setDevice(at::Device(static_cast<c10::DeviceType>(i), device)); } } } worker_device = device; }Copy the code

In addition to set_device(CPU_DEVICE) being called for initialization, Engine::thread_init is also called for starting the device thread, setting the device sequence number. This sequence number corresponds to the ReadyQueue.

auto Engine::start_device_threads() -> void { for (int i = 0; i < num_devices; ++i) { std::thread t(&Engine::thread_init, this, i, device_ready_queues_[i], true); t.detach(); } } void Engine::thread_init(int device, const std::shared_ptr<ReadyQueue>& ready_queue, bool should_increment) { ... set_device(device); . }Copy the code

3.9.2 Ready Queue

In the previous section, we had the following code to get the queue.

Input_buffer.device (); // Obtain the CPU queue using input_buffer.device(). auto queue = ready_queue(graph_task->cpu_ready_queue_, input_buffer.device());Copy the code

Let’s look at how the current queue is obtained based on the CPU queue of this GraphTask and the configured input device.

  • Call InputBuffer:: Device () to get the set device from the input, if set, use the device, otherwise use the CPU.
  • If CPU, use CPU_ready_queue, otherwise use device_ready_queues_.

Further analysis is as follows:

Each GraphTask has its own CPU queue, but its GPU Queues are shared by all GraphTasks.

// CPU ready queue is per GraphTask, but CUDA device ready queues are shared across all graph tasks auto Engine::ready_queue(std::shared_ptr<ReadyQueue> cpu_ready_queue, at::Device device) -> std::shared_ptr<ReadyQueue>{ if (device.type() == at::kCPU || device.type() == at::DeviceType::Meta) { // return the cpu ready queue passed in return cpu_ready_queue; } else { // See Note [Allocating GPUs to autograd threads] return device_ready_queues_.at(device.index()); }}Copy the code

InputBuffer:: Device Retrieves the device from the input parameter. This returns if there is a configured device, otherwise at::kCPU.

auto InputBuffer::device() const -> at::Device { // Since we pick the first non-CPU tensor, this won't work with // mixed device-type operations (e.g., an op that is both CUDA // and XLA). This is *incredibly* unlikely, so we don't worry // about it. for (auto& var : buffer) { if (var.defined()) { auto device = var.device(); if (device.type() ! = at::kCPU) { return device; } } } // Only report to the CPU thread if there really were no tensors // from other devices. return at::kCPU; }Copy the code

3.10 The main process waits

At the end of Engine::execute, the main process enters the wait state with the following code

// Avoid a refcount bump for the Future, since we check for refcount in // DistEngine (see TORCH_INTERNAL_ASSERT(futureGrads.use_count() == 1) // in Dist_engine.cpp). // The main process blocks waiting for graph_Task ->future_result_. auto& fut = graph_task->future_result_; fut->wait(); return fut->value().toTensorVector();Copy the code

Here STD ::shared_ptr
<: ::future="" ivalue="">
future_result_; To communicate between threads.

The main thread waits with wait, and the worker thread notifies the main process of completion with markCompleted.

0x04 Starts the thread

Because of the complexity of the threading part, we will analyze it separately from the startup part.

Because large models tend to have too many gradients, PyTorch uses multithreading. To deal with various cases, PyTorch defines a thread variable worker_Device. The threads generated by the engine are assigned a “worker_Device” that specifies which device they are processing work for. This variable is initialized at the following location:

  • In CUDA, the creation time of XLA device threads is initialized as they are waiting to work on their own devices.
  • Initialize before the graph task of the CPU thread executes, because for each backward call, we use caller threads to drive engine execution.
static constexpr int NO_DEVICE = -2; static constexpr int CPU_DEVICE = -1; // Threads spawned by the engine are assigned a 'worker_device' specifying // what device they process work for. This variable is initialized at: // 1. thread creation time for CUDA, XLA device threads, as they are // spinning threads waiting for works on their device. // 2. before the graph task execution for CPU threads, as for each // backward call we use the caller thread to drive engine execution. // This is used when handling reentrant  backwards calls; // See Note [Reentrant backwards] static thread_local int worker_device = NO_DEVICE;Copy the code

4.1 Starting the Device thread

4.4.1 call

Within the execute_with_graph_task method, start start_Device_threads using initialize_device_threads_pool.

std::shared_ptr<at::ivalue::Future> Engine::execute_with_graph_task( const std::shared_ptr<GraphTask>& graph_task, STD ::shared_ptr<Node> graph_root, InputBuffer&& input_buffer) {// The worker thread initialize_device_threads_pool() is started first;Copy the code

STD ::call_once is used here to ensure that the start_device_threads member function is called only once during the entire process cycle, that is, the device thread is generated only once.

void Engine::initialize_device_threads_pool() {
  track_bad_autograd_forks();
  std::call_once(start_device_threads_flag_, &Engine::start_device_threads, this);
}
Copy the code

4.1.2 Number of threads

In the engine, the number of worker threads is determined by the number of devices. If there are n devices, n device worker threads will be started. For example, if there are two Gpus, start two device worker threads. But each GraphTask has its own CPU worker thread (which we’ll cover in a moment).

The ReadyTask corresponding to the GPU worker thread is a member variable of Engine.

std::vector<std::shared_ptr<ReadyQueue>> device_ready_queues_;
Copy the code

In this case, the ready queue index of the two GPU worker threads is 0 and 1 respectively.

Device_ready_queues_ is defined in the engine, which also indicates that device queues are shared between all Graphtasks, while CPU queues are unique to each GraphTask. The device thread is started in the start_device_threads function, which calls STD :: Thread to start the thread and detach it to run independently.

4.1.3 Starting the device thread

Start_device_threads is used to start device threads, the number of device threads is dependent on the number of devices, so that NUM_DEVICES threads work together in the background to process the tasks in GraphTask.

  • Use deviceCount to get the number of devices num_devices.

  • The number of device threads to start is then determined based on the number of devices.

  • Create as many readyqueues as there are worker threads. These readyqueues are managed on top of device_readY_queues_ of the Engine. Device_ready_queues_ is the management GPU.

  • Create a device thread. Each thread is STD :: Thread, and the corresponding ReadyQueue, device_ready_queues_[I], and Engine (which has only one Engine instance throughout the process life cycle) are passed in at build time. Queue can then rely on Engine’s sharing of Device_readY_queues_ to transfer working objects between threads.

  • In contrast, GraphTask has a ReadyQueue (CPU_ready_queue_) that runs cpu-related worker threads. CPU worker threads are designed to handle back-propagated CPU work. When GraphTask finishes working on a GPU, the next NodeTask should switch to the CPU, so GraphTask needs to remember its own CPU_ready_queue_ to send a message to CPU_ready_queue_.

    Notice that CPU_ready_queue_ is GraphTask’s proprietary ReadyQueue.

Auto Engine::start_device_threads() -> void {// See Note [Allocating GPUs to autograd threads] // Use deviceCount to get the number of devices Num_devices. c10::DeviceIndex num_devices = 0; for (const auto& impl_atomic : c10::impl::device_guard_impl_registry) { auto* impl = impl_atomic.load(); if (impl) { num_devices = std::max(num_devices, impl->deviceCount()); } } // allocate one thread for every GPU device (but colocate GPUs of different // types), And pre-allocate the device_ready_queues_ to ensure safe reading on it. // Create multiple readyqueues, Number of readyqueues is the same as number of worker threads device_ready_queues_ = STD ::vector< STD ::shared_ptr<ReadyQueue>>(num_devices); for (auto& queue : device_ready_queues_) { queue.reset(new ReadyQueue()); } thread_pool_shared_ = std::make_shared<ThreadPoolShared>(); // create device thread for (int I = 0; i < num_devices; ++i) { std::thread t(&Engine::thread_init, this, i, device_ready_queues_[i], true); t.detach(); // Wait for the threads to start {STD ::unique_lock< STD ::mutex> LK (non_reentrant_device_thread_mutex_); while(non_reentrant_device_thread_count_.load() ! = static_cast<uint32_t>(num_devices)) { non_reentrant_device_thread_condvar_.wait(lk); }}}Copy the code

After the device thread starts, it uses wait to block in its corresponding ReadyQueue. The main thread or other worker threads wake up the device thread by using push(NodeTask) operation on the ReadyQueue of a device thread.

4.2 Thread Initialization

The thread will call thread_init to initialize it.

  • Configure the device for the thread.
  • Call init_local_ready_queue to initialize the local queue.
  • Thread_main is called as the thread body function.

The code for initializing the local queue is as follows:

void Engine::init_local_ready_queue(std::shared_ptr<ReadyQueue> ready_queue) {
  if (ready_queue) {
    // if ready_queue provided in the caller, use the caller's ready_queue to initialize local_ready_queue
    local_ready_queue = std::move(ready_queue);
  } else if (!local_ready_queue){
    // otherwise if local_ready_queue not allocated, allocate a new ready_queue
    local_ready_queue = std::make_shared<ReadyQueue>();
  }
}
Copy the code

Each Autograd worker thread is associated with a ready queue, which is the thread’s workflow. Local_ready_queue uses STD ::shared_ptr as the local thread pointer.

// Every autograd worker thread is associated with a ready queue, which specifies
// the stream of work of this thread to do. This shared_ptr is a thread_local
// pointer to each thread's ready_queue, and it should be initialized via the
// Engine::init_local_ready_queue() call in each corresponding thread before execution.
//
// The CUDA, XLA threads are shared among all invocations of backwards via
// device_ready_queues_, while CPU threads are dedicated to processing CPU work for
// the backward they invoked. So any given graph task maintains its own cpu_ready_queue_
// where you should send work for it to be done
//
// For reentrant backward calls, if we spawn new thread from the current thread
// because we reached the maximum depth, the new thread will just reuse the same
// ReadyQueue with the parent thread for performance improvement.
// see Note [Reentrant backwards] for more details.
// NOLINTNEXTLINE(cppcoreguidelines-avoid-non-const-global-variables)
static thread_local std::shared_ptr<ReadyQueue> local_ready_queue = nullptr;
Copy the code

The specific initialization code is as follows:

void Engine::thread_init(int device, const std::shared_ptr<ReadyQueue>& ready_queue, bool should_increment) {
  if (should_increment) {
    increment_non_reentrant_thread_count();
  }

  at::init_num_threads();

  // Note [Allocating GPUs to autograd threads]
  // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  // What's our strategy here?  Originally, the autograd engine was written
  // with only CUDA in mind.  We allocate one thread to handle all CPU
  // operations, and a thread per CUDA device.
  //
  // But what if we have OTHER devices?  There are two plausible
  // strategies:
  //
  //  - We can allocate threads equal to max(num_cuda_devices, num_xla_devices,
  //    ...) and colocate cuda device 0 with xla device 0
  //  - We can allocate threads equal to sum(num_cuda_devices, num_xla_devices,
  //    ...) keeping everyone separate.
  //
  // We don't have any good reason to prefer one or the other, so we've
  // arbitrarily picked to colocate devices.  Maybe the other approach is
  // better.
  set_device(device);

  // initialize each device thread's thread local ready queue with the ready queue
  // that is created before the thread initialization
  init_local_ready_queue(ready_queue);

  std::shared_ptr<GraphTask> graph_task = nullptr;
  thread_main(graph_task);
  if (should_increment) {
    // Decrement the count during shutdown if we incremented earlier.
    decrement_non_reentrant_thread_count();
  }
}
Copy the code

The logic is as follows, generating a series of worker threads and also generating device_readY_queues_ :

+------------------------+ +-----------------------+ | GraphTask | | Main Thread | | | |-----------------| | | | cpu_ready_queue_+-------> | CPU ReadyQueue | <-----------+ local_ready_queue | | | +-----------------+ | | | | | | +------------------------+ +-----------------------+ +------------------------+ | Engine | +----------------------+ +-----------------------+ | | | Device ReadyQueues | | Worker Thread 1 | | | | | | | | device_ready_queues_ +---> | | | | | | | +-------------+ | | | | | | | ReadyQueue 1| <-------------+ local_ready_queue | +------------------------+ | + -- -- -- -- -- -- -- -- -- -- -- -- -- + | | | | | | | | | | | | | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | |. | |. | |. | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | | Worker Thread 2 | | +-------------+ | | | | | ReadyQueue 2| <-------------+ local_ready_queue | | +-------------+ | | | | | | | | | +-----------------------+ | +-------------+ | | | ReadyQueue 3| | | +-------------+ | | | +----------------------+Copy the code

Thread_main is then called for the thread body, which we’ll examine below.

4.3 Reentrant backpropagation

This example

As you can see from PyTorch’s test code, back propagation is also called in back propagation.

In this case, there are multiple GraphTasks within the Engine.

class MyFunction(Function): @staticmethod def forward(ctx, x): return x @staticmethod def backward(ctx, x): order.append("MyFunction") return x class Reentrant(Function): @staticmethod def forward(ctx, x): with torch.enable_grad(): ctx.x = Variable(x.detach(), requires_grad=True) ctx.x = ctx.x - 1 return ctx.x.detach() @staticmethod def backward(ctx, x): order.append("Reentrant") if ctx.x < 0: return x with torch.enable_grad(): Reentrant.apply(ctx.x).backward() return x a = myfunction.apply (torch. Tensor (6.0, Requires_grad =True) B = reentrant.apply (torch. Tensor (9.0, requires_grad=True)) V = a * b v.Brown ()Copy the code

4.3.2 Design concept

Here is PyTorch’s design philosophy

To understand the reentrant backward problem, we must note two aspects of the current implementation of the Autograd engine:

    1. When you call Engine::execute(), you want to block until differentiation is complete so that the final result variable can be passed back.
    1. When the engine is running, there is one worker thread running on top of each work queue, and each work queue is pinned to the specific device performing the operation.

The problem is, suppose you call backward() inside a worker thread.

According to property (1), we should block until the nesting task is complete. However, according to attribute (2), this worker thread is responsible for handling the tasks assigned to it; We’d better not be blocked. Because in that case, all of our reverse executions (including the one we just started) would be deadlocked!

So, we maintain a pool of threads waiting for work to be allocated.

When a reentrant backward call occurs, the current thread is blocked and a thread in the pool is awakened to complete the blocking task and any other tasks assigned to the worker thread. If no thread is available, a new thread is generated. The new thread will continue to process tasks in the same ReadyQueue as the parent worker thread. When the current GraphTask completes, the parent worker thread waiting for the task is notified and the current thread is returned to the thread pool.

4.3.3 implementation

Engine:: execute_with_graph_Task calls the following code to add a new thread to the thread pool when reentrant backward propagation is detected beyond the maximum recursion depth.

    if (current_depth >= max_recursion_depth_) {
      // See Note [Reentrant backwards]
      // If reached the max depth, switch to a different thread
      add_thread_pool_task(graph_task);
    }
Copy the code

Related data structures are as follows:

struct ThreadPoolShared { // Data structures used by the threads for executing reentrant backwards // tasks. See Note [Reentrant backwards] // Number of available threads for processing new GraphTasks. unsigned int num_workers_; // The threads will wait on work_ to be notified of GraphTasks std::condition_variable work_; // To protect reads and writes to graphtask_queue_ and num_workers_ // and for synchronizing creating new threads when needed std::mutex mutex_; // Workers will process the GraphTasks added to this queue. A GraphTask is // allocated inside Engine::execute and lives  for the duration of execute std::queue<std::weak_ptr<GraphTask>> graphtasks_queue_; ThreadPoolShared() : num_workers_(0) {} };Copy the code

The code for add_thread_pool_task is as follows.

  • This determines whether the GraphTask queue has reached its maximum, and if not, creates a new thread.
  • Put graph_task into the queue graphTasks_queue_.
  • The execution function for the new thread is reentrant_thread_init, which waits onthread_pool_shared_->work_Above the law.
  • There will bethread_pool_shared_->work_.notify_one()Let the new thread run.
void Engine::add_thread_pool_task(const std::weak_ptr<GraphTask>& graph_task) {
  std::unique_lock<std::mutex> lck(thread_pool_shared_->mutex_);
  // There may already be some items on the graphtasks_queue_ added by other
  // threads but not enough workers to get to the new task that will be
  // added
  bool create_thread = (thread_pool_shared_->num_workers_ <= thread_pool_shared_->graphtasks_queue_.size());
  thread_pool_shared_->graphtasks_queue_.push(graph_task);
  // Don't need to be holding the lock while actually creating the thread
  lck.unlock();
  if (create_thread) {
    std::thread t(&Engine::reentrant_thread_init, this);
    t.detach();
  }
  // This works even if new thread is created because wait() will test the
  // predicate before waiting
  thread_pool_shared_->work_.notify_one();
}
Copy the code

The new thread executes the function reentrant_thread_init as follows:

  • Share cpu_ready_queue with the Graph_Task’s original thread.
  • It gets the GraphTask from graphtasks_queue_ and assigns it to graph_task.
  • It is then executed using thread_main(graph_task).
// Reentrant call will re-use the graph_task's owner thread ready_queue for // queueing tasks (NOTE: this is not true in the async_mode of the engine). // While we can create separate ready queue for each new reentrant //  thread, but sharing the same cpu_ready_queue with parent thread is a // performance improvement and cuda thread still have to do  the same thing. void Engine::reentrant_thread_init() { at::init_num_threads(); auto tp_shared = thread_pool_shared_; while(true) { std::unique_lock<std::mutex> lk(tp_shared->mutex_); ++thread_pool_shared_->num_workers_; tp_shared->work_.wait(lk, [&tp_shared]{ return ! tp_shared->graphtasks_queue_.empty(); }); --thread_pool_shared_->num_workers_; auto task = tp_shared->graphtasks_queue_.front(); tp_shared->graphtasks_queue_.pop(); lk.unlock(); std::shared_ptr<GraphTask> graph_task; if (! (graph_task = task.lock())) { continue; } set_device(graph_task->owner_); // set the local_ready_queue to the ready queue on the graph_task->owner_ device local_ready_queue = ready_queue_by_index(graph_task->cpu_ready_queue_, graph_task->owner_); total_depth = graph_task->reentrant_depth_; thread_main(graph_task); // call thread function}}Copy the code

4.4 the main thread

In addition to the above thread, the engine has a main thread. This is denoted by NO_DEVICE. As shown earlier, CPU_DEVICE is also used for temporary reentrant checks, but it is still the main thread.

static constexpr int CPU_DEVICE = -1;
static constexpr int NO_DEVICE = -2;
Copy the code

4.5 Process Analysis

Let’s parse the generation relationship between threads using execute_with_graph_task.

std::shared_ptr<at::ivalue::Future> Engine::execute_with_graph_task( const std::shared_ptr<GraphTask>& graph_task, std::shared_ptr<Node> graph_root, InputBuffer&& input_buffer) { initialize_device_threads_pool(); Input_buffer.device () specifies the device to run on, so depending on the device, Queue auto queue = ready_queue(graph_task->cpu_ready_queue_, input_buffer.device()); Set_device (CPU_DEVICE); if (worker_device == NO_DEVICE) {if (worker_device == NO_DEVICE) {if (worker_device == NO_DEVICE) { graph_task->owner_ = worker_device; // set the graph_task owner to the current device queue->push(NodeTask(graph_task, std::move(graph_root), std::move(input_buffer))); thread_main(graph_task); Worker_device = NO_DEVICE; worker_device = NO_DEVICE; } else {// worker_device is any devices (i.e. CPU, CUDA): this is a re-entrant // backward call from that device. graph_task->owner_ = worker_device; Queue ->push(NodeTask(graph_task, STD ::move(graph_root), STD ::move(input_buffer))); If (current_depth >= max_recursion_depth_) {// Reach the maximum reentrancingdepth, where a new thread add_thread_pool_task(graph_task) is started; // add_thread_pool_task is GPU thread or CPU thread, depending on worker_device} else {++total_depth; ++current_depth; thread_main(graph_task); // Thread_main is still executed by the main thread, internally blocked by pop waiting --current_depth; --total_depth; } return graph_task->future_result_;} return graph_task->future_result_; }Copy the code

The specific thread relationship is as follows:

  1. The main thread uses push(NodeTask) to insert NodeTask 0 into graphTask.cpu_ready_queue_.
  2. The main thread uses thread_main to fetch NodeTask 0 from graphTask.cpu_ready_queue_, assuming the device index of NodeTask 0 is 1.
  3. The main thread uses thread_main to insert NodeTask 1 into device 1’s ReadyQueue.
  4. Device thread 1 blocks on Device 1’s ReadyQueue 1 and is woken up to fetch NodeTask 1.
  5. Device thread 1 processes NodeTask 1 to get its next edge, and if the device on that edge is Device 2, a NodeTask 2 device is generated. Then insert NodeTask 2 into ReadyQueue 2.
  6. Device thread 2 blocks on Device 2’s ReadyQueue 2, wakes up, retrieves NodeTask 2, and continues processing.
+-------------------------+ | GraphTask | | | | cpu_ready_queue_ | | + | | | | +-------------------------+ | +------------------+ | | Main Thread | v | | 1 +--------+---------+ | |push(NodeTask0)| | +-------------------+ | +--------------------->+ CPU ReadyQueue | | Worker Thread 1 | | | | | 4 | | | local_ready_queue| +---+--------------+ pop(NodeTask1) | | | | 2 pop() | +------------------> | | +--------------------------+ | | | | | | +--------------------+ | | | | | | | Device ReadyQueues | | | local_ready_queue | | | | | | | | | | | | 3 | | | | | | | |push(NodeTask1)| +-------------+ | | | | | +------------------------> | ReadyQueue 1+-----+ | | | | | +-------------+ |  5 | | | | | |push(NodeTask2)| | +------------------+ | | +---------------+ | | | | | | | | | +-------------------+ | | | | +-----------+ +-------------------+ +---------------------------+ | | | | Worker Thread 2 | | Engine | | | | | | | |  | v | | | | | | +----------+--+ | | | | device_ready_queues_ +-----> | | ReadyQueue 2+------------------------> | | | |  +-------------+ |pop(NodeTask2) | | | | | | 6 | local_ready_queue | +---------------------------+ | | | | | | | | | +-------------+ | | | | | ReadyQueue 3| | | | | +-------------+ | | | | | | | +--------------------+ +-------------------+Copy the code

The mobile phone is as follows:

0xEE Personal information

★★★★ Thoughts on life and technology ★★★★★

Wechat official account: Rosie’s Thoughts

0 XFF reference

Explain the network construction in Pytorch

PyTorch’s optimizer

Distribution of PyTorch

PyTorch’s Tensor

PyTorch’s Tensor

PyTorch’s Tensor

PyTorch dynamic diagram (part 2)

PyTorch dynamic diagram (part 1)