Pytorch changelog

Changelog History

Page 3

• v1.5.0-rc5

  April 20, 2020

• v1.5.0-rc4

  April 18, 2020

• v1.5.0-rc3

  April 09, 2020

• v1.5.0-rc2

  March 28, 2020

• v1.5.0-rc1

  March 19, 2020

• v1.4.1

  March 23, 2020

• v1.4.0 Changes

  January 16, 2020

  🚀 PyTorch 1.4.0 Release Notes

  • Highlights
  • Backwards Incompatible Changes
    • Python
    • JIT
    • C++
  • 🆕 New Features
    • torch.optim
    • Distributed
    • RPC [Experimental]
    • JIT
    • Mobile
  • 👌 Improvements
    • Distributed
    • JIT
    • Mobile
    • Named Tensors
    • C++ API
    • AMD Support
    • ONNX
    • Quantization
    • Visualization
    • Other Improvements
  • 🐛 Bug Fixes
    • Distributed
    • RPC
    • C++ API
    • JIT
    • Quantization
    • Mobile
    • Other Bug fixes
  • 🗄 Deprecations
  • 🐎 Performance

  🚀 The PyTorch v1.4.0 release is now available.

  🚀 The release contains over 1,500 commits and a significant amount of effort in areas spanning existing areas like JIT, ONNX, Distributed, Performance and Eager Frontend Improvements and improvements to experimental areas like mobile and quantization. It also contains new experimental features including rpc-based model parallel distributed training and language bindings for the Java language (inference only).

  🚀 PyTorch 1.4 is the last release that supports Python 2. For the C++ API, it is the last release that supports C++11: you should start migrating to Python 3 and building with C++14 to make the future transition from 1.4 to 1.5 easier.

  Highlights

  🏗 PyTorch Mobile - Build level customization

  🚀 Following the experimental release of PyTorch Mobile in the 1.3 release, PyTorch 1.4 adds additional mobile support including the ability to customize build scripts at a fine-grain level. This allows mobile developers to optimize library size by only including the operators used by their models and, in the process, reduce their on device footprint significantly. Initial results show that, for example, a customized MobileNetV2 is 40% to 50% smaller than the prebuilt PyTorch mobile library. Learn more about how to create your own custom builds, and please engage with the community on the PyTorch forums to provide any feedback you have.

  Distributed Model Parallel Training [Experimental]

  🚀 With the scale of models, such as RoBERTa, continuing to increase into the billions of parameters, model parallel training has become ever more important to help researchers push the limits. This release provides a distributed RPC framework to support distributed model parallel training. It allows for running functions remotely and referencing remote objects without copying the real data around, and provides autograd and optimizer APIs to transparently run backwards and update parameters across RPC boundaries.

  👀 To learn more about the APIs and the design of this feature, see the links below:

  • 📚 API documentation
  • 📄 Distributed Autograd design doc
  • 📄 Remote Reference design doc

  👀 For the full tutorials, see the links below:

  • A full RPC tutorial
  • Examples using model parallel training for reinforcement learning and with an LSTM

  As always, you can connect with community members and discuss more on the forums.

  Java bindings [Experimental]

  🚀 In addition to supporting Python and C++, this release adds experimental support for Java bindings. Based on the interface developed for Android in PyTorch Mobile, the new bindings allow you to invoke TorchScript models from any Java program. Note that the Java bindings are only available for Linux for this release, and for inference only. We expect support to expand in subsequent releases. See the code snippet below for how to use PyTorch within Java:

  📚 Learn more about how to use PyTorch from Java here, and see the full Javadocs API documentation here.

  Pruning

  👍 Pruning functionalities have been added to PyTorch in the nn.utils.prune module. This provides out-of-the-box support for common magnitude-based and random pruning techniques, both structured and unstructured, both layer-wise and global, and it also enables custom pruning from user-provided masks.

  To prune a tensor, first select a pruning technique among those available in nn.utils.prune (or implement your own by subclassing BasePruningMethod).

  from torch.nn.utils import prune t = torch.rand(2, 5) p = prune.L1Unstructured(amount=0.7) pruned\_tensor = p.prune(t)
  

  To prune a module, select one of the pruning functions available in nn.utils.prune (or implement your own) and specify which module and which parameter within that module pruning should act on.

  m = nn.Conv2d(3, 1, 2) prune.ln\_structured(module=m, name='weight', amount=5, n=2, dim=1)
  

  Pruning reparametrizes the module by turning weight (in the example above) from a parameter to an attribute, and replacing it with a new parameter called weight_orig (i.e. appending "_orig" to the initial parameter name) that stores the unpruned version of the tensor. The pruning mask is stored as a buffer named weight_mask (i.e. appending "_mask" to the initial parameter name). Pruning is applied prior to each forward pass by recomputing weight through a multiplication with the updated mask using PyTorch's forward_pre_hooks.

  Iterative pruning is seamlessly enabled by repeatedly calling pruning functions on the same parameter (this automatically handles the combination of successive masks by making use of a PruningContainer under the hood).

  👍 nn.utils.prune is easily extensible to support new pruning functions by subclassing the BasePruningMethod base class and implementing the compute_mask method with the instructions to compute the mask according to the logic of the new pruning technique.

  Backwards Incompatible Changes

  Python

  ⏱ torch.optim: It is no longer supported to use Scheduler.get_lr() to obtain the last computed learning rate. to get the last computed learning rate, call Scheduler.get_last_lr() instead. (26423)

  🆕 Learning rate schedulers are now “chainable,” as mentioned in the New Features section below. Scheduler.get_lr was sometimes used for monitoring purposes to obtain the current learning rate. But since Scheduler.get_lr is also used internally for computing new learning rates, this actually returns a value that is “one step ahead.” To get the last computed learning rate, use Scheduler.get_last_lr instead.

  ⚡️ Note that optimizer.param_groups[0]['lr'] was in version 1.3.1 and remains in 1.4.0 a way of getting the current learning rate used in the optimizer.

  Tensor.unfold on a 0-dimensional Tensor now properly returns a 1-dimensional Tensor.

  Version 1.3.1	Version 1.4.0
  >>> torch.tensor(5).unfold(dimension=0, size=1, step=1)

  tensor(5) | >>> torch.tensor(5).unfold(dimension=0, size=1, step=1) tensor([5]) |

  0️⃣ torch.symeig now return a 0-element eigenvectors tensor when eigenvectors=False (the default).

  Version 1.3.1	Version 1.4.0
  >>> torch.symeig(torch.randn(3,3)).eigenvectors.shape

  torch.Size([3, 3]) | >>> torch.symeig(torch.randn(3,3)).eigenvectors.shape torch.Size([0]) |

  JIT

  • Make torch.jit.get_trace_graph private (it is now torch.jit._get_trace_graph) (29149)
    • This function was intended only for ONNX integration; use traced_module.graph instead, like:
    • traced_module = torch.jit.trace(my_module, example_inputs)
      traced_graph = traced_module.graph
  • @property on ScriptModules has been disabled (28395)
    • Scripted @property accesses were silently broken before, where we would evaluate the the get function once and store that as the attribute permanently. They properly error now; a workaround is to make your @property a regular method.
  • 🚚 Custom ops: torch::jit::RegisterOperators has been removed, use torch::RegisterOperators instead (28229). The usage and behavior should remain the same.
  • Removetorch.jit._register_* bindings from Python (e.g. torch.jit._register_attribute). These were private functions that were not intended to be used. (29499)

  C++

  [C++] The distinction between Tensor and Variable has been eliminated at the C++ level. (28287)

  🔀 This change simplifies our C++ API and matches previous changes we did at the python level that merged Tensors and Variables into a single type.

  This change is unlikely to affect user code; the most likely exceptions are:

  Argument-dependent lookup for torch::autograd may no longer work. This can break because Variable is now defined as an alias for Tensor (using Variable = Tensor;). In this case, you must explicitly qualify the calls to torch::autograd functions.

  Because Variable and Tensor are now the same type, code which assumes that they are different types (e.g., for the purposes of templating, or std::enable_if checks) will not work until you delete the (now) redundant overload/specialization.

  Some operators may trace differently. If this happens, please file a bug. The most likely situations are:

  1. There are now more operations in your trace than before (usually, calls to aten::empty)
  2. There are now less operations in your trace than before (e.g., the trace complains that "there is no observable dependence" with the inputs)

  [C++] arguments in torch::nn::LinearOptions are renamed to match the Python API. (27382)

  • Arguments that are renamed:
    • in -> in_features
    • out -> out_features
    • with_bias -> bias

  [C++] arguments in torch::nn::Conv{1,2,3}dOptions are renamed to match the Python API. (28917) (29838)

  • Arguments that are renamed:
    • input_channels -> in_channels
    • output_channels -> out_channels
    • with_bias -> bias

  [C++] torch::nn::Conv{1,2,3}dOptions no longer has the transposed argument. (31005)

  • If users have transposed originally set to true in torch::nn::Conv{1,2,3}dOptions, they should migrate their code to use torch::nn::ConvTranspose{1,2,3}d layers instead.

  [C++] All Reduction enums for torch::nn layers and functionals are changed to have torch::KEnumNAME syntax. (27942, 26837)

  • Example: previously, to specify “mean” as the reduction method in a torch::nn layer or functional, we would use torch::Reduction::Mean. Now, torch::Reduction::Mean has been renamed to the shorter torch::kMean.

  [C++] torch::tensor constructor is improved to match Python API behavior. (28523) (29632) (29066)

  • 🛠 Shape checking fixes
    • Example 1: previously, torch::tensor({{1}, {2}}) produced a tensor of sizes {2}. Now, it produces a tensor of sizes {2, 1}.
    • Example 2: previously, torch::tensor(1.1) produced a 1-dim tensor. Now it produces a 0-dim tensor.
  • Type inference improvements
    • Example 1: previously, C++ torch::tensor with a double (e.g. torch::tensor(1.1)) or a (nested) braced-init-list of doubles (e.g. torch::tensor({{1.1, 2.2}}) produces a tensor with dtype torch::kDouble. Now it produces a tensor with dtype torch::get_default_dtype().
    • Example 2: previously, C++ torch::tensor with an integer type (e.g. torch::tensor(1)) or a (nested) braced-init-list of integer types (e.g. torch::tensor({{1, 2}})) produces a tensor with the same dtype. Now it always produces a tensor of dtype torch::kLong (aka. int64_t).
    • Example 3: previously, when passed a TensorOptions without a dtype set to the torch::tensor constructor, it always produces a tensor of dtype torch::get_default_dtype(). Now it produces a tensor of different dtypes based on the dtype of the braced-init-list and the default dtype.
  • Passing a std::initializer_list (NOT braced-init-list) to torch::tensor will no longer compile, and the user should pass the equivalent braced-init-list to torch::tensor instead. For example, write torch::tensor({1.1, 1.2}) instead of torch::tensor(std::initializer_list<double>({1.1, 1.2})).

  [C++] Some activation modules’ forward function now take Tensor instead of Tensor& as input. (28501)

  torch::nn layers affected: ELU / SELU / Hardtanh / LeakyReLU / ReLU / ReLU6 / RReLU / CELU
  This change ensures that the above layers can be used in a torch::nn::Sequential module. If your C++ model uses any of the above layers, you must recompile your C++ code with the new libtorch binary.

  🆕 New Features

  torch.optim

  ⏱ Learning rate schedulers (torch.optim.lr_scheduler) now support “chaining.” This means that two schedulers can be defined and stepped one after the other to compound their effect, see example below. Previously, the schedulers would overwrite each other.

  >>> import torch
  >>> from torch.optim import SGD
  >>> from torch.optim.lr_scheduler import ExponentialLR, StepLR
  >>>
  >>> model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]
  >>> optimizer = SGD(model, 0.1)
  >>>
  >>> scheduler1 = ExponentialLR(optimizer, gamma=0.9)
  >>> scheduler2 = StepLR(optimizer, step_size=3, gamma=0.1)
  >>>
  >>> for epoch in range(4):
  >>> print(epoch, scheduler2.get_last_lr()[0])
  >>>
  >>> optimizer.step()
  >>> scheduler1.step()
  >>> scheduler2.step()
  
  0 0.1
  1 0.09000000000000001
  2 0.08100000000000002
  3 0.00729000000000002
  4 0.00656100000000002
  

  Distributed

  • ➕ Add allgather_coalesced API to ProcessGroup (28634,29059)
  • ➕ Add abort API in ProcessGroupGloo Send/Recv Work (29928).
  • ➕ Add --no_python flag to allow using a bash script wrapper in the launch command (29144).

  RPC [Experimental]

  📚 torch.distributed.rpc is a newly introduced package. It contains basic building blocks to run functions remotely in model training and inference, which will be useful for scenarios like distributed model parallel or implementing parameter server frameworks. More specifically, it contains four pillars: RPC, Remote Reference, Distributed Autograd, and Distributed Optimizer. Please refer to the documentation and the tutorial for more details.

  • 🔀 Add rpc_sync and rpc_async for builtin operators and Python user functions (23228, 23569, 28392).
  • ➕ Add remote and RRef for builtin operators and Python user functions (25169, 25499).
  • Distributed Autograd - FAST mode backward pass implementation. (27022, 27576).
  • ↔ Integrate remote and RRef with distributed autograd (28630, 28656).
  • ➕ Add a distributed optimizer (29304, 30062).
  • ➕ Add python API for get_gradients() method to retrieve gradients from distributed autograd context. (28926).
  • 👌 Support creating local RRefs on local values and to-self remote calls (28948, 29634).
  • 👌 Support custom pickler for RPC (30185).
  • ➕ Add default RPC agent options based on the backend type (30201).
  • ➕ Add local shutdown to ProcessGroup agent (30330).

  JIT

  • script::Module: implement more of of the nn.Module API (28828)
    • In particular, adds the (optionally recursive) methods that iterate over submodules, parameters, etc.
    • Adds a pybind-like attr() method to simplify attribute access.
  • ➕ Add support for @staticmethod on ScriptModules (27163)
  • 👌 Support Module Containers as Iterables (26465)
  • 👌 Support Iterables In List Comprehensions (26768)
  • 👍 Dictionaries now preserve insertion order, and OrderedDict is supported (26465)
  • ➕ Add support for hasattr() (29332)
  • TorchScript classes can now be callable (26743)
  • ➕ Add clone_instance for ScriptModules (30168)
  • ➕ Add torch.memory_format support to the TorchScript (28544)
  • Custom forward() is now allowed on container modules (28988)
  • Calls to submodules are now preserved in the traced graph (29261)
  • ➕ Add support for module containers to be used as iterables (28255)
  • 👉 Make JIT Serialization support arbitrary std::function<> IO (28039)
  • 👌 Support layout()in script (27100)
  • Methods and functions are no longer inlined in the serialized file format (26706)

  Mobile

  • 🏗 Build level customization
    • Add custom build script to only include selected operators (30144).
    • Dump operator names used by a script module (29374, 30467).
    • Disable JIT optimizer in Android wrapper for mobile custom build (30285).
    • FBJNI Gradle ABI_FILTERS parameter (30135).

  👌 Improvements

  Distributed

  👌 Improvements

  • ➕ Add timeout support in ProcessGroupNCCL (27224).
  • Ensure that DDP wrapped module has parameters that require gradients (25858).
  • Making torch/csrc/cuda NCCL usage safe for NCCL 2.5 (29014).
  • ✅ Enable test_distributed for ROCm but only with NCCL backend (28814).

  RPC Improvements

  • 🔀 Separate out RPC to rpc_sync and rpc_async APIs (26570).
  • 👉 Make python user function serialization format to be consistent with builtin operators (27136).
  • Clean up distributed autograd context on all participants on exit (27951).
  • 👌 Improve error handling for distributed autograd engine. (27940).
  • Scope pybind11 functions to torch.distributed.{autograd,rpc} (27529).
  • Lift rpc_timeout to RpcAgent to make it reusable for other RpcAgent implementations. (29341).
  • Support sending message to self in process_group_agent (29253).
  • Properly shutdown RPC even in the case of clean_shutdown=False. (29148).
  • Ensure initializedContextIds_ map is cleaned up appropriately in distributed autograd engine. (29787).
  • ➕ Add hash and equality operators for WorkerInfo (29958).
  • ➕ Add RpcAgentOptions struct type to bundle arguments for different RpcAgents (29972).
  • ⏱ Mark timeout FutureMessages and throw exceptions in ProcessGroupAgent (29601).
  • 👻 Re-throw python remote exception when using remote reference to itself (29930).
  • 0️⃣ By default ignore RRef leaks during shutdown (30217).

  📚 Documentation

  • ➕ Add Design doc for Distributed Autograd Engine (29175, 30068, 29927)
  • ➕ Add Design doc for Remote Reference (30066).
  • ➕ Add documentation page for torch.distrbuted.rpc (29276, 28030, 29971, 30160, 30050, 30069, 30179, 30218, 30240, 30243, 30259).

  MISC

  • ➕ Add known worker IDs to distributed autograd context (26324).
  • Minor tweaks to RPC message API (28326).
  • 📇 Rename PythonUDF{Call,Resp} (27530).
  • 👉 Use std::shared_ptr for DistAutogradContext (29770).
  • Mark c10d::~NCCLUtils as noexcept (29118).

  JIT

  • 🚚 Move custom passes to last optimization step (29256)
  • Represent the original Python name of a module type the same way in traced and scripted modules. (29912)
  • 🖨 Only print original SourceRange on highlight (29708)
  • Error message and ergonomic improvements:
    • Show full call stack in TorchScript exception even when calls were inlined. (29911)
    • Reduce error context from 10 -> 3 (26765)
    • Fix error report highlight for unmatched type annotation (27195)
    • Make default string arguments in schemas human readable (27088)
    • Print which output didn't have dependence during trace checking. (29047)
  • 👌 Improvements to save/load and serialization performance:
    • Modules can now share JIT types if their implementation is the same, improving save/load performance (26666)
    • Improve float pickling speed. (28553)
    • Pickler: convert std::stringstream cases for improved performance. (29351)
    • Buffer to speed Unpickler (27727)
    • Buffer in Pickler to improve performance. (27720)
    • In torch::save() avoid zip compressing small header records. (28180)
    • String optimizations related to serialization. (28230)
  • Clean up serialized source format (28129)
  • API for finding a common ancestor block for a pair of nodes (28864)
  • 👉 Make inserted child module names unique (27237)
  • 👍 Better hashing for constant pool (27733)
  • 👌 Improve error messages when a method or attribute is missing (27110)
  • 🖨 Display original source range in Node::print (27524)
  • Always use the closure to resolve variable names (27515)

  Mobile

  • 👌 Improve Java API / JNI
    • Add module method to allow explicitly destructing native part (27090).
    • Add methods to write image tensor content to buffer (27359).
    • Various improvements to Android API (27454, 27455).
    • Add support for PyTorch JNI build (29412, 42faf961c8, d22f61432d).
    • Various fixes to PyTorch JNI (29350, 29861, 30206, 30207).
  • 👌 Improve support for older Android NDK
    • Introduce math_compat.h for older Android versions (28567).
    • Define std::strtoll for older Android (28603).
  • 👌 Improve error message, documentation, debuggability
    • Enable full error message for mobile builds (29926).
    • Update iOS README.md (27145).
    • Update Android README.md (28533).
    • Rename function parameters to avoid [-Werror,-Wshadow] (30276).
    • Fix exception message in Java Tensor (30776).
  • 👌 Improve support for benchmark and profiling
    • Add Android and iOS test app for benchmark and profiling (28405, 28406, 28469, 28622).
    • Integration with mobile benchmark in PEP (28437).
    • Subscribe for record function and if android do atrace (28708).
  • 👌 Improve build / CI
    • Improve Android Gradle build and publishing (26833, 27389, 29262, 29738).
    • Misc fixes to the Android test project (27453).
    • Improve XCode build script (27358, 28996, 29002).
    • Add testing code to iOS CI jobs (27593, 27594, 27784, 30133).
    • Misc fixes to the iOS TestApp (27591, 28356, 28809, 29247, 29962, 29963).
    • Add support for host build to pytorch_android (27662,27664).
    • Add host build Gradle publishing (29749).
    • Add mobile build CI with host toolchain (30292).

  Named Tensors

  • 👍 torch.addcdiv, torch.addcmul Added named tensor support (28975).
  • 👍 torch.{ones,zeros,full,rand,randn}_like Added named tensor support (28981).
  • 👍 torch.cdist Added named tensor support (29129).
  • 👍 torch.equal Added named tensor support (29322).
  • ➕ Added named tensor support for comparison ops (27162).
  • 🛠 Tensor.align_to Fixed error message (27221).
  • Tensor.align_to Make method-only. (27304).
  • Tensor.align_to Accept partially named tensors (27308).
  • 🛠 torch.mean(Tensor, Dimname) Fixed autograd support (29199).
  • Tensor.unflatten Fix when dim is a negative integer (#31208) (31432).
  • 🛠 Fix type errors in examples about Named Tensor (27828).

  C++ API

  🆕 New torch::nn modules

  • Convolution layers
    • torch::nn::ConvTranspose{1,2,3}d / Unfold (29721) (27809).
  • Pooling layers
    • torch::nn::AdaptiveAvgPool{1, 2, 3}d / MaxUnpool{1, 2, 3}d / LPPool{1, 2}d / FractionalMaxPool{2,3}d (26808, 26818, 26819) (26896, 26915, 27027) (27800, 28492, 29584) (29933).
  • Loss layers
    • torch::nn::HingeEmbeddingLoss / CosineEmbeddingLoss /MultiMarginLoss (27101) (27345) (27424) (27770).
    • torch::nn::TripletMarginLoss / SoftMarginloss / MultiLabelMargin / MarginRankingLoss / MultiLabelSoftMarginLoss (27713, 27956) (27660) (27659) (29000) (27669).
    • torch::nn::MSELoss / KLDivLoss / BCELoss / SmoothL1Loss / PoissonNLLLoss / BCEWithLogitsLoss (27156) (28806) (30146) (27661) (28755) (28783).
    • torch::nn::NLLLoss / CrossEntropyLoss / CTCLoss (29812) (28654).
  • Normalization Layers
    • torch::nn::LayerNorm / InstanceNorm{1,2,3}d / BatchNorm{1,2,3}d / GroupNorm / LocalResponseNorm / CrossMapLRN2d (28032) (28790) (28176, 28936) (29920) (28759) (29039).
  • Activation Layers
    • torch::nn::ELU / LeakyReLU / SELU / PReLU / ReLU / ReLU6 / RRelu / CELU / GLU (27028) (27059) (27434) (27429) (27435) (27436) (27437) (27487) (29922).
    • torch::nn::Sigmoid / LogSigmoid / LogSoftmax / Softmax / Softmax2d / Softplus / Softmin / Softsign / Softshrink / Hardshrink / Hardtanh / Tanh / Threshold (27488) (27060) (27462) (27446) (27509) (27489) (27459) (27535) (27534) (27035) (27537) (27038) (27536) (27538).
  • ⬇️ Dropout Layers
    • torch::nn::Dropout / Dropout{2, 3}d / AlphaDropout / FeatureAlphaDropout (29761) (28424).
  • Padding Layers
    • torch::nn::ReflectionPad{1, 2}d / ReplicationPad{1,2,3}d / ZeroPad2d / ConstantPad{1,2,3}d (28538) (28539) (28540) (28541).
  • Embedding layers
    • torch::nn::Embedding / EmbeddingBag (26358).
  • Linear layers
    • torch::nn::Bilinear / Flatten (26082) (28072).
  • Vision layers
    • torch::nn::Upsample / PixelShuffle (28413) (28140).

  🆕 New torch::nn::functional functions

  • Convolution functions
    • torch::nn::functional::conv{1,2,3}d / conv_transpose{1,2,3}d / fold / unfold (28917) (29721) (28732) (27809).
  • Pooling functions
    • torch::nn::functional::adaptive_avg_pool{1, 2, 3}d / lp_pool{1, 2}d / fractional_max_pool{2, 3}d / fractional_max_pool{2, 3}d_with_indices (26808, 26818, 26819) (27800, 28492) (29584) (29933).
  • Loss functions
    • torch::nn::functional::hinge_embedding_loss / multi_margin_loss / multilabel_soft_margin_loss / triplet_margin_loss / soft_margin_loss / margin_ranking_loss (27101) (27424) (27669) (27713) (27660) (29000).
    • torch::nn::functional::poisson_nll_loss / nll_loss / cross_entropy / binary_cross_entropy_with_logits (28755) (29812) (28783).
    • torch::nn::functional::l1_loss / kl_div / mse_loss / binary_cross_entropy / smooth_l1_loss / ctc_loss (27156) (28806) (30146) (27661) (28654).
  • Normalization functions
    • torch::nn::functional::layer_norm / instance_norm / clip_grad_norm_ / batch_norm / group_norm / local_response_norm / normalize (28032) (28790, 30684) (26140, 29584, 30216) (28176, 28936) (29920) (28759) (27280).
  • Activation functions
    • torch::nn::functional::elu / leaky_relu / selu / prelu / relu / relu6 / rrelu / celu / glu / gelu (27028) (27059) (27434) (27429) (27435) (27436) (27437) (27487) (29922) (28433).
    • torch::nn::functional:: log_sigmoid/ log_softmax / softmax / softplus / softmin / softsign / softshrink / hardshrink / tanhshrink / hardtanh / gumbel_softmax / threshold (27060) (27462) (27446) (27489) (27459) (27535) (27534) (27035) (27537) (27038) (28121) (27538).
  • Embedding functions
    • torch::nn::functional::embedding / embedding_bag / one_hot (28669) (29673) (27177).
  • Linear functions
    • torch::nn::functional::linear / bilinear (27382) (26082).
  • Padding functions
    • torch::nn::functional::pad (26601, 28760).
  • Vision functions
    • torch::nn::functional::affine_grid / grid_sample / interpolate / pixel_shuffle (27263) (28354) (28413) (28140).
  • Distance functions
    • torch::nn::functional::pdist (27122).
  • Utility functions
    • torch::nn::utils::clip_grad_value_ / parameters_to_vector / vector_to_parameters (28736, 29584) (30216) (29267).

  👍 AMD Support

  • 🆕 New features integration
    • Enabled RCCL Integration (23884, 27383, 27518, 29385)
    • Enabled rocTX and rocTracer Integration (27416)
    • Improved hiprtc integration (27390)
    • bfloat16 enablement (initial) on ROCm (27719)
  • 🏗 Build/CI
    • Upgrade to ROCm 2.9 (27417)
    • Upgrade ROCm CI to Python3.6 (30119, 27353)
    • Distribute hipify scripts as part of torch package (27425)
    • Build and test gfx908 architecture (27388)
    • Add torch.version.hip (29815).
    • Build fixes (29547, 29009)

  ONNX

  🚀 In PyTorch 1.4, we have mainly focused on expanding the coverage for ONNX Opset 11, and enabling exporting torchvision models. Most of the torchvision models can be exported to ONNX (Opset 11, with fixed input size), including FasterRCNN, MaskRCNN, and KeypointRCNN. We have also enhanced export support for some tensor indexing scenarios, with more enhancements to come in the next release. In addition, 20+ new PyTorch operators are enabled in ONNX exporter.

  Expanding Coverage for ONNX Opset 11

  • 👍 torch.sort/torch.topk are supported in Opset 11 (25739)
  • torch.size/torch.squeeze/torch.unsqueeze/torch.mm/torch.index_fill/torch.index_copy are supported in Opset 11 (27578)
  • torch.masked_select/torch.masked_scatter are supported in Opset 11 (25949)
  • 👍 torch.arange is supported in Opset 11 (26875)
  • avg_pool, constant_pad_nd, reflection_pad, replication_pad Support enhanced in Opset 11 (28225)
  • 👍 torch.hardtanh is supported in Opset 11 (30169)
  • Enable ONNX constant folding for opset 11 (29011)

  Exporting More Torch Operators/Models to ONNX

  • torch.remainder is enabled in exporter (24410)
  • torch.unfold is enabled in exporter (24970)
  • torch.slice/torch.select with negative index are enabled in exporter (25273, 26549)
  • torch.ones/torch.ones_like/torch.zeros/torch.zeros_like/torch.full/torch.full_like with default dtype are enabled in exporter (27577)
  • torch.unbind is enabled in exporter (27247)
  • torch.nn.functional.interpolate export is enhanced (27179, 27566, 28560, 29489)
  • torch.det is enabled in exporter (26958)
  • torch.group_norm is enabled in exporter (27071)
  • torch.meshgrid is enabled in exporter (26037)
  • torch.randn/torch.randn_like are enabled in exporter (28470, 29354)
  • torch.weight_norm enabled in exporter (28618)
  • torch.scalar_tensor is enabled in exporter (28713)
  • torch.logdet is enabled in exporter (29767)
  • torch.batch_norm 2D with affine=False is enabled in exporter (29458)
  • torch.bitshift is enabled in exporter (28210)

  ✅ Enhancing Export/Test Infra

  • ✅ Use deepcopy inputs in ONNX ORT test cases (27186)
  • Return NotImplemented from all binary math ops (27423).
  • Disabling ONNX IR v4 sematics for opset 8 or lower (28990)
  • ➕ Add ONNX tests for torchvision models (30121)
  • Keep output type information while exporting ONNX graph (25906)

  Quantization

  🐎 Quantization updates correspond to a mix of bug-fixes and feature improvements, with feature improvements adding improved operator coverage and performance improvements. We have also made a lot of progress towards enabling graph mode quantization support.

  • 🔋 Feature improvements:
    • Enabling intra-op parallelism (26692).
    • Enabling inplace relu (28710).
    • Quantized Tensor support copy (28612).
    • Add quantized torch mean implementation (27675).
    • Add quantized avg_pool2d for pytorch mobile (27631).
    • Add nn.quantized.Conv3d (29813).
    • Adding inplace quantized relu6 (29245).
    • Fast histogram observer (29790).
    • PackedSequence support for quantized LSTM (29585).
    • Improve legacy QuantizedLinear functions to reduce overhead (29773).
    • Add support for quantized operator conversion from PT to C2 via ONNX (29694).
    • enable per channel dynamic quantization (30122).
  • 👍 Scripting support:
    • Make PerChannelMinMaxObserver scriptable using torch.jit.ignore (29416).
    • Make HistogramObserver scriptable with @torch.jit.ignore (27950).
    • Fix tracing for dynamic quantized LSTM (29331).

  Visualization

  • 🛠 Fixed graph visualization: displaying proper names after recent JIT changes (30244)
  • 👌 Support logging embedding for TensorBoard visualizations to generic filesystem (27716)

  Other Improvements

  • torch.argmax/argmin Allow half type (28787).
  • torch.cuda.memory_stats / memory_summary instrumentation for CUDA memory allocator (27361).
  • torch.set_num_threads Allow calling multiple times with TBB (27190).
  • torch.set_num_threads Allow calling multiple times in parallel native (27947).
  • torch.logical_xor Allow non-bool tensors (27248).
  • torch.promote_types Nicer error message. (27941).
  • torch.batch_norm_elemt Add an out-variant (27621).
  • torch.lerp Implement derivative with respect to weight (28219).
  • 👍 torch.complex32 Add type promotion support (27929).
  • 👍 torch.unique Support bool tensors (28374).
  • torch.reshape Improve backward for viewable geometries (28901).
  • torch.lu Generalized factorization (28608).
  • torch.equal Add the intra-op parallelism (28810).
  • torch.randint Accept generator=None (29748).
  • torch.bfloat16 Enabled for cuda (27259).
  • torch.multinomial Enable for torch.half (29266).
  • nn.RNN Respect the current stream in cudnn (27026).
  • nn.RNN Preserve nonlinearity attribute (28058).
  • 👍 nn.Linear Support 0-batch size. (27211).
  • nn.functional.binary_cross_entropy implement double backwards (26983).
  • 👍 nn.AdaptiveAvgPool2d Add support for NHWC memory format (24396).
  • nn.GELU Add GELU activation (28944).
  • nn.LayerNorm Handle batch size of zero (28614).
  • 👍 nn.BatchNorm Add NHWC support on cudnn (23861).
  • 👍 nn.BatchNorm2d support torch.channels_last (28982).
  • nn.BatchNorm2d Handle empty inputs (30035).
  • nn.LayerNorm Enable the intra-op parallelism (28464).
  • nn.utils.prune Add pruning functionality (24076).
  • nn.Sequential Make iterable (28987).
  • dtype.is_signed Ability to differentiate signed dtypes (29511).
  • ⏱ optim.lr_scheduler.MultiplicativeLRAdd new multiplicative learning rate scheduler. (27254).
  • 👍 cuda.comm.scatter, gather Add channel-last support (28077).
  • at::parallel_for Choose number of OMP threads based on GRAIN_SIZE (26963).
  • 👍 Return NotImplemented from unsupported tensor arithmetic operators (26507).
  • Automatically select proper tqdm submodule (27108).
  • 📜 Pickle support for sparse tensors (27062).
  • 👍 Vectorized complex unary and binary op support. (26500).
  • 👍 Complex support for reduce and linpack ops on CPU (27653).
  • 👍 Complex support for compare and pointwise ops on CPU (28735).
  • 👉 Make PyTorch Python 3.8 compatible (29302).
  • ⚠ Buffer python warning to avoid deadlocks (26613).
  • 👉 Use NNPACK for strided convolutions. (29084).

  🐛 Bug Fixes

  Distributed

  • Ensure NCCL error handling code is disabled for NCCL versions < 2.4 (27124).
  • 🛠 Fix segmentation fault in FileStore with concurrent accesses. (28812).
  • 🛠 Fix DDP incompatibility issue with nn.MultiheadAttention (26826).

  RPC

  • ➕ Add ProcessGroupAgent termination detection algorithm (26984).
  • 🛠 Fix pybind11 warnings in Python RPC handler implementation (27284).
  • Defer creating ProcessGroupAgent listener thread until contexts are initialized (28013).
  • 🛠 Fix Python RPC handler exit crash (27251).
  • 🛠 Fix distributed autograd initialization (29069).
  • Always include autograd context id in rpc_* / remote requests (29781).
  • 👉 Make RRefContext singleton leaky, deal with module destruct order race. (30172).

  🛠 C++ API Bug Fixes

  • at::Tensor::requires_grad_ now supported (26332).
  • 👍 torch::isfinite now supported (30083).
  • torch::nn::modules_ordered_dict is deprecated (28774).
  • ➕ Add reset_parameters to torch::nn modules (29832).
  • 👍 Allow passing undefined Tensor to Module::register_parameter (27948).
  • Exclude undefined tensors in the result of Module::parameters() / named_paramters() / buffers() / named_buffers() (30626).
  • Include hierarchy information in C++ API loading error messages (28499).
  • 🛠 Fix a bug: the C++ L-BFGS optimizer does not work properly if there are one or more registered tensors with no grad in the model (27606).
  • 🚚 Use c10::variant-based enums for Nonlinearity and FanMode (27933). Support for torch::nn::init::Nonlinearity and torch::nn::init::FanMode will be removed in 1.5.

  JIT

  • 👉 Make dropout properly condition on training. (29436)
  • 🛠 Fix aten::grad to return optional list (29577)
  • 🛠 Fix torch.arange dtype
  • 🛠 Fix type sharing on loaded ScriptModules (29826)
  • 🛠 Fix type sharing between traced modules (29583)
  • 0️⃣ Check for mutable default parameters (29833)
  • 🛠 Fix tracing of autograd functions (29791)
  • Check for unrolled loop in break & continue (29474)
  • 🛠 Fix negative string indexing (22700)
  • 👉 Make jit.trace_module reentrant (29411)
  • 🛠 Fix jit outplace tracing and reapply changes to _like operators. (28839)
  • Properly guard against inheritance on TorchScript classes (28407)
  • 🛠 Fix when giving jit format warning about unsupported options (28616)
  • 🛠 Fix handling of function attributes. (28569)
  • 🛠 Fix pushLong() issue in pickler. (28057)
  • 🛠 Fix broken name mangling (27511)
  • 🛠 Fix segfault while printing value type for an error msg in emitListComprehension (27261)
  • 🛠 Fix toIValue dict iteration (26856)
  • 🛠 Fix race condition in Function::optimized_graph(). (27012)
  • Sanitize module names on legacy import (27764)
  • Python None should have its type inferred as NoneType (26665)
  • Properly set existing attributes under recursive script (27514)

  Quantization

  • Skip copy_same_type_transpose_ for quantized tensor (29609).
  • ➕ Add note that cuda quantization is not supported (27829).
  • 📇 Rename _intrinsic to intrinsic (27194).
  • 👍 Better error message for quantized dispatch (28635).
  • ⚡️ Update the misleading comments for zero_points and scale in dynamic quant linear module [1/2] (28767).
  • Avoid the misleading zero_point and scale [2/2] (28827).
  • ➕ Add the warning message for API with linear modules (28766).
  • Do not insert observers for empty sequential modules (28384).
  • 🛠 Fix the padding issue of quantized average pool operator (28260).

  Mobile

  • 🛠 Fix deadlock issues in ThreadPool (29885).
  • Disable ProfilingGraphExecutorImpl for mobile (30067).

  🛠 Other Bug fixes

  torch.kthvalue Fix CUDA shared memory out of bound access in findPattern (28989).

  torch.save Fix source files not being saved (28965).

  torch.load Fix OSError loading files larger than 2GB. (27125).

  torch.linspace clearer error message for negative step sizes. (28274).

  torch.histc Add range checks to avoid segfaults (27712).

  torch.lu Fix threadlocal issue on cpu (28546).

  torch.max_pool2d Limit tensor size to max CUDA grid size (28931).

  torch.renorm Fix a memory leak in CUDA renorm. (29873).

  torch.index_add Fix bug in atomicAdd on CUDA for some dtypes (29231).

  torch.addmm Fix handling of empty tensors (28613).

  nn.CTCLoss Fix incorrect gradient for large target sizes (27460).

  nn.functional.ctc_loss Fix incorrect gradient on cudnn (27039).

  nn.Embedding Incorrect gradient at padding_idx in cuda kernel. (27731).

  nn.LayerNorm Fix an illegal memory access error (28196).

  nn.Conv2d handle zero stride (28784).

  nn.PoissonNLLLoss Fix incorrect result with full=True (28637).

  nn.AvgPool2d fix an overflow for 2^31-1 sized inputs (30793).

  nn.RNNBase Fix an issue with use of children of RNN third party device types (28562).

  🔧 nn.Upsample Fix “invalid configuration argument” error (28927).

  nn.Upsample Fix a CUDA launch config failure (29016).

  ⏱ optim.lr_scheduler.OneCycleLR Correctly handle div_factor parameter (28217).

  🚚 PackedSequence.to Ensure all tensors are moved (27245).

  EventList.total_average Fix a regression caused by missing iadd (27498).

  Tensor.record_stream Ensure stream is recorded for shifted view tensors (27371).

  torch.hub Handle branch names containing a slash. (27960).

  🛠 Fix error handling in Magma kernels (29003).

  🛠 Fix avx for c++14 (28207).

  🛠 Fix illegal memory access thread safety issue in sparse CUDA (29426).

  🗄 Deprecations

  🚀 Python 2 support is deprecated and will not be supported in the 1.5 release.

  ⏱ torch.optim: Scheduler.step(epoch) is now deprecated; use Scheduler.step() instead. (26432)

  For example:

  >>> for epoch in range(10):
  >>> optimizer.step()
  >>> scheduler.step(epoch)
  DeprecationWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
    warnings.warn(EPOCH_DEPRECATION_WARNING, DeprecationWarning)
  

  [C++] C++11 is deprecated and will not be supported in the 1.5 release.

  [C++] Tensor::is_variable() has been deprecated. As noted in the Backwards Incompatible Changes section, the distinction between variable and non-variable has been eliminated, so this check is no longer meaningful. Generally, is_variable() will now return true except in some special circumstances (see 29653 for more details). (29653)

  [C++] torch::nn::modules_ordered_dict has been deprecated. It is generally no longer necessary and can just be removed. (28774)

  🗄 torch.jit.quantized API has been deprecated in favor of torch.quantization.quantize_dynamic (28766)

  🐎 Performance

  🐎 A benchmark suite is available to easily measure the performance of operators with a range of input shapes. The generated benchmark data fully characterize the performance of operators in terms of execution time. For more details see README.md in the benchmarks/operator_benchmark directory.

  • 🐎 torch.nn.functional.threshold, torch.nn.functional.layer_norm, torch.cdist Performance of threshold (CPU), layer norm (CUDA) and cdist operations was improved (27155,27634, 25799)
  • 🐎 torch.Tensor.fill_ Performance for half and bfloat16 types on CPU was improved (28397).
  • torch.nn.MaxPool2d implementation for channels_last format was added (24872)
  • There is a fast pass reducing the overheads of pointwise operations relying on TensorIterator under certain conditions (contiguous inputs, no broadcast) (29180).
  • Overheads of operations with scalars/number literals was improved (29915).
  • In case of type promotion on the GPU, the values are converted on the fly, without explicit casting of the full tensor (30018).
  • 🐎 reorder_dimensions in TensorIterator favors output write locality, improving overall performance when operating on discontiguous tensors (28615).
  • Float pickling speed was improved (28553).
  • GRAIN_SIZE for intra-op parallelization was unified between TH and ATen operations (28770)
  • 🐎 tensor.numel devirtualized, improving performance (27294)

• v1.4.0.a0

  October 08, 2019

• v1.3.1 Changes

  November 07, 2019

  🛠 Significant Fixes

  🛠 Type Promotion: fixed a bug where type promotion, combined with non-contiguous tensors could compute incorrect results. (28253)

  Version 1.3.0	Version 1.3.1
  >>> a = torch.tensor([[True, True],

                    [<span class="pl-c1">False</span>, <span class="pl-c1">True</span>]])
  

  # get a non-contiguous tensor >>> a_transpose = a.t() # type promote by comparing across dtypes (bool -> long) >>> a_transpose == 0 # POTENTIALLY INCORRECT VALUES | >>> a = torch.tensor([[True, True], [False, True]]) # get a non-contiguous tensor >>> a_transpose = a.t() # type promote by comparing across dtypes (bool -> long) >>> a_transpose == 0 tensor([[False, True], [False, False]]) |

  🛠 Type Promotion / Indexing: Fixed a Bug that Allowed Mixed-Dtype Indexing and assignment could lead to incorrect results. Mixed dtype operations of this form are currently disabled, as they were in 1.2. (28231)

  Version 1.3.0	Version 1.3.1
  >>> a = torch.ones(5, 2, dtype=torch.float)

  >>> b = torch.zeros(5, dtype=torch.long) >>> a[:, [1]] = b.unsqueeze(-1) >>> a # POTENTIALLY INCORRECT VALUES | >>> a = torch.ones(5, 2, dtype=torch.float) >>> b = torch.zeros(5, dtype=torch.long) >>> a[:, [1]] = b.unsqueeze(-1) RuntimeError: expected dtype Float but got dtype Long |

  🛠 torch.where(condition, x, y): fixed a bug on CPU where incorrect results could be returned if x and y were of different dtypes. Mixed dtype operations of this form are currently disabled, as they were in version 1.2. (29078)

  Version 1.3.0	Version 1.3.1
  >>> x = torch.randn(2, 3)

  >>> y = torch.randint(0, 10, (2, 3)) >>> torch.where(x < 0, x, y) tensor(...) # POTENTIALLY INCORRECT VALUES | >>> x = torch.randn(2, 3) >>> y = torch.randint(0, 10, (2, 3)) >>> torch.where(x < 0, x, y) RuntimeError: expected scalar type Float but found Long |

  🛠 Other Fixes

  • 👍 torch.argmax: fix regression on CUDA that disabled support for torch.float16 inputs. (28915)
  • NamedTensor: fix Python refcounting bug with Tensor.names. (28922)
  • 👍 Quantization: support deepcopy for quantized tensors. (28612)
  • 👍 Quantization: support nn.quantized.ReLU with inplace=True. (28710)
  • 📚 Documentation: torch.lgamma and torch.polygamma are now documented. (28964)

• v1.3.0 Changes

  October 10, 2019

  Table of Contents

  • 💥 Breaking Changes
  • Highlights
    • [Experimental]: Mobile Support
    • [Experimental]: Named Tensor Support
    • [Experimental]: Quantization support
    • Type Promotion
    • Deprecations
  • 🆕 New Features
    • TensorBoard: 3D Mesh and Hyperparameter Support
    • Distributed
    • Libtorch Binaries with C++11 ABI
    • New TorchScript features
  • 👌 Improvements
    • C++ Frontend Improvements
    • Autograd
    • New torch::nn modules
    • New torch::nn::functional functions
    • tensor Construction API
    • Other C++ Improvements
    • Distributed Improvements
    • Performance Improvements
    • JIT Improvements
    • ONNX Exporter Improvements
    • Adding Support for ONNX IR v4
    • Adding Support for ONNX Opset 11
    • Exporting More Torch Operators/Models to ONNX
    • Enhancing ONNX Export Infra
    • Other Improvements
  • 🐛 Bug Fixes
    • TensorBoard Bug Fixes
    • C++ API Bug fixes
    • JIT
    • Other Bug Fixes
  • 📚 Documentation Updates
    • Distributed
    • JIT
    • Other documentation improvements

  💥 Breaking Changes

  Type Promotion: Mixed dtype operations may return a different dtype and value than in previous versions. (22273, 26981)

  👍 Previous versions of PyTorch supported a limited number of mixed dtype operations. These operations could result in loss of precision by, for example, truncating floating-point zero-dimensional tensors or Python numbers.

  📚 In Version 1.3, PyTorch supports NumPy-style type promotion (with slightly modified rules, see full documentation). These rules generally will retain precision and be less surprising to users.

  Version 1.2	Version 1.3
  >>> torch.tensor(1) + 2.5

  tensor(3) >>> torch.tensor([1]) + torch.tensor(2.5) tensor([3]) >>> torch.tensor( True ) + 5 tensor(True) | >>> torch.tensor(1) + 2.5 tensor(3.5000) >>> torch.tensor([1]) + torch.tensor(2.5) tensor([3.5000]) >>> torch.tensor(True) + 5 tensor(6) |

  Type Promotion: in-place operations whose result_type is a lower dtype category (bool < integer < floating-point) than the in-place operand now throw an Error. (22273, 26981)

  Version 1.2	Version 1.3
  >>> int_tensor = torch.tensor(1)

  >>> int_tensor.add_(1.5) tensor(2) >>> bool_tensor = torch.tensor(True) >>> bool_tensor.add_(5) tensor(True) | >>> int_tensor = torch.tensor(1) >>> int_tensor.add_(1.5) RuntimeError: result type Float cannot be cast to the desired output type Long >>> bool_tensor = torch.tensor(True) >>> bool_tensor.add_(5) RuntimeError: result type Long cannot be cast to the desired output type Bool |

  📄 These rules can be checked at runtime via torch.can_cast.

  torch.flatten: 0-dimensional inputs now return a 1-dim tensor. (25406).

  Version 1.2	Version 1.3
  >>> torch.flatten(torch.tensor(0))

  tensor(0) | >>> torch.flatten(torch.tensor(0)) tensor([0]) |

  nn.functional.affine_grid: when align_corners = True, changed the behavior of 2D affine transforms on 1D data and 3D affine transforms on 2D data (i.e., when one of the spatial dimensions has unit size).

  Previously, all grid points along a unit dimension were considered arbitrarily to be at -1, now they are considered to be at 0 (the center of the input image).

  🚚 torch.gels: removed deprecated operator, use torch.lstsq instead. (26480).

  📌 utils.data.DataLoader: made a number of Iterator attributes private (e.g. num_workers, pin_memory). (22273)

  [C++] Variable::backward will no longer implicitly create a gradient for non-1-element Variables. Previously, a gradient tensor of all 1s would be implicitly created . This behavior matches the Python API. (26150)

  auto x = torch::randn({5, 5}, torch::requires_grad());
  auto y = x * x;
  y.backward()
  // ERROR: "grad can be implicitly created only for scalar outputs"
  

  [C++] All option specifiers (e.g. GRUOptions::bidirectional_) are now private, use the function variants (GRUOptions::bidirectional(...)) instead. (26419).

  Highlights

  👍 [Experimental]: Mobile Support

  🚀 In PyTorch 1.3, we are launching experimental support for mobile. Now you can run any TorchScript model directly without any conversion. Here are the full list of features in this release:

  • 👌 Support for full TorchScript inference on mobile;
  • Prebuilt LibTorch libraries for Android/iOS on JCenter/CocoaPods;
  • Java wrapper for Android with functionality to cover common inference cases (loading and invoking the model);
  • 👌 Support for all forward ops on mobile CPU (backward ops are not supported yet);
  • ⚡️ Some optimized fp32 operator implementations for ARM CPUs (based on Caffe2Go);
  • ⚡️ Some optimized int8 operator implementations for ARM CPUs (based on QNNPACK);

  ✅ We decided not to create a new framework for mobile so that you can use the same APIs you are already familiar with to run the same TorchScript models on Android/iOS devices without any format conversion. This way you can have the shortest path from research ideas to production-ready mobile apps.

  The tutorials, demo apps and download links for prebuilt libraries can be found at: https://pytorch.org/mobile/

  🚀 This is an experimental release. We are working on other features like customized builds to make PyTorch smaller, faster and better for your specific use cases. Stay tuned and give us your feedback!

  👍 [Experimental]: Named Tensor Support

  👍 Named Tensors aim to make tensors easier to use by allowing users to associate explicit names with tensor dimensions. In most cases, operations that take dimension parameters will accept dimension names, avoiding the need to track dimensions by position. In addition, named tensors use names to automatically check that APIs are being used correctly at runtime, providing extra safety. Names can also be used to rearrange dimensions, for example, to support "broadcasting by name" rather than "broadcasting by position".

  Create a named tensor by passing a names argument into most tensor factory function.

  \>\>\> tensor = torch.zeros(2, 3, names=('C', 'N')) tensor([[0., 0., 0.], [0., 0., 0.]], names=('C', 'N'))
  

  Named tensors propagate names across operations.

  \>\>\> tensor.abs() tensor([[0., 0., 0.], [0., 0., 0.]], names=('C', 'N'))
  

  Rearrange to a desired ordering by using align_to .

  \>\>\> tensor = tensor.align\_to('N', 'C', 'H', 'W')\>\>\> tensor.names, tensor.shape (('N', 'C', 'H', 'W'), torch.Size([3, 2, 1, 1]))
  

  📚 And more! Please see our documentation on named tensors.

  👍 [Experimental]: Quantization support

  👍 PyTorch now supports quantization from the ground up, starting with support for quantized tensors. Convert a float tensor to a quantized tensor and back by:

  x = torch.rand(10,1, dtype=torch.float32)
  xq = torch.quantize_per_tensor(x, scale = 0.5, zero_point = 8, dtype=torch.quint8)
  # xq is a quantized tensor with data represented as quint8
  xdq = x.dequantize()
  # convert back to floating point
  

  👍 We also support 8 bit quantized implementations of most common operators in CNNs, including:

  • Tensor operations:
    • view, clone, resize, slice
    • add, multiply, cat, mean, max, sort, topk
  • Modules/Functionals (in torch.nn.quantized)
    • Conv2d
    • Linear
    • Avgpool2d, AdaptiveAvgpool2d, MaxPool2d, AdaptiveMaxPool2d
    • Interpolate
    • Upsample
  • 👍 Fused operations for preserving better accuracy (in torch.nn.intrinsic)
    • ConvReLU2d, ConvBnReLU2d, ConvBn2d
    • LinearReLU
    • add_relu

  👍 We also support dynamic quantized operators, which take in floating point activations, but use quantized weights (in torch.nn.quantized.dynamic).

  • LSTM
  • Linear

  👍 Quantization also requires support for methods to collect statistics from tensors and calculate quantization parameters (implementing interface torch.quantization.Observer). We support several methods to do so:

  • MinMaxObserver
  • MovingAverageMinMaxObserver
  • PerChannelMinMaxObserver
  • MovingAveragePerChannelMinMaxObserver
  • HistogramObserver

  👍 For quantization aware training, we support fake-quantization operators and modules to mimic quantization during training:

  • torch.fake_quantize_per_tensor_affine, torch.fake_quantize_per_channel_affine
  • torch.quantization.FakeQuantize

  👍 In addition, we also support workflows in torch.quantization for:

  • post-training dynamic quantization
  • static post training quantization
  • quantization aware training

  All quantized operators are compatible with TorchScript.

  📚 For more details, see the documentation at: https://pytorch.org/docs/master/quantization.html

  Type Promotion

  Arithmetic and comparison operations may now perform mixed-type operations that promote to a common dtype.

  This below example was not allowed in version 1.2. In version 1.3, the same code returns a tensor with dtype=torch.float32.

  >>> torch.tensor([1], dtype=torch.int) + torch.tensor([1], dtype=torch.float32)
  

  📚 See the full documentation for more details.

  • torch.result_type Provide function to determine result of mixed-type operations (26012).
  • torch.can_cast Expose casting rules for type promotion (26805).
  • torch.promote_types Expose promotion logic (26655).

  🗄 Deprecations

  nn.functional.affine_grid / nn.functional.grid_sample: USING The Align_CORNER Default value is now deprecated, because it will be changed in 1.4 release.

  🚀 The align_corner parameter was added in this release; the behavior in the previous release was equivalent to setting the parameter to True. This is also the current default value but it will be changed to False from 1.4 release. Note that using the default will trigger a warning as demonstrated below; set the value explicitly to remove the warning.

  >>> torch.nn.functional.affine_grid(torch.randn(1,2,3),
                                      (1,3,2,2))
  UserWarning: Default grid_sample and affine_grid behavior will be changed
  to align_corners=False from 1.4.0. 
  See the documentation of grid_sample for details.
  ...
  
  >>> torch.nn.functional.affine_grid(torch.randn(1,2,3),
                                      (1,3,2,2),
                                      align_corners=True)
  # NO WARNING!
  ...
  

  🗄 [C++] Deprecate torch::Tensor::data<T>() in favor of torch::Tensor::data_ptr<T>() (24847, 24886).

  🆕 New Features

  👍 TensorBoard: 3D Mesh and Hyperparameter Support

  torch.utils.tensorboard supports 3D mesh and points plus hyperparameter logging. More details can be found in the documentation for SummaryWriter with add_mesh and add_hparams.

  A simple example exercising both methods:

  from torch.utils.tensorboard import SummaryWriter
  
  vertices_tensor = torch.as_tensor([
      [1, 1, 1],
      [-1, -1, 1],
      [1, -1, -1],
      [-1, 1, -1],
  ], dtype=torch.float).unsqueeze(0)
  colors_tensor = torch.as_tensor([
      [255, 0, 0],
      [0, 255, 0],
      [0, 0, 255],
      [255, 0, 255],
  ], dtype=torch.int).unsqueeze(0)
  faces_tensor = torch.as_tensor([
      [0, 2, 3],
      [0, 3, 1],
      [0, 1, 2],
      [1, 3, 2],
  ], dtype=torch.int).unsqueeze(0)
  
  with SummaryWriter() as w:
      w.add_mesh('my_mesh', vertices=vertices_tensor, colors=colors_tensor, faces=faces_tensor)
      for i in range(5):
          w.add_hparams({'lr': 0.1*i, 'bsize': i},
                        {'hparam/accuracy': 10*i, 'hparam/loss': 10*i})
  

  Distributed

  🚀 This release adds macOS support for torch.distributed with the Gloo backend. You can more easily switch from development (e.g. on macOS) to deployment (e.g. on Linux) without having to change a single line of code. The prebuilt binaries for macOS (stable and nightly) include support out of the box.

  • ⬇️ torch.distributed.all_reduce_coalesced Support allreduce of a list of same-device tensors (24949, 25470, 24876)
  • torch.distributed.all_reduce Add bitwise reduction ops (BAND, BOR, BXOR) (26824)

  Libtorch Binaries with C++11 ABI

  🏗 We now provide Libtorch binaries for building applications compatible with the C++11 ABI. The download links for libtorch binaries with C++11 ABI can be found in https://pytorch.org/ “QUICK START LOCALLY”.

  🆕 New TorchScript features

  • ➕ Add not in support for TorchScript (23637).
  • You can now raise exceptions in one side of an if branch (23565).
  • ➕ Add torch.jit.is_scripting() API (25955).
  • 👉 Make assertions like x is not None unwrap the optional type of x (23949).
  • ➕ Add dictionary augmented assignment (+=) support to TorchScript (23639).
  • 👌 Support grad and data attribute for tensor in TorchScript (23842).
  • ➕ Add @ignore for TorchScript classes (23614).
  • 👌 Support nn.GRU in script (23266).
  • 👌 Support tensor as a key type in TorchScript (23638).
  • ➕ Add support for ModuleDict (25715).
  • Bind set_grad_enabled() into TorchScript (25350).
  • ➕ Add in membership checks for lists (25796).
  • ➕ Add tuple keyword (25474).
  • Add __getitem__ to class types (25664).
  • Add __setitem__ to class types (25750).
  • 👉 Make JIT dicts ordered, matching Python 3.6+ semantics (26465).
  • ➕ Added invert bitwise operation to TorchScript (22324).
  • ➕ Add min() and max() for lists to TorchScript (26351).
  • 👌 Support iterables and ranges in list comprehensions (26768).

  👌 Improvements

  C++ Frontend Improvements

  👍 We are on our way to better API parity between our Python and C++ frontends. Specifically, we made the following improvements:

  Autograd

  • Tensor autograd APIs
    • torch::Tensor::data Added (26008).
    • torch::Tensor::grad Don’t create a gradient for non-1-element Variables [BC-breaking] (26150).
    • torch::Tensor::is_leaf Added (26186).
    • torch::Tensor::output_nr Added (26216).
    • torch::Tensor::_version Added (26217).
  • ➕ Add support for custom autograd functions in C++ API
    • For example usage, please see the PR description and test cases in (23572, 23628, and 23803)
  • torch::autograd::backward and torch::autograd::grad (24342)
  • torch::autograd::Variable::register_hook (24393).

  🆕 New torch::nn modules

  • Containers
    • torch::nn::ModuleList (24317).
  • Linear layers
    • torch::nn::Identity (26713).
  • Convolution layers
    • torch::nn::Fold (24160).
  • Pooling layers
    • torch::nn::MaxPool1d / MaxPool2d / MaxPool3d (24860, 26521).
    • torch::nn::AvgPool1d / AvgPool2d / AvgPool3d (25800).
    • torch::nn::AdaptiveMaxPool1d / AdaptiveMaxPool2d / AdaptiveMaxPool3d (26755, 26772, 26775).
  • Loss functions
    • torch::nn::L1Loss (25902).
  • Distance functions
    • torch::nn::CosineSimilarity (26424)
    • torch::nn::PairwiseDistance (26424)

  🆕 New torch::nn::functional functions

  • Pooling functions
    • torch::nn::functional::max_pool1d / max_pool2d / max_pool3d (26262).
    • torch::nn::functional::max_pool1d_with_indices / max_pool2d_with_indices / max_pool3d_with_indices (26521).
    • torch::nn::functional::avg_pool1d / avg_pool2d / avg_pool3d (26262).
    • torch::nn::functional::adaptive_max_pool1d / adaptive_max_pool2d / adaptive_max_pool3d (26755, 26772, 26775).
    • torch::nn::functional::adaptive_max_pool1d_with_indices / adaptive_max_pool2d_with_indices / adaptive_max_pool3d_with_indices (26755, 26772, 26775).
  • Distance functions
    • torch::nn::functional::cosine_similarity (26424).
    • torch::nn::functional::pairwise_distance (26424).

  tensor Construction API

  • ➕ Add support for multidimensional inputs to torch::tensor (26210, 26890, 26756).
    • From now on, we can use torch::tensor({{1, 2}, {3, 4}}) in C++ to construct the same tensor as torch.tensor([[1, 2], [3, 4]]) in Python. Some caveats are noted in this comment.
  • ➕ Add support for bool and BFloat16 dtypes to torch::tensor (23337).

  Other C++ Improvements

  • ➕ Add torch::nn::Module::unregister_module function, for unregistering a submodule from a torch::nn::Module (26088).

  Distributed Improvements

  • ⏱ torch.distributed Detect and handle NCCL errors appropriately instead of blocking peers until timeout in ProcessGroupNCCL (25012, 25905)
  • torch.distributed Make scatter/gather arguments optional (25575)
  • torch.distributed.launch Add a -m flag to allow users to launch python modules (24910).
  • 🌲 torch.distributed Add function to get NCCL version for logging (26583).
  • ⏱ torch.distributed Add timeout parameter to connect function in TCPStore (26554).
  • ⏱ torch.distributed use timeout in connect function to prevent against infinite loop (26364).
  • 🔀 torch.nn.modules.batchnorm Allow SyncBatchNorm to run without DDP in inference mode (24815)

  🐎 Performance Improvements

  • torch.argmax/argmin Rewrite as TensorIterator reductions (26181).
  • torch.erfinv Vectorize unary operator (26629).
  • torch.sin/cos/tan Use intrinsics for trigonometric functions on CPU (26431).
  • 🛠 Fix possible deadlock in SharedCache inside a forked child proc (25158).
  • torch.qr Fix a regression (23591).
  • nn.Conv Use Caffe2's implementation of grouped depthwise 3x3 convolutions (26556).
  • nn.Conv Use parallel_for in DepthwiseConvKernel (26879).
  • nn.Conv Change shape for conv and unary ops (25477).
  • Fix pin_memory_thread not exiting quickly (23646).
  • Increase predefined_minimum_secs to reduce variation (23734).
  • ✨ Enhance Tensor indexSelect performance (23055).
  • 0️⃣ Separate input shapes to reduce default execution time (24136).
  • constraints.lower_cholesky Vectorize LowerCholeskyTransform (24131).
  • Speed up an integer to the power of a positive integer on CPU (26020).
  • [ROCm] Enable jit fusion (22872).
  • [ROCm] Use MIOpen for transpose convolutions (26172).

  JIT Improvements

  • 🏁 Enable CPU fused kernel on Windows (25578).
  • 🔦 Expose an API to iterate all the registered operators (23207).
  • Include recursive class compilations in error call stack (23454).
  • Substantial improvements to saved model format speed and size.
    • Compress debug symbols when serializing TorchScript models. (23659).
    • Compress all non-Tensor components of a serialized TorchScript model. (23723).
    • Perform string uniquing by value in pickle serialization. (23741).
    • Implement a bunch of pickle serialization features that optimize for size. (23759).
    • Implement more size-oriented opcodes in the depickler. (26454).
  • Cache node operators to speed up optimization (24827).
  • 👍 Allow forward hooks in tracing (23613).
  • ➕ Add Pickler C++ API (23241).
  • Open up AliasAnalysisKind for any ops (23810).
  • ➕ Add the ability to compile exports on traced modules (24298).
  • 👉 Make NoneType a subtype of Optional[T] (25361).

  ONNX Exporter Improvements

  🚀 In PyTorch 1.3, we have added support for exporting graphs with ONNX IR v4 semantics, and set it as default. We have achieved good initial coverage for ONNX Opset 11, which was released recently with ONNX 1.6. Further enhancement to Opset 11 coverage will follow in the next release. We have enabled export for about 20 new PyTorch operators. Also, we have focused on enabling the export for all models in torchvision. We have introduced some necessary groundwork for that in this release, e.g., accepting PyTorch models with inputs/outputs of Dict or String. We continue to work on torchvision models, such as FasterRCNN and MaskRCNN, to enable their export.

  ➕ Adding Support for ONNX IR v4

  • Provide an option to exclude the weights from model inputs (#23284)
  • 0️⃣ Make graph inputs without weights as default (#26146)

  ➕ Adding Support for ONNX Opset 11

  • 👍 Introduce ONNX Opset 11 support (#23739)
  • ➕ Add export for torch.Interpolate in Opset 11 (#24805, #27179)
  • ➕ Add export for tensor.gather, tensor.scatter and tensor.scatter_add in Opset 11 (#24790)
  • ➕ Add export for tensor.clamp in Opset 11 (#25797)
  • ➕ Add export for torch.topk and torch.sort in Opset 11 (#25739)

  Exporting More Torch Operators/Models to ONNX

  • Export torch.pixel_shuffle (#23739)
  • Export torch.multinomial (#23581)
  • Export torch.norm’s frobenius_norm (#23536)
  • Export torch.std (#22310)
  • Export torch.empty and torch.empty_like (#24166)
  • Export torch.rsqrt (#24153)
  • Export torch.log1p (#25808)
  • Export torch.unique (#25050)
  • Export torch.gelu (#24475)
  • Export tensor.index_fill and tensor.index_copy (#23052)
  • Export torch.round (#26126)
  • Export torch.baddbmm (#25738)
  • Export torch.remainder (#24410)
  • Export torch.cumsum (#24476)
  • Export tensor.size with negative axis (#26436)
  • Export RNN/LSTM with h0/c0 initial state (#22813)

  Enhancing ONNX Export Infra

  • Enable exporting PyTorch models which have Dict and String as inputs and outputs (#25889)
  • Systematically solving mismatched types caused by implicit type conversion for binary arithmetic operators by adding an ONNX type conversions pass. (#24378)
  • Correctly validate dynamic axes names. (#23974)
  • ✅ Enable ONNX Runtime tests for Opset 10 and partially for Opset 11 (#22993)

  Other Improvements

  • Error checking: many operators now perform strides check of the output tensor and errors if it contains inner overlaps that would result in incorrect result (23063).
  • torch.det/logdet/slogdet Allowing batching (22909).
  • torch.logical_not Add new operator (23839).
  • torch.logical_xor Add new operator (23847).
  • ⚡️ torch.symeig Improve the stability of gradient updates (23018).
  • torch.eye Enable for bool and half (24148).
  • torch.tril / triu Enable for bool and half (24163).
  • 👍 torch.logical_not/xor support non-bool tensors. (23916, 23978).
  • 📜 torch.index_select Implement indexing methods for sparse tensors (24937).
  • torch.lu_solve Enable broadcasting of batch dimensions (24333).
  • torch.cholesky Enable batches greater than 262140 (24438).
  • torch.det Simplify generation of singular matrices to avoid numerical issue on PowerPC (25773).
  • torch.erfinv In the CUDA implementation, use erfinv() for double to preserve accuracy (25337).
  • torch.erfinv Add a float version of erfinv on CPU (26070).
  • ⚡️ torch.cuda.stream Updates autograd engine to respect streams set in forward (8354).
  • torch.backends.mkldnn.enabled Allow disabling MKLDNN at runtime (25459).
  • torch.cholesky_solve Add derivative (26185).
  • torch.cholesky_inverse Add derivative (26451).
  • torch.polygamma Ensure that n is non-negative (26294).
  • torch.pinverse Enable batching (26095).
  • torch.digamma/trigamma Fix type mismatches on CUDA (25791).
  • torch.where Enable for bool tensor on CUDA (26430).
  • 0️⃣ torch.load default encoding change to 'utf-8' (26421).
  • torch.repeat_interleave Respect the current stream (26946).
  • torch.bernoulli_ Implement for bool tensors (25076).
  • torch.norm Fix nuclear norm with requires_grad=True (26303).
  • torch.hub.download_url_to_file Make function public (26723).
  • nn.modules.conv add padding_mode to repr (23996).
  • 👍 nn.Transformer Extend to support BERT (gelu) (24181).
  • 👍 nn.BatchNorm2d Add support for non-affine batch norm with float stats and half inputs (22750).
  • nn.Parameter Fix type hints (25586).
  • nn.CTCLoss Improve error message (26325).
  • nn.Conv Allow batch size of 0 (26214).
  • nn.LSTM/GRU enable double backward for non-cudnn (26660).
  • optim.Adagrad Add epsilon argument (24980).
  • 0️⃣ optim.LBFGS Change default tolerance_grad to 1e-7 (25240).
  • ⏱ optim.lr_scheduler.OneCycleLR Add new 1cycle learning rate scheduler (25324).
  • ⚡️ optimizer.step Fix type annotation (26930).
  • 👍 bfloat16 Add support for sub, mul, and div on CPU (22851).
  • bfloat16 Enabled comparison ops on CPU (24182).
  • bfloat16 Enabled masked methods (24183).
  • bfloat16 Enabled torch.mm and torch.mv (24224).
  • 🌲 bfloat16 Enable log_softmax and CrossEntropyLoss (24457).
  • bfloat16 Enabled conv methods (26167).
  • bfloat16 Enabled dtype on CUDA (26407).
  • 👀 quasirandom.SobolEngine Use random seed if not specified (24884).
  • utils.data.dataloader Add possible out of shared memory error message (25730).
  • cuda.set_rng_state Add type hint (26200).
  • 👍 Zero sized tensor support for repeat_interleave (23717).
  • Recommend ~ and bitwise_not() when user tries to apply neg (-) on a bool tensor. (23621).
  • 🛠 Fix double backward of inplace op on view (23502).
  • autograd.grad Validate shapes of outputs (25349).
  • Enable libflame as a LAPACK choice (25795).
  • 🛠 Fix race condition in CUDA initialization (25788).
  • ⚡️ Include iteration_ in SGD optimizer serialization (26906).
  • [C++] torch::tensor Fix an ambiguous overload issues in constructor (26890).
  • [XLA] Check device before accessing data_ptr in PackLayer (26056).
  • [XLA] Allow overwriting catch-all kernels (25947).

  🐛 Bug Fixes

  🛠 TensorBoard Bug Fixes

  • SummaryWriter.add_graph: Fix empty graph output in some cases (25599).
  • ⚡️ Update Caffe2 contrib TensorBoard logging to not require TensorFlow (25259).
  • SummaryWriter.make_video: Fix write_gif call to moviepy for newer lib (21218).

  🛠 C++ API Bug fixes

  • 🛠 Fixes mismatch of device and data type when computing step_size in LBFGS optimizer (25909).

  JIT

  • 🛠 Fix list comprehension that change the type of the original iterable (24271).
  • 🛠 Fix double copying of constants during recursive scripting (24412).
  • 🛠 Fix frontend error message (23576).
  • Clear recursive error stack on each compilation (23458).
  • 🛠 Fix bugs in assignment to optionals (25059).
  • 👉 Make torch.jit.Attribute work when PYTORCH_ENABLED=0 (23851).
  • 🛠 Fix unicode in comments causing compilation errors (24218).
  • Correctly raise an error if an nn.Module has not been initialized but you try to script it (24852).
  • 🛠 Fix annotated assignment to variables (25094).
  • dictPop: dereference dict.find() iterator before calling dict.erase() (25056).
  • 🛠 fix closures which always throw. (25278).
  • ➕ Add source location to class instantiation error (24990).
  • 🛠 Fix AliasAnalysisKind::PURE on MSVC (25375).
  • Emit script function calls during tracing. (25089).
  • Resolve NamedTuple types properly in Python (26443).
  • 🛠 Fix schema matching of tuples to vartype lists (25944).
  • Correctly preserve ignored function return value type (25262).
  • 🛠 Fix missing newline in compiled from source range highlight (25802).
  • 🛠 Fix use-after-free bug in optional (25965).
  • 🛠 Fix torch.arange traced as constant (25363).
  • Preserve module names in recursive script (24505).
  • Properly resolve ignored module method type annotations (26683).
  • 👉 Make is_optional check more robust (26312).
  • 🛠 Fix builtin lookup for Python functions (26688).
  • Typevar matching fix + implicit conversions from Scalar to int/float (26453).
  • 🛠 Fix range for non-int inputs and pow implementation (26926).

  🛠 Other Bug Fixes

  • 📌 torch.is_pinned pin_memory should not copy on already pinned tensors (23484).
  • torch.cdist Fix incorrect gradients on CUDA non-batch tensors (22915).
  • 🏁 torch.from_numpy Fix failure on windows for int32 (25139).
  • torch.tensor Fix memory leak creating a tensor from numpy (24267).
  • torch.index Don't save self in index backward (25594).
  • torch.bincount Fix int32 overflow on CUDA (25748).
  • torch.bernoulli Fix the distribution sampler (26864).
  • torch.pow Fix precision (25476).
  • torch.cdist Fix gradient computation when first arg is 1xn (26254).
  • ➕ torch.scatter_add_ Fix scatter CPU kernel when (input size, src size) > index size (25839).
  • 🛠 nn.ConvTranspose2d Fixed an error with float16 inputs and weights on CUDA. (23552).
  • nn.CTCLoss Fix zero-length targets on CUDA (23298).
  • nn.Conv2d Correct an overflow in an error message (25146).
  • optim.Adam apply a small mathematical fix. (23737).
  • 👷 dataloader Fix IndexError on shutdown if not all workers are started (23761).
  • Tensor.repeat Fix crash on for 0 repeats (23766).
  • torch.pin_memory only use one thread (25111).
  • distributions.Uniform,HalfCauchy,Gamma Fix log_prob when value is a float (23017).
  • 🛠 Fix typing error for Padding with asymmetric signatures (24895).
  • Avoid race condition in intrusive_ptr.reset_() (24464).
  • torch.hub: Fix SSL cert issue for hub in Python 2 (25042).
  • 🛠 Fix int overflow issue in CUDA kernels. (24818).
  • Module.cuda Fix type hints (25018).
  • 🛠 Fix bug in assertNotEqual for int tensors (25412).
  • 🛠 Fix 'in' return true incorrectly (24156).
  • 🛠 Fix bugs in bulk loader when batch_size=None or with namedtuple (26065).
  • 🛠 Fix serialization issue in big endian arch (26383).
  • 🛠 Fix Vec256::abs() for floating point when applied on -0.0 (26422).
  • 🛠 Fix cyclic reference in _LRScheduler (25776).
  • 🛠 Fix a build failure on s390x (26233).
  • [XLA] Fix tensor construction from array (24283).

  📚 Documentation Updates

  Distributed

  • torch.distributed Error phrasing in torch.distributed helper functions (25574)
  • torch.distributions.negative_binomial clarified ambiguous doc string in NegativeBinomial (25923)

  JIT

  • ➕ Add technical documentation for the serialization format (23456).
  • 🛠 Fix trace docs (24191).
  • ➕ Add trace_module to docs (24258).
  • Cleanup distinction around script and trace (24208).
  • 🛠 Fix item() call in docs (25404).
  • ⚡️ Misc doc updates / fixes (24371, 24445).

  📚 Other documentation improvements

  • 📚 torch.record_stream Add documentation (24078).
  • torch.fold Describe the relation between fold and unfold operations (24840).
  • torch.argmax Fix incorrect doc (23775).
  • 📄 torch.random add docs (23553).
  • 📄 torch.empty_strided Add docs (23735).
  • torch.bitwise_not Document for bool tensors (23800).
  • 📚 torch.cdist Add documentation (25221).
  • ⚡️ torch.where Update parameter names in doc (25554).
  • torch.atan2 Clarify and correct the doc (26180).
  • 📚 nn.functional.bilinear Added documentation (24951).
  • nn.functional.upsample Fix align_corners doc (23707).
  • 🛠 nn.Transformer Fixed an error in the example (24837).
  • 📚 optim.lr_scheduler.CosineAnnealingWarmRestarts Add documentation (25421).
  • ⚡️ optim.SGD Updated with subscripts (23985).
  • optim.RMSprop Highlighting in the doc that square root comes before adding epsilon (26735).
  • ⚠ autograd.detect_anomaly Add a warning (26615).
  • 👌 Improve dataloader docs on when auto-batching is disabled (23671).
  • ⚡️ Updated docs and added deprecation warnings to acknowledge a bool tensor (22261).
  • Document benchmarking practice for CUDA (23910).
  • ➕ Add ASAN instructions to CONTRIBUTING.md (24848).

• « First
• ‹ Prev
• 1
• 2
• 3
• 4
• 5
• Next ›
• Last »