1 Tensor在Python层面的继承体系
2 Tensor在C++层面的继承体系
- 2.1 Storage
- 2.2 autogradmeta

本文参考

PyTorch的Tensor（上）
PyTorch源码浅析(5)：Python扩展
PYTORCH C++ API

本文基于PyTorch1.7.0，https://github.com/pytorch/pytorch/tree/v1.7.0 如果本文有不清楚或者不正确的地方，请在评论区指正

1 Tensor在Python层面的继承体系

[torch/tensor.py, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/tensor.py#L40]
class Tensor(torch._C._TensorBase):
    ...
[torch/nn/parameter.py, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/nn/parameter.py#L6]
class Parameter(torch.Tensor):
    ...

torch._C._TensorBase是Tensor的父类，_TensorBase通过THPVariable_initModule(module)添加到torch._C。torch._C模块的初始化见本专栏PyTorch初始化—initModule。
接着往下分析THPVariable_initModule(module)

[torch/csrc/autograd/python_variable.cpp, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/csrc/autograd/python_variable.cpp#L787]
bool THPVariable_initModule(PyObject *module)
{
  static std::vector<PyMethodDef> methods;
  THPUtils_addPyMethodDefs(methods, torch::autograd::variable_methods);
  THPUtils_addPyMethodDefs(methods, extra_methods);
  THPVariableType.tp_methods = methods.data();
  if (PyType_Ready(&THPVariableType) < 0)
    return false;
  Py_INCREF(&THPVariableType);
  PyModule_AddObject(module, "_TensorBase",   (PyObject *)&THPVariableType);
  torch::autograd::initTorchFunctions(module);
  torch::autograd::initTensorImplConversion(module);
  return true;
}

可以看到，在PyModule_AddObject(module, “_TensorBase”, (PyObject *)&THPVariableType)往module中（即torch._C）注册了_TensorBase类。THPUtils_addPyMethodDefs(methods, torch::autograd::variable_methods)往THPVariableType中添加了variable_methods。

[torch/csrc/autograd/generated/python_variable_methods.cpp]
PyMethodDef variable_methods[] = {
  // These magic methods are all implemented on python object to wrap NotImplementedError
  {"__add__", castPyCFunctionWithKeywords(TypeError_to_NotImplemented_<THPVariable_add>), METH_VARARGS | METH_KEYWORDS, NULL},
  {"__radd__", castPyCFunctionWithKeywords(TypeError_to_NotImplemented_<THPVariable_add>), METH_VARARGS | METH_KEYWORDS, NULL},
  {"__iadd__", castPyCFunctionWithKeywords(TypeError_to_NotImplemented_<THPVariable_add_>), METH_VARARGS | METH_KEYWORDS, NULL},
  ...
};

2 Tensor在C++层面的继承体系

在上面我们知道Python层面的Tensor类最终对应到C++层面的THPVariableType。接下来将介绍Tensor在C++层面的继承体系。

[torch/csrc/autograd/python_variable.cpp, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/csrc/autograd/python_variable.cpp#L723]
PyTypeObject THPVariableType = {
  PyVarObject_HEAD_INIT(nullptr, 0)
  "torch._C._TensorBase",                      /* tp_name */
  sizeof(THPVariable),                         /* tp_basicsize */
  ...
  (destructor)THPVariable_dealloc,             /* tp_dealloc */
  ...
  THPVariable_pynew                            /* tp_new */
};

Tensor类对应THPVariableType，即类型对象，Tensor类实例化之后的对象对应THPVariable，即实例对象。【见cpython类型对象和实例对象的分析】

[torch/csrc/autograd/python_variable.h, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/csrc/autograd/python_variable.h#L12]
struct THPVariable {
    PyObject_HEAD
    // Payload
    torch::autograd::Variable cdata;
    // Hooks to be run on backwards pass (corresponds to Python attr
    // '_backwards_hooks', set by 'register_hook')
    PyObject* backward_hooks = nullptr;
};

THPVariable的关键字段是cdata，对应的类型是torch::autograd::Variable。

[torch/csrc/autograd/variable.h, https://github.com/pytorch/pytorch/blob/v1.7.0/torch/csrc/autograd/variable.h#L31]
using Variable = at::Tensor;

因此，可以看到THPVariable依赖的Variable本质上是at::Tensor。

at::Tensor在build/aten/src/Aten/core/TensorBody.h 中定义。

[build/aten/src/Aten/core/TensorBody.h]
// Tensor is a "generic" object holding a pointer to the underlying TensorImpl object, which
// has an embedded reference count.
class TORCH_API Tensor {
 ...
 protected:
  c10::intrusive_ptr<TensorImpl, UndefinedTensorImpl> impl_;
};

Tensor中存储着指向TensorImpl的指针impl_。

[c10/core/TensorImpl.h]
/**
 * The low-level representation of a tensor, which contains a pointer
 * to a storage (which contains the actual data) and metadata (e.g., sizes and
 * strides) describing this particular view of the data as a tensor.
 */
struct C10_API TensorImpl : public c10::intrusive_ptr_target {
  ...
protected:
  ...
  Storage storage_;

private:
  // This pointer points to an AutogradMeta struct that stores autograd-specific fields
  // (such as grad_ / grad_fn_ / grad_accumulator_).
  // This pointer always has unique ownership (meaning only one TensorImpl can own it
  // at a time).
  std::unique_ptr<c10::AutogradMetaInterface> autograd_meta_ = nullptr;

protected:
  std::unique_ptr<c10::NamedTensorMetaInterface> named_tensor_meta_ = nullptr;
  c10::VariableVersion version_counter_;
  PyObject* pyobj_ = nullptr;
  SmallVector<int64_t,5> sizes_;
  SmallVector<int64_t,5> strides_;
  int64_t storage_offset_ = 0;
  int64_t numel_ = 1;
  caffe2::TypeMeta data_type_;
  bool reserved_ : 1;
};

TensorImpl中比较重要的两个属性storage和autograd_meta，前者用于数据存储，后者用于自动微分。

2.1 Storage

[c10/core/Storage.h]
struct C10_API Storage {
 ...
 protected:
  c10::intrusive_ptr<StorageImpl> storage_impl_;
};

Storage保存了指向StorageImpl的指针storageimpl。

StorageImpl

// https://github.com/pytorch/pytorch/blob/v1.7.0/c10/core/TensorImpl.h#L316
struct C10_API StorageImpl final : public c10::intrusive_ptr_target {
private:
DataPtr data_ptr_;
size_t size_bytes_;
bool resizable_;
bool received_cuda_;
Allocator* allocator_;
};

StorageImpl中的dataptr指向DataPtr

[c10/core/Allocator.h]
// A DataPtr is a unique pointer (with an attached deleter and some
// context for the deleter) to some memory, which also records what
// device is for its data.
//
// nullptr DataPtrs can still have a nontrivial device; this allows
// us to treat zero-size allocations uniformly with non-zero allocations.
//
class C10_API DataPtr {
private:
c10::detail::UniqueVoidPtr ptr_;
Device device_;
};

[c10/util/UniqueVoidPtr.h]
class UniqueVoidPtr {
private:
// Lifetime tied to ctx_
void* data_;
std::unique_ptr<void, DeleterFnPtr> ctx_;
};

DataPtr中的ptr_指向UniqueVoidPtr。

2.2 autogradmeta

TensorImpl中另一个属性autogradmeta的类型为AutogradMetaInterface。

[c10/core/TensorImpl.h]
struct C10_API AutogradMetaInterface {
virtual void set_requires_grad(bool requires_grad, at::TensorImpl* self_impl) = 0;
virtual bool requires_grad() const = 0;
virtual at::Tensor& mutable_grad() = 0;
virtual const at::Tensor& grad() const = 0;
virtual const at::Tensor& fw_grad(uint64_t level, const at::Tensor& self) const = 0;
virtual void set_fw_grad(const at::Tensor& new_grad, const at::Tensor& self, uint64_t level, bool is_inplace_op) = 0;
virtual ~AutogradMetaInterface();
};

实际上该autogradmeta会指向AutogradMeta ```cpp [torch/csrc/autograd/variable.h] //~~~~~~~~~~~~~~~~~~ // AutogradMeta //~~~~~~~~~~~~~~~~~~

/// Each Variable has one unique AutogradMeta struct, which stores autograd /// metadata fields that are necessary for tracking the Variable’s autograd history. /// As an optimization, a Variable may store a nullptr, in lieu of a default /// constructed AutogradMeta.

struct TORCHAPI AutogradMeta : public c10::AutogradMetaInterface { … std::string name; Variable grad; std::shared_ptr grad_fn; std::weakptr grad_accumulator; std::sharedptr fw_grad; std::vector> hooks; std::shared_ptr cpp_hooks_list; … // Only meaningful on leaf variables (must be false otherwise) bool requires_grad; /// Sets the requires_grad property of Variable. This should be true for /// leaf variables that want to accumulate gradients, and false for all other /// variables. void setrequires_grad(bool requires_grad, at::TensorImpl* self_impl) override { TORCH_CHECK( !requires_grad || isDifferentiableType(at::typeMetaToScalarType(self_impl->dtype())), “Only Tensors of floating point and complex dtype can require gradients”); requires_grad = requires_grad; }

bool requiresgrad() const override { return requires_grad || gradfn; } }; ``` 至此，Tensor的继承体系基本摸清了。
下一节将介绍一个Tensor是如何构建的。