Backend是MNN对计算设备的抽象。MNN当前已经支持CPU、Vulkan、OpenCL、Metal等Backend，只在计算设备暂未支持时新增Backend，新增Op，请参阅新增Op文档。

声明

所有新增Backend都需继承Backend类，并实现所有纯虚函数。

  1. class XPUBackend final : public Backend {
  2.   XPUBackend(MNNForwardType type, MemoryMode mode);
  3.   virtual ~XPUBackend();
  4.   virtual Execution* onCreate(const std::vector<Tensor*>& inputs, const std::vector<Tensor*>& outputs, const MNN::Op* op) override;
  5.   virtual void onExecuteBegin() const override;
  6.   virtual void onExecuteEnd() const override;
  7.   virtual bool onAcquireBuffer(const Tensor* tensor, StorageType storageType) override;
  8.   virtual bool onReleaseBuffer(const Tensor* tensor, StorageType storageType) override;
  9.   virtual bool onClearBuffer() override;
  10.   virtual void onCopyBuffer(const Tensor* srcTensor, const Tensor* dstTensor) const override;
  11. }

构造与销毁

Backend构造时，可以额外指定内存环境，在内存受限环境中，应避免非必要的内存使用。可以在构造函数中，完成对计算设备访问的必要初始化，如GPU下预加载shader等。

  1. /** backend memory mode */
  2. enum MemoryMode {
  3.     /** use memory without limit. */
  4.     NORMAL = 0,
  5.     /** use memory thriftily. */
  6.     LIMIT = 1
  7. };
  8. /**
  9.  * @brief initializer.
  10.  * @param type  forward type.
  11.  * @param mode  memory mode.
  12.  */
  13. Backend(MNNForwardType type, MemoryMode mode = NORMAL);

Execution创建

Backend需要通过onCreate为op创建出exection实例：

  1. virtual Execution* onCreate(const std::vector<Tensor*>& inputs, const std::vector<Tensor*>& outputs, const MNN::Op* op) override;

可以在方法内根据op类型创建，但更建议提供注册接口：

  1. class XPUBackend final : public Backend {
  2.     // ...
  3.     class Creator {
  4.     public:
  5.         /**
  6.          * @brief create execution for given input, op on metal backend.
  7.          * @param inputs    given input tensors.
  8.          * @param op        given op.
  9.          * @param backend   metal backend.
  10.          * @return created execution if supported, NULL otherwise.
  11.          */
  12.         virtual Execution *onCreate(const std::vector<Tensor *> &inputs, const MNN::Op *op,
  13.                                     Backend *backend) const = 0;
  14.     };
  15.     /**
  16.      * @brief register creator for given op type.
  17.      * @param type      given op type.
  18.      * @param creator   registering creator.
  19.      */
  20.     static void addCreator(OpType type, Creator *creator);
  21.     // ...
  22. };
  23. template <class T>
  24. class XPUCreatorRegister {
  25. public:
  26.     /**
  27.      * @brief initializer. register T creator for given op type.
  28.      * @param type  given op type.
  29.      */
  30.     XPUCreatorRegister(OpType type) {
  31.         T *test = new T;
  32.         XPUBackend::addCreator(type, test);
  33.     }
  34. };

这样，Op Execution中，就可以通过注册追加Op类型：

  1. class XPUPoolingCreator : public XPUBackend::Creator {
  2. public:
  3.     virtual Execution *onCreate(const std::vector<Tensor *> &inputs, const MNN::Op *op, Backend *backend) const {
  4.         return new XPUPooling(backend, op->main_as_Pool());
  5.     }
  6. };
  7. static XPUCreatorRegister<XPUPoolingCreator> __reg(OpType_Pooling);

内存管理

Backend通过onAcquireBuffer为tensor分配内存，通过onReleaseBuffer为tensor释放内存。内存有三种存储模式：STATIC内存不复用，一般用于op常量存储；DYNAMIC内存可复用，一般用于变量存储；DYNAMIC_SEPERATE内存在pipeline间可复用，一般用于pipeline常量存储。onAcquireBuffer和onReleaseBuffer中可以不实际分配/释放内存，只记录内存用量变更，在onAllocateBuffer调用时，再根据用量计算出优化方案，一次性完成分配/释放。

  1. /** backend buffer storage type */
  2. enum StorageType {
  3.     /**
  4.      use NOT reusable memory.
  5.      - allocates memory when `onAcquireBuffer` is called.
  6.      - releases memory when `onReleaseBuffer` is called or when the backend is deleted.
  7.      - do NOTHING when `onClearBuffer` is called.
  8.      */
  9.     STATIC,
  10.     /**
  11.      use reusable memory.
  12.      - allocates or reuses memory when `onAcquireBuffer` is called. prefers reusing.
  13.      - collects memory for reuse when `onReleaseBuffer` is called
  14.      - releases memory when `onClearBuffer` is called or when the backend is deleted.
  15.      */
  16.     DYNAMIC,
  17.     /**
  18.      use NOT reusable memory.
  19.      - allocates memory when `onAcquireBuffer` is called.
  20.      - do NOTHING when `onReleaseBuffer` is called.
  21.      - releases memory when `onClearBuffer` is called or when the backend is deleted.
  22.      */
  23.     DYNAMIC_SEPERATE
  24. };
  25. /**
  26.  * @brief allocate buffer of tensor for given storage type.
  27.  * @param tensor        buffer provider.
  28.  * @param storageType   buffer storage type.
  29.  * @return success or not.
  30.  */
  31. virtual bool onAcquireBuffer(const Tensor* tensor, StorageType storageType) = 0;
  32. /**
  33.  * @brief release buffer of tensor for given storage type.
  34.  * @param tensor        buffer provider.
  35.  * @param storageType   buffer storage type.
  36.  * @return success or not.
  37.  */
  38. virtual bool onReleaseBuffer(const Tensor* tensor, StorageType storageType) = 0;

在所有内存都分配完成后，backend会收到onAllocateBuffer回调：

  1. /**
  2.  * @brief callback after all buffers needed by backend ops were allocated.
  3.  * @return success or not. (result not used currently)
  4.  */
  5. virtual bool onAllocateBuffer() {
  6.     return true;
  7. }

Backend在调用onClearBuffer时，需要释放所有DYNAMIC和DYNAMIC_SEPERATE存储模式的内存：

  1. /**
  2.  * @brief clear all dynamic buffers.
  3.  * @return success or not.
  4.  */
  5. virtual bool onClearBuffer() = 0;

此外，backend还需要负责tensor数据的拷贝：

  1. /**
  2.  * @brief copy buffer from tensor to tensor.
  3.  * @param srcTensor source buffer provider.
  4.  * @param dstTensor dest buffer provider.
  5.  */
  6. virtual void onCopyBuffer(const Tensor* srcTensor, const Tensor* dstTensor) const = 0;

拷贝可能在backend内部，也可能在backend与CPU backend之间。
拷贝需要处理Tensor间的布局转换，相同布局时，可以直接拷贝数据；不同布局，如NHWC和NC4HW4，则一般需要做特殊转换。

Pipeline回调

Backend在pipeline执行的各个周期都会收到回调，onResizeBegin和onResizeEnd在调整内存分配前后调用（op的onResize会在此间调用）；onExecuteBegin和onExecuteEnd在op执行前后调用（op的onExecute会在此间调用）；onWaitFinish相对特殊，由用户主动调用，异步执行的pipeline需要同步等待完成。

  1. /**
  2.  * @brief callback before resize ops.
  3.  */
  4. virtual void onResizeBegin();
  5. /**
  6.  * @brief callback after resize ops.
  7.  */
  8. virtual void onResizeEnd();
  9. /**
  10.  * @brief callback before executing ops.
  11.  */
  12. virtual void onExecuteBegin() const = 0;
  13. /**
  14.  * @brief callback after executing ops.
  15.  */
  16. virtual void onExecuteEnd() const = 0;
  17. /**
  18.  * @brief wait for all async execution to be finished.
  19.  * @return success or not.
  20.  */
  21. virtual bool onWaitFinish();

注册Backend

最后，定义Backend Creator，注册方法中调用MNNInsertExtraBackendCreator就可以完成Backend的注册，这里的注册方法需要在BackendRegister.cpp中声明并调用：

  1. class XPUBackendCreator : public BackendCreator {
  2.     virtual Backend *onCreate(const Backend::Info &info) const {
  3.         return new MetalBackend;
  4.     }
  5. };
  6. void registerCPUBackendCreator() {
  7.     MNNInsertExtraBackendCreator(MNN_FORWARD_CPU, new CPUBackendCreator);
  8. };

使用cmake编译时，完成代码修改后，也需要相应修改CMakeLists.txt。