Overview

There are two levels of abstraction in MNN for inference, Interpreter and Session. Interpreter is the holder of the model data; Session is created by Interpreter and is the holder of the inference data. Multiple inferences can share the same model data, that is, multiple Sessions can share an Interpreter.

When no more session needs to be created and no more training data needs to be updated, Interpreter can release model data through the releaseModel function to save memory.

Create Interpreter

There are two ways to create an Interpreter:

  • Create from disk file

    1. /**
    2. * @brief create net from file.
    3. * @param file  given file.
    4. * @return created net if success, NULL otherwise.
    5. */
    6. static Interpreter* createFromFile(const char* file);

  • Create from memory buffer

    1. /**
    2. * @brief create net from buffer.
    3. * @param buffer    given data buffer.
    4. * @param size      size of data buffer.
    5. * @return created net if success, NULL otherwise.
    6. */
    7. static Interpreter* createFromBuffer(const void* buffer, size_t size);

The Interpreter instance returned by the function is created by new, and must be released by delete when it is no longer needed to avoid memory leaks.

Create Session

Sessions are typically created via Interpreter::createSession:

  1. /**
  2.  * @brief create session with schedule config. created session will be managed in net.
  3.  * @param config session schedule config.
  4.  * @return created session if success, NULL otherwise.
  5.  */
  6. Session* createSession(const ScheduleConfig& config);

The Session instance returned by the function is managed by Interpreter. It must be released by calling Interpreter::releaseSession when it is no longer needed to avoid memory leaks.

Simple Usage

In general, there is no need to set additional scheduling configuration. The function will automatically recognize the scheduling path, input and output according to the model structure, for example:

  1. ScheduleConfig conf;
  2. Session* session = interpreter->createSession(conf);

Scheduling configuration

The scheduling configuration is defined as follows:

  1. /** session schedule config */
  2. struct ScheduleConfig {
  3.     /** which tensor should be kept */
  4.     std::vector<std::string> saveTensors;
  5.     /** forward type */
  6.     MNNForwardType type = MNN_FORWARD_CPU;
  7.     /** number of threads in parallel */
  8.     int numThread = 4;
  10.     /** subpath to run */
  11.     struct Path {
  12.         std::vector<std::string> inputs;
  13.         std::vector<std::string> outputs;
  15.         enum Mode {
  16.             /**
  17.              * Op Mode
  18.              * - inputs means the source op, can NOT be empty.
  19.              * - outputs means the sink op, can be empty.
  20.              * The path will start from source op, then flow when encounter the sink op.
  21.              * The sink op will not be compute in this path.
  22.              */
  23.             Op = 0,
  25.             /**
  26.              * Tensor Mode (NOT supported yet)
  27.              * - inputs means the inputs tensors, can NOT be empty.
  28.              * - outputs means the outputs tensors, can NOT be empty.
  29.              * It will find the pipeline that compute outputs from inputs.
  30.              */
  31.             Tensor = 1
  32.         };
  34.         /** running mode */
  35.         Mode mode = Op;
  36.     };
  37.     Path path;
  39.     /** backup backend used to create execution when desinated backend do NOT support any op */
  40.     MNNForwardType backupType = MNN_FORWARD_CPU;
  42.     /** extra backend config */
  43.     BackendConfig* backendConfig = nullptr;
  44. };

In the inference, the main selection backend is specified by type, and the default is CPU. The alternate backend specified by backupType, and is enabled when the main selection backend does not support operators in the model.

The inference path includes all operators from inputs to outputs of the path. When not specified, it is automatically identified based on the model structure. To save memory, MNN reuses tensor memory excluding outputs. If you need to preserve the results of the intermediate tensor, you can use saveTensors to avoid memory reuse.

When inferring, the number of threads can be modified by numThread. But the real number of threads depends on the deployment environment:

  • iOS, GCD is used, ignores the configuration;
  • When MNN_USE_THREAD_POOL is enabled, the number of threads depends on the number of threads configured for the first time;
  • OpenMP, the number of threads is set globally, and the actual number of threads depends on the number of threads configured last time;

In addition, additional parameters for the backend can be set via backendConfig. See below for details.

Backend Configuration

The backend configuration is defined as follows:

  1. struct BackendConfig {
  2.     enum MemoryMode {
  3.         Memory_Normal = 0,
  4.         Memory_High,
  5.         Memory_Low
  6.     };
  8.     MemoryMode memory = Memory_Normal;
  10.     enum PowerMode {
  11.         Power_Normal = 0,
  12.         Power_High,
  13.         Power_Low
  14.     };
  16.     PowerMode power = Power_Normal;
  18.     enum PrecisionMode {
  19.         Precision_Normal = 0,
  20.         Precision_High,
  21.         Precision_Low
  22.     };
  24.     PrecisionMode precision = Precision_Normal;
  26.     /** user defined context */
  27.     void* sharedContext = nullptr;
  28. };

memory, power, and precision are memory, power, and precision preferences, respectively. Backends that support these options are adjusted as they are executed; if any option is not supported, it is ignored.

The sharedContext is used when customizing backend, and users can assign values according to their needs.

Create Session with Multiple Paths

When you need a more complex configuration on inference paths, you can create the session with scheduling configuration group:

  1. /**
  2.  * @brief create multi-path session with schedule configs. created session will be managed in net.
  3.  * @param configs session schedule configs.
  4.  * @return created session if success, NULL otherwise.
  5.  */
  6. Session* createMultiPathSession(const std::vector<ScheduleConfig>& configs);

**
Each scheduling configuration can configure paths and options independently .