!169 Serving, r1.1.2

From: @xu-yfei
Reviewed-by: @zhangyinxia,@zhoufeng54
Signed-off-by: @zhoufeng54

README.md

@@ -45,12 +45,15 @@ MindSpore Serving depends on the MindSpore training and inference framework. The
 
 Use the pip command to install Serving. Perform the following steps:
 
-- Download the .whl package from the MindSpore Serving page and install it.
+- Download the .whl package from the [MindSpore Serving page](https://www.mindspore.cn/versions/en) and install it.
 
     ```python
-    pip install https://ms-release.obs.cn-north-4.myhuaweicloud.com/1.1.0/Serving/ascend/ubuntu_x86/mindspore_serving-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
+    > - `{version}` denotes the version of MindSpore Serving. For example, when you are downloading MindSpore Serving 1.1.0, `{version}` should be 1.1.0.
+    > - `{arch}` denotes the system architecture. For example, the Linux system you are using is x86 architecture 64-bit, `{arch}` should be `x86_64`. If the system is ARM architecture 64-bit, then it should be `aarch64`.
+
 - Install Serving using the source code.
 
     Download the [source code](https://gitee.com/mindspore/serving) and go to the `serving` directory.
@@ -67,21 +70,21 @@ Use the pip command to install Serving. Perform the following steps:
 
     ```shell
     # Ascend 310
-    sh build.sh -e d -V 310
+    sh build.sh -e ascend -V 310
     # Ascend 910
-    sh build.sh -e ascend
+    sh build.sh -e ascend -V 910
     ```
 
-    In the preceding information, `build.sh` is the build script file in the `serving` directory, Take the x86 system as an example. After the build is complete, find the .whl installation package of MindSpore in the `serving/third_party/mindspore/build/package/` directory and install it.
+    In the preceding information, `build.sh` is the build script file in the `serving` directory. After the build is complete, find the .whl installation package of MindSpore in the `serving/third_party/mindspore/build/package/` directory and install it.
 
     ```python
-    pip install mindspore_ascend-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_ascend-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
     Find the .whl installation package of Serving in the `serving/build/package/` directory and install it.
 
     ```python
-    pip install mindspore_serving-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
 Run the following commands to verify the installation. Import the Python module. If no error is reported, the installation is successful.

README_CN.md

@@ -45,12 +45,15 @@ MindSpore Serving依赖MindSpore训练推理框架，安装完[MindSpore](https:
 
 使用pip命令安装，安装方式如下：
 
-- 请从MindSpore Serving下载页面下载并安装whl包。
+- 请从[MindSpore Serving下载页面](https://www.mindspore.cn/versions)下载并安装whl包。
 
     ```python
-    pip install https://ms-release.obs.cn-north-4.myhuaweicloud.com/1.1.0/Serving/ascend/ubuntu_x86/mindspore_serving-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
+    > - `{version}`表示MindSpore Serving版本号，例如下载1.1.0版本MindSpore Serving时，`{version}`应写为1.1.0。
+    > - `{arch}`表示系统架构，例如使用的Linux系统是x86架构64位时，`{arch}`应写为`x86_64`。如果系统是ARM架构64位，则写为`aarch64`。
+
 - 源码编译安装。
 
     下载[源码](https://gitee.com/mindspore/serving)，下载后进入`serving`目录。
@@ -67,21 +70,21 @@ MindSpore Serving依赖MindSpore训练推理框架，安装完[MindSpore](https:
 
     ```shell
     # Ascend 310
-    sh build.sh -e d -V 310
+    sh build.sh -e ascend -V 310
     # Ascend 910
-    sh build.sh -e ascend
+    sh build.sh -e ascend -V 910
     ```
 
-    其中，`build.sh`为`serving`目录下的编译脚本文件，以x86系统为例，编译完后，在`serving/third_party/mindspore/build/package/`目录下找到MindSpore的whl安装包进行安装：
+    其中，`build.sh`为`serving`目录下的编译脚本文件，编译完后，在`serving/third_party/mindspore/build/package/`目录下找到MindSpore的whl安装包进行安装：
 
     ```python
-    pip install mindspore_ascend-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_ascend-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
     同时在`serving/build/package/`目录下找到Serving的whl安装包进行安装：
 
     ```python
-    pip install mindspore_serving-1.1.0-cp37-cp37m-linux_x86_64.whl
+    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl
     ```
 
 执行以下命令，验证安装结果。导入Python模块不报错即安装成功：

RELEASE.md

@@ -1,11 +1,24 @@
+# MindSpore Serving 1.1.1 Release Notes
+
+## Major Features and Improvements
+
+* Adapts new C++ inference interface for MindSpore version 1.1.1.
+
+## Bug fixes
+
+* [BUGFIX] Fix bug in transforming result of type int16 in python Client.
+* [BUGFIX] Fix bytes type misidentified as str type after python preprocess and postprocess.
+* [BUGFIX] Fix bug releasing C++ tensor data when it's wrapped as numpy object sometimes.
+* [BUGFIX] Update RuntimeError to warning log when check Ascend environment failed.
+
 # MindSpore Serving 1.1.0 Release Notes
 
 ## Major Features and Improvements
 
 ### Ascend 310 & Ascend 910 Serving Framework
 
-Support gRPC and RESTful API.
-Support simple Python API for Client and Server.
-Support Model configuration，User can customize preprocessing & postprocessing for model.
-Support multiple models，Multiple models can run simultaneously.
-Support Model batching，Multiple instances will be split and combined to meet the batch size requirements of the model.
+* Support gRPC and RESTful API.
+* Support simple Python API for Client and Server.
+* Support Model configuration，User can customize preprocessing & postprocessing for model.
+* Support multiple models，Multiple models can run simultaneously.
+* Support Model batching，Multiple instances will be split and combined to meet the batch size requirements of the model.

cmake/check_requirements.cmake

@@ -1,7 +1,7 @@
 ## define customized find fucntions, print customized error messages
 function(find_required_package pkg_name)
     find_package(${pkg_name})
-    if (NOT ${pkg_name}_FOUND)
+    if(NOT ${pkg_name}_FOUND)
         message(FATAL_ERROR "Required package ${pkg_name} not found, please install the package and try building mindspore_serving again.")
     endif()
 endfunction()
@@ -9,31 +9,36 @@ endfunction()
 ## find python, quit if the found python is static
 set(Python3_USE_STATIC_LIBS FALSE)
 find_package(Python3 COMPONENTS Interpreter Development)
-if (Python3_FOUND)
+if(Python3_FOUND)
     message("Python3 found, version: ${Python3_VERSION}")
-    message("Python3 library path: ${Python3_LIBRARY_DIRS}")
+    message("Python3 library path: ${Python3_LIBRARY}")
     message("Python3 interpreter: ${Python3_EXECUTABLE}")
-    else()
+elseif(Python3_LIBRARY AND Python3_EXECUTABLE AND
+        ${Python3_VERSION} VERSION_GREATER_EQUAL "3.7.0" AND ${Python3_VERSION} VERSION_LESS "3.8.0")
+    message(WARNING "Maybe python3 environment is broken.")
+    message("Python3 library path: ${Python3_LIBRARY}")
+    message("Python3 interpreter: ${Python3_EXECUTABLE}")
+else()
     message(FATAL_ERROR "Python3 not found, please install Python>=3.7.5, and set --enable-shared "
             "if you are building Python locally")
-endif ()
+endif()
 
 ## packages used both on windows and linux
-if (DEFINED ENV{MS_PATCH_PATH})
+if(DEFINED ENV{MS_PATCH_PATH})
     find_program(Patch_EXECUTABLE patch PATHS $ENV{MS_PATCH_PATH})
     set(Patch_FOUND ${Patch_EXECUTABLE})
-else ()
+else()
     find_package(Patch)
-endif ()
-if (NOT Patch_FOUND)
+endif()
+if(NOT Patch_FOUND)
     message(FATAL_ERROR "Patch not found, please set environment variable MS_PATCH_PATH to path where Patch is located, "
             "usually found in GIT_PATH/usr/bin on Windows")
-endif ()
+endif()
 message(PATCH_EXECUTABLE = ${Patch_EXECUTABLE})
 
 find_required_package(Threads)
 
 ## packages used on Linux
-if (NOT CMAKE_SYSTEM_NAME MATCHES "Windows")
+if(NOT CMAKE_SYSTEM_NAME MATCHES "Windows")
     find_required_package(OpenSSL)
 endif()

example/resnet/client.py

@@ -53,10 +53,15 @@ def run_classify_top5():
     """Client for servable resnet50 and method classify_top5"""
     client = Client("localhost", 5500, "resnet50", "classify_top5")
     instances = []
-    for image in read_images():
-        instances.append({"image": image})
+    for image in read_images():  # read multi image
+        instances.append({"image": image})  # input `image`
     result = client.infer(instances)
     print(result)
+    for result_item in result:  # result for every image
+        label = result_item["label"]  # result `label`
+        score = result_item["score"]  # result `score`
+        print("label result", label)
+        print("score result", score)
 
 
 def run_restful_classify_top1():

mindspore_serving/CMakeLists.txt

@@ -27,7 +27,7 @@ message("serving using grpc_DIR : " ${gPRC_DIR})
 find_package(gRPC CONFIG REQUIRED)
 message(STATUS "Using gRPC ${gRPC_VERSION}")
 
-set(_GRPC_GRPCPP gRPC::grpc++)
+set(_GRPC_GRPCPP gRPC::grpc++ gRPC::grpc)
 set(_REFLECTION gRPC::grpc++_reflection)
 
 if (CMAKE_CROSSCOMPILING)
@@ -116,6 +116,8 @@ target_include_directories(_mindspore_serving PRIVATE ${pybind11_INCLUDE_DIRS})
 target_link_libraries(_mindspore_serving PRIVATE serving_common)
 set_property(TARGET _mindspore_serving PROPERTY POSITION_INDEPENDENT_CODE TRUE)
 
+target_link_options(serving_common PRIVATE -Wl,-init,mindspore_serving_log_init)
+
 # user set path
 
 if (MS_WHL_LIB_PATH)

mindspore_serving/ccsrc/common/log.cc

@@ -21,6 +21,8 @@
 namespace mindspore {
 namespace serving {
 
+int g_ms_serving_log_level = LOG_WARNING;
+
 #undef Dlog
 #define Dlog(module_id, level, format, ...)                   \
   do {                                                        \
@@ -78,7 +80,7 @@ static std::string GetProcName() {
   return app_name.substr(pos + 1);
 }
 
-static std::string GetLogLevel(ERROR_LEVEL level) {
+static std::string GetLogLevel(MsLogLevel level) {
   switch (level) {
     case LOG_DEBUG:
       return "DEBUG";
@@ -93,7 +95,7 @@ static std::string GetLogLevel(ERROR_LEVEL level) {
 }
 
 // convert MsLogLevel to corresponding glog level
-static int GetGlogLevel(ERROR_LEVEL level) {
+static int GetGlogLevel(MsLogLevel level) {
   switch (level) {
     case LOG_DEBUG:
     case LOG_INFO:
@@ -107,12 +109,269 @@ static int GetGlogLevel(ERROR_LEVEL level) {
 }
 
 void LogWriter::OutputLog(const std::ostringstream &msg) const {
-  auto submodule_name = "Serving";
+  if (log_level_ < g_ms_serving_log_level) {
+    return;
+  }
+  auto submodule_name = "SERVING";
   google::LogMessage("", 0, GetGlogLevel(log_level_)).stream()
     << "[" << GetLogLevel(log_level_) << "] " << submodule_name << "(" << getpid() << "," << GetProcName()
     << "):" << GetTime() << " "
     << "[" << file_ << ":" << line_ << "] " << func_ << "] " << msg.str() << std::endl;
 }
 
+static MsLogLevel GetGlobalLogLevel() { return static_cast<MsLogLevel>(FLAGS_v); }
+
+static std::string GetEnv(const std::string &envvar) {
+  const char *value = ::getenv(envvar.c_str());
+
+  if (value == nullptr) {
+    return std::string();
+  }
+
+  return std::string(value);
+}
+
+enum LogConfigToken {
+  INVALID,      // indicate invalid token
+  LEFT_BRACE,   // '{'
+  RIGHT_BRACE,  // '}'
+  VARIABLE,     // '[A-Za-z][A-Za-z0-9_]*'
+  NUMBER,       // [0-9]+
+  COMMA,        // ','
+  COLON,        // ':'
+  EOS,          // End Of String, '\0'
+  NUM_LOG_CFG_TOKENS
+};
+
+static const char *g_tok_names[NUM_LOG_CFG_TOKENS] = {
+  "invalid",        // indicate invalid token
+  "{",              // '{'
+  "}",              // '}'
+  "variable",       // '[A-Za-z][A-Za-z0-9_]*'
+  "number",         // [0-9]+
+  ",",              // ','
+  ":",              // ':'
+  "end-of-string",  // End Of String, '\0'
+};
+
+static inline bool IsAlpha(char ch) { return (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z'); }
+
+static inline bool IsDigit(char ch) { return ch >= '0' && ch <= '9'; }
+
+class LogConfigLexer {
+ public:
+  explicit LogConfigLexer(const std::string &text) : buffer_(text) {
+    cur_idx_ = 0;
+    cur_token_ = LogConfigToken::INVALID;
+  }
+  ~LogConfigLexer() = default;
+
+  // skip white space, and return the first char after white space
+  char SkipWhiteSpace() {
+    while (cur_idx_ < buffer_.size()) {
+      char ch = buffer_[cur_idx_];
+      if (ch == ' ' || ch == '\t') {
+        ++cur_idx_;
+        continue;
+      }
+      return ch;
+    }
+    return '\0';
+  }
+
+  LogConfigToken GetNext(std::string *const ptr) {
+    char ch = SkipWhiteSpace();
+    // clang-format off
+    static const std::map<char, LogConfigToken> single_char_map = {
+      {'{', LogConfigToken::LEFT_BRACE},
+      {'}', LogConfigToken::RIGHT_BRACE},
+      {',', LogConfigToken::COMMA},
+      {':', LogConfigToken::COLON},
+      {'\0', LogConfigToken::EOS},
+    };
+    // clang-format on
+
+    auto iter = single_char_map.find(ch);
+    if (iter != single_char_map.end()) {
+      if (ptr != nullptr) {
+        *ptr = std::string() + ch;
+      }
+      ++cur_idx_;
+      return iter->second;
+    } else if (IsAlpha(ch)) {
+      std::ostringstream oss;
+      do {
+        oss << ch;
+        ch = buffer_[++cur_idx_];
+      } while (cur_idx_ < buffer_.size() && (IsAlpha(ch) || IsDigit(ch) || ch == '_'));
+      if (ptr != nullptr) {
+        *ptr = std::string(oss.str());
+      }
+      return LogConfigToken::VARIABLE;
+    } else if (IsDigit(ch)) {
+      std::ostringstream oss;
+      do {
+        oss << ch;
+        ch = buffer_[++cur_idx_];
+      } while (cur_idx_ < buffer_.size() && IsDigit(ch));
+      if (ptr != nullptr) {
+        *ptr = std::string(oss.str());
+      }
+      return LogConfigToken::NUMBER;
+    }
+    return LogConfigToken::INVALID;
+  }
+
+ private:
+  std::string buffer_;
+  size_t cur_idx_;
+
+  LogConfigToken cur_token_;
+  std::string cur_text_;
+};
+
+class LogConfigParser {
+ public:
+  explicit LogConfigParser(const std::string &cfg) : lexer(cfg) {}
+  ~LogConfigParser() = default;
+
+  bool Expect(LogConfigToken expected, LogConfigToken tok) {
+    if (expected != tok) {
+      MSI_LOG_WARNING << "`, but got `" << g_tok_names[tok] << "`. The whole configuration will be ignored.";
+      return false;
+    }
+    return true;
+  }
+
+  // The text of config MS_SUBMODULE_LOG_v is in the form {submodule1:log_level1,submodule2:log_level2,...}.
+  // Valid values of log levels are: 0 - debug, 1 - info, 2 - warning, 3 - error
+  // e.g. MS_SUBMODULE_LOG_v={PARSER:0, ANALYZER:2, PIPELINE:1}
+  std::map<std::string, std::string> Parse() {
+    std::map<std::string, std::string> log_levels;
+
+    bool flag_error = false;
+    std::string text;
+    auto tok = lexer.GetNext(&text);
+    // empty string
+    if (tok == LogConfigToken::EOS) {
+      return log_levels;
+    }
+
+    if (!Expect(LogConfigToken::LEFT_BRACE, tok)) {
+      return log_levels;
+    }
+
+    do {
+      std::string key, val;
+      tok = lexer.GetNext(&key);
+      if (!Expect(LogConfigToken::VARIABLE, tok)) {
+        flag_error = true;
+        break;
+      }
+
+      tok = lexer.GetNext(&text);
+      if (!Expect(LogConfigToken::COLON, tok)) {
+        flag_error = true;
+        break;
+      }
+
+      tok = lexer.GetNext(&val);
+      if (!Expect(LogConfigToken::NUMBER, tok)) {
+        flag_error = true;
+        break;
+      }
+
+      log_levels[key] = val;
+      tok = lexer.GetNext(&text);
+    } while (tok == LogConfigToken::COMMA);
+
+    if (!flag_error && !Expect(LogConfigToken::RIGHT_BRACE, tok)) {
+      flag_error = true;
+    }
+
+    if (flag_error) {
+      log_levels.clear();
+    }
+    return log_levels;
+  }
+
+ private:
+  LogConfigLexer lexer;
+};
+
+bool ParseLogLevel(const std::string &str_level, MsLogLevel *ptr_level) {
+  if (str_level.size() == 1) {
+    int ch = str_level.c_str()[0];
+    ch = ch - '0';  // substract ASCII code of '0', which is 48
+    if (ch >= LOG_DEBUG && ch <= LOG_ERROR) {
+      if (ptr_level != nullptr) {
+        *ptr_level = static_cast<MsLogLevel>(ch);
+      }
+      return true;
+    }
+  }
+  return false;
+}
+
+void InitSubModulesLogLevel() {
+  // initialize submodule's log level using global
+  auto global_log_level = GetGlobalLogLevel();
+  g_ms_serving_log_level = global_log_level;
+
+  // set submodule's log level
+  auto submodule = GetEnv("MS_SUBMODULE_LOG_v");
+  MSI_LOG(DEBUG) << "MS_SUBMODULE_LOG_v=`" << submodule << "`";
+  LogConfigParser parser(submodule);
+  auto configs = parser.Parse();
+  for (const auto &cfg : configs) {
+    if (cfg.first == "SERVING") {
+      MsLogLevel submodule_log_level;
+      if (!ParseLogLevel(cfg.second, &submodule_log_level)) {
+        MSI_LOG(WARNING) << "Illegal log level value " << cfg.second << " for " << cfg.first << ", ignore it.";
+        continue;
+      }
+      g_ms_serving_log_level = submodule_log_level;
+    }
+  }
+}
+
 }  // namespace serving
 }  // namespace mindspore
+
+extern "C" {
+#if defined(_WIN32) || defined(_WIN64)
+__attribute__((constructor)) void common_log_init(void) {
+#else
+void common_log_init(void) {
+#endif
+  // do not use glog predefined log prefix
+  FLAGS_log_prefix = false;
+  // set default log level to WARNING
+  if (mindspore::serving::GetEnv("GLOG_v").empty()) {
+    FLAGS_v = mindspore::serving::LOG_WARNING;
+  }
+
+  // set default log file mode to 0640
+  if (mindspore::serving::GetEnv("GLOG_logfile_mode").empty()) {
+    FLAGS_logfile_mode = 0640;
+  }
+  std::string logtostderr = mindspore::serving::GetEnv("GLOG_logtostderr");
+  // default print log to screen
+  if (logtostderr.empty()) {
+    FLAGS_logtostderr = true;
+  } else if (logtostderr == "0" && mindspore::serving::GetEnv("GLOG_log_dir").empty()) {
+    FLAGS_logtostderr = true;
+    MSI_LOG(WARNING) << "`GLOG_log_dir` is not set, output log to screen.";
+  }
+  mindspore::serving::InitSubModulesLogLevel();
+}
+
+// shared lib init hook
+#if defined(_WIN32) || defined(_WIN64)
+__attribute__((constructor)) void mindspore_serving_log_init(void) {
+#else
+void mindspore_serving_log_init(void) {
+#endif
+  common_log_init();
+}
+}

mindspore_serving/ccsrc/common/log.h

@@ -37,9 +37,10 @@ static constexpr int GetRelPathPos() noexcept {
            : 0;
 }
 
-#define SERVING_FILE_NAME                                                                     \
-  (sizeof(__FILE__) > GetRelPathPos() ? static_cast<const char *>(__FILE__) + GetRelPathPos() \
-                                      : static_cast<const char *>(__FILE__))
+#define SERVING_FILE_NAME                                                        \
+  (sizeof(__FILE__) > mindspore::serving::GetRelPathPos()                        \
+     ? static_cast<const char *>(__FILE__) + mindspore::serving::GetRelPathPos() \
+     : static_cast<const char *>(__FILE__))
 
 class LogStream {
  public:
@@ -97,7 +98,7 @@ class LogStream {
   std::shared_ptr<std::stringstream> sstream_;
 };
 
-enum ERROR_LEVEL {
+enum MsLogLevel {
   LOG_DEBUG,
   LOG_INFO,
   LOG_WARNING,
@@ -107,7 +108,7 @@ enum ERROR_LEVEL {
 
 class MS_API LogWriter {
  public:
-  LogWriter(const char *file, int line, const char *func, ERROR_LEVEL log_level)
+  LogWriter(const char *file, int line, const char *func, MsLogLevel log_level)
       : file_(file), line_(line), func_(func), log_level_(log_level) {}
   ~LogWriter() = default;
 
@@ -131,23 +132,34 @@ class MS_API LogWriter {
   const char *file_;
   int line_;
   const char *func_;
-  ERROR_LEVEL log_level_;
+  MsLogLevel log_level_;
 };
 
-#define MSILOG_IF(level)                                                                                      \
+extern int g_ms_serving_log_level MS_API;
+
+#define MSILOG_IF(level, condition)                                                         \
+  static_cast<void>(0),                                                                     \
+    !(condition) ? std::string()                                                            \
+                 : mindspore::serving::LogWriter(SERVING_FILE_NAME, __LINE__, __FUNCTION__, \
+                                                 mindspore::serving::LOG_##level) < mindspore::serving::LogStream()
+
+#define MSILOG_NOIF(level)                                                                                    \
   mindspore::serving::LogWriter(SERVING_FILE_NAME, __LINE__, __FUNCTION__, mindspore::serving::LOG_##level) < \
     mindspore::serving::LogStream()
 
+#define IS_OUTPUT_ON(level) (mindspore::serving::LOG_##level) >= mindspore::serving::g_ms_serving_log_level
+
 #define MSILOG_THROW                                                                                            \
   mindspore::serving::LogWriter(SERVING_FILE_NAME, __LINE__, __FUNCTION__, mindspore::serving::LOG_EXCEPTION) ^ \
     mindspore::serving::LogStream()
 
 #define MSI_LOG(level) MSI_LOG_##level
 
-#define MSI_LOG_DEBUG MSILOG_IF(DEBUG)
-#define MSI_LOG_INFO MSILOG_IF(INFO)
-#define MSI_LOG_WARNING MSILOG_IF(WARNING)
-#define MSI_LOG_ERROR MSILOG_IF(ERROR)
+#define MSI_LOG_DEBUG MSILOG_IF(DEBUG, IS_OUTPUT_ON(DEBUG))
+#define MSI_LOG_INFO MSILOG_IF(INFO, IS_OUTPUT_ON(INFO))
+#define MSI_LOG_WARNING MSILOG_IF(WARNING, IS_OUTPUT_ON(WARNING))
+#define MSI_LOG_ERROR MSILOG_IF(ERROR, IS_OUTPUT_ON(ERROR))
+
 #define MSI_LOG_EXCEPTION MSILOG_THROW
 
 #define MSI_EXCEPTION_IF_NULL(ptr)                                   \

mindspore_serving/ccsrc/common/proto_tensor.cc

@@ -241,7 +241,7 @@ Status GrpcTensorHelper::CreateInstanceFromRequest(const proto::PredictRequest &
   MethodSignature method_signature;
   if (!servable_signature.GetMethodDeclare(request_spec->method_name, &method_signature)) {
     return INFER_STATUS_LOG_ERROR(INVALID_INPUTS)
-           << "Method " << method_name << " is not registed for servable " << servable_name;
+           << "Method " << method_name << " is not registered for servable " << servable_name;
   }
 
   // instance
@@ -268,7 +268,7 @@ Status GrpcTensorHelper::CreateReplyFromInstances(const proto::PredictRequest &r
   MethodSignature method_signature;
   if (!servable_signature.GetMethodDeclare(method_name, &method_signature)) {
     return INFER_STATUS_LOG_ERROR(INVALID_INPUTS)
-           << "Method " << method_name << " is not registed for servable " << servable_name;
+           << "Method " << method_name << " is not registered for servable " << servable_name;
   }
   *reply->mutable_servable_spec() = request.servable_spec();
 

mindspore_serving/ccsrc/common/status.h

@@ -67,8 +67,8 @@ class Status {
   MSI_LOG_ERROR << msg;                       \
   status = mindspore::serving::Status(type, msg)
 
-#define INFER_STATUS_LOG_ERROR(code) mindspore::serving::Status(code) = MSI_LOG_ERROR
-#define INFER_STATUS_LOG_WARNING(code) mindspore::serving::Status(code) = MSI_LOG_WARNING
+#define INFER_STATUS_LOG_ERROR(code) mindspore::serving::Status(code) = MSILOG_NOIF(ERROR)
+#define INFER_STATUS_LOG_WARNING(code) mindspore::serving::Status(code) = MSILOG_NOIF(WARNING)
 
 }  // namespace mindspore::serving
 

mindspore_serving/ccsrc/common/tensor_base.h

@@ -51,7 +51,7 @@ enum DataType {
 class TensorBase;
 using TensorBasePtr = std::shared_ptr<TensorBase>;
 
-class MS_API TensorBase {
+class MS_API TensorBase : public std::enable_shared_from_this<TensorBase> {
  public:
   TensorBase() = default;
   virtual ~TensorBase() = default;

mindspore_serving/ccsrc/master/restful/http_process.cc

@@ -257,7 +257,8 @@ Status RestfulService::CheckReqJsonValid(const json &js_msg) {
   }
 
   if (count != 1) {
-    return INFER_STATUS_LOG_ERROR(INVALID_INPUTS) << "key 'instances' should exit and only exit one time";
+    return INFER_STATUS_LOG_ERROR(INVALID_INPUTS)
+           << "key 'instances' expects to exist once, but actually " << count << " times";
   }
   return SUCCESS;
 }
@@ -680,7 +681,7 @@ Status RestfulService::RunRestful(const std::shared_ptr<RestfulRequest> &restful
   MSI_TIME_STAMP_END(ParseRequest)
   if (status != SUCCESS) {
     std::string error_msg = status.StatusMessage();
-    std::string msg = "Parser reqeust failed, " + error_msg;
+    std::string msg = "Parser request failed, " + error_msg;
     status = msg;
     return status;
   }
@@ -730,7 +731,7 @@ Status RestfulService::ParseRequest(const std::shared_ptr<RestfulRequest> &restf
       status = ParseInstancesMsg(js_msg, request);
       break;
     default:
-      return INFER_STATUS_LOG_ERROR(FAILED) << "restful reqeust only support instances mode";
+      return INFER_STATUS_LOG_ERROR(FAILED) << "restful request only support instances mode";
   }
 
   return status;

mindspore_serving/ccsrc/master/restful/restful_request.cc

@@ -53,6 +53,7 @@ std::string DecomposeEvRequest::UrlQuery(const std::string &url, const std::stri
     if (end_pos != std::string::npos) {
       return url.substr(start_pos + key_size, end_pos - start_pos - key_size);
     }
+    return url.substr(start_pos + key_size);
   }
   return "";
 }
@@ -134,7 +135,12 @@ Status DecomposeEvRequest::Decompose() {
   if (url_.find(UrlKeyVersion) != std::string::npos) {
     auto version_str = UrlQuery(url_, UrlKeyVersion);
     try {
-      version_ = std::stol(version_str);
+      auto version = std::stol(version_str);
+      if (version < 0 || version >= UINT32_MAX) {
+        return INFER_STATUS_LOG_ERROR(INVALID_INPUTS)
+               << "please check url, version number range failed, request version number " << version_str;
+      }
+      version_ = static_cast<uint32_t>(version);
     } catch (const std::invalid_argument &) {
       ERROR_INFER_STATUS(status, INVALID_INPUTS, "please check url, the keyword:[version] must contain.");
       return status;
@@ -187,7 +193,6 @@ Status RestfulRequest::RestfulReplay(const std::string &replay) {
     ERROR_INFER_STATUS(status, INVALID_INPUTS, "replay_buffer_ is nullptr");
     return status;
   }
-  MSI_LOG_INFO << "replay message: " << replay;
   evbuffer_add(replay_buffer_, replay.data(), replay.size());
   evhttp_send_reply(decompose_event_request_->event_request_, HTTP_OK, "Client", replay_buffer_);
   return status;
@@ -195,7 +200,7 @@ Status RestfulRequest::RestfulReplay(const std::string &replay) {
 
 Status RestfulRequest::ErrorMessage(Status status) {
   Status error_status(SUCCESS);
-  nlohmann::json error_json = {{"error_message", status.StatusMessage()}};
+  nlohmann::json error_json = {{"error_msg", status.StatusMessage()}};
   std::string out_error_str = error_json.dump();
   if ((error_status = RestfulReplay(out_error_str)) != SUCCESS) {
     return error_status;

mindspore_serving/ccsrc/master/restful/restful_server.cc

@@ -114,7 +114,7 @@ Status RestfulServer::CreatRestfulServer(int time_out_second) {
   event_http_ = evhttp_new(event_base_);
   if (event_http_ == nullptr) {
     status = INFER_STATUS_LOG_ERROR(SYSTEM_ERROR)
-             << "Serving Error: RESTful server start failed, create http server faild";
+             << "Serving Error: RESTful server start failed, create http server failed";
     free_event_base();
     return status;
   }
@@ -148,7 +148,7 @@ Status RestfulServer::StartRestfulServer() {
 
   if (listener == nullptr) {
     status = INFER_STATUS_LOG_ERROR(SYSTEM_ERROR)
-             << "Serving Error: RESTful server start failed, create http listener faild, port " << restful_port_;
+             << "Serving Error: RESTful server start failed, create http listener failed, port " << restful_port_;
     free_event_base();
     free_evhttp();
     return status;
@@ -156,7 +156,7 @@ Status RestfulServer::StartRestfulServer() {
   auto bound = evhttp_bind_listener(event_http_, listener);
   if (bound == nullptr) {
     status = INFER_STATUS_LOG_ERROR(SYSTEM_ERROR)
-             << "Serving Error: RESTful server start failed, bind http listener to server faild, port "
+             << "Serving Error: RESTful server start failed, bind http listener to server failed, port "
              << restful_port_;
     evconnlistener_free(listener);
     free_event_base();

mindspore_serving/ccsrc/python/serving_py.cc

@@ -27,6 +27,9 @@
 namespace mindspore::serving {
 
 PYBIND11_MODULE(_mindspore_serving, m) {
+  // avoid as numpy object memory copy in PyTensor::AsPythonData
+  py::class_<TensorBase, TensorBasePtr>(m, "Tensor_");
+
   py::class_<PyPreprocessStorage, std::shared_ptr<PyPreprocessStorage>>(m, "PreprocessStorage_")
     .def(py::init<>())
     .def_static("get_instance", &PyPreprocessStorage::Instance)

mindspore_serving/ccsrc/python/tensor_py.cc

@@ -196,7 +196,7 @@ py::object PyTensor::AsPythonData(TensorBasePtr tensor, bool copy) {
     py::buffer_info info(reinterpret_cast<void *>(const_cast<uint8_t *>(data)), sizeof(uint8_t),
                          py::format_descriptor<uint8_t>::format(), 1, shape, strides);
     if (!copy) {
-      py::array self;
+      py::object self = py::cast(tensor.get());
       return py::array(py::dtype(info), info.shape, info.strides, info.ptr, self);
     } else {
       return py::array(py::dtype(info), info.shape, info.strides, info.ptr);
@@ -210,7 +210,7 @@ py::object PyTensor::AsPythonData(TensorBasePtr tensor, bool copy) {
                          static_cast<ssize_t>(tensor_shape.size()), shape, strides);
 
     if (!copy) {
-      py::array self;
+      py::object self = py::cast(tensor.get());
       return py::array(py::dtype(info), info.shape, info.strides, info.ptr, self);
     } else {
       return py::array(py::dtype(info), info.shape, info.strides, info.ptr);
@@ -230,25 +230,25 @@ InstanceData PyTensor::AsInstanceData(const py::tuple &tuple) {
   InstanceData instance_data;
   for (auto &item : tuple) {
     TensorBasePtr tensor = nullptr;
-    if (py::isinstance<py::str>(item)) {
-      tensor = std::make_shared<Tensor>();
-      tensor->set_data_type(serving::kMSI_String);
-      auto val = item.cast<std::string>();
-      tensor->add_bytes_data(reinterpret_cast<const uint8_t *>(val.data()), val.length());
-    } else if (py::isinstance<py::bytes>(item)) {
+    if (py::isinstance<py::bytes>(item)) {  // bytes can be seen as str, so check bytes first
       tensor = std::make_shared<Tensor>();
       tensor->set_data_type(serving::kMSI_Bytes);
       auto val = std::string(item.cast<py::bytes>());
       tensor->add_bytes_data(reinterpret_cast<const uint8_t *>(val.data()), val.length());
+    } else if (py::isinstance<py::str>(item)) {
+      tensor = std::make_shared<Tensor>();
+      tensor->set_data_type(serving::kMSI_String);
+      auto val = item.cast<std::string>();
+      tensor->add_bytes_data(reinterpret_cast<const uint8_t *>(val.data()), val.length());
     } else if (py::isinstance<py::bool_>(item)) {
       auto val = item.cast<bool>();
       tensor = std::make_shared<Tensor>(serving::kMSI_Bool, std::vector<int64_t>(), &val, sizeof(val));
     } else if (py::isinstance<py::int_>(item)) {
-      auto val = item.cast<int32_t>();
-      tensor = std::make_shared<Tensor>(serving::kMSI_Int32, std::vector<int64_t>(), &val, sizeof(val));
+      auto val = item.cast<int64_t>();
+      tensor = std::make_shared<Tensor>(serving::kMSI_Int64, std::vector<int64_t>(), &val, sizeof(val));
     } else if (py::isinstance<py::float_>(item)) {
-      auto val = item.cast<float>();
-      tensor = std::make_shared<Tensor>(serving::kMSI_Float32, std::vector<int64_t>(), &val, sizeof(val));
+      auto val = item.cast<double>();
+      tensor = std::make_shared<Tensor>(serving::kMSI_Float64, std::vector<int64_t>(), &val, sizeof(val));
     } else {
       try {
         tensor = PyTensor::MakeTensorNoCopy(py::cast<py::array>(item));

mindspore_serving/ccsrc/worker/context.cc

@@ -52,7 +52,7 @@ Status ServableContext::SetDeviceTypeStr(const std::string &device_type) {
     type = kDeviceTypeNotSpecified;
   } else {
     return INFER_STATUS_LOG_ERROR(FAILED)
-           << "Unsupport device type '" << device_type
+           << "Unsupported device type '" << device_type
            << "', only support 'Ascend', 'Davinci'(same with 'Ascend') and None, case ignored";
   }
   SetDeviceType(type);

mindspore_serving/ccsrc/worker/inference/inference.h

@@ -28,13 +28,14 @@
 #include "common/log.h"
 #include "common/status.h"
 #include "include/api/types.h"
+#include "include/api/data_type.h"
 
 namespace mindspore {
 namespace serving {
 
-using api::ModelType;
-using api::ModelType::kMindIR;
-using api::ModelType::kOM;
+using mindspore::ModelType;
+using mindspore::ModelType::kMindIR;
+using mindspore::ModelType::kOM;
 
 struct TensorInfo {
   size_t size;  // -1: unspecified
@@ -76,9 +77,6 @@ class MS_API InferSession {
   virtual std::vector<TensorInfo> GetInputInfos(uint32_t model_id) const = 0;
   virtual std::vector<TensorInfo> GetOutputInfos(uint32_t model_id) const = 0;
   virtual ssize_t GetBatchSize(uint32_t model_id) const = 0;
-  virtual TensorBasePtr MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const {
-    return nullptr;
-  }
   virtual bool CheckModelSupport(DeviceType device_type, ModelType model_type) const { return true; }
 };
 
@@ -207,12 +205,12 @@ static inline LogStream &operator<<(LogStream &stream, DeviceType device_type) {
   return stream;
 }
 
-static inline LogStream &operator<<(LogStream &stream, api::ModelType model_type) {
+static inline LogStream &operator<<(LogStream &stream, mindspore::ModelType model_type) {
   switch (model_type) {
-    case api::kMindIR:
+    case mindspore::kMindIR:
       stream << "MindIR";
       break;
-    case api::kOM:
+    case mindspore::kOM:
       stream << "OM";
       break;
     default:

mindspore_serving/ccsrc/worker/inference/mindspore_model_wrap.cc

@@ -36,36 +36,44 @@ Status MindSporeModelWrap::FinalizeEnv() {
   return SUCCESS;
 }
 
-api::DataType TransInferDataType2ApiTypeId(DataType data_type) {
-  const std::map<DataType, api::DataType> type2id_map{
-    {serving::kMSI_Unknown, api::kMsUnknown}, {serving::kMSI_Bool, api::kMsBool},
-    {serving::kMSI_Int8, api::kMsInt8},       {serving::kMSI_Uint8, api::kMsUint8},
-    {serving::kMSI_Int16, api::kMsInt16},     {serving::kMSI_Uint16, api::kMsUint16},
-    {serving::kMSI_Int32, api::kMsInt32},     {serving::kMSI_Uint32, api::kMsUint32},
-    {serving::kMSI_Int64, api::kMsInt64},     {serving::kMSI_Uint64, api::kMsUint64},
-    {serving::kMSI_Float16, api::kMsFloat16}, {serving::kMSI_Float32, api::kMsFloat32},
-    {serving::kMSI_Float64, api::kMsFloat64},
+mindspore::DataType TransInferDataType2ApiTypeId(DataType data_type) {
+  const std::map<DataType, mindspore::DataType> type2id_map{
+    {serving::kMSI_Unknown, mindspore::DataType::kTypeUnknown},
+    {serving::kMSI_Bool, mindspore::DataType::kNumberTypeBool},
+    {serving::kMSI_Int8, mindspore::DataType::kNumberTypeInt8},
+    {serving::kMSI_Uint8, mindspore::DataType::kNumberTypeUInt8},
+    {serving::kMSI_Int16, mindspore::DataType::kNumberTypeInt16},
+    {serving::kMSI_Uint16, mindspore::DataType::kNumberTypeUInt16},
+    {serving::kMSI_Int32, mindspore::DataType::kNumberTypeInt32},
+    {serving::kMSI_Uint32, mindspore::DataType::kNumberTypeUInt32},
+    {serving::kMSI_Int64, mindspore::DataType::kNumberTypeInt64},
+    {serving::kMSI_Uint64, mindspore::DataType::kNumberTypeUInt64},
+    {serving::kMSI_Float16, mindspore::DataType::kNumberTypeFloat16},
+    {serving::kMSI_Float32, mindspore::DataType::kNumberTypeFloat32},
+    {serving::kMSI_Float64, mindspore::DataType::kNumberTypeFloat64},
   };
   auto it = type2id_map.find(data_type);
   if (it == type2id_map.end()) {
     MSI_LOG_WARNING << "Unsupported MSI data type " << data_type;
-    return api::kMsUnknown;
+    return mindspore::DataType::kTypeUnknown;
   } else {
     return it->second;
   }
 }
 
-DataType TransTypeId2InferDataType(api::DataType type_id) {
-  const std::map<api::DataType, DataType> id2type_map{
-    {api::kMsUnknown, kMSI_Unknown}, {api::kMsBool, kMSI_Bool},     {api::kMsFloat64, kMSI_Float64},
-    {api::kMsInt8, kMSI_Int8},       {api::kMsUint8, kMSI_Uint8},   {api::kMsInt16, kMSI_Int16},
-    {api::kMsUint16, kMSI_Uint16},   {api::kMsInt32, kMSI_Int32},   {api::kMsUint32, kMSI_Uint32},
-    {api::kMsInt64, kMSI_Int64},     {api::kMsUint64, kMSI_Uint64}, {api::kMsFloat16, kMSI_Float16},
-    {api::kMsFloat32, kMSI_Float32},
+DataType TransTypeId2InferDataType(mindspore::DataType type_id) {
+  const std::map<mindspore::DataType, DataType> id2type_map{
+    {mindspore::DataType::kTypeUnknown, kMSI_Unknown},       {mindspore::DataType::kNumberTypeBool, kMSI_Bool},
+    {mindspore::DataType::kNumberTypeFloat64, kMSI_Float64}, {mindspore::DataType::kNumberTypeInt8, kMSI_Int8},
+    {mindspore::DataType::kNumberTypeUInt8, kMSI_Uint8},     {mindspore::DataType::kNumberTypeInt16, kMSI_Int16},
+    {mindspore::DataType::kNumberTypeUInt16, kMSI_Uint16},   {mindspore::DataType::kNumberTypeInt32, kMSI_Int32},
+    {mindspore::DataType::kNumberTypeUInt32, kMSI_Uint32},   {mindspore::DataType::kNumberTypeInt64, kMSI_Int64},
+    {mindspore::DataType::kNumberTypeUInt64, kMSI_Uint64},   {mindspore::DataType::kNumberTypeFloat16, kMSI_Float16},
+    {mindspore::DataType::kNumberTypeFloat32, kMSI_Float32},
   };
   auto it = id2type_map.find(type_id);
   if (it == id2type_map.end()) {
-    MSI_LOG_WARNING << "Unsupported data id " << type_id;
+    MSI_LOG_WARNING << "Unsupported data id " << static_cast<int>(type_id);
     return kMSI_Unknown;
   } else {
     return it->second;
@@ -80,28 +88,30 @@ Status MindSporeModelWrap::LoadModelFromFile(serving::DeviceType device_type, ui
   MSI_EXCEPTION_IF_NULL(model_id);
   std::string device_type_str;
   if (device_type == kDeviceTypeAscendMS) {
-    device_type_str = api::kDeviceTypeAscend910;
+    device_type_str = mindspore::kDeviceTypeAscend910;
   } else if (device_type == kDeviceTypeAscendCL) {
-    device_type_str = api::kDeviceTypeAscend310;
+    device_type_str = mindspore::kDeviceTypeAscend310;
   } else {
     MSI_LOG_EXCEPTION << "Only support Ascend310 or Ascend910 in MindSporeModelWrap";
   }
 
-  std::shared_ptr<api::Model> model = nullptr;
+  std::shared_ptr<mindspore::Model> model = nullptr;
   try {
-    api::Context::Instance().SetDeviceTarget(device_type_str).SetDeviceID(device_id);
-    auto graph = api::Serialization::LoadModel(file_name, model_type);
-    model = std::make_shared<api::Model>(api::GraphCell(graph));
+    mindspore::GlobalContext::SetGlobalDeviceTarget(device_type_str);
+    mindspore::GlobalContext::SetGlobalDeviceID(device_id);
+    auto graph = mindspore::Serialization::LoadModel(file_name, model_type);
+    auto context = TransformModelContext(other_options);
+    model = std::make_shared<mindspore::Model>(mindspore::GraphCell(graph), context);
   } catch (std::runtime_error &ex) {
     MSI_LOG_ERROR << "Load model from file failed, model file: " << file_name << ", device_type: '" << device_type_str
                   << "', device_id: " << device_id << ", model type: " << model_type << ", options: " << other_options;
     return FAILED;
   }
-  api::Status status = model->Build(other_options);
-  if (!status.IsSuccess()) {
+  mindspore::Status status = model->Build();
+  if (!status.IsOk()) {
     MSI_LOG_ERROR << "Load model from file failed, model file: " << file_name << ", device_type: '" << device_type_str
                   << "', device_id: " << device_id << ", model type: " << model_type << ", options: " << other_options;
-    return Status(FAILED, status.StatusMessage());
+    return Status(FAILED, status.ToString());
   }
   model_index_++;
   *model_id = model_index_;
@@ -120,6 +130,41 @@ Status MindSporeModelWrap::LoadModelFromFile(serving::DeviceType device_type, ui
   return SUCCESS;
 }
 
+std::shared_ptr<Context> MindSporeModelWrap::TransformModelContext(const std::map<std::string, std::string> &options) {
+  using ContextStrFun = std::function<void(const std::shared_ptr<Context> &, const std::string &)>;
+  ContextStrFun set_output_type = [](const std::shared_ptr<Context> &context, const std::string &val) {
+    // "FP32", "FP16", "UINT8"
+    if (val == "FP32") {
+      mindspore::ModelContext::SetOutputType(context, mindspore::DataType::kNumberTypeFloat32);
+    } else if (val == "FP16") {
+      mindspore::ModelContext::SetOutputType(context, mindspore::DataType::kNumberTypeFloat16);
+    } else if (val == "UINT8") {
+      mindspore::ModelContext::SetOutputType(context, mindspore::DataType::kNumberTypeUInt8);
+    } else {
+      MSI_LOG_ERROR << "Set model context output type failed, unknown data type " << val;
+    }
+  };
+  std::map<std::string, ContextStrFun> option_map = {
+    {"acl_option.insert_op_config_file_path", mindspore::ModelContext::SetInsertOpConfigPath},
+    {"acl_option.input_format", mindspore::ModelContext::SetInputFormat},
+    {"acl_option.input_shape", mindspore::ModelContext::SetInputShape},
+    {"acl_option.output_type", set_output_type},
+    {"acl_option.precision_mode", mindspore::ModelContext::SetPrecisionMode},
+    {"acl_option.op_select_impl_mode", mindspore::ModelContext::SetOpSelectImplMode},
+  };
+  auto context = std::make_shared<mindspore::ModelContext>();
+  for (auto &item : options) {
+    const auto &key = item.first;
+    const auto &value = item.second;
+    auto it = option_map.find(key);
+    if (it != option_map.end()) {
+      MSI_LOG_INFO << "Set context options, key: " << key << ", value: " << value;
+      it->second(context, value);
+    }
+  }
+  return context;
+}
+
 Status MindSporeModelWrap::GetModelInfos(ApiModelInfo *api_model_info) {
   MSI_EXCEPTION_IF_NULL(api_model_info);
   auto model = api_model_info->model;
@@ -138,70 +183,46 @@ Status MindSporeModelWrap::GetModelInfos(ApiModelInfo *api_model_info) {
       }
     }
   };
-  auto shape_element_num = [](const std::vector<int64_t> &shape) -> size_t {
-    size_t elements_nums = std::accumulate(shape.begin(), shape.end(), 1LL, std::multiplies<size_t>());
-    return elements_nums;
-  };
-  auto get_tensor_info_from_tensor = [find_batch_size, shape_element_num, api_model_info](
-                                       const std::vector<int64_t> &shape, const api::DataType &data_type,
-                                       const size_t mem_size, int input_index = -1) {
+  auto get_tensor_info_from_tensor = [](const mindspore::MSTensor &ms_tensor) {
     serving::TensorInfo tensor_info;
-    tensor_info.shape = shape;
-    tensor_info.data_type = TransTypeId2InferDataType(data_type);
-    tensor_info.size = mem_size;
+    tensor_info.shape = ms_tensor.Shape();
+    tensor_info.data_type = TransTypeId2InferDataType(ms_tensor.DataType());
+    tensor_info.size = ms_tensor.DataSize();
     if (tensor_info.size == 0) {
-      tensor_info.size = TensorBase::GetTypeSize(tensor_info.data_type) * shape_element_num(tensor_info.shape);
-    }
-    auto list = api_model_info->without_batch_dim_inputs;
-    if (std::find(list.begin(), list.end(), input_index) == list.end()) {
-      find_batch_size(tensor_info.shape);
+      auto &shape = tensor_info.shape;
+      size_t elements_nums = std::accumulate(shape.begin(), shape.end(), 1LL, std::multiplies<size_t>());
+      tensor_info.size = TensorBase::GetTypeSize(tensor_info.data_type) * elements_nums;
     }
     return tensor_info;
   };
   {  // input infos
-    std::vector<std::string> names;
-    std::vector<std::vector<int64_t>> shapes;
-    std::vector<api::DataType> data_types;
-    std::vector<size_t> mem_sizes;
-    api::Status status = model->GetInputsInfo(&names, &shapes, &data_types, &mem_sizes);
-    if (!status.IsSuccess()) {
-      return Status(FAILED, status.StatusMessage());
-    }
-    if (names.size() != shapes.size() || names.size() != data_types.size() || names.size() != mem_sizes.size()) {
-      return INFER_STATUS_LOG_ERROR(FAILED)
-             << "Get input infos failed, names size: " << names.size() << ", shapes size: " << shapes.size()
-             << ", data_types size: " << data_types.size() << ", mem_sizes: " << mem_sizes.size();
-    }
-    for (size_t i = 0; i < names.size(); i++) {
-      api_model_info->input_names.push_back(names[i]);
-      auto tensor_info = get_tensor_info_from_tensor(shapes[i], data_types[i], mem_sizes[i], i);
+    auto input_infos = model->GetInputs();
+    for (size_t i = 0; i < input_infos.size(); i++) {
+      auto &info = input_infos[i];
+      auto tensor_info = get_tensor_info_from_tensor(info);
       if (tensor_info.data_type == kMSI_Unknown) {
-        return INFER_STATUS_LOG_ERROR(FAILED) << "Unknown input api data type " << data_types[i];
+        return INFER_STATUS_LOG_ERROR(FAILED)
+               << "Unknown input mindspore data type " << static_cast<int>(info.DataType());
+      }
+      const auto &list = api_model_info->without_batch_dim_inputs;
+      if (std::find(list.begin(), list.end(), i) == list.end()) {
+        find_batch_size(tensor_info.shape);
       }
       api_model_info->input_tensor_infos.push_back(tensor_info);
+      api_model_info->input_names.push_back(info.Name());
     }
   }
   {  // output infos
-    std::vector<std::string> names;
-    std::vector<std::vector<int64_t>> shapes;
-    std::vector<api::DataType> data_types;
-    std::vector<size_t> mem_sizes;
-    api::Status status = model->GetOutputsInfo(&names, &shapes, &data_types, &mem_sizes);
-    if (!status.IsSuccess()) {
-      return Status(FAILED, status.StatusMessage());
-    }
-    if (names.size() != shapes.size() || names.size() != data_types.size() || names.size() != mem_sizes.size()) {
-      return INFER_STATUS_LOG_ERROR(FAILED)
-             << "Get output infos failed, names size: " << names.size() << ", shapes size: " << shapes.size()
-             << ", data_types size: " << data_types.size() << ", mem_sizes: " << mem_sizes.size();
-    }
-    for (size_t i = 0; i < names.size(); i++) {
-      api_model_info->output_names.push_back(names[i]);
-      auto tensor_info = get_tensor_info_from_tensor(shapes[i], data_types[i], mem_sizes[i]);
+    auto output_infos = model->GetOutputs();
+    for (auto &info : output_infos) {
+      auto tensor_info = get_tensor_info_from_tensor(info);
       if (tensor_info.data_type == kMSI_Unknown) {
-        return INFER_STATUS_LOG_ERROR(FAILED) << "Unknown output api data type " << data_types[i];
+        return INFER_STATUS_LOG_ERROR(FAILED)
+               << "Unknown output mindspore data type " << static_cast<int>(info.DataType());
       }
+      find_batch_size(tensor_info.shape);
       api_model_info->output_tensor_infos.push_back(tensor_info);
+      api_model_info->output_names.push_back(info.Name());
     }
   }
   if (!first_dim_same) {
@@ -221,16 +242,21 @@ Status MindSporeModelWrap::UnloadModel(uint32_t model_id) {
 
 Status MindSporeModelWrap::ExecuteModel(uint32_t model_id, const RequestBase &request, serving::ReplyBase *reply) {
   MSI_EXCEPTION_IF_NULL(reply);
-  FuncMakeInBuffer func_in = [&request](size_t index) {
+  FuncMakeInBuffer func_in = [&request](size_t index, const std::string &name) {
     auto input_tensor = request[index];
-    return api::Buffer(input_tensor->data(), input_tensor->data_size());
+    return mindspore::MSTensor::CreateRefTensor(name, TransInferDataType2ApiTypeId(input_tensor->data_type()),
+                                                input_tensor->shape(), const_cast<uint8_t *>(input_tensor->data()),
+                                                input_tensor->data_size());
   };
 
-  FuncMakeOutTensor func_out = [&reply](const api::Buffer &result_tensor, DataType data_type,
+  FuncMakeOutTensor func_out = [&reply](const mindspore::MSTensor &result_tensor, DataType data_type,
                                         const std::vector<int64_t> &shape) {
+    if (result_tensor.IsDevice()) {
+      MSI_LOG_EXCEPTION << "Can not support device type tensor";
+    }
     auto tensor = reply->add();
     MSI_EXCEPTION_IF_NULL(tensor);
-    tensor->set_data(result_tensor.Data(), result_tensor.DataSize());
+    tensor->set_data(result_tensor.Data().get(), result_tensor.DataSize());
     tensor->set_data_type(data_type);
     tensor->set_shape(shape);
   };
@@ -240,17 +266,17 @@ Status MindSporeModelWrap::ExecuteModel(uint32_t model_id, const RequestBase &re
 Status MindSporeModelWrap::ExecuteModel(uint32_t model_id, const std::vector<TensorBasePtr> &request,
                                         std::vector<TensorBasePtr> *reply) {
   MSI_EXCEPTION_IF_NULL(reply);
-  FuncMakeInBuffer func_in = [&request](size_t index) {
+  FuncMakeInBuffer func_in = [&request](size_t index, const std::string &name) {
     auto &input_tensor = request[index];
-    auto api_buffer_wrap = std::dynamic_pointer_cast<ApiBufferTensorWrap>(input_tensor);
-    if (api_buffer_wrap) {
-      return api_buffer_wrap->GetBuffer();
-    } else {
-      return api::Buffer(input_tensor->data(), input_tensor->data_size());
-    }
+    return mindspore::MSTensor::CreateRefTensor(name, TransInferDataType2ApiTypeId(input_tensor->data_type()),
+                                                input_tensor->shape(), const_cast<uint8_t *>(input_tensor->data()),
+                                                input_tensor->data_size());
   };
-  FuncMakeOutTensor func_out = [&reply](const api::Buffer &result_tensor, DataType data_type,
+  FuncMakeOutTensor func_out = [&reply](const mindspore::MSTensor &result_tensor, DataType data_type,
                                         const std::vector<int64_t> &shape) {
+    if (result_tensor.IsDevice()) {
+      MSI_LOG_EXCEPTION << "Can not support device type tensor";
+    }
     auto tensor = std::make_shared<ApiBufferTensorWrap>(result_tensor);
     tensor->set_data_type(data_type);
     tensor->set_shape(shape);
@@ -273,14 +299,14 @@ Status MindSporeModelWrap::ExecuteModelCommon(uint32_t model_id, size_t request_
     return INFER_STATUS_LOG_ERROR(FAILED) << "Inputs size not match, request inputs size " << request_size
                                           << ", model inputs size " << input_names.size();
   }
-  std::vector<api::Buffer> inputs;
+  std::vector<mindspore::MSTensor> inputs;
   for (size_t i = 0; i < input_names.size(); i++) {
-    inputs.push_back(in_func(i));
+    inputs.push_back(in_func(i, input_names[i]));
   }
-  std::vector<api::Buffer> outputs;
-  api::Status status = model->Predict(inputs, &outputs);
-  if (!status.IsSuccess()) {
-    MSI_LOG_ERROR << "Predict failed: " << status.StatusMessage();
+  std::vector<mindspore::MSTensor> outputs;
+  mindspore::Status status = model->Predict(inputs, &outputs);
+  if (!status.IsOk()) {
+    MSI_LOG_ERROR << "Predict failed: " << status.ToString();
     return FAILED;
   }
   if (outputs.size() != output_names.size()) {
@@ -331,34 +357,24 @@ ssize_t MindSporeModelWrap::GetBatchSize(uint32_t model_id) const {
   return model_info.batch_size;
 }
 
-TensorBasePtr MindSporeModelWrap::MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const {
-  return std::make_shared<ApiBufferTensorWrap>(data_type, shape);
-}
-
 bool MindSporeModelWrap::CheckModelSupport(DeviceType device_type, ModelType model_type) const {
   std::string device_type_str;
   switch (device_type) {
     case kDeviceTypeAscendMS:
-      device_type_str = api::kDeviceTypeAscend910;
+      device_type_str = mindspore::kDeviceTypeAscend910;
       break;
     case kDeviceTypeAscendCL:
-      device_type_str = api::kDeviceTypeAscend310;
+      device_type_str = mindspore::kDeviceTypeAscend310;
       break;
     default:
       return false;
   }
-  return api::Model::CheckModelSupport(device_type_str, model_type);
+  return mindspore::Model::CheckModelSupport(device_type_str, model_type);
 }
 
 ApiBufferTensorWrap::ApiBufferTensorWrap() = default;
 
-ApiBufferTensorWrap::ApiBufferTensorWrap(DataType type, const std::vector<int64_t> &shape)
-    : type_(type), shape_(shape) {
-  size_t data_len = itemsize() * TensorBase::element_cnt();
-  buffer_.ResizeData(data_len);
-}
-
-ApiBufferTensorWrap::ApiBufferTensorWrap(const api::Buffer &buffer) : buffer_(buffer) {}
+ApiBufferTensorWrap::ApiBufferTensorWrap(const mindspore::MSTensor &tensor) : tensor_(tensor) {}
 
 ApiBufferTensorWrap::~ApiBufferTensorWrap() = default;
 

mindspore_serving/ccsrc/worker/inference/mindspore_model_wrap.h

@@ -34,7 +34,7 @@ struct ApiModelInfo {
   std::vector<serving::TensorInfo> input_tensor_infos;
   std::vector<std::string> output_names;
   std::vector<serving::TensorInfo> output_tensor_infos;
-  std::shared_ptr<api::Model> model;
+  std::shared_ptr<mindspore::Model> model;
   uint32_t batch_size = 0;
   std::string device_type;
   uint32_t device_id = 0;
@@ -69,27 +69,25 @@ class MindSporeModelWrap : public InferSession {
 
   ssize_t GetBatchSize(uint32_t model_id) const override;
 
-  TensorBasePtr MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const override;
-
   bool CheckModelSupport(DeviceType device_type, ModelType model_type) const override;
 
  private:
   std::unordered_map<uint32_t, ApiModelInfo> model_map_;
   uint32_t model_index_ = 0;
 
-  using FuncMakeInBuffer = std::function<api::Buffer(size_t index)>;
+  using FuncMakeInBuffer = std::function<mindspore::MSTensor(size_t index, const std::string &name)>;
   using FuncMakeOutTensor =
-    std::function<void(const api::Buffer, DataType data_type, const std::vector<int64_t> &shape)>;
+    std::function<void(const mindspore::MSTensor, DataType data_type, const std::vector<int64_t> &shape)>;
   Status ExecuteModelCommon(uint32_t model_id, size_t request_size, const FuncMakeInBuffer &in_func,
                             const FuncMakeOutTensor &out_func);
   Status GetModelInfos(ApiModelInfo *model_info);
+  std::shared_ptr<Context> TransformModelContext(const std::map<std::string, std::string> &other_options);
 };
 
 class ApiBufferTensorWrap : public TensorBase {
  public:
   ApiBufferTensorWrap();
-  ApiBufferTensorWrap(DataType type, const std::vector<int64_t> &shape);
-  explicit ApiBufferTensorWrap(const api::Buffer &buffer);
+  explicit ApiBufferTensorWrap(const mindspore::MSTensor &buffer);
   ~ApiBufferTensorWrap() override;
 
   void set_data_type(DataType type) override { type_ = type; }
@@ -98,28 +96,26 @@ class ApiBufferTensorWrap : public TensorBase {
   void set_shape(const std::vector<int64_t> &shape) override { shape_ = shape; }
   std::vector<int64_t> shape() const override { return shape_; }
 
-  const uint8_t *data() const override { return static_cast<const uint8_t *>(buffer_.Data()); }
-  size_t data_size() const override { return buffer_.DataSize(); }
+  const uint8_t *data() const override { return static_cast<const uint8_t *>(tensor_.Data().get()); }
+  size_t data_size() const override { return tensor_.DataSize(); }
 
-  bool resize_data(size_t data_len) override { return buffer_.ResizeData(data_len); }
-  uint8_t *mutable_data() override { return static_cast<uint8_t *>(buffer_.MutableData()); }
+  bool resize_data(size_t data_len) override { MSI_LOG_EXCEPTION << "ApiBufferTensorWrap not support resize data"; }
+  uint8_t *mutable_data() override { return static_cast<uint8_t *>(tensor_.MutableData()); }
 
   // For kMSI_String and kMSI_Bytes
-  void clear_bytes_data() override { MSI_LOG_EXCEPTION << "Not support for api::Buffer Tensor"; }
+  void clear_bytes_data() override { MSI_LOG_EXCEPTION << "Not support for mindspore::Buffer Tensor"; }
   void add_bytes_data(const uint8_t *data, size_t bytes_len) override {
-    MSI_LOG_EXCEPTION << "Not support for api::Buffer Tensor";
+    MSI_LOG_EXCEPTION << "Not support for mindspore::MSTensor Tensor";
   }
-  size_t bytes_data_size() const override { MSI_LOG_EXCEPTION << "Not support for api::Buffer Tensor"; }
+  size_t bytes_data_size() const override { MSI_LOG_EXCEPTION << "Not support for mindspore::Buffer Tensor"; }
   void get_bytes_data(size_t index, const uint8_t **data, size_t *bytes_len) const override {
-    MSI_LOG_EXCEPTION << "Not support for api::Buffer Tensor";
+    MSI_LOG_EXCEPTION << "Not support for mindspore::MSTensor Tensor";
   }
 
-  api::Buffer GetBuffer() const { return buffer_; }
-
  private:
   DataType type_ = kMSI_Unknown;
   std::vector<int64_t> shape_;
-  api::Buffer buffer_;
+  mindspore::MSTensor tensor_;
 };
 
 }  // namespace serving

mindspore_serving/ccsrc/worker/model.cc

@@ -30,8 +30,4 @@ std::vector<TensorInfo> AscendModelServable::GetOutputInfos() const { return ses
 
 uint64_t AscendModelServable::GetBatchSize() const { return session_->GetBatchSize(model_id_); }
 
-TensorBasePtr AscendModelServable::MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const {
-  return session_->MakeInferenceTensor(data_type, shape);
-}
-
 }  // namespace mindspore::serving

mindspore_serving/ccsrc/worker/model.h

@@ -39,9 +39,6 @@ class ServableBase {
   virtual std::vector<TensorInfo> GetInputInfos() const = 0;
   virtual std::vector<TensorInfo> GetOutputInfos() const = 0;
   virtual uint64_t GetBatchSize() const = 0;
-  virtual TensorBasePtr MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const {
-    return nullptr;
-  }
 };
 
 class AscendModelServable : public ServableBase {
@@ -55,7 +52,6 @@ class AscendModelServable : public ServableBase {
   std::vector<TensorInfo> GetInputInfos() const override;
   std::vector<TensorInfo> GetOutputInfos() const override;
   uint64_t GetBatchSize() const override;
-  TensorBasePtr MakeInferenceTensor(DataType data_type, const std::vector<int64_t> &shape) const override;
 
  private:
   std::shared_ptr<serving::InferSession> session_{nullptr};

mindspore_serving/ccsrc/worker/work_executor.cc

@@ -192,14 +192,12 @@ void WorkExecutor::InitPrePostprocess() {
 
 void WorkExecutor::InitInputTensors() {
   inference_inputs_.clear();
-  for (const auto &input_info : input_infos_) {
-    TensorBasePtr tensor = servable_->MakeInferenceTensor(input_info.data_type, input_info.shape);
-    if (tensor == nullptr) {
-      tensor = std::make_shared<Tensor>();
-      tensor->set_data_type(input_info.data_type);
-      tensor->set_shape(input_info.shape);
-      tensor->resize_data(input_info.size);
-    }
+  for (size_t i = 0; i < input_infos_.size(); i++) {
+    auto &input_info = input_infos_[i];
+    auto tensor = std::make_shared<Tensor>();
+    tensor->set_data_type(input_info.data_type);
+    tensor->set_shape(input_info.shape);
+    tensor->resize_data(input_info.size);
     inference_inputs_.push_back(tensor);
   }
 }

mindspore_serving/client/python/client.py

@@ -92,7 +92,7 @@ def _create_numpy_from_tensor(tensor):
         ms_service_pb2.MS_BOOL: np.bool,
         ms_service_pb2.MS_INT8: np.int8,
         ms_service_pb2.MS_UINT8: np.uint8,
-        ms_service_pb2.MS_INT16: ms_service_pb2.MS_INT16,
+        ms_service_pb2.MS_INT16: np.int16,
         ms_service_pb2.MS_UINT16: np.uint16,
         ms_service_pb2.MS_INT32: np.int32,
         ms_service_pb2.MS_UINT32: np.uint32,

mindspore_serving/log.py

@@ -0,0 +1,522 @@
+# Copyright 2020 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+"""
+log module
+"""
+import sys
+import os
+import stat
+import time
+import logging
+from logging.handlers import RotatingFileHandler
+import traceback
+import threading
+import platform
+
+if platform.system() != "Windows":
+    import fcntl
+
+__all__ = ['get_level', 'get_log_config']
+
+# The lock for setting up the logger
+_setup_logger_lock = threading.Lock()
+
+# When getting the logger, Used to check whether
+# the logger already exists
+_global_logger = None
+
+# The flag for enable console output
+_std_on = '1'
+# The flag for disable console output
+_std_off = '0'
+# Rotating max bytes, default is 50M
+_logger_def_max_bytes = '52428800'
+# Rotating backup count, default is 30
+_logger_def_backup_count = '30'
+# The default log level
+_logger_def_level = '2'
+
+# Log level name and level mapping
+_name_to_level = {
+    'ERROR': 40,
+    'WARNING': 30,
+    'INFO': 20,
+    'DEBUG': 10,
+}
+
+# GLog level and level name
+_gloglevel_to_name = {
+    '3': 'ERROR',
+    '2': 'WARNING',
+    '1': 'INFO',
+    '0': 'DEBUG',
+}
+
+# The mapping of logger configurations to glog configurations
+_confmap_dict = {'level': 'GLOG_v', 'console': 'GLOG_logtostderr', 'filepath': 'GLOG_log_dir',
+                 'maxBytes': 'logger_maxBytes', 'backupCount': 'logger_backupCount'}
+
+
+class _MultiCompatibleRotatingFileHandler(RotatingFileHandler):
+    """Inherit RotatingFileHandler for multiprocess compatibility."""
+
+    def rolling_rename(self):
+        """Rolling rename log files and set permission of Log file"""
+        for i in range(self.backupCount - 1, 0, -1):
+            sfn = self.rotation_filename("%s.%d" % (self.baseFilename, i))
+            dfn = self.rotation_filename("%s.%d" % (self.baseFilename, i + 1))
+            if os.path.exists(sfn):
+                if os.path.exists(dfn):
+                    os.remove(dfn)
+                # Modify the permission of Log file
+                os.chmod(sfn, stat.S_IREAD)
+                os.rename(sfn, dfn)
+
+    def doRollover(self):
+        """Override doRollover for multiprocess compatibility
+        and setting permission of Log file"""
+        if self.stream:
+            self.stream.close()
+            self.stream = None
+
+        # Attain an exclusive lock with bloking mode by `fcntl` module.
+        with open(self.baseFilename, 'a') as file_pointer:
+            if platform.system() != "Windows":
+                fcntl.lockf(file_pointer.fileno(), fcntl.LOCK_EX)
+
+        if self.backupCount > 0:
+            self.rolling_rename()
+
+        dfn = self.rotation_filename(self.baseFilename + ".1")
+        if os.path.exists(dfn):
+            os.remove(dfn)
+        # Modify the permission of Log file
+        os.chmod(self.baseFilename, stat.S_IREAD)
+        self.rotate(self.baseFilename, dfn)
+
+        with open(self.baseFilename, 'a'):
+            # Modify the permission of Log file
+            os.chmod(self.baseFilename, stat.S_IREAD | stat.S_IWRITE)
+
+        if not self.delay:
+            self.stream = self._open()
+
+
+class _DataFormatter(logging.Formatter):
+    """Log formatter"""
+
+    def __init__(self, sub_module, fmt=None, **kwargs):
+        """
+        Initialization of logFormatter.
+
+        Args:
+            sub_module (str): The submodule name.
+            fmt (str): Specified format pattern. Default: None.
+        """
+        super(_DataFormatter, self).__init__(fmt=fmt, **kwargs)
+        self.sub_module = sub_module.upper()
+
+    def formatTime(self, record, datefmt=None):
+        """
+        Override formatTime for uniform format %Y-%m-%d-%H:%M:%S.SSS.SSS
+
+        Args:
+            record (str): Log record.
+            datefmt (str): Date format.
+
+        Returns:
+            str, formatted timestamp.
+        """
+        created_time = self.converter(record.created)
+        if datefmt:
+            return time.strftime(datefmt, created_time)
+
+        timestamp = time.strftime('%Y-%m-%d-%H:%M:%S', created_time)
+        msecs = str(round(record.msecs * 1000))
+        # Format the time stamp
+        return f'{timestamp}.{msecs[:3]}.{msecs[3:]}'
+
+    def format(self, record):
+        """
+        Apply log format with specified pattern.
+
+        Args:
+            record (str): Format pattern.
+
+        Returns:
+            str, formatted log content according to format pattern.
+        """
+        # NOTICE: when the Installation directory of mindspore changed,
+        # ms_home_path must be changed
+        ms_install_home_path = 'mindspore'
+        idx = record.pathname.rfind(ms_install_home_path)
+        if idx >= 0:
+            # Get the relative path of the file
+            record.filepath = record.pathname[idx:]
+        else:
+            record.filepath = record.pathname
+        record.sub_module = self.sub_module
+        return super().format(record)
+
+
+def _get_logger():
+    """
+    Get logger instance.
+
+    Returns:
+        Logger, a logger.
+    """
+    if _global_logger:
+        return _global_logger
+
+    kwargs = _get_env_config()
+    _verify_config(kwargs)
+    logger = _setup_logger(_adapt_cfg(kwargs))
+    return logger
+
+
+def _adapt_cfg(kwargs):
+    """
+    Glog configurations converted to logger configurations.
+
+    Args:
+        kwargs (dict): The dictionary of log configurations.
+
+            - console (str): Whether to output log to stdout.
+            - level (str): Log level.
+            - filepath (str): The path for saving logs, if console is false, a file path must be assigned.
+            - maxBytes (str): The Maximum value of a log file for rotating, only valid if console is false.
+            - backupCount (str): The count of rotating backup log files, only valid if console is false.
+
+    Returns:
+        Dict, the input parameter dictionary.
+    """
+    kwargs['level'] = _gloglevel_to_name.get(kwargs.get('level', _logger_def_level))
+    kwargs['console'] = not kwargs.get('console') == _std_off
+    kwargs['maxBytes'] = int(kwargs.get('maxBytes', _logger_def_max_bytes))
+    kwargs['backupCount'] = int(kwargs.get('backupCount', _logger_def_backup_count))
+    return kwargs
+
+
+def info(msg, *args, **kwargs):
+    """
+    Log a message with severity 'INFO' on the MindSpore logger.
+
+    Examples:
+        >>> from mindspore_serving import log as logger
+        >>> logger.info("The arg(%s) is: %r", name, arg)
+    """
+    _get_logger().info(msg, *args, **kwargs)
+
+
+def debug(msg, *args, **kwargs):
+    """
+    Log a message with severity 'DEBUG' on the MindSpore logger.
+
+    Examples:
+        >>> from mindspore_serving import log as logger
+        >>> logger.debug("The arg(%s) is: %r", name, arg)
+    """
+    _get_logger().debug(msg, *args, **kwargs)
+
+
+def error(msg, *args, **kwargs):
+    """Log a message with severity 'ERROR' on the MindSpore logger."""
+    _get_logger().error(msg, *args, **kwargs)
+
+
+def warning(msg, *args, **kwargs):
+    """Log a message with severity 'WARNING' on the MindSpore logger."""
+    _get_logger().warning(msg, *args, **kwargs)
+
+
+def get_level():
+    """
+    Get the logger level.
+
+    Returns:
+        str, the Log level includes 3(ERROR), 2(WARNING), 1(INFO), 0(DEBUG).
+
+    Examples:
+        >>> import os
+        >>> os.environ['GLOG_v'] = '0'
+        >>> from mindspore_serving import log as logger
+        >>> logger.get_level()
+    """
+    # level and glog level mapping dictionary
+    level_to_glog_level = dict(zip(_name_to_level.values(), _gloglevel_to_name.keys()))
+
+    return level_to_glog_level.get(_get_logger().getEffectiveLevel())
+
+
+def _get_formatter():
+    """
+    Get the string of log formatter.
+
+    Returns:
+        str, the string of log formatter.
+    """
+    formatter = '[%(levelname)s] %(sub_module)s(%(process)d:' \
+                '%(thread)d,%(processName)s):%(asctime)s ' \
+                '[%(filepath)s:%(lineno)d] %(message)s'
+    return formatter
+
+
+def _get_env_config():
+    """
+    Get configurations from environment variables.
+
+    Returns:
+        Dict, the dictionary of configurations.
+    """
+    config_dict = {}
+    for key, env_value in _confmap_dict.items():
+        value = os.environ.get(env_value)
+        if value:
+            config_dict[key] = value.strip()
+    return config_dict
+
+
+def _verify_config(kwargs):
+    """
+    Verify log configurations.
+
+    Args:
+        kwargs (dict): The dictionary of log configurations.
+
+            - console (str): Whether to output log to stdout.
+            - level (str): Log level.
+            - filepath (str): The path for saving logs, if console is false, a file path must be assigned.
+            - maxBytes (str): The Maximum value of a log file for rotating, only valid if console is false.
+            - backupCount (str): The count of rotating backup log files, only valid if console is false.
+    """
+    # Check the input value of level
+    level = kwargs.get('level', None)
+    if level is not None:
+        _verify_level(level)
+
+    # Check the input value of console
+    console = kwargs.get('console', None)
+    file_path = kwargs.get('filepath', None)
+
+    if console is not None:
+        if not console.isdigit() or console not in (_std_off, _std_on):
+            raise ValueError(f'Incorrect value, The value of {_confmap_dict["console"]} must be 0 or 1,'
+                             f' Output log to console, configure to 1.')
+
+        if console == _std_off and not file_path:
+            raise ValueError(f'When {_confmap_dict["console"]} is set to 0, The directory of '
+                             f'saving log must be set, {_confmap_dict["filepath"]} cannot be empty.')
+
+        # Check the input value of filepath
+        if console == _std_off and file_path is not None:
+            file_real_path = os.path.realpath(file_path)
+            if not os.path.exists(file_real_path):
+                raise ValueError(f'The file path does not exist. '
+                                 f'{_confmap_dict["filepath"]}:{file_path}')
+
+        # Check the input value of maxBytes
+        max_bytes = kwargs.get('maxBytes', None)
+        if console == _std_off and max_bytes is not None:
+            if not max_bytes.isdigit():
+                raise ValueError(f'Incorrect value, The value of {_confmap_dict["maxBytes"]} must be positive integer. '
+                                 f'{_confmap_dict["maxBytes"]}:{max_bytes}')
+
+        # Check the input value of backupCount
+        backup_count = kwargs.get('backupCount', None)
+        if console == _std_off and backup_count is not None:
+            if not backup_count.isdigit():
+                raise ValueError(f'Incorrect value, The value of {_confmap_dict["backupCount"]} must be positive '
+                                 f'integer. {_confmap_dict["backupCount"]}:{backup_count}')
+
+
+def _verify_level(level):
+    """
+    Verify log level.
+
+    Args:
+        level (str): The log level.
+    """
+    level_name = _gloglevel_to_name.get(level, None)
+
+    # Check the value of input level
+    if level_name not in _name_to_level:
+        raise ValueError(f'Incorrect log level:{level}, Please check the log level configuration, '
+                         f'desired log level :{_gloglevel_to_name}')
+
+
+def get_log_config():
+    """
+    Get logger configurations.
+
+    Returns:
+        Dict, the dictionary of logger configurations.
+
+    Examples:
+        >>> import os
+        >>> os.environ['GLOG_v'] = '1'
+        >>> os.environ['GLOG_logtostderr'] = '0'
+        >>> os.environ['GLOG_log_dir'] = '/var/log/mindspore'
+        >>> os.environ['logger_maxBytes'] = '5242880'
+        >>> os.environ['logger_backupCount'] = '10'
+        >>> from mindspore_serving import log as logger
+        >>> logger.get_log_config()
+    """
+    logger = _get_logger()
+    handler = logger.handlers[0]
+    config_dict = {}
+    config_dict['GLOG_v'] = get_level()
+    config_dict['GLOG_logtostderr'] = _std_on
+
+    if handler.name == 'FileHandler':
+        config_dict['GLOG_logtostderr'] = _std_off
+        # Separating file path and name
+        file_path_and_name = os.path.split(handler.baseFilename)
+        config_dict['GLOG_log_dir'] = file_path_and_name[0]
+        config_dict['logger_maxBytes'] = handler.maxBytes
+        config_dict['logger_backupCount'] = handler.backupCount
+    return config_dict
+
+
+def _clear_handler(logger):
+    """Clear the handlers that has been set, avoid repeated loading"""
+    for handler in logger.handlers:
+        logger.removeHandler(handler)
+
+
+def _find_caller(stack_info=False):
+    """
+    Find the stack frame of the caller.
+
+    Override findCaller on the logger, Support for getting log record.
+    Find the stack frame of the caller so that we can note the source
+    file name, function name and line number.
+
+    Args:
+        stack_info (bool): If the value is true, print stack information to the log. Default: False.
+
+    Returns:
+        tuple, the tuple of the frame data.
+    """
+    # pylint: disable=protected-access
+    f = sys._getframe(3)
+    sinfo = None
+    # log_file is used to check caller stack frame
+    log_file = os.path.normcase(f.f_code.co_filename)
+    f = f.f_back
+    rv = "(unknown file)", 0, "(unknown function)", None
+    while f:
+        co = f.f_code
+        filename = os.path.normcase(co.co_filename)
+        if log_file == filename:
+            f = f.f_back
+            continue
+        if stack_info:
+            sinfo = _get_stack_info(f)
+        rv = (co.co_filename, f.f_lineno, co.co_name, sinfo)
+        break
+    return rv
+
+
+def _get_stack_info(frame):
+    """
+    Get the stack informations.
+
+    Args:
+        frame(frame): the frame requiring informations.
+
+    Returns:
+        str, the string of the stack informations.
+    """
+    sinfo = None
+    stack_prefix = 'Stack (most recent call last):\n'
+    sinfo = stack_prefix + "".join(traceback.format_stack(frame))
+    return sinfo
+
+
+def _setup_logger(kwargs):
+    """
+    Set up the logger.
+
+    Args:
+        kwargs (dict): The dictionary of log configurations.
+
+            - console (bool): Whether to output log to stdout. Default: True.
+            - level (str): Log level. Default: WARNING.
+            - filepath (str): The path for saving logs, if console is false, a file path must be assigned.
+            - maxBytes (int): The Maximum value of a log file for rotating, only valid if console is false.
+              Default: 52428800.
+            - backupCount (int): The count of rotating backup log files, only valid if console is false. Default: 30.
+
+    Returns:
+        Logger, well-configured logger.
+    """
+
+    # The name of Submodule
+    sub_module = 'SERVING'
+    # The name of Base log file
+    log_name = 'mindspore.log'
+
+    global _global_logger
+
+    _setup_logger_lock.acquire()
+    try:
+        if _global_logger:
+            return _global_logger
+
+        logger = logging.getLogger(name=f'{sub_module}.{log_name}')
+        # Override findCaller on the logger, Support for getting log record
+        logger.findCaller = _find_caller
+        console = kwargs.get('console', True)
+        # Set log level
+        logger.setLevel(kwargs.get('level', logging.WARNING))
+        # Set "propagate" attribute to False, stop searching up the hierarchy,
+        # avoid to load the handler of the root logger
+        logger.propagate = False
+        # Get the formatter for handler
+        formatter = _get_formatter()
+
+        # Clean up handle to avoid repeated loading
+        _clear_handler(logger)
+
+        # Set streamhandler for the console appender
+        if console:
+            console_handler = logging.StreamHandler(sys.stderr)
+            console_handler.name = 'StreamHandler'
+            console_handler.formatter = _DataFormatter(sub_module, formatter)
+            logger.addHandler(console_handler)
+
+        # Set rotatingFileHandler for the file appender
+        else:
+            # filepath cannot be null, checked in function _verify_config ()
+            logfile_dir = os.path.realpath(kwargs.get('filepath'))
+            file_name = f'{logfile_dir}/{log_name}'
+            logfile_handler = _MultiCompatibleRotatingFileHandler(
+                filename=file_name,
+                # Rotating max bytes, default is 50M
+                maxBytes=kwargs.get('maxBytes', _logger_def_max_bytes),
+                # Rotating backup count, default is 30
+                backupCount=kwargs.get('backupCount', _logger_def_backup_count),
+                encoding='utf8'
+            )
+            logfile_handler.name = 'FileHandler'
+            logfile_handler.formatter = _DataFormatter(sub_module, formatter)
+            logger.addHandler(logfile_handler)
+
+        _global_logger = logger
+
+    finally:
+        _setup_logger_lock.release()
+    return _global_logger

mindspore_serving/master/__init__.py

@@ -14,7 +14,6 @@
 # ============================================================================
 """MindSpore Serving Master"""
 
-from mindspore_serving.worker import _check_version
 from ._master import start_grpc_server, start_restful_server, start_master_server, stop
 
 __all__ = []

mindspore_serving/master/_master.py

@@ -17,6 +17,7 @@
 import threading
 from functools import wraps
 from mindspore_serving.worker import check_type
+from mindspore_serving import log as logger
 from mindspore_serving._mindspore_serving import Master_
 
 _wait_and_clear_thread = None
@@ -27,6 +28,7 @@ def _start_wait_and_clear():
     """Start thread waiting for catch ctrl+c, and clear env"""
 
     def thread_func():
+        logger.info("Serving master: wait for Ctrl+C to exit ------------------------------------")
         print("Serving master: wait for Ctrl+C to exit ------------------------------------")
         Master_.wait_and_clear()
 

mindspore_serving/worker/_check_version.py

@@ -16,6 +16,7 @@
 import os
 import sys
 from pathlib import Path
+from mindspore_serving import log as logger
 
 
 class AscendEnvChecker:
@@ -103,53 +104,53 @@ class AscendEnvChecker:
                 else:
                     os.environ['LD_LIBRARY_PATH'] = self.tbe_path
             else:
-                raise EnvironmentError(
+                logger.warning(
                     f"No such directory: {self.tbe_path}, Please check if Ascend 910 AI software package is "
-                    "installed correctly.")
+                    f"installed correctly.")
 
         if Path(self.op_impl_path).is_dir():
             sys.path.append(self.op_impl_path)
         else:
-            raise EnvironmentError(
+            logger.warning(
                 f"No such directory: {self.op_impl_path}, Please check if Ascend 910 AI software package is "
-                "installed correctly.")
+                f"installed correctly.")
 
         if Path(self.cce_path).is_dir():
             os.environ['PATH'] = self.cce_path + ":" + os.environ['PATH']
         else:
-            raise EnvironmentError(
+            logger.warning(
                 f"No such directory: {self.cce_path}, Please check if Ascend 910 AI software package is "
-                "installed correctly.")
+                f"installed correctly.")
 
         if self.op_path is None:
             pass
         elif Path(self.op_path).is_dir():
             os.environ['ASCEND_OPP_PATH'] = self.op_path
         else:
-            raise EnvironmentError(
+            logger.warning(
                 f"No such directory: {self.op_path}, Please check if Ascend 910 AI software package is "
-                "installed correctly.")
+                f"installed correctly.")
 
     def _check_env(self):
         """ascend dependence path check"""
         if self.path is None or self.path_check not in self.path:
-            print("Can not find ccec_compiler(need by mindspore-ascend), please check if you have set env "
-                  "PATH, you can reference to the installation guidelines https://www.mindspore.cn/install")
+            logger.warning("Can not find ccec_compiler(need by mindspore-ascend), please check if you have set env "
+                           "PATH, you can reference to the installation guidelines https://www.mindspore.cn/install")
 
         if self.python_path is None or self.python_path_check not in self.python_path:
-            print(
+            logger.warning(
                 "Can not find tbe op implement(need by mindspore-ascend), please check if you have set env "
                 "PYTHONPATH, you can reference to the installation guidelines "
                 "https://www.mindspore.cn/install")
 
         if self.ld_lib_path is None or not (self.ld_lib_path_check_fwk in self.ld_lib_path and
                                             self.ld_lib_path_check_addons in self.ld_lib_path):
-            print("Can not find driver so(need by mindspore-ascend), please check if you have set env "
-                  "LD_LIBRARY_PATH, you can reference to the installation guidelines "
-                  "https://www.mindspore.cn/install")
+            logger.warning("Can not find driver so(need by mindspore-ascend), please check if you have set env "
+                           "LD_LIBRARY_PATH, you can reference to the installation guidelines "
+                           "https://www.mindspore.cn/install")
 
         if self.ascend_opp_path is None or self.ascend_opp_path_check not in self.ascend_opp_path:
-            print(
+            logger.warning(
                 "Can not find opp path (need by mindspore-ascend), please check if you have set env ASCEND_OPP_PATH, "
                 "you can reference to the installation guidelines https://www.mindspore.cn/install")
 

mindspore_serving/worker/_worker.py

@@ -16,6 +16,7 @@
 
 import threading
 from functools import wraps
+from mindspore_serving import log as logger
 from mindspore_serving._mindspore_serving import Worker_
 from . import context
 from .task import _start_py_task
@@ -28,6 +29,7 @@ def _start_wait_and_clear():
     """Waiting for Ctrl+C, and clear up environment"""
 
     def thread_func():
+        logger.info("Serving worker: wait for Ctrl+C to exit ------------------------------------")
         print("Serving worker: wait for Ctrl+C to exit ------------------------------------")
         Worker_.wait_and_clear()
 
@@ -179,6 +181,7 @@ def start_servable_in_master(servable_directory, servable_name, version_number=0
             "Ascend" means the device type can be Ascend910 or Ascend310, etc.
             "Davinci" has the same meaning as "Ascend".
             None means the device type is determined by the MindSpore environment.
+        device_id (int): The id of the device the model loads into and runs in.
 
     Examples:
         >>> import os

mindspore_serving/worker/check_type.py

@@ -33,7 +33,7 @@ def check_and_as_str_tuple_list(arg_name, strs):
             if not item:
                 raise RuntimeError(f"The item of parameter '{arg_name}' should not be empty str")
             if item in str_list:
-                raise RuntimeError(f"The item name '{item}' in parameter '{arg_name}' should not be repeated")
+                raise RuntimeError(f"The item value '{item}' in parameter '{arg_name}' should not be repeated")
             str_list.append(item)
 
     return tuple(strs)
@@ -53,18 +53,25 @@ def check_bool(arg_name, bool_val):
         raise RuntimeError(f"Parameter '{arg_name}' should be bool, but actually {type(bool_val)}")
 
 
-def check_int(arg_name, int_val, mininum=None, maximum=None):
+def check_int(arg_name, int_val, mininum=None, maximum=None, is_tuple_item=False):
     """Check whether the input parameters are reasonable int input"""
+    if not is_tuple_item:
+        prefix = f"Parameter '{arg_name}'"
+    else:
+        prefix = f"The item value '{int_val}' in parameter '{arg_name}'"
+
+    if isinstance(int_val, bool):
+        raise RuntimeError(f"{prefix} should be int, but actually {type(int_val)}")
     if not isinstance(int_val, int):
-        raise RuntimeError(f"Parameter '{arg_name}' should be int, but actually {type(int_val)}")
+        raise RuntimeError(f"{prefix} should be int, but actually {type(int_val)}")
     if mininum is not None and int_val < mininum:
         if maximum is not None:
-            raise RuntimeError(f"Parameter '{arg_name}' should be in range [{mininum},{maximum}]")
-        raise RuntimeError(f"Parameter '{arg_name}' should be >= {mininum}")
+            raise RuntimeError(f"{prefix} should be in range [{mininum},{maximum}]")
+        raise RuntimeError(f"{prefix} should be >= {mininum}")
     if maximum is not None and int_val > maximum:
         if mininum is not None:
-            raise RuntimeError(f"Parameter '{arg_name}' should be in range [{mininum},{maximum}]")
-        raise RuntimeError(f"Parameter '{arg_name}' should be <= {maximum}")
+            raise RuntimeError(f"{prefix} should be in range [{mininum},{maximum}]")
+        raise RuntimeError(f"{prefix} should be <= {maximum}")
 
 
 def check_ip_port(arg_name, port):
@@ -85,11 +92,9 @@ def check_and_as_int_tuple_list(arg_name, ints, mininum=None, maximum=None):
     if isinstance(ints, (tuple, list)):
         int_list = []
         for item in ints:
-            if not isinstance(item, int):
-                raise RuntimeError(f"The item of parameter '{arg_name}' should be int, but actually {type(item)}")
             if item in int_list:
-                raise RuntimeError(f"The item name '{item}' in parameter '{arg_name}' should not be repeated")
-            check_int(arg_name, item, mininum, maximum)
+                raise RuntimeError(f"The item value '{item}' in parameter '{arg_name}' should not be repeated")
+            check_int(arg_name, item, mininum, maximum, True)
             int_list.append(item)
 
     return tuple(ints)

mindspore_serving/worker/register/method.py

@@ -22,6 +22,7 @@ from easydict import EasyDict
 from mindspore_serving._mindspore_serving import ServableStorage_, MethodSignature_, PredictPhaseTag_
 from mindspore_serving.worker.common import get_func_name, get_servable_dir
 from mindspore_serving.worker import check_type
+from mindspore_serving import log as logger
 from .preprocess import register_preprocess, check_preprocess
 from .postprocess import register_postprocess, check_postprocess
 
@@ -101,7 +102,7 @@ def _wrap_fun_to_pipeline(fun, input_count):
     argspec_len = len(inspect.signature(fun).parameters)
     if argspec_len != input_count:
         raise RuntimeError(f"function {fun.__name__} input args count {argspec_len} not match "
-                           f"registed in method count {input_count}")
+                           f"registered in method count {input_count}")
 
     @wraps(fun)
     def call_func(instances):
@@ -393,7 +394,7 @@ def _get_method_def_func_meta(method_def_func):
         if inputs_count <= 0:
             raise RuntimeError(f"Invalid '{func_name}' invoke args")
 
-        print(f"call type '{func_name}', inputs count {inputs_count}, outputs count {outputs_count}")
+        logger.info(f"call type '{func_name}', inputs count {inputs_count}, outputs count {outputs_count}")
         func_meta[func_name] = [inputs_count, outputs_count]
 
     if _call_servable_name not in func_meta:
@@ -459,11 +460,12 @@ def register_method(output_names):
             output_tensors = (output_tensors,)
         if len(output_tensors) != len(output_names):
             raise RuntimeError(
-                f"Method return output size {len(output_tensors)} not match registed {len(output_names)}")
+                f"Method return output size {len(output_tensors)} not match registered {len(output_names)}")
 
         method_def_context_.returns = [item.as_pair() for item in output_tensors]
-        print("------------Register method: method_name", method_def_context_.method_name,
-              ", servable_name", method_def_context_.servable_name, ", inputs", input_names, ", outputs", output_names)
+        logger.info(f"Register method: method_name {method_def_context_.method_name} "
+                    f", servable_name {method_def_context_.servable_name}, inputs: {input_names}, outputs: "
+                    f"{output_names}")
 
         global _servable_storage
         _servable_storage.register_method(method_def_context_)

mindspore_serving/worker/register/postprocess.py

@@ -16,6 +16,7 @@
 
 from mindspore_serving._mindspore_serving import PostprocessStorage_
 from mindspore_serving.worker.common import get_servable_dir, get_func_name
+from mindspore_serving import log as logger
 
 
 def check_postprocess(postprocess_name, inputs_count, outputs_count):
@@ -60,5 +61,5 @@ def register_postprocess(func, inputs_count, outputs_count):
     func_name = get_func_name(func)
     name = servable_name + "." + func_name
 
-    print("------------Register postprocess", name, inputs_count, outputs_count)
+    logger.info(f"Register postprocess {name} {inputs_count} {outputs_count}")
     postprocess_storage.register(func, name, inputs_count, outputs_count)

mindspore_serving/worker/register/preprocess.py

@@ -16,6 +16,7 @@
 
 from mindspore_serving._mindspore_serving import PreprocessStorage_
 from mindspore_serving.worker.common import get_servable_dir, get_func_name
+from mindspore_serving import log as logger
 
 
 def check_preprocess(preprocess_name, inputs_count, outputs_count):
@@ -60,5 +61,5 @@ def register_preprocess(func, inputs_count, outputs_count):
     func_name = get_func_name(func)
     name = servable_name + "." + func_name
 
-    print("------------Register preprocess", name, inputs_count, outputs_count)
+    logger.info(f"Register preprocess {name} {inputs_count} {outputs_count}")
     preprocess_storage.register(func, name, inputs_count, outputs_count)

mindspore_serving/worker/register/servable.py

@@ -17,6 +17,7 @@
 from mindspore_serving._mindspore_serving import ServableMeta_
 from mindspore_serving.worker import check_type
 from mindspore_serving.worker.common import get_servable_dir
+from mindspore_serving import log as logger
 from .method import _servable_storage
 
 
@@ -53,7 +54,8 @@ def declare_servable(servable_file, model_format, with_batch_dim=True, options=N
             check_type.check_str("options key", k)
             check_type.check_str(k + " value", w)
     elif isinstance(options, AclOptions):
-        options = _as_options_map(options)
+        # pylint: disable=protected-access
+        options = options._as_options_map()
     elif options is not None:
         raise RuntimeError(f"Parameter 'options' should be None, dict of <str,str> or AclOptions, but "
                            f"gotten {type(options)}")
@@ -65,9 +67,9 @@ def declare_servable(servable_file, model_format, with_batch_dim=True, options=N
         meta.without_batch_dim_inputs = without_batch_dim_inputs
 
     _servable_storage.declare_servable(meta)
-    print("------------Declare servable, servable_name:", meta.servable_name,
-          ", servable_file:", servable_file, ", model_format:", model_format, ", with_batch_dim:", with_batch_dim,
-          ", options:", options, ", without_batch_dim_inputs:", without_batch_dim_inputs)
+    logger.info(f"Declare servable, servable_name: {meta.servable_name} "
+                f", servable_file: {servable_file} , model_format: {model_format},  with_batch_dim: {with_batch_dim} "
+                f", options: {options}, without_batch_dim_inputs: {without_batch_dim_inputs}")
 
 
 class AclOptions:
@@ -201,20 +203,19 @@ class AclOptions:
                                f"'high_precision', actually given '{val}'")
         self.op_select_impl_mode = val
 
-
-def _as_options_map(acl_options):
-    """Transfer AclOptions to dict of str,str"""
-    options = {}
-    if acl_options.insert_op_cfg_path:
-        options['mindspore.option.insert_op_config_file_path'] = acl_options.insert_op_cfg_path
-    if acl_options.input_format:
-        options['mindspore.option.input_format'] = acl_options.input_format
-    if acl_options.input_shape:
-        options['mindspore.option.input_shape'] = acl_options.input_shape
-    if acl_options.output_type:
-        options['mindspore.option.output_type'] = acl_options.output_type
-    if acl_options.precision_mode:
-        options['mindspore.option.precision_mode'] = acl_options.precision_mode
-    if acl_options.op_select_impl_mode:
-        options['mindspore.option.op_select_impl_mode'] = acl_options.op_select_impl_mode
-    return options
+    def _as_options_map(self):
+        """Transfer AclOptions to dict of str,str"""
+        options = {}
+        if self.insert_op_cfg_path:
+            options['acl_option.insert_op_config_file_path'] = self.insert_op_cfg_path
+        if self.input_format:
+            options['acl_option.input_format'] = self.input_format
+        if self.input_shape:
+            options['acl_option.input_shape'] = self.input_shape
+        if self.output_type:
+            options['acl_option.output_type'] = self.output_type
+        if self.precision_mode:
+            options['acl_option.precision_mode'] = self.precision_mode
+        if self.op_select_impl_mode:
+            options['acl_option.op_select_impl_mode'] = self.op_select_impl_mode
+        return options

mindspore_serving/worker/task.py

@@ -20,6 +20,7 @@ import logging
 from mindspore_serving._mindspore_serving import Worker_
 from mindspore_serving.worker.register.preprocess import preprocess_storage
 from mindspore_serving.worker.register.postprocess import postprocess_storage
+from mindspore_serving import log as logger
 
 
 class ServingSystemException(Exception):
@@ -105,9 +106,9 @@ class PyTask:
                     self.result_batch.append(output)
 
                 get_result_time = time.time()
-                print(f"-----------------{self.task_name} get result "
-                      f"{last_index} ~ {last_index + len(self.result_batch) - 1} cost time",
-                      (get_result_time - get_result_time_end) * 1000, "ms")
+                logger.info(f"{self.task_name} get result "
+                            f"{last_index} ~ {last_index + len(self.result_batch) - 1} cost time "
+                            f"{(get_result_time - get_result_time_end) * 1000} ms")
 
                 self.push_result_batch()
                 break
@@ -119,7 +120,7 @@ class PyTask:
             except ServingSystemException as e:
                 raise e
             except Exception as e:  # catch exception and try next
-                print("{self.task_name} get result catch exception: ")
+                logger.warning(f"{self.task_name} get result catch exception: {e}")
                 logging.exception(e)
                 self.push_failed(1)  # push success results and a failed result
                 self.temp_result = self._handle_task_continue()
@@ -149,7 +150,7 @@ class PyTask:
             outputs = self.task_info["fun"](instance_list[self.index:])
             return outputs
         except Exception as e:
-            print(f"{self.task_name} invoke catch exception: ")
+            logger.warning(f"{self.task_name} invoke catch exception: ")
             logging.exception(e)
             self.push_failed(len(instance_list) - self.index)
             return None
@@ -217,7 +218,7 @@ class PyTaskThread(threading.Thread):
 
     def run(self):
         """Run tasks of preprocess and postprocess, switch to other type of process when some instances are handled"""
-        print("start py task for preprocess and postprocess, switch_batch", self.switch_batch)
+        logger.info(f"start py task for preprocess and postprocess, switch_batch {self.switch_batch}")
         preprocess_turn = True
         while True:
             try:
@@ -257,10 +258,10 @@ class PyTaskThread(threading.Thread):
                     preprocess_turn = True
 
             except Exception as e:
-                print("py task catch exception and exit: ")
+                logger.error(f"py task catch exception and exit: {e}")
                 logging.exception(e)
                 break
-        print("end py task for preprocess and postprocess")
+        logger.info("end py task for preprocess and postprocess")
         Worker_.stop()
 
 

setup.py

@@ -23,7 +23,7 @@ from setuptools import setup, find_packages
 from setuptools.command.egg_info import egg_info
 from setuptools.command.build_py import build_py
 
-version = '1.1.0'
+version = '1.1.2'
 
 backend_policy = os.getenv('BACKEND_POLICY')
 commit_id = os.getenv('COMMIT_ID').replace("\n", "")

tests/ut/cpp/stub/include/api/cell.h

@@ -0,0 +1,133 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_CELL_H
+#define MINDSPORE_INCLUDE_API_CELL_H
+#include <string>
+#include <vector>
+#include <map>
+#include <memory>
+#include "include/api/status.h"
+#include "include/api/types.h"
+#include "include/api/graph.h"
+
+namespace mindspore {
+class InputAndOutput;
+using Input = InputAndOutput;
+using Output = InputAndOutput;
+
+class MS_API CellBase {
+ public:
+  CellBase() = default;
+  virtual ~CellBase() = default;
+  virtual std::vector<Output> Construct(const std::vector<Input> &inputs) { return {}; }
+  virtual std::shared_ptr<CellBase> Clone() const = 0;
+  virtual Status Run(const std::vector<MSTensor> &inputs, std::vector<MSTensor> *outputs) { return kSuccess; }
+  std::vector<Output> operator()(const std::vector<Input> &inputs) const;
+};
+
+template <class T>
+class MS_API Cell : public CellBase {
+ public:
+  virtual ~Cell() = default;
+  std::shared_ptr<CellBase> Clone() const override { return std::make_shared<T>(static_cast<const T &>(*this)); }
+};
+
+class MS_API ParameterCell final : public Cell<ParameterCell> {
+ public:
+  ParameterCell() = default;
+  ~ParameterCell() override = default;
+
+  ParameterCell(const ParameterCell &);
+  ParameterCell &operator=(const ParameterCell &);
+
+  ParameterCell(ParameterCell &&);
+  ParameterCell &operator=(ParameterCell &&);
+
+  explicit ParameterCell(const MSTensor &);
+  ParameterCell &operator=(const MSTensor &);
+
+  explicit ParameterCell(MSTensor &&);
+  ParameterCell &operator=(MSTensor &&);
+
+  MSTensor GetTensor() const { return tensor_; }
+
+ private:
+  MSTensor tensor_;
+};
+
+class MS_API OpCellBase : public CellBase {
+ public:
+  explicit OpCellBase(const std::string &name) : name_(name) {}
+  ~OpCellBase() override = default;
+  const std::string &GetOpType() const { return name_; }
+
+ protected:
+  std::string name_;
+};
+
+template <class T>
+class MS_API OpCell : public OpCellBase, public std::enable_shared_from_this<T> {
+ public:
+  explicit OpCell(const std::string &name) : OpCellBase(name) {}
+  ~OpCell() override = default;
+  std::shared_ptr<CellBase> Clone() const override { return std::make_shared<T>(static_cast<const T &>(*this)); }
+};
+
+class MS_API GraphCell final : public Cell<GraphCell> {
+ public:
+  class GraphImpl;
+
+  GraphCell() = default;
+  ~GraphCell() override = default;
+
+  explicit GraphCell(const Graph &);
+  explicit GraphCell(Graph &&);
+  explicit GraphCell(const std::shared_ptr<Graph> &);
+
+  const std::shared_ptr<Graph> &GetGraph() const { return graph_; }
+  Status Run(const std::vector<MSTensor> &inputs, std::vector<MSTensor> *outputs) override;
+  std::vector<MSTensor> GetInputs();
+  std::vector<MSTensor> GetOutputs();
+
+ private:
+  friend class ModelImpl;
+  Status Load();
+
+  std::shared_ptr<Graph> graph_;
+  std::shared_ptr<GraphImpl> executor_;
+};
+
+class MS_API InputAndOutput {
+ public:
+  InputAndOutput();
+  ~InputAndOutput() = default;
+
+  // no explicit
+  InputAndOutput(const MSTensor &);  // NOLINT(runtime/explicit)
+  InputAndOutput(MSTensor &&);       // NOLINT(runtime/explicit)
+
+  InputAndOutput(const std::shared_ptr<CellBase> &, const std::vector<InputAndOutput> &, int32_t index);
+
+  int32_t GetIndex() const { return index_; }
+  void SetIndex(int32_t index) { index_ = index; }
+
+ private:
+  std::shared_ptr<CellBase> cell_;
+  std::vector<InputAndOutput> prev_;
+  int32_t index_;
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_CELL_H

tests/ut/cpp/stub/include/api/context.h

@@ -0,0 +1,64 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_CONTEXT_H
+#define MINDSPORE_INCLUDE_API_CONTEXT_H
+
+#include <map>
+#include <any>
+#include <string>
+#include <memory>
+#include "include/api/types.h"
+
+namespace mindspore {
+constexpr auto kDeviceTypeAscend310 = "Ascend310";
+constexpr auto kDeviceTypeAscend910 = "Ascend910";
+
+struct MS_API Context {
+  virtual ~Context() = default;
+  std::map<std::string, std::any> params;
+};
+
+struct MS_API GlobalContext : public Context {
+  static std::shared_ptr<Context> GetGlobalContext();
+
+  static void SetGlobalDeviceTarget(const std::string &device_target);
+  static std::string GetGlobalDeviceTarget();
+
+  static void SetGlobalDeviceID(const uint32_t &device_id);
+  static uint32_t GetGlobalDeviceID();
+};
+
+struct MS_API ModelContext : public Context {
+  static void SetInsertOpConfigPath(const std::shared_ptr<Context> &context, const std::string &cfg_path);
+  static std::string GetInsertOpConfigPath(const std::shared_ptr<Context> &context);
+
+  static void SetInputFormat(const std::shared_ptr<Context> &context, const std::string &format);
+  static std::string GetInputFormat(const std::shared_ptr<Context> &context);
+
+  static void SetInputShape(const std::shared_ptr<Context> &context, const std::string &shape);
+  static std::string GetInputShape(const std::shared_ptr<Context> &context);
+
+  static void SetOutputType(const std::shared_ptr<Context> &context, enum DataType output_type);
+  static enum DataType GetOutputType(const std::shared_ptr<Context> &context);
+
+  static void SetPrecisionMode(const std::shared_ptr<Context> &context, const std::string &precision_mode);
+  static std::string GetPrecisionMode(const std::shared_ptr<Context> &context);
+
+  static void SetOpSelectImplMode(const std::shared_ptr<Context> &context, const std::string &op_select_impl_mode);
+  static std::string GetOpSelectImplMode(const std::shared_ptr<Context> &context);
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_CONTEXT_H

tests/ut/cpp/stub/include/api/data_type.h

@@ -0,0 +1,44 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef MINDSPORE_INCLUDE_API_DATA_TYPE_H_
+#define MINDSPORE_INCLUDE_API_DATA_TYPE_H_
+
+namespace mindspore {
+enum class DataType : int {
+  kTypeUnknown = 0,
+  kObjectTypeString = 12,
+  kObjectTypeList = 13,
+  kObjectTypeTuple = 14,
+  kObjectTypeTensorType = 17,
+  kNumberTypeBool = 30,
+  kNumberTypeInt8 = 32,
+  kNumberTypeInt16 = 33,
+  kNumberTypeInt32 = 34,
+  kNumberTypeInt64 = 35,
+  kNumberTypeUInt8 = 37,
+  kNumberTypeUInt16 = 38,
+  kNumberTypeUInt32 = 39,
+  kNumberTypeUInt64 = 40,
+  kNumberTypeFloat16 = 42,
+  kNumberTypeFloat32 = 43,
+  kNumberTypeFloat64 = 44,
+  kNumberTypeEnd = 46,
+  // add new enum here
+  kInvalidType = INT32_MAX,
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_DATA_TYPE_H_

tests/ut/cpp/stub/include/api/graph.h

@@ -0,0 +1,45 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_GRAPH_H
+#define MINDSPORE_INCLUDE_API_GRAPH_H
+
+#include <cstddef>
+#include <string>
+#include <vector>
+#include <map>
+#include <memory>
+#include "include/api/status.h"
+#include "include/api/types.h"
+
+namespace mindspore {
+class MS_API Graph {
+ public:
+  class GraphData;
+  explicit Graph(const std::shared_ptr<GraphData> &graph_data);
+  explicit Graph(std::shared_ptr<GraphData> &&graph_data);
+  explicit Graph(std::nullptr_t);
+  ~Graph();
+
+  enum ModelType ModelType() const;
+  bool operator==(std::nullptr_t) const;
+
+ private:
+  friend class GraphCell;
+  friend class ModelImpl;
+  std::shared_ptr<GraphData> graph_data_;
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_GRAPH_H

tests/ut/cpp/stub/include/api/model.h

@@ -0,0 +1,55 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_MODEL_H
+#define MINDSPORE_INCLUDE_API_MODEL_H
+
+#include <string>
+#include <vector>
+#include <map>
+#include <memory>
+#include <utility>
+#include "include/api/status.h"
+#include "include/api/types.h"
+#include "include/api/graph.h"
+#include "include/api/cell.h"
+
+namespace mindspore {
+class ModelImpl;
+struct Context;
+
+class MS_API Model {
+ public:
+  explicit Model(const std::vector<Output> &network, const std::shared_ptr<Context> &model_context = nullptr);
+  explicit Model(const GraphCell &graph, const std::shared_ptr<Context> &model_context = nullptr);
+  ~Model();
+  Model(const Model &) = delete;
+  void operator=(const Model &) = delete;
+
+  Status Build();
+  Status Resize(const std::vector<MSTensor> &inputs, const std::vector<std::vector<int64_t>> &dims);
+
+  Status Predict(const std::vector<MSTensor> &inputs, std::vector<MSTensor> *outputs);
+
+  std::vector<MSTensor> GetInputs();
+  std::vector<MSTensor> GetOutputs();
+
+  static bool CheckModelSupport(const std::string &device_type, ModelType model_type);
+
+ private:
+  std::shared_ptr<ModelImpl> impl_;
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_MODEL_H

tests/ut/cpp/stub/include/api/serialization.h

@@ -0,0 +1,39 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_SERIALIZATION_H
+#define MINDSPORE_INCLUDE_API_SERIALIZATION_H
+
+#include <string>
+#include <vector>
+#include <map>
+#include <memory>
+#include "include/api/status.h"
+#include "include/api/types.h"
+#include "include/api/model.h"
+#include "include/api/graph.h"
+
+namespace mindspore {
+class MS_API Serialization {
+ public:
+  static Graph LoadModel(const void *model_data, size_t data_size, ModelType model_type);
+  static Graph LoadModel(const std::string &file, ModelType model_type);
+  static Status LoadCheckPoint(const std::string &ckpt_file, std::map<std::string, Buffer> *parameters);
+  static Status SetParameters(const std::map<std::string, Buffer> &parameters, Model *model);
+  static Status ExportModel(const Model &model, ModelType model_type, Buffer *model_data);
+  static Status ExportModel(const Model &model, ModelType model_type, const std::string &model_file);
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_SERIALIZATION_H

tests/ut/cpp/stub/include/api/status.h

@@ -0,0 +1,138 @@
+/**
+ * Copyright 2020 Huawei Technologies Co., Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef MINDSPORE_INCLUDE_API_STATUS_H
+#define MINDSPORE_INCLUDE_API_STATUS_H
+
+#include <string>
+#include <ostream>
+#include <climits>
+
+namespace mindspore {
+enum CompCode : uint32_t {
+  kCore = 0x00000000u,
+  kMD = 0x10000000u,
+  kME = 0x20000000u,
+  kMC = 0x30000000u,
+  kLite = 0xF0000000u,
+};
+
+enum StatusCode : uint32_t {
+  kSuccess = 0,
+  // Core
+  kCoreFailed = kCore | 0x1,
+
+  // MD
+  kMDOutOfMemory = kMD | 1,
+  kMDShapeMisMatch = kMD | 2,
+  kMDInterrupted = kMD | 3,
+  kMDNoSpace = kMD | 4,
+  kMDPyFuncException = kMD | 5,
+  kMDDuplicateKey = kMD | 6,
+  kMDPythonInterpreterFailure = kMD | 7,
+  kMDTDTPushFailure = kMD | 8,
+  kMDFileNotExist = kMD | 9,
+  kMDProfilingError = kMD | 10,
+  kMDBoundingBoxOutOfBounds = kMD | 11,
+  kMDBoundingBoxInvalidShape = kMD | 12,
+  kMDSyntaxError = kMD | 13,
+  kMDTimeOut = kMD | 14,
+  kMDBuddySpaceFull = kMD | 15,
+  kMDNetWorkError = kMD | 16,
+  kMDNotImplementedYet = kMD | 17,
+  // Make this error code the last one. Add new error code above it.
+  kMDUnexpectedError = kMD | 127,
+
+  // ME
+  kMEFailed = kME | 0x1,
+  kMEInvalidInput = kME | 0x2,
+
+  // MC
+  kMCFailed = kMC | 0x1,
+  kMCDeviceError = kMC | 0x2,
+  kMCInvalidInput = kMC | 0x3,
+  kMCInvalidArgs = kMC | 0x4,
+
+  // Lite  // Common error code, range: [-1, -100）
+  kLiteError = kLite | (0x0FFFFFFF & -1),           /**< Common error code. */
+  kLiteNullptr = kLite | (0x0FFFFFFF & -2),         /**< NULL pointer returned.*/
+  kLiteParamInvalid = kLite | (0x0FFFFFFF & -3),    /**< Invalid parameter.*/
+  kLiteNoChange = kLite | (0x0FFFFFFF & -4),        /**< No change. */
+  kLiteSuccessExit = kLite | (0x0FFFFFFF & -5),     /**< No error but exit. */
+  kLiteMemoryFailed = kLite | (0x0FFFFFFF & -6),    /**< Fail to create memory. */
+  kLiteNotSupport = kLite | (0x0FFFFFFF & -7),      /**< Fail to support. */
+  kLiteThreadPoolError = kLite | (0x0FFFFFFF & -8), /**< Error occur in thread pool. */
+
+  // Executor error code, range: [-100,-200)
+  kLiteOutOfTensorRange = kLite | (0x0FFFFFFF & -100), /**< Failed to check range. */
+  kLiteInputTensorError = kLite | (0x0FFFFFFF & -101), /**< Failed to check input tensor. */
+  kLiteReentrantError = kLite | (0x0FFFFFFF & -102),   /**< Exist executor running. */
+
+  // Graph error code, range: [-200,-300)
+  kLiteGraphFileError = kLite | (0x0FFFFFFF & -200), /**< Failed to verify graph file. */
+
+  // Node error code, range: [-300,-400)
+  kLiteNotFindOp = kLite | (0x0FFFFFFF & -300),        /**< Failed to find operator. */
+  kLiteInvalidOpName = kLite | (0x0FFFFFFF & -301),    /**< Invalid operator name. */
+  kLiteInvalidOpAttr = kLite | (0x0FFFFFFF & -302),    /**< Invalid operator attr. */
+  kLiteOpExecuteFailure = kLite | (0x0FFFFFFF & -303), /**< Failed to execution operator. */
+
+  // Tensor error code, range: [-400,-500)
+  kLiteFormatError = kLite | (0x0FFFFFFF & -400), /**< Failed to checking tensor format. */
+
+  // InferShape error code, range: [-500,-600)
+  kLiteInferError = kLite | (0x0FFFFFFF & -500),   /**< Failed to infer shape. */
+  kLiteInferInvalid = kLite | (0x0FFFFFFF & -501), /**< Invalid infer shape before runtime. */
+
+  // User input param error code, range: [-600, 700)
+  kLiteInputParamInvalid = kLite | (0x0FFFFFFF & -600), /**< Invalid input param by user. */
+};
+
+class Status {
+ public:
+  Status() : status_code_(kSuccess) {}
+  Status(enum StatusCode status_code, const std::string &status_msg = "")  // NOLINT(runtime/explicit)
+      : status_code_(status_code), status_msg_(status_msg) {}
+  Status(const StatusCode code, int line_of_code, const char *file_name, const std::string &extra = "");
+
+  ~Status() = default;
+
+  enum StatusCode StatusCode() const { return status_code_; }
+  const std::string &ToString() const { return status_msg_; }
+
+  friend std::ostream &operator<<(std::ostream &os, const Status &s);
+
+  bool operator==(const Status &other) const { return status_code_ == other.status_code_; }
+  bool operator==(enum StatusCode other_code) const { return status_code_ == other_code; }
+  bool operator!=(const Status &other) const { return status_code_ != other.status_code_; }
+  bool operator!=(enum StatusCode other_code) const { return status_code_ != other_code; }
+
+  explicit operator bool() const { return (status_code_ == kSuccess); }
+  explicit operator int() const { return static_cast<int>(status_code_); }
+
+  static Status OK() { return Status(StatusCode::kSuccess); }
+
+  bool IsOk() const { return (StatusCode() == StatusCode::kSuccess); }
+
+  bool IsError() const { return !IsOk(); }
+
+  static std::string CodeAsString(enum StatusCode c);
+
+ private:
+  enum StatusCode status_code_;
+  std::string status_msg_;
+};
+}  // namespace mindspore
+#endif  // MINDSPORE_INCLUDE_API_STATUS_H

tests/ut/cpp/stub/include/api/types.h

@@ -16,15 +16,33 @@
 #ifndef MINDSPORE_INCLUDE_API_TYPES_H
 #define MINDSPORE_INCLUDE_API_TYPES_H
 
+#include <cstddef>
 #include <string>
 #include <vector>
 #include <memory>
-
+#include "include/api/data_type.h"
+
+// refer to https://gcc.gnu.org/wiki/Visibility
+#if defined _WIN32 || defined __CYGWIN__
+#ifdef BUILDING_DLL
+#ifdef __GNUC__
+#define MS_API __attribute__((dllexport))
+#else
+#define MS_API __declspec(dllexport)  // Note: actually gcc seems to also supports this syntax.
+#endif
+#else
+#ifdef __GNUC__
+#define MS_API __attribute__((dllimport))
+#else
+#define MS_API __declspec(dllimport)  // Note: actually gcc seems to also supports this syntax.
+#endif
+#endif
+#else
 #define MS_API __attribute__((visibility("default")))
+#endif
 
 namespace mindspore {
-namespace api {
-enum ModelType {
+enum ModelType : uint32_t {
   kMindIR = 0,
   kAIR = 1,
   kOM = 2,
@@ -33,52 +51,38 @@ enum ModelType {
   kUnknownType = 0xFFFFFFFF
 };
 
-enum DataType {
-  kMsUnknown = 0,
-  kMsBool = 1,
-  kMsInt8 = 2,
-  kMsInt16 = 3,
-  kMsInt32 = 4,
-  kMsInt64 = 5,
-  kMsUint8 = 6,
-  kMsUint16 = 7,
-  kMsUint32 = 8,
-  kMsUint64 = 9,
-  kMsFloat16 = 10,
-  kMsFloat32 = 11,
-  kMsFloat64 = 12,
-  // insert new data type here
-  kInvalidDataType = 0xFFFFFFFF
-};
-
-class MS_API Tensor {
+class MS_API MSTensor {
  public:
-  Tensor();
-  Tensor(const std::string &name, DataType type, const std::vector<int64_t> &shape, const void *data, size_t data_len);
-  ~Tensor();
+  class Impl;
 
-  const std::string &Name() const;
-  void SetName(const std::string &name);
+  static MSTensor CreateTensor(const std::string &name, DataType type, const std::vector<int64_t> &shape,
+                               const void *data, size_t data_len) noexcept;
+  static MSTensor CreateRefTensor(const std::string &name, DataType type, const std::vector<int64_t> &shape,
+                                  const void *data, size_t data_len) noexcept;
 
-  api::DataType DataType() const;
-  void SetDataType(api::DataType type);
+  MSTensor();
+  explicit MSTensor(const std::shared_ptr<Impl> &impl);
+  MSTensor(const std::string &name, DataType type, const std::vector<int64_t> &shape, const void *data,
+           size_t data_len);
+  ~MSTensor();
 
+  const std::string &Name() const;
+  enum DataType DataType() const;
   const std::vector<int64_t> &Shape() const;
-  void SetShape(const std::vector<int64_t> &shape);
+  int64_t ElementNum() const;
 
-  const void *Data() const;
+  std::shared_ptr<const void> Data() const;
   void *MutableData();
   size_t DataSize() const;
 
-  bool ResizeData(size_t data_len);
-  bool SetData(const void *data, size_t data_len);
+  bool IsDevice() const;
 
-  int64_t ElementNum() const;
-  static int GetTypeSize(api::DataType type);
-  Tensor Clone() const;
+  MSTensor Clone() const;
+  bool operator==(std::nullptr_t) const;
 
  private:
-  class Impl;
+  friend class ModelImpl;
+  explicit MSTensor(std::nullptr_t);
   std::shared_ptr<Impl> impl_;
 };
 
@@ -101,19 +105,5 @@ class MS_API Buffer {
   class Impl;
   std::shared_ptr<Impl> impl_;
 };
-
-constexpr auto kModelOptionDumpCfgPath = "mindspore.option.dump_config_file_path";
-constexpr auto kModelOptionDvppCfgPath = "mindspore.option.dvpp_config_file_path";
-constexpr auto kModelOptionInsertOpCfgPath = "mindspore.option.insert_op_config_file_path";  // aipp config file
-constexpr auto kModelOptionInputFormat = "mindspore.option.input_format";                    // nchw or nhwc
-// Mandatory while dynamic batch: e.g. "input_op_name1: n1,c2,h3,w4;input_op_name2: n4,c3,h2,w1"
-constexpr auto kModelOptionInputShape = "mindspore.option.input_shape";
-constexpr auto kModelOptionDynamicBatchSize = "mindspore.option.dynamic_batch_size";
-constexpr auto kModelOptionDynamicImageSize = "mindspore.option.dynamic_image_size";
-constexpr auto kModelOptionDynamicDims = "mindspore.option.dynamic_dims";
-constexpr auto kModelOptionSerialInput = "mindspore.option.serial_inputs_name";  // separated by ';'
-constexpr auto kModelOptionOutputNode = "mindspore.option.output_node";          // e.g. "node_name1:0;node_name2:1"
-constexpr auto kModelOptionOutputType = "mindspore.option.output_type";  // "FP32", "UINT8" or "FP16", default as "FP32"
-}  // namespace api
 }  // namespace mindspore
 #endif  // MINDSPORE_INCLUDE_API_TYPES_H

third_party/mindspore

@@ -1 +1 @@
-Subproject commit 7a3f34247db5a6c7a00b00b16d2bfa2ff94e0b08
+Subproject commit a7bbe134b54174a30169d978bef3a58599af1aa1