Huawei_Technology
/
MindSpore-Serving
Not watched
1
0
Fork 0
Code
Releases 3 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
178 Commits
4 Branches
2.6 MB
9 Download
 
 
 
 
 
 
Tree: 1f5d8102a5
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '1f5d8102a5'
${ noResults }
“xujincai” 1f5d8102a5 fix in hccl config 5 years ago
.gitee Add Gitee pull request template 5 years ago
cmake Serving, dym load mindspore.so 5 years ago
docs fix architecture.png 5 years ago
example add st to serving 5 years ago
mindspore_serving fix in hccl config 5 years ago
scripts serving update cpplint and pylint 5 years ago
tests fix in hccl config 5 years ago
third_party Serving, ci ut bug fix and update mindspore 5 years ago
.clang-format ：serving 0.1.0 5 years ago
.gitignore ：serving 0.1.0 5 years ago
.gitmodules ：serving 0.1.0 5 years ago
CMakeLists.txt serving ut tests 5 years ago
LICENSE Add Apache 2.0 license file 5 years ago
NOTICE Add Apache 2.0 license file 5 years ago
README.md Serving gpt3 fix 5 years ago
README_CN.md Serving gpt3 fix 5 years ago
RELEASE.md fix readme 5 years ago
build.sh Serving, support restful float16 output support 5 years ago
requirements_test.txt serving, ut5 5 years ago
setup.py serving: update MindSpore, AclOptions API 5 years ago

README.md

MindSpore Serving

查看中文

Overview

MindSpore Serving is a lightweight and high-performance service module that helps MindSpore developers efficiently deploy online inference services in the production environment. After completing model training on MindSpore, you can export the MindSpore model and use MindSpore Serving to create an inference service for the model.

MindSpore Serving architecture:

Currently, the MindSpore Serving nodes include client, master, and worker. On a client node, you can directly deliver inference service commands through the gRPC or RESTful API. Servable model service is deployed on a worker node. Servable indicates a single model or a combination of multiple models and provides different services in various methods. A master node manages all worker nodes and their model information, as well as managing and distributing tasks. The master and worker nodes can be deployed in the same process or in different processes. Currently, the client and master nodes do not depend on specific hardware platforms. The worker node supports only the Ascend 310 and Ascend 910 platforms. GPUs and CPUs will be supported in the future.
MindSpore Architecture

MindSpore Serving provides the following functions:

  • gRPC and RESTful APIs on clients
  • Pre-processing and post-processing of assembled models
  • Batch. Multiple instance requests are split and combined to meet the batch size requirement of the model.
  • Simple Python APIs on clients

Installation

MindSpore Serving depends on the MindSpore training and inference framework. Therefore, install MindSpore and then MindSpore Serving.

Installing Serving

Use the pip command to install Serving. Perform the following steps:

  • Download the .whl package from the MindSpore Serving page and install it.

    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl
    • {version} denotes the version of MindSpore Serving. For example, when you are downloading MindSpore Serving 1.1.0, {version} should be 1.1.0.
    • {arch} denotes the system architecture. For example, the Linux system you are using is x86 architecture 64-bit, {arch} should be x86_64. If the system is ARM architecture 64-bit, then it should be aarch64.

  • Install Serving using the source code.

    Download the source code and go to the serving directory.

    Method 1: Specify the path of the installed or built MindSpore package on which Serving depends and install Serving.

    sh build.sh -p $MINDSPORE_LIB_PATH

    In the preceding information, build.sh is the build script file in the serving directory, and $MINDSPORE_LIB_PATH is the lib directory in the installation path of the MindSpore software package, for example, softwarepath/mindspore/lib. This path contains the library files on which MindSpore depends.

    Method 2: Directly build Serving. The MindSpore package is built together with Serving. You need to configure the environment variables for MindSpore building.

    # Ascend 310
sh build.sh -e ascend -V 310
# Ascend 910
sh build.sh -e ascend -V 910

    In the preceding information, build.sh is the build script file in the serving directory. After the build is complete, find the .whl installation package of MindSpore in the serving/third_party/mindspore/build/package/ directory and install it.

    pip install mindspore_ascend-{version}-cp37-cp37m-linux_{arch}.whl

    Find the .whl installation package of Serving in the serving/build/package/ directory and install it.

    pip install mindspore_serving-{version}-cp37-cp37m-linux_{arch}.whl

Run the following commands to verify the installation. Import the Python module. If no error is reported, the installation is successful.

from mindspore_serving import master
from mindspore_serving import worker

Configuring Environment Variables

To run MindSpore Serving, configure the following environment variables:

Quick Start

MindSpore-based Inference Service Deployment is used to demonstrate how to use MindSpore Serving.

Documents

Developer Guide

For more details about the installation guide, tutorials, and APIs, see MindSpore Python API.

Community

Governance

MindSpore Open Governance

Communication

Contributions

Welcome to MindSpore contribution.

Release Notes

RELEASE

License

Apache License 2.0

Delete task

Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.

cancel
confirm

A lightweight and high-performance service module that helps MindSpore developers efficiently deploy online inference services in the production environment.

C++ Python Shell Text CMake other