Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Model] add Paddle.js web demo #392

Merged
merged 18 commits into from
Oct 20, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
35 changes: 35 additions & 0 deletions examples/application/js/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@

# 前端AI应用

人工智能技术的快速发展带动了计算机视觉、自然语言处理领域的产业升级。另外,随着PC和移动设备上算力的稳步增强、模型压缩技术迭代更新以及各种创新需求的不断催生,在浏览器中部署AI模型实现前端智能已经具备了良好的基础条件。
针对前端部署AI深度学习模型困难的问题,百度开源了Paddle.js前端深度学习模型部署框架,可以很容易的将深度学习模型部署到前端项目中。

## Paddle.js简介

[Paddle.js](https://github.com/PaddlePaddle/Paddle.js)是百度`PaddlePaddle`的web方向子项目,是一个运行在浏览器中的开源深度学习框架。`Paddle.js`可以加载`PaddlePaddle`动转静的模型,经过`Paddle.js`的模型转换工具`paddlejs-converter`转换成浏览器友好的模型,易于在线推理预测使用。`Paddle.js`支持`WebGL/WebGPU/WebAssembly`的浏览器中运行,也可以在百度小程序和微信小程序环境下运行。

简言之,利用Paddle.js,我们可以在浏览器、小程序等前端应用场景上线AI功能,包括但不限于目标检测,图像分割,OCR,物品分类等AI能力。

## Web Demo使用

在浏览器中直接运行官方demo参考[文档](./web_demo/README.md)

|demo名称|web demo目录|可视化|
|-|-|-|
|目标检测|[ScrewDetection/FaceDetection](./web_demo/demo/src/pages/cv/detection/)| <img src="https://user-images.githubusercontent.com/26592129/196874536-b7fa2c0a-d71f-4271-8c40-f9088bfad3c9.png" height="200px">|
|人像分割背景替换|[HumanSeg](./web_demo//demo/src/pages/cv/segmentation/HumanSeg)|<img src="https://user-images.githubusercontent.com/26592129/196874452-4ef2e770-fbb3-4a35-954b-f871716d6669.png" height="200px">|
|物体识别|[GestureRecognition/ItemIdentification](./web_demo//demo/src/pages/cv/recognition/)|<img src="https://user-images.githubusercontent.com/26592129/196874416-454e6bb0-4ebd-4b51-a88a-8c40614290ae.png" height="200px">|
|OCR|[TextDetection/TextRecognition](./web_demo//demo/src/pages/cv/ocr/)|<img src="https://user-images.githubusercontent.com/26592129/196874354-1b5eecb0-f273-403c-aa6c-4463bf6d78db.png" height="200px">|


## 微信小程序Demo使用

在微信小程序运行官方demo参考[文档](./mini_program/README.md)

|名称|目录|
|-|-|
|OCR文本检测| [ocrdetecXcx](./mini_program/ocrdetectXcx/) |
|OCR文本识别| [ocrXcx](./mini_program/ocrXcx/) |
|目标检测| coming soon |
|图像分割| coming soon |
|物品分类| coming soon |
126 changes: 126 additions & 0 deletions examples/application/js/mini_program/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@

# Paddle.js微信小程序Demo

- [1.简介](#1)
- [2. 项目启动](#2)
* [2.1 准备工作](#21)
* [2.2 启动步骤](#22)
* [2.3 效果展示](#23)
- [3. 模型推理pipeline](#3)
- [4. 常见问题](#4)

<a name="1"></a>
## 1.简介


本目录下包含文本检测、文本识别小程序demo,通过使用 [Paddle.js](https://github.com/PaddlePaddle/Paddle.js) 以及 [Paddle.js微信小程序插件](https://mp.weixin.qq.com/wxopen/plugindevdoc?appid=wx7138a7bb793608c3&token=956931339&lang=zh_CN) 完成在小程序上利用用户终端算力实现文本检测框选效果。

<a name="2"></a>
## 2. 项目启动

<a name="21"></a>
### 2.1 准备工作
* [申请微信小程序账号](https://mp.weixin.qq.com/)
* [微信小程序开发者工具](https://developers.weixin.qq.com/miniprogram/dev/devtools/download.html)
* 前端开发环境准备:node、npm
* 小程序管理后台配置服务器域名,或打开开发者工具【不校验合法域名】

详情参考:https://mp.weixin.qq.com/wxamp/devprofile/get_profile?token=1132303404&lang=zh_CN)

<a name="22"></a>
### 2.2 启动步骤

#### **1. 克隆Demo代码**
```sh
git clone https://github.com/PaddlePaddle/FastDeploy
cd FastDeploy/examples/application/js/mini_program
```

#### **2. 进入小程序目录,安装依赖**

```sh
# 运行文本识别demo,进入到ocrXcx目录
cd ./ocrXcx && npm install
# 运行文本检测demo,进入到ocrdetectXcx目录
# cd ./ocrdetectXcx && npm install
```

#### **3. 微信小程序导入代码**
打开微信开发者工具 --> 导入 --> 选定目录,输入相关信息

#### **4. 添加 Paddle.js微信小程序插件**
小程序管理界面 --> 设置 --> 第三方设置 --> 插件管理 --> 添加插件 --> 搜索 `wx7138a7bb793608c3` 并添加
[参考文档](https://developers.weixin.qq.com/miniprogram/dev/framework/plugin/using.html)

#### **5. 构建依赖**
点击开发者工具中的菜单栏:工具 --> 构建 npm

原因:node_modules 目录不会参与编译、上传和打包中,小程序想要使用 npm 包必须走一遍“构建 npm”的过程,构建完成会生成一个 miniprogram_npm 目录,里面会存放构建打包后的 npm 包,也就是小程序真正使用的 npm 包。*
[参考文档](https://developers.weixin.qq.com/miniprogram/dev/devtools/npm.html)

<a name="23"></a>
### 2.3 效果展示

<img src="https://user-images.githubusercontent.com/43414102/157648579-cdbbee61-9866-4364-9edd-a97ac0eda0c1.png" width="300px">

<a name="3"></a>
## 3. 模型推理pipeline

```typescript
// 引入 paddlejs 和 paddlejs-plugin,注册小程序环境变量和合适的 backend
import * as paddlejs from '@paddlejs/paddlejs-core';
import '@paddlejs/paddlejs-backend-webgl';
const plugin = requirePlugin('paddlejs-plugin');
plugin.register(paddlejs, wx);

// 初始化推理引擎
const runner = new paddlejs.Runner({modelPath, feedShape, mean, std});
await runner.init();

// 获取图像信息
wx.canvasGetImageData({
canvasId: canvasId,
x: 0,
y: 0,
width: canvas.width,
height: canvas.height,
success(res) {
// 推理预测
runner.predict({
data: res.data,
width: canvas.width,
height: canvas.height,
}, function (data) {
// 获取推理结果
console.log(data)
});
}
});
```

<a name="4"></a>
## 4. 常见问题
### 4.1 出现报错 `Invalid context type [webgl2] for Canvas#getContext`

可以不管,不影响正常代码运行和demo功能

### 4.2 预览看不到结果

建议尝试真机调试

### 4.3 微信开发者工具出现黑屏,然后出现超多报错

重启微信开发者工具

### 4.4 模拟和真机调试结果不一致;模拟检测不到文本等

可以以真机为准;

模拟检测不到文本等可以尝试随意改动下代码(增删换行等)再点击编译


### 4.5 手机调试或运行时出现 长时间无反应等提示

请继续等待,模型推理需要一定时间


12 changes: 12 additions & 0 deletions examples/application/js/mini_program/ocrXcx/app.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/* global wx, App */
import * as paddlejs from '@paddlejs/paddlejs-core';
import '@paddlejs/paddlejs-backend-webgl';
// eslint-disable-next-line no-undef
const plugin = requirePlugin('paddlejs-plugin');
plugin.register(paddlejs, wx);

App({
globalData: {
Paddlejs: paddlejs.Runner
}
});
12 changes: 12 additions & 0 deletions examples/application/js/mini_program/ocrXcx/app.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"pages": [
"pages/index/index"
],
"plugins": {
"paddlejs-plugin": {
"version": "2.0.1",
"provider": "wx7138a7bb793608c3"
}
},
"sitemapLocation": "sitemap.json"
}
72 changes: 72 additions & 0 deletions examples/application/js/mini_program/ocrXcx/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions examples/application/js/mini_program/ocrXcx/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"name": "paddlejs-demo",
"version": "0.0.1",
"description": "",
"main": "app.js",
"dependencies": {
"@paddlejs/paddlejs-backend-webgl": "^1.2.0",
"@paddlejs/paddlejs-core": "^2.1.18",
"d3-polygon": "2.0.0",
"js-clipper": "1.0.1",
"number-precision": "1.5.2"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC"
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading