[关闭]
@TedZhou 2022-12-26T14:27:31.000000Z 字数 2567 阅读 237

微信小程序引入TensorflowJS实现Pose-detection

wechat tenserflow js movenet


添加TensorflowJS插件

登录小程序管理后台,打开“第三方设置”,添加插件TensorflowJS。本插件封装了TensorFlow.js库,利用小程序WebGL API给第三方小程序调用时提供GPU加速。

在app.json里添加插件配置:

  1. "plugins": {
  2. "tfjsPlugin": {
  3. "version": "0.2.0",
  4. "provider": "wx6afed118d9e81df9"
  5. }
  6. }

引入TensorFlow.js npm包

在小程序根目录添加package.json:

  1. {
  2. "name": "yourProject",
  3. "version": "0.0.1",
  4. "main": "dist/index.js",
  5. "license": "Apache-2.0",
  6. "dependencies": {
  7. "@tensorflow-models/pose-detection": "^2.0.0",
  8. "@tensorflow/tfjs-backend-webgl": "^3.21.0",
  9. "@tensorflow/tfjs-converter": "^3.21.0",
  10. "@tensorflow/tfjs-core": "^3.21.0",
  11. "fetch-wechat": "^0.0.3"
  12. }
  13. }

先npm install安装,再小程序开发者工具构建npm,关闭“将JS编译成ES5”的选项。

如果运行时出现找不到pose包的错误,尝试npm install @mediapipe/pose
(会自动引入最新版本"@mediapipe/pose": "^0.5.1635988162")

在app.onLaunch调用configPlugin插件初始化

  1. app.onLaunch = function () {
  2. tfjsPlugin.configPlugin({
  3. fetchFunc: fetchWechat.fetchFunc(),// polyfill fetch function
  4. tf,// inject tfjs runtime
  5. webgl,// inject backend
  6. canvas: wx.createOffscreenCanvas(),// provide webgl canvas
  7. });
  8. }

下载模型文件

tensorflow的预训练模型是放在墙外,需要下载下来放可访问的服务器上:
https://tfhub.dev/google/tfjs-model/movenet/singlepose/lightning/4

加载Movenet模型

  1. const poseDetection = require('@tensorflow-models/pose-detection')
  2. page.onLoad = async function(){
  3. const modelUrl = 'https://<host>/tfjs/tfjs-model/movenet/singlepose/lightning/4/model.json'
  4. const modelConfig = {
  5. modelUrl,
  6. modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING,
  7. };
  8. this.movenet = await poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, modelConfig).catch(console.error)
  9. }

使用摄像头逐帧识别

在页面wxml添加camera组件

  1. <camera device-position="front" flash="off"></camera>
  2. <button type="primary" bindtap="poseDetect">识别</button>
  1. page.poseDetect = function(){
  2. this.data.ts = Date.now()
  3. this.cameraCtx = this.cameraCtx || wx.createCameraContext()
  4. //侦听相机视频流
  5. this.cameraLisenter = this.cameraLisenter || this.cameraCtx.onCameraFrame(frame => {
  6. if (this.estimatePoses.posing || Date.now() < this.data.ts+100) return//限速
  7. this.estimatePoses(frame).then(() => this.data.ts = Date.now())
  8. })
  9. //toggle start/stop posing
  10. if (!this.data.posing){
  11. this.cameraLisenter && this.cameraLisenter.start()
  12. }else{
  13. this.cameraLisenter && this.cameraLisenter.stop()
  14. }
  15. this.data.posing = !this.data.posing
  16. }
  17. page.estimatePoses = async function(frame){
  18. if (this.estimatePoses.posing) return//限流
  19. this.estimatePoses.posing = true
  20. const data = {//把视频帧转为estimatePoses要求的格式
  21. data: new Uint8Array(frame.data),
  22. height: Number(frame.height),
  23. width: Number(frame.width)
  24. }
  25. console.time('posing')
  26. const poses = await this.movenet.estimatePoses(data).catch(console.error)
  27. if (poses.length){//检测到人体
  28. // this.drawPoses(poses[0].keypoints)
  29. console.log(poses)
  30. }
  31. console.timeEnd('posing')
  32. this.estimatePoses.posing = false
  33. }
添加新批注
在作者公开此批注前,只有你和作者可见。
回复批注