2023-07-04

How to use a trained yolov8 object box detection model directly in tensorflow js in pure javascript? [closed]

I'm currently trying to use a trained yolov8 object box detection model (for 13 ish classes) from ultralytics, directly in tensorflow js in pure javascript.

I don't want to use roboflow js, onnx etc for running it ; or any framework like node-js implementation etc to serve it.. I want to have a fully working yolov8n model detecting custom objects in images/videos/webcam direclty on browser using only tensorflow js.

In the purpose of having something quite usefull (more than 10fps in webcam), i know that i have to use a quantized model for it. So my second question : What's also the code needed to use a custom yolov8n quantized model ?

I looked pretty much at hundreds of links, topics etc on the subject but i haven't found any for this particular use case. I'm also open to someone that has the full solution for an ssd-mobilenet custom training/deploying solution in pure tensorflow js for object detection with boxes.

You would be very nice and serving a lot of people i think if you share the full solution with codes on how to do it here !

Ps : I know that it's possible but any examples I'have found yet are either using node-js, or are for object classification etc... I haven't found any for my specific purpose : pure smooth tjfs yolov8 object detection with boxes.

[EDIT] If you're willing to help : Just give us the code to use a custom trained yolov8n on image/webcam using tf.loadGraphModel for a normal/16-quantized/8-quantized to work as a working example for this use-case that is nowhere on the internet.

[EDIT 2] I basically came up with this code as of right now... It seems to correclty pass the image to the model but i don't understand the output. What's the catch here ? :

html

js tf.ready().then(() => {

    const modelPath = "https://localhost/test_php/yolov8_model_json";

    tf.tidy(() => {

      tf.loadGraphModel(modelPath, { fromTFHub: true }).then((model) => {
        const mysteryImage = document.getElementById("mystery");


        const tfimg = tf.browser.fromPixels(mysteryImage).toInt();
          const tfimg_res = tf.image.resizeBilinear(tfimg, [640, 640]);
          const expandedimg = tfimg_res.transpose([0, 1, 2]).expandDims();
          
          // const predictions = await model.executeAsync(expandedimg);
          const predictions = model.predict(expandedimg);

          console.log(predictions);

          // get image width, height from output
          const imageWidth = predictions.shape[2]
          const imageHeight = predictions.shape[1]

          console.log('imageWidth:');
          console.log(imageWidth);

          console.log('imageHeight:');
          console.log(imageHeight);

          // create a regular array from the model's output prediction (tensor)
          const map = Array.from(predictions.dataSync())

          console.log('map:');
          console.log(map);

          // create an image with the segmentation color data
          let imageData = new ImageData(imageWidth, imageHeight);
          imageData.data.set(map);

          let canvas = document.getElementById('canvas')
          let ctx = canvas.getContext('2d')
          ctx.putImageData(imageData, 640, 640)
          console.log('processOutput completed:', imageData)

output : console log



No comments:

Post a Comment