Get reference pictures from preview to learn a neural network

[Hanshogeland 0]

Hi,

I’m working on a project where I have trained a neural network model ”CNN” with data from the preview in Cura. The CNN are capable of predicting the outcom of the print - pass failed for each layer, and I’d like to write a plugin to Cura where I’d get pictures from a couple of angles from each layer in the preview. These pictures will then be sent via SSH to the SBC ”Jetson Nano” having duet 3 board connected. This will enable the possibillty to verify the print on the fly. So before I dive into the Plugin writing I’m wondering if it’s even possible to generate pictures using code in a plugin?

[ahoeben 1,770]

You could have a look at how the UFP Writer plugin creates thumbnails.

[nallath 1,104]

Interesting idea! I'd love to hear about your progress. if you need any specific advice on how to make the plugin, feel free to contact me.

[Hanshogeland 0]

Hi, Thx for you answers.

I have started to try to at least get the cura plugin to start and I´ve gotten so far that I cover the basic Menu item, however I copied the code from the UFP writer and tried to get the method rolling, but I´m stuck at understanding how to get the stream into the method, since I want to to run the script from the menu item I guess I need to add something by the start to access the stream. As you´ll see below I´m pretty novice on this - the out commented Write method need the input stream.

 

from typing import List
from typing import cast
from Charon.VirtualFile import VirtualFile #To open UFP files.
from Charon.OpenMode import OpenMode #To indicate that we want to write to UFP files.
from io import StringIO #For converting g-code to bytes.
from UM.Application import Application
from UM.Logger import Logger
from UM.Mesh.MeshWriter import MeshWriter #The writer we need to implement.
from UM.MimeTypeDatabase import MimeTypeDatabase, MimeType
from UM.PluginRegistry import PluginRegistry #To get the g-code writer.
from PyQt5.QtCore import QBuffer
from cura.Snapshot import Snapshot
from cura.Utils.Threading import call_on_qt_thread
from UM.i18n import i18nCatalog
from UM.Extension import Extension
from UM.PluginRegistry import PluginRegistry
from UM.Scene.SceneNode import SceneNode
from UM.Scene.Selection import Selection
from UM.Message import Message
from cura.CuraApplication import CuraApplication
import os

i18n_catalog = i18nCatalog("CreateTrainingDataPlugin")

class CreateTrainingDataPlugin(Extension, MeshWriter):
    def __init__(self):
        super().__init__()
        self.addMenuItem(i18n_catalog.i18n("Create training data pictures"), self.doExtendedCreateTrainingPics)
        #self.addMenuItem(i18n_catalog.i18n("Write"), self.doExtendedWrite)

        self._message = None
        MimeTypeDatabase.addMimeType(
            MimeType(
                name = "application/x-ufp",
                comment = "Ultimaker Format Package",
                suffixes = ["ufp"]
            )
        )
        self._snapshot = None

    def doExtendedCreateTrainingPics(self):
        self.doCreateTrainingdata(True)
    
    #@call_on_qt_thread
    #def doExtendedWrite(self):
    #   self.write(stream)
    @call_on_qt_thread
    def doCreateTrainingdata(self, extended_mode):
        self._message = Message(i18n_catalog.i18nc("@info:status", "Creating .PNG pics"), title = i18n_catalog.i18nc("@title", "PNG Pics"))
        self._message.show()
        #self.write()

    def _createSnapshot(self, *args):
        # must be called from the main thread because of OpenGL
        Logger.log("d", "Creating thumbnail image...")
        try:
            self._snapshot = Snapshot.snapshot(width = 300, height = 300)
        except Exception:
            Logger.logException("w", "Failed to create snapshot image")
            self._snapshot = None

    @call_on_qt_thread
    def write(self, stream, nodes, mode = MeshWriter.OutputMode.BinaryMode):
        archive = VirtualFile()
        archive.openStream(stream, "application/x-ufp", OpenMode.WriteOnly)

        #Store the g-code from the scene.
        archive.addContentType(extension = "gcode", mime_type = "text/x-gcode")
        gcode_textio = StringIO() #We have to convert the g-code into bytes.
        gcode_writer = cast(MeshWriter, PluginRegistry.getInstance().getPluginObject("GCodeWriter"))
        success = gcode_writer.write(gcode_textio, None)
        if not success: #Writing the g-code failed. Then I can also not write the gzipped g-code.
            self.setInformation(gcode_writer.getInformation())
            return False
        gcode = archive.getStream("/3D/model.gcode")
        gcode.write(gcode_textio.getvalue().encode("UTF-8"))
        archive.addRelation(virtual_path = "/3D/model.gcode", relation_type = "http://schemas.ultimaker.org/package/2018/relationships/gcode")

        self._createSnapshot()

        #Store the thumbnail.
        if self._snapshot:
            archive.addContentType(extension = "png", mime_type = "image/png")
            thumbnail = archive.getStream("/Metadata/thumbnail.png")

            thumbnail_buffer = QBuffer()
            thumbnail_buffer.open(QBuffer.ReadWrite)
            thumbnail_image = self._snapshot
            thumbnail_image.save(thumbnail_buffer, "PNG")

            thumbnail.write(thumbnail_buffer.data())
            archive.addRelation(virtual_path = "/Metadata/thumbnail.png", relation_type = "http://schemas.openxmlformats.org/package/2006/relationships/metadata/thumbnail", origin = "/3D/model.gcode")
        else:
            Logger.log("d", "Thumbnail not created, cannot save it")

 

[Cuq 162]

Hello

 

I'm certainly missing something but I d'ont know what ? I'm trying to use the same code to create a SnapShot .. I can get a file but the result is absolutly not correct. If Im using the plugin UPFWriter I got the result thumbnail.png  ( the first image) and my result is my_thumbnail.png  (the second image) ?

 

Actual source code 

# Copyright (c) 2020
# The SimpleShapes plugin is released under the terms of the AGPLv3 or higher.

from PyQt5.QtCore import QObject
from PyQt5.QtCore import QBuffer

from UM.Extension import Extension
from cura.CuraApplication import CuraApplication
from cura.Snapshot import Snapshot
from cura.Utils.Threading import call_on_qt_thread

from UM.Application import Application
from UM.Logger import Logger
from UM.Message import Message
from UM.MimeTypeDatabase import MimeTypeDatabase, MimeType

from UM.i18n import i18nCatalog
catalog = i18nCatalog("cura")

class CreateSnapShot(Extension, QObject,):
    
    def __init__(self, parent = None) -> None:
        QObject.__init__(self, parent)
        Extension.__init__(self)

        # attention pas le même nom que le menu
        self.addMenuItem(catalog.i18nc("@item:inmenu", "Snap"), self.doExtendedCreateTrainingPics)
        
        self._snapshot = None
       
 
    def doExtendedCreateTrainingPics(self):
        self.doCreateTrainingdata(True)

    #def doExtendedWrite(self):
    #   self.write(stream)
    @call_on_qt_thread
    def doCreateTrainingdata(self, extended_mode):
        self._write()
        self._message = Message(catalog.i18nc("@info:status", "Creating .PNG pics"), title = catalog.i18nc("@title", "PNG Pics"))
        self._message.show()
        
        
    def _createSnapshot(self, *args):
        # must be called from the main thread because of OpenGL
        Logger.log("d", "Creating thumbnail image...")
        try:
            self._snapshot = Snapshot.snapshot(width = 300, height = 300)
        except Exception:
            Logger.logException("w", "Failed to create snapshot image")
            self._snapshot = None
            
    @call_on_qt_thread
    def _write(self):

        self._createSnapshot()

        #Store the thumbnail.
        if self._snapshot:
            
            thumbnail_image = self._snapshot
            thumbnail_image.save("C:/temp/thumbnail.png", "PNG")
            Logger.log("d", "Thumbnail creation")
            
        else:
            Logger.log("d", "Thumbnail not created, cannot save it")
            

 

Edited May 27, 2020 by Cuq

[nallath 1,104]

This puzzled me a bit as well, it seems that the snapshot doesn't work correctly.

So for now you can get it to work by creating a job and creating the snapshots inside that job.

[Cuq 162]

1 hour ago, nallath said:

This puzzled me a bit as well, it seems that the snapshot doesn't work correctly.
 

 

Thanks for your answer @nallath; I don't think it's an issue from the function snapshot. Because if I use the UFP Writer plugin , the result is correct ( first ) image.  And normaly my code start from the same source .. But there is a real problem when I write the PNG file.

 

I'm thinking of an initialization that isn't done or a difference between the Save and Write method used in the UFP plugin code. But I can't figure out what's missing.

 

thumbnail_buffer = QBuffer()
thumbnail_buffer.open(QBuffer.ReadWrite)
thumbnail_image = self._snapshot
thumbnail_image.save(thumbnail_buffer, "PNG")

thumbnail.write(thumbnail_buffer.data())

versus

thumbnail_image = self._snapshot
thumbnail_image.save("C:/temp/thumbnail.png", "PNG")

 

Edited May 28, 2020 by Cuq

[Cuq 162]

tested also with :       

thumbnail_buffer = QBuffer()
thumbnail_buffer.open(QBuffer.ReadWrite)
thumbnail_image = self._snapshot
thumbnail_image.save(thumbnail_buffer, "PNG")
filehandle = open(Filename, 'w+b')
filehandle.write(thumbnail_buffer.data())
filehandle.close()

Same result 

[nallath 1,104]

The write UFP gode is called from a job, which means it's started from a non main thread. The @call_on_qt_thread ensures that it is later called on the main thread.

 

The problem seems to be that when the snapshot is directly called from the main thread, it gives the weird results. That's also why i suggested the job work around. So to get this to work:

Create a "WriteScreenshotJob" (subclass off Job), let the screenhot take happen in the run and ensure that the snapshot bit is run in a function with the call_on_qt_thread decorator.

[Cuq 162]

Thanks for your answer .. but your explanation are far away from my knowledges in Python .. So I will wait a wild and expted to find one day such type of sample code on Github 🙂

Edited May 29, 2020 by Cuq

[Cuq 162]

tested with 

class InterCallObject:
    def __init__(self):
        self.finish_event = threading.Event()
        self.result = None


def call_on_qt_thread(func):
    def _call_on_qt_thread_wrapper(*args, **kwargs):
        def _handle_call(ico, *args, **kwargs):
            ico.result = func(*args, **kwargs)
            ico.finish_event.set()
        
        inter_call_object = InterCallObject()
        new_args = tuple([inter_call_object] + list(args)[:])
        Logger.log("d", "new_args = %s", new_args)
        CuraApplication.getInstance().callLater(_handle_call, *new_args, **kwargs)
        inter_call_object.finish_event.wait()
        return inter_call_object.result
    return _call_on_qt_thread_wrapper

 

but   inter_call_object.finish_event.wait()   enter in a infinite loop

 

Edited May 29, 2020 by Cuq

[Cuq 162]

On 5/29/2020 at 12:06 PM, Cuq said:

Thanks for your answer .. but your explanation are far away from my knowledges in Python .. So I will wait a wild and expted to find one day such type of sample code on Github 🙂

 

Never Give up !  3 years later. : https://github.com/5axes/CuraSettingsWriter

 

 

 

At the same time I learned that you can embedded the image directly in the HTML file.