Jump to content

Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd


Daid

Recommended Posts

Posted (edited) · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

This is day 5 of "Inside the Ultimaker 3", Developer mode & Linux/Systemd

 

-

 

 

Other days:

Day 1 - GCode

Day 2 - Remote access part 1

Day 3 - Remote access part 2

Day 4 - Electronics

 

-

 

 

Today we will take a short look at developer mode. Why? Because I want to make it clear what it does before misconceptions happen.

 

DEVELOPER MODE - ACTIVATED!

Developer mode can be activated from the {SYSTEM}->{Maintenance}->{Diagnostics}->{Developer mode}. Activating this will restart your machine. The reason for this is that it makes it easier for the rest of the code.

The rest of the code can just assume that whatever the developer mode switch was on startup, that is what it is during the rest of the time the machine is on.

 

-

 

 

Why did we add a {developer mode} switch, why is it off by default?

Well. This is the most important bit. Developer mode makes you machine insecure!. With this mode on, some safety mechanisms are disabled, and remote login is enabled. This means you do not just open the machine for yourself.

You potentially also open up the machine for hackers on your local network. So take extreme care with this. If you do not know what you are doing, developer mode is NOT needed.

 

What does it do?

First off, and most importantly. It activate the ssh server. This server allows for remote login trough the network. Linux users will be very familiar with this.

Note that you can completely mess up your system with this. Which is why I'm not giving too much information on how to use this right now. As you can access, delete, change any file on the printer.

Secondly, it changes the main menu so it always shows the IP address on the printer. We are doing so much with so many printers at the office, that a quick look at the IP address is extremely useful.

Next up, it when an error/warning happens, next to the normal message that you get, you get the internal developer message as well. This can provide more information, or confuse you even more.

 

Inside the printer - Linux

Now, we will get nerdy here. So unless you are a Linux nerd, the rest here will sound like voodoo.

To start off with some basics. We are running:

 

  • A custom image, based on Debian stable (jessie)
  • We are using systemd. Some Linux people love to hate this. We love it.
  • If we don't need it, it is most likely not installed.
  • Most of our own code runs in Python3. So python3 is available on the system.
  • apt-get is available, but if you run apt-get update you will run out of disk space. This is due to the fact that the root partition is not big enough to contain the huge package list from debian. Run mount /dev/mmcblk0p3 /var/lib/apt/ first. You could also resize your root partition. But that is a bit harder.

 

 

Linux - systemd

First off. I love systemd. It makes so many things easier. We are using it for a few main things:

 

  1. Starting up the system
  2. Logging
  3. Bringing down the system for a system update

 

 

Linux - systemd - Starting up

Our linux system starts up with systemd. Systemd is some fancy piece of software that handles all kinds of startup and stop action, as well as some more things.

Systemd does this by the means of services. You can read a lot about this online. So I'll just go into the very basics. Our own services are defined in:

 

/etc/systemd/system

 

And we have a whole bunch of files there that start with "griffin". Most simple example, we have the file:

 

/etc/systemd/system/griffin.nfc.service

 

This file handles the NFC service. The NFC service is the part of the system that talks with the NFC hardware to talk with the (optional) NFC tags in spools. The rest of the system has (almost) no knowledge of NFC. I've seperated concerns like that.

If you are on the system, you can just read it with cat, as it's a normal text file. It will show you how this service is started, and what it depends on. As for example, the nfc service depends on the printer service to run before it can do it's thing.

The reason for this is, it needs to monitor the printer service, as the printer service knows when a material change is happening. And that's when the NFC service needs to do it's thing. So the NFC service listens to the printer service.

 

-

 

 

Now, to start/stop a service we can use systemctl start griffin.nfc.service or systemctl stop griffin.nfc.service. But you will most likely cause problems if you stop indivitual services if you have no clue what you are doing.

 

Linux - systemd - Logging

All these services become really interresting once you add the logging. Systemd adds logging for us as well. But, as added advantage over the "classic" Linux logging, we can filter in this logging.

A really common (almost muscle memory) command that we use a lot is:

 

journalctl -fu griffin.printer

 

This looks at the logging of just the printer service, and "follows" this logging. Which means that the process you just started won't exit and that every new log message that is added will be directly shown on your screen. I quite often have this running during debugging of issues.

There are quite a few cool things you can do with journalctl, other examples:

 

#Only show the log from this bootjournalctl -b 0#Show the log from the previous bootjournalctl -b -1#Show the last 1000 log lines from the printer servicejournalctl -n 1000 -u griffin.printer

 

 

Linux - systemd - System update

When we want to do a system update, we have a bit of a problem. As we are running on this system. So we need to update ourselves, while on the move.

Once again, systemd really helps us here. We have a special service that we can start, called "update.rootfs.service". This service is intended to run on it's own, without anything else. Systemd as a function for this called "isolate".

The isolate function means only that service runs, and it's dependencies. Our update service has no dependencies, so that means everything else is brought down once we isolate that service.

 

-

 

 

When the update.rootfs.service is isolate, we know the rest of the system is not running, and it is safe to update those files. You can check /usr/lib/systemupdate/initiate_update.sh for details on how this update process works.

It uses rsync, which means only the changed files will be copied over on a system update. Meaning that the amount of writes to disk is actually quite limited, and the chance of the system no longer starting up if the power shuts off during an update is really low.

Note, this isolating of the update.rootfs.service is done AFTER signature verification. The full update process also checks if the firmware update file is signed by our private key. To prevent man-in-the-middle-attacks on system updates.

 

Closing words

I understand if this was really really really technical and nerdy. But, if you are toying with the UM3 system, and getting your hands dirty. This explains a few really basic things.

Tomorrow I will go into Active Leveling, as I think that's really interesting subject.

 

-

 

 

 

-

 

 

Disclaimer: Any information presented here could be wrong. I did my best to proof read everything, but it could conflict with the actual behavior of the printer.

Edited by Guest
  • Like 5
Link to post
Share on other sites

Posted (edited) · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

Wow, thats cool! Can we migrate a system to a sata drive without any loss of functionality or just using sd with more volume? It does not cause conflict of versions with subsequent update of Debian ? Is it possible to add new functionality to the Griffin using free GPIO?

Edited by Guest
  • Link to post
    Share on other sites

    Posted (edited) · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    @Daid, I didn't have the time yet to read all your information about the UM3 interns. But I will. As this is by far the most complete description I've seen here about electronics and software in an Ultimaker product. You get a huge thank you from me!

    I'm convinced that these threads will be used for many months...

    Edited by Guest
  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

    • Like 1
    Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

     

    Its pretty easy.

    Look at core dumps

     

    find / -xdev -name core -ls -o  -path "/lib*" -prune

     

    Remove already installed packages

     

    apt-get autoremove --purge 

     

    And clean outdated kernels, if they exist

     

    dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

     

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

     

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

     

    Its pretty easy.

    Look at core dumps

     

     

    find / -xdev -name core -ls -o  -path "/lib*" -prune

     

    Remove already installed packages

     

    apt-get autoremove --purge 

     

    And clean outdated kernels, if they exist

     

    dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

     

     

    nothing is every easy..

    as soon as i try something with apt-get :

    E: Write error - write (28: No space left on device)

  • Link to post
    Share on other sites

    Posted (edited) · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

     

     

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

     

    Its pretty easy.

    Look at core dumps

     

     

    find / -xdev -name core -ls -o  -path "/lib*" -prune

     

    Remove already installed packages

     

    apt-get autoremove --purge 

     

    And clean outdated kernels, if they exist

     

    dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

     

     

    nothing is every easy..

    as soon as i try something with apt-get :

    E: Write error - write (28: No space left on device)

     

    Even

    sudo apt-get clean

    show these error?

    Edited by Guest
  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

     

    @Sandervg, why in my last comment part of the text is bold? I don't use any formatting for these :D, just for the code part.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

     

     

     

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

     

    Its pretty easy.

    Look at core dumps

     

     

    find / -xdev -name core -ls -o  -path "/lib*" -prune

     

    Remove already installed packages

     

    apt-get autoremove --purge 

     

    And clean outdated kernels, if they exist

     

    dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

     

     

    nothing is every easy..

    as soon as i try something with apt-get :

    E: Write error - write (28: No space left on device)

     

    Even

     

    sudo apt-get clean

     

     show these error?

     

    no that one works.. for the rest:

    Reading package lists... Error!

    E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)

    E: Failed to truncate file - ftruncate (9: Bad file descriptor)

    E: The package lists or status file could not be parsed or opened.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

     

     

     

    so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

     

    Its pretty easy.

    Look at core dumps

     

     

    find / -xdev -name core -ls -o  -path "/lib*" -prune

     

    Remove already installed packages

     

    apt-get autoremove --purge 

     

    And clean outdated kernels, if they exist

     

    dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

     

     

    nothing is every easy..

    as soon as i try something with apt-get :

    E: Write error - write (28: No space left on device)

     

    Even

     

    sudo apt-get clean

     

     show these error?

     

    no that one works.. for the rest:

    Reading package lists... Error!

    E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)

    E: Failed to truncate file - ftruncate (9: Bad file descriptor)

    E: The package lists or status file could not be parsed or opened.

  • Link to post
    Share on other sites

    Posted (edited) · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Try

     

    ps aux | grep mlocate

     

    and post results

    Maybe updatedb.mlocate process take too much space. If yes, just kill them

    Edited by Guest
  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Try

     

    ps aux | grep mlocate

     

    and post results

    Maybe updatedb.mlocate process take too much space. If yes, just kill them

     

    root@ultimakersystem-ccbdd3000318:~# ps aux | grep mlocate

    root 15615 0.0 0.0 2068 532 pts/0 S+ 23:45 0:00 grep mlocate

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Run these: sudo find / -path /var/ -prune -or -size +250M -print

    Whats size of /usr/bin/updatedb.mlocate?

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Run these: sudo find / -path /var/ -prune -or -size +250M -print

    Whats size of /usr/bin/updatedb.mlocate?

     

    no results from the first cmdlet..

    and that second file does not exist..

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    @Sandervg, why in my last comment part of the text is bold? I don't use any formatting for these :D, just for the code part.

     

    We are looking into this bug. If you have any more feedback, please post it in the forum feedback post :)

    Last week we found out that I saw some quotes in bold, and my colleague didn't. But after logging in as me, he did. Not sure if this is related, but that is under investigation now.

    If you edit your post, and post it again without any edits, does it change back to normal?

    • Like 1
    Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Thanks for these great blog posts! It really helps understanding the system.

    When I run

    mount /dev/mmcblk0p3 /var/lib/apt/

    before running

    apt-get update

    I still run out of disk space. Am I missing something?

     

     ......Hit http://security.debian.org stable/updates/main Translation-enReading package lists... Error!E: Write error - write (28: No space left on device)E: IO Error saving source cacheE: The package lists or status file could not be parsed or opened.

     

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    While this post is probably antient by now, just to let this lingering questing not go unanswered:

    apt-get update stores files in both /var/lib/apt and /var/cache/apt. Removing both these directories solves the disk full issue with regards to files generated from apt.

    rm -r /var/lib/apt /var/cache/apt

    Having said that, there are two methods to get apt working. One is to use tmpfs (we did this in the latest firmwares, but there's problems and risks there, such as running out of memory)

    mkdir -p /var/lib/apt /var/cache/apt

    mount -t tmpfs /var/lib/apt

    mount -t tmpfs /var/cache/apt

    That said, what I (used to) do a lot, is mount /dev/mmcblk0p3 /home/ultimaker/tmp

    mkdir -p /home/ultimaker/tmp/apt_cache

    mkdir -p /home/ultimaker/tmp/apt_lib

    ln -s /home/ultimaker/tmp/apt_cache /var/cache/apt

    ln -s /home/ultimaker/tmp/apt_lib /var/lib/apt

    after that, after each reboot you still need to mount mmclk0p3, but you have plenty of space to do apt-get update :)

    • Like 2
    Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Hello, I started to change the firmware, but I don't know which command I can use to edit files. Also I wanted to know which commands I can use with the ssh connection for example "nano" or another console editor.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    I've been wanting to create a service that saves the .g files when using 'Print Over Network' from Cura. Either by USB or to NAS. 

     

    Is there a tmp mount point?

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd
    On 10/25/2016 at 7:08 PM, Daid said:

    You could also resize your root partition. But that is a bit harder.

    Hi Daid,

    could you elaborate on how to do this? 

    I'm familiar with resizing partitions/filesystems on x86/non-embedded device, by doing an online-resize or booting into a live-linux-system.

    F2FS does not seem to support online resizing. 

     

    My goal is to install some more python packages for a research project.

     

    Regards, Jan

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    You are right that F2FS does not support online resizing. This is also made more difficult by the fact that we allocated the rest of the space for Cura Connect.

     

    Updating to a 5.X release will also resize your partitions to the same size as the S5 firmware. And this update is available at the moment, so that might give you enough space.

     

    Another way would be to boot from the internal SD card slot and do the same method as with a live-linux-system.

     

    Finally, you could also opt for the dirty method, where you place for example /usr/share on the 2nd partition, and symlink /usr/share to that partition.

     

    As always, if you mess up, it's your own risk.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Hi Daid, thanks for your fast response. :)

    11 minutes ago, Daid said:

    Updating to a 5.X release will also resize your partitions to the same size as the S5 firmware. And this update is available at the moment, so that might give you enough space.

    Do you have a number how much space would be available? I'm currently at 16MB free space, installing python3-pip will take about 180MB. 

     

    The other two options are also fine. I'll probably take the live-linux route, if the 5.X firmware does not provide enough space.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    The 5.X firmware has about 900MB for the root partition, up from the 500MB, so that should help a lot.

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Are there any commands that can be sent to get the printer out of an "aborted" state?  We are trying to do some remote management on machines.  As an example, I have a machine right now that I sent a print to, but it is stuck in "Aborted" state.  I suspect it may be waiting for a confirmation button press (this is an S5).  I have tried logging in as root, and issuing a reboot command, but it comes back up in the same state.

     

  • Link to post
    Share on other sites

    Posted · Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

    Is there a way to update the Ultimaker S5 through developer mode/ssh?  I have a need script these firmware updates over my network.

    • Like 1
    Link to post
    Share on other sites

    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now
    • Our picks

      • Introducing Universal Cura Projects in the UltiMaker Cura 5.7 beta
        Strap in for the first Cura release of 2024! This 5.7 beta release brings new material profiles as well as cloud printing for Method series printers, and introduces a powerful new way of sharing print settings using printer-agnostic project files! Also, if you want to download the cute dinosaur card holder featured below, it was specially designed for this release and can be found on Thingiverse! 
          • Like
        • 10 replies
      • S-Line Firmware 8.3.0 was released Nov. 20th on the "Latest" firmware branch.
        (Sorry, was out of office when this released)

        This update is for...
        All UltiMaker S series  
        New features
         
        Temperature status. During print preparation, the temperatures of the print cores and build plate will be shown on the display. This gives a better indication of the progress and remaining wait time. Save log files in paused state. It is now possible to save the printer's log files to USB if the currently active print job is paused. Previously, the Dump logs to USB option was only enabled if the printer was in idle state. Confirm print removal via Digital Factory. If the printer is connected to the Digital Factory, it is now possible to confirm the removal of a previous print job via the Digital Factory interface. This is useful in situations where the build plate is clear, but the operator forgot to select Confirm removal on the printer’s display. Visit this page for more information about this feature.
          • Like
        • 0 replies
    ×
    ×
    • Create New...