Inside the Ultimaker 3 - Day 5 - Developer mode & Linux/Systemd

[Daid 306]

This is day 5 of "Inside the Ultimaker 3", Developer mode & Linux/Systemd

 

-

 

 

Other days:

Day 1 - GCode

Day 2 - Remote access part 1

Day 3 - Remote access part 2

Day 4 - Electronics

 

-

 

 

Today we will take a short look at developer mode. Why? Because I want to make it clear what it does before misconceptions happen.

 

DEVELOPER MODE - ACTIVATED!

Developer mode can be activated from the {SYSTEM}->{Maintenance}->{Diagnostics}->{Developer mode}. Activating this will restart your machine. The reason for this is that it makes it easier for the rest of the code.

The rest of the code can just assume that whatever the developer mode switch was on startup, that is what it is during the rest of the time the machine is on.

 

-

 

 

Why did we add a {developer mode} switch, why is it off by default?

Well. This is the most important bit. Developer mode makes you machine insecure!. With this mode on, some safety mechanisms are disabled, and remote login is enabled. This means you do not just open the machine for yourself.

You potentially also open up the machine for hackers on your local network. So take extreme care with this. If you do not know what you are doing, developer mode is NOT needed.

 

What does it do?

First off, and most importantly. It activate the ssh server. This server allows for remote login trough the network. Linux users will be very familiar with this.

Note that you can completely mess up your system with this. Which is why I'm not giving too much information on how to use this right now. As you can access, delete, change any file on the printer.

Secondly, it changes the main menu so it always shows the IP address on the printer. We are doing so much with so many printers at the office, that a quick look at the IP address is extremely useful.

Next up, it when an error/warning happens, next to the normal message that you get, you get the internal developer message as well. This can provide more information, or confuse you even more.

 

Inside the printer - Linux

Now, we will get nerdy here. So unless you are a Linux nerd, the rest here will sound like voodoo.

To start off with some basics. We are running:

 

• A custom image, based on Debian stable (jessie)
  
• We are using systemd. Some Linux people love to hate this. We love it.
  
• If we don't need it, it is most likely not installed.
  
• Most of our own code runs in Python3. So python3 is available on the system.
  
• apt-get is available, but if you run apt-get update you will run out of disk space. This is due to the fact that the root partition is not big enough to contain the huge package list from debian. Run mount /dev/mmcblk0p3 /var/lib/apt/ first. You could also resize your root partition. But that is a bit harder.
  

 

 

Linux - systemd

First off. I love systemd. It makes so many things easier. We are using it for a few main things:

 

1. Starting up the system
   
2. Logging
   
3. Bringing down the system for a system update
   

 

 

Linux - systemd - Starting up

Our linux system starts up with systemd. Systemd is some fancy piece of software that handles all kinds of startup and stop action, as well as some more things.

Systemd does this by the means of services. You can read a lot about this online. So I'll just go into the very basics. Our own services are defined in:

 

/etc/systemd/system

 

And we have a whole bunch of files there that start with "griffin". Most simple example, we have the file:

 

/etc/systemd/system/griffin.nfc.service

 

This file handles the NFC service. The NFC service is the part of the system that talks with the NFC hardware to talk with the (optional) NFC tags in spools. The rest of the system has (almost) no knowledge of NFC. I've seperated concerns like that.

If you are on the system, you can just read it with cat, as it's a normal text file. It will show you how this service is started, and what it depends on. As for example, the nfc service depends on the printer service to run before it can do it's thing.

The reason for this is, it needs to monitor the printer service, as the printer service knows when a material change is happening. And that's when the NFC service needs to do it's thing. So the NFC service listens to the printer service.

 

-

 

 

Now, to start/stop a service we can use systemctl start griffin.nfc.service or systemctl stop griffin.nfc.service. But you will most likely cause problems if you stop indivitual services if you have no clue what you are doing.

 

Linux - systemd - Logging

All these services become really interresting once you add the logging. Systemd adds logging for us as well. But, as added advantage over the "classic" Linux logging, we can filter in this logging.

A really common (almost muscle memory) command that we use a lot is:

 

journalctl -fu griffin.printer

 

This looks at the logging of just the printer service, and "follows" this logging. Which means that the process you just started won't exit and that every new log message that is added will be directly shown on your screen. I quite often have this running during debugging of issues.

There are quite a few cool things you can do with journalctl, other examples:

 

#Only show the log from this bootjournalctl -b 0#Show the log from the previous bootjournalctl -b -1#Show the last 1000 log lines from the printer servicejournalctl -n 1000 -u griffin.printer

 

 

Linux - systemd - System update

When we want to do a system update, we have a bit of a problem. As we are running on this system. So we need to update ourselves, while on the move.

Once again, systemd really helps us here. We have a special service that we can start, called "update.rootfs.service". This service is intended to run on it's own, without anything else. Systemd as a function for this called "isolate".

The isolate function means only that service runs, and it's dependencies. Our update service has no dependencies, so that means everything else is brought down once we isolate that service.

 

-

 

 

When the update.rootfs.service is isolate, we know the rest of the system is not running, and it is safe to update those files. You can check /usr/lib/systemupdate/initiate_update.sh for details on how this update process works.

It uses rsync, which means only the changed files will be copied over on a system update. Meaning that the amount of writes to disk is actually quite limited, and the chance of the system no longer starting up if the power shuts off during an update is really low.

Note, this isolating of the update.rootfs.service is done AFTER signature verification. The full update process also checks if the firmware update file is signed by our private key. To prevent man-in-the-middle-attacks on system updates.

 

Closing words

I understand if this was really really really technical and nerdy. But, if you are toying with the UM3 system, and getting your hands dirty. This explains a few really basic things.

Tomorrow I will go into Active Leveling, as I think that's really interesting subject.

 

-

 

 

 

-

 

 

Disclaimer: Any information presented here could be wrong. I did my best to proof read everything, but it could conflict with the actual behavior of the printer.

Edited October 25, 2016 by Guest

[ivan-lipunov 15]

Wow, thats cool! Can we migrate a system to a sata drive without any loss of functionality or just using sd with more volume? It does not cause conflict of versions with subsequent update of Debian ? Is it possible to add new functionality to the Griffin using free GPIO?

Edited October 25, 2016 by Guest

[Dim3nsioneer 558]

@Daid, I didn't have the time yet to read all your information about the UM3 interns. But I will. As this is by far the most complete description I've seen here about electronics and software in an Ultimaker product. You get a huge thank you from me!

I'm convinced that these threads will be used for many months...

Edited October 25, 2016 by Guest

[korneel 176]

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

[ivan-lipunov 15]

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

 

Its pretty easy.

Look at core dumps

 

find / -xdev -name core -ls -o  -path "/lib*" -prune

 

Remove already installed packages

 

apt-get autoremove --purge 

 

And clean outdated kernels, if they exist

 

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

 

[korneel 176]

 

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

 

Its pretty easy.

Look at core dumps

 

 

find / -xdev -name core -ls -o  -path "/lib*" -prune

 

Remove already installed packages

 

apt-get autoremove --purge 

 

And clean outdated kernels, if they exist

 

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

 

 

nothing is every easy..

as soon as i try something with apt-get :

E: Write error - write (28: No space left on device)

[ivan-lipunov 15]

 

 

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

 

Its pretty easy.

Look at core dumps

 

 

find / -xdev -name core -ls -o  -path "/lib*" -prune

 

Remove already installed packages

 

apt-get autoremove --purge 

 

And clean outdated kernels, if they exist

 

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

 

 

nothing is every easy..

as soon as i try something with apt-get :

E: Write error - write (28: No space left on device)

 

Even

sudo apt-get clean

show these error?

Edited October 29, 2016 by Guest

[ivan-lipunov 15]

 

@Sandervg, why in my last comment part of the text is bold? I don't use any formatting for these , just for the code part.

[korneel 176]

 

 

 

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

 

Its pretty easy.

Look at core dumps

 

 

find / -xdev -name core -ls -o  -path "/lib*" -prune

 

Remove already installed packages

 

apt-get autoremove --purge 

 

And clean outdated kernels, if they exist

 

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

 

 

nothing is every easy..

as soon as i try something with apt-get :

E: Write error - write (28: No space left on device)

 

Even

 

sudo apt-get clean

 

 show these error?

 

no that one works.. for the rest:

Reading package lists... Error!

E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)

E: Failed to truncate file - ftruncate (9: Bad file descriptor)

E: The package lists or status file could not be parsed or opened.

[korneel 176]

 

 

 

so let's say i ran an apt-get update before reading your text and /dev/root is full.. how would one solve this theoretical scenario?

 

Its pretty easy.

Look at core dumps

 

 

find / -xdev -name core -ls -o  -path "/lib*" -prune

 

Remove already installed packages

 

apt-get autoremove --purge 

 

And clean outdated kernels, if they exist

 

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

 

 

nothing is every easy..

as soon as i try something with apt-get :

E: Write error - write (28: No space left on device)

 

Even

 

sudo apt-get clean

 

 show these error?

 

no that one works.. for the rest:

Reading package lists... Error!

E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)

E: Failed to truncate file - ftruncate (9: Bad file descriptor)

E: The package lists or status file could not be parsed or opened.

[ivan-lipunov 15]

Try

 

ps aux | grep mlocate

 

and post results

Maybe updatedb.mlocate process take too much space. If yes, just kill them

Edited November 5, 2016 by Guest

[korneel 176]

Try

 

ps aux | grep mlocate

 

and post results

Maybe updatedb.mlocate process take too much space. If yes, just kill them

 

root@ultimakersystem-ccbdd3000318:~# ps aux | grep mlocate

root 15615 0.0 0.0 2068 532 pts/0 S+ 23:45 0:00 grep mlocate

[ivan-lipunov 15]

Run these: sudo find / -path /var/ -prune -or -size +250M -print

Whats size of /usr/bin/updatedb.mlocate?

[korneel 176]

Run these: sudo find / -path /var/ -prune -or -size +250M -print

Whats size of /usr/bin/updatedb.mlocate?

 

no results from the first cmdlet..

and that second file does not exist..

[SandervG 1,522]

@Sandervg, why in my last comment part of the text is bold? I don't use any formatting for these , just for the code part.

 

We are looking into this bug. If you have any more feedback, please post it in the forum feedback post

Last week we found out that I saw some quotes in bold, and my colleague didn't. But after logging in as me, he did. Not sure if this is related, but that is under investigation now.

If you edit your post, and post it again without any edits, does it change back to normal?

[companje 2]

Thanks for these great blog posts! It really helps understanding the system.

When I run

mount /dev/mmcblk0p3 /var/lib/apt/

before running

apt-get update

I still run out of disk space. Am I missing something?

 

 ......Hit http://security.debian.org stable/updates/main Translation-enReading package lists... Error!E: Write error - write (28: No space left on device)E: IO Error saving source cacheE: The package lists or status file could not be parsed or opened.

 

[Olliver 6]

While this post is probably antient by now, just to let this lingering questing not go unanswered:

apt-get update stores files in both /var/lib/apt and /var/cache/apt. Removing both these directories solves the disk full issue with regards to files generated from apt.

rm -r /var/lib/apt /var/cache/apt

Having said that, there are two methods to get apt working. One is to use tmpfs (we did this in the latest firmwares, but there's problems and risks there, such as running out of memory)

mkdir -p /var/lib/apt /var/cache/apt

mount -t tmpfs /var/lib/apt

mount -t tmpfs /var/cache/apt

That said, what I (used to) do a lot, is mount /dev/mmcblk0p3 /home/ultimaker/tmp

mkdir -p /home/ultimaker/tmp/apt_cache

mkdir -p /home/ultimaker/tmp/apt_lib

ln -s /home/ultimaker/tmp/apt_cache /var/cache/apt

ln -s /home/ultimaker/tmp/apt_lib /var/lib/apt

after that, after each reboot you still need to mount mmclk0p3, but you have plenty of space to do apt-get update

[YoP3D 0]

Hello, I started to change the firmware, but I don't know which command I can use to edit files. Also I wanted to know which commands I can use with the ssh connection for example "nano" or another console editor.

[Adam Kicks 0]

I've been wanting to create a service that saves the .g files when using 'Print Over Network' from Cura. Either by USB or to NAS. 

 

Is there a tmp mount point?

[jansen 0]

On 10/25/2016 at 7:08 PM, Daid said:

You could also resize your root partition. But that is a bit harder.

Hi Daid,

could you elaborate on how to do this? 

I'm familiar with resizing partitions/filesystems on x86/non-embedded device, by doing an online-resize or booting into a live-linux-system.

F2FS does not seem to support online resizing. 

 

My goal is to install some more python packages for a research project.

 

Regards, Jan

[Daid 306]

You are right that F2FS does not support online resizing. This is also made more difficult by the fact that we allocated the rest of the space for Cura Connect.

 

Updating to a 5.X release will also resize your partitions to the same size as the S5 firmware. And this update is available at the moment, so that might give you enough space.

 

Another way would be to boot from the internal SD card slot and do the same method as with a live-linux-system.

 

Finally, you could also opt for the dirty method, where you place for example /usr/share on the 2nd partition, and symlink /usr/share to that partition.

 

As always, if you mess up, it's your own risk.

[jansen 0]

Hi Daid, thanks for your fast response. :)

11 minutes ago, Daid said:

Updating to a 5.X release will also resize your partitions to the same size as the S5 firmware. And this update is available at the moment, so that might give you enough space.

Do you have a number how much space would be available? I'm currently at 16MB free space, installing python3-pip will take about 180MB. 

 

The other two options are also fine. I'll probably take the live-linux route, if the 5.X firmware does not provide enough space.

[Daid 306]

The 5.X firmware has about 900MB for the root partition, up from the 500MB, so that should help a lot.

[jeffroe 3]

Are there any commands that can be sent to get the printer out of an "aborted" state?  We are trying to do some remote management on machines.  As an example, I have a machine right now that I sent a print to, but it is stuck in "Aborted" state.  I suspect it may be waiting for a confirmation button press (this is an S5).  I have tried logging in as root, and issuing a reboot command, but it comes back up in the same state.

 

[SteveRichard 2]

Is there a way to update the Ultimaker S5 through developer mode/ssh?  I have a need script these firmware updates over my network.