Cura not working - No space left on device

[SmartBlug 0]

I cannot use my printer anymore (Extended 3)

 

On the log :

ultimakersystem-ccbdd300033b flipflop_wsgi.py[1314]: OSError: [Errno 28] No space left on device: '/var/lib/griffin/cluster/preferences.json.new'

 

If I connect in dev mode : 

/dev/mmcblk1p2    963584 828416         0 100% /
devtmpfs          481308      0    481308   0% /dev
tmpfs             515100      0    515100   0% /dev/shm
tmpfs             515100   7840    507260   2% /run
tmpfs               5120      0      5120   0% /run/lock
tmpfs             515100      0    515100   0% /sys/fs/cgroup
tmpfs             515100      0    515100   0% /var/lib/apt
tmpfs             515100      0    515100   0% /tmp
tmpfs             515100      0    515100   0% /var/cache/apt
/dev/mmcblk1p3   2775040  26632   2527224   2% /var/spool/cluster
 

Is there a way to clean the file system ?

 

Thanks,

[Smithy 1,141]

Have you tried a reboot of the printer? Maybe there are some cleanups during the boot process, but I don't know.

[SmartBlug 0]

Yes, of course. 

I also tried to reset cura settings but same problem. 

[Smithy 1,141]

The only idea I have left is that you check /var/log for large logfiles. I would do that at least on a normal Linux system.

But I've never heard of the printer running out of disk space. Maybe someone else knows more, sorry!

[SmartBlug 0]

I deleted some of the logs and get back 100k...

 

/var/log/journal is big, how to safely delete this journal ?

 

size of main folders :

526231    /
215146    /usr
168876    /var
99980    /usr/lib
91755    /var/lib
87435    /usr/share
83935    /var/lib/connman
83910    /var/lib/connman/ethernet_ccbdd300033b_cable
75247    /var/log
74847    /var/log/journal
74844    /var/log/journal/5e702514274833c0ebe690cc5ccc224d
62364    /opt
53676    /lib
50743    /opt/qt
46852    /usr/share/fonts
42043    /opt/qt/lib
39929    /usr/lib/python3
39926    /usr/lib/python3/dist-packages
37227    /usr/share/fonts/opentype
37224    /usr/share/fonts/opentype/noto
27626    /usr/lib/arm-linux-gnueabihf
26975    /lib/modules
26971    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede
25823    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede/kernel
23787    /usr/bin
21829    /usr/lib/python3.4
14635    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede/kernel/drivers
13239    /usr/lib/python3/dist-packages/numpy
12523    /usr/share/i18n
11618    /opt/pyqt
11568    /lib/udev
11428    /opt/pyqt/PyQt5
9486    /usr/share/i18n/locales
8704    /lib/arm-linux-gnueabihf
8289    /usr/share/okuda
8036    /usr/lib/python3/dist-packages/cluster
7911    /usr/lib/python3/dist-packages/cluster/app
7840    /run
7412    /usr/share/griffin
7267    /usr/lib/python3/dist-packages/sqlalchemy
7237    /usr/share/okuda/resources
6606    /usr/lib/python3/dist-packages/cluster/app/static
6440    /run/log/journal/5e702514274833c0ebe690cc5ccc224d
6440    /run/log/journal
6440    /run/log
 

[Smithy 1,141]

You mean the directory /var/log/journal ?

It should be safe to delete log files inside the directory, but keep the directory itself. 

 

Are there many log files inside? Because normally there is log rotation, maybe not working or configured on the printer.

If there is only one big logfile, you can also truncate the file with just

> logfilename

instead of delete them. The > is important which means you redirect the input, which is nothing to the file, resulting in a 0 byte file. But just a delete should also be no problem.

 

But I am not sure if this is enough, the directory has 74MB. Maybe there is something else which consumes so much disk space.  

[SmartBlug 0]

I removed the folder in /var/log/journal and get back 58Mb 😉

It seems it was logs not properly deleted in the past by the software (probably due to bug in old releases)

 

now, only

-rw-r-----  1 root systemd-journal 8388608 Dec 15 07:40 system.journal
-rw-r-----+ 1 root systemd-journal 8388608 Dec 15 07:38 user-1000.journal
 

I'll monitor to see if this folder size continue to increase or not...

[Whitespace 2]

I have the same problem.  I suspect it's avahi-daemon logging too much (see below):

 

- I was printing fine

- after some number of prints Cura Connect would break

- reading logs on the touchscreen showed that no space was left on /

- couldn't enable dev mode to ssh in

- would reset cura connect — losing all my stats — printer would work again for a couple of days, then run out of space again

- this time though I am able to SSH in and can verify that /var/log/journal/** is taking up 89MB of 806 available on /

 

root@ultimakersystem-ccbdd30087d3:/var/log/journal/c9c140d1e794462fbc6facaa6f36863f# ls -l
total 90252
-rw-r-----  1 root systemd-journal   53248 Jan  1  1970 system.journal
-rw-r-----  1 root systemd-journal 8388608 Jan  1  1970 system@00000008ccb660c6-1b488519ed25a7ee.journal~
-rw-r-----  1 root systemd-journal 8388608 May 21 17:32 system@8a780352b4d74a25abba2411be124d36-000000000001d96a-0005a62956cc9fe7.journal
-rw-r-----  1 root systemd-journal 8388608 May 21 21:06 system@8a780352b4d74a25abba2411be124d36-0000000000020038-0005a62be43bd14a.journal
-rw-r-----  1 root systemd-journal 8388608 May 22 00:00 system@8a780352b4d74a25abba2411be124d36-000000000002253e-0005a62ee1b94778.journal
-rw-r-----  1 root systemd-journal 8388608 May 22 00:32 system@8a780352b4d74a25abba2411be124d36-0000000000025072-0005a631525ec98c.journal
-rw-r-----  1 root systemd-journal 8388608 May 22 01:04 system@8a780352b4d74a25abba2411be124d36-0000000000027c8b-0005a631c3fec32d.journal
-rw-r-----  1 root systemd-journal 8388608 May 20 23:35 system@ebae5b2b5a4646dabb9d3c59a8d6fe95-0000000000018b9f-0005a61aead30f5f.journal
-rw-r-----  1 root systemd-journal 8388608 May 21 03:07 system@ebae5b2b5a4646dabb9d3c59a8d6fe95-000000000001b68c-0005a61cd98f111f.journal
-rw-r-----+ 1 root systemd-journal 8388608 May 22 01:08 user-1000.journal
-rw-r-----+ 1 root systemd-journal 8388608 May 22 01:04 user-1000@6376e02ffff348eea29dbb59865292ca-0000000000000000-0000000000000000.journal
-rw-r-----+ 1 root systemd-journal 8388608 May 21 03:07 user-1000@6376e02ffff348eea29dbb59865292ca-000000000001d767-0005a61fba8f8837.journal

 

Tailing logs with journalctl -r, I see that avahi-daemon is spamming useless logs to disk:

Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.137.
Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.32.
Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::4da:c6cf:bdab:6026.
Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::21:efd5:4317:132e.
Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::4aa6:b8ff:fe83:bbfc.
Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.173.

Also other systems are logging too much to STDOUT.

[Whitespace 2]

For others who want to debug here are some commands you might find useful:

df -h                      # shows disk usage in human-readable terms
journalctl --disk-usage    # shows how much disk space logs are taking
ls -l /var/log/journal/**/ # list all journald logs
rm /var/log/journal/**/*@* # delete all archived logs (logs containing an @ symbol)

Removing those logs saved me a bunch of space.  I also edited /etc/systemd/journald.conf and halved SystemMaxUse to 32M, but that didn't seem to really work.

Edited June 1, 2020 by Whitespace
typo

[Whitespace 2]

I suspect that avahi-daemon is logging too much.  The installed version is 0.6.31, and the very next version — 0.6.32 — fixes these warnings from appearing in logs: https://github.com/lathiat/avahi/blob/master/docs/NEWS#L155

 

The installed version is over four years old :(

 

I can't install it via apt-get, so I disabled avahi-daemon by modifying /etc/default/avahi-daemon and stopping it with update-rc.d avahi-daemon stop

[gr5 2,071]

@Whitespace what kind of printer do you have?  Your profile says you have only an S5?  Is this on your S5?

[Guest maht]

@CarloK - any ideas about this?

[CarloK 201]

I'm currently working on another project, not on the S3/S5 so anything I mention here is from own experience.

 

The bad part is, we are using old software. The good part is, we are using old software.

Many bugs have been solved since the original release and I'd like to see many of our libraries updated to newer versions. The good part is that the software is working on thousands of printers with relative few problems and most of the problems are perhaps annoying, but at least known. 

 

@Whitespace what changed in your setup for these errors to start occuring? Is it a new printer or did you perform a software update?
What firmware version are you running?

[territerriterri 10]

Hey @Whitespace, can you try doing a Factory Reset instead of Cura Connect reset and see if this helps? 

 

[Whitespace 2]

Yes this is on an S5. I haven’t had it for a while, so I don’t know how often it’s had this problem. I upgraded it to the latest firmware (5.5?) before the most recent 5.6.

 

It has had space issues since 5.5, but I can’t confirm it had it prior to that.

 

I will say that the network disconnection issues seem to be very related to the space issues.

 

Cura Connect reset works, but if this happens again I’ll try a factory reset.

 

Also I recently upgraded to 5.6 so I will see how that works.

 

Thanks all!

[jseriff 3]

I have this same issue on an S5.  The logs fill up with ~85MB over several days, filling the device entirely.  At that point, the S5 shows as being on the network, but is basically inoperable.

 

There are three ways I've found to get the printer back into an operable state:

 

1) Factory Reset

2) Cura Connect Reset

3) SSH into the printer, and clean the log files directory, then reboot the printer (via the CLI)

 

Once one of those three steps is taken, the printer will be operable again for between 2 and 10 days, at which point the cycle repeats.

 

The root cause appears to be network connectivity issues - the printer appears to have network connectivity issues when under load (the laptop next to the printer does not exhibit any connectivity issues, and the printer is showing full signal).  I see excessive avahi logs, as well as the printer's software having trouble getting firmware version information, or hitting its own API, causing it to log large python stack traces.

 

Long term, it would be great to get a more stable network connection, as this has other issues (Cura will frequently not be able to show the status of the print, despite the printer thinking it has a network connection - this will come and go as a print is operating).  Short term, monitoring the size of the log directory and removing old files, or modifying the rotation settings would be a great help.

[Whitespace 2]

Quick update: after upgrading to 5.6 I have not had any space or network connectivity issues.

[jseriff 3]

Unfortunately, I am on 5.7.2, and continue to see the issue.  I have not seem any changes in the issue in the last 12 months, and I generally upgrade to the latest firmware within a few weeks of its release.