Jump to content
Ultimaker Community of 3D Printing Experts

Cura not working - No space left on device


Recommended Posts

Posted · Cura not working - No space left on device

I cannot use my printer anymore (Extended 3)

 

On the log :

ultimakersystem-ccbdd300033b flipflop_wsgi.py[1314]: OSError: [Errno 28] No space left on device: '/var/lib/griffin/cluster/preferences.json.new'

 

If I connect in dev mode : 

/dev/mmcblk1p2    963584 828416         0 100% /
devtmpfs          481308      0    481308   0% /dev
tmpfs             515100      0    515100   0% /dev/shm
tmpfs             515100   7840    507260   2% /run
tmpfs               5120      0      5120   0% /run/lock
tmpfs             515100      0    515100   0% /sys/fs/cgroup
tmpfs             515100      0    515100   0% /var/lib/apt
tmpfs             515100      0    515100   0% /tmp
tmpfs             515100      0    515100   0% /var/cache/apt
/dev/mmcblk1p3   2775040  26632   2527224   2% /var/spool/cluster
 

Is there a way to clean the file system ?

 

Thanks,

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    The only idea I have left is that you check /var/log for large logfiles. I would do that at least on a normal Linux system.

    But I've never heard of the printer running out of disk space. Maybe someone else knows more, sorry!

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I deleted some of the logs and get back 100k...

     

    /var/log/journal is big, how to safely delete this journal ?

     

    size of main folders :

    526231    /
    215146    /usr
    168876    /var
    99980    /usr/lib
    91755    /var/lib
    87435    /usr/share
    83935    /var/lib/connman
    83910    /var/lib/connman/ethernet_ccbdd300033b_cable
    75247    /var/log
    74847    /var/log/journal
    74844    /var/log/journal/5e702514274833c0ebe690cc5ccc224d
    62364    /opt
    53676    /lib
    50743    /opt/qt
    46852    /usr/share/fonts
    42043    /opt/qt/lib
    39929    /usr/lib/python3
    39926    /usr/lib/python3/dist-packages
    37227    /usr/share/fonts/opentype
    37224    /usr/share/fonts/opentype/noto
    27626    /usr/lib/arm-linux-gnueabihf
    26975    /lib/modules
    26971    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede
    25823    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede/kernel
    23787    /usr/bin
    21829    /usr/lib/python3.4
    14635    /lib/modules/4.14.32-ultimaker-00309-geaace6d4aede/kernel/drivers
    13239    /usr/lib/python3/dist-packages/numpy
    12523    /usr/share/i18n
    11618    /opt/pyqt
    11568    /lib/udev
    11428    /opt/pyqt/PyQt5
    9486    /usr/share/i18n/locales
    8704    /lib/arm-linux-gnueabihf
    8289    /usr/share/okuda
    8036    /usr/lib/python3/dist-packages/cluster
    7911    /usr/lib/python3/dist-packages/cluster/app
    7840    /run
    7412    /usr/share/griffin
    7267    /usr/lib/python3/dist-packages/sqlalchemy
    7237    /usr/share/okuda/resources
    6606    /usr/lib/python3/dist-packages/cluster/app/static
    6440    /run/log/journal/5e702514274833c0ebe690cc5ccc224d
    6440    /run/log/journal
    6440    /run/log

     

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    You mean the directory /var/log/journal ?

    It should be safe to delete log files inside the directory, but keep the directory itself. 

     

    Are there many log files inside? Because normally there is log rotation, maybe not working or configured on the printer.

    If there is only one big logfile, you can also truncate the file with just

    > logfilename

    instead of delete them. The > is important which means you redirect the input, which is nothing to the file, resulting in a 0 byte file. But just a delete should also be no problem.

     

    But I am not sure if this is enough, the directory has 74MB. Maybe there is something else which consumes so much disk space.  

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I removed the folder in /var/log/journal and get back 58Mb 😉

    It seems it was logs not properly deleted in the past by the software (probably due to bug in old releases)

     

    now, only

    -rw-r-----  1 root systemd-journal 8388608 Dec 15 07:40 system.journal
    -rw-r-----+ 1 root systemd-journal 8388608 Dec 15 07:38 user-1000.journal
     

    I'll monitor to see if this folder size continue to increase or not...

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I have the same problem.  I suspect it's avahi-daemon logging too much (see below):

     

    - I was printing fine

    - after some number of prints Cura Connect would break

    - reading logs on the touchscreen showed that no space was left on /

    - couldn't enable dev mode to ssh in

    - would reset cura connect — losing all my stats — printer would work again for a couple of days, then run out of space again

    - this time though I am able to SSH in and can verify that /var/log/journal/** is taking up 89MB of 806 available on /

     

    root@ultimakersystem-ccbdd30087d3:/var/log/journal/c9c140d1e794462fbc6facaa6f36863f# ls -l
    total 90252
    -rw-r-----  1 root systemd-journal   53248 Jan  1  1970 system.journal
    -rw-r-----  1 root systemd-journal 8388608 Jan  1  1970 system@00000008ccb660c6-1b488519ed25a7ee.journal~
    -rw-r-----  1 root systemd-journal 8388608 May 21 17:32 system@8a780352b4d74a25abba2411be124d36-000000000001d96a-0005a62956cc9fe7.journal
    -rw-r-----  1 root systemd-journal 8388608 May 21 21:06 system@8a780352b4d74a25abba2411be124d36-0000000000020038-0005a62be43bd14a.journal
    -rw-r-----  1 root systemd-journal 8388608 May 22 00:00 system@8a780352b4d74a25abba2411be124d36-000000000002253e-0005a62ee1b94778.journal
    -rw-r-----  1 root systemd-journal 8388608 May 22 00:32 system@8a780352b4d74a25abba2411be124d36-0000000000025072-0005a631525ec98c.journal
    -rw-r-----  1 root systemd-journal 8388608 May 22 01:04 system@8a780352b4d74a25abba2411be124d36-0000000000027c8b-0005a631c3fec32d.journal
    -rw-r-----  1 root systemd-journal 8388608 May 20 23:35 system@ebae5b2b5a4646dabb9d3c59a8d6fe95-0000000000018b9f-0005a61aead30f5f.journal
    -rw-r-----  1 root systemd-journal 8388608 May 21 03:07 system@ebae5b2b5a4646dabb9d3c59a8d6fe95-000000000001b68c-0005a61cd98f111f.journal
    -rw-r-----+ 1 root systemd-journal 8388608 May 22 01:08 user-1000.journal
    -rw-r-----+ 1 root systemd-journal 8388608 May 22 01:04 user-1000@6376e02ffff348eea29dbb59865292ca-0000000000000000-0000000000000000.journal
    -rw-r-----+ 1 root systemd-journal 8388608 May 21 03:07 user-1000@6376e02ffff348eea29dbb59865292ca-000000000001d767-0005a61fba8f8837.journal

     

    Tailing logs with journalctl -r, I see that avahi-daemon is spamming useless logs to disk:

    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.137.
    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.32.
    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::4da:c6cf:bdab:6026.
    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::21:efd5:4317:132e.
    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host fe80::4aa6:b8ff:fe83:bbfc.
    Jun 01 21:54:15 ultimakersystem-ccbdd30087d3 avahi-daemon[1214]: Invalid response packet from host 192.168.1.173.

    Also other systems are logging too much to STDOUT.

  • Link to post
    Share on other sites
    Posted (edited) · Cura not working - No space left on device

    For others who want to debug here are some commands you might find useful:

    df -h                      # shows disk usage in human-readable terms
    journalctl --disk-usage    # shows how much disk space logs are taking
    ls -l /var/log/journal/**/ # list all journald logs
    rm /var/log/journal/**/*@* # delete all archived logs (logs containing an @ symbol)

    Removing those logs saved me a bunch of space.  I also edited /etc/systemd/journald.conf and halved SystemMaxUse to 32M, but that didn't seem to really work.

    Edited by Whitespace
    typo
    • Like 1
    Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I suspect that avahi-daemon is logging too much.  The installed version is 0.6.31, and the very next version — 0.6.32 — fixes these warnings from appearing in logs: https://github.com/lathiat/avahi/blob/master/docs/NEWS#L155

     

    The installed version is over four years old :(

     

    I can't install it via apt-get, so I disabled avahi-daemon by modifying /etc/default/avahi-daemon and stopping it with update-rc.d avahi-daemon stop

    • Like 1
    Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    @Whitespace what kind of printer do you have?  Your profile says you have only an S5?  Is this on your S5?

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    @CarloK - any ideas about this?

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I'm currently working on another project, not on the S3/S5 so anything I mention here is from own experience.

     

    The bad part is, we are using old software. The good part is, we are using old software.

    Many bugs have been solved since the original release and I'd like to see many of our libraries updated to newer versions. The good part is that the software is working on thousands of printers with relative few problems and most of the problems are perhaps annoying, but at least known. 

     

    @Whitespace what changed in your setup for these errors to start occuring? Is it a new printer or did you perform a software update?
    What firmware version are you running?

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    Hey @Whitespace, can you try doing a Factory Reset instead of Cura Connect reset and see if this helps? 

     

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    Yes this is on an S5. I haven’t had it for a while, so I don’t know how often it’s had this problem. I upgraded it to the latest firmware (5.5?) before the most recent 5.6.

     

    It has had space issues since 5.5, but I can’t confirm it had it prior to that.

     

    I will say that the network disconnection issues seem to be very related to the space issues.

     

    Cura Connect reset works, but if this happens again I’ll try a factory reset.

     

    Also I recently upgraded to 5.6 so I will see how that works.

     

    Thanks all!

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    I have this same issue on an S5.  The logs fill up with ~85MB over several days, filling the device entirely.  At that point, the S5 shows as being on the network, but is basically inoperable.

     

    There are three ways I've found to get the printer back into an operable state:

     

    1) Factory Reset

    2) Cura Connect Reset

    3) SSH into the printer, and clean the log files directory, then reboot the printer (via the CLI)

     

    Once one of those three steps is taken, the printer will be operable again for between 2 and 10 days, at which point the cycle repeats.

     

    The root cause appears to be network connectivity issues - the printer appears to have network connectivity issues when under load (the laptop next to the printer does not exhibit any connectivity issues, and the printer is showing full signal).  I see excessive avahi logs, as well as the printer's software having trouble getting firmware version information, or hitting its own API, causing it to log large python stack traces.

     

    Long term, it would be great to get a more stable network connection, as this has other issues (Cura will frequently not be able to show the status of the print, despite the printer thinking it has a network connection - this will come and go as a print is operating).  Short term, monitoring the size of the log directory and removing old files, or modifying the rotation settings would be a great help.

    • Like 3
    Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    Quick update: after upgrading to 5.6 I have not had any space or network connectivity issues.

  • Link to post
    Share on other sites
    Posted · Cura not working - No space left on device

    Unfortunately, I am on 5.7.2, and continue to see the issue.  I have not seem any changes in the issue in the last 12 months, and I generally upgrade to the latest firmware within a few weeks of its release.

  • Link to post
    Share on other sites

    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now
    ×
    ×
    • Create New...