Ultimaker split screen firmware bug

[chase_c 0]

I know this topic has been discussed for over a year now and no firmware fix has been released. This thread was started to see if the community can come together and find the root cause of the issue.

 

Issues that have been observed (only when wifi is active):

• When the machine is powered on, eventually the screen will split
  • When the screen split is present, I get random blobs in my prints.
  • I cannot connect to Cura and start/manage prints from my computer

 

Here is a list of things that I have done so far:

• Contacted Ultimaker NA Zendesk support
  • Support told me this is not a firmware issue, but instead is either the wifi card or Olimex board.
    • Both were replaced with brand new parts and the issue persists
  • Reverted to old firmware 4.3.3, and on this specific firmware
    • I do not witness any of the screen splitting issues
    • However, my automatic bed leveling is not working correctly
  • Support told me it was must be "primarily" a network issue.
    • My 2.4GHz SSID is separate from my 5GHz SSID
    • SSID does not contain special characters
    • Multiple 2.4GHz channels have been tested with identical results (split screen)
• Contacted the distributor where I purchased my machine
  • Distributor worked with Ultimaker to get me another Olimex and wifi board

 

Observations supporting this being a firmware issue:

• Most importantly, Ultimaker staff has confirmed this is a firmware artifact/bug and that they were supposedly working on a fix over a year ago (May 1, 2019).
  • Ultimaker 3 display broken after firmware upgrade
• I see no screen split issues on 4.3.3.
• All hardware that could cause this issue has been replaced with brand new parts on my machine, but the issue persists only on newer firmware revisions.
• My guess is that this may be an issue where the newer firmware exposes wifi networking edge cases.
  • These issues are not present ever over LAN, but I do not have the luxury of running ethernet cables across my house.

 

Some people may consider the older firmware to be an acceptable workaround, but I do not. These devices were sold as machines that would get frequent live updates in the form of improved functionality and features. Now a firmware bug is causing my device to be unreliable, at best. The only reason I am resorting to asking for community help in root causing this issue is that I have no other reasonable recourse. Ultimaker support is not being helpful at all, and I've not been asked once to supply any logs or assist with troubleshooting the issue.

 

If there is nothing physically wrong with a machine, customers should not be expected to just deal with having to pick and choose features for over a year. This needs to be fixed.

Edited June 29, 2020 by chase_c

[ahoeben 1,952]

I realize you probably want this issue fixed instead of a workaround, but if Wifi seems to be the cause of the problem and you can't run wires, then you might try using a simple access point to do the wifi-ing and connect a short ethernet cable between the access point and the printer.

[Szahari 30]

In the investigation of the issue, its been noticed that the following lines are always found in the log files UM3 printers that are affected by the shifted / split screen:

 

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel:

Modules linked in: nls_iso8859_1 nls_cp437 vfat fat lm75 af_packet arc4 gpio_pca953x ath9k_htc uvcvideo ath9k_common videobuf2_vmalloc leds_pca963x videobuf2_memops ath9k_hw videobuf2_v4l2 ath videobuf2_core uas mac80211 videodev evdev media cfg80211 rc_cec sun4i_backend sun4i_ts axp20x_usb_power gpio_axp209 axp20x_pek axp20x_adc industrialio at24 sun4i_gpadc nvmem_sunxi_sid sun4i_lradc_keys sunxi_wdt pwm_sun4i ahci_sunxi libahci_platform libahci libata sun4i_ss sun4i_drm_hdmi cec des_generic sun4i_hdmi_i2c rc_core spi_sun4i sun4i_tcon sun4i_drm drm_kms_helper cfbfillrect cfbimgblt cfbcopyarea rotary_encoder drm pwm_beeper cpufreq_dt thermal_sys hwmon i2c_dev rfkill_gpio rfkill sha1_arm_neon sha1_arm sha1_generic ipv6 ssd1307fb syscopyarea sysfillrect sysimgblt fb_sys_fops [last unloaded: gpio_pca953x]

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: CPU: 0 PID: 7 Comm: ksoftirqd/0 Tainted: G        W       4.14.32-ultimaker-00309-geaace6d4aede #3

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: Hardware name: Allwinner sun7i (A20) Family

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b01102bc>] (unwind_backtrace) from [<b010c17c>] (show_stack+0x10/0x14)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b010c17c>] (show_stack) from [<b05e54bc>] (dump_stack+0x84/0x98)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b05e54bc>] (dump_stack) from [<b012beb0>] (__warn+0xe8/0x100)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b012beb0>] (__warn) from [<b012bf30>] (warn_slowpath_null+0x20/0x28)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b012bf30>] (warn_slowpath_null) from [<af43cc88>] (ieee80211_rx_napi+0x848/0x9ac [mac80211])

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<af43cc88>] (ieee80211_rx_napi [mac80211]) from [<af574370>] (ath9k_rx_tasklet+0x13c/0x17c [ath9k_htc])

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<af574370>] (ath9k_rx_tasklet [ath9k_htc]) from [<b0130ca8>] (tasklet_action+0x74/0x110)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0130ca8>] (tasklet_action) from [<b0101644>] (__do_softirq+0xfc/0x224)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0101644>] (__do_softirq) from [<b0130dd4>] (run_ksoftirqd+0x38/0x50)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0130dd4>] (run_ksoftirqd) from [<b014c468>] (smpboot_thread_fn+0x11c/0x198)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b014c468>] (smpboot_thread_fn) from [<b01487d0>] (kthread+0x124/0x154)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b01487d0>] (kthread) from [<b0108208>] (ret_from_fork+0x14/0x2c)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: ---[ end trace a088063bd8ec7c9d ]---

Jun 12 19:44:59 ultimakersystem-ccbdd300169b connmand[259]: wlan0 {newlink} index 3 address 00:0E:8E:72:24:A1 mtu 1500

Jun 12 19:44:59 ultimakersystem-ccbdd300169b connmand[259]: wlan0 {newlink} index 3 operstate 2 <DOWN>

 

Maybe someone in the community understands a bit more whats happening there?

All evidence we have supports the fact that a local network configuration setting is causing a corruption in the data. Because of htat,tTogether with the logs, any and all information about your local network configuration may be helpful in the investigation of this issue:

- is it home or work network

- any peculiarities that stand out for your network? Many devices connected?

- Are you using a wifi repeater?

[chase_c 0]

 

2 hours ago, Szahari said:

In the investigation of the issue, its been noticed that the following lines are always found in the log files UM3 printers that are affected by the shifted / split screen:

 

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel:

Modules linked in: nls_iso8859_1 nls_cp437 vfat fat lm75 af_packet arc4 gpio_pca953x ath9k_htc uvcvideo ath9k_common videobuf2_vmalloc leds_pca963x videobuf2_memops ath9k_hw videobuf2_v4l2 ath videobuf2_core uas mac80211 videodev evdev media cfg80211 rc_cec sun4i_backend sun4i_ts axp20x_usb_power gpio_axp209 axp20x_pek axp20x_adc industrialio at24 sun4i_gpadc nvmem_sunxi_sid sun4i_lradc_keys sunxi_wdt pwm_sun4i ahci_sunxi libahci_platform libahci libata sun4i_ss sun4i_drm_hdmi cec des_generic sun4i_hdmi_i2c rc_core spi_sun4i sun4i_tcon sun4i_drm drm_kms_helper cfbfillrect cfbimgblt cfbcopyarea rotary_encoder drm pwm_beeper cpufreq_dt thermal_sys hwmon i2c_dev rfkill_gpio rfkill sha1_arm_neon sha1_arm sha1_generic ipv6 ssd1307fb syscopyarea sysfillrect sysimgblt fb_sys_fops [last unloaded: gpio_pca953x]

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: CPU: 0 PID: 7 Comm: ksoftirqd/0 Tainted: G        W       4.14.32-ultimaker-00309-geaace6d4aede #3

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: Hardware name: Allwinner sun7i (A20) Family

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b01102bc>] (unwind_backtrace) from [<b010c17c>] (show_stack+0x10/0x14)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b010c17c>] (show_stack) from [<b05e54bc>] (dump_stack+0x84/0x98)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b05e54bc>] (dump_stack) from [<b012beb0>] (__warn+0xe8/0x100)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b012beb0>] (__warn) from [<b012bf30>] (warn_slowpath_null+0x20/0x28)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b012bf30>] (warn_slowpath_null) from [<af43cc88>] (ieee80211_rx_napi+0x848/0x9ac [mac80211])

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<af43cc88>] (ieee80211_rx_napi [mac80211]) from [<af574370>] (ath9k_rx_tasklet+0x13c/0x17c [ath9k_htc])

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<af574370>] (ath9k_rx_tasklet [ath9k_htc]) from [<b0130ca8>] (tasklet_action+0x74/0x110)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0130ca8>] (tasklet_action) from [<b0101644>] (__do_softirq+0xfc/0x224)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0101644>] (__do_softirq) from [<b0130dd4>] (run_ksoftirqd+0x38/0x50)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b0130dd4>] (run_ksoftirqd) from [<b014c468>] (smpboot_thread_fn+0x11c/0x198)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b014c468>] (smpboot_thread_fn) from [<b01487d0>] (kthread+0x124/0x154)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: [<b01487d0>] (kthread) from [<b0108208>] (ret_from_fork+0x14/0x2c)

Jun 12 19:44:59 ultimakersystem-ccbdd300169b kernel: ---[ end trace a088063bd8ec7c9d ]---

Jun 12 19:44:59 ultimakersystem-ccbdd300169b connmand[259]: wlan0 {newlink} index 3 address 00:0E:8E:72:24:A1 mtu 1500

Jun 12 19:44:59 ultimakersystem-ccbdd300169b connmand[259]: wlan0 {newlink} index 3 operstate 2 <DOWN>

 

Maybe someone in the community understands a bit more whats happening there?

All evidence we have supports the fact that a local network configuration setting is causing a corruption in the data. Because of htat,tTogether with the logs, any and all information about your local network configuration may be helpful in the investigation of this issue:

- is it home or work network

- any peculiarities that stand out for your network? Many devices connected?

- Are you using a wifi repeater?

 

 

Looks like a partial stack trace to me. I would like to see where those errors are logged in the source. Since the wifi module is an off-the-shelf part, it should be easy enough to look into the operation state being set to dig further into the issue.

[chase_c 0]

5 hours ago, ahoeben said:

I realize you probably want this issue fixed instead of a workaround, but if Wifi seems to be the cause of the problem and you can't run wires, then you might try using a simple access point to do the wifi-ing and connect a short ethernet cable between the access point and the printer.

 

I appreciate the suggestion, but a workaround is just a non-starter. I didn't pay partial price for the printer, so I don't want partial features. The wifi functionality was a key feature that convinced me to purchase this specific machine. This issue has been known for over a year, and it would affect me even if I purchased the system yesterday because of some random network setting that nobody can pinpoint.

 

This wasn't a $200 printer. I paid ~$4500 out of my own pocket for home use and I expect it to function as it was advertised. Not to mention, with this bug they have tanked my resell value because I cannot guarantee other people won't face the same issue.

[chase_c 0]

@Szahari My general network details are:

• Netgear R8500
• Separate SSID for 2.4GHz and 5GHz
• SSID has no special characters or spaces
• Tried multiple channels for 2.4GHz
• Very few devices connected to 2.4GHz
  • Only 1 or 2 devices
• 2.4GHz set to 1Gbps mode

If you have any other specific questions I'm happy to answer them.

[ahoeben 1,952]

2 hours ago, chase_c said:

I appreciate the suggestion, but a workaround is just a non-starter

 

Hey, you are asking the community, so you get a community-answer... The community cannot fix the firmware for you.

[chase_c 0]

I didn't mean to hurt your feelings, but I hope you don't speak for the community as a whole. That's not a great way to invite new people to the forums. My only suggestion is to be proactive in looking for a better solution than bypass the wifi feature completely.

 

It isn't outside of the realm of possibility for the community to come together and troubleshoot until we pinpoint the issue. As I've said before, it seems highly possible that it's just a network setting on some routers that the newer firmware revisions are not liking. This issue is not present on firmware version 4.3.3.

 

The "fix" may not require an official firmware release to make it function correctly, although that would be preferable.

 

 

[ahoeben 1,952]

Ok, I don’t have an UM3, and you are asking for something I cannot provide. So there is nothing I can do for you then.

[chase_c 0]

No problem! I appreciate your responses. :)

[gr5 2,178]

And I don't use wifi on my UM3.  Your community keeps shrinking.  Sorry! 

 

I know from experience that the firmware is complicated - or more accurately - there's a lot there.  An overwhelming amount of code.  And this issue looks like a difficult one to fix - I'm guessing some kind of memory overflow/stack overflow versus a simple bug.  Memory issues are really hard to track down.  And usually not easily repeatable.

 

The only thing I might try if I were you is replacing conman with the version they put in github.  Or trying to get the latest version for the installed operating system.  I don't know how to do that exactly (apt get??).  But a lot of googling and a lot of reading might help you update conman to the latest version meant for this particular flavor of linux.

 

In another thread on this forum, recently (in the year 2020), is a discussion of updating certain aspects of linux on the UM3 (or possibly S3 or S5 - it's all the same).  The guy updated some parts of linux and it helped some issue he had. Probably network related.  So you might want to do a google search for that thread.  Definitely use google with "site:ultimaker.com" as part of the search and restrict to posts in the last 6 months.  That guy seemed quite knowledgeable and what he said would give you a huge start on how to update the linux on your um3.

[gr5 2,178]

By the way - if you are going to be doing all this you need a recovery plan in case you brick linux.  You can do this with a micro SD card install.  Your local reseller can help you with that.  Or you can actually connect directly to the linux box with a serial cable.  That way if you break something that keeps the install from getting past the ssh daemon you can still connect to your machine.  I sell the cables here.  Keep in mind that this is kind of advanced stuff if you are new to linux and serial terminals and such.  The microSD card is a simpler solution but it resets everything back to factory install when you might just need to delete a single semicolon or something:

 

https://thegr5store.com/store/index.php/catalogsearch/result/?q=olimex

[chase_c 0]

Thanks @gr5 but I've been out of town for the last week. I'm hoping to dig into this in the next week or two. Finding time to debug this is a bit of a challenge. :)