Printcore in head slot X is taking too long to warm up - Ultimainboard reboot happening in the background

[jimre 0]

I would like to greet everyone in the forum! This is my first post.

 

Our UM3 printer is struggling with the issue mentioned in the topic. (It was in service, the Ultimainboard was already replaced, but it didn't solved the problem.) The printing stops randomly after 12-13 hours printing time. The error message is misleading as there is no problem with the printhead. The problem is in the background that the +24V power line is switched off.

 

I've put the printer in developer mode, so I could log in and check the logs while the printer is working and even when it is locked and the LCD screen shows the error message. I see this in the log:

Jan 31 04:48:04 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:04,353 INFO     transportLayer  BUILD:Jan 27 2019 22:52:07
Jan 31 04:48:04 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:04,365 INFO     transportLayer  Detected board with id #3
Jan 31 04:48:04 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:04,373 INFO     transportLayer  ADS1015 #0x48 ref.voltage=958
Jan 31 04:48:04 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:04,376 INFO     transportLayer  Marlin started
Jan 31 04:48:24 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:24,399 WARNING  transportLayer  ProtoError:Sequence number is unexpected (receive
d, expected): 73,0
Jan 31 04:48:24 ultimakersystem-ccbdd3007324 python3[931]: 2021-01-31 04:48:24,400 WARNING  transportLayer  Protocol error from Marlin received, send buffer:
 [MessageTuple(sequence_number=73, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ecae618>, in_planner
_buffer=False), MessageTuple(sequence_number=74, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x82b7d46
8>, in_planner_buffer=False), MessageTuple(sequence_number=75, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda
> at 0x82b7ddb0>, in_planner_buffer=False), MessageTuple(sequence_number=76, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<l
ocals>.<lambda> at 0x82b7d618>, in_planner_buffer=False), MessageTuple(sequence_number=77, message='M105', callback_function=<function ApplicationLayer.__get
IdleCommand.<locals>.<lambda> at 0x82b7d3d8>, in_planner_buffer=False)]

 

For me it looks like that the Atmel microcontroller on the Ultimainboard rebooted. The firmware version and the "Marlin started" message indicates that. The next message may be related to this as the Ultimainboard waits for sequence '0' (as it just started) but it gets '73'. The protocol error is probably also related to this that as it might continue in the middle of a GCODE transmission.

 

According to my knowledge when the Atmel2560 (re)starts every pin is set to input. Which is probably also true to the pin controlling the relay switching the +24V supply for the heaters and the stepper motors. So the power for the stepper motors and the heating is switched off by the "reboot". It won't enable them again, only recognize that the temperature of the head drops below the critical limit and I see many "cold extrusion prevented" messages in the log and after a while "waitForHeatableObjectsStep Waiting for hotend-1: 31.0/210.0 °C (wait_tolerance: 0.0, progress: 0.00)" messages are logged until the the Linux rebooted or the griffin service restarted. At that point the LCD displays the error and the printer's user interface stopped. Meanwhile if I ssh into the printer I can send GCODE commands like switch on/off the fan(M106), switch on/off the +24V relay(M80/M81). After switching the +24V back I can home the head (G28 X/Y/Z). There was no response to the temperature commands (M104/109 or M140/190) until I haven't restarted the griffin.printer.service on the Linux. After that the bed temperature can be set and it heated up the bed. If I reboot the Linux everything goes back to normal.

 

For me the background of the original error message is clear, but the reason is completely not.

 

The printer uses the latest 2.1.4 version of the Ultimainboard and the latest version of the firmware (both on Linux and on the Atmel part). I don't think that there is a power interruption as the Olimex board always worked continuously, so the +5V voltage which is needed for both is present. I did not observed any interruption in the SSH connection. The printer stops suddenly during the normal printing.

 

Last time when this error appeared I dismantled the electronics. Resoldered the Atmel on the board and the connector to the printhead, cleaned the contacts on the board and on the head, then put everything back together. It worked without interruption nearly in 7x24 for more than two weeks. Before that "repair" it stopped in a day four times in a row. Around Christmas it was good for two weeks without interruption, but that time I've printed short jobs. 

 

Do you have any idea why the Atmel board reboots? Some external reset or a watchdog reset?

 

Any help or suggestion greatly appreciated.

 

Best regards, Imre

 

[ndu 0]

Did you manage to solve this issue?

Our Atmel board also started rebooting during a print.

 

I also put the printer into dev mode to get logs, and as you see below it seems like the same issue. However, I never got the error "Printcore in head slot X is taking too long to warm up", as you did.

The error occurred in a PETG print, with relatively high temps. I was thinking it might be the powersupply having issues, so I will test with lower bed temp and lower print core temp, to see if I still get the error.

Snippet from logs:

Feb 16 08:25:42 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:25:42,221 INFO     timeEstimator   getTimeRemaining() total_time 20:20:12 scaling_factor 0.9526796062937224 compensated_total_time 19:22:27.579336 time remaining 18:02:26.854937 pure_cura_time_remaining 19:00:11.275388
Feb 16 08:26:44 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:26:44,481 INFO     transportLayer  BUILD:Jan 27 2019 22:52:07
Feb 16 08:26:44 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:26:44,486 INFO     transportLayer  Detected board with id #2
Feb 16 08:26:44 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:26:44,492 INFO     transportLayer  ADS1015 #0x48 ref.voltage=958
Feb 16 08:26:44 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:26:44,495 INFO     transportLayer  Marlin started
Feb 16 08:27:04 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:04,508 WARNING  transportLayer  ProtoError:Sequence number is unexpected (received, expected): 189,0
Feb 16 08:27:04 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:04,510 WARNING  transportLayer  Protocol error from Marlin received, send buffer: [MessageTuple(sequence_number=189, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12d68>, in_planner_buffer=False), MessageTuple(sequence_number=190, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e88>, in_planner_buffer=False), MessageTuple(sequence_number=191, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12db0>, in_planner_buffer=False), MessageTuple(sequence_number=192, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e40>, in_planner_buffer=False), MessageTuple(sequence_number=193, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12930>, in_planner_buffer=False)]
Feb 16 08:27:04 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:04,617 WARNING  transportLayer  Resend data request for sequence 0, but not found, before adjusting: [MessageTuple(sequence_number=189, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12d68>, in_planner_buffer=False), MessageTuple(sequence_number=190, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e88>, in_planner_buffer=False), MessageTuple(sequence_number=191, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12db0>, in_planner_buffer=False), MessageTuple(sequence_number=192, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e40>, in_planner_buffer=False), MessageTuple(sequence_number=193, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12930>, in_planner_buffer=False)]
Feb 16 08:27:04 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:04,619 WARNING  transportLayer  Resend data request for sequence 0, but not found, after adjusting: [MessageTuple(sequence_number=0, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12d68>, in_planner_buffer=False), MessageTuple(sequence_number=1, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e88>, in_planner_buffer=False), MessageTuple(sequence_number=2, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12db0>, in_planner_buffer=False), MessageTuple(sequence_number=3, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12e40>, in_planner_buffer=False), MessageTuple(sequence_number=4, message='M105', callback_function=<function ApplicationLayer.__getIdleCommand.<locals>.<lambda> at 0x9ed12930>, in_planner_buffer=False)]
Feb 16 08:27:04 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:04,737 INFO     transportLayer   too long extrusion prevented
Feb 16 08:27:46 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:46,092 INFO     transportLayer   cold extrusion prevented
Feb 16 08:27:46 ultimakersystem-ccbdd30032e5 python3[874]: 2021-02-16 08:27:46,257 INFO     transportLayer   cold extrusion prevented