Very bad bug in temperature reading

[jonnybischof 60]

Hi community

I've just discovered an awful bug that ruined several prints and part of my hotend:

If the thermocouple sensor is disconnected while the UM is running, then there is no error. The temperature reading just remains at it's last value.

If you ask "why the heck would you disconnect the sensor?!?": Disconnecting is what I did to simulate a broken wire (I suspected that would be the culprit, and I was right...).

So, my wire that runs in parallel to the Bowden tube is broken (aka slack joint). It has connection most of the time, but in a very certain position the connection breaks, and the friggin' temperature reading just freezes o.O.

Of course this happened while the heater was ON, meaning the heater was just kept on heating and heating until God knows what temperature and the printer didn't realize it because of the broken wire.

And I wondered why the thing started smoking and clogging up the nozzle all the time...

I left the printer running with the wire disconnected while I wrote this. It still shows 67°C.

This is so unacceptable it hurts. THIS CAN'T BE, DAMMIT! This is a safety relevant feature, THERE HAS TO BE AN ERROR MESSAGE and of course the heater needs to be switched off if there is no temperature reading. Who programmed this sh!-- ? That is so basic, I still can't believe it...

Where does this have to be reported to be fixed ASAP?

I'm using a custom firmware from Ginge's Marlinbuilder, build date december 20, 2013. I have the hex file around.

/edit:

After 10 more minutes the temperature now reads 70°C even though there was no sensor wire connected...

/edit:

Recreated the error multiple times now. It definetly does behave the way I described...

 

[Dim3nsioneer 557]

Just to confirm I got it right: the connection between amplifier board and UM shield is broken, not the connection between thermocouple and amplifier board?

 

[jonnybischof 60]

You're right. Sorry I wasn't clear on that...

 

[geeks 1]

I had the same thing!

I built a reliable connection with a new connector. Sometimes it pops! Since the temperature is never stable!

 

[Daid 305]

It's a bit of a hardware problem. As the ADC remains floating, and what happens differs from chip to chip in this case. Some chips keep exactly the same value, others drift up or down.

Now, I do know a software fix for this hardware problem. What you could do is every X samples, disable the ADC, set the pint as input, and enable the internal pull-up resistor for a fraction of the time. This would pull the floating line towards 5V which will cause a MAXTEMP on the next read.

I never got around to actually verify of the hardware can handle this and implement it in the firmware. (The temperature code is quite fragile) It is however, the last remaining bug in the temperature code. It can also only happen during printing, as if the wire is disconnected before printing, another failsafe will trigger.

Also, quite often, if the wire gets disconnected, you do get a MIN or MAX temp error due to noise you generate during the disconnect.

 

[geeks 1]

Ok

 

Me, I had imagined a simple idea.

 

If the temperature is adjusted to 200 ° C for example. Exceeding +5 ° C and -5 ° C for 1 minute is allowed. If this time is exceeded, stope printing and error trigger.

 

[Dim3nsioneer 557]

I would rather vote for a solution which physically stops the floating. As Daid has pointed out, the floating could be quite small and setting limits wouldn't trigger anything. On the other hand, floating could be large and then the hotend could get very hot within one minute (as it usually needs two minutes from room temperature to 200°C).

Certainly the issue has some seriosity as Jonny has clearly pointed out. I don't know about other RepRap electronics, but maybe other boards running with Marlin are not that different for the temperature sensing. So the issue could be far more widespread than 'just' within Ultimakers. Depending on your workload, Daid, it might be an idea to put that generally into the Marlin developer group?

 

[Daid 305]

It's a problem that only happens with the thermocouple circuit in the UM-Origonal. (The UM2 has a PT100 circuit which does not have this problem)

Another workaround would be to put a 20k resistor pullup on the board, but that might change your temperature readings.

 

[geeks 1]

I prefer 100 times with a secure pull-up resistor.

 

But will we have a patch that will make the choice in settings?

 

Without more schematic, which of the two pins is connected to the resistor?

 

[Dim3nsioneer 557]

Safety by hardware is always better than safety by software. It's e.g. higher rated in safety checks for a CE declaration.

The 20k pull-up resistor would mean changing the Marlin configuration tab; maybe extending it with a 'new' sensor type.The values themselves could be calculated with a small spice simulation.

It might get nasty if UM would decide to deliver future UM1 with such a hardware modification. A version check by the firmware must then be possible.

 

[Daid 305]

There are internal pull-ups in the AVR chip that can do this, but it requires quite a bit of patching and testing of the temperature code. So it would not require a board modification. But you can add a 100kOhm resistor on the board right now if you want. Shouldn't effect the temperature readout too much and would prevent this problem.

(And the next version of the UM-Origonal will most likely have the UM2 electronics with the PT100 circuit. As this also gets rid of the electronics fan and solves a few other issues)

 

[geeks 1]

Ok to add a 100K resistor.

 

UM2 on this modiffication is already present for the PT 100?

 

[Dim3nsioneer 557]

A PT100 doesn't need that modification. If the connection breaks, the electronics sees infinite resistance which results in the highest possible temperature. And this will certainly trigger an error message and a switch-off of the hotend.

 

[Dim3nsioneer 557]

[...]

(And the next version of the UM-Origonal will most likely have the UM2 electronics with the PT100 circuit. As this also gets rid of the electronics fan and solves a few other issues)

 

Spoiler alert!

 

[gr5 2,167]

Or go with a 1 megohm if you are worried. Or even 10 megohm should be enough to notice this problem.

johnny - for my UM Original, if that wire is loose, I get a max temp error instantly.

 

[Dim3nsioneer 557]

I made the test with my UM1 today and disconnected the cable comming from the UM arduino shield on the amplifier board. The result was pretty much the same as Jonny described it.

I get the max temp error if I disconnect a wire from the thermocouple on the amplifier board.

 

[Daid 305]

Spoiler alert!

 

With the UM2 board being so much better then the UM-Origonal board, it wouldn't be difficult to expect this somewhere sometime in the future. It reduces part count in our storage facility, and we'll have less failures (no more broken stepper drivers, no more broken electronics fans). It also adds a few extra features like software controlled current settings for the stepper drivers.

And, a currently unused, safety circuit that can be used for extra protection. There is also room on it for an analog Z endstop. So more room for tinkering there.

But, we're not there yet. It does require some changes in the kit here and there.

As to fix this bug in firmware, you would have to add 1 or more extra steps in this state machine:

https://github.com/ErikZalm/Marlin/blob/Marlin_v1/Marlin/temperature.cpp#L1104

 

[geeks 1]

A PT100 doesn't need that modification. If the connection breaks, the electronics sees infinite resistance which results in the highest possible temperature. And this will certainly trigger an error message and a switch-off of the hotend.

Ok :cool:

 

[jonnybischof 60]

Ok, you're talking of connectors snapping off. I had the issue that the cable actually broke (slack joint) which is not surprising because the cable moves all the time.

It would be a good idea to use a cable which is actually made for being flexed: high flex cable. I know, these cost about 5 times more than standard cables. But for a flexing cabling, you should use highly flexible cables (makes sense, no?).

I will switch out the cable myself, as I don't need most of the wires in the standard cable anyways and can save some weight there.

/edit:

Maybe, instead of using a hflex cable, it would already suffice to improve the strain relief and make it very rigid so that the parts where the sleeve is cut off can't flex at all. I believe this is where the slack joint is located, but I'm not sure. I'll add that to my list for the "UM black edition".

 

[luotinen 0]

Confirmed the fire hazard exists on two machines:

- Ultimaker1 v2

- Ultimaker1 v3

Steps to confirm if this danger influences your machine:

1. Set nozzle temperature

2. Pull out the sensor wire on top of the extruder block to simulate sensor malfunction

3. The machine should safety shutdown and display an error message, if it does not, kill the power and reconnect the sensor wire.

 

[gr5 2,167]

I wouldn't call it a "fire hazard" - more of a self destruct hazard.

UM has tested this machine by running full power through the head for a very long time - long past when it is "destroyed". The machine gets damaged of course but there isn't much of a fire hazard unless you put gasoline or other flammables on the printer (things that burn below 300C should definitely cause a problem):

http://www.engineeringtoolbox.com/fuels-ignition-temperatures-d_171.html

 

[anon4321 16]

If you are going to address this, You should also address this:

http://umforum.ultimaker.com/index.php?/topic/7070-feature-suggestion-nozzle-heater-runaway-protection/?p=65778

*******NOTE: if my analysis of the above code is correct. Pulling the temp sensor HIGH is BAD unless the code is fixed. UNLESS, the condition is detected elsewhere. Is there code that produces the MAX TEMP error and would it detect this? I think so...

However, I think for the most consistent and safest checking, I think the changes I suggested should be made. The architect in me says the temp checking should be handled in one place and should be defined as the temp being within the expected range within the expected time. This check would handle fail to 0 and fail to MAXTEMP. The only thing it doesn't handle is fail to continuous reporting of the current value which I believe should be handled by the pull up resistor fix.

 

[anon4321 16]

So on the hardware side, it looks like if you solder a 100K in the R23 (or R21 for dual extrusion) would fix the floating ADC issue. Correct?

 

[anon4321 16]

Jonny, regarding the slack joint (at the head end I assume),

***cough see second pic *** cough *** shameless plug ***

https://www.youmagine.com/designs/um-original-bowden-tube-clips-and-retainer

*** cough ***

 

[luotinen 0]

I just assumed it was burning as white smoke kept coming out.

Has anyone tried if installing a resistor fixes this issue?