Jump to content
Ultimaker Community of 3D Printing Experts
Sign in to follow this  
jonnybischof

Very bad bug in temperature reading

Recommended Posts

Hi community

I've just discovered an awful bug that ruined several prints and part of my hotend:

If the thermocouple sensor is disconnected while the UM is running, then there is no error. The temperature reading just remains at it's last value.

If you ask "why the heck would you disconnect the sensor?!?": Disconnecting is what I did to simulate a broken wire (I suspected that would be the culprit, and I was right...).

So, my wire that runs in parallel to the Bowden tube is broken (aka slack joint). It has connection most of the time, but in a very certain position the connection breaks, and the friggin' temperature reading just freezes o.O.

Of course this happened while the heater was ON, meaning the heater was just kept on heating and heating until God knows what temperature and the printer didn't realize it because of the broken wire.

And I wondered why the thing started smoking and clogging up the nozzle all the time...

I left the printer running with the wire disconnected while I wrote this. It still shows 67°C.

This is so unacceptable it hurts. THIS CAN'T BE, DAMMIT! This is a safety relevant feature, THERE HAS TO BE AN ERROR MESSAGE and of course the heater needs to be switched off if there is no temperature reading. Who programmed this sh!-- ? That is so basic, I still can't believe it...

Where does this have to be reported to be fixed ASAP?

I'm using a custom firmware from Ginge's Marlinbuilder, build date december 20, 2013. I have the hex file around.

/edit:

After 10 more minutes the temperature now reads 70°C even though there was no sensor wire connected...

/edit:

Recreated the error multiple times now. It definetly does behave the way I described...

 

Share this post


Link to post
Share on other sites

It's a bit of a hardware problem. As the ADC remains floating, and what happens differs from chip to chip in this case. Some chips keep exactly the same value, others drift up or down.

Now, I do know a software fix for this hardware problem. What you could do is every X samples, disable the ADC, set the pint as input, and enable the internal pull-up resistor for a fraction of the time. This would pull the floating line towards 5V which will cause a MAXTEMP on the next read.

I never got around to actually verify of the hardware can handle this and implement it in the firmware. (The temperature code is quite fragile) It is however, the last remaining bug in the temperature code. It can also only happen during printing, as if the wire is disconnected before printing, another failsafe will trigger.

Also, quite often, if the wire gets disconnected, you do get a MIN or MAX temp error due to noise you generate during the disconnect.

 

Share this post


Link to post
Share on other sites

I would rather vote for a solution which physically stops the floating. As Daid has pointed out, the floating could be quite small and setting limits wouldn't trigger anything. On the other hand, floating could be large and then the hotend could get very hot within one minute (as it usually needs two minutes from room temperature to 200°C).

Certainly the issue has some seriosity as Jonny has clearly pointed out. I don't know about other RepRap electronics, but maybe other boards running with Marlin are not that different for the temperature sensing. So the issue could be far more widespread than 'just' within Ultimakers. Depending on your workload, Daid, it might be an idea to put that generally into the Marlin developer group?

 

Share this post


Link to post
Share on other sites

It's a problem that only happens with the thermocouple circuit in the UM-Origonal. (The UM2 has a PT100 circuit which does not have this problem)

Another workaround would be to put a 20k resistor pullup on the board, but that might change your temperature readings.

 

Share this post


Link to post
Share on other sites

Safety by hardware is always better than safety by software. It's e.g. higher rated in safety checks for a CE declaration.

The 20k pull-up resistor would mean changing the Marlin configuration tab; maybe extending it with a 'new' sensor type.The values themselves could be calculated with a small spice simulation.

It might get nasty if UM would decide to deliver future UM1 with such a hardware modification. A version check by the firmware must then be possible.

 

Share this post


Link to post
Share on other sites

There are internal pull-ups in the AVR chip that can do this, but it requires quite a bit of patching and testing of the temperature code. So it would not require a board modification. But you can add a 100kOhm resistor on the board right now if you want. Shouldn't effect the temperature readout too much and would prevent this problem.

(And the next version of the UM-Origonal will most likely have the UM2 electronics with the PT100 circuit. As this also gets rid of the electronics fan and solves a few other issues)

 

Share this post


Link to post
Share on other sites

I made the test with my UM1 today and disconnected the cable comming from the UM arduino shield on the amplifier board. The result was pretty much the same as Jonny described it.

I get the max temp error if I disconnect a wire from the thermocouple on the amplifier board.

 

Share this post


Link to post
Share on other sites

Spoiler alert! :D

 

With the UM2 board being so much better then the UM-Origonal board, it wouldn't be difficult to expect this somewhere sometime in the future. It reduces part count in our storage facility, and we'll have less failures (no more broken stepper drivers, no more broken electronics fans). It also adds a few extra features like software controlled current settings for the stepper drivers.

And, a currently unused, safety circuit that can be used for extra protection. There is also room on it for an analog Z endstop. So more room for tinkering there.

But, we're not there yet. It does require some changes in the kit here and there.

As to fix this bug in firmware, you would have to add 1 or more extra steps in this state machine:

https://github.com/ErikZalm/Marlin/blob/Marlin_v1/Marlin/temperature.cpp#L1104

 

Share this post


Link to post
Share on other sites

Ok, you're talking of connectors snapping off. I had the issue that the cable actually broke (slack joint) which is not surprising because the cable moves all the time.

It would be a good idea to use a cable which is actually made for being flexed: high flex cable. I know, these cost about 5 times more than standard cables. But for a flexing cabling, you should use highly flexible cables (makes sense, no?).

I will switch out the cable myself, as I don't need most of the wires in the standard cable anyways and can save some weight there.

/edit:

Maybe, instead of using a hflex cable, it would already suffice to improve the strain relief and make it very rigid so that the parts where the sleeve is cut off can't flex at all. I believe this is where the slack joint is located, but I'm not sure. I'll add that to my list for the "UM black edition".

 

Share this post


Link to post
Share on other sites

Confirmed the fire hazard exists on two machines:

- Ultimaker1 v2

- Ultimaker1 v3

Steps to confirm if this danger influences your machine:

1. Set nozzle temperature

2. Pull out the sensor wire on top of the extruder block to simulate sensor malfunction

3. The machine should safety shutdown and display an error message, if it does not, kill the power and reconnect the sensor wire.

 

Share this post


Link to post
Share on other sites

I wouldn't call it a "fire hazard" - more of a self destruct hazard.

UM has tested this machine by running full power through the head for a very long time - long past when it is "destroyed". The machine gets damaged of course but there isn't much of a fire hazard unless you put gasoline or other flammables on the printer (things that burn below 300C should definitely cause a problem):

http://www.engineeringtoolbox.com/fuels-ignition-temperatures-d_171.html

 

Share this post


Link to post
Share on other sites

If you are going to address this, You should also address this:

http://umforum.ultimaker.com/index.php?/topic/7070-feature-suggestion-nozzle-heater-runaway-protection/?p=65778

*******NOTE: if my analysis of the above code is correct. Pulling the temp sensor HIGH is BAD unless the code is fixed. UNLESS, the condition is detected elsewhere. Is there code that produces the MAX TEMP error and would it detect this? I think so...

However, I think for the most consistent and safest checking, I think the changes I suggested should be made. The architect in me says the temp checking should be handled in one place and should be defined as the temp being within the expected range within the expected time. This check would handle fail to 0 and fail to MAXTEMP. The only thing it doesn't handle is fail to continuous reporting of the current value which I believe should be handled by the pull up resistor fix.

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Our picks

    • How to 3D print with reinforced engineering materials
      Ultimaker is hosting a webinar where we explain how you can achieve and maintain a high print success rate using these new reinforced engineering materials. Learn from Ultimaker's Product Manager of Materials and top chemical engineer Bart van As how you can take your 3D printing to that next level.
      • 0 replies
    • "Back To The Future" using Generative Design & Investment Casting
      Designing for light-weight parts is becoming more important, and I’m a firm believer in the need to produce lighter weight, less over-engineered parts for the future. This is for sustainability reasons because we need to be using less raw materials and, in things like transportation, it impacts the energy usage of the product during it’s service life.
        • Like
      • 12 replies
×

Important Information

Welcome to the Ultimaker Community of 3D printing experts. Visit the following links to read more about our Terms of Use or our Privacy Policy. Thank you!