These are testing times...

[burtoogle 513]

OK, so there's quite a discussion going on in the Cura 3.5 thread about quality control and testing of Cura (and related firmware, etc.) and I thought I would start this new thread to make a particular point and give people a place to discuss it and related matters. Here goes...

 

For various reasons, the products UM make evolve. The hardware is fairly static (years) but the software is a much more dynamic thing (months). The software is continually getting enhanced with new features and capabilities, old bugs getting squashed, etc. Almost always, each release brings not only goodies but new bugs and incompatibilities with existing platforms and OSes.

 

Obviously, each new release needs a lot of testing on all 3 supported platforms (with their various versions of OS and drivers, etc.) before it can be considered trustworthy. We know that UM do a lot of testing. They could probably do more but even if they did a lot more, it wouldn't catch every bug and issue. The problem space is too big with too many variables. So this is where the community of Cura users can step up and make a real difference. Hands up those of you who use Cura with a non-UM printer and never bother to try the beta that is made available before each release. [Quite a lot of people looking a bit sheepish out there.] Why not try it? Why not give something back to UM? I'm not talking about money. All it takes is a bit of disk space and some time to install the beta and try out slicing and (ideally) printing a few of your projects. If all looks good, fine, you've done your bit. If there's a problem, create an issue on github (or post on the forum) and that's even more valuable.

 

Yes, the software (Cura) is free for everybody to use but if you're using it on a non-UM printer then you really should be helping with the beta testing because, at the end of the day, that helps everyone, yourself included. If you don't bother to try the beta and the new release doesn't work well for you, who's problem is that? Yours!

 

Of course, it would be nice if those people who have purchased a UM printer could try the betas also if they possibly can but I don't feel that they are in anyway as obliged as the non-UM printer owners to do so.

 

So to sum up, the Cura user community can make a huge difference to the quality of each Cura release. Why be a freeloader when you can be a valuable member of the community?

 

 

[JohnInOttawa 104]

Starting this thread is a good idea. 

 

Just a thought - it might be that Ultimaker sells to a world full of freeloaders, or perhaps the user community perceives barriers or risk in beta testing. 

 

I know most of the time my printers are running there's a commitment or deadline in sight.  Can I make more time to test things?  Probably.   Would I know how to do so methodically and effectively, without testing the same 25% of features that everyone else uses and totally missing the corner case that ruins the mood?  Probably not.

 

Then there's background.  Would I, personally, think of running a beta version of CNC milling control software?  Not a chance.  The risk of having a head crash with a 15000 rpm bit disintegrating is too great.  Not the same level of threat here, but my point is, we tend to develop an approach to testing depending on where we come from.

 

All this to say:  Is there a need to better engage the user community in beta testing and proving?  Yes?  Then why don't we build that bridge as a community?  Rather than relying on random chance for a user out there to stumble on something, even in Beta,  and then complain, is there a way to identify what features are up for change in advance so those who use those sections can take notice, set aside the time and shake down the product?  Maybe there is specific data that needs to be reported in a usable format following even a successful test.  Just thinking out loud.

 

I agree with you, these products are only as good as we let them be.  New features mean more complex testing and more edges for something to maybe catch on.  Let's help each other be a better testing community.  How do we do that?

 

-John

[Shadowman 53]

I agree that the thread can have value however, I think the pervasive issue is Ultimaker deploying software as “ready for prime time” rather than as a “beta”.

 

Certainly I can and will share issues as evidenced and yet Ultimaker and the Cura development team need to create a comprehensive testing routine supported by those desirous and willing to participate. I certainly don’t have time to do so; I have been doing so for years with numerous 3D printers which is why I stroked the big check believing that I was now simply an “owner and user” of a proven package and not a member of the development team.

 

Albeit different; I don’t purchase a new car expecting to be the tester, I don’t purchase paper printer expecting to be a tested, etc..

 

When I make a purchase I evaluate many things including what I am willing to pay for a product and or service some of which is what I feel it took to bring it to market i.e. the development costs. You see; I wasn’t looking for the lowest price or best value but rather the best product; one that I can trust and use without making the operating of it a full time job.

 

When anyone makes the decision to invest in a Ultimaker product; particularly the last 2 offerings; the 3 and S5 then it is Ultimaker’s resonsibity to do as needed to make certain that it arrives ready to perform as solicited “without” an unspoken caveat about how everyone is doing their best.

 

This is what Ultimaker told me and blindly I trusted them.

 

Now I can’t speak for others and yet I can reiterate that we invested roughly $8K for a S5 with spares and materials; an amount that most certainly puts this printer in a non hobbiest catagory. 

 

Again; I see value in thread as long as it is not created with the expectation that those that purchased the printers are now somehow responsible for the ongoing development of the associated software. As if to say; “hey” you purchased it and now you expect it to work; LOL without your help this can’t happen. 

 

I am purposely being blunt because the issue is being placed in-part on the backs of the owners of the Ultimaker printers; at least this is part of what I read.

 

Again I say; where is Ultimaker’s Official statement?

[ahoeben 1,948]

Now you're acting as if Ultimaker employees came into your house and killed your cat. Noone is telling you you have to do anything; you don't have to test beta versions and you don't have to upgrade to Cura 3.5.

 

Cura 3.4.1 - though not perfect - still works, right? If you have removed it from your computer, you can still download it from software.ultimaker.com

 

I am all for more testing, and not releasing a "stable" version before it is proven stable by internal and community testing - instead of releasing because it is that time of the week - but lets not make more of the issues found in Cura 3.5 than what they are.

[P3D 46]

7 hours ago, Shadowman said:

Again I say; where is Ultimaker’s Official statement?

 

Maybe they're busy preparing Cura 3.5.1 with the bugs resolved...

 

All jokes aside, while I do share your concerns and have been bitten by a Cura 3.5.0 bug, I also have to say that most commercial software that is more stable and bug-free than Cura also has fewer features and a snail-like development pace - or is extremely expensive.

 

I also have seen much worse than Cura, especially when comparing manufacturer supplied software... Particularly, the nightmare that ABBs RobotStudio is comes to mind. Things working with version x.0.1 but not with x.0.2, and still other things only working with a version from 2 years ago, random crashes etc. (I do hope there was some improvement since when I had to use it, but I doubt it)

 

This doesn't solve the Cura bugs, of course, and I also think they should have done more testing (the bug I experienced with Breakaway supports in particular should never have entered a release version), but it puts things in perspective a little bit.

[kmanstudios 1,120]

32 minutes ago, P3D said:

snail-like development pace

Let us see...slower dev releases that are very stable VS. quick and buggy. I would go with stable every day.

 

33 minutes ago, P3D said:

I also have seen much worse than Cura

So have I. But also much better. Unfortunately, the software industry as a whole is on this unwritten fast release cycle that requires 'recalls' (recent win 10 pull back due to files being deleted), Autodesk products requiring pathces for 6 months out of any given year because of their rush. The list goes on and it is endemic within the software industry as a whole.

 

Say what you want, but before everybody got into a race to the bottom, software was a lot more stable. The current excuse is that there are so many lines of code blah, blah, blah. That would sorta imply the need for slowing down a bit and making things stable.

 

It is also very important for the company to present as solid of a solution as possible when entering into a higher competitive arena.

37 minutes ago, P3D said:

(the bug I experienced with Breakaway supports in particular should never have entered a release version)

Agreed. That is a big bugaboo.

 

38 minutes ago, P3D said:

I also think they should have done more testing

Kinda makes for the slower and more solid case.

 

2 hours ago, ahoeben said:

proven stable by internal and community testing - instead of releasing because it is that time of the week

Agreed, but as stated above, seems to be the way software is released these days and has been for years now. It is an issue across the board and not just Cura.

[P3D 46]

Aaaand the official statement is here: 

 

[jens3 11]

Based on previous experiences, I consider any x.x.0 an alpha release and x.x.1 a beta release. A lot of times the beta version works well enough, especially with the user community having come up with work around's.

While I am firmly in the house of 'I want the product t be stable rather than having fancy new features', I must admit that every time a x.x.0 release comes around I am like a kid in a candy store and load it up, just to be disappointed and saying to myself 'next time wait for x.x.1 dummy'. You'd think I would learn but alas ......

I think re-naming should go a long way towards letting people know that if you are running a business do not work with the latest.

I also believe there should be at least one more version , x.x.5, that is strictly a bug fix issue without new features. If .0 is Alpha, .1 is Beta then .5 should be considered the actual release.

It is nice to see .1 shortly after .0 as a 'we have fixed show stoppers'. .5 should come well down the road when the development team has had the time to catch up with things.

[Shadowman 53]

On 10/16/2018 at 12:31 AM, ahoeben said:

Now you're acting as if Ultimaker employees came into your house and killed your cat. Noone is telling you you have to do anything; you don't have to test beta versions and you don't have to upgrade to Cura 3.5.

 

Cura 3.4.1 - though not perfect - still works, right? If you have removed it from your computer, you can still download it from software.ultimaker.com

 

I am all for more testing, and not releasing a "stable" version before it is proven stable by internal and community testing - instead of releasing because it is that time of the week - but lets not make more of the issues found in Cura 3.5 than what they are.

Your comment is interesting but carries no weight with me.

 

As a business using a product targeted for a business that I am told is ready for prime time I expect “nothing” less.

 

Your comments set the stage for acceptance at a level that is unacceptable.

 

For example; when you go to update your firmware it states whether a tested stable version or in testing as such you can make a conscious decision. In the case with Ultimaker Cura 3.5 we were told that it was not only ready for prime time but encouraged to load it via the pervasive pop-up when opening any previous version of Ultimaker Cura. 

 

If the chain of events as assiciated with this deplyment caused you no issues and or concerns then I suspect that your printer is not depended upon during the course of your business.

 

I do respect your position however, let me make “very” clear that the premature deployment of Ultimaker Cura 3.5 caused me significant issues with us initially believing that we had secondary issues that we spent 2 days attempting to fix until we realized that the culprit was “exclusively” Ultimaker Cura 3.5. If this is not understood by you then no words conveyed by me will make my position more clear.

 

I do suspect that many lessons have been learned and I also expect that future deployments will have much more consideration given as such will be as they should be; stable and fully functional on the core level.

 

Three things;

 

1. Remove that pervasive pop-up within earlier versions encouraging one to udate.

2. If features are not fully tested within the program then so note it such as was the case with Tree Supprts.

3. Make darn certain that the core tools within are fully functional such as “Breakaway Support”; this one being broken and the subsequent casual it should be fixed by Ultimaker Cura release 3.5.6 completely dumbfounds and troubled me.

 

I do appreciate the comments as shared by the many that have been affected while also appreciate the development teams work.

 

Like everyone else I look for this to be an anomaly; albeit a significant one that we all learn from.

 

Developers, slower fully tested deployments.

 

Users, consider any deployment as beta until you confirm that within your world it performs as needed.

 

 

Edited October 18, 2018 by Shadowman
Spelling

[ahoeben 1,948]

Note that I am not an Ultimaker employee, I am just a guy who purchased Ultimaker printers before, who contributed a fair amount of code to Cura, and who has been fairly active in this community. My views are not the views of Ultimaker, and I am tired of this discussion so I will refrain from further participating in it.

[nallath 1,120]

 

6 hours ago, Shadowman said:

1. Remove that pervasive pop-up within earlier versions encouraging one to udate.

2. If features are not fully tested within the program then so note it such as was the case with Tree Supprts.

2

Tree supports are in the experimental category. Which ya know, kind of give away that its well... Experimental. The popup issue has been resolved.

 

[Shadowman 53]

2 hours ago, nallath said:

 

Tree supports are in the experimental category. Which ya know, kind of give away that its well... Experimental. The popup issue has been resolved.

 

I appreciate your clarification of my position.

 

Yes, Tree Supports are experimental as such “use at your own risk”; this was clearly defined and understood.

 

With you and the development team having eliminated the popup issue; I thank you.

[AirBronto 8]

I'm not sure how feasible this is, but what would help me a lot is to have the ability to run different versions of the Cura applications completely in parallel with each other, sharing no files or profile settings at all.  Like a lot of people, I've spent dozens of hours fine-tuning my 3.4.1 profile, dialing in well over 100 options and reconfiguring them for multiple nozzle sizes and materials.  I tested the 3.5 beta (the beta versions seem to have their own paths for profiles and printers so I didn't have to worry about my production settings), but I refrained from updating to the 3.5 release because I was nervous about any process that modified/touched my profiles.  I've made backups of them, but in previous versions (I believe it was going from 3.3 to 3.4), I had an issue where the upgraded software seemed to have changed my profiles just enough to cause strange issues when I tried to revert.  I wish I could remember the details, but essentially updating to the new release version ended up hiding settings in the old version and I couldn't get them back even after uninstalling the new version.  It also silently modified my starting GCODE for one of my machines, and reverted a few of my custom settings.  Since I've made so many changes to so many settings, one or two misconfigurations could mean hours of work for me.

 

My ideal scenario would be that when I download Cura 3.5 release, it copies everything like profiles, machines, and materials from my 3.4 folder into a 3.5 folder, and then only upgrades the newly copied files only (meaning that my original 3.4 profiles, machines and materials are completely unchanged by the upgrade).  That way I could have 3.4 and 3.5 open at the same time for A/B testing, but I wouldn't have to worry about the 3.5 upgrade making small changes to my 3.4 install.  I personally prefer the quick-release schedule because I like new features and enjoy tinkering with things.  However, I'd like to be able to do it in a way that I can load up the new release, experiment freely with it, change any settings I want, enable new features, and then after I've completely messed everything up I can easily just say 'screw it,' open up my older version and know that everything is 100% unchanged, no reversion or backups or reconfiguration needed.

[Shadowman 53]

16 minutes ago, AirBronto said:

I'm not sure how feasible this is, but what would help me a lot is to have the ability to run different versions of the Cura applications completely in parallel with each other, sharing no files or profile settings at all.  Like a lot of people, I've spent dozens of hours fine-tuning my 3.4.1 profile, dialing in well over 100 options and reconfiguring them for multiple nozzle sizes and materials.  I tested the 3.5 beta (the beta versions seem to have their own paths for profiles and printers so I didn't have to worry about my production settings), but I refrained from updating to the 3.5 release because I was nervous about any process that modified/touched my profiles.  I've made backups of them, but in previous versions (I believe it was going from 3.3 to 3.4), I had an issue where the upgraded software seemed to have changed my profiles just enough to cause strange issues when I tried to revert.  I wish I could remember the details, but essentially updating to the new release version ended up hiding settings in the old version and I couldn't get them back even after uninstalling the new version.  It also silently modified my starting GCODE for one of my machines, and reverted a few of my custom settings.  Since I've made so many changes to so many settings, one or two misconfigurations could mean hours of work for me.

 

My ideal scenario would be that when I download Cura 3.5 release, it copies everything like profiles, machines, and materials from my 3.4 folder into a 3.5 folder, and then only upgrades the newly copied files only (meaning that my original 3.4 profiles, machines and materials are completely unchanged by the upgrade).  That way I could have 3.4 and 3.5 open at the same time for A/B testing, but I wouldn't have to worry about the 3.5 upgrade making small changes to my 3.4 install.  I personally prefer the quick-release schedule because I like new features and enjoy tinkering with things.  However, I'd like to be able to do it in a way that I can load up the new release, experiment freely with it, change any settings I want, enable new features, and then after I've completely messed everything up I can easily just say 'screw it,' open up my older version and know that everything is 100% unchanged, no reversion or backups or reconfiguration needed.

A very well thought out and presented proposal.

 

I agree with you and think that such a program would as a byproduct create the basis for one; being business or hobbiest to contribute to the ongoing development when compared that which you personally experienced in past and many; including myself experienced with the deployment of Ultimaker Cura 3.5 afterwhich albeit desirous of we are extremely reluctant to participate as such contribute to new deployments.

 

In my opinion this was a great suggestion.

 

Thank you.

 

Edited October 18, 2018 by Shadowman
Spelling

[nallath 1,120]

Quote

My ideal scenario would be that when I download Cura 3.5 release, it copies everything like profiles, machines, and materials from my 3.4 folder into a 3.5 folder, and then only upgrades the newly copied files only (meaning that my original 3.4 profiles, machines and materials are completely unchanged by the upgrade).  That way I could have 3.4 and 3.5 open at the same time for A/B testing, but I wouldn't have to worry about the 3.5 upgrade making small changes to my 3.4 install.

This is exactly the way Cura handles it right now. 

Edited October 18, 2018 by nallath

[Shadowman 53]

9 minutes ago, nallath said:

This is exactly the way Cura handles it right now. 

 

Not meaning to create an argument; “suppose to” maybe a better comment as the automatic migrating of profiles between versions into Ultimaker Cura 3.5 has not worked as many have shared these last several days.

 

Furthermore; I suggest taking the idea of running parallel versions a step further and encourage such a practice rather than simply presenting an upgrade path.

 

A thread that introduces folks to to idea of parallel versions and the associated benefits and reduced risk because I am certain that the vast majority of folks would never consider it.

 

It is simply not a typical upgrade practice.

 

Just a thought.

[JohnInOttawa 104]

So, I think this should go without saying, but this forum is a place where we tackle issues, and not people, right?

 

I see individuals leaving the discussion just when we need their input the most.  If we truly want the best out of this product, the only way I can see that happening is if we use our resources - that relies on engagement.

 

Not to disagree with anyone's level of frustration.  But I do not think the energy from that should be aimed at a person.

 

FWIW

John

[Shadowman 53]

5 minutes ago, JohnInOttawa said:

So, I think this should go without saying, but this forum is a place where we tackle issues, and not people, right?

 

I see individuals leaving the discussion just when we need their input the most.  If we truly want the best out of this product, the only way I can see that happening is if we use our resources - that relies on engagement.

 

Not to disagree with anyone's level of frustration.  But I do not think the energy from that should be aimed at a person.

 

FWIW

John

Agreed

 

If referring to any of my comments and or suggestions they are all program and not people related.

 

Takes care

[AirBronto 8]

1 hour ago, nallath said:

This is exactly the way Cura handles it right now. 

 

I just installed Cura 3.4.1 and 3.5.1 on a fresh Windows 10 virtual machine... and you're right!  The settings changes in the 3.5.1 version did NOT go back and modify the changes in 3.4.1.  I'm not sure what my earlier problem was, but it's not a problem anymore.  Thanks!  Now I can run 3.5.1 and test it without worrying about losing my 3.4.1 settings.

[Shadowman 53]

8 minutes ago, AirBronto said:

 

I just installed Cura 3.4.1 and 3.5.1 on a fresh Windows 10 virtual machine... and you're right!  The settings changes in the 3.5.1 version did NOT go back and modify the changes in 3.4.1.  I'm not sure what my earlier problem was, but it's not a problem anymore.  Thanks!  Now I can run 3.5.1 and test it without worrying about losing my 3.4.1 settings.

Thank you for sharing your successful conclusion.

[nallath 1,120]

1 hour ago, Shadowman said:

 

Not meaning to create an argument; “suppose to” maybe a better comment as the automatic migrating of profiles between versions into Ultimaker Cura 3.5 has not worked as many have shared these last several days.

He was talking about that installing 3.5 would ruin his 3.4 setup, which is obviously not the case. Cura first copies the older version (so in this case; copy 3.4 to 3.5 files) and subsequently upgrades them (if needed).

This is irrespective of if the upgrading step (which can modify your profiles) actually succeeded.

[Shadowman 53]

12 minutes ago, nallath said:

He was talking about that installing 3.5 would ruin his 3.4 setup, which is obviously not the case. Cura first copies the older version (so in this case; copy 3.4 to 3.5 files) and subsequently upgrades them (if needed).

This is irrespective of if the upgrading step (which can modify your profiles) actually succeeded.

Understood 

 

Then to clarify my position; when upgrading the profiles should;

 

1. Remain with the original as they are

2. Port into the new version 

 

Thus creating parallel programs.

 

By the way the Ultimaker Cura update popup is still present when loading an earlier version of Cura.

 

Thank you 

[nallath 1,120]

2 minutes ago, Shadowman said:

Then to clarify my position; when upgrading the profiles should;

 

1. Remain with the original as they are

2. Port into the new version 

 

Thus creating parallel programs.

This is what happens. Obviously once the "branch" has been made, no changes made in an older version will ever be ported to the newer version (unless you delete the newer config data, in which case the copy & upgrade dance starts again)

 

3 minutes ago, Shadowman said:

By the way the Ultimaker Cura update popup is still present when loading an earlier version of Cura.

Yes. That's because it's going to be in the next version.

[Shadowman 53]

2 minutes ago, nallath said:

This is what happens. Obviously once the "branch" has been made, no changes made in an older version will ever be ported to the newer version (unless you delete the newer config data, in which case the copy & upgrade dance starts again)

 

Yes. That's because it's going to be in the next version.

Wonderful and thank you again.

 

For what it’s worth.

 

With an open discussion this is great example of the value associated with;

 

1. A punch list of issues being tended to.

 

2. Release notes presented as the incremental updates are released. 

 

Again, just thoughts and “not” critiques.

[nallath 1,120]

Yeah, i'd love to provide a completely open issue tracker. Unfortunately, we ocasionally have to work on things we want to keep a secret a bit longer, so it was decided to close our JIRA tracker.