RANT: I almost lost faith in E10 today

Off Topic
1

Evening Everyone,

I hate to write a post like this, as I’ve been a staunch supporter for E10 and all of it’s benefits specifically to Admins and developers.

We have had several critical issues since upgrading to 10.2.300.15 including different customers being included on invoices via the automated get shipments oob process, being able to create the same shipment number multiple times via the TF shipment BO, consistent freezing and random WCF server errors (we are EMS St hosted). EMS terminal server drop outs consistently so our users lose connectivity. Our AR department is fed up.

Support hasnt helped much. data fixes they send us don’t work (I go thru the business objects and update myself), and even to submit a problem to development the issue needs to be “re-creatable” . Data fixes are a quick bandaid to a potential underlying issue.

We invoiced a customer for $60K of orders that weren’t theirs- there were two sales orders & two customers on the same invoice. Which isn’t/shouldn’t be possible.

It is very frustrating because support said they could not send it to development since it cannot be reproduced. We needed to reach out to specific people to escalate. I understand the need to reproduce the issue to try and find the root cause, but we also cannot incorrectly invoice customers at random.We’ve done thousands of invoices over the past month but 22 were incorrect – why those 22 were affected, no one knows. We use the “get shipment” function for auto invoicing. So, this isn’t manual/human error, it’s OOB Epicor functionality. There’s clearly something happening, and it’s unacceptable to be told that nothing can be done about it unless we can reproduce the issue.

Where is the quality control / regression testing? Where is the performance testing? I’ve followed all recommendations/ and our servers are supposedly set up to spec with ST hosting , and it still takes our users 5-10 seconds per line in SO entry (with all BPMs/customization disabled). (We were one of the customers who helped bring to light the SO copy performance issue resolved in 300.9)

I like the flexibility and control over this system I do, but what about focusing on quality/performance first, then IOT functionality later. It’s like lipstick on a pig.

If anyone had any recommendations on what we can do please feel free to pm me. I have done in depth performance testing with the perf diag tool, BPM optimizing , app customization optimaztion etc.

We can do 60fps real time graphics in high fidelity but we can’t update a small dataset (comparatively) on an order in under 10 seconds? Come on. My DOOM level (4k textures, map, props) (already way bigger than an order dataset on a home desktop mind you)loads faster than saving an order on an enterprise grade Xeon server

Let’s move this system into the 21st century out of the 20th.
Sorry!
Phew! Thanks

steve-jobs-quotes
steve-jobs-quotes.jpg1200×628 47.3 KB

3 Likes
2

@jgehling It sounds like there might be a bug with Customer.ConsolidateLinesPerPart?

ConsolidateLinesPerPart: Indicates that the shipment lines will be consolidated within the invoice for the same part, the same order, and the same packing slip.

I would look at the invoices that are combining and see if this checkbox is checked in Customer Setup. If not, look for anything that the invoice lines have in common. If so, you will have to start testing different combinations in a test system to see if you can get it to replicate. If you can replicate it, then post the steps and we should be able to find where the bug is and report it to Epicor.

Your pain is definitely felt by everyone on this forum. I would get your CAM involved immediately because you are one of Epicor’s success stories for their hosted service.. Once you report the bug to support it’s up to your CAM to get it pushed through and fixed!

I’ll let people a lot smarter than myself tell you exactly why the performance is so bad.

1 Like
3

Thanks John. I’ll check that issue out. I’m relatively new to helping developing and administering our ERP 10 platform( in terms of this company, i managed ERP 10 for years at a prior company) that is why I’m bringing these core issues up. YES, we have come a long way with the reporting and integration vehicles in E10, I just want to get us to the next level. E10 is a great platform, but it is a platform that requires a great deal of care and proper implementation from all areas of a business (sales,supply chain, finance, IT)

4

One of the earliest ERP projects (non-Epicor) I was involved in the chief executive stated he expected some disruption/noise/inconvenience at go live but if we were unable to ship to an invoice customers properly on system (like the case of one large SAP customer unable to invoice on system for months) then he would consider the go live a failure. I’ve kept that with me and every ERP implementation I have been involved in thus far it has never been a go live issue.

I’m assuming you did robust testing/pilot pre-upgrade so I would get your higher ups to speak to Epicor higher ups as this is functionality that should and needs to work.

A bit of work but the next time it happens restore the last live back up to test (assuming you do this if hosted) and replicate the transactions to shipment on an error invoice and then see if it does the same thing when you invoice. If it does, then you have a real example and you can go back to the backup of live, recreate test and work through it with Epicor.

1 Like
5

Another option now available to Epicor is Time Travel Debugging, which also works for SaaS users. TTD attaches to a process and records everything. It creates a large dataset that allows developers the ability to run debug as if it was live without impacting the production system. TTD is available with Visual Studio 2019 Enterprise edition.

5 Likes
6

Stop showing off with your 2019 technology :grinning:

3 Likes
7

How often do you do backups? Do you have a backup of the day before that invoice was created?
If so restore that to a Db and try to replicate? Does it happen again ? If so there are things that can be done to debug.
Let me know and I’ll try to help you through it

4 Likes
8

@jgehling don’t pass this offer up :wink:

1 Like
9

Good Morning,

Sorry for the rant. I realize its easy for me to stand here on my soap box and sometimes it’s too easy to fall into the rabbit hole of complaining, (and what triggered me was leaving the home screen up (no ERP apps open to go get a cup of coffee and coming back to this error:
image

Thank you everyone for your responses. EMS backs up all of our databases nightly at midnight. We definitely did try re-creating the invoice issue again in our test environment after a refresh and the bug did not re-appear (the invoice for the same customer for the same order was automatically generated successfully). This invoice issue uses the Erp.Internal.AR.CreateInvFromShip.dll as they were not created manually. This has gotten traction and our database is being sent to development (but only after much pressure and reaching out to senior support staff directly)

10

Hey James,
We have been on Epicor since 10.1.400 and upgraded to 10.2.300 this January, and we have never seen am invoice issue like this before. We did do all necessary testing before going live on this version and everything went extremely well. This issue is just so sporadic and cannot be reproduced on command.

11

Jeff, I assumed you had tested thoroughly as I said in my OP. My point is that despite everything you have done, your business is losing faith in invoicing - 22 invoices being totally wrong even out of thousands is just not right- get your management to start to put some pressure on Epicor management about this release impacting on your ability to trade.

2 Likes
12

I feel the same way… Epicor is great at the fact that you can customize much of it in order to fit your needs, but support and specifically Development are hard tow work areas at Epicor.
As you said, most of the times support/development will reject the issue if you can’t replicate it which is awful. There are a lot of processes run in the background so we as users are up to a challenge to find out what exactly is provoking that behaviour.
Escalating doesn’t do a thing to solve issues, and when you ask to some insights of how Epicor operates that process so we(the customer) can test and test until we can get a way to replicate the response is nonexistent. No info about it is relayed to the customer(which is no complete surprise of course)
We are a company that actually have quite of bugs discovered, probably it is also our “fault” in the fact that we are quite a special industry, but come on. I had an issue with the configurator, which rendered the operation useless since the configurator took 55 minutes to close. The solution took several months to actually being adressed by Dev and still they would fix it on 10.2.200 (at the time not even 10.2.100 was available), of course we manage to pushed for solution on 10.1.600 but it took a lot of time and convincing.
Another awful project managment is CSF, they tend to solve only the basics and “hard code” solutions that are no good, even when you try to explain them exactly the issue and how it would be solved for any customer.
I have cases with MRP, MES, WorkQueue, etc. all gone to Development, but the solutions take months normally, from the moment you can persuade support to address it to Dev to the actual solution which pretty much always is bound to be solved on another release or minor version, not even a patch…

Well I pretty much vented, I guess I just needed to get started haha. But bottom line, you are not alone.

1 Like
13

Just to add to this rant - we all use Excel, we buy it for a couple hundred bucks a license, and nobody ever checks the values in a column to make sure the “Sum” function is actually correctly summing the values.

With ERP software, the assumption is that it won’t work, and that has to be determined and fixed prior to implementation.

Being that ERP software is one hell of a lot more expensive than a spreadsheet, it seems to me that our ERP suppliers have lost vision of their customer’s needs. The assumption that it works properly is only made by the folks creating the code. The marketplace fully understands that it’s highly likely that the product will not work as advertised.

3 Likes
14

My Users sent SO Acknowledgements and didnt check the Total when Taxes were included on the SO (which Epicor fixed in 10.2) but now when I send Customers an Invoice with the surprise addition of tax on a million dollar custom built fire truck, they are not obligated to pay it – since our SO Acknowledgement was our Legal Agreement. Sucks to lose alot of money. Its easy to say well you should have checked the Report… At what time do I stop doubting the ERP Software? and start questioning my sanity instead of the Computers.

Not all is perfect, definitely room for improvement. Regression testing should be improved on ERP Vendors side… Even an issue on BOM could cause a ripple effect of Vehicle Recalls, in a bundle of 800 materials that 1 strap or screw for example comes in at 0 qty and is missed, causing an airbag to not kick-in :slight_smile: That can also be caused by human error, absolutely - but theoretically let’s assume Engineers triple checked it all before Checking In the BOM and it just got messed up on Job Entry.

Id like to use it like I use a Calculator, 5x5 = 25 and not 5x5=25(let me verify that just incase 25/5=5)

I wonder if anyone doubts TurboTax or HR Block, like prints out the PDF and goes to a Tax Lawyer and gets a second opinion - or - do you just have faith in it? When I spend 79.99 I usually trust it blindly :smiley:

4 Likes
15

I’ve had/have errors in mine, and when I go to a professional, he gets me out of self employment tax for some limited freelance work that I do. He has to send a letter to the IRS to do it. Turbotax won’t do that for you. So I just have to justify if the taking an afternoon off work and paying $400 is going to be worth it compared to the $80 I spend on turbo tax. Sometimes it is, sometimes it isn’t.

16

Wish this was something new. I remember implementing a new release of Syteline (another ERP) in 2000 - a much bigger undertaking than what we do today - specifically to fix some major bugs. We found some new bugs which caused significant problems, which, wait for it were fixed in the next release which was being released imminently.

Been in this game a while and around about the mid-90’s previously stable systems introduced GUI/windows compatibility and we started to accept broken, fixed in the next version, software. First 10 years or so th of teh 21st century things stabilized not least as lots of players ran out of cash to enhance their systems and quite a lot of vendor rationalization. Even things like mobile integration were half ass*d bolt ons rather than trying to provide the same experience.

Past ten plus years have just been constant flux/change with everything from mainstream virtualization through SAAS/evergreen IT through IOT to real time analytics to device agnostic ux. Throw in Epicor ditching the RDBMS/4GL they had used previously and perhaps things are being spread to thin.

We all love that Epicor is very much a “core platform” that we can easily customize to do lots of cool stuff. But as businesses, and back to the OP we expect the basic estimate to invoice and all steps in between process to work without really checking (like the excel analogy above). It is the mods/customizations and the more niche, esoteric modules that we expect to have to test/expect issues with.

I know this started as a rant about E10 but I can’t help think that costly monolithic ERP is a bit of an aging paradigm and scalable easy to “stack” best of breed apps may well be the future.

We could run our business on spreadsheets/access db’s but that would be like Fred Flintstone’s car. ERP sold as a Porsche/Ferrari/rolls Royce but can’t help thinking that, to quote Top Gear, it is an Alfa Romeo that it looks great and fun when it works but you better be prepared for all the times that you will be on the side of the road with smoke coming out of the engine.

3 Likes
17

FIAT - Fix It Again, Tony

1 Like
18

Run it on Dell servers - Doesn’t Ever Last Long

19

This is a warped idea that’s been infecting the entire tech industry. It’s led to major issues going unfixed for years in supposedly premier software from top companies. I’d consider identifying and fixing bugs without clear steps to reproduce to be a basic skill requirement for any mid-level developer.

20

A planes service log:

Pilot: “Engine noise at altitudes greater than 10,000 feet”

Tech: “Cannot reproduce in hanger”

2 Likes