We have had some serious issues with forms taking forever to load and hanging when changing values on some screens, like Quote Entry.
We are on Kinetic 2022.1, the problem existed prior to the upgrade, we were on Epicor 10.2.5 previously.
I’m not sure if this is to do with bad windows updates, maybe a bad server config, or network. I have ran the Performance and Diagnostics tool and looks good to me. I don’t see any of the “bad” updates listed in control panel either. I’m running out of ideas and this is causing a lot of pain for our employees.
We have a SQL server running on bare metal, but our app server is virtualized in vcenter. Speaking of, we were running some really old server for the longest time, it was sluggish back then, but not to the extent we see today. We virtualized almost all of our servers on a new HCI cluster consisting of 4 VxRails. Each server has a Xeon Gold 6230R with 384GB of RAM. Storage is mechanical, they do have a SSD for cache. I also just upgraded our entire network, simplifying our setup and uplifting many access switches to 10GB.
I’m thinking this is software related, both our servers and network have been upgraded significantly and we still have problems.
Any advice or help is also appreciated. Thanks in advance!
Take some trace logs of slow things happening, include server trace, BPM logging, and BAQ logging, no need to include datasets or changes… In the server trace part of the log it will list all BPMs and data directives, and their execution times… Usually this is due to bad BPMs or too many changelogs, etc…
If you post some traces here I’m sure we can help narrow the issue…
@josecgomez did anything official ever come out? I thought the compression change was causing some other situation of freezing for @jgiese.wci … although I am not sure I understood the whole back and forth about all that. Should we hold off from making that compression change.
There is a case (affected by network bandwidth and proximity / speed) where this change can cause some clients to lock up. It is an underlying issue with .NET 4.8 and asynchronous / synchronous web calls. Epicor has been working on a fix around it and are actively testing it.
Once that is done they will publish official guidance on this. So yes hold off for now, or make the change and watch for issues. We are actually able to work with this just fine because we are far enough from our server SaaS where only the Task Agent is affected by lock ups.
The compression trick didn’t seem to do the trick for us, but thanks for sharing.
I have a trace for Quote Entry though, it takes nearly 20 seconds to load the form, then my biggest complaint is that changing the “Expected Quantity” takes 10-15 seconds to process. I think the trace is onto something though, I see lines like:
That is milliseconds… This means the server is responding in time, the issue appears to be the client computer… How is your client setup? Is it installed on your PC, or do you connect through a terminal server session?
Not sure if helpful, but the web access does feel faster, we have not adopted it yet though. Also, I notice there appears to be some kind of “friction”, for lack of a better word, when dragging some forms around my screen. Job and Quote Entry seem to be the worst. I’m on a decent computer, and task manager doesn’t show any resources being heavily used either.
Do you have a customization on those screens that are slowest? Also, are you the only one with the issue, or does everyone (other PCs) have the same symptoms?
TBH, your additional description, like the “friction” when moving forms around, makes it sound like the CPU is being used at 100% or close to it… You might have picked up some bitcoin miner virus? I have seen these things go as far as to alter the performance counters to make the Task manager look like everything is fine…
There are some customizations, yes, but problems persist even if I load the base form. Everyone has the same issues. I don’t think it would be a virus, we get the same issues even on a fresh load of Windows. I also get the same issues using the client on the server directly. Everything else seems to run fine, the only issue as far as I can tell is the Epicor client.
Can you drop that trace in the performance and diagnostic tool? What do you see when you do that
I tried but you modified the file so its not valid any longer. The stuff you redacted is just username and machine name I’m not sure that’s sensitive. But it makes the diagnostics tool blow up any chance you can get us a full clean (unedited) trace?
I’m curious how large these quotes are I’d be curious too to see a network trace see how much data you are pulling down. Any chance you know how to run fiddler classic?
