We’ve gone live with the switch to 10.2 today, combined with a new server, a change to VMs instead of all-on-one-box and SQL Server 2017 from 2012.
Users are screaming about dreadful performance, which is not what we want to hear after all the investment. Epicor did the install and set-up, and are looking into it, but I’d be glad if anyone has any tips in the meantime.
Nothing much shows up in the Performance Tool, and definitely nothing related to the loudest complaints. Config check says “fail” on SQL query performance, but not by much (naturally we’d prefer it not to fail at all, but it doesn’t seem to be the main issue).
It seems to be the really basic things that are the problem, like loading data into Sales Order Entry, waiting for search lists to populate.
Any clues?
aidacra
(Nathan your friendly neighborhood Support Engineer)
2
Did your users report these issues when they were testing 10.2 before the official cut-over?
Testing was in a VM with limited resources, so yes, they reported it felt sluggish. But in testing, not enough so to stop the switchover. Going live is the first time everybody has been on the system at once and seems to have tipped the balance.
Use TCPING to ping between the nodes in your environment. For example App server to SQL on 1433. See what the speed is there. If its 1ms or below start looking at SQL and or OS settings.
Logging in is fine.
Reports are fine.
Opening forms is slow, some more than others.
Searching is slow.
Switching between lines on a sales order is slow.
Saving is not particularly slow, beyond what would be expected because of loading new data.
The server looks fine. Config reports it’s running very fast (it should, for the spec). No spikes, memory looks normal. The database VM has dedicated Raid 10 disks set up as per Epicor spec, local SSDs.
I overlooked this in my server setup when we went from 9 to 10 and it made Epicor 10x slower than it should have been.
Are you sure C-States disabled in your host BIOS?
This is another ‘dumb’ one but I’ve seen servers (both the VM and or the Physical hypervisor) set to power saving modes in the BIOS and/or in Windows. This will cause these same symptoms. Make sure everything is on Maximum Performance.
Do you have BPMs in place on the stuff that is slow. If you do are you ever joining tt and db tables together directly in queries either in code OR in the query designer widgets?
Thanks. I didn’t do the actual server set-up, just hooked up Epicor people with the server people and got them talking. But I’ve checked and this is apparently all as it should be.
We do, but I believe all BPMs are well under control at this point unless there are structural changes between 10.1 and 10.2 which are tripping them up.
It seems unlikely, though, because no such things are showing up in Performance Monitor, and that’s where they usually stick out.
In fact, the really odd thing is that very little from the worst affected areas is showing up in Performance Monitor stats. Yet in the user client, they’re seeing way too much blue circle.
If you are running the fat client on each desktop try running the client on the app server itself in a remote desktop session. See if the you get the same performance issues on the server or if it is just limited to the client on the desktops.
If you have your VM’s Setup correctly and SQL Setup correctly it will work just fine.
We run Cisco Blades + VMWare. If you are on VMWare make sure you are NOT on HW11 and atleast HW13 or HW8 due to some bugs… In addition your Virtual Host should have all firmware’s examined and up-to-date.
Then reach out to Epicor for a “Health Check” and find out your bottle necks You should run diskspd aka sqlio etc… make sure you pass all PDT Epicor Health Checks.
Also since you are on VM make sure its not balooning and look at all stats including your NetApp/QNAP/SAN/NAS
Thanks for the tips.
It’s a Hyper-V set-up, but the hardware management etc is outsourced here so I’m in the hands of others for the details of that. It is all set up according to a spec from Epicor, and Epicor have said they’re happy with the environment performance. They have done the health check, and I’ve done the PDT stuff myself. The key problem seems to be SQL query performance, but they don’t know why since it’s all been set up by the book according to them.
One of the people we’re paying to do this stuff is meant to be looking at the parts I don’t have access to tomorrow, so I’m just crossing my fingers until then now.
We are running into this same situation with 10.2.100 right now. We have done the same with health checks and PDT. I do have access to the servers and SAN and we have all bios and server setting set according to Epicor, in high performance. We migrated from 9.05.701 to 10.2.100.8 so we were never on an earlier version of Epicor. The install was done on 10.2 so many of the timeout settings were already set for us. I am currently working on a ticket with support, so I will update if there is more to follow.
We have disabled all BPMs in our Test environment still seem to see very slow performance, however ours seems to be reports timing out and we do see the same with opening forms and order lines.
On a side note, when running the PDT, I do notice a bit of a difference between running the X86 and X64 version of the tool. Network average time remains the same, but using the X64 actually seems to perform slower for average server time. Is anyone else seeing this? Could there be some utilities using the X64 making it run slower such as the Task Agent?
How many CPUs/Cores and possibly what Processor do you have. Can you Remote Desktop into your Hyper-V or even APP Server and run CPU-Z | Softwares | CPUID
Your Hyper-V Host should easily hit something like:
One thing I did notice If i go to IIS Pool Settings and I can’t recall; but there is a setting to make your poll multi-threaded in your machine or web.config - once I did that it actually hurt me; once I left it alone all was fast again… shoot it was from one of @aidacra guides on performance tuning
You should run diskspd aka sqlio etc… make sure you pass all PDT Epicor Health Checks.
