The past couple days we have started getting a ‘System.OutOfMemoryException thrown’ error on our transaction server. It seems to be triggered by a few key tasks, like trying to do a print preview on ‘Invoice Edit List’, which you generate from:
Invoice Entry (populated from a specific invoice group) … Actions Menu: Group … Edit List. The specific details of the error are:
Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message: RunTask: Exception of type ‘System.OutOfMemoryException’ was thrown.
If I look at the resource meter, RAM usage is high but there still quite a bit available. Has anyone else run into this, and how did you resolve.
We also get that error when attempting to ship a credit card order. After generating the shipping label and the packing list, the user is then prompted: “Is this pack list ready to Ship?” When they click ‘Yes’, the system would normally trigger a charge to the customers credit card, and then mark the shipment as ‘Shipped’. But when they click ‘Yes’ now they’re getting the error.
Check for BPM execution on any of those touch points and/or poorly written BAQs that may be invoked via dynamic query or BPM. Check your database server performance as well.
Did anything change week over week?
Everything was working ‘normal’ up through EOD last Friday. As far as I know, there weren’t any changes to BAQs, BPMs or dashboards over the weekend, but I’ll confirm. I do know were starting to run low on RAM accessible to our SQL VM. I’m suspecting that VM may in fact be the culprit, as it’s taking much longer just to log ont that VM or to bring up SSMS.
There are some other threads about this. It started happening to various people starting a few months back. I have been dealing with it since Sept maybe. Same thing just seemingly started happening one day. Some days are better than others. Today was the first time it’s happened in a while. No way to reproduce and no real indication of what’s wrong. We have all come to the conclusion it has to be some windows update because it usually started on a monday and no changes in Epicor version.
I’m guessing the error shown below might be related. Has anyone else seen this, and do you know what process would need authentication? We haven’t changed anything that would require the use of credentials.
Transport authentication failed.
Service: net.tcp://epitrans/ERP10/Ice/Lib/SessionMod.svc
ClientIdentity:
ActivityId:
SecurityNegotiationException: The server has rejected the client credentials. —> InvalidCredentialException: The server has rejected the client credentials. —> Win32Exception: The logon attempt failed
That’s pretty off center of the rest of the reports. I would say that’s unrelated to this. You are seeing a server side error. We are seeing a client side error.
A related question about the SessionMod.svc error:
That service is accessible under REST. I ran across this post …
… about logging out of the REST api. Is it necessary to log out after using the API? I have been developing something that uses REST but haven’t pushed it out yet. Is it possible that each time I did a test that a ‘door’ was left open on our network, and that open door is responsible for the persistent ‘event’ entry on our app server?
The following was posted elsewhere and he shared it with me:
Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message: RunTask: Exception of type ‘System.OutOfMemoryException’ was thrown.
Stack Trace:
at System.Text.StringBuilder.ToString()
at Erp.Internal.PE.PEMessage.TryReadUntilFileAccessible(String fileName) in C:_projects\ERP\RL10.2.300.0\Source\Server\Internal\PE\PEMessage\PEMessage.cs:line 245
at Erp.Internal.PE.PEMessage.UpdatePostInfo(String newPostInfo, String postingUID) in C:_projects\ERP\RL10.2.300.0\Source\Server\Internal\PE\PEMessage\PEMessage.cs:line 197
at Erp.Internal.PE.LogView.UpdatePostInfo() in C:_Releases\ERP\UD10.2.300.40\Source\Server\Internal\PE\PEData\LogView.cs:line 178
at Erp.Internal.PE.LogView.FinalTag() in C:_Releases\ERP\UD10.2.300.40\Source\Server\Internal\PE\PEData\LogView.cs:line 580
at Erp.Internal.PE.Pre_PostBase._RunProcess(Int64 instance_TaskNum, String groupID, PostingMode mode, Boolean skipPEDataInit) in C:_Releases\ERP\UD10.2.300.40\Source\Server\Internal\PE\Pre_PostBase\Pre_PostBase.cs:line 212
Solution
Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message: RunTask: Exception of type ‘System.OutOfMemoryException’ was thrown.
Exception caught in: Epicor.ServiceModel
Error Detail
Description: Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message: RunTask: Exception of type ‘System.OutOfMemoryException’ was thrown.
Program: CommonLanguageRuntimeLibrary
Method: ToString
Original Exception Type: OutOfMemoryException
Framework Method: TryReadUntilFileAccessible
Framework Line Number: 0
Framework Column Number: 0
Framework Source: TryReadUntilFileAccessible at offset 504 in file:line:column :0:0
Resolution
PROBLEM RESOLUTION:
Disable Posting Engine (PE Logging) in the database in question.
Open Epicor client,
Navigate to Financial Management
Select General Ledger
Select General Operations
Click PE Log Viewer.
Click on the Transaction types tab and click Unmark All button
Click Save.
Go to the Settings tab, press the Clear PE File button
Click Save
Close PE Log Viewer window.
Test to see if the error can be reproduced.
NOTE:
• PE Logging should NEVER be enabled in a production database!
• Any OutOfMemoryException errors when working within a financial module where the server and/or Epicor client isn’t really running out of memory could be caused by PE logging being enabled.
• The Performance and Diagnostic Tool (PDT) Config Check has a rule regarding PE rules. Any PE rule being enabled against the database in question will cause this specific PDT rule to FAIL.
NOTE: This can also create the following error when saving invoice
• "Failed to create directory C:\Epicor\ERP10\ERP10.0.700\ClientDeployment\Log.System Error Access to the path ‘C:\Epicor\ERP10\ERP10.0.700\ClientDeployment’ is denied.
Notes
Troubleshooting that was already performed:
• Confirmed that the Epicor server had plenty of available physical RAM.
• Switched to Epicor64.exe (64bit client) = same error.
• Restart of Epicor server = same error.
• Redeploy appserver process on Epicor server = same error.
• Purged .NET temp files (C:\WINDOWS\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files) = same error.
• Provisioned a new appserver process pointing to the same database on Windows server and pointed client to that = DO NOT receive error.
Now I’m curious to see if other reports are due to the same reason, I would have never thought about checking the pe log file.
I agree that it should only be turned on if you are troubleshooting the posting engine, I wonder how many transactions would be logged for you to reach 1.2gb, months maybe?
Just sharing the same experience in our Epicor System. This week, during User Training we facing the same problem. We have check our PE Log in Server directory and already reach 2 GB Then, we delete this log and the problem disappear.
PS: If we have two separate machine of Application Server for Interactive and Reporting App Server, PE Log should be in Report App Server.
Hi Juari. We got the same issue today; we followed the instructions and we couldn’t open the log file, however we see it’s 25GB. We decided to delete it, and renamed the old one just in case. Is this the only root cause for OutofMemory Exception error in Epicor? Thanks for your support.