Task "Running" but Nowhere To Be Found

josecgomez · January 21, 2020, 2:41pm

Epicor Version 10.2.300.12
We have now for MONTHS been chasing this down and at this point I just need to see if someone else has any other ideas, or at the very least just document it for myself on here.

We have a custom report written by Epicor CSG which runs mostly fine most of the time, though from time to time it will “hang” as an ACTIVE task forever, the only way to fix it is to manually kill the task agent, appservers reboot, remove the task and try again

We have 2 AppServers which are dedicated exclusively to running System Tasks (R1 and R2) after much troubleshooting we have set it up to log Reporting, Routing, BPM, Verbose Logging and have had no luck. Today I came in and submitted 2 of the same tasks and it happened.

AppConfig Settings

<configuration>
  <Trace
    disabled="false"
    verbose="true"
    logLocation="ServerLog.txt"
    detailedExceptions="true"
    maxLogFileSize="10mb"
    logCount="20">
    <TraceFlags>
      <add
        uri="profile://ice/fw/reporting" />
      <add
        uri="profile://ice/fw/printrouting" />
      <add
        uri="trace://ice/log/CSG" />
      <add
        uri="trace://ice/fw/BPM" />
      <add
        uri="trace://ice/log" />
    </TraceFlags>
  </Trace>
</configuration>

Task 1 took about 27 minutes to run (that’s a long time , normally a few seconds) Task 2 is still running, since I caught it in the act I decided to do some more sleuthing just for documentation sake.

Task 1 Shows up in the ServerLog.txt and System Agent History and Report History. Task was submited at 8:35 AM and ran for 27 Minuites, here is the Entry on the ServerLog in Server R1

<Op Utc="2020-01-21T13:35:19.3008131Z" act="Ice:Lib:RunTask/RunTaskSvcContract/RunTask" dur="1621551.7466" cli="10.164.135.60:54900" usr="nrosenbaum" machine="84799-E10-R1" pid="3792" tid="112">
  <RunTaskInfo msg="SysTaskNum: 1841274, Task Description: CSG Print Commissions Documents, RunProcedure: Erp.Internal.UD.CSGPrintCommissionRptLib" />
  <CSG msg="START: instanceTaskNum=1841274, outputFileName=\\84799-E10-A2\EpicorData2\Reports\nrosenbaum\CSG Print Commissions Documents2183546.xml" />
  <CSG msg="PARAMETERS&#xD;&#xA;SalesRepList : 1083&#xD;&#xA;SalesTerList : &#xD;&#xA;StatementNumList : &#xD;&#xA;CommPeriod : 2020-1&#xD;&#xA;ProcessRegister : True&#xD;&#xA;CommissionStatement : False&#xD;&#xA;Facesheet : True&#xD;&#xA;CommissionSumary : False&#xD;&#xA;CommStatFormat : 1&#xD;&#xA;" />
  <CSG msg="outputFileName2=\\84799-E10-A2\EpicorData2\Reports\nrosenbaum\CSG Print Commissions Documents2183546.xml&#xD;&#xA;tFiscalYear=2020&#xD;&#xA;tFiscalPeriod=1&#xD;&#xA;tStatList=&#xD;&#xA;tSalesRepList=1083&#xD;&#xA;tSalesTerList=" />
  <ReportRendered reportPath="/EpicorLive10SSRS/reports/CustomReports/PrintCommission/PrintCommission" SSRSRenderFormat="PDF" bytesRendered="4864" duration="1620776.7447" />
  <CSG msg="END" />
</Op>

Note the duration of 1620776.7 miliseonds (27 minutes)

The second Task has been running since 8:45 ( it is now 9:36 )

And it is just sitting there in the “Active” queue however it is nowhere to be found on ANY log. That is this task, and this task number show ACTIVE in the System Agent but have somehow not made it to any log. ServerLog or Event Viewer on either of the Task Servers, it is like a ghost.

Interestingly enough I can submit other tasks and they work fine, AR Print etc, they all work. But this particular Task is just a an “ACTIVE” ghost. No Error Entrires in Event Viewer under Task Agent or App Server Node, not a whisper of them in the ServerLog NOTHING… It is as if this Task was Marked ACTIVE and never actually started.

Does anyone have any ideas here? I’m about to blow a gasket!

josecgomez · January 21, 2020, 3:10pm

Ok After waiting longer (over an hour an a half now) and continuing to monitor the logs I finally see an Entry for the Task.
I got this entry in the Log Doesn’t help me diagnose the issue but at least it told it me it “Timed Out” (however that’s not all that helpful) I’ll keep digging

<EpiConnection oper="Opened" spid="659" />
  <ContextBound name="Ice.Services.BO.SysTaskSvcFacade" contextid="6adc2d6d-60e3-498d-baee-5b5451649ef3" defaultcontext="False" />
  <ContextBound name="Ice.Services.BO.SysTaskSvc" contextid="6adc2d6d-60e3-498d-baee-5b5451649ef3" defaultcontext="False" />
  <Trace uri="trace://ice/fw/perf/Initialize" duration="0.1062" root="False" />
  <DataContext msg="New DataContext ID:658b40ff-8035-483c-9874-8a4f1130bc2c created on SPID:" />
  <EpiConnection oper="Opened" spid="655" />
  <EpiConnection oper="Disposed" spid="655" />
  <Trace uri="trace://ice/fw/perf/Initialize" duration="9.0594" root="True" />
  <EFQueryCache count="29" />
  <DataContext msg="Enlisting in Transaction 886773a3-5036-4f14-8d75-f9fb42c33d66:10, SPID:659" />
  <EpiCommand hashCode="-3638886854476197823" lockHints="UPDLOCK" queryType="SELECT" spid="659" table="[Ice].[SysTask]">
    <parameters>
      <parameter name="p__linq__0" type="Boolean"><![CDATA[False]]></parameter>
      <parameter name="p__linq__1" type="String"><![CDATA[Erp.Internal.UD.CSGPrintCommissionRptLib]]></parameter>
      <parameter name="p__linq__2" type="Int64"><![CDATA[1841295]]></parameter>
    </parameters><![CDATA[
SELECT TOP (1) 
    [Extent1].[SysTaskNum] AS [SysTaskNum], 
    [Extent1].[TaskDescription] AS [TaskDescription], 
    [Extent1].[TaskType] AS [TaskType], 
    [Extent1].[StartedOn] AS [StartedOn], 
    [Extent1].[EndedOn] AS [EndedOn], 
    [Extent1].[SubmitUser] AS [SubmitUser], 
    [Extent1].[TaskStatus] AS [TaskStatus], 
    [Extent1].[Company] AS [Company], 
    [Extent1].[AgentID] AS [AgentID], 
    [Extent1].[AgentSchedNum] AS [AgentSchedNum], 
    [Extent1].[AgentTaskNum] AS [AgentTaskNum], 
    [Extent1].[RunProcedure] AS [RunProcedure], 
    [Extent1].[InitiatorSource] AS [InitiatorSource], 
    [Extent1].[ActivityMsg] AS [ActivityMsg], 
    [Extent1].[History] AS [History], 
    [Extent1].[TaskNote] AS [TaskNote], 
    [Extent1].[LastActivityOn] AS [LastActivityOn], 
    [Extent1].[UserPIDInfo] AS [UserPIDInfo], 
    [Extent1].[ProcessID] AS [ProcessID], 
    [Extent1].[IsSystemTask] AS [IsSystemTask], 
    [Extent1].[SysRevID] AS [SysRevID], 
    [Extent1].[SysRowID] AS [SysRowID], 
    [Extent1].[ProgressPercent] AS [ProgressPercent]
    FROM [Ice].[SysTask] AS [Extent1] WITH (UPDLOCK) 
    WHERE ([Extent1].[History] = @p__linq__0) AND ([Extent1].[RunProcedure] = @p__linq__1) AND ([Extent1].[SysTaskNum] = @p__linq__2)
]]></EpiCommand>
  <DBStatement type="SysTask" duration="44.2568" rowCount="1" hashCode="-3638886854476197823" dbContextId="6adc2d6d-60e3-498d-baee-5b5451649ef3">
    <stackTrace>
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler.InvokeSingle(Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs" line="0" />
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler+&lt;&gt;c__DisplayClass35_0`5[TContext,TArg1,TArg2,TArg3,TResult].&lt;Compile&gt;b__0(TContext context, TArg1 arg1, TArg2 arg2, TArg3 arg3)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs" line="1139" />
      <stackFrame method="SysTask Ice.Services.BO.SysTaskSvc.SelectSysTaskBySysTaskNumWithUpdLockQuery(Boolean history, String runProcedure, Int64 sysTaskNum)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTask.Queries.cs" line="99" />
      <stackFrame method="Void Ice.Services.BO.SysTaskSvc.SetSysTaskStatusForPendingTask(Int64 sysTaskNum, String runProcedure, String taskStatus, String messageToLog)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTask.cs" line="281" />
      <stackFrame method="Void Ice.Services.BO.SysTaskSvcFacade.SetSysTaskStatusForPendingTask(Int64 sysTaskNum, String runProcedure, String taskStatus, String messageToLog)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTaskSvcFacade.cs" line="248" />
      <stackFrame method="Object .SyncInvokeSetSysTaskStatusForPendingTask(Object , Object[] , Object[] )" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Object System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Object Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs" line="59" />
      <stackFrame method="Object Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs" line="28" />
      <stackFrame method="Object Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs" line="23" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[TChannel,TItem,UChannel].InnerTryReceiveCompletedCallback(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.TransportDuplexSessionChannel+TryReceiveAsyncResult.OnReceive(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.OnReceiveComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Net.LazyAsyncResult.Complete(IntPtr userToken)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)" module="System.dll" file="" line="0" />
      <stackFrame method="Int32 System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)" module="System.dll" file="" line="0" />
      <stackFrame method="Boolean System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.ConnectionStream+IOAsyncResult.OnAsyncIOComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)" module="CommonLanguageRuntimeLibrary" file="" line="0" />
    </stackTrace>
  </DBStatement>
  <EFQueryCache count="31" />
  <EpiCommand hashCode="-1111364414381793029" queryType="SELECT,SELECT" spid="659" table="[Ice].[SysTaskLog]">
    <parameters>
      <parameter name="p__linq__0" type="Int64"><![CDATA[1841295]]></parameter>
      <parameter name="p__linq__1" type="Int32"><![CDATA[0]]></parameter>
    </parameters><![CDATA[
SELECT TOP (1) 
    [Project1].[SysTaskNum] AS [SysTaskNum], 
    [Project1].[EntryNum] AS [EntryNum], 
    [Project1].[EnteredOn] AS [EnteredOn], 
    [Project1].[IsError] AS [IsError], 
    [Project1].[MsgText] AS [MsgText], 
    [Project1].[SysRevID] AS [SysRevID], 
    [Project1].[SysRowID] AS [SysRowID], 
    [Project1].[MsgType] AS [MsgType]
    FROM ( SELECT 
        [Extent1].[SysTaskNum] AS [SysTaskNum], 
        [Extent1].[EntryNum] AS [EntryNum], 
        [Extent1].[EnteredOn] AS [EnteredOn], 
        [Extent1].[IsError] AS [IsError], 
        [Extent1].[MsgText] AS [MsgText], 
        [Extent1].[SysRevID] AS [SysRevID], 
        [Extent1].[SysRowID] AS [SysRowID], 
        [Extent1].[MsgType] AS [MsgType]
        FROM [Ice].[SysTaskLog] AS [Extent1]
        WHERE ([Extent1].[SysTaskNum] = @p__linq__0) AND ([Extent1].[EntryNum] > @p__linq__1)
    )  AS [Project1]
    ORDER BY [Project1].[EntryNum] DESC, [Project1].[SysTaskNum] DESC
]]></EpiCommand>
  <DBStatement type="SysTaskLog" duration="24.6221" rowCount="0" hashCode="-1111364414381793029" dbContextId="6adc2d6d-60e3-498d-baee-5b5451649ef3">
    <stackTrace>
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler.InvokeSingle(Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs" line="0" />
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler+&lt;&gt;c__DisplayClass34_0`4[TContext,TArg1,TArg2,TResult].&lt;Compile&gt;b__0(TContext context, TArg1 arg1, TArg2 arg2)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs" line="1110" />
      <stackFrame method="Void Ice.Core.WriteTaskLog.WriteSysTaskLogMessage(IceContext context, Int64 taskNum, Boolean appendMessage, Boolean isError, String newMessage, MsgType messsageType)" module="Ice.Core.TaskBase.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Internal\Lib\TaskLib\TaskBase\WriteTaskLog\WriteTaskLog.cs" line="145" />
      <stackFrame method="Void Ice.Services.BO.SysTaskSvc.WriteToTaskLog(String NewMessage, Int64 TaskNum)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTask.cs" line="330" />
      <stackFrame method="Void Ice.Services.BO.SysTaskSvc.SetSysTaskStatusForPendingTask(Int64 sysTaskNum, String runProcedure, String taskStatus, String messageToLog)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTask.cs" line="304" />
      <stackFrame method="Void Ice.Services.BO.SysTaskSvcFacade.SetSysTaskStatusForPendingTask(Int64 sysTaskNum, String runProcedure, String taskStatus, String messageToLog)" module="Ice.Services.BO.SysTask.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Server\Services\BO\SysTask\SysTaskSvcFacade.cs" line="248" />
      <stackFrame method="Object .SyncInvokeSetSysTaskStatusForPendingTask(Object , Object[] , Object[] )" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Object System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Object Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs" line="59" />
      <stackFrame method="Object Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs" line="28" />
      <stackFrame method="Object Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs" line="23" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[TChannel,TItem,UChannel].InnerTryReceiveCompletedCallback(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.TransportDuplexSessionChannel+TryReceiveAsyncResult.OnReceive(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.OnReceiveComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Net.LazyAsyncResult.Complete(IntPtr userToken)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)" module="System.dll" file="" line="0" />
      <stackFrame method="Int32 System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)" module="System.dll" file="" line="0" />
      <stackFrame method="Boolean System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.ConnectionStream+IOAsyncResult.OnAsyncIOComplete(Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)" module="System.dll" file="" line="0" />
      <stackFrame method="Void System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)" module="CommonLanguageRuntimeLibrary" file="" line="0" />
    </stackTrace>
  </DBStatement>
  <ContextBound name="Ice.Triggers.Default.CreateTrigger`2[[Ice.IceDataContext, Epicor.System, Version=3.2.300.0, Culture=neutral, PublicKeyToken=37a5ccb872c00aec],[Ice.Tables.SysTaskLog, Ice.Data.Model, Version=3.2.300.0, Culture=neutral, PublicKeyToken=476c49c300052a33]]" contextid="6adc2d6d-60e3-498d-baee-5b5451649ef3" defaultcontext="False" />
  <Trigger table="SysTaskLog" type="Added" pk="SysTaskLog&lt;~&gt;0&lt;~&gt;0" rowId="e82209d7-e4a1-4d22-a03b-24a25a7c6d32" duration="0.0115" />
  <Trigger table="SysTaskLog" type="Added" pk="SysTaskLog&lt;~&gt;1841295&lt;~&gt;1" rowId="e82209d7-e4a1-4d22-a03b-24a25a7c6d32" duration="0" comment="Trigger already queued to run" />
  <EpiCommand queryType="UPDATE,SELECT" spid="659" table="[Ice].[SysTask],[Ice].[SysTask]">
    <parameters>
      <parameter name="@0" type="DateTime2"><![CDATA[1/21/2020 2:45:15 PM]]></parameter>
      <parameter name="@1" type="String"><![CDATA[Error]]></parameter>
      <parameter name="@2" type="Boolean"><![CDATA[True]]></parameter>
      <parameter name="@3" type="DateTime2"><![CDATA[1/21/2020 2:45:15 PM]]></parameter>
      <parameter name="@4" type="Guid"><![CDATA[8ee60b28-d257-4da8-aa79-7f877f76cc27]]></parameter>
    </parameters><![CDATA[
UPDATE [Ice].[SysTask]
SET [EndedOn] = @0, [TaskStatus] = @1, [History] = @2, [LastActivityOn] = @3
WHERE ([SysRowID] = @4)
SELECT [SysRevID]
FROM [Ice].[SysTask]
WHERE @@ROWCOUNT > 0 AND [SysRowID] = @4
]]></EpiCommand>
  <EpiCommand queryType="INSERT,SELECT" spid="659" table="[Ice].[SysTaskLog],[Ice].[SysTaskLog]">
    <parameters>
      <parameter name="@0" type="Int64"><![CDATA[1841295]]></parameter>
      <parameter name="@1" type="Int32"><![CDATA[1]]></parameter>
      <parameter name="@2" type="DateTime2"><![CDATA[1/21/2020 2:45:16 PM]]></parameter>
      <parameter name="@3" type="Boolean"><![CDATA[False]]></parameter>
      <parameter name="@4" type="String"><![CDATA[EpicorLive10_R2: A communication error occurred trying to run task ID 1841295 for agent SystemTaskAgent on the application server (User: nrosenbaum, Task Description: CSG Print Commissions Documents).  If this continues to happen investigate if you need to increase the receive and send timeouts in your web.config.
Error Detail:
The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '1.00:00:00'.
Stack Trace:

Server stack trace: 
   at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at Sys
tem.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityDuplexChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Ice.Contracts.RunTaskSvcContract.RunTask(Int64 ipTaskNu]]></parameter>
      <parameter name="@5" type="Guid"><![CDATA[e82209d7-e4a1-4d22-a03b-24a25a7c6d32]]></parameter>
      <parameter name="@6" type="String"><![CDATA[ServerInfo]]></parameter>
    </parameters><![CDATA[
INSERT [Ice].[SysTaskLog]([SysTaskNum], [EntryNum], [EnteredOn], [IsError], [MsgText], [SysRowID], [MsgType])
VALUES (@0, @1, @2, @3, @4, @5, @6)
SELECT [SysRevID]
FROM [Ice].[SysTaskLog]
WHERE @@ROWCOUNT > 0 AND [SysRowID] = @5
]]></EpiCommand>
  <ContextBound name="Ice.Triggers.Default.WriteTrigger`2[[Ice.IceDataContext, Epicor.System, Version=3.2.300.0, Culture=neutral, PublicKeyToken=37a5ccb872c00aec],[Ice.Tables.SysTaskLog, Ice.Data.Model, Version=3.2.300.0, Culture=neutral, PublicKeyToken=476c49c300052a33]]" contextid="6adc2d6d-60e3-498d-baee-5b5451649ef3" defaultcontext="False" />
  <Trigger table="SysTaskLog" type="Modified" pk="SysTaskLog&lt;~&gt;0&lt;~&gt;0" rowId="5c39ea1c-4dc0-4fba-bc09-091ff5de5d9d" duration="0.0844" runningTriggers="Added SysTaskLog e82209d7-e4a1-4d22-a03b-24a25a7c6d32 --- " />
  <ContextBound name="Ice.Triggers.SysTask.WriteTrigger" contextid="6adc2d6d-60e3-498d-baee-5b5451649ef3" defaultcontext="False" />
  <Trigger table="SysTask" type="Modified" pk="SysTask&lt;~&gt;1841295" rowId="8ee60b28-d257-4da8-aa79-7f877f76cc27" duration="0.0486" runningTriggers="Modified SysTask 8ee60b28-d257-4da8-aa79-7f877f76cc27 --- " />
  <Session msg="Skipping session save, no updates: c3d510dc-7ea2-47bc-a4da-e9bdca796687" />
  <EpiConnection oper="Disposed" spid="659" />
  <Expression hc="-7699591423694899699"><![CDATA[(ctx, userID_ex) => Queryable.FirstOrDefault(Queryable.Where(
  ctx.SysUserFile,
  row => row.UserID == userID_ex
))]]></Expression>
  <Expression hc="-3638886854476197823"><![CDATA[(ctx, history_ex, runProcedure_ex, sysTaskNum_ex) => Queryable.FirstOrDefault(Queryable.Where(
  WithExtension.With(
    ctx.SysTask,
    UpdLock
  ),
  row => row.History == history_ex && row.RunProcedure == runProcedure_ex && row.SysTaskNum == sysTaskNum_ex
))]]></Expression>
  <Expression hc="-1111364414381793029"><![CDATA[(ctx, Instance_TaskNum, _EntryNum) => Queryable.LastOrDefault(Queryable.Where(
  ctx.SysTaskLog,
  row => row.SysTaskNum == Instance_TaskNum && row.EntryNum > _EntryNum
))]]></Expression>
  <Expression hc="1436528654627131299"><![CDATA[(ctx, secCode_ex, tenantID_ex) => Queryable.FirstOrDefault(Queryable.Select(
  Queryable.Where(
    Queryable.Join(
      ctx.Security,
      ctx.SysCompany,
      row => row.Company,
      company => company.Company,
      (row, company) => new <>f__AnonymousType16<Security,SysCompany>(
        row,
        company
      )
    ),
    <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.company.TenantID == tenantID_ex && <>h__TransparentIdentifier0.row.SecCode == secCode_ex && <>h__TransparentIdentifier0.row.CompanyVisibility == 5
  ),
  <>h__TransparentIdentifier0 => <>h__TransparentIdentifier0.row
))]]></Expression>
  <Expression hc="3572190417463890142"><![CDATA[(ctx, secCode_ex) => Queryable.FirstOrDefault(Queryable.Where(
  ctx.Security,
  row => row.SecCode == secCode_ex && row.CompanyVisibility == 10
))]]></Expression>
  <Expression hc="3893034440758979422"><![CDATA[(ctx, secCode_ex, company_ex) => Queryable.FirstOrDefault(Queryable.Where(
  ctx.Security,
  row => row.Company == company_ex && row.SecCode == secCode_ex && row.CompanyVisibility == 0
))]]></Expression>
  <Expression hc="5516295100864980672"><![CDATA[(ctx, key_ex) => Queryable.FirstOrDefault(Queryable.Where(
  ctx.SysUserComp,
  row => row.Company == key_ex.CompanyID && row.UserID == key_ex.UserID
))]]></Expression>
  <Expression hc="6967163624834185525"><![CDATA[(ctx, company_ex, sysFunction_ex) => Queryable.Any(Queryable.Where(
  ctx.SysLock,
  row => row.Company == company_ex && row.SysFunction == sysFunction_ex
))]]></Expression>
</Op>
<Security msg="PrimaryIdentity name: dsmgr AuthenticationType: UsernameValidator" machine="84799-E10-R1" pid="10588" tid="48" />
<AuthorizedUser msg="dsmgr" machine="84799-E10-R1" pid="10588" tid="48" />
<FileSystemReader msg="Loading assembly System.Collections.Immutable.dll from folder e:\epicor\websites\EpicorLive10\Server\Assemblies\" machine="84799-E10-R1" pid="10588" tid="46" />
<AssemblyResolver msg="Assembly System.Collections.Immutable is added to assembly cache" machine="84799-E10-R1" pid="10588" tid="46" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="46" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="47" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="9" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="46" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="50" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" />
<DataModelLoading msg="contextType=ErpContext" machine="84799-E10-R1" pid="10588" tid="48" /><Op Utc="2020-01-21T14:45:16.1098092Z" act="Ice:Lib:SessionMod/SessionModSvcContract/SetCompany" dur="213.6496" cli="10.89.20.104:49494" usr="dsmgr" machine="84799-E10-R1" pid="10588" tid="48">
  <Sql queries="10" cacheHits="1" time="96.83" qryTypeCount="9" />
  <SessionSetting msg="Deserialized ErpSessionSettings. AppServerURL =  : ClientComputerName =  : ClientDateFormat =  : ClientTerminalID = 0 : CompanyID = CONS : CompanyName = Consolidation : CountryCode =  : CountryGroupCode =  : FormatCultureName = en-US : DesignMode = False : ExternalSecurityToken =  : InstallationID = 3d93c47b-fce3-4274-b3bd-09d411d002de : LanguageID =  : LastAccessed = 1/21/2020 2:45:00 PM : LicenseExpirationTime = 1/21/2020 3:00:00 PM : MobileConnect = False : PlantID = 000 : ProductCode =  : ProductName =  : SessionID = e7d495cc-09fc-4881-9d1f-6f00b6686ef4 : SessionType = 00000003-b615-4300-957b-34956697f040 : SolutionID =  : TaskClientID =  : Timeout = 900 : TimezoneOffset =  : UserID = dsmgr : CurrencyInformation = " />
  <SessionSetting msg="Deserialized ErpSessionSettings. DefaultUOM =  : EmployeeID =  : WorkstationDescription =  : WorkstationID = " />
  <Session msg="Session found in Cache: e7d495cc-09fc-4881-9d1f-6f00b6686ef4" />
  <Session msg="Restoring existing Session from cache: e7d495cc-09fc-4881-9d1f-6f00b6686ef4 for User:dsmgr in Company:CONS" />
  <DataContext msg="New DataContext ID:0d3deac6-737a-4c2f-83c5-951d54f53dd8 created on SPID:" />
  <EpiConnection oper="Opened" spid="659" />
  <EFQueryCache count="38" />
  <EpiCommand hashCode="3893034440758979422" queryType="SELECT" spid="659" table="[Ice].[Security]">
    <parameters>
      <parameter name="p__linq__0" type="String"><![CDATA[CONS]]></parameter>
      <parameter name="p__linq__1" type="String"><![CDATA[Lib.SessionMod.SetCompany]]></parameter>
    </parameters><![CDATA[
SELECT TOP (1) 
    [Extent1].[Company] AS [Company], 
    [Extent1].[SecCode] AS [SecCode], 
    [Extent1].[EntryList] AS [EntryList], 
    [Extent1].[NoEntryList] AS [NoEntryList], 
    [Extent1].[SecurityMgr] AS [SecurityMgr], 
    [Extent1].[Description] AS [Description], 
    [Extent1].[ParentSecCode] AS [ParentSecCode], 
    [Extent1].[ParentSeqNum] AS [ParentSeqNum], 
    [Extent1].[Disconnected] AS [Disconnected], 
    [Extent1].[ShipRecv] AS [ShipRecv], 
    [Extent1].[ShopSupervisor] AS [ShopSupervisor], 
    [Extent1].[MaterialHandler] AS [MaterialHandler], 
    [Extent1].[ProductionWorker] AS [ProductionWorker], 
    [Extent1].[ServTech] AS [ServTech], 
    [Extent1].[CustWebAvailable] AS [CustWebAvailable], 
    [Extent1].[CGCCode] AS [CGCCode], 
    [Extent1].[SystemCode] AS [SystemCode], 
    [Extent1].[SystemFlag] AS [SystemFlag], 
    [Extent1].[SysRevID] AS [SysRevID], 
    [Extent1].[SysRowID] AS [SysRowID], 
    [Extent1].[GlobalSecurityMgr] AS [GlobalSecurityMgr], 
    [Extent1].[CompanyVisibility] AS [CompanyVisibility]
    FROM [Ice].[Security] AS [Extent1]
    WHERE ([Extent1].[Company] = @p__linq__0) AND ([Extent1].[SecCode] = @p__linq__1) AND (0 = [Extent1].[CompanyVisibility])
]]></EpiCommand>
  <DBStatement type="Security" duration="0.8708" rowCount="0" hashCode="3893034440758979422" dbContextId="0d3deac6-737a-4c2f-83c5-951d54f53dd8">
    <stackTrace>
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler.InvokeSingle(Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs" line="0" />
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler+&lt;&gt;c__DisplayClass34_0`4[TContext,TArg1,TArg2,TResult].&lt;Compile&gt;b__0(TContext context, TArg1 arg1, TArg2 arg2)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs" line="1110" />
      <stackFrame method="SecurityTableItem Ice.TableCache.SecurityTableCache.GetByKey(String companyID, String secCode, DataScope companyVisibility)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Cache\TableCache\SecurityTableCache.cs" line="164" />
      <stackFrame method="SecurityTableItem Ice.TableCache.SecurityTableCache.GetByKey(String companyID, String secCode)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Cache\TableCache\SecurityTableCache.cs" line="116" />
      <stackFrame method="Boolean Ice.Manager.Security.CheckSecurityAccess(String secCode, UserFileItem user, String companyID)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Manager\Security.cs" line="306" />
      <stackFrame method="Boolean Ice.Security.AuthorizationManager.CheckAccess(OperationContext operationContext, Message&amp; message)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Security\AuthorizationManager.cs" line="111" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[TChannel,TItem,UChannel].InnerTryReceiveCompletedCallback(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1+AsyncQueueReader[T].Set(Item item)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1[T].EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1[T].EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SingletonChannelAcceptor`3[ChannelInterfaceType,TChannel,QueueItemType].Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.HttpPipeline+EnqueueMessageAsyncResult.CompleteParseAndEnqueue(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Channels.HttpPipeline+EnqueueMessageAsyncResult.HandleParseIncomingMessage(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="IAsyncResult System.ServiceModel.Channels.HttpPipeline+EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="AsyncCompletionResult System.ServiceModel.Channels.HttpChannelListener`1+HttpContextReceivedAsyncResult`1[TChannel,TListenerChannel].ProcessHttpContextAsync()" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="IAsyncResult System.ServiceModel.Channels.HttpChannelListener`1[TChannel].BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)" module="CommonLanguageRuntimeLibrary" file="" line="0" />
    </stackTrace>
  </DBStatement>
  <EpiConnection oper="Disposed" spid="659" />
  <DataContext msg="New DataContext ID:1a899b33-b4a9-4258-b8a6-b5466b3c216c created on SPID:" />
  <EpiConnection oper="Opened" spid="659" />
  <EFQueryCache count="38" />
  <EpiCommand hashCode="1436528654627131299" queryType="SELECT,SELECT,SELECT" spid="659" table="[Ice].[Security],[Ice].[SysCompany_UD],[Ice].[SysCompany]">
    <parameters>
      <parameter name="p__linq__0" type="String"><![CDATA[]]></parameter>
      <parameter name="p__linq__1" type="String"><![CDATA[Lib.SessionMod.SetCompany]]></parameter>
    </parameters><![CDATA[
SELECT 
    [Limit1].[Company] AS [Company], 
    [Limit1].[SecCode] AS [SecCode], 
    [Limit1].[EntryList] AS [EntryList], 
    [Limit1].[NoEntryList] AS [NoEntryList], 
    [Limit1].[SecurityMgr] AS [SecurityMgr], 
    [Limit1].[Description] AS [Description], 
    [Limit1].[ParentSecCode] AS [ParentSecCode], 
    [Limit1].[ParentSeqNum] AS [ParentSeqNum], 
    [Limit1].[Disconnected] AS [Disconnected], 
    [Limit1].[ShipRecv] AS [ShipRecv], 
    [Limit1].[ShopSupervisor] AS [ShopSupervisor], 
    [Limit1].[MaterialHandler] AS [MaterialHandler], 
    [Limit1].[ProductionWorker] AS [ProductionWorker], 
    [Limit1].[ServTech] AS [ServTech], 
    [Limit1].[CustWebAvailable] AS [CustWebAvailable], 
    [Limit1].[CGCCode] AS [CGCCode], 
    [Limit1].[SystemCode] AS [SystemCode], 
    [Limit1].[SystemFlag] AS [SystemFlag], 
    [Limit1].[SysRevID] AS [SysRevID], 
    [Limit1].[SysRowID] AS [SysRowID], 
    [Limit1].[GlobalSecurityMgr] AS [GlobalSecurityMgr], 
    [Limit1].[CompanyVisibility] AS [CompanyVisibility]
    FROM ( SELECT TOP (1) 
        [Extent1].[Company] AS [Company], 
        [Extent1].[SecCode] AS [SecCode], 
        [Extent1].[EntryList] AS [EntryList], 
        [Extent1].[NoEntryList] AS [NoEntryList], 
        [Extent1].[SecurityMgr] AS [SecurityMgr], 
        [Extent1].[Description] AS [Description], 
        [Extent1].[ParentSecCode] AS [ParentSecCode], 
        [Extent1].[ParentSeqNum] AS [ParentSeqNum], 
        [Extent1].[Disconnected] AS [Disconnected], 
        [Extent1].[ShipRecv] AS [ShipRecv], 
        [Extent1].[ShopSupervisor] AS [ShopSupervisor], 
        [Extent1].[MaterialHandler] AS [MaterialHandler], 
        [Extent1].[ProductionWorker] AS [ProductionWorker], 
        [Extent1].[ServTech] AS [ServTech], 
        [Extent1].[CustWebAvailable] AS [CustWebAvailable], 
        [Extent1].[CGCCode] AS [CGCCode], 
        [Extent1].[SystemCode] AS [SystemCode], 
        [Extent1].[SystemFlag] AS [SystemFlag], 
        [Extent1].[SysRevID] AS [SysRevID], 
        [Extent1].[SysRowID] AS [SysRowID], 
        [Extent1].[GlobalSecurityMgr] AS [GlobalSecurityMgr], 
        [Extent1].[CompanyVisibility] AS [CompanyVisibility]
        FROM  [Ice].[Security] AS [Extent1]
        INNER JOIN  (SELECT [Extent3].[Company] AS [Company], [Extent3].[TenantID] AS [TenantID]
            FROM  [Ice].[SysCompany_UD] AS [Extent2]
            INNER JOIN [Ice].[SysCompany] AS [Extent3] ON [Extent2].[ForeignSysRowID] = [Extent3].[SysRowID] ) AS [Join1] ON [Extent1].[Company] = [Join1].[Company]
        WHERE ([Join1].[TenantID] = @p__linq__0) AND ([Extent1].[SecCode] = @p__linq__1) AND (5 = [Extent1].[CompanyVisibility])
    )  AS [Limit1]
]]></EpiCommand>
  <DBStatement type="Security" duration="1.1916" rowCount="0" hashCode="1436528654627131299" dbContextId="1a899b33-b4a9-4258-b8a6-b5466b3c216c">
    <stackTrace>
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler.InvokeSingle(Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs" line="0" />
      <stackFrame method="TResult Epicor.Data.DBExpressionCompiler+&lt;&gt;c__DisplayClass34_0`4[TContext,TArg1,TArg2,TResult].&lt;Compile&gt;b__0(TContext context, TArg1 arg1, TArg2 arg2)" module="Epicor.System.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs" line="1110" />
      <stackFrame method="SecurityTableItem Ice.TableCache.SecurityTableCache.GetByKey(String companyID, String secCode, DataScope companyVisibility)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Cache\TableCache\SecurityTableCache.cs" line="167" />
      <stackFrame method="SecurityTableItem Ice.TableCache.SecurityTableCache.GetByKey(String companyID, String secCode)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Cache\TableCache\SecurityTableCache.cs" line="122" />
      <stackFrame method="Boolean Ice.Manager.Security.CheckSecurityAccess(String secCode, UserFileItem user, String companyID)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Manager\Security.cs" line="306" />
      <stackFrame method="Boolean Ice.Security.AuthorizationManager.CheckAccess(OperationContext operationContext, Message&amp; message)" module="Epicor.Ice.dll" file="C:\_Releases\ICE\ICE3.2.300.12\Source\Framework\Epicor.Ice\Security\AuthorizationManager.cs" line="111" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.AuthorizationBehavior.Authorize(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc&amp; rpc)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SecurityChannelListener`1+ReceiveItemAndVerifySecurityAsyncResult`2[TChannel,TItem,UChannel].InnerTryReceiveCompletedCallback(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1+AsyncQueueReader[T].Set(Item item)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1[T].EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.InputQueue`1[T].EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.SingletonChannelAcceptor`3[ChannelInterfaceType,TChannel,QueueItemType].Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Channels.HttpPipeline+EnqueueMessageAsyncResult.CompleteParseAndEnqueue(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.ServiceModel.Channels.HttpPipeline+EnqueueMessageAsyncResult.HandleParseIncomingMessage(IAsyncResult result)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Boolean System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="IAsyncResult System.ServiceModel.Channels.HttpPipeline+EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="AsyncCompletionResult System.ServiceModel.Channels.HttpChannelListener`1+HttpContextReceivedAsyncResult`1[TChannel,TListenerChannel].ProcessHttpContextAsync()" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="IAsyncResult System.ServiceModel.Channels.HttpChannelListener`1[TChannel].BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)" module="System.ServiceModel.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)" module="System.ServiceModel.Activation.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)" module="System.ServiceModel.Internals.dll" file="" line="0" />
      <stackFrame method="Void System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)" module="CommonLanguageRuntimeLibrary" file="" line="0" />
    </stackTrace>
  </DBStatement>
  <EpiConnection oper="Disposed" spid="659" />

ERPSysAdmin · January 21, 2020, 6:05pm

I have seen a “GHOST” process in our System Monitor a few times and we are on 10.1.600. One time it was MRP which showed as “ACTIVE” for days but I couldn’t cancel/delete it from the System Monitor and I tried stopping the Task Agent, Recycling AND Stopping and Re-Starting the AppServer and even rebooting the Server but I couldn’t remove it. So I submitted a ticket to Epicor and they told me to Stop the Task Agent(s) and Delete everything from the Ice.SysTask table. I believe it is okay to delete the records in this table when the Task Agent is down because it contains old records of Processes that are submitted to the System Monitor\Task Agent.

Here are the instructions the Epicor tech sent me:

I try to be very careful running this but here are a few SQL programs I keep handy because I’ve had to use them a few (maybe 4) times:

Maybe a clean-up of your Ice.SysTask table will help? I’m not sure where you can begin to figure out why you are getting these “GHOST” processes if you don’t get good logs. I think ours have mainly been caused by our AppServer memory usage suddenly growing out of control and no choice but to Recycle the AppServer during the time that the (MRP) process was running. Maybe you are having some connection issues or maybe there are some Timeout Settings that you can adjust? Just thinking out loud… it is interesting to me that it started running and logged an entry after an hour and a half.

josecgomez · January 21, 2020, 6:43pm

So, After much more digging through endless logs I found this

Notice the RowCount vs TimeDataRetrieval and TimeProcessing… It is taking an hour or more to retrieve 300 records from the Intermediary Database. However the plot thickens because once I restart the
SSRS Report Service, I can run the same report with the exact same number / amount of data in seconds…

So it appears something is happening that is causing SSRS Report Service to… get bogged down and then it can’t figure out how to perform? Any ideas?

ERPSysAdmin · January 21, 2020, 7:40pm

Do you think you have some “parameter sniffing” problems with that report? And maybe when you restart the SSRS Report Service it clears out the Query Execution Plan Cache and starts with a new plan and it runs faster.

When a query is sometimes fast and sometimes slow, for the same input parameters, and you swear nothing else in the environment is changing, that’s often a case of parameter sniffing.

Brent Ozar explains:

josecgomez · January 21, 2020, 7:45pm

I doubt it, remember there are new tables created for every report run

Table_GUID so any given query will be distinctly and drastically different from one another… (I think)

Good idea though, I’ll do some reading on it just in case.

JeffLeBert · January 21, 2020, 11:20pm

Did you know SSRS keeps some very nice logs of what it is doing? Basically there is an ExecutionLog3 SQL view that you can query that tells you all kinds of stuff. I’ve been able to figure out SSRS specific issues from that in the past. You get information like what query was run, how long it took and now many rows it returned. It’s been a while since I’ve played with this but it is the next place I would go and is already on and available to you!

https://docs.microsoft.com/en-us/sql/reporting-services/report-server/reporting-services-log-files-and-sources?view=sql-server-ver15

josecgomez · January 22, 2020, 1:36am

Yes @JeffLeBert! That’s how I figured out the above DataAccess time vs render time etc

What I seem to not be able to find an explanation for is why the report with the same exact data will behave one way now and a different way later ( gets worse with use, I’m leaning toward a memory leak in SSRS itself ? )

JeffLeBert · January 22, 2020, 1:45am

Sorry, missed the obvious screen shots. In any case, it’s good to get the link to SSRS logging again.

You can get memory leaks in SSRS. The easiest way is to create the issue in the report’s VB code. Look for shared (same as static in C#) fields. I would think you would have to hold on to a LOT of data or call the report a lot of times to get anything noticeable. Still possible.

josecgomez · January 22, 2020, 1:57am

This report when run with full data is several hundred pages (don’t ask why I lost that battle)

hkeric.wci · January 22, 2020, 2:00am

The only time I have these issues, yet the SSRS Log shows things running fast, is when:

We have issues with our Disks (so once we moved it to new SSDs and it went down from minutes to seconds
When someone clicks Sync Dataset

You are on the cloud, perhaps Epicors Cloud Disks are being pushed.

By the way did someone forget to set the Ram Limit on Report Server DB? SQL Likes to eat all the ram when it can. Can you check the Database Max Memory Settings etc? (thinking about how it was fast again after you restarted)

Also I wonder what your Main SSRS Server Log Shows, I know we had a report throw like 500 errors similar to:

Hally · July 10, 2020, 7:11am

While my comment might be of little relevance, I was experiencing a similar problem and couldn’t understand why when I opened system monitor and used the show all tasks option I was not seeing everything.

I tried doing a show grid filters, because there was a lot of data and when I enabled it, I noticed that there was another filter enabled on another column. Turning it off showed all the missing rows!

I must have had a filter enabled at some stage, but on reopening the system monitor the row header did not indicate that the grid filters were set…Been bugging me for ages.

A bit more of a test on this and it looks like if you set grid filters and save layouts, close out of Epicor, log back in, open system monitor any indication of the grid filters are gone but behind the scenes the filter is still applied. It’s not until you click on grid filters do you see the existing filter.