We recently noticed that our SQL server is running .NET version 4.7 and our app and SSRS servers are still running 4.6.1. We don’t understand how our SQL server got upgraded but should we run into any issues if we upgrade them all to 4.7?
I would assume not but we don’t know. Epicor says they should all be the same but haven’t told us if being 4.7 is a bad thing.
From a runtime perspective 4.7 SHOULD be backwards compat as per Microsoft’s efforts. There always seems to be something that is broken on an admin API though which is why we go through the certification / verifying / ratifying / choose your term.
I don’t know if we have stated a delivery vehicle for 4.7 but have been looking at it.
So don’t think @aidacra will be happy with you but it may work
The reason i asked is because we are having an issue that Epicor support has been working on for weeks and has been unable to identify what is going on. Our backflush process is failing to process seemingly randmonly on some jobs. We have been unable to identify a pattern or reason. The error we see in the backflush log is: ‘The partner transaction manager has disabled its support for remote/network transactions.’
When we look at the app server event viewer we see this error at the same time:
Query failed to run
The underlying provider failed on EnlistTransaction.
(ctx, company_ex, personID_ex) => ctx.Person.Where(row => ((row.Company == company_ex) AndAlso (row.PersonID == personID_ex))).FirstOrDefault()
We are grasping at straws right now. Any insight would be greatly appreciated. The issue is getting worse and we have been passed from team to team, from person to person at Epicor and they all seem stumped. We are on 10.1.500.31.
That’s definitely a DTC transaction issue. Are you intending on a DTC Transaction with an integration or have a BPM in that area or?
We have the backflush process scheduled to run every 5 min (continuous has never worked). It errors on select jobs even running with BPM turn off (all customizations for that matter). About 10% off all jobs fail to backflush through this process. Even restoring Live to Pilot environment, they fail there too. The only way we get those jobs to backflush is to open them in Job Closing. Just opening them in Job Closing causes them to backflush and it does so without error.
What is worse is that the exception handling on the backflush process does not allow the process to continue around the failing jobs. No backflushing will process until we get the bad ones out of the way and then all 100’s after will run correctly (or at least until it hits another bad one). So we have to monitor it continually or our productivity metrics posted live in manufacturing are all out of whack.
If you have a ticket open with @aidacra, you are half way there. The DTC escalation issue pops up from time to time and is either a coding issue in core product or a BPM. If you disabled the bpms and have given Nathan the stack trace, they should be able to start digging. The apps folks have experts in understanding the issue now - unravelling where it goes off the rails is more entertaining.
Not sure if it has made it to Nathan @aidacra yet. I hope soon. We have been passed all around Epicor for weeks now. We uploaded our DB last week and our case is CS0000747767.
The import part of all of this is that the same jobs fail in our testing environments, they fail with customizations turned off, we have stack traces, and specific error message. Hopefully someone can figure this out.
Reviewed the case: if the analyst currently working on the issue can’t repro the issue with the latest backup of your database provided earlier today, then it would be an issue my team would pursue. I suspect though that the analyst will be able to repro the issue based on the error and the apps development team will ultimately provide the resolution.
You have the stack trace? That should speed up analysis
We will make sure it is in the ticket. This is it:
Program Ice.Services.Lib.RunTask raised an unexpected exception with the following message: RunTask: The underlying provider failed on EnlistTransaction.
Stack Trace:
at System.Data.EntityClient.EntityConnection.EnlistTransaction(Transaction transaction)
at System.Data.Objects.ObjectContext.EnsureConnection()
at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at System.Data.Objects.CompiledQuery.ExecuteQuery[TResult](ObjectContext context, Object[] parameterValues)
at Epicor.Data.DBExpressionCompiler.GetResult[TContext,TQuery,TResult](Func`3 executeQuery, Cache cacheSetting, TContext dataContext, TQuery query) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 441
at Epicor.Data.DBExpressionCompiler.InvokeSingle[TContext,TQuery,TResult](Expression expression, Cache currentCacheSetting, Boolean cacheQuery, TContext dataContext, Func`2 getDataCacheKey, Func`2 compileQuery, Func`3 executeQuery) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\DBExpressionCompiler.cs:line 302
at Epicor.Data.DBExpressionCompiler.<>c__DisplayClass34_0`4.<Compile>b__0(TContext context, TArg1 arg1, TArg2 arg2) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\DBExpressionCompiler.Generated.cs:line 1110
at Erp.Internal.Lib.JobRcpnt.DoPerson(String In_PersonId, Int32 In_AlertNum) in c:\_Releases\ERP\RL10.1.500\Source\Server\Internal\Lib\JobRcpnt\JobRcpnt.cs:line 225
at Erp.Internal.Lib.JobRcpnt._JobRcpnt(String In_JobNum, Int32 In_AlertNum, String In_EmailList, String& RcpntList) in c:\_Releases\ERP\RL10.1.500\Source\Server\Internal\Lib\JobRcpnt\JobRcpnt.cs:line 118
at Erp.Internal.Lib.XAP05.Do_1150(String In_ParmList, Int32 In_AlertNum) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\Shared\XAP05\XAP05.cs:line 1794
at Erp.Internal.Lib.XAP05._XAP05(Int32 In_AlertNum, String In_ParmList, String& VContinue) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\Shared\XAP05\XAP05.cs:line 420
at Erp.Internal.Lib.JobOperWrite._JobOperWrite(JobOper JobOper, JobOper OldJobOper, Boolean RollUp) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\JobOperShared\JobOperWrite\JobOperWrite.i.cs:line 1013
at Erp.Internal.Lib.JobOperShWrite.DoJobOper(JobOper JobOper, JobOper OLDJobOper) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\JobOperShared\JobOperShWrite\JobOperShWrite.cs:line 1298
at Erp.Triggers.JobOper.WriteTrigger.write(JobOper JobOper, JobOper OldJobOper) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Db\Triggers\JobOper\write.cs:line 221
at Ice.Triggers.TriggerQueue.ExecuteWriteTrigger(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 296
at Ice.Triggers.TriggerQueue.RunWriteTriggerInNewLevel(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord, Boolean forAddedRow) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 157
at Ice.Triggers.TriggerQueue.<>c__DisplayClass11_0.<RunWriteTrigger>b__1() in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 148
at Ice.Triggers.TriggerQueue.RunAtNewLevel(Func`1 buildTriggerRunState, Action action) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 501
at Ice.Triggers.TriggerQueue.RunTriggers(IceDataContext context) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 83
at Ice.IceDataContext.RunUntilAllTriggersHaveExecuted() in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\IceDataContext.cs:line 542
at Ice.Triggers.TriggerQueue.RunAtNewLevel(Func`1 buildTriggerRunState, Action action) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 501
at Ice.IceDataContext.Validate[TLinqRow](TLinqRow row) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\IceDataContext.cs:line 299
at Erp.Triggers.LaborDtl.WriteTrigger.Write(LaborDtl LaborDtl, LaborDtl OLaborDtl) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Db\Triggers\LaborDtl\Write.cs:line 1525
at Ice.Triggers.TriggerQueue.ExecuteWriteTrigger(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 296
at Ice.Triggers.TriggerQueue.RunWriteTriggerInNewLevel(IceDataContext context, LinqRow modifiedRecord, LinqRow originalRecord, Boolean forAddedRow) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 157
at Ice.Triggers.TriggerQueue.<>c__DisplayClass11_0.<RunWriteTrigger>b__1() in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 148
at Ice.Triggers.TriggerQueue.RunAtNewLevel(Func`1 buildTriggerRunState, Action action) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 501
at Ice.Triggers.TriggerQueue.RunTriggers(IceDataContext context) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Triggers\TriggerQueue.cs:line 83
at Ice.IceDataContext.RunUntilAllTriggersHaveExecuted() in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\IceDataContext.cs:line 542
at Erp.Internal.Lib.BFLaborD.CreateProduction() in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\BFLaborD\BFLaborD.cs:line 380
at Erp.Internal.Lib.BFLaborD.processLaborDtl(Guid LaborDtlRowid, String& oMessage) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\BFLaborD\BFLaborD.cs:line 220
at Erp.Internal.Lib.BFLaborD._BFLaborD(Guid LaborDtlRowid, String& oMessage) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\Lib\BFLaborD\BFLaborD.cs:line 139
at Erp.Internal.JC.BackflushLaborProcess.RunProcess(Int64 InstanceTaskNum, String OutputFileName) in D:\_Releases\ERP\UD10.1.500.31\Source\Server\Internal\JC\BackflushLaborProcess\BackflushLaborProcess.cs:line 174
at Ice.Hosting.TaskCaller.InnerExecuteTask(IceDataContext newContext) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 93
at Ice.Hosting.TaskCaller.ExecuteTask() in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.Ice\Hosting\TaskCaller\TaskCaller.cs:line 54
at Ice.Lib.RunTask.BpmFriendlyTaskLauncher.Run(String sessionIdPrefix, IceContext db, Action taskRunner) in D:\_Releases\ICE\3.1.500.31\Source\Server\Services\Lib\RunTask\BpmFriendlyTaskLauncher.cs:line 63
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in D:\_Releases\ICE\3.1.500.31\Source\Server\Services\Lib\RunTask\RunTask.cs:line 526
Inner Exception:
The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)
Stack Trace:
at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)
at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)
at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)
at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)
at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)
at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.EnlistTransaction(Transaction transaction)
at System.Data.SqlClient.SqlConnection.EnlistTransaction(Transaction transaction)
at Epicor.Data.Provider.EpiConnection.EnlistTransaction(Transaction transaction) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Data\EpiProvider2\EpiConnection.cs:line 209
at System.Data.EntityClient.EntityConnection.EnlistTransaction(Transaction transaction)
at Ice.Services.Lib.RunTaskSvc.InnerRunTask(Int64 ipTaskNum, Boolean suppressTransaction) in D:\_Releases\ICE\3.1.500.31\Source\Server\Services\Lib\RunTask\RunTask.cs:line 526
at Ice.Services.Lib.RunTaskSvcFacade.RunTask(Int64 ipTaskNum) in D:\_Releases\ICE\3.1.500.31\Source\Server\Services\Lib\RunTask\RunTaskSvcFacade.cs:line 87
at SyncInvokeRunTask(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at Epicor.Hosting.OperationBoundInvoker.InnerInvoke(Object instance, Func`2 func) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 59
at Epicor.Hosting.OperationBoundInvoker.Invoke(Object instance, Func`2 func) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Hosting\OperationBoundInvoker.cs:line 28
at Epicor.Hosting.Wcf.EpiOperationInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) in D:\_Releases\ICE\3.1.500.31\Source\Framework\Epicor.System\Hosting\Wcf\EpiOperationInvoker.cs:line 23
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SecurityChannelListener`1.ReceiveItemAndVerifySecurityAsyncResult`2.InnerTryReceiveCompletedCallback(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
at System.Net.AsyncProtocolRequest.CompleteRequest(Int32 result)
at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)The team currently working on it was not able to repro the issue on Epicor’s end. This has happened before and sucks because we seem to drop into no mans land and are left to fend for ourselves.
We are in desperate need of assistance to get this issue resolved. If there is anything you can do to push this along that would be great.
It’s not no mans land if you have it repro on your side without BPM - it just needs a little encouragement to help it along that you are continuing to provide (I assume).
If it can’t be repro’d within Support, that is more or less the end of the application support triage as they work with the base application.
When we bring a database in-house, my DB Load team executes a tailoring database script to allow it to work within our environment. As of 10.1, the script does the following–one of them could be your X-factor.
WE REMOVE:
WE DISABLE:
WE UPDATE(OR CREATE):
WE COMPLETE AS NEEDED:
EDIT:
We only load databases at the latest point update for the release that the database is currently on. So, as of today, any 10.1.400.x database processed by support would be loaded at 10.1.400.38, for 10.1.600.x that would be 10.1.600.18 (etc).
@aidacra
If application support triage has reached the end of their support, how does our case move on to the next team? Who helps us figure out our X-factor?
Application support’s primary initial objective when testing an issue is to determine if the behavior is intended or not, and if not, if the cause is due to the base application or (generically) “something else”. Based on their testing the cause is not the base application at the latest point update so must be “something else”.
When this happens, we have to determine what could be unique with how a customer is running the application in their environment vs how Support does. The X-factor could be one of the items I listed yesterday regarding the steps a customer database goes through when we load it within Support.
We disabled all BPMs and retested and the result is still the same. We have not tested .33.
Are you unable to test with our custimizations still on? Are you able to test at .31?
Upgrading is ever easy. We run through a whole testing process that can take weeks. Every upgrade we have applied fixed some issues but always introduces new ones. We actually found 2 issues when we migrated from 10.1.500.12 to 10.1.500.31 that have been acknowledged as new bugs and scheduled to be fixed in 10.2.200. 
We aren’t suggesting you upgrade your production to .33, just an environment that isn’t production with a backup of your current live database to determine if this one issue is resolved in the new release.
We upgraded our testing environment to 10.1.500.33 and it still fails. Now what is the next step?
I provided a list of the things that support does when we bring a database in-house. How many of them have you eliminated?
Additionally, as you are working with Support I would recommend you reach out to the analyst you are already working with regarding your testing your findings so they can assist you in determining appropriate next steps if you haven’t done so already.