I’m encountering a strange issue and I get the sense like the solution is right under my nose but I have still been unable to find it.
The situation is as follows:
We migrated Epicor 9 to Epicor 10.2.300 just over a month ago.
Since going live, we have found about two dozen of our customers receive two or three copies of all documents being sent out via APR. It has consistently been the same customers while over a hundred others are fine. Reviewing these customers in Customer maintenance reveals nothing is shared by these problem customers that is unique from the functioning customers.
As I have been investigating deeper I discovered two things which I believe are the key to finding the solution:
the issue does not occur if I use a different email address on the customer record for testing (an internal email or a private external email address).
If I use an external site like MXToolbox to do an smtp test to these customers emails, they all show a “failure” on the smtp transaction time test because they took longer than 15 ms to respond.
We use an internal Exchange server to send out and I can see in the logs that three distinct emails are being generated at exactly 30 seconds apart.
We have validated the Breaking/Routing configuration is good (even support has looked at it) and that there isn’t anything filtering or targeting these specific customers.
In System Monitor I see errors for these emails that send multiple copies. The error reads: An error occurred while performing print routing action Email Report:
The operation has timed out.
I’m suspecting there is a setting in IIS that has a 30 second timeout or something like that which is triggering Epicor to create a second and sometimes third request to send the email, even though it does legitimately get sent out every time. I haven’t been able to find that setting however and I haven’t seen any settings within Epicor itself or the web.config file which would drive this.
I figured I would reach out in case anyone has any ideas or has encountered something like this before. Any ideas would be appreciated.
Not sure why they would resend from Epicor if timed out. I would venture to bet there is something in the Epicor code causing that. I am not aware of any setup within Epicor where you specify email timeout settings and email retries.
Is there any commonality between the customers email address that is getting duplicate emails other than the email takes long to respond? It is interesting that the System Monitor is showing an error for these.
Hi Kristine. We have found no other commonality and if I change the email address for testing, it works fine. I send over two dozen test copies to be sure it wasn’t happening randomly when I first was troubleshooting. (I neglected to say that we did have this stop happening for one customer after we were asked to change the contact information).
Hi. Al yes we break on order number, but this happens with Orders, Packing Slips, Customer Statements and Invoices. The duplication is consistent across all document types so long as the contact is the same.
If I resend the exact same problematic documents to a different email address, they do not duplicate even though I have left the data set unchanged.
Good thought but unfortunately I chased that one down already to find a dead end.
I am guessing you have your answer when you say that it stopped happening for one customer after you changed the contact information to another domain. The print routing looks fine.
Thank you for suggestions everyone. I reached out to get an updated list of examples from our team who send out and monitor the APR communications and I received some more examples that now change the conditions. We have a few customers now who have had duplicates or triplicates sent but only two or three instances out of fifty in the past month. I confirmed the customer record has not been changed since go live. I also BAQ’d the contact email and custom field data and re imported it on a specific customer as a test and that still failed this morning.
@regency-mho - Have you confirmed that the dups only (and always) happen when there is an issue sending the first email?
Does this happen if a single document is sent? The reason for this question is that the receiving email server might not like getting simultaneous multiple emails (addressed for the same user), from an outside domain. Might be some sort of spam detection.
Do the recipients in question use a third party mail filtering service like SkyScan Service?
Hi Calvin, if I understand you correctly, yes I can confirm that we duplicates are sent, there is not a separate “successful” task in System Monitor, nor is there a copy of the email that proceeds the duplicates at an earlier time.
Yes this happens to single documents. We do Order Acknowledgements and Packing Slips individually. We also have seen this happen printing to APR from Invoice Tracker. I was wondering about spam detection myself but each email is sent out (as seen from our email server logs) as an individual email so the only spamming would be the dups.
I do not know what filtering or security service each of the recipients are using. Many of them are small business owners using a third party email service and don’t have a technical team who can provide me with that information.
I would suggest you continue to push Epicor support on. They may not have tried this type of scenario out during development/testing phases. Especially since you see the dups in the system monitor. We send upwards of 100+ AR invoices out via APR every day and I haven’t heard any of our customers receiving duplicate emails. Interesting scenario.
Hi Al, in regards to the result set of the data, we have this for SO ACK, AR Invoice, Packing Slip Confirmations, Customer Statements and RMA.
I reviewed the reportdata for two instances today that happened where three copies were sent out but there is only one line on the invoice. I confirmed there is in fact only one record for each.
We also tried a change yesterday to ARForm to break on Company, CustomerNum and InvoiceNum instead of just InvoiceNum to see if we had any different results, but nothing changed on the batch of invoices going out this morning, for whatever that is worth.
Hi Kristine, to clarify we do not see duplicates in System Monitor. We only see one task entry but with the error message mentioned earlier. I see duplicates in our email logs, from feedback from the customer (obviously) and we also started to cc and internal mailbox to track these activity better so I can see the duplicates there as well.
@regency-mho - My bad. I mis-read your message earlier. Either way you see the error in system monitor and the email log show multiple emails going out. Is there anything written in the server log that would give you a trace that a email request possibly is sent from Epicor multiple times.