Anyone else experiencing sporadic clock in/job in/out times with MES?

After the 2024.1.15 update, we’ve started noticing sporadic times when shop floor personnel clock in or job in/out via MES. Has anyone else noticed this? Sometimes there appears to be a rhyme or reason (+5 hours, +4 hours), but other times the offset is something else. This sometimes occurs within the same session (e.g. clock-in to shift via MES is right, then start activity is wrong). It’s almost like the requests are being sent to different servers in different time zones. Obviously, this will prove to be a pain for payroll and our production activities will have crazy durations during this period. Looking at the LaborHed table directly, all of the incorrect rows have all time related fields set to the wrong value, so I’m unsure if there’s a creative way to correct these in bulk. I’m open to suggestions :slight_smile:

We opened and escalated an EpicCare case on Monday but have yet to hear from anyone.

Was there a change in .15 that requires some sort of config change I’m not aware of?

[edit]Or could this be some lingering effect of the Kinetic outage Sunday evening?[/edit]

2 Likes

I haven’t checked this in particular but we are experiencing wide ranging issues with timestamps throughout the system. Trying to figure out a pattern.

What’s worse is that we are seeing times for creating lots that are in the same minute of each other, two will be 5 hours off, and one will be correct.

This is going to be a major headache.

I would ask that others look around at their timestamps and see how wide ranging this is.

4 Likes

when I see 4 or 5 hour differences, I smell a Time Zone issue. All Browser MES or Classic MES, or a mixture?

3 Likes

Interesting thought. Could be difference between local and your cloud datacenter time?

1 Like

We’re still 100% classic MES. The offset is not reliable however, so if it is the difference between local and cloud, it’s bouncing between several timezones.

2 Likes

Yeah, Epicor is hosted in Azure Central, which is 5 hours (CST) from GMT.

Just wondering if some code running on the server is recognizing the TZ, but other code is just using GMT. It’s a thought. :person_shrugging:

And time zone is changing this weekend! Woot!

2 Likes

It’s a thought, but I believe we are pulling server time, and it’s not isolated to just us creating parts.

My silo reporting has half of them correct, the other is 5 hours off, and they are both being reported from the same server, and the same process/function.

1 Like

I think they need to check the times on their app servers.

I just checked our silo reporting process, I am recording both the local time, and the server time, and one batch on both is 5 hours ahead, while the other is correct.

Edit: To be clear:

Silos 1-15 → Local time & Server Time Correct
Silos 16-27 → Local time & Server Time +5 hours

Reported from same server, approximately a minute from each other.

2 Likes

Is the clock in/out minute messed up the same way?

Clock in / out displayed times are redundant to LaborDtl ClockInMinute / ClockOutMinute, aka
dateadd(minute, LaborDtl.ClockOutMinute, cast('19531030' as datetime)
… Unless your particular server records that at the server’s timezone instead of yours, in which case something like
datediff(minute, '19531030', cast(getutcdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Pacific Standard Time' as datetime)

1 Like

My ticket:

CS0004621204 - Time off Inconsistently by time zone offset (+5 hours in the future for us) from same processes.

1 Like

Plot thickens, at least with one part.

On my silo stuff, the dates that seem to be not adjusted come in as

2024-10-30T10:30:50

But this, 2024-10-30T10:39:42.0267281+00:00

Turned into this 2024-10-30T15:39:42.0267281+00:00


That doesn’t help much with my lot errors though, as they should all be the exact same format coming in.

2 Likes

ISO date problem back?

3 Likes

Mannnnnn I just want to upgrade already haha, first elevated licenses, bad tokens, now timestamps are all over the place?

Seems to be a cloud issue though from what I understand? Anyone on prem having the same issues?

2 Likes

We got the .15 update this past weekend and are seeing odd times in the system monitor. Reports are showing created 4 hours in the future (we are est) and some Last Actions are correct but not all.

3 Likes

Not sure if it’s related, but CEDT (Europe) ended last Sunday, 27 October 2024.

Guess Big Sky GIF by ABC Network

This was sent in the same batch, but 8 different bpm calls.
4 are correct, 4 are +5 hours.

PCID Server MfgTime Local Date String Custom Local Date String ISO Sent As String Server Mfg Date Local Parsed Date Server Mfg Seconds Local Seconds
5161019 10/30/2024 6:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:14 10/30/2024 10/30/2024 67816 49813
5161019 10/30/2024 6:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:17 10/30/2024 10/30/2024 67818 49813
5161019 10/30/2024 6:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:19 10/30/2024 10/30/2024 67820 49813
5161019 10/30/2024 1:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:21 10/30/2024 10/30/2024 49822 49813
5161019 10/30/2024 1:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:23 10/30/2024 10/30/2024 49824 49813
5161019 10/30/2024 1:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:24 10/30/2024 10/30/2024 49826 49813
5161019 10/30/2024 1:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:26 10/30/2024 10/30/2024 49827 49813
5161019 10/30/2024 6:50 PM 2024/10/DD_13:50:13.211 10/30/2024 1:50:13 PM 10/30/2024 13:50:28 10/30/2024 10/30/2024 67829 49813

Edit: y’all ignore my custom date string, I screwed it up, it’s just debug / testing data

3 Likes

Anyone know if I can check the id or something of which appserver it’s running on?
Trying to see if there is some round robin stuff going on.

Not sure how to tell the appserver. I’m part of the Cloud portal beta progam and only have access to a Start and Stop task agent button. (not for Live yet) No mention of any IDs.

I’m not seeing the same issue in Pilot or Addtional DBs so you may be onto something.

Try this.

Environment.MachineName;
3 Likes

I will in a minute, I just remembered, I have a portal, and I didn’t do the one thing I always tell my users…

Reboot GIF by Schitt's Creek