As yezdi says: Set up NTP within your ESXi servers.
I suggest you use the public NTP server 'cloud' of ntp.org - e.g. 0.de.pool.ntp.org, 1.de.pool.ntp.org etc.pp.
One more thing: I was having issues synchronizing against our LANCOM firewalls. this simply wouldn't work. Our Linux VMs showed the same behaviour.
So make your time sources as diverse as possible. And do not use your windows VMs :-)
HTH
Dirk