thread

This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.

https://www.fz-juelich.de/en/news/archive/press-release/2026/mystery-of-60-second-fluctuations-in-power-grids-solved

It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.

Ok, so this went surprisingly viral. What I learned from the responses so far:

The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.

Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.

Our industry learns and unlearns this phenomenon again and again.

(I personally certainly have experienced this in my line of work, too, but...)

People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.

So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.

@hzulla Yeah, lots of people responsible for distributed systems haven’t been taught the ways of thinking about them.

My work team just deployed a scheduled I/O-heavy job (storage snapshots on a non-CoW filesystem) to about a hundred physical servers. The original plan was to push out exactly the same config to all of them. Everybody understood the problem once I pointed it out, but it’s just not something most people consider.

@hzulla People also tend to schedule things on the hour/half-hour, or at multiples of five minutes, in part because cron syntax makes that easier.

@hzulla In fairness, devs sticking to the minute mark isn't really something you want to fix (because you can't). Instead you have to recognize what causes it and give devs the tools to avoid it by default. Because there's currently not a good way to express "once a minute, regardless of where in the period" on many platforms

@hzulla One does however wonder about the effect of switching domestic scale consumption/generation on the half hour when (in the UK) the electricity price changes. Obvs you want to switch your battery from fully charging to fully discharging exactly on the half hour because that makes you the most money ... but what if everybody does this?

@TimWardCam @hzulla It's called 'herding' by NESO and (eg) the 10-minute randomisation in car charging start is intended to blunt this...