thread

Hanno Zulla @hzulla@infosec.exchange

This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one major cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.

https://www.fz-juelich.de/en/news/archive/press-release/2026/mystery-of-60-second-fluctuations-in-power-grids-solved

Mystery of 60-Second Fluctuations in Power Grids Solved Why do our grids show a brief spike in electricity demand every minute? A team led by Dirk Witthaut from Forschungszentrum Jülich has now demonstrated that this is due to regular power checks carried out by large-scale battery storage systems. These findings could help to make our electricity supply more robust, as the team reports in an article published in Nature Energy. www.fz-juelich.de
Hanno Zulla @hzulla@infosec.exchange

It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.

Hanno Zulla @hzulla@infosec.exchange

Ok, so this went surprisingly viral. What I learned from the responses so far:

The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.

Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.

Our industry learns and unlearns this phenomenon again and again.

(I personally certainly have experienced this in my line of work, too, but...)

People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.

So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.

Zimmie @bob_zim@infosec.exchange

@hzulla Yeah, lots of people responsible for distributed systems haven’t been taught the ways of thinking about them.

My work team just deployed a scheduled I/O-heavy job (storage snapshots on a non-CoW filesystem) to about a hundred physical servers. The original plan was to push out exactly the same config to all of them. Everybody understood the problem once I pointed it out, but it’s just not something most people consider.

std::polar @sehe@fosstodon.org

@hzulla In fairness, devs sticking to the minute mark isn't really something you want to fix (because you can't). Instead you have to recognize what causes it and give devs the tools to avoid it by default. Because there's currently not a good way to express "once a minute, regardless of where in the period" on many platforms

majick @majick@mefi.social

@hzulla "We solved this problem 40 years ago, you fools!" is the eternal curse, the burden we bear and always will. It used to be spikes in IO and to some extent compute. It's been network traffic for a long time. Energy is just another resource to manage in systems; cpuidle probably made it a little bit worse.

Now scale that to datacenter-scale operations and, yeah, I can see how this becomes one of those facepalm moments we don't deserve to have over and over again.

At this point in my life, the only innovation I can imagine leaving behind of any value is filling old potholes like this for once and for all. All the preventable pitfalls everyone keeps pitching needlessly into. The obligation isn't just to today's users, it's to those who inherit tomorrow's users.

Tim Ward ⭐🇪🇺🔶 #FBPE @TimWardCam@c.im

@hzulla One does however wonder about the effect of switching domestic scale consumption/generation on the half hour when (in the UK) the electricity price changes. Obvs you want to switch your battery from fully charging to fully discharging exactly on the half hour because that makes you the most money ... but what if everybody does this?

replying to @TimWardCam@c.im
DamonHD @DamonHD@mastodon.social

@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.

I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.