thread

It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.

Ok, so this went surprisingly viral. What I learned from the responses so far:

The main "problem" is the synchronised clock and too many developers using the minute mark for task scheduling default settings.

Having mass-deployed systems doing a thing at the exact same time causing trouble has been known to sysadmins for decades. Solutions readily exist for various task schedulers. However, inexperienced developers and maintainers still default to using the minute mark.

Our industry learns and unlearns this phenomenon again and again.

(I personally certainly have experienced this in my line of work, too, but...)

People may be aware of the problem of synchronised tasks on mass-deployed software for their own distributed system, but I'm mostly surprised that the effect is even visible within the global electricity grid.

So while simple solutions exist since forever, it's unclear how our industry can keep this knowledge alive to avoid future generations doing this little mistake all over again and again.

@hzulla Yeah, lots of people responsible for distributed systems haven’t been taught the ways of thinking about them.

My work team just deployed a scheduled I/O-heavy job (storage snapshots on a non-CoW filesystem) to about a hundred physical servers. The original plan was to push out exactly the same config to all of them. Everybody understood the problem once I pointed it out, but it’s just not something most people consider.

@hzulla People also tend to schedule things on the hour/half-hour, or at multiples of five minutes, in part because cron syntax makes that easier.

@hzulla In fairness, devs sticking to the minute mark isn't really something you want to fix (because you can't). Instead you have to recognize what causes it and give devs the tools to avoid it by default. Because there's currently not a good way to express "once a minute, regardless of where in the period" on many platforms

@hzulla One does however wonder about the effect of switching domestic scale consumption/generation on the half hour when (in the UK) the electricity price changes. Obvs you want to switch your battery from fully charging to fully discharging exactly on the half hour because that makes you the most money ... but what if everybody does this?

@TimWardCam @hzulla It's called 'herding' by NESO and (eg) the 10-minute randomisation in car charging start is intended to blunt this...

@DamonHD @hzulla Nobody has told me to offset my #HomeAssistant automations - so I haven't! And it wouldn't be enforceable anyway, would it?

@TimWardCam @hzulla Early adopters are unlikely to bring down the grid by themselves, and the big VPP players are expected to avoid herding also. This is for mainstream timed use, to smear those times.

I haven't run HA, but inserting even a few seconds randomly before bringing on a timed big load will reduce demand slew rate, so I do it routinely, as I have for servers. Random numbers and primes are two of my favourite robustness tools.

@DamonHD @TimWardCam @hzulla My favourite solution is to forget to enable the NTP server.

@TimWardCam @DamonHD @hzulla the solcast forecasting plugin does this. Too many installs hammering the api at the same time. Not the same as the grid, but still, noticeable