This happens. Recently we had a problem in production where our database grew by a factor of 10 in just a few minutes due to a replication glitch. Of course it took down the whole application as we ran out of space.
Some things just happen and all head room and monitoring cannot save you if things go seriously wrong. You cannot prepare for everything in life and IT I guess. It is part of the job.
Bad things can happen but that’s why you build disaster recovery into the infrastructure. Especially with a compqny as big as Toyota, you can’t have a single point of failure like this. They produce over 13,000 cars per day. This failure cost them close to 300,000,000 dollars just in cars.
The IT people that want to implement that disaster recovery plan do not make the purchasing decisions. It takes an event like this to get the retards in the C-suite listen to IT staff.
In my experience, the C-Suite dicks will put the hammer down on someone and maybe fire a couple of folks. They’ll demand a summary of what happened and what will be done to stop it from happening again. IT will provide legit options to resolve this long term, but because that comes with a price tag they’ll be told to fix it with “process changes” and the cycle continues.
If they give IT money that’s less for themselves at EOY for bonuses so it’s a big concern /s
Yea, fair point regarding the single point of failure. I guess it was one of those scenarios that should just never happen.
I am sure it won’t happen again though.
As I said it can just happen even though you have redundant systems and everything. Sometimes you don’t think about that one unlikely scenario and boom.
Idiots, they ought to have switched to tabs for indenting. Everybody knows that.
Jokes aside, this got me thinking, is there actual difference in term of file size?
Yes, 4 spaces are 3 more bytes than a tab.
But if you compress the source files there would be no difference.
And who needs 100% quality images. Just set jpeg quality to 60% and be done with it.
I blame lean philosophy. Keeping spare parts and redundancy is expensive so definitely don’t do it…which is just rolling the dice until it comes up snake eyes and your plant shuts down.
It’s the “save 5% yearly and stop trying to avoid a daily 5% chance of disaster”
Over prepared is silly, but so is under prepared.
They were under prepared.
I work in a manufacturing company that was owned by the founder for 50 years until about 4 years ago when he retired. He disagreed with a lot of the ideas behind lean manufacturing so we had like 5 years worth of inventory sitting in our warehouse.
When the new management came in, there was a lot of squawking about inefficiency, how wasteful it was to keep so much raw material on the shelf, and how we absolutely needed to sell it off or get rid of it.
Then a funny little thing happened in 2020.
Suddenly, we were the only company in our industry still churning out product. Other companies were calling us, desperate to buy our products or even just our raw material. We saw MASSIVE growth the next two years and came out of the pandemic better than ever. And it was mostly thanks to the old owners view that “Just In Time” manufacturing was BS.
Cool story, but a once every 150 years pandemic is hardly a good reason to keep wasting money on storing stuff. A fire or a flood was much more likely to wipe it all out in 50 years.
Even in your anecdote the owner never actually benefited from the extra costs.
Depending on what you’re producing costs to maintain extra inventory of raw materials can be massive and for the company the size of Toyota, multiply that by million.
You’re not wrong, but the real answer is somewhere in between.
Even in your anecdote the owner never actually benefited from the extra costs.
Imagine doing something or having a life/business philosophy that doesn’t exist for your own soul benefit, and exists maybe for the benefit of others.
Removed by mod
Sysadmin pro tip: Keep a 1-10GB file of random data named DELETEME on your data drives. Then if this happens you can get some quick breathing room to fix things.
Also, set up alerts for disk space.
10GB is nothing in an enterprise datastore housing PBs of data. 10GB is nothing for my 80TB homelab!
It not going to bring the service online, but it will prevent a full disk from letting you do other things. In some cases SSH won’t work with a full disk.
Even better, cron job every 5 mins and if total remaining space falls to 5% auto delete the file and send a message to sys admin
At that point just set the limit a few gig higher and don’t have the decoy file at all
Sends a message and gets the services ready for potential shutdown. Or implements a rate limit to keep the service available but degraded.
Also, if space starts decreasing much more rapidly than normal.
Free disk space is just inventory and therefor wasteful.
Kanban
There’s some irony to every tech company modeling their pipeline off Toyota’s Kanban system…
Only for Toyota to completely fuck up their tech by running out of disk space for their system to exist on. Looks like someone should have put “Buy more hard drives” to the board.
not to mention the lean process effed them during fukashima and covid, with a breakdown in logistics and a shortage of chips, meant that their entire mode of operating shut down, as they had no capacity to deal with any outages in any of their systems. Maybe that has happened again, just in server land.
It wasn’t just Fukushima. There was a massive flood in Thailand at the same time that shut down a load of suppliers. It was a really bad bit of luck but they did learn from that.
Oh yeah, this is all from memory.
This is a fun read in the wake of learning about all the personal data car manufacturers have been collecting
Storage has never been cheaper.
There’s going to be a seppuku session in somebody’s IT department.
well well, someone will be fired
This is the best summary I could come up with:
TOKYO, Sept 6 (Reuters) - A malfunction that shut down all of Toyota Motor’s (7203.T) assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
The system halt followed an error due to insufficient disk space on some of the servers and was not caused by a cyberattack, the world’s largest automaker by sales said in a statement on Wednesday.
“The system was restored after the data was transferred to a server with a larger capacity,” Toyota said.
The issue occurred following regular maintenance work on the servers, the company said, adding that it would review its maintenance procedures.
Two people with knowledge of the matter had told Reuters the malfunction occurred during an update of the automaker’s parts ordering system.
Toyota restarted operations at its assembly plants in its home market on Wednesday last week, a day after the malfunction occurred.
The original article contains 159 words, the summary contains 159 words. Saved 0%. I’m a bot and I’m open source!
Wow, what a useful bot!
Free space is a wasted disk space.
They paid for the whole disk, they’re going to use the whole disk.
Search for non system files older than 2020 and lmza zip archive that shit stat!
You have now archived the encryption keys. Nothing can communicate and everything begins failing.
Ctrl+z?
Server less DBs ftw
Serverless just means that the user doesn’t manage the capacity by themselves. This scenario can happen easily if the serverless provider is as incompetent as the Toyota admins.
Serverless means you don’t have control over it yourself when something goes wrong, so you can’t fix it yourself.
