Earlier this year I had a small bug in my homepage. It caused a 100% usage of the cpu and an increase of 1gb of the log files. This is how I fixed it.
Disk Usage
On 2022-04-13, after I came home from work, I deployed a new version of my page with only updates gems and the used ruby version. No big deal. Buy while watching some Star Trek, I noticed that the CPU usage of my server jumped to 100% and the disk usage did also increase!
My first assumption was that my server was hacked. Since my switch to using Ruby on Rails for my homepage, someone was checking posible week routes. It didn't border me, because they where checking for a PHP deploy (.git in the public directory).
Path to the solution
So I checked the logs. Using tail revealed a Ruby error. But when I run tail with a higher number of lines (tail -n 20), was the output totaly different! Every time I ran it!
This caused me to check the size of my log files: Over 1gb! It increased from some low mb file to over 1gb in under an hour! There wasn't some upload happening, just logging!
tail -n 150 production.log
Did finally reveal the cause: In webmention_job.rb Net::HTTP wasn't defined and Good-Job did re-queue the job! Something that they did change since then.
So import net/http and deploy and CPU usage did decrease! The bug was gone!
All what was left for me to do, was to setup Log Rotation, and finish the Star Trak episode.
Why did it happen/What did I learn
Other than in Javascript, where imports are file scoped. In Ruby they are process scoped. One of the updated gems didn't import "net/http" anymore. In both development and test did "webmock" import "net/http". But not in production. And because "net/http" is not a gem, but a Ruby core library, didn't Rails import it, too.
Christopher Astfalk
