well that's a first..
finished up some code + ingest of some big data before going to sleep. ended up going to bed at like ~4 AM. completely knocked out (different planet). get awoken by dogs going crazy. takes a while to figure out what's going on. oh there's someone at the door. whose that? that's my boss. why is my boss at my door. hold on am i dreaming? open door:
"production is down"
fuck.
still wondering if this is real.
go get my laptop. fix production (took like 30 minutes).

what a wreck.
edit: great way to start page 565.
eh!
tldr: my boss literally drove ~35 minutes to my house to wake up me, telling me production (site/systems etc) was broken. that's so insane. feels awful. everything is good now, probably didn't cause too much damage (down 2 hours). but still, complete train wreck.
oh, the reason? not 100% sure yet but, max_connect_errors was triggered from our mysql db, which "firewalled" all of our internal instances. it seems kind of flukey, related to perhaps being 'overwhelmed' or ingesting tons of "duped out" records (which isn't an error to the code, a dup is fine).
so the db basically blocked our backend servers.
just disabled max_connect_errors...