So I’m working on a server from home.
I do a cat /sys/class/net/eth0/operstate and it says unknown despite the interface being obviously up, since I’m SSH’ing into the box.
I try to explicitely set the interface up to force the status to say up with ip link set eth0 up. No joy, still unknown.
Hmm… maybe I should bring it down and back up.
So I do ip link set eth0 down and… I drive 15 miles to work to do the corresponding ip link set eth0 up
50 years using Unix and I’m still doing this… 😥
Use removedpit by Red Hat. It gives you a GUI to make networking changes*, and will check if the connection still works before making the change. If the connection doesn’t work (like the ip addresses changed), it will undo the change and then warn you. You can then either force the change through or leave it be.
*via NetworkManager only.
I’ve done this kind of thing remotely in screen with
ifdown eth0 ; sleep 10 ; ifup eth0 ;I knew a guy who did this and had to fly to Germany to fix it because he didn’t want to admit what he’d done.
Why don’t you use chained commands, or better yet simply create an alias that chains down/up, then use the alias instead?
Because I plain forgot I was remote. It’s as simple and as stupid as that.
We’ve all been there. If you do this stuff for a living, you’ve done that way more than once.
I formated an OS drive by mistake last night, thought it was my flash drive…
This is why IPMI is so important.
Can also use Pi KVM to add a similar capability to non server grade hardware that doesn’t have it. I did that for a workstation once.
Yup, I use PiKVM, too. Fun fact, PiKVM’s first content commit is a clone of my DIY IPMI repo 😉
Look, it’s’a me: https://github.com/pikvm/pikvm/commit/70eebd5c59da26dc3f6ad56730adbb616055f4e5#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R4
Awesome job!
Thanks! Yea, it was a really fun project to make back before there were any real options. And I’m glad the PiKVM team could expand upon it.
Somewhere along the way I lost the “based on” credit, likely whenever they fully modernized the stack. I wasn’t really keeping track, but did find it humorous when LTT said the creator complained someone based another project on them. I was like “Hmmmmmmm…” but just laughed because I didn’t make it for it to stagnate like it had been with me.
Every network engineer must lock themselves out of a node at some point, it is a rite of passage.
I have a failsafe service for one of my servers, it pings the router and if it hasn’t reached it once for an entire hour then it will reboot the server.
This won’t save me from all mistakes but it will prevent firewall, link state, routing and a few other issues when I’m not present.
Lol I’ve locked myself out of so many random cloud and remote instances like this that now I always make a sleep chain or a kill timer with tmux/screen.
Usually like:
./risky_dumb_script.sh ; sleep 30 ; ./undo.sh
Or
./risky_dumb.script.sh
Which starts with a 30 second sleep, and:
(tmux) sleep 300 ; kill PID
A few months ago I accidentally dd’d ~3GiB to the beginning of one of the drives in a 4 drive array… That was fun to rebuild.
Like 3 weeks ago on my (testing) server I accidentally DD’d a Linux ISO to the first drive in my storage array (I had some kind of jank manual “LVM” bullshit I set up with odd mountpoints to act as a NAS, do not recommend), no Timeshift, no Btrfs snapshot. It gave me the kick in the pants I needed to stop trying to use a macbook air with 6 external hard drives as a server though. Also gave me the kick in the pants I needed to stop using volatile naming conventions in my fstab.
I was scared to move the cloud for this reason. I was used to running to the server room and the KVM if things went south. If that was frozen, usually unplugging the server physically from the switch would get it calm down.
Now Amazon supports a direct console interface like KVM and you can virtually unplug virtual servers from their virtual servers too.
It’s VMs within VMs within VMs.
