An agent fixed our web server. I never typed a command.
A web page on one of the lab's servers goes down, and the problem goes to an agent in plain language. It connects over SSH, finds the failed service, traces the logs back to what stopped it, restarts it, and checks the page from outside the machine. Nobody typed a command on the server.
Chapters
- 0:00The report: the web page is down
- 0:16Handing it to the agent in plain language
- 0:33In over SSH: nothing listening on port 80
- 0:43Finding the real web server, stopped
- 0:59The logs, the restart, and the outside check
- 1:33Full disclosure, and what matters
Transcript
The narration, as written. The voice is synthetic.
0:00The report: the web page is down
Hi, I'm Hal. A user says the web page on Ubuntu1 is down. Let's look. The server is at 192.168.3.11. Connection refused. Hard reload, so it's not cached. Still refused. The problem is real.
0:16Handing it to the agent in plain language
Now I hand it to the agent. One plain-language request: a user can't reach the web page on Ubuntu1, verify it and find out why. One piece of direction: there's an SSH key on this machine it can use. That's the whole hand-off.
0:33In over SSH: nothing listening on port 80
It finds the SSH config, resolves the host, and gets in. Nothing is listening on port 80. Not nginx, not Apache. Neither is installed.
0:43Finding the real web server, stopped
It keeps looking and finds the actual web server: lighttpd, or "lighty," if you've been around it. Enabled, but in a failed state. The config is valid and it starts fine by hand. Not a broken install. Something stopped it.
0:59The logs, the restart, and the outside check
It goes to the logs and works the timeline. Seventeen seconds before the failure, someone logged in over SSH and ran sudo su. That's the lead. It restarts the service. Notice it didn't stop to ask me. For this taste, I gave it the go-ahead up front. On a system you care about, that's a choice you make on purpose. We'll come back to it. Listening on port 80. A 200 from the page. Then it checks the firewall and confirms the page is reachable from outside the box, not just from localhost. Back to the browser, hard reload, and the page is up.
1:33Full disclosure, and what matters
Full disclosure: I staged this. That sudo su session was me, stopping the server to see if the agent would find it. It did, and it told me. Here's what matters. I never connected to that server. I never typed a command on it. I described the problem and watched. The setup behind this, the key, the access, the guardrails, is its own set of walkthroughs. Those are coming.