Something broke: the shop’s API answers 500
A troubleshooting challenge: the shop’s API fails with 500. Find out why on the server, free the space the right way and prove it works from outside.
No walkthrough this time. Customers can browse the shop, but adding anything to the cart fails, and http://web/api/products answers 500 Internal Server Error. Find out why, and fix it so it stays fixed. The checklist only ticks when the shop really works again.
Everything you need, you've used before:
curlfrom your laptop to see the symptom, and again at the end to prove the fix;ssh web, thensystemctl statusandsudo journalctl -ufor what the program said;df -h(how full each disk is) anddu -h(what uses the space), withsort -h, which sorts sizes like 4.0K, 12M and 16G properly:
sudo du -h /var | sort -h | tail -n 5
One thing that catches everyone once: on Linux, deleting a file that a program still has open removes its name, but the space is only freed when the program lets go of it. sudo lsof +L1 lists files like that, deleted and still open. Restarting the program frees them. Emptying a file instead of deleting it (sudo truncate -s 0 file) frees the space at once, even while it's open.
The machines are pretend ones inside this page, so you can't break anything that matters. Reset puts it all back.
Your turn
Make http://web/api/products answer again, and leave web’s disk with room to spare.
Your task
Make http://web/api/products answer again, and leave web’s disk with room to spare.
- Find what filled the disk (not done yet)
- web’s disk has room again (not done yet)
- The API answers again (not done yet)
The checklist ticks itself off as you work in the terminal: any way that gets the result counts.
Hints come one at a time, then one way to do it. Try each before the next.
What this lesson uses, in one place.
df -h- Sizes like 2.0M or 512G More on df
du -h- Sizes like 4.0K, 12K or 1.5M More on du
sort -h- Sort sizes like 4.0K, 12M and 2.1G (what du -h prints) by their real size More on sort
lsof +L1- Open files that have been deleted More on lsof
truncate- Shrink or empty a file
Every everyday command on one page: the Linux commands cheat sheet.