Skip to content

Something broke: the shop’s API answers 500

NinjaLesson 63 of 6612 min

A troubleshooting challenge: the shop’s API fails with 500. Find out why on the server, free the space the right way and prove it works from outside.

No walkthrough this time. Customers can browse the shop, but adding anything to the cart fails, and http://web/api/products answers 500 Internal Server Error. Find out why, and fix it so it stays fixed. The checklist only ticks when the shop really works again.

Everything you need, you've used before:

  • curl from your laptop to see the symptom, and again at the end to prove the fix;
  • ssh web, then systemctl status and sudo journalctl -u for what the program said;
  • df -h (how full each disk is) and du -h (what uses the space), with sort -h, which sorts sizes like 4.0K, 12M and 16G properly:
sudo du -h /var | sort -h | tail -n 5

One thing that catches everyone once: on Linux, deleting a file that a program still has open removes its name, but the space is only freed when the program lets go of it. sudo lsof +L1 lists files like that, deleted and still open. Restarting the program frees them. Emptying a file instead of deleting it (sudo truncate -s 0 file) frees the space at once, even while it's open.

The machines are pretend ones inside this page, so you can't break anything that matters. Reset puts it all back.

Your turn

Make http://web/api/products answer again, and leave web’s disk with room to spare.

Next