— Created: Xiaoke, 2017/05/01 20:40 CST
— Last modified: Xiaoke, 2017/05/01 21:52 CST

Today, during a regular check of my own MQTT site, I found the last data record of the MQTT messages stopped at the previous night. By checking the services running using ps -ef | grep exe_name; systemctl status service_name, I found MySQL daemon mysqld somehow mysteriously stopped. I installed a bunch of updates last night and mysqld probably failed to restart then. A manual start of the daemon through sudo systemctl start mysql also lead to error messages. After combing through googling results, by chance I found that my system might run out of disk space. Command df -h confirmed an almost 100% usage on the root directory. Although I was using a 20G economic hosting service, it was still not possible for the Ubuntu operating system to occupy that much space, considering the limited programs I was running.

Now it fell on me to find the murderer of the disk space, the largest files. Command sudo du -a / | sort -n -r | head -n 20 conveniently facilitated this purpose, and the results indicated an almost 15G of the /var/log directory. This was an exciting yet surprising discovery, since the problem emerged, but in an unexpected location. A further du -ha on /var/log directory showed that the file /var/log/syslog was about 14G in size.

You may wonder what happened to the syslog file, since normally this file won't grow that wildly. Opening this file was not practical considering its size. A check of the last few lines of the file by 'tail -n 20 syslog' displayed familiar printed messages from a python program I once wrote. A further check of syslog file showed that most of the log displayed were the messages from that python program. That python program was part of a daemon service, and had a couple of print statements which I didn't comment out. I thought those messages would be printed to /dev/null as soon as the program was executed as a systemd service. But actually, all the printed messages were packed into the /var/log/syslog file and inflated it to an absurdly large size.

Simply remove the syslog file may cause problems, since it is used by various running programs of the system. The command sudo cat /dev/null > /var/log/syslog turned out to effectively reduce the size of the file to 0. After this, by df -h, a 5G root directory appeared and mysql service started successfully.

So, when executed as a system service, all printed messages in python will be logged into /var/log/syslog. Do comment out the print messages after debugging.