Overcoming Disk Space Challenges in Server Launch

Launching a server to distribute digital Kanjideck files quickly turned into a crisis when hundreds of users flooded in, overwhelming the system's 40GB disk space. The server, a modest Hetzner machine running NixOS, was unable to handle the load due to a 2.2GB file among others. The logs revealed a persistent error indicating insufficient storage, which halted file access and email communications. With disk usage maxed out, I scrambled to identify and clear space, discovering that Plausible Analytics and the /nix/store were major space consumers.

Efforts to clear the /nix/store failed due to the lack of space, prompting a desperate attempt to free up space by deleting logs and truncating database tables. However, these measures were insufficient as space continued to be consumed rapidly. The solution came in the form of mounting the /nix/store on a separate volume, which alleviated the immediate pressure and allowed the server to function, albeit with some users still experiencing issues downloading the large file.

Further investigation revealed that nginx's default proxy settings were inadequate for handling the large file, as the proxy_max_temp_file_size was set too low. Adjusting this setting allowed the file to be served successfully. Yet, a deeper issue was uncovered: nginx was buffering the file to temporary storage, consuming vast amounts of space. Disabling proxy buffering resolved the issue, stabilizing disk usage at a manageable level.

In hindsight, the crisis highlighted the importance of understanding server configurations and the impact of proxy settings on disk usage. The experience underscored the need for careful planning and the value of experience in managing server resources under pressure.