One pitch-black, bitterly cold night, while I was deep in sleep, a frantic phone call jolted me awake. The caller reported that bandwidth usage across several data centers had surged dramatically and needed immediate attention; otherwise, the IDC provider would pull our network cables. I glanced at the clock鈥攊t was around 2 a.m. What a miserable situation!
I first logged into the monitoring system to check the traffic. The server that normally had the highest traffic was maxing out its 1G bandwidth (I didn’t manage to grab a screenshot at the time due to the urgency). Under normal circumstances, its peak bandwidth would hover steadily between 600M-700M/s, as shown in the image below:
for i in `ls $Path`; do
grep -a -r apk $Path/$i/* | strings |grep "KEY:" >/tmp/cache_list$i.txt
grep -v apk$ /tmp/cache_list$i.txt >> /tmp/del$i.txt
rm -rf `grep -v apk$ /tmp/cache_list$i.txt|awk -F: '{print $1}'`
#echo $Path/$i
sleep 60
done
rm -rf /tmp/cache_list*
Understanding this script requires the specific context, and for certain reasons I won’t analyze it further here. In short, the script’s function was to search for certain files within the cache directory and delete them if they existed.
The result of this operation was that cached files were deleted shortly after being created. When a user needed to download that file, the edge server had no cache and had to go back to the source (fetching from the relay server). Under normal conditions, these files would be cached for a long time, but because of this script, they were wiped out again after a short while. This led to a massive, continuous stream of origin pull requests, which caused the bandwidth spike. To avoid immediate risk, I didn’t delete the script directly, but commented it out in the crontab schedule. I checked each edge server one by one and commented out this task.
Watching the traffic graphs, the bandwidth consumption gradually decreased, and after 10