Title: Enabling GZIP Compression in Nginx Reverse Proxy Scenario
Date: 2022-10-17 00:48:11
Tags: []
Published: true
HideInList: false
Feature:
IsTop: false
The situation arose from a website cluster project I was working on, where the database was queried every time the sitemap was crawled.
With a large number of spiders, the performance impact was significant, so I implemented pre-caching for the sitemap queries (to minimize performance impact).
The domain was using a CDN with Cloudflare (CF), and the traffic consumption was only around 1m per second for 200 requests.
After changing the sitemap query method, I added a reverse proxy layer for load balancing due to other reasons.
As a result, the traffic consumption increased to around 12m per second for about 160 requests, resulting in approximately 1t of traffic consumption per day.
The server had bidirectional traffic, and the reverse proxy server would shut down after running for 15 days, unable to handle the increased spider traffic in the later stages.
The reason was found to be the frequent crawling of the website sitemap by the spiders, with each request being around 120k in size.
In the setup of "Main Server - Reverse Proxy - CF CDN," GZIP compression was not effective in the communication between the main server and the reverse proxy. GZIP compression only took effect from the CF CDN to the client.
This led to excessive traffic consumption between the servers, and at this point, I had to turn to the almighty Google to find a solution.
Solution: https://www.jianshu.com/p/d24cb42d96d5