Railgun protocol for traffic compression

Railgun protocol

CloudFlare CDN-provider has developed in the past year a Railgun protocol to compress network traffic. Binary protocol, written in a programming language Google Go, transmits hashes web pages and diff between them, with support for versioning. After implementation of technology the compression in some cases goes up to 99.6%, which is unattainable using gzip. 4Chan and Imgur Railgun sites enjoyed the non-cached traffic reduction by about 50%.

Recently CloudFlare announced that the protocol is implemented to support Amazon Web Services, and about 30 of the largest hosting companies. The installation of Railgun now is easier than ever i.e. there are plugins for WordPress, Joomla, Drupal and other CMS, released packages for most popular distributions of Linux and BSD.

How is works?

Any CDN-provider tries to cache the maximum amount of content. But in reality, to cache only about 66% of the objects, while the remaining 34% is re-requested from the server in case of updates. To compress the traffic and was created Railgun.

Railgun use almost the same algorithms as the processing sequence of images in high definition video. High compression ratio in video codecs is achieved by not compressing individual frames, and the differences between adjacent frames. This allows you to compress a frame rate of one million pixels in a few kilobytes. Theoretically, you can compress it all in a single byte in case it is not different from the previous frame. Similar like done in web pages. If you change the web page in reality only a small part is changed, and the change is enough to pass between the current version and the one that the client received the last time.

In technical terms Railgun obtains two components as the sender (sender) and the receiver (listener). Senders are being installed in each data center of CDN-CloudFlare network worldwide. The receiver is a software component provided by one of CloudFlare’s commercial partners. There is a TCP-connection being established between the sender and the recipient through secure TLS where asynchronous HTTP-requests are performed by Railgun binary data. As the web client system Railgun looks like a proxy server, but in reality it is a specialized system with specific functions. One of them is the compression of content that being cached by synchronizing versions of pages. When you update the version of the page on the network is transmitted only change between the previous and the new version.

The majority of tests demonstrated the maximum compression being achieved on news sites with heavy traffic in terms of the attendance. For example, a binary comparison of the main page of the site reddit.com reveals the change in average 2.15% for 5 minutes and 3.16% for one hour. Home page of The New York Times experience binary changes by 0.6% in five minutes and by 3% in an hour. For the main page BBC News, the figures are 0.4% and 2%, respectively. In general, for the most popular sites in the second request diff sometimes so small that it fits in a single packet TCP.