Proxy array load balancing types for big ISP

Proxy array load balancing

A proxy array is a group of proxy (cache) servers that work together as one logical cache. Load balancing decides which member serves which request, so that no proxy is overloaded and the total cache is used efficiently.

1. DNS round robin. The proxy name has several A records; DNS returns them in rotation. Simple and cheap, but not aware of load or failures (cached DNS answers continue to point to a dead proxy), and the same URL may be cached in many proxies (duplicate copies, low hit rate).

2. Hardware / layer-4 or layer-7 switch (load balancer). A switch in front of the proxies spreads connections by round robin, least connections or URL-based rules. It detects failed proxies, but it may be a single point of failure and costs money.

3. Hash-based (non-redundant) array, e.g. CARP. The URL (or host name) is hashed and the result selects exactly one proxy. Each object lives in only one proxy, so the total cache size is the sum of all members and the hit rate is high. Adding or removing a proxy remaps only a small fraction of the URLs.

4. Redundant array (mirrored / replicated). Each object is stored in two or more proxies, or traffic is directed to a pair. A proxy failure is hidden from users, but the effective cache is smaller.

5. Hierarchical / ICP (sibling) caching. Proxies ask their neighbours with ICP before going to the origin. This gives some sharing but adds inter-proxy traffic and delay.

 Clients --> [Front end: switch / PAC file]
                 |        |        |
             hash(URL) selects one proxy
              [P1]     [P2]     [P3]  ...  --> Internet

Mechanism appropriate for a big ISP

A hash-based array (CARP) combined with a layer-4 load balancer for redundancy is best.

• An ISP has millions of users and a huge set of objects. With hashing, every URL is cached once, so the combined disk of all proxies acts as one very large cache and the hit ratio is high; round robin would duplicate popular objects and waste disk space.
• It scales linearly: proxies are added as traffic grows, and because of the hashing scheme only about 1/n of the cached URLs are lost when one is added.
• There is no inter-proxy query traffic (unlike ICP), so no extra delay or load.
• Requests are distributed according to configurable weights (capacity), which suits proxies of different sizes.
• Failure handling: health checks remove a dead proxy and its share is re-hashed to the others; a pair of load balancers (VRRP/HSRP) or WCCP routers avoids a single point of failure.
• Transparent redirection (WCCP or policy routing) removes the need to configure client browsers, which is impractical for an ISP.

Hence an ISP uses a CARP-style non-redundant array for efficiency, with redundant front-end devices and health monitoring for availability.