--- title: "Large fixed-size chunks in GFS" subject: "Big Data Technologies" chapter: 2 key: "semester_8/big_data_technologies:gfs-large-fixed-chunks" source: https://www.bctengineer.com/q/semester_8/big_data_technologies/gfs-large-fixed-chunks site: BCT Engineering --- # Large fixed-size chunks in GFS Big Data Technologies · Chapter 2: Google File System - Asked in: 2073 Magh, 10 marks - Video: [Google File System - Paper that inspired Hadoop](https://www.youtube.com/watch?v=eRgFNW4QFDc) — Defog Tech - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/gfs-large-fixed-chunks - Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-2 ## Question Why do we have large and fixed sized Chunks in GFS? What can be the demerits of that design? ## Short answer Large fixed 64 MB chunks reduce metadata size and client-master interactions and suit large sequential files. Demerits are hot spots for small popular files, metadata waste for many small files, poor random small accesses, and slower re-replication. ## Answer GFS stores each file as **fixed-size chunks of 64 MB**, much larger than the few-KB blocks of ordinary file systems. Each chunk is a Linux file on a chunkserver, identified by a 64-bit chunk handle. #### Why large, fixed-size chunks (advantages) 1. **Less metadata:** fewer chunks mean the master's table is small. One 64 MB chunk needs under 64 bytes of metadata, so all metadata fits in the master's memory, enabling fast operations. 2. **Fewer client-master interactions:** reads and writes on the same chunk need only one request to the master. A client can cache chunk locations for a multi-TB working set. 3. **Reduced network overhead:** a client doing many operations on a chunk can keep a persistent TCP connection to one chunkserver. 4. **Suits the workload:** applications read/write multi-GB files sequentially, so large chunks match the access pattern. 5. **Simple allocation and lookup:** fixed size makes it easy to compute the chunk index from a byte offset (offset / 64 MB), and simplifies replica placement, load balancing and re-replication. 6. **Lazy space allocation:** chunk files are allocated only as needed, so internal fragmentation is avoided. #### Demerits 1. **Hot spots:** a small file may consist of one chunk. If many clients access it at the same time (e.g. an executable file being started by hundreds of machines), the chunkservers holding it are overloaded. GFS mitigates this with a higher replication factor and staggered start times. 2. **Wasted space for small files:** a small file still takes a whole chunk entry, and with many small files the master's metadata grows without the benefit of large chunks (lazy allocation limits the disk waste, not the metadata). 3. **Inefficiency for random access of small data:** a whole large chunk must be managed even for small reads. 4. **Slower re-replication and recovery:** copying a 64 MB chunk takes more time and bandwidth than a small block. 5. **Load imbalance:** a few big chunks make it harder to balance load evenly. 6. **Unsuitable for many small files** and for low-latency random workloads. #### Summary Large fixed-size chunks reduce metadata size and master traffic and suit large sequential files, but they cause hot spots for small, popular files and are poor for small-file workloads. GFS reduces the hot-spot problem by higher replication of popular files and by throttling start-up reads. ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. Why does a 64 MB chunk size let GFS keep all metadata in the master's memory? A. Chunk metadata is compressed on disk B. Fewer chunks per file means the master's table is small (under 64 bytes per chunk) C. Chunkservers store the metadata instead D. Each file has only one chunk Answer: B. Large chunks mean few chunks, and each needs under 64 bytes of metadata, so the whole table fits in master memory. (Key point: Less metadata) 2. How do large chunks reduce load on the master? A. Clients never contact the master B. The master caches all data C. Reads and writes on the same chunk need only one master request, and locations can be cached D. The master copies data to clients Answer: C. A client asks the master once per chunk and caches locations, so many operations on a big chunk need no further master contact. (Key point: Fewer client-master interactions) 3. Why can large chunks cause a hot spot? A. A small file occupies one chunk, so many clients reading it overload the few chunkservers holding it B. Large chunks are replicated too many times C. The master cannot find big chunks D. Checksums become slow Answer: A. A small popular file (e.g. an executable started by hundreds of machines) lives in one chunk on few servers, which get overloaded. (Key point: Hot spots) 4. What does fixed chunk size make easy? A. Avoiding replication B. Removing the master C. Eliminating checksums D. Computing the chunk index from a byte offset (offset / 64 MB) Answer: D. With fixed size, the chunk index is simply offset divided by 64 MB, which also simplifies placement and load balancing. (Key point: Computing chunk index) ## Related in other subjects - [Design wind pressure, large openings, 100 yr](https://www.bctengineer.com/question.md?key=semester_6%2Fdesign_of_steel_and_timber_structure%3Awind-large-openings-hill-160m) — Design of Steel and Timber Structure, chapter 8 - [Why do we need DFT](https://www.bctengineer.com/question.md?key=semester_7%2Fdigital_signal_analysis_and_processing%3Awhy-need-dft) — Digital Signal Analysis and Processing, chapter 7, asked 6× - [T/F: rural design, transformer size first](https://www.bctengineer.com/question.md?key=semester_8%2Ftransmission_and_distribution_system_design%3Atf-rural-transformer-size-first) — Transmission and Distribution System Design, chapter 7 - [Large vs small scale models; propagation mechanisms](https://www.bctengineer.com/question.md?key=semester_7%2Fwireless_communication%3Alarge-small-scale-propagation) — Wireless Communication, chapter 3