Chapter 4: File System Management & I/O System

4.2. File System Implementation

GV: Nguyễn Thị Thanh Vân

Operating System Concepts – 10th Edition Silberschatz, Galvin and Gagne ©2018

Outline

 File-System Structure
 File-System Operations
 Directory Implementation
 Allocation Methods
 Free-Space Management
 Efficiency and Performance
 Recovery
 Example: WAFL File System

Operating System Concepts – 10th Edition 14.2 Silberschatz, Galvin and Gagne ©2018

1
 Objectives

 Describe the details of implementing local file systems and

directory structures
 Discuss block allocation and free-block algorithms and trade-offs
 Explore file system efficiency and performance issues
 Look at recovery from file system failures
 Describe the WAFL file system as a concrete example

Operating System Concepts – 10th Edition 14.3 Silberschatz, Galvin and Gagne ©2018

File-System Structure
 File structure
• Logical storage unit
• Collection of related information
 File system resides on secondary storage (disks)
• Provided user interface to storage, mapping logical to physical
• Provides efficient and convenient access to disk by allowing data to
be stored, located retrieved easily
 Disk provides in-place rewrite and random access
• I/O transfers performed in blocks of sectors (usually 512 bytes)
 File control block (FCB) – storage structure consisting of information
about a file
 Device driver controls the physical device
 File system organized into layers

Operating System Concepts – 10th Edition 14.4 Silberschatz, Galvin and Gagne ©2018

2
 Layered File System

Operating System Concepts – 10th Edition 14.5 Silberschatz, Galvin and Gagne ©2018

File System Layers

 Device drivers manage I/O devices at the I/O control layer
• Given commands like “read drive1, cylinder 72, track 2, sector 10,
into memory location 1060” outputs low-level hardware specific
commands to hardware controller
 Basic file system given command like “retrieve block 123” translates
to device driver
 Also manages memory buffers and caches (allocation, freeing,
replacement)
• Buffers hold data in transit
• Caches hold frequently used data
 File organization module understands files, logical address, and
physical blocks
 Translates logical block # to physical block #
 Manages free space, disk allocation

Operating System Concepts – 10th Edition 14.6 Silberschatz, Galvin and Gagne ©2018

3
 File System Layers (Cont.)

 Logical file system manages metadata information

• Translates file name into file number, file handle, location by
maintaining file control blocks (inodes in UNIX)
• Directory management
• Protection
 Layering useful for reducing complexity and redundancy, but adds
overhead and can decrease performance
 Logical layers can be implemented by any coding method according to
OS designer

Operating System Concepts – 10th Edition 14.7 Silberschatz, Galvin and Gagne ©2018

File System Layers (Cont.)

 Many file systems, sometimes many within an operating system

• Each with its own format:
• CD-ROM is ISO 9660;
• Unix has UFS, FFS;
• Windows has FAT, FAT32, NTFS as well as floppy, CD, DVD Blu-
ray,
• Linux has more than 130 types, with extended file system ext3
and ext4 leading; plus distributed file systems, etc.)
• New ones still arriving – ZFS, GoogleFS, Oracle ASM, FUSE

Operating System Concepts – 10th Edition 14.8 Silberschatz, Galvin and Gagne ©2018

4
 File-System Operations

 We have system calls at the API level, but how do we implement their
functions?
• On-disk and in-memory structures
 Boot control block contains info needed by system to boot OS from that
volume
• Needed if volume contains OS, usually first block of volume
 Volume control block (superblock, master file table) contains volume
details
• Total # of blocks, # of free blocks, block size, free block pointers or array
 Directory structure organizes the files
• Names and inode numbers, master file table

Operating System Concepts – 10th Edition 14.9 Silberschatz, Galvin and Gagne ©2018

File-System Implementation (Cont.)

 Per-file File Control Block (FCB) contains many details about the file
• Typically inode number, permissions, size, dates
• NFTS stores into in master file table using relational DB structures

Operating System Concepts – 10th Edition 14.10 Silberschatz, Galvin and Gagne ©2018

5
 In-Memory File System Structures

 Mount table storing file system mounts, mount points, file system types
 System-wide open-file table contains a copy of the FCB of each file
and other info
 Per-process open-file table contains pointers to appropriate entries in
system-wide open-file table as well as other info
 The following figure illustrates the necessary file system structures
provided by the operating systems
 Plus buffers hold data blocks from secondary storage
 Open returns a file handle for subsequent use
 Data from read eventually copied to specified user process memory
address

Operating System Concepts – 10th Edition 14.11 Silberschatz, Galvin and Gagne ©2018

In-Memory File System Structures

Figure (a) refers to opening a file
Figure (b) refers to reading a file

Operating System Concepts – 10th Edition 14.12 Silberschatz, Galvin and Gagne ©2018

6
 Directory Implementation

 Linear list of file names with pointer to the data blocks

• Simple to program
• Time-consuming to execute
 Linear search time
 Could keep ordered alphabetically via linked list or use B+ tree
 Hash Table – linear list with hash data structure
• Decreases directory search time
• Collisions – situations where two file names hash to the same location
• Only good if entries are fixed size, or use chained-overflow method

Operating System Concepts – 10th Edition 14.13 Silberschatz, Galvin and Gagne ©2018

Allocation Methods - Contiguous

 An allocation method refers to how disk blocks are allocated for files:
 Each file occupies set of contiguous blocks
• Best performance in most cases
• Simple – only starting location (block #) and length (number of blocks)
are required
• Problems include:
 Finding space for file,
 Knowing file size,
 External fragmentation, need for compaction off-line (downtime)
or on-line

Operating System Concepts – 10th Edition 14.14 Silberschatz, Galvin and Gagne ©2018

7
 Contiguous Allocation

 Mapping from logical to physical (block

size =512 bytes)

LA/512

 Block to be accessed = starting address

+ Q
 Displacement into block = R

 Contiguous allocation is easy to

implement but has limitations, and is
therefore not used in modern ile systems.

Operating System Concepts – 10th Edition 14.15 Silberschatz, Galvin and Gagne ©2018

Extent-Based Systems

 Many newer file systems (i.e., Veritas File System) use a modified
contiguous allocation scheme
 Extent-based file systems allocate disk blocks in extents
 An extent is a contiguous block of disks
• Extents are allocated for file allocation
• A file consists of one or more extents

Operating System Concepts – 10th Edition 14.16 Silberschatz, Galvin and Gagne ©2018

8
 Allocation Methods - Linked

 Each file a linked list of blocks

• File ends at nil pointer
• No external fragmentation
• Each block contains pointer to next block
• No compaction, external fragmentation
• Free space management system called when new block needed
• Improve efficiency by clustering blocks into groups but increases
internal fragmentation
• Reliability can be a problem
• Locating a block can take many I/Os and disk seeks

Operating System Concepts – 10th Edition 14.17 Silberschatz, Galvin and Gagne ©2018

Linked Allocation
 Each file is a linked list of disk blocks: blocks may be scattered anywhere
on the disk
 Scheme. Ex: a file of 5 blocks: start at block 9, and continue at block 16,
then block 1, then block 10, and finally block 25

Q
 Mapping LA/511
R

 Block to be accessed is the Qth

block in the linked chain of
blocks representing the file.
 Displacement into block = R + 1

Operating System Concepts – 10th Edition 14.18 Silberschatz, Galvin and Gagne ©2018

9
 Allocation Methods – FAT
 FAT (File Allocation Table) - An important variation on linked allocation
• Beginning of volume has table, indexed by block number
• Much like a linked list, but faster on disk and cacheable
• New block allocation simple
 Ex:
• a file consisting of disk blocks
217, 618, and 339

 Each file has its own index block(s) of

pointers to its data blocks
 Logical view

index table
Operating System Concepts – 10th Edition 14.19 Silberschatz, Galvin and Gagne ©2018

Example of Indexed Allocation

Operating System Concepts – 10th Edition 14.20 Silberschatz, Galvin and Gagne ©2018

10
 Combined Scheme: UNIX UFS
4K bytes per block, 32-bit addresses

More index blocks than can be addressed with 32-bit file pointer

Operating System Concepts – 10th Edition 14.26 Silberschatz, Galvin and Gagne ©2018

Performance
 Best method depends on file access type
• Contiguous great for sequential and random
 Linked good for sequential, not random
 Declare access type at creation -> select either contiguous or linked
 Indexed more complex
• Single block access could require 2 index block reads then data
block read
• Clustering can help improve throughput, reduce CPU overhead
 For NVM, no disk head so different algorithms and optimizations
needed
• Using old algorithm uses many CPU cycles trying to avoid non-
existent head movement
• With NVM goal is to reduce CPU cycles and overall path needed
for I/O

Operating System Concepts – 10th Edition 14.27 Silberschatz, Galvin and Gagne ©2018

11
 Free-Space Management
 File system maintains free-space list to track available
blocks/clusters (Using term “block” for simplicity)
• the free-space list is implemented as a bitmap or bit vector.
 Bit vector or bit map (n blocks)
0 1 2 n-1 1  block[i] free
bit[i] =
… 0  block[i] occupied
 Ex: 001111001111110001100000011100000 ...
 Block number calculation:
(number of bits per word) * (number of 0-value words) + offset of first 1 bit
CPUs have instructions to return offset within word of first “1” bit

• Bit map requires extra space

Example:
block size = 4KB = 212 bytes
disk size = 240 bytes (1 terabyte)
n = 240/212 = 228 bits (or 32MB)
if clusters of 4 blocks -> 8MB of memory
Operating System Concepts – 10th Edition 14.29 Silberschatz, Galvin and Gagne ©2018

Linked Free Space List on Disk

 Linked list (free list) - Another approach to free-space

management. Advantage:
• Cannot get contiguous space easily
• No waste of space
• No need to traverse the entire list
(if # free blocks recorded)

 Ex: blocks 2, 3, 4, 5, 8, 9, 10, 11, 12, 13, 17, 18, 25, 26, and
27 were free and the rest of the blocks were allocated.

Operating System Concepts – 10th Edition 14.30 Silberschatz, Galvin and Gagne ©2018

12
 Free-Space Management (Cont.)
 Grouping: A modification of the free-list approach
• Modify linked list to store address of next n-1 free blocks in first
free block, plus a pointer to next block that contains free-block-
pointers (like this one)

 Counting: Another approach

• Because space is frequently contiguously used and freed, with
contiguous-allocation allocation, extents, or clustering
 Keep address of first free block and count of following free
blocks
 Free space list then has entries containing addresses and
counts

Operating System Concepts – 10th Edition 14.31 Silberschatz, Galvin and Gagne ©2018

Free-Space Management (Cont.)

 Space Maps
• Used in ZFS (Solaris, OS: huge numbers of files, directories)
• Consider meta-data I/O on very large file systems
 Full data structures like bit maps couldn’t fit in memory ->
thousands of I/Os
• Divides device space into metaslab units and manages metaslabs
 Given volume can contain hundreds of metaslabs
• Each metaslab has associated space map
 Uses counting algorithm
• But records to log file rather than file system
 Log of all block activity, in time order, in counting format
• Metaslab activity -> load space map into memory in balanced-tree
structure, indexed by offset
 Replay log into that structure

 Combine contiguous free blocks into single entry

Operating System Concepts – 10th Edition 14.32 Silberschatz, Galvin and Gagne ©2018

13
 Free-Space Management (Cont.)

TRIMing Unused Blocks

 HDDS overwrite in place so need only free list
 Blocks not treated specially when freed
• Keeps its data but without any file pointers to it, until overwritten
 Storage devices not allowing overwrite (like NVM) suffer badly with
same algorithm
• Must be erased before written, erases made in large chunks
(blocks, composed of pages) and are slow
• TRIM is a newer mechanism for the file system to inform the
NVM storage device that a page is free
 Can be garbage collected or if block is free, now block can be
erased

Operating System Concepts – 10th Edition 14.33 Silberschatz, Galvin and Gagne ©2018

Efficiency and Performance

 Efficiency dependent on:

• Disk allocation and directory algorithms
• Types of data kept in file’s directory entry
• Pre-allocation or as-needed allocation of metadata structures
• Fixed-size or varying-size data structures

Operating System Concepts – 10th Edition 14.34 Silberschatz, Galvin and Gagne ©2018

14
 Efficiency and Performance (Cont.)

 Performance
• Keeping data and metadata close together
• Buffer cache – separate section of main memory for frequently used
blocks
• Synchronous writes sometimes requested by apps or needed by OS
 No buffering / caching – writes must hit disk before acknowledgement
 Asynchronous writes more common, buffer-able, faster
• Free-behind and read-ahead – techniques to optimize sequential access
• Reads frequently slower than writes

Operating System Concepts – 10th Edition 14.35 Silberschatz, Galvin and Gagne ©2018

Page Cache

 A page cache caches pages rather than disk blocks using virtual memory
techniques and addresses
 Memory-mapped I/O uses a page cache
 Routine I/O through the file system uses the buffer (disk) cache
 This leads to the following figure
I/O Without a Unified Buffer Cache

I/O using a unified buffer cache.

Operating System Concepts – 10th Edition 14.36 Silberschatz, Galvin and Gagne ©2018

15
 Recovery

 Consistency checking – compares data in directory structure with

data blocks on disk, and tries to fix inconsistencies
• Can be slow and sometimes fails
 Use system programs to back up data from disk to another storage
device (magnetic tape, other magnetic disk, optical)
 Recover lost file or disk by restoring data from backup

Operating System Concepts – 10th Edition 14.37 Silberschatz, Galvin and Gagne ©2018

Log Structured File Systems

 Log structured (or journaling) file systems record each metadata
update to the file system as a transaction
 All transactions are written to a log
• A transaction is considered committed once it is written to the log
(sequentially)
• Sometimes to a separate device or section of disk
• However, the file system may not yet be updated
 The transactions in the log are asynchronously written to the file
system structures
• When the file system structures are modified, the transaction is
removed from the log
 If the file system crashes, all remaining transactions in the log must
still be performed
 Faster recovery from crash, removes chance of inconsistency of
metadata

Operating System Concepts – 10th Edition 14.38 Silberschatz, Galvin and Gagne ©2018

16
 Example: WAFL File System
 Used on Network Appliance “Filers” – distributed file system
appliances
 “Write-anywhere file layout”
 Serves up NFS, CIFS, http, ftp
 Random I/O optimized, write optimized
• NVRAM for write caching
 Similar to Berkeley Fast File System, with extensive modifications
 The WAFL File Layout

Operating System Concepts – 10th Edition 14.39 Silberschatz, Galvin and Gagne ©2018

Snapshots in WAFL

Operating System Concepts – 10th Edition 14.40 Silberschatz, Galvin and Gagne ©2018

17
 The Apple File System

Operating System Concepts – 10th Edition 14.41 Silberschatz, Galvin and Gagne ©2018

End of Chapter 4.2

Operating System Concepts – 10th Edition Silberschatz, Galvin and Gagne ©2018

18
 Linked Allocation

 Each file is a linked list of disk blocks: blocks may be scattered

anywhere on the disk

block = pointer

 Mapping
Q
LA/511
R
Block to be accessed is the Qth block in the linked chain of blocks
representing the file.

Displacement into block = R + 1

Operating System Concepts – 10th Edition 14.43 Silberschatz, Galvin and Gagne ©2018

19