Disco: Running Commodity Operating Systems on Scalable Multiprocessors
One-line Summary
Disco uses virtual machines to run multiple commodity operating systems on large-scale shared-memory multiprocessors. Disco VMM hides NUMA-ness from non-NUMA aware OSes, requires low effort to implement, and introduces moderate overhead due to virtualization.
Paper Structure Outline
Introduction
Problem Description
A Return to Virtual Machine Monitors
Challenges Facing Virtual Machines
Disco: A Virtual Machine Monitor
Disco's Interface
Implementation of Disco
Virtual CPUs
Virtual Physical Memory
NUMA Memory Management
Virtual I/O Devices
Copy-on-write Disks
Virtual Network Interface
Running Commodity Operating Systems
Necessary Changes for MIPS Architecture
Device Drivers
Changes to the HAL
Other Changes to IRIX
SPLASHOS: A Specialized Operating System
Experimental Results
Experimental Setup and Workloads
Execution Overheads
Memory Overheads
Scalability
Dynamic Page Migration and Replication
Related Work
System Software for Scalable Shared Memory Machines
Virtual Machine Monitors
Other System Software Structuring Techniques
ccNUMA Memory Management
Conclusions
Background & Motivation
The motivation is to enable existing commodity operating systems to handle Non-Uniform Memory Access (NUMA) architectures. Instead of modifying existing operating systems to run on scalable shared-memory multiprocessors, an additional layer (VM monitor) is inserted between the hardware and the OS.
Course notes by Prof. Andrea. Left: SMP (symmetrical multiprocessor uniform memory access machine), right: cc-NUMA
Cache-coherent Non-Uniform Memory Architecture (cc-NUMA) makes hardware scalable, while SMP ensures the same performance to all memory from everywhere. Both ensure correctness, though.
Design and Implementation
Disco is a layer between OSes and hardware
The advantages of using virtual machines in the context of this work are:
The Disco layer understands the NUMA architecture
It's a portability layer
Monitors are smaller and easier to understand & trust than operating systems
Allows to run different OSes concurrently (almost unmodified)
The drawbacks of using virtual machines are:
Overhead: cost of virtualizing
Time: VMM (Disco) acts as an emulator. Most instructions can just run, but privileged instructions + TLB instructions must be trapped & emulated
Pmake & Database do a lot of syscalls, often traps into Disco, which then goes to kernel. The extra 16% overhead for those workloads is due to the extra work handling TLB misses. The kernel time being less is because Disco zeros the pages (does work instead of IRIX).
Disco does a good job sharing buffer cache space across VMs and sharing IRIX text.
No migration + replication, just looking at how much more scalable is Disco than IRIX due to optimizations of not having locks in which IRIX does a bad job at. IRIX on 8-processor cc-NUMA machine. 2VM -> 8VM actually improves because Disco does not have bad lock
Much less time accessing remote memory, more local memory.
This paper started off VMWare (which was founded by authors of Disco in 1998 and successfully commercialized this work) and revived virtual machines for the next 20 years. Now VMs are commodities, and every cloud provider and virtually every enterprise uses VMs today.
New Vocabulary
IRIX: A variety of UNIX System V with BSD extensions.
