Chipsets and the Memory Map

I’m going to write a few articles about the inside of a computer in an attempt to explain how the modern kernel works. Hobbyists and programmers who are interested in these things are often inexperienced, and I hope to help them. We focus on Linux, Windows, and Intel processors. Exploring internals is a hobby of mine. I have written a lot of code running in kernel mode, but haven’t done it in a long time. In this first article, LET’s describe the Intel-based motherboard layout, how the CPU accesses memory, and the system memory mapping.

First, let’s take a look at how the components of today’s Intel computers are connected. Here’s a look at the main components of the motherboard, complete with ugly color schemes:

Schematic of a modern motherboard, north bridge and South bridge make up the chipset.

Looking at this diagram, the key thing to remember is that the CPU has no idea what it’s connecting to. The CPU communicates with the outside world through its pins, regardless of what the outside world is, whether it is a motherboard in a computer, an oven, a network router, a brain-computer interface, or a CPU testing platform. There are three main ways for a CPU to communicate with the outside world: memory address space, I/O address space, and interrupt. For now, we only care about the motherboard and memory.

On the motherboard, the CPU’s interface to the outside world is the front-side Bus that connects one end to the North Bridge. Whenever the CPU reads or writes memory, it passes through this Bus. The CPU uses some pins to send the physical memory address it wants to read and write, while other pins are responsible for sending and receiving read and write data. The Intel Core 2 QX6600 processor has 33 pins for transmitting physical memory addresses (so it can access 233 storage units) and 64 pins for sending and receiving data (so data is transmitted in 64-bit or 8-byte blocks). This allows the CPU to address 64GB (233 by 8 bytes) of physical memory, although most chipsets can only handle 8GB of RAM.

Address buses are one-way.

Now, the problem is that we tend to think of memory only in terms of RAM, which is what our programs are reading and writing all the time. Granted, most memory requests from programs are routed through the Northbridge to the RAM module, but most is not all. Physical memory addresses are also used to communicate with various devices on the motherboard (this communication is called memory-mapped I/O). These devices include video cards, most PCI cards (such as scanners or SCSI cards), and flash memory for the BIOS.

When Northbridge receives a physical address request, it decides where the physical address goes: RAM? Or graphics card? This routing policy is determined by memory address mapping. For each region of physical memory address, the memory map knows which device that region belongs to. Most of the addresses are mapped to RAM, but when they are not mapped, the memory map tells Northbridge which device should service the requests for those addresses. These memory address maps away from RAM modules create a history hole between 640KB and 1MB in PC memory. When memory addresses are reserved for graphics cards and PCI devices, the holes are even larger. This is why 4GB of RAM is problematic for 32-bit operating systems. The /proc/iomem file on Linux lists these address-range mappings in full. The figure below shows a typical memory map for the top 4GB physical memory addresses in an Intel PC:

The first 4 GB memory layout on Intel systems

The actual address and range depends on the specific motherboard and the devices present in the computer, but most Core 2 systems are close to this layout. All brown areas are mapped away from RAM. Keep in mind that these are physical addresses used on the motherboard bus. Inside the CPU (for example, in the programs we run and write), memory addresses refer to logical addresses that must be translated by the CPU into physical addresses before they can be accessed on the memory bus.

The rules for converting logical addresses to physical addresses are complex, depending on which mode the CPU is running in (real mode, 32-bit protected mode, 64-bit protected mode). Regardless of the translation mechanism, THE CPU pattern determines how much physical memory can be accessed. For example, if the CPU is running in 32-bit protected mode, it can only address up to 4GB (PAE is an exception, of course, but we’ll leave it at that). Since the 1GB or so high physical address is mapped to other devices on the motherboard, the CPU can only efficiently use about 3GB of RAM (sometimes less — I have a Vista machine that has only 2.4GB available). If the CPU was in real mode, it could access only 1MB of physical memory (which was the only mode that early Intel processors could provide). A CPU running in 64-bit mode can physically address 64GB of space (though few chipsets support that much RAM). In 64-bit mode, physical sites above the total system memory can be used to gain access to RAM areas that were stolen by motherboard devices. This is called reclaiming Memory, and it’s done with the help of the chipset.

That’s all you need to prepare for the next article, which describes the boot process that jumps into the kernel from boot to BootLoader. If you want to learn more about the topic of this article, I recommend you read the Intel Manual. I’m big into primary sources overall, but the Intel manuals in particular are well written and accurate. Here are a few:

  • For Intel G35 Chipset Records the typical Core 2 processor Chipset. This is the main source of this article.
  • For Intel Core 2 quad-core Q6000 Sequence is a processor Datasheet. It records every pin in the processor (there aren’t that many, really, after you group them). Although it is mysterious in some places, it is still very delicious.
  • Intel Software Developer’s Manuals were quite good. They are not dense tomes, but beautifully explain all sorts of things about architecture. Volumes 1 and 3A are full of dry stuff (don’t be put off by the name, “Volume” is small and you can read it selectively).
  • Padraig Brady suggested I link to Ulrich Drepper’s excellent paper on memory. I was going to do this in an article on memory, but after all, it’s okay to link people.