Configuration of the Memory Hole on 64-bit AMD Systems

The Problem

From the official Tyan FAQ:

=

Why does my OS see less than the total memory installed when I install 4GB or more of memory (typically 512MB less)?

The BIOS needs to overlay the APIC, ACPI Table, AGP Aperture and PCI MMIO (Memory-mapped I/O [see PCI Spec 2.3, Section 3.2.2 for more information]) over the last 512MB of the 4GB physical address space. OS accessible memory and these structures cannot both exist at the same place and this portion of DRAM is hidden and unavailable to the OS.

In order to reclaim this lost memory, it must be remapped to another area. This remapping needs to be enables in BIOS, supported by the CPU and the installed OS. Unfortunately, none of this information is in the motherboard or CPU manual.

Our System Configuration

Processor: Dual AMD 246 (2.0 GHz)

RAM: 4 X 1 GB DDR ECC RAM

Video Card: Nvidia 7800GT

Hard Drive: SATA 250GB

OS: Fedora Core 3; 64-bit

The Solution

1. Go to the BIOS page (F2 when booting). We have Phoenix BIOS revision 1.02 (10/2005)
2. Go to Advanced --> Hammer Configuration --> Memory Hole
3. Set Memory Hole Remapping to Software
4. Set Memory Hole amount to Manual 64 MB (seemed to automatically go to 1536 upon reboot - this works too)
5. Set IOMMU to enabled
6. Set IOMMU amount to 64 MB
7. Save and Reboot (F10)

Note:

In FC3, the stock kernel 2.6.9 did not work with the Memory Hole remapping enabled. We needed to update to the latest 2.6.12.2 legacy kernel.

Xwindows did not work upon first boot. To fix:

• Boot using a KNOPPIX "live" boot CD.
• Mount the / partition on the hard drive
• Edit the file /etc/inittab, change the default boot mode from 5 (graphics) to 3 (text)
• Edit /boot/grub/grub.conf to disable the hiddenmenu
• Reboot from hard drive.
• Download the 64-bit NVIDIA driver from NVIDIA.com
• Install (do this after updating kernel to 2.6.12, needs to be done whenever kernel is updated)
• During installation, tell it to recompile -- don't bother trying to download a version for the specific kernel
• Tell it to reconfigure the /etc/X11/xorg.conf file for you
• Test with startx. Video now works, but defaults to 800X600.
• Edit /etc/X11/xorg.conf. Relevant sections quoted here:

   
Section "Monitor"
        Identifier   "Monitor0"
        VendorName   "Monitor Vendor"
        ModelName    "Dell 1901FP (Digital)"
        HorizSync    30.0 - 80.0
        VertRefresh  56.0 - 76.0
        Option      "dpms"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Videocard0"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Viewport    0 0
        Depth       24
        Modes      "1280x1024"
    EndSubSection
EndSection

IOMMU Configuration

• After several hours of running both computers at ~99%, using a shared networked drive for processing, got overflows in IOMMU.
• This resulted in unmounting the hard drive, then remounting it read-only, thus ending all processes
• To prevent this, increased the IOMMU amount under "Memory Hole" Configuration in BIOS
  • previous (default) value was 64 MB.
  • Increasing to 128 MB or 256 MB helped, but overflow eventually occurred
  • Increased IOMMU to 512 MB -- seems OK (4-20-2006)
  • Memory leak in IOMMU: set both systems to 2 GB, takes 7-10 days to overflow
  • To prevent problems, reboot systems weekly

BIOS Update

• June 1 2006: found recent BIOS on TYAN site (1.02, March 2006), updated on cem02 and cem03 (old BIOS Oct 2005, 1.02)
• cem03: updated without problem
• cem02: memory error upon update. System would not boot until CMOS erased. Reset BIOS settings. startx would not work (Nvidia card not found), ethernet unreliable
• cem02: Retried BIOS update, nvidia card still not working. Reinstalled old BIOS (1.02) - everything ok again.
• cem02: still has problem on IRQ 7 shortly after booting (always had this problem), however, no visible effects

• Set ALLOWTOPICVIEW =

-- BillRice - 21 Mar 2006