Linux SMP FAQ

Linux SMP FAQ
David Mentr�, [email protected]
v0.47, 20 november 1998

This FAQ review main issues (and I hope solutions) related to SMP con
figuration under Linux.
______________________________________________________________________

Table of Contents

1. Introduction

2. What's new ?

3. Questions related to any architectures

3.1 Kernel side
3.2 User side
3.3 SMP Programming
3.3.1 Parallelization methods
3.3.2 The C Library
3.3.3 Languages, Compilers and debuggers
3.3.4 Other libraries
3.3.5 Other points about SMP Programming

4. Intel architecture specific questions

4.1 Why it doesn't work on my machine?
4.2 Possible causes of crash
4.3 Motherboard specific information
4.3.1 Motherboards with known problems
4.3.2 Motherboards with NO known problems
4.4 How to obtain maximum performance ? (not strictly SMP related)
4.4.1 From disks

5. Useful pointers

5.1 Various
5.2 Multithreaded programs and library
5.3 SMP specific patches
5.4 Parallelizing/Optimizing Compilers for 586/686 machines (

6. Glossary

7. List of contributors

______________________________________________________________________

1. Introduction

Linux can work on SMP (Symetric Multi-Processors) machines. SMP
support has started with the 2.0 family and has been improved in the
2.1 (future 2.2) saga.

FAQ maintained by David Mentr� ([email protected]). The latest
edition of this FAQ can be found at

� http://www.irisa.fr/prive/mentre/smp-faq/ (France)

� http://www.phy.duke.edu/brahma/smp-faq/ (USA)

If you want to contribute to this FAQ, I would prefer a diff against
the SGML version <http://www.irisa.fr/prive/mentre/smp-faq/smp-
faq.sgml> of this document, but any remarks (in plain text) will be
greatly appreciated. If you send me an email about this FAQ, please
include a tag like [Linux SMP FAQ] in the Subject: field of your e-
mail. It helps me automatically sort mails (and you will have a faster
reply ;)).

This FAQ is an improvement of a first draft made by Chris Pirih.

All information contained in this FAQ is provided "as is." All
warranties, expressed, implied or statutory, concerning the accuracy
of the information of the suitability for any particular use are
hereby specifically disclaimed. While every effort has been taken to
ensure the accuracy of the information contained in this FAQ, the
authors assume(s) no responsibility for errors or omissions, or for
damages resulting from the use of the information contained herein.

2. What's new ?

v0.47, 20 november 1998

� Added that 2.0.36 as the MTRR patch (related to the BogoMips
problem)

v0.46, 10 november 1998

� Update about Epox KP6-LS motherboards

v0.45, 25 october 1998

� Corrected an error regarding /proc/stat file

� Added a pointer to CESDIS Ethernet Linux Drivers site

v0.44, 14 october 1998

� Updated the link to the web page: Motherboards rumored to run
Linux SMP

� Added Jakob explanation how to time SMP systems with 2.0 kernels

v0.43, 9 september 1998

� Updated first question in section 3.1

� Updated mt-Mesa link: multi-threaded is now included as
experimental in the Mesa distribution

v0.42, 2 september 1998

� Minor cosmetic update in sect 3.3

� Two links (multithreaded Mesa and SMP performance) marked
outdated

� Updated the item about threads and exceptions in C++ (sect 3.3)

v0.41, 1 september 1998

� Added a major section: "3.3 SMP Programming" written by Jakob
�stergaard

� moved some item of section "3.2 User side" in sect 3.3

v0.40, 27 august 1998

� Updated section 3.1, item 7: processor affinity

v0.39, 27 august 1998

� Updated needed Award BIOS version for Tyan motherboards (hASCII)

� Added an item on IRQ in the crash section (me and hASCII)

� Added good support of Asus P2B-DS (Ulf Rompe)

� Added another smp-list archive in pointer section (Hank
Leininger)

v0.38, 8 august 1998

� Added a pointer to the Linux Threads FAQ

v0.37, 30 July 1998

� Emil Briggs is working on parallel plugins for Gimp (see "Is
there any threaded programs or library?", sect. "User side")

v0.36, 26 July 1998

� Thanks to Jakob �stergaard, two changes in "Possible causes of
Crash"

� Changed 2.0.33 to 2.0.35 (latest stable)

� Added a "BIOS related causes of failure"

v0.35, 14 July 1998

� Added N440BX Server Board in Motherboards with NO problems

� Added a succes story for GigaByte motherboard with BIOS upgrade

� Added a "How to obtain maximum performance ?" section (waiting
for your contributions ;)

v0.34, 10 june 1998

� Added a "Parallelizing/Optimizing Compilers for 586/686
machines" section in section "Useful Pointers", thanks to Sumit
Roy

� Corrected a mispelling, "Asus P/I-UP5" is in fact "Asus P/I-
P65UP5"

v0.33, 3 june 1998

� Yet another success story for a GigaByte DLX Motherboard.

� A tip for Tyan motherboards, disable the "DRAM Fast Leadoff"
BIOS option

v0.32, 27 may 1998

� Asus P/I-UP5 added in the motherboard-with-NO-problem section

v0.31, 18 may 1998

� Elitegroup P6LX2-A works with 2.1.100 and 101

� Bugs should be reported to [email protected]

v0.30, 12 may 1998

� SuperMicro is now in the motherboard-with-NO-problem section

v0.29, 11 may 1998

� A success story for a GigaByte 686 motherboard with 2.1.101

� Added a new item in the "User Side" section: "Is there any
threaded programs or library?"

� OpenGL Mesa library is beeing multithreaded. Cool! See the new
section for details.

v0.28, 09 may 1998

� A US mirror of this FAQ is now available (see Introduction)

� Merge of the two confusing Gigabyte 686 entries

v0.27, 05 may 1998

� New info for the Adaptec and TekRam drivers

� Micronics W6-LI motherboard works under SMP

3. Questions related to any architectures

3.1. Kernel side

1. Does Linux support multi-threading? If I start two or more
processes, will they be distributed among the available CPUs?

Yes. Processes and kernel-threads are distributed among processors.
2. What kind of architectures are supported in SMP?

From Alan Cox:
SMP is supported in 2.0 on the hypersparc (SS20, etc.) systems
and Intel 486, Pentium or higher machines which are Intel
MP1.1/1.4 compliant. Richard Jelinek adds: right now, systems
have been tested up to 4 CPUs and the MP standard (and so Linux)
theoretically allows up to 16 CPUs.

SMP support for UltraSparc, SparcServer, Alpha and PowerPC
machines is in progress in 2.1.x.

From Ralf B�chle:
MIPS, m68k and ARM does not support SMP; the latter two probly
won't ever.

That is, I'm going to hack on MIPS-SMP as soon as I get a SMP
box ...

3. How do I make a Linux SMP kernel?

Uncomment the SMP=1 line in the main Makefile
(/usr/src/linux/Makefile).

AND

enable "RTC support" (from Robert G. Brown). Note that inserting
RTC support actually doesn't afaik prevent drift, but according to
a discussion [Robert G. Brown] remember from a year ago or so it
can prevent lockup when the clock is read at boot time. A note from
Richard Jelinek says also that activating the Enhanced RTC is
necessary to get the second CPU working (identified) on some
original Intel Mainboards.

AND

do NOT enable APM! APM and SMP are not compatible, and your system
will almost certainly (or at least probably ;)) crash under boot if
APM is enabled (Jakob Oestergaard). Alan Cox confirms this : 2.1.x
turns APM off for SMP boxes. Basically APM is undefined in the
presence of SMP systems, and anything could occur.

You must rebuild all your kernel and kernel modules when changing
to and from SMP mode. Remember to make modules and make
modules_install (from Alan Cox).

4. How do I make a Linux non-SMP kernel?

Comment the SMP=1 line in the Makefile (and not set SMP to 0).

You must rebuild all your kernel and kernel modules when changing
to and from SMP mode. Remember to make modules and make
modules_install.

5. How can I tell if it worked?

cat /proc/cpuinfo

Typical output (dual PentiumII):

______________________________________________________________________
processor : 0
cpu : 686
model : 3
vendor_id : GenuineIntel
stepping : 3
fdiv_bug : no
hlt_bug : no
fpu : yes
fpu_exception : yes
cpuid : yes
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmo
v mmx
bogomips : 267.06

processor : 1
cpu : 686
model : 3
vendor_id : GenuineIntel
stepping : 3
fdiv_bug : no
hlt_bug : no
fpu : yes
fpu_exception : yes
cpuid : yes
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic 11 mtrr pge mca cmo
v mmx
bogomips : 267.06
______________________________________________________________________

6. What is the status of converting the kernel toward finer grained
locking and multithreading?

2.1.x has signal handling, interrupts and some I/O stuff fine grain
locked. The rest is gradually migrating. All the scheduling is SMP
safe

7. Does Linux SMP support processor affinity?

Standard kernel
No and Yes. There is no way to force a process onto specific
CPU's but the linux scheduler has a processor bias for each
process, which tends to keep processes tied to a specific CPU.

Patch
Yes. Look at PSET - Processor Sets for the Linux kernel
<http://isunix.it.ilstu.edu/~thockin/pset/>:

The goal of this project is to make a source compatible
and functionally equivalent version of pset (as defined
by SGI - partially removed from their IRIX 6.4 kernel)
for Linux. This enables users to determine which
processor or set of processors a process may run on.
Possible uses include forcing threads to seperate proces
sors, timings, security (a `root' only CPU?) and probably
more.

8. Where should one report SMP bugs to?

Please report bugs to [email protected].

3.2. User side

1. Do I really need SMP?

If you have to ask, you probably don't. :)

2. How does one display mutiple cpu performance?

Thanks to Samuel S. Chessman, here is some useful utilities:

Character based:
http://www.cs.inf.ethz.ch/~rauch/procps.html

Basically, it's procps v1.12.2 (top, ps, et. al.) and some
patches to support SMP.

Graphic:
xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included)
the cpuX entry in /proc/stat file.

The official homepage for xosview is:
http://lore.ece.utexas.edu/~bgrayson/xosview.html

The various forissier's kernel patches are at: http://www-
isia.cma.fr/~forissie/smp_kernel_patch/

3. How can I enable more than 1 process for my kernel compile?

use:

___________________________________________________________________
# make [modules|zImage|bzImages] MAKE="make -jX"
where X=max number of processes.
WARNING: This won't work for "make dep".
___________________________________________________________________

With a 2.1.x like kernel, see also the file
/usr/src/linux/Documentation/smp for specific instruction.

BTW, since running multiple compilers allows a machine with sufficient
memory to use use the otherwise wasted CPU time during I/O caused
delays make MAKE="make -j 2" -j 2 actually even helps on uniprocessor
boxes (from Ralf B�chle).

4. Why the time given by the time command is false ? (from Joel
Marchand)

In the 2.0 series, the result given by the time command is false.
The sum user+system is right *but* the spreading between user and
system time is false.

More precisely: "The explanation is, that all time spent in
processors other than the boot cpu is accounted as system time. If
you time a program, add the user time and the system time, then you
timing will be almost right, except for also including the system
time that is correctly accounted for" (Jakob �stergaard).

This bug in corrected in 2.1 series.

5. How will my application perform under SMP?

Look at SMP Performance of Linux [OUTDATED]
<http://www.interlog.com/~mackin/linux-smp.html> which gives useful
hints how to bench a specific machine (from a post made by Cameron
MacKinnon).

3.3. SMP Programming

Section made by Jakob �stergaard.

This section is intended to outline what works, and what doesn't, when
it comes to programming multi-threaded software for SMP Linux.

3.3.1. Parallelization methods

1. POSIX Threads

2. PVM / MPI Message Passing Libraries

3. fork() -- Multiple processes

Since both fork() and PVM/MPI processes usually does not share memory,
but either communicate by means of IPC or a messaging API, they will
not be described further in this section. They are not very specific
to SMP, since they are used just as much - or more - on uniprocessor
computers, and clusters thereof.

Only POSIX Threads provide us with multiple threads sharing ressources
like - especially - memory. The exact same thing that makes a SMP
machine special, by allowing many processors to share their memory. To
use both (or more ;) processors of an SMP, use a kernel-thread
library. A good library is the LinuxThreads, a pthread library made by
Xavier Leroy <http://pauillac.inria.fr/~xleroy/linuxthreads/> which is
now integrated with glibc2 (aka libc6).

Multi-threading has never been really popular in the UN*X world
though. For some reason, applications requiring multiple processes or
threads, has mostly been written using fork(). Therefore, when using
the thread approach, one runs into problems of incompatible (not
thread-ready) libraries, compilers, and debuggers. GNU/Linux is no
exception to this. Hopefully the next few sections will sched a little
light over what is currently possible, and what is not.
3.3.2. The C Library

Older C libraries are not thread-safe. It is very important that you
use GNU LibC (glibc), also known as libc6. Earlier versions are
ofcourse possible to use, but it will cause you much more trouble than
upgrading your system will, well probably :)

If you want to use GDB to debug your programs, you should get the
patches or SRPMS available for glibc at
http://odin.appliedtheory.com/debug_thread_rpms/RedHat_5.1/ which will
allow GDB to work with threads.

It is not nessecary to patch glibc in any way just to make it work
with threads. If you do not need to debug the software (this could be
true for all machines that are not development workstations), there is
no need to patch glibc.

3.3.3. Languages, Compilers and debuggers

There is a wealth of programming languages available for GNU/Linux,
and many of them can be made to use threads one way or the other (some
languages like Ada and Java even have threads as primitives in the
language).

This section will, however, currently only describe C and C++. If you
have experience in SMP Programming with other languages, please
enlighten us.

GNU C and C++, as well as the EGCS C and C++ compilers work with the
thread support from the standard C library (glibc). There are however
a few issues:

1. When compiling C or C++, use the -D_REENTRANT define in the
compiler command line. This is necessary to make certain error-
handling functions work.

2. When using C++, If two threads throw exceptions concurrently, the
program will segfault. The compiler does not generate thread-safe
exception code.

The workaround is to put a
pthread_mutex_lock(&global_exception_lock) in the constructor(s) of
every class you throw(), and to put the corresponding
pthread_mutex_unlock(...) in the destructor. It's ugly, but it
works. This solution was given by Markus Ferch.

The GNU Debugger GDB as of version 4.17, is incapable of handling
threads, at least in any sensible way. There is however a patched
version available at
http://odin.appliedtheory.com/debug_thread_rpms/RedHat_5.1/, for
RedHat Linux at least. It's probably quite easy to use the standalone
patches on any other GNU/Linux distribution. Note that you must also
patch your glibc.

Note that core-dumps are of no use, when using multiple threads.
Somehow the core dump is attached to one of the currently running
threads, and not to the program as a whole. Therefore, whenever you
are debugging anything, run it from the debugger.

Hint: If you have a thread running haywire, like eating 100% CPU time,
and you cannot seem to figure out why, here is a nice way to find out
what's going on: Run the program straight from the shell, no GDB. Make
the thread go haywire. Use top to get the PID of the process. Run GDB
like gdb program pid. This will make GDB attach itself to the process
with the PID you specified, and stop the thead. Now you have a GDB
session with the offending thread, and can use bt and the likes to see
what is happening.

3.3.4. Other libraries

ElectricFence: This library is not thread safe. It should however be
possible to make it work in SMP environments, by inserting mutex locks
in the ElectricFence code.

3.3.5. Other points about SMP Programming

1. Where can I found more information about parallel programming?

Look at the Linux Parallel Processing HOWTO
<http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

Lots of useful information can be found at Parallel Processing
using Linux <http://yara.ecn.purdue.edu/~pplinux/>

Look also at the Linux Threads FAQ <http://linas.org/linux/threads-
faq.html>

2. Is there any threaded programs or library?

Yes. For programs, you should look at: Multithreaded programs on
linux <http://www.e.kth.se/~e94_bek/mthread.html> (I love
hyperlinks, did you know that ? ;))

As far as library are concerned, there is:

OpenGL Mesa library
Thanks to David Buccarelli, Andreas Schiffler and Emil Briggs,
it exists in a multithreaded version (right now [1998-05-11],
there is a working version that provides speedups of 5-30% on
some OpenGL benchmarks). The multithreaded stuff is now included
in the regular Mesa distribution as an experimental option. For
more information, look at the Mesa library
<http://www.ssec.wisc.edu/~brianp/Mesa.html>

BLAS
Pentium Pro Optimized BLAS and FFTs for Intel Linux
<http://www.cs.utk.edu/~ghenry/distrib/>

Multithreaded BLAS routines are not available right now, but a
dual proc library is planned for 1998-05-27, see Blas News
<http://www.cs.utk.edu/~ghenry/distrib/blasnews> for details.

The GIMP
Emil Briggs, the same guy who is involved in multithreaded Mesa,
is also working on multithreaded The GIMP plugins. Look at
http://nemo.physics.ncsu.edu/~briggs/gimp/index.html for more
info.

4. Intel architecture specific questions

4.1. Why it doesn't work on my machine?

1. Can I use my Cyrix/AMD/non-Intel CPU in SMP?

Short answer: no.

Long answer: Intel claims ownership to the APIC SMP scheme, and
unless a company licenses it from Intel they may not use it. There
are currently no companies that have done so. (This of course can
change in the future) FYI - Both Cyrix and AMD support the non-
proprietary OpenPIC SMP standard but currently there are no
motherboards that use it.

2. Why doesn't my old Compaq work?

Put it into MP1.1/1.4 compliant mode.

3. Why doesnt my ALR work?

From Robert Hyatt : ALR Revolution quad-6 seems quite safe, while
some older revolution quad machines without P6 processors seem
"iffy"...

4. Why does SMP go so slowly? or Why does one CPU show a very low
bogomips value while the first one is normal?

From Alan Cox: If one of your CPU's is reporting a very low
bogomips value the cache is not enabled on it. Your vendor probably
provides a buggy BIOS. Get the patch to work around this or better
yet send it back and buy a board from a competent supplier.

Another work-around is to use the 2.0.36 kernel. It contains the
MTRR patch which should solve this problem (select option "Handle
buggy SMP BIOSes with bad MTRR setup" in the "General setup" menu).

5. I've heard IBM machines have problems

Some IBM machines have the MP1.4 bios block in the EBDA, allowed
but not supported by <2.1.80. Please update to the right kernel.

There is an old 486SLC based IBM SMP box. Linux/SMP requires
hardware FPU support.

6. Is there any advantage of Intel MP 1.4 over 1.1 specification?

Nope (according to Alan :) ), 1.4 is just a stricker specs of 1.1.

7. Why does the clock drift so rapidly when I run linux SMP?

This is known problem with IRQ handling and long kernel locks in
the 2.0 series kernels. Consider upgrading to a later 2.1 kernel
(not garenteed to work).

From Jakob Oestergaard: Or, consider running xntpd. That should
keep your clock right on time. (I think that I've heard that
enabling RTC in the kernel also fixes the clock drift. It works for
me! but I'm not sure whether that's general or I'm just being
lucky)

8. Why are my CPU's numbered 0 and 2 instead of 0 and 1 (or some other
odd numbering)?

The CPU number is assigned by the MB manufacturer and doesn't mean
anything. Ignore it.

9. My SMP system is locking up all the time. Black screen, nothing in
the logs. Help!

If you're running a 2.0 kernel, consider upgrading to later 2.0.32+
kernels or apply Leonard Zubkoff's deadlock patch. If you still
have deadlocks, apply Ingo Molnar's deadlock detection patch and
post the results (against your System.map) to linux-smp or linux-
kernel. You might also consider running a 2.1 kernel.

4.2. Possible causes of crash

You'll find in this section some possible reasons for a crash of an
SMP machine (credits are due to Jakob �stergaard for this part). As
far as I (david) know, theses problems are Intel specific.

� Cooling problems

From Ralf B�chle: [Related to case size and fans] It's important
that the air is flowing. It of course can't where cables etc. are
preventing this like in too small cases. On the other side I've
seen oversized cases causing big problems. There are some tower
cases on the market that actually are worse for cooling than
desktops. In short, the right thing is thinking about aerodynamics
in the case. Extra cases for hot peripherals are usefull as well.

� Bad memory

Don't buy too cheap RAM and don't use mixed RAM modules on a
motherboard that is picky about it.

Especially Tyan motherboards are known to be picky about RAM speed
(see the Tyan paragraph below for a possible solution).

� Bad combination of different stepping CPUs

Check /proc/cpuinfo to see that your CPUs are same stepping.

� You are running 2.0.35 aren't you ?

If you run 2.0.31 or 2.1.xx you can't be sure that SMP is stable.
2.0.35 is the right kernel for a production system. 2.1.xx kernels
perform better, but they are development releases and should NOT be
considered stable!

� If your system is unstable, then DON'T overclock it!

...and even if it is stable, DON'T overclock.

From Ralf B�chle: Overclocking causes very subtile problems. I
have a nice example, one of my overclocked old machines
misscomputes a couple of pixels of a 640 x 400 fractal. The
problem is only visible when comparing them using tools. So better
say never, nuncas, jamais, niemals overclock.

� 2.0.x kernel and fast ethernet (from Robert G. Brown)

2.0.X kernels on high performance fast ethernet systems have
significant (and known) problems with a race/deadlock condition in
the networking interrupt handler.

The solution is to get the latest 100BT development drivers from
CESDIS Linux Ethernet device drivers site
<http://cesdis.gsfc.nasa.gov/linux/drivers/> (ones that define
SMPCHECK).

� A bug in the 440FX chipset (from Emil Briggs)

If you had a system using the 440FX chipset then your problem with
the lockups was possibly due to a documented errata in the chipset.
Here is a reference

References: Intel 440FX PCIset 82441FX (PMC) and 82442FX (DBX)
Specification Update. pg. 13

http://www.intel.com/design/pcisets/specupdt/297654.htm

The problem can be fixed with a BIOS workaround (Or a kernel patch)
and in fact David Wragg wrote a patch that's included with Richard
Gooch's MTTR patch. For more information and a fix look here:

http://nemo.physics.ncsu.edu/~briggs/vfix.html

� DONT run emm386.exe before booting linux SMP

From Mark Duguid, dumb rule #1 with W6LI motherboards. ;)

� If the machine reboots/freezes after a while, there can be two good
BIOS + memory related reasons (from Jakob �stergaard)

� If the BIOS has settings like "memory hole at 16M" and/or "OS/2
memory > 64MB", try disabling them both. Linux does not always
react well with theese options.

� If you have more than 64 MB of memory in the machine, and you
specified the exact number manually in the LILO configuration, you
should specify one MB less than you actually have in the machine.
If you have 128 MB, you lilo.conf line looks like:
append="mem=127M"

� Be aware of IRQ related problems

Sometime, some cards are not recognized or can trigger IRQ
conflicts. Try shuffle cards on slots in different ways.

Contributed by hASCII : removing an " append="hisax=9,2,3"" line in
lilo.conf allowed using a kernel from the 2.1.xx series with
activated ISDN + Hisax support. Kernels from the 2.0.xx series
doesn't make problems like this.

Some hardware is also known to cause problems. This includes:

� Adaptec SCSI controllers

Latest news (05 may 1998):

The latest version of this driver (5.0.13 at this writting) should
be SMP safe (both in 2.0 and 2.1 kernels). Doug had verified it
through code review and intensive tests (on a dual PII system).

This said, some machines still have problems. Doug has kindly given
some explanations:

� Doug: The simple fact of the matter is that this version of the
aic7xxx driver places a larger demand on DMA transfers than the old
driver did, especially for for smaller DMA transactions. So, the
point should be made that this version of the driver *is* more
demanding on your system than either the 4.1 or 4.1.1 driver, and
that certain marginal systems may get hit by this problem (such as
my own P133 system is).

� In 2.1 kernels, they are still problems (with IRQ) with the kernel
(2.1.99) and they are not related to this specific driver. People
are working on this.

� 3Com 3c905 cards

Some work, some don't. Try disabling busmastering if your system is
unstable.

4.3. Motherboard specific information

Please note: Some more specific information can be found with the list
of Motherboards rumored to run Linux SMP <http://www.nlug.org/smp/>

4.3.1. Motherboards with known problems

� GA686DLX (Andrew Crane)

Same BIOS related BogoMIPS problem as other motherboards.

Solution from Alan Cox: Congratulations, send the bill for your
hair damage to the supplier. You have yet another SMP box with
faulty bios. There is a patch for 2.0.x on www.uk.linux.org and
there are people working on generic MTRR handling for 2.1.x

From Claus-Justus Heine: I'm able to boot above MB with 2.1.90 +
Ingo's apic + Richard's mtrr patches. From David Mentr�: the kernel
> 2.1.99 have the MTRR patch included and Florian Hinzmann told me
that with 2.1.101, both CPUs are detected fine.

David Maslen tells: Fortunately I suspect these problems have been
fixed, and would like to report my success so that others won't be
put off what seems to be a nice motherboard. [...] My system has
been running for about 3 days without a lockup (72hrs) Kernel
2.1.103 compiled easily. /proc/cpuinfo says the cpus are both
there. This was also true of 2.0.33.

Other possible solution: BIOS upgrade (see http://www.giga-
byte.com).

(14 July 1998)Volker Reichelt confirms that with a BIOS upgrade
from v1.14 to v1.20 everything seems to work fine.

� EPoX KP6-LS (Christopher Allen Wing, 16 march 1998)

It appears to have the same BIOS related BogoMIPS problem as other
motherboards. (one CPU only gives about 3 BogoMIPS, the other gives
the full amount) All 2.0.x kernels lock up soon after booting, late
2.1.x kernels run slowly but don't seem to lock up. There is no
BIOS upgrade available (yet). I wrote the manufacturer but have not
received a reply.

(update, 10 november 1998) First of all, there are 2 revisions on
the KP6-LS: (1) Revision 2 (production release, has a power-leak on
the second processor connector - To fix this problem, contact Epox
support and they will guide you for further assistence. - For some
boards it is needed to replace the entire board. Cause the leak is
corrected in the middle of this revision). (2) Revision 3
(production release). For both the revisions is a patch avaliable
from their site (http://www.epox.com/support/bios.html). Please
make sure that Epox has been contacted about the bios upgrade
before applying. After applying the patch, kernel 2.0.35 and all
other expirimental kernels from 2.1.110 up to 2.1.125 report a
correct BogoMIPS index. (Niels Ammerlaan)

� Tyan

Tyan motherboards are known to be picky about RAM speed (Jakob
Oestergaard).

From Doug Ledford about the onboard aic-7895 SCSI controller (for
which he wrote the driver): "BTW, make sure you have at least BIOS
version 1.16 on that Tyan motherboard. The 1.15 and below BIOS
versions have a bug relating to IRQ allocation for the 7895 SCSI
controller" (submitted by Szakacsits Szabolcs).

(27 august 1998) From hASCII: If you have a tyan with award bios,
the newest version you can get is 1.02.

But, as motherboard problems are more close to grayshade than black
and white ;), Richard Jelinek (from S.u.S.E.) tells that they
sell/sold several Tyan Titan Pro (Dual-PPro) Motherboards. They
work perfectly with SMP.

(3 june 1998) maybe an explanation found by Leonard Zubkoff and
told tp me by Sean Reifschneider: The Tyan Titan Pro motherboards
with 3.03 BIOS version (dated 10/31/96) has an option labled "DRAM
Fast Leadoff", which MUST be disabled.

� MS-6114

More details for this motherboard at
http://www.msi.com.tw/product/6114/6114.htm
<http://www.msi.com.tw/product/6114/6114.htm>

Solution: BIOS upgrade

Somebody experienced solid hangs (nothing in the log files) under
constant load of about 5 running processes within less than 12
hours with AMI BIOS v1.1. v1.4b3 runs without problems.

4.3.2. Motherboards with NO known problems

� AIR P6NDP and P6NDI (Leonard N. Zubkoff)

My primary production machine is based on an AIR P6NDP and one of
my test machines uses a P6NDI. Both seem to be fine motherboards
in my experience. The P6NDI BIOS is a little conservative in its
programming of the Natoma chipset for 50ns EDO, but a minor tweek
to one register in rc.local took care of that.

� AIR 54CDP (Chris Mauritz)

You can also list the following motherboard as working with no
problems:

AIR 54CDP motherboard / EISA/PCI / onboard aic7870 / dual P120 /
Redhat 5.0 (2.0.32 and 2.0.33 kernels)

However, Jon Lewis adds the following comments: Has Chris tried
putting multiple PCI cards in these motherboards...say a SCSI card
and a network card? Due to BIOS brain damage, the 54CDP insists on
having all PCI devices share a single IRQ. AIR lead me on for 2
weeks and then said "there have been / will be no updates for your
board". This means I can't have Tulip network cards and PCI SCSI
without hacking the drives (which seems to work, though I was
cautioned not to do it).

� HP XU 6/200 (Jean-Francois Micouleau)

Works with 2.0 and 2.1 kernels. Some problems under high network
load with 2.0.x kernel. Works under 2.1.78 with Ingo Molnar IO-APIC
patch.

� Elitegroup P6FX2-A (Benedikt Heinen)

Had this mainboard running with ONE PPro on it for several months,
and since about a year, it's running without problems with TWO PPro
200MHz. The only crashes this machine ever experienced were before
Leonard Zubkoff's deadlock-patches for Linux 2.0.30... ;)

Elitegroup P6FX2-A / ISA/PCI / Dual PPro200 / Debian "hamm"

� Elitegroup P6LX2-A (Ulf Nielsen)

This motherboard works with a post 2.1.99 kernel (100 and 101
tested). All kernels pre 2.1.100 does not work on P6LX2-A.

� QDI P6I440LX/DP "Legend IV" (Tony Kocurko)

The QDI P6I440LX/DP "Legend IV" dual Pentium II motherboard now
boots properly with Linux-SMP. As of 0326 hours on 15 April, there
is a necessary BIOS upgrade to version 1.6 at the Canadian site
(http://www.qdi.ca). Previously, the Legend IV would bring up the
first processor normally, but the /proc/cpuinfo file would show the
second processor barely alive. The BIOS update seems to have
corrected the problem.

� Asus P2L97-DS

ASUS P2L97-DS works great under Linux-SMP (from Richard Jelinek).

� Asus P/I-P65UP5 (Jukka Tainio)

I have run it with both 2x233 MMX Pentium and 2x200 PPRO. Both
boards are in heavy www-proxy use and work without problems. Kernel
version is 2.0.33.

� Asus P2B-DS (Ulf Rompe)

Asus P2B-DS is working without problems. It runs with two PII-266,
the onboard Adaptec 7890 (Patch from
ftp://ftp.dialnet.net/pub/linux/aic7xxx/ required) in U2W mode, an
Elsa Victory Erazor AGP an some obscure other hardware.

The only thing missing is support for the new SMP-aware APM, but
that is a general Linux problem.

� Micronics W6-LI

From Mark Garlanger: 2.0.33 is very stable on the board, and the
2.1.xx kernels are improving. People have also been able to run
other OSes including Windows NT(yuck...), FreeBSD-SMP, Rhapsody.

� SuperMicro

Thomas Schenk is running about 80 SMP machines with SuperMicro
motherboards without any problem. Kernel used: 2.0.33. No "bad
bogomips" problem for him (was I [david] wrong about this ?).

� N440BX Server Board

Moni Hollmann using a N440BX Server Board unter Linux (SMP) and no
Problems occured so far. RAM: 256 MB, 2x Pentium/II- 350
(Deschutes).

4.4. How to obtain maximum performance ? (not strictly SMP related)

4.4.1. From disks

Some useful information pointed out by Robert G. Brown, one of the
bigest contributor to this FAQ.

Date: Wed, 8 Jul 1998 09:12:41 +1000 (EST)
From: "Andrew Mc.Ghee" <[email protected]>

I believe there was work done on this at - (I've got the report sitting in
front of me, but there is no http reference, must have used UNIX netscape
to print it, but it was done by Eric Hendriks at cesdis.gsfc.nasa.gov)

Distilling it down, the paper presents

Interface Disk Arrangement Cpu Usage Throughput
=============================================================
IDE-PIO 4 Disks / 2 Channels 98% read 7.53 MB/s
58% write 4.84 MB/s
IDE-DMA 4 Disks / 2 Channels 34% read 11.78 MB/s
57% write 9.21 MB/s
IDE-DMA 2 Disks / 2 Channels 31% ----- 9.87 MB/s
SCSI 4 Disks / 1 Channel 53% read 13.41 MB/s
92% write 11.93 MB/s
SCSI 4 Disks / 2 Channels 77% read 21.99 MB/s
97% write 13.44 MB/s

The test where done using RAID0 via the md system, but of course the test
involved using a large, single contiguous data transfer.

Andrew Mc.Ghee
Mechanical Engineering
University of Queensland

5. Useful pointers

5.1. Various

� Parallel Processing using Linux
<http://yara.ecn.purdue.edu/~pplinux/>

� Linux Parallel Processing HOWTO
<http://yara.ecn.purdue.edu/~pplinux/PPHOWTO/pphowto.html>

� (outdated) Linux SMP home page
<http://www.uk.linux.org/SMP/title.html>

� linux-smp mailing list

To subscribe, send subscribe linux-smp in the message body at
[email protected]

To unsubscribe, send unsubscribe linux-smp in the message body at
[email protected]

Linux SMP archives <http://www.linuxhq.com/lnxlists/linux-smp/>

Linux SMP archives at progressive-comp.com <http://www.progressive-
comp.com/Lists/?l=linux-smp&r=1&w=2#linux-smp>

� pthread library made by Xavier Leroy
<http://pauillac.inria.fr/~xleroy/linuxthreads/>

� Motherboards rumored to run Linux SMP <http://www.nlug.org/smp/>

� procps <http://www.cs.inf.ethz.ch/~rauch/procps.html>

� xosview <http://lore.ece.utexas.edu/~bgrayson/xosview.html>

� SMP Performance of Linux [OUTDATED]
<http://www.interlog.com/~mackin/linux-smp.html>

� CESDIS Linux Ethernet device drivers site
<http://cesdis.gsfc.nasa.gov/linux/drivers/>

5.2. Multithreaded programs and library

� Linux Threads FAQ <http://linas.org/linux/threads-faq.html>

� Multithreaded programs on linux
<http://www.e.kth.se/~e94_bek/mthread.html>

� Pentium Pro Optimized BLAS and FFTs for Intel Linux
<http://www.cs.utk.edu/~ghenry/distrib/> (not available right now,
but a dual proc library is planned for 5/27/98, see Blas News
<http://www.cs.utk.edu/~ghenry/distrib/blasnews> for details)

� Mesa library <http://www.ssec.wisc.edu/~brianp/Mesa.html> (with
experimental multi-threading)

� Parallel plugins for The GIMP
<http://nemo.physics.ncsu.edu/~briggs/gimp/index.html>

5.3. SMP specific patches

� Forissier kernel patches <http://www-
isia.cma.fr/~forissie/smp_kernel_patch/>

� Patch for a bug in the 440FX chipset
<http://nemo.physics.ncsu.edu/~briggs/vfix.html>

� MTRR patch (latest version: 1.9)
<http://www.atnf.csiro.au/~rgooch/kernel-patches.html>

� PSET - Processor Sets for the Linux kernel
<http://isunix.it.ilstu.edu/~thockin/pset/>

5.4. ( Sumit Roy ) Parallelizing/Optimizing Compilers for 586/686
machines

� Pentium Compiler Group <http://www.goof.com/pcg/> creators of pgcc

� Absoft <http://www.absoft.com/> , Fortran 90 and Fortran 77
compilers

� The Portland Group, Inc. <http://www.pgroup.com/>, supports the
OpenMP <http://www.openmp.org> standard for Fortran parallelization
on Linux

� Pacific-Sierra Research Corporation <http://www.psrv.com/>, has a
free F90 compiler for Linux, as well as parallelizing compilers for
SMP Linux

� Applied Parallel Research <http://s006.infomall.org/index.html>,
currently have parallelizing compilers for WinNT

6. Glossary

� SMP Symetric Multi-Processors

� APIC Advanced Programmable Interrupt Controler

� thread A thread is a processor activity in a process. The same
process can have multiple threads. Those threads share the process
address space and can therefore share data.

� pthread Posix thread, threads defined by the Posix standard.

7. List of contributors

Many thanks to those who help me to maintain this FAQ :

� Tigran A. Aivazian

� Niels Ammerlaan

� Emil Briggs

� Robert G. Brown

� Samuel S. Chessman

� Alan Cox

� Andrew Crane

� Mark Duguid

� Jocelyne Erhel

� Byron Faber

� Mark Garlanger

� hASCII

� Claus-Justus Heine

� Benedikt Heinen

� Florian Hinzmann

� Moni Hollmann

� Robert Hyatt

� Richard Jelinek

� Tony Kocurko

� Doug Ledford

� Hank Leininger

� Cameron MacKinnon

� Joel Marchand

� David Maslen

� Chris Mauritz

� Jean-Francois Micouleau

� Ulf Nielsen

� Jakob Oestergaard

� Ulf Rompe

� Jean-Michel Rouet

� Ralf B�chle

� Volker Reichelt

� Sean Reifschneider

� Sumit Roy

� Thomas Schenk

� Szakacsits Szabolcs

� Jukka Tainio

� Simen Timian Thoresen

� El Warren

� Christopher Allen Wing

� Leonard N. Zubkoff