SPiN skrev:
Det var den, ja. Varför är den orättvis? FreeBSD gick det ju väldigt bra för.
Gammal som gatan? FreeBSD 5.1 och Linux 2.6 används ju.
Hur 5.1 presterar saknar värde idag, eftersom 5.2.1 är ute, med många förbättringar. SCHED_ULE istället för SCHED_4BSD och nya KSE-implementeringar m.m. Han uppdaterade t.o.m till -CURRENT, hur står det till i skallen då?
Han gjorde misstaget att mäta kraften hos 5.1, utan att uppmärksamma att det finns skräpkod i 5.x-serien som försämrar prestandan. Det räcker inte att bara stänga av DEBUGGING. Han verkar inte ens ha läst UPDATING där det tydligt står:
NOTE TO PEOPLE WHO THINK THAT FreeBSD 5.x IS SLOW:
FreeBSD 5.x has many debugging features turned on, in
both the kernel and userland. These features attempt to detect
incorrect use of system primitives, and encourage loud failure
through extra sanity checking and fail stop semantics. They
also substantially impact system performance. If you want to
do performance measurement, benchmarking, and optimization,
you'll want to turn them off. This includes various WITNESS-
related kernel options, INVARIANTS, malloc debugging flags
in userland, and various verbose features in the kernel. Many
developers choose to disable these features on build machines
to maximize performance.
När det gäller OpenBSD så gav Bert de Jong ett bra svar:
http://marc.theaimsgroup.com/?l=openbsd-misc&m=106660195122460&w=2
List: openbsd-misc
Subject: Re: OpenBSD Benchmarked... results: poor!
From: Bert de Jong <bert () ohreally ! com>
Date: 2003-10-19 22:15:04
I read the tests and many of the comments on them.
I was very curious about the test, but very disappointed with everything after
I read it.
The person who performed the tests was very naive when he said this ought to
be the end of flamewars about network performance/scalability among the
Linux/*BSD users. His test was just not good enough to be meaningful to
anyone in the respect of offering acceptable conclusions.
A proper test would be
- Performed on proper hardware. I mean, doing a network performance test on an
old laptop with inadequate harddisk size, with four very much server-oriented
operating systems, one of which (OpenBSD) couldn't even be installed on the
same part of the harddisk as the others! That's like organising a race
between a Lamborghini, a Lotus, a Porsche and a Ferrari in downtown New York
during rush hour, and disqualifying the Ferrari on the grounds that it is
difficult to park backwards.
- Unbiased. This one was very much biased against OpenBSD, and in favour of
Linux/'Leanux', as follows from many of the comments made by the tester.
- Performed by somebody who knew enough about installing and running all of
the OS's involved to run tests on them (the tester seems knowledgeable about
Linux, but is totally clueless as far as for instance OpenBSD is concerned.)
- Performed with a test programme that wasn't developed with a bias toward one
of the OS's in the test (the test programme involved was developed on Linux,
later 'ported to' *BSD.)
- Described plain fact, by someone who would be objective and who would avoid
showing emotions about the subject, much less a general favour or disgust
toward the OS's tested. This is especially important because of the sensitive
nature of the test subject. There are so many flamewars already!
- Described withoud prejudice even /if/ the tester happened to be more
familiar or friendly with one of the OS's tested. That would really help
improve the value of the test. Unprejudiced==scientifical==professional==a
virtue.
- Carefully giving minute details about the test conditions (hardware,
software, test programme details, OS installation details..) This test wasn't
remotely accurate, look how it even fails to mention at what time
OpenBSD-CURRENT was downloaded. That's crucial information.
- Compare equally. The test is already invalid because it compares an ancient
-STABLE NetBSD with a -CURRENT FreeBSD.
- Be clear about its subject. The test focuses exclusively on network
performance/scalability, and then goes on to praise or totally disqualify the
OS's tested solely on this ground. The test doesn't look at crucial aspects
like security, maintainability, documentation, correctness of design,
etcetera, but will still not hesitate to draw very un-subtle conclusions.
- Ask those who are in the know for comments on the test conclusions before
making the results public.
In this precise case, it would have prevented several stupid factual mistakes
from being published (the OpenBSD installation problem and IPv6
idiosyncrasies for instance.)
- Call into the test all relevant players, or at least represent different
groups properly.
Much as I dislike it - I'm an open source adept as well - Windows is used as a
server system on the public internet by many organisations. It should be
compared with the other systems in this test, and be given a fair chance. I
wouldn't have been surprised if it performed very well, seeing that the
TCP/IP stack of modern Windows versions has been largely copied from
FreeBSD..
Also I would have liked it if there were at least one proprietary Unix system
(such as Solaris) in the test. Just for the sake of the comparison.
- Look closer into the reasons, backgrounds, pros and cons of faults that the
tests find. Again, if the tester had done this, he would have found that some
of the badness he found wasn't a design mistake, but a design decision based
on healthily made trade-offs (security trade-offs in the case of, not
surprisingly, OpenBSD.)
- Bert
Att testet har uppdaterats efter de påtryckningar han har fått påvisar ju bara hur dåligt det var gjort från början.
