Discussion:
Hardware compatibility test: draft proposal
(too old to reply)
Wouter Verhelst
2008-08-20 04:24:07 UTC
Permalink
Hi,

As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware to test
whether Debian works on their system. The rationale for such a test:
while most of us know how Debian works and how one should test whether
their hardware actually works with Debian, it is not reasonable to
expect the same of hardware vendors for /all/ GNU/Linux distributions
out there. According to Bdale, currently all other major operating
system vendors, including the commercial Linux distributions, already
provide such a test to the hardware vendors. For this purpose, it is
okay if such a test is interactive to some extent (after all, it is
something that you would run on a hardware prototype in a lab, not on
each and every production machine), although I'm thinking a hardware
compatibility test could be useful in more cases, where it might be
better if it wasn't interactive.

So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
such a system could need:

- It should be modular. People who maintain driver packages for
particular hardware may want to write additional tests that a vendor
may want to run; and if this particular package supports it, the
driver package maintainer may want to provide pointers to a particular
package so that an inexperienced user may be able to configure their
hardware after running the test themselves.
- The different tests should each be able to communicate what type of
hardware they test for, whether that particular piece of hardware has
been found, and whether it actually works. The test of whether
something works may require that network connectivity, hard disks, or
other similar things are available in or to the system, as applicable.
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
- The vendor should be able to influence the score of a test by
explicitly stating that particular hardware isn't available. If the
vendor really wants to build a laptop without wireless drivers in this
day and age, then it's obviously okay if no wireless drivers were
detected. If, however, the vendor is not insane, then the failure to
detect a wireless chipset should clearly influence the score.

So, that's probably it at this point. I should have a look at bdale's
talk once it becomes available at meetings-archive.debian.net, but that
doesn't yet seem to be the case. If someone who was at bdale's talk has
anything to add, that'd be welcome. If someone could think of anything
even without being at bdale's talk, that's welcome too, of course.

Now, with the above in mind, and after having considered Holger's
proposal to do this with Debian Live[2], I think the following generic
spec should cut it, but I'm open to other suggestions at this point.
It's also not very detailed yet, but since no code has been written yet,
that doesn't really matter at this point.

- A base package 'debian-hct' will provide a basic infrastructure for
these tests to run in and an initscript that actually runs them. It
will also contain some tests that are useful for /any/ system, such as
"do we find something that looks like a harddisk controller" etc.
- Additional packages may provide tests. Packages that do so should say
'Provides: hardware-compatibility-test' in their control file.
- Tests are found in /etc/hw-compat-tests. This directory will have
subdirectories, one for each of 'hard disk controller', 'wired network
interfaces', 'wireless network interfaces', etc. The scripts in this
directory will run in asciibetical order, so that, e.g., drivers that
need firmware to be loaded can ensure this firmware is actually loaded
before allowing the generic test for this class of hardware to be ran.
- Scripts in the subdirectories, when ran, should end their output with
on a single line the letter 'S', followed by a colon, followed by two
numbers separated by a '/'; the first is the score, the second the
maximum possible score for this script. All output that precedes the
score is redirected to a file for the user to inspect later. Scripts
should make sure to avoid having a line that starts with 'S:' in this
preceding output, perhaps by escaping it with a leading space or some
such (this leading space will not be stripped).
- If no hardware is found that would be supported by the driver this
script checks for, then the second number, the maximum possible
score for this script is, by definition, 0.
- If the hardware being tested for can only be supported by the use of
non-free drivers or firmware, then the script must mention this fact
through the output of the letter 'n' after the actual score/maximum
possible score; e.g., it would say something like 'S:1/1n'. I still
need to flesh out how to best handle this particular bit of
information (it doesn't have to be all the same way; e.g., the
requirement of non-free drivers for the wireless interface is less
problematic when one wants to install than is the requirement of
non-free drivers for a hard disk). Suggestions are welcome.
- Scripts may use debconf to ask the user for input if they need it.
- Scripts should not just test for availability of hardware. Instead,
they should test the actual functionality; e.g., tests for a network
interface should be done by trying to DHCP off that interface, X.org
drivers should try to start X and ask for input using a graphical
dialog, and tests for a hard disk should be done by trying to read
some data from the disk.
- If multiple scripts in a particular subsystem directory offer
maximum possible scores, their values will be added to eachother. If
a script offers 0 as maximum possible score, it will be ignored,
even if it also outputs 'n' to signal non-free usage.
- A script must not assume input to be available from the user, nor
can it expect a controlling terminal. It can only assume that stdout
works and that debconf is available.
- If none of the scripts in a particular directory indicates it sees
some hardware (i.e., all scripts offer 0 as maximum possible score),
the system will ask the user whether this is expected. If this is
expected, the subsystem will be ignored for the final score; if not,
the final score for this subsystem will be 0/1.
- After all the scores have been collected, the framework will multiply
or divide the scores for each of the different subsystem so they fit
within a predefined scheme that ends up giving the system a score on a
scale of 1 to 100, which is then presented to the user along with
details about the tests that were run and the. It is not
necessary that every piece of hardware is given the same amount of
consideration in this final score; e.g., we will want to consider the
proper functioning of a hard disk to be of much more value than the
proper functioning of a fingerprint reader.
- A Debian Live image will be provided that will install the
'debian-hct' package plus all packages that say 'Provides:
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.

Thoughts?

(Back to bed now. Jet lag, gotta love it)

[1] http://grep.be/blog/en/computer/debian/hardware_test and
.../hardware_test_followup
[2] http://debian-live.alioth.debian.org/
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Frank Lichtenheld
2008-08-21 17:54:17 UTC
Permalink
Post by Wouter Verhelst
So, that's probably it at this point. I should have a look at bdale's
talk once it becomes available at meetings-archive.debian.net, but that
doesn't yet seem to be the case. If someone who was at bdale's talk has
anything to add, that'd be welcome. If someone could think of anything
even without being at bdale's talk, that's welcome too, of course.
http://meetings-archive.debian.net/pub/debian-meetings/2008/debconf8/high/508_hp.comgodebian.ogg

Gruesse,
--
Frank Lichtenheld <***@debian.org>
www: http://www.djpig.de/
Petter Reinholdtsen
2008-08-21 18:27:51 UTC
Permalink
[Wouter Verhelst]
Post by Wouter Verhelst
As I mentioned in my blog[1], I kindof like the suggestion that
Bdale came up with during Debconf that we write a hardware
compatibility test of sorts that hardware vendors could run on their
own hardware to test whether Debian works on their system.
I saw Bdales talk and I like it too. I did not spend time thinking
about your proposal, but wanted to let you know how we test
compatibility in a Debian Edu installation.

It is done in a installation where PXE installing work, and the first
step in the compatibility test is to install Debian Edu automatically
with all questions preseeded. If installation work, the hard drive
work.

If the KDE login manager show up (kdm), the graphics card work.

Next step is to log in, and if a small sound is played, the sound card
is working.

Start a web browser, and visit <URL:http://www.skolelinux.org/>. If
it work, the network card is functioning.

Next is 3D graphics. Select"Science & Math->Stellarium" from the K
meny, and see if the program is snappy. If it is, accelerated OpenGL
is working and the 3D stuff should work fine.

Insert a USB memory stick, and see if a dialog box pop up after a few
seconds to ask what should be done. If it does, USB is working.

Last, run nvram-wakeup and see if the motherboard and BIOS version is
supported. If it is, shutdown-at-night will work independently of
Wake On LAN support.

I hope a Debian compatibility test will check similar things.

Happy hacking,
--
Petter Reinholdtsen
Henrique de Moraes Holschuh
2008-08-22 00:43:51 UTC
Permalink
Post by Petter Reinholdtsen
It is done in a installation where PXE installing work, and the first
step in the compatibility test is to install Debian Edu automatically
with all questions preseeded. If installation work, the hard drive
work.
[...]

Your idea is good, and a simple way to partially test things, yes.

I'd add a pre-run of the Linux firmware kit, and also grep the kernel log
for any ACPI strings. IMHO, one should reject outright any machines that
outputs any warnings from the firmware kit, unless the vendor promises to
fix it. Even warnings. These things come back to bite you later.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
Chris Lamb
2008-08-22 10:05:49 UTC
Permalink
Post by Wouter Verhelst
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware
I think this is a great idea, and I generally agree with your decisions and
assumptions.
Post by Wouter Verhelst
- Scripts should not just test for availability of hardware. Instead,
they should test the actual functionality; e.g., tests for a network
interface should be done by trying to DHCP off that interface, X.org
drivers should try to start X and ask for input using a graphical
dialog, and tests for a hard disk should be done by trying to read
some data from the disk.
I think this the most important paragraph of all.

What I think is missing is some really concrete info on just how various
hardware items would be tested. For example, in the case of ethernet
adaptors, I feel that simply successfully DHCP-ing on an interface is really
not an acceptable test.

As an example, Etch's kernel contains various modules (such as sky2) that
"kinda" work on today's hardware - whilst the driver would probably pass a
DHCP test, the actual performance or reliability of the device is completely
inadequate. Wireless devices would pose an even more difficult problem, as
support for the various encryption modes that the device supports tend to be
developed at different times, etc.

(I'm fairly confident these comments have parallels with other hardware
categories, I just give networking examples that most readers might be able
to relate to)

Whilst I am aware that a testsuite could never be 100% conclusive (and I'm
sure you are aware too) my underlying enquiry here is trying to work out
what level of confidence you are aiming for.

(Hm, this didn't mean to be so negative..)
Post by Wouter Verhelst
- A Debian Live image will be provided that will install the
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
Please let me know if you would like any help with Live image foo.


Regards,
--
Chris Lamb, UK ***@chris-lamb.co.uk
GPG: 0x634F9A20
Wouter Verhelst
2008-08-23 23:52:44 UTC
Permalink
Post by Chris Lamb
Post by Wouter Verhelst
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware
I think this is a great idea, and I generally agree with your decisions and
assumptions.
Post by Wouter Verhelst
- Scripts should not just test for availability of hardware. Instead,
they should test the actual functionality; e.g., tests for a network
interface should be done by trying to DHCP off that interface, X.org
drivers should try to start X and ask for input using a graphical
dialog, and tests for a hard disk should be done by trying to read
some data from the disk.
I think this the most important paragraph of all.
What I think is missing is some really concrete info on just how various
hardware items would be tested.
Well, that's the hard bit, isn't it? ;-)

I didn't go into much detail here yet, simply because I feel this is
something we'll have to figure out as we write everything.
Post by Chris Lamb
For example, in the case of ethernet adaptors, I feel that simply
successfully DHCP-ing on an interface is really not an acceptable
test.
As an example, Etch's kernel contains various modules (such as sky2) that
"kinda" work on today's hardware - whilst the driver would probably pass a
DHCP test, the actual performance or reliability of the device is completely
inadequate. Wireless devices would pose an even more difficult problem, as
support for the various encryption modes that the device supports tend to be
developed at different times, etc.
(I'm fairly confident these comments have parallels with other hardware
categories, I just give networking examples that most readers might be able
to relate to)
Whilst I am aware that a testsuite could never be 100% conclusive (and I'm
sure you are aware too) my underlying enquiry here is trying to work out
what level of confidence you are aiming for.
Yes, good point.

The main problem is, at this point I'm not actually all that sure of it,
really. On the one hand, I want to test as much as is possible. On the
other hand, that is going to get very hard very quickly, and would
probably take forever to develop.

We'll probably need to find a bit of a compromise here: initially, test
for at least basic functionality, and test additional functionality if
we find out at some point that common modern hardware has problems where
basic functionality isn't available.
Post by Chris Lamb
(Hm, this didn't mean to be so negative..)
Negative? What's negative? ;)
Post by Chris Lamb
Post by Wouter Verhelst
- A Debian Live image will be provided that will install the
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
Please let me know if you would like any help with Live image foo.
That's the final step. If/when I get there, I will. If I still need it
then...
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Ben Hutchings
2008-08-30 12:20:54 UTC
Permalink
Post by Chris Lamb
Post by Wouter Verhelst
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware
I think this is a great idea, and I generally agree with your decisions and
assumptions.
Post by Wouter Verhelst
- Scripts should not just test for availability of hardware. Instead,
they should test the actual functionality; e.g., tests for a network
interface should be done by trying to DHCP off that interface, X.org
drivers should try to start X and ask for input using a graphical
dialog, and tests for a hard disk should be done by trying to read
some data from the disk.
I think this the most important paragraph of all.
What I think is missing is some really concrete info on just how various
hardware items would be tested. For example, in the case of ethernet
adaptors, I feel that simply successfully DHCP-ing on an interface is really
not an acceptable test.
<snip>

At a bare minimum the HCT should try to run the driver's offline
self-test (ethtool -t $dev offline). For a more comprehensive (and
generic) test we would really want to use two machines, though an
external loopback cable may suffice.

My employer outsources much of the generic network test development to
Oktet Labs. Their network test environment
<http://www.oktetlabs.ru/test_env.rhtml> is distributed under GPL though
it is not freely downloadable. It might be worth asking them to provide
it to Debian.

Ben.
--
Ben Hutchings
It is easier to write an incorrect program than to understand a correct one.
Giacomo Catenazzi
2008-08-22 10:55:34 UTC
Permalink
Post by Wouter Verhelst
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
Yes, I find the talk very interesting.
Post by Wouter Verhelst
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
/me too
Post by Wouter Verhelst
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
No. I don't think we should provide profiles.
The check should be:
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).

Additionally a component test should be furnished, for component
hardware vendor. (e.g. a network card vendor would test
only the network card, not the whole system).
Post by Wouter Verhelst
- The vendor should be able to influence the score of a test by
explicitly stating that particular hardware isn't available. If the
vendor really wants to build a laptop without wireless drivers in this
day and age, then it's obviously okay if no wireless drivers were
detected. If, however, the vendor is not insane, then the failure to
detect a wireless chipset should clearly influence the score.
See above.
The score should be done only "negatively", i.e. when a
hardware is found but:
- doesn't work properly
- it is buggy (as for the BIOS: linux have some work-around for
broken BIOS, but the test should alert vendor)
- it requires manual configuration
(partly could be a problem of Debian, but it could be also
a problem in hardware: wrong identification strings/number)
- requires non-free stuff
- ...
Post by Wouter Verhelst
Now, with the above in mind, and after having considered Holger's
proposal to do this with Debian Live[2], I think the following generic
spec should cut it, but I'm open to other suggestions at this point.
It's also not very detailed yet, but since no code has been written yet,
that doesn't really matter at this point.
- A base package 'debian-hct' will provide a basic infrastructure for
these tests to run in and an initscript that actually runs them. It
will also contain some tests that are useful for /any/ system, such as
"do we find something that looks like a harddisk controller" etc.
we really want "debian" in package name?
Should we do a more broad project?
Post by Wouter Verhelst
- Additional packages may provide tests. Packages that do so should say
'Provides: hardware-compatibility-test' in their control file.
and Depends it should on 'debian-hct', to have a common interface.
Post by Wouter Verhelst
- Tests are found in /etc/hw-compat-tests. This directory will have
subdirectories, one for each of 'hard disk controller', 'wired network
interfaces', 'wireless network interfaces', etc. The scripts in this
directory will run in asciibetical order, so that, e.g., drivers that
need firmware to be loaded can ensure this firmware is actually loaded
before allowing the generic test for this class of hardware to be ran.
I don't like to have it in /etc. IMHO it is better to have it in
/var/lib/ or ... and copied to a root directory on the target system.
Post by Wouter Verhelst
- A Debian Live image will be provided that will install the
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
the BIOS testing and memory testing should be included on the
basic test.

I'm not so sure that the test should be packages.
The system is too different on early boot from a Debian system.
I think it is a lot simpler to do a separate project, using
building infrastructure as d-i or debian-live.

IMHO the target CD should be very easy to create and it
should be complete. (but maybe allowing additional external
kernel and modules).

The "modular" thing should be put mainly on the testing
development side.
Post by Wouter Verhelst
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.
This should be discussed after we have comprehensive tests, and
probably to LI. I don't want to see packages with logo of 10
distributions. Marketing people could choose to include on packages
only few distribution, which could be negative for other
distribution (and maybe for debian).

ciao
cate
Post by Wouter Verhelst
Thoughts?
(Back to bed now. Jet lag, gotta love it)
[1] http://grep.be/blog/en/computer/debian/hardware_test and
.../hardware_test_followup
[2] http://debian-live.alioth.debian.org/
Joe Smith
2008-08-22 21:14:30 UTC
Permalink
Post by Giacomo Catenazzi
Post by Wouter Verhelst
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
Yes, I find the talk very interesting.
Post by Wouter Verhelst
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
/me too
Post by Wouter Verhelst
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
No. I don't think we should provide profiles.
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).
Ok. What does the test do if it notices that a pecie of hardware exists, but
cannot identify it?
Should it warn about that?

The long list of hardware not found may be a problem. Let's picture a modern
desktop PC.
"No floppy drive found."
"No touchpad found."
"No webcam found."
"No firewire found."
"No modem found." (This is common enough these days, but we would definately
want to test any found modem due to the proliferation of weird winmodems
these days).
"No physics acellerator found."
"No PC card slot found"
"No HD-DVD drive found"
"No Blu-ray drive found"
"No RAID controller found"
"No gigabit ethernet found"
"No EFI firmware found" (This theoretical PC is not a Intel Mac.)
"No fingerprint scanner found".


(The list could go on and on).

Perhaps a few of those could be merged into other tests, like a single
optical drive test, but in some of those cases it would then be important
that a list of hardware that was found is printed. That way the manufacturer
can be sure that both optical drives (in a two optical drive PC) were found.

If there is any hardware that cannot be tested with only a single test
script due to major implementation differences between all brands of
hardware (and no current unified software interface), we would want to avoid
having a notice printed for each possible manufacture of the device.
Giacomo A. Catenazzi
2008-08-25 16:51:47 UTC
Permalink
Post by Joe Smith
Post by Giacomo Catenazzi
Post by Wouter Verhelst
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
Yes, I find the talk very interesting.
Post by Wouter Verhelst
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
/me too
Post by Wouter Verhelst
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
No. I don't think we should provide profiles.
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).
Ok. What does the test do if it notices that a pecie of hardware exists,
but cannot identify it?
Should it warn about that?
I think that 100% (but maybe very few exceptions) of the new hardware
can be detected and identify the class of hardware.

Possible problem could arise with some USB gadget, but class list
is frequently updated, and in this case we must warn the user
about unsupported hardware. (the same with firewire, PCI, and some other
buses).
Post by Joe Smith
Perhaps a few of those could be merged into other tests, like a single
optical drive test, but in some of those cases it would then be
important that a list of hardware that was found is printed. That way
the manufacturer can be sure that both optical drives (in a two optical
drive PC) were found.
I agree, but probably we should ask Bdale about how the manufacturers
will do the test and what we can expect about it.
Post by Joe Smith
If there is any hardware that cannot be tested with only a single test
script due to major implementation differences between all brands of
hardware (and no current unified software interface), we would want to
avoid having a notice printed for each possible manufacture of the
device.
Hmm. If hardware cannot be fully detected (e.g. in this case: if we
cannot detect what kind of the protocol is used), it is an error, which
should be notified, so to correct the test, the driver or the hardware.
[i.e. the burden of hardware identification should not be left to
our users]. The handle of such cases IMHO should not have a
high priority.

Probably not always it is possible to do such identification
(IIRC recently on kernel and Linus had such problem with
Intel e1000 "variant"), but I think it is rare.

ciao
cate
Wouter Verhelst
2008-08-22 18:10:36 UTC
Permalink
Post by Giacomo Catenazzi
Post by Wouter Verhelst
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
Yes, I find the talk very interesting.
Post by Wouter Verhelst
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
/me too
Post by Wouter Verhelst
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
No. I don't think we should provide profiles.
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
That will create ambiguity, which I think we should avoid at all cost.
If you tell a user of this test that "it seems to work, but you should
check this and this and this output to make sure we didn't miss
anything", then such a test is worthless; hardware vendors *will* miss
this kind of thing.
Post by Giacomo Catenazzi
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).
I'm afraid I'm not familiar with this piece of software; and as I'm
offline right now, I can't look it up, either.

However, I don't think checking by listing the hardware is the best way
to go, in any regard. A script that checks for all hardware available in
a system can't know about each and every piece of hardware out there;
especially not if new interfaces are made that the script doesn't yet
know about. Giving a piece of hardware a perfect score because our
script failed to notice there's some unsupported wireless interface
connected is not the way to go.
Post by Giacomo Catenazzi
Additionally a component test should be furnished, for component
hardware vendor. (e.g. a network card vendor would test
only the network card, not the whole system).
I'm not sure that's very useful. A network card vendor has two options:
either they design a card based on an existing chipset that has drivers
already in the kernel (in which case it will probably just work, unless
they use previously-unknown PCI IDs, in which case it won't), or they
will create a totally new chipset for their network card, in which case
it won't work. A vendor is usually aware of this fact, and giving them a
"test" for something they already know about isn't very useful.

If you disagree, of course don't let me stand in your way. Just don't
expect me to work on it ;-)
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- The vendor should be able to influence the score of a test by
explicitly stating that particular hardware isn't available. If the
vendor really wants to build a laptop without wireless drivers in this
day and age, then it's obviously okay if no wireless drivers were
detected. If, however, the vendor is not insane, then the failure to
detect a wireless chipset should clearly influence the score.
See above.
The score should be done only "negatively", i.e. when a
- doesn't work properly
- it is buggy (as for the BIOS: linux have some work-around for
broken BIOS, but the test should alert vendor)
- it requires manual configuration
(partly could be a problem of Debian, but it could be also
a problem in hardware: wrong identification strings/number)
- requires non-free stuff
- ...
Yes, I agree; that's the only sane way to do it. If we don't find any
issues at all, a system should get a 100% score; only when we do find
some issue should the score be lower.
Post by Giacomo Catenazzi
Post by Wouter Verhelst
Now, with the above in mind, and after having considered Holger's
proposal to do this with Debian Live[2], I think the following generic
spec should cut it, but I'm open to other suggestions at this point.
It's also not very detailed yet, but since no code has been written yet,
that doesn't really matter at this point.
- A base package 'debian-hct' will provide a basic infrastructure for
these tests to run in and an initscript that actually runs them. It
will also contain some tests that are useful for /any/ system, such as
"do we find something that looks like a harddisk controller" etc.
we really want "debian" in package name?
Should we do a more broad project?
I'm not opposed to making this useful to more distributions than just
Debian; however, I also think that we should provide an official
"Debian" test that will test whether something will work with Debian
Stable, rather than just a random Linux kernel plus some random software
that's available on some random website. The former is something
tangible; the latter isn't.
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- Additional packages may provide tests. Packages that do so should say
'Provides: hardware-compatibility-test' in their control file.
and Depends it should on 'debian-hct', to have a common interface.
Yes, probably.
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- Tests are found in /etc/hw-compat-tests. This directory will have
subdirectories, one for each of 'hard disk controller', 'wired network
interfaces', 'wireless network interfaces', etc. The scripts in this
directory will run in asciibetical order, so that, e.g., drivers that
need firmware to be loaded can ensure this firmware is actually loaded
before allowing the generic test for this class of hardware to be ran.
I don't like to have it in /etc. IMHO it is better to have it in
/var/lib/ or ... and copied to a root directory on the target system.
Yeah, totally. This is an artifact of me having written this at 4AM-ish.
I wrote /etc because I was thinking this should be something "like"
initscripts; but it's most likely better placed in /usr/share or some
such.
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- A Debian Live image will be provided that will install the
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
the BIOS testing and memory testing should be included on the
basic test.
Yes, probably.
Post by Giacomo Catenazzi
I'm not so sure that the test should be packages.
The system is too different on early boot from a Debian system.
In what way? Note that a Debian Live system *is* a normal Debian system;
you build a Debian Live image by giving the debian-live script a bunch
of packages; it then installs them into a chroot, changes a few things
so that the system will actually boot from CD-ROM, and then create an
ISO image from it.

How is that "too different" from a Debian system? Or are you talking
about something else?
Post by Giacomo Catenazzi
I think it is a lot simpler to do a separate project, using
building infrastructure as d-i or debian-live.
IMHO the target CD should be very easy to create and it
should be complete. (but maybe allowing additional external
kernel and modules).
That's my goal, too.
Post by Giacomo Catenazzi
The "modular" thing should be put mainly on the testing
development side.
Yeah, I'm not suggesting we provide vendors with a bunch of packages;
the packages thing is a way for us to make it easier to actually develop
this without having one maintainer who has to write/accept/maintain
*all* the tests in the system. But hey, this is debian-devel, not
debian-announce ;-)
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.
This should be discussed after we have comprehensive tests,
Sure.
Post by Giacomo Catenazzi
and probably to LI. I don't want to see packages with logo of 10
distributions.
To me, it's not about seeing a logo on a package. It's about seeing
another vendor besides HP make a commitment that "Yes, Debian will be
supported. We won't run away scared to the hills if you say that you're
running Debian." Because that's what most vendors will do currently.

I think you'll also find aving a generic "Linux" hardware compatibility
test isn't very useful; if we provide hardware vendors with such a
thing, they'll go "I ran that test, and it said 'pass'; but my
shiny-new-network-card-chipset-that's-supported-in-the-latest-development-kernel-only
isn't supported in Debian? That's probably a bug in Debian, then".

And it doesn't even have to be "too new" stuff; it may also be that some
particular bit of hardware is supported by some free out-of-tree driver
which is supported by some distributions, but not necessarily by all of
them. Would you test for that in a generic "Linux" compatibility test?
If you don't, you'll fail on an amount of hardware that works perfectly
well in most distributions; and if you do, you'll succeed on hardware
that won't be supported by a number of distributions, possibly the one
the hardware vendor cares about most (because, say, they use it
themselves for their webservers).

No, I think we *do* need a Debian-specific test.
Post by Giacomo Catenazzi
Marketing people could choose to include on packages only few
distribution, which could be negative for other distribution (and
maybe for debian).
I think you'll find that they do that anyway. The more logos, the
merrier, because it sells good. Never mind that you can make those up, of
course.
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Giacomo A. Catenazzi
2008-08-25 17:11:32 UTC
Permalink
Post by Wouter Verhelst
Post by Giacomo Catenazzi
Post by Wouter Verhelst
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
Yes, I find the talk very interesting.
Post by Wouter Verhelst
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
/me too
Post by Wouter Verhelst
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
No. I don't think we should provide profiles.
"complete": test all hardware that it is installed on system.
If the system doesn't have the video-card or the wifi just it doesn't
test it, but it write a notice.
That will create ambiguity, which I think we should avoid at all cost.
If you tell a user of this test that "it seems to work, but you should
check this and this and this output to make sure we didn't miss
anything", then such a test is worthless; hardware vendors *will* miss
this kind of thing.
I really think there are no such cases. Eventually we found
unknown hardware, but I really think that on modern hardware
we will not be undetected hardware (but on very new bus,
in which case the hardware vendor care much more).
Post by Wouter Verhelst
Post by Giacomo Catenazzi
Listening the available hardware is pretty trivial (see i.e. my
AutoKernConf).
I'm afraid I'm not familiar with this piece of software; and as I'm
offline right now, I can't look it up, either.
However, I don't think checking by listing the hardware is the best way
to go, in any regard. A script that checks for all hardware available in
a system can't know about each and every piece of hardware out there;
especially not if new interfaces are made that the script doesn't yet
know about. Giving a piece of hardware a perfect score because our
script failed to notice there's some unsupported wireless interface
connected is not the way to go.
I meant the kdetect script, which list the hardware attached to
various buses.
You are right, the other program try to give a name and a kernel
driver to hardware, which is impossible to do completely.

But you can list the hardware of pretty all buses, which
usually some information about the type.
My script collect work of kernel and other programs.
Additionally "dmidecode" collects a lot more information
about memory, cache, motherboard, etc, and
there is already some program to find sensors
(my script find only the sensor found by the kernel,
the other program is kernel-indipendent).
Now there is also a fan-control script that
check the fan (but maybe using dmidecode or i2c)

I really think we can detect nearly all hardware.
The hardware that could be missed, it is the hardware that
probably the vendor will forget (sensor, fan and other
small and probably fast changing piece of hardware).
Post by Wouter Verhelst
Post by Giacomo Catenazzi
Additionally a component test should be furnished, for component
hardware vendor. (e.g. a network card vendor would test
only the network card, not the whole system).
either they design a card based on an existing chipset that has drivers
already in the kernel (in which case it will probably just work, unless
they use previously-unknown PCI IDs, in which case it won't), or they
will create a totally new chipset for their network card, in which case
it won't work. A vendor is usually aware of this fact, and giving them a
"test" for something they already know about isn't very useful.
If you disagree, of course don't let me stand in your way. Just don't
expect me to work on it ;-)
Maybe we should discuss with Bdale. I'm not sure about your reasoning,
for two reasons:
- hardware vendor will do a new version, but it still use old driver
(USB and PCI have an additional version value, along the ID)
- a lot of drivers support multiple hardware: if you read the kernel,
you will see driver with a long list of ID.
Some drivers (IDE,...) check only the class of hardware

so I think there could be error on hardware that we could detect.

But my point was about one of my mouse: it work good on windoze,
but not on my systems ("automatic backspace after click").
For this reason I would like to have component tests.

Anyway I think it is more an interface problem, not a really
problem, and it is not the primary goal.


-- to late --
I'll read and answer the bottom part of the email tomorrow ;-)

ciao
cate
Post by Wouter Verhelst
Post by Giacomo Catenazzi
I'm not so sure that the test should be packages.
The system is too different on early boot from a Debian system.
In what way? Note that a Debian Live system *is* a normal Debian system;
you build a Debian Live image by giving the debian-live script a bunch
of packages; it then installs them into a chroot, changes a few things
so that the system will actually boot from CD-ROM, and then create an
ISO image from it.
How is that "too different" from a Debian system? Or are you talking
about something else?
Post by Giacomo Catenazzi
I think it is a lot simpler to do a separate project, using
building infrastructure as d-i or debian-live.
IMHO the target CD should be very easy to create and it
should be complete. (but maybe allowing additional external
kernel and modules).
That's my goal, too.
Post by Giacomo Catenazzi
The "modular" thing should be put mainly on the testing
development side.
Yeah, I'm not suggesting we provide vendors with a bunch of packages;
the packages thing is a way for us to make it easier to actually develop
this without having one maintainer who has to write/accept/maintain
*all* the tests in the system. But hey, this is debian-devel, not
debian-announce ;-)
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.
This should be discussed after we have comprehensive tests,
Sure.
Post by Giacomo Catenazzi
and probably to LI. I don't want to see packages with logo of 10
distributions.
To me, it's not about seeing a logo on a package. It's about seeing
another vendor besides HP make a commitment that "Yes, Debian will be
supported. We won't run away scared to the hills if you say that you're
running Debian." Because that's what most vendors will do currently.
I think you'll also find aving a generic "Linux" hardware compatibility
test isn't very useful; if we provide hardware vendors with such a
thing, they'll go "I ran that test, and it said 'pass'; but my
shiny-new-network-card-chipset-that's-supported-in-the-latest-development-kernel-only
isn't supported in Debian? That's probably a bug in Debian, then".
And it doesn't even have to be "too new" stuff; it may also be that some
particular bit of hardware is supported by some free out-of-tree driver
which is supported by some distributions, but not necessarily by all of
them. Would you test for that in a generic "Linux" compatibility test?
If you don't, you'll fail on an amount of hardware that works perfectly
well in most distributions; and if you do, you'll succeed on hardware
that won't be supported by a number of distributions, possibly the one
the hardware vendor cares about most (because, say, they use it
themselves for their webservers).
No, I think we *do* need a Debian-specific test.
Post by Giacomo Catenazzi
Marketing people could choose to include on packages only few
distribution, which could be negative for other distribution (and
maybe for debian).
I think you'll find that they do that anyway. The more logos, the
merrier, because it sells good. Never mind that you can make those up, of
course.
Giacomo Catenazzi
2008-08-26 07:02:03 UTC
Permalink
Post by Wouter Verhelst
Post by Giacomo Catenazzi
I'm not so sure that the test should be packages.
The system is too different on early boot from a Debian system.
In what way? Note that a Debian Live system *is* a normal Debian system;
you build a Debian Live image by giving the debian-live script a bunch
of packages; it then installs them into a chroot, changes a few things
so that the system will actually boot from CD-ROM, and then create an
ISO image from it.
How is that "too different" from a Debian system? Or are you talking
about something else?
ok. I was thinking about doing hardware test a lot earlier
(i.e. in initramfs).
But probably it is not really needed, and we probably could assume
that system is enough reasonable.
Anyway, we can try with debian-live, and ev. move things
earlier.
Post by Wouter Verhelst
Post by Giacomo Catenazzi
The "modular" thing should be put mainly on the testing
development side.
Yeah, I'm not suggesting we provide vendors with a bunch of packages;
the packages thing is a way for us to make it easier to actually develop
this without having one maintainer who has to write/accept/maintain
*all* the tests in the system. But hey, this is debian-devel, not
debian-announce ;-)
Post by Giacomo Catenazzi
Post by Wouter Verhelst
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.
This should be discussed after we have comprehensive tests,
Sure.
Post by Giacomo Catenazzi
and probably to LI. I don't want to see packages with logo of 10
distributions.
To me, it's not about seeing a logo on a package. It's about seeing
another vendor besides HP make a commitment that "Yes, Debian will be
supported. We won't run away scared to the hills if you say that you're
running Debian." Because that's what most vendors will do currently.
I think you'll also find aving a generic "Linux" hardware compatibility
test isn't very useful; if we provide hardware vendors with such a
thing, they'll go "I ran that test, and it said 'pass'; but my
shiny-new-network-card-chipset-that's-supported-in-the-latest-development-kernel-only
isn't supported in Debian? That's probably a bug in Debian, then".
And it doesn't even have to be "too new" stuff; it may also be that some
particular bit of hardware is supported by some free out-of-tree driver
which is supported by some distributions, but not necessarily by all of
them. Would you test for that in a generic "Linux" compatibility test?
If you don't, you'll fail on an amount of hardware that works perfectly
well in most distributions; and if you do, you'll succeed on hardware
that won't be supported by a number of distributions, possibly the one
the hardware vendor cares about most (because, say, they use it
themselves for their webservers).
No, I think we *do* need a Debian-specific test.
The LinuxFirmwareKit has few kernels. IIRC redhat, suse and
latest vanilla, so vendors could check the support of various
distributions.

I think we could be done in the same way: we allow installing/selecting
other kernels. The tests are the same, only the kernel could change,
and "obviously, the default kernel is the Debian stable kernel.

In this manner, vendor could test new-hardware with requires new-kernel
but on other side, vendors have information about Debian kernel, so they
could ev. try to backport drivers for us.

I think doing it Debian specific is more difficult, because it
requires a lot more work.
we should share some burden to upstreams (I'm thinking mainly about
xorg: test about 2D, 3D accelerations, etc.; i2c, etc.).

ciao
cate
Mikhail Gusarov
2008-08-23 15:22:34 UTC
Permalink
Twas brillig at 06:24:07 20.08.2008 UTC+02 when ***@debian.org did gyre and gimble:

[]

JFYI, there's great piece of software which can be reused: http://www.inquisitor.ru/about/

--
Continue reading on narkive:
Loading...