The linux-kernel mailing list FAQ

These frequently asked questions are divided in various categories. Please contribute any category and Q/A that you may find relevant. You can also add your answer to any question that has already been answered, if you have additional information to contribute.

The official site is: http://www.tux.org/lkml/ (this is in the east coast of the U.S.A). Many thanks to Sam Chessman and David Niemi for hosting the FAQ on a high-bandwidth, professionally managed Linux server. The following mirrors are available (and are updated at the same time as the official site):

• http://www.atnf.csiro.au/~rgooch/linux/docs/lkml/ in Sydney, Australia
• http://www.ras.ucalgary.ca/~rgooch/linux/docs/lkml/ in Calgary, Canada
• http://www.kernel.org/pub/linux/docs/lkml/ in the west coast of the U.S.A.

Hot off the Presses

vger.kernel.org has enabled ECN. You may need to switch ISP in order to receive linux-kernel email. See the section on ECN for more details.

Go to http://www.atnf.csiro.au/~rgooch/linux/docs/kernel-newsflash.html for newflashes about official kernel releases.
NOTE: this page is no longer maintained. If there is an alternative page, please let me know.

Index

• Basic Linux kernel documentation
• Contributors and some special expressions
• Related mailing lists
• Question Index
  1. General questions
  2. Driver specific questions
  3. Mailing list questions
  4. "How do I" questions
  5. "Who's who" questions
  6. CPU questions
  7. OS questions
  8. Compiler/binutils questions
  9. Feature specific questions
  10. "What's changed between kernels 2.0.x and 2.2.x" questions
  11. Primer documents
  12. Kernel Programming Questions
  13. Mysterious kernel messages
  14. Odd kernel behaviour
  15. Programming Religion
  16. User-space Programming Questions
• Answers
• Contributing

Basic Linux kernel documentation

• The Linux Kernel Hackers' Guide, compiled by Michael K. Johnson of Red Hat fame. Includes among other documents selected Q/As from the linux-kernel mailing list.
• The Linux Kernel book, by David A. Rusling, available in various formats from the Linux Documentation Project and mirrors. Still being worked on, but explains clearly the main structure of the Linux kernel.
• The Linux FAQ by Robert Kiesling has many high quality Q/As.
• The Linux Kernel HOWTO by Brian Ward. Fundamental reading for anybody wanting to post to the linux-kernel mailing list.
• A completely new Kernelhacking-HOWTO at http://www.kernelhacking.org/. Currently work in progress, but already contains some useful information.
• Various Linux HOWTOs on specific questions, such as the BogoMips mini-HOWTO by Wim van Dorst. These are all by definition LDP documents.
• The Linux kernel source code for any particular kernel version that you may be using. Note that there is a /Documentation directory which holds some very useful text files about drivers, etc. Also check the MAINTAINERS file in the kernel source root directory.
• Some drivers even have Web pages, with additional up to date information e.g. the network drivers by Donald Becker, etc. Check the Hardware section in the LDP site.
• Similarly, Linux implementations for some CPU architectures have dedicated Web pages, mailing lists, and sometimes even a HOWTO e.g. the Linux Alpha HOWTO by Neal Crook. Check the LDP site and its mirrors for Web links to the various architecture specific sites.
• Linux device drivers, a book written by Alessandro Rubini. C. Scott Ananian reviewed it for Amazon.com.
• Linux kernel internals, a book by Michael Beck (Editor) et al. Also reviewed for Amazon.com.
• Another useful site is: http://www.kernelnewbies.org/
• Here is a general guide on how to ask questions in a way that greatly improves your chances of getting a reply: http://www.catb.org/~esr/faqs/smart-questions.html. If you have a bug to report, you should also read http://www.chiark.greenend.org.uk/~sgtatham/bugs.html.
  Extra instructions, specific to the Linux kernel are available here.

Contributors and some special expressions

• AC : Alan Cox
• AV : Alexander Viro
• ADB: Andrew D. Balsa
• CP : Colin Plumb
• DBE: Daniel Bergstrom
• DSM: David S. Miller (co-postmaster)
• DW : David Woodhouse
• JBG: Jan-Benedict Glaw
• KGB: Krzysztof G. Baranowski
• KO : Keith Owens
• MEA: Matti E. Aarnio (co-postmaster)
• MRW: Matthew Wilcox
• PG : Paul Gortmaker
• RC : Ralph Corderoy
• REG: Richard E. Gooch (FAQ maintainer)
• REW: Roger E. Wolff
• RML: Robert M. Love
• RRR: Rafael R. Reilova
• TAC: Thomas A. Cort
• TJ : Trevor Johnson
• TYT: Theodore Y. Ts'o
• VKh: Vassilii Khachaturov

• AFAIK = As Far As I Know
• AKA = Also Known As
• ASAP = As Soon As Possible
• BTW = By The Way (used to introduce some piece of information or question that is on a different topic but may be of interest)
• COLA = comp.os.linux.announce (newsgroup)
• ETA = Estimated Time of Arrival
• FAQ = Frequently Asked Question
• FUD = Fear, Uncertainty and Doubt
• FWIW = For What It's Worth
• FYI = For Your Information
• IANAL = I Am Not A Lawyer
• IIRC = If I Recall Correctly
• IMHO = In My Humble Opinion
• IMNSHO = In My Not-So-Humble Opinion
• IOW = In Other Words
• LART = Luser Attitude Readjustment Tool (quoting Al Viro: "Anything you use to forcibly implant the clue into the place where luser's head is")
• LUSER = pronounced "loser", a user who is considered to indeed be a loser (idiot, drongo, wanker, dim-wit, fool, etc.)
• OTOH = On The Other Hand
• PEBKAC = Problem Exists Between Keyboard And Chair
• ROTFL = Rolling On The Floor Laughing
• RSN = Real Soon Now
• RTFM = Read The Fucking Manual (original definition) or Read The Fine Manual (if you want to pretend to be polite)
• TANSTAAFL = There Ain't No Such Thing As A Free Lunch (contributed by David Niemi, quoting Robert Heinlein in his science fiction novel 'The Moon is a Harsh Mistress')
• THX = Thanks (thank you)
• TIA = Thanks In Advance
• WIP = Work In Progress
• WRT = With Respect To

Related mailing lists

• The linux-net@vger.kernel.org mailing list is for networking user questions. Subscribe by sending
  subscribe linux-net in the message body to majordomo@vger.kernel.org
• The netdev@vger.kernel.org mailing list is for network development (not user questions). Subscribe by sending
  subscribe netdev in the message body to majordomo@vger.kernel.org

Question Index

Section 1 - General questions

1. Why do you use "GNU/Linux" sometimes and just "Linux" in other parts of the FAQ?
2. What is an experimental kernel version?
3. What is a production kernel?
4. What is a feature freeze?
5. What is a code freeze?
6. What is a f.g.hhprei kernel?
7. Where do I get the latest kernel source?
8. Where do I get extra kernel patches?
9. What is a patch?
10. How do I make a patch suitable for the linux kernel list?
11. How do I apply a patch?
12. What's vger?
13. What is a CVS tree? Where can I find more information about CVS?
14. Is there a CVS tutorial?
15. How do I get my patch into the kernel?
16. Why does the kernel tarball contain a directory called linux/ instead of linux-x.y.z/ ?
17. What's the difference between the official kernels and Alan Cox's -ac series of patches?
18. What does it mean for a module to be tainted?
19. What is this about GPLONLY symbols?
20. Do I have to use GIT to send patches?
21. Who maintains the kernel?
22. The kernel doesn't compile cleanly. What shall I do ?

Section 2 - Driver specific questions

1. Driver such and such is broken!
2. Here is a new driver for hardware XYZ.
3. Is there support for my card TW-345 model C in kernel version f.g.hh?
4. Who maintains driver such and such?
5. I want to write a driver for card TW-345 model C, how do I get started?
6. I want to get the docs, but they want me to sign an NDA (Non-Disclosure Agreement).
7. I want/need/must have a driver for card TW-345 model C! Won't anybody write one for me?
8. What's this major/minor device number thing?
9. Why aren't WinModems supported?
10. Modern CPUs are very fast, so why can't I write a user mode interrupt handler?
11. Do I need to test my driver against all distributions?

Section 3 - Mailing list questions

1. How do I subscribe to the linux-kernel mailing list?
2. How do I unsubscribe from the linux-kernel mailing list?
3. Do I have to be subscribed to post to the list?
4. Is there an archive for the list?
5. How can I search the archive for a specific question?
6. Are there other ways to search the Web for information on a particular Linux kernel issue?
7. How heavy is the traffic on the list?
8. What kind of question can I ask on the list?
9. What posting style should I use for the list?
10. Is the list moderated?
11. Can I be ejected from the list?
12. Are there any implicit rules on this list that I should be aware of?
13. How do I post to the list?
14. Does the list get spammed?
15. I am not getting any mail anymore from the list! Is it down or what?
16. Is there an NNTP gateway somewhere for the mailing list?
17. I want to post a Great Idea (tm) to the list. What should I do?
18. There is a long thread going on about something completely offtopic, unrelated to the kernel, and even some people who are in the "Who's who" section of this FAQ are mingling in it. What should I do to fight this "noise"?
19. Can we have the Subject: line modified to help mail filters?
20. Can we have a Reply-To: header automatically added to the list traffic?
21. Can I post job offers/requests to the list?
22. Why do I get bounces when I send private email to some people?
23. Why don't you split the list, such as having one each for the development and stable series?

Section 4 - "How do I" questions

1. How do I post a patch?
2. How do I capture an Oops?
3. How do I post an Oops?
4. I think I found a bug, how do I report it?
5. What information should go into a bug report?
6. I found a bug in an "old" version of the kernel, should I report it?
7. How do I compile the kernel?
8. How do check if the running kernel is tainted?

Section 5 - "Who's who" questions

1. Who is in charge here?
2. Why don't we have a Linux Kernel Team page, same as there are for other projects?
3. Why doesn't <any of the below> answer my mails? Isn't that rude?
4. Why do I get bounces when I send private to email to some of these people?
5. Who is Matti Aarnio?
6. Who is H. Peter Anvin?
7. Who is Donald Becker?
8. Who is Alan Cox?
9. Who is Richard E. Gooch?
10. Who is Paul Gortmaker?
11. Who is Bill Hawes?
12. Who is Mark Lord?
13. Who is Larry McVoy?
14. Who is David S. Miller?
15. Who is Linus Torvalds?
16. Who is Theodore Y. T'so?
17. Who is Stephen Tweedie?
18. Who is Roger Wolff?

Other OS developers:

• Who is Prof. Douglas Comer (Xinu)?
• Who is Richard M. Stallman aka RMS (GNU)?
• Who is Prof. Andrew Tanenbaum (MINIX)?

Section 6 - CPU questions

1. What is the "best" CPU for GNU/Linux?
2. What is the fastest CPU for GNU/Linux?
3. I want to implement the Linux kernel for CPU Hyper123, how do I get started?
4. Why is my Cyrix 6x86/L/MX detected by the kernel as a Cx486?
5. What about those x86 CPU bugs I read about?
6. I grabbed the standard kernel tarball from ftp.kernel.org or some mirror of it, and it doesn't compile on the Sparc, what gives?
7. Does the Linux kernel execute the Halt instruction to power down the CPU?
8. I have a non-Intel x86 CPU. What is the [best|correct] kernel config option for my CPU?
9. What CPU types does Linux run on?

Section 7 - OS questions

1. OS $toomuch has this Nice feature, so it must be better than GNU/Linux.
2. Why doesn't the Linux kernel have a graphical boot screen like $toomuch OS?
3. The kernel in OS CTE-variant has this Nice-very-nice feature, can I port it to the Linux kernel?
4. How about adding feature Nice-also-very-nice to the Linux kernel?
5. Are there more bugs in later versions of the Linux kernel, compared to earlier versions?
6. Why does the Linux kernel source code keep getting larger and larger?
7. The kernel source is HUUUUGE and takes too long to download. Couldn't it be split in various tarballs?
8. What are the licensing/copying terms on the Linux kernel?
9. What are those references to "bazaar" and "cathedral"?
10. What is this "World Domination" thing?
11. What are the plans for future versions of the Linux kernel?
12. Why does it show BogoMips instead of MHz in the kernel boot message?
13. I installed kernel x.y.z and package foo doesn't work anymore, what should I do?
14. People talk about user space vs. kernel space. What's the advantage of each?
15. What are threads?
16. Can I use threads with GNU/Linux?
17. You mean threads are implemented in user space? Why not in kernel space? Wouldn't that be more efficient?
18. Can GNU/Linux machines be clustered?
19. How well does Linux scale for SMP?
20. Can I lock a process/thread to a CPU?
21. How efficient are threads under Linux?
22. How does the Linux networking/TCP stack work?
23. Can we put the networking/TCP stack into user-space?

Section 8 - Compiler/binutils questions

1. I downloaded the newest kernel and it doesn't even compile! What's wrong?
2. What are the recommended compiler/binutils for building kernels?
3. Why the recommended compiler? I like xyz-compiler better.
4. Can I compile the kernel with gcc 2.8.x, egcs, (add your xyz compiler here)? What about optimizations? How do I get to use -O99, etc.?
5. I compiled the kernel with xyz compiler and get the following warnings/errors/strange behavior, should I post a bug report to the list? Should I post a patch?
6. Why does my kernel compilation stops at random locations with: "Internal compiler error: program cc1 caught fatal signal 11."?
7. What compiler flags should I use to compile modules?
8. Why do I get unresolved symbols like foo__ver_foo in modules?
9. Why do I get unresolved symbols with __bad_ in the name?

Section 9 - Feature specific questions

1. GNU/Linux Y2K compliance?
2. What is the maximum file size supported under ext2fs? 2 GB?
3. GGI/KGI or the Graphics Interface in Kernel Space debate?
4. How do I get more than 16 SCSI disks?
5. What's devfs and why is it a Good Idea (tm)?
6. Linux memory management? Zone allocation?
7. How many open files can I have?
8. When will the Linux accept(2) bug be fixed?
9. What about STREAMS? I noticed Caldera has a STREAMS package, when will that go in the kernel source proper?
10. I need encryption and steganography. Why isn't it in the kernel?
11. How about an undelete facility in the kernel?
12. How about tmpfs for Linux?
13. What is the maximum file size/filesystem size?
14. Linux uses lots of swap while I still have stuff in cache. Isn't this wrong?
15. Why don't we add resource forks/streams to Linux filesystems like NT has?
16. Why don't we internationalise kernel messages?

Section 10- "What's changed between kernels 2.0.x and 2.2.x" questions

1. Size (source and executable)?
2. Can I use a 2.2.x kernel with a distribution based on a 2.0.x kernel?
3. New filesystems supported?
4. Performance?
5. New drivers not available under 2.0.x?
6. What are those __initxxx macros?
7. I have seen many posts on a "Memory Rusting Effect". Under what circumstances/why does it occur?
8. Why does ifconfig show incorrect statistics with 2.2.x kernels?
9. My pseudo-tty devices don't work any more. What happened?
10. Can I use Unix 98 ptys?
11. Capabilities?
12. Kernel API changes

Section 11- Primer documents

Note that writing the longer text/Web page on some relevant Linux kernel topic and providing a Q/A in this section confers you instant Guru status. Some people would *kill* for this. Now go and write your stuff. ;)

1. What's a primer document and why should I read it first?
2. How about having I/O completion ports?
3. What is the VFS and how does it work?
4. What's the Linux kernel's notion of time?
5. Is there any magic in /proc/scsi that I can use to rescan the SCSI bus?

Section 12- Kernel Programming Questions

1. When is cli() needed?
2. Why do I see sometimes a cli()-sti() pair, and sometimes a save_flags-cli()-restore_flags sequence?
3. Can I call printk() when interrupts are disabled?
4. What is the exact purpose of start_bh_atomic() and end_bh_atomic()?
5. Is it safe to grab the global kernel lock multiple times?
6. When do I need to initialise variables?

Section 13- Mysterious kernel messages

1. What exactly does a "Socket destroy delayed" mean?
2. What do I do about "inconsistent MTRRs"?
3. Why does my kernel report lots of "DriveStatusError BadCRC" messages?
4. Why does my kernel report lots of "APIC error" messages?

Section 14- Odd kernel behaviour

1. Why is kapmd using so much CPU time?
2. Why does the 2.4 kernel report Connection refused when connecting to sites which work fine with earlier kernels?
3. Why does the kernel now report zero shared memory?
4. Why does lsmod report a use count of -1 for some modules? Is this a bug?
5. Why doesn't the kernel see all of my RAM?
6. I've mounted a filesystem in two different places and it worked. Why?

Section 15- Programming Religion

1. Why is the Linux kernel written in C/assembly?
2. Why don't we rewrite it all in assembly language for processor Mega666?
3. Why don't we rewrite the Linux kernel in C++?
4. Why is the Linux kernel monolithic? Why don't we rewrite it as a microkernel?
5. Why don't we replace all the goto's with C exceptions?
6. Why are the kernel developers so dismissive of new techniques?

Section 16- User-space Programming Questions

1. Why does setsockopt() double SO_RCVBUF?

Answers

Section 1 - General questions

1. Why do you use "GNU/Linux" sometimes and just "Linux" in other parts of the FAQ?
   • (ADB) In this FAQ, we have tried to use the word "Linux" or the expression "Linux kernel" to designate the kernel, and GNU/Linux to designate the entire body of GNU/GPL'ed OS software, as found in the various distributions. We prefer to call a cat, a cat, and a GNU, a GNU. ;-)
     The purpose of the FAQ is to provide information on the Linux kernel and avoid debates on e.g. semantics issues. Further discussion of the relationship between GNU software and Linux can be found at http://www.gnu.org/gnu/linux-and-gnu.html.
     BTW, it seems many people forget that the linux kernel mailing list is a forum for discussion of kernel-related matters, not GNU/Linux in general; please do not bring up this subject on the list.
2. What is an experimental kernel version?
   • (ADB) Linux kernel versions are divided in two series: experimental (odd series e.g. 1.3.xx or 2.1.x) and production (even series e.g. 1.2.xx, 2.0.xx, 2.2.x, 2.4.x and so on). The experimental series are fast moving versions which are used to test new features, algorithms, device drivers, etc. By their own nature the experimental kernels may behave in unpredictable ways, so one may experience data losses, random machine lockups, etc.
3. What is a production kernel?
   • (ADB) Production or stable kernels have a well defined feature set, a low number of known bugs, and tried and proven drivers. They are released less frequently than the experimental kernels, but even so some "vintages" are considered better than others. GNU/Linux distributions are usually based on chosen stable kernel versions, not necessarily the latest production version.
4. What is a feature freeze?
   • (ADB) A feature freeze is when Linus announces on the linux-kernel list that he will not consider any more features until the release of a new stable kernel version. Usually the net effect of such an announcement is that on the following days people on the list propose a flurry of new features before Linus really enforces the feature freeze. ;-)
5. What is a code freeze?
   • (ADB) A code freeze is more restrictive than a feature freeze; it means only severe bug fixes are accepted. This is a short phase that usually precedes the creation of a new stable kernel tree.
6. What is a f.g.hhprei kernel?
   • (ADB) These are intermediate pre-release versions of version f.g.hh. Note that usually i < 5, but e.g. 2.0.34prei was available with i = 1 to 16. Sometimes "pre" is replaced by the initials of the developer putting together the kernel revision, e.g. 2.1.105ac4 means the 4th intermediate release of kernel version 2.1.105 by Alan Cox.
7. Where do I get the latest kernel source?
   • (ADB) The primary site for the Linux kernel (experimental and production) sources is hosted by Transmeta (the company Linus Torvalds used to work for) on a dedicated Web server at http://www.kernel.org/. This site is mirrored across the world, and has pointers to mirrors for each country. You can go directly to a mirror for your country by going to http://www.CODE.kernel.org/ where "CODE" is the appropriate country code. For example, "au" is the country code for Australia, so the principle mirror site for Australia is http://www.au.kernel.org/
   • (REG) You may also access tarballs and patches directly via ftp from ftp://ftp.CODE.kernel.org/pub/linux/kernel/ which is where Linus distributes his kernels from. Other notable kernel hackers have directories under the people directory, which is where they keep their kernel patches. The testing directory is where Linus puts pre-release patches. The pre-release patches are mainly intended for other developers, so they can stay in sync with changes in Linus' source tree. These are often highly experimental and may crash or cause filesystem corruption. Use at your own risk.

     Note that Linus and Marcelo are using GIT to manage their kernel source trees, and it is more convenient for them to make snapshots of their latest trees available via GIT, rather than make patches. If you want access to these snapshots (which are merely a work in progress, and may be buggy), there are several access methods available:

     CVS:		:pserver:anonymous@cvs.kernel.org:/home/cvs/linux-2.[45]
     Subversion:	svn://svn.kernel.org/linux-2.[46]/trunk
     

   • (JBG) Linux is no longer maintained with the BitKeeper source code management system, but with GIT, a tool Linus wrote after BitKeeper was no longer available to all developers. You can browse Linus's latest kernel source as well as all other people's projects hosted on kernel.org. There's also a nice Overview of GIT and some helper tools as well as a complete Tutorial to get you into using GIT.
8. Where do I get extra kernel patches?
   • (REG) There are many places which provide various extra patches to the kernel for new features. One fairly good archive is available at: http://www.linuxhq.com/.
9. What is a patch?
   • (RRR) A patch file (as it refers to the Linux kernel) is an ASCII text file that contains the differences between the original code and the new code, plus some additional information such as filenames and line numbers. The patch program (man patch) can then apply the patch to an existing kernel source tree.
10. How do I make a patch suitable for the linux kernel list?
    • (REG) Here are some basic guidelines for posting patches. For information on how to generate patches, see the entry by RRR below.
      • Ensure the patch does not have trailing control-M characters on each line. A number of broken tools used to encode patches add control-M for "DOS compatibility". This breaks many versions of patch, so be sure to configure your tools properly, or use unbroken tools, otherwise your patch will be silently deleted.
      • Include the patch inline in your email, in plain text. Do not post it as a base64 MIME attachment. Many people will not be able to read your patch, and thus your patch will be deleted without comment.
      • This FAQ previously advised posting a URL to a patch if the patch is large. This is no longer recommended. The preferred way to submit a large patch is to break it up into logical chunks, with a descriptive comment for each, and post each piece with a subject line like
        "[PATCH] cleanup of foo driver [1/5]".
        Do not start a new thread for each chunk - rather, post each chunk as a followup to the previous chunk. You may want to begin with an explanatory post, and label it something like
        "[PATCH] cleanup of foo driver [0/5]".
        See Documentation/SubmittingPatches for more information.
      • If you want Linus or one of the primary maintainers (i.e. Marcelo, David) to apply your patch, you must Cc: them explicitly, otherwise your patch will be ignored.
      • When sending patches to Linus or one of the primary maintainers, you must include the patch inline, in plain text, no matter how large the patch.
      • If you want to send a patch to the list for comment, and also send it to Linus/primary maintainer for inclusion, and the patch is large, you may wonder how to reconcile the conflicting requirements. The solution is obvious: post the URL to the mailing list, wait for comments, and later send the patch, inline, to Linus/primary maintainer. Yes, this is more work for you. No, we don't care.
      • If you have a mailer that eats whitespace or causes similar corruption, then FIX YOUR MAILER, don't expect to be able to take the easy solution and MIME encode your patch.
      Finally, I've seen one person question the veracity of these guidelines, stating that the rules are rather more relaxed, and this FAQ is being over zealous. Fortunately, the King Penguin himself responded to this, so I include his words on this, so that there can be no doubt:

      If I get a patch in an attachment (other than a "Text/PLAIN" type
      attachment with no mangling and that pretty much all mail readers and
      all tools will see as a normal body), I simply WILL NOT apply it unless
      I have strong reason to. I usually wont even bother looking at it,
      unless I expected something special from the sender.
      
      Really. Don't send patches as attachments. 
      
      		Linus
      

    • A caveat applies for people using a Mozilla Mail client. Andrew Morton noted that Mozilla mangles spaces in column zero when patches are included in the message body. Fortunately, Mozilla Mail sends patch attachments as type text/plain or text/x-patch (depending on the presence of a file extension), so it's safe to send patches as attachments instead.
    • (RRR) To make a patch you use the diff program (read the info file for diff). The easiest way to do this is to set up two source trees under /usr/src, set a symlink "/usr/src/linux" to point to the modified tree, and diff one tree against the other. The file /usr/src/Documentation/CodingStyle has more specific information, read it. Things to remember:
      • Always specify unified (-u) diff format.
      • Avoid making formatting changes to the source that make the diff needlessly larger. Watch out for editors that convert tabs to spaces or vice versa.
      • Unless you have specific reasons, diff against the latest official source tree. Otherwise, your patch is likely to be ignored. Either way, specify in your post against what you've diff'ed.
      • Make sure your diff includes only the intended changes in your patch, not every other patch you have made to your source tree. Usually patches are limited to a few files, or directories. It is best to only diff the relevant files i.e. if I only made changes to the file driver_xyz.c under drivers/net, then I would use the following commands (assuming you have the original source tree named "linux-2.1.105", and the modified tree pointed at by the symlink "linux"):

        cd /usr/src
        diff -u linux-2.1.105/drivers/net/driver_xyz.c            \
                        linux/drivers/net/driver_xyz.c  > my_patch
        

      • The following two should go without saying: the arguments to diff are first source (the original, unmodified file(s)), and then destination (your modified version of the file(s)), otherwise you get a reversed patch (and lots of people wondering what you're smoking). Also, make sure your patch applies and compiles cleanly.
      • Of course you need to set up two identical source directories to be able to diff the tree later. A nice trick -- requiring a little bit of consideration, though -- is to create the modified source tree from hard links to the original source tree:

        tar xzvf linux-2.1.anything.tar.gz
        mv linux linux-2.1.anything.orig
        cp -al linux-2.1.anything.orig linux-2.1.anything
        

        This will hardlink every source file from the original tree to a new location; it is very fast, since it does not need to create some 80+ megabytes of files. You can now apply patches to the linux-2.1.anything source tree, since patch does not change the original files but move them to filename.orig, so the contents of the hard-linked file will not be changed.

        Assuming that your editor does the same thing, too (moving original files to backup files before writing out changed ones) you can also freely edit within the hardlinked tree. If your editor does not handle files this way, you need to make a copy of each file before editing it, like this:

        cp driver_xyz.c temporary; mv temporary driver_xyz.c
        

        You can use file permissions to remind you to do this. Just remove write permissions from all the files in the directory you are working in:

        chmod -w *.c
        

        The changed tree can be diffed at high speed, since most files don't just have indentical contents, they are identical files in both trees. Naturally removing that tree is quite fast, too. Thanks to Janos Farkas <chexum@shadow.banki.hu> for this trick.

      • Finally, review the patch file (the format is not that complicated) before posting, and include all relevant information as to the nature of the patch. In particular, specify: why is this patch needed/useful, and what exactly does it fix/improve.
11. How do I apply a patch?
    • (TAC) (From /usr/src/linux/README) You can upgrade between releases by patching. Patches are distributed in the traditional gzip and the new bzip2 format. To install by patching, get all the newer patch files, enter the top-level directory of the unpacked kernel source tree and execute:
      gzip -cd patchXX.gz | patch -p1 or:
      bzip2 -dc patchXX.bz2 | patch -p1
      (repeat xx for all versions bigger than the version of your current source tree, in order) and you should be ok. You may want to remove the backup files (xxx~ or xxx.orig), and make sure that there are no failed patches (xxx# or xxx.rej). If there are, either you or me has made a mistake.
      Alternatively, the script patch-kernel can be used to automate this process. It determines the current kernel version and applies any patches found. Use it thus:
      scripts/patch-kernel .
      The first argument in the command is the location of the kernel source. Patches are applied from the current directory, but an alternative directory can be specified as the second argument.
    • (RRR) To apply kernel patches please take a look at the kernel README file (/usr/src/linux/README) under "Installing the kernel". There is also a good explanation on the Linux HQ Project site.
12. What's vger?
    • (REG) "vger" is the name of the machine which hosts the LKML server. This server also hosts a number of other linux-related mailing lists. More information about the server is available at http://vger.kernel.org/
13. What is a CVS tree? Where can I find more information about CVS?
    • (REG) "CVS" is short for Concurrent Versions System, a Source Code Management system. Check out the CVS Bubbles page.
14. Is there a CVS tutorial somewhere?
    • (ADB) Here is a CVS tutorial which you can find online:
      • An interactive CVS tutorial.
      Getting a general idea of how CVS works takes about 15 minutes (highly recommended). Note that there are various graphical front ends to CVS, so you don't have to learn the usual assortment of cryptic commands.
15. How do I get my patch into the kernel?
    • (RRR) Depending on your patch there are several ways to get it into the kernel. The first thing is to determine under which maintainer does your code fall into (look in the MAINTAINERS file). If your patch is only a small bugfix and you're sure that it is 'obviously correct', then by all means send it to the appropriate maintainer and post it to the list. If there is urgency to the bugfix (i.e. a major security hole) you can also send it to Linus directly, but remember he's likely to ignore random patches unless they are "obviously correct" to him, have the maintainer's approval, or have been well tested and meet the first condition. In case you're wondering what constitutes well tested, here's another important bit: one purpose of the list is to get patches peer-reviewed and well-tested. Now, if your patch is relatively big, i.e. a rewrite of a large code section or a new device driver, then to conserve bandwidth and disk-space just post an announcement to the list with a link to the patch. Lastly, if you're not too sure about your patch yet, want some feedback from the maintainer, or wish to avoid open-season flaming on work-in-progress, then use private email.
    • (REG) If there is no specific maintainer for the part of the kernel you want to patch, then you have three main options:
      • send it to linux-kernel@vger.kernel.org and hope someone picks it up and feeds it to Linus, or maybe Linus himself will pick it up (don't count on it)
      • send it to linux-kernel and Cc: Linus Torvalds <torvalds@osdl.org> and hope Linus will apply it. Note that Linus operates like a black box. Do not expect a response from him. You will need to check patches he releases to see if he applied your patch. If he doesn't apply your patch, you will need to resend it (often many times). If after weeks or months and many patch releases he still hasn't applied it, maybe you should give up. He probably doesn't like it
      • send it to linux-kernel and Cc: Alan Cox <alan@redhat.com>. Alan is better at responding to email, and will queue your patch and resend it to Linus periodically, so you can forget about it. He also serves as a good taste tester. If Alan accepts your patch, it's more likely that Linus will too. If he doesn't like your patch, you will probably get an email saying so. Expect it to be terse.
16. Why does the kernel tarball contain a directory called linux/ instead of linux-x.y.z/ ?
    • (DW) Because that's the way Linus wants it. It makes applying many consecutive patches simpler, because the directory doesn't need to be renamed each time, and it also makes life easier for Linus.
17. What's the difference between the official kernels and Alan Cox's -ac series of patches?
    • (REG, contributed by Erik Mouw) Alan's kernel can be seen as a test bed for Linus' kernels. While Linus is very conservative and only applies obvious and well tested patches to the 2.4 kernel, Alan maintains a set of kernel patches that contains new concepts, more and/or newer drivers, and more intrusive patches. If the patches prove themselves stable, Alan submits them to Linus to include them into the official kernel.
18. What does it mean for a module to be tainted?
    • (REG, contributed by John Levon) Some vendors distribute binary modules (i.e. modules without available source code under a free software license). As the source is not freely available, any bugs uncovered whilst such modules are loaded cannot be investigated by the kernel hackers. All problems discovered whilst such a module is loaded must be reported to the vendor of that module, not the Linux kernel hackers and the linux-kernel mailing list. The tainting scheme is used to identify bug reports from kernels with binary modules loaded: such kernels are marked as "tainted" by means of the MODULE_LICENSE tag. If a module is loaded that does not specify an approved license, the kernel is marked as tainted. The canonical list of approved license strings is in linux/include/linux/module.h.
      "oops" reports marked as tainted are of no use to the kernel developers and will be ignored. A warning is output when such a module is loaded. Note that you may come across module source that is under a compatible license, but does not have a suitable MODULE_LICENSE tag. If you see a warning from modprobe or insmod for a module under a compatible license, please report this bug to the maintainers of the module, so that they can add the necessary tag.
    • (KO) If a symbol has been exported with EXPORT_SYMBOL_GPL then it appears as unresolved for modules that do not have a GPL compatible MODULE_LICENSE string, and prints a warning. A module can also taint the kernel if you do a forced load. This bypasses the kernel/module verification checks and the result is undefined, when it breaks you get to keep the pieces.
    • (KO) According to Alan Cox, a license of "BSD without advertisement clause" is not a suitable free software license. This license type allows binary only modules without source code. Any modules in the kernel tarball with this license should really be "Dual BSD/GPL".
19. What is this about GPLONLY symbols?
    • (REG) By default, symbols are exported using EXPORT_SYMBOL, so they can be used by loadable modules. During the 2.4 series, a new export directive EXPORT_SYMBOL_GPL was added. This is almost the same thing, except that the symbol can only be accessed by modules which have a GPL compatible licence (note that this includes dual-licenced BSD/GPL code). This new directive was added for these reasons:
      • To clarify the ambiguous legal ground on which non-GPL (particularly proprietary) modules lie. A strict reading of the GPL prohibits loading proprietary modules into the kernel. While Linus has consistently stated that proprietary modules are allowed (i.e. he has granted an explicit exemption), it is not clear that he is able to speak for all developers who have contributed to the Linux kernel. While many think Linus' edict means that all contributed code falls under this exemption granted by Linus, not everyone agrees that this is a legally sound argument. The new EXPORT_SYMBOL_GPL directive makes the licence conditions explicit, and thus removes the legal ambiguity.
      • To allow choice for developers who wish, for their own reasons, to contribute code which cannot be used by proprietary modules. Just as a developer has the right to distribute code under a proprietary licence, so too may a developer distribute code under an anti-proprietary licence (i.e. strict GPL).
      Note that Linus has stated that existing symbols will not be switched to GPL-only. Developers of proprietary modules for Linux need not fear. Furthermore, it is quite unlikely that Linus will look favourably upon the introduction of new core driver APIs which are restricted to GPL-only modules. This would not be in the best interests of Linux. Linus has forwarded me a message he sent to someone else to clarify his views. Note that since that time, several developers have eroded the number of non-GPL only symbols by writing new (usually better) infrastructure and interfaces and deprecating the older interfaces. The newer interfaces are often tagged as GPL-only. In addition, there are some "kernel janitors" who aggressively submit patches to remove all symbols (whether GPL-only or not) which are not used by code shipped with the kernel source tree.
20. Do I have to use GIT to send patches?
    • (REG) Absolutely not. Some kernel developers, including Linus and Marcelo, have chosen to use GIT to manage their kernel source trees, but this does not mean you need to use GIT yourself to maintain your trees or submit patches. Many notable kernel developers continue to maintain their source trees using other tools and techniques, and continue to send conventional patches.
21. Who maintains the kernel?
    • (REG) Originally, Linus Torvalds maintained the kernel. As the kernel has matured, he has delegated maintenance for older stable versions to others, while he continues development of the latest "bleeding edge" release. As of 27-MAY-2002, the following kernel versions are maintained by these people:
      • 2.0 David Weinehall <tao@acc.umu.se>
      • 2.2 Alan Cox <alan@lxorguk.ukuu.org.uk>
      • 2.4 Marcelo Tosatti <mtosatti@redhat.com>
      • 2.6 Linus Torvalds <torvalds@osdl.org>
22. The kernel doesn't compile cleanly. What shall I do?
    • (REG) First make sure you have the latest version of that kernel series. Perhaps a pre-patch already has a fix. If not, search the list archives for a fix. Don't contribute to noise on the list by asking a question that may already have been answered.
      If the problem has not yet been fixed, try digging into the code yourself and post a fix to the mailing list. You'll be famous! Beware that making broken code compile just for the sake of a clean 'make bzImage modules' doesn't count as a fix, and your fix will be discarded, ignored or flamed.

Section 2 - Driver specific questions

1. Driver such and such is broken!
   • (RRR) Try to be more specific. Please, provide information on your particular setup (see Qs How do I make a bug report?) Also see the Q: "kernel x.y.z broken!" below.
   • (ADB) That's the worst possible way to start a thread. Please try to reach the author of the driver first and report the "broken" driver to him. Constructive criticism is welcome, usually.
2. Here is a new driver for hardware XYZ.
   • (REW) Good work! Please try to find a few people that also have the XYZ hardware and have them test it on their configuration (e.g. by posting a message on a newsgroup). No it won't go in the standard kernel before some people have tested it.
     Testing will take a while. In the mean time, kernel development will continue, and you will have to rewrite your patch for the most recent version before Linus might consider it.
     As a whole new driver is most likely more than a few pages long, we'd prefer it if you would put the actual driver up for ftp instead of posting it to the list. Post the URL and the description that tells us what your driver does for which hardware.
3. Is there support for my card TW-345 model C in kernel version f.g.hh?
   • (REW) First check if your card is detected at boot time. It usually is. Second see if you might need to configure something like modules.conf for your card. Third see if there is a file with the card name in the kernel sources. (e.g. you have a Buslogic card, and there is a buslogic.c file in the kernel sources, you're in luck.). Next, grep for the manufacturer name through ALL the kernel sources. And try the model number of your card. Also try to find the largest chip on your card and grep for the chip number on that thing. Realize that 53C80 chips might be named 5380 in the kernel. Other chips don't have their middle name removed.
     Nothing yet? Now check DejaNews, using the same arguments you used to grep the kernel source. There are 99.99% chances that somebody has exactly the same card TW-345 model C.
     Ok. That's what you can do without bothering anyone. If all this doesn't lead somewhere, you should really ask this question on a newsgroup like comp.os.linux.hardware.
4. Who maintains driver such and such?
   • (RRR) Have a look at the /usr/src/linux/MAINTAINERS file, this is the most authoritative source. Also check the source code for the driver itself; in both cases, check the latest version of the kernel that you have available. Some drivers have specific Web pages and sometimes even a dedicated mailing list. Check those first. If you cannot contact the maintainer then as a last resort post a short message to the list. In any case, keep in mind that maintainers are usually very busy people and most of them work on Linux for free and in their spare time, so don't expect an immediate response. Some maintainers get just too many mails in too small periods of time to be able to answer them all, so please be kind to them.
5. I want to write a driver for card TW-345 model C, how do I get started?
   • (REW) Good initiative! First a piece of advise: are you up to this? Ten times as many projects like this get started as get finished. Also, make sure that you're not doing double work. Make sure that such a driver is not already available: read Q/A 2.3 above...
     First prepare yourself. Get the docs, read them (OK, you're allowed to start skipping stuff if you've gotten to the part "detailed register descriptions"). Next, get the Linux kernel source, find a driver that drives similar hardware to the one you're going to work on, and read THAT. (I usually use the smallest one I can find: wc -l *.c | sort -n | head -4).
     Ok. You've thought about it. Now the question is, do you have technical documentation for your card? You can reverse engineer the driver for MS operating systems, but having the documentation is MUCH easier.
     In the dark old ages (70s to middle of the 80s), you got a complete technical description with every card you could get. This is no longer the case. Anyway, contact your vendor and politely ask them for the "device driver kit" or the "technical manual" for the card.
     Try the head office and your local office at the same time. Local offices occasionally have bad photo copies that they give out before you get an official rejection from the head office. In that case whom you got the documentation from becomes confidential information. Don't put the guy's name in the source.
     If you can't get the technical documentation, consider giving up and investing in a competitors product (and tell the manufacturer about this). Not given up yet? Ok. Next step is to find out what the DOS driver does. Try to get the card to work while you run it in a microsoft emulator (dosemu or WINE). This will allow you to program these tools to log the I/O accesses of the driver. This will give you a large list of I/O accesses that the driver did. If you're good, you might be able to see patterns, and deduce how the driver works. From there you might be able to write a working driver. Good luck! You'll need it.
6. I want to get the docs, but they want me to sign an NDA (Non-Disclosure Agreement).
   • (REW) Some people find this a tremendous problem. Some companies just want to know who has the docs to their hardware, and don't mind if you write a GPL-ed driver. In that case, there is really no problem: just tell them what you intend to do and ask them to acknowledge in writing that they've understood what you're saying. In that case, you can get your driver into the standard kernel, but you cannot send out the docs to anybody who wants to work on the driver. They will have to rely on the comments in the source.
     Other companies (just like Netscape) themselves signed NDAs that forbids them to disclose information to you.
     Some really think that they have trade secrets in the interface towards the software, and intend to keep them secret. Those won't allow you to write a driver and then put the source on the net. Be careful with these.
   • (ADB) The first and only NDA I ever received instantly found its way to the wastebasket. I would advise anybody who gets an NDA to refuse to sign it, if it refers to anything that may/will be put under GNU/GPL. Of course, for contract work this doesn't apply.
7. I want/need/must have a driver for card TW-345 model C! Won't anybody write one for me?
   • (REW) Some Linux developers will settle for a beer, and develop the driver for you. Others want a "free sample" of the hardware and will then go ahead and write the driver.
     If you need more than a few of the cards or you manufacture the cards yourself, you can consider paying one of the commercial Linux device driver companies to get a commercially backed, officially maintained device driver.
8. What's this major/minor device number thing?
   • (REG) Device numbers are the traditional Unix way to provide a mapping between the filesystem and device drivers. A device number is a combination of a major number and a minor number. Currently Linux has 8 bit majors and minors. When you open a device file (character or block device) the kernel takes the major number from the inode and indexes into a table of driver structure pointers. The specific driver structure is then used to call the driver open() method, which in turn may interpret the minor number. There are two tables: one for character devices and one for block devices, each are 256 entries maximum. Obviously, there must be agreement between device numbers used in a driver and files in /dev. The kernel source has the file Documentation/devices.tex which lists all the official major and minor numbers. H. Peter Anvin (HPA) maintains this list. If you write a new driver (for public consumption), you will need to get a major number allocated by HPA. See the Q/A on devfs for an improved (IMHO) mechanism for handling device drivers.
9. Why aren't WinModems supported?
   • (REG, quoting Edward S. Marshall) The problem is the lack of specifications for this hardware. Most companies producing so-called "WinModems" refuse to provide specifications which would allow non-Microsoft operating systems to use them.
     The basic issue is that they don't work like a traditional modem; they don't have a DSP, instead making the CPU do all the work. Hence, you can't talk to them like a traditional modem, and you need to run the modem driver as a realtime task, or you'll have serious data loss issues under any kind of load. They're simply a poor design.
   • (REG) Note that some people have been putting effort into reverse engineering some WinModems, so you may be lucky and find that yours is now supported. If not, it's time to get a refund and buy a real modem.
     Note that modems have to be approved by the appropriate statutory or regulatory body for standards compliance (to make sure they don't send crap down the line and blow up the exchange). With WinModems, the driver software needs to be certified as well as the hardware. It's harder to get approval for Open Source drivers, since it usually costs money to obtain approval. Also, in theory, it's easier to modify an Open Source driver, so it would no longer be compliant. In reality, 99.999% of users don't even know there is source code for the driver, so "Standards Compliance" may well be a smoke-screen for manfacturers who don't want to bother with non-WinTel systems. If certification was the only problem, manufacturers could release binary-only drivers.
   • (DW)The good news is that a certain amount of WinModem hardware is now supported. The bad news is that that is just the tip of the iceberg. Although the WinModems can now be used, they have functionality similar to that of a sound card - all the modulation and demodulation has to be performed by the host CPU. Work is progressing on this front too - see http://www.linmodems.org/ for more up-to-date information.
10. Modern CPUs are very fast, so why can't I write a user mode interrupt handler?
    • (REG, quoting Pete Zaitcev) This is not a question of having enough CPU cycles to waste them on mode switches. Rather, the current Linux architecture does not allow it. User processes run with interrupts enabled. Thus, any interrupt handler must deactivate the particular interrupt source before a process is scheduled to run, or an interrupt storm results. The deactivation is done in a device specific manner, so at least a small device driver must be present in kernel mode.
11. Do I need to test my driver against all distributions?
    • (REG, MEA) There are minor detail changes in between each kernel version (even in stable series), and depending on what configuration options are used (basically SMP or not), certain things like spinlocks may or may not reserve space in structures, and may or may not need to be called (are even optimized away in non-SMP systems), meaning that a binary driver compiled for SMP might not work with a non-SMP kernel. And vice versa.
      Also different vendors tend to inject different things into their kernel patch-sets, which again may subtly change data layouts, etc. In stable kernel series great pains are suffered at maintenance so that data layouts of in-kernel APIs (and API calls themselves) are not changed. Nevertheless something may change making binary drivers to fail in mysterious ways.
      Subtle memory changes may appear with i386-PAE mode (large memory machines which can't map all of RAM into the kernel at the same time).
      Because of these differences, a driver compiled for one version of the kernel, or one vendor's kernel, is not likely to work with another kernel. Thus, if you are distributing a binary-only driver, you will have a significant support load compiling drivers for different kernels. If you are distributing a driver in source form, then, provided the driver is well-written (i.e. does not make assumptions about byte ordering or word sizes and uses standard kernel interfaces), the driver should be portable across kernel versions and architecture types. It will of course have to be compiled by end-users for their particular kernel. Distribution maintainers are likely to provide pre-compiled drivers, thus most end-users won't need to compile the driver themselves.

Section 3 - Mailing list questions

"Test" messages are very, very inappropriate on the lkml or any other list, for that matter. If you want to know whether the subscribe succeeded, wait for a couple of hours after you get a reply from the mailing list software saying it did. You'll undoubtedly get a number of list messages. If you want to know whether you can post, you must have something important to say, right? After you have read the following paragraphs, compose a real letter, not a test message, in an editor, saving the body of the letter in the off chance your post doesn't succeed. Then post your letter to lkml. Please remember that there are quite a number of subscribers, and it will take a while for your letter to be reflected back to you. An hour is not too long to wait.

(REG) The essential point to remember when posting to the linux-kernel mailing list is that there are a lot of very busy people reading the list. No matter how important you think you are, it is most likely that there are many people on the list who are more important than you. "Important" is not measured by the amount of money you have, how much your question is worth to your company or how desperate you are for an answer, rather, it is measured by how much you contribute to the linux kernel.

With that in mind, you should make sure that you are not wasting the time of other people on the list. Write for maximum efficiency of reading. It doesn't matter if it takes twice as long for you to compose a more readable message, if it halves the time a hundred key kernel developers spend trying to decode your message. Ignoring good taste and consideration is most likely to result in you being ignored.

1. How do I subscribe to the linux-kernel mailing list?
   • (ADB) Think again before you subscribe. Do you really want to get that much traffic in your mailbox? Are you so concerned about Linux kernel development that you will patch your kernel once a week, suffer through the oopses, bugs and the resulting time and energy losses? Are you ready to join the Order of the Great Penguin, and be called a "Linux geek" for the rest of your life? Maybe you're better off reading the weekly "Kernel Traffic" summary at http://www.kerneltraffic.org/.
     OK, if you still want to read linux-kernel in its full glory, send the line "subscribe linux-kernel your_email@your_ISP" in the body of the message to majordomo@vger.kernel.org (don't include the " characters, and of course replace the fake email address with your true address). You have been warned!
   • (MEA) Quite often I see things like what this summary report tells:

     FAILED:
       <smtp cedar-republic.com edmond@cedar-republic.com 60000>: ...\
             <<- RCPT To:<edmond@cedar-republic.com>
             ->> 550 <edmond@cedar-republic.com>... we do not relay
     

     Feeding this address to a page at URL: http://vger.kernel.org/mxverify.html yields information that ONE of their backup MX servers refuses to send email thru to them. Thus whenever all other servers fail to be reachable, that one ruins their email connectivity.

     Do make sure YOU don't have this very problem!
     See http://vger.kernel.org/majordomo-info.html for information on Majordomo.

2. How do I unsubscribe from the linux-kernel mailing list?
   • (ADB) At the bottom of each and every message sent by the linux-kernel mailing list server one can read:
     -
     To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
     the body of a message to majordomo@vger.kernel.org
     See http://vger.kernel.org/majordomo-info.html for information on Majordomo.
3. Do I have to be subscribed to post to the list?
   • (ADB) No, you don't have to be subscribed to the list to post to it. The address of the list is linux-kernel@vger.kernel.org. And you should indicate on your message that you wish to be personally CC'ed the answers/comments posted to the list in response to your posting.
   • (REG) It is, however, generally considered good netiquette to be subscribed to a list (or a newsgroup for that matter) and lurk for a while before posting. That way you can learn what's considered an appropriate post and what isn't.
     Don't treat the list as your personal helpdesk. Remember that the list is a community.
4. Is there an archive for the list?
   • (REG) There are many. Here are some:
     • http://www.uwsg.indiana.edu/hypermail/linux/kernel/ has a search by word/subject capability.
     • http://marc.theaimsgroup.com/?l=linux-kernel keeps a collection of Linux related list archives.
     • http://groups.google.com/groups?hl=en&q=fa.linux.kernel&meta= is a Google interface to the fa.linux.kernel newsgroup, which is in turn fed from the mailing list.
     • http://gossamer-threads.com/lists/linux/kernel/ has an easy interface and an appealing format (click on a thread, shows all posts in a thread with posts clearly delimited).
     Here are some more resources:
     • "Kernel Traffic" at http://www.kerneltraffic.org/ provides a weekly summary of the discussions on the list, and archives previous summaries.
5. How can I search the archive for a specific question?
   • (ADB) Use simple keywords which refer to the issue that matters to you. For example, if you are investigating an oops that happens whenever you plug in a network adapter NIC-007, use "NIC-007" or "oops NIC-007". As soon as you have found a link to a message that interests you, try to follow the thread. Remember that you will almost always get more information by carefully searching the archive than by posting a question to the list itself.
6. Are there other ways to search the Web for information on a particular Linux kernel issue?
   • (ADB) Sure. Before you check the list archives, you can search DejaNews and AltaVista (simultaneously, if your browser allows you to open various windows). You can also follow some links on the Linux Documentation Project site.
7. How heavy is the traffic on the list?
   • (TAC) List traffic is very heavy; the average number of messages per day is 290 [10/2002 - 04/2003]. That's over 8,700 message a month!!!
   • (ADB) You really don't want to read each and every posting to the list. If you are concerned with list traffic, I suggest you temporarily try the digest lists, which will be much easier on your mailbox (thanks to A. Wik for this suggestion).
   • (REG) There is a weekly summary called "Kernel Traffic" at http://www.kerneltraffic.org/, which can save you a lot of time.
8. What kind of question can I ask on the list?
   • (ADB) The basic rule is to avoid asking questions that have been asked before, or that are irrelevant to other list users, or that are off topic. Please use your good sense.
   • (REG) Remember that this is a list for the discussion of kernel development. If you have some ideas or bug reports to contribute, this is the place. User space issues are not appropriate for this forum. If you find a bug in the C library or some application, it doesn't belong on linux-kernel.
9. What posting style should I use for the list?
   • (REG, contributed by thunder7@xs4all.nl) When following up a post on the kernel mailing list, please think before you quote. Since everybody else on the list also got the original post, don't quote it entirely. Highlight only the points that you really need to understand your arguments. Make sure the quoted part is recognizable as such, by ensuring each quoted line starts with a > (or more >>, in case of multi-level quoting). Don't quote signatures, entire patches, entire config files or entire posts. Don't quote the standard signature. The kernel-list is crowded enough already, let's take care!
   • (REG) Be aware that your message is far more likely to be deleted without being read if you have too much quoted material before your reply.
   • (REG) And please reply after the quoted text, not before it (as per RFC 1855). It's very confusing to see a reply before the quoted context. And it's embarrassing: it makes you look like a newbie. Change your mailer if necessary, if the one you have makes it hard to do reply-after-quoting.
     I know some people like to quote the entire message they are replying to, so they put their reply right at the top so people won't give up after the first page of quoted material. Don't do it. It's annoying. Just learn to stop quoting everything. No-one wants to see it all anyway (list archives allow people to see everything if they missed it). You're not helping yourself anyway, as you're more likely to be ignored if you reply-before-quoting.
   • (REG) Please don't use tabs or multiple spaces to quote text. Use the "> " sequence instead. Using whitespace to quote text makes it difficult to differentiate between what's quoted and the reply. And don't try to be cute or "different" and use some other character like "}" or whatever. Again, it's confusing. It wastes people's time. Write for maximum efficiency of reading.
   • (REG) Please try to have halfway reasonable spelling and grammar. When reading text with really bad spelling or grammar, people stall while trying to parse your post. Don't think you're being "artistic" by stripping out all punctuation characters. Linux-kernel is not an online gallery, it's a communications medium. Write for maximum efficiency of reading.
   • (REG) Please don't have long, inflammatory, controversial or offensive signatures (see RFC 1855). The rule of thumb is no more than 4 lines of 80 characters each.
   • (PG) Don't attach huge files to your post. One major culprit is people attaching their kernel .config file to their post. These can be in excess of 1000 lines, and will grow more as kernel options are continuously added. If the contents of your .config file are relevant to your post then attach the output of

     grep ^C .config

     or

     grep "=[y|m]" .config

     .
   • (MEA) Some structures are forbidden as they appear to be used way too much in SPAM mail. Specifically, messages with Content-Type: text/html either as the only (primary) message, or as ANY of component sub-messages are considered spam, and rejected outright without any info to the sender.
     Also, any message with header matching the regular expression: X-Mailing-List:.*@vger.kernel.org is considered to be LOOPING somewhere, and is thus diverted to list-owner.
   • (REG) If you are using stuck using Microsoft Outlook or Outlook Express, which have flawed quoting algorithms, you should apply one of the following fixes:
     • http://home.in.tum.de/~jain/software/oe-quotefix/ for Outlook Express
     • http://home.in.tum.de/~jain/software/outlook-quotefix/ for Outlook
     These fixes make these mailers more standards compliant.
10. Is the list moderated?
    • (ADB) No, the linux-kernel list is not moderated.
11. Can I be ejected from the list?
    • (ADB) It is technically possible, but I have never heard of anybody being ejected from the linux-kernel list.
    • (REW) But you will if you post questions or answers that are asked and answered on this FAQ. ;-)
    • (MEA) Oh definitely, all you need to have is malfunctioning email system which does not accept email to you -- e.g. check your domain backup MX servers by using the tool at: http://vger.kernel.org/mxverify.html
      It is known that over the years the keepers of vger's lists have removed some people after getting sufficiently annoyed with them, but there you really must try to exceed yourself, and will likely get lots of peer pressure before getting kicked off.
      Another way to quickly get yourself removed is to use the program called "fetchmail" -- which in itself is not all that bad, but apparently it is far too easy to accidentally re-post email to addresses which the visible RFC 822 headers contain -- that is, what the original sender used, like: To: linux-kernel@vger.kernel.org The result is duplicate messages on the mailing list. If you let that happen, you can be quite sure that your subscription will be removed as soon as possible.
12. Are there any implicit rules on this list that I should be aware of?
    • (ADB) Here are a few implicit rules which you should be aware of:
      • Stick to the subject. This is a Linux kernel list, mainly for developers.
      • Use English only!
      • Don't post in HTML format! If you are using IE or Netscape, please turn off HTML formatting for your posts to the kernel list.
      • If you use that other OS, make sure your mailer doesn't use Charset="Windows*" as those posts will be blocked.
      • If you will be asking a question, before you post to the list, try to find the answer in the available documentation or in the list archives. Just remember that 99% of the questions on this list have already been answered at least once. Usually the first answer is the most detailed, so the archives contain far better information than you will get from somebody who has answered the same question a dozen times or more.
      • Be precise, clear and concise, whether asking a question or making a comment or announcing a bug, posting a patch or whatever. Post facts, avoid opinions.
      • Be nice, there is no need to be rude. Avoid expressions that may be interpreted as aggressive towards other list participants, even if the subject being treated is particularly relevant to you and/or controversial.
      • Don't drag on with controversies. Don't try to have the last word. You will eventually have the last word, but meanwhile you'll have lost all your sympathy credit.
      • A line of code is worth a thousand words. If you think of a new feature, implement it first, then post to the list for comments.
      • It's very easy to criticize someone else's code, but when you write something for the first time, it's not that simple. If you find a bug, a mistake, or something that could be perfected, don't immediately post a comment such as "This piece of code is crap, how did it get into the kernel?". Contact the author of the code, explain the issue, and try to get the point across in a simple, humble way. Do that a few times and you will get a lot of credit as a good code debugger. Then when you write a piece of code people will pay attention to you.
      • Don't flame beginners that ask the wrong questions. This adds noise to the list. Send them a private mail pointing them to a source of information e.g. this FAQ.
    • (MEA) If you post HTML, your email won't make it to the lists (see section 3.9).
    • (RC) Ensure your email doesn't match any of the regular expressions in vger's Majordomo's taboo list of regular expressions else it will be silently dropped. This matches seemingly innocuous words like `Deutschland' as in `Sitecom Deutschland GmbH'.
    • (REG) Don't post post any religious or political material, including in your signature. Doing it in the body of a message will anger people, as it's always off-topic and is a waste of bandwidth (remember that even in the 21st century, many people are still being gouged by the second for bandwidth by their ISP or telco or both).
      Including this unwanted material in your signature is less obnoxious, but is pointless at best (preaching to the converted). Most people will ignore it, and many will be prone to ignore the content of your message, recognising you are a wanker. If you want to be taken seriously, leave the soap-box at home. Limit your posts to technical issues.
13. How do I post to the list?
    • (REG) You send a message to the address linux-kernel@vger.kernel.org
14. Does the list get spammed?
    • (ADB) The linux-kernel list is no longer spammed, you will rarely if ever find a commercial posting to the list itself. OTOH once you post to the list, expect to get a few undesirable mails in the following days. Unfortunately some people watch the list and think it's a good idea to pick names from it. There are many ways to avoid spam, check the dedicated anti spam sites on the list. I learned many things this way.
    • (REW) Although the list maintainers do their best to keep the list spam free, it is not possible to do this 100%. Some of the good kernel development people cannot keep up with the volume on linux-kernel. But they do occasionally post. Therefore we need to keep the submissions open for "everybody". Some of the other important people have two or three Email addresses. They too need to post from different addresses. Consequently something that looks like a submission from a valid return address tends to go on the list. There is nothing an automated filtering system can do about it.
      The end result is about one spam a month. It happens. The maintainer will get a flood of mail about it and he will block the domain it came from. Please don't bother the list about it, don't add noise. Don't post "This guy is a jerk if he spams this list". Don't post "I traced him, you can mail bomb him at this address". Don't post "I traced him, bother his postmaster at such and such".
15. I am not getting any mail anymore from the list! Is it down or what?
    • (ADB) Majordomo is an intelligent mail list server. If for any reason your email address is unavailable, after some retries you will be automatically unsubscribed.
    • (REW) On the other hand, accidents with the mailing list server have happened. These have wiped out the whole subscription list once or twice. Just resubscribe. Majordomo will get you a nice note saying you're still subscribed if suddenly everybody went dumb. Don't post "Just testing: Is the list working? I didn't get any mail for a few days now".
    • (MEA) You may get unsubscribed because MTAs relaying traffic to you get bounces for some reason. One thing to verify is that your email routing data in the DNS is valid, e.g. feed your address to the input box at: http://vger.kernel.org/mxverify.html
    • (MEA) VGER and/or one of its fanout boxes may be in overload. Usually system keepers notice the situation, and it becomes fixed within 1-2 days without messages being lost, but we don't track the entire world. Asking help from postmaster@vger.kernel.org could expedite the issue. Asking help on lists WILL NOT help, doing so just puts more load on the system!
16. Is there an NNTP gateway somewhere for the mailing list?
    • (RRR) Yes there is the newsgroup fa.linux.kernel, but you can only read the mailing list there, not post directly. Posting to the list must go by email to linux-kernel@vger.kernel.org.
      Here's the dejanews URL, if your NNTP host don't have the group http://www.dejanews.com/bg.xp?level=fa.linux.kernel
    • (REG) Unfortunately not all news servers have the fa heirarchy. You can access the fa.linux.kernel by going to http://groups.google.com/groups?hl=en&q=fa.linux.kernel&meta=
    • (REG, contributed by Gunter Ohrner) Yes, GMANE offers a bidirectional gateway at nntp:gmane.linux.kernel at server news.gmane.org with additional web access at http://blog.gmane.org/gmane.linux.kernel.
17. I want to post a Great Idea (tm) to the list. What should I do?
    • (REG) OK, that's great. Now:
      • First make sure that your idea is relevant to kernel development. Perhaps your idea is better implemented in the C library, or perhaps in a new library? Before posting to linux-kernel, be sure it really is a kernel issue.
      • OK, so you have this great idea for the kernel. Are you sure someone hasn't thought of it before? Reading all of this document is a good starting point. Also search the mailing list archives to see if that topic has been raised before.
      • Now you have verified that you have an idea none has suggested before. For the best response, code up an implementation/kernel patch and post that to the kernel list when you announce your idea. If you provide code, you can be sure someone will try it out and give you comments. If you don't know anything about kernel hacking, this is a good time to start learning:-) By the time you've implemented your idea, you'll be able to call yourself a Linux Guru.
      • If you really can't code something up, but still would like your idea implemented, post a message to the kernel list. Be as clear and precise as possible, so that people can understand your idea quickly. If you are lucky, someone who likes your idea may find the time to implement it. If nobody steps forward to implement it, you're out of luck: remember, we're all volunteers and we all have too many things to do as it is.
      • If you get a negative response to your idea, don't get offended, after all, we all have different notions on what is a Good Idea (tm) and a Bad Idea (tm). If someone is rude to you, please resist the temptation to carry on a war on the list. Instead, email them privately saying that you don't like their rudeness. If everybody is polite, but just strongly disagrees with your idea, be careful not to push it too hard. If people haven't understood the point you are making, try explaining it a different way. But if people understand your idea but maintain it is flawed, it's time to stop pushing it. Pushing harder will just get you ignored.
      • If you're convinced you're right, despite what everybody else says, stop talking about it and implement it! If you're right, you'll have the last laugh.
    • (ADB) Good code (i.e. documented, elegant, efficient) and some benchmarking data showing your Great Idea performs well will go a long way to show you're right.
18. There is a long thread going on about something completely offtopic, unrelated to the kernel, and even some people who are in the "Who's who" section of this FAQ are mingling in it. What should I do to fight this "noise"?
    • (REW, ADB) Ignore it.
    • (REG) Don't send a response to the kernel list under any circumstances. If you feel compelled to respond, do so privately informing the person that the message was offtopic. Or set up a procmail recipe to drop all messages for that thread: that way you'll never see the thread again.
19. Can we have the Subject: line modified to help mail filters?
    • (REG) The usual proposition is that a string like [LINUX-KERNEL] is prepended to the subject line.
      This question has been raised many times before, and the answer has always been "no" or "there are better ways to filter email". The majority of the developers, and all (?) of the list maintainers take this position. Some of the reasons are:
      • It would increase the size of the Subject: line. This is a problem, as it limits the amount of useful information that can be seen in the Subject: line, making it harder to scan through a list of subject lines looking for interesting subjects.
      • It can lead to the Subject: line from hell, since some mailers and users don't behave sanely. Imagine the following:
        RE: [LINUX-KERNEL] Re: [LINUX-KERNEL] RE: [LINUX-KERNEL] Re: [LINUX-KERNEL] Critical security flaw in 2.666.0
        That's a lot of characters. The useful information will very likely be lost due to line truncation.
      • It doesn't work for cross-posted messages, as the subject line for a single email will change depending on which list it was sent via. Not only can this confuse simple-minded filtering recipes, it can also break threaded mail readers (people may end up reading the same message twice).
      • Cross-posting will make the Subject: line from hell problem more frequent. Imagine the following:
        RE: [LINUX-KERNEL] Re: [LINUX-SCSI] RE: [LINUX-KERNEL] Re: [LINUX-SCSI] RE: [LINUX-KERNEL] Re: [LINUX-SCSI] Critical security flaw in 2.666.0
        See? It just gets worse. Give it up, Subject: line modification is a bad idea.
      • The correct way to filter is to base your recipe on the X-Mailing-List: line, which should always have "linux-kernel@vger.kernel.org".
        An example procmail recipe would look like this:

        # Linux-kernel list
        :0: /var/lib/emacs/lock/!home!fred!mfilter!linux!kernel
        * ^X-Mailing-List:.*linux-kernel@vger\.kernel\.org
        /home/fred/mfilter/linux/kernel
        

        People subscribed to linux-kernel-digest@lists.us.dell.com, which uses GNU Mailman, may want to use something like this:

        # linux-kernel-digest
        :0
        * ^X-BeenThere: linux-kernel-digest@lists\.us\.dell\.com
        /home/fred/mfilter/linux/kernel-digest
        

        People using mailagent might try this in their .rules file (thanks to Martin Smith <martin@sharrow.demon.co.uk>):

        To CC: /linux-kernel@vger.kernel.org/
        { SPLIT -adi ~/Kernel }
        

        Similarly to procmail you can omit the mail folder from the split command. This causes the split messages to go back into the mailagent queue for further processing.

        Most mailers with filtering capabilities can be similarly configured. If not, then you can simply install procmail. If perchance you're running a damaged OS that can't filter properly, and there is no procmail port for it, then you should either upgrade, or accept that you won't be able to filter linux-kernel. Don't bother asking for a subject line modification.

      • If you really want to get the feel of a toy mailing list, you can write a procmail recipe which will modify the Subject: line.
        An example procmail recipe would look like this:

        # Linux-kernel list
        :0 f
        * ^X-Mailing-List:.*linux-kernel@vger\.kernel\.org
        | sed -e 's/^Subject: /Subject: [TOY-LINUX-KERNEL] /'
        

        Warning: if you do this, be careful to edit your Subject: line when replying to messages from the list, otherwise you risk being ignored or kill-filed.
20. Can we have a Reply-To: header automatically added to the list traffic?
    • (DW) Some mailing lists automatically add a Reply-To: header to the mails which go through them, forcing people to reply to the list, rather than replying personally to the original poster. This is a bad idea for many reasons which won't be listed here. See Chip Rosenthal's excellent summary Reply-To: Munging Considered Harmful for more explanation.
21. Can I post job offers/requests to the list?
    • (REG) Of course not! This is a technical development list, not a job exchange. You may find this site useful: http://www.hotlinuxjobs.com/
22. Why do I get bounces when I send private email to some people?
    • (REG) This could be for a variety of reasons, such as temporary problems with mail delivery. Your email may also be blocked (permanently rejected) by that individual or their ISP. This often happens if you send email from a machine or domain which is listed in the MAPS RBL, DUL and ORBS lists. These lists have been set up to protect people against spam. See http://www.mail-abuse.org/ and http://www.orbs.org/ for more information on these lists.
      NOTE that these lists aren't trying to block you personally, they are trying to block known spammers or spammer-friendly sites (RBL and ORBS), or uncontrolled dial-up users (DUL). If you are being blocked, it probably means you have the misfortune to be using an ISP that is not a good net citizen and thus has been added to the RBL or ORBS lists. In some cases, you may be blocked because your ISP has volunteered their dial-up IP address ranges to the DUL, in which case you should be using their approved mail relay rather than sending email out directly from your host.
      You must NOT post a message to the kernel list about this, as the people there cannot and will not help you. Nor should you use the list as a means of getting a message through to the individual you are trying to contact. This is not what the list is for.
      If you are intent on making a fool of yourself in public, follow the same path as too many others before you, and complain on the kernel list about how unfair it is that you are being blocked because your ISP is bad. Expect sympathy from some, flames from others and silence from most. The net gain will be that your mail will still be blocked by the anti-spam lists, many people will ignore you in future emails (because you've made a fool of yourself), and you may find yourself in the killfiles of some people (i.e. you personally are being blocked because some people are fed up with you and don't want to hear anything more from you).
      If you actually want your mails to no longer be blocked, get your ISP to clean up their act, or switch to a decent ISP. If you are required to use your ISP's mail relay, but it is crippled somehow, complain to your ISP or switch to one with competent staff.
      If your ISP is unresponsive and you don't have an alternative ISP you could switch to, you'll just have to accept that an increasing fraction of people will block your email (as more and more people subscribe to the anti-spam lists). There's no point in shouting at the people who are defending themselves against spam (no-one is obliged to receive any and all email), go pester the spammers instead.
23. Why don't you split the list, such as having one each for the development and stable series?
    • (REG, by "hacksaw") It's true that the lkml is a high traffic list and can be a lot to handle. However, splitting the list wouldn't help, since most developers would just subscribe to both lists. In fact, there would then be extra traffic, because of the number of issues that hit both the development and stable kernels, or even farther back!

Section 4 - "How do I" questions

1. How do I post a patch?
   • (ADB) I assume you made the patch following the general instructions found above. Now write a short post describing your patch, the version of the kernel it applies to, your tests, the feedback you would like to get, etc. This should fit in 10 lines. Attach your patch and a one line README file describing it very succinctly, and mentioning your name and email (either as two ASCII files or as a MIME encoded tarball). In the subject of your post, put: [PATCH] <the driver name or piece of code patched>, kernel <kernel version>. Send. Wait.
     The small README file insures that your patch will not start circulating around the net without people noticing your name. If you don't care about copyright and/or your patch is trivial, you can skip tarring the files, just gzip the patch file and attach it to your post.
   • (REG) Note that Linus does not read linux-kernel very much. So if you want him to see a patch, you will need to send it to him directly (say by Cc:ing him if you post to the list). Note that Linus likes to be able to read patches in plain ASCII, so anything that is uuencoded or MIMEd is likely to go straight to the bit-bucket. If because your patch is large you only send a URL, send a plain-text copy to Linus privately.
     Also note that Linus drops patches silently when he is too busy (which is always:-), so if you don't see it in the next kernel patch, send it again. Oh, and don't expect him to tell you he's applied the patch, either.
2. How do I capture an Oops?
   • (REG, quoting Keith Owens) If an Oops is recoverable then the text appears first in the kernel message buffer (/proc/kmsg). You can use the dmesg command to print the contents but most of the time klogd and syslogd will automatically capture the Oops and write it to your log files.

     Sometimes an Oops is so bad that the kernel is completely hung. When this occurs, almost anything that requires kernel support is also dead. In particular most interrupt driven subsystems are unusable, especially after the dreaded "Aiee, killing interrupt handler" message. Since most disk controllers use interrupts, no disk I/O is possible so the Oops does not get written to the log files. The same problem applies to logging over the network, most network cards require interrupt handlers.

     In a complete hang, you have three options.

     • Write the Oops down by hand from the screen and type it in after you have rebooted. This is the only option if you have not planned for a kernel hang.
     • If you plan ahead and install a serial console linked to another machine (read linux/Documentation/serial-console.txt) then you can capture the Oops report on the other machine. By far the easiest and most reliable option.
     • Since kernel 2.3.10 it has also been possible to use a parallel port line printer as a console. You can either attach a real printer, or another computer with EPP (Enhanced Parallel Port) support, which pretends to be a printer.
     • There have been patches on linux-kernel to save the log somewhere in hardware. Unfortunately these patches are very hardware specific. Search the l-k archives for "Oops assist", "OOPS output over reboot" and "KMSGDUMP". Most of these patches require that the keyboard still works and even that can be useless when the kernel hangs.
     Other operating systems can save the log even when the machine hangs, why doesn't Linux? Any OS that can save the log after a catastrophic kernel failure must do so without kernel support, that typically means using the underlying hardware. Alas the ix86 hardware does not provide enough support for this, in particular most BIOS will clear memory on reset, destroying any data in storage.
3. How do I post an Oops?
   • (ADB) Assuming you have found a genuine Oops (those are rare nowadays, but they happen), you should post the relevant portions of your system log, kernel configuration file and kernel symbol map, plus a description of your hardware and the circumstances under which the Oops occurred. Can the Oops be triggered by any particular method? Did it happen after you changed any part of your hardware configuration?
     Don't post your oops report before you have checked linux/Documentation/oops-tracing.txt, the relevant paragraphs in linux/README, the ksymoops C program in linux/scripts/ksymoops which has another README, and the gdb man and info pages (thanks to Paul Kimoto for this tip). These documents describe the basic procedure for kernel oops tracing. Good trace info makes it much easier to understand and solve apparently weird oopses.
   • (REG) Don't even bother posting an Oops if you haven't run it through ksymoops to decode the symbol addresses. The report will be ignored because it contains too little useful information.
     Make sure you copy the correct System.map file into /boot or into the modules directory, otherwise you will get incorrect results.
   • (REG, quoting "The Doctor What") There are some situations that make a kernel oops useless. The two most obvious are if your are overclocking your CPU or running VMWARE's vmmon. The reason is that overclocking can introduce random bit errors, while VMWARE's vmmon has the ability to (and does) change parts of the kernel. In both cases, data in the kernel, as reported by the oops, won't be useful.
4. I think I found a bug, how do I report it?
   • (ADB) A bug differs very slightly from an oops, actually. An oops is when the kernel detects that something has gone wrong. A bug is when something (in the kernel, presumably) doesn't behave the way it should, either with a driver or in some kernel algorithm. If you can detect this misbehaviour, you may or may not be getting an oops.
     Perhaps the most important step is to determine under which conditions this misbehaviour can be triggered, and whether it is reproducible.
5. What information should go in a bug report?
   • (ADB) Does it affect system security? Is it related to a specific driver/hardware configuration? Did you manage to identify the piece(s) of kernel code concerned? It really depends on the kind of bug you found.
   • (TYT) Please follow general good bug reporting guidelines: remember, the developers don't have access to your system, and they're not mind readers. Tell us which kernel version, and what your hardware is (if you're not sure, more detail is better than less). At the very least, tell us what processor and motherboard you have, how much memory, how many and what kind of disks (IDE, SCSI, etc.), what kind of disk controllers you have, what other expansion boards (specify whether they're PCI or ISA or some other bus). Also useful: what version of gcc and binutils were used to compile the kernel.
     Try to find a simple, reliable way to trigger the problem. Telling the developer that they have to set up some complicated application environment (especially if it involves some ghastly expensive proprietary software like SAP or Oracle :-) may cause the developer to hit the 'd' key and move on.
     In general, raw data is better than jumping to conclusions. If you want to give your guesses in your bug reports, they're of course welcome, but this is not a substitute for raw data. Many problems are not what they first seem. A hardware problem can masquerade as a VM problem. A device driver or VM problem can cause the filesystem code to notice a discrepancy, and flag a warning. Even if you're sure that the problem isn't a hardware problem, or some other theory that the developer advances, the scientific method demands that you do a test to rule these sorts of things out. Sometimes, you will get surprised.....
     If you get a kernel oops message, it's useless unless you give us the proper symbolic information. This used to mean sending relevant pieces out of System.map. Fortunately, with the latest syslogd/klogd, this is much simpler (check the man page of klogd to see if your version has this feature; if it doesn't, you should upgrade to the latest version, and probably to a modern distribution). Make sure that you have the System.map file installed the appropriate place so that klogd can find it (the standard search path is in the /boot, /, and /usr/src/linux directories).
     If the system oops and then dies without a chance for klogd to record the information into a syslog file, copy down the oops message exactly, and then use the ksymoops (see the man page) to get the symbolic information out. Remember, the raw numbers by themselves will generally not be useful.
     If you can, try to isolate the problem to a specific kernel version. Knowledge that it worked in version 2.2.17, as well as 2.3.0-test6, but it stopped working in 2.3.0-test7-pre1, is extremely helpful, and will save developers a lot of time. (If you're comfortable disecting patches, fell free, taking apart the individual file changes and try to isolate to a particular change.)
   • (REG) You did of course read REPORTING-BUGS from the kernel source tree first, didn't you?
6. I found a bug in an "old" version of the kernel, should I report it?
   • (CP) Only if it hasn't been fixed yet. The best thing to do is to try to repeat it with a new version of the kernel. If not, you have to figure out if it's been fixed yet. The kernel release announcements and patch descriptions from Jitterbug are also useful. Failing that, look for discussion of the bug in linux-kernel and check the patches between your kernel and the latest ones for relevant changes.
     If you can't find your bug mentioned, and you're not running a truly ancient kernel, posting a bug report is worthwhile. You can probably expect a request of the form "try it with the latest kernel" or "try it with this patch" in response. If there's a reason why you can't run the latest kernel (like it's your main dialin server and you don't want to mess with it), saying it in your original report will save some explaining later.
7. How do I compile the kernel?
   • (REG) See the Kernel HOWTO for some information. Also, there are people at http://www.kernelnewbies.org/ who are usually willing to help.
   • (William Stearns) The Buildkernel script walks you through an entire kernel build, including downloading the necessary files, patching the source, building the kernel and modules, installing the lot into lilo, and optionally building pcmcia-cs, cipe, and freeswan code for that kernel. Download and install either the tar or rpm version of the script, and run one of the following commands:

     buildkernel NEWESTSTABLE #To build the most recent stable kernel.
     buildkernel NEWESTBETA   #To build the most recent beta kernel.
     buildkernel 2.4.7        #If you know the version you wish to build.
     

8. How do I check if the running kernel is tainted?
   • (Kalin Kozhuharov) The short answer is:

     cat /proc/sys/kernel/tainted
     

     If it is "0", then the running kenrel is NOT tainted. Otherwise - it is tainted.
     See the Tainted kernel document from Novell/SUSE Linux for more information.

Section 5 - "Who's who" questions

1. Who is in charge here?
   • (ADB) Do you mean "Who takes decisions relative to the mailing list?" or do you mean "Who takes decisions relative to the Linux kernel"? If the former: there is relatively little to decide when it comes to the mailing list. Majordomo, once correctly setup, will manage the list in an autonomous fashion. In any case, you can always reach the Majordomo-owner for the list, if you have a very specific question about the list mechanism itself. When it comes to kernel development management and decision making, see the answer to Question 7.8 below.
2. Why don't we have a Linux Kernel Team page, same as there are for other projects?
   • (ADB) Perhaps because there is no Linux Kernel Team, per se. Also because so many people contributed to the Linux kernel that it would be a tough task to setup and maintain such a page. Finally, although this is not a rule, most Linux kernel contributors prefer to keep a low profile, for various reasons.
3. Why doesn't <any of the below> answer my mails? Isn't that rude?
   • (ADB) Probably because of sheer lack of time to answer each email that gets sent to them. What would you do if you got 1000 mails in your mailbox, from one day to the next? They don't mean to be rude, however.
     One hint: if you attach to your mail a genuinely useful piece of good quality code that you wrote, there are good chances that it will be answered (choose a good subject line, too). If you ask a dozen beginner's questions, the truth is, there are zero chances that you will get even the simplest reply pointing to some source of information.
     Aside from that, you may get "mail rejected" error messages if you try to contact some major contributors of the list. It is due to the spam filtering systems used by them. Please complain about it to your ISP and don't post to the list about spam !! .
   • (REG) Some people also have very aggressive mail filtering which rejects (non-list) messages from people they don't know, asking for a re-send with a password (this stops SPAM dead). If you mail to someone and receive such an automatic response, don't get upset. Remember, a person's mailbox is their personal property.
     Also, some people maintain "guru lists" and only read posts on linux-kernel by someone on their guru list, other people's posts go to /dev/null. This is done because there are too many questions asked on linux-kernel which shouldn't be (which is why people should read this FAQ first!), and people can't cope with the load. If you post to the list and want make sure a specific individual will see the message, Cc: that person.
4. Why do I get bounces when I send private email to some of these people?
   • (REG) Some people, like Alan Cox, bounce messages. Read this to find out why and what you can do about it.
5. Who is Matti Aarnio?
   • (MEA) He is principally a ZMailer hacker, and a co-postmaster of vger.kernel.org.
     Sometimes he finds also cycles to hack on the kernel, and you see some patches from him. (e.g. initial work on Large File Summit; files over 2G in size, was his)
6. Who is H. Peter Anvin?
7. Who is Donald Becker?
8. Who is Alan Cox?
   • (AC) Alan Cox supervises the 2.0.34/35/36 kernel releases, works on the Mac68K port, the SGI port, 2.0 networking, modular sound, video capture and helps collect up and sort patches to the kernel. He gets to do all this and sleep because the nice guys at Red Hat pay him to hack Linux.
9. Who is Richard E. Gooch?
   • (REG himself) "I've written various utilities and kernel patches which you can find here including the MTRR, devfs and fastpoll patches. My PhD in Computer Science was on the topic of Astronomical Visualization , which is my current research interest. This is what I work on when I don't get distracted by kernel hacking. See my home page to find out more about me."
10. Who is Paul Gortmaker?
    • (ADB, OK'ed by Paul) Paul has contributed various pieces of kernel code over the last few years, among other things the Real Time Clock driver. He is also the maintainer of the 8390 based network drivers (NE-2000, etc.), and wrote the Linux Ethernet HOWTO and the Boot-Prompt HOWTO.
11. Who is Mark Lord?
12. Who is Larry McVoy?
13. Who is David S. Miller?
    • (DSM) David Miller is mainly known for the porting work he has done, primarily for the 32-bit and 64-bit Sparc platforms although he has made significant contributions to the MIPS effort as well. He is also the current maintainer of the IP networking layer in the kernel and likes to address general performance and scalability problems all over as his time permits.
14. Who is Linus Torvalds?
15. Who is Theodore Y. T'so?
    • (TYTSO) Theodore Ts'o has over the years written, rewritten, or supported Posix Job Control, the high level tty driver, the serial driver, the ramdisk support, e2fsck/e2fsprogs, and other bits and pieces of code in and near the kernel. He is currently a member of the Technical Board of Linux International. His day job at MIT is concerned with Kerberos and other network security and I/T architecture issues. He is also a member of the Internet Engineering Task Force, where he serves as a member of the Security Area Directorate.
16. Who is Roger Wolff?
    • (REW himself) "I wrote the kmalloc that still drives linux-2.0.x. I wrote the Specialix and Olicom device drivers. I currently write Linux device drivers for a living. Contact me if you need one."

Other OS developers

• Who is Prof. Douglas Comer?
  • (Prof. Comer) Dr. Douglas Comer is a full professor of Computer Science at Purdue University, where he teaches courses on operating systems and computer networks. He has written numerous research papers and textbooks, and currently heads several networking research projects.
    He has been involved in TCP/IP and internetworking since the late 1970s, and is an internationally recognized authority. He designed and implemented X25NET and Cypress networks, and the Xinu operating system. He is director of the Internetworking Research Group at Purdue, editor of Software - Practice and Experience, and a former member of the Internet Architecture Board.
    Dr. Comer completed the original version of Xinu (and wrote "The Xinu approach" book) in 1979. Since then, Xinu has been expanded and ported to a wide variety of platforms, including: IBM PC, Macintosh, Digital Equipment Corporation VAX and DECStation 3100, Sun Microsystems Sun 2, Sun 3 and Sparcstations, and Intel Pentium. It has been used as the basis for many research projects. Furthermore, Xinu has been used as an embedded system in products by companies such as Motorola, Mitsubishi, Hewlett-Packard, and Lexmark. There is a full TCP/IP stack, and even the original version of Xinu (for the PDP-11) supported arbitrary processes and network I/O.
• Who is Richard M. Stallman?
  • (RMS) Richard Stallman is the founder of the GNU project, launched in 1984 to develop the free operating system GNU (an acronym for "GNU's Not Unix"), and thereby give computer users the freedom that most of them have lost. GNU is free software: everyone is free to copy it and redistribute it, as well as to make changes either large or small.
    Today, Linux-based variants of the GNU system, based on the kernel Linux developed by Linus Torvalds, are in widespread use. There are estimated to be over 10 million users of GNU/Linux systems today.
    Richard Stallman is the principal author of the GNU C Compiler, a portable optimizing compiler which was designed to support diverse architectures and multiple languages. The compiler now supports over 30 different architectures and 7 programming languages.
    Stallman also wrote the GNU symbolic debugger (GDB), GNU Emacs, and various other GNU programs.
    Stallman received the Grace Hopper Award from the Association for Computing Machinery for 1991 for his development of the first Emacs editor in the 1970s. In 1990 he was awarded a MacArthur Foundation fellowship, and in 1996 an honorary doctorate from the Royal Institute of Technology in Sweden. In 1998 he received the Electronic Frontier Foundation's Pioneer award along with Linus Torvalds.
• Who is Prof. Andrew Tanenbaum?
  • (Prof. Tanenbaum) Andrew S. Tanenbaum has an S.B. degree from MIT and a Ph.D. from the University of California at Berkeley. He is currently a Professor of Computer Science at the Vr