KLOGD(8)           Linux System Administration           KLOGD(8)

NAME
       klogd - kernel log daemon.

SYNOPSIS
       klogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [
       -p ] [ -s ] [ -k fname ] [ -v ] [ -x ]

DESCRIPTION
       klogd is a system daemon which intercepts and  logs  Linux
       kernel messages.

OPTIONS
       -c n   Sets  the  default log level of console messages to
              n.

       -d     Enable debugging mode.  This will generate LOTS  of
              output to stderr.

       -f file
              Log  messages to the specified filename rather than
              to the syslog facility.

       -i -I  Signal the currently executing klogd daemon.   Both
              of  these switches control the loading/reloading of
              symbol information.  The -i switch signals the dae-
              mon  to  reload  the kernel module symbols.  The -I
              switch signals for a reload of both the static ker-
              nel symbols and the kernel module symbols.

       -n     Avoid auto-backgrounding. This is needed especially
              if the klogd is started and controlled by  init(8).

       -o     Execute  in  'one-shot' mode.  This causes klogd to
              read and log all the messages that are found in the
              kernel  message  buffers.   After a single read and
              log cycle the daemon exits.

       -p     Enable paranoia.  This option controls  when  klogd
              loads  kernel  module  symbol information.  Setting
              this switch causes klogd to load the kernel  module
              symbol  information  whenever  an  Oops  string  is
              detected in the kernel message stream.

       -s     Force klogd to use the system call interface to the
              kernel message buffers.

       -k file
              Use the specified file as the source of kernel sym-
              bol information.

       -v     Print version and exit.

       -x     Omits EIP translation and there  doesn't  read  the
              System.map.

OVERVIEW
       The functionality of klogd has been typically incorporated
       into other versions of syslogd but this seems to be a poor
       place for it.  In the modern Linux kernel a number of ker-
       nel messaging issues such as sourcing, prioritization  and
       resolution  of kernel addresses must be addressed.  Incor-
       porating kernel logging into a separate process  offers  a
       cleaner separation of services.

       In  Linux  there  are  two potential sources of kernel log
       information:  the  /proc  file  system  and  the   syscall
       (sys_syslog)  interface,  although ultimately they are one
       and the same.   Klogd  is  designed  to  choose  whichever
       source  of  information  is the most appropriate.  It does
       this by first checking for the presence of a mounted /proc
       file  system. If this is found the /proc/kmsg file is used
       as the source of kernel log information. If the proc  file
       system  is  not mounted klogd uses a system call to obtain
       kernel messages.  The command line switch (-s) can be used
       to  force  klogd  to  use the system call interface as its
       messaging source.

       If kernel messages are directed through the syslogd daemon
       the  klogd  daemon,  as of version 1.1, has the ability to
       properly prioritize kernel messages. Prioritization of the
       kernel  messages  was added to it at approximately version
       0.99pl13 of the kernel. The raw kernel messages are of the
       form:

              <[0-7]>Something said by the kernel.

       The  priority of the kernel message is encoded as a single
       numeric digit enclosed inside the <>  pair.   The  defini-
       tions  of these values is given in the kernel include file
       kernel.h.  When a message is received from the kernel  the
       klogd  daemon  reads  this  priority level and assigns the
       appropriate priority level to the syslog message.  If file
       output  (-f)  is  used the prioritization sequence is left
       pre-pended to the kernel message.

       The klogd daemon also allows the ability to alter the pre-
       sentation  of kernel messages to the system console.  Con-
       sequent with the prioritization of kernel messages was the
       inclusion  of default messaging levels for the kernel.  In
       a stock kernel the the default console log level is set to
       7.   Any  messages with a priority level numerically lower
       than 7 (higher priority) appear on the console.

       Messages of priority level 7 are considered to be  'debug'
       messages  and  will  thus not appear on the console.  Many
       administrators, particularly in a multi-user  environment,
       prefer  that  all  kernel messages be handled by klogd and
       either directed to a file or to the syslogd daemon.   This
       prevents  'nuisance'  messages such as line printer out of
       paper or disk change detected from cluttering the console.

       By  default  the  klogd  daemon  executes a system call to
       inhibit all kernel messages (except for panics) from being
       displayed  on  the  console.  The -c switch can be used to
       alter this behavior.  The argument given to the -c  switch
       specifies  the  priority  level  of messages which will be
       directed to the console.  Note that messages of a priority
       value  LOWER than the indicated number will be directed to
       the console.

              For example, to have the kernel  display  all  mes-
              sages with a priority level of 3 (KERN_ERR) or more
              severe the following command would be executed:

                   klogd -c 4

       The definitions of the numeric values for kernel  messages
       are  given  in the file kernel.h which can be found in the
       /usr/include/linux directory if  the  kernel  sources  are
       installed.  These values parallel the syslog priority val-
       ues which are defined in the file syslog.h  found  in  the
       /usr/include/sys sub-directory.

       The klogd daemon can also be used in a 'one-shot' mode for
       reading the kernel message  buffers.   One  shot  mode  is
       selected  by specifying the -o switch on the command line.
       Output will be directed to either the syslogd daemon or to
       an alternate file specified by the -f switch.

              For  example, to read all the kernel messages after
              a system boot and record  them  in  a  file  called
              krnl.msg the following command would be given.

                   klogd -o -f ./krnl.msg

KERNEL ADDRESS RESOLUTION
       If  the  kernel detects an internal error condition a gen-
       eral protection fault will be triggered.  As part  of  the
       GPF  handling  procedure  the  kernel  prints out a status
       report indicating the state of the processor at  the  time
       of  the  fault.  Included in this display are the contents
       of the microprocessor's registers,  the  contents  of  the
       kernel  stack  and  a tracing of what functions were being
       executed at the time of the fault.

       This information is  EXTREMELY  IMPORTANT  in  determining
       what  caused the internal error condition.  The difficulty
       comes when a kernel developer  attempts  to  analyze  this
       information.   The  raw numeric information present in the
       protection fault printout is of very  little  use  to  the
       developers.   This is due to the fact that kernels are not
       identical and the addresses of variable locations or func-
       tions  will  not  be the same in all kernels.  In order to
       correctly diagnose the cause of failure a kernel developer
       needs  to  know what specific kernel functions or variable
       locations were involved in the error.

       As part of the kernel compilation  process  a  listing  is
       created which specified the address locations of important
       variables and function in the kernel being compiled.  This
       listing is saved in a file called System.map in the top of
       the kernel directory source tree.  Using  this  listing  a
       kernel developer can determine exactly what the kernel was
       doing when the error condition occurred.

       The process of resolving the numeric  addresses  from  the
       protection fault printout can be done manually or by using
       the ksymoops program  which  is  included  in  the  kernel
       sources.

       As  a  convenience  klogd  will  attempt to resolve kernel
       numeric addresses to their symbolic forms if a kernel sym-
       bol  table is available at execution time.  A symbol table
       may be specified by using the -k  switch  on  the  command
       line.   If  a  symbol file is not explicitly specified the
       following filenames will be tried:

       /boot/System.map
       /System.map
       /usr/src/linux/System.map

       Version information is supplied in the system maps  as  of
       kernel 1.3.43.  This version information is used to direct
       an intelligent search of the list of symbol tables.   This
       feature  is useful since it provides support for both pro-
       duction and experimental kernels.

       For example a production kernel  may  have  its  map  file
       stored  in  /boot/System.map.   If an experimental or test
       kernel is compiled with  the  sources  in  the  'standard'
       location of /usr/src/linux the system map will be found in
       /usr/src/linux/System.map.  When klogd  starts  under  the
       experimental  kernel  the  map in /boot/System.map will be
       bypassed in favor of the map in /usr/src/linux/System.map.

       Modern kernels as of 1.3.43 properly format important ker-
       nel addresses so that they will be recognized  and  trans-
       lated  by  klogd.   Earlier  kernels require a source code
       patch be applied to the kernel  sources.   This  patch  is
       supplied with the sysklogd sources.

       The  process  of analyzing kernel protections faults works
       very well with a static kernel.   Additional  difficulties
       are  encountered  when attempting to diagnose errors which
       occur in loadable kernel modules.  Loadable kernel modules
       are used to implement kernel functionality in a form which
       can be loaded or unloaded at will.  The  use  of  loadable
       modules is useful from a debugging standpoint and can also
       be useful in decreasing the amount of memory required by a
       kernel.

       The  difficulty with diagnosing errors in loadable modules
       is due to the dynamic nature of the kernel modules.   When
       a module is loaded the kernel will allocate memory to hold
       the module, when the module is unloaded this  memory  will
       be returned back to the kernel.  This dynamic memory allo-
       cation makes it impossible to produce  a  map  file  which
       details  the  addresses of the variable and functions in a
       kernel loadable module.  Without this location map  it  is
       not possible for a kernel developer to determine what went
       wrong if a protection fault involves a kernel module.

       klogd has support for dealing with the problem of diagnos-
       ing protection faults in kernel loadable modules.  At pro-
       gram start time or in response to a signal the daemon will
       interrogate the kernel for a listing of all modules loaded
       and the addresses in memory they are loaded at.   Individ-
       ual  modules  can also register the locations of important
       functions when the module is  loaded.   The  addresses  of
       these  exported  symbols  are  also determined during this
       interrogation process.

       When a protection fault occurs an attempt will be made  to
       resolve kernel addresses from the static symbol table.  If
       this fails the symbols from the currently  loaded  modules
       are  examined  in an attempt to resolve the addresses.  At
       the very minimum this allows klogd to indicate which load-
       able  module was responsible for generating the protection
       fault.  Additional information may  be  available  if  the
       module  developer  chose to export symbol information from
       the module.

       Proper and accurate resolution of addresses in kernel mod-
       ules  requires  that klogd be informed whenever the kernel
       module status changes.  The -i and -I switches can be used
       to  signal  the  currently  executing  daemon  that symbol
       information be reloaded.  Of  most  importance  to  proper
       resolution  of module symbols is the -i switch.  Each time
       a kernel module is loaded or removed from the  kernel  the
       following command should be executed:

       klogd -i

       The  -p switch can also be used to insure that module sym-
       bol information is up  to  date.   This  switch  instructs
       klogd  to  reload the module symbol information whenever a
       protection fault is  detected.   Caution  should  be  used
       before  invoking the program in 'paranoid' mode.  The sta-
       bility of the kernel  and  the  operating  environment  is
       always  under  question  when  a  protection fault occurs.
       Since the klogd daemon must execute system calls in  order
       to  read the module symbol information there is the possi-
       bility that the system may be too unstable to capture use-
       ful  information.   A much better policy is to insure that
       klogd is updated whenever a module is loaded or  unloaded.
       Having  uptodate  symbol  information loaded increases the
       probability of properly resolving a protection fault if it
       should occur.

       Included in the sysklogd source distribution is a patch to
       the modules-2.0.0 package which allows the  insmod,  rmmod
       and modprobe utilities to automatically signal klogd when-
       ever a module is inserted  or  removed  from  the  kernel.
       Using  this  patch will insure that the symbol information
       maintained in klogd is always consistent with the  current
       kernel state.

SIGNAL HANDLING
       The  klogd  will respond to eight signals: SIGHUP, SIGINT,
       SIGKILL, SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2  and  SIGCONT.
       The SIGINT, SIGKILL, SIGTERM and SIGHUP signals will cause
       the daemon to close its kernel log sources  and  terminate
       gracefully.

       The SIGTSTP and SIGCONT signals are used to start and stop
       kernel logging. Upon receipt of a SIGTSTP signal the  dae-
       mon  will  close its log sources and spin in an idle loop.
       Subsequent receipt of a SIGCONT signal will cause the dae-
       mon  to  go  through  its  initialization sequence and re-
       choose an input source.  Using SIGSTOP and SIGCONT in com-
       bination  the  kernel  log  input can be re-chosen without
       stopping and restarting the daemon.  For  example  if  the
       /proc  file  system is to be un-mounted the following com-
       mand sequence should be used:

            # kill -TSTP pid
            # umount /proc
            # kill -CONT pid

       Notations will be made in the system  logs  with  LOG_INFO
       priority documenting the start/stop of logging.

       The SIGUSR1 and SIGUSR2 signals are used to initiate load-
       ing/reloading of kernel symbol  information.   Receipt  of
       the SIGUSR1 signal will cause the kernel module symbols to
       be reloaded.  Signaling the daemon with SIGUSR2 will cause
       both  the static kernel symbols and the kernel module sym-
       bols to be reloaded.

       Provided that the System.map file is placed in  an  appro-
       priate  location  the signal of generally greatest useful-
       ness is the SIGUSR1 signal.  This signal is designed to be
       used   to  signal  the  daemon  when  kernel  modules  are
       loaded/unloaded.  Sending this signal to the daemon  after
       a kernel module state change will insure that proper reso-
       lution of symbols will occur if a protection fault  occurs
       in the address space occupied by a kernel module.

FILES
       /proc/kmsg
              One Source for kernel messages klogd
       /var/run/klogd.pid
              The file containing the process id of klogd
       /boot/System.map, /System.map, /usr/src/linux/System.map
              Default locations for kernel system maps.

BUGS
       Probably numerous.  Well formed context diffs appreciated.

AUTHOR
       The  klogd  was   originally   written   by   Steve   Lord
       (lord@cray.com), Greg Wettstein made major improvements.

       Dr. Greg Wettstein (greg@wind.enjellic.com)
       Enjellic Systems Development

       Oncology Research Divsion Computing Facility
       Roger Maris Cancer Center
       Fargo, ND 58122

Version 1.3              24 November 1995                       1