[BACK]Return to index.html CVS log [TXT][DIR] Up to [local] / nagios / check_hw_sensors

Diff for /nagios/check_hw_sensors/index.html between version 1.2 and 1.6

version 1.2, 2006/05/04 02:31:29 version 1.6, 2007/01/06 03:27:53
Line 1 
Line 1 
 <html><head><title>Nagios Check - check_hw_sensors</title></head>  <html><head><title>Nagios Check - check_hw_sensors</title></head>
     <body>      <body>
         <h1>Nagios Check - check_hw_sensors</h1>          <h1>Nagios Check - check_hw_sensors</h1>
         <p>check_hw_sensors plugin for Nagios monitors sysctl hw.sensors on OpenBSD</p>          <p>check_hw_sensors plugin for Nagios monitors sysctl hw.sensors on OpenBSD</p>
         <p>With the new sensor framework in OpenBSD 3.9, I wanted to be able to monitor the new hw.sensors from <a href='http://www.nagios.org/'>Nagios</a> and this is what I have.  It is currently a work in progress although it does seem to work just fine.  The documentation is a bit thin and I don't know how reliable it is.  I would be happy to accept patches.  Send them to <a href='mailto:andrew+nagios@rraz.net'>andrew+nagios@rraz.net</a>.  I know the docs aren't as good as I would like, so if there are places that need clarification, please let me know!</p>          <p>With the new sensor framework in OpenBSD 3.9, I wanted to be able to monitor the new hw.sensors from <a href='http://www.nagios.org/'>Nagios</a> and this is what I have.  The documentation is a bit thin and I don't know how reliable it is.  I would be happy to accept patches.  Send them to <a href='mailto:andrew+nagios@rraz.net'>andrew+nagios@rraz.net</a>.  I know the docs aren't as good as I would like, so if there are places that need clarification, please let me know!</p>
         <p>What I think is really kewl about this plugin is that it can use the same sensorsd.conf as sensorsd.  That means that they can be easily kept in sync.  But, since Nagios supports both warning and critical alerts, it turned out really handy that sensorsd ignores any additional capabilities in the file.  The addtional capabilities check_hw_sensors supports are described below.  If you have an /etc/sensorsd.conf with the checks you want, it can be run as simply as 'check_hw_sensors -f'.</p>                  <p>New in this release is support for the new 2 level sensors in OpenBSD 4.0-current.  They seem way more better, and I may change some stuff to only support that version after I get all my machines moved to 4.1</p>
         <p><center><b><a href='check_hw_sensors-1.14.tar.gz'>Download the current version here</a></b></center></p>                  <p>It has the ability to check the sensors that report their status.  Since many sensors support this, it can make the size of your sensorsd.conf much smaller.  For example, check_hw_sensors will automatically check these two sensors:
         <h4>                  <ul>
           Please be sure to support the <a href="http://www.openbsd.org">OpenBSD</a>                          <li>hw.sensors.76=esm0, Fan 4, 3629 RPM, OK</li>
           project by purchasing                          <li>hw.sensors.77=esm0, Fan 5, 0 RPM, CRITICAL</li>
           <a href="http://www.openbsd.org/items.html">CDs</a>,                  </ul>
           <a href="http://www.openbsd.org/tshirts.html">T-shirts</a>, or making a                  It will report the status listed to Nagios.  For 76, it would be OK, for 77 it would be CRITICAL.  You don't need to put anything in a config file to support those.</p>
           <a href="http://www.openbsd.org/donations.html">donation</a>.          <p>What I think is really kewl about this plugin is that it can use the same sensorsd.conf as sensorsd.  That means that they can be easily kept in sync.  But, since Nagios supports both warning and critical alerts, it turned out really handy that sensorsd ignores any additional capabilities in the file.  The addtional capabilities check_hw_sensors supports are described below.  If you have an /etc/sensorsd.conf with the checks you want, it can be run as simply as 'check_hw_sensors -f'.  If you only want to check the sensors that report their status, you can ever run it as just 'check_hw_sensors'.</p>
           <br />          <p>TODO:
           These finances ensure that OpenBSD will continue to exist, and          <ul>
           will remain <a href="http://www.openbsd.org/policy.html">free</a>              <li>need real documentation.</li>
           for everyone to use and reuse as they see fit.                          <li>the RANGE using the colon to separate probably screws up the getcap of sensorsd.conf, so it should probably get replaced with a dash or somesuch</li>
         </h4>          </ul>
         <pre>          </p>
     check_hw_sensors (-f [&lt;FILENAME&gt;]|(-s &lt;hw.sensors id&gt; -w limit -c limit))          <p><center><b><a href='check_hw_sensors-1.22.tar.gz'>Download the current version here</a></b></center></p>
           <h4>
 Usage:            Please be sure to support the <a href="http://www.openbsd.org">OpenBSD</a>
     -f, --filename=FILE            project by purchasing
         FILE to load checks from (defaults to /etc/sensorsd.conf)            <a href="http://www.openbsd.org/items.html">CDs</a>,
     -s, --sensor=ID            <a href="http://www.openbsd.org/tshirts.html">T-shirts</a>, or making a
         ID of a single sensor.  "-s 0" means hw.sensors.0.            <a href="http://www.openbsd.org/donations.html">donation</a>.
     -w, --warning=RANGE or single ENTRY            <br />
         Exit with WARNING status if outside of RANGE or if != ENTRY            These finances ensure that OpenBSD will continue to exist, and
     -c, --critical=RANGE or single ENTRY            will remain <a href="http://www.openbsd.org/policy.html">free</a>
         Exit with CRITICAL status if outside of RANGE or if != ENTRY            for everyone to use and reuse as they see fit.
           </h4>
     -h (--help)       usage help          <pre>
         </pre>      check_hw_sensors [-i] (-f [&lt;FILENAME&gt;]|(-s &lt;hw.sensors id&gt; [-w limit] [-c limit]))
         <p>FILE is in the same format as <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd.conf'>sensorsd.conf(5)</a> plus some additional entries.  These additional entries in the file are ignored by <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd'>sensorsd(8)</a>.  </p>  
   Usage:
         <p>check_hw_sensors understands the following entries:<br>      -i, --ignore-status
         low, high, crit, warn, crit.low, crit.high, warn.low, warn.high</p>          Don't check the status of sensors that report it.
       -f, --filename=FILE
         <p>An ENTRY depends on the type.  The descriptions in <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd.conf'>sensorsd.conf(5)</a>          FILE to load checks from (defaults to /etc/sensorsd.conf)
         can be used when appropriate, or you can use the following:      -s, --sensor=ID
           ID of a single sensor.  "-s 0" means hw.sensors.0.
         <ul>      -w, --warning=RANGE or single ENTRY
             <li>volts_dc, fanrpm or raw<br>          Exit with WARNING status if outside of RANGE or if != ENTRY
             Anything that includes digits.      -c, --critical=RANGE or single ENTRY
             Both the value of the check and the value of the sensor          Exit with CRITICAL status if outside of RANGE or if != ENTRY
             response that are not either a digit or period are stripped          </pre>
             and then the two resultant values are compared.</li>          <p>FILE is in the same format as <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd.conf'>sensorsd.conf(5)</a> plus some additional entries.  These additional entries in the file are ignored by <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd'>sensorsd(8)</a>.  </p>
   
             <li>temp<br>          <p>check_hw_sensors understands the following entries:<br>
             Can be as above, but if the entry has an F in it,                  low, high, crit, warn, crit.low, crit.high, warn.low, warn.high,
             it compares farenheit, otherwise it uses celcius.</li>                  ignore, status</p>
   
             <li>indicator or drive<br>          <p>An ENTRY depends on the type.  The descriptions in <a href='http://www.openbsd.org/cgi-bin/man.cgi?query=sensorsd.conf'>sensorsd.conf(5)</a>
             does a case sensitive match of each          can be used when appropriate, or you can use the following:
             entry in the comma separated list and if it does not match  
             any of the entries, it matches the status.  Does not support any low or high entries.</li>          <ul>
         </ul>              <li>fanrpm, volts_dc, amps, watthour, amphour, integer (raw), percent, lux or timedelta<br>
               Anything that includes digits.
         <p>The entries 'crit' or 'warn' (or the -c or -w on the command line)              Both the value of the check and the value of the sensor
         may be a RANGE or a comma separated list of acceptable values.              response that are not either a digit or period are stripped
         The comma separated list of values contains a list of things that              and then the two resultant values are compared.</li>
         will NOT cause the status.  This is possibly counterintuitive, but  
         you are more likely to know good values than bad values.</p>              <li>temp<br>
               Can be as above, but if the entry has an F in it,
         <p>A RANGE is a low ENTRY and a high ENTRY separated by a colon (:).              it compares farenheit, otherwise it uses celcius.</li>
         It can also be low: or :high with the other side left blank to only  
         make the single check..<p>              <li>indicator or drive<br>
               does a case sensitive match of each
         <p>check_hw_sensors (nagios-plugins 1.4.2) 1.14<br>              entry in the comma separated list and if it does not match
         The nagios plugins come with ABSOLUTELY NO WARRANTY. You may redistribute              any of the entries, it sets the status.</li>
         copies of the plugins under the terms of the GNU General Public License.          </ul>
         For more information about these matters, see the file named COPYING.</p>  
         <h3>Example Sensorsd.conf</h3>          <p>The entries 'crit' or 'warn' (or the -c or -w on the command line)
         <pre>          may be a RANGE or a comma separated list of acceptable values.
 # $OpenBSD$          The comma separated list of values contains a list of things that
 # $ RedRiver: sensorsd.conf,v 1.1 2006/05/03 21:48:42 andrew Exp $          will NOT cause the status.  This is possibly counterintuitive, but
           you are more likely to know good values than bad values.</p>
 #  
 # Sample sensorsd.conf file. See sensorsd.conf(5) for details.          <p>A RANGE is a low ENTRY and a high ENTRY separated by a colon (:).
 # This one has examples for use with nagios check_hw_sensors          It can also be low: or :high with the other side left blank to only
 # Actual sensors on a 2450 are below.          make the single check..<p>
 #  
                   <p>An entry marked "ignore" will cause that sensor to be skipped.
 # hw.sensors.0=esm0, Motherboard, raw, 0                  Generally used with status checking of all sensors to ignore sensors you
 # hw.sensors.1=esm0, CPU 1, OK, temp, 28.00 degC / 82.40 degF                  don't care about or that report incorrectly.</p>
 hw.sensors.1:high=50C:warn.high=40C  
                   <p>If you are using --ignore-status, you can still check the status of
 # hw.sensors.2=esm0, CPU 2, OK, temp, 30.00 degC / 86.00 degF                  individual sensors with a status entry.</p>
 hw.sensors.2:high=50C:warn.high=40C  
           <p>check_hw_sensors (nagios-plugins 1.4.2) 1.22<br>
 # hw.sensors.3=esm0, Mainboard, OK, temp, 21.50 degC / 70.70 degF          The nagios plugins come with ABSOLUTELY NO WARRANTY. You may redistribute
 hw.sensors.3:high=40C:warn.high=30C          copies of the plugins under the terms of the GNU General Public License.
           For more information about these matters, see the file named COPYING.</p>
 # hw.sensors.4=esm0, CPU 1 Core, OK, volts_dc, 1.69 V  
 hw.sensors.4:high=1.85V:warn.high=1.8V:low=1.60V:warn.low=1.65V          <h3>Example sensorsd.conf</h3>
           <pre>
 # hw.sensors.5=esm0, CPU 2 Core, OK, volts_dc, 1.70 V  # hw.sensors.acpibat0.volt0=7.40 V DC, (voltage), OK
 hw.sensors.5:high=1.85V:warn.high=1.8V:low=1.60V:warn.low=1.65V  # hw.sensors.acpibat0.volt1=8.30 V DC, (current voltage), OK
   # hw.sensors.acpibat0.watthour0=57.72 Wh, (last full capacity)
 # hw.sensors.6=esm0, Motherboard +5V, OK, volts_dc, 4.95 V  # hw.sensors.acpibat0.watthour1=0.00 Wh, (warning capacity)
 hw.sensors.6:high=5.1V:warn.high=5.05V:low=4.90V:warn.low=4.85V  # hw.sensors.acpibat0.watthour2=0.12 Wh, (low capacity)
   # hw.sensors.acpibat0.watthour3=57.72 Wh, (remaining capacity)
 # hw.sensors.7=esm0, Motherboard +12V, OK, volts_dc, 11.94 V  hw.sensors.acpibat0.watthour3:warn.low=50 Wh:crit.low=30 Wh
 hw.sensors.7:high=12.15V:warn.high=12.1V:low=11.8V:warn.low=11.85V  # hw.sensors.acpibat0.raw0=2, (battery charging), OK
   # hw.sensors.acpibat0.raw1=99, (rate)
 # hw.sensors.8=esm0, Motherboard +3.3V, OK, volts_dc, 3.27 V  # hw.sensors.acpiac0.indicator0=On, (power supply)
 hw.sensors.8:high=3.5V:warn.high=3.4V:low=3.15V:warn.low=3.2V  hw.sensors.acpiac0.indicator0:crit=On
   # hw.sensors.acpitz0.temp0=62.95 degC, (zone temperature)
 # hw.sensors.9=esm0, Motherboard +2.5V, OK, volts_dc, 2.48 V  hw.sensors.acpitz0.temp0:warn.high=65 degC:crit.high=75 degC
 hw.sensors.9:high=2.75V:warn.high=2.6V:low=2.25V:warn.low=2.4V          </pre>
                   <h3>CVS log for check_hw_sensors</h3>
 # hw.sensors.10=esm0, Motherboard GTL Term, OK, volts_dc, 1.49 V                  <pre>
 hw.sensors.10:high=1.75V:warn.high=1.6V:low=1.25V:warn.low=1.4V  RCS file: /cvs/scripts/Admin scripts/check_hw_sensors/check_hw_sensors,v
   Working file: check_hw_sensors
 # hw.sensors.11=esm0, Motherboard Battery, OK, volts_dc, 2.93 V  head: 1.22
 hw.sensors.11:high=3.1V:warn.high=3.05V:low=2.75V:warn.low=2.8V  branch:
   locks: strict
 # hw.sensors.12=esm0, Chassis Intrusion, indicator, Off  access list:
 hw.sensors.12:crit=Off:warn=Off  symbolic names:
   keyword substitution: kv
 # hw.sensors.13=esm0, Fan 1, OK, fanrpm, 3526 RPM  total revisions: 22;    selected revisions: 22
 hw.sensors.13:low=3000:warn.low=3250  description:
   ----------------------------
 # hw.sensors.14=esm0, Fan 2, OK, fanrpm, 3569 RPM  revision 1.22
 hw.sensors.14:low=3000:warn.low=3250  date: 2007/01/06 03:16:41;  author: andrew;  state: Exp;  lines: +11 -4
   Support the new dual level sensors
 # hw.sensors.15=esm0, Fan 3, OK, fanrpm, 3563 RPM  ----------------------------
 hw.sensors.15:low=3000:warn.low=3250  revision 1.21
   date: 2006/12/05 16:26:27;  author: andrew;  state: Exp;  lines: +5 -5
 # hw.sensors.16=esm0, Backplane, raw, 0  new better example for 4.0
 # hw.sensors.17=esm0, Backplane Top, OK, temp, 14.50 degC / 58.10 degF  and fix the s/drive\s// from the data, not the type
 hw.sensors.17:high=35C:warn.high=25C  ----------------------------
   revision 1.20
 # hw.sensors.18=esm0, Backplane Bottom, OK, temp, 22.00 degC / 71.60 degF  date: 2006/12/05 00:17:47;  author: andrew;  state: Exp;  lines: +35 -26
 hw.sensors.18:high=40C:warn.high=30C  Match sensors differently depending on OS Version from the Config module.
   
 # hw.sensors.19=esm0, Backplane +5V, OK, volts_dc, 4.97 V  Also support checks on the other sensor types and document that.
 hw.sensors.19:high=5.1V:warn.high=5.05V:low=4.90V:warn.low=4.85V  
   and refactor the way I return a $sensor-&gt;{'status'} from ~10 lines to 1.
 # hw.sensors.20=esm0, Backplane SCSI A Connected, indicator, On  ----------------------------
 hw.sensors.20:crit=On:warn=On  revision 1.19
   date: 2006/12/04 23:33:53;  author: andrew;  state: Exp;  lines: +8 -3
 # hw.sensors.21=esm0, Backplane SCSI A External, OK, volts_dc, 4.70 V  add a regex for the 'percent' type of sensor
 hw.sensors.21:high=5.1V:warn.high=5.05V:low=4.60V:warn.low=4.65V  ----------------------------
   revision 1.18
 # hw.sensors.22=esm0, Backplane SCSI B Connected, indicator, Off  date: 2006/12/02 02:15:17;  author: andrew;  state: Exp;  lines: +74 -9
 hw.sensors.22:crit=Off:warn=Off  fix it for the output from OpenBSD 4.0
   ----------------------------
 # hw.sensors.23=esm0, Drive 0, drive, online  revision 1.17
 hw.sensors.23:crit=online,empty:warn=online  date: 2006/10/25 23:30:23;  author: andrew;  state: Exp;  lines: +4 -7
   get the docs up to match the new version
 # hw.sensors.24=esm0, Drive 1, drive, online  ----------------------------
 hw.sensors.24:crit=online,empty:warn=online  revision 1.16
   date: 2006/10/25 18:36:46;  author: andrew;  state: Exp;  lines: +4 -4
 # hw.sensors.25=esm0, Drive 2, drive, empty  Stuff in CVS should output nagios format
 hw.sensors.25:crit=online,empty:warn=online,empty  ----------------------------
   revision 1.15
 # hw.sensors.26=esm0, Drive 3, drive, empty  date: 2006/10/25 18:35:59;  author: andrew;  state: Exp;  lines: +73 -46
 hw.sensors.26:crit=online,empty:warn=online,empty  add support for the status as reported by the sensors.  it is teh r0x0r!
   ----------------------------
 # hw.sensors.27=esm0, Backplane Control 2, raw, 17  revision 1.14
 # hw.sensors.28=esm0, Backplane +3.3V, OK, volts_dc, 3.28 V  date: 2006/05/04 01:30:29;  author: andrew;  state: Exp;  lines: +6 -4
 hw.sensors.28:high=3.5V:warn.high=3.4V:low=3.2V:warn.low=3.25V  I thought I checked this in already
   ----------------------------
 # hw.sensors.29=safte0, temp0, OK, temp, 14.44 degC / 58.00 degF  revision 1.13
 hw.sensors.29:high=35C:warn.high=25C  date: 2006/05/03 22:16:42;  author: andrew;  state: Exp;  lines: +7 -6
   Some more fixing of the help
 # hw.sensors.30=safte0, temp1, OK, temp, 22.22 degC / 72.00 degF  ----------------------------
 hw.sensors.30:high=40C:warn.high=30C  revision 1.12
         </pre>  date: 2006/05/03 21:54:43;  author: andrew;  state: Exp;  lines: +51 -30
         <h3>Example output from sysctl hw.sensors from that box</h3>  Get the help output cleaned up.  Still not 100% what I want, but so far so good.
         <pre>  ----------------------------
 hw.sensors.0=esm0, Motherboard, raw, 0  revision 1.11
 hw.sensors.1=esm0, CPU 1, OK, temp, 30.00 degC / 86.00 degF  date: 2006/05/03 20:01:09;  author: holligan;  state: Exp;  lines: +4 -4
 hw.sensors.2=esm0, CPU 2, OK, temp, 31.00 degC / 87.80 degF  updated and clarified help
 hw.sensors.3=esm0, Mainboard, OK, temp, 19.50 degC / 67.10 degF  ----------------------------
 hw.sensors.4=esm0, CPU 1 Core, OK, volts_dc, 1.69 V  revision 1.10
 hw.sensors.5=esm0, CPU 2 Core, OK, volts_dc, 1.70 V  date: 2006/05/03 03:31:22;  author: andrew;  state: Exp;  lines: +27 -42
 hw.sensors.6=esm0, Motherboard +5V, OK, volts_dc, 4.95 V  A bunch of cleanup and some kewl refactoring into loops.
 hw.sensors.7=esm0, Motherboard +12V, OK, volts_dc, 11.93 V  
 hw.sensors.8=esm0, Motherboard +3.3V, OK, volts_dc, 3.27 V  Still need to find a way to refactor the checks that are so similar!
 hw.sensors.9=esm0, Motherboard +2.5V, OK, volts_dc, 2.48 V  ----------------------------
 hw.sensors.10=esm0, Motherboard GTL Term, OK, volts_dc, 1.49 V  revision 1.9
 hw.sensors.11=esm0, Motherboard Battery, OK, volts_dc, 2.94 V  date: 2006/05/03 02:26:47;  author: andrew;  state: Exp;  lines: +8 -8
 hw.sensors.12=esm0, Chassis Intrusion, indicator, Off  It now doesn't do nonexistent checks (for some stuff anyway)
 hw.sensors.13=esm0, Fan 1, OK, fanrpm, 3514 RPM  Also changed the &lt;br /&gt; to &lt;br&gt;.  Not valid XHTML whatever, but it does get stripped before getting sent to my pager.
 hw.sensors.14=esm0, Fan 2, OK, fanrpm, 3582 RPM  ----------------------------
 hw.sensors.15=esm0, Fan 3, OK, fanrpm, 3570 RPM  revision 1.8
 hw.sensors.16=esm0, Backplane, raw, 0  date: 2006/05/02 21:23:29;  author: andrew;  state: Exp;  lines: +6 -16
 hw.sensors.17=esm0, Backplane Top, OK, temp, 14.50 degC / 58.10 degF  Better looking output for the web page.
 hw.sensors.18=esm0, Backplane Bottom, OK, temp, 22.50 degC / 72.50 degF  ----------------------------
 hw.sensors.19=esm0, Backplane +5V, OK, volts_dc, 4.97 V  revision 1.7
 hw.sensors.20=esm0, Backplane SCSI A Connected, indicator, On  date: 2006/05/02 20:03:53;  author: andrew;  state: Exp;  lines: +5 -5
 hw.sensors.21=esm0, Backplane SCSI A External, OK, volts_dc, 4.70 V  oops, that's an array ref!
 hw.sensors.22=esm0, Backplane SCSI B Connected, indicator, Off  ----------------------------
 hw.sensors.23=esm0, Drive 0, drive, online  revision 1.6
 hw.sensors.24=esm0, Drive 1, drive, online  date: 2006/05/02 19:59:47;  author: andrew;  state: Exp;  lines: +7 -5
 hw.sensors.25=esm0, Drive 2, drive, empty  Better output for the OK checks
 hw.sensors.26=esm0, Drive 3, drive, empty  ----------------------------
 hw.sensors.27=esm0, Backplane Control 2, raw, 17  revision 1.5
 hw.sensors.28=esm0, Backplane +3.3V, OK, volts_dc, 3.28 V  date: 2006/05/02 19:49:29;  author: andrew;  state: Exp;  lines: +19 -12
 hw.sensors.29=safte0, temp0, OK, temp, 15.00 degC / 59.00 degF  Only show details for things other than OK, cuZ we are limited in the amount of data we can return :-(
 hw.sensors.30=safte0, temp1, OK, temp, 22.78 degC / 73.00 degF  ----------------------------
         </pre>  revision 1.4
         <p>Andrew Fresh &lt;<a href='mailto:andrew@mad-techies.org'>andrew@mad-techies.org</a>&gt;</p>  date: 2006/05/02 15:54:42;  author: andrew;  state: Exp;  lines: +44 -21
         <p><small>$RedRiver: index.html,v 1.1 2006/05/03 23:50:17 andrew Exp $</small></p>  Some cleanup, as well as making it output a single line like nagios supposedly likes.
     </body>  ----------------------------
 </html>  revision 1.3
   date: 2006/05/02 01:39:23;  author: andrew;  state: Exp;  lines: +3 -3
   fix the help getopts.
   ----------------------------
   revision 1.2
   date: 2006/05/02 01:29:33;  author: andrew;  state: Exp;  lines: +427 -11
   Adding the sensors from one of the routers, cuZ there were a lot and I can use it for testing.
   
   Also, now the check_hw_sensors now seems to be OK.  I need to put it on a few machines and set up the checks now.  If it works for the rest of the week, I can clean it up and mebbe put it on teh interweb and post to undeadly.
   ----------------------------
   revision 1.1
   date: 2006/05/01 18:11:23;  author: andrew;  state: Exp;
   add this so I can check it out on a box for testing
   =============================================================================
                   </pre>
           <p>Andrew Fresh &lt;<a href='mailto:andrew@mad-techies.org'>andrew@mad-techies.org</a>&gt;</p>
           <p><small>$RedRiver: index.html,v 1.5 2006/12/05 16:44:14 andrew Exp $</small></p>
       </body>
   </html>
   

Legend:
Removed from v.1.2  
changed lines
  Added in v.1.6

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>