Perl in OpenBSD

Andrew Fresh

&u1F421;🐡 I'm andrew, so many email addresses for this talk s/andrew\K@cpan/afresh1/ GSG, I'll be hanging out at our booth. * not using BSD, but I have gotten one of the ops folks using it at home * I assume you know what perl is, but . . .

What is OpenBSD

A complete BSD licensed operating system

The only OS to have perl integrated into the base

Focus on correctness of code

Sane defaults, removal of knobs, things Just Work.

OpenSSH, LibreSSL, tmux, OpenNTPd, OpenBGPd, OpenOSPFd, ospf6d, OpenSMTPd, httpd, cwm, PF, relayd, carp, ifstated, OpenLDAP, OpenSNMPd, slowcgi, spamd, OpenIKEd, mandoc, mg, anoncvs, strlcpy, strlcat, strtonum, sudo

* I know at merlyn and tchrist are OpenBSD users already. * Only OS AFAIK * BSD in general and OpenBSD specifically (licence, history, goals) - best development platform, developer oriented - provide robust, secure, standards based software anyone can use for any purpose - security; crypto; fix problems quickly; - "Try to be the #1 most secure operating system." * It's awesome * knobs are for knobs * OpenBSD, conservative or cutting edge * so many cool things: OpenSSH, tmux, libReSSL, cwm, httpd https://twitter.com/lestrrat/status/608013726222938113 https://twitter.com/mwlauthor/status/608086709423767552

Preferred License 

Copyright (c) YYYY YOUR NAME HERE 


Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.


THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

My history

Worked at an ISP that needed updates

My friend recommended OpenBSD

For one thing, it came with perl. This was amazing.

Features seemed to show up as I needed them

* Worked at ISP starting in 1998 * Some good stories from working there. * friend recommended OpenBSD * Features like ospfd and bgpd * OpenBSD introduced me to perl in ~'99 * I probably didn't use perl until 5.8 * I'm now working as a perl developer * "so i arrived at openbsd. let's go back and look at the history of why perl was there".

Justification for Import

perl still fits the niche between C and shell

easily make changes without upstream approval

The main reasons

It's hard to work with strings safely in the shell

The FreeBSD pkg tools were horribly written in C

Why was it imported? * It fit niche between shell and C, and still does. * changes without consulting upstream * strings in shell scripts * freebsd pkg Not saying there's a connection, but tchrist & millert were neighbors

History

5.003 imported by downsj@ in 1996

Taken over by millert@ who imported

2010 he ran out of time, 5.14 was lost

Imported in 1996 by Jason Downs (rentrak) I didn't even know what perl was then, was working for a friend who was running an ISP on NT 3.51 Todd Miller had a pretty impressive run, he's moved on to other things What does that mean?

Integration

Installs as /usr/bin/perl

lives in /usr/src/gnu/usr.sbin/perl

Makefile.bsdwrapper ties it into the build system

Still a few customizations in hints/openbsd.sh

Pre-build unicore files, some archs can't take enough memory

Uses symlinks in obj/ to build in a read-only tree

perl is a part of Unix, so integration on that is not significatly different. In the most minimal supported install of OpenBSD, it's ~200M total the "gnu" there is somewhat misnamed, should be "non-BSD", easy division, 4-clause BSD and fewer restictions, or not. upstream for hints/openbsd.sh and one of the first patches I pushed upstream Was warned that getting things upstream is terribly, horrible, no good, very bad. Pre-built unicore because on some archectures there is no machine that can hold enough memory to build them. Pushed a speed-up patch for the symlink creation upstream

Notable uses of perl

usr.sbin/adduser

date: 1996/09/28 05:58:34; author: downsj
Perl adduser(8) from FreeBSD. in base

usr.bin/vi/build/recover

date: 1999/10/11 20:07:19; author: millert
"Rewrite in perl for safety and paranoia."
right away imported adduser * I'm actually in the middle of a rewrite of this. * It isn't terrible but certainly could be improved. Script to (safely) recover nvi edit sessions commit message++ "Rewrite in perl for safety and paranoia."

makewhatis

date: 2000/02/03 18:10:48; author: espie
Switch to perl: *much* faster (about ten times), *much* more flexible, simpler to understand (no more sed), and no temporary files.

used for 14 years until it was replaced with C in Ingo's great mandoc rewrite

date: 2014/04/18 10:00:48; author: schwarze
Switch to the new makewhatis(8)/apropos(1)/whatis(1) combo.

This is a good example of the development process. Use the best thing available at the time. You should definitely check out Ingo's work, he has some good talks online. A different perspective, you can use the newest `feature`s because you don't need backwards compat, but can't use things from the CPAN.

Security features

Uses the system MD5 implementation

Uses the system malloc

Uses arc4random unless you've asked for "reproducible" non-random

MD5, better, assembler, not really security system malloc, slower but much more secure Was some annoyance about the drand48 implementation not being easily overridable. (arc4random) * A Replacement Call 4 Random * not RC4, ChaCha20 * whole talks by Theo about it. * come to BSD Cons or YouTube

Back to me

I had some free time and wanted to contribute back.

Helped schwarze@ rewrite security(8)

Asked how I could help, mentioned I liked perl

It was suggested I get a new version ready for import.

I think to scare me away, I did not catch on.

5.16.3 got cleaned up and ready, sthen@ imported it.

libexec/security -- ingo -- what is this script? A couple years later, had time for other things, asked what. They said, "can you get a new version of perl ready for import?" I think they really meant "go away" Not exactly the perl I meant, but so far it has worked out. The first version that was my fault was 5.16.3 with help from sthen@ What did I do to get it ready?

Local patches

I've separated out 47 different patches

Mostly for integration with the build system

More annoyances that need patches

I turned the single diff between "stock perl" and "OpenBSD perl" into a set of patch files that makes it much easier to work with. I actually maintain them on github. nearly half of which I know I need to find time to push upstream The only non-core module we add to the distribution is OpenBSD::MkTemp although are others installed. It is an interface to the C to mkstemp() and mkstemps() and mkdtemp() used by the packaging system. * perl doesn't use NGROUPS_MAX * `make; make` builds perl twice (.PHONY) * this list is not complete

What did I do then?

Feb of 2014, I got afresh1@openbsd.org

I imported 5.18.2 in March of 2014

Followed by 5.20.1 in November of 2014

Recently imported 5.20.2 in April of 2015

I got an account just so I could attempt to break the tree by importing 5.18.2. "are you sure you want the 1?" The 1 actually confuses the commit stats script, I don't get any credit there. got my account during lock, so although the update was ready, the tree was not OpenBSD release every 6 months in May/November. 3 months big changes, 3 months stability, 2 weeks of no changes OpenBSD 5.8 == 20 years of on-time OpenBSD releases I was ahead of the game for 5.20 and timing hit with the lock, so had 5.18.2 for 5.6 but 5.20.1 for 5.7 and 5.20.2 for 5.8 5.20 did give me some trouble on vax, a compiler bug. We'll see in the future, probably .1 releases around unlock for November release and .2 around May.

Upstreamed Patches

mostly cleanups for Configure script so far There was worry about getting the change for system malloc upstream It has fortunately not been too bad getting patches upstream it may have been in the past, but all it took was some adjustments for coding style. when I submitted the symlink patch, there was surprise anyone used it. Tony Cook fixed it for old awk on Solaris 11 -- old awk story!! And a few module patches that haven't made it in yet. commit e61ccd964839921b5f9ec9d1e9d648c4b24926a9 Author: Tony Cook <tony@develop-help.com> Date: Wed Jun 11 14:52:26 2014 +1000 Andrew Fresh is now a perl author

Maintaining patches

17 architectures, including VAX

Before import:

Next up, back on dev releases and trying to push patches upstream. 17 arches. Most of my issues with getting 5.20 imported ended up being compiler bugs on vax. But, tracking those down made vax more stable for everything. The testing is fairly basic, I get it ready and run the new version as system perl on my laptop and build system. That means building all my own packages and testing out the package system. Get a porter to run a full bulk build of all ports and update any ports that have failures or figure out why. Import - I have a script that I keep tuning to try to avoid screwing it up. Now that it's in . . .

What's it used for

libexec/security

regress/

libtool

pkg_*

dpb - Distributed Package Build

pkg-config

security: The first code my name was actually on in OpenBSD. Actually may still be the only code that has my copyright, need to fix that. regress: (smokers) just haven't been able to convince my lazy brain to be that productive. libtool: AFAIK the only non GNU libtool implementation. Written in perl by espie@ One of the next places I want to look, to try to get the GNU libtool regress test to run pkg_* + dpb: Also written by espie@ Where he seems to spend most of his time He has good talks about this, especially making dpb work pkg-config provides paths to *packages* that need them reimplementation of freedesktop.org version

Installing Packages

pkg_add p5-Mojolicious

It's then relatively easy to reproduce your setup

    pkg_info -mq > ~/pkg_list # save
    pkg_add  -z -l ~/pkg_list # restore

OpenBSD 5.7 has (fairly) current
Catalyst, Mojolicious, Dancer2 and DBIx::Class

pkg_add -- that's it, installing packages is recommended Rebuilding a system (can also save automatically installed packages) What's the difference between the base system, ports, and packages BSD != Mac, Linux or Windows. pick stable or -current and don't mix * only security fixes to stable * -current is *supposed* to always work The end goal is packages but what is the ports tree?

Perl Ports

You can install from the CPAN

It's much easier to maintain OpenBSD with a port

It's really easy to do

I have a tool I use that does most of the heavy lifting

There's a GSOC project to make it better

Giannis Tsaraias

Why make a port? So easy! * can install from CPAN * maintenance much easer with a port Google Summer of Code project!! Giannis Tsaraias <yannis@cryptolab.net> University of Crete in Greece

Creating Perl Ports

Perl port Makefiles are usually 5 lines plus the dependency list.

I use
https://github.com/afresh1/openbsd-module-ports

What's in a port? create Makefile + pkg/DESCR, generate distinfo + pkg/PLIST I use my script, but I'm hoping for good changes from the GSOC

Sample Makefile

# $OpenBSD$
COMMENT =     generate S5 slideshow from POD source
DISTNAME =    Pod-S5-0.09
CATEGORIES =  productivity

# Same as perl and CC BY-SA
PERMIT_PACKAGE_CDROM =  Yes
MODULES =      cpan
RUN_DEPENDS =  devel/p5-Pod-Tree \
               devel/p5-Syntax-Highlight-Engine-Kate
.include <bsd.port.mk>

Usually ~5 lines Makefiles can get pretty crazy with flavors, multi-packages, testing setup, and custom installation routines. flavors; multi-packages; spinning up mysql/postgres instances; (DBIx::Class) you can run "make test" and it runs the perl tests. The goal eventually is to make the ports tree a fancy CPAN client for the simple cases.

My Future Plans

Keeping perl up-to-date

lots of patches left for upstreaming

keeping my ports up to date

finishing up my adduser(8) rewrite

looking at libtool

My TODO just goes on-and-on

Up-to-date perl; upstream patches; update ports; adduser rewrite; libtool and lots, lots more. If you're interested in OpenBSD ...

More Information

http://www.openbsd.org

mailing lists

man pages

OpenBSD documentation is pretty amazing

openbsd.org; FAQ; man pages; mailing lists "RTFM" is often to get rid of you, it's actually a good idea in OpenBSD.

Thank you!

afresh1@openbsd.org
andrew@cpan.org
andrew@afresh1.com
afresh@grantstreet.com
@AFresh1 on twitter

BSD Pizza Night (Portland, OR)
last Thursday of every month

http://www.openbsd.org/donations.html
http://www.openbsdfoundation.org
http://www.openbsdstore.com

http://cvs.afresh1.com/~andrew/talks

Thank you Grantstreet Group is hiring BSD Pizza Night If you use any of the things I mentioned: ssh, Mac OS X, iOS (iPhone/iPad), Android, Linux, the word Hackathon, the Internet Donations / Buy CDs Questions? http://mdocml.bsd.lv/openbsd_projects.html