Perl in OpenBSD
- @AFresh1 on Twitter
I'm andrew, so many email addresses for this talk
GSG, I'll be hanging out at our booth.
* not using BSD, but I have gotten one of the ops folks using it at home
* I assume you know what perl is, but . . .
What is OpenBSD
A complete BSD licensed operating system
The only OS to have perl integrated into the base
Focus on correctness of code
Sane defaults, removal of knobs, things Just Work.
OpenSSH, LibreSSL, tmux, OpenNTPd, OpenBGPd, OpenOSPFd, ospf6d, OpenSMTPd,
httpd, cwm, PF, relayd, carp, ifstated, OpenLDAP, OpenSNMPd, slowcgi, spamd,
OpenIKEd, mandoc, mg, anoncvs, strlcpy, strlcat, strtonum, sudo
* I know at merlyn and tchrist are OpenBSD users already.
* Only OS AFAIK
* BSD in general and OpenBSD specifically
(licence, history, goals)
- best development platform, developer oriented
- provide robust, secure, standards based software anyone can use for any purpose
- security; crypto; fix problems quickly;
- "Try to be the #1 most secure operating system."
* It's awesome
* knobs are for knobs
* OpenBSD, conservative or cutting edge
* so many cool things: OpenSSH, tmux, libReSSL, cwm, httpd
Copyright (c) YYYY YOUR NAME HERE
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Worked at an ISP that needed updates
- I ended up in charge of anything tech
My friend recommended OpenBSD
- which fit the budget I was given.
- and changed my life.
For one thing, it came with perl.
This was amazing.
Features seemed to show up as I needed them
* Worked at ISP starting in 1998
* Some good stories from working there.
* friend recommended OpenBSD
* Features like ospfd and bgpd
* OpenBSD introduced me to perl in ~'99
* I probably didn't use perl until 5.8
* I'm now working as a perl developer
* "so i arrived at openbsd.
let's go back and look at the history of why perl was there".
Justification for Import
perl still fits the niche between C and shell
easily make changes without upstream approval
The main reasons
It's hard to work with strings safely in the shell
The FreeBSD pkg tools were horribly written in C
Why was it imported?
* It fit niche between shell and C, and still does.
* changes without consulting upstream
* strings in shell scripts
* freebsd pkg
Not saying there's a connection, but tchrist & millert were neighbors
5.003 imported by downsj@ in 1996
Taken over by millert@ who imported
- 5.004_04, 5.005_03,
- 5.6.0, 5.6.1,
- 5.8.0, 5.8.2, 5.8.3, 5.8.5, 5.8.6, 5.8.8,
- 5.10.0, 5.10.1,
- and 5.12.2
2010 he ran out of time, 5.14 was lost
Imported in 1996 by Jason Downs
I didn't even know what perl was then,
was working for a friend who was running an ISP on NT 3.51
Todd Miller had a pretty impressive run, he's moved on to other things
What does that mean?
Makefile.bsdwrapper ties it into the build system
Still a few customizations in hints/openbsd.sh
Pre-build unicore files, some archs can't take enough memory
Uses symlinks in
obj/ to build in a read-only tree
perl is a part of Unix, so integration on that is not significatly different.
In the most minimal supported install of OpenBSD, it's ~200M total
the "gnu" there is somewhat misnamed, should be "non-BSD",
easy division, 4-clause BSD and fewer restictions, or not.
upstream for hints/openbsd.sh and one of the first patches I pushed upstream
Was warned that getting things upstream is terribly, horrible, no good, very bad.
Pre-built unicore because on some archectures there is no machine that can
hold enough memory to build them.
Pushed a speed-up patch for the symlink creation upstream
Notable uses of perl
date: 1996/09/28 05:58:34; author: downsj
Perl adduser(8) from FreeBSD. in base
date: 1999/10/11 20:07:19; author: millert
"Rewrite in perl for safety and paranoia."
right away imported adduser
* I'm actually in the middle of a rewrite of this.
* It isn't terrible but certainly could be improved.
Script to (safely) recover nvi edit sessions
"Rewrite in perl for safety and paranoia."
date: 2000/02/03 18:10:48; author: espie
Switch to perl: *much* faster (about ten times), *much* more flexible,
simpler to understand (no more sed), and no temporary files.
used for 14 years until it was replaced with C in Ingo's great mandoc rewrite
date: 2014/04/18 10:00:48; author: schwarze
Switch to the new makewhatis(8)/apropos(1)/whatis(1) combo.
This is a good example of the development process.
Use the best thing available at the time.
You should definitely check out Ingo's work, he has some good talks online.
A different perspective, you can use the newest `feature`s because you don't
need backwards compat, but can't use things from the CPAN.
Uses the system MD5 implementation
Uses the system malloc
- slow, security features, randomization
Uses arc4random unless you've asked for "reproducible" non-random
MD5, better, assembler, not really security
system malloc, slower but much more secure
Was some annoyance about the drand48 implementation not being easily overridable.
* A Replacement Call 4 Random
* not RC4, ChaCha20
* whole talks by Theo about it.
* come to BSD Cons or YouTube
Back to me
I had some free time and wanted to contribute back.
Helped schwarze@ rewrite
Asked how I could help, mentioned I liked perl
It was suggested I get a new version ready for import.
I think to scare me away, I did not catch on.
5.16.3 got cleaned up and ready, sthen@ imported it.
libexec/security -- ingo -- what is this script?
A couple years later, had time for other things, asked what.
They said, "can you get a new version of perl ready for import?"
I think they really meant "go away"
Not exactly the perl I meant, but so far it has worked out.
The first version that was my fault was 5.16.3 with help from sthen@
What did I do to get it ready?
I've separated out 47 different patches
Mostly for integration with the build system
More annoyances that need patches
I turned the single diff between "stock perl" and "OpenBSD perl" into a set of patch files that makes it much easier to work with. I actually maintain them on github.
nearly half of which I know I need to find time to push upstream
The only non-core module we add to the distribution is OpenBSD::MkTemp
although are others installed. It is an interface to the C to mkstemp() and mkstemps() and mkdtemp() used by the packaging system.
* perl doesn't use NGROUPS_MAX
* `make; make` builds perl twice (.PHONY)
* this list is not complete
What did I do then?
Feb of 2014, I got email@example.com
I imported 5.18.2 in March of 2014
Followed by 5.20.1 in November of 2014
Recently imported 5.20.2 in April of 2015
I got an account just so I could attempt to break the tree by importing 5.18.2.
"are you sure you want the 1?"
The 1 actually confuses the commit stats script, I don't get any credit there.
got my account during lock, so although the update was ready, the tree was not
OpenBSD release every 6 months in May/November.
3 months big changes, 3 months stability, 2 weeks of no changes
OpenBSD 5.8 == 20 years of on-time OpenBSD releases
I was ahead of the game for 5.20 and timing hit with the lock, so
had 5.18.2 for 5.6 but 5.20.1 for 5.7 and 5.20.2 for 5.8
5.20 did give me some trouble on vax, a compiler bug.
We'll see in the future, probably .1 releases around unlock for November release and .2 around May.
- Correct architectures and versions for hints
Sun Jun 1 11:00:39 2014 -0700
- Default to using OpenBSD's malloc
Sat May 31 10:30:40 2014 -0700
- international currency formatting (POSIX.1-2008)
Wed Jul 9 03:30:00 2014 -0700
- Better handling of symlink creation
Sat May 31 15:37:52 2014 -0700
mostly cleanups for Configure script so far
There was worry about getting the change for system malloc upstream
It has fortunately not been too bad getting patches upstream
it may have been in the past, but all it took was some adjustments
for coding style.
when I submitted the symlink patch, there was surprise anyone used it.
Tony Cook fixed it for old awk on Solaris 11
-- old awk story!!
And a few module patches that haven't made it in yet.
Author: Tony Cook <firstname.lastname@example.org>
Date: Wed Jun 11 14:52:26 2014 +1000
Andrew Fresh is now a perl author
17 architectures, including VAX
- update all patches, removing upstreamed
- look for regressions on any architectures
- bulk ports build of all ports
- running it on my system for "a while"
- test import against a local cvs tree copy
Next up, back on dev releases and trying to push patches upstream.
Most of my issues with getting 5.20 imported ended up being compiler bugs on vax. But, tracking those down made vax more stable for everything.
The testing is fairly basic, I get it ready and run the new version as system perl on my laptop and build system. That means building all my own packages and testing out the package system. Get a porter to run a full bulk build of all ports and update any ports that have failures or figure out why.
Import - I have a script that I keep tuning to try to avoid screwing it up.
Now that it's in . . .
What's it used for
dpb - Distributed Package Build
security: The first code my name was actually on in OpenBSD.
Actually may still be the only code that has my copyright, need to fix that.
regress: (smokers) just haven't been able to convince my lazy brain to be that productive.
libtool: AFAIK the only non GNU libtool implementation. Written in perl by espie@
One of the next places I want to look, to try to get the GNU libtool regress test to run
pkg_* + dpb: Also written by espie@
Where he seems to spend most of his time
He has good talks about this, especially making dpb work
pkg-config provides paths to *packages* that need them
reimplementation of freedesktop.org version
It's then relatively easy to reproduce your setup
pkg_info -mq > ~/pkg_list # save
pkg_add -z -l ~/pkg_list # restore
OpenBSD 5.7 has (fairly) current
Catalyst, Mojolicious, Dancer2 and DBIx::Class
pkg_add -- that's it, installing packages is recommended
Rebuilding a system (can also save automatically installed packages)
What's the difference between the base system, ports, and packages
BSD != Mac, Linux or Windows.
pick stable or -current and don't mix
* only security fixes to stable
* -current is *supposed* to always work
The end goal is packages but what is the ports tree?
You can install from the CPAN
It's much easier to maintain OpenBSD with a port
It's really easy to do
I have a tool I use that does most of the heavy lifting
There's a GSOC project to make it better
Why make a port? So easy!
* can install from CPAN
* maintenance much easer with a port
Google Summer of Code project!!
Giannis Tsaraias <email@example.com>
University of Crete in Greece
Creating Perl Ports
- Create Makefile and pkg/DESCR
- generate distinfo and pkg/PLIST
- and you're done
Perl port Makefiles are usually 5 lines plus the dependency list.
What's in a port?
create Makefile + pkg/DESCR, generate distinfo + pkg/PLIST
I use my script, but I'm hoping for good changes from the GSOC
COMMENT = generate S5 slideshow from POD source
DISTNAME = Pod-S5-0.09
CATEGORIES = productivity
# Same as perl and CC BY-SA
PERMIT_PACKAGE_CDROM = Yes
MODULES = cpan
RUN_DEPENDS = devel/p5-Pod-Tree \
Usually ~5 lines
Makefiles can get pretty crazy with flavors, multi-packages, testing setup, and custom installation routines.
flavors; multi-packages; spinning up mysql/postgres instances; (DBIx::Class)
you can run "make test" and it runs the perl tests.
The goal eventually is to make the ports tree a fancy CPAN client for the simple cases.
My Future Plans
Keeping perl up-to-date
lots of patches left for upstreaming
keeping my ports up to date
finishing up my
looking at libtool
My TODO just goes on-and-on
Up-to-date perl; upstream patches; update ports; adduser rewrite; libtool
and lots, lots more.
If you're interested in OpenBSD ...
- FAQ and man pages are there
OpenBSD documentation is pretty amazing
openbsd.org; FAQ; man pages; mailing lists
"RTFM" is often to get rid of you, it's actually a good idea in OpenBSD.
@AFresh1 on twitter
BSD Pizza Night (Portland, OR)
last Thursday of every month
Grantstreet Group is hiring
BSD Pizza Night
If you use any of the things I mentioned:
ssh, Mac OS X, iOS (iPhone/iPad), Android, Linux, the word Hackathon, the Internet
Donations / Buy CDs