Of ports and men

There has been some discussion lately about if and how to "revamp" the ports system to make it more usable by general users, which was in part started by a post on PostgreSQL.org. Unfortunately there has been very little feedback from users themselves - which is probably a mistake, but also - there was very little feedback from the population (not a particularily small one) that is the cross-section of users and developers. Some ideas were presented, but at the end it all started revolving around banding the gaps and smaller improvements that will, I think, be practically invisible to the end-users.

I'd like to present some of my ideas here.

The biggest single idea I was hoping to get traction is the necessity of introducing a "stable" ports branch, which doesn't have to include all ports but for starts some more-than-minimal subset (e.g. the "FAPP" set). The main reasons for this are:

  1. Introducing ABI stability between the ports - so if some popular library changes (like libjpeg, libpng, glib), hundreds of ports on a working production system don't have to be recompiled. (in the defense of the current system - this recompiling works and is automated)
  2. Introducing binary packages and faster patch time - because maintaining binary packages where everything changes after such an ABI breakage is next to useless.

Unfortunately, the general feeling is that this cannot be done because of infrastructure limitations and because of port maintainers' time - they would be hard pressed to maintain separate sets of ports (the current "HEAD" branch and one or more "stable" branches). I see the infrastructure limitations can be serious but these are merely technical problems - they can be solved automatically one way or the other. The second problem - new demands on ports maintainers' time - is more significant.

I'm not a big ports maintainer - I semi-maintain only two ports - but I don't think it would be particularily hard to maintain a stable branch of the ports tree because of one simple reason: the porting work is already done and "maintaining" the stable branch mostly revolves around updating the ports with newer distfiles (source files), where applicable. Of course there are edge cases - such as that for security reasons some ports (shared libraries mostly) simply *must* break ABI, but I think this could be solved by at least "borrowing" patches and ideas from similar systems that have such a "stable packages" idea successfuly implemented (e.g. most of the big Linux distributions).

To test this, and to prove (mostly to myself) it would be doable, I've proposed a Google Summer of Code project. This is probably the last time I'd be eligible for it. Here's what I proposed:

A frequently requested FreeBSD feature is a more stable ports tree and a binary package distribution mechanism. This proposal aims to offer at least an experimental implementation of both, in a way that offers user a subset of available ports (from the ports tree) in an ABI-stable way which will also be maintained for at least the period of 12 months from the SoC project start, together with their binary packages and a rsync-based distribution method for both. The proposed technologies for building this infrastructure include reliance on existing FreeBSD/unix tools and libraries, with Lua as the glue language (the specifics can be determined later, before the SoC project starts).

The results of this project would be twofold: an infrastructure that could be used for this purpose in the future (with the possibility of import into the base system) if accepted by the community of developers, and the documented experience, made available via a blog, a wiki or similar communication or documentation infrastructure, of the good and bad sides of creating and using such a system. Since the results of this project will be made public ASAP, and it would be a fairly high-profile project, it is expected that users will participate by giving feedback.

Project Description:
A selection of packages will be made from the current ports tree (the everchanging ports' "HEAD"), which will be maintained as close as possible to an an "overlay tree" for the ports tree, with the distinction that the overlay tree will be self-contained in terms of dependencies and will be maintained in an ABI-stable way. The primary goal of maintaining this "stable" tree is to provide users with patches to the ported applications without causing ABI or shared libs breakage, which are currently very visible with the volatile nature of the ports tree. For the sake of possible future development, this "stable" tree will currently be marked as intended for the FreeBSD 8.x series (future possible versions will probably introduce ABI breakage but will be marked for a future version of FreeBSD). The secondary goal of this proposal is to create a robust method of distributing binary packages, possibly by using binary diffs, which will speed up distribution and patching. The binary packages will be built on the earliest supported FreeBSD release from the series.

In no case will there be attempts to circumvent, replace or disable the usual ports tree and binary packages. Full compatibility (with regards to the ports and build infrastructure) between the "stable" tree and the regular ports tree will be maintained. The "stable" tree ports will be updated from the regular tree ports as long as there are no ABI or shared libs inconsistencies between them. Unless an unforseen situation arises, binary packages will be the regular binary packages, compatible with pkg_add et al.

This "stable" ports tree will be maintained for a time period which effectively includes 9-10 months after the SoC project officially ends.

Technical Details:
I propose that the selection of ports to maintain in an ABI-stable way consists of (or a superset of): apache22, php5-*, perl, python, sqlite3, postgresql84-*, mysql55-* and their dependencies. The starting points of these ports will be maintained as they are/were when the stable tree "branches" from the regular ports tree. It is possible that some build options (make config) will differ from the ports' defaults for the binary packages, in which case they will be documented and maintained. As is now, users can choose to follow binary packages or build ports from source, with reasonable compatibility between them. It is outside the scope of this project to try to split ports which build or install multiple entities (development libraries, binaries, other resources) into separate ports, or do any such "surgery" on existing ports. Unless the project's scope changes (which would require public discussion and consensus), the system's port database format (/var/db/pkg et al) will not be modified.

A single user-accessible utility will be created to manage the entire client side of the infrastructure, including updating the "stable" ports tree, installing, removing and updating ports and packages. The project/maintainer side might include multiple utilities, as needed.

Hosting for the projects' results (ports, packages, etc.) will be either in the FreeBSD infrastructure or (at least during the initial SoC timeframe) provided by me.

Benefits for FreeBSD:
Possibly the greatest benefits from the FreeBSD project will be the experience and feedback in creating and maintaining this experimental infrastructure, including the evaluation of how hard it is to maintain such infrastructure (and the stable branch), recorded for the purpose of being used by the developers. The entire infrastructure will be created and maintained with the possibility of its inclusion in the FreeBSD base system, if the developer community agrees on it and sufficient port maintainers sign up for the effort.

Of course, absolute stability is not possible - even the systems with very rigid rules (like Debian) occasionally produce bad "stable" packages - we're all human. To err is human, but to have a VCS history of past code to slide to is what computers are for.

#1 Re: Of ports and men

Added on 2010-04-11T12:24 by Andreas Nilsson

Very interesting. The ports-system could use some improvements...


I'd love to be able to see what ports will be installed as deps to the one I'm currently thinking of installing. I like the idea of having a 'wrapper' round the ports-system to make it easier to deal with.


One more drastic idea is perhaps to try and unfiy the BSDs handling of third party applications, i.e. swith to pkgsrc.

#2 Re: Of ports and men

Added on 2010-04-11T13:12 by Andrew Pantyukhin

Ivan, any effort towards the ports system will be dearly appreciated, but be ready to spend over 90% of time in politics, rather than technical matters. History shows, only very tiny, evolutionary and almost invisible improvements have a chance to survive and blossom in ports' infrastructure.

My opinion is, since the ports system is so behemothly slow to move, it's a perfect target for extensive and deep research. Of course, there's not much chance anyone is going to fund it.

Like Andreas noticed, joining forces with pkgsrc, OpenBSD, Gentoo, Arch, Nix and others could provide the necessary momentum, but unlike src devs, key ports people have never shown much interest for cooperation with other systems. Internal bikesheds seem enough.


Good luck!

#3 Re: Of ports and men

Added on 2010-04-11T16:24 by Michael Moll

I would love to see a complete switch to pkgsrc, both sides would profit from that!

#4 Re: Of ports and men

Added on 2010-04-11T17:36 by Ivan Voras

pkgsrc is smaller than ports - that will never happen. OTOH nothing is stopping people from using pkgsrc on FreeBSD.

#5 Re: Of ports and men

Added on 2010-04-11T17:46 by Justin Sherrill

Having a quarterly "stable" release of pkgsrc has actually reduced my labor, and I imagine would do the same for ports. 

I'm trying to do full binary builds for DragonFly (i386 and x86_64, current and development release) which is a lot of time-consuming building to track.  Having a relatively stable branch to track means major updates are confined to about every 3 months, rather than "at any time and in any frequency", as it is now if you have updates to libjpeg, perl, php, etc.

#6 Re: Of ports and men

Added on 2010-04-11T17:54 by Ivan Voras

What does "stable" mean in the context of pkgsrc? Do minor updates and bugfixes still go into the stable branch or is it simply a frozen snapshot of the state at a point in time? If the second, I think it doesn't really help anyone. E.g. a PHP release (X.Y) might have point releases (X.Y.z) more than quarterly, and since it's such a big source of security holes, it's necessary to have it updated ASAP.

#7 Re: Of ports and men

Added on 2010-04-11T20:46 by Michael Moll

> pkgsrc is smaller than ports

That's right, of course the missing ports/packages would have to be ported over first...

> nothing is stopping people from using pkgsrc on FreeBSD.

pkgsrc works mostly well on FreeBSD but only very few people use this combination, so at present it would always take more time to get FreeBSD specific patches and the like into pkgsrc. I don't know if e.g. X.org from pkgsrc would work as-is on FreeBSD

> Do minor updates and bugfixes still go into the stable branch or is it simply a frozen snapshot of the state at a point in time?

Security updates and other highly critical updates are pulled into the stable branch also.

#8 Re: Of ports and men

Added on 2010-04-12T10:38 by lissyara

It's good idea

#9 Re: Of ports and men

Added on 2010-04-13T06:36 by Jacob Myers

If I may add my 2c here, a good improvement would be greater standardisation of knobs and being able to disable the libdialog menu in favour of defining what knobs you want ahead of time in another file (/etc/knobs.conf?). Something like this:





Not sure if that's feasible, though.

#10 Re: Of ports and men

Added on 2010-04-17T18:45 by Francisco Reyes

Doesn't the PBI system in PC-BSD already does what the article is trying to do? Although I don't use PC-BSD, if I recall how the PBI system works it encapsulates all needed libraries and executables for the PBI so that it is self contained and independant of other ports or libraries.

#11 PBI's

Added on 2010-04-17T19:15 by Ivan Voras

Sort of - but while it's ok for user applications like OpenOffice it scales badly for server-side stuff. How do you package Perl in a PBI? Do you include every single addon and library (there are over 4000 of them in the ports system + users can install their own), some of which are incompatible with each other? Same with Python, PHP, Ruby, Apache, X11 or just about everything that is not a single "application" but a framework for other things. And if you e.g. go that route (package *everything*) you will find that for innocous thing like PHP you require everything else - openssl, X11, MySQL to satisfy all the dependancies.

#12 Re: PBI's

Added on 2010-04-17T20:35 by Oliver Herold

PBI's are producing a massive overhead. Just because I have maybe certain resources doesn't mean I have to spend them just for fun. Okay it's a solution for a certain kind of desktop user, but it should be lightyears away from base.

Comments !