The biggest single idea I was hoping to get traction is the necessity of introducing a "stable" ports branch, which doesn't have to include all ports but for starts some more-than-minimal subset (e.g. the "FAPP" set). The main reasons for this are:
- Introducing ABI stability between the ports - so if some popular library changes (like libjpeg, libpng, glib), hundreds of ports on a working production system don't have to be recompiled. (in the defense of the current system - this recompiling works and is automated)
- Introducing binary packages and faster patch time - because maintaining binary packages where everything changes after such an ABI breakage is next to useless.
Unfortunately, the general feeling is that this cannot be done because of infrastructure limitations and because of port maintainers' time - they would be hard pressed to maintain separate sets of ports (the current "HEAD" branch and one or more "stable" branches). I see the infrastructure limitations can be serious but these are merely technical problems - they can be solved automatically one way or the other. The second problem - new demands on ports maintainers' time - is more significant.
I'm not a big ports maintainer - I semi-maintain only two ports - but I don't think it would be particularily hard to maintain a stable branch of the ports tree because of one simple reason: the porting work is already done and "maintaining" the stable branch mostly revolves around updating the ports with newer distfiles (source files), where applicable. Of course there are edge cases - such as that for security reasons some ports (shared libraries mostly) simply *must* break ABI, but I think this could be solved by at least "borrowing" patches and ideas from similar systems that have such a "stable packages" idea successfuly implemented (e.g. most of the big Linux distributions).
To test this, and to prove (mostly to myself) it would be doable, I've proposed a Google Summer of Code project. This is probably the last time I'd be eligible for it. Here's what I proposed:
Synopsis:
----------
A frequently requested FreeBSD feature is a
more stable ports tree and a binary package distribution mechanism. This
proposal aims to offer at least an experimental implementation of both,
in a way that offers user a subset of available ports (from the ports
tree) in an ABI-stable way which will also be maintained for at least
the period of 12 months from the SoC project start, together with their
binary packages and a rsync-based distribution method for both. The
proposed technologies for building this infrastructure include reliance
on existing FreeBSD/unix tools and libraries, with Lua as the glue
language (the specifics can be determined later, before the SoC project
starts).
The results of this project would be twofold: an
infrastructure that could be used for this purpose in the future (with
the possibility of import into the base system) if accepted by the
community of developers, and the documented experience, made available
via a blog, a wiki or similar communication or documentation
infrastructure, of the good and bad sides of creating and using such a
system. Since the results of this project will be made public ASAP, and
it would be a fairly high-profile project, it is expected that users
will participate by giving feedback.
Project Description:
--------------------
A
selection of packages will be made from the current ports tree (the
everchanging ports' "HEAD"), which will be maintained as close as
possible to an an "overlay tree" for the ports tree, with the
distinction that the overlay tree will be self-contained in terms of
dependencies and will be maintained in an ABI-stable way. The primary
goal of maintaining this "stable" tree is to provide users with patches
to the ported applications without causing ABI or shared libs breakage,
which are currently very visible with the volatile nature of the ports
tree. For the sake of possible future development, this "stable" tree
will currently be marked as intended for the FreeBSD 8.x series (future
possible versions will probably introduce ABI breakage but will be
marked for a future version of FreeBSD). The secondary goal of this
proposal is to create a robust method of distributing binary packages,
possibly by using binary diffs, which will speed up distribution and
patching. The binary packages will be built on the earliest supported
FreeBSD release from the series.
In no case will there be
attempts to circumvent, replace or disable the usual ports tree and
binary packages. Full compatibility (with regards to the ports and build
infrastructure) between the "stable" tree and the regular ports tree
will be maintained. The "stable" tree ports will be updated from the
regular tree ports as long as there are no ABI or shared libs
inconsistencies between them. Unless an unforseen situation arises,
binary packages will be the regular binary packages, compatible with
pkg_add et al.
This "stable" ports tree will be maintained for a
time period which effectively includes 9-10 months after the SoC project
officially ends.
Technical Details:
------------------
I
propose that the selection of ports to maintain in an ABI-stable way
consists of (or a superset of): apache22, php5-*, perl, python, sqlite3,
postgresql84-*, mysql55-* and their dependencies. The starting points
of these ports will be maintained as they are/were when the stable tree
"branches" from the regular ports tree. It is possible that some build
options (make config) will differ from the ports' defaults for the
binary packages, in which case they will be documented and maintained.
As is now, users can choose to follow binary packages or build ports
from source, with reasonable compatibility between them. It is outside
the scope of this project to try to split ports which build or install
multiple entities (development libraries, binaries, other resources)
into separate ports, or do any such "surgery" on existing ports. Unless
the project's scope changes (which would require public discussion and
consensus), the system's port database format (/var/db/pkg et al) will
not be modified.
A single user-accessible utility will be created
to manage the entire client side of the infrastructure, including
updating the "stable" ports tree, installing, removing and updating
ports and packages. The project/maintainer side might include multiple
utilities, as needed.
Hosting for the projects' results (ports,
packages, etc.) will be either in the FreeBSD infrastructure or (at
least during the initial SoC timeframe) provided by me.
Benefits
for FreeBSD:
----------------------
Possibly the greatest benefits
from the FreeBSD project will be the experience and feedback in
creating and maintaining this experimental infrastructure, including the
evaluation of how hard it is to maintain such infrastructure (and the
stable branch), recorded for the purpose of being used by the
developers. The entire infrastructure will be created and maintained
with the possibility of its inclusion in the FreeBSD base system, if the
developer community agrees on it and sufficient port maintainers sign
up for the effort.
Of course, absolute stability is not possible - even the systems with very rigid rules (like Debian) occasionally produce bad "stable" packages - we're all human. To err is human, but to have a VCS history of past code to slide to is what computers are for.
#1 Re: Of ports and men
Very interesting. The ports-system could use some improvements...
I'd love to be able to see what ports will be installed as deps to the one I'm currently thinking of installing. I like the idea of having a 'wrapper' round the ports-system to make it easier to deal with.
One more drastic idea is perhaps to try and unfiy the BSDs handling of third party applications, i.e. swith to pkgsrc.
#2 Re: Of ports and men
Ivan, any effort towards the ports system will be dearly appreciated, but be ready to spend over 90% of time in politics, rather than technical matters. History shows, only very tiny, evolutionary and almost invisible improvements have a chance to survive and blossom in ports' infrastructure.
My opinion is, since the ports system is so behemothly slow to move, it's a perfect target for extensive and deep research. Of course, there's not much chance anyone is going to fund it.
Like Andreas noticed, joining forces with pkgsrc, OpenBSD, Gentoo, Arch, Nix and others could provide the necessary momentum, but unlike src devs, key ports people have never shown much interest for cooperation with other systems. Internal bikesheds seem enough.
Good luck!
#3 Re: Of ports and men
I would love to see a complete switch to pkgsrc, both sides would profit from that!
#4 Re: Of ports and men
pkgsrc is smaller than ports - that will never happen. OTOH nothing is stopping people from using pkgsrc on FreeBSD.
#5 Re: Of ports and men
Having a quarterly "stable" release of pkgsrc has actually reduced my labor, and I imagine would do the same for ports.
I'm trying to do full binary builds for DragonFly (i386 and x86_64, current and development release) which is a lot of time-consuming building to track. Having a relatively stable branch to track means major updates are confined to about every 3 months, rather than "at any time and in any frequency", as it is now if you have updates to libjpeg, perl, php, etc.
#6 Re: Of ports and men
What does "stable" mean in the context of pkgsrc? Do minor updates and bugfixes still go into the stable branch or is it simply a frozen snapshot of the state at a point in time? If the second, I think it doesn't really help anyone. E.g. a PHP release (X.Y) might have point releases (X.Y.z) more than quarterly, and since it's such a big source of security holes, it's necessary to have it updated ASAP.
#7 Re: Of ports and men
> pkgsrc is smaller than ports
That's right, of course the missing ports/packages would have to be ported over first...
> nothing is stopping people from using pkgsrc on FreeBSD.
pkgsrc works mostly well on FreeBSD but only very few people use this combination, so at present it would always take more time to get FreeBSD specific patches and the like into pkgsrc. I don't know if e.g. X.org from pkgsrc would work as-is on FreeBSD
> Do minor updates and bugfixes still go into the stable branch or is it simply a frozen snapshot of the state at a point in time?
Security updates and other highly critical updates are pulled into the stable branch also.
#8 Re: Of ports and men
It's good idea
#9 Re: Of ports and men
If I may add my 2c here, a good improvement would be greater standardisation of knobs and being able to disable the libdialog menu in favour of defining what knobs you want ahead of time in another file (/etc/knobs.conf?). Something like this:
PORTAUDIO=False
package.FOOFLAG=True
Not sure if that's feasible, though.
#10 Re: Of ports and men
Doesn't the PBI system in PC-BSD already does what the article is trying to do? Although I don't use PC-BSD, if I recall how the PBI system works it encapsulates all needed libraries and executables for the PBI so that it is self contained and independant of other ports or libraries.
#11 PBI's
Sort of - but while it's ok for user applications like OpenOffice it scales badly for server-side stuff. How do you package Perl in a PBI? Do you include every single addon and library (there are over 4000 of them in the ports system + users can install their own), some of which are incompatible with each other? Same with Python, PHP, Ruby, Apache, X11 or just about everything that is not a single "application" but a framework for other things. And if you e.g. go that route (package *everything*) you will find that for innocous thing like PHP you require everything else - openssl, X11, MySQL to satisfy all the dependancies.
#12 Re: PBI's
PBI's are producing a massive overhead. Just because I have maybe certain resources doesn't mean I have to spend them just for fun. Okay it's a solution for a certain kind of desktop user, but it should be lightyears away from base.