Go (golang) packaging, part 2

[Hilko Bengen]

Michael,

after re-reading your GoPackaging Wiki page and digging around a bit in
the sources for the go tool, I think that only minimal changes to go(1)
are needed for its build system to "Just Work" without breaking
expectations:

1. Debian user's/admin's perspective: As I have already written
elsewhere in this thread, using upstream build systems without extra
parameters usually results in libraries and binaries being installed to
/usr/local. This is not what "sudo go get" currently (version 1:1.0.2-2)
does -- it happily puts both source and binary files into GOROOT
(=/usr/lib/go). This is bound to break things in interesting ways at
some point.

One way to fix this would be changing the default GOPATH setting from an
empty string to a path that has "/usr/local/lib/go" as its first entry.

Another quick solution might be changing go(1)'s behavior so that it
will still look for libraries in GOROOT, just not attempt to modify
files there.

2. Software developer's perspective: Since the text "How to Write Go
Code" at http://golang.org/doc/code.html tells me right away that
setting GOPATH is something I need to do, changing the GOPATH default
value should not cause too much confusion.

3. Debian package maintainer's perspective: I disagree with your
position that "Go libraries [...] should not be used directly for Go
development". If I put together a Debian package for a Go library,
developers should still have the option to use it. And it should be
easy, preferably be installed into the default GOPATH, so a second entry
(after /usr/local/lib/go) does not seem unreasonable.

Upstream has not made it clear whether 3rd partylibraries should be
installed into GOROOT or somewhere else. There still are a few
unanswered questions about how go(1) should behave, and what bugs are
still lurking in that code. Perhaps upstream will change go(1) so that a
concept of third-party libraries is possible. Until then, putting
libraries into a separate directory such as /usr/lib/gocode seems like a
good idea.

But what about gccgo? Apparently, go(1) can be used with gccgo, but
gccgo-compiled code cannot be linked with golang-go-compiled code. So,
perhaps it makes sense to install the libraries produced by each
compiler into separate directory hierarchies -- /usr/lib/gocode/golang
and /usr/lib/gocode/gccgo?

Cheers,
-Hilko

[Michael Stapelberg]

Hi Matthias,

2. -fno-split-stack
Otherwise I could not link executables and got the following error
/usr/bin/ld.bfd.real: cindex: hidden symbol `__morestack' in
/usr/lib/gcc/x86_64-linux-gnu/4.7/libgcc.a/morestack.o) is referenced
by DSO
/usr/bin/ld.bfd.real: final link failed: Bad value

better use -fuse-ld=ld.gold, or in a packaging context, build-depend on
binutils-gold. gold should be available on all archs where gccgo is available.

This doesn’t work unfortunately:

1. -fuse-ld=ld.gold results in:
gccgo: error: unrecognized command line option ‘-fuse-ld=ld.gold’
This is with gccgo 4:4.7.2-1 (current Debian testing)

2. Installing binutils-gold doesn’t help either. When calling
gccgo -Wl,-debug -I /tmp/godyn/src -shared -fPIC -o /tmp/build/libsparse.so ./set.go
I get:

ld_file_name = /usr/bin/ld
c_file_name = /usr/bin/gccgo
nm_file_name = /usr/bin/nm
strip_file_name = /usr/bin/strip
c_file = /tmp/ccks2q2o.c
o_file = /tmp/ccjfIMxS.o
COLLECT_GCC_OPTIONS = '-fsplit-stack' '-I' '/tmp/godyn/src' '-shared' '-fPIC' '-o' '/tmp/build/libsparse.so' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
COLLECT_GCC = gccgo
COMPILER_PATH = /usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH = /usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/

/usr/bin/ld --sysroot=/ --build-id --no-add-needed --eh-frame-hdr -m elf_x86_64 --hash-style=both -shared -o /tmp/build/libsparse.so /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. /tmp/ccRsBUuQ.o -lgobegin -lgo -lm --wrap=pthread_create -lgcc_s -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
/usr/bin/ld.gold.real: error: /tmp/ccRsBUuQ.o: could not convert call to '__morestack' to '__morestack_non_split'
collect2: error: ld returned 1 exit status
[Leaving /tmp/ccks2q2o.c]
[Leaving /tmp/ccjfIMxS.o]
[Leaving /tmp/ccC2k82l.ld]
[Leaving /tmp/ccP48tyP.le]
[Leaving /tmp/build/libsparse.so]

[Michael Stapelberg]

Hi Hilko,

Post by Hilko Bengen
/usr/local. This is not what "sudo go get" currently (version 1:1.0.2-2)
does -- it happily puts both source and binary files into GOROOT
(=/usr/lib/go). This is bound to break things in interesting ways at
some point.

Indeed. This is when having an empty GOPATH, right?

Post by Hilko Bengen
One way to fix this would be changing the default GOPATH setting from an
empty string to a path that has "/usr/local/lib/go" as its first entry.
Another quick solution might be changing go(1)'s behavior so that it
will still look for libraries in GOROOT, just not attempt to modify
files there.

I prefer the latter. We should ask upstream about their point of view on
this first, though.

Post by Hilko Bengen
3. Debian package maintainer's perspective: I disagree with your
position that "Go libraries [...] should not be used directly for Go
development". If I put together a Debian package for a Go library,
developers should still have the option to use it. And it should be
easy, preferably be installed into the default GOPATH, so a second entry
(after /usr/local/lib/go) does not seem unreasonable.
Upstream has not made it clear whether 3rd partylibraries should be
installed into GOROOT or somewhere else. There still are a few
unanswered questions about how go(1) should behave, and what bugs are
still lurking in that code. Perhaps upstream will change go(1) so that a
concept of third-party libraries is possible. Until then, putting
libraries into a separate directory such as /usr/lib/gocode seems like a
good idea.

I am slightly confused. First you say you disagree, but then you admit
that there still are problems and agree with regards to placing
libraries into /usr/lib/gocode after all? :-)

Post by Hilko Bengen
But what about gccgo? Apparently, go(1) can be used with gccgo, but
gccgo-compiled code cannot be linked with golang-go-compiled code. So,
perhaps it makes sense to install the libraries produced by each
compiler into separate directory hierarchies -- /usr/lib/gocode/golang
and /usr/lib/gocode/gccgo?

I’d say it makes more sense to pick one compiler instead.

[Hilko Bengen]

Post by Michael Stapelberg

Post by Hilko Bengen
/usr/local. This is not what "sudo go get" currently (version 1:1.0.2-2)
does -- it happily puts both source and binary files into GOROOT
(=/usr/lib/go). This is bound to break things in interesting ways at
some point.

Indeed. This is when having an empty GOPATH, right?

Right. Anf if I have interpreted the code correctly, "go get -u
<package>" will at the moment try to update the first occurence of the
package that it finds in $GOROOT or $GOPATH.

Post by Michael Stapelberg

Post by Hilko Bengen
One way to fix this would be changing the default GOPATH setting from an
empty string to a path that has "/usr/local/lib/go" as its first entry.
Another quick solution might be changing go(1)'s behavior so that it
will still look for libraries in GOROOT, just not attempt to modify
files there.

I prefer the latter. We should ask upstream about their point of view on
this first, though.

Maybe they can also be persuaded to put the two roles that have been
assigned to GOPATH into separate variables: A search path and an
installation target.

Post by Michael Stapelberg

Post by Hilko Bengen
3. Debian package maintainer's perspective: I disagree with your
position that "Go libraries [...] should not be used directly for Go
development". If I put together a Debian package for a Go library,
developers should still have the option to use it. And it should be
easy, preferably be installed into the default GOPATH, so a second entry
(after /usr/local/lib/go) does not seem unreasonable.
Upstream has not made it clear whether 3rd partylibraries should be
installed into GOROOT or somewhere else. There still are a few
unanswered questions about how go(1) should behave, and what bugs are
still lurking in that code. Perhaps upstream will change go(1) so that a
concept of third-party libraries is possible. Until then, putting
libraries into a separate directory such as /usr/lib/gocode seems like a
good idea.

I am slightly confused. First you say you disagree, but then you admit
that there still are problems and agree with regards to placing
libraries into /usr/lib/gocode after all? :-)

I may have misunderstood what you meant with "used directly for Go
development". Ah well, let me try again: I would very much like to see a
number of useful Go library packages distributed by Debian that:

- make the process for developing or building executables (and other
libraries) as easy as or easier than using "go get". (This is what I
understand as direct usage.)

- make it possible and easy to package such executables and libraries
for Debian. Specific DBMS support packages for the database/sql
interface come to mind...

I agree with putting libraries into a directory separate from $GOROOT
because I consider it the lesser evil as long as the go(1) questions
aren't sorted out.

Post by Michael Stapelberg

Post by Hilko Bengen
But what about gccgo? Apparently, go(1) can be used with gccgo, but
gccgo-compiled code cannot be linked with golang-go-compiled code. So,
perhaps it makes sense to install the libraries produced by each
compiler into separate directory hierarchies -- /usr/lib/gocode/golang
and /usr/lib/gocode/gccgo?

I’d say it makes more sense to pick one compiler instead.

To get things started with as little effort as possible, pick one
compiler and package some libraries and executable packages, such as
codesearch. I'm all in favor of this compiler being golang-go.

It will also make sense to support gccgo[1] at some point -- if only for
portability across all Debian platforms. Picking an extensible naming
scheme directory hierarchy (such as /usr/lib/gocode/$IMPL) would make
this possible from the start.

By the way: Where does the idea for the "gocode" directory name come
from?

Cheers,
-Hilko

[1] and possibly other implementations that may come with yet different
incompatible binary interfaces...

[Michael Stapelberg]

Hi Hilko,

Sure. See the Makefile at the end of this mail. Please note that I
[...]

Thanks for the instructions. I reproduced them and got shared libraries
plus dynamically linked binaries.

Aside from details about the split stack flags, now one big question
arises:

Assuming we ship Go libraries compiled as shared libraries, where do we
get the SONAME from? There is no mechanism for Go libraries to declare
an ABI break. Inventing one and asking all upstream projects to adopt it
seems unlikely to succeed. Ignoring SONAMEs altogether could make
software break on the user’s installation. Then again, the same could
happen to every interpreted language, e.g. Perl.

What do you think?

[Paul Wise]

Post by Michael Stapelberg
Assuming we ship Go libraries compiled as shared libraries, where do we
get the SONAME from? There is no mechanism for Go libraries to declare
an ABI break. Inventing one and asking all upstream projects to adopt it
seems unlikely to succeed. Ignoring SONAMEs altogether could make
software break on the user’s installation. Then again, the same could
happen to every interpreted language, e.g. Perl.

Could it be automatically created based on some sort of hash of the
exported symbols and their argument lists?

If not, based on your discussions with upstream I would say that the
culture of the Go development community is incompatible with shared
libraries. So your choices are:

Convince Go upstream to add an officially-supported and encouraged
shared library mechanism that includes ABI mechanisms.

Invent some automatic, Debian-specific method for ABI stuff. Like
Provides: go-foo-abi-<hash> and put that info in shlibs.

Use static libraries, Built-Using and frequent binNMUs.

[Steve McIntyre]

Post by Paul Wise

Post by Michael Stapelberg
Assuming we ship Go libraries compiled as shared libraries, where do we
get the SONAME from? There is no mechanism for Go libraries to declare
an ABI break. Inventing one and asking all upstream projects to adopt it
seems unlikely to succeed. Ignoring SONAMEs altogether could make
software break on the user’s installation. Then again, the same could
happen to every interpreted language, e.g. Perl.

FFS, yet another new language where the implementors have refused to
think ahead and consider ABI handling? Idiots. :-(

Post by Paul Wise
Could it be automatically created based on some sort of hash of the
exported symbols and their argument lists?
If not, based on your discussions with upstream I would say that the
culture of the Go development community is incompatible with shared
Convince Go upstream to add an officially-supported and encouraged
shared library mechanism that includes ABI mechanisms.
Invent some automatic, Debian-specific method for ABI stuff. Like
Provides: go-foo-abi-<hash> and put that info in shlibs.
Use static libraries, Built-Using and frequent binNMUs.

Considering the mess that we already have with (for example) Haskell
in this respect, I would vote strongly against accepting or pretending
to support any more languages in this vein.

[Chow Loong Jin]

Post by Steve McIntyre
FFS, yet another new language where the implementors have refused to
think ahead and consider ABI handling? Idiots. :-(

I totally agree with you here.

Post by Steve McIntyre
Considering the mess that we already have with (for example) Haskell
in this respect, I would vote strongly against accepting or pretending
to support any more languages in this vein.

This won't really help matters much -- people who want/need to use the language
will install it for themselves out of the package manager, leading to an even
bigger mess. Otherwise someone will just end up supporting Debian packages out
of Debian, which is also suboptimal.

[Joachim Breitner]

Hi,

Post by Steve McIntyre

Post by Paul Wise

Post by Michael Stapelberg
Assuming we ship Go libraries compiled as shared libraries, where do we
get the SONAME from? There is no mechanism for Go libraries to declare
an ABI break. Inventing one and asking all upstream projects to adopt it
seems unlikely to succeed. Ignoring SONAMEs altogether could make
software break on the user’s installation. Then again, the same could
happen to every interpreted language, e.g. Perl.

FFS, yet another new language where the implementors have refused to
think ahead and consider ABI handling? Idiots. :-(

Post by Paul Wise
Could it be automatically created based on some sort of hash of the
exported symbols and their argument lists?
If not, based on your discussions with upstream I would say that the
culture of the Go development community is incompatible with shared
Convince Go upstream to add an officially-supported and encouraged
shared library mechanism that includes ABI mechanisms.
Invent some automatic, Debian-specific method for ABI stuff. Like
Provides: go-foo-abi-<hash> and put that info in shlibs.
Use static libraries, Built-Using and frequent binNMUs.

Considering the mess that we already have with (for example) Haskell
in this respect, I would vote strongly against accepting or pretending
to support any more languages in this vein.

At least to me my work on Haskell in Debian feels more than pretending,
and from personal experience with the creators of the language, I have
strong doubts that they are Idiots.

In fact I don’t see how you can have modern features like
cross-module-inlining without having to potentially recompile depending
packages.

And it is clearly not (anymore) the case that ABIs are not handled in
Haskell. In fact they are handled in a very precise way that allows us
to guarantee on the package level that user’s installations won’t get
broken. I think our priorities should be the user’s experience, and we
should be willing to accept a little infrastructural complications on
our side for that goal.

Also, I’d like to point out that the challenges provided by languages
like OcaML and Haskell have motivated improvements to our infrastructure
that benefit the project in general, such as the BD-Uninstallable
support in wanna-build that has prevented a lot of failed builds (and
hence manual intervention).

Greetings,
Joachim

[Adam Borowski]

Post by Joachim Breitner
At least to me my work on Haskell in Debian feels more than pretending,
and from personal experience with the creators of the language, I have
strong doubts that they are Idiots.
In fact I don’t see how you can have modern features like
cross-module-inlining without having to potentially recompile depending
packages.
And it is clearly not (anymore) the case that ABIs are not handled in
Haskell. In fact they are handled in a very precise way that allows us
to guarantee on the package level that user’s installations won’t get
broken. I think our priorities should be the user’s experience, and we
should be willing to accept a little infrastructural complications on
our side for that goal.

It's not a matter of "a little infrastructural complication", it's about
having the slightest chance of reasonable security support -- or even
regular bug fixes, when multiple layers of libraries are involved.

If there is a bug in library A, if you use static linking, you need to
rebuild every single library B that uses A, then rebuild every C that uses
B, then finally every single package in the archive that uses any of these
libraries.

Just imagine what would happen if libc6 would be statically linked, and a
security bug happens inside (like, in the stub resolver). Rebuilding the
world on every update might be viable for a simple scientific task[1], but
not at all for a distribution.

Static linking also massively increases memory and disk use; this has
obvious performance effects if there's more than one executable running
on the system.


[1]. Simple as for the number of diverse packages/systems involved.

[Joachim Breitner]

Hi,

Post by Adam Borowski
It's not a matter of "a little infrastructural complication", it's about
having the slightest chance of reasonable security support -- or even
regular bug fixes, when multiple layers of libraries are involved.
If there is a bug in library A, if you use static linking, you need to
rebuild every single library B that uses A, then rebuild every C that uses
B, then finally every single package in the archive that uses any of these
libraries.
Just imagine what would happen if libc6 would be statically linked, and a
security bug happens inside (like, in the stub resolver). Rebuilding the
world on every update might be viable for a simple scientific task[1], but
not at all for a distribution.

why not?

I agree that it is not desirable, but it is possible, and if it it were
possible easily (e.g. with little human interaction), this would be an
indicate of a very good infrastructure.

And in fact we have it: binNMUs and buildds enable us to rebuild large
parts¹ of the distribution after such a change (otherwise, maintaining
Haskell wouldn’t be feasible).

Post by Adam Borowski
Static linking also massively increases memory and disk use; this has
obvious performance effects if there's more than one executable running
on the system.

True, static linking has its disadvantages. But this is unrelated to the
problem of supporting languages with often-changing ABIs.

Greetings,
Joachim

¹ I do it regularly for more than 2.5% of all our source packages, and –
besides machine power – I see no reason why this would be impossible for
larger fractions of the archive.

[Don Armstrong]

Post by Joachim Breitner

Post by Adam Borowski
Just imagine what would happen if libc6 would be statically
linked, and a security bug happens inside (like, in the stub
resolver). Rebuilding the world on every update might be viable
for a simple scientific task[1], but not at all for a
distribution.

I agree that it is not desirable, but it is possible, and if it it
were possible easily (e.g. with little human interaction), this
would be an indicate of a very good infrastructure.

It's certainly possible, and our infrastructure enables us to even
consider doing it, but it's still a design flaw in the underlying
languages and their interfaces that should be addressed and resolved.

We need to be better at communicating to upstreams who are designing
and maintaining packages which do these kinds of things the problems
that it causes for people who want to use their packages in stable
production systems.


Don Armstrong

[Hilko Bengen]

Post by Adam Borowski
If there is a bug in library A, if you use static linking, you need to
rebuild every single library B that uses A, then rebuild every C that
uses B, then finally every single package in the archive that uses any
of these libraries.

But wouldn't it be great if all the people wishing for something that
combines the strengths of Debian's and Gentoo's idea of software
packaging for years finally got the system they wanted as a side effect
of automating such rebuilds?

Cheers,
-Hilko

[Enrico Tassi]

Post by Joachim Breitner
At least to me my work on Haskell in Debian feels more than pretending,
and from personal experience with the creators of the language, I have
strong doubts that they are Idiots.

They are not, they are very smart, but they are academic people
with a very little idea of what it means to build a real software
distribution like Debian. I love type systems to, but whenever I talk
about static linking with in the academic context, they think it just
saves few KB of disk space.

I can't assert the same for go's designers, but for different reasons
they ended up incurring in the same design flaw. They want a binary to
be shipped on their production servers, and be sure that no matter how
crappy they are, it will work (no missing .so, no .so ABI problem).
Or at least, this is what I've understood.

Here we build a realistic system where a security update can be
made just by recompiling and pullig 1 package, not recompiling the whole
archive. Static linking imposes so.

To me it is just a problem of scale. You see the problems of static
linking only on a reasonably large scale. And Debian is huge...

Ciao

[Jon Dowland]

Not all C libraries are distributed from one central site and they
certainly don't expect you to use a central package installation system.

So much more the shame for C. Those are *improvements* that Perl came up
with (well, actually, that the TeX community came up with and Perl copied)
that made the ecosystem much nicer.

On that note, Rusty Russell's "CCAN" is interesting:
http://ccodearchive.net/

[Jon Dowland]

As a Haskell developer, I find cabal much more convenient than nothing,
in the situation where the library I want is not packaged by Debian yet.
If I want my haskell libraries and programs to reach a wide audience, I
need to learn Cabal anyway.

If you are writing libraries to add to the language, then I don't consider
you a normal developer using the language.

Well, my point stands if I were writing a mere Haskell user-oriented program
for that matter.

You generally don't have to because things are in Debian archives already.

It can be a chicken and egg problem. I see .debs often when a repository does
exist (spotify, dropbox I think, google chrome) and many situations where a
repository does not (humble indie bundle)

If you want bleeding edge, then you are not a normal user and you
certainly aren't a system administrator that wants to keep a controlled
system they can reproduce.

I must admit I'm losing touch of precisely what you are arguing here. I guess
it's not "everything that matters will be packaged with Debian" hence the
previous paragraph re: external apt repositories. Yet, I don't suppose you're
arguing that availability in an external apt repository is any guarantee of
quality (Or at least I hope you're not). I don't think we're necessarily
talking about bleeding edge, either. If something is not packaged in Debian,
it's not necessarily bleeding edge.

I know dpkg --get-selections will tell me all the software installed on
the system so I can do the same on another one. If yet another package
maanger gets involved I have to know about it and do something different
to handle that. That's not a good thing.

True. But you also lose lots of other information, such as what is marked
automatic, the contents of your debconf DB and corresponding changes in
/etc, non-corresponding changes in /etc… dpkg --get-selections is not and
has never been a solution to the problem you are describing.

[Jon Dowland]

Using Debian packages is a *means*, not an *end*. Sometimes in these
discussions I think people lose sight of the fact that, at the end of the
day, the goal is not to construct an elegantly consistent system composed
of theoretically pure components. That's a *preference*, but there's
something that system is supposed to be *doing*, and we will do what we
need to do in order to make the system functional.

And in particular, where a problem cannot be solved in pure Debian, I don't
want Debian to interfere with the bit of the solution that lives outside of
its domain. That may include not attempting to package/patch/alter/adjust
upstream systems like Go that have a different philosophical approach. The
worst case scenario IMHO is some people invest a lot of time to make the
Debianized-Go stuff quite divergent from upstream, people's expectations of
how things behave in Go-land are broken when they access Go-via-Debian, the
job is never quite complete and so we get extra bugs, and a new upstream
community relationship is marred. This is a much worse outcome than not
attempting to package Go at all, IMHO.

I guess I'd quite like the boundaries of responsibility to be very clear,
when I'm forced to have such boundaries.

[Adam Borowski]

Post by Jon Dowland
And in particular, where a problem cannot be solved in pure Debian, I don't
want Debian to interfere with the bit of the solution that lives outside of
its domain. That may include not attempting to package/patch/alter/adjust
upstream systems like Go that have a different philosophical approach. The
worst case scenario IMHO is some people invest a lot of time to make the
Debianized-Go stuff quite divergent from upstream, people's expectations of
how things behave in Go-land are broken when they access Go-via-Debian

Just think what would happen if every single language and environment would
be allowed its own packaging system. This way lies madness. Yes, packaging
needs to be beaten into shape, even if this goes against upstream wishes.

[Hilko Bengen]

Post by Adam Borowski

The worst case scenario IMHO is some people invest a lot of time to
make the Debianized-Go stuff quite divergent from upstream, people's
expectations of how things behave in Go-land are broken when they
access Go-via-Debian

Just think what would happen if every single language and environment
would be allowed its own packaging system. This way lies madness.

Not if there's a clear separation between the language's solution and
dpkg-land. Mind you, almost every language already has its own packaging
system or two. I really don't see any big deal here.

Post by Adam Borowski
Yes, packaging needs to be beaten into shape, even if this goes
against upstream wishes.

Just put up a boundary between dpkg and everything else. This is
actually quite easy because Debian packages generally don't install
anything to /usr/local. From there it's just a maatter of ensuring that
each packaging system respects the line. And I expect that that's
exactly what is going to happen in the case of the go(1) tool.

Cheers,
-Hilko

[Neil Williams]

On Tue, 05 Feb 2013 23:44:30 +0100

Post by Hilko Bengen

Post by Adam Borowski

The worst case scenario IMHO is some people invest a lot of time to
make the Debianized-Go stuff quite divergent from upstream, people's
expectations of how things behave in Go-land are broken when they
access Go-via-Debian

Just think what would happen if every single language and environment
would be allowed its own packaging system. This way lies madness.

Not if there's a clear separation between the language's solution and
dpkg-land. Mind you, almost every language already has its own packaging
system or two. I really don't see any big deal here.

Then don't package Go at all and leave it entirely outside the realm of
dpkg - no dependencies allowed in either direction, no files created
outside /usr/local for any reason, no contamination of the apt or dpkg
cache data. If what you want is complete separation, why is there even
a long running thread on integration?

Post by Hilko Bengen

Post by Adam Borowski
Yes, packaging needs to be beaten into shape, even if this goes
against upstream wishes.

Just put up a boundary between dpkg and everything else. This is
actually quite easy because Debian packages generally don't install
anything to /usr/local. From there it's just a maatter of ensuring that
each packaging system respects the line. And I expect that that's
exactly what is going to happen in the case of the go(1) tool.

Then why bother discussing packaging Go if it isn't going to be
packaged, it's just going to invent it's own little ghetto
in /usr/local?

If Go wants to be packaged, it complies by the requirements of
packaging. If it wants to live the life of a hermit and disappear up
itself, that's fine but then it doesn't get the privilege of
interacting with the rest of Debian. It's just a user download.

[Jon Dowland]

Then don't package Go at all and leave it entirely outside the realm of dpkg
- no dependencies allowed in either direction, no files created outside
/usr/local for any reason, no contamination of the apt or dpkg cache data. If
what you want is complete separation, why is there even a long running thread
on integration?

That's one possible solution, and a low-risk one at that. The others carry the
risk of doing the job badly, especially if there is not enough resource to
implement them going forwards.

Then why bother discussing packaging Go if it isn't going to be packaged,
it's just going to invent it's own little ghetto in /usr/local?

That seems pretty perjorative. The reason Go has "invented it's own little
ghetto" is to solve the distribution problem. The reason they want to solve it
is because, despite our best efforts, we haven't solved it. Pretending we have
done helps nobody. From Go's perspective, we are a bit player. There's a
pattern of misplaced arrogance and pride that permeates -devel from time to
time whenever difficult integration discussions come up (systemd included) that
really doesn't help. Let's not overstate our importance in the wider world, it
does not help us one bit. Step by step we'll just close our doors to the rest
of the Universe and slide further into irrelevance.

If Go wants to be packaged, it complies by the requirements of packaging.

Go doesn't want anything: It's a programming language and environment, not a
sentient being. The authors of Go are probably not that bothered about it being
packaged, seeing as they've put energy into solving the distribution problem
themselves, at the same time making it more difficult to distribution-package.
The people who want to see it packaged are people who want to see Debian users
be able to conveniently interact with Go-land.

[Russ Allbery]

Post by Neil Williams
If Go wants to be packaged, it complies by the requirements of
packaging. If it wants to live the life of a hermit and disappear up
itself, that's fine but then it doesn't get the privilege of interacting
with the rest of Debian. It's just a user download.

Debian packaging isn't a reward that we give to upstream software authors
in exchange for doing things right. Most software gets Debian packages
because people who use Debian want to use it and don't like dealing with
unpackaged software. Upstream authors are often indifferent.

[Hilko Bengen]

If what you want is complete separation, why is there even a long
running thread on integration?

Sorry if I failed to make myself clear:

I want excellent Debian packages of the compiler/runtime/tools *and*
libraries *and* still make it possible for our users to use upstream's
half-a-package-manager if they so desire. Complete separation is
something entirely different.

Then why bother discussing packaging Go if it isn't going to be
packaged, it's just going to invent it's own little ghetto in
/usr/local?

We already have directory structures in place in /usr/local for several
other languages and "software ecosystems":

$ ls /usr/local/lib/
eclipse ocaml python2.6 python3.1 site_ruby
luarocks perl python2.7 python3.2
$ ls /usr/local/share
applications emacs games man sgml xml
ca-certificates fonts hibernate perl texmf zsh

I am pretty sure that if you asked about packaging software in the
Python, Perl, Ruby, Java, Lua communities, you would get recommendations
to not use Debian packages at all and get pointers to what the
respective community considers a solution to the packaging problem (if
they see it as a problem at all). This likely involve replacing the
language compiler/runtime itself.

So what? Just take the freedom given to you through the software
licenses and ignore those parts of their view of how things ought to
work and work with the useful parts instead.

Cheers,
-Hilko

[Roland Mas]

Hilko Bengen, 2013-02-06 14:46:11 +0100 :

[...]

Post by Hilko Bengen
I am pretty sure that if you asked about packaging software in the
Python, Perl, Ruby, Java, Lua communities, you would get recommendations
to not use Debian packages at all and get pointers to what the
respective community considers a solution to the packaging problem (if
they see it as a problem at all).

I can only speak about Python and Perl, but I don't remember *ever*
having been told to use their deployment system instead of the packaged
versions of the interpreter and modules. The closest I've seen is
something like "if you're running CentOS or RHEL, then you'll need this
plethora of modules that are not packaged, so please use our
language-specific system to install them instead".

So it is possible for a language community to work with the
distributors rather than against. The fact that some communities
(insert your favourite pet peeve here) choose not to shouldn't be
construed as an impossibility.

Roland.

[Hilko Bengen]

Post by Roland Mas
[...]

Post by Hilko Bengen
I am pretty sure that if you asked about packaging software in the
Python, Perl, Ruby, Java, Lua communities, you would get recommendations
to not use Debian packages at all and get pointers to what the
respective community considers a solution to the packaging problem (if
they see it as a problem at all).

I can only speak about Python and Perl, but I don't remember *ever*
having been told to use their deployment system instead of the packaged
versions of the interpreter and modules. The closest I've seen is
something like "if you're running CentOS or RHEL, then you'll need this
plethora of modules that are not packaged, so please use our
language-specific system to install them instead".

I have heard exactly what I described at Perl conferences more than
once. The root cause may have been that some versions of RHEL still ship
Perl 5.8.something and ancient, broken versions of some modules, but I
had the impression that some (not all) people over-generalized this view
to every Linux distribution, including Debian.

If you need or want to run the current stable Perl (5.16.2) and the
latest-greatest modules on wheezy, it's going to be in your best
interest to use things like perlbrew and local::lib (which are both
shipped with Debian).

That upstream's preferences may differ from ours is not even a problem,
as long as no-one tries to enforce his values on users. I don't see such
attempts in the Perl and Go communities.

Post by Roland Mas
So it is possible for a language community to work with the
distributors rather than against.

And it's possible for Debian to at least *try* to work with any language
community.

Simply calling people "idiots" when one hasn't yet understood the other
community's values/interests/position does not help, of course. ;-)

Cheers,
-Hilko

[Barry Warsaw]

I can only speak about Python and Perl, but I don't remember *ever* having
been told to use their deployment system instead of the packaged versions of
the interpreter and modules. The closest I've seen is something like "if
you're running CentOS or RHEL, then you'll need this plethora of modules that
are not packaged, so please use our language-specific system to install them
instead".

Speaking with many hats on, I think Debian Python has done a very admirable
job of integrating the Python ecosystem with Debian.

It's never going to be perfect, and there are certainly difficult outliers,
but in most cases, things work pretty well. OTOH, Python itself has several
strategies for dealing with difficult situations, with probably the most
notable being virtualenv (and upstream's built-in venv in Python 3.3+). You
can even mix and match, i.e. if some Debian packaged versions are fine but you
need one or two missing or out-of-date packages from the Cheeseshop, you can
`virtualenv --site-system-packages` that generally works pretty well.

Where things get tricky is if you have multiple applications that need
different versions of its dependencies. Say Debian has python-foo 1.2 which
application Bar needs, but application Baz needs python-foo 2.0. Despite
years of discussion, in Debian, Ubuntu, and upstream, we really don't have a
good multi-version story. I think that's forgivable though because it's a
Really Hard Problem and nobody's stepped up to pay a team of 5 gurus for the
25 man years of non-stop work it would probably take to solve (both
technically and socially ;). This problem isn't particularly limited to
Python.

I'm also encouraged by the albeit slow work on upstream packaging technology
(PEPs and code) to help improve the overall packaging story. I had many
discussions with various packaging folks, and good developers from Fedora and
other distros, about ways to make it easier to integrate PyPI packaging with
distros, Linux-based or otherwise. I thought we'd made a lot of good
progress, but some of the drivers moved on to other things. I'm hoping Nick
Coghlan's efforts at Pycon 2013 can help motivate a revival of this work[1].

Cheers,
-Barry

[1]
https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/

[Russ Allbery]

Post by Barry Warsaw
Where things get tricky is if you have multiple applications that need
different versions of its dependencies. Say Debian has python-foo 1.2
which application Bar needs, but application Baz needs python-foo 2.0.
Despite years of discussion, in Debian, Ubuntu, and upstream, we really
don't have a good multi-version story. I think that's forgivable though
because it's a Really Hard Problem and nobody's stepped up to pay a team
of 5 gurus for the 25 man years of non-stop work it would probably take
to solve (both technically and socially ;). This problem isn't
particularly limited to Python.

I keep being tempted to go off on a rant about how we have all of these
modern, sophisticated, much more expressive programming languages, and yet
still none of them handle ABI versioning as well as C does. Normal
versioning problems that we just take for granted in C, such as allowing
the coexistence of two versions of the same library with different ABIs on
the system at the same time without doing the equivalent of static
linking, vary between ridiculously difficult and completely impossible in
every other programming language that I'm aware of (with the partial
exception of C++ where it is possible to use the same facilities as C,
just considerably more difficult).

In fact, most new languages seem to be *regressing* here. Both Perl and
Python were already fairly bad at this, Java bailed on the problem and
shoved it under the carpet of a version of static linking, and now Go is
even worse than they are and is explicitly embracing static linking.
Sigh.

It's immensely frustrating that this doesn't appear to be an interesting
problem to language designers, and even brand-new, sophisticated languages
that are trying to be at the cutting edge of language design do not make
any serious attempt at solving the ABI versioning problem.

Everyone could stand to learn something from the REST and web services
community, where ABI versioning is a well-known and widely accepted
problem that regularly prompts considerable discussion and careful thought
and a variety of useful proposals for addressing it with different
tradeoffs. It's kind of sad when one's brand-new, cutting-edge
programming language does a worse job at a foundational requirement of
language ecosystems than the Facebook API.

[Joachim Breitner]

Hi,

Post by Russ Allbery
Normal
versioning problems that we just take for granted in C, such as allowing
the coexistence of two versions of the same library with different ABIs on
the system at the same time without doing the equivalent of static
linking, vary between ridiculously difficult and completely impossible in
every other programming language that I'm aware of

things are not that bad in Haskell: Precisely due to the rapidly
changing ABIs, and hence usually very tightly versioned build
dependencies in the library metadata, Haskell (i.e. Cabal) has very good
support for installing multiple versions of a library being installed at
the same time. Work is in progress to even support multiple builds of
the same library (e.g. foo-1.0 built against bar-1.0 and foo-1.0 built
against bar-2.0) to be installed at the same time.

Of course this is not a feature that helps us a lot in Debian, where we
usually want to provide one single version of each library. But that is
due to our choosing, and not a shortcoming of the language ecosystem.
And we have had exceptions (parsec, QuickCheck) when there are two
common major versions of a library, so it is possible.

Greetings,
Joachim

[Steve Langasek]

Post by Russ Allbery
I keep being tempted to go off on a rant about how we have all of these
modern, sophisticated, much more expressive programming languages, and yet
still none of them handle ABI versioning as well as C does. Normal
versioning problems that we just take for granted in C, such as allowing
the coexistence of two versions of the same library with different ABIs on
the system at the same time without doing the equivalent of static
linking, vary between ridiculously difficult and completely impossible in
every other programming language that I'm aware of (with the partial
exception of C++ where it is possible to use the same facilities as C,
just considerably more difficult).
In fact, most new languages seem to be *regressing* here. Both Perl and
Python were already fairly bad at this, Java bailed on the problem and
shoved it under the carpet of a version of static linking, and now Go is
even worse than they are and is explicitly embracing static linking.
Sigh.

Actually, if you look closely, you'll find that the traditional Java .jar
linking resolver precisely mirrors the behavior of the C linker on Solaris
from the same era (allows you to link dynamically, but requires top-level
objects to be linked at build time with all the recursive dependencies of
the libraries it uses). So it's not so much that they failed to learn from
C in that case, as that they failed to learn from the Free Unices of the
era.

But for Go, there's certainly no excuse. They simply have ignored the
problem by deciding that it doesn't matter to their deployment model,
leaving distributions high and dry.

[Florian Weimer]

Post by Steve Langasek
Actually, if you look closely, you'll find that the traditional Java
.jar linking resolver precisely mirrors the behavior of the C linker
on Solaris from the same era (allows you to link dynamically, but
requires top-level objects to be linked at build time with all the
recursive dependencies of the libraries it uses).

I don't think this is actually true. The Class-Path manifest
attribute has been around since almost forever (1.2 probably). It
serves the same purpose as the NEEDED attribute in ELF, and avoids the
need for dependencies to bubble up to the final link.

Most Java linking is done differently, probably because relatively few
people now about the Class-Path attribute. Others use other module
systems not part of the JDK, of course. But the Class-Path attribute
alone gets you very far.

[Matthew Woodcraft]

Post by Russ Allbery
I keep being tempted to go off on a rant about how we have all of
these modern, sophisticated, much more expressive programming
languages, and yet still none of them handle ABI versioning as well as
C does. Normal versioning problems that we just take for granted in C,
such as allowing the coexistence of two versions of the same library
with different ABIs on the system at the same time without doing the
equivalent of static linking, vary between ridiculously difficult and
completely impossible in every other programming language that I'm
aware of (with the partial exception of C++ where it is possible to
use the same facilities as C, just considerably more difficult).
In fact, most new languages seem to be *regressing* here.

I don't think it's as clear-cut as that.

Debian handles multiple versions of C libraries at _runtime_ well, but I
think its support for C libraries still leaves a good deal to be
desired: it doesn't let you install multiple versions of -dev packages,
and it doesn't provide much in the way of tools to help you manage
multiple versions of libraries-and-headers that you've installed outside
the packaging system.


I think, for someone who wants an OS for software development, and wants
or needs to program against library versions newer than those that
Debian ships, Debian has better support for some of the newer languages
than it does for C.

(For example, as I understand it Python's virtualenv/venv stuff lets you
express "I want to see the standard library shipped with Debian's
Python, but otherwise only the locally-installed libraries I specify".
That's annoying to do with C because all the headers are jumbled
together in /usr/include.)

-M-

[Neil Williams]

On Thu, 7 Feb 2013 20:16:18 +0000

Post by Matthew Woodcraft
I don't think it's as clear-cut as that.
Debian handles multiple versions of C libraries at _runtime_ well, but I
think its support for C libraries still leaves a good deal to be
desired: it doesn't let you install multiple versions of -dev packages,

That depends on the -dev package, many are versioned where that is
appropriate. I have libgtk-3.0-dev and libgtk2.0-dev installed,
amongst others. It all depends on how long it is likely to take for all
of the reverse dependencies to migrate.

It's a bit more work for the maintainer but a SONAME change in a
library still needs a trip through NEW, so a new source package vs a
few new binary packages isn't that hard. If there are packages which
aren't providing suitable migration paths, file bugs.

More often than not, there is simply no need to have libfoo1-dev and
libfoo2-dev - libfoo2 & it's libfoo-dev just live in experimental whilst
the reverse deps migrate to the new API.

Post by Matthew Woodcraft
(For example, as I understand it Python's virtualenv/venv stuff lets you
express "I want to see the standard library shipped with Debian's
Python, but otherwise only the locally-installed libraries I specify".
That's annoying to do with C because all the headers are jumbled
together in /usr/include.)

Not true. pkg-config isolates the headers between packages whilst
retaining dependencies and only the headers you specify get
included by the compiler anyway. .h files in /usr/include are the
exception, most are in package-specific sub-directories of /usr/include.

[Russ Allbery]

Post by Neil Williams

Post by Matthew Woodcraft
I don't think it's as clear-cut as that.
Debian handles multiple versions of C libraries at _runtime_ well, but
I think its support for C libraries still leaves a good deal to be
desired: it doesn't let you install multiple versions of -dev packages,

It can if upstream versions the headers. A lot of upstreams don't do
this, but some do, and C provides a perfectly reasonable facility for
doing so (add the API version to the include paths). Many common C
libraries, from Glib to APR, use this facility.

You certainly *can* break versioning in C, but my point is rather that C
provides all the facilities that you need to do this and they are in
reasonably common use in the C community. (It's common not to bother to
version APIs because it's usually reasonably easy to maintain API
compatibility if you care, since you can add new fields to structs and add
new interfaces without changing the API even if the ABI changes.)

ABI versioning is much more important than API versioning because ABI
versioning affects end users who just want to install packages and have
them work, whereas API versioning only affects developers who by
definition have access to the source and can probably update things.

One of the challenges of interpreted (or JIT-compiled) languages like Perl
and Python is that the API and the ABI are the same thing, which means
that you have to version the API in order to version the ABI. This is
unavoidable given the nature of the language, but is actually quite
awkward, since versioning the API generally requires that the source code
of all callers be aware of the versioning and update to use the new
version. With C libraries with ABI versioning but no API versioning (the
most common case), one often doesn't need to make any changes at all to
the caller to move to a newer version of the ABI.

Post by Neil Williams
More often than not, there is simply no need to have libfoo1-dev and
libfoo2-dev - libfoo2 & it's libfoo-dev just live in experimental whilst
the reverse deps migrate to the new API.

Right. Plus, if upstream isn't versioning the API, the -dev packages
won't be coinstallable, and thus are somewhat pointless.

I could see how one could go about supporting something like this in a
language like Perl or Python. It would require introducing the concept of
an API version at the language level, and the caller would be required to
declare the API version at the time it loads the module. (Experience in
the REST world, specifically with the Facebook API, says that allowing the
caller to omit the API version and providing some default is a bad idea
and produces fragile clients.) So, to take a Perl example (since that's
what I'm most familiar with), the current:

use Date::Parse 2.20;

where the number is the minimum required version of Date::Parse (which is
kind of useless since it doesn't set a maximum version, and even with
support for setting a maximum version is annoying because it requires the
caller know release versions instead of API versions), one would instead
have something like:

use_api Date::Parse 2;

which would load version 2 of the Date::Parse API.

Then, the module would declare what API versions it supports, and the
installation paths would have to be restructured so that modules providing
different versions would be co-installable. One way backward-compatible
way to do that in Perl would be to install Date::Parse as both:

Date/Parse/_API_2.pm
Date/Parse.pm

Old Perls would use the second path. New Perls, given the above
declaration, would only load the first path and would throw an error if it
didn't exist. After a transition, or with an install option, you could
drop the second path, and then you could co-install two versions of
Date::Parse that provided different APIs because the next API version
would install:

Date/Parse/_API_3.pm

and clients that did:

use_api Date::Parse 3;

would get that one instead. Note that this would also allow a single
source package to provide multiple versions of the API by just installing
multiple _API_n.pm modules.

This is all quite ugly, and is mostly just a thought experiment. I'm sure
one could devise better schemes. What surprises me is that no one has
done this in any of the newer languages. (You really do want support for
this built into the language, since you want a bunch of help constructing
the install paths, supporting the automatic logic behind use_api, and
ideally you want the language to be able to figure out whether you changed
APIs at least enough to warn you about it.)

[Paul Wise]

Post by Barry Warsaw
Speaking with many hats on, I think Debian Python has done a very admirable
job of integrating the Python ecosystem with Debian.

One of the pain points for users (I've had folks ask me this
face-to-face) with that stuff was site-packages vs dist-packages. With
your various Python hats on, can you explain why not just use
"packages" instead of "site-packages" and "dist-packages"? The right
way (IMO) would have been to put site packages in
/usr/local/lib/pythonX.Y/packages and dist ones in
/usr/lib/pythonX.Y/packages. Right now I have
/usr/local/lib/pythonX.Y/dist-packages and
/usr/lib/pythonX.Y/dist-packages, why is /usr/local dist-packages
instead of site-packages? /usr/local is clearly not the location for
distro installed packages.

Why did Debian have to invent /usr/share/pyshared and symlink farms in
/usr/lib/pythonX.Y instead of upstream having something like that in
the default install and search paths?

The location of .pyc files that are built at install time doesn't feel
FHS-correct to me, /var/cache/python/X.Y/ seems better.

Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

[Matthias Klose]

Post by Paul Wise

Post by Barry Warsaw
Speaking with many hats on, I think Debian Python has done a very admirable
job of integrating the Python ecosystem with Debian.

One of the pain points for users (I've had folks ask me this
face-to-face) with that stuff was site-packages vs dist-packages. With
your various Python hats on, can you explain why not just use
"packages" instead of "site-packages" and "dist-packages"?

Sure, anything else than "site-packages" would have worked.

Post by Paul Wise
The right
way (IMO) would have been to put site packages in
/usr/local/lib/pythonX.Y/packages and dist ones in
/usr/lib/pythonX.Y/packages. Right now I have
/usr/local/lib/pythonX.Y/dist-packages and
/usr/lib/pythonX.Y/dist-packages, why is /usr/local dist-packages
instead of site-packages? /usr/local is clearly not the location for
distro installed packages.

This came up several times (e.g. [1], [2]). See [3] for the rationale for the
naming and the extra directories in /usr/local.

Post by Paul Wise
Why did Debian have to invent /usr/share/pyshared and symlink farms in
/usr/lib/pythonX.Y instead of upstream having something like that in
the default install and search paths?

Having separate copies of .py files would have been an option. Splitting .so and
.py files across directories not, because having different directories for third
party packages in python2.x can break the import behaviour, does break split-out
debug information, doesn't work for gir files, and more. Please search back the
archives until 2003, if you are really interested in this.

Post by Paul Wise
The location of .pyc files that are built at install time doesn't feel
FHS-correct to me, /var/cache/python/X.Y/ seems better.

The location is correct if you include these files in the packages. So why
change it if you regenerate them? Byte-compilation and pyshared is only a means
to make pure python code independent of the python version. This is changed in
Python3 upstream [4]. Please address any outstanding issues upstream, then maybe
provide a backport.

Post by Paul Wise
Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

I don't have any issue to build packages using dh_python2 and dh_python3. Please
file a bug report if you do have such issues. It is my understanding that
dh_python2/dh_python3 are stable and should still be available in jessie, but
then using the underlying pybuild system. If you want to improve the tools,
please do. Sarcasm doesn't do any improvement by itself. Please join Piotr.

Matthias

[1] https://lists.debian.org/debian-python/2008/03/msg00021.html
[2] https://lists.debian.org/debian-python/2009/02/msg00002.html
[3] https://lists.debian.org/debian-devel/2009/02/msg00431.html
[4] http://www.python.org/dev/peps/pep-3147/

[Barry Warsaw]

Okay, fortunately, no bands are practicing tonight and no kids need homework
help, so let's see if I can answer some of these questions. :)

Post by Barry Warsaw
Speaking with many hats on, I think Debian Python has done a very admirable
job of integrating the Python ecosystem with Debian.

One of the pain points for users (I've had folks ask me this face-to-face)
with that stuff was site-packages vs dist-packages. With your various Python
hats on, can you explain why not just use "packages" instead of
"site-packages" and "dist-packages"?

Fundamentally, this comes down to a conflict between Python's historical
defaults and Debian's interpretation of the FHS. Let me just stipulate that
I'm not casting blame, or saying that anybody is doing anything wrong. I'm
not interested in that discussion, though I've had it many times. It is what
it is.

Old timers like me will remember the days when *nix systems reserved
/usr/local for stuff you downloaded and installed from source (i.e. most
everything on a usable system :). There was no /opt or FHS. This was
codified in the first auto-configuration scripts. I don't remember when
Python adopted the configure regime, but as long as I can remember (going back
at least to 1994), a default build-from-source of Python installed into
/usr/local. When site-packages was added,
/usr/local/lib/pythonX.Y/site-packages was the most logical place to put it.

Predating my involvement with Debian, I remember problem reports where
developers of Python, and others who install from source for various reasons,
would break their systems when they used the wrong Python executable to
install third party packages outside of the Debian packaging system. This was
because Debian allowed /usr/local/lib/pythonX.Y/site-packages to be used for
third party packages installed outside the Debian packaging system, using the
*system* Python, i.e. /usr/bin/python. This meant that if I installed
something for /usr/bin/python into /usr/local/lib/pythonX.Y/site-packages it
could easily break my /usr/local/bin/python, and possibly vice versa.

I think it was at a Pycon years ago that Matthias and I discussed this
problem. At the time (and probably still so), it didn't seem like either
Debian or Python was going to change its policy, so we had to find a way to
avoid the conflict and let both communities live in peace. Matthias's
solution was the use of dist-packages for Debian's system Python, which would
be ignored by a /usr/local/bin Python. Also, system Python would ignore
/usr/local/lib/pythonX.Y/site-packages (but not .../dist-packages), thus
avoiding all conflict. It seemed elegant at the time, and I still think this
is a reasonable compromise, even though it does cause some tooling problems,
which have to be patched in Debian.

The right way (IMO) would have been to put site packages in
/usr/local/lib/pythonX.Y/packages and dist ones in
/usr/lib/pythonX.Y/packages. Right now I have
/usr/local/lib/pythonX.Y/dist-packages and /usr/lib/pythonX.Y/dist-packages,
why is /usr/local dist-packages instead of site-packages? /usr/local is
clearly not the location for distro installed packages.

That was my position, i.e. that system Python shouldn't have any path from
/usr/local on sys.path, but that was very strongly (at the time) disputed by
Debian users. To be fair, the Debian users at the time (and maybe still do)
say that the right solution is for a default from-source build of Python to
install into /opt/local and not /usr/local, but again, that would conflict
with years of established use by upstream.

That's the historical background as I remember it anyway.

Why did Debian have to invent /usr/share/pyshared and symlink farms in
/usr/lib/pythonX.Y instead of upstream having something like that in
the default install and search paths?

Because upstream doesn't really care (or didn't until my PEPs 3147 and 3149 in
Python 3.2) about multiple versions of packages co-existing in harmony, and
because upstream Python requires .pyc files to live next to (FSVO, see below)
the .py files.

Debian was the first place that I recall where multiple versions of Python
could be co-installed. Let's say you have both Python 2.6 and 2.7 installed,
and you have a module called foo.py that is source-level compatible with both.
The problem is that Python has never guaranteed that .pyc files would be
compatible across Python versions. It's never said they wouldn't be, but in
practice the byte code cached in .pyc files always changes, due to new
features or bug fixes in the interpreter between major version numbers.

So in Debian you have a situation where you want to share foo.py across all
supported and installed Pythons, but where you cannot share .pyc files because
they aren't compatible. You want to share .py files 1) to keep package sizes
smaller, 2) to consume less disk space, 3) because you don't actually know
which versions of Python the target system has installed. Just because
version W of Debian supports PythonX.Y and PythonA.B doesn't mean your system
has both installed, so you'd rather not pay for the penalty of packaging up
two identical foo.py's for both of them, just because they'll live in
different locations on the file system. And they'd have to live in different
paths because of Python's requirement for nearby .pyc files combined with
cross-version incompatibility of .pyc files.

(Aside: there's no getting around paying this cost for extension modules since
they are binary .so files, but there are *way* fewer of these than
pure-Python, theoretically cross-version compatible source files.)

There have been several regimes to manage this, all of them to the best of my
knowledge using symlink farms to manage the sharing of .py files with the
version-specific-ness of .pyc files. IMHO, dh_python2 takes the best approach
to this, but previous regimes such as python-support, and probably
python-central are still in use.

This was finally solved by my work on PEPs 3147 and 3149, which introduced the
__pycache__ directory in Python 3.2 and tagged .so and .pyc file names.
(Aside: __pycache__ isn't strictly necessary to support this, but was a nice
additional feature suggested by Guido.)

Now in the Python 3 world, you *can* co-install multiple versions and even
though the .pyc and .so files are still version-specific, they can co-exist
peacefully. PythonX.Y will only try to load foo.cpython-XY.pyc and ignore
foo.cpython-AB.pyc, instead of overwriting it, which would have happened
before. Unfortunately, this work came too late to be included for Python 2,
so we still need the symlimk farms for that (obsolete <wink>) version.

But if you look at how we do Python 3 packages now, you'll see
/usr/lib/python3/dist-packages with shared .py source files, version-specific
.pyc inside __pycache__ directories, and ABI tagged .so files co-existing with
no symlink farms. Three cheers for progress!

(Aside: you'll still see a /usr/lib/python3.X but that's for version-specific
stdlib only.)

The location of .pyc files that are built at install time doesn't feel
FHS-correct to me, /var/cache/python/X.Y/ seems better.

It probably is, but upstream Python can only handle .pyc files living next to
(or in the post PEP 3147 world, very nearby) their .py files. I suppose you
could use Python 3.3's importlib to write an importer that codified this
policy, but leaving aside whether it would be worth it, you'd probably have a
similar (or worse) tooling problem as with dist-packages, since there's
probably many packages that assume the .pyc lives near the .py (and some have
even had bugs caused by the PEP 3147 reorganization alone, not all of which
are fixed I'm sure).

Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

Not exactly. dh_python2 and dh_python3 are really good IMHO, but one problem
is that while dh has a lot of helpers to make it easy to write d/rules files
for common case setup.py based Python 2 packages, it doesn't know anything
about Python 3. Take a look at all the overrides you have to add for
libraries that are both Python 2 and 3 compatible, as described in
http://wiki.debian.org/Python/LibraryStyleGuide

Among the things that pybuild improves is dh support for Python 3, so you
really can almost always write just a 3 line d/rules file, even for libraries
that support both Python 2 and 3, with automatic running of unittests, etc.
That's win enough, IMHO.

Piotr can perhaps speak in more detail about it, but pybuild is more ambitious
still, and IIUC, really just builds on top of dh_python2 and dh_python3 by
supporting several different upstream build systems (default is to
auto-detect, e.g. distutils-based, configure-based, etc.) with lots of
overrides and customization possible. For example, there are several ways
that a library's test suite can be invoked so having good auto-detection of
that, along with convenient ways to customize it are important.

The pybuild manpage has all the gory details, but I think with pybuild, we're
finally able to promote really easy to write d/rules files for the majority of
Python packages across both major version stacks.

Hope that helps!
-Barry

[Steve Langasek]

Post by Paul Wise
Why did Debian have to invent /usr/share/pyshared and symlink farms in
/usr/lib/pythonX.Y instead of upstream having something like that in
the default install and search paths?

This is all resolved now in python3. There is no more /usr/share/pyshared
for python3, and all packages install directly to
/usr/lib/python3/dist-packages.

It took a while to get there, which is regrettable, but when such changes
need to be done upstream in order to be done right, and you have a large
established user base like python does, these things can be slow moving.

Post by Paul Wise
The location of .pyc files that are built at install time doesn't feel
FHS-correct to me, /var/cache/python/X.Y/ seems better.

I think this is a minor point, honestly. Also, I believe there are other
precedents in Debian for this kind of install-time bytecode compilation
(emacs lisp, IIRC?).

Post by Paul Wise
Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

pybuild is a necessary adjunct to dh_python3 that provides the dh(1) build
system integration, not an alternate build helper tool. I don't think
"breeding like rabbits" is anywhere near the mark.

[Chow Loong Jin]

Post by Steve Langasek

Post by Paul Wise
Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

pybuild is a necessary adjunct to dh_python3 that provides the dh(1) build
system integration, not an alternate build helper tool. I don't think
"breeding like rabbits" is anywhere near the mark.

Well, relative to other languages, I think Python's had the most changes with
regards to build helper tools -- there was dh_pycentral, and dh_pysupport in the
past which did more or less the same thing in different ways, and now we have
dh_python2, and dh_python3. In contrast, Mono stuff have only had the dh_cli*
set of things, and Java only had the javahelper bunch of things.

[Russ Allbery]

Post by Chow Loong Jin
Well, relative to other languages, I think Python's had the most changes
with regards to build helper tools -- there was dh_pycentral, and
dh_pysupport in the past which did more or less the same thing in
different ways, and now we have dh_python2, and dh_python3. In contrast,
Mono stuff have only had the dh_cli* set of things, and Java only had
the javahelper bunch of things.

Java also has Maven helper tools (maven-debian-helper and
maven-repo-helper).

The iterating on Python tools has partly been because there were various
problems with doing the right thing and various possible workarounds for
integration issues that were then resolved in part by upstream changing
things there so that Debian could do what we need to do. I don't think
the quantity of tools is itself a sign of a problem. (There have been
other things around Python package maintenance that have been problems,
though, but seem to be getting better.)

[Steve Langasek]

Post by Chow Loong Jin

Post by Steve Langasek

Post by Paul Wise
Debian's Python build helper tools are still breeding like rabbits,
there is a new one in experimental. I guess because the current ones
dh_python2/dh_python3 don't handle packages that contain only code
that runs on both python2 and python3 without changes.

pybuild is a necessary adjunct to dh_python3 that provides the dh(1) build
system integration, not an alternate build helper tool. I don't think
"breeding like rabbits" is anywhere near the mark.

Well, relative to other languages, I think Python's had the most changes
with regards to build helper tools -- there was dh_pycentral, and
dh_pysupport in the past which did more or less the same thing in
different ways, and now we have dh_python2, and dh_python3.

Yes, but this is not proliferation. There's one standard tool for python2 -
dh_python2 - and one for python3 - dh_python3. (The languages have
sufficiently different build-time and install-time requirements that it
makes sense to have one for each.)

Post by Chow Loong Jin
In contrast, Mono stuff have only had the dh_cli* set of things, and Java
only had the javahelper bunch of things.

Heh. No, there's javahelper, and maven-debian-helper, and
maven-repo-helper, and I'm pretty sure there was another one for another
repo system besides maven.

[Chow Loong Jin]

Post by Steve Langasek
[...]

Post by Chow Loong Jin
Well, relative to other languages, I think Python's had the most changes
with regards to build helper tools -- there was dh_pycentral, and
dh_pysupport in the past which did more or less the same thing in
different ways, and now we have dh_python2, and dh_python3.

Yes, but this is not proliferation. There's one standard tool for python2 -
dh_python2 - and one for python3 - dh_python3. (The languages have
sufficiently different build-time and install-time requirements that it
makes sense to have one for each.)

Agreeably so, but I don't think the justification applies for having both
python-support and python-central. Thank $deity those have been replaced by
dh_python2. A significant number of packages still show up in reverse-depends
-b, though.

Post by Steve Langasek

Post by Chow Loong Jin
In contrast, Mono stuff have only had the dh_cli* set of things, and Java
only had the javahelper bunch of things.

Heh. No, there's javahelper, and maven-debian-helper, and
maven-repo-helper, and I'm pretty sure there was another one for another
repo system besides maven.

Hmm, okay, I hadn't known about those. I guess that makes the Mono build helpers
the outlier then. Why can't everyone else be as awesome as us and have proper
Debhelper-based build helpers written in Perl? ;-)

[Jon Dowland]

For cpan there is even dh-make-perl. The solution then is to make
equivelant scripts for other languages. The solution is NOT to use some
other package installation system.

Although I've never used dh-make-perl myself, I'm lead to believe that it
is perhaps *the* most successful tool of its type (that is, of things that
create .debs from packages in an alternative repository system like CPAN,
gems, cabal, etc.), and that it works as reliably as it does (which as Russ
points out is not 100% by any means) by relying on data from CPAN. So it's
hard to use it as an argument against such external package systems.

[Russ Allbery]

Post by Jon Dowland
Although I've never used dh-make-perl myself, I'm lead to believe that
it is perhaps *the* most successful tool of its type (that is, of things
that create .debs from packages in an alternative repository system like
CPAN, gems, cabal, etc.), and that it works as reliably as it does
(which as Russ points out is not 100% by any means) by relying on data
from CPAN. So it's hard to use it as an argument against such external
package systems.

Indeed, that's exactly right. dh-make-perl works as well as it does in
part because upstream has a fairly good package management system with
explicit dependencies and good package metadata, all of which dh-make-perl
can download and make use of.

I used the tool back before integration with that information was
available, and it required much more work to build the package. Now, it
can mostly do things like figure out the dependencies without a lot of
human assistance.

[Florian Weimer]

I drew a different conclusion from Ian's messages the thread you
mentioned (see the quotes below). Apparently, one *can* build shared
libraries using gccgo, but they are not currently usable using dlopen().
My impression was that this means that regular use of shared libraries
*is* possible with gccgo.

The problem with shared libraries in Go is that the API guarantee for
Go itself allows changes which break the shared library ABI, such as
adding struct fields (thus changing struct sizes and offsets). Sizes
and offsets are compiled directly into the code, without a relocation.
Even with shared libraries, we will have to recompile all dependent
packages in many cases.

This binary compatibility issue was addressed in the Java front end,
which had similar issues.

This is not unlike C or C++, of course, but some library authors there
have a more stringent attitude towards ABI compatibility and build API
change guidelines based on that. (Technically, we even have to
recompile all library packages when we make major changes to eglibc
because the static libraries are tied to a very specific eglibc
version because the symbols aren't bound to versions at that point.
So we probably shouldn't complain to loudly about other languages not
getting this completely right.)

Fedora hasn't got a solution for this, either, FWIW. OpenSUSE seems
to have support in their build infrastructure for soname bumps, which
could be used for this as well.