Boost logo

Boost-Build :

From: Ali Azarbayejani (ali_at_[hidden])
Date: 2003-06-02 08:57:15


Vladimir Prus wrote:
> The question really was: why build system should handle patching --- why not
> extract/patch once, and live with it. You've answered my question.

That was actually my first design (to extract and patch once), but
occasionally a new or revised patch gets checked in and you want users
to be able simply to update from the repository and re-build and end up
with the right answer. This requires proper dependencies.

>>The user needs to convey the information that 'foo/b' requires patching,
>>so such a declaration is necessary. The only way I know to represent
>>the declared information in the BBv2 system is by creating a main target.
> Since we're speaking about the right way, not what's possible currently, I
> might imagine something like
> patch-tree patch-foo : foo patches ;
> So, when the user gets new patches, he would run
> bjam patch-foo ;
> and the sources will be updated.

Yes, fact this is the high-level rule that I already use, and Dave
proposed the same solution. But it's desirable to have a lower-level
single-file patch rule as well that doesn't depend upon a tree
structure...there may be occasion to patch a file in some other context
and you wouldn't want to have to build a single-file tree just for that

So "patch-tree" can be implemented in terms of "patch-file". For
modularity and maintenance reasons, this is desirable. Thus with the
"patch-tree" high-level rule, the individual files are not expressed
explicitly, but they would still end up as main-targets if "patch-tree"
is implemented with "patch-file". Of course, you could implement
"patch-tree" as its own updating rule, but that would defy modular
design and present maintenance and extensibility issues.

> The situation where "foo/b" is both a source and a target resembles me what I
> had with gettext support (that I've did locally for V1). The translations
> file is converted into machine form. But the translation files is itself
> updated from sources. The way I did it, translation files was updated from
> sources every time, and that was troublesome: I don't want them updated
> unless I'm willing to correct translations.
> So, I've decided that in V2, translations will only be updated after explicit
> request. In the case above, it probably makes some sense too. Imagine:
> user has locally modified "foo/b", and new version of patch for the file
> arrives. He rebuilds and "foo/b" is patched. Original version is stored.
> Another version of patch arrives, he does the same again, and now the original
> version of file is gone. I'm not sure this is real problem

Actually, this is a real problem because patches are always stored as a
patch from the original file in the tar. The solution is that if a
patch changes, the file must be re-extracted and then re-patched.

This actually presented some issues with BBv2 as well, which I had to
work around. The extract action has to depend upon the patches (so that
if a patch changes, the original files get re-extracted).

At one time, I asked why feature <dependency> can't refer to a plain
file (rather than just main targets) and you asked me for a use case.
This is the use case. I need to say something like

AllPatches = foo/b foo/d

extract foo : foo.tar.gz : <dependency>patches/$(AllPatches) ;

But, as I now know, <dependency> isn't intended for establishing such
relationships, so I had to make all patches into main targets. And
that's another (bad) reason why the patches need to be main targets.

This use case illustrates why I propose the following requirements on
the system:

(1) A source file should be usable wherever (almost?) a main-target or
intermediate file is usable. In particular, you should be able to
express a dependency on either a main-target or a file.

(2) Source files can be in subdirectories (or sibling directories, or
any directory for that matter); they should be expressable naturally
using pathnames with slashes in them. N.B. This patching business is
not a degenerate example either; of the C++ libraries I have from my
users, a majority of them use subdirectories in a single project to
organize their source files for a single library.

(3) Consequently, Target IDs should support references to both
main-targets and files, and should support pathnames and ids with slashes.

>>Also, the extracted or patched file must be a main target because other
>>external projects will want to use these extracted and patched files and
>>there's no way to properly refer to them if they are not main-targets.
> That's an argument! I still wonder, however:
> 1. Why other project need to access patches files. Can you explain in more
> detail?

Yes. In my modular structure, I have one kind of module ("Package
Module") that is responsible for unpacking and patching elements of a
tarfile and another kind of module ("Software Module") that is
responsible for building libraries and/or executables. A Software
Module doesn't care whether its source files come directly from the
repository or whether they come from a Package Module. Thus, to build a
library from source that comes from a packaged tarfile, a Software
Module must reach into the Package Module to obtain the final sources
(extracted and patched).

In BB terminology, Package Module and Software Module are different
kinds of "project". And they are not necessarily related (although
currently, the related Software Modules belong to the Package Module).

> 2. The same effect can be achieved by subprojects: i.e. placing Jamfile in foo
> and declared main target "b" there. It might not be ok for your use case. is not OK. I don't have control over "foo" or its
contents...that comes straight from an unmodified third-party tarfile.

> The
> problem is that "foo/b" looks like you declare a target in different
> directory or different project.

Aha! I agree with you completely, but I believe this is a problem with
target id syntax.

> Further, there's another question: if you
> possible have (using @ to separate target name)
> location_at_foo/b
> location/foo_at_b
> is there potential for confusion?

I don't think there's confusion.

The first one is target "foo/b" in project "location/Jamfile" or file
"foo/b" relative to project "location/Jamfile" (i.e. "location/foo/b").

The second one is target "b" in project "location/foo/Jamfile" or file
"b" relative to project "location/foo/Jamfile" (i.e. "location/foo/b").

This seems crystal clear to me. I don't think that it is a problem that
file "location/foo/b" can potentially be referred to in several ways
(also "._at_location/foo/b"), implying membership of a source file in
multiple projects at once. It would be unusual to have
"location/Jamfile" AND "location/foo/Jamfile" AND be referring to file
"location/foo/b" as a source of both projects, but, on the other hand,
it is conceivable that you would WANT to have a source serve two
separate projects and the notation is crystal clear.

> 3. Do users define all those main target manually? If not, probably the best
> solution is to define one main-target to apply all patches, and make external
> targets depend on specific file, not on main target. That would still require
> separating target and project with something else than "@".

Currently the targets representing the source files (patched or
unpatched) are listed manually. There really is no way around
this...someone has to indicate what the source files of interest are.
The patches, on the other hand, are stored in a tree, so they are
detected and applied automatically.

Your solution is along the lines of what Dave proposed. Unfortunately,
they probably all need to be main-targets because all of the "source
files" are technically built targets (technically the "sources" of the
Package Module are the tar.gz file and the patches). The interesting
part, however, is that if the external project can simply refer to them
as "package_at_foo/b", the external project does not need to know whether
"foo/b" is (a) a plain file, (b) a main-target representing an extracted
(unpatched) file, or (c) a main-target representing an extracted and
patched file.


