Step One: Suggested File System Layout

For the DragonFly builds I'm using:

- /bulklog for the bulk meta data and log files of each package

- /scratch as WRKOBJDIR

- /distfiles as DISTDIR

- /packages as PACKAGES

- separate physical file systems for /var and /usr/pkg

All file systems but those are read-only, /tmp and /dev need platform

specific handling either as tmpfs or link. ptyfs helps for /dev.

For parallel builds, /bulklog, /packages and /usr/pkgsrc are supposed to

be shared (null mounts, NFS). /usr/pkg and PKG_DBDIR need to be

per-node and /scratch should be per-node.

The defaults scripts will remove /usr/pkg and PKG_DBDIR on the master

node as well. When using chroot sandboxes for the build, the master can

share a sandbox with one of the build instances.

Step Two: Preparing pbulk prefix

Install pkgtools/pbulk and possibly helper programs like screen into

LOCALBASE=/usr/pkg_bulk, PKG_DBDIR=/usr/pkg_bulk/pkgdb. A full

bootstrap is the recommended approach. Copy /usr/pkg_bulk to the client

nodes for parallel builds.

The initial bootstrap and install may be performed with the following commands:

cd /usr/pkgsrc/bootstrap

./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb

If you have already performed a bootstrap into /usr/pkg_bulk then you may need

to remove that directory (rm -fr /usr/pkg_bulk) prior to starting the new

bootstrap.

pbulk may be installed into /usr/pkg_bulk with the following:

cd /usr/pkgsrc/pkgtools/pbulk

env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake install

The mk.conf file in /usr/pkg_bulk/etc/mk.conf should already have the

correct LOCALBASE and PKG_DBDIR settings.

Step Three: Configure pbulk

(a)

Setup general build parameters in mk.conf. Below is an example

fragment to add. On NetBSD this is typically /etc/mk.conf. For

non-NetBSD systems we will create a mk.conf in a temporary location,

/tmp/mk.conf for example, and include it in the bootstrap that will

be built in step (b).

(b) [not for NetBSD]

Build a binary bootstrap kit and override the included mk.conf with the

file created in (a). Compress the tarball with gzip. This is a new

bootstrap that is different from the one created in Step Two above.

This bootstrap kit is created by the following:

cd /usr/pkgsrc/bootstrap

./bootstrap --mk-fragment /tmp/mk.conf \

--gzip-binary-kit bootstrap_kit.tar.gz

Note that we have specified the make file fragment, /tmp/mk.conf, created

in step (a) above.

(c)

Edit /usr/pkg_bulk/etc/pbulk.conf:

- For NetBSD:

- bootstrapkit should be empty.

- make must be changed to /usr/bin/make.

- For non-NetBSD:

- bootstrapkit should be changed to the tarball in (b). Both master

and client nodes access this, so keep it in a shared location.

- master_mode=yes needs a proper list of client nodes.

- base_url goes into the report mail, the rsync targets are used

for uploads.

- for limited bulk builds, set limited_list to a file with one

location per line ("x11/gtk2" for example). Packages that are required

as dependencies will be added automatically.

Step Four: Running pbulk

/usr/pkg_bulk/bin/bulkbuild runs the full build. You may wish to run

this inside of screen (misc/screen) since a full bulk build may take

a very long time.

/usr/pkg_bulk/libexec/pbulk/{pre-build,scan,build,report,upload} are the

individual build phases. They depend on the execution of the phase

before. Exceptions are report and upload which both depend on build.

/usr/pkg_bulk/bin/bulkbuild-restart can be used to resume an aborted

build. If it failed after the actual build phase, report and upload should

be called individually though.

--- sample mk.conf fragment ---

WRKOBJDIR = /scratch

PKGSRCDIR = /usr/pkgsrc

DISTDIR = /distfiles

PACKAGES = /packages

FAILOVER_FETCH= yes

X11_TYPE= modular

SKIP_LICENSE_CHECK= yes

ALLOW_VULNERABLE_PACKAGES= yes

PKG_DEVELOPER?= yes

####

PBULK_CACHE_DIRECTORY when set collects the build options of packages,

so these need to be extracted only once per pbulk run. This may be

a directory shared between pbulk clients (like bulklog) but doesn't

have to be.

You need to clear it between pbulk runs when you updated pkgsrc,

or packages that change default options or their dependents may stop

building.