Commits

Anonymous committed cd00ea7

package audits should pass pkglint
incrementals should work better

Comments (0)

Files changed (16)

 	    Getting started with the Userland Consolidation
 
 
-Building the bits
+Getting Started
+
+    This README provides a very brief overview of the gate, how to retrieve
+    a copy, and how to build it.  Detailed documentation about the Userland
+    gate can be found in the 'doc' directory.  Questions or comments about
+    the gate can be addressed to userland-discuss@opensolaris.org.
+
+Overview
 
     The Userland consolidation maintains a Mercurial gate at
-    
+
         ssh://anon@hg.opensolaris.org//hg/userland/gate
-        
+
     This gate contains build recipies, patches, IPS manifests, etc. necessary
     to download, prep, build, test, package and publish open source software.
-    In order to build the contents of the Userland gate, you need to clone it.
-    Since you are reading this, you probably already have, but in any event
-    you can do so with the following command
+    The build infrastructure is similiar to that of the SFW consolidation in
+    that it makes use of herarchical Makefiles which provide dependency and
+    recipe information for building the components.  In order to build the
+    contents of the Userland gate, you need to clone it.  Since you are
+    reading this, you probably already have.
+
+Getting the Bits
+
+    As mentioned, the gate is stored in a Mercurial repository.  In order to
+    build or develop in the gate, you will need to clone it.  You can do so
+    with the following command
     
-    $ hg clone ssh://anon@hg.opensolaris.org//hg/userland/gate /scratch/clone
-    
-    In order to build the bits either individually or collectively, you must
-    set the WS_TOP environment variable to point to the top of your workspace.
-    
-    $ export WS_TOP=/scratch/clone
-    
-    To build and publish the entire contents of the gate, you can use
-    
-    $ cd /scratch/clone
-    $ gmake publish
-    
-    To build and publish a specific component you need to initialize the
-    workspace by building the tools and creating a repository to publish
-    your results in.  The easiest way to do this is to
-    
-    $ cd /scratch/clone
-    $ gmake setup
-    
-    Once you have initialize the the workspace, you can build individual
-    components by
-    
-    $ cd /scratch/clone/components/(component)
-    $ gmake publish
-    
-    All of the bits are are built will be published to the repository created
-    by the setup step (file:///scratch/clone/repo/)  If you build the entire
-    contents of the gate, individual build logs for each component will be
-    located at /scratch/clone/logs/(target):(component).log
+      $ hg clone ssh://anon@hg.opensolaris.org//hg/userland/gate /scratch/clone
 
+    This will create a replica of the various pieces that are checked into the
+    source code management system, but it does not retrieve the community
+    source archives associated with the gate content.  To download the
+    community source associated with your cloned workspace, you will need to
+    execute the following:
 
+      $ export WS_TOP=/scratch/clone
+      $ cd /scratch/clone/components
+      $ gmake download
+
+    This will use GNU make and the downloading tool in the gate to walk through
+    all of the component directories downloading and validating the community
+    source archives from the gate machine or their canonical source repository.
+
+    There are two variation to this that you may find interesting.  First, you
+    can cause gmake(1) to perform it's work in parallel by adding '-j (jobs)'
+    to the command line.  Second, if you are only interested in working on a
+    particular component, you can change directories to that component's
+    directory and use 'gmake download' from that to only get it's source
+    archive.
+
+Building the Bits.
+
+    You can build individual components or the contents of the entire gate.
+    Regardless of how you build the gate, you must set WS_TOP in the calling
+    environment to point to the top of your workspace. Ex:
+
+      $ export WS_TOP=/scratch/clone
+
+  Component build
+
+    If you are only working on a single component, you can just build it using
+    following:
+
+      setup the workspace for building components
+
+        $ cd ${WS_TOP}/components ; gmake setup
+
+      build the individual component
+
+        $ cd (component-dir) ; gmake publish
+
+  Complete Top Down build  
+
+    Complete top down builds are also possible by simply running
+
+      $ cd ${WS_TOP}/components
+      $ gmake package-install
+
+    The 'package-install' target will build each component, publish it to the
+    workspace IPS repo and install it in the running environment.  As a result,
+    it is strongly recommended that you only perform complete top down builds
+    in a zone.  Tools to help facilitate build zone creation will be integrated
+    shortly.  If the zone you create to build your workspace in does not have
+    networking enabled, you can pre-download any community source archives into
+    your workspace from the global with:
+
+      $ cd ${WS_TOP}/components
+      $ gmake download
+
+  You can add parallelism to your builds by adding '-j (jobs)' to your gmake
+  command line arguments.
+
+  The gate should only incrementally build what it needs to based on what has
+  changed since you last built it.

components/Makefile

 include components.mk
 
 
-download:	TARGET = download
-prep:		TARGET = prep
-build:		TARGET = build
-install:	TARGET = install
-publish:	TARGET = publish
-validate:	TARGET = validate
-clean:		TARGET = clean
-clobber:	TARGET = clobber
+download:		TARGET = download
+prep:			TARGET = prep
+build:			TARGET = build
+install:		TARGET = install
+publish:		TARGET = publish
+install-packages:	TARGET = install-packages
+validate:		TARGET = validate
+clean:			TARGET = clean
+clobber:		TARGET = clobber
 prep build install publish:		TEMPLATE_ZONE=$(ZONE)
 prep build install publish:		LOG = >$(WS_LOGS)/$@.$(TARGET).log 2>&1
 
 .DEFAULT:	publish
 
-download prep build install publish validate: $(COMPONENT_DIRS)
+download prep build install publish install-packages validate: $(COMPONENT_DIRS)
 
 COMPONENT_DIRS.nosetup =	$(COMPONENT_DIRS:%=%.nosetup)
 
 	echo "export PATH WS_TOP" >>$@
 
 $(COMPONENT_DIRS):	setup FORCE
-	@cd $@ ; echo "$(TARGET) $@" ; \
+	@cd $@ && echo "$(TARGET) $@" && \
 	 $(BASS_O_MATIC) $(TEMPLATE_ZONE:%=--template-zone %) \
-	 		 $(@:%=--component %) --make $(TARGET) $(LOG)
+			 $(@:%=--component %) --make $(TARGET) $(LOG)
 
 $(COMPONENT_DIRS.nosetup):	FORCE
-	@cd $(@:%.nosetup=%) ; echo "$(TARGET) $(@:%.nosetup=%)" ; \
+	@cd $(@:%.nosetup=%) && echo "$(TARGET) $(@:%.nosetup=%)" && \
 	 $(BASS_O_MATIC) $(TEMPLATE_ZONE:%=--template-zone %) \
-	 		 $(@:%.nosetup=--component %) --make $(TARGET) $(LOG)
+			 $(@:%.nosetup=--component %) --make $(TARGET) $(LOG)
 
 # depends.mk is auto-generated by the build tools, bass-o-matic.py in particular
 depends.mk:	components.mk

components/bind/Makefile

 COMPONENT_NAME=		bind
 COMPONENT_VERSION=	9.7.0-P2
 IPS_PKG_NAME =		network/dns/bind
-COMPONENT_CLASSIFICATION="Applications/Internet utility"
+COMPONENT_CLASSIFICATION="Applications/Internet"
 COMPONENT_DESCRIPTION=	"Berkeley Internet Name Domain server and utilities"
 COMPONENT_SRC=		$(COMPONENT_NAME)-$(COMPONENT_VERSION)
 COMPONENT_PROJECT_URL =	http://www.isc.org/software/bind

components/bind/bind.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license COPYRIGHT license="ISC"
 dir path=etc
 dir path=usr
 link path=usr/lib/liblwres.so.60 target=liblwres.so.60.0.0
 link path=usr/sbin/named-compilezone target=named-checkzone
 link path=usr/share/man/man8/named-compilezone.8 target=named-checkzone.8
-depend fmri=pkg:/SUNWcs@0.5.11-0.143 type=require
-depend fmri=pkg:/library/libxml2@2.7.6-0.143 type=require
-depend fmri=pkg:/library/security/openssl@0.9.8.14-0.143 type=require
-depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require
-depend fmri=pkg:/system/library/math@0.5.11-0.143 type=require
-depend fmri=pkg:/system/library@0.5.11-0.143 type=require
+depend fmri=pkg:/library/libxml2 type=require
+depend fmri=pkg:/library/security/openssl type=require
+depend fmri=pkg:/library/zlib type=require
+depend fmri=pkg:/system/library/math type=require
+depend fmri=pkg:/system/library type=require
 
 
 

components/file/file.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license COPYING license="BSD-like"
 dir path=usr
 dir path=usr/bin
     variant.arch=$(ARCH)
 link path=usr/lib/libmagic.so target=libmagic.so.1.0.0
 link path=usr/lib/libmagic.so.1 target=libmagic.so.1.0.0
-depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require
-depend fmri=pkg:/system/library@0.5.11-0.143 type=require
+depend fmri=pkg:/library/zlib type=require
+depend fmri=pkg:/system/library type=require
 
 
 

components/lcms/Makefile

 #
 # Copyright (c) 2010, Oracle and/or it's affiliates.  All rights reserved.
 #
+
 include ../../make-rules/shared-macros.mk
 
 COMPONENT_NAME=		lcms
 REQUIRED_FILE +=	/usr/lib/$(MACH64)/libtiff.so
 REQUIRED_FILE +=	/usr/lib/libjpeg.so
 REQUIRED_FILE +=	/usr/lib/libtiff.so
+
+include ../../make-rules/depend.mk

components/lcms/lcms.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license COPYING license="LCMS"
 dir path=usr
 dir path=usr/bin
 file path=usr/bin/wtpt variant.arch=$(ARCH)
 file path=usr/include/lcms/icc34.h
 file path=usr/include/lcms/lcms.h
-file path=usr/lib/$(MACH64)/liblcms.la variant.arch=$(ARCH)
+#file path=usr/lib/$(MACH64)/liblcms.la variant.arch=$(ARCH)
 file path=usr/lib/$(MACH64)/liblcms.so.1.0.19 variant.arch=$(ARCH)
 file path=usr/lib/$(MACH64)/pkgconfig/lcms.pc variant.arch=$(ARCH)
-file path=usr/lib/liblcms.la variant.arch=$(ARCH)
+#file path=usr/lib/liblcms.la variant.arch=$(ARCH)
 file path=usr/lib/liblcms.so.1.0.19 variant.arch=$(ARCH)
 file path=usr/lib/pkgconfig/lcms.pc
 file path=usr/share/man/man1/icc2ps.1
     variant.arch=$(ARCH)
 link path=usr/lib/liblcms.so target=liblcms.so.1.0.19
 link path=usr/lib/liblcms.so.1 target=liblcms.so.1.0.19
-depend fmri=pkg:/image/library/libjpeg@0.5.11-0.143 type=require
-depend fmri=pkg:/image/library/libtiff@0.5.11-0.143 type=require
-depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require
-depend fmri=pkg:/system/library/math@0.5.11-0.143 type=require
-depend fmri=pkg:/system/library@0.5.11-0.143 type=require
+depend fmri=pkg:/image/library/libjpeg type=require
+depend fmri=pkg:/image/library/libtiff type=require
+depend fmri=pkg:/library/zlib type=require
+depend fmri=pkg:/system/library/math type=require
+depend fmri=pkg:/system/library type=require
 
 
 

components/libmng/libmng.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license LICENSE license="BSD-Like"
 dir path=usr
 dir path=usr/include
     variant.arch=$(ARCH)
 link path=usr/lib/libmng.so target=libmng.so.1.0.0
 link path=usr/lib/libmng.so.1 target=libmng.so.1.0.0
-depend fmri=pkg:/image/library/libjpeg@0.5.11-0.146 type=require
-depend fmri=pkg:/library/lcms@1.19-0.146 type=require
-depend fmri=pkg:/library/zlib@1.2.3-0.146 type=require
-depend fmri=pkg:/system/library/math@0.5.11-0.146 type=require
-depend fmri=pkg:/system/library@0.5.11-0.146 type=require
+depend fmri=pkg:/image/library/libjpeg type=require
+depend fmri=pkg:/library/lcms type=require
+depend fmri=pkg:/library/zlib type=require
+depend fmri=pkg:/system/library/math type=require
+depend fmri=pkg:/system/library type=require

components/pycups/pycups.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license COPYING license="GPLv2"
 dir path=usr
 dir path=usr/lib
 file path=usr/lib/python2.6/vendor-packages/64/cups.so variant.arch=$(ARCH)
 file path=usr/lib/python2.6/vendor-packages/cups-1.0-py2.6.egg-info
 file path=usr/lib/python2.6/vendor-packages/cups.so variant.arch=$(ARCH)
-depend fmri=pkg:/library/print/cups-libs@1.4.2-0.143 type=require
-depend fmri=pkg:/runtime/python-26@2.6.4-0.143 type=require
-depend fmri=pkg:/system/library@0.5.11-0.143 type=require
+depend fmri=pkg:/library/print/cups-libs type=require
+depend fmri=pkg:/runtime/python-26 type=require
+depend fmri=pkg:/system/library type=require
 
 
 

components/rsync/rsync.p5m

 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 license COPYING license="GPLv2"
 dir path=usr
 dir path=usr/bin
 file path=usr/bin/rsync variant.arch=$(ARCH)
 file path=usr/share/man/man1/rsync.1
 file path=usr/share/man/man5/rsyncd.conf.5
-depend fmri=pkg:/system/library@0.5.11-0.143 type=require
+depend fmri=pkg:/system/library type=require
 
 
 

make-rules/configure.mk

 CONFIGURE_OPTIONS += CXX="$(CCC)"
 CONFIGURE_OPTIONS.32 += --bindir=/usr/bin
 CONFIGURE_OPTIONS.32 += --libdir=/usr/lib
-CONFIGURE_OPTIONS.64 = --bindir=/usr/bin/$(MACH64)
+CONFIGURE_OPTIONS.64 += --bindir=/usr/bin/$(MACH64)
 CONFIGURE_OPTIONS.64 += --libdir=/usr/lib/$(MACH64)
 
 $(COMPONENT_SRC)/build-$(MACH32)/.configured:	BITS=32
 	($(RM) -rf $(@D) ; $(MKDIR) $(@D))
 	$(COMPONENT_PRE_CONFIGURE_ACTION)
 	(cd $(@D) ; $(ENV) $(CONFIGURE_ENV) $(CONFIG_SHELL) \
-		../configure $(CONFIGURE_OPTIONS) $(CONFIGURE_OPTIONS.$(BITS)))
+		../configure $(CONFIGURE_OPTIONS))
 	$(COMPONENT_POST_CONFIGURE_ACTION)
 	$(TOUCH) $@
 

make-rules/ips.mk

 PKGFMT =	/usr/bin/pkgfmt
 PKGMOGRIFY =	/usr/bin/pkgmogrify
 PKGSEND =	/usr/bin/pkgsend
+PKGLINT =	/usr/bin/pkglint
 
 # Package headers should all pretty much follow the same format
 METADATA_TEMPLATE =		$(WS_TOP)/transforms/manifest-metadata-template
 ifeq	($(IPS_PKG_NAME),)
 	IPS_PKG_NAME =	$(COMPONENT_NAME)
 endif
+ifeq	($(COMPONENT_SUMMARY),)
+	COMPONENT_SUMMARY =	$(COMPONENT_DESCRIPTION)
+endif
 IPS_COMPONENT_VERSION =	$(COMPONENT_VERSION)
 
 .DEFAULT:		publish
 
-.SECONDARY:		$(GENERATED).fdeps
+.SECONDARY:
 
 publish:		install $(COMPONENT_SRC)/.published
 
 	cat $(CANONICAL_MANIFESTS) | $(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \
  		sed -e '/^$$/d' -e '/^#.*$$/d' | sort -u | $(PKGFMT) >$@
 
-$(MANIFEST_BASE)-%.compare:		$(MANIFEST_BASE)-%.p5m
+$(MANIFEST_BASE)-%.compare:		$(MANIFEST_BASE)-%.p5m canonical-manifests
 	$(PKGMOGRIFY) $(PKG_OPTIONS) $(COMPARISON_TRANSFORMS) $< >$@
 
 manifest-compare:	$(COMBINED).compare $(GENERATED).compare
 	@echo "Manifest comparison"
 	@pkgdiff $(GENERATED).compare $(COMBINED).compare
 
+$(MANIFEST_BASE)-%.linted:	$(MANIFEST_BASE)-%.resolved
+	@echo "Linting $(@:$(MANIFEST_BASE)-%.linted=%) manifest"
+	$(PKGLINT) $<
+	$(TOUCH) $@
+
+manifest-checks:	manifest-compare $(PUBLISHED:%.published=%.linted)
+
 # mogrify the canonical manifest(s) 
 #
-$(MANIFEST_BASE)-%.resolved:	%.p5m manifest-compare
+$(MANIFEST_BASE)-%.resolved:	%.p5m canonical-manifests
 	$(PKGMOGRIFY) $(PKG_OPTIONS) $< $(PUBLISH_TRANSFORMS) >$@
 
-$(MANIFEST_BASE)-%.published:	$(MANIFEST_BASE)-%.resolved
+$(MANIFEST_BASE)-%.published:	$(MANIFEST_BASE)-%.resolved manifest-checks
 	$(PKGSEND) -s $(PKG_REPO) publish --fmri-in-manifest \
 		-d $(PROTO_DIR) -d . $<
 	$(TOUCH) $@
 
-$(COMPONENT_SRC)/.published:	manifest-compare $(PUBLISHED)
+$(COMPONENT_SRC)/.published:	$(PUBLISHED)
 	$(TOUCH) $@
 
 print-package-names:	canonical-manifests
 		$(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \
  		sed -e '/^$$/d' -e '/^#.*$$/d' | sort -u
 
-canonical-manifests:	$(CANONICAL_MANIFESTS)
+install-packages:	publish
+	@if [ $(IS_GLOBAL_ZONE) = 0 -o x$(ROOT) != x ]; then \
+	    cat $(CANONICAL_MANIFESTS) $(WS_TOP)/transforms/print-paths | \
+		$(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \
+ 		sed -e '/^$$/d' -e '/^#.*$$/d' -e 's;/;;' | sort -u | \
+		(cd $(PROTO_DIR) ; pfexec /bin/cpio -dump $(ROOT)) ; \
+	else ; \
+	    echo "unsafe to install package(s) automatically" ; \
+        fi
+
+$(MOGRIFIED):	install
+
+canonical-manifests:	$(CANONICAL_MANIFESTS) Makefile $(PATCHES)
 ifeq	($(strip $(CANONICAL_MANIFESTS)),)
 	# If there were no canonical manifests in the workspace, nothing will
 	# be published and we should fail.  A sample manifest can be generated
 	@pkg search -H -l '<$(DEPENDS:%=% OR) /bin/true>' \
 		| sed -e 's/pkg:\/\(.*\)@.*/REQUIRED_PKGS += \1/g' >$@
 
+pre-prep:	required-pkgs.mk
+
+
 CLEAN_PATHS +=	required-pkgs.mk

make-rules/prep.mk

 	$(GPATCH) -d $(@D) $(GPATCH_FLAGS) < $<
 	$(TOUCH) $@
 
-$(COMPONENT_ARCHIVE):
+$(COMPONENT_ARCHIVE):	Makefile
 	$(FETCH) --file $@ \
 		$(COMPONENT_ARCHIVE_URL:%=--url %) \
 		$(COMPONENT_ARCHIVE_HASH:%=--hash %)
+	$(TOUCH) $@
 
-$(COMPONENT_SRC)/.unpacked:	$(COMPONENT_ARCHIVE)
+$(COMPONENT_SRC)/.unpacked:	$(COMPONENT_ARCHIVE) Makefile $(PATCHES)
+	$(RM) -r $(COMPONENT_SRC)
 	$(UNPACK) $(UNPACK_ARGS) $(COMPONENT_ARCHIVE)
 	$(TOUCH) $@
 

make-rules/shared-macros.mk

 CONSOLIDATION =	userland
 PUBLISHER =	$(CONSOLIDATION)-build
 
+IS_GLOBAL_ZONE =	$(shell /usr/sbin/zoneadm list | grep -c global)
+ROOT =			/
+
 # get the most recent build number from the last mercurial tag
 LAST_HG_TAG =	$(shell hg tags -q | head -2 | tail -1)
 LAST_BUILD_NUM = $(LAST_HG_TAG:build-%=%)
 BUILD_NUM =		0.$(shell expr $(LAST_BUILD_NUM) + 1)
 BUILD_VERSION =		$(OS_VERSION)-$(BUILD_NUM)
 
+
 COMPILER =		studio
 BITS =			32
 PYTHON_VERSION =	2.6

transforms/manifest-metadata-template

 set name=info.upstream_url value=$(COMPONENT_PROJECT_URL)
 set name=info.source_url value=$(COMPONENT_ARCHIVE_URL)
 set name=org.opensolaris.consolidation value=$(CONSOLIDATION)
+set variant.arch=$(ARCH)
 
 #license $(COPYRIGHT_FILE) license='$(COPYRIGHTS)'
 

transforms/publish-cleanup

 
 <transform set -> edit value "\s$" "">
 <transform set value="^\s*$" -> drop>
-<transform set name=info.classification value=".+:" -> drop>
+<transform set name=info.classification value=".+:\s*$" -> drop>
 
 # drop the license actions from the manifests, we aren't going to compare them
 <transform license -> drop>