Overview

- repoSight design specs (work in progress)
  António Meireles, October 2009

* Initial Problem Statement

  Right now, there are no simple tools to help on the various stages of repository management,
  at a qualitative level.

  It is true that we have a lot of fine and granular control available on the repository actions,
  via conary/rbuild but we lack an higher level tool that makes some sense of everything that is
  on a given set of repositories. The issue gets more serious as the repositories, and the
  communities working on them, get larger.  

  There is a serious weakness right now in the way we handle and consume metadata (package descriptions
  et al.). Albeit conary has proper support for it, we never got to have it right yet, resulting in a
  very sub-optimal experience - namely with PackageKit, the most obvious consumer of it. 

  We don't have a single, pluggable, tool that eases the process of tracking either upstream, or 3rd
  party repositories or distros, to search for changes, nor we have set a meaningfull abstraction to
  take advantage of such a tool.

  As the repositories grew larger the job of keeping groups sane and consistent gets harder and more
  complex, as there we have no tools to abstract it. 

* Solution Proposal

  to make metadata the core of our nervous system, and make as much as possible driven/monitored thru it. 

  Introducing repoSight.
  repoSight is a modular set of libraries to be consumed in several different ways - from a command-line
  tool to foresight-policy, to a web-based front-end. They will consume and make sense of what is set in
  metadata, in order to get management tasks simpler, obvious and error free. Said metadata is to be
  placed in a file, per :source, called INFO.

  Initially, our primary focus is on a command-line tool, that will check, per package, the INFO file and
  make sense of it. As we get forward into the foresight3 tree, it is important to make sure it is consistent
  and compliant with our own goals.

* The INFO file format

  * Every INFO file is a plain text based file to be parsed by iniparse (http://code.google.com/p/iniparse/),
    and using a ConfigParser like syntax.

  * the first 'block' of every INFO file must have the exact same name of the package:source it belongs to.

  * the first block must have 'summary', 'description', 'URL' and 'licensing' entries, where 'summary' is the
    shortDescription, 'description' the extended one, 'URL' the upstream canonical home, and 'licencing' a comma
    separated list of the licences that apply. Unless the obvious cases where it doesn't apply a good place to 
    look for the contents of this fields is on the Fedora tree.

  * 'URL' can only be set once per INFO file and that must happen in the first block.

  * INFO files can have several blocks. they can reference just a package (if main :source produces several ones),
    a :component (eg: [:devel]), or a mix of the two (eg: [foo:devel]).  

  * in addition to the fields above, every 1st block must have a 'tracks' field, with the following syntax... 

    tracks: rpath  # rPL is the canonical upstream of the package, and we should look for changes there
    tracks: foresight # the package is exclusive for foresight, and so it makes no sense to try to look for 
    	    	        changes elsewhere
    tracks: koji a.b.c-d.fcXY # this means that the baseline for the given package is the the a.b.c-d.fcXY version
                                from fedora (rigth now and fore simplicity, we only are caring about a koji 'plugin',
				in the future we expect to have several ones, covering other distros, and also major
				open-source projects, consuming for example gnome/xorg jhbuild files.)
				As soon as we have support for more than one plugin, they'd get separated  by commas. 

    arch: x86|x86_64|x86-x86_64|universal| # arch in wich the given package should be built, by Default is universal
                                           which means all arches we care (x86 and x86_64) should get built. x86 means
					   pkg only available on x86, idem for x86_64. x86-x86_64 covers the special
					   case of a package getting multiflavored in x86_64 (e.g. glibc) shiping
					   in the x86_64 version both x86 and x86_64 bits (this syntax may need
					   refinement in the case we find one of these cases in a x86_64 specific pkg)
					   A priori this field only is called from the first block. If we get packages
					   with variable components according to the arch, we set it too on their block.
    alias: koji foo, gnome bar		   # optional field. means that our packaging naming differs from the one used
    	   	     	   		   somewhere else, and hence tells the plugin that monitors that 3rd party about
					   it. can be called in any block. If called in a block other than the 1st one
					   means that albeit the primary package name is common, some of the pkgSpeced
					   components have custom naming. (example above assumes besides the koji plugin
					   the existence of a gnome plugin too - not yet written)
    sets: bootstrap, world		   # 'sets' a package belongs too. mandatory on 1st block, optional in the others.
    	  	     			   if not set in other blocks, data in 1st one will be used. this will be used 
					   to populate groups, and operations across labels. (FIXME: more details later)


* the command-line tool 
  
  deliverables

   at a repo level
    that (description) metadata set is consistent, as (all pkgSpec stuff should have proper descriptions)
    that all flavors are properly built and with matching versions
    that the latest :source version is the one built [if not advise/sugest proper actions]

   competitive analisys

    ability to tell us our per package status vs 3rd parties, and suggest courses of action if any.  
    patch analysis, ability to detect if a patch come from elsewhere (say koji) and if is still relevant.
                    
NOTE: As of Oct/26/2009 all of {bash,binutils,bzip2,conary,conary-policy,coreutils,cpio,db,debugedit,diffutils,elementtree,elfutils,expat,file,findutils,gawk,gmp,grep,gzip,libtermcap,libtool,libxml2,make,MAKEDEV,openssl,patch,pcre,popt,pycrypto,python,python-setuptools,sed,setup,sgmlop,sqlite,tar,texinfo,unifdef,userspace-kernel-headers,zlib}=foreplay.rpath.org@bootstrap:toolchain pass with some degree of success the very early rough version of repoSight.