Source

smart-oxe / IDEAS

Full commit

Configuration file
------------------

- Write YAML based config file?

Commands
--------

- Introduce a mixed-mode command, allowing to install/remove/upgrade.

Guessing unused packages
------------------------

<acme> a heur?stica levaria em considera??o: "idade" do pacote (i.e. h? quanto
tempo ele foi instalado), tamanho, fato de nenhum outro precisar dele, tempo de
acesso a seus componentes, etc

Downloading and committing
--------------------------

- Partial installer may copy to the fixed disk only these entries
  for which there are missing pre-requires.

Removable repositories
----------------------

- Ask channel to check if repository is available somewhere.
- Remove packages for unavailable media and ask to replace
  it when the current transaction has been committed.
- Removable repositories may be remote as well.
- Use localmedia:// for repositories which may be stored on any
  local media (not *one* path like /mnt/cdrom).
- Use a file in the removable media in a standard place
  to inform the repository type/position/name/etc.
- Implement autodetection of channels by introducing an
  autodetect() function in the channel info file.
- Implement plugins which detect different kinds of
  localmedia (autofs, fstab, supermount, etc), with different
  notions of mounting/unmounting/etc.

Mirror system
-------------

- Store history of downloads (bytes, download time, errors)
- History queue is shared by all servers, and has a limited lenght
- Compute penality based on stored history (penality >= 0)
- All servers start with penality = 0
- Use servers with the lowest penality

Priority system
---------------

- Packages and channels have priorities
- Both may be overriden by user configuration
- The final package priority is obtained by taking the user
  priority for the package, if given, or by taking the highest
  channel priority for the given package and summing with the
  package priority.
- Upgrades happen always to a package with the same or a higher
  priority.
- Downgrades may happen if packages with higher priorities are
  found somewhere.
- Priority is used to compute the best choice when multiple options
  are available to satisfy a dependency.

Repositories
------------

- Implement an "auto-repository", which downloads the needed info and
  uses that info to load one or more repositories (mirrors?).

Remote systems
--------------

- Implement RemoteChannel, RemoteLoader, and RemotePackageManager.
- Implement "daemon" command, which waits for connections, and when
  connected, creates a local Control with the given channel
  information, loads packages for the given channels, and sends
  the loaded packages for being used in a remote loader. It also
  handles downloading and committing packages to local a
  PackageManager when instructed by a remote Control.

Random
------

- Implement ISO downloading system.

RPM coloring
------------

<jbj> color of a package is union of file colors within.
<jbj> color of a dependency is union of file colors that have the dependency.
<jbj> color of a transaction is union of package colors.
<jbj> %_transaction_color is affinity mask.
<jbj> technically color=1 upgrades color=1, color=2 upgrades color=2. different
      arch is incidental.
<jbj> and color for file maps eaxctly to elf32/elf64 magic in file.
<jbj> dependency color is computed, not stored. there are arrays to attch
      variable number of dependencies to files, gory but straightforward.
<jbj> so starting pt and count saved for each file, depends dict contains 'P'
      or 'R' in 0xff000000 iirc, to determine whether index is into
      provides/requires, 0x00ffffff is index into {N,EVR,F} tag arrays.
<jbj> disjoint dependency graphs, each color is it's own graph, not too scary.
<jbj> the far more important issue is that files, not packages, have
      dependencies attached. that means a ppkg starts to become a lightweight
      container manifest rather than current heavyweight blob.
<jbj> btw, all the arrays are now sorted too, so bsearch becomes possible.
<jbj> hash still better, but rpm always been addicted to bsearch for uglix
      portability hysteria ;-)
<jbj> many-to-one is impossible map. one-to-many through filelist, see rpm -q
      --file{require,provide} 
<jbj> --filerequires displays only DEPENDSDICT(X,N) iff
      value & 0xff000000 == 'R'
<jbj> filedependsN is the number of deps starting at filedpendsX indexed
      through dependsdict that are attached to a file.

<niemeyer> Hiho!
<jbj> hey!
<niemeyer> Do you have some minutes for a small discussion?
<jbj> sure
<niemeyer> It's about upgrading packages with two different architectures installed in the same machine.
<niemeyer> What do you belive to be the correct way to upgrade these packages?
<jbj> ah, okay ;-)
<niemeyer> Should both architectures be fetcher in parallel?
<niemeyer> fetched
<jbj> there's "correct" and there is what is implemented.
<jbj> the general problem is quite hard.
<niemeyer> I understand that..
<jbj> however, the goal was to smoosh distros for 2 arches together w/o any change.
<niemeyer> What I want is letting users happy with an upgrader that deals with the way rpm handles the problem.
<jbj> that simplifies, so arch1 (or actually color1) upgrades only color1, ditto color2
<niemeyer> So, it's more about what to do, than about how rpm implements it.
<jbj> what to do ... ?
<niemeyer> Yes.. I mean:
<jbj> details please.
<niemeyer> Ok, now I have two packages: .i386.rpm and .ia64.rpm, should I upgrade both at the same time? What if only a new version of the ia64 is available? Should i386 be uninstalled?
<niemeyer> How did you see that (hard) problem when you envisioned support in rpm?
<jbj> ah, so you worry about how to choose what gets upgraded?
<niemeyer> Yes!
<jbj> well traditionally there was archScore() to prefer i586 over i486 on i686.
<niemeyer> Yes.. I'm using that currently.
<jbj> that works for each of two distro arches, the 2 (say i386 and x86_64) are disjoint on elf32/elf64 color.
<jbj> so what is needed is test on color before doing traditional archScore() to choose what gets upgraded.
<jbj> it's basically 2 choices, one choice for each color.
<niemeyer> Is there an easy way to do that?
<niemeyer> (test on color)
<jbj> and then throw the choices into same or different transactions.
<jbj> sure, there are methods to get the color of all rpm objects.
<jbj> the color of an object is just the union of the file colors contained/attached in the object.
<niemeyer> So I can just ask, using the rpm API, for the color of a header, for instance?
<jbj> so a pkg that has an elf32 file is color = 1
<jbj> headers are not rpm objects ;-)
<niemeyer> Are they not? :)
<jbj> but headerGetEntry on file colors, or the ints together. that is the header/package color.
<jbj> headers are being carefully yanked out of the rpmlib API, rpmlib uses ds/fi/te etc everywhere possible.
<jbj> yes, rpmtsAddInstallElement and rpmdbMatchIterator still use header, it ain't exactly easy abandoning the most important data structure in rpm.
<niemeyer> So there's no way to ask the color of a package but going through the maps of colors in the package and compute the per-file color?
<jbj> nope. files have color, all other colors are computed, to localize in case I need to change something.
<niemeyer> Btw, I still have our last conversation on colors here. I promise I won't bother you with the same questions. :)
<jbj> np, you'll be back for more when you discover ia64 hackery ;-)
<niemeyer> Hehehe :)
<niemeyer> Well.. I guess I'll have to figure out how to compute the color of a package using file colors then.
<jbj> headerGetEntry on FILECOLORS, loop over array color |= fcolor[i]
<jbj> color &= 0x3
<jbj> that is pkg color
<jbj> hdr color too
<niemeyer> Cool.. will use that. Thanks!
<niemeyer> Then, I'll have to attach a color to every package, and figure upgrades using a per-color logic, as you suggested.
<niemeyer> Ok.. I think I'm starting to see the light. :)
<niemeyer> Now, let's get into per-color dependencies..
* niemeyer runs..

Package list sorting
--------------------

<jbj> niemeyer: just the voices in my head. ther is 1 degree of freedom for
tsort. FIFO/LIFO are perfectly ok, but the goal of ordering in an install is to
create as many leaf nodes as soon as possible, number of immediate descendants
in sub-tree is/was easy weight (because available in Knuth's tsort algorithm).
better weight is number of nodes in sub-tree, not just descendants. --chainsaw
seemed obvious thing to do
<jbj> heh, stare at mutisets, think about permutation groups and cycles.
multiply inherited dependencies (i.e. node has multiple possible "parents") is
then a forbidden permutation on the permutation group.

Debian repository
-----------------

Fix of the "whois"-package to work with the new registrar of the .org-domains
(Added 2003-01-31, last checked 2004-04-16) (Download as text) - maintained by
dreamind at dreamind dot de 
Packages: whois
Architectures: i386
deb http://www.dreamind.de/debian/ woody-fixes main 
deb-src http://www.dreamind.de/debian/ woody-fixes main
      
Files:

http://www.br.debian.org/mirror/list
http://download.unesp.br/linux/debian/

http://www.dreamind.de/debian/dists/woody-fixes/main/binary-i386/Packages.gz
http://www.dreamind.de/debian/dists/woody-fixes/main/binary-i386/Packages
http://www.dreamind.de/debian/dists/woody-fixes/main/binary-i386/whois_4.5.25-1_i386.deb

Manual of dpkg:

http://sistemac.carnet.hr/paketi/dpkg-full/

About ordering:

http://lists.debian.org/deity/1998/03/msg00036.html

Debian Policy:

http://www.debian.org/doc/debian-policy/

Slackware repository
--------------------

slackpkg: http://slackpkg.sourceforge.net/
swaret: http://www.swaret.org/
slapt-get: http://software.jaos.org/

http://www.linuxtux.org/~juhl/slackpacks/

ftp://ftp.nluug.nl/pub/os/Linux/distr/slackware/slackware-current/
ftp://ftp.slackware.com/pub/slackware/slackware-current/

Mandrake repository
-------------------

ftp://ftp.planetmirror.com/pub/plf/mandrake/cooker

Red Carpet repository
---------------------

http://open-carpet.org/
ftp://ftp.ximian.com/pub/
http://rpms.subpop.net/channels.xml.gz

YUM repository
--------------

http://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/repodata/

Examples of cases where the upgrading algorithm is better
---------------------------------------------------------

- Package A requires a dependency provided by B-1.0. B-1.0
  is then upgraded to B-2.0, but the dependency needed by A
  is moved to package C. When an upgrade is done, will
  package B-1.0 be upgraded to B-2.0 and C be installed,
  or will A be removed?
  (http://bugzilla.atrpms.net/show_bug.cgi?id=160)

- Package A requires an implicit dependency, which is provided
  by package B, C and D. OTOH, package A has an explicit
  dependency on package C. Will package B and D be selected
  for installation, even though the dependency is satisfied
  by C, which must necessarily be installed?

- Package A is obsoleted by B and C, and they don't conflict.
  Will package B and C be selected for installation?

High level interface
--------------------

- Implement a higher level interface for package selection.

http://bugzilla.conectiva.com.br/show_bug.cgi?id=11124
http://fedora.redhat.com/projects/anaconda-installer/images/screenshot-0002.png
http://people.debian.org/~mvo/synaptic/ideas/synaptic-app-idea2.png


Parallel support
----------------

http://www.happyassassin.net/2005/05/04/a-quick-guide-to-urpmi-parallel/


Alternative logic for priorities?
---------------------------------

<Insount> You know, I think the most common use scenario is:
<Insount> 1. By default, track the latest version from channel 'stable'.
<Insount> 2. If don't find the package there, look for it in in 'unstable' or 'extra' or 'superextra', in this order. And track the latest version you find in the topmost amond these.
<niemeyer> Insount: What if a relation can't be solved without a package from an auxilar channel?
<Insount> 3. For packages foo and bar, track the version in 'unstable' (even if they're in 'stable'), and get the minimal dependencies from there too.
<Insount> EOF
<Insount> niemeyer: Bring in whatever packages you need, but always pick the first feasible combination, judging only by priority at the forking point rather than the whole combination.
<Insount> niemeyer: That is, search the tree of combinations in depth-first order, ordering by priority and each node, and return the first feasible combination.
<Insount> s/and/at/
<niemeyer> Insount: What about the choices that follow from the possibility of bringin in a package from the alternative repository?
<Insount> niemeyer: Bring them in, choosing the "best" version for each -- judging by the priorities for that package alone.
<Insount> niemeyer: For each package, you try versions in decreasing priority order. For each of them, add all dependneices (recursion). Once you have a consistent set, stop.

vim:ts=4:sw=4:et