Clone wiki

kallithea / Meeting2015AndrewMads

Random notes from Andrew and Mads meetup in Copenhagen May 2015.

Talked about Kallithea development, status, options, dreams, plans, and random stuff. Everything is personal opinions and probably outdated. Nothing official and no decisions on behalf of others. The notes have been edited and may or may not reflect actual discussions or opinions.

Steps towards using Bootstrap and less

  • refactorings that makes it possible to enable Bootstrap
  • enable bootstrap.css (and js?) (including removing unnecessary parts of style.css)
  • better tooltips
  • progress bar
  • remove old crappy css
  • move to LESS, committed (not generated on the fly)
  • easy way to (re-)generate less
  • NOW we start changing the look’n’feel
  • automatic rebuild on demand?
  • conclusion: Andrew will refactor/reorder upstream parts that are “ready”, Mads will test “probably ready” parts … when they probably are ready

Security

  • it would be nice to have a session list in the profile? + drop all/individual sessions button. Mads thinks it is hard to get a list when using beaker, Andrew will research.
  • rework password resets: don’t send any passwords, just a password reset link
    • password reset URL has three parameters: user id, timestamp of the reset request and a hash of the following:
      • user id
      • (part of?) old (current at that moment) password hash ← required to ensure links work only once (salt)
      • session cookie ← to prevent unrelated sessions from misusing the link
      • timestamp of the reset request ← to add time limits, hardcode 24h for the moment
      • session secret … or part of?
    • but reinventing security is wrong, also research existing options
    • Alternatives involve similar solutions, so proposed algorithm is probably at least not bad
    • Proof of concept patch proposed (without timeout handling) review pending
  • TODO: mail address verification
    • same as password reset … except also hashing address?
    • also require the user to specify the password again (because it allows changing the password)

Local users can change their own username

  • should only be allowed when self registration is enabled?
  • should it be a separate knob?
  • should admin be able to change anyway?
  • also change name of personal folder?
  • see also discussion: “should usernames go away?”

API

  • review features
  • review documentation - should be generated from actual API
  • review security / access control, should be like normal web interface
  • be less paranoid about api keys for user self and admin?

Group access control

  • use case: complete list of users and groups is confidential - also other members of groups you are a member of
  • currently: full user list in page templates (changeset/changeset.html and more!..), for @mention completion (and add reviewers)
    • should be constrained to users/groups with explicit repo access
    • only sent when users actually can comment (load on demand possible, not necessarily secure; also exposes user’s behaviour, might raise privacy concerns — rms would hate us)
  • currently: API lists all groups and members
    • group access - what does it do? should it just be fixed?
    • and what does “Inherit defaults” mean?
  • user group solution (which may or may not be how it currently is):
    • possible “modes” of (non)relationship between user and group:
      • user can’t see group [[=none?]] ← [private group?]
      • user can see group and grant group access to adminned repos and thus also see who is a member (both name and profile page?) [[=read]]
      • user can manage members of the group [[=write]]
      • user can rename group and add/remove admins [[=admin]]
    • consequences
      • seeing what a group has access to follows from being member or admin
      • it is not relevant to be able to just see a group name - it should either be invisible or readable so access can be granted
      • user can be member of a group without being able to see it - it will seem like explicit access
  • user listing solution - which other users should a user be able to see (when repo admin and granting access or @mentioning)?
    • when granting permissions: only users that users can see through read access to a group
    • when @mentioning: only users with access to the repo
  • default user:
    • current model and UI is confusing because
      • ‘default user’ can be anonymous user … which obviously also must be inherited to all real authenticated users
      • ‘default user’ can exclude anonymous and only be authenticated users (approximately “all” users)
      • other users and groups (whatever that means) can be excluded from inheriting from this ‘default’ user
    • instead
      • create an almost-normal group with automatic membership and configurable permissions that will be given to everybody, authenticated or not, make it easy and clear how to disable that group
        • kiilerix: it should be a group because it has the properties of a group and there is no such thing as inheritance between individuals
        • andrewsh: it should be a user because there’s no way to distinguish unauthenticated users from each other, they effectively are the same role so best represented by a user, not a group — at least, internally; the UI might be different though.
          • kiilerix: but each anonymous user creates an ad hoc user that is a member of that group - there is more than one anonymous user!
            • andrewsh: no, there isn’t really more than one anonymous user, I think, it’s just (in(de)finite number of) instances of the same incognito user.
        • possible conclusion: it might be both a user and a group … or special in some other way ;-)
      • create an almost-normal group with automatic membership and configurable permissions that will be given to all authenticated users (but not anonymous users), make it easy and clear how to disable that group
      • leave it to admins (and ldap) to define and maintain groups with “almost all users” — no “opt out”
      • note re user listing above: users will be able to see all other others by being given read access to these groups

Allow login using user’s email address instead of a user name

  • DOWIT

Should usernames go away

  • should be able to log in with email address
  • for organizations, username is given by email address
  • but nice … and needed for personal folders/profile pages
    • except if you think nice repo urls is more important than nice profile urls ;-)
  • perhaps make it configurable - some will only use email addresses, others might want to hide email addresses (even though often leaked in commits)

Release schedule

  • try to make a major release after every major Mercurial release
    • 0.3 in August with
      • bootstrap
      • ssh
      • per repo settings/config?
  • minor releases on demand, perhaps monthly

Data(base) model: how would a perfect model look like?

  • “Series” instead of pull requests?
  • Database upgrades

Auth stacking

  • we need something - don’t know what yet
  • need for multiple auth sources
    • use case: local auth + multiple ldap + container
    • inspiration from pam?
  • fall-through or tree structure? conditionals? is “use first match” always enough? auth must come first?
  • multi factor auth?
    • no issue; will either be side channel or two factors in password field
  • partly fall-through?
    • example: get auth from one source, user info from another? (pam/container auth, local/ldap userinfo)
    • ability to map/combine user details from multiple sources? .username = ldap1.username,ldap2.username .fullname = pam.fullname
    • perhaps like ldap which already has option for specifying which ldap field to use for what
    • how to fit in existing auth configuration? or new data model?
  • Auth API is ugly - plugins are weird
  • Mads minimal proposal that might give 99%:
    • make it possible to add same plugin multiple times by adding .1, .2, .3 (in settings.auth_plugins)
      • auth_ldap,auth_ldap.1,auth_ldap.2 or
      • auth_ldap,auth_ldap=1,auth_ldap=2
    • (automatically add kallithea.lib.auth_modules - like hgext works)
    • add this suffix to settings.app_settings_name key names
    • make auth plugin fall through all modules but never overwrite what has been collected
    • for each plugin, add ui flags & settings keys (suffix after plugin name) for whether it provides auth, name, email

Javascript - tables?

  • which lib to use? (has PR for jquery datatables)
  • kill paging, replace with on demand scrolling?

Minimize html / dom size?

  • lots of redundancy in what we send

Template escaping - avoid |n

  • json and other data - data attributes
  • better helper functions so we don’t need |n
  • how to handle expansion of URLs in translated/generated strings?

Refactorings

  • need for cleanup in lots of other areas

Web Framework future

  • pylons
    • reimplementing following pylons best practice would be nice and perhaps make it easier to upgrade … but a wasted effort ;-)
  • pyramid
    • big player and trustworthy … perhaps not
    • presumably most pylons-ish option … perhaps not
    • it might be possible to migrate one library at a time
  • django
    • too high level?
    • too tightly integrated, not many options
  • cherrypy?
    • small
  • flask?
    • low level
  • morepath
    • just routing?
  • web.py
    • or a teddy bear or some other toy

Cross-site cross-VCS collaboration

  • Beautiful dream!
  • OAuth support too? GitHub and Bitbucket APIs can give some inspiration
  • Staging area
    • Avoid serving anonymous content. Use for own pushes too? Store as bundles? Unionrepo-ish. Perhaps pushgate-ish, automatically creating PRs.

Renaming

  • Instead of repo URLs with repo ID, keep records of all names of repos an redirect from old to new name. When reusing an old name for creating a renamed repo, give a warning.

URL scheme

  • Mads’ guiding principles
    • must be consistent & predictable & pretty & short
      • very few characters are URL safe
      • very very few characters are safe from browser URL escaping
    • must start with repo name “from root”
    • must be safe even though we don’t have fixed 2 level as gh & bb
    • URLs are for stable objects - views have query parameters
      • are branches “something” or queries? queries!
      • don’t encode “random stuff” in paths - use parameters!
  • HTTP_X_PARTIAL_XHR - replace with something less odd
  • JSON API URLs?

“API”

  • should continue with json-rpc for consistency
  • rest api would be nice too, should expose same controllers as json- rpc
  • ajax from web pages should call REST too, replace HTTP_X_PARTIAL_XHR
  • html controllers should also expose raw data without template rendering
  • apikey authentication in URLs is weird and error prone (server logs, etc)

Other Outstanding PRs / mailing list ideas

SSH

  • Wip: https://bitbucket.org/conservancy/kallithea/pull- request/140/ssh-v6 , reviewed and discussed.
  • Possibly, switch to using Paramiko in the future: http://www.paramiko.org/ : ‘Paramiko is a Python (2.6+, 3.3+) implementation of the SSHv2 protocol, providing both client and server functionality. While it leverages a Python C extension for low level cryptography ( http://pycrypto.org ), Paramiko itself is a pure Python interface around SSH networking concepts.’
  • Alternatively, implement a “rsh server” and leave the ssh part to sshd.

Per Repo settings

  • https://bitbucket.org/conservancy/kallithea/pull-request/39/allow-to-enable-or-disable-hooks-on-a-per/diff
    • using an association table makes it easy to reuse … but perhaps complex to maintain?
    • simple db migration step, avoids duplicating whole db ...
  • requirements:
    • admin - repo defaults indicates existing per repo settings: private, stats, locking, downloads (boolean fields on repository in database)
    • hg settings & extensions per repo (instead of manual .hg/hgrc editing) … and there is probably a case for similar git settings
    • per repo hooks and hook parameters - also git
    • notify per repo (but that might just be a hook)
    • theming? (not really, but perhaps, one day …)
    • other global config?
    • some kind of fancy group inheritance … but not in first milestone
  • stored today
    • bools on repos
    • settings table - 99% really global (perhaps except stylify_metatags, dashboard_items, repository_fields)
    • ui table, 1:1 database representation of mercurial config, contains stuff we want to make per repo, not much else … except git coverage
  • easy solution
    • add a nullable repo field to the ui table
    • need a way to distinguish between VCS-specific stuff and our own; see what TortoiseHg does and do the same? Perhaps, use a [kallithea] section?

UI testing

  • would be very nice to have, would improve quality

DXR

  • something like this could replace some search & browsing

LDAP groups

Rename PR to Series or something like that

  • common name for a set of changes
    • (sad that “changeset” has been taken, this could be a “changesetset”)
  • avoids hardcoding the “policy” about what to do after looking at the changes
  • common name for
    • WIP progress visualization development tool (could be auto update?)
    • informal FYI review
    • formal review with approval
    • request for “integration” (pull, merge, rebase, graft)
  • a 1 changeset series == the changeset - it should be possible to “add reviewers” to a single changeset
  • means we need a way to specify policy … but free text in description could work for a starter, just like we now can use “pull requests” for lots of other things
  • the name doesn’t align very well with (future) issue tracking — or should it not?
    • assuming pull request, patch review and an issue is three kinds of the very same thing (in GitHub, every PR is also an issue, for example)
    • it is important that issues reported by the users are different from a series
    • if we assume they aren’t, issues (tickets?) and series can be different entities referencing each other — then it’s okay
    • (mads doesn’t get this ;-) )

Issue tracker?

  • Mads will implement “persistent comments as todo items” anyway … which pretty much is issue tracking. Let’s see where that goes and how general and upstreamable it will be ...

Downloads

workflow engine / state machine on PRs?

  • support custom work flows
  • easy integration with other systems (CI?, external review tool)

Updated