1. vim-community Avatar vim-community
  2. Untitled project
  3. vim-pi
  4. Issues

Create some kind of API which makes vim-pi easy to use without vim

Issue #80 invalid
ZyX_I created an issue

Currently data is scattered across various files. While it is convenient for creating database, it makes vim-pi impossible to use without cloning.

Suggestion: set up a bitbucket hook to automatically publish data in a fixed format to some location when master bookmark is updated.

Suggestion: make “fixed format” be vam#install#Pool() with modifications.

Suggestion: make “some location” be not version-controlled. All data necessary to regenerate data in “some location” will be located in https://bitbucket.org/vimcommunity/vim-pi.

Suggestion: “some location” should have multiple mirrors. Official mirror list is maintained in vim-pi repository.

Currently no suggestions about a list of mirrors. Probably bitbucket downloads.

Comments (23)

  1. Shougo

    Yes. I think those features are needed. In current version, I must use vam#install#Pool() for get repository. But I don't use external API for it. The database should be merged.

  2. marco-oweber

    We could also think about moving the contents into a database so that it can be edited online.. That's what I had in mind. Then people could contribute their changes easily

  3. ZyX_I reporter

    Having a script like in #82 will make contributing easier. I do not see much advantages in the database populated from some website, but you lose ability to backtrack in history that VCS gives you “for free” (obviously unless you code it) which is sometimes useful (e.g. like when we were doing renamings analysis).

    To save bandwidth API for querying database will be helpful. But it makes mirroring harder. Mirrors are rather helpful because here (in Russia) you can find website not available (blocked) because it happened to share the same IP with a website marked as containing something illegal. Not to mention that hosting may be down for other reasons.

    I should also ask whether you are going to compete with www.vim.org. Because you already have such database there.

  4. marco-oweber

    Well - the goal would be to replace vim.org with a) wiki b) database (and source lookup / review/ grepping) etc. Eg think about vam.mawercer.de - that's a hacky PHP script. Accesssing a database would be easier. But it works - so maybe its not best to rewrite everything at the moment unless we have to.

  5. ZyX_I reporter

    Do you have support from Bram? I remember that there were some threads in vim-dev, but no details. Only impression of achieving nothing there. Also some project without any development progress.

    Unless you have support it is like building castles on the sand. You cannot replace vim.org without upstream agreeing to replace it or an enormous amount of resources spend for various kinds of ads.

    And, by the way, do we have at least an API draft for this database? I think @Shougo will stop waiting for it long before it is finished. I have not ever written a single line of code accessing database except for quick-and-dirty google bookmarks starred items to Yandex.Maps bookmarks database (sqlite3) converter containing something like two (DELETE and INSERT) SQL commands. And zero experience with creating web applications.

  6. marco-oweber

    For this vim-pi project we need a single file to lookup changes ideas/drafts. I recommend vim-pi's readme (on bitbucket). Whether we shut down discussion here is not important to me. We can have a transition period and even upload changes to both repositories for quite a while. Its important to me to know and keep track about what's going on. And I invite you and Shougo to keep this README up to date. About "forking Vim" - yes - I know that that might happen. I already wasted too much time on "workarounds". I just don't know how to move forward yet (eg because this might mean writing a new language) - haven't had time yet - and this will take time. Till I know what I want I'd like to maximize value/effort ratio.

  8. ZyX_I reporter

    I am thinking about what for this database may be used. And got the idea that the following changes are needed that affect what is stored here:

    1. Descriptions. For searching within vim.
    2. (Possibly) READMEs for searching from web API. Not until we have a hosting.

    3. Change the whole storage to more portage-like format. I.e. directory structure should be

      addons.json
addons.json.xz
plugins/{plugin name}/addon-info.json
plugins/{plugin name}/vo-post-install.hdsl
plugins/{plugin name}/dev-post-install.hdsl

      addons.json will contain only minimal data: a dictionary that looks like

      {"{plugin name}":{"files":["vo-post-install.hdsl","dev-post-install.hdsl"],"desc":"This is a good plugin","snr":123,"version":"4.2","addon-info-sources-sha256":"3d62ba556908a8551cb632aaaa8775d981d5a663b8f37b915032aa2750beefc1"}}

      . addons.json is here only for plugin managers to 1. know what addons we know about, 2. perform search on these addons, 3. determine which files to download and 4. know whether update is needed. Note about "addon-info-sources-sha256": it is performed on a JSON dump of "sources" key contents. Used hash is SHA256 since vim now has appropriate function available. It is not supposed to be computed not by vim-pi cron job though: this is why I suggest having same key in addon-info below.

      addons.json most likely will be hosted not here (i.e. in some hosting suggested for generated data).

      addon-info.json should have the following structure:

      {
    "name": "{plugin name}",
    "description": "This is a good plugin",
    "author": "User <user@example.org>",
    "vim-org-script-nr": 123,
    "sources-sha256": "3d62ba556908a8551cb632aaaa8775d981d5a663b8f37b915032aa2750beefc1",
    "sources": {
        "stable": [{
            "type": "git",
            "url": "https://bitbucket.org/user/{plugin name}",
            "branch": "master",
            "hooks": {
                "post-install": "dev-post-install"
            }
        }, {
            "type": "git",
            "url": "https://bitbucket.org/user/{plugin name}",
            "tag-regex": "release-.*",
            "hooks": {
                "post-install": "dev-post-install"
            }
        }, {
            "type": "git",
            "url": "https://bitbucket.org/user/{plugin name}",
            "tag": "release-4.2",
            "hooks": {
                "post-install": "dev-post-install"
            }
        }, {
            "type": "archive",
            "version": "4.2",
            "archive-name": "{plugin name}.tar.xz",
            "unpack": ["xz", "tar", [
                "# Hoodospel script line 1",
                "# The whole list is here in order not to make other package managers perform guessing",
                "# In particular HDSL script is required if you cannot just use unpack tools",
                "# hoping this script will never be needed because there are post-install hooks"
            ]],
            "url": "http://www.vim.org/scripts/download_script.php?src_id=234",
            "hooks": {
                "post-install": "vo-post-install"
            }
        }],
        "development": [{
            "type": "git",
            "url": "https://bitbucket.org/user/{plugin name}",
            "branch": "develop",
            "hooks": {
                "post-install": "dev-post-install"
            }
        }, {
            "type": "archive",
            "url": "https://bitbucket.org/vimcommunity/vim-pi/get/develop.zip",
            "unpack": ["zip", ["REQUIRE 1.0", "STRIP_DIRECTORY DETECT"]],
            "hooks": {
                "post-install": "dev-post-install"
            }
        }]
    },
    "dependencies": {
        "foo": {},
        "bar": {"optional": true}
    }
}

      Notable changes:

      1. "repository" key got moved to "sources" list.
      2. All sources are seen at once, it is up to PM and/or user to decide which to use. Most plugins will have "stable":[{"type":"archive",…}],"development":[{"type":"git",…},{"type":"archive","url":"https://bitbucket.org] only though. In the current format you can have only one or the other and it is a blocker: we cannot post output of vam#install#Pool() because.
      3. PM are now supposed to be able to also check out specific VCS branches and tags (in the example plugin uses git-flow with master branch for stable releases and develop branch for development, all releases in master are tagged with release-{version}).
      4. Vim-pi has taken the job of PM of determining how to unpack sources.
      5. Vim-pi is now also listing zip archive generated on github/bitbucket.
      6. archive_name got renamed to archive-name, vim_script_nr got renamed to vim-org-script-nr for consistency (no underscores).
      7. There is now additional version key: sources-sha256.
      8. Hooks are now attached to sources and not to their kinds.
      9. Dependencies may be optional.
      10. All hooks are in files.
      11. Just a note: hook files may source another files. This means that PM should download files based on addons.json and not addon-info.json. Hooks are placed in hooks.hdsl.d directory.

      The processing of this data in package managers should be now much simplified:

      1. Download addon-info for chosen plugin.
      2. Use some settings to determine sources that are needed. It is supposed that PM will take the first source it is able to install that matches user settings.
      3. If it does use hoodospel then it just passes the whole dictionary to it. For hoodospel code is simplified since I do not have to replicate the unpacking logic that is already in autoget.py.

    Reasoning behind this all:

    1. As I said, current format does not allow distributing www.vim.org and SCM sources at once.
    2. It deletes the database files zoo. Main reasoning behind having this zoo is that currently there is no nice way to reuse hooks, edit vim files, or follow renames (I am talking about patchinfo.vim and lots of numbers inside). In new variant there are more ways to do so.
    3. It allows more applications: if somebody will want to implement a package manager that (like its big brothers) allows installation of specific version new format will not stand in his way.
    4. As addon-info.json file is JSON there are not much problems in editing it with programs.
    5. Some parts of VAM/NeoBundle may now be moved to hoodospel since it will be in any case required for installation (due to hooks) and there is not much sense in duplicating unpacking/installation code.
  9. marco-oweber

    I'm not sure I want to spend the time on it. Your design does make sense.

    sources-sha256 should be per source. What to do about git? Do you want have a trusted hash for "latest known revision"? I think that github solves this issue by using https reasonably well. Otherwise you might have more trouble than value.

    People use tags rather than branches.

    "unpack": ["xz", "tar", [

    I'm unsure whether its the task of us to tell the unpack order. You could create a shell dsl to be followed - but again - what for - current VAM implementation works.

    I want to focus on more important things in my life.

  10. ZyX_I reporter

    sources-sha256 should be per source.

    Why? It is a way to tell PM whether or not to look for updates. I.e. if recorded sources-sha256 matches current sources-sha256 PM has absolutely nothing to do when it was requested to update (unless PM options are changed, but it is PM own business). More specifically: PM does not have to redownload addon-info.json when it was requested to update. sources-sha256 is in addon-info.json for the only purpose of making PM not to recompute and record it somewhere: the main reasoning is comparing with sha256 in addons.json. I cannot have per-sources hash and use it in addons.json. Or I could have a list there, but it is pointless: this list will just become a new hash. For per-source versioning there is version key.

    What to do about git? Do you want have a trusted hash for "latest known revision"? I think that github solves this issue by using https reasonably well. Otherwise you might have more trouble than value.

    It is not about trusted downloading. Hash is computed out of sources dictionary contents. Not out of downloaded archive or else. Though it is maybe a better idea to compute hash using addon-info.json file itself, but that means I cannot put this hash into the addon-info.json file forcing PM to store it somewhere separately from addon-info.json.

    I'm unsure whether its the task of us to tell the unpack order. You could create a shell dsl to be followed - but again - what for - current VAM implementation works.

    Not actually. Current VAM implementation contains unpacking code. And autoget.py implementation contains unpacking code (and, by the way, the latter is more robust as it fallbacks to libmagic). I suggested this to make it possible to remove this duplication.

  11. marco-oweber

    The sha detail is not important. If you have "master" checkout sha will not change and you still want to check .. calculating a hash should be very cheap, so the client reading the json can do so anyway. http://titanpad.com/AcUfZQjXUz That's what we (I and garbas) came up with long time ago. We imagined keys such as "relations", "tags", "maintainers", too.

  12. ZyX_I reporter

    What “master” checkout? One is supposed to have neither git nor full checkout of vim-pi repository. If PM does check out repository it just can cope ignoring hash entirely.

  13. marco-oweber

    I'd like to add that a downside of portage is disk IO. Performance will drop if you have one file per package. Upside: git storage would be even nicer because the text file doesn't have to be restored if only some plugins get added. I feel speed is more important (?). We could still have a summary file. So this is a reminder only

  15. Tim Lebedkov

    I would not call a package database format like addons.json an API. As far as I can see everybody agrees on having a database to be one or many files accessed via HTTP. How can we decide now? By voting? Who is allowed to vote?

  16. ZyX_I reporter

    @tim_lebedkov Voting for technical decisions is the worst idea you can ever imagine. Database will be many files accessed via HTTP(S) because nobody here has suggested anything regarding hosting, actual API design and is willing/has time to write a server. This is also easy to maintain.

  17. Tim Lebedkov

    I tried to fork the wiki, but this does not seem to be possible. As far as I understand it is also not possible to make .md files inside of the project visible as a web site. Could somebody please add me to the project so I can clone the wiki?

  21. ZyX_I reporter

    Notable changes: addon-info was renamed to plugin-info (since it is Vim plugin index this is logical), instead of listing all sources in top-level plugin-info.json file in the database it is suggested to split everything into top-level plugin-info.json file with common definitions and version-specific plugin-info.json which will be merged with the top-level one. I also added forks support and renamed a number of keys (previously some keys used underscores and some dashes).

  24. Log in to comment
Assignee
–
Type
enhancement
Priority
blocker
Status
invalid
Component
API
Milestone
vim-pi
Votes
0
Watchers
3
Jira: the preferred issue tracker for Bitbucket. Join the team!