Enable retrieval of individual files or entire directories

Issue #330 resolved
lefish created an issue


We just started using rhodecode and really like it. Thanks!

One thing we're trying to do as part of our build and deployment scripts is retrieve the latest (or, potentially any revision) of a directory within our repository, without having to pull the entire repository (it's large, and we often need just one part of it for some deployment tasks).

It appears that we currently can retrieve a version of a file using this URL:

https://myrepository.com/[repository name]/raw/[changeset #]/path/to/my/file.txt

and, even better, it looks like we can specify the word "tip" instead of a specific changeset so we're guaranteed to get the latest version.

But we need to retrieve an entire directory, and either need to download all the contents (zipped or not), or, at the very least, an easy-to-parse list of the directory's contents, which can iterate over and use the above method to grab each file individually.

Wondering if that's feasible, and something you'd consider adding?

Thanks, David Fisher

Comments (14)

  1. Marcin Kuzminski repo owner

    RhodeCode file browser have build in file search, this file search uses a ajax call with special header to retrieve a JSON list of all files and directories in given directory. I think you can easy mimic this with simple curl command.

    curl -X GET --header 'X-PARTIAL-XHR:true' 'https://secure.rhodecode.org/rhodecode/nodelist/tip/docs'

    This will return a JSON list of all nodes, i think it's easy to parse.

  2. lefish reporter

    Thanks for your response. I'm trying that, but the http response is redirecting me and presenting the login form. I tried authenticating using my API key like so:

    curl -kLX GET --header --data-binary '{"api_key":"[my api key]"}' 'X-PARTIAL-XHR:true' 'https://sourcecode.monsooncommerce.com/[repo name]/nodelist/tip/path/to/dir'

    But that didn't work. I'm not trying to execute any methods via the API, just to authenticate in order for the other parts of the URL to accomplish something. Does that API allow me to do that somehow? Any way to automate the authentication?

    Thanks again!

  3. Marcin Kuzminski repo owner

    ups, forgot that might now work if there's no public access to repository, i guess a dirty hack would be to write a script that can log in with a cookie and that request the file-list, but it's kind of dirty hack.

    I'll add an API method for listing files into rhodecode, sounds like trivial to do for me.

  4. lefish reporter
    • changed status to new

    HI Marcin,

    One more question: when do you expect to do another stable release w/installer (non-Beta) that will include yesterday's changes?

    And once that is released, can we install over our existing installation and maintain our current configuration and settings? If not, is there an easy way to incorporate the changes you made yesterday while keeping our current config? We're running v0.07.


  5. lefish reporter

    Hi again Marcin,

    I just realize that if we need to retrieve a single file using curl or wget and the method described in comment #1, we need to authenticate by username/pw or API key. You added the API method to get a directory list (yay!), but we still can't pull individual files in an automated fashion because we can't authenticate. I think to make the directory listing API call useful, it would need a companion API call to allow individual file (or complete directory) retrieval.

    I'm hesitant to ask you about adding that, but figured I'd toss that out there.

    Thanks again. David

  6. Matt Zuba

    It sounds as if you're probably coming from a subversion system and migrating to Mercurial and trying to do what you used to do. Even if that's not the case, you seem to be trying to do something that Mercurial wasn't natively made to handle.

    For longevity reasons, you may want to reconsider your repository layout and look into using subrepositories (http://mercurial.selenic.com/wiki/Subrepository). Just a thought... :)

  7. Marcin Kuzminski repo owner

    Each view in rhodecode can easily be accesible with API KEY, just change the

    @LoginRequired() into @LoginRequired(api_access=True) on decorated views to enable api access, then it's possible to access resource by putting in api_key=<key> into GET param.

  8. lefish reporter

    Thanks Marcin. I'm assuming the decorated views are the .py files in the controllers directory? The "view" I'm specifically looking for is the "raw" file contents. I don't see a "raw" view or .py file, however. What view handles the requests for raw file source?


  9. Shawn Gray

    I was able to download an individual file, with the help of these posts by using this command:

    curl https://bitbucket.org/[name]/[repository-name]/raw/[revision]/[filename] > [directory]/[filename]

    This was very helpful when I just needed to get one file from the repository, for example I want to just go back to what the file was before editing... By using this method my file is restored right away without the need to clone the entire tree, which if I have other files modified, but have not updated the local repository on the BitBucket server could cause problems.

    Any way thanks for the help, as I said with the guidance of these posts I was able to figure this out, and it is very helpful when I need to restore just one file right away without restoring the whole repo.

  10. Log in to comment