How to get pages of Repositories

[Zanson]

when running Repositories->all() i only get the 1st page of results back (10 repos). I see a next key when print_r on the results, but i do not know how to get a 2nd page of results.

// login
$repos->setCredentials(new Bitbucket\API\Authentication\Basic($bb_user, $bb_pass));

# get list of pull requests
$list    = $repos->all($accountname);
$listAll = json_decode($list->getContent(), true);
foreach ($listAll['values'] as $k => $val) {
    echo $val['full_name'] . "\n";
}

[Alexandru Guzinschi]

Sadly for the moment there is no easy way to accomplish this task, but it will be nice to have.

In the meantime, you could do something like this to get the next page with results:

if (isset($listAll['next'])) {
    $list2 = $repos->getClient()->setApiVersion('2.0')->request($listAll['next']);
    $listAll2 = json_decode($list2->getContent(), true);
}

[Zanson]

Excelent, that works. Thanks for the quick response. Much appreciated.

[Zanson]

Created a local function for this. i'm not sure about the rest of the api calls, but it works well for looping and merging all the results into 1 large array.

$repos->setCredentials(new Bitbucket\API\Authentication\Basic($bb_user, $bb_pass));

function getAllRepoListings($request) {
    global $repos;

    $list = json_decode($request->getContent(), true);

    if (isset($list['next'])) {
        $listMerge = getAllRepoListings($repos->getClient()->setApiVersion('2.0')->request($list['next']), $repos);

        return array_merge($listMerge, $list['values']);
    }

    return $list['values'];
}

$list = getAllRepoListings($repos->all($accountname));

[Onni Hakala]

Really helpful, thanks for sharing!

[Peter Csonka]

Damn, I looked for in the documentation, then in the source code for a solution, and I actually felt stupid, because I did not find anything... Now I know why. :) Thanks for the function Jason!

[Mark Mitchell]

Here is a version which can be dropped into a class that doesn't use globals. Just set your class variable accountName to the bitbucket team/account name. Thanks for the idea Jason!

    /**
     * List all the repositories
     *
     * @return array
     * @throws \Exception
     */
    public function listRepositories() {
        $repositories = new API\Repositories();
        $repositories->getClient()->addListener(
            new API\Http\Listener\OAuthListener($this->getAuthParams())
        );
        return $this->getAllRepoListings($repositories, $repositories->all($this->accountName));
    }

    /**
     * Recursively get all the repositories
     *
     * @param API\Repositories $repositories
     * @param \Buzz\Message\MessageInterface $response
     * @return array
     */
    function getAllRepoListings($repositories, $response) {
        $list = json_decode($response->getContent(), true);
        if (isset($list['next'])) {
            return array_merge(
                $this->getAllRepoListings(
                    $repositories,
                    $repositories->getClient()->setApiVersion('2.0')->request($list['next'])
                ),
                $list['values']
            );
        }
        return $list['values'];
    }

[Alexandru Guzinschi]

@carcus88 Just an opinion. Considering that you run that recursively, maybe you should check if accountName is not empty before making the first request.

If you a send a null parameter to Repositories:all it will start to fetch all public repositories available globally inside Bitbucket, which will eat resources until the script execution time limit kicks in and kills the script (if its not disabled).

[Tit Petric]

I've adapted the above to not use recursion, and to give a check if the username is empty. I slightly extended it with a callback just to give it an optional progress indicator.

function getAllRepoListings($repositories, $username, $callback = false) {
        if (empty($username)) {
                throw new Exception("Listing all global public repositories is not supported");
        }
        $result = $repositories->all($username);
        $retval = array();
        while (true) {
                if ($callback !== false) {
                        $callback();
                }
                $page = json_decode($result->getContent(), true);
                $retval = empty($retval) ? $page['values'] : array_merge($retval, $page['values']);
                if (!empty($page['next'])) {
                        $result = $repositories->getClient()->setApiVersion('2.0')->request($page['next']);
                        continue;
                }
                break;
        }
        return $retval;
}

The callback parameter is optional, as I said I'm using it to display progress. A full example of use would be something like this:

$repos = new Bitbucket\API\Repositories;
$repos->setCredentials($this->getCredentials());

$config = $this->getConfig();

echo "Fetching repositories ";
$repositories = $this->getAllRepoListings($repos, $config['username'], function() { echo "."; } );
echo " Done.\n";
echo "Retrieved " . count($repositories) . " repositories.\n";

foreach ($repositories as $repository) {
        $name = $repository['full_name'];
        if (preg_match('/' . $pattern . '/', $name)) {
                echo $name . "\n";
        }
}

Specifically, I'm using it to match my numerous repositories against preg patterns:

# ./sync ls .+api.+
Fetching repositories .................... Done.
Retrieved 196 repositories.
titpetric/api-tinker
titpetric/leanpub-api-foundations
titpetric/mmc-api-test

And these are just the repositories which are owned by my account, overall I have about double that. There's no way I'm managing deploy keys, webhooks and permissions for such a number of repositories by hand. @vimishor - if you'd like to include this in some form to the library I can fork/give you a PR. It just depends on how/where/in what shape you'd like this :)

[Alexandru Guzinschi]

@titpetric Sure, you could implement this feature by changing the signature of Bitbucket\API\Repositories::all().

The only thing I care about is not to break BC if possible and in order to do that you would need to leave the current behavior in place (i.e paginated response) and to return a MessageInterface instead of an array like you do in your example.

A signature like public function all($owner = null, $paginated = true, $callback = null) I think that will do and you can do your thing only when $paginated is false. At the end you can replace values from the last response with the entire array of fetched repositories and return that response.

If you have a better idea, I'm open to suggestions.

[Alexandru Guzinschi]

Added Pager info in FAQ and Changelog

related: #17

todo: Try to also support v1 of the API.

→ <<cset 0b1758d136f3>>

[Alexandru Guzinschi]

Added listener to allow Pager to work with API v1

Because Pager expects some metadata specific to V2 of the API to be present in the response, would not work with collections coming from v1 of the API.

ApiOneCollectionListener will add to each v1 response (which contains a collection), the metadata specific to v2 which is expected by Pager.

ref: #17

→ <<cset 185aea0a2ae3>>

[Alexandru Guzinschi]

I implemented a simple Pager in 0b1758d136f3 and 185aea0a2ae3 and I will appreciate any feedback from anyone who has the time to test it.

For the moment who wants to test it, needs to use the develop branch and because the documentation from the website is not yet updated, I will leave a code snippet here as a simple example:

<?php
$repo = new \Bitbucket\API\Repositories();
// auth here
$page = new \Bitbucket\API\Http\Response\Pager($repo->getClient(), $repo->all('gentlero'));

// get current page
$response = $page->getCurrent();

// next page
$response = $page->fetchNext();

// previous page
$response = $page->fetchPrevious();

// get all pages
$response = $page->fetchAll();

Note: Any method of Pager that is prefixed with fetch implies that a new HTTP request will be made.

Expectations: Pager should work the same with API v1 and v2; if not, please open a new issue.

[Matthias Alt]

Hello there,

is there a plan to get this Pager in a upcoming release? I would like to get rid of 'dev-develop' in my application, but I need this Pager-Class.

Regards Matthias

[Alexandru Guzinschi]

@sd_alt My initial plan was to close a few more issues before I tag 1.0 (specially #49), but I didn't manage to do that for now, for various reasons.

I think that best choice to move forward is to leave everything as is for now, release v1.0 and deal with caching and everything else in future releases.

I will take care of that next week.

[Matthias Alt]

@vimishor Thanks for your reply. Thats absolutely fine.

Regards Matthias

[Alexandru Guzinschi]

Feature available in v1.0.0

[razvan]

I found an issue while I was using the pager to get the contents of a directory. https://developer.atlassian.com/bitbucket/api/2/reference/resource/repositories/%7Busername%7D/%7Brepo_slug%7D/src/%7Bnode%7D/%7Bpath%7D

I used the fetchAll method. Bitbucket's response contains the exact URL to access in order to get the next set of results. The problem is that the pager modifies that URL (and it shouldn't) by adding the "format" parameter. This alteration causes a 500 status code.

[Alexandru Guzinschi]

@rzds Can you please open a new issue and also provide some sample code to reproduce the issue ?

Thanks in advance.

[razvan]

Oh... It's not pager who adds the format parameter. It's actually the client. This will be harder to fix, I think. Unfortunately, I'm a little caught up in something at the moment.

[Alexandru Guzinschi]

@rzds see this issue for further updates.

Thanks for report.