Check Repository Group instead of specific Repository

[Tim Klicks]

Hej Dmitrii,

First of all, let me say that your plugin is exactly what we searched for. And that it is for free is really really great and pushes our Jira functionality to another level.

However, I would like to ask, if it is possible to check a, like we call it, Repository Group instead of one specific repository. For example, we have the following gitlab structure here:

docker-services/pdf-service docker-services/jenkins-service docker-services/nginx-service . . .

Every week, there will be a at least one new repository created inside of 'docker-services' Repository Group. At the moment, this leads to the following error: *error: https://gitlab.deepmap.net/api/v3/projects/microservices/repository/branches/master *

This would mean, that we would have to create this new repos in the jigit plugin in order to be checked too. For our purposes, it would be great if we could create the Repository Group as a whole.

Do you think, that is possible?

Best Regards, Tim.

[Dmitrii Apanasevich]

Hi Tim!

Let's clarify some things. As far as I understand, there is a new repository created every week and you have to configure it's indexing in Jigit. All your repositories' names match the pattern docker-services/*. And you want to setup indexing of all existing and future repositories inside docker-services. Right?

In what case do you get an error you mentioned?

[Tim Klicks]

Hej Dmitrii,

Thanks for your swift response on this issue. I will try to clarify our situation.

We have our own hosted gitlab here and we have different repositories, like 'microservices' or so on. Inside these repositories, we have different projects, for example 'microservices/pdf-service' or 'microservices/app-config-service' and so on

Our developers are working inside the projects, like 'microservices/pdf-service', and are improving the code or add new features there and log/push their changes to this project using the Jira Ticket/Issue identifier. At the moment, we have to add every project as a repository in Jigit, so we would have 'microservices_pdf-service' for checking 'microservices/pdf-service', 'microservices_app-config-service' for checking 'microservices/app-config-service', and so on in the overview.

As our developers are also adding new services, they create projects inside 'microservices' at least once per week themselves and we don't know about it. This would lead to projects not being created in Jigit and thus not being visible in the Issue Tracking later on.

Our question is now, if there is a possibility to add the whole gitlab repository 'microservices' as a Jigit repository, so that it would automatically scan all the gitlab projects under 'microservices'. At the moment, we are receiving this error: error: https://gitlab.deepmap.net/api/v3/projects/microservices/repository/branches/master

Hope it is clearer now.

Best Regards, Tim.

[Dmitrii Apanasevich]

Our question is now, if there is a possibility to add the whole gitlab repository 'microservices' as a Jigit repository, so that it would automatically scan all the gitlab projects under 'microservices'. At the moment, we are receiving this error: error: https://gitlab.deepmap.net/api/v3/projects/microservices/repository/branches/master

O! I get it. You get this error when you try to add microservices group as a separate repository at config page. Unfortunatelly, it's not possible now.

I have to take a look at API to answer it is possible or not. I'll comment later.

[Dmitrii Apanasevich]

Ok, suitable API exists both in GitLab and GitHub:

• https://docs.gitlab.com/ce/api/groups.html#list-a-group-39-s-projects
• https://developer.github.com/v3/repos/#list-organization-repositories

This feature will index all branches of all repositories inside a group. You won't have a possibility to index only specific branches. Is it ok?

The only question is: Do you use subgroups or microservices group has only projects (repositories) inside?

[Tim Klicks]

Hej Dmitrii,

Thanks for checking the gitlab and github APIs and coming back so fast. From what I can say, we do not use subgroups under the microservices group, we only have projects underneath it.

When you say 'You won't have a possibility to index only specific branches', you probably mean once we have added microservices group, or?

For me, it would be great, if your plugin would have some kind of radio button, which is giving us the possibility to say if we want to either create/index a single project and branch (just like it is right now) or if we want to create/index a whole group.

Do you think this is possible?

Best Regards, Tim.

[Dmitrii Apanasevich]

When you say 'You won't have a possibility to index only specific branches', you probably mean once we have added microservices group, or?

I meant that project group will have only option of indexing all branches without a possibility to select specific ones.

* For me, it would be great, if your plugin would have some kind of radio button, .... Do you think this is possible?*

Yes, implementation will be something like you've described.

[Tim Klicks]

Hej Dmitrii,

Sounds great. We are looking forward to the new version which will also be able to index whole project groups. We will keep an eye on the releases ;-)

Best Regards, Tim.

[Tim Klicks]

Hej Dmitrii,

Do you have rough time frame when this feature will be added?

Best Regards, Tim.

[Dmitrii Apanasevich]

Hi Tim,

I'm afraid of giving you false hopes, but I think I'll have done this feature by the end of january.

[Tim Klicks]

Hej Dmitrii,

When I read the first words of your reply, I almost got a heart attack, because I thought you wouldn't include this feature. End of january is totally fine with me and I did not want to put pressure on you.

Have a nice christmas holiday and I am/we are looking for this feature in a further release.

Best Regards, Tim.

[Dmitrii Apanasevich]

Hi Tim,

I've implemented this feature and prepared release version. It has passed all automated and manual tests, but I need a fresh look. Could you test it? I'll attach it ti this thread if you agree.

[Dmitrii Apanasevich]

Resolves #12: indexing of a repository group.

Adds new feature that allows indexing of every repository (with all branches) whithin a group.

→ <<cset 2c93b9387e2c>>

[Surat Mukker]

Dmitrii,

We recently installed your plugin and it seems to be doing what we need to do. However we use github and we have multiple repositories as well.

We would be happy to try out and help test your changes as it will save us a lot of effort to add the repositories manually. Kindly let us know how we can try out the updated plugin.

[Dmitrii Apanasevich]

Hi Surat,

I've attached the new version.

All what has changed in UI is just a name of a field in settings page. Now you should key in Path and select type of the content in it: a single repository or a group (organization in terms of GitHub). For example, for organization https://github.com/flyway path is flyway.

So, you need to install the new version of add-on, add your organization at settings page (check whether your settings are ok by pressing Test connection). Then disable indexing of all single respositories (that you've configured before) inside this group so that not to duplicate commits. Later, when you will be sure that organization indexing works pefrectly, you can press Reindex button for disabled repositories. This action removes commits related to a repo. And then delete disabled repositories by pressing Delete button.

Feel free to ask questions and give feedback here.

[Surat Mukker]

Thanks Dimitri for such a quick turn around.

We have successfully installed the updated plugin.

The test connection works with the new plugin. waiting to see if it starts indexing the repos. Will update once have more information or if there are any questions.

[Dmitrii Apanasevich]

I've got an email about a comment on this issue with a piece of a stacktrace, but it looks like you've deleted it (hope, you've understood the root cause). But, nonetheless, I decided to give you a useful link: https://bitbucket.org/dmitry_apanasevich/jira-jigit-plugin/issues/16/error-null-commit-diff-for-sha1-while#comment-41764204

One of Jigit users has already solved a similar problem.

Hope it'll help you.

I look forward to hearing good news from you soon.

[Surat Mukker]

Hi Dmitrii,

Thanks for your tip.

Yes I had deleted the comment as immediately after posting it realized that it was not directly related to the plugin but a mysql error, so thought of googling first Found other posts regarding the character set encoding and set the values for the database, but that hasn't fixed the problem.

Haven't figured out the table/column that is complaining about the encoding. The exception is not printing the column name but rather 'TITLE'. Any clue regarding which table or column the character encoding should be set ?

Looked through the database and found the tables being used by the plugin and they are being populated.

Checked the table AO_E8B6CC_GIT_HUB_EVENT and found the tickets to check and the commit messages are showing up in JIRA.

It seems the indexer is starting from the earliest checkins and older tickets are being updated first. It is on July 2016 right now so it will take a while to catch up since indexer is running into github API limits, every so often.

[Dmitrii Apanasevich]

Any clue regarding which table or column the character encoding should be set ?

Of course. The exception is related to table AO_9F079B_COMMIT. Generally, all tables with names that starts from AO_9F079B_ are related to Jigit. The table you mentioned is not a Jigit table.

It seems the indexer is starting from the earliest checkins and older tickets are being updated first. It is on July 2016 right now so it will take a while to catch up since indexer is running into github API limits, every so often.

Indexer doesn't work with commits ordered by date. It starts working from HEAD and then goes down the tree. Algorithm traverses a tree in a level-order manner (breadth-first search). The result looks similar to an indexing from the last to the first commit.

Actually, Jigit settings page shows a quantity of indexed commits related to a group\organization. Does this number match the number of commits in your GitHub repositories? I wonder if the exception is related to the indexing process or JIRA throws it when a user clicks to a Jigit tabpage?

[Tim Klicks]

Hej Dmitrii,

Please excuse the very delayed answer, but I was on vacation and had no access to my mails... I have tested the new version today and for me (own hosted gitlab) it works perfectly.

Thanks a lot for taking care of this issue - I really appreciate this.

Best Regards, Tim.

[Surat Mukker]

Hi Dmitrii,

Sorry for the delay. Was able to update the table today and will watch for exceptions.

Meanwhile,

• out of 100+ repositories in the github, jigit is indexing 30 of these. Is there a limit on the number of repositories being synced ?

select distinct (repo_name) from AO_9F079B_COMMIT;

• There is an warning in the log file

17-Jan-2018 13:30:42.991 WARNING [http-nio-8085-exec-6] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://jira.plume.tech/rest/bitbucket/1.0/repository/47/sync, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.
2018-01-17 13:30:42,992 http-nio-8085-exec-6 INFO anonymous 810x89955x1 - 192.30.252.45,127.0.0.1 /rest/bitbucket/1.0/repository/47/sync [c.a.j.p.d.r.external.v1.RepositoryResource] Postcommit hook started synchronization for repository [47].
2018-01-17 13:30:42,993 http-nio-8085-exec-6 WARN anonymous 810x89955x1 - 192.30.252.45,127.0.0.1 /rest/bitbucket/1.0/repository/47/sync [c.a.j.p.dvcs.service.RepositorySyncServiceImpl] Sync requested but repository with id 47 does not exist anymore.

• I tried to do a reindex, but that resulted in an exception

2018-01-17 13:31:15,023 http-nio-8085-exec-1 ERROR surat 811x89956x1 pa095y 96.95.217.146,127.0.0.1 /rest/jigit/1.0/repo/plume-design/clear [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: null
jigit.indexer.api.LimitExceededException
        at jigit.indexer.api.github.GitHubErrorListener.onError(GitHubErrorListener.java:36)
        at api.client.http.ApiHttpRequester.handleError(ApiHttpRequester.java:108)
        at api.client.http.ApiHttpRequester.parse(ApiHttpRequester.java:96)
        at api.client.http.ApiHttpRequester.withResultOf(ApiHttpRequester.java:66)
        at api.client.http.ApiHttpRequester.withResultOf(ApiHttpRequester.java:58)
        at jigit.client.github.GitHubOrganizationsAPI.repositories(GitHubOrganizationsAPI.java:33)
        at jigit.indexer.repository.RepoType$2.repositories(RepoType.java:33)

[Dmitrii Apanasevich]

Hi Tim,

*Please excuse the very delayed answer, but I was on vacation and had no access to my mails... I have tested the new version today and for me (own hosted gitlab) it works perfectly.

Thanks a lot for taking care of this issue - I really appreciate this.*

It's totally ok. Thanks for testing, I'll publish a new version of add-on soon in the Marketplace. It'll be the same code that you've already installed from the attachment above but with a new version number.

I'll be grateful if you post a review for the add-on here. Not just a review about this feature but a review of the plugin. Whether it's useful\user-friendly\easy-to-use and so on.

[Dmitrii Apanasevich]

Hi Surat,

*Meanwhile,

out of 100+ repositories in the github, jigit is indexing 30 of these. Is there a limit on the number of repositories being synced ?*

There is a limit of API calls at GitHub side, so you either have to wait or to ask GitHub support about increasing this limit for you.

Have you seen your commits on issue tabpages for that 30 repositories?

There is an warning in the log file

It's not related to Jigit. It looks like logs from add-on for Bitbucket integration.

I tried to do a reindex, but that resulted in an exception

It is because of you've ran out allowed GitHub API calls. Actually, Jigit setting page displays a warning if this limit is exceeded.

[Dmitrii Apanasevich]

Hi Surat,

Do you have any news?

[Surat Mukker]

Hi Dmitrii,

Sorry have not had any time to spend with the plugin for a week now.

But it seems the group option will not work for us as we have a lot of active repositories.

Will update you once I get some time to spend with it.

[Dmitrii Apanasevich]

Hi Surat,

But it seems the group option will not work for us as we have a lot of active repositories.

Could you give some more details about the problem? You don't see recent commits on issue tabs? Or maybe it's related only to some specific repositories whereas others are indexed totally?

I look forward to hearing from you. Feel free to ask questions.

[Surat Mukker]

Hi Dmitrii,

A lot of active repositories means that Github API rate limit of 5000 per hour is reached very quickly, before it can index all changes. So only some of the repositories are getting indexed. Since there is no way to choose which repos to index (other than adding them one by one and not using the group option). For the repositories getting indexed the commits are showing up on the JIRA issues.

The plugin is working as expected, it is not working for our use case.

Thanks for your help and quick responses and turn around times.

[Dmitrii Apanasevich]

Ok, I see the problem. As you've mentioned, the only way using Jigit in your case is to add repos one by one.