1. Okapi Framework Avatar Okapi Framework
  2. Okapi and localization
  3. Okapi
  4. Issues

Please publish Okapi Core/Steps/Filters to Maven Central

Issue #670 resolved
Sean Flanigan created an issue

I know this has been on the radar for a long time, but it would be good to have an issue to track, even if it could take a while.

An earlier discussion from 2014: https://groups.google.com/d/topic/okapi-devel/-K5CiuJTcmk/discussion

Our use case: Zanata's client-side and common code is in Maven Central (via OSSRH), but we've kept the Zanata's server-side code out of Central because of dependencies which aren't in Central. At this point, we think all our dependencies are in Central except Okapi. It would be nice to knock over that last bowling pin.

Comments (19)

  2. Sean Flanigan reporter

    Let me know if I can help.

    Regarding some of the things discussed previously, I would recommend keeping it simple: keep the versions as they are, keep testing the modules together, keep releasing them together (with the same version) from a single git repo.

    Splitting into multiple git repos with sub-modules was also discussed, but I don't think the maven-release-plugin likes sub modules. If repo size is a concern, note that I was able to reduce the .git directory from 301MB to 237MB using https://rtyley.github.io/bfg-repo-cleaner/ to purge test_large.txt (672MB) from the history. Alternatively, if I purge all obsolete files larger than 5MB (mostly test documents), I can get it down to 143MB. Note that this would require rewriting history, but that's the only way to reduce a git repo's size anyway. In any case, I think it's better to keep these issues separate.

  3. Chase Tingley

    Thanks Sean, I forgot about that discussion.

    Reading through that thread, there was a lot of discussion publishing them in various bundles (ie, all filters in a single artifact). Thinking about it again now, is there a good reason to do that? Why not just publish the individual artifacts as we define them in the project?

  4. Sean Flanigan reporter

    Why not just publish the individual artifacts as we define them in the project?

    That seems perfectly fine to me. Better than combining them into larger jars, I would think. I think it's mainly a matter of deploying the artifacts as they are, but to Sonatype's Maven repositories instead of to Cloudbees'.

    There's a guide for OSSRH here: http://central.sonatype.org/pages/ossrh-guide.html

    This is important too: http://central.sonatype.org/pages/requirements.html The GPG signing requirement (for releases) can be a nuisance from a CI point of view, because you need a way of getting the GPG keys to the CI build node. Unless you make releases from a single machine (as we do for Zanata).

  5. Chase Tingley

    I went through that process once for a small library I wrote, it was pretty straightforward, but I was building locally and wasn't automating everything through a CI. @nmihai_2000 has been doing a lot of the build tinkering recently, maybe he has some thoughts.

  6. Mihai Nita

    There are safe ways to have the GPG in jenkins (but accessible to people who have admin access to Jenkins) If that is too broad, then I can dig to see if there is a way to input some kind of password in Jenkins when you start the build "by hand" to unlock the GPG (so that even the Jenkins admins don't see it if they don't know the password)

    =====

    I've been through the OSSRH process a while ago for a small project of mine, and was pretty painless.

    Publishing the current maven build results is probably easier, without merging them into a "monster jar" The current "monster jar" does not really follow the "normal" way of listing dependencies (they are listed in the MANIFEST.MF file, instead of being in a META-INF/maven subfolder, in a pom.xml and pom.properties)

    We will need to supply javadoc and sources (http://central.sonatype.org/pages/requirements.html). Normally we don't generate these .jar files. Might need to fix any potential failures in generating the javadoc (we only generate javadoc for core) And the javadoc for non-core stuff is not that great, to be fair :-) But I often find the sources very handy...

    There is no need to publish the test jars.

    =====

    Size: The compiled jar files are about 5.6MB The tests about 46MB (but I see no good reason to publish them :-) The sources in one jar are 2.6MB (we "pack" them in with the "merged" library), so splitting them will add some more (zip headers :-), but I don't expect it to go beyond 3MB.

    =====

    I have used the cloudbees repo to build using maven and download okapi automatically. The good thing with independent modules is that you can control exactly what to download. The cons is that you need to know exactly what you need. To write an application that supports all formats (something like Rainbow, or Tikal) you need to manually list all the filters and all the steps and all the connectors. Since they are largely independent, they will not be "dragged in" as maven dependencies.

    It might be possible to create a "bootstrap jar" that lists everything as a dependency, but with not code. So if you want to support all you list the "okapi-all" artifact in your pom... But I have never tried it

    Mihai

  8. Mihai Nita

    Instead of a bootstrap jar for okapi-all, I think you can create a pom artifact, which has the other jars as dependencie.

    That's what I was proposing with the "bootstrap jar", because I am not sure you can deploy a pom to maven, has to be a jar, no?

    So when you pom.xml (like the one you describe) you end up with an empty jar (no code), that has the pom.xml in the META-INF/maven subfolder. And then you can depend on the okapi-all, the jar with the pom-only gets downloaded, and that one "drags" with it all the modules.

    I call the "monster jar with all the code inside" a "shadow jar" (maybe incorrectly?), but that's what I think I've found (at least in some parts of) the internet :-)

    As a (not so much of a fan or user) of ant, or regular java user (no maven / gradle / ivy magic) I like the "shadow jar" for one reason only: when you use it in a project it is easy to associate a javadoc / sources. It is easy to depend on 100 jar(s): you just add the folder to the classpath. But then telling Eclipse on Intellij where are the sources for each mini-jar... no so much fun :-)

    Mihai

  9. Sean Flanigan reporter

    That's what I was proposing with the "bootstrap jar", because I am not sure you can deploy a pom to maven, has to be a jar, no?

    Pure pom artifacts are okay too. See https://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-pom-best-practice.html#pom-relationships-sect-grouping-deps. It shows how to define the pom artifact, and how to consume it from another project (you just add it as a dependency with <type>pom</type>.

    Eclipse and IntelliJ can find source jars just fine for Maven projects, as long as the source jars have been uploaded to the Maven repository. Both IDES should offer to download the relevant source jar when you open a class which has no source yet.

    So if you don't care about Ant, there's no reason for monster jars that I can see[1]. Smaller, modular artifacts are the way to go.

    [1] for libraries, that is: packaging everything you need into an uber-jar sometimes makes sense for end-user applications.

    In any case, I think creating okapi-all.pom or okapi-all.jar would be best considered as a separate task from uploading to Maven Central.

  10. Mihai Nita

    Eclipse and IntelliJ can find source jars just fine for Maven projects

    I know... but not everybody want to use Maven (in fact I don't use Okapi through Maven, I only use Maven for Okapi development :-)

    But the publishing to Maven will be modular artifacts, definitely. People who don't use Maven will not get their Okapi from a Maven repository :-)

    I have opened an issue with Sonatype: https://issues.sonatype.org/browse/OSSRH-37100

  11. Mihai Nita

    @ Yves: can you please create a JIRA user and send it my way? I will add you to the ticket so that you can deploy artifacts.

    Same for everybody else who might need to release Okapi

    Although it is technically possible that the release and the publishing is done by different people :-) And should also be possible to add more people later. I was just impatient to get this ball rolling (after sleeping on it for months and months :-)

  15. Mihai Nita

    I have also spent some time to read the whole thread from the old thread (https://groups.google.com/d/topic/okapi-devel/-K5CiuJTcmk/discussion)

    And there are many questions there that are valid, and are not yet answered. So the Sonatype process is in motion, but it does not mean that all the problems are solved :-) There is some "deciding" on what to do, and then doing it, before we can do a real publish... (well, we can probably do one easily, but not the quality one would like :-)

  16. Mihai Nita

    Added Chase, sorry, I did not see it in time...

    The case was approved, it looks like we can release any time we want now... Except that we are not quite ready, I think :-)

    M.

  20. Log in to comment
Assignee
–
Type
task
Priority
major
Status
resolved
Milestone
M36
Version
–
Votes
1
Watchers
3
Jira: the preferred issue tracker for Bitbucket. Join the team!