-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Conversation
…ion of the dependency plugin by writing a pom.xml file dynamically.
…ing of specification methods. Instead, focus on using the mvn: version of the gemname when needed and use the maven_name everywhere else as that is what is used by the dependency resolver and runtime.
For those of us that don't use Java libraries via Maven, can you please describe why you wrote this patch, and why you think it is a good idea to add this code and functionality into the core of Bundler? Thanks. |
Hey sorry I should have added some more detail. Basically, Maven, among other things, provides dependency management for Java applications. Integrating the ability to include Java dependencies specified via Maven into JRuby projects would seem like a natural fit with Bundler (for those who use Bundler with JRuby). I have explained in more detail this effort at my blog (http://hokiesuns.blogspot.com/2012/02/bundler-maven-for-your-jruby-projects.html). I hope that this helps more! |
I think the reverse direction can be better. So, if maven can download and install gems (or just call bundler), then it would be more useful, as you can deploy your ruby/rails application via maven, instead instructing bundler to do some magic with stuffs what is totally unrelated with ruby. To make clear: I think bundler is basically developed for ruby applications and it supports jruby too (because it is just an another ruby implementation), but it is not dedicated to support java applications in any way. Java applications uses maven, and I think bundler cannot be and would not be a replacement for maven in any context. However, If maven can handle gems and/or bundler, similar as it currently handles e.g. ant tasks, it would be more useful and more nice, because developers can do deployment less steps than current solutions, because maven can handle easily a lot of known application servers. So, summarizing: I am very skeptic about this idea. |
I think you are misunderstanding what I am trying to do. I am not proposing using Bundler to deploy/manage dependencies of Java applications in any way. I am saying that my JRuby applications that need to leverage Java dependencies should be done in Bundler using Maven as a proxy (as opposed to me going to the web and downloading the jar and dependencies and include locally) or worse manage a pom.xml file in my JRuby application with some out of band link between them. Having Maven call Bundler is a good thought and someone else (I think I referenced it in my blog) has done that but I am not a fan of using Maven in my JRuby application because:
I want to leverage Maven in Bundler for the dependency management function only as that is what Bundler does well. |
Hi guys, I think this could be a interesting approach, considering we can have gems with jar dependencies like this, |
@ANithian but bundler is pure-ruby stuff, and it is not contains platform-dependent codes, except some workarounds what is needed for running on some systems. But your code and your idea is absolute platform-dependent: it can be run only JRuby. Other platforms cannot gather any value from it. And again, I do not think it is a job of bundler. If we talk about it, we can talk about how bundler can install some dependent packages on Linux e.g. via PackageKit too. The two thing is not too far from each other. I think the way where we are is bad. Even if the idea is can make your (and some others) life easier and the implementation is possible perfect, it is not a job of bundler. Bundler is targeted for handling ruby dependencies. Period. Similar as it will die if C extensions cannot be compiled because a missing dependencies (think to nokogiri and libxml2-dev), it can leave skipped the Java dependencies too. These are not important from bundler's point of view, and managing of them is not a job of bundler. But, this is my opinion. I do not know point of view of @carlhuda but if I were the developer of this stuff, I wouldn't accept this feature request. But I am not responsible to talk about the state of developer(s). |
I am not sure why the slippery slope that somehow Bundler dealing with Java dependencies therefore means that Bundler could manage OS dependencies. The whole point of JRuby, as far as I thought, was that it was a way to be able to build Ruby projects that can leverage the power of Java libraries; therefore, my argument is that supporting Java libraries should be something that a Bundler that works in JRuby should do. If there is a way for me to take my changes and create a bundler-java extension that works, then please let me know how I can do that and I will be more than happy to undo the pull request and put my changes into a gem that extends bundler so that in a Gemfile, certain DSL around maven dependencies will be understood. |
What is the argument opposite for Maven? I do not think you run rails application alone in production mode. If I were start JRuby project, I will use maven for deployment, as it can do it easily (mvn deploy), like capistrano does. From there is the one step to adding Java library dependencies for maven (what is could be an easy job if you use an IDE what know pom.xml, like NetBeans, what can complete groupid/artifactid/versoin for Ctrl+Space). I can imagine and more agree with extension what can works like "gemspec" statement, but bit other: call maven and let it to process pom.xml to installing dependencies (and/or build Java libs in current project). This is a better compromissum rather than extending Gemfile DSL and Bundler to handle dependencies what
Handling it a lillte more untouched from Bundler can be better, and more useful. And can be make more lazy dependency between Bundler and maven. Maven's complexity is not an argument, if you learn it, it will be simple. My other argument is if you let maven to do it's work, you can build some project-related Java codes too (e.g. from src folder) what is the perfect area of Maven. |
I think that maybe there are two use cases which are useful here:
In this case you're right, making maven install the JRuby dependencies is absolutely the right way to go, and it wouldn't make sense to have bundler run under JRuby to install maven dependencies for the Java application.
This is a case where having a java platform specific extension to bundler's DSL would make a lot of sense. In fact, in my mind this is a natural extension of what the java platform functionality in bundler's existing DSL already does. Namely, because if your application is primarily ruby with a few Java libraries under the covers, it's useful and desirable to have one dependency installation technique which is applied uniformly for all dependencies. It is in fact very powerful, because it allows you to use any maven repository as though it were a gem all the way down to the deployment script. I believe that this can be added as an extension to bundler, but if bundler opts to support a more full segment of the java functionality alongside its support for the JRuby platform, this is not a bad step in that direction, and, for non-JRuby users it doesn't present any extra difficulty provided this add-on doesn't impact existing functionality (which, from the commits, it doesn't appear to do). From my understanding of this code, it actually takes a set of maven dependencies and makes them behave like gems both from a use-case and dependency management perspective. As an application developer, this seems like a big win to me, because, when I choose JRuby as a platform for some I application or web service I do it because I like Java's library support and the underpinnings of the JVM, not because I have a burning desire to learn a new technique for dependency management. The more the JRuby ecosystem can hide the complexity of Java without obstructing its power the better. |
I do not know Bundler internals, but I wouldn't like to see this in the core of bundler. As an extension/plugin/addon, yeah, it is an absolutelly awesome idea. But in the context of the core Bundler code it is very platform-specific thing and I think start going on this way is not a best idea. The PackageKit example was a very-very extreme example for the ending of this way but if we allow a small thing we can expect a need for a biggest things. Again, I not argue against the Maven, but I argue against doing platform-specific package handling in Bundler code. When I say platform, I mean Ruby platform what is currently JRuby. |
jRuby already ships rubygems with maven support. However, bundler has no notion of maven so being able to install a maven jar as a gem has no use when you can't include it in your Gemfile. I would love to see this commit go into Bundler. |
The only complaint I had with the maven gemify support in rubygems is that it downloads and packages the jar as part of the gem rather than referencing the possibly already downloaded jar that exists in your maven repo. This isn't a huge deal to change but I figured if you have to download stuff for the custom plugin then why not use the repo. I don't know if people package gems with the app or use Bundler to re-download the gem in the new environment (which is what I would personally do). |
I understand the reasoning for this feature, but doing it within bundler feels like the wrong place to do it. Since jruby is the only implementation that can utilize this functionality and jruby already has this exact functionality built-in their rubygems, the right way to handle this is to have bundler call into rubygems at the proper places to allow the code already in jruby to handle these maven deps. This allows jruby to own the code related to maven, which is how it should be. They can release fixes, etc on their own schedule without depending on the bundler team. I'd like to understand how bundler currently does not honor the code within jruby to handle maven deps. Could someone familiar with the situation please comment? |
A bit of background... The "maven gem" feature has been developing in JRuby for several years, and only in JRuby 1.6 did we ship a "preview" capability that allowed installing maven artifacts as gems. We do this by patching two places in RubyGems: spec retrieval (and logic to find that spec based on a gem name), and gem retrieval. Spec retrieval is hooked to recognize that we're installing a maven group:artifact style name and build a gemspec from that library's maven pom.xml file. The resulting gemspec has all the typical information, with dependencies (from maven), project information, and so on transfered from the pom file. Gem retrieval then fetches the actual library from maven and wraps a thin gem around it. The gem never really exists anywhere...it is created dynamically from the maven artifact. This works very well in JRuby's copy of RubyGems, but it has never gained wide traction primarily because Bundler does not hit our patched code. Bundler does its own fetching and spec resolution, which means it does not recognize maven artifacts and does not know how to install them. This pull request intends to fix that. @hron84 The change is made in bundler because bundler does its own thing and doesn't see our patched logic in RubyGems. If bundler could use the logic from JRuby, we'd happily put the logic in JRuby. Since it does not, we're left with a choice: have this incredibly useful feature of JRuby not work in Bundler, or patch Bundler to support it. As for whether platform-specific features should be in Bundler or RubyGems...I can easily argue that C extensions, which don't work consistently across all implementations, represent a platform-specific feature. We're not even asking for something as complex as building C extensions...we're simply asking for support to wrap a jar file with a gem wrapper on the fly. Truth be told, this could be made into a general feature as well, if other Ruby impls are interested in being able to fetch jar files from Maven as gems (e.g. for use with RJB). @evanphx As mentioned above, it's not possible to tweak RubyGems 1.8.x right now in a way that Bundler will see, so we're forced to patch Bundler directly. Believe me, we don't like it either, but we don't have another choice at the moment. The diffs for our copy of RubyGems are, as always, available in our forked repository. The current branch we're using is here: https://github.com/jruby/rubygems/tree/jruby-rubygems_1_8_15. Most of the changes are related to the Maven support. One last note: For JRuby 1.7, we will not be using the : character to separate group:artifact:version, since it's disallowed as a path character on Windows. I believe we will use the @ character, but the decision is not yet final. |
Thanks Charles that was quite useful and needed. The reason I made my own version of the maven_gemify is that I wanted to avail of Maven's already in-built ways to download dependencies (albeit in its own local repo) and not wrap the jar into the gem directly but rather require the jar that lives elsewhere on the file system. Also, this patch simply serves as a syntactic way to bridge two dependency management systems using an adapter that in truth exists within JRuby. The version I wrote was just to do it differently (because I wanted to use a jar that could theoretically exist on the filesystem if you already are using Maven). I don't see another way around this other than to patch Bundler to provide this syntactic sugar to bridge these two systems together. |
@headius Could you help me understand the difference between this code and the code in JRuby? I've started compare them, and a lot seems different. |
@ANithian I am in favor of using local jars and normal maven retrieval mechanisms too. We should try to unify this code. |
@headius Sure thing this would be awesome! How would you like me to proceed? I can take my maven_gemify2 and submit it as a pull request to JRuby if you wish. @evanphx There are two major things I did in this patch.
|
This seems reasonable but the main point that is still missing is having one thing manage both java and Ruby dependencies together. Jbundler would be a nice wrapper on maven but still force me to use bundler and where would the maven dependencies be loaded in my rails app? |
@mkristain I don't understand how jbundler and bundler would cooperate. If you have a jruby-gem that requires a jruby jar from maven, how do the versions get locked correctly? Especially in a situation where two gems require the same jar, but slightly different versions, but one gem is in It seems like jbundler should be a plugin for bundler:
So that a single |
I agree Christian, it can't be an isolated project that will only call bundle when we're dealing with JRuby-specific gems that depend on Maven artifact (which should be preferred to embedding the jars in the gem). So, your suggestion is adding some kind of plugin support to Bundler, right? How would plugins be found? Shouldn't we declare a "gem 'jbundler'" before calling "plugin 'jbundler'"? But this way it wouldn't understand the extensions the first time the Gemfile is parsed. Or should we issue a "gem install-plugin jbundler" first, and that would create some sort of .bundle/plugins file that would be loaded first before trying to parse the Gemfile? I know this is not the hardest part of the Maven-Bundler integration but this is one that should also be solved. A while ago I suggested in the JRuby list that we could try to add some hook to Bundler so that we could develop a separate gem that could register some handlers to Bundler hooks and maybe in the future we could try to issue a pull request to Bundler to automatically call that gem instead if a Mvnfile exists... But actually as soon as this feature is completed someone will ask for Ivy support or something else. With this in mind, maybe a "plugins" framework could be a great add-on to Bundler, but it should be better defined how that would work... |
Okay, so here is an attempt to flesh this out a little more:
Bundler can read the Gemfile twice. The first pass is for plugins, the second for everything else. The 'plugin' directive would only allow gems -- bundler is a ruby project, and writing plugins should be ruby. However, a plugin, once loaded, can load other things (e.g. jars via maven). In the first pass, only On the second pass, The advantage of having the Should the "bundler plugin" idea be added to a new pull request or issue? |
This seems like a reasonable approach to me. Also, I don't mind if Bundler will use a different logic from Maven because the Gemfile.lock will remain the same. Currently Java applications also have to deal with conflicts created by Maven resolution rules too, but adding exceptions manually so this also could be applied to the Bundler approach. There is an important difference though. As far as I understand, Maven will never fail on resolving dependencies and will always choose some version. In the other side, I think that Bundler won't allow you to have two gems in the dependency chain where A depends on 'dep=1.0' and B depends on 'dep=1.1'. Another issue, as pointed by Kristian, is that Maven allows any string as a version specification, while Gems don't. So, I'm not sure how to deal with this. Should Bundler abstract that idea to support versions other than the currently supported ones or should plugins map those strings to something that could be gem-compatible? |
as @rosenfeld said maven version resolution is not compatible with as far I understand things, having one file Gemfile for declaring the the situation becomes tricky when you want a rubgems with a dependency "jar 'org.slf4j:sf4j-simple', '1.5.6'" which bundler and rubygems just ignore, but jbundler could obey it. |
My example above didn't include versions for the So basically, below the covers Bundler just hands the This probably means that |
mapping maven artifacts onto rubygems I did the last year or so and things just became more and more messy. there are a couple of more things which will never work with rubygems. so I am eager to find a way handle things properly. ruby-maven gem does handle both Gemfile resolution and jar file resolution with maven and uses bundler only to generate the lock file - the lock file for jar dependencies is not needed since it is done the maven way (and I never ran into version ranges in maven until now). actually ruby-maven already obeys the requirements I mentioned before (http://blog.mkristian.tk/2011/09/jruby-and-rubygems-and-javaclassloader.html) so the question is if bundler will be able to achieve a similar solution. |
I kind of suspected that you couldn't lock versions for maven. Well, you could, but you'd have to bypass maven when loading from a 'locked' project (i.e. download the jars directly). But there would be no way of checking of dependency problems when downloading, which would suck. |
So one thought still remains. I don't think a separate MvnFile works for reasons of still needing integration with Bundler although to me, the pom.xml IS the lock file as you have to define versions. Also I am not sold on a separate DSL on Maven unless it's directly integrated into Bundler. The plugin idea works but this seems like a fair bit of refactoring on Bundler? Also in doing this patch, I had to make some hard-coded changes which I wasn't thrilled with. That being said, a simpler integration of the two could revolve around NOT gemifying maven dependencies for all the reasons mentioned here but rather just using the pom.xml as the "lock" file and upon Bundler.setup, invoke the plugin that prints the classpath and simply require all those jars. I do want to make sure that boot time of a Rails project isn't slowed down by downloading Maven dependencies so the Gemfile.lock would have to store some meta-information about the pom.xml so as to make sure the "install" phase does the jar download and the "setup" phase simply does a mass require of jars. |
As a developer, I'd prefer something that works like bundler: I
|
I agree... I was just trying to see if there was a way to make it so that it all sits in bundler and avoids another DSL on top of Maven. Pom.xml files are like Gemfiles you are right but since they also mandate a version in the dependency it also serves a lock file for given a pom.xml you are ensured to get a version of the dependency compatible with your code since you either requested a version or a range of them (admittedly I have never used version ranges before). |
since the next version of maven will come with different DSLs and it I really do not see the need of a pom.xml anywhere. when I have a if I get a hook to have jbundler be a plugin for bundler, that would git clone abc but putting a 'j' infront of it could do the job as well. and with some luck the jbundler DSL becomes the maven DSL, at least I currently the overhead of jbundler after an initial setup of the so I still do not see what does NOT WORK with external Mvnfile - since rubygems uses a specification file to allow "dependency managers" to so again how does a gem declare a dependency to a jar file ? once that |
to demonstrate what I mean I setup an example on http://github.com/mkristian/jbundler which shows how to setup gems with jar dependencies. how maven resolve version conflicts - compare the slf4j version of my_project/Gemfile with gem_with_jar/gem_with_jar.gemspec but I do admit a tighter bundler integration would be nicer since the jar resolution now takes place on the first run. $bundle install would be the better place for the resolution to happen. next I will add version ranges like "~>1.2.3" for jar dependencies . . . |
just one last remark of mine: jbundler is now in state where you manage you jar dependencies with an extra "Mvnfile",
groups would be no problem if it is possible to ask Bundler from a running application which groups were set up !? |
hi, just pushed a first gem of jbundler. jbundler tries to manage jar add to you Gemfile gem 'jbundler' any jar dependency of your project goes into Mvnfile. something like: jar 'org.slf4j:slf4j-simple', '~> 1.6.2' you need to use "Bundler.require" to setup your gems environment and the jbundler does the resolving at the first run of your software (not it will create a Mvnfile.lock to lock down the version for the above jar 'org.slf4j:slf4j-simple', '1.6.2' IMPORTANT any little problem you encounter will help to improve things when hope it will help your project. |
Thank you very much for your effort, @mkristian! Much appreciated :) |
I'm running into this problem right now and have come up short with solutions. Is jbundler the best way to go? Are there any other solutions? Is no one else really doing this in their projects? |
Hi @arthurkalm, have you tried jbundler? It worked for me. Let us know in the JRuby mailing list if you run into any troubles with it. |
I don't use JRuby myself, so I'll leave this ticket open for discussion. On May 21, 2012, at 11:34 AM, Arthur Kalmensonreply@reply.github.com wrote:
|
@rosenfeld I gave jbundler a try but I couldn't get it installed. I was going to go down the path of trying to get all my gems installed using JRuby's maven-gem with rake. |
Hey @arthurkalm, I've just tried in a fresh RVM gemset on JRuby 1.6.7 and it worked for me. Could you please try this application? |
Well, there is some cleanup to be done in the jbundler gem for sure, but at least it works :) There is still some debugging output in the released gem but that should be easily fixed. |
Also, please try to install Maven 3 first. At least I had it already installed, so maybe it does make some difference. If you're on Debian unstable like me, this means 'apt-get install maven'. |
With regards to verbosity, I've just asked @mkristian to release a new version without it: |
Thanks @rosenfeld, things are sort of working now. I still can't depend on internal projects, but I'll keep issues going on jbundler. I guess this would be the official way of integrating bundler with maven? Are people using any other methods? Are people even doing this at all? |
Hi @arthurkalm, currently I'm not working with JRuby or with Maven artifacts on my Rails applications yet. But I don't think I would have a problem if I decided to. In the company I worked on previously we used to use Sonatype Nexus for hosting our custom internal libraries: http://www.sonatype.org/nexus/go Why do you think that wouldn't work for you? You are not required to use it but it is better than manually "mvn install" your custom library dependencies. |
Also, if you decide to package your internal projects as gems, jbundler will also support it according to its documentation (haven't tested it, but I believe it works because the author is using it on his real projects). |
@rosenfeld we're also using Nexus and I've created a new issue on jbundler. Anyway, we should probably continue the conversation there, don't want to spam everyone on this pull request :). Thanks for your help. |
I have included support for maven dependencies using my own re-written maven_gemify library that relies on the underlying maven dependency plugin. I suspect though that this could be replaced with the native maven_gemify library without much hassle but I haven't tried.