Modified readme2modulestrings.py so that it will work in a MinGW Windows environment.

#656 Merged at e663af4
Repository
daniel_fort
Branch
Windows-port
Repository
hudson
Branch
unified
Author
  1. Daniel Fort
Reviewers
Description

It may not be elegant but it works for compiling the command line tools on a Windows system set up with a simple MinGW install. Not cygwin that requires .dll's for the binaries to work or Linux in a virtual machine. Building raw2dng requires pull request #655 lv_rec: use fseeko/fseeko64 depending on platform as in mlv_dump.c. I'm keeping the pull requests separate because the raw2dng.c change is sort of a no-brainer copy of what is already working in mlv_dump.c and this one required lots of testing to get it to work. One "side effect" of this is that it is now possible to build ML in MinGW in Windows as long as the ARM cross compiler is installed in the home directory. However, I have not been able to get a working autoexec.bin yet.

[EDIT: Found the issue with autoexec.bin, xor_chk not working in Windows.]

[EDIT2: Found the solution to the autoexec.bin issue, thanks to @dmilligan for pointing it out. Pull request #658 build_tools for Windows.

Comments (22)

  1. Johanan Pandone

    Wierd that you can't get a working autoexec...are you doing "make clean" first? also try "make zip" and try the autoexec bin from the zip that's created. Sometimes my autoexec has problems when I do regular make, but if I do make zip and copy entire contents from zip file it works perfect....

  2. Daniel Fort author

    It may be something misconfigured or missing from my MinGW installation. I just wiped it out and started from scratch, making notes and screenshots along the way for a possible tutorial, but still no working autoexec.bin.

  3. Marsu42

    Don't let me detain you trying to compile with bare-bones mingw, but I used it for some time and found it's really not worth the hassle. The (launchpad) arm gcc compiler is a windows exe anyway even under cygwin, so it's only about the make system and a couple of helper files - plus tex, if you generate the docs. Even with mingw, most people are likely to use msys as a shell which is an outdated cygwin fork.

    Cygwin is much more maintained these days, automatically updated and even available in x64. For all but the very most die hard users, I'd +1 making ML's compile including all modules work 100% with cygwin (which it necessarily doesn't out of the box, line endings and such) as it's the easiest way for win users to set up an environment to hack the ML code or get some branches/pull requests merged.

    Some time ago I even uploaded a complete cygwin + arm-gcc zip package - unpack it anywhere, run a shell script and bang there's your most current autoexec.bin

    1. Daniel Fort author

      This pull request isn't for compiling ML, though it can almost do that, it is for compiling the command line tools. Building cr2hdr.exe, mlv_dump.exe and raw2dng.exe in Cygwin will result in binary files that depend on Cygwin1.dll and possibly other Cygwin dll files. Wouldn't it be better not to have these dependencies for distribution?

      Note that I have a related pull request open, lv_rec: use fseeko/fseeko64 depending on platform as in mlv_dump.c. Apparently mlv_dump is already set up for compiling in a Windows environment. With these two pull requests it is possible to also build cr2hdr and raw2dng in Windows.

      Note that these changes do not affect cross compiling from OS-X or Linux to create Windows binaries which I believe is what most people are doing.

      1. Marsu42

        Ah, right - but it's both (nearly) the same as even the .exe tools use the base ML system and the arm gcc compiler? But even under cygwin many companion tools still need fixes in the Makefile, it would be certainly nice to have auto-detection for cygwin & msys (i.e. "mingw").

        1. Daniel Fort author

          "...even the .exe tools use the base ML system and the arm gcc compiler?" Not really, those are compiled by the host system's gcc compiler.

          "...it would be certainly nice to have auto-detection for cygwin & msys (i.e. "mingw")." That's the intent of this pull request--though it looks like Cygwin works more like Linux while Mac OS-X and Windows need some special handling.

    2. Daniel Fort author

      By the way--it appears that the problem with creating a working autoexec.bin under a Windows MinGW environment is with xor_chk. Copying a Cygwin built xor_chk.exe and including the required cygwin1.dll in the build_tools directory then building a platform in a Windows MinGW/msys environment will produce a working autoexec.bin. In any case, I'm straying off topic, this pull request is only for compiling the command line tools but there is hope for more.

      1. Marsu42

        My experience with mingw using "real" windows exe builds of unix tools (i.e. not msys) is that you keep thinking "I'll just need to fix this one tiny flaw, and everything will be peachy" ... and then the next problem occurs, for example some features of ML don't work or whatever. A lot of unix authors simply refuse to include patches for windows into their sources, so you keep applying manual patches, recompiling, testing until you grow gray hair.

        Personally as written, I don't think having cygwin.dll-less versions of intermediary tools is worth the hassle, the speed loss is minor as the biggest problem of the windows kernel is the lack of fork() which for example makes a ./configure much faster in a Linux vm than under native Windows no matter what.

        1. Daniel Fort author

          Thanks for sharing your experience with MinGW. I was wondering if you have ever tried building "real" windows exe builds of the command line tools in Cygwin but not requiring cygwin1.dll by using a cross compiler like it is possible to do in Linux and OS-X. Looks like something to try.

          In any case, this is a little nerdy but in the spirit of a post by a1ex in the ML's Goal? topic on the forum. To paraphrase: "...it's more like a hobby project...while also sharpening programming skills and sharing knowledge...everybody is free to work on whatever he feels like; there's no hard to-do list." I'm not trying to pick the best way to compile ML and the command line tools within the ML source tree, I'm trying to maximize the portability of the source code just to see how far it can go. MinGW/msys is simply an interesting challenge.

          I started a topic on the forum, Compiling ML on Windows with msys/MinGW - The Sequel. Please check it out if you are interested in how this pull request came about and why I'm defending it.

          1. Marsu42

            Pleas don't mis-understand me - there's no need for you to "defend" your pull request, I just want to save you lots of frustration by commenting my personal experiences with "real" windows unix exe tools w/o the cygwin/msys dependency. I was very nerdy for a long time as well, I even tried building ML with the (first deprecated, now abandoned) MS unix kernel layer. But it all comes down to one experience: If some environment isn't properly supported you're in for endless patching, searching obscure newsgroup posts from decades ago and a lot of frustration.

            Btw I guess the "GNU for Windows" project isn't updated anymore for a reason :-), it's probably not worth the trade-off hassle vs. gain by getting proper windows paths w/o the unix emulation layer, esp. if you have to assembly all the "native" windows tools for yourself while there's a handy cygwin installer downloading all the dependencies available.

            Otherwise I am more than happy you're on improving the win build system, because at least if you're targeting msys-mingw this will automatically benefit cygwin as well. Btw, no, I'm not working for Redhat - just trying to share my experiences :-)

            1. Daniel Fort author

              Ha ha--funny stuff. I meant defend as in defending a thesis. Part of my motivation for working on this is to learn what it takes to support many platforms, both cameras and operating systems.

              I started out trying to use the original MinGW but in my journey found there are a pair of newer projects called MSYS2 and MinGW-w64 on SourceForge. These projects are being actively developed and can be setup either like Cygwin or MinGW. I was half-way kidding when I suggested cross compiling to Windows in Cygwin but MSYS2 combined with MinGW-w64 can actually do that. I'm working on a tutorial on how to set up a development environment with these tools but first I want to get the installation process simplified as much as possible.

              Back to this pull request. The shell script is pretty handy on its own, go to any directory and type the path to last_change_info.sh and it will return a formatted Mercurial report of the last change in that directory. I wish that it didn't duplicate the same code in the shell script that is in the Python script but this is the only way I can get it working on all platforms. I test on OS-X, Linux, Cygwin and of course msys/MinGW on Windows.

              1. Marsu42

                Interesting there's been some msys2 development again, let's see how long it lasts - after all, cygwin is maintainted by a full-time paid employee at redhat.

                As for mingw-w64: If you look further on sourceforge, there are a bunch of compiler/toolchain projects based on the basic mingw-w64 code, been there, done that :-p and ended up with cygwin once again. The cygwin setup has the option to download/update native mingw-w64 for both i686 and x86_64 - most important with all the dev libs and headers you'll ever need, they're not the same as for unix as a lot need mingw-specific patches. If you didn't try that option, I strongly suggest you have a look :-)

                Anyway, so much for my newest cygwin commercial and may your pull requests be accepted :-)

                  1. Daniel Fort author

                    I tried to compile ML with babun but couldn't get it to work. It is an interesting project though. I use a Mac to do pretty much everything, this is an interesting side project. I remember first using Cygwin a long time ago, it still has its loyal followers--eh @Marsu42 ?

                    1. Marsu42

                      :-) I nearly forgot about cygwin as it was unmaintained for quite a long time - but with the microsoft unix kernel being abandoned in win10, it's back again and the newest cygwin versions are actively developed, including a x64 port. It's really that I was so fed up getting manual msys+mingw "nearly" to work I by now know how convenient it is to have all relevant patches in question incorporated into the mingw compilers and dev headers/libs you can simply download instead of installing patching everything yourself.

                      Personally, I find cygwin ideal for quickly compiling ML in windows, the "download and use" cygwin zip I build sometime ago was only a few megs large (excluding the arm gcc compiler).

                1. Daniel Fort author

                  That full-time paid employee at Red Hat has a name, Corinna Vinschen. Funny I could find that but I can't seem to track down the option to create native Windows binaries. Are you talking about "-mno-cygwin smartctl" ? According to the F.A.Q. on the Cygwin website about static-linking:

                  Can I build a Cygwin program that does not require cygwin1.dll at runtime?

                  No. If your program uses the Cygwin API, then your executable cannot run without cygwin1.dll. In particular, it is not possible to statically link with a Cygwin library to obtain an independent, self-contained executable.

                  If this is an issue because you intend to distribute your Cygwin application, then you had better read and understand https://cygwin.com/licensing.html, which explains the licensing options. Unless you purchase a special commercial license from Red Hat, then your Cygwin application must be Open Source.

                  1. Marsu42

                    Yeah, the static linking of the cygwin dll was often requested but won't happen due to legal issues - cygwin compiles aren't to be distributed in "stealth", you found the paragraph. So you always have to put the dll into your path.

                    But cygwin is basically the same as msys fork, Cygwin simply has the advantage of being maintainted and having an installer/updater so you always use the latest toolchain and libs. You can download the mingw i686 and x64 compilers with the cygwin setup and then simply cross-compile to native windows - i.e. if you put the mingw compiler into your ml makefiles you'll end up with an exe that doesn't require the cygwin dll. All your ml tools like cr2hdr and so on won't depend on it.

                    Btw1 if you're using something with automake, simply do "./configure --host=x86_64-w64-mingw32" and you'll get a native windows exe - if the code in question doesn't depend on anything unix-ish that needs to be emulated with cygwin.

                    Btw2 for many more recent cygwin packages, use the "cygwin ports" repo, run the setup with "setup-x86_64.exe -K http://cygwinports.org/ports.gpg" and select the respective ftp link (more on the cygwin ports website on this).

                    1. Daniel Fort author

                      Wow, great stuff. You are the Cygwin guru!

                      Looks like I've got some more testing to do, this time on Cygwin. I take it that your Btw1 doesn't apply to ML because we aren't using automake. However, your Btw2 looks very intriguing.

                      Darn, right when I finally got MSYS/MinGW working perfectly.

                      1. Marsu42

                        Another soul converted to the cause :-p ... but as for your working msys/mingw setup, rejoice, once time passes you'd have to put work into it again to keep all the toolchain & lib/header stuff up to date while with cygwin you just have to run the setup again.

                        As for "cygwin guru" :-) ... it took me a looooong time to understand what cygwin, msys and mingw (different flavors like mingw64, mingw-w64) exactly is and how cross compiling from cygwin to native win32 or win64 works. I guess this is the reason why all the manual mingw and msys2 stuff is still being popular while cygwin does the same thing but with less work.

  4. Daniel Fort author

    Revisited running the shell script on line 121 and it is working on all build platforms. No need for the "if branch." (Thanks a1ex--I feel like when I take my car to the mechanic and it works fine.)

    Tested on OS-X, Linux, Cygwin, msys/MinGW and msys2/MinGW-w64