1. gyrokinetics Avatar gyrokinetics
  2. gs2
  3. gs2
  4. Issues

find regextype fails on mac

Issue #58 resolved
Ryusuke Numata created an issue

Giving -regextype option to find in tests/Makefile.test_utils causes an error on mac because find of mac does not have such an option.

find: -regextype: unknown primary or operator

I wonder if it's necessary to use "-regextype sed".

Comments (21)

  3. Ryusuke Numata reporter

    It seems the PR#97 is ok.

    But, I'm getting a strange behavior related to find. Whenever I compile something, it gives me the illegal option error three times.

    find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]

    For example, below is the output when I compile constants. I don't know where the find command is issued. This is nothing to do with tests/Makefile.test_utils.

    numata@tequila2% make constants.o
Makefile.tequila2:2: *** This is ~/include/Makefile.tequila2 *** (<-- this is just output from my local GK_SYSTEM Makefile)
cpp  -DFCOMPILER=_GFORTRAN_ -ffreestanding -P -traditional -I/opt/local/include  -DGIT_HASH='"e95cc21f01e4184ba0fe4490f1e5d4d939829034"' -DGIT_BRANCH='"bugfix/bsd-compatible-find"' -DGIT_STATE='"modified"' -DGIT_PATCH_FILE='".patch.make"' -DRUN_NAME_SIZE=2000  -DGK_SYSTEM='"tequila2"' -DMPI -DISO_C_BINDING -DNEW_DIAG -DFFT=_FFTW3_ -DNETCDF -DNETCDF_PARALLEL -DHDF -DHAVE_MPI -DFORTRAN_NETCDF -DISO_C_BINDING -DPARALLEL utils/constants.fpp -o utils/constants.f90
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
h5pfc  -ffree-line-length-none -fno-backslash -fdefault-real-8 -fdefault-double-8  -O3 -I. -I/Volumes/Data/numata/sim_local/UMD/gs2/utils -I/Volumes/Data/numata/sim_local/UMD/gs2/geo -I..    -I/opt/local/include  -I/opt/local/include     -c utils/constants.f90
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
find: illegal option -- r
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]
rm utils/constants.f90
  4. David Dickinson

    It seems that the find version in use does not interpret -regex as a valid whole option but is trying to interpret this as -r which is not recognised. Could you report the output of find --version (or similar). I think the reason this is shown three times is that it is called once for each type of test (unit, linear and nonlinear) as a part of defining the list_<unit/linear/nonlinear>_tests recipe - I'm not sure this is the case but I suspect it must be as I can't find any other use of find (it's a bit odd though as the call to find should really only be made as a part of running the recipe).

  5. Peter Hill

    -regex turns out to be non-POSIX, but is in both GNU and recent BSD versions. I'm pretty sure Mac OS X uses very old BSD binutils, so we have to stick to actual POSIX options. @rnumata I've updated PR #97 that will hopefully work properly for you now.

  6. Ryusuke Numata reporter

    Now, I get (again, three times)

    find: illegal option -- t
usage: find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression]
       find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]

    According to the online manual, find accepts -regex and -type. If I use those options from terminal, it's fine. But, from Makefile, it's not ok.

  7. Peter Hill

    It sounds almost like it's eating the - before the option

    Could you report the output of:
    - make --version
    - gmake --version
    - find --version

  9. Ryusuke Numata reporter 
    numata@tequila2% make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

    There's no option like --version for find of mac os. I also have gmake and gfind installed, but not used. (gmake gives the same result)

  10. Ryusuke Numata reporter

    Yes, there may be some issue on how variables are expanded. Who (make or shell) expands $, and when are they expanded?

  11. Ryusuke Numata reporter

    I figured out the reason for this error.

    The "export" command in tests/Makefile.tests_and_benchmarks forces to evaluate getListSource etc. where find is invoked. During this evaluation the argument $(1) is blank, so the command "find -type f ..." is executed. The "find" command on mac gives error when the path is not given, while that of linux does not care.

    SYNOPSIS of mac's find
    find [-H | -L | -P] [-EXdsx] [-f path] path ... [expression] find [-H | -L | -P] [-EXdsx] -f path [path ...] [expression]

    SYNOPSIS of linux's find
    find [-H] [-L] [-P] [-D debugopts] [-Olevel] [path...] [expression]

    Can the "export" command be removed?

  12. Ryusuke Numata reporter

    Here's a sample Makefile to reproduce the problem

    export

define find
$(shell find $(1) -type f)
endef

hello:
        @echo "Hello, world!"
  13. David Dickinson

    Thanks for providing this update.

    If the export in Makefile.tests_and_benchmarks is removed then I think this may prevent Makefile.utils from being successfully included in subsequent files such as Makefile.tests_root_common (at least that's what happens for me without any other changes as TEST_DIR is then evaluated as empty), which I'd expect to lead to an error from make. We can avoid this by adding export to the top of Makefile but that may well undo the fix of removing it in the tests file (and potentially exports much more).

    One work around for now may be to add ifneq ("$(MAKECMDGOALS)", "list_tests") near the top of Makefile.test_utils and an endif at the bottom -- without a mac it seems to be hard to test this, so this work around may not actually help. Another way is to just ignore the error by adding 2>/dev/null after the find commands, this is obviously not optimal.

    I think a more reasonable workaround is to replace define getListDir with define getListDir = (and the same for getListSource). Adding the trailing = makes this function definition deferred so it should never be evaluated until needed, at which point it should be called in the correct way (i.e. with an argument). Unfortunately I believe this might require make version >= 3.82, although I think that earlier versions will just ignore the define completely if it has a trailing =.

    edit: It seems the final proposed fix might not actually lead to a suitable workaround.

  15. David Dickinson

    @rnumata yes that might be a possibility.

    I've been able to partially reproduce this locally now by adding a non-existent directory into the find command used by getSourceDir and I can see corresponding error messages when running make commands that shouldn't invoke this command.

  16. David Dickinson

    With my version of the issue I was able to fix this by replacing

    $(sort $(dir $(basename $(shell find $(1) -regextype sed -regex ".*/.*.f.."))))

    with

    $(if $(1),$(sort $(dir $(basename $(shell find $(1) -regextype sed -regex ".*/.*.f..")))))

    (and similar for getListSource). This ensures that if $(1) hasn't been set then the function is a null function.

  17. David Dickinson

    Makes getListSource and getListDir null functions when called without arguments.

    This can happen whilst make parses the Makefile.

    The fix hopefully avoids distracting errors on systems such as mac which require a path to be passed to find.

    See issue #58

    → <<cset 670b9eca3a05>>

  18. David Dickinson

    I've added a branch bugfix/avoid_calling_user_functions_with_no_argument_makefile that hopefully provides this fix.

  19. Ryusuke Numata reporter

    I've merged the branch bugfix/bsd-compatible-find. Now, the problem is resolved.

    See pull request #97.

    I think this fix does not just avoid the errors, but also reduces compilation time by avoiding unnecessary find command calls.

  22. Log in to comment
Assignee
Type
bug
Priority
major
Status
resolved
Component
–
Milestone
–
Version
–
Votes
0
Watchers
2
Jira: the preferred issue tracker for Bitbucket. Join the team!