how do I use include_parts for including all the *.c file to env.Program

[Tamar Levy]

is that possible to do.

Thank, Tamar

[Jason Kenny]

I am a little unclear on what you are asking.

In general a normal part file doing a glob would look like: Import('') ... #get files src_files=Pattern(src_dir = 'src', includes = ['.cpp','*.c']).files() ret=env.Program("hello",src_files) env.InstallTarget(ret)

You can also use the SCons Glob() functions as well

If you have something else in mind please give me more detail so i can provide a better answer.

[Tamar Levy]

Hi Jason,

thanks for answering my question. I think I probably confused a little bit, I ment that lets say the function Part() is including all the source file to be compiled. so I was thinking maybe there is a better way for env.Program to grab all the source files. and anyway I do not understand what is the role of function Part(), if you kindly show me an example that will be great.

Thanks, Tamar

[Jason Kenny]

Hi Tamar,

I have documentation you can find here... http://parts.tigris.org/doc/PartsUserGuide.pdf

The quick answer to your questions is that Part(...) defines some component of the build defines in a part file. SCons by default is defines to do a recursive make style. Parts lets one define a more component/project based setup in which you can easily share different components in a plugin play manner. If you have cloned Part locally you should see a number of small simple examples under the samples directory. One of the main values of Parts is to all the user to define a component to build and to easy share state between different components. So for example in classic makefile or SConstructs you have ma have some tree of items to make the main "product". If for example a base leaf component was updated and outputted a new .so object to link with, this would normally require all dependent components to update there makefiles or sconscripts files to link with the new name. In Parts, dependent components would just get the new value pass to it, requiring no modification to the other build files. There is more of course, but the main idea is that a "Part" defines an object that contain meta data needed for other Parts to build against it. This allow sharing of data better, allowing it to scale better to changes, use on different platforms, etc than a classic make or even Sconstruct file would be able to do without lots of effort. Adding or removing a component is easy as adding or removing the Part() call. Of course if a component DependsOn() another component the build sill stop with an error that it did not fine a component matching the requirements defined in the DependsOn() call

Ideally there has been the idea to allow a Part load a different type of file ( such as a make or msbuild file and go from there) However i have not done that yet as that requires a lot of work, and I still have some items to clean up in the core.

Feel free to ask me any other questions you may have

[Tamar Levy]

Thanks Jason for your respond. I saw that when Im calling Part Im basically providing other part files that define or declare all the .c and .h files for compilation. in env.Program() we suppose to pass the name of the compiled object and also all the .h and .c files, is there a way to do that if they already being declared when calling the Part function? if my question are not relevant please let me know Im a new user of scons and just lately Im strating to understand how it is being used. anyway I have another question regrading scons that I posted in stack overflow: http://stackoverflow.com/questions/41211569/is-there-any-way-to-add-arm-compiler-to-scons-toolchain

Thanks, Tamar

[Jason Kenny]

Yes. You provide a Parts file that is like a sconscript file in SCons. The difference is that Part() will set up a number of items and states and add various other functions to help make defining what to build easier. A common practice is to define a file like this: https://bitbucket.org/sconsparts/parts/src/d097472d8f49ecf5603d0b05553d00024e245db0/samples/subpart1/proxy/driver/driver.parts?at=master&fileviewer=file-view-default

This file just uses the PartName() and version number values as the output for the so/dll in this example.

I don't have the ability to define a set of files to build directly as you suggest. However if you understand what Parts does, you could define a template part file that would take some variables and do the right thing based on the values. You would need to define the part to take a value if it should be a program or shared object, static lib, etc output. So what you suggest is possible. if you define the right template.

Just a note on that .. SCons scans the files for headers, so you might what to add extra paths to CPPPATH to be scanned. By default Parts that depend on other Parts have this value passed along auto-magically so you don't need to do it. However if you have a third-party library you need to link with, you will want to add those paths possibly.

So the question on stack overflow. I have an enhanced toolchain setup in Parts. I talk about it in the documentation I had provided you. This would allow you to define a toolchain and configuration for the tools you have. I add a notion of a HOST and TARGET platform and a version number for tool selection. For gcc you probably only need to add an extra entry to the parts.tools.GnuCommon.common.gcc ( and g++) objects for the compiler you want to refer to. this can be done via adding an extension in the .part-site/tools directory. something like:

gcc.Register( # we assume that the system has the correct libraies installed to do a cross build # or that the user add the extra check for the stuff the need hosts=[SystemPlatform('posix','x86_64')], targets=[SystemPlatform('posix','arm')], info=[ GnuInfo( #standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[ PathFinder(['/usr/bin']) ], opt_dirs=[ '/opt/' ], script=None, subst_vars={}, shell_vars={'PATH':'${GCC.INSTALL_ROOT}'}, test_file='gcc' ) ] )

I know I have a few items I need to fix here, but these are more on making it easier to do or supporting stuff like rhel dev toolchains. The other big item would be adding a file to your parts-site/toolchain directory to define a name of the toolchain you want to have. Again this area is not 100% what I wanted yet..but is good enough to handle everything thrown at it so far.

given that you could define a command like:

scons --tc=gcc --target=arm

I have example in which you can use the android NDK ( however i probably need to up date the code to support newer NDK drops) in which you would say "scons --tc gcc --target android-x86" or "scons --tc gcc --target android-arm"

Some other basic notes...

tools are found based on the what valid targets are define in the toolsetting object for the tool and host platform. The target platform by default is the value of the host platform, unless it is give a different value. Currently it is in the form of OS-ARCH. If you only provide one of the values, such as OS the ARCH value will be the value of the host, so the example --target= android would map to android-x86_64 on my current system.

I did not say anything about configuration yet, and unless you have the other stuff working I would not worry about it as the default gcc one should be ok, till you what to extend or replace it with you own values for a cross case that matters to you.

If you don't want to use Parts to help here, you can do this yourself ( But this is an open project, so I always like new users :-) ), I see you have your own sbuild setup you working on. Just keep in mind you will want to mange host and target values, different environments per host-target-configuration setup as well a number of other items as well. Depending on how you look at it, In Parts I take a more strict view that the tools being used need to found and verified to be what you want, vs just something i found on the path ( I don't know where that path has been..) to help with duplicating exact environments for a build. On Linux SCons tend to be more "liberal" as will be happy if the gcc you are using is sourced on the path, The tool I provide overrides this a bit to allow selecting a version, and preferring a more exact match when we can get it. So if this was just raw SCons, you could just set your arm gcc first on the path and Scons would use it.

Parts add a --verbose and --trace argument. you can add --verbose=type1,type2 value as well to help filter what is dumped. By default the value is all, which is everything. You might find this useful to understand what is happening better in Parts.

Hopefully this is helpful. let me know what else you need.

[Tamar Levy]

Hi Jason,

I tried adding hosts=[SystemPlatform('posix','x86_64')], targets=[SystemPlatform('posix','arm')], info=[ GnuInfo( #standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[ PathFinder(['/usr/bin']) ], opt_dirs=[ '/opt/' ], script=None, subst_vars={}, shell_vars={'PATH':'${GCC.INSTALL_ROOT}'}, test_file='arm-linux-gnueabi-gcc' ) ] )

I still get an error when I run: scons --tc=gcc --target=arm "scons: *** No version of GCC was found on the system for target posix-arm"

if I only specify scons --target=arm I get "scons: *** No version of GXX was found on the system for target posix-arm"

I tried adding the file gcc_posix-any_posix-arm.py and g++_posix-any_posix-arm is the sample helloworld it still seems that it doesnt recognize it (I get the same error) is there any example in the samples that solves this issue that I have? I want to see some example with a tool that is not defined in scons ToolChain as arm-linux-gnueabi-gcc.

Thanks, Tamar

[Tamar Levy]

Hi Jason,

where should I defined the name of the linker? is it in gcc.py or should it be in SConstruct?

Thanks, Tamar

[Jason Kenny]

HI,

sorry dealing with kids out of school today..

The linker for gcc/g++ is defined by the gnulink.py mixed with stuff in link.py and smartlink.py. However for Parts we have some stuff in binutil.py which i think you can ignore ( it was added when there was some hiccup in binutils which required some people to build with a different binutils than what was bound to the compiler. I think by default the linker is based on the gcc or g++ so it should just work. Unless you need to call "ld" directly I would just have it use the default gcc/g++ as link pass through. SCons has some code that notices you compiler C++ code in the set of object being linked, so it should call your g++ in those cases else gcc.

I have a very simple example of a tools in samples/custombuilder which makes a fake "optool" It does some extra stuff which I would avoid if you can in that it adds a InstallOpt and SdkOpt API function the set of InstallXXX and SdkXXX functions I have. This might be helpful for your example.

The message of: "scons: *** No version of GXX was found on the system for target posix-arm" means Parts could not find a running gcc or g++ tool.

This could be a number of reason for this. I need to refactor this code as I find it more complex than I feel it should be to add a tool as I need to clean up the toolsetting and ToolInfo code, refactor code that that did not work out as I thought it should, ie it needs to be easier to use and more informative ( wiser now... This was one of my project this break, but life might hold me back)

As far as your case goes I think, without looking at it, I have a bug in the logic in gnutools with how it reads /opt/ that pattern was based on information used when I was at Intel. I have learned since then the person giving me this info did know what he was doing, so the pattern is wrong. I think you can have better luck if you change the part with :

install_scanner=[ PathFinder(['/usr/bin']) ],

to have the path point to the "/opt/gcc<the rest of the path>/bin" path ( I forgot the standard path here.. I would need to look it up or build a compiler again) I think that should help you in this case or get you closer. The GnuInfo was based on that most posix/gnu based tools have a common pattern in how it install tools. The /opt pattern like i said is not correct it looks for something like /opt/gcc-4.1 or something which is missing data about the target. Given this code is fixes, this should run better out of the box. ( pull requests welcome :-) ). I also need o address issues in this code to support RHEL dev toolschain as they have a very different look up pattern.

if you have the path for you compiler under /opt I probably can throw you an example that should work with 24-48 hours. Give me reason to write new sample and fix up my tools code to find better patterns

Hope this help!

Jason

[Tamar Levy]

Hi Jason,

Thanks for spending time responding to me, this is very much appreciated. the thing is that my compiler is not under "/opt/" it is under "/usr/bin", it seems like is does not find gcc, but frankly I do not understand what scons cannot find, when printing --verbose I didnt see any search of scons for the compiler "'arm-linux-gnueabi-gcc", how can I know that the printed error "scons: *** No version of GCC was found on the system for target posix-arm" really refers to scons cannot find "'arm-linux-gnueabi-gcc"? it looks like there is missing part here with:

gcc.Register( # we assume that the system has the correct libraies installed to do a cross build # or that the user add the extra check for the stuff the need hosts=[SystemPlatform('posix','x86_64')], targets=[SystemPlatform('posix','arm')], info=[ GnuInfo( #standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[ PathFinder(['/usr/bin']) ], opt_dirs=[ '/opt/' ], script=None, subst_vars={}, shell_vars={'PATH':'${GCC.INSTALL_ROOT}'}, test_file='arm-linux-gnueabi-gcc' ) ] )

it probably not just defining this in gcc.py because it feels like it doesn't even look for arm-linux-gnueabi-gcc when Im running: scons --tc=gcc --target=arm

[Tamar Levy]

Hi Jason,

you asked me to try to add this code

gcc.Register( # we assume that the system has the correct libraies installed to do a cross build # or that the user add the extra check for the stuff the need hosts=[SystemPlatform('posix','x86_64')], targets=[SystemPlatform('posix','arm')], info=[ GnuInfo( #standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[ PathFinder(['/usr/bin']) ], opt_dirs=[ '/opt/' ], script=None, subst_vars={}, shell_vars={'PATH':'${GCC.INSTALL_ROOT}'}, test_file='arm-linux-gnueabi-gcc' ) ] )

and I added it to the gcc.py under part-site/configurations/debug

and scons probably seems to ignore it so I out it in the installation part file here: /usr/local/lib/python2.7/dist-packages/parts/tools/GnuCommon/gcc.py

and now I get this error: scons: Reading SConscript files ... scons: done reading SConscript files. scons: *** No targets specified and no Default() targets found. Stop. Found nothing to build

which looks like a little bit progress even though I do not know how to solve it

so I believe that code should be under GnuCommon in the installation folder but how can I overwrite it in my project instead of changing it in the installation folder?

Thanks, Tamar

[Jason Kenny]

I will try to send you a sample.. of course it might be useful to add this in Parts proper. In parts I disable the default target of '.'

Just add "all"

ie scons all --tc...

[Jason Kenny]

Add a sample to how to add a custom gcc. This sample uses Ubuntu arm cross compiler as an example as at this moment it is not in the default setup for Parts.

[Jason Kenny]

Hi

I just had a few minutes to make a sample for you. It is attached to the issue.

Hopefully this solve the problem you are having. Just down load it and run "scons --target=arm all" For parts --tc default to "default" toolchain which will select gcc, given the icc compiler is not installed.

Hope this help solve your issue. I should take some time to add arm support for gcc, on Ubuntu while I am at it to the core.

[Tamar Levy]

Hi Jason,

Thanks for the samples. now I understand what I was missing it works greately for arm but Im also trying to enable it to arm64 with using different compiler: parts.tools.GnuCommon.common.gcc.Register( # we assume that the system has the correct libraies installed to do # a cross build or that the user add the extra check for the stuff the need hosts=[SystemPlatform('posix', 'x86_64')], targets=[SystemPlatform('posix', 'amd64')], info=[ parts.tools.GnuCommon.common.GnuInfo( # standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[PathFinder(['/usr/bin'])], opt_dirs=['/opt/'], script=None, subst_vars={}, shell_vars={'PATH': '${GCC.INSTALL_ROOT}'}, test_file='aarch64-linux-gnu-gcc') ] )

but I get an error: SCons Error: Error: arm64 is not a valid --target_platform value

is it because arm64 is not defined in scons as an architecture? so in that case how do you compile for 64bit on a 64bit arm architecture?

Thanks, Tamar

[Jason Kenny]

Hi,

Parts has a mapping for AMD64 ( and Intel64) to x86_64 as they are the same thing. Why SystemPlatform had an error, would be some bug I need to fix. It should have worked. I will see about fixing that issue in Parts. The work around is to use x86_64 and skip the marketing names AMD and Intel uses.

[Tamar Levy]

Hi Jason,

thanks for the quick answer. Im so sorry but the error above is when I used arm64 I also tried amd64 and it never worked, could it be that arm64 is not defined as one of the scons default architectures support?

[Jason Kenny]

Hi,

you had hosts=[SystemPlatform('posix', 'x86_64')], targets=[SystemPlatform('posix', 'amd64')],

but yes I don't have a mapping for arm64. I was not sure how to deal with arm as unlike x86 setups, arm to arm chips are not guaranteed to work with arm chip different manufacturers.

However It is easy for me to add arm64 to the setup. You can as well in the parts-site. Just add a file to .part-site/pieces ( and name will do in here as long as it does not replace existing "pieces" in parts)

something like:

import parts.api.platforms

parts.api.platforms.AddArchitecture('arm64')

Should solve your issue. Note! that you can call this again to add other "valid" mappings to arm64 you might want to allow.. ie

parts.api.platforms.AddArchitecture("arm_64','arm64')  # would map an arm_64 value arm64

[Tamar Levy]

Hi Jason,

Thanks a lot
it works like magic Im starting to like scons :)

[Tamar Levy]

Hi Jason,

One more question if I also want to support another compiler: arm-linux-gnueabihf-gcc for arm do I need to create another folder ToolChain which will differentiate between arm-linux-gnueabihf-gcc and arm-linux-gnueabi-gcc how are we dealing with that situation using scons?

Thanks, Tamar

[Tamar Levy]

Hi Jason,

I think maybe the best way is to add another two architecture: 1. arm-linux-gnueabi 2. arm-linux-gnueabihf

and then for each one to attach the right compiler right?

Thanks, Tamar

[Jason Kenny]

Hi,

Glad to here Parts and SCons is working out well for you.

As far as you last question. I have not added arm values to the platform as I stated before because I have been unclear on what the right values to add are. I would suggest that you add new architecture to deal with the case of the arm compilers. Much like you added arm64, you might need to add some sort of arm_hf (again not sure what the right values are arm should be here) but this would allow you define a case for the different arm cases you are dealing with. If you have a good understanding of what arm architectures are there, I would be happy to add them to the core Parts code. Likewise you can add them yourself with the parts.api.platforms.AddArchitecture API with the correct Toolsetting additions for gcc. This is basically you last post. I would also suggest that you keep the OS are out of the platform.. having something like arm_gnueabi would be better as i try to use - as the main separator of different groups.

I have some plans to add to system platforms to allow more details about the platform in it name, such as more detailed value such as CPU type ( vs general architecture like x86) that would be optional to define. Have not done this yet as it has to be correct as once it is use it not easy to remove. "hacks" in which people can extend the existing values for to be used for targets are safe as they are in the users control.

Hope that help!

[Jason Kenny]

Based on my understanding comments this question is resolved. Added a sample to master to help others in the future

[Marc Valle]

Thanks for the sample Jason!

[Tamar Levy]

Hi Jason,

Im runnihg into problems compiling c++ code using parts with scons all --target=arm. my problem is that Im using /usr/bin/objcopy instaed of usr/arm-linux-gnueabi/bin/objcopy, do you know exactly where can I set objcopy? do I need to set it up somewhere here? if yes how can I do it? parts.tools.GnuCommon.common.gcc.Register( # compilation for Linux armel architecture system can be done from any Linux x86_64 system hosts=[SystemPlatform('posix', 'x86_64')], targets=[SystemPlatform('posix', 'arm')], info=[ parts.tools.GnuCommon.common.GnuInfo( # default binary location for arm-linux-gnueabi-gcc compiler install_scanner=[PathFinder(['/usr/bin'])], opt_dirs=['/opt/'], script=None, subst_vars={}, shell_vars={'PATH': '${GCC.INSTALL_ROOT}'}, test_file='arm-linux-gnueabi-gcc') ] )

Thanks, Tamar

[Jason Kenny]

Hi,

I think the issue is that for various reason we made a binutil tool to allow for some odd cases in which one needed to use different binutils that the one with the compiler ( or to make sure that version was used). It was an odd advance usage case that I would like to remove, but it keep coming up as a needed item for cross builds as the binutils are not always bound to the compiler as expected. I think all you need to do is to define some cases for arm binutils in your toolchain.

I think you look in parts/tools/gnucommon/binutils.py you will be at a good starting point of what you want to extend in your parts-site directory. Hopefully this is a good start to your problem

[Tamar Levy]

Hi Jason,

Thanks for your help, I did what you have said and created binutil.py file with this code init:

from common import binutils, GnuInfo from parts.tools.Common.Finders import PathFinder,ScriptFinder from parts.platform_info import SystemPlatform from parts.tools.Common.ToolInfo import ToolInfo import parts.tools.Common import SCons.Util

from SCons.Debug import logInstanceCreation

parts.tools.GnuCommon.common.binutils.Register( # we assume that the system has the correct libraies installed to do a cross build # or that the user add the extra check for the stuff the need hosts=[SystemPlatform('posix','x86'),SystemPlatform('posix','x86_64')], targets=[SystemPlatform('posix','arm')], info=[ BinutilInfo( #standard location, however there might be # some posix offshoot that might tweak this directory # so we allow this to be set install_scanner=[ PathFinder(['/usr/bin']) ], opt_dirs=[], script=None, subst_vars={ 'OBJCOPY':'${BINUTILS.INSTALL_ROOT}/arm-linux-gnueabi-objcopy', 'AR':'${BINUTILS.INSTALL_ROOT}/arm-linux-gnueabi-ar', }, shell_vars={'BINUTILS_INSTALL_ROOT':'${BINUTILS.INSTALL_ROOT}'}, test_file='arm-linux-gnueabi-ld', opt_pattern=binutils_pattern ) ] )

unfortunately when I try to build it doesnt pick the objcopy that I referenced because I get the same error: "objcopy: Unable to recognize the format of the input file"

I think I have something wrong in my implementation, d you have any idea from looking at this code what is wrong?

Thanks, Tamar

[Jason Kenny]

was this based on a standard Ubuntu package?

[Tamar Levy]

I dont understand your question. I just copied it from the file that you directed my to and added support from arm compiled on regular linux (x86 or x86_64)

Thanks, Tamar

[Jason Kenny]

Sorry, mean the compiler package. I am not sure what to say at the moment so I was looking for a way to possible to get a better understanding of what happened. I thought this was a standard compiler package I did not have direct support for in Parts. If it is, i can try to take a peak and see what is going on.

[Tamar Levy]

Hi Jason,

I can only tell you that when I try to compile in debug mode for arm on linux Im failing because its uses the wrong objcopy instead of using /usr/bin/arm-linux-gnueabi-objcopy it uses /usr/bin/objcopy and this is what Im trying to solve. I think that problem come into surface when you try to compile c++ code.

Thanks, Tamar

[Tamar Levy]

Hi Jason,

I think I know where is the problem, I download the whole parts source code and I saw where Im failing basically so this is my error Creating PDB for test objcopy: Unable to recognise the format of the input file

and when I try to see where is that print of "Creating PDB for" is coming from I saw its coming from parts/tools/gnulink.py from a function called _setUpPdbActions. it fails in the command : $OBJCOPY --only-keep-debug $PDBFLAGS ${TARGET} ${TARGET.attributes.pdb}

I tried to see what is that $OBJCOPY is referring to and I saw its using /usr/bin/objcopy which is problematic because this is an arm build that I do on x86 machine. I dont know if there is way to set $OBJCOPY to /usr/bin/arm-linux-gnueabi-objcopy because if there is its going to solve my problem I really hope I gave you enough information about the problem that Im having, I tried to make you a sample but it doesnt uses gnulink.py probably and that why I cannot reproduce the problem.

Thanks. Tamar

[Jason Kenny]

I looked in to this a bit. The OBJCOPY should be set on line 185 of tools/gnulink.py However the code gets a little odd, and I think we have a bug. @karbazol Had originally made this code. I think a bug is in the logic at here The code has some logic to deal with stuff for the Intel Phi cards. Ideally this should be setting the $OBJCOPY to the correct value based on the value of your toolinfo object you had defined. I will try to dig into this more once I get a few free cycles.

[Eugene Leskinen]

Hi guys, I think Tamar should do something like this

# python
binutils.Register(
    # we assume that the system has the correct libraries installed to do a cross build
    # or that the user add the extra check for the stuff the need
    hosts=[SystemPlatform('posix','x86'),SystemPlatform('posix','x86_64')],
    targets=[SystemPlatform('posix','arm')],
    info=[
    BinutilInfo(
        #standard location, however there might be
        # some posix offshoot that might tweak this directory
        # so we allow this to be set
        install_scanner=[
            PathFinder(['/usr/arm-linux-gnueabi/bin/'])
            ],
        opt_dirs=[
                '/opt/'
            ],
        script=None,
        subst_vars={
            'OBJCOPY':'${BINUTILS.INSTALL_ROOT}/objcopy',
            'AR':'${BINUTILS.INSTALL_ROOT}/ar',
        },
        shell_vars={'BINUTILS_INSTALL_ROOT':'${BINUTILS.INSTALL_ROOT}'},
        test_file='ld',
        opt_pattern=binutils_pattern
        )
    ]
)

Please take a note at PathFinder argument

[Tamar Levy]

Hi Eugene/Jason,

Where do I need to put this code? I tried to put it under part-site/tools/g++.py, is that a correct place to put it? because I still get the same error. anyway I succeeded to create a sample for you that replicate the error. I attached the sample to this email. this is the command that I run: scons build:: utest:: --target=arm --cfg=debug.

Thanks, Tamar

[Tamar Levy]

Hi Eugene/Jason,

Where do I need to put this code? I tried to put it under part-site/tools/g++.py, is that a correct place to put it? because I still get the same error. anyway I succeeded to create a sample for you that replicate the error. I attached the sample to this email. this is the command that I run: scons build:: utest:: --target=arm --cfg=debug.

Thanks, Tamar

[Tamar Levy]

Hi Jason,

Did you have a chance to take a look at the sample I sent you?

Thanks, Tamar

[Jason Kenny]

Hi Tamarlev,

Sorry for the delay. This problem seemed to be complicated by some module confusion in the Parts code. I made rename and updated the sample in master to fix the issue ( there is a PR #24 ). Hopefully this should address the problem for you.