speed issue: global variables

[Aaron Bartell]

My overall chroot process is now around 15 minutes (I am the only one on the machine) so I am looking for processing-time culprits. One culprit is the way I added global variables and the time it takes to find/replace them for each line.

As a reminder, here is what I mean by "global variables" (i.e. myuser, var1, var2):

$ ./chroot_setup.sh chroot_minimal.lst /QOpenSys/home/aaron myuser=aaron2 var1=val1 var2=val2

Instead of doing a separate sed for each xxxx.lst line I propose we create a timestamped temporary file that iterates through all global variables and composes a single sed to be run once. Below is a diff of my proposed change. This takes 30 seconds of processing down to less than 1 second.

##Thoughts on this performance fix? Are there better ways to code this? (I am still a shell scripting newbie)

diff --git a/chroot_setup.sh b/chroot_setup.sh
index 606f5f0..8f235fe 100755
--- a/chroot_setup.sh
+++ b/chroot_setup.sh
@@ -185,20 +185,6 @@ function chroot_setup {
         action=$name
       ;;
       *)
-        # replace command args with values
-        key=""
-        val=""
-        for i in $(echo $CHROOT_VARG | tr "=" "\n")
-        do
-          if ([ $key ]); then
-            val=$i
-            name=$(echo $name | sed s"|$key|$val|g")
-            key=""
-            val=""
-          else
-            key=$i
-          fi
-        done
         case "$action" in
           ":file")
              chroot_setup $name
@@ -281,9 +267,27 @@ if (($qopen==0)); then
 fi
 # run operation
 # $PS1='ranger$ '
+key=""
+val=""
+sed_cmd=""
+for i in $(echo $CHROOT_VARG | tr "=" "\n")
+do
+  if ([ $key ]); then
+    val=$i
+    sed_cmd="$sed_cmd -e 's|$key|$val|g'"
+    key=""
+    val=""
+  else
+    key=$i
+  fi
+done
+CHROOT_LIST_TMP=$CHROOT_LIST.$(date "+%Y%m%d.%H%M%S").tmp
+eval "sed $sed_cmd $CHROOT_LIST > $CHROOT_LIST_TMP"
 case "$opr" in
   -g)
-    chroot_setup $CHROOT_LIST
+    chroot_setup $CHROOT_LIST_TMP
+    rm $CHROOT_LIST_TMP
     echo "============"
     echo "chroot command:"
     echo "  > chroot $CHROOT_DIR /bin/bsh"

[Former user]

Well, around 15 mins ... not clear what expectations are suppose to target.

So, yes, any avoided fork/exec, which occurs via $(utility | utility) will speed things up.

-- but --

Shoot from hip is not a good way to make performance improvements, aka, you may work really hard for 30 seconds on 15 minutes. I suggest you instrument the scripts with a new trace variable TRACE_MY_FUNCTION=1,0, then use some sort of date/time calculation to see where big old 15 minute 'long running' functions actually exists. Therein, if you find 90% of time is IFS (tar, cp, mv, etc), well, mmm, nice way to say, mmm, well, gee, you are out of luck, and, not waste your time polishing the scripts for 30 seconds.

[Aaron Bartell]

Well, around 15 mins ... not clear what expectations are suppose to target.

I'd love to get it down to under 2 minutes. I might be smokin' rope though.

I miscalculated with my initial post. I actually went from 15 minutes down to a little over 7 (wowza!)

The nls stuff is another big culprit (630MB in size and probably 30 seconds). Working on solutions there also.

I suggest you instrument the scripts with a new trace variable TRACE_MY_FUNCTION=1,0, then use some sort of date/time calculation to see where big old 15 minute 'long running' functions actually exists.

I added time in front of the various chroot_xxxxx functions and that gave me my insight. The TRACE_MY_FUNCTION is a good idea and I will add it as a separate issue so I can remember to implement it later.

##With all that said, are you fine with the aforementioned diff of chroot_setup.sh?

[Former user]

No i do not like this change at all. Creating another file is not a good idea. Keep everything 'global in memory if you want, but do not start creating other files.

[Former user]

Mmm ... i am thinking ... essentially you are turning chroot_setup.sh into a mini compiler, that is, generating another "run-able", based input templates. In general, i am not for building another compiler, they inevitably run foul of 'another extension' to the language (aka, python again), therefore, my comment above ... i do not like this change at all.

However ...

You are likely the ultimate test case for this chroot effort, therefore, we probably should simply go with your temp create idea ... so ... go ahead ... commit the change to GIT (push it up).

BTW -- i am feeling (allowed feelings), not all your fancy provisioning stuff is making into this project. Bluntly, will you be adding your 'profile' previsioning customization techniques/scripts ... or ... do you consider these efforts proprietary (essential feed your business bulldog ... bark, bark)???

[Aaron Bartell]

BTW -- i am feeling (allowed feelings), not all your fancy provisioning stuff is making into this project. Bluntly, will you be adding your 'profile' previsioning customization techniques/scripts ... or ... do you consider these efforts proprietary (essential feed your business bulldog ... bark, bark)???

I am using the exact same chroot_setup.sh and pkg_setup.sh as ibmichroot. I obviously have my own xxxxx.lst files. I have a lot of Ruby code involved to bring everything together (i.e. DB2 interaction) because ibmichroot doesn't address everything (i.e. conditional logic and DB2 interaction). I am not convinced (yet) that we should have conditional logic in xxxx.lst files and instead that should be a call out to a program (.sh, .rb, .py, etc).

To address your profile provisioning question, here is an example:

:system
CRTUSRPRF USRPRF(myuser) PASSWORD(mypassword) HOMEDIR('mydir/./home/myuser')

./chroot_setup.sh xxxxx.lst /QOpenSys/dir1 myuser=usr1 mypassword=safetyfirst mydir=/QOpenSys/dir1

In the above call to chroot_setup.sh I generate the user profile/password and everything else dynamically in a Ruby script. Is that the fancy stuff you are wanting me to include?

[Former user]

Mmm ... no, not 100% sure my own thinking ... but ... i don't think we want to add or become 'set-up dependent' on ruby scripts (or python, php, nodejs). I think the best idea is to remain sortr of 'shell only', or, least common denominator that is on all machines (aka, /bin/bsh from PASE).

my feelings ...

Ok, yes,

Aaron usage ... i see CRTUSRPRF, implication being that in your admin 'usage/pattern', set-up a 'chroot environment' that includes setting up a user profile that goes with 'each' chroot.

Tony usage ... My intent chroot_setup.sh was that user profile already existed, and, admin was setting up a 'home' for development activities (or stand-alone jail web server, etc.).

So, at the moment we have ability to do either without duplicating work, aka, use each other stuff without forcing any single security model. Mmm ... current chroot_setup.sh may be best of both worlds ... therefore keep going this path, also your mini-compile /tmp/file_date.lst thing is fine.

However, i am wondering if you have a pattern (aaron_profile_and_all.lst), that should be included in the project (not so far to be ruby, php, python). Do you have thought or a new lst?

[Aaron Bartell]

i don't think we want to add or become 'set-up dependent' on ruby scripts (or python, php, nodejs).

I agree. I like the pure nature of ibmichroot (shell scripts are my new friend - love them).

Mmm ... current chroot_setup.sh may be best of both worlds

Agreed. We've made it flexible enough to serve multiple scenarios/purposes. Let's keep on this path.

therefore keep going this path, also your mini-compile /tmp/file_date.lst thing is fine.

Ha ha. You keep warming up to the idea more all the time ;-) I initially started going the route of doing it entirely in memory (i.e. sed into variable) but then I needed the "compiled" .lst file for debugging purposes, so then I determined the temp file was a good thing - especially since it gets deleted at the end.

However, i am wondering if you have a pattern (aaron_profile_and_all.lst), that should be included in the project (not so far to be ruby, php, python). Do you have thought or a new lst?

That would take time to put together and test (at least 2 days) - not something I have right now. Would probably be quicker if I took one of your how_to_<lang>.txt files and made it into a xxxx.lst file. This would need to wait until after the new year because I have too much stuff going on right now (opensource.common.org event and four articles due before end of year).

[Aaron Bartell]

Resolved with this commit.