1. tuck
  2. pymite


Dean Hall  committed ce34afb

Fixes issue 234
Add support for platform-specific module loading

  • Participants
  • Parent commits d47b0e6
  • Branches v10

Comments (0)

Files changed (13)

File docs/src/PlatformMarshalledModules.rst

View file
  • Ignore whitespace
+Platform Marshalled Modules
+Adding Modules to P14p After Compile-Time
+:Author: Dean Hall
+P14p has always provided the ability to create Python modules and import them in
+your program, provided the modules are added to the P14p binary at compile-time.
+What's been missing is the ability to dynamically add and use modules after
+compile-time.  Platform Marshalled Modules (PMM) is a new feature that allows
+the ability to import a marshalled module in a platform-specific fashion.  This
+new feature provides the following benefits:
+#. A platform may store marshalled modules on nearly *any* form of information
+   storage media.  This includes SEEPROM, auxiliary Flash areas, SD Cards, file
+   systems or even network-accessible storage.
+#. Compile p14p once and put your main program in a PMM so it can be easily
+   modified.  For your platform, you could establish a convention such as the
+   platform's main.py contains simply ``import autoexec``.  Then you write your
+   application in autoexec.py, convert it to autoexec.pmm and store the
+   resulting bytes in your platform's storage area.
+#. Override built-in modules.  If you've compiled P14p and distributed it in an
+   embedded system and later find a bug in one of the built-in modules, you can
+   install a corrected version of that module in the platform storage area.
+   P14p will search the platform storage area first by default and use the
+   corrected module from there.
+The ``posix64`` platform has an example of PMM that uses the platform's file
+system as the storage area.  So the marshalled modules are stored as simple
+files with the ``.pmm`` file extension.  Using the ``.pmm`` file extension is
+the recommended convention for any platform that has a file system.
+The rest of this document explains the implementation of PMM and how to use it.
+Using PMM
+There are two pieces to using PMM in P14p.  First, the port of P14p to a
+specific platform *must* implement and register a platform-specific function
+that fetches the PMM data and turns it into a Module in RAM.  Second, the
+end-user programmer must use the converter tool, ``src/tools/pmPyToPmm.py``, to
+marshal his module into a PMM byte stream and store that bytestream in a manner
+that allows the platform to search for it by knowing only the module's name.
+For example, if the module ``telemetry.py`` is written, the program that uses
+the telemetry module has a line of code such as ``import telemetry`` and so
+P14p's VM will only have the string ``"telemetry"`` to use when looking up the
+module in the platform's storage area.  Now, let's dig into the two pieces to
+using PMM in detail.
+Implement the Platform Loading Function
+With the introduction of PMM, P14p's platform porting layer has one new API::
+    PmReturn_t plat_loadCodeObject(pPmObj_t pname, pPmObj_t *r_cob);
+:Aside: The function is called "plat_loadCodeObject" because inside the P14p VM,
+        all the information from a compiled Python module is held in a structure
+        called a Code Object.  A Code Object structure can be located in a
+        microcontroller's flash memory or RAM.  When the P14p VM needs to import
+        a module, the VM finds a code object of the matching name and "imports"
+        it using ``mod_new`` which effectively turns the code object into a
+        module.  Further aside: In Python a code object can hold the
+        representation of a module, a class or a function.  Likewise, in P14p,
+        the C structure PmFunc_t is used to hold the contents of a function,
+        class or module object.
+This function is OPTIONAL.  Porting P14p to a new platform does not require that
+this function exists.  However, if you want to add the PMM feature to your
+platform, you need to implement this function and register it during
+``pm_init()``.  Let's look at what ``plat_loadCodeObject()`` has to do.
+First, we can see from the function's arguments that it expects a name and
+returns by reference a cob, or Code Object.  The function also returns a
+PmReturn_t status.  This direct return value should be ``PM_RET_OK`` if the
+module was found and loaded into RAM, ``PM_RET_NO`` if the module was not found,
+or one of the ``PM_RET_EX_*`` values defined in ``pm.h``, which is how an
+exception is reported to the VM.
+Inside of ``plat_loadCodeObject()`` the following actions must be performed:
+#. Search the platform storage area to see if a PMM with the matching name is
+   found.  If a matching PMM is not found, return ``PM_RET_NO``.  In the posix64
+   platform example, I search the current working directory for a file with a
+   matching name (and .pmm extension).
+#. If the platform storage area is NOT in a directly addressable memory  (RAM or
+   Flash is directly addressable; SEEPROM or a file is not), the PMM's bytes
+   must be copied into RAM so they can be read by P14p's Marshal function.  In
+   the posix64 platform example, I allocate a chunk of memory and copy the PMM
+   file's contents into it.  Note that P14p memory chunks are limited to 2040
+   bytes (as defined by ``HEAP_MAX_LIVE_CHUNK_SIZE``) at the time of this
+   writing, so this limits the size of the ``.pmm`` file on the posix64
+   platform.
+#. Call ``marshal_load()`` and give it the address to the start of the PMM
+   bytes.  ``marshal_load()`` will return a code object by reference if the PMM
+   file described a module.
+#. Check for any errors and report them (as exceptions) if necessary.
+   Otherwise, return the code object by reference and return ``PM_RET_OK``.  The
+   VM will take care of everything else.
+Register the Platform Loading Function
+Great, you've made ``plat_loadCodeObject()``, now what?  Well, the VM doesn't
+know about your function yet.  You have to explicitly register your function.
+In the posix64 example, I perform registration in ``plat_init()``.  I cannot
+think of a reason to perform this registration elsewhere, so I highly recommend
+it be done there.  Here is the code to do it::
+    mod_setPlatLoadCodeObjectFunctionPointer(plat_loadCodeObject);
+As you can see, the registration function is part of ``mod.c`` because that's
+where ``mod_import()``, the function that calls your ``plat_loadCodeObject()``,
+resides.  The registration function is expecting the function pointer as its
+only argument.  As long as your load function matches the signature of
+``plat_loadCodeObject`` as defined in ``src/vm/plat_interface.h`` everything
+should work fine.
+Creating a PMM
+I can't call this section "Creating a PMM file" because not all platforms will
+store PMM data in a file.  However, PMM creation takes place on the desktop
+using a converter tool that takes a Python source file as input and produces a
+``.pmm`` file as output.  So PMM data can be thought of as a file and I use the
+two terms interchangably.
+The converter tool ``src/tools/pmPyToPmm.py`` is fairly short and easy to
+comprehend.  The tool runs on the command line and accepts one python source
+file as its one optional argument.  If a filename is given as an argument, a
+file with the same base name and ``.pmm`` extension is created as output.
+If no argument is given, the tool expects input via ``stdin`` and produces
+output on ``stdout``.
+The converter tool only does two things: compiles the given Python source to a
+code object and marshal's the code object to a byte string.  It is important to
+notice that the marshal tool being used is P14p's own ``pmMarshal``.
+``pmMarshal`` is conceptually comparable to Python's ``marshal``, but has syntax
+differences that make the two marshal formats incompatible.  This
+incompatibility was necessary because P14p's code objects are significantly
+different than CPython's.
+With the ``.pmm`` file created, you must copy the *exact* binary data from the
+``.pmm`` file into the platform's storage area.  How this is accomplished is
+specific to the platform and storage medium.
+IMPORTANT: There is nothing in the ``.pmm`` data that indicates the name of the
+module.  So the platform-porting programmer must create a way to find the
+``.pmm`` data (and its length in bytes!) by name.  I mention the length because
+the ``marshal_load()`` requires the length of the data bytes as an argument.  In
+the posix64 example, I use ``fseek()`` and ``ftell()`` to discover the size of
+the ``.pmm`` file.

File src/lib/string.py

View file
  • Ignore whitespace
     i = strtol(pc, &pend, base);
     /* Raise ValueError if there was a conversion error */
-    if (*pend != C_NULL)
+    if (*pend != '\\0')
         PM_RAISE(retval, PM_RET_EX_VAL);
         return retval;

File src/platform/posix64/SConscript

View file
  • Ignore whitespace
-HAVE_FLOAT = "little endien"
 env_dbg = Environment(variables = vars,
     CPPPATH = ["#src/vm", "#src/platform/posix64/build"],
     CCFLAGS = cflags + cflags_dbg,
-    LINKFLAGS = "-Wl,-map $mapfile")
+    LINKFLAGS = "-lm -Wl,-map $mapfile")
 env = env_dbg
 if not sys.platform.startswith("win"):
     env['ARFLAGS'] = "rcs"

File src/platform/posix64/plat.c

View file
  • Ignore whitespace
     ualarm(1000, 1000);
+    mod_setPlatLoadCodeObjectFunctionPointer(plat_loadCodeObject);
     return PM_RET_OK;
 #endif /* HAVE_DEBUG_INFO */
+ * The Posix64 platform looks in the current directory of the filesystem
+ * to find a module of the matching name (with .pmm extension).
+ * If one is found, it is assumed to contain a marshalled code object.
+ */
+PmReturn_t plat_loadCodeObject(pPmObj_t pname, pPmObj_t *r_cob)
+    char cwd[FILENAME_MAX];
+    char *pchar;
+    uint16_t len;
+    FILE *fp;
+    long fsize;
+    uint8_t *pchunk;
+    PmReturn_t retval;
+    uint8_t objid;
+    *r_cob = C_NULL;
+    /* Get the current directory or return failure */
+    if (getcwd(cwd, FILENAME_MAX) == C_NULL)
+    {
+        return PM_RET_NO;
+    }
+    /* Scan to end of path */
+    for (pchar = cwd; *pchar != '\0'; pchar++);
+    /* Return failure if there is not enough room to hold the filename */
+    len = ((pchar - cwd) + ((pPmString_t)pname)->length + sizeof(ext));
+    if (len >= FILENAME_MAX)
+    {
+        return PM_RET_NO;
+    }
+    /* Append directory seperator and given name to the path */
+    *pchar = '/'; pchar++;
+    sli_memcpy(pchar, ((pPmString_t)pname)->val, ((pPmString_t)pname)->length);
+    pchar += ((pPmString_t)pname)->length;
+    /* Append p14p's marshal file extension and terminate */
+    sli_memcpy(pchar, ext, sizeof(ext));
+    pchar += sizeof(ext);
+    *pchar = '\0';
+    /* Return failure if the file won't open */
+    fp = fopen(cwd, "rb");
+    if (fp == C_NULL)
+    {
+        return PM_RET_NO;
+    }
+    /* Get the size of the file */
+    fseek(fp, 0, SEEK_END);
+    fsize = ftell(fp);
+    fseek(fp, 0, SEEK_SET);
+    /* Return failure if file is too big or an exception if not enough memory */
+    if ((fsize + sizeof(PmObjDesc_t)) > 65535) return PM_RET_NO;
+    retval = heap_getChunk((uint16_t)(fsize + sizeof(PmObjDesc_t)), &pchunk);
+    PM_RETURN_IF_ERROR(retval);
+    heap_gcPushTempRoot((pPmObj_t)pchunk, &objid);
+    {
+        /*
+         * Read the file into the buffer
+         * (offset by sizeof(OD) so GC doesn't clobber the data)
+         */
+        if (fsize != fread(pchunk + sizeof(PmObjDesc_t), 1, fsize, fp))
+        {
+            fclose(fp);
+            return PM_RET_NO;
+        }
+        /* Un-marshal the file contents */
+        retval = marshal_load(pchunk + sizeof(PmObjDesc_t), fsize, r_cob);
+        PM_RETURN_IF_ERROR(retval);
+    }
+    heap_gcPopTempRoot(objid);
+    heap_freeChunk((pPmObj_t)pchunk);
+    /* Raise exception if it is not a Code Object */
+    if (OBJ_GET_TYPE(*r_cob) != OBJ_TYPE_COB)
+    {
+        PM_RAISE(retval, PM_RET_EX_TYPE);
+        return retval;
+    }
+    return PM_RET_OK;

File src/tests/system/SConscript

View file
  • Ignore whitespace
     env['ARFLAGS'] = "rcs"
 # Generate pmfeatures.h
-pmfeatures_py = File("#src/platform/posix/pmfeatures.py")
+pmfeatures_py = File("#src/platform/posix64/pmfeatures.py")
 pmfeatures_h = env.Command("pmfeatures.h", pmfeatures_py,
     "src/tools/pmGenPmFeatures.py $SOURCE > $TARGET")
 # Copy desktop plat source files
 plat_h = env.Command("#src/tests/system/build/plat.h",
-                     "#src/platform/posix/plat.h",
+                     "#src/platform/posix64/plat.h",
                      Copy("$TARGET", "$SOURCE"))
 plat_c = env.Command("#src/tests/system/build/plat.c",
-                     "#src/platform/posix/plat.c",
+                     "#src/platform/posix64/plat.c",
                      Copy("$TARGET", "$SOURCE"))
 # Generate code objs and types

File src/tools/pmCoCreator.py

View file
  • Ignore whitespace
     # HACK: append co_filename and co_name to the co_consts tuple to reduce
     # the size of the code object structure by two pointers.
     # co_filename and co_name are extracted via the co API in codeobj.c
-    k = list(fco['co_consts'])
-    k.extend([fco['co_filename'], fco['co_name']])
-    fco['co_consts'] = tuple(k)
+    fco['co_consts'] += (fco['co_filename'], fco['co_name'],)
     # Prep data to fill into the CO structure definition
     d = {}

File src/tools/pmPyToPmm.py

View file
  • Ignore whitespace
+#! /usr/bin/env python
+# This file is Copyright 2012 Dean Hall.
+# This file is part of the Python-on-a-Chip program.
+# Python-on-a-Chip is free software: you can redistribute it and/or modify
+# it under the terms of the GNU LESSER GENERAL PUBLIC LICENSE Version 2.1.
+# Python-on-a-Chip is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# A copy of the GNU LESSER GENERAL PUBLIC LICENSE Version 2.1
+# is seen in the file COPYING up one directory from this.
+# Compiles a python source file and converts it to a marshalled code object.
+# Also known as a P14p Marshalled Module (pmm)
+import sys, os.path
+import pmMarshal
+# The following must match epynomous var in src/vm/marshal.h
+def pyToPmm(fn):
+    if fn: fin = open(fn, 'r')
+    else: fin = sys.stdin
+    src = "".join(fin.readlines())
+    co = compile(src, str(fn), "exec",)
+    com = pmMarshal.dumps(co)
+    if fn: fout = open(os.path.splitext(sys.argv[1])[0] +
+                       PM_MARSHAL_FILE_EXTENSION,
+                       'wb')
+    else: fout = sys.stdout
+    fout.write(com)
+if __name__ == "__main__":
+    fn = None
+    if len(sys.argv) != 1:
+        fn = sys.argv[1]
+    pyToPmm(fn)

File src/vm/heap.c

View file
  • Ignore whitespace
             retval = heap_gcMarkObj((pPmObj_t)((pPmCob_t)pobj)->co_consts);
             retval = heap_gcMarkObj((pPmObj_t)((pPmCob_t)pobj)->co_cellvars);
-            PM_RETURN_IF_ERROR(retval);
         case OBJ_TYPE_MOD:

File src/vm/marshal.h

View file
  • Ignore whitespace
 PmReturn_t marshal_load(uint8_t *ps, uint16_t len, pPmObj_t *r_po);
 PmReturn_t marshal_dump(pPmObj_t po, pPmObj_t *r_ps);

File src/vm/module.c

View file
  • Ignore whitespace
 #include "pm.h"
+static Mod_platLoadCodeObject_t mod_platLoadCodeObjectFunctionPointer = C_NULL;
 mod_new(pPmObj_t pco, pPmObj_t *pmod)
         return retval;
-    /* Try to find the module in the table */
-    for (i = 0; i < pm_global_module_table_len_ptr->val; i++)
+    /* #234: Support platform-specific module loading */
+    if (mod_platLoadCodeObjectFunctionPointer != C_NULL)
-        if (string_compare(pm_global_module_table[i].pnm, (pPmString_t)pstr) == C_SAME)
+        retval = mod_platLoadCodeObjectFunctionPointer(pstr, (pPmObj_t *)&pco);
+        /* Return now if an exception occured */
+        if ((retval != PM_RET_NO) && (retval != PM_RET_OK))
-            pco = pm_global_module_table[i].pco;
-            break;
+            return retval;
-    /* If img was not found, raise ImportError */
+    /* If the platform-callback didn't run or didn't find the code object */
+    if (retval != PM_RET_OK)
+    {
+        /* Try to find the module in the table */
+        for (i = 0; i < pm_global_module_table_len_ptr->val; i++)
+        {
+            if (string_compare(pm_global_module_table[i].pnm, (pPmString_t)pstr) == C_SAME)
+            {
+                pco = pm_global_module_table[i].pco;
+                break;
+            }
+        }
+    }
+    /* If code obj was not found, raise ImportError */
     if (pco == C_NULL)
         PM_RAISE(retval, PM_RET_EX_IMPRT);
     return retval;
+mod_setPlatLoadCodeObjectFunctionPointer(Mod_platLoadCodeObject_t pfunc)
+    mod_platLoadCodeObjectFunctionPointer = pfunc;
+    return PM_RET_OK;

File src/vm/module.h

View file
  • Ignore whitespace
+/** #234: Function pointer for platform-specific loading callback */
+typedef PmReturn_t(*Mod_platLoadCodeObject_t)(pPmObj_t, pPmObj_t*);
  * Creates a Module Obj for the given Code Obj.
 PmReturn_t mod_import(pPmObj_t pstr, pPmObj_t *pmod);
+ * Stores the given function pointer to be used within mod_import()
+ * so that code objects can be searched and loaded in a platform-specific manner.
+ *
+ * @param   pfunc Pointer to callback function that will load a code object
+ * @return  Return status (always PM_RET_OK)
+ */
+PmReturn_t mod_setPlatLoadCodeObjectFunctionPointer(Mod_platLoadCodeObject_t pfunc);
 #endif /* __MODULE_H__ */

File src/vm/plat_interface.h

View file
  • Ignore whitespace
 #define __PLAT_H__
  * \file
- * \brief PyMite's Porting Interface 
+ * \brief PyMite's Porting Interface
 void plat_reportError(PmReturn_t result);
+ * OPTIONAL: It is NOT necessary to implement this function on a platform.
+ *           Implementing this function will allow the ability to load code
+ *           from data storage areas including Flash, SEEPROM or a filesystem.
+ * USAGE:    In plat_init(), register this function by calling::
+ *           mod_setPlatLoadCodeObjectCallback(plat_loadCodeObject);
+ * Finds a code object matching the given name.
+ * The code object must be in addressable memory (RAM or Flash);
+ * if it is not, the code object must be copied to RAM.
+ * Used by mod_import() to import modules using a platform-specific method.
+ * For more information on this function, see
+ * docs/src/PlatformMarshalledModules.rst
+ */
+PmReturn_t plat_loadCodeObject(pPmObj_t pname, pPmObj_t *r_cob);
 #endif /* __PLAT_H__ */

File src/vm/pm.h

View file
  • Ignore whitespace
 /** null for C code */
-#define C_NULL 0
+#define C_NULL ((void *)0)
 /** false for C code */
 #define C_FALSE (uint8_t)0