1. zjes
  2. rope_py3k

Commits

Ali Gholami Rudi  committed 135ead2

Handling strings in following lines in patched ast

  • Participants
  • Parent commits 7c2d7a0
  • Branches trunk

Comments (0)

Files changed (4)

File docs/dev/issues.txt

View file
  • Ignore whitespace
 Cache Invalidation And Parameter Cache
 ======================================
 
-`PyFunction` holds a cache of is parameters.  The reason this cache is
-not moved to the `ParameterName` is that in SOI we need to set the
-parameters and then we start analyzing the return value.
-
-The main problem with this approach is that the cache references other
-modules and python elements.  I think this happens because SOI might
-analyze calls in the reverse order of imports.  But fixing it needs
-lots of work and maybe changing rope considerably.
+When retrieving pyobjects from objectdb no dependencies between
+pymodules is added.  This might lead to references to invalidated
+modules.
 
 We should use an approach that:
 
-* Never should allow access an invalidated object
-* Does not stop garbage collection from freeing invalidated objects
+* Never should allow access to an invalidated object
+* Should not stop garbage collection from freeing invalidated objects
 * Should be easy to test
 
 The current approach fails in all of these.
 
 Why weakref cannot be used in ConcludedData?
   Concluded data can hold anything.  It can hold a `PyObject` whose
-  type might be invalidated.  But since that pyobject is not
+  type might be a `PyDefinedObject`.  But since that pyobject is not
   referenced it is considered garbage.
 
 
   observers
 
 
-Adding `PyModule.is_valid()`
-----------------------------
-
-This requires introducing `_PyObjectRef`, `_PyNameRef`, and probably
-`_PyObjectListRef`.  We'll also be able to remove the list of
-concluded objects in `PyModule`.  These objects should use weak
-references.
-
-But this does not seem to work::
-
-  class DependentRef(object):
-      """The data remains valid as long as the dependency remains valid"""
-
-      def __init__(self, dependency=None, data=None):
-          self.dependency = weak(dependency)
-          self.data = weak(data)
-
-      def get(self):
-          if self._is_dependency_valid():
-              return self.data
-          self.data = None
-
-
-  class PyObjectRef(object):
-
-      def __init__(self, pyobject=None):
-          self.data = weak(data)
-
-      def get(self):
-          if self.is_valid(self.data):
-              return self.data
-          self.data = None
-
-
-Invalidating Imported Modules In Both Directions
-------------------------------------------------
-
-If other approaches fails I have to use this.  We have to handle
-loops, by the way.  In the end, it does not solve all of the problems!
-
-
 Better Concluded Data
 =====================
 

File docs/dev/workingon.txt

View file
  • Ignore whitespace
 Small Stories
 =============
 
-- Every change makes all pymodules forget all concluded data
+- Handling strings in following lines in `patchedast`
 
-* Using weakref to prevent unnecessary invalidation?
-* Moving `PyCore.create_module()` and `PyCore.create_package()`?
-* Adding `rope.base.refs`?
-* Invalidation problems and function parameter cache
-* Removing function parameter cache using set on individual `ParameterName`\s
+* Refactoring `rope.refactor.patchedast`
+* Moving `PyCore.create_module()` and `PyCore.create_package()`? Where
+  to? `libutils` maybe?
 
-* Handling strings in following lines in `patchedast`
 * Using `svn` instead of `pysvn`
 * Extracting subexpressions; look at `extracttest` for more info
 * Create ... and implicit interfaces?

File rope/refactor/patchedast.py

View file
  • Ignore whitespace
         self.source = _Source(source)
         self.children = children
         self.lines = codeanalyze.SourceLinesAdapter(source)
+        self.children_stack = []
 
     Number = object()
     String = object()
         if hasattr(node, 'region'):
             raise RuntimeError('Node <%s> has been already patched!' %
                                node.__class__.__name__)
+        self.children_stack.append(base_children)
         children = []
         formats = []
         suspected_start = self.source.offset
         start = suspected_start
         first_token = True
-        for child in base_children:
+        while base_children:
+            child = base_children.pop(0)
             if child is None:
                 continue
             offset = self.source.offset
                 token_start = child.region[0]
             else:
                 if child is self.String:
-                    region = self.source.consume_string()
+                    region = self.source.consume_string(
+                        end=self._find_next_statement_start())
                 elif child is self.Number:
                     region = self.source.consume_number()
                 elif child == '!=':
         if self.children:
             node.sorted_children = children
         node.region = (start, self.source.offset)
+        self.children_stack.pop()
 
     def _handle_parens(self, children, start, formats):
         """Changes `children` and returns new start"""
                 index += 1
         return start, opens
 
+    def _find_next_statement_start(self):
+        for children in reversed(self.children_stack):
+            for child in children:
+                if isinstance(child, ast.stmt):
+                    return self.lines.get_line_start(child.lineno)
+        return len(self.source.source)
+
     _operators = {'And': 'and', 'Or': 'or', 'Add': '+', 'Sub': '-', 'Mult': '*',
                   'Div': '/', 'Mod': '%', 'Pow': '**', 'LShift': '<<',
                   'RShift': '>>', 'BitOr': '|', 'BitAnd': '&', 'BitXor': '^',
         self._handle(node, children)
 
     def _Module(self, node):
-        self._handle(node, node.body, eat_spaces=True)
+        self._handle(node, list(node.body), eat_spaces=True)
 
     def _Name(self, node):
         self._handle(node, [node.id])
         self.offset = new_offset + len(token)
         return (new_offset, self.offset)
 
-    def consume_string(self):
+    def consume_string(self, end=None):
         if _Source._string_pattern is None:
             original = codeanalyze.get_string_pattern()
             pattern = r'(%s)((\s|\\\n|#[^\n]*\n)*(%s))*' % \
                       (original, original)
             _Source._string_pattern = re.compile(pattern)
         repattern = _Source._string_pattern
-        return self._consume_pattern(repattern)
+        return self._consume_pattern(repattern, end)
 
     def consume_number(self):
         if _Source._number_pattern is None:
         lines = self.source[:self.offset].split('\n')
         return (len(lines), len(lines[-1]))
 
-    def _consume_pattern(self, repattern):
+    def _consume_pattern(self, repattern, end=None):
         while True:
-            match = repattern.search(self.source, self.offset)
+            if end is None:
+                end = len(self.source)
+            match = repattern.search(self.source, self.offset, end)
             if self._good_token(match.group(), match.start()):
                 break
             else:

File ropetest/refactor/patchedasttest.py

View file
  • Ignore whitespace
             'Assign', ['Name' , ' ', '=', ' (', 'Str', ')'])
         checker.check_children('Str', ["'1' \n'2'"])
 
-    # XXX: Handle this case
-    def xxx_test_not_concatenating_strings_on_separate_lines(self):
+    def test_not_concatenating_strings_on_separate_lines(self):
         source = "'1'\n'2'\n"
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
+        checker.check_children('Module', ['', 'Expr', '\n', 'Expr', '\n'])
 
     def test_long_integer_literals(self):
         source = "0x1L + a"