Commits

Anonymous committed a569f23

Using _ast module in patchedast and similarfinder

Comments (0)

Files changed (10)

 New Features
 ============
 
-* Adding custom source folders in project ``config.py``
-* A simple UI for performing restructurings; ``C-c r x``
-* Restructurings
-
-From this release rope will no longer support Python ``2.4`` and rope
-``0.5`` was the last version that supported it.
-
-Custom Source Folders
----------------------
-
-By default rope searches the project for finding source folders
-(folders that should be searched for finding modules).  You can add
-paths to that list using ``source_folders`` config.  Note that rope
-guesses project source folders correctly most of the time.  You can
-also extend python path using ``python_path`` config.
-
-Restructurings
---------------
-
-Restructuring support is one of the main goals of the ``0.6`` release
-of rope.  `rope.refactor.restructure` can be used for performing
-restructurings.  Currently a simple dialog has been added for
-performing them, but you cannot add checks to your restructuring in
-that dialog, yet.  But the full functionality is available if you're
-using rope as a library.
-
-A restructuring is a program transformation; not as well defined as
-other refactorings like rename.  Let's see some examples.
-
-Example 1
-'''''''''
-
-In its basic form we have a pattern and a goal.  Consider we were not
-aware of the ``**`` operator and wrote our own ::
-
-  def pow(x, y):
-      result = 1
-      for i in range(y):
-          result *= x
-      return result
-
-  print pow(2, 3)
-
-Now that we know ``**`` exists we want to use it wherever `pow` is
-used (There might be hundreds of them!).  We can use a pattern like::
-
-  pattern = 'pow(${?param1}, ${?param2})'
-
-Goal can be some thing like::
-
-  goal = '${?param1} ** ${?param2}'
-
-Note that ``${...}`` is used to match something in the pattern.  If
-names that appear in ``${...}`` start with a leading ``?`` every
-expression at that point will match, otherwise only the specified name
-will match (This form is not useful if you're not using checks that is
-described later).
-
-You can use the matched names in goal and they will be replaced with
-the string that was matched in each occurrence.  So the outcome of our
-restructuring will be::
-
-  def pow(x, y):
-      result = 1
-      for i in range(y):
-          result *= x
-      return result
-
-  print 2 ** 3
-
-It seems to be working but what if `pow` is imported in some module or
-we have some other function defined in some other module that uses the
-same name and we don't want to change it.  Checks come to rescue.  Each
-restructuring gets a ``checks`` parameter in its constructor.  It can
-be a dictionary.  Its keys are pattern names that appear in the
-pattern (the names in ``${...}``) or it can be pattern names plus any
-of ``.object`` or ``.type``.  The values can be rope
-`rope.base.pyobject.PyObject` or `rope.base.pynames.PyNames` objects.
-
-For solving the above problem we change our `pattern`.  But `goal`
-remains the same::
-
-  pattern = '${?pow_func}(${?param1}, ${?param2})'
-  goal = '${?param1} ** ${?param2}'
-
-Consider the name of the module containing our `pow` function is
-`mod`.  ``checks`` can be::
-
-  checks = {}
-  mod = project.get_pycore().get_module('mod')
-  pow_pyname = mod.get_attribute('pow')
-  checks['?pow_func'] = pow_pyname
-
-Note that project is an instance of `rope.base.project.Project`.  We
-can perform the restructuring now::
-
-  from rope.refactor import restructure
-
-  restructuring = restructure.Restructure(project, pattern, goal, checks)
-  project.do(restructuring.get_changes())
-
-`PyName`\s and `PyObject`\s are used to describe names and objects in
-rope.  Each name in a program (a `PyName`) might reference an object
-(a `PyObject`) that has a type (a `PyObject`).
-
-
-Example 2
-'''''''''
-
-As another example consider::
-
-  class A(object):
-
-      def f(self, p1, p2):
-          print p1
-          print p2
-
-
-  a = A()
-  a.f(1, 2)
-
-Later we decide that `A.f()` is doing too much and we want to divide
-it to `A.f1()` and `A.f2()`::
-
-  class A(object):
-
-      def f(self, p1, p2):
-          print p1
-          print p2
-
-      def f1(self, p):
-          print p
-
-      def f2(self, p):
-          print p2
-
-
-  a = A()
-  a.f(1, 2)
-
-But who's going to fix all those nasty occurrences (Actually this
-situation can be handled using inline method refactoring but this is
-just an example; Consider inline refactoring is not implemented yet!).
-Restructurings come to rescue::
-
-  pattern = '${?inst}.f(${?p1}, ${?p2})'
-  goal = '${?inst}.f1(${?p1}); ${?inst}.f2(${?p2})\n'
-  
-  mod = project.get_pycore().get_module('my.mod')
-  a_class_pyname = mod.get_attribute('A')
-  a_class_pyobject = a_pyname.get_object()
-  checks = {}
-  checks['?inst.type'] = a_class_pyobject
-
-We can perform the restructuring using `Restructure` class as shown
-above.  We will have::
-
-  class A(object):
-
-      def f(self, p1, p2):
-          print p1
-          print p2
-
-      def f1(self, p):
-          print p
-
-      def f2(self, p):
-          print p2
-
-
-  a = A()
-  a.f1(1); a.f2(2)
-
-Issues
-''''''
-
-Adding checks is not available in the restructuring dialog, yet.  The
-main reason is that I couldn't find a user-friendly way for specifying
-them (though I have something in mind for the next release).  If you
-have any idea I'll be glad to hear.
-
-The other constraint that restructurings have is that pattern names
-can only appear in at the start of an expression.  For instance
-``var.${name}`` is invalid.  These situations can usually be fixed by
-specifying good checks, for example on the type of `var` and using a
-``${var}.name`` pattern.
+* 
 
 
 Getting Started

docs/dev/stories.txt

 * Renaming and moving normal files/folders
 
 
+* Finding similar statements when extracting variable/method
+
+
+> Public Release 0.6m2 : June 3, 2007
+
+
 * Enhanced restructuring dialog
-
-
-* Finding similar statements when extracting variable/method

docs/dev/workingon.txt

-Small Stories
-=============
+Using `_ast` instead of `compiler
+=================================
+
+- Using `_ast` in `patchedast`
+- Using `_ast` in `similarfinder`
+
+* Handling strings in following lines in `patchedast`
+* Using `PyModule.ast` in `similarfinder`
 
 * Should we call `pycore.resource_to_pyobject()` for all modules
 * The ability to limit the files to restructure

docs/user/overview.txt

 parameters that are passed to a function.
 
 
+Restructurings
+--------------
+
+Restructuring support is one of the main goals of the ``0.6`` release
+of rope.  `rope.refactor.restructure` can be used for performing
+restructurings.  Currently a simple dialog has been added for
+performing them, but you cannot add checks to your restructuring in
+that dialog, yet.  But the full functionality is available if you're
+using rope as a library.
+
+A restructuring is a program transformation; not as well defined as
+other refactorings like rename.  Let's see some examples.
+
+Example 1
+'''''''''
+
+In its basic form we have a pattern and a goal.  Consider we were not
+aware of the ``**`` operator and wrote our own ::
+
+  def pow(x, y):
+      result = 1
+      for i in range(y):
+          result *= x
+      return result
+
+  print pow(2, 3)
+
+Now that we know ``**`` exists we want to use it wherever `pow` is
+used (There might be hundreds of them!).  We can use a pattern like::
+
+  pattern = 'pow(${?param1}, ${?param2})'
+
+Goal can be some thing like::
+
+  goal = '${?param1} ** ${?param2}'
+
+Note that ``${...}`` is used to match something in the pattern.  If
+names that appear in ``${...}`` start with a leading ``?`` every
+expression at that point will match, otherwise only the specified name
+will match (This form is not useful if you're not using checks that is
+described later).
+
+You can use the matched names in goal and they will be replaced with
+the string that was matched in each occurrence.  So the outcome of our
+restructuring will be::
+
+  def pow(x, y):
+      result = 1
+      for i in range(y):
+          result *= x
+      return result
+
+  print 2 ** 3
+
+It seems to be working but what if `pow` is imported in some module or
+we have some other function defined in some other module that uses the
+same name and we don't want to change it.  Checks come to rescue.  Each
+restructuring gets a ``checks`` parameter in its constructor.  It can
+be a dictionary.  Its keys are pattern names that appear in the
+pattern (the names in ``${...}``) or it can be pattern names plus any
+of ``.object`` or ``.type``.  The values can be rope
+`rope.base.pyobject.PyObject` or `rope.base.pynames.PyNames` objects.
+
+For solving the above problem we change our `pattern`.  But `goal`
+remains the same::
+
+  pattern = '${?pow_func}(${?param1}, ${?param2})'
+  goal = '${?param1} ** ${?param2}'
+
+Consider the name of the module containing our `pow` function is
+`mod`.  ``checks`` can be::
+
+  checks = {}
+  mod = project.get_pycore().get_module('mod')
+  pow_pyname = mod.get_attribute('pow')
+  checks['?pow_func'] = pow_pyname
+
+Note that project is an instance of `rope.base.project.Project`.  We
+can perform the restructuring now::
+
+  from rope.refactor import restructure
+
+  restructuring = restructure.Restructure(project, pattern, goal, checks)
+  project.do(restructuring.get_changes())
+
+`PyName`\s and `PyObject`\s are used to describe names and objects in
+rope.  Each name in a program (a `PyName`) might reference an object
+(a `PyObject`) that has a type (a `PyObject`).
+
+
+Example 2
+'''''''''
+
+As another example consider::
+
+  class A(object):
+
+      def f(self, p1, p2):
+          print p1
+          print p2
+
+
+  a = A()
+  a.f(1, 2)
+
+Later we decide that `A.f()` is doing too much and we want to divide
+it to `A.f1()` and `A.f2()`::
+
+  class A(object):
+
+      def f(self, p1, p2):
+          print p1
+          print p2
+
+      def f1(self, p):
+          print p
+
+      def f2(self, p):
+          print p2
+
+
+  a = A()
+  a.f(1, 2)
+
+But who's going to fix all those nasty occurrences (Actually this
+situation can be handled using inline method refactoring but this is
+just an example; Consider inline refactoring is not implemented yet!).
+Restructurings come to rescue::
+
+  pattern = '${?inst}.f(${?p1}, ${?p2})'
+  goal = '${?inst}.f1(${?p1}); ${?inst}.f2(${?p2})\n'
+  
+  mod = project.get_pycore().get_module('my.mod')
+  a_class_pyname = mod.get_attribute('A')
+  a_class_pyobject = a_pyname.get_object()
+  checks = {}
+  checks['?inst.type'] = a_class_pyobject
+
+We can perform the restructuring using `Restructure` class as shown
+above.  We will have::
+
+  class A(object):
+
+      def f(self, p1, p2):
+          print p1
+          print p2
+
+      def f1(self, p):
+          print p
+
+      def f2(self, p):
+          print p2
+
+
+  a = A()
+  a.f1(1); a.f2(2)
+
+Issues
+''''''
+
+Adding checks is not available in the restructuring dialog, yet.  The
+main reason is that I couldn't find a user-friendly way for specifying
+them (though I have something in mind for the next release).  If you
+have any idea I'll be glad to hear.
+
+The other constraint that restructurings have is that pattern names
+can only appear in at the start of an expression.  For instance
+``var.${name}`` is invalid.  These situations can usually be fixed by
+specifying good checks, for example on the type of `var` and using a
+``${var}.name`` pattern.
+
+
 Object Inference
 ================
 
 you're new to rope), though that is not recommended.
 
 
+Custom Source Folders
+---------------------
+
+By default rope searches the project for finding source folders
+(folders that should be searched for finding modules).  You can add
+paths to that list using ``source_folders`` config.  Note that rope
+guesses project source folders correctly most of the time.  You can
+also extend python path using ``python_path`` config.
+
+
 Outline
 =======
 
 """rope, a python refactoring library and IDE"""
 
-VERSION = '0.6m1'
+VERSION = '0.6m2'
             if isinstance(child, _ast.AST):
                 result.append(child)
     return result
+
+
+def call_for_nodes(node, callback, recursive=False):
+    """If callback returns `True` the child nodes are skipped"""
+    result = callback(node)
+    if recursive and not result:
+        for child in get_child_nodes(node):
+            call_for_nodes(child, callback, recursive)
+
+
+def get_children(node):
+    result = []
+    if node._fields is not None:
+        for name in node._fields:
+            child = getattr(node, name)
+            result.append(child)
+    return result

rope/refactor/patchedast.py

-import compiler.ast
-import compiler.consts
 import re
 
-from rope.base import codeanalyze, exceptions
+from rope.base import ast, codeanalyze, exceptions
 
 
 def get_patched_ast(source, sorted_children=False):
     """Adds `region` and `sorted_children` fields to nodes"""
-    return patch_ast(compiler.parse(source), source, sorted_children)
+    return patch_ast(ast.parse(source), source, sorted_children)
 
 
-def patch_ast(ast, source, sorted_children=False):
-    """Patches the given ast
+def patch_ast(node, source, sorted_children=False):
+    """Patches the given node
     
-    After calling, each node in `ast` will have a new field named
+    After calling, each node in `node` will have a new field named
     `region` that is a tuple containing the start and end offsets
     of the code that generated it.
     
     them.
 
     """
-    if hasattr(ast, 'region'):
-        return ast
+    if hasattr(node, 'region'):
+        return node
     walker = _PatchingASTWalker(source, children=sorted_children)
-    call_for_nodes(ast, walker)
-    return ast
+    ast.call_for_nodes(node, walker)
+    return node
 
 
-def call_for_nodes(ast, callback, recursive=False):
-    """If callback returns `True` the child nodes are skipped"""
-    result = callback(ast)
-    if recursive and not result:
-        for child in ast.getChildNodes():
-            call_for_nodes(child, callback, recursive)
-
 def node_region(patched_ast_node):
     """Get the region of a patched ast node"""
     return patched_ast_node.region
     """
     result = []
     for child in patched_ast_node.sorted_children:
-        if isinstance(child, compiler.ast.Node):
+        if isinstance(child, ast.AST):
             result.append(write_ast(child))
         else:
             result.append(child)
 class MismatchedTokenError(exceptions.RopeError):
     pass
 
+
 class _PatchingASTWalker(object):
 
     def __init__(self, source, children=False):
         self.source = _Source(source)
         self.children = children
+        self.lines = codeanalyze.SourceLinesAdapter(source)
 
     Number = object()
     String = object()
 
     def __call__(self, node):
-        method = getattr(self, 'visit' + node.__class__.__name__, None)
+        method = getattr(self, '_' + node.__class__.__name__, None)
         if method is not None:
             return method(node)
         # ???: Unknown node; What should we do here?
                            node.__class__.__name__)
 
     def _handle(self, node, base_children, eat_parens=False, eat_spaces=False):
+        if hasattr(node, 'region'):
+            raise RuntimeError('Node <%s> has been already patched!' %
+                               node.__class__.__name__)
         children = []
         formats = []
         suspected_start = self.source.offset
             if child is None:
                 continue
             offset = self.source.offset
-            if isinstance(child, compiler.ast.Node):
-                call_for_nodes(child, self)
+            if isinstance(child, ast.AST):
+                ast.call_for_nodes(child, self)
                 token_start = child.region[0]
             else:
                 if child is self.String:
                 index += 1
         return start, opens
 
-    def visitAdd(self, node):
-        self._handle(node, [node.left, '+', node.right])
+    _operators = {'And': 'and', 'Or': 'or', 'Add': '+', 'Sub': '-', 'Mult': '*',
+                  'Div': '/', 'Mod': '%', 'Pow': '**', 'LShift': '<<',
+                  'RShift': '>>', 'BitOr': '|', 'BitAnd': '&', 'BitXor': '^',
+                  'FloorDiv': '//', 'Invert': '~', 'Not': 'not', 'UAdd': '+',
+                  'USub': '-', 'Eq': '==', 'NotEq': '!=', 'Lt': '<',
+                  'LtE': '<=', 'Gt': '>', 'GtE': '>=', 'Is': 'is',
+                  'IsNot': 'is not', 'In': 'in', 'NotIn': 'not in'}
 
-    def visitAnd(self, node):
-        self._handle(node, self._child_nodes(node.nodes, 'and'))
+    def _get_op(self, node):
+        return self._operators[node.__class__.__name__].split(' ')
 
-    def visitAssAttr(self, node):
-        self._handle(node, [node.expr, '.', node.attrname])
+    def _Attribute(self, node):
+        self._handle(node, [node.value, '.', node.attr])
 
-    def visitAssList(self, node):
-        children = self._child_nodes(node.nodes, ',')
-        children.insert(0, '[')
-        children.append(']')
+    def _Assert(self, node):
+        children = ['assert', node.test]
+        if node.msg:
+            children.append(',')
+            children.append(node.msg)
         self._handle(node, children)
 
-    def visitAssName(self, node):
-        self._handle(node, [node.name])
-
-    def visitAssTuple(self, node):
-        self._handle_tuple(node)
-
-    def visitAssert(self, node):
-        children = ['assert', node.test]
-        if node.fail:
-            children.append(',')
-            children.append(node.fail)
+    def _Assign(self, node):
+        children = self._child_nodes(node.targets, '=')
+        children.append('=')
+        children.append(node.value)
         self._handle(node, children)
 
-    def visitAssign(self, node):
-        children = self._child_nodes(node.nodes, '=')
-        children.append('=')
-        children.append(node.expr)
+    def _AugAssign(self, node):
+        children = [node.target]
+        children.extend(self._get_op(node.op))
+        children.extend(['=', node.value])
         self._handle(node, children)
 
-    def visitAugAssign(self, node):
-        self._handle(node, [node.node, node.op, node.expr])
+    def _Repr(self, node):
+        self._handle(node, ['`', node.value, '`'])
 
-    def visitBackquote(self, node):
-        self._handle(node, ['`', node.expr, '`'])
+    def _BinOp(self, node):
+        children = [node.left] + self._get_op(node.op) + [node.right]
+        self._handle(node, children)
 
-    def visitBitand(self, node):
-        self._handle(node, self._child_nodes(node.nodes, '&'))
+    def _BoolOp(self, node):
+        self._handle(node, self._child_nodes(node.values,
+                                             self._get_op(node.op)[0]))
 
-    def visitBitor(self, node):
-        self._handle(node, self._child_nodes(node.nodes, '|'))
-
-    def visitBitxor(self, node):
-        self._handle(node, self._child_nodes(node.nodes, '^'))
-
-    def visitBreak(self, node):
+    def _Break(self, node):
         self._handle(node, ['break'])
 
-    def visitCallFunc(self, node):
-        children = [node.node, '(']
-        children.extend(self._child_nodes(node.args, ','))
-        if node.star_args is not None:
-            if node.args:
+    def _Call(self, node):
+        children = [node.func, '(']
+        args = list(node.args) + node.keywords
+        children.extend(self._child_nodes(args, ','))
+        if node.starargs is not None:
+            if args:
                 children.append(',')
-            children.extend(['*', node.star_args])
-        if node.dstar_args is not None:
-            if node.args:
+            children.extend(['*', node.starargs])
+        if node.kwargs is not None:
+            if args:
                 children.append(',')
-            children.extend(['**', node.dstar_args])
+            children.extend(['**', node.kwargs])
         children.append(')')
         self._handle(node, children)
 
-    def visitClass(self, node):
+    def _ClassDef(self, node):
         children = []
         children.extend(['class', node.name])
         if node.bases:
             children.extend(self._child_nodes(node.bases, ','))
             children.append(')')
         children.append(':')
-        if node.doc is not None:
-            children.append(self.String)
-        children.append(node.code)
+        children.extend(node.body)
         self._handle(node, children)
 
-    def visitCompare(self, node):
+    def _Compare(self, node):
         children = []
-        children.append(node.expr)
-        for op, child in node.ops:
-            children.append(op)
-            children.append(child)
+        children.append(node.left)
+        for op, expr in zip(node.ops, node.comparators):
+            children.extend(self._get_op(op))
+            children.append(expr)
         self._handle(node, children)
 
-    def visitConst(self, node):
-        value = repr(node.value)
-        if isinstance(node.value, (int, long, float, complex)):
-            value = self.Number
-        if isinstance(node.value, basestring):
-            value = self.String
-        self._handle(node, [value])
+    def _Delete(self, node):
+        self._handle(node, ['del'] + self._child_nodes(node.targets, ','))
 
-    def visitContinue(self, node):
+    def _Num(self, node):
+        self._handle(node, [self.Number])
+
+    def _Str(self, node):
+        self._handle(node, [self.String])
+
+    def _Continue(self, node):
         self._handle(node, ['continue'])
 
-    def visitDecorators(self, node):
-        self._handle(node, ['@'] + self._child_nodes(node.nodes, '@'))
-
-    def visitDict(self, node):
+    def _Dict(self, node):
         children = []
         children.append('{')
-        for index, (key, value) in enumerate(node.items):
-            children.extend([key, ':', value])
-            if index < len(node.items) - 1:
-                children.append(',')
+        if node.keys:
+            for index, (key, value) in enumerate(zip(node.keys, node.values)):
+                children.extend([key, ':', value])
+                if index < len(node.keys) - 1:
+                    children.append(',')
         children.append('}')
         self._handle(node, children)
 
-    def visitDiscard(self, node):
+    def _Ellipsis(self, node):
+        self._handle(node, ['...'])
+
+    def _Expr(self, node):
+        self._handle(node, [node.value])
+
+    def _Exec(self, node):
         children = []
-        if not self._is_none_or_const_none(node.expr):
-            children.append(node.expr)
+        children.extend(['exec', node.body])
+        if node.globals:
+            children.extend(['in', node.globals])
+        if node.locals:
+            children.extend([',', node.locals])
         self._handle(node, children)
 
-    def visitDiv(self, node):
-        self._handle(node, [node.left, '/', node.right])
-
-    def visitEllipsis(self, node):
-        self._handle(node, ['...'])
-
-    def visitExpression(self, node):
-        self._handle(node, [node.node])
-
-    def visitExec(self, node):
-        children = []
-        children.extend(['exec', node.expr])
-        if node.locals:
-            children.extend(['in', node.locals])
-        if node.globals:
-            children.extend([',', node.globals])
+    def _For(self, node):
+        children = ['for', node.target, 'in', node.iter, ':']
+        children.extend(node.body)
+        if node.orelse:
+            children.extend(['else', ':'])
+            children.extend(node.orelse)
         self._handle(node, children)
 
-    def visitFloorDiv(self, node):
-        self._handle(node, [node.left, '//', node.right])
-
-    def visitFor(self, node):
-        children = ['for', node.assign, 'in', node.list, ':', node.body]
-        if node.else_:
-            children.extend(['else', ':', node.else_])
+    def _ImportFrom(self, node):
+        children = ['from']
+        if node.level:
+            children.append('.' * node.level)
+        children.extend([node.module, 'import'])
+        children.extend(self._child_nodes(node.names, ','))
         self._handle(node, children)
 
-    def visitFrom(self, node):
-        children = ['from']
-        if hasattr(node, 'level') and node.level > 0:
-            children.append('.' * node.level)
-        children.extend([node.modname, 'import'])
-        for index, (name, alias) in enumerate(node.names):
-            children.append(name)
-            if alias is not None:
-                children.extend(['as', alias])
-            if index < len(node.names) - 1:
-                children.append(',')
+    def _alias(self, node):
+        children = [node.name]
+        if node.asname:
+            children.extend(['as', node.asname])
         self._handle(node, children)
 
-    def visitFunction(self, node):
+    def _FunctionDef(self, node):
         children = []
         if node.decorators:
-            children.append(node.decorators)
-        children.extend(['def', node.name, '('])
-        children.extend(self._arg_list(node.argnames, node.defaults,
-                                       node.flags))
+            for decorator in node.decorators:
+                children.append('@')
+                children.append(decorator)
+        children.extend(['def', node.name, '(', node.args])
         children.extend([')', ':'])
-        if node.doc:
-            children.append(self.String)
-        children.append(node.code)
+        children.extend(node.body)
         self._handle(node, children)
 
-    def _arg_list(self, argnames, defaults, flags):
+    def _arguments(self, node):
         children = []
-        args = list(argnames)
-        dstar_args = None
-        if flags & compiler.consts.CO_VARKEYWORDS:
-            dstar_args = args.pop()
-        star_args = None
-        if flags & compiler.consts.CO_VARARGS:
-            star_args = args.pop()
-        defaults = [None] * (len(args) - len(defaults)) + list(defaults)
+        args = list(node.args)
+        star_args = node.vararg
+        dstar_args = node.kwarg
+        defaults = [None] * (len(args) - len(node.defaults)) + list(node.defaults)
         for index, (arg, default) in enumerate(zip(args, defaults)):
             if index > 0:
                 children.append(',')
             if args:
                 children.append(',')
             children.extend(['**', dstar_args])
-        return children
+        self._handle(node, children)
 
     def _add_args_to_children(self, children, arg, default):
         if isinstance(arg, (list, tuple)):
                 children.append(token)
         children.append(')')
 
-    def visitGenExpr(self, node):
-        self._handle(node, [node.code], eat_parens=True)
+    def _GeneratorExp(self, node):
+        children = [node.elt]
+        children.extend(node.generators)
+        self._handle(node, children, eat_parens=True)
 
-    def visitGenExprFor(self, node):
-        children = ['for', node.assign, 'in', node.iter]
+    def _comprehension(self, node):
+        children = ['for', node.target, 'in', node.iter]
         if node.ifs:
-            children.extend(node.ifs)
+            for if_ in node.ifs:
+                children.append('if')
+                children.append(if_)
         self._handle(node, children)
 
-    def visitGenExprIf(self, node):
-        self._handle(node, ['if', node.test])
-
-    def visitGenExprInner(self, node):
-        self._handle(node, [node.expr] + node.quals)
-
-    def visitGetattr(self, node):
-        self._handle(node, [node.expr, '.', node.attrname])
-
-    def visitGlobal(self, node):
+    def _Global(self, node):
         children = self._child_nodes(node.names, ',')
         children.insert(0, 'global')
         self._handle(node, children)
 
-    def visitIf(self, node):
-        children = ['if', node.tests[0][0], ':', node.tests[0][1]]
-        for test, body in node.tests[1:]:
-            children.extend(['elif', test, ':', body])
-        if node.else_:
-            children.extend(['else', ':', node.else_])
+    def _If(self, node):
+        if self._is_elif(node):
+            children = ['elif']
+        else:
+            children = ['if']
+        children.extend([node.test, ':'])
+        children.extend(node.body)
+        if node.orelse:
+            if len(node.orelse) == 1 and self._is_elif(node.orelse[0]):
+                pass
+            else:
+                children.extend(['else', ':'])
+            children.extend(node.orelse)
         self._handle(node, children)
 
-    def visitIfExp(self, node):
-        return self._handle(node, [node.then, 'if', node.test,
-                                   'else', node.else_])
+    def _is_elif(self, node):
+        if not isinstance(node, ast.If):
+            return False
+        offset = self.lines.get_line_start(node.lineno) + node.col_offset
+        word = self.source[offset:offset + 4]
+        # XXX: This is a bug; the offset does not point to the first
+        alt_word = self.source[offset - 5:offset - 1]
+        return 'elif' in (word, alt_word)
 
-    def visitImport(self, node):
+    def _IfExp(self, node):
+        return self._handle(node, [node.body, 'if', node.test,
+                                   'else', node.orelse])
+
+    def _Import(self, node):
         children = ['import']
-        for index, (name, alias) in enumerate(node.names):
-            if index != 0:
-                children.append(',')
-            children.append(name)
-            if alias is not None:
-                children.extend(['as', alias])
+        children.extend(self._child_nodes(node.names, ','))
         self._handle(node, children)
 
-    def visitInvert(self, node):
-        self._handle(node, ['~', node.expr])
+    def _keyword(self, node):
+        self._handle(node, [node.arg, '=', node.value])
 
-    def visitKeyword(self, node):
-        self._handle(node, [node.name, '=', node.expr])
+    def _Lambda(self, node):
+        self._handle(node, ['lambda', node.args, ':', node.body])
 
-    def visitLambda(self, node):
-        children = ['lambda']
-        children.extend(self._arg_list(node.argnames, node.defaults,
-                                       node.flags))
-        children.extend([':', node.code])
+    def _List(self, node):
+        self._handle(node, ['['] + self._child_nodes(node.elts, ',') + [']'])
+
+    def _ListComp(self, node):
+        children = ['[', node.elt]
+        children.extend(node.generators)
+        children.append(']')
         self._handle(node, children)
 
-    def visitLeftShift(self, node):
-        self._handle(node, [node.left, '<<', node.right])
+    def _Module(self, node):
+        self._handle(node, node.body, eat_spaces=True)
 
-    def visitList(self, node):
-        self._handle(node, ['['] + self._child_nodes(node.nodes, ',') + [']'])
+    def _Name(self, node):
+        self._handle(node, [node.id])
 
-    def visitListCompFor(self, node):
-        children = ['for', node.assign, 'in', node.list]
-        if node.ifs:
-            children.extend(node.ifs)
-        self._handle(node, children)
-
-    def visitListCompIf(self, node):
-        self._handle(node, ['if', node.test])
-
-    def visitListComp(self, node):
-        self._handle(node, ['[', node.expr] + node.quals + [']'])
-
-    def visitMod(self, node):
-        self._handle(node, [node.left, '%', node.right])
-
-    def visitModule(self, node):
-        doc = None
-        if node.doc is not None:
-            doc = self.String
-        self._handle(node, [doc, node.node], eat_spaces=True)
-
-    def visitMul(self, node):
-        self._handle(node, [node.left, '*', node.right])
-
-    def visitName(self, node):
-        self._handle(node, [node.name])
-
-    def visitNot(self, node):
-        self._handle(node, ['not', node.expr])
-
-    def visitOr(self, node):
-        self._handle(node, self._child_nodes(node.nodes, 'or'))
-
-    def visitPass(self, node):
+    def _Pass(self, node):
         self._handle(node, ['pass'])
 
-    def visitPower(self, node):
-        self._handle(node, [node.left, '**', node.right])
-
-    def visitPrint(self, node):
-        children = self._base_print(node)
-        children.append(',')
-        self._handle(node, children)
-
-    def visitPrintnl(self, node):
-        self._handle(node, self._base_print(node))
-
-    def _base_print(self, node):
+    def _Print(self, node):
         children = ['print']
         if node.dest:
             children.extend(['>>', node.dest])
-            if node.nodes:
+            if node.values:
                 children.append(',')
-        children.extend(self._child_nodes(node.nodes, ','))
-        return children
-
-    def visitRaise(self, node):
-        children = ['raise']
-        if node.expr1:
-            children.append(node.expr1)
-        if node.expr2:
+        children.extend(self._child_nodes(node.values, ','))
+        if not node.nl:
             children.append(',')
-            children.append(node.expr2)
-        if node.expr3:
-            children.append(',')
-            children.append(node.expr3)
         self._handle(node, children)
 
-    def visitReturn(self, node):
+    def _Raise(self, node):
+        children = ['raise']
+        if node.type:
+            children.append(node.type)
+        if node.inst:
+            children.append(',')
+            children.append(node.inst)
+        if node.tback:
+            children.append(',')
+            children.append(node.tback)
+        self._handle(node, children)
+
+    def _Return(self, node):
         children = ['return']
-        if not self._is_none_or_const_none(node.value):
+        if node.value:
             children.append(node.value)
         self._handle(node, children)
 
-    def _is_none_or_const_none(self, node):
-        return node is None or (isinstance(node, compiler.ast.Const) and
-                                node.value == None)
+    def _Sliceobj(self, node):
+        children = []
+        for index, slice in enumerate(node.nodes):
+            if index > 0:
+                children.append(':')
+            if slice:
+                children.append(slice)
+        self._handle(node, children)
 
-    def visitRightShift(self, node):
-        self._handle(node, [node.left, '>>', node.right])
+    def _Index(self, node):
+        self._handle(node, [node.value])
 
-    def visitSlice(self, node):
-        children = [node.expr, '[']
+    def _Subscript(self, node):
+        self._handle(node, [node.value, '[', node.slice, ']'])
+
+    def _Slice(self, node):
+        children = []
         if node.lower:
             children.append(node.lower)
         children.append(':')
         if node.upper:
             children.append(node.upper)
-        children.append(']')
+        if node.step:
+            children.append(':')
+            children.append(node.step)
         self._handle(node, children)
 
-    def visitSliceobj(self, node):
+    def _TryFinally(self, node):
         children = []
-        for index, slice in enumerate(node.nodes):
-            if index > 0:
-                children.append(':')
-            if not self._is_none_or_const_none(slice):
-                children.append(slice)
+        if len(node.body) != 1 or not isinstance(node.body[0], ast.TryExcept):
+            children.extend(['try', ':'])
+        children.extend(node.body)
+        children.extend(['finally', ':'])
+        children.extend(node.finalbody)
         self._handle(node, children)
 
-    def visitStmt(self, node):
-        self._handle(node, node.nodes)
-
-    def visitSub(self, node):
-        self._handle(node, [node.left, '-', node.right])
-
-    def visitSubscript(self, node):
-        self._handle(node, [node.expr, '['] +
-                            self._child_nodes(node.subs, ',') + [']'])
-
-    def visitTuple(self, node):
-        self._handle_tuple(node)
-
-    def visitTryFinally(self, node):
-        children = []
-        if not isinstance(node.body, compiler.ast.TryExcept):
-            children.extend(['try', ':'])
-        children.extend([node.body, 'finally', ':', node.final])
+    def _TryExcept(self, node):
+        children = ['try', ':']
+        children.extend(node.body)
+        children.extend(node.handlers)
+        if node.orelse:
+            children.extend(['else', ':'])
+            children.extend(node.orelse)
         self._handle(node, children)
 
-    def visitTryExcept(self, node):
-        children = ['try', ':', node.body]
-        for exception, name, body in node.handlers:
-            children.append('except')
-            if exception:
-                children.append(exception)
-            if name:
-                children.extend([',', name])
-            children.extend([':', body])
-        if node.else_:
-            children.extend(['else', ':', node.else_])
-        self._handle(node, children    )
+    def _excepthandler(self, node):
+        children = ['except']
+        if node.type:
+            children.append(node.type)
+        if node.name:
+            children.extend([',', node.name])
+        children.append(':')
+        children.extend(node.body)
+        self._handle(node, children)
 
-    def _handle_tuple(self, node):
-        if node.nodes:
-            self._handle(node, self._child_nodes(node.nodes, ','),
+    def _Tuple(self, node):
+        if node.elts:
+            self._handle(node, self._child_nodes(node.elts, ','),
                          eat_parens=True)
         else:
             self._handle(node, ['(', ')'])
 
-    def visitUnaryAdd(self, node):
-        self._handle(node, ['+', node.expr])
+    def _UnaryOp(self, node):
+        children = self._get_op(node.op)
+        children.append(node.operand)
+        self._handle(node, children)
 
-    def visitUnarySub(self, node):
-        self._handle(node, ['-', node.expr])
-
-    def visitYield(self, node):
+    def _Yield(self, node):
         children = ['yield']
         if node.value:
             children.append(node.value)
         self._handle(node, children)
 
-    def visitWhile(self, node):
-        children = ['while', node.test, ':', node.body]
-        if node.else_:
-            children.extend(['else', ':', node.else_])
+    def _While(self, node):
+        children = ['while', node.test, ':']
+        children.extend(node.body)
+        if node.orelse:
+            children.extend(['else', ':'])
+            children.extend(node.orelse)
         self._handle(node, children)
 
-    def visitWith(self, node):
-        children = ['with', node.expr]
-        if node.vars:
-            children.extend(['as', node.vars])
-        children.extend([':', node.body])
+    def _With(self, node):
+        children = ['with', node.context_expr]
+        if node.optional_vars:
+            children.extend(['as', node.optional_vars])
+        children.append(':')
+        children.extend(node.body)
         self._handle(node, children)
 
     def _child_nodes(self, nodes, separator):
     def consume_string(self):
         if _Source._string_pattern is None:
             original = codeanalyze.get_string_pattern()
-            pattern = r'(%s)((\s|\\\n|#[^\n]*\n)*(%s))*' % (original, original)
+            pattern = r'(%s)((\s|\\\n|#[^\n]*\n)*(%s))*' % \
+                      (original, original)
             _Source._string_pattern = re.compile(pattern)
         repattern = _Source._string_pattern
         return self._consume_pattern(repattern)
 
     def consume_not_equal(self):
         if _Source._not_equals_pattern is None:
-            _Source._not_equals_pattern = re.compile('<>|!=')
+            _Source._not_equals_pattern = re.compile(r'<>|!=')
         repattern = _Source._not_equals_pattern
         return self._consume_pattern(repattern)
 

rope/refactor/similarfinder.py

 """This module can be used for finding similar code"""
-import compiler.ast
 import re
 
-from rope.base import codeanalyze, evaluate, exceptions, pyobjects
+from rope.base import codeanalyze, evaluate, exceptions, pyobjects, ast
 from rope.refactor import patchedast, sourceutils, occurrences
 
 
     """A class for finding similar expressions and statements"""
 
     def __init__(self, source, start=0, end=None):
-        ast = compiler.parse(source)
-        self._init_using_ast(ast, source, start, end)
+        node = ast.parse(source)
+        self._init_using_ast(node, source, start, end)
 
-    def _init_using_ast(self, ast, source, start, end):
+    def _init_using_ast(self, node, source, start, end):
         self.start = start
         self.end = len(source)
         if end is not None:
             self.end = end
-        if not hasattr(ast, 'sorted_children'):
-            self.ast = patchedast.patch_ast(ast, source)
+        if not hasattr(node, 'sorted_children'):
+            self.ast = patchedast.patch_ast(node, source)
 
     def get_matches(self, code):
         """Search for `code` in source and return a list of `Match`\es
 
         `code` can contain wildcards.  ``${name}`` matches normal
         names and ``${?name} can match any expression.  They can
-        only appear in `compiler.ast.Name` and `compiler.ast.AssName`.
+        only appear in `_ast.Name`.
         You can use `Match.get_ast()` for getting the node that has
         matched a given pattern.
         """
 
     def _create_pattern(self, expression):
         expression = self._replace_wildcards(expression)
-        ast = compiler.parse(expression)
+        node = ast.parse(expression)
         # Getting Module.Stmt.nodes
-        nodes = ast.node.nodes
-        if len(nodes) == 1 and isinstance(nodes[0], compiler.ast.Discard):
+        nodes = node.body
+        if len(nodes) == 1 and isinstance(nodes[0], ast.Expr):
             # Getting Discard.expr
-            wanted = nodes[0].expr
+            wanted = nodes[0].value
         else:
             wanted = nodes
         return wanted
     """
 
     def __init__(self, pymodule, checks, start=0, end=None):
-        super(CheckingFinder, self)._init_using_ast(
-            pymodule.get_ast(), pymodule.source_code, start, end)
+        # XXX: was distabled to use the new ast
+        #        super(CheckingFinder, self)._init_using_ast(
+        #            pymodule.get_ast(), pymodule.source_code, start, end)
+        super(CheckingFinder, self).__init__(pymodule.source_code, start, end)
         self.pymodule = pymodule
         self.checks = checks
 
     def _evaluate_node(self, node):
         scope = self.pymodule.get_scope().get_inner_scope_for_line(node.lineno)
         expression = node
-        if isinstance(expression, compiler.ast.AssName):
+        if isinstance(expression, ast.Name) and \
+           isinstance(expression.ctx, ast.Store):
             start, end = patchedast.node_region(expression)
             text = self.pymodule.source_code[start:end]
             return evaluate.get_string_result(scope, text)
     def find_matches(self):
         if self.matches is None:
             self.matches = []
-            patchedast.call_for_nodes(self.body, self._check_node,
-                                      recursive=True)
+            ast.call_for_nodes(self.body, self._check_node, recursive=True)
         return self.matches
 
     def _check_node(self, node):
             self.matches.append(ExpressionMatch(node, mapping))
 
     def _check_statements(self, node):
-        if not isinstance(node, compiler.ast.Stmt):
-            return
-        for index in range(len(node.nodes)):
-            if len(node.nodes) - index >= len(self.pattern):
-                current_stmts = node.nodes[index:index + len(self.pattern)]
+        for child in ast.get_children(node):
+            if isinstance(child, (list, tuple)):
+                self.__check_stmt_list(child)
+
+    def __check_stmt_list(self, nodes):
+        for index in range(len(nodes)):
+            if len(nodes) - index >= len(self.pattern):
+                current_stmts = nodes[index:index + len(self.pattern)]
                 mapping = {}
                 if self._match_stmts(current_stmts, mapping):
                     self.matches.append(StatementMatch(current_stmts, mapping))
 
     def _match_nodes(self, expected, node, mapping):
-        if isinstance(expected, (compiler.ast.Name, compiler.ast.AssName)):
-           if self.ropevar.is_normal(expected.name):
+        if isinstance(expected, ast.Name):
+           if self.ropevar.is_normal(expected.id):
                return self._match_normal_var(expected, node, mapping)
-           if self.ropevar.is_any(expected.name):
+           if self.ropevar.is_any(expected.id):
                return self._match_any_var(expected, node, mapping)
+        if not isinstance(expected, ast.AST):
+            return expected == node
         if expected.__class__ != node.__class__:
             return False
 
-        children1 = expected.getChildren()
-        children2 = node.getChildren()
+        children1 = ast.get_children(expected)
+        children2 = ast.get_children(node)
         if len(children1) != len(children2):
             return False
         for child1, child2 in zip(children1, children2):
-            if isinstance(child1, compiler.ast.Node):
+            if isinstance(child1, ast.AST):
                 if not self._match_nodes(child1, child2, mapping):
                     return False
+            elif isinstance(child1, (list, tuple)):
+                if not isinstance(child2, (list, tuple)):
+                    return False
+                for c1, c2 in zip(child1, child2):
+                    if not self._match_nodes(c1, c2, mapping):
+                        return False
             else:
                 if child1 != child2:
                     return False
 
     def _match_normal_var(self, node1, node2, mapping):
         if node2.__class__ == node1.__class__ and \
-           self.ropevar.get_base(node1.name) == node2.name:
-            mapping[self.ropevar.get_base(node1.name)] = node2
+           self.ropevar.get_base(node1.id) == node2.id:
+            mapping[self.ropevar.get_base(node1.id)] = node2
             return True
         return False
 
     def _match_any_var(self, node1, node2, mapping):
-        name = self.ropevar.get_base(node1.name)
+        name = self.ropevar.get_base(node1.id)
         if name not in mapping:
             mapping[name] = node2
             return True

ropetest/refactor/patchedasttest.py

 import unittest
 
+from rope.base import ast
 from rope.refactor import patchedast
 from ropetest import testutils
 
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         start = source.index('10')
-        checker.check_region('Const(10)', start, start + 2)
+        checker.check_region('Num', start, start + 2)
 
     def test_integer_literals_and_sorted_children(self):
         source = 'a = 10\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         start = source.index('10')
-        checker.check_children('Const(10)', ['10'])
+        checker.check_children('Num', ['10'])
 
     def test_ass_name_node(self):
         source = 'a = 10\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         start = source.index('a')
-        checker.check_region('AssName', start, start + 1)
-        checker.check_children('AssName', ['a'])
+        checker.check_region('Name', start, start + 1)
+        checker.check_children('Name', ['a'])
 
     def test_assign_node(self):
         source = 'a = 10\n'
         start = source.index('a')
         checker.check_region('Assign', 0, len(source) - 1)
         checker.check_children(
-            'Assign', ['AssName', ' ', '=', ' ', 'Const'])
+            'Assign', ['Name', ' ', '=', ' ', 'Num'])
 
     def test_add_node(self):
         source = '1 + 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Add', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' ', 'Const(2)'])
+            'BinOp', ['Num', ' ', '+', ' ', 'Num'])
+
+    def test_lshift_node(self):
+        source = '1 << 2\n'
+        ast = patchedast.get_patched_ast(source, True)
+        checker = _ResultChecker(self, ast)
+        checker.check_region('BinOp', 0, len(source) - 1)
+        checker.check_children(
+            'BinOp', ['Num', ' ', '<<', ' ', 'Num'])
 
     def test_and_node(self):
         source = 'True and True\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('And', 0, len(source) - 1)
+        checker.check_region('BoolOp', 0, len(source) - 1)
         checker.check_children(
-            'And', ['Name', ' ', 'and', ' ', 'Name'])
+            'BoolOp', ['Name', ' ', 'and', ' ', 'Name'])
 
     def test_basic_closing_parens(self):
         source = '1 + (2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        two_start = source.index('2')
-        checker.check_region('Const(2)', two_start, two_start + 1)
-        checker.check_children('Const(2)', ['2'])
-        checker.check_region('Add', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' (', 'Const(2)', ')'])
+            'BinOp', ['Num', ' ', '+', ' (', 'Num', ')'])
 
     def test_basic_opening_parens(self):
         source = '(1) + 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Const(1)', 1, 2)
-        checker.check_children('Const(1)', ['1'])
-        checker.check_region('Add', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Add', ['(', 'Const(1)', ') ', '+', ' ', 'Const(2)'])
+            'BinOp', ['(', 'Num', ') ', '+', ' ', 'Num'])
 
     def test_basic_opening_biway(self):
         source = '(1) + (2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Add', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Add', ['(', 'Const(1)', ') ', '+', ' (', 'Const(2)', ')'])
+            'BinOp', ['(', 'Num', ') ', '+', ' (', 'Num', ')'])
 
     def test_basic_opening_double(self):
         source = '1 + ((2))\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Add', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' ((', 'Const(2)', '))'])
+            'BinOp', ['Num', ' ', '+', ' ((', 'Num', '))'])
 
     def test_handling_comments(self):
         source = '(1 + #(\n2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' #(\n', 'Const(2)'])
+            'BinOp', ['Num', ' ', '+', ' #(\n', 'Num'])
 
     def test_handling_parens_with_spaces(self):
         source = '1 + (2\n    )\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' (', 'Const(2)', '\n    )'])
+            'BinOp', ['Num', ' ', '+', ' (', 'Num', '\n    )'])
 
     def test_handling_strings(self):
         source = '1 + "("\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Add', ['Const(1)', ' ', '+', ' ', 'Const'])
+            'BinOp', ['Num', ' ', '+', ' ', 'Str'])
 
     def test_handling_implicit_string_concatenation(self):
         source = "a = '1''2'"
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Assign', ['AssName' , ' ', '=', ' ', "Const('12')"])
-        checker.check_children('Const', ["'1''2'"])
+            'Assign', ['Name' , ' ', '=', ' ', 'Str'])
+        checker.check_children('Str', ["'1''2'"])
 
     def test_handling_implicit_string_concatenation_line_breaks(self):
         source = "a = '1' \\\n'2'"
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Assign', ['AssName' , ' ', '=', ' ', "Const('12')"])
-        checker.check_children('Const', ["'1' \\\n'2'"])
+            'Assign', ['Name' , ' ', '=', ' ', 'Str'])
+        checker.check_children('Str', ["'1' \\\n'2'"])
+
+    def test_handling_explicit_string_concatenation_line_breaks(self):
+        source = "a = ('1' \n'2')"
+        ast = patchedast.get_patched_ast(source, True)
+        checker = _ResultChecker(self, ast)
+        checker.check_children(
+            'Assign', ['Name' , ' ', '=', ' (', 'Str', ')'])
+        checker.check_children('Str', ["'1' \n'2'"])
+
+    def test_not_cancatenating_strings_on_separate_lines(self):
+        source = "'1'\n'2'\n"
+        ast = patchedast.get_patched_ast(source, True)
+        checker = _ResultChecker(self, ast)
 
     def test_long_integer_literals(self):
         source = "0x1L + a"
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Add', ['Const' , ' ', '+', ' ', 'Name'])
-        checker.check_children('Const', ['0x1L'])
+            'BinOp', ['Num' , ' ', '+', ' ', 'Name'])
+        checker.check_children('Num', ['0x1L'])
 
     def test_complex_number_literals(self):
         source = "1.0e2j + a"
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'Add', ['Const' , ' ', '+', ' ', 'Name'])
-        checker.check_children('Const', ['1.0e2j'])
+            'BinOp', ['Num' , ' ', '+', ' ', 'Name'])
+        checker.check_children('Num', ['1.0e2j'])
 
     def test_ass_attr_node(self):
         source = 'a.b = 1\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('AssAttr', 0, source.index('=') - 1)
-        checker.check_children('AssAttr', ['Name', '', '.', '', 'b'])
+        checker.check_region('Attribute', 0, source.index('=') - 1)
+        checker.check_children('Attribute', ['Name', '', '.', '', 'b'])
 
     def test_ass_list_node(self):
         source = '[a, b] = 1, 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('AssList', 0, source.index(']') + 1)
-        checker.check_children('AssList', ['[', '', 'AssName', '', ',',
-                                           ' ', 'AssName', '', ']'])
+        checker.check_region('List', 0, source.index(']') + 1)
+        checker.check_children('List', ['[', '', 'Name', '', ',',
+                                        ' ', 'Name', '', ']'])
 
     def test_ass_tuple(self):
-        source = 'a, b = 1, 2\n'
+        source = 'a, b = range(2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('AssTuple', 0, source.index('=') - 1)
+        checker.check_region('Tuple', 0, source.index('=') - 1)
         checker.check_children(
-            'AssTuple', ['AssName', '', ',', ' ', 'AssName'])
+            'Tuple', ['Name', '', ',', ' ', 'Name'])
 
     def test_ass_tuple2(self):
-        source = '(a, b) = 1, 2\n'
+        source = '(a, b) = range(2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('AssTuple', 0, source.index('=') - 1)
+        checker.check_region('Tuple', 0, source.index('=') - 1)
         checker.check_children(
-            'AssTuple', ['(', '', 'AssName', '', ',', ' ', 'AssName', '', ')'])
+            'Tuple', ['(', '', 'Name', '', ',', ' ', 'Name', '', ')'])
 
     def test_assert(self):
         source = 'assert True\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Assert', 0, len(source) - 1)
         checker.check_children(
-            'Assert', ['assert', ' ', 'Name', '', ',', ' ', 'Const'])
+            'Assert', ['assert', ' ', 'Name', '', ',', ' ', 'Str'])
 
     def test_aug_assign_node(self):
         source = 'a += 1\n'
         start = source.index('a')
         checker.check_region('AugAssign', 0, len(source) - 1)
         checker.check_children(
-            'AugAssign', ['Name', ' ', '+=', ' ', 'Const'])
+            'AugAssign', ['Name', ' ', '+', '', '=', ' ', 'Num'])
 
     def test_back_quotenode(self):
         source = '`1`\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Backquote', 0, len(source) - 1)
+        checker.check_region('Repr', 0, len(source) - 1)
         checker.check_children(
-            'Backquote', ['`', '', 'Const(1)', '', '`'])
+            'Repr', ['`', '', 'Num', '', '`'])
 
     def test_bitand(self):
         source = '1 & 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Bitand', 0, len(source) - 1)
+        checker.check_region('BinOp', 0, len(source) - 1)
         checker.check_children(
-            'Bitand', ['Const(1)', ' ', '&', ' ', 'Const(2)'])
+            'BinOp', ['Num', ' ', '&', ' ', 'Num'])
 
     def test_bitor(self):
         source = '1 | 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Bitor', 0, len(source) - 1)
         checker.check_children(
-            'Bitor', ['Const(1)', ' ', '|', ' ', 'Const(2)'])
+            'BinOp', ['Num', ' ', '|', ' ', 'Num'])
 
     def test_call_func(self):
         source = 'f(1, 2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('CallFunc', 0, len(source) - 1)
+        checker.check_region('Call', 0, len(source) - 1)
         checker.check_children(
-            'CallFunc', ['Name', '', '(', '', 'Const(1)', '', ',',
-                         ' ', 'Const(2)', '', ')'])
+            'Call', ['Name', '', '(', '', 'Num', '', ',',
+                     ' ', 'Num', '', ')'])
 
     def test_call_func_and_keywords(self):
         source = 'f(1, p=2)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'CallFunc', ['Name', '', '(', '', 'Const(1)', '', ',',
-                         ' ', 'Keyword', '', ')'])
+            'Call', ['Name', '', '(', '', 'Num', '', ',',
+                     ' ', 'keyword', '', ')'])
 
     def test_call_func_and_start_args(self):
         source = 'f(1, *args)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'CallFunc', ['Name', '', '(', '', 'Const(1)', '', ',',
-                         ' ', '*', '', 'Name', '', ')'])
+            'Call', ['Name', '', '(', '', 'Num', '', ',',
+                     ' ', '*', '', 'Name', '', ')'])
 
     def test_call_func_and_only_dstart_args(self):
         source = 'f(**kwds)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'CallFunc', ['Name', '', '(', '', '**', '', 'Name', '', ')'])
+            'Call', ['Name', '', '(', '', '**', '', 'Name', '', ')'])
 
     def test_class_node(self):
         source = 'class A(object):\n    """class docs"""\n    pass\n'
         checker.check_region('Class', 0, len(source) - 1)
         checker.check_children(
             'Class', ['class', ' ', 'A', '', '(', '', 'Name', '', ')',
-                      '', ':', '\n    ', '"""class docs"""', '\n    ', 'Stmt'])
+                      '', ':', '\n    ', 'Expr', '\n    ', 'Pass'])
 
     def test_class_with_no_bases(self):
         source = 'class A:\n    pass\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Class', 0, len(source) - 1)
         checker.check_children(
-            'Class', ['class', ' ', 'A', '', ':', '\n    ', 'Stmt'])
+            'Class', ['class', ' ', 'A', '', ':', '\n    ', 'Pass'])
 
     def test_simple_compare(self):
         source = '1 < 2\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Compare', 0, len(source) - 1)
         checker.check_children(
-            'Compare', ['Const(1)', ' ', '<', ' ', 'Const(2)'])
+            'Compare', ['Num', ' ', '<', ' ', 'Num'])
 
     def test_multiple_compare(self):
         source = '1 < 2 <= 3\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Compare', 0, len(source) - 1)
         checker.check_children(
-            'Compare', ['Const(1)', ' ', '<', ' ', 'Const(2)', ' ',
-                        '<=', ' ', 'Const(3)'])
+            'Compare', ['Num', ' ', '<', ' ', 'Num', ' ',
+                        '<=', ' ', 'Num'])
 
     def test_decorators_node(self):
         source = '@d\ndef f():\n    pass\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Decorator', 0, 2)
-        checker.check_children('Decorator', ['@', '', 'Name'])
+        checker.check_region('FunctionDef', 0, len(source) - 1)
+        checker.check_children(
+            'FunctionDef',
+            ['@', '', 'Name', '\n', 'def', ' ', 'f', '', '(', '', 'arguments',
+             '', ')', '', ':', '\n    ', 'Pass'])
 
     def test_function_node(self):
         source = 'def f():\n    pass\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_region('Function', 0, len(source) - 1)
-        checker.check_children('Function', ['def', ' ', 'f', '', '(', '',
-                                            ')', '', ':', '\n    ', 'Stmt'])
+        checker.check_children('Function', ['def', ' ', 'f', '', '(', '', 'arguments', '',
+                                            ')', '', ':', '\n    ', 'Pass'])
 
     def test_function_node2(self):
         source = 'def f(p1, **p2):\n    """docs"""\n    pass\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Function', 0, len(source) - 1)
         checker.check_children(
-            'Function', ['def', ' ', 'f', '', '(', '', 'p1', '', ',',
-                         ' ', '**', '', 'p2', '', ')', '', ':', '\n    ',
-                         '"""docs"""', '\n    ', 'Stmt'])
+            'Function', ['def', ' ', 'f', '', '(', '', 'arguments',
+                         '', ')' , '', ':', '\n    ', 'Expr', '\n    ', 'Pass'])
+        checker.check_children(
+            'arguments', ['Name', '', ',',
+                          ' ', '**', '', 'p2'])
 
     def test_function_node_and_tuple_parameters(self):
         source = 'def f(a, (b, c)):\n    pass\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Function', 0, len(source) - 1)
         checker.check_children(
-            'Function', ['def', ' ', 'f', '', '(', '', 'a', '', ',', ' ', '(',
-                         '', 'b', '', ',', ' ', 'c', '', ')', '', ')' , '',
-                         ':', '\n    ', 'Stmt'])
+            'Function', ['def', ' ', 'f', '', '(', '', 'arguments',
+                         '', ')' , '', ':', '\n    ', 'Pass'])
+        checker.check_children(
+            'arguments', ['Name', '', ',', ' ', 'Tuple'])
 
     def test_dict_node(self):
         source = '{1: 2, 3: 4}\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Dict', 0, len(source) - 1)
         checker.check_children(
-            'Dict', ['{', '', 'Const(1)', '', ':', ' ', 'Const(2)', '', ',',
-                     ' ', 'Const(3)', '', ':', ' ', 'Const(4)', '', '}'])
+            'Dict', ['{', '', 'Num', '', ':', ' ', 'Num', '', ',',
+                     ' ', 'Num', '', ':', ' ', 'Num', '', '}'])
 
     def test_div_node(self):
         source = '1 / 2\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Div', 0, len(source) - 1)
-        checker.check_children('Div', ['Const(1)', ' ', '/', ' ', 'Const(2)'])
+        checker.check_region('BinOp', 0, len(source) - 1)
+        checker.check_children('BinOp', ['Num', ' ', '/', ' ', 'Num'])
 
     def test_simple_exec_node(self):
         source = 'exec ""\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_region('Exec', 0, len(source) - 1)
-        checker.check_children('Exec', ['exec', ' ', 'Const'])
+        checker.check_children('Exec', ['exec', ' ', 'Str'])
 
     def test_exec_node(self):
         source = 'exec "" in locals(), globals()\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Exec', 0, len(source) - 1)
         checker.check_children(
-            'Exec', ['exec', ' ', 'Const', ' ', 'in',
-                     ' ', 'CallFunc', '', ',', ' ', 'CallFunc'])
+            'Exec', ['exec', ' ', 'Str', ' ', 'in',
+                     ' ', 'Call', '', ',', ' ', 'Call'])
 
     def test_for_node(self):
         source = 'for i in range(1):\n    pass\nelse:\n    pass\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('For', 0, len(source) - 1)
         checker.check_children(
-            'For', ['for', ' ', 'AssName', ' ', 'in', ' ', 'CallFunc', '',
-                    ':', '\n    ', 'Stmt', '\n',
-                    'else', '', ':', '\n    ', 'Stmt'])
+            'For', ['for', ' ', 'Name', ' ', 'in', ' ', 'Call', '',
+                    ':', '\n    ', 'Pass', '\n',
+                    'else', '', ':', '\n    ', 'Pass'])
 
     def test_normal_from_node(self):
         source = 'from x import y\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('From', 0, len(source) - 1)
+        checker.check_region('ImportFrom', 0, len(source) - 1)
         checker.check_children(
-            'From', ['from', ' ', 'x', ' ', 'import', ' ', 'y'])
+            'ImportFrom', ['from', ' ', 'x', ' ', 'import', ' ', 'alias'])
+        checker.check_children('alias', ['y'])
 
     @testutils.run_only_for_25
     def test_from_node(self):
         source = 'from ..x import y as z\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('From', 0, len(source) - 1)
+        checker.check_region('ImportFrom', 0, len(source) - 1)
         checker.check_children(
-            'From', ['from', ' ', '..', '', 'x', ' ', 'import', ' ', 'y',
-                     ' ', 'as', ' ', 'z'])
+            'ImportFrom', ['from', ' ', '..', '', 'x', ' ',
+                           'import', ' ', 'alias'])
+        checker.check_children('alias', ['y', ' ', 'as', ' ', 'z'])
 
     def test_simple_gen_expr_node(self):
         source = 'zip(i for i in x)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('GenExpr', 4, len(source) - 2)
+        checker.check_region('GeneratorExp', 4, len(source) - 2)
         checker.check_children(
-            'GenExprInner', ['Name', ' ', 'GenExprFor'])
+            'GeneratorExp', ['Name', ' ', 'comprehension'])
         checker.check_children(
-            'GenExprFor', ['for', ' ', 'AssName', ' ', 'in', ' ', 'Name'])
+            'comprehension', ['for', ' ', 'Name', ' ', 'in', ' ', 'Name'])
 
     def test_gen_expr_node_handling_surrounding_parens(self):
         source = '(i for i in x)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('GenExpr', 0, len(source) - 1)
+        checker.check_region('GeneratorExp', 0, len(source) - 1)
         checker.check_children(
-            'GenExpr', ['(', '', 'GenExprInner', '', ')'])
+            'GeneratorExp', ['(', '', 'Name', ' ', 'comprehension', '', ')'])
 
     def test_gen_expr_node2(self):
         source = 'zip(i for i in range(1) if i == 1)\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_children(
-            'GenExprFor', ['for', ' ', 'AssName', ' ', 'in', ' ', 'CallFunc',
-                           ' ', 'GenExprIf'])
-        checker.check_children('GenExprIf', ['if', ' ', 'Compare'])
+            'comprehension', ['for', ' ', 'Name', ' ', 'in', ' ', 'Call',
+                              ' ', 'if', ' ', 'Compare'])
 
     def test_get_attr_node(self):
         source = 'a.b\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_region('Getattr', 0, len(source) - 1)
-        checker.check_children('Getattr', ['Name', '', '.', '', 'b'])
+        checker.check_region('Attribute', 0, len(source) - 1)
+        checker.check_children('Attribute', ['Name', '', '.', '', 'b'])
 
     def test_global_node(self):
         source = 'global a, b\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('If', 0, len(source) - 1)
         checker.check_children(
-            'If', ['if', ' ', 'Name', '', ':', '\n    ', 'Stmt', '\n',
-                   'else', '', ':', '\n    ', 'Stmt'])
+            'If', ['if', ' ', 'Name', '', ':', '\n    ', 'Pass', '\n',
+                   'else', '', ':', '\n    ', 'Pass'])
 
     def test_if_node2(self):
         source = 'if True:\n    pass\nelif False:\n    pass\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('If', 0, len(source) - 1)
         checker.check_children(
-            'If', ['if', ' ', 'Name', '', ':', '\n    ', 'Stmt', '\n',
-                   'elif', ' ', 'Name', '', ':', '\n    ', 'Stmt'])
+            'If', ['if', ' ', 'Name', '', ':', '\n    ', 'Pass', '\n',
+                   'If'])
+
+    def test_if_node3(self):
+        source = 'if True:\n    pass\nelse:\n' \
+                 '    if True:\n        pass\n'
+        ast = patchedast.get_patched_ast(source, True)
+        checker = _ResultChecker(self, ast)
+        checker.check_region('If', 0, len(source) - 1)
+        checker.check_children(
+            'If', ['if', ' ', 'Name', '', ':', '\n    ', 'Pass', '\n',
+                   'else', '', ':', '\n    ', 'If'])
 
     def test_import_node(self):
         source = 'import a, b as c\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Import', 0, len(source) - 1)
         checker.check_children(
-            'Import', ['import', ' ', 'a', '', ',', ' ', 'b', ' ',
-                       'as', ' ', 'c'])
+            'Import', ['import', ' ', 'alias', '', ',', ' ', 'alias'])
 
     def test_lambda_node(self):
         source = 'lambda a, b=1, *z: None\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Lambda', 0, len(source) - 1)
         checker.check_children(
-            'Lambda', ['lambda', ' ', 'a', '', ',', ' ', 'b', '', '=', '',
-                       'Const(1)', '', ',', ' ', '*', '', 'z', '', ':',
-                       ' ', 'Name'])
+            'Lambda', ['lambda', ' ', 'arguments', '', ':', ' ', 'Name'])
+        checker.check_children(
+            'arguments', ['Name', '', ',', ' ', 'Name', '', '=', '',
+                          'Num', '', ',', ' ', '*', '', 'z'])
 
     def test_list_node(self):
         source = '[1, 2]\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('List', 0, len(source) - 1)
         checker.check_children(
-            'List', ['[', '', 'Const(1)', '', ',', ' ', 'Const(2)', '', ']'])
+            'List', ['[', '', 'Num', '', ',', ' ', 'Num', '', ']'])
 
     def test_list_comp_node(self):
         source = '[i for i in range(1) if True]\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('ListComp', 0, len(source) - 1)
         checker.check_children(
-            'ListComp', ['[', '', 'Name', ' ', 'ListCompFor', '', ']'])
+            'ListComp', ['[', '', 'Name', ' ', 'comprehension', '', ']'])
         checker.check_children(
-            'ListCompFor', ['for', ' ', 'AssName', ' ', 'in', ' ',
-                            'CallFunc', ' ', 'ListCompIf'])
-        checker.check_children('ListCompIf', ['if', ' ', 'Name'])
+            'comprehension', ['for', ' ', 'Name', ' ', 'in', ' ',
+                              'Call', ' ', 'if', ' ', 'Name'])
 
     def test_simple_module_node(self):
         source = 'pass\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_region('Module', 0, len(source))
-        checker.check_children('Module', ['', 'Stmt', '\n'])
+        checker.check_children('Module', ['', 'Pass', '\n'])
 
     def test_module_node(self):
         source = '"""docs"""\npass\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
         checker.check_region('Module', 0, len(source))
-        checker.check_children('Module', ['', '"""docs"""', '\n', 'Stmt', '\n'])
+        checker.check_children('Module', ['', 'Expr', '\n', 'Pass', '\n'])
+        checker.check_children('Str', ['"""docs"""'])
 
     def test_not_and_or_nodes(self):
         source = 'not True or False\n'
         ast = patchedast.get_patched_ast(source, True)
         checker = _ResultChecker(self, ast)
-        checker.check_children('Discard', ['Or'])
-        checker.check_children('Or', ['Not', ' ', 'or', ' ', 'Name'])
+        checker.check_children('Expr', ['BoolOp'])
+        checker.check_children('BoolOp', ['UnaryOp', ' ', 'or', ' ', 'Name'])
 
     def test_print_node(self):
         source = 'print >>out, 1,\n'
         checker = _ResultChecker(self, ast)
         checker.check_region('Print', 0, len(source) - 1)
         checker.check_children('Print', ['print', ' ', '>>', '', 'Name', '',
-                                         ',', ' ', 'Const(1)', '', ','])
</