Commits

Armin Rigo  committed a5b0f97 Merge

merge heads

  • Participants
  • Parent commits 4c7b26f, 8315532

Comments (0)

Files changed (54)

File pypy/doc/jit/index.rst

 
 - Hooks_ debugging facilities available to a python programmer
 
+- Virtualizable_ how virtualizables work and what they are (in other words how
+  to make frames more efficient).
 
 .. _Overview: overview.html
 .. _Notes: pyjitpl5.html
 .. _Hooks: ../jit-hooks.html
+.. _Virtualizable: virtualizable.html

File pypy/doc/jit/virtualizable.rst

+
+Virtualizables
+==============
+
+**Note:** this document does not have a proper introduction as to how
+to understand the basics. We should write some. If you happen to be here
+and you're missing context, feel free to pester us on IRC.
+
+Problem description
+-------------------
+
+The JIT is very good at making sure some objects are never allocated if they
+don't escape from the trace. Such objects are called ``virtuals``. However,
+if we're dealing with frames, virtuals are often not good enough. Frames
+can escape and they can also be allocated already at the moment we enter the
+JIT. In such cases we need some extra object that can still be optimized away,
+despite existing on the heap.
+
+Solution
+--------
+
+We introduce virtualizables. They're objects that exist on the heap, but their
+fields are not always in sync with whatever happens in the assembler. One
+example is that virtualizable fields can store virtual objects without
+forcing them. This is very useful for frames. Declaring an object to be
+virtualizable works like this:
+
+    class Frame(object):
+       _virtualizable_ = ['locals[*]', 'stackdepth']
+
+And we use them in ``JitDriver`` like this::
+
+    jitdriver = JitDriver(greens=[], reds=['frame'], virtualizables=['frame'])
+
+This declaration means that ``stackdepth`` is a virtualizable **field**, while
+``locals`` is a virtualizable **array** (a list stored on a virtualizable).
+There are various rules about using virtualizables, especially using
+virtualizable arrays that can be very confusing. Those will usually end
+up with a compile-time error (as opposed to strange behavior). The rules are:
+
+* Each array access must be with a known positive index that cannot raise
+  an ``IndexError``. Using ``no = jit.hint(no, promote=True)`` might be useful
+  to get a constant-number access. This is only safe if the index is actually
+  constant or changing rarely within the context of the user's code.
+
+* If you initialize a new virtualizable in the JIT, it has to be done like this
+  (for example if we're in ``Frame.__init__``)::
+
+    self = hint(self, access_directly=True, fresh_virtualizable=True)
+
+  that way you can populate the fields directly.
+
+* If you use virtualizable outside of the JIT – it's very expensive and
+  sometimes aborts tracing. Consider it carefully as to how do it only for
+  debugging purposes and not every time (e.g. ``sys._getframe`` call).
+
+* If you have something equivalent of a Python generator, where the
+  virtualizable survives for longer, you want to force it before returning.
+  It's better to do it that way than by an external call some time later.
+  It's done using ``jit.hint(frame, force_virtualizable=True)``

File pypy/doc/whatsnew-head.rst

 .. branch: kill-typesystem
 Remove the "type system" abstraction, now that there is only ever one kind of
 type system used.
+
+.. branch: kill-gen-store-back-in
+Kills gen_store_back_in_virtualizable - should improve non-inlined calls by
+a bit
+
+.. branch: dotviewer-linewidth
+.. branch: reflex-support

File pypy/interpreter/generator.py

File contents unchanged.

File pypy/interpreter/pyframe.py

         # CO_OPTIMIZED: no locals dict needed at all
         # NB: this method is overridden in nestedscope.py
         flags = code.co_flags
-        if flags & pycode.CO_OPTIMIZED: 
-            return 
+        if flags & pycode.CO_OPTIMIZED:
+            return
         if flags & pycode.CO_NEWLOCALS:
             self.w_locals = self.space.newdict(module=True)
         else:
                 executioncontext.return_trace(self, self.space.w_None)
                 raise
             executioncontext.return_trace(self, w_exitvalue)
-            # clean up the exception, might be useful for not
-            # allocating exception objects in some cases
-            self.last_exception = None
+            # it used to say self.last_exception = None
+            # this is now done by the code in pypyjit module
+            # since we don't want to invalidate the virtualizable
+            # for no good reason
             got_exception = False
         finally:
             executioncontext.leave(self, w_exitvalue, got_exception)
                 break
             w_value = self.peekvalue(delta)
             self.pushvalue(w_value)
-        
+
     def peekvalue(self, index_from_top=0):
         # NOTE: top of the stack is peekvalue(0).
         # Contrast this with CPython where it's PEEK(-1).
         nlocals = self.pycode.co_nlocals
         values_w = self.locals_stack_w[nlocals:self.valuestackdepth]
         w_valuestack = maker.slp_into_tuple_with_nulls(space, values_w)
-        
+
         w_blockstack = nt([block._get_state_(space) for block in self.get_blocklist()])
         w_fastlocals = maker.slp_into_tuple_with_nulls(
             space, self.locals_stack_w[:nlocals])
         else:
             w_exc_value = self.last_exception.get_w_value(space)
             w_tb = w(self.last_exception.get_traceback())
-        
+
         tup_state = [
             w(self.f_backref()),
             w(self.get_builtin()),
             w(f_lineno),
             w_fastlocals,
             space.w_None,           #XXX placeholder for f_locals
-            
+
             #f_restricted requires no additional data!
             space.w_None, ## self.w_f_trace,  ignore for now
 
             ncellvars = len(pycode.co_cellvars)
             cellvars = cells[:ncellvars]
             closure = cells[ncellvars:]
-        
+
         # do not use the instance's __init__ but the base's, because we set
         # everything like cells from here
         # XXX hack
 
     ### line numbers ###
 
-    def fget_f_lineno(self, space): 
+    def fget_f_lineno(self, space):
         "Returns the line number of the instruction currently being executed."
         if self.w_f_trace is None:
             return space.wrap(self.get_last_lineno())
         except OperationError, e:
             raise OperationError(space.w_ValueError,
                                  space.wrap("lineno must be an integer"))
-            
+
         if self.w_f_trace is None:
             raise OperationError(space.w_ValueError,
                   space.wrap("f_lineno can only be set by a trace function."))
         if ord(code[new_lasti]) in (DUP_TOP, POP_TOP):
             raise OperationError(space.w_ValueError,
                   space.wrap("can't jump to 'except' line as there's no exception"))
-            
+
         # Don't jump into or out of a finally block.
         f_lasti_setup_addr = -1
         new_lasti_setup_addr = -1
                         if addr == self.last_instr:
                             f_lasti_setup_addr = setup_addr
                         break
-                    
+
             if op >= HAVE_ARGUMENT:
                 addr += 3
             else:
                 addr += 1
-                
+
         assert len(blockstack) == 0
 
         if new_lasti_setup_addr != f_lasti_setup_addr:
             block = self.pop_block()
             block.cleanup(self)
             f_iblock -= 1
-            
+
         self.f_lineno = new_lineno
         self.last_instr = new_lasti
-            
+
     def get_last_lineno(self):
         "Returns the line number of the instruction currently being executed."
         return pytraceback.offset2lineno(self.pycode, self.last_instr)
             self.f_lineno = self.get_last_lineno()
             space.frame_trace_action.fire()
 
-    def fdel_f_trace(self, space): 
-        self.w_f_trace = None 
+    def fdel_f_trace(self, space):
+        self.w_f_trace = None
 
     def fget_f_exc_type(self, space):
         if self.last_exception is not None:
             if f is not None:
                 return f.last_exception.w_type
         return space.w_None
-         
+
     def fget_f_exc_value(self, space):
         if self.last_exception is not None:
             f = self.f_backref()
             if f is not None:
                 return space.wrap(f.last_exception.get_traceback())
         return space.w_None
-         
+
     def fget_f_restricted(self, space):
         if space.config.objspace.honor__builtins__:
             return space.wrap(self.builtin is not space.builtin)

File pypy/module/pypyjit/interp_jit.py

 from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.interpreter.pycode import PyCode, CO_GENERATOR
 from pypy.interpreter.pyframe import PyFrame
-from pypy.interpreter.pyopcode import ExitFrame
+from pypy.interpreter.pyopcode import ExitFrame, Yield
 from opcode import opmap
 
-PyFrame._virtualizable2_ = ['last_instr', 'pycode',
-                            'valuestackdepth', 'locals_stack_w[*]',
-                            'cells[*]',
-                            'last_exception',
-                            'lastblock',
-                            'is_being_profiled',
-                            'w_globals',
-                            'w_f_trace',
-                            ]
+PyFrame._virtualizable_ = ['last_instr', 'pycode',
+                           'valuestackdepth', 'locals_stack_w[*]',
+                           'cells[*]',
+                           'last_exception',
+                           'lastblock',
+                           'is_being_profiled',
+                           'w_globals',
+                           'w_f_trace',
+                           ]
 
 JUMP_ABSOLUTE = opmap['JUMP_ABSOLUTE']
 
                 self.valuestackdepth = hint(self.valuestackdepth, promote=True)
                 next_instr = self.handle_bytecode(co_code, next_instr, ec)
                 is_being_profiled = self.is_being_profiled
+        except Yield:
+            self.last_exception = None
+            w_result = self.popvalue()
+            jit.hint(self, force_virtualizable=True)
+            return w_result
         except ExitFrame:
+            self.last_exception = None
             return self.popvalue()
 
     def jump_absolute(self, jumpto, ec):

File pypy/module/pypyjit/test_pypy_c/test_containers.py

             i8 = int_lt(i5, i7)
             guard_true(i8, descr=...)
             guard_not_invalidated(descr=...)
-            p10 = call(ConstClass(ll_int_str), i5, descr=<Callr . i EF=3>)
+            p10 = call(ConstClass(ll_str__IntegerR_SignedConst_Signed), i5, descr=<Callr . i EF=3>)
             guard_no_exception(descr=...)
             i12 = call(ConstClass(ll_strhash), p10, descr=<Calli . r EF=0>)
             p13 = new(descr=...)

File pypy/module/pypyjit/test_pypy_c/test_generators.py

         log = self.run(main, [500])
         loop, = log.loops_by_filename(self.filepath)
         assert loop.match_by_id("generator", """
+            cond_call(..., descr=...)
             i16 = force_token()
             p45 = new_with_vtable(ConstClass(W_IntObject))
             setfield_gc(p45, i29, descr=<FieldS .*>)

File pypy/tool/pypyjit.py

File contents unchanged.

File pypy/tool/pypyjit_child.py

         return lltype.nullptr(T)
     interp.heap.malloc_nonmovable = returns_null     # XXX
 
-    from rpython.jit.backend.llgraph.runner import LLtypeCPU
+    from rpython.jit.backend.llgraph.runner import LLGraphCPU
     #LLtypeCPU.supports_floats = False     # for now
-    apply_jit(interp, graph, LLtypeCPU)
+    apply_jit(interp, graph, LLGraphCPU)
 
 
 def apply_jit(interp, graph, CPUClass):

File rpython/annotator/test/test_annrpython.py

         from rpython.rlib.jit import hint
 
         class A:
-            _virtualizable2_ = []
+            _virtualizable_ = []
         class B(A):
             def meth(self):
                 return self
         from rpython.rlib.jit import hint
 
         class A:
-            _virtualizable2_ = []
+            _virtualizable_ = []
 
         class I:
             pass

File rpython/jit/backend/arm/opassembler.py

         pmc.B_offs(self.mc.currpos(), c.EQ)
         return pos
 
-    def _call_assembler_reset_vtoken(self, jd, vloc):
-        from rpython.jit.backend.llsupport.descr import FieldDescr
-        fielddescr = jd.vable_token_descr
-        assert isinstance(fielddescr, FieldDescr)
-        ofs = fielddescr.offset
-        tmploc = self._regalloc.get_scratch_reg(INT)
-        self.mov_loc_loc(vloc, r.ip)
-        self.mc.MOV_ri(tmploc.value, 0)
-        self.mc.STR_ri(tmploc.value, r.ip.value, ofs)
-
     def _call_assembler_load_result(self, op, result_loc):
         if op.result is not None:
             # load the return value from (tmploc, 0)

File rpython/jit/backend/llgraph/runner.py

         self.fieldname = fieldname
         self.FIELD = getattr(S, fieldname)
 
+    def get_vinfo(self):
+        return self.vinfo
+
     def __repr__(self):
         return 'FieldDescr(%r, %r)' % (self.S, self.fieldname)
 
     translate_support_code = False
     is_llgraph = True
 
-    def __init__(self, rtyper, stats=None, *ignored_args, **ignored_kwds):
+    def __init__(self, rtyper, stats=None, *ignored_args, **kwds):
         model.AbstractCPU.__init__(self)
         self.rtyper = rtyper
         self.llinterp = LLInterpreter(rtyper)
         class MiniStats:
             pass
         self.stats = stats or MiniStats()
+        self.vinfo_for_tests = kwds.get('vinfo_for_tests', None)
 
     def compile_loop(self, inputargs, operations, looptoken, log=True, name=''):
         clt = model.CompiledLoopToken(self, looptoken.number)
         except KeyError:
             descr = FieldDescr(S, fieldname)
             self.descrs[key] = descr
+            if self.vinfo_for_tests is not None:
+                descr.vinfo = self.vinfo_for_tests
             return descr
 
     def arraydescrof(self, A):
     forced_deadframe = None
     overflow_flag = False
     last_exception = None
+    force_guard_op = None
 
     def __init__(self, cpu, argboxes, args):
         self.env = {}
         if self.forced_deadframe is not None:
             saved_data = self.forced_deadframe._saved_data
             self.fail_guard(descr, saved_data)
+        self.force_guard_op = self.current_op
+    execute_guard_not_forced_2 = execute_guard_not_forced
 
     def execute_guard_not_invalidated(self, descr):
         if self.lltrace.invalid:
         #     res = CALL assembler_call_helper(pframe)
         #     jmp @done
         #   @fastpath:
-        #     RESET_VABLE
         #     res = GETFIELD(pframe, 'result')
         #   @done:
         #
             vable = lltype.nullptr(llmemory.GCREF.TO)
         #
         # Emulate the fast path
-        def reset_vable(jd, vable):
-            if jd.index_of_virtualizable != -1:
-                fielddescr = jd.vable_token_descr
-                NULL = lltype.nullptr(llmemory.GCREF.TO)
-                self.cpu.bh_setfield_gc(vable, NULL, fielddescr)
+        #
         faildescr = self.cpu.get_latest_descr(pframe)
         if faildescr == self.cpu.done_with_this_frame_descr_int:
-            reset_vable(jd, vable)
             return self.cpu.get_int_value(pframe, 0)
         elif faildescr == self.cpu.done_with_this_frame_descr_ref:
-            reset_vable(jd, vable)
             return self.cpu.get_ref_value(pframe, 0)
         elif faildescr == self.cpu.done_with_this_frame_descr_float:
-            reset_vable(jd, vable)
             return self.cpu.get_float_value(pframe, 0)
         elif faildescr == self.cpu.done_with_this_frame_descr_void:
-            reset_vable(jd, vable)
             return None
-        #
+
         assembler_helper_ptr = jd.assembler_helper_adr.ptr  # fish
         try:
             result = assembler_helper_ptr(pframe, vable)

File rpython/jit/backend/llsupport/assembler.py

 from rpython.jit.backend.llsupport.memcpy import memcpy_fn
 from rpython.jit.backend.llsupport.symbolic import WORD
 from rpython.jit.metainterp.history import (INT, REF, FLOAT, JitCellToken,
-    ConstInt, BoxInt)
+    ConstInt, BoxInt, AbstractFailDescr)
 from rpython.jit.metainterp.resoperation import ResOperation, rop
 from rpython.rlib import rgc
 from rpython.rlib.debug import (debug_start, debug_stop, have_debug_prints,
 class GuardToken(object):
     def __init__(self, cpu, gcmap, faildescr, failargs, fail_locs, exc,
                  frame_depth, is_guard_not_invalidated, is_guard_not_forced):
+        assert isinstance(faildescr, AbstractFailDescr)
         self.cpu = cpu
         self.faildescr = faildescr
         self.failargs = failargs
 
         jmp_location = self._call_assembler_patch_je(result_loc, je_location)
 
-        # Path B: fast path.  Must load the return value, and reset the token
+        # Path B: fast path.  Must load the return value
 
-        # Reset the vable token --- XXX really too much special logic here:-(
-        if jd.index_of_virtualizable >= 0:
-            self._call_assembler_reset_vtoken(jd, vloc)
         #
         self._call_assembler_load_result(op, result_loc)
         #

File rpython/jit/backend/llsupport/descr.py

 FLAG_STRUCT   = 'X'
 FLAG_VOID     = 'V'
 
-class FieldDescr(AbstractDescr):
+class ArrayOrFieldDescr(AbstractDescr):
+    vinfo = None
+
+    def get_vinfo(self):
+        return self.vinfo
+
+class FieldDescr(ArrayOrFieldDescr):
     name = ''
     offset = 0      # help translation
     field_size = 0
 # ____________________________________________________________
 # ArrayDescrs
 
-class ArrayDescr(AbstractDescr):
+class ArrayDescr(ArrayOrFieldDescr):
     tid = 0
     basesize = 0       # workaround for the annotator
     itemsize = 0
     lendescr = None
     flag = '\x00'
+    vinfo = None
 
     def __init__(self, basesize, itemsize, lendescr, flag):
         self.basesize = basesize

File rpython/jit/backend/llsupport/test/ztranslation_test.py

         # - floats neg and abs
 
         class Frame(object):
-            _virtualizable2_ = ['i']
+            _virtualizable_ = ['i']
 
             def __init__(self, i):
                 self.i = i
                 self.val = val
 
         class Frame(object):
-            _virtualizable2_ = ['thing']
+            _virtualizable_ = ['thing']
 
         driver = JitDriver(greens = ['codeno'], reds = ['i', 'frame'],
                            virtualizables = ['frame'],

File rpython/jit/backend/model.py

     total_compiled_loops = 0
     total_compiled_bridges = 0
     total_freed_loops = 0
-    total_freed_bridges = 0    
+    total_freed_bridges = 0
 
     # for heaptracker
     # _all_size_descrs_with_vtable = None
     def arraydescrof(self, A):
         raise NotImplementedError
 
-    def calldescrof(self, FUNC, ARGS, RESULT):
+    def calldescrof(self, FUNC, ARGS, RESULT, extrainfo):
         # FUNC is the original function type, but ARGS is a list of types
         # with Voids removed
         raise NotImplementedError
     def bh_copyunicodecontent(self, src, dst, srcstart, dststart, length):
         raise NotImplementedError
 
-
 class CompiledLoopToken(object):
     asmmemmgr_blocks = None
     asmmemmgr_gcroots = 0

File rpython/jit/backend/test/runner_test.py

File contents unchanged.

File rpython/jit/backend/x86/assembler.py

                                                         allblocks)
         self.target_tokens_currently_compiling = {}
         self.frame_depth_to_patch = []
+        self._finish_gcmap = lltype.nullptr(jitframe.GCMAP)
 
     def teardown(self):
         self.pending_guard_tokens = None
         self.mov(fail_descr_loc, RawEbpLoc(ofs))
         arglist = op.getarglist()
         if arglist and arglist[0].type == REF:
-            gcmap = self.gcmap_for_finish
+            if self._finish_gcmap:
+                self._finish_gcmap[0] |= r_uint(1) # rax
+                gcmap = self._finish_gcmap
+            else:
+                gcmap = self.gcmap_for_finish
             self.push_gcmap(self.mc, gcmap, store=True)
         else:
             # note that the 0 here is redundant, but I would rather
         #
         return jmp_location
 
-    def _call_assembler_reset_vtoken(self, jd, vloc):
-        from rpython.jit.backend.llsupport.descr import FieldDescr
-        fielddescr = jd.vable_token_descr
-        assert isinstance(fielddescr, FieldDescr)
-        vtoken_ofs = fielddescr.offset
-        self.mc.MOV(edx, vloc) # we know vloc is on the current frame
-        self.mc.MOV_mi((edx.value, vtoken_ofs), 0)
-        # in the line above, TOKEN_NONE = 0
-
     def _call_assembler_load_result(self, op, result_loc):
         if op.result is not None:
             # load the return value from the dead frame's value index 0
         assert 0 < offset <= 127
         self.mc.overwrite(jmp_location - 1, chr(offset))
 
+    def store_force_descr(self, op, fail_locs, frame_depth):
+        guard_token = self.implement_guard_recovery(op.opnum,
+                                                    op.getdescr(),
+                                                    op.getfailargs(),
+                                                    fail_locs, frame_depth)
+        self._finish_gcmap = guard_token.gcmap
+        self._store_force_index(op)
+        self.store_info_on_descr(0, guard_token)
+
     def force_token(self, reg):
         # XXX kill me
         assert isinstance(reg, RegLoc)

File rpython/jit/backend/x86/regalloc.py

 from rpython.jit.codewriter.effectinfo import EffectInfo
 from rpython.jit.metainterp.history import (Box, Const, ConstInt, ConstPtr,
     ConstFloat, BoxInt, BoxFloat, INT, REF, FLOAT, TargetToken)
-from rpython.jit.metainterp.resoperation import rop
+from rpython.jit.metainterp.resoperation import rop, ResOperation
 from rpython.rlib import rgc
 from rpython.rlib.objectmodel import we_are_translated
 from rpython.rlib.rarithmetic import r_longlong, r_uint
         #if jump_op is not None and jump_op.getdescr() is descr:
         #    self._compute_hint_frame_locations_from_descr(descr)
 
+    def consider_guard_not_forced_2(self, op):
+        self.rm.before_call(op.getfailargs(), save_all_regs=True)
+        fail_locs = [self.loc(v) for v in op.getfailargs()]
+        self.assembler.store_force_descr(op, fail_locs,
+                                         self.fm.get_frame_depth())
+        self.possibly_free_vars(op.getfailargs())
+
     def consider_keepalive(self, op):
         pass
 

File rpython/jit/codewriter/codewriter.py

File contents unchanged.

File rpython/jit/codewriter/effectinfo.py

     OS_UNI_COPY_TO_RAW          = 113
 
     OS_JIT_FORCE_VIRTUAL        = 120
+    OS_JIT_FORCE_VIRTUALIZABLE  = 121
 
     # for debugging:
     _OS_CANRAISE = set([
 EffectInfo.MOST_GENERAL = EffectInfo(None, None, None, None,
                                      EffectInfo.EF_RANDOM_EFFECTS,
                                      can_invalidate=True)
+EffectInfo.LEAST_GENERAL = EffectInfo([], [], [], [],
+                                      EffectInfo.EF_ELIDABLE_CANNOT_RAISE,
+                                      can_invalidate=False)
 
 
 def effectinfo_from_writeanalyze(effects, cpu,

File rpython/jit/codewriter/jtransform.py

             op1 = SpaceOperation('str_guard_value', [op.args[0], c, descr],
                                  op.result)
             return [SpaceOperation('-live-', [], None), op1, None]
+        if hints.get('force_virtualizable'):
+            return SpaceOperation('hint_force_virtualizable', [op.args[0]], None)
         else:
             log.WARNING('ignoring hint %r at %r' % (hints, self.graph))
 

File rpython/jit/codewriter/test/test_policy.py

 
 def test_access_directly_but_not_seen():
     class X:
-        _virtualizable2_ = ["a"]
+        _virtualizable_ = ["a"]
     def h(x, y):
         w = 0
         for i in range(y):

File rpython/jit/metainterp/blackhole.py

         from rpython.jit.metainterp import quasiimmut
         quasiimmut.do_force_quasi_immutable(cpu, struct, mutatefielddescr)
 
+    @arguments("r")
+    def bhimpl_hint_force_virtualizable(r):
+        pass
+
     @arguments("cpu", "d", returns="r")
     def bhimpl_new(cpu, descr):
         return cpu.bh_new(descr)

File rpython/jit/metainterp/compile.py

         rstack._stack_criticalcode_start()
         try:
             deadframe = cpu.force(token)
+            # this should set descr to ResumeGuardForceDescr, if it
+            # was not that already
             faildescr = cpu.get_latest_descr(deadframe)
             assert isinstance(faildescr, ResumeGuardForcedDescr)
             faildescr.handle_async_forcing(deadframe)
             rstack._stack_criticalcode_stop()
 
     def handle_async_forcing(self, deadframe):
-        from rpython.jit.metainterp.resume import force_from_resumedata
+        from rpython.jit.metainterp.resume import (force_from_resumedata,
+                                                   AlreadyForced)
         metainterp_sd = self.metainterp_sd
         vinfo = self.jitdriver_sd.virtualizable_info
         ginfo = self.jitdriver_sd.greenfield_info
-        all_virtuals = force_from_resumedata(metainterp_sd, self, deadframe,
-                                             vinfo, ginfo)
+        # there is some chance that this is already forced. In this case
+        # the virtualizable would have a token = NULL
+        try:
+            all_virtuals = force_from_resumedata(metainterp_sd, self, deadframe,
+                                                 vinfo, ginfo)
+        except AlreadyForced:
+            return
         # The virtualizable data was stored on the real virtualizable above.
         # Handle all_virtuals: keep them for later blackholing from the
         # future failure of the GUARD_NOT_FORCED

File rpython/jit/metainterp/executor.py

File contents unchanged.

File rpython/jit/metainterp/heapcache.py

File contents unchanged.

File rpython/jit/metainterp/history.py

         descr_ptr = cpu.ts.cast_to_baseclass(descr_gcref)
         return cast_base_ptr_to_instance(AbstractDescr, descr_ptr)
 
+    def get_vinfo(self):
+        raise NotImplementedError
 
 class AbstractFailDescr(AbstractDescr):
     index = -1

File rpython/jit/metainterp/optimizeopt/test/test_optimizebasic.py

         }
         self.optimize_loop(ops, expected, call_pure_results)
 
+    def test_guard_not_forced_2_virtual(self):
+        ops = """
+        [i0]
+        p0 = new_array(3, descr=arraydescr)
+        guard_not_forced_2() [p0]
+        finish(p0)
+        """
+        self.optimize_loop(ops, ops)
+
 
 class TestLLtype(BaseTestOptimizeBasic, LLtypeMixin):
     pass

File rpython/jit/metainterp/optimizeopt/test/test_optimizeopt.py

         """
         self.optimize_loop(ops, expected)
 
+    def test_force_virtualizable_virtual(self):
+        ops = """
+        [i0]
+        p1 = new_with_vtable(ConstClass(node_vtable))
+        cond_call(1, 123, p1, descr=clear_vable)
+        jump(i0)
+        """
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected)
+
     def test_setgetfield_counter(self):
         ops = """
         [p1]

File rpython/jit/metainterp/optimizeopt/test/test_util.py

     asmdescr = LoopToken() # it can be whatever, it's not a descr though
 
     from rpython.jit.metainterp.virtualref import VirtualRefInfo
+
     class FakeWarmRunnerDesc:
         pass
     FakeWarmRunnerDesc.cpu = cpu
     vrefinfo = VirtualRefInfo(FakeWarmRunnerDesc)
     virtualtokendescr = vrefinfo.descr_virtual_token
     virtualforceddescr = vrefinfo.descr_forced
+    FUNC = lltype.FuncType([], lltype.Void)
+    ei = EffectInfo([], [], [], [], EffectInfo.EF_CANNOT_RAISE,
+                    can_invalidate=False,
+                    oopspecindex=EffectInfo.OS_JIT_FORCE_VIRTUALIZABLE)
+    clear_vable = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, ei)
+
     jit_virtual_ref_vtable = vrefinfo.jit_virtual_ref_vtable
     jvr_vtable_adr = llmemory.cast_ptr_to_adr(jit_virtual_ref_vtable)
 

File rpython/jit/metainterp/optimizeopt/virtualize.py

 class OptVirtualize(optimizer.Optimization):
     "Virtualize objects until they escape."
 
+    _last_guard_not_forced_2 = None
+
     def new(self):
         return OptVirtualize()
 
             return
         self.emit_operation(op)
 
+    def optimize_GUARD_NOT_FORCED_2(self, op):
+        self._last_guard_not_forced_2 = op
+
+    def optimize_FINISH(self, op):
+        if self._last_guard_not_forced_2 is not None:
+            guard_op = self._last_guard_not_forced_2
+            self.emit_operation(op)
+            guard_op = self.optimizer.store_final_boxes_in_guard(guard_op, [])
+            i = len(self.optimizer._newoperations) - 1
+            assert i >= 0
+            self.optimizer._newoperations.insert(i, guard_op)
+        else:
+            self.emit_operation(op)
+
     def optimize_CALL_MAY_FORCE(self, op):
         effectinfo = op.getdescr().get_extra_info()
         oopspecindex = effectinfo.oopspecindex
                 return
         self.emit_operation(op)
 
+    def optimize_COND_CALL(self, op):
+        effectinfo = op.getdescr().get_extra_info()
+        oopspecindex = effectinfo.oopspecindex
+        if oopspecindex == EffectInfo.OS_JIT_FORCE_VIRTUALIZABLE:
+            value = self.getvalue(op.getarg(2))
+            if value.is_virtual():
+                return
+        self.emit_operation(op)
+
     def optimize_VIRTUAL_REF(self, op):
         # get some constants
         vrefinfo = self.optimizer.metainterp_sd.virtualref_info
             self.do_RAW_MALLOC_VARSIZE_CHAR(op)
         elif effectinfo.oopspecindex == EffectInfo.OS_RAW_FREE:
             self.do_RAW_FREE(op)
+        elif effectinfo.oopspecindex == EffectInfo.OS_JIT_FORCE_VIRTUALIZABLE:
+            # we might end up having CALL here instead of COND_CALL
+            value = self.getvalue(op.getarg(1))
+            if value.is_virtual():
+                return
         else:
             self.emit_operation(op)
 

File rpython/jit/metainterp/pyjitpl.py

 from rpython.rlib.jit import Counters
 from rpython.rlib.objectmodel import we_are_translated, specialize
 from rpython.rlib.unroll import unrolling_iterable
-from rpython.rtyper.lltypesystem import lltype, rclass
+from rpython.rtyper.lltypesystem import lltype, rclass, rffi
 
 
 
             opnum = rop.GUARD_TRUE
         else:
             opnum = rop.GUARD_FALSE
-        self.generate_guard(opnum, box)
+        self.metainterp.generate_guard(opnum, box)
         if not switchcase:
             self.pc = target
 
         value = box.nonnull()
         if value:
             if not self.metainterp.heapcache.is_class_known(box):
-                self.generate_guard(rop.GUARD_NONNULL, box, resumepc=orgpc)
+                self.metainterp.generate_guard(rop.GUARD_NONNULL, box,
+                                               resumepc=orgpc)
         else:
             if not isinstance(box, Const):
-                self.generate_guard(rop.GUARD_ISNULL, box, resumepc=orgpc)
+                self.metainterp.generate_guard(rop.GUARD_ISNULL, box,
+                                               resumepc=orgpc)
                 promoted_box = box.constbox()
                 self.metainterp.replace_box(box, promoted_box)
         return value
     def _opimpl_getfield_gc_greenfield_any(self, box, fielddescr, pc):
         ginfo = self.metainterp.jitdriver_sd.greenfield_info
         if (ginfo is not None and fielddescr in ginfo.green_field_descrs
-            and not self._nonstandard_virtualizable(pc, box)):
+            and not self._nonstandard_virtualizable(pc, box, fielddescr)):
             # fetch the result, but consider it as a Const box and don't
             # record any operation
             resbox = executor.execute(self.metainterp.cpu, self.metainterp,
     opimpl_raw_load_i = _opimpl_raw_load
     opimpl_raw_load_f = _opimpl_raw_load
 
+    @arguments("box")
+    def opimpl_hint_force_virtualizable(self, box):
+        self.metainterp.gen_store_back_in_vable(box)
+
     @arguments("box", "descr", "descr", "orgpc")
     def opimpl_record_quasiimmut_field(self, box, fielddescr,
                                        mutatefielddescr, orgpc):
         descr = QuasiImmutDescr(cpu, box, fielddescr, mutatefielddescr)
         self.metainterp.history.record(rop.QUASIIMMUT_FIELD, [box],
                                        None, descr=descr)
-        self.generate_guard(rop.GUARD_NOT_INVALIDATED, resumepc=orgpc)
+        self.metainterp.generate_guard(rop.GUARD_NOT_INVALIDATED,
+                                       resumepc=orgpc)
 
     @arguments("box", "descr", "orgpc")
     def opimpl_jit_force_quasi_immutable(self, box, mutatefielddescr, orgpc):
             do_force_quasi_immutable(self.metainterp.cpu, box.getref_base(),
                                      mutatefielddescr)
             raise SwitchToBlackhole(Counters.ABORT_FORCE_QUASIIMMUT)
-        self.generate_guard(rop.GUARD_ISNULL, mutatebox, resumepc=orgpc)
+        self.metainterp.generate_guard(rop.GUARD_ISNULL, mutatebox,
+                                       resumepc=orgpc)
 
-    def _nonstandard_virtualizable(self, pc, box):
+    def _nonstandard_virtualizable(self, pc, box, fielddescr):
         # returns True if 'box' is actually not the "standard" virtualizable
         # that is stored in metainterp.virtualizable_boxes[-1]
-        if (self.metainterp.jitdriver_sd.virtualizable_info is None and
-            self.metainterp.jitdriver_sd.greenfield_info is None):
-            return True      # can occur in case of multiple JITs
-        standard_box = self.metainterp.virtualizable_boxes[-1]
-        if standard_box is box:
-            return False
         if self.metainterp.heapcache.is_nonstandard_virtualizable(box):
             return True
-        eqbox = self.metainterp.execute_and_record(rop.PTR_EQ, None,
-                                                   box, standard_box)
-        eqbox = self.implement_guard_value(eqbox, pc)
-        isstandard = eqbox.getint()
-        if isstandard:
-            self.metainterp.replace_box(box, standard_box)
-        else:
-            self.metainterp.heapcache.nonstandard_virtualizables_now_known(box)
-        return not isstandard
+        if box is self.metainterp.forced_virtualizable:
+            self.metainterp.forced_virtualizable = None
+        if (self.metainterp.jitdriver_sd.virtualizable_info is not None or
+            self.metainterp.jitdriver_sd.greenfield_info is not None):
+            standard_box = self.metainterp.virtualizable_boxes[-1]
+            if standard_box is box:
+                return False
+            vinfo = self.metainterp.jitdriver_sd.virtualizable_info
+            if vinfo is fielddescr.get_vinfo():
+                eqbox = self.metainterp.execute_and_record(rop.PTR_EQ, None,
+                                                           box, standard_box)
+                eqbox = self.implement_guard_value(eqbox, pc)
+                isstandard = eqbox.getint()
+                if isstandard:
+                    self.metainterp.replace_box(box, standard_box)
+                    return False
+        if not self.metainterp.heapcache.is_unescaped(box):
+            self.emit_force_virtualizable(fielddescr, box)
+        self.metainterp.heapcache.nonstandard_virtualizables_now_known(box)
+        return True
+
+    def emit_force_virtualizable(self, fielddescr, box):
+        vinfo = fielddescr.get_vinfo()
+        token_descr = vinfo.vable_token_descr
+        mi = self.metainterp
+        tokenbox = mi.execute_and_record(rop.GETFIELD_GC, token_descr, box)
+        condbox = mi.execute_and_record(rop.PTR_NE, None, tokenbox,
+                                       history.CONST_NULL)
+        funcbox = ConstInt(rffi.cast(lltype.Signed, vinfo.clear_vable_ptr))
+        calldescr = vinfo.clear_vable_descr
+        self.execute_varargs(rop.COND_CALL, [condbox, funcbox, box],
+                             calldescr, False, False)
 
     def _get_virtualizable_field_index(self, fielddescr):
         # Get the index of a fielddescr.  Must only be called for
 
     @arguments("box", "descr", "orgpc")
     def _opimpl_getfield_vable(self, box, fielddescr, pc):
-        if self._nonstandard_virtualizable(pc, box):
+        if self._nonstandard_virtualizable(pc, box, fielddescr):
             return self._opimpl_getfield_gc_any(box, fielddescr)
         self.metainterp.check_synchronized_virtualizable()
         index = self._get_virtualizable_field_index(fielddescr)
 
     @arguments("box", "box", "descr", "orgpc")
     def _opimpl_setfield_vable(self, box, valuebox, fielddescr, pc):
-        if self._nonstandard_virtualizable(pc, box):
+        if self._nonstandard_virtualizable(pc, box, fielddescr):
             return self._opimpl_setfield_gc_any(box, valuebox, fielddescr)
         index = self._get_virtualizable_field_index(fielddescr)
         self.metainterp.virtualizable_boxes[index] = valuebox
 
     @arguments("box", "box", "descr", "descr", "orgpc")
     def _opimpl_getarrayitem_vable(self, box, indexbox, fdescr, adescr, pc):
-        if self._nonstandard_virtualizable(pc, box):
+        if self._nonstandard_virtualizable(pc, box, fdescr):
             arraybox = self._opimpl_getfield_gc_any(box, fdescr)
             return self._opimpl_getarrayitem_gc_any(arraybox, indexbox, adescr)
         self.metainterp.check_synchronized_virtualizable()
     @arguments("box", "box", "box", "descr", "descr", "orgpc")
     def _opimpl_setarrayitem_vable(self, box, indexbox, valuebox,
                                    fdescr, adescr, pc):
-        if self._nonstandard_virtualizable(pc, box):
+        if self._nonstandard_virtualizable(pc, box, fdescr):
             arraybox = self._opimpl_getfield_gc_any(box, fdescr)
             self._opimpl_setarrayitem_gc_any(arraybox, indexbox, valuebox,
                                              adescr)
 
     @arguments("box", "descr", "descr", "orgpc")
     def opimpl_arraylen_vable(self, box, fdescr, adescr, pc):
-        if self._nonstandard_virtualizable(pc, box):
+        if self._nonstandard_virtualizable(pc, box, fdescr):
             arraybox = self._opimpl_getfield_gc_any(box, fdescr)
             return self.opimpl_arraylen_gc(arraybox, adescr)
         vinfo = self.metainterp.jitdriver_sd.virtualizable_info
             promoted_box = resbox.constbox()
             # This is GUARD_VALUE because GUARD_TRUE assumes the existance
             # of a label when computing resumepc
-            self.generate_guard(rop.GUARD_VALUE, resbox, [promoted_box],
-                                resumepc=orgpc)
+            self.metainterp.generate_guard(rop.GUARD_VALUE, resbox,
+                                           [promoted_box],
+                                           resumepc=orgpc)
             self.metainterp.replace_box(box, constbox)
             return constbox
 
     def opimpl_guard_class(self, box, orgpc):
         clsbox = self.cls_of_box(box)
         if not self.metainterp.heapcache.is_class_known(box):
-            self.generate_guard(rop.GUARD_CLASS, box, [clsbox], resumepc=orgpc)
+            self.metainterp.generate_guard(rop.GUARD_CLASS, box, [clsbox],
+                                           resumepc=orgpc)
             self.metainterp.heapcache.class_now_known(box)
         return clsbox
 
     def opimpl_jit_merge_point(self, jdindex, greenboxes,
                                jcposition, redboxes, orgpc):
         resumedescr = compile.ResumeAtPositionDescr()
-        self.capture_resumedata(resumedescr, orgpc)
+        self.metainterp.capture_resumedata(resumedescr, orgpc)
 
         any_operation = len(self.metainterp.history.operations) > 0
         jitdriver_sd = self.metainterp.staticdata.jitdrivers_sd[jdindex]
         # xxx hack
         if not self.metainterp.heapcache.is_class_known(exc_value_box):
             clsbox = self.cls_of_box(exc_value_box)
-            self.generate_guard(rop.GUARD_CLASS, exc_value_box, [clsbox],
-                                resumepc=orgpc)
+            self.metainterp.generate_guard(rop.GUARD_CLASS, exc_value_box,
+                                           [clsbox], resumepc=orgpc)
         self.metainterp.class_of_last_exc_is_const = True
         self.metainterp.last_exc_value_box = exc_value_box
         self.metainterp.popframe()
         except ChangeFrame:
             pass
 
-    def generate_guard(self, opnum, box=None, extraargs=[], resumepc=-1):
-        if isinstance(box, Const):    # no need for a guard
-            return
-        metainterp = self.metainterp
-        if box is not None:
-            moreargs = [box] + extraargs
-        else:
-            moreargs = list(extraargs)
-        metainterp_sd = metainterp.staticdata
-        if opnum == rop.GUARD_NOT_FORCED:
-            resumedescr = compile.ResumeGuardForcedDescr(metainterp_sd,
-                                                   metainterp.jitdriver_sd)
-        elif opnum == rop.GUARD_NOT_INVALIDATED:
-            resumedescr = compile.ResumeGuardNotInvalidated()
-        else:
-            resumedescr = compile.ResumeGuardDescr()
-        guard_op = metainterp.history.record(opnum, moreargs, None,
-                                             descr=resumedescr)
-        self.capture_resumedata(resumedescr, resumepc)
-        self.metainterp.staticdata.profiler.count_ops(opnum, Counters.GUARDS)
-        # count
-        metainterp.attach_debug_info(guard_op)
-        return guard_op
-
-    def capture_resumedata(self, resumedescr, resumepc=-1):
-        metainterp = self.metainterp
-        virtualizable_boxes = None
-        if (metainterp.jitdriver_sd.virtualizable_info is not None or
-            metainterp.jitdriver_sd.greenfield_info is not None):
-            virtualizable_boxes = metainterp.virtualizable_boxes
-        saved_pc = self.pc
-        if resumepc >= 0:
-            self.pc = resumepc
-        resume.capture_resumedata(metainterp.framestack, virtualizable_boxes,
-                                  metainterp.virtualref_boxes, resumedescr)
-        self.pc = saved_pc
-
     def implement_guard_value(self, box, orgpc):
         """Promote the given Box into a Const.  Note: be careful, it's a
         bit unclear what occurs if a single opcode needs to generate
             return box     # no promotion needed, already a Const
         else:
             promoted_box = box.constbox()
-            self.generate_guard(rop.GUARD_VALUE, box, [promoted_box],
-                                resumepc=orgpc)
+            self.metainterp.generate_guard(rop.GUARD_VALUE, box, [promoted_box],
+                                           resumepc=orgpc)
             self.metainterp.replace_box(box, promoted_box)
             return promoted_box
 
             if resbox is not None:
                 self.make_result_of_lastop(resbox)
             self.metainterp.vable_after_residual_call()
-            self.generate_guard(rop.GUARD_NOT_FORCED, None)
+            self.metainterp.generate_guard(rop.GUARD_NOT_FORCED, None)
             if vablebox is not None:
                 self.metainterp.history.record(rop.KEEPALIVE, [vablebox], None)
             self.metainterp.handle_possible_exception()
         self.portal_trace_positions = []
         self.free_frames_list = []
         self.last_exc_value_box = None
+        self.forced_virtualizable = None
         self.partial_trace = None
         self.retracing_from = -1
         self.call_pure_results = args_dict_box()
                 print jitcode.name
             raise AssertionError
 
+    def generate_guard(self, opnum, box=None, extraargs=[], resumepc=-1):
+        if isinstance(box, Const):    # no need for a guard
+            return
+        if box is not None:
+            moreargs = [box] + extraargs
+        else:
+            moreargs = list(extraargs)
+        metainterp_sd = self.staticdata
+        if opnum == rop.GUARD_NOT_FORCED or opnum == rop.GUARD_NOT_FORCED_2:
+            resumedescr = compile.ResumeGuardForcedDescr(metainterp_sd,
+                                                         self.jitdriver_sd)
+        elif opnum == rop.GUARD_NOT_INVALIDATED:
+            resumedescr = compile.ResumeGuardNotInvalidated()
+        else:
+            resumedescr = compile.ResumeGuardDescr()
+        guard_op = self.history.record(opnum, moreargs, None,
+                                             descr=resumedescr)
+        self.capture_resumedata(resumedescr, resumepc)
+        self.staticdata.profiler.count_ops(opnum, Counters.GUARDS)
+        # count
+        self.attach_debug_info(guard_op)
+        return guard_op
+
+    def capture_resumedata(self, resumedescr, resumepc=-1):
+        virtualizable_boxes = None
+        if (self.jitdriver_sd.virtualizable_info is not None or
+            self.jitdriver_sd.greenfield_info is not None):
+            virtualizable_boxes = self.virtualizable_boxes
+        saved_pc = 0
+        if self.framestack:
+            frame = self.framestack[-1]
+            saved_pc = frame.pc
+            if resumepc >= 0:
+                frame.pc = resumepc
+        resume.capture_resumedata(self.framestack, virtualizable_boxes,
+                                  self.virtualref_boxes, resumedescr)
+        if self.framestack:
+            self.framestack[-1].pc = saved_pc
+
     def create_empty_history(self):
         self.history = history.History()
         self.staticdata.stats.set_history(self.history)
             self.raise_continue_running_normally(live_arg_boxes, jitcell_token)
 
     def compile_done_with_this_frame(self, exitbox):
-        self.gen_store_back_in_virtualizable()
         # temporarily put a JUMP to a pseudo-loop
+        self.store_token_in_vable()
         sd = self.staticdata
         result_type = self.jitdriver_sd.result_type
         if result_type == history.VOID:
         if target_token is not token:
             compile.giveup()
 
+    def store_token_in_vable(self):
+        vinfo = self.jitdriver_sd.virtualizable_info
+        if vinfo is None:
+            return
+        vbox = self.virtualizable_boxes[-1]
+        if vbox is self.forced_virtualizable:
+            return # we already forced it by hand
+        force_token_box = history.BoxPtr()
+        # in case the force_token has not been recorded, record it here
+        # to make sure we know the virtualizable can be broken. However, the
+        # contents of the virtualizable should be generally correct
+        self.history.record(rop.FORCE_TOKEN, [], force_token_box)
+        self.history.record(rop.SETFIELD_GC, [vbox, force_token_box],
+                            None, descr=vinfo.vable_token_descr)
+        self.generate_guard(rop.GUARD_NOT_FORCED_2, None)
+
     def compile_exit_frame_with_exception(self, valuebox):
-        self.gen_store_back_in_virtualizable()
+        self.store_token_in_vable()
         sd = self.staticdata
         token = sd.loop_tokens_exit_frame_with_exception_ref[0].finishdescr
         self.history.record(rop.FINISH, [valuebox], None, descr=token)
         self.virtualref_boxes[i+1] = self.cpu.ts.CONST_NULL
 
     def handle_possible_exception(self):
-        frame = self.framestack[-1]
         if self.last_exc_value_box is not None:
             exception_box = self.cpu.ts.cls_of_box(self.last_exc_value_box)
-            op = frame.generate_guard(rop.GUARD_EXCEPTION,
-                                      None, [exception_box])
+            op = self.generate_guard(rop.GUARD_EXCEPTION,
+                                     None, [exception_box])
             assert op is not None
             op.result = self.last_exc_value_box
             self.class_of_last_exc_is_const = True
             self.finishframe_exception()
         else:
-            frame.generate_guard(rop.GUARD_NO_EXCEPTION, None, [])
+            self.generate_guard(rop.GUARD_NO_EXCEPTION, None, [])
 
     def handle_possible_overflow_error(self):
-        frame = self.framestack[-1]
         if self.last_exc_value_box is not None:
-            frame.generate_guard(rop.GUARD_OVERFLOW, None)
+            self.generate_guard(rop.GUARD_OVERFLOW, None)
             assert isinstance(self.last_exc_value_box, Const)
             assert self.class_of_last_exc_is_const
             self.finishframe_exception()
         else:
-            frame.generate_guard(rop.GUARD_NO_OVERFLOW, None)
+            self.generate_guard(rop.GUARD_NO_OVERFLOW, None)
 
     def assert_no_exception(self):
         assert self.last_exc_value_box is None
         if vinfo is not None:
             self.virtualizable_boxes = virtualizable_boxes
             # just jumped away from assembler (case 4 in the comment in
-            # virtualizable.py) into tracing (case 2); check that vable_token
-            # is and stays NULL.  Note the call to reset_vable_token() in
-            # warmstate.py.
+            # virtualizable.py) into tracing (case 2); if we get the
+            # virtualizable from somewhere strange it might not be forced,
+            # do it
             virtualizable_box = self.virtualizable_boxes[-1]
             virtualizable = vinfo.unwrap_virtualizable_box(virtualizable_box)
-            assert not vinfo.is_token_nonnull_gcref(virtualizable)
+            if vinfo.is_token_nonnull_gcref(virtualizable):
+                vinfo.reset_token_gcref(virtualizable)
             # fill the virtualizable with the local boxes
             self.synchronize_virtualizable()
         #
                                                         virtualizable)
             self.virtualizable_boxes.append(virtualizable_box)
 
-    def gen_store_back_in_virtualizable(self):
+    def gen_store_back_in_vable(self, box):
         vinfo = self.jitdriver_sd.virtualizable_info
         if vinfo is not None:
             # xxx only write back the fields really modified
             vbox = self.virtualizable_boxes[-1]
+            if vbox is not box:
+                # ignore the hint on non-standard virtualizable
+                # specifically, ignore it on a virtual
+                return
+            if self.forced_virtualizable is not None:
+                # this can happen only in strange cases, but we don't care
+                # it was already forced
+                return
+            self.forced_virtualizable = vbox
             for i in range(vinfo.num_static_extra_boxes):
                 fieldbox = self.virtualizable_boxes[i]
                 descr = vinfo.static_field_descrs[i]
                     self.execute_and_record(rop.SETARRAYITEM_GC, descr,
                                             abox, ConstInt(j), itembox)
             assert i + 1 == len(self.virtualizable_boxes)
+            # we're during tracing, so we should not execute it
+            self.history.record(rop.SETFIELD_GC, [vbox, self.cpu.ts.CONST_NULL],
+                                None, descr=vinfo.vable_token_descr)
 
     def replace_box(self, oldbox, newbox):
         assert isinstance(oldbox, Box)

File rpython/jit/metainterp/resoperation.py

     'GUARD_NO_OVERFLOW/0d',
     'GUARD_OVERFLOW/0d',
     'GUARD_NOT_FORCED/0d',      # may be called with an exception currently set
+    'GUARD_NOT_FORCED_2/0d',    # same as GUARD_NOT_FORCED, but for finish()
     'GUARD_NOT_INVALIDATED/0d',
     '_GUARD_LAST', # ----- end of guard operations -----
 
     'VIRTUAL_REF/2',         # removed before it's passed to the backend
     'READ_TIMESTAMP/0',
     'MARK_OPAQUE_PTR/1b',
+    # this one has no *visible* side effect, since the virtualizable
+    # must be forced, however we need to execute it anyway
     '_NOSIDEEFFECT_LAST', # ----- end of no_side_effect operations -----
 
     'SETARRAYITEM_GC/3d',

File rpython/jit/metainterp/resume.py

 # because it needs to support optimize.py which encodes virtuals with
 # arbitrary cycles and also to compress the information
 
+class AlreadyForced(Exception):
+    pass
+
 class Snapshot(object):
     __slots__ = ('prev', 'boxes')
 
 
 def capture_resumedata(framestack, virtualizable_boxes, virtualref_boxes,
                        storage):
-    n = len(framestack)-1
-    top = framestack[n]
-    _ensure_parent_resumedata(framestack, n)
-    frame_info_list = FrameInfo(top.parent_resumedata_frame_info_list,
-                                top.jitcode, top.pc)
-    storage.rd_frame_info_list = frame_info_list
-    snapshot = Snapshot(top.parent_resumedata_snapshot,
-                        top.get_list_of_active_boxes(False))
+    n = len(framestack) - 1
     if virtualizable_boxes is not None:
         boxes = virtualref_boxes + virtualizable_boxes
     else:
         boxes = virtualref_boxes[:]
-    snapshot = Snapshot(snapshot, boxes)
-    storage.rd_snapshot = snapshot
+    if n >= 0:
+        top = framestack[n]
+        _ensure_parent_resumedata(framestack, n)
+        frame_info_list = FrameInfo(top.parent_resumedata_frame_info_list,
+                                    top.jitcode, top.pc)
+        storage.rd_frame_info_list = frame_info_list
+        snapshot = Snapshot(top.parent_resumedata_snapshot,
+                            top.get_list_of_active_boxes(False))
+        snapshot = Snapshot(snapshot, boxes)
+        storage.rd_snapshot = snapshot
+    else:
+        storage.rd_frame_info_list = None
+        storage.rd_snapshot = Snapshot(None, boxes)
 
 #
 # The following is equivalent to the RPython-level declaration:
             return len(numb.nums)
         index = len(numb.nums) - 1
         virtualizable = self.decode_ref(numb.nums[index])
-        if self.resume_after_guard_not_forced == 1:
-            # in the middle of handle_async_forcing()
-            assert vinfo.is_token_nonnull_gcref(virtualizable)
-            vinfo.reset_token_gcref(virtualizable)
-        else:
-            # just jumped away from assembler (case 4 in the comment in
-            # virtualizable.py) into tracing (case 2); check that vable_token
-            # is and stays NULL.  Note the call to reset_vable_token() in
-            # warmstate.py.
-            assert not vinfo.is_token_nonnull_gcref(virtualizable)
+        # just reset the token, we'll force it later
+        vinfo.reset_token_gcref(virtualizable)
         return vinfo.write_from_resume_data_partial(virtualizable, self, numb)
 
     def load_value_of_type(self, TYPE, tagged):

File rpython/jit/metainterp/test/test_recursive.py

                 self.i8 = i8
                 self.i9 = i9
 
-
         def loop(n):
             i = 0
             o = A(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
                 self.i8 = i8
                 self.i9 = i9
 
-
         def loop(n):
             i = 0
             o = A(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
         # exactly the same logic as the previous test, but with 'frame.j'
         # instead of just 'j'
         class Frame(object):
-            _virtualizable2_ = ['j']
+            _virtualizable_ = ['j']
             def __init__(self, j):
                 self.j = j
 
                 self.val = val
 
         class Frame(object):
-            _virtualizable2_ = ['thing']
+            _virtualizable_ = ['thing']
 
-        driver = JitDriver(greens = ['codeno'], reds = ['i', 'frame'],
+        driver = JitDriver(greens = ['codeno'], reds = ['i', 's', 'frame'],
                            virtualizables = ['frame'],
                            get_printable_location = lambda codeno : str(codeno))
 
 
         def portal(codeno, frame):
             i = 0
+            s = 0
             while i < 10:
-                driver.can_enter_jit(frame=frame, codeno=codeno, i=i)
-                driver.jit_merge_point(frame=frame, codeno=codeno, i=i)
+                driver.can_enter_jit(frame=frame, codeno=codeno, i=i, s=s)
+                driver.jit_merge_point(frame=frame, codeno=codeno, i=i, s=s)
                 nextval = frame.thing.val
                 if codeno == 0:
                     subframe = Frame()
                     subframe.thing = Thing(nextval)
                     nextval = portal(1, subframe)
+                    s += subframe.thing.val
                 frame.thing = Thing(nextval + 1)
                 i += 1
             return frame.thing.val
 
         res = self.meta_interp(main, [0], inline=True)
+        self.check_resops(call=0, cond_call=0) # got removed by optimization
         assert res == main(0)
 
     def test_directly_call_assembler_virtualizable_reset_token(self):
+        py.test.skip("not applicable any more, I think")
         from rpython.rtyper.lltypesystem import lltype
         from rpython.rlib.debug import llinterpcall
 
                 self.val = val
 
         class Frame(object):
-            _virtualizable2_ = ['thing']
+            _virtualizable_ = ['thing']
 
         driver = JitDriver(greens = ['codeno'], reds = ['i', 'frame'],
                            virtualizables = ['frame'],
                 self.val = val
 
         class Frame(object):
-            _virtualizable2_ = ['thing']
+            _virtualizable_ = ['thing']
 
         driver = JitDriver(greens = ['codeno'], reds = ['i', 'frame'],
                            virtualizables = ['frame'],
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['l[*]', 's']
+            _virtualizable_ = ['l[*]', 's']
 
             def __init__(self, l, s):
                 self = hint(self, access_directly=True,
                 self.val = val
 
         class Frame(object):
-            _virtualizable2_ = ['thing']
+            _virtualizable_ = ['thing']
 
         driver = JitDriver(greens = ['codeno'], reds = ['i', 'frame'],
                            virtualizables = ['frame'],

File rpython/jit/metainterp/test/test_tracingopts.py

                                     virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['l[*]', 's']
+            _virtualizable_ = ['l[*]', 's']
 
             def __init__(self, a, s):
                 self = jit.hint(self, access_directly=True, fresh_virtualizable=True)

File rpython/jit/metainterp/test/test_virtualizable.py

 from rpython.jit.codewriter.policy import StopAtXPolicy
 from rpython.jit.metainterp.optimizeopt.test.test_util import LLtypeMixin
 from rpython.jit.metainterp.test.support import LLJitMixin
-from rpython.jit.metainterp.warmspot import get_translator
+from rpython.jit.metainterp.warmspot import get_translator, get_stats
+from rpython.jit.metainterp.resoperation import rop
 from rpython.rlib.jit import JitDriver, hint, dont_look_inside, promote, virtual_ref
 from rpython.rlib.rarithmetic import intmask
 from rpython.rtyper.annlowlevel import hlstr
         return lltype_to_annotation(lltype.Void)
 
     def specialize_call(self, hop):
-        op = self.instance    # the LLOp object that was called
         args_v = [hop.inputarg(hop.args_r[0], 0),
                   hop.inputconst(lltype.Void, hop.args_v[1].value),
                   hop.inputconst(lltype.Void, {})]
         ('vable_token', llmemory.GCREF),
         ('inst_x', lltype.Signed),
         ('inst_node', lltype.Ptr(LLtypeMixin.NODE)),
-        hints = {'virtualizable2_accessor': FieldListAccessor()})
-    XY._hints['virtualizable2_accessor'].initialize(
+        hints = {'virtualizable_accessor': FieldListAccessor()})
+    XY._hints['virtualizable_accessor'].initialize(
         XY, {'inst_x': IR_IMMUTABLE, 'inst_node': IR_IMMUTABLE})
 
     xy_vtable = lltype.malloc(rclass.OBJECT_VTABLE, immortal=True)
                 n -= 1
         def f(n):
             xy = self.setup()
+            promote_virtualizable(xy, 'inst_x')
             xy.inst_x = 10000
             m = 10
             while m > 0:
                 g(xy, n)
                 m -= 1
+            promote_virtualizable(xy, 'inst_x')
             return xy.inst_x
         res = self.meta_interp(f, [18])
         assert res == 10180
             return xy.inst_x
         res = self.meta_interp(f, [20])
         assert res == 134
-        self.check_simple_loop(setfield_gc=1, getfield_gc=0)
-        self.check_resops(setfield_gc=2, getfield_gc=3)
+        self.check_simple_loop(setfield_gc=1, getfield_gc=0, cond_call=1)
+        self.check_resops(setfield_gc=2, getfield_gc=4)
 
     # ------------------------------
 
         ('inst_x', lltype.Signed),
         ('inst_l1', lltype.Ptr(lltype.GcArray(lltype.Signed))),
         ('inst_l2', lltype.Ptr(lltype.GcArray(lltype.Signed))),
-        hints = {'virtualizable2_accessor': FieldListAccessor()})
-    XY2._hints['virtualizable2_accessor'].initialize(
+        hints = {'virtualizable_accessor': FieldListAccessor()})
+    XY2._hints['virtualizable_accessor'].initialize(
         XY2, {'inst_x': IR_IMMUTABLE,
               'inst_l1': IR_IMMUTABLE_ARRAY, 'inst_l2': IR_IMMUTABLE_ARRAY})
 
             while m > 0:
                 g(xy2, n)
                 m -= 1
+            promote_virtualizable(xy2, 'inst_l2')
             return xy2.inst_l2[0]
         assert f(18) == 10360
         res = self.meta_interp(f, [18])
         res = self.meta_interp(f, [20], enable_opts='')
         assert res == expected
         self.check_simple_loop(setarrayitem_gc=1, setfield_gc=0,
-                               getarrayitem_gc=1, arraylen_gc=1, getfield_gc=1)
+                               getarrayitem_gc=1, arraylen_gc=1, getfield_gc=2)
 
     # ------------------------------
 
             while m > 0:
                 g(xy2, n)
                 m -= 1
-            return xy2.parent.inst_l2[0]
+            parent = xy2.parent
+            promote_virtualizable(parent, 'inst_l2')
+            return parent.inst_l2[0]
         assert f(18) == 10360
         res = self.meta_interp(f, [18])
         assert res == 10360
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self.x = x
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['l[*]', 's']
+            _virtualizable_ = ['l[*]', 's']
 
             def __init__(self, l, s):
                 self.l = l
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self.x = x
                               virtualizables = ['frame'])
 
         class BaseFrame(object):
-            _virtualizable2_ = ['x[*]']
+            _virtualizable_ = ['x[*]']
 
             def __init__(self, x):
                 self.x = x
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class Y:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class Y:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class FooBarError(Exception):
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
         class SomewhereElse:
             pass
                               virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['stackpos', 'stack[*]']
+            _virtualizable_ = ['stackpos', 'stack[*]']
 
         def f(n):
             frame = Frame()
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self = hint(self, access_directly=True)
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self = hint(self, access_directly=True)
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y', 'z']
+            _virtualizable_ = ['x', 'y', 'z']
 
             def __init__(self, x, y, z=1):
                 self = hint(self, access_directly=True)
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x[*]']
+            _virtualizable_ = ['x[*]']
 
             def __init__(self, x, y):
                 self = hint(self, access_directly=True,
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self = hint(self, access_directly=True)
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self.x = x
                                 virtualizables = ['frame'])
 
         class Frame(object):
-            _virtualizable2_ = ['x', 'y']
+            _virtualizable_ = ['x', 'y']
 
             def __init__(self, x, y):
                 self = hint(self, access_directly=True)
 
     def test_inlining(self):
         class Frame(object):
-            _virtualizable2_ = ['x', 'next']
+            _virtualizable_ = ['x', 'next']
 
             def __init__(self, x):
                 self = hint(self, access_directly=True)