Source

python-peps / pep-0319.txt

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
PEP: 319
Title: Python Synchronize/Asynchronize Block
Version: $Revision$
Last-Modified: $Date$
Author: Michel Pelletier <michel@users.sourceforge.net>
Status: Rejected
Type: Standards Track
Created: 24-Feb-2003
Python-Version: 2.4?
Post-History: 


Abstract

    This PEP proposes adding two new keywords to Python, `synchronize'
    and 'asynchronize'.  

Pronouncement

    This PEP is rejected in favor of PEP 343.

The `synchronize' Keyword

    The concept of code synchronization in Python is too low-level.
    To synchronize code a programmer must be aware of the details of
    the following pseudo-code pattern:

        initialize_lock()

        ...

        acquire_lock()
        try:
            change_shared_data()
        finally:
            release_lock()

    This synchronized block pattern is not the only pattern (more
    discussed below) but it is very common.  This PEP proposes
    replacing the above code with the following equivalent:

        synchronize:
            change_shared_data()

    The advantages of this scheme are simpler syntax and less room for
    user error.  Currently users are required to write code about
    acquiring and releasing thread locks in 'try/finally' blocks;
    errors in this code can cause notoriously difficult concurrent
    thread locking issues.


The `asynchronize' Keyword  

    While executing a `synchronize' block of code a programmer may
    want to "drop back" to running asynchronously momentarily to run
    blocking input/output routines or something else that might take a
    indeterminate amount of time and does not require synchronization.
    This code usually follows the pattern:

        initialize_lock()

        ...

        acquire_lock()
        try:    
            change_shared_data()
            release_lock()             # become async
            do_blocking_io()
            acquire_lock()             # sync again
            change_shared_data2()

        finally:
            release_lock()

    The asynchronous section of the code is not very obvious visually,
    so it is marked up with comments.  Using the proposed
    'asynchronize' keyword this code becomes much cleaner, easier to
    understand, and less prone to error:

        synchronize:
            change_shared_data()

            asynchronize:
               do_blocking_io()

            change_shared_data2()

    Encountering an `asynchronize' keyword inside a non-synchronized
    block can raise either an error or issue a warning (as all code
    blocks are implicitly asynchronous anyway).  It is important to
    note that the above example is *not* the same as:

        synchronize:
            change_shared_data()

        do_blocking_io()

        synchronize:
            change_shared_data2()

    Because both synchronized blocks of code may be running inside the
    same iteration of a loop, Consider:

        while in_main_loop():
            synchronize:
                change_shared_data()

                asynchronize:
                   do_blocking_io()

                change_shared_data2()

    Many threads may be looping through this code.  Without the
    'asynchronize' keyword one thread cannot stay in the loop and
    release the lock at the same time while blocking IO is going on.
    This pattern of releasing locks inside a main loop to do blocking
    IO is used extensively inside the CPython interpreter itself.


Synchronization Targets

    As proposed the `synchronize' and `asynchronize' keywords
    synchronize a block of code.  However programmers may want to
    specify a target object that threads synchronize on.  Any object
    can be a synchronization target.

    Consider a two-way queue object: two different objects are used by
    the same `synchronize' code block to synchronize both queues
    separately in the 'get' method:

        class TwoWayQueue:
            def __init__(self):
                self.front = []
                self.rear = []

            def putFront(self, item):
                self.put(item, self.front)

            def getFront(self):
                item = self.get(self.front)
                return item

            def putRear(self, item):
                self.put(item, self.rear)

            def getRear(self):
                item = self.get(self.rear)
                return item

            def put(self, item, queue):
                synchronize queue:
                    queue.append(item)

            def get(self, queue):
                synchronize queue:
                    item = queue[0]
                    del queue[0]
                    return item

    Here is the equivalent code in Python as it is now without a
    `synchronize' keyword:

        import thread

        class LockableQueue:

            def __init__(self):
                self.queue = []
                self.lock = thread.allocate_lock()

        class TwoWayQueue:
            def __init__(self):
                self.front = LockableQueue()
                self.rear = LockableQueue()

            def putFront(self, item):
                self.put(item, self.front)

            def getFront(self):
                item = self.get(self.front)
                return item

            def putRear(self, item):
                self.put(item, self.rear)

            def getRear(self):
                item = self.get(self.rear)
                return item

            def put(self, item, queue):
                queue.lock.acquire()
                try:
                    queue.append(item)
                finally:
                    queue.lock.release()

            def get(self, queue):
                queue.lock.acquire()
                try:
                    item = queue[0]
                    del queue[0]
                    return item
                finally:
                    queue.lock.release()

    The last example had to define an extra class to associate a lock
    with the queue where the first example the `synchronize' keyword
    does this association internally and transparently.


Other Patterns that Synchronize

    There are some situations where the `synchronize' and
    `asynchronize' keywords cannot entirely replace the use of lock
    methods like `acquire' and `release'.  Some examples are if the
    programmer wants to provide arguments for `acquire' or if a lock
    is acquired in one code block but released in another, as shown
    below.

    Here is a class from Zope modified to use both the `synchronize'
    and `asynchronize' keywords and also uses a pool of explicit locks
    that are acquired and released in different code blocks and thus
    don't use `synchronize':

        import thread
        from ZServerPublisher import ZServerPublisher

        class ZRendevous:

            def __init__(self, n=1):
                pool=[]
                self._lists=pool, [], []

                synchronize:
                    while n > 0:
                        l=thread.allocate_lock()
                        l.acquire()
                        pool.append(l)
                        thread.start_new_thread(ZServerPublisher,
                                                (self.accept,))
                        n=n-1

            def accept(self):
                synchronize:
                    pool, requests, ready = self._lists
                    while not requests:
                        l=pool[-1]
                        del pool[-1]
                        ready.append(l)

                        asynchronize:
                            l.acquire()

                        pool.append(l)

                    r=requests[0]
                    del requests[0]
                    return r

            def handle(self, name, request, response):
                synchronize:
                    pool, requests, ready = self._lists
                    requests.append((name, request, response))
                    if ready:
                        l=ready[-1]
                        del ready[-1]
                        l.release()

    Here is the original class as found in the
    'Zope/ZServer/PubCore/ZRendevous.py' module.  The "convenience" of
    the '_a' and '_r' shortcut names obscure the code:

        import thread
        from ZServerPublisher import ZServerPublisher

        class ZRendevous:

            def __init__(self, n=1):
                sync=thread.allocate_lock()
                self._a=sync.acquire
                self._r=sync.release
                pool=[]
                self._lists=pool, [], []
                self._a()
                try:
                    while n > 0:
                        l=thread.allocate_lock()
                        l.acquire()
                        pool.append(l)
                        thread.start_new_thread(ZServerPublisher,
                                                (self.accept,))
                        n=n-1
                finally: self._r()

            def accept(self):
                self._a()
                try:
                    pool, requests, ready = self._lists
                    while not requests:
                        l=pool[-1]
                        del pool[-1]
                        ready.append(l)
                        self._r()
                        l.acquire()
                        self._a()
                        pool.append(l)

                    r=requests[0]
                    del requests[0]
                    return r
                finally: self._r()

            def handle(self, name, request, response):
                self._a()
                try:
                    pool, requests, ready = self._lists
                    requests.append((name, request, response))
                    if ready:
                        l=ready[-1]
                        del ready[-1]
                        l.release()
                finally: self._r()

    In particular the asynchronize section of the `accept' method is
    not very obvious.  To beginner programmers, `synchronize' and
    `asynchronize' remove many of the problems encountered when
    juggling multiple `acquire' and `release' methods on different
    locks in different `try/finally' blocks.


Formal Syntax

    Python syntax is defined in a modified BNF grammar notation
    described in the Python Language Reference [1].  This section
    describes the proposed synchronization syntax using this grammar:

        synchronize_stmt: 'synchronize' [test] ':' suite
        asynchronize_stmt: 'asynchronize' [test] ':' suite
        compound_stmt: ... | synchronized_stmt | asynchronize_stmt
        
    (The '...' indicates other compound statements elided).


Proposed Implementation

    The author of this PEP has not explored an implementation yet.
    There are several implementation issues that must be resolved.
    The main implementation issue is what exactly gets locked and
    unlocked during a synchronized block.

    During an unqualified synchronized block (the use of the
    `synchronize' keyword without an target argument) a lock could be
    created and associated with the synchronized code block object.
    Any threads that are to execute the block must first acquire the
    code block lock.

    When an `asynchronize' keyword is encountered in a `synchronize'
    block the code block lock is unlocked before the inner block is
    executed and re-locked when the inner block terminates.

    When a synchronized block target is specified the object is
    associated with a lock.  How this is implemented cleanly is
    probably the highest risk of this proposal.  Java Virtual Machines
    typically associate a special hidden lock object with target
    object and use it to synchronized the block around the target
    only.


Backward Compatibility

    Backward compatibility is solved with the new `from __future__'
    Python syntax [2], and the new warning framework [3] to evolve the
    Python language into phasing out any conflicting names that use
    the new keywords `synchronize' and `asynchronize'.  To use the
    syntax now, a developer could use the statement:

        from __future__ import threadsync  # or whatever

    In addition, any code that uses the keyword `synchronize' or
    `asynchronize' as an identifier will be issued a warning from
    Python.  After the appropriate period of time, the syntax would
    become standard, the above import statement would do nothing, and
    any identifiers named `synchronize' or `asynchronize' would raise
    an exception.


PEP 310 Reliable Acquisition/Release Pairs

    PEP 310 [4] proposes the 'with' keyword that can serve the same
    function as 'synchronize' (but no facility for 'asynchronize').
    The pattern:

        initialize_lock()

        with the_lock:
            change_shared_data()

    is equivalent to the proposed:

        synchronize the_lock:
            change_shared_data()

    PEP 310 must synchronize on an exsiting lock, while this PEP
    proposes that unqualified 'synchronize' statements synchronize on
    a global, internal, transparent lock in addition to qualifiled
    'synchronize' statements.  The 'with' statement also requires lock
    initialization, while the 'synchronize' statment can synchronize
    on any target object *including* locks.

    While limited in this fashion, the 'with' statment is more
    abstract and serves more purposes than synchronization.  For
    example, transactions could be used with the 'with' keyword:

        initialize_transaction()

        with my_transaction:
            do_in_transaction()

        # when the block terminates, the transaction is committed.

    The 'synchronize' and 'asynchronize' keywords cannot serve this or
    any other general acquire/release pattern other than thread
    synchronization.


How Java Does It

    Java defines a 'synchronized' keyword (note the grammatical tense
    different between the Java keyword and this PEP's 'synchronize')
    which must be qualified on any object.  The syntax is:

        synchronized (Expression) Block 

    Expression must yeild a valid object (null raises an error and
    exceptions during 'Expression' terminate the 'synchronized' block
    for the same reason) upon which 'Block' is synchronized.


How Jython Does It

    Jython uses a 'synchronize' class with the static method
    'make_synchronized' that accepts one callable argument and returns
    a newly created, synchronized, callable "wrapper" around the
    argument.


Summary of Proposed Changes to Python

    Adding new `synchronize' and `asynchronize' keywords to the
    language.


Risks

    This PEP proposes adding two keywords to the Python language. This
    may break code.

    There is no implementation to test.

    It's not the most important problem facing Python programmers
    today (although it is a fairly notorious one).

    The equivalent Java keyword is the past participle 'synchronized'.
    This PEP proposes the present tense, 'synchronize' as being more
    in spirit with Python (there being less distinction between
    compile-time and run-time in Python than Java).


Dissenting Opinion

    This PEP has not been discussed on python-dev.
        

References

    [1] The Python Language Reference
        http://docs.python.org/reference/

    [2] PEP 236, Back to the __future__, Peters
        http://www.python.org/dev/peps/pep-0236/

    [3] PEP 230, Warning Framework, van Rossum
        http://www.python.org/dev/peps/pep-0230/

    [4] PEP 310, Reliable Acquisition/Release Pairs, Hudson, Moore
        http://www.python.org/dev/peps/pep-0310/


Copyright

    This document has been placed in the public domain.



Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End: