Commits

hideki nara committed 24bd0be

動作ダイアグラムとか追加とか

Comments (0)

Files changed (17)

source/_static/djcelery_diagram/djcelery_diagram_ondemand.jpg

Added
New image

source/_static/djcelery_diagram/djcelery_diagram_periodic.jpg

Added
New image

source/async_usecase.rst

     :depth: 2
     :class: talks-contents
 
+
+.. raw:: html
+ 
+    <a  class="popup-image" id="popup-image" href="http://cdn.thenextweb.com/files/2010/08/stress_one.gif">
+        <img src="http://cdn.thenextweb.com/files/2010/08/stress_one.gif" width="300px"/>
+    </a>
+
+
 作業を貯めてあとでやる
 =======================
 

source/canvas.rst

     >>>> h1.apply_async()
     <AsyncResult: b1cc1a33-4d82-4684-937b-0a836ce6bca0>
 
+複製
+-----
+
+
+.. code-block:: python
+
+    >>> t2=t.clone()
+    >>> t2
+    
+    app.tasks.add(2, 5)
+    >>> t
+    app.tasks.add(2, 5)
+    
+    >>> t2.args = (100,200)
+    >>> t
+    app.tasks.add(2, 5)
+    
+    >>> t2
+    app.tasks.add(100, 200)
+    
+    >>> r=t2.delay()
+    >>> r.result
+    300
+
+
 celey.canvas.Signature = dict + α
 ================================================
 
 partial
 ------------
 
-immutable
-------------
+タスクの引数は部分的に変更可能。
+
+.. code-block:: python
+
+    @task
+    def add(x,y):
+        return x+y
+    
+    
+.. code-block:: python
+    
+    >>> from app.tasks import \*
+    >>> t=add.s(2)
+
+    >>> type(t)
+    <class 'celery.canvas.Signature'>
+    
+    >>> str(t)
+    'app.tasks.add(2)'
+    
+    >>> r=t.delay()
+    >>> r.result
+    
+    TypeError('add() takes exactly 2 arguments (1 given)',)
+    
+    >>> r=t.delay(3)
+    >>> r.result
+    5
+    
+    
+    >>> t.args = t.args + (5,)
+    >>> r.result
+    7
 
 callbacks
 ------------
 
+- タスクの後に実行するタスクを指定可能 -> ワークフロー(Canvas)
+
+immutable
+------------
+
+
+-  普遍 = 親タスクの結果を見ない
+
+.. code-block:: python 
+
+    @task
+    def add(x,y):
+        return x+y
+    
+
+.. code-block:: python 
+
+    @task
+    def linkedon( \*args,**kwargs):
+        return str(args) + str(kwargs)
+    
+    
+.. code-block:: python 
+
+    >>> t=add.apply_async((9,3),link=linkedon.subtask((),{'msg':'fine'}))
+    >>> t.children[0].result
+    "(12,){'msg': 'fine'}"
+    
+    >>> t=add.apply_async((9,3),link=linkedon.subtask((),{'msg':'fine'},immutable=True))
+    >>> t.children[0].result
+    "(){'msg': 'fine'}"
+    
+ショートカット:s(),si()
+----------------------------
+
+.. code-block:: python 
+
+    >>> t=add.apply_async((9,3),link=linkedon.s((),{'msg':'fine'}))
+    >>> t.children[0].result
+    "(12, (), {'msg': 'fine'}){}"
+
+    
+    >>> t=add.apply_async((9,3),link=linkedon.si((),{'msg':'fine'}))
+    >>> t.children[0].result
+    "((), {'msg': 'fine'}){}"
+
+
+
 Groups(グループ内のTaskを平行して実行する)
 ========================================================
 
+まとめられる。
+
+.. code-block:: python
+
+    >>> from celery import group
+    >>> res=group(add.s(1,2),add.s(2,3),add.s(3,4))()
+
+    >>> res.result
+
+    Traceback (most recent call last):
+    File "<console>", line 1, in <module>
+    AttributeError: 'GroupResult' object has no attribute 'result'
+
+    >>> res.get()
+    [3, 5, 7]
+
+GroupResult
+
+.. code-block:: python
+
+    >>> type(res)
+    <class 'celery.result.GroupResult'>
+
+    >>> res.id
+    '7bbf1f1b-eafa-4924-a8a2-5d924ff1f131'
+
+    >>> res.children
+    (<AsyncResult: 56bca686-7eec-405a-a030-87cf4ede955f>, 
+    <AsyncResult: 41d543d4-400b-4727-a388-02813605d7c2>, 
+    <AsyncResult: 1c3f0564-ac7d-4848-9caf-f72e6244ae23>)
+
+
+タスク付け足し
+
+.. code-block:: python
+
+    >>> g.tasks = g.tasks + (add.s(4,5),)
+    >>> g.tasks
+    (app.tasks.add(1, 2), app.tasks.add(2, 3), app.tasks.add(3, 4), app.tasks.add(4, 5))
+
+    >>> res=g()
+    >>> res.get()
+    [3, 5, 7, 9]
+
+
+サブタスクg を用意
+
+.. code-block:: python
+
+    >>> g=group(add.s(1,1),add.s(2,2))
+
+同期呼び出し
+
+.. code-block:: python
+
+    >>> res=g.apply()
+    >>> type(res)
+    <class 'celery.result.GroupResult'>
+
+    >>> res.id
+    'eb068a37-a463-4ea4-9366-19e0fba95c8d'
+
+    >>> res.children
+    [<EagerResult: 302ee475-19d0-4347-b131-b378850c69b7>, <EagerResult: 36dc241d-1e9f-4057-a011-c64003010e58>]
+
+ここで、 グループタスク eb068a37-a463-4ea4-9366-19e0fba95c8d はProduceされません。
+
+
+非同期呼び出し
+
+.. code-block:: python
+
+
+    >>> res = g.apply_async()
+
+    >>> type(res)
+    <class 'celery.result.GroupResult'>
+
+    >>> res.id
+    '45356237-100b-442b-80dd-757faf015494'
+
+    >>> res.children
+    (<AsyncResult: 302ee475-19d0-4347-b131-b378850c69b7>, <AsyncResult: 36dc241d-1e9f-4057-a011-c64003010e58>)
+
+ここで、 グループタスク 45356237-100b-442b-80dd-757faf01549 はProduceされます。
+
+呼び出しグループタスク自体は異なるが、サブタスクは同じ
+
+.. code-block:: python
+
+    >>> g().id == g.apply_async().id
+    False
+
+    >>> g.delay().id == g.apply_async().id
+    False
+    
+    >>> g().children == g.apply_async().children
+    True
+
+    >>> g.delay().children == g.apply_async().children
+    True
+
+
+まぜるとか
+
+.. code-block:: python
+
+    >>> from app.tasks import *
+    >>> from celery import group
+    >>> from celery import chain
+
+    >>> g=group(add.s(1,2),add.s(2,3),add.s(3,4))
+    >>> c=chain(add.s(2, 2), add.s(4), add.s(8))
+
+    >>> g2=group(g,c)
+    >>> res=g2()
+
+    >>> type(res)
+    <class 'celery.result.GroupResult'>
+
+    >>> res.children
+    (<AsyncResult: 37643ceb-f8e6-4b4d-a0dc-4a99f7214f07>, <AsyncResult: 0be55acc-596d-4297-abf8-0a1207467d9f>)
+
+    >>> map(lambda r : r.get(), res.children)
+    [<GroupResult: 37643ceb-f8e6-4b4d-a0dc-4a99f7214f07 [cc123480-7995-4fd9-9618-41fb5b7b095a, 6b4db3a6-7253-4036-8873-2c69ab0e9879, f24740b1-5cb0-4905-aed5-ac31a2096eb9]>, 
+    16]
+
+    >>> map(lambda r : r.get(), res.children[0].children)
+    [[3, 5, 7]] 
+ 
 Chains( Taskのコールバークを連続的に行う)
 =================================================
 
+(((2+2) + 4 )  + 8 ) = 16 をやってみる
+
+.. code-block:: python 
+
+    >>> from celery import chain
+    >>> res = chain(add.s(2, 2), add.s(4), add.s(8))()
+
+    >>> res.result
+    16
+
+こうもかける :
+
+.. code-block:: python 
+
+    >>> res=(add.s(2, 2) | add.s(4) | add.s(8))()
+    >>> res.result
+    16
+
+immutable (状態変更不可能)だとchainする意味はあるか?
+
+.. code-block:: python 
+
+    >>> res=(add.si(1, 1) | add.si(4,1) | add.si(8,1))()
+    >>> res.result
+    9
+
+    >>> res.parent.result
+    5
+
+    >>> res.parent.parent.result
+    2
+
+サブタスク c を作る
+
+.. code-block:: python
+
+
+    >>> c=(add.s(2, 2) | add.s(4) | add.s(8))
+    >>> type(c)
+    <class 'celery.canvas.chain'>
+
+非同期呼び出し
+
+.. code-block:: python
+
+    >>> res=c()
+    >>> type(res)
+    <class 'celery.result.AsyncResult'>
+
+    >>> res.id
+    '0d7d902e-eb95-4bdd-b984-997397355f44'
+
+    >>> res.children == None
+    True
+
+    >>> print res.get(), res.parent.get(), res.parent.parent.get()
+    16 8 4
+
+    >>> c.apply_async()
+    <AsyncResult: 0d7d902e-eb95-4bdd-b984-997397355f44>
+
+同期呼び出し
+
+.. code-block:: python
+
+    >>> res=c.apply()
+    >>> type(res)
+    <class 'celery.result.EagerResult'>
+
+    >>> print res.get(), res.parent.get(), res.parent.parent.get()
+    16 8 4
+
+    >>> res.id
+    '0d7d902e-eb95-4bdd-b984-997397355f44'
+
+このidは先に非同期呼び出しした物と同じID。深い!
+ただし、別途subtaskを作り直して、apply()するとProduceされません。
+
+
+その他
+========
+
 Chords (コールバック付きGroups)
-=================================================
+-------------------------------------------------
 
 - header group + body group
 
 Map (引数リストの要素に対してTaskをを作る)
-===============================================================
+-------------------------------------------------
 
 - map( callbable, list ) みたい
 
 StarMap (Mapへのそれぞれの引数が \*args )
-===============================================================
+-------------------------------------------------
 
 Chunks(引数の一覧を等分割してGroup化)
-===============================================================
+-------------------------------------------------
 

source/celery.rst

 Celeryとは?
 ====================================
 
+- http://celeryproject.org/
 
 
     Celery is an asynchronous task queue/job queue based on distributed message passing.
 Task Queue(タスクキュー)
 ====================================
 
+
+- http://docs.celeryproject.org/en/latest/getting-started/introduction.html#what-is-a-task-queue
+
+    Task queues are used as a mechanism to distribute work across threads or machines.
+
+    A task queue’s input is a unit of work, 
+    called a task, dedicated worker processes then constantly monitor the queue for new work to perform.
+
+    Celery communicates via messages using a broker to mediate between clients and workers. 
+    To initiate a task a client puts a message on the queue, 
+    the broker then delivers the message to a worker.
+
+    A Celery system can consist of multiple workers and brokers, 
+    giving way to high availability and horizontal scaling.
+
+    Celery is written in Python, 
+    but the protocol can be implemented in any language. 
+    So far there’s RCelery for the Ruby programming language, and a PHP client, 
+    but language interoperability can also be achieved by using webhooks.
+
 Asynchronous(非同期)
 ====================================
 
 Distributed Message Passing(分散メッセージ送信)
 ========================================================
 
+- メッセージ
+
+    - 「C++というオブジェクト指向言語では「メンバ関数の呼び出し」と呼びます。しかしObjective-Cでは明確に通信(メッセージ)と言われています。」
+
+- 分散
+
+    - 間にブローカーが入れば分散なのかな?   
+    - RabbitMQ  : http://www.rabbitmq.com/distributed.html
+    - WebSphere MQ (+ JMS?)
+    - MSMQ ( + COM+ Event?)
+
+
 Real=Time & Scheduling(リアルタイム/スケジュール)
 =======================================================
 

source/celery_constructs.rst

 Kombu 
 ====================================
 
+- http://kombu.readthedocs.org/en/latest/index.html
+
 Message Brokerage
 ------------------------------------------
 

source/concurrency.rst

     <AsyncResult: 55513afb-3c1a-41bc-b296-8b64026c30fd>
 
 
-.. figure::  _static/concurrency/concurrency_result.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/concurrency/concurrency_result.png"/>
+        <img src="_static/concurrency/concurrency_result.png"/>
+    </a>
 
 
 --concurrency指定しなくても1プロセスは--billiard-forkされます

source/djcelery_diagram.rst

     :depth: 2
     :class: talks-contents
 
-TODO
+On Demand
 ==============
 
-TODO:Djang-celery のダイアグラムを作る
+.. raw:: html
+ 
+    <a  class="popup-image" id="popup-image" href="_static/djcelery_diagram/djcelery_diagram_ondemand.jpg">
+        <img src="_static/djcelery_diagram/djcelery_diagram_ondemand.jpg" width="300px"/>
+    </a>
+
+taskは少なくとも同じコード
+------------------------------
+
+各ノードは別のコンピュータでかまわない(分散)
 ---------------------------------------------
+
+Periodic
+==============
+
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/djcelery_diagram/djcelery_diagram_periodic.jpg">
+        <img src="_static/djcelery_diagram/djcelery_diagram_periodic.jpg" width="300px"/>
+    </a>
+
+
+--beat --scheduler=djcelery.schedulers.DatabaseScheduler
+------------------------------------------------------------------
+

source/djkombu.rst

             print e
             pass
 
-.. image:: _static/run/simple01/admin_list.png
-
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/run/simple01/admin_list.png"/>
+        <img src="_static/run/simple01/admin_list.png"/>
+    </a>
 
 何が起きているか?
 =====================
 Queue
 ^^^^^^^^^^^
 
-.. image:: _static/run/simple01/admin_queue_list.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/run/simple01/admin_queue_list.png"/>
+        <img src="_static/run/simple01/admin_queue_list.png"/>
+    </a>
 
 タスクdelay呼び出し:Messageに追加
 -------------------------------------------------------------------------
 Message
 ^^^^^^^^^^
 
-.. image:: _static/run/simple01/admin_message_list.png  
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/run/simple01/admin_message_list.png"/>
+        <img src="_static/run/simple01/admin_message_list.png"/>
+    </a>
 
-.. image:: _static/run/simple01/admin_message.png  
-
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/run/simple01/admin_message.png"/>
+        <img src="_static/run/simple01/admin_message.png"/>
+    </a>
 
 ワーカーがMessageからレコード取り出しメッセージ復元
 -------------------------------------------------------
 AsyncResult =  TaksMeta モデル( task states )
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. image:: _static/run/simple01/admin_task_state.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/run/simple01/admin_task_state.png"/>
+        <img src="_static/run/simple01/admin_task_state.png"/>
+    </a>
 
 
 
-
 Messaging framework for Python
 ================================
 
+- http://kombu.readthedocs.org/en/latest/introduction.html
+
 :: 
 
     The aim of Kombu is to make messaging in Python as easy as possible 
 作業分担
 =========
 
-- http://kombu.readthedocs.org/en/latest/userguide/examples.html
-
 アプリケーション側 : Celery
 ---------------------------------------------
 
 メッセージイングブローカー側 : Kombu
 ---------------------------------------------
 
+Kombu:タスクキューの例
+-------------------------
 
-.. figure:: _static/kombu-title-image.png
-    :name:  popup-image
-    :class: popup-image
+- http://bit.ly/kombu_task_queue
 +hdknr/@hdknr
 ---------------------
 
+- http://gplus.to/hdknr
 - https://twitter.com/hdknr
 
 http://www.linkedin.com/in/hdknr/ja
 デジタルサイネージ(っぽい) (2001年〜)
 ------------------------------------------
 
+.. raw:: html
 
-.. figure:: _static/me/vision.jpg
-
+    <a  class="figure" id="card_fan_out" href="_static/me/vision.jpg">
+        <img src="_static/me/vision.jpg" width="300px" />
+    </a>
 
 メールマーケティング(2000年〜)
 ------------------------------
 OpenID Connect (2008年〜)
 -------------------------------------
 
-.. figure:: _static/me/OpenIDConnect-Map-v7.png
-    
-    OpenID Connect Protocol
-    ( from http://openid.net/connect/ )
+.. raw:: html
+
+    <a  class="figure" id="card_fan_out" href="_static/me/OpenIDConnect-Map-v7.png">
+        <img src="_static/me/OpenIDConnect-Map-v7.png" width="300px" />
+    </a>
 
 - http://openid.net/wg/connect/
 - #idcon 14th ~ BYOResume or DEAD! ~ ( http://idcon.doorkeeper.jp/events/1689 )

source/monitor.rst

 django-celery : Django Admin UI
 ==================================================
 
-.. figure:: _static/monitor/monitor_admin_taskstate_list.png 
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/monitor/monitor_admin_taskstate_list.png"/>
+        <img src="_static/monitor/monitor_admin_taskstate_list.png"/>
+    </a>
+
 
 djcelery.models.TaskState
 ----------------------------
 .. figure:: _static/monitor/monitor_console_taskstate.png
 
 .. figure:: _static/monitor/monitor_console_taskresult.png          
+

source/periodic.rst

 Periodic : タスク
 ----------------------------
 
-.. figure:: _static/periodic/periodic_task.png
+タスクの設定
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    タスクの設定
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_task.png"/>
+        <img src="_static/periodic/periodic_task.png"/>
+    </a>
 
-.. figure:: _static/periodic/periodic_arguments.png
+タスクへの引数設定
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    タスクへの引数設定
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_arguments.png"/>
+        <img src="_static/periodic/periodic_arguments.png"/>
+    </a>
 
-.. figure:: _static/periodic/periodic_options.png
+タスクの実行オプションの設定
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    タスクの実行オプションの設定
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_options.png"/>
+        <img src="_static/periodic/periodic_options.png"/>
+    </a>
+
 
 Interval : 毎秒/分/時 
 ----------------------------
 
-.. figure:: _static/periodic/periodic_interval.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_interval.png"/>
+        <img src="_static/periodic/periodic_interval.png"/>
+    </a>
 
+5分が長いので、1分に変更し、引数にメッセージも入れた。途中変更も反映。
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. figure:: _static/periodic/periodic_result.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_result.png"/>
+        <img src="_static/periodic/periodic_result.png"/>
+    </a>
 
-    5分が長いので、1分に変更し、引数にメッセージも入れた。途中変更も反映。
 
 Crontab : crontab時間指定
 ----------------------------------
 
-.. figure:: _static/periodic/periodic_crontab.png
 
-.. figure:: _static/periodic/periodic_crontab_task.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_crontab.png"/>
+        <img src="_static/periodic/periodic_crontab.png"/>
+    </a>
 
-    IntervalかCrontabかはどちらか一方でのみ指定
+IntervalかCrontabかはどちらか一方でのみ指定
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. figure:: _static/periodic/periodic_crontab_result.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_crontab_task.png"/>
+        <img src="_static/periodic/periodic_crontab_task.png"/>
+    </a>
+
+
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/periodic/periodic_crontab_result.png"/>
+        <img src="_static/periodic/periodic_crontab_result.png"/>
+    </a>
 
 
 Django-Admin で変更すればスケジュール変わります

source/routing.rst

         Queue('soup', Exchange('soup'), routing_key='soup'),
     )
 
+AMQP: RabbitMQのルーティングを制御
+========================================
 
+
+
+

source/security.rst

 - プロジェクトごとにバーチャル環境を用意して、BROKER_URLが異なるようにする
 
 
-.. figure:: _static/security/security_rabbitmq_virtual_host.png
+.. raw:: html
+ 
+    <a  class="figure" id="periodic" href="_static/security/security_rabbitmq_virtual_host.png"/>
+        <img src="_static/security/security_rabbitmq_virtual_host.png"/>
+    </a>
+
 
 .. code-block:: python
 

source/signal.rst

 シグナルいろいろ
 ====================
 
+- http://celery.readthedocs.org/en/latest/userguide/signals.html
+
 Task
 -----
 
+- task_sent
+- task_prerun
+- task_postrun
+- task_success
+- task_failure
+- task_revoked
+
+
 Worker
 ----------
 
+
+- celeryd_after_setup
+- celeryd_init
+- worker_init
+- worker_ready
+- worker_process_init
+- worker_shutdown
+
+
 Celerybeat
 --------------------
 
+- beat_init
+- beat_embedded_init
+
+
 Eventlet
 ------------
 
+
+- eventlet_pool_started
+- eventlet_pool_preshutdown
+- eventlet_pool_postshutdown
+- eventlet_pool_apply
+
+
 Logging
 ------------
 
+- setup_logging
+- after_setup_logger
+- after_setup_task_logger
+

source/task_queue.rst

 基本的デザインパターン
 ------------------------------------
 
+簡単なFIFOキュー
+^^^^^^^^^^^^^^^^^^
+
+- 「 Queue モジュールは、多生産者-多消費者(multi-producer, multi-consumer)キューを実装します。
+
+   これは、複数のスレッドの間で情報を安全に交換しなければならないときのマルチスレッドプログラミングで特に有益です。
+
+   このモジュールの Queue クラスは、必要なすべてのロックセマンティクスを実装しています。
+
+   これはPythonのスレッドサポートの状況に依存します。 threading モジュールを参照してください。」
+    
+    - http://www.python.jp/doc/release/library/queue.html
+
+.. code-block:: python
+
+    import Queue
+    
+    q = Queue.Queue()
+    
+    for i in range(5):
+        q.put(i)
+    
+    while not q.empty():
+        print q.get()
+
+しかし要件が複雑になるにつれて....
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- http://www.doughellmann.com/PyMOTW-ja/Queue/
+
 enqueue/dequue
 ------------------------------------