SlideShare a Scribd company logo
1 of 176
Download to read offline
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators:
The Final Frontier
David Beazley (@dabeaz)
http://www.dabeaz.com
Presented at PyCon'2014, Montreal
1
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
2
• GeneratorTricks for Systems Programmers (2008)
http://www.dabeaz.com/generators/
• A flying leap into generator awesomeness
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
3
• A Curious Course on Coroutines and Concurrency (2009)
http://www.dabeaz.com/coroutines/
• Wait, wait? There's more than iteration?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Today's Installment
4
• Everything else you ever wanted to know
about generators, but were afraid to try
• Part 3 of a trilogy
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Requirements
5
• You need Python 3.4 or newer
• No third party extensions
• Code samples and notes
http://www.dabeaz.com/finalgenerator/
• Follow along if you dare!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Disclaimer
6
• This is an advanced tutorial
• Assumes general awareness of
• Core Python language features
• Iterators/generators
• Decorators
• Common programming patterns
• I learned a LOT preparing this
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Will I Be Lost?
7
• Although this is the third part of a series, it's
mostly a stand-alone tutorial
• If you've seen prior tutorials, that's great
• If not, don't sweat it
• Be aware that we're focused on a specific use of
generators (you just won't get complete picture)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Focus
8
• Material in this tutorial is probably not
immediately applicable to your day job
• More thought provoking and mind expanding
• from __future__ import future
practical
utility
bleeding
edge
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part I
9
Preliminaries - Generators and Coroutines
(rock)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators 101
• yield statement defines a generator function
10
def countdown(n):
while n > 0:
yield n
n -= 1
• You typically use it to feed iteration
for x in countdown(10):
print('T-minus', x)
• A simple, yet elegant idea
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Under the Covers
11
• Generator object runs in response to next()
>>> c = countdown(3)
>>> c
<generator object countdown at 0x10064f900>
>>> next(c)
3
>>> next(c)
2
>>> next(c)
1
>>> next(c)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
StopIteration
>>>
• StopIteration raised when function returns
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
12
• Generators as "iterators" misses the big picture
• There is so much more to yield
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators as Pipelines
• Stacked generators result in processing pipelines
• Similar to shell pipes in Unix
13
generator
input
sequence
for x in s:generator generator
def process(sequence):
for s in sequence:
... do something ...
yield item
• Incredibly useful (see prior tutorial)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines 101
• yield can receive a value instead
14
def receiver():
while True:
item = yield
print('Got', item)
• It defines a generator that you send things to
recv = receiver()
next(recv) # Advance to first yield
recv.send('Hello')
recv.send('World')
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines and Dataflow
15
• Coroutines enable dataflow style processing
source coroutine
coroutine
send() send()
• Publish/subscribe, event simulation, etc.
coroutine
coroutine
send()
send()
coroutine
send()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Fundamentals
• The yield statement defines a generator function
16
def generator():
...
... yield ...
...
• The mere presence of yield anywhere is enough
• Calling the function creates a generator instance
>>> g = generator()
>>> g
<generator object generator at 0x10064f120>
>>>
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advancing a Generator
• next(gen) - Advances to the next yield
17
def generator():
...
...
yield item
...
• Returns the yielded item (if any)
• It's the only allowed operation on a newly
created generator
• Note: Same as gen.__next__()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Sending to a Generator
• gen.send(item) - Send an item to a generator
18
def generator():
...
item = yield
...
...
yield value
• Wakes at last yield, returns sent value
• Runs to the next yield and emits the value
g = generator()
next(g) # Advance to yield
value = g.send(item)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Closing a Generator
• gen.close() - Terminate a generator
19
def generator():
...
try:
yield
except GeneratorExit:
# Shutting down
...
• Raises GeneratorExit at the yield
• Only allowed action is to return
• If uncaught, generator silently terminates
g = generator()
next(g) # Advance to yield
g.close() # Terminate
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Raising Exceptions
• gen.throw(typ [, val [,tb]]) - Throw exception
20
def generator():
...
try:
yield
except RuntimeError as e:
...
...
yield val
• Raises exception at yield
• Returns the next yielded value (if any)
g = generator()
next(g) # Advance to yield
val = g.throw(RuntimeError,
'Broken')
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator ReturnValues
• StopIteration raised on generator exit
21
def generator():
...
yield
...
return result
• Return value (if any) passed with exception
• Note: Python 3 only behavior (in Python 2,
generators can't return values)
g = generator()
try:
next(g)
except StopIteration as e:
result = e.value
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Delegation
• yield from gen - Delegate to a subgenerator
22
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Allows generators to call other generators
• Operations take place at the current yield
• Return value (if any) is returned
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Delegation Example
• Chain iterables together
23
def chain(x, y):
yield from x
yield from y
• Example:
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> for x in chain(a, b):
... print(x,end=' ')
...
1 2 3 4 5 6
>>> c = [7,8,9]
>>> for x in chain(a, chain(b, c)):
... print(x, end=' ')
...
1 2 3 4 5 6 7 8 9
>>>
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Mini-Reference
• Generator instance operations
24
gen = generator()
next(gen) # Advance to next yield
gen.send(item) # Send an item
gen.close() # Terminate
gen.throw(exc, val, tb) # Raise exception
result = yield from gen # Delegate
• Using these, you can do a lot of neat stuff
• Generator definition
def generator():
...
yield
...
return result
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 2
25
and now for something completely different
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Common Motif
• Consider the following
26
f = open()
...
f.close()
lock.acquire()
...
lock.release()
db.start_transaction()
...
db.commit()
start = time.time()
...
end = time.time()
• It's so common, you'll see it everywhere!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Managers
• The 'with' statement
27
with open(filename) as f:
statement
statement
...
with lock:
statement
statement
...
• Allows control over entry/exit of a code block
• Typical use: everything on the previous slide
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• It's easy to make your own (@contextmanager)
28
import time
from contextlib import contextmanager
@contextmanager
def timethis(label):
start = time.time()
try:
yield
finally:
end = time.time()
print('%s: %0.3f' % (label, end-start)
• This times a block of statements
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
29
• Usage
with timethis('counting'):
n = 1000000
while n > 0:
n -= 1
• Output
counting: 0.023
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Another example: temporary directories
30
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
with tempdir() as dirname:
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Whoa,Whoa, Stop!
• Another example: temporary directories
31
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
outdir = tempfile.mkdtemp()
try:
yield outdir
finally:
shutil.rmtree(outdir)
• Example
with tempdir() as dirname:
...
What is this?
• Not iteration
• Not dataflow
• Not concurrency
• ????
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Under the covers
32
with obj:
statements
statements
statements
...
statements
• If an object implements these methods it can
monitor entry/exit to the code block
obj.__enter__()
obj.__exit__()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager
• Implementation template
33
class Manager(object):
def __enter__(self):
return value
def __exit__(self, exc_type, val, tb):
if exc_type is None:
return
else:
# Handle an exception (if you want)
return True if handled else False
• Use:
with Manager() as value:
statements
statements
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager Example
• Automatically deleted temp directories
34
import tempfile
import shutil
class tempdir(object):
def __enter__(self):
self.dirname = tempfile.mkdtemp()
return self.dirname
def __exit__(self, exc, val, tb):
shutil.rmtree(self.dirname)
• Use:
with tempdir() as dirname:
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Alternate Formulation
• @contextmanager is just a reformulation
35
import tempfile, shutil
from contextlib import contextmanager
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
finally:
shutil.rmtree(dirname)
• It's the same code, glued together differently
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• How does it work?
36
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
finally:
shutil.rmtree(dirname)
• Think of "yield" as scissors
• Cuts the function in half
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• Each half maps to context manager methods
37
@contextmanager
def tempdir():
dirname = tempfile.mkdtemp()
try:
yield dirname
statements
statements
statements
...
finally:
shutil.rmtree(dirname)
__enter__
__exit__
user statements ('with' block)
• yield is the magic that makes it possible
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• There is a wrapper class (Context Manager)
38
class GeneratorCM(object):
def __init__(self, gen):
self.gen = gen
def __enter__(self):
...
def __exit__(self, exc, val, tb):
...
• And a decorator
def contextmanager(func):
def run(*args, **kwargs):
return GeneratorCM(func(*args, **kwargs))
return run
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• enter - Run the generator to the yield
39
class GeneratorCM(object):
def __init__(self, gen):
self.gen = gen
def __enter__(self):
return next(self.gen)
def __exit__(self, exc, val, tb):
...
• It runs a single "iteration" step
• Returns the yielded value (if any)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• exit - Resumes the generator
40
class GeneratorCM(object):
...
def __exit__(self, etype, val, tb):
try:
if etype is None:
next(self.gen)
else:
self.gen.throw(etype, val, tb)
raise RuntimeError("Generator didn't stop")
except StopIteration:
return True
except:
if sys.exc_info()[1] is not val: raise
• Either resumes it normally or raises exception
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Full Disclosure
• Actual implementation is more complicated
• There are some nasty corner cases
• Exceptions with no associated value
• StopIteration raised inside a with-block
• Exceptions raised in context manager
• Read source and see PEP-343
41
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Discussion
• Why start with this example?
• A completely different use of yield
• Being used to reformulate control-flow
• It simplifies programming for others (easy
definition of context managers)
• Maybe there's more... (of course there is)
42
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
43
Call me, maybe
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
44
Call me, maybe
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Async Processing
• Consider the following execution model
45
def func(args):
...
...
...
return result
run_async(func, args)
main thread
• Examples: Run in separate process or thread,
time delay, in response to event, etc.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example:Thread Pool
46
from concurrent.futures import ThreadPoolExecutor
def func(x, y):
'Some function. Nothing too interesting'
import time
time.sleep(5)
return x + y
pool = ThreadPoolExecutor(max_workers=8)
fut = pool.submit(func, 2, 3)
r = fut.result()
print('Got:', r)
• Runs the function in a separate thread
• Waits for a result
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
47
>>> fut = pool.submit(func, 2, 3)
>>> fut
<Future at 0x1011e6cf8 state=running>
>>>
• Future - A result to be computed later
• You can wait for the result to return
>>> fut.result()
5
>>>
• However, this blocks the caller
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
48
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
result = fut.result()
print('Got:', result)
• Alternatively, you can register a callback
• Triggered upon completion
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exceptions
49
>>> fut = pool.submit(func, 2, 'Hello')
>>> fut.result()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 395, in result
return self.__get_result()
File "/usr/local/lib/python3.4/concurrent/futures/_base.py",
line 354, in __get_result
raise self._exception
File "/usr/local/lib/python3.4/concurrent/futures/thread.py",
line 54, in run
result = self.fn(*self.args, **self.kwargs)
File "future2.py", line 6, in func
return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>>
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures w/Errors
50
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• Error handling with callbacks
• Exception propagates out of fut.result() method
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
51
def run():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(result_handler)
def result_handler(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• Consider the structure of code using futures
• Meditate on it... focus on the code.
• This seems sort of familiar
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Callback Hell?
52
• No, no, no.... keep focusing.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
53
def entry():
fut = pool.submit(func, 2, 3)
fut.add_done_callback(exit)
def exit(fut):
try:
result = fut.result()
print('Got:', result)
except Exception as e:
print('Failed: %s: %s' % (type(e).__name__, e))
• What if the function names are changed?
• Wait! This is almost a context manager (yes)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Futures
54
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
run_inline_future(do_func)
• Thought: Maybe you could do that yield trick
• The extra callback function is eliminated
• Now, just one "simple" function
• Inspired by @contextmanager
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
55
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
56
• This twisted idea has been used before...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Preview
57
t = Task(gen)
• There are two separate parts
• Part 1:Wrapping generators with a "task"
• Part 2: Implementing some runtime code
run_inline_future(gen)
• Forewarning: It will bend your mind a bit
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
58
• Will continue to use threads for examples
• Mainly because they're easy to work with
• And I don't want to get sucked into an event loop
• Don't dwell on it too much
• Key thing: There is some background processing
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
59
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
• Problem: Stepping through a generator
• Involves gluing callbacks and yields together
def do_func(x, y):
yield pool.submit(func, x, y)
result =
print('Got:', result)
add_done_callback()
cut
enter
exit
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
60
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
61
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Task class wraps
around and represents
a running generator.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
62
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Advance the
generator to the next
yield, sending in a
value (if any)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
63
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Attach a callback to
the produced Future
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
64
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None):
try:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
result = fut.result()
! self.step(result)
Collect result and
send back into the
generator
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
65
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y):
time.sleep(1)
return x + y
def do_func(x, y):
result = yield pool.submit(func, x, y)
print('Got:', result)
t = Task(do_func(2, 3))
t.step()
• Try it:
• Output:
Got: 5
• Yes, it works
Note: must initiate
first step of the task
to get it to run
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
66
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y):
time.sleep(1)
return x + y
def do_many(n):
while n > 0:
result = yield pool.submit(func, n, n)
print('Got:', result)
n -= 1
t = Task(do_many(10))
t.step()
• More advanced: multiple yields/looping
• Yes, this works too.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
67
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
68
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
send() or throw()
depending on
success
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
69
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
def _wakeup(self, fut):
try:
result = fut.result()
self.step(result, None)
except Exception as exc:
self.step(None, exc)
Catch exceptions
and pass to next
step as appropriate
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Error Example
70
def do_func(x, y):
try:
result = yield pool.submit(func, x, y)
print('Got:', result)
except Exception as e:
print('Failed:', repr(e))
t = Task(do_func(2, 'Hello'))
t.step()
• Try it:
• Output:
Failed: TypeError("unsupported operand type(s) for +:
'int' and 'str'",)
• Yep, that works too.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
71
• This whole thing is rather bizarre
• Execution of the inlined future takes place all on
its own (concurrently with other code)
• The normal rules don't apply
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Consider
72
def recursive(n):
yield pool.submit(time.sleep, 0.001)
print('Tick:', n)
Task(recursive(n+1)).step()
Task(recursive(0)).step()
• Infinite recursion?
• Output:
Tick: 0
Tick: 1
Tick: 2
...
Tick: 1662773
Tick: 1662774
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 4
73
yield from yield from yield from yield from future
(maybe)
source: @UrsulaWJ
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
74
• Focus on the future
• Not the past
• Not now
• Yes, the future.
• No, really, the future.
(but not the singularity)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
75
class Task:
def __init__(self, gen):
self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
pass
...
generator must only
produce Futures
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
76
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• It's trying to delay the execution of a user-
supplied inlined future until later.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
77
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• No
Traceback (most recent call last):
...
AttributeError: 'generator' object has no attribute
'add_done_callback'
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
78
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• This is busted
• gen is a generator, not a Future
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
79
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
for f in gen:
yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• Idea: Just iterate the generator manually
• Make it produce the required Futures
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
80
def after(delay, gen):
'''
Run an inlined future after a time delay
'''
yield pool.submit(time.sleep, delay)
for f in gen:
yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• No luck.The result gets lost somewhere
Got: None
• Hmmm.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
81
def after(delay, gen):
yield pool.submit(time.sleep, delay)
result = None
try:
while True:
f = gen.send(result)
result = yield f
except StopIteration:
pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!
Got: 5
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
82
def after(delay, gen):
yield pool.submit(time.sleep, delay)
result = None
try:
while True:
f = gen.send(result)
result = yield f
except StopIteration:
pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!
Got: 5
manual running of
generator with
results (ugh!)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (4th Attempt)
83
def after(delay, gen):
yield pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• A better solution: yield from
• 'yield from' - Runs the generator for you
Got: 5
• And it works! (yay!)
• Awesome
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
PEP 380
• yield from gen - Delegate to a subgenerator
84
def generator():
...
yield value
...
return result
def func():
result = yield from generator()
• Transfer control to other generators
• Operations take place at the current yield
• Far more powerful than you might think
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Conundrum
85
def after(delay, gen):
yield pool.submit(time.sleep, delay)
yield from gen
• "yield" and "yield from"?
• Two different yields in the same function
• Nobody will find that confusing (NOT!)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
86
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
87
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
• No. 'yield' and 'yield from' not interchangeable:
Traceback (most recent call last):
...
TypeError: 'Future' object is not iterable
>>>
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
??????
88
(Can it be made to work?)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Iterable Futures
89
def patch_future(cls):
def __iter__(self):
! if not self.done():
yield self
return self.result()
cls.__iter__ = __iter__
from concurrent.futures import Future
patch_future(Future)
• A simple ingenious patch
• It makes all Future instances iterable
• They simply produce themselves and the result
• It magically makes 'yield from' work!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
All Roads Lead to Future
90
yield self
• Future is the only thing
that actually yields
• Everything else delegates
using 'yield from'
• Future terminates the
chain
generator
generator
generator
Future
yield from
yield from
yield from
run()
next(gen)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Decorator
91
import inspect
def inlined_future(func):
assert inspect.isgeneratorfunction(func)
return func
• Generators yielding futures is its own world
• Probably a good idea to have some demarcation
• Does nothing much at all, but serves as syntax
@inlined_future
def after(delay, gen):
yield from pool.submit(time.sleep, delay)
yield from gen
• Alerts others about what you're doing
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Wrangling
92
t = Task(gen)
t.step()
• The "Task" object is just weird
• No way to obtain a result
• No way to join with it
• Or do much of anything useful at all
Task
runs
????
t.step()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
93
class Task(Future):
def __init__(self, gen):
!super().__init__()
! self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
self.set_result(exc.value)
• This tiny tweak makes it much more interesting
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
94
class Task(Future):
def __init__(self, gen):
!super().__init__()
! self._gen = gen
def step(self, value=None, exc=None):
try:
if exc:
fut = self._gen.throw(exc)
else:
fut = self._gen.send(value)
fut.add_done_callback(self._wakeup)
except StopIteration as exc:
self.set_result(exc.value)
• This tiny tweak makes it much more interesting
A Task is a Future
Set its result upon
completion
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
95
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
• Obtaining the result of task
• So, you create a task that runs a generator
producing Futures
• The task is also a Future
• Right. Got it.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
96
@inlined_future
def do_func(x, y):
result = yield pool.submit(func, x, y)
return result
t = Task(do_func(2,3))
t.step()
...
print("Got:", t.result())
class Task(Future):
...
def step(self, value=None, exc=None):
try:
...
except StopIteration as exc:
self.set_result(exc.value)
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Runners
97
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
def run_inline_future(fut):
t = start_inline_future(fut)
return t.result()
• You can make utility functions to hide details
result = run_inline_future(do_func(2,3))
print('Got:', result)
• Example: Run an inline future to completion
• Example: Run inline futures in parallel
t1 = start_inline_future(do_func(2, 3))
t2 = start_inline_future(do_func(4, 5))
result1 = t1.result()
result2 = t2.result()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Step Back Slowly
98
• Built a generator-based task system for threads
Tasks
@inline_future
run_inline_future()
Threads
pools
concurrent.future
Futures
submit
result
• Execution of the future hidden in background
• Note: that was on purpose (for now)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
99
• Ideas are the foundation asyncio coroutines
Tasks
@coroutine
run_until_complete()
Event Loop
asyncio
Futures
result
• In fact, it's almost exactly the same
• Naturally, there are some details with event loop
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Simple Example
100
import asyncio
def func(x, y):
return x + y
@asyncio.coroutine
def do_func(x, y):
yield from asyncio.sleep(1)
return func(x, y)
loop = asyncio.get_event_loop()
result = loop.run_until_complete(do_func(2,3))
print("Got:", result)
• asyncio "hello world"
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
101
import asyncio
@asyncio.coroutine
def echo_client(reader, writer):
while True:
line = yield from reader.readline()
if not line:
break
resp = b'Got:' + line
writer.write(resp)
writer.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(
asyncio.start_server(echo_client, host='', port=25000)
)
loop.run_forever()
• asyncio - Echo Server
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Be on the Lookout!
102
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103
(source: globalpost.com)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 5
104
"Gil"
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Python Threads
105
• Threads, what are they good for?
• Answer: Nothing, that's what!
• Damn you GIL!!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actually...
106
• Threads are great at doing nothing!
time.sleep(2) # Do nothing for awhile
data = sock.recv(nbytes) # Wait around for data
data = f.read(nbytes)
• In fact, great for I/O!
• Mostly just sitting around
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
107
• Threads are weak for computation
• Global interpreter lock only allows 1 CPU
• Multiple CPU-bound threads fight each other
• Could be better
http://www.dabeaz.com/GIL
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Solution
108
• Naturally, we must reinvent the one thing that
threads are good at
• Namely, waiting around.
• Event-loops, async, coroutines, green threads.
• Think about it: These are focused on I/O
(yes, I know there are other potential issues with threads, but work with me here)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
109
• Event-loops have their own issues
• Don't bug me, I'm blocking right now
(source: chicagotribune.com)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Standard Solution
110
• Delegate the work out to a process pool
python
python
python
python
python
main program
CPU
CPU
CPU
CPU
• multiprocessing, concurrent.futures, etc.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
111
• Didn't we just do this with inlined futures?
def fib(n):
return 1 if n <= 2 else (fib(n-1) + fib(n-2))
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
pool = ProcessPoolExecutor(4)
result = run_inline_future(compute_fibs(35))
• It runs without crashing (let's ship it!)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
112
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34))
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
113
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34))
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
9.56s
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
114
• Can you launch tasks in parallel?
t1 = start_inline_future(compute_fibs(34))
t2 = start_inline_future(compute_fibs(34))
result1 = t1.result()
result2 = t2.result()
• Sequential execution
run_inline_future(compute_fibs(34))
run_inline_future(compute_fibs(34)) 9.56s
• Recall (from earlier)
def start_inline_future(fut):
t = Task(fut)
t.step()
return t
4.78s
Inlined tasks running
outside confines of
the GIL?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Execution Model
115
• The way in which it works is a little odd
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
print(threading.current_thread())
val = yield from pool.submit(fib, i)
result.append(val)
return result
add
this
• Output: (2 Tasks)
<_MainThread(MainThread, started 140735086636224)>
<_MainThread(MainThread, started 140735086636224)>
<Thread(Thread-1, started daemon 4320137216)>
<Thread(Thread-1, started daemon 4320137216)>
<Thread(Thread-1, started daemon 4320137216)>
...
????
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Process Pools
116
• Process pools involve a hidden result thread
main thread python
python
CPU
CPU
submit
result_thread
• result thread reads returned values
• Sets the result on the associated Future
• Triggers the callback function (if any)
results
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Issue
117
• Our inlined future switches execution threads
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
main thread
result thread
• Switch occurs at the first yield
• All future execution occurs in result thread
• That could be a little weird (especially if it blocked)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Important Lesson
118
• If you're going to play with control flow, you
must absolutely understand possible implications
under the covers (i.e., switching threads across
the yield statement).
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Insight
119
• The yield is not implementation
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
val = yield from pool.submit(fib, i)
result.append(val)
return result
• You can implement different execution models
• You don't have to follow a formulaic rule
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Thread Execution
120
• Variant: Run generator entirely in a single thread
def run_inline_thread(gen):
value = None
exc = None
while True:
try:
if exc:
fut = gen.throw(exc)
else:
fut = gen.send(value)
try:
value = fut.result()
exc = None
except Exception as e:
exc = e
except StopIteration as exc:
return exc.value
• It just steps through... no callback function
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
121
• Try it again with a thread pool (because why not?)
@inlined_future
def compute_fibs(n):
result = []
for i in range(n):
print(threading.current_thread())
val = yield from pool.submit(fib, i)
result.append(val)
return result
tpool = ThreadPoolExecutor(8)
t1 = tpool.submit(run_inline_thread(compute_fibs(34)))
t2 = tpool.submit(run_inline_thread(compute_fibs(34)))
result1 = t1.result()
result2 = t2.result()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
122
• Output: (2 Threads)
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
<Thread(Thread-1, started 4319916032)>
<Thread(Thread-2, started 4326428672)>
...
(works perfectly)
4.60s
(a bit faster)
• Processes, threads, and futures in perfect harmony
• Uh... let's move along. Faster. Must go faster.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Big Idea
123
• You can mold and adapt generator execution
• That yield statement: magic!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 6
124
Fake it until you make it
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actors
125
• There is a striking similarity between coroutines
and actors (i.e., the "actor" model)
• Features of Actors
• Receive messages
• Send messages to other actors
• Create new actors
• No shared state (messages only)
• Can coroutines serve as actors?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
126
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
idea: use generators
to define a kind of
"named" actor task
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Attempt 1
127
_registry = { }
def send(name, msg):
_registry[name].send(msg)
def actor(func):
def wrapper(*args, **kwargs):
gen = func(*args, **kwargs)
next(gen)
_registry[func.__name__] = gen
return wrapper
• Make a central coroutine registry and a decorator
• Let's see if it works...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
128
• A very simple example
@actor
def printer():
while True:
msg = yield
print('printer:', msg)
printer()
n = 10
while n > 0:
send('printer', n)
n -=1
• It seems to work (maybe)
printer: 10
printer: 9
printer: 8
...
printer: 1
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
129
• Recursive ping-pong (inspired by Stackless)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
ping
pong
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
130
ping 0
pong 1
Traceback (most recent call last):
File "actor.py", line 36, in <module>
send('ping', 0)
File "actor.py", line 8, in send
_registry[name].send(msg)
File "actor.py", line 24, in ping
send('pong', n + 1)
File "actor.py", line 8, in send
_registry[name].send(msg)
File "actor.py", line 31, in pong
send('ping', n + 1)
File "actor.py", line 8, in send
_registry[name].send(msg)
ValueError: generator already executing
• Alas, it does not work
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Problems
131
• Important differences between actors/coroutines
• Concurrent execution
• Asynchronous message delivery
• Although coroutines have a "send()", it's a normal
method call
• Synchronous
• Involves the call-stack
• Does not allow recursion/reentrancy
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 1
132
class Actor(threading.Thread):
def __init__(self, gen):
super().__init__()
self.daemon = True
self.gen = gen
self.mailbox = Queue()
self.start()
def send(self, msg):
self.mailbox.put(msg)
def run(self):
while True:
msg = self.mailbox.get()
self.gen.send(msg)
• Wrap the generator with a thread
• Err...... no.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 2
133
_registry = { }
_msg_queue = deque()
def send(name, msg):
_msg_queue.append((name, msg))
def run():
while _msg_queue:
name, msg = _msg_queue.popleft()
_registry[name].send(msg)
• Write a tiny message scheduler
• send() simply drops messages on a queue
• run() executes as long as there are messages
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
134
• Recursive ping-pong (reprise)
@actor
def ping():
while True:
n = yield
print('ping %d' % n)
send('pong', n + 1)
@actor
def pong():
while True:
n = yield
print('pong %d' % n)
send('ping', n + 1)
ping()
pong()
send('ping', 0)
run()
ping
pong
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
135
ping 0
pong 1
ping 2
pong 3
ping 4
ping 5
ping 6
pong 7
...
... forever
• It works!
• That's kind of amazing
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Comments
136
• It's still kind of a fake actor
• Lacking in true concurrency
• Easily blocked
• Maybe it's good enough?
• I don't know
• Key idea: you can bend space-time with yield
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 7
137
A TerrifyingVisitor
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Let's Write a Compiler
138
• Well, an extremely simple one anyways...
• Evaluating mathematical expressions
2 + 3 * 4 - 5
• Why?
• Because eval() is for the weak, that's why
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
139
• Lexing : Make tokens
2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
• Parsing : Make a parse tree
[NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
140
• Evaluation :Walk the parse tree
PLUS
NUM
(2)
TIMES
NUM
(3)
NUM
(4)
MINUS
NUM
(5)
9
• It's almost too simple
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
141
import re
from collections import namedtuple
tokens = [
r'(?P<NUM>d+)',
r'(?P<PLUS>+)',
r'(?P<MINUS>-)',
r'(?P<TIMES>*)',
r'(?P<DIVIDE>/)',
r'(?P<WS>s+)',
]
master_re = re.compile('|'.join(tokens))
Token = namedtuple('Token', ['type','value'])
def tokenize(text):
scan = master_re.scanner(text)
return (Token(m.lastgroup, m.group())
for m in iter(scan.match, None)
if m.lastgroup != 'WS')
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
142
text = '2 + 3 * 4 - 5'
for tok in tokenize(text):
print(tok)
Token(type='NUM', value='2')
Token(type='PLUS', value='+')
Token(type='NUM', value='3')
Token(type='TIMES', value='*')
Token(type='NUM', value='4')
Token(type='MINUS', value='-')
Token(type='NUM', value='5')
• Example:
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Parsing
143
• Must match the token stream against a grammar
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
• An expression is just a bunch of terms
2 + 3 * 4 - 5
term + term - term
• A term is just one or more factors
term term
factor factor * factor
(2) (3) (4)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
144
expr ::= term { +|- term }*
term ::= factor { *|/ factor}*
factor ::= NUM
def expr():
! term()
! while accept('PLUS','MINUS'):
term()
!print('Matched expr')
def term():
! factor()
while accept('TIMES','DIVIDE'):
factor()
print('Matched term')
def factor():
if accept('NUM'):
print('Matched factor')
else:
raise SyntaxError()
Encode the grammar
as a collection of
functions
Each function steps
through the rule
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
145
def parse(toks):
lookahead, current = next(toks, None), None
def accept(*toktypes):
nonlocal lookahead, current
if lookahead and lookahead.type in toktypes:
current, lookahead = lookahead, next(toks, None)
return True
def expr():
term()
while accept('PLUS','MINUS'):
term()
print('Matched expr')
...
expr()
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
146
class Node:
_fields = []
def __init__(self, *args):
for name, value in zip(self._fields, args):
setattr(self, name, value)
class BinOp(Node):
_fields = ['op', 'left', 'right']
class Number(Node):
_fields = ['value']
• Need some tree nodes for different things
• Example:
n1 = Number(3)
n2 = Number(4)
n3 = BinOp('*', n1, n2)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
147
def parse(toks):
...
def expr():
left = term()
while accept('PLUS','MINUS'):
left = BinOp(current.value, left)
left.right = term()
return left
def term():
left = factor()
while accept('TIMES','DIVIDE'):
left = BinOp(current.value, left)
left.right = factor()
return left
def factor():
if accept('NUM'):
return Number(int(current.value))
else:
raise SyntaxError()
return expr()
Building nodes
and hooking
them together
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Our Little Parser
148
text = '2 + 3*4 - 5'
toks = tokenize(text)
tree = parse(toks)
• Story so far...
BinOp('-',
BinOp('+',
Number(2),
BinOp('*',
Number(3),
Number(4)
)
),
Number(5)
)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
149
class NodeVisitor:
def visit(self, node):
return getattr(self,
'visit_' + type(node).__name__)(node)
• The "Visitor" pattern
• Example:
class MyVisitor(NodeVisitor):
def visit_Number(self, node):
print(node.value)
def visit_BinOp(self, node):
self.visit(node.left)
self.visit(node.right)
print(node.op)
MyVisitor().visit(tree)
2
3
4
*
+
5
-
output
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
150
class Evaluator(NodeVisitor):
def visit_Number(self, node):
! return node.value
def visit_BinOp(self, node):
leftval = self.visit(node.left)
rightval = self.visit(node.right)
if node.op == '+':
return leftval + rightval
elif node.op == '-':
return leftval - rightval
elif node.op == '*':
return leftval * rightval
elif node.op == '/':
return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
9
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Digression
151
• Last 12 slides a whole graduate CS course
• Plus at least one additional Python tutorial
• Don't worry about it
• Left as an exercise...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Death Spiral
152
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
toks = tokenize(text)
tree = parse(toks)
val = Evaluate().visit(tree)
• And it almost works...
Traceback (most recent call last):
File "compiler.py", line 100, in <module>
val = Evaluator().visit(tree)
File "compiler.py", line 63, in visit
return getattr(self, 'visit_' + type(node).__name__)(node)
File "compiler.py", line 80, in visit_BinOp
leftval = self.visit(node.left)
...
RuntimeError: maximum recursion depth exceeded while calling a
Python object
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
153
class Evaluator(NodeVisitor):
def visit_Number(self, node):
! return node.value
def visit_BinOp(self, node):
leftval = self.visit(node.left)
rightval = self.visit(node.right)
if node.op == '+':
return leftval + rightval
elif node.op == '-':
return leftval - rightval
elif node.op == '*':
return leftval * rightval
elif node.op == '/':
return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
!%*@*^#^#
Recursion
(damn you to hell)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
154
+
0 + 1 + 2 + 3 + 4 ... + 999
0
+
+
+
+
2
3
998
999
1
...
A deeply
nested tree
structure
Blows up!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155
• The visitor pattern is bad idea
• Better: Functional language with pattern
matching and tail-call optimization
I ToldYou So
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
156
How do you NOT do something?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
157
How do you NOT do something?
(yield?)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
158
class Evaluator(NodeVisitor):
def visit_Number(self, node):
! return node.value
def visit_BinOp(self, node):
leftval = yield node.left
rightval = yield node.right
if node.op == '+':
return leftval + rightval
elif node.op == '-':
return leftval - rightval
elif node.op == '*':
return leftval * rightval
elif node.op == '/':
return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
Nope. Not doing
that recursion.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
159
class NodeVisitor:
def genvisit(self, node):
result = getattr(self,
'visit_' + type(node).__name__)(node)
if isinstance(result, types.GeneratorType):
result = yield from result
return result
• Step 1:Wrap "visiting" with a generator
• Thinking: No matter what the visit_() method
produces, the result will be a generator
• If already a generator, then just delegate to it
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
160
>>> v = Evaluator()
>>> n = Number(2)
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 2
>>>
• Example:A method that simply returns a value
• Result: Carried as value in StopIteration
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
161
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
<__main__.Number object at 0x1058525c0>
>>> gen.send(_.value)
<__main__.Number object at 0x105852630>
>>> gen.send(_.value)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
• A method that yields nodes (iteration)
Again, note the return
mechanism
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
162
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> gen = v.genvisit(n)
>>> gen
<generator object genvisit at 0x10070ab88>
>>> gen.send(None)
<__main__.Number object at 0x1058525c0>
>>> gen.send(_.value)
<__main__.Number object at 0x105852630>
>>> gen.send(_.value)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
• A method that yields nodes
def visit_Number(self, node):
return node.value
Manually carrying out this
method in the example
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
163
class NodeVisitor:
def visit(self, node):
stack = [ self.genvisit(node) ]
! result = None
while stack:
try:
node = stack[-1].send(result)
stack.append(self.genvisit(node))
result = None
except StopIteration as exc:
! stack.pop()
result = exc.value
! return result
• Step 2: Run depth-first traversal with a stack
• Basically, a stack of running generators
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Transcendence
164
# Make '0+1+2+3+4+...+999'
text = '+'.join(str(x) for x in range(1000))
toks = tokenize(text)
tree = parse(toks)
val = Evaluate().visit(tree)
print(val)
• Does it work?
• Yep
499500
• Yow!
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
165
class Evaluator(NodeVisitor):
def visit_BinOp(self, node):
leftval = yield node.left
rightval = yield node.right
if node.op == '+':
result = leftval + rightval
...
return result
• Each yield creates a new stack entry
• Returned values (via StopIteration) get
propagated as results
generator
generator
generator
stack
yield
yield return
return
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
166
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> stack = [ v.genvisit(n) ]
>>> stack[-1].send(None)
<__main__.Number object at 0x1058525c0>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 3
>>> stack.pop()
>>> stack[-1].send(3)
<__main__.Number object at 0x105852630>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 4
>>> stack.pop()
>>> stack[-1].send(4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
Nodes are visited and
generators pushed onto
a stack
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
167
>>> v = Evaluator()
>>> n = BinOp('*', Number(3), Number(4))
>>> stack = [ v.genvisit(n) ]
>>> stack[-1].send(None)
<__main__.Number object at 0x1058525c0>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 3
>>> stack.pop()
>>> stack[-1].send(3)
<__main__.Number object at 0x105852630>
>>> stack.append(v.genvisit(_))
>>> stack[-1].send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 4
>>> stack.pop()
>>> stack[-1].send(4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
>>>
Results propagate via
StopIteration
12 (Final Result)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168
Final Words
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Historical Perspective
169
• Generators seem to have started as a simple
way to implement iteration (Python 2.3)
• Took an interesting turn with support for
coroutines (Python 2.5)
• Taken to a whole new level with delegation
support in PEP-380 (Python 3.3).
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Control Flow Bending
170
• yield statement allows you to bend control-flow
to adapt it to certain kinds of problems
• Wrappers (context managers)
• Futures/concurrency
• Messaging
• Recursion
• Frankly, it blows my mind.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
171
• Inclusion of asyncio in standard library may be a
game changer
• To my knowledge, it's the only standard library
module that uses coroutines/generator
delegation in a significant manner
• To really understand how it works, need to have
your head wrapped around generators
• Read the source for deep insight
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Is it Proper?
172
• Are coroutines/generators a good idea or not?
• Answer: I still don't know
• Issue: Coroutines seem like they're "all in"
• Fraught with potential mind-bending issues
• Example:Will there be two standard libraries?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Two Libraries?
173
Python
Standard Library
Standard
coroutine library
(asyncio and friends)
?????
• If two different worlds, do they interact?
• If so, by what rules?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Personal Use
174
• My own code is dreadfully boring
• Generators for iteration: Yes.
• Everything else: Threads, recursion, etc. (sorry)
• Nevertheless:There may be something to all of
this advanced coroutine/generator business
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Bit More Information
175
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thanks!
176
• I hope you got some new ideas
• Please feel free to contact me
http://www.dabeaz.com
• Also, I teach Python classes (shameless plug)
@dabeaz (Twitter)
• Special Thanks:
Brian Curtin, Ken Izzo, George Kappel, Christian Long,
Michael Prentiss,Vladimir Urazov, Guido van Rossum

More Related Content

What's hot

SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++
SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++
SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++David Beazley (Dabeaz LLC)
 
Using SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonUsing SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonDavid Beazley (Dabeaz LLC)
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetWalter Heck
 
Do more than one thing at the same time, the Python way
Do more than one thing at the same time, the Python wayDo more than one thing at the same time, the Python way
Do more than one thing at the same time, the Python wayJaime Buelta
 
오픈소스로 시작하는 인공지능 실습
오픈소스로 시작하는 인공지능 실습오픈소스로 시작하는 인공지능 실습
오픈소스로 시작하는 인공지능 실습Mario Cho
 
How to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor NettyHow to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor NettyVMware Tanzu
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking systemJesse Vincent
 
Language Sleuthing HOWTO with NLTK
Language Sleuthing HOWTO with NLTKLanguage Sleuthing HOWTO with NLTK
Language Sleuthing HOWTO with NLTKBrianna Laugher
 
Docker Online Meetup #3: Docker in Production
Docker Online Meetup #3: Docker in ProductionDocker Online Meetup #3: Docker in Production
Docker Online Meetup #3: Docker in ProductionDocker, Inc.
 
Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.Workhorse Computing
 
PyCon Taiwan 2013 Tutorial
PyCon Taiwan 2013 TutorialPyCon Taiwan 2013 Tutorial
PyCon Taiwan 2013 TutorialJustin Lin
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingDmitry Spodarets
 
The Common Debian Build System (CDBS)
The Common Debian Build System (CDBS)The Common Debian Build System (CDBS)
The Common Debian Build System (CDBS)Peter Eisentraut
 
Sphinx autodoc - automated api documentation - PyCon.KR 2015
Sphinx autodoc - automated api documentation - PyCon.KR 2015Sphinx autodoc - automated api documentation - PyCon.KR 2015
Sphinx autodoc - automated api documentation - PyCon.KR 2015Takayuki Shimizukawa
 

What's hot (20)

Python in Action (Part 2)
Python in Action (Part 2)Python in Action (Part 2)
Python in Action (Part 2)
 
Perl-C/C++ Integration with Swig
Perl-C/C++ Integration with SwigPerl-C/C++ Integration with Swig
Perl-C/C++ Integration with Swig
 
Python in Action (Part 1)
Python in Action (Part 1)Python in Action (Part 1)
Python in Action (Part 1)
 
SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++
SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++
SWIG : An Easy to Use Tool for Integrating Scripting Languages with C and C++
 
Using SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonUsing SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with Python
 
PuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of PuppetPuppetCamp SEA 1 - Use of Puppet
PuppetCamp SEA 1 - Use of Puppet
 
Python at Facebook
Python at FacebookPython at Facebook
Python at Facebook
 
Do more than one thing at the same time, the Python way
Do more than one thing at the same time, the Python wayDo more than one thing at the same time, the Python way
Do more than one thing at the same time, the Python way
 
How do event loops work in Python?
How do event loops work in Python?How do event loops work in Python?
How do event loops work in Python?
 
오픈소스로 시작하는 인공지능 실습
오픈소스로 시작하는 인공지능 실습오픈소스로 시작하는 인공지능 실습
오픈소스로 시작하는 인공지능 실습
 
How to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor NettyHow to Avoid Common Mistakes When Using Reactor Netty
How to Avoid Common Mistakes When Using Reactor Netty
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking system
 
Language Sleuthing HOWTO with NLTK
Language Sleuthing HOWTO with NLTKLanguage Sleuthing HOWTO with NLTK
Language Sleuthing HOWTO with NLTK
 
The Internals of "Hello World" Program
The Internals of "Hello World" ProgramThe Internals of "Hello World" Program
The Internals of "Hello World" Program
 
Docker Online Meetup #3: Docker in Production
Docker Online Meetup #3: Docker in ProductionDocker Online Meetup #3: Docker in Production
Docker Online Meetup #3: Docker in Production
 
Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.Shared Object images in Docker: What you need is what you want.
Shared Object images in Docker: What you need is what you want.
 
PyCon Taiwan 2013 Tutorial
PyCon Taiwan 2013 TutorialPyCon Taiwan 2013 Tutorial
PyCon Taiwan 2013 Tutorial
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance Computing
 
The Common Debian Build System (CDBS)
The Common Debian Build System (CDBS)The Common Debian Build System (CDBS)
The Common Debian Build System (CDBS)
 
Sphinx autodoc - automated api documentation - PyCon.KR 2015
Sphinx autodoc - automated api documentation - PyCon.KR 2015Sphinx autodoc - automated api documentation - PyCon.KR 2015
Sphinx autodoc - automated api documentation - PyCon.KR 2015
 

Similar to Generators: The Final Frontier

Advanced Python, Part 2
Advanced Python, Part 2Advanced Python, Part 2
Advanced Python, Part 2Zaar Hai
 
A git workflow for Drupal Core development
A git workflow for Drupal Core developmentA git workflow for Drupal Core development
A git workflow for Drupal Core developmentCameron Tod
 
A Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsA Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsMichael Pirnat
 
Python decorators
Python decoratorsPython decorators
Python decoratorsAlex Su
 
Using and contributing to the next Guice
Using and contributing to the next GuiceUsing and contributing to the next Guice
Using and contributing to the next GuiceAdrian Cole
 
Introduction to CUDA C: NVIDIA : Notes
Introduction to CUDA C: NVIDIA : NotesIntroduction to CUDA C: NVIDIA : Notes
Introduction to CUDA C: NVIDIA : NotesSubhajit Sahu
 
Friend this-new&delete
Friend this-new&deleteFriend this-new&delete
Friend this-new&deleteShehzad Rizwan
 
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...Athens Big Data
 
Functional Programming from OO perspective (Sayeret Lambda lecture)
Functional Programming from OO perspective (Sayeret Lambda lecture)Functional Programming from OO perspective (Sayeret Lambda lecture)
Functional Programming from OO perspective (Sayeret Lambda lecture)Ittay Dror
 
Python Coroutines, Present and Future
Python Coroutines, Present and FuturePython Coroutines, Present and Future
Python Coroutines, Present and Futureemptysquare
 
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...Fons Sonnemans
 
PyCon KR 2019 sprint - RustPython by example
PyCon KR 2019 sprint  - RustPython by examplePyCon KR 2019 sprint  - RustPython by example
PyCon KR 2019 sprint - RustPython by exampleYunWon Jeong
 
Whats new in ES2019
Whats new in ES2019Whats new in ES2019
Whats new in ES2019chayanikaa
 
Cassandra nice use cases and worst anti patterns no sql-matters barcelona
Cassandra nice use cases and worst anti patterns no sql-matters barcelonaCassandra nice use cases and worst anti patterns no sql-matters barcelona
Cassandra nice use cases and worst anti patterns no sql-matters barcelonaDuyhai Doan
 
Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"LogeekNightUkraine
 
Creating sub zero dashboard plugin for apex with google
Creating sub zero dashboard plugin for apex with googleCreating sub zero dashboard plugin for apex with google
Creating sub zero dashboard plugin for apex with googleRoel Hartman
 
Practical git for developers
Practical git for developersPractical git for developers
Practical git for developersWim Godden
 

Similar to Generators: The Final Frontier (20)

Advanced Python, Part 2
Advanced Python, Part 2Advanced Python, Part 2
Advanced Python, Part 2
 
A git workflow for Drupal Core development
A git workflow for Drupal Core developmentA git workflow for Drupal Core development
A git workflow for Drupal Core development
 
A Few of My Favorite (Python) Things
A Few of My Favorite (Python) ThingsA Few of My Favorite (Python) Things
A Few of My Favorite (Python) Things
 
Python decorators
Python decoratorsPython decorators
Python decorators
 
Using and contributing to the next Guice
Using and contributing to the next GuiceUsing and contributing to the next Guice
Using and contributing to the next Guice
 
Introduction to CUDA C: NVIDIA : Notes
Introduction to CUDA C: NVIDIA : NotesIntroduction to CUDA C: NVIDIA : Notes
Introduction to CUDA C: NVIDIA : Notes
 
Friend this-new&delete
Friend this-new&deleteFriend this-new&delete
Friend this-new&delete
 
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
 
Functional Programming from OO perspective (Sayeret Lambda lecture)
Functional Programming from OO perspective (Sayeret Lambda lecture)Functional Programming from OO perspective (Sayeret Lambda lecture)
Functional Programming from OO perspective (Sayeret Lambda lecture)
 
Layout10
Layout10Layout10
Layout10
 
Python Coroutines, Present and Future
Python Coroutines, Present and FuturePython Coroutines, Present and Future
Python Coroutines, Present and Future
 
Docker by Example - Basics
Docker by Example - Basics Docker by Example - Basics
Docker by Example - Basics
 
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...
TechDays 2016 - Developing websites using asp.net core mvc6 and entity framew...
 
Ddev workshop t3dd18
Ddev workshop t3dd18Ddev workshop t3dd18
Ddev workshop t3dd18
 
PyCon KR 2019 sprint - RustPython by example
PyCon KR 2019 sprint  - RustPython by examplePyCon KR 2019 sprint  - RustPython by example
PyCon KR 2019 sprint - RustPython by example
 
Whats new in ES2019
Whats new in ES2019Whats new in ES2019
Whats new in ES2019
 
Cassandra nice use cases and worst anti patterns no sql-matters barcelona
Cassandra nice use cases and worst anti patterns no sql-matters barcelonaCassandra nice use cases and worst anti patterns no sql-matters barcelona
Cassandra nice use cases and worst anti patterns no sql-matters barcelona
 
Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"
 
Creating sub zero dashboard plugin for apex with google
Creating sub zero dashboard plugin for apex with googleCreating sub zero dashboard plugin for apex with google
Creating sub zero dashboard plugin for apex with google
 
Practical git for developers
Practical git for developersPractical git for developers
Practical git for developers
 

More from David Beazley (Dabeaz LLC)

Using Python3 to Build a Cloud Computing Service for my Superboard II
Using Python3 to Build a Cloud Computing Service for my Superboard IIUsing Python3 to Build a Cloud Computing Service for my Superboard II
Using Python3 to Build a Cloud Computing Service for my Superboard IIDavid Beazley (Dabeaz LLC)
 
A Curious Course on Coroutines and Concurrency
A Curious Course on Coroutines and ConcurrencyA Curious Course on Coroutines and Concurrency
A Curious Course on Coroutines and ConcurrencyDavid Beazley (Dabeaz LLC)
 
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...David Beazley (Dabeaz LLC)
 
WAD : A Module for Converting Fatal Extension Errors into Python Exceptions
WAD : A Module for Converting Fatal Extension Errors into Python ExceptionsWAD : A Module for Converting Fatal Extension Errors into Python Exceptions
WAD : A Module for Converting Fatal Extension Errors into Python ExceptionsDavid Beazley (Dabeaz LLC)
 
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...David Beazley (Dabeaz LLC)
 

More from David Beazley (Dabeaz LLC) (7)

Using Python3 to Build a Cloud Computing Service for my Superboard II
Using Python3 to Build a Cloud Computing Service for my Superboard IIUsing Python3 to Build a Cloud Computing Service for my Superboard II
Using Python3 to Build a Cloud Computing Service for my Superboard II
 
Interfacing C/C++ and Python with SWIG
Interfacing C/C++ and Python with SWIGInterfacing C/C++ and Python with SWIG
Interfacing C/C++ and Python with SWIG
 
A Curious Course on Coroutines and Concurrency
A Curious Course on Coroutines and ConcurrencyA Curious Course on Coroutines and Concurrency
A Curious Course on Coroutines and Concurrency
 
Writing Parsers and Compilers with PLY
Writing Parsers and Compilers with PLYWriting Parsers and Compilers with PLY
Writing Parsers and Compilers with PLY
 
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...
An Embedded Error Recovery and Debugging Mechanism for Scripting Language Ext...
 
WAD : A Module for Converting Fatal Extension Errors into Python Exceptions
WAD : A Module for Converting Fatal Extension Errors into Python ExceptionsWAD : A Module for Converting Fatal Extension Errors into Python Exceptions
WAD : A Module for Converting Fatal Extension Errors into Python Exceptions
 
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...
Why Extension Programmers Should Stop Worrying About Parsing and Start Thinki...
 

Recently uploaded

Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 

Recently uploaded (20)

Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 

Generators: The Final Frontier

  • 1. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators: The Final Frontier David Beazley (@dabeaz) http://www.dabeaz.com Presented at PyCon'2014, Montreal 1
  • 2. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators 2 • GeneratorTricks for Systems Programmers (2008) http://www.dabeaz.com/generators/ • A flying leap into generator awesomeness
  • 3. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators 3 • A Curious Course on Coroutines and Concurrency (2009) http://www.dabeaz.com/coroutines/ • Wait, wait? There's more than iteration?
  • 4. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Today's Installment 4 • Everything else you ever wanted to know about generators, but were afraid to try • Part 3 of a trilogy
  • 5. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Requirements 5 • You need Python 3.4 or newer • No third party extensions • Code samples and notes http://www.dabeaz.com/finalgenerator/ • Follow along if you dare!
  • 6. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Disclaimer 6 • This is an advanced tutorial • Assumes general awareness of • Core Python language features • Iterators/generators • Decorators • Common programming patterns • I learned a LOT preparing this
  • 7. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Will I Be Lost? 7 • Although this is the third part of a series, it's mostly a stand-alone tutorial • If you've seen prior tutorials, that's great • If not, don't sweat it • Be aware that we're focused on a specific use of generators (you just won't get complete picture)
  • 8. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Focus 8 • Material in this tutorial is probably not immediately applicable to your day job • More thought provoking and mind expanding • from __future__ import future practical utility bleeding edge
  • 9. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part I 9 Preliminaries - Generators and Coroutines (rock)
  • 10. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators 101 • yield statement defines a generator function 10 def countdown(n): while n > 0: yield n n -= 1 • You typically use it to feed iteration for x in countdown(10): print('T-minus', x) • A simple, yet elegant idea
  • 11. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Under the Covers 11 • Generator object runs in response to next() >>> c = countdown(3) >>> c <generator object countdown at 0x10064f900> >>> next(c) 3 >>> next(c) 2 >>> next(c) 1 >>> next(c) Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> • StopIteration raised when function returns
  • 12. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 12 • Generators as "iterators" misses the big picture • There is so much more to yield
  • 13. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators as Pipelines • Stacked generators result in processing pipelines • Similar to shell pipes in Unix 13 generator input sequence for x in s:generator generator def process(sequence): for s in sequence: ... do something ... yield item • Incredibly useful (see prior tutorial)
  • 14. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines 101 • yield can receive a value instead 14 def receiver(): while True: item = yield print('Got', item) • It defines a generator that you send things to recv = receiver() next(recv) # Advance to first yield recv.send('Hello') recv.send('World')
  • 15. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines and Dataflow 15 • Coroutines enable dataflow style processing source coroutine coroutine send() send() • Publish/subscribe, event simulation, etc. coroutine coroutine send() send() coroutine send()
  • 16. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Fundamentals • The yield statement defines a generator function 16 def generator(): ... ... yield ... ... • The mere presence of yield anywhere is enough • Calling the function creates a generator instance >>> g = generator() >>> g <generator object generator at 0x10064f120> >>>
  • 17. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advancing a Generator • next(gen) - Advances to the next yield 17 def generator(): ... ... yield item ... • Returns the yielded item (if any) • It's the only allowed operation on a newly created generator • Note: Same as gen.__next__()
  • 18. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Sending to a Generator • gen.send(item) - Send an item to a generator 18 def generator(): ... item = yield ... ... yield value • Wakes at last yield, returns sent value • Runs to the next yield and emits the value g = generator() next(g) # Advance to yield value = g.send(item)
  • 19. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Closing a Generator • gen.close() - Terminate a generator 19 def generator(): ... try: yield except GeneratorExit: # Shutting down ... • Raises GeneratorExit at the yield • Only allowed action is to return • If uncaught, generator silently terminates g = generator() next(g) # Advance to yield g.close() # Terminate
  • 20. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Raising Exceptions • gen.throw(typ [, val [,tb]]) - Throw exception 20 def generator(): ... try: yield except RuntimeError as e: ... ... yield val • Raises exception at yield • Returns the next yielded value (if any) g = generator() next(g) # Advance to yield val = g.throw(RuntimeError, 'Broken')
  • 21. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator ReturnValues • StopIteration raised on generator exit 21 def generator(): ... yield ... return result • Return value (if any) passed with exception • Note: Python 3 only behavior (in Python 2, generators can't return values) g = generator() try: next(g) except StopIteration as e: result = e.value
  • 22. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Delegation • yield from gen - Delegate to a subgenerator 22 def generator(): ... yield value ... return result def func(): result = yield from generator() • Allows generators to call other generators • Operations take place at the current yield • Return value (if any) is returned
  • 23. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Delegation Example • Chain iterables together 23 def chain(x, y): yield from x yield from y • Example: >>> a = [1, 2, 3] >>> b = [4, 5, 6] >>> for x in chain(a, b): ... print(x,end=' ') ... 1 2 3 4 5 6 >>> c = [7,8,9] >>> for x in chain(a, chain(b, c)): ... print(x, end=' ') ... 1 2 3 4 5 6 7 8 9 >>>
  • 24. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Mini-Reference • Generator instance operations 24 gen = generator() next(gen) # Advance to next yield gen.send(item) # Send an item gen.close() # Terminate gen.throw(exc, val, tb) # Raise exception result = yield from gen # Delegate • Using these, you can do a lot of neat stuff • Generator definition def generator(): ... yield ... return result
  • 25. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 2 25 and now for something completely different
  • 26. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Common Motif • Consider the following 26 f = open() ... f.close() lock.acquire() ... lock.release() db.start_transaction() ... db.commit() start = time.time() ... end = time.time() • It's so common, you'll see it everywhere!
  • 27. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Managers • The 'with' statement 27 with open(filename) as f: statement statement ... with lock: statement statement ... • Allows control over entry/exit of a code block • Typical use: everything on the previous slide
  • 28. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • It's easy to make your own (@contextmanager) 28 import time from contextlib import contextmanager @contextmanager def timethis(label): start = time.time() try: yield finally: end = time.time() print('%s: %0.3f' % (label, end-start) • This times a block of statements
  • 29. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management 29 • Usage with timethis('counting'): n = 1000000 while n > 0: n -= 1 • Output counting: 0.023
  • 30. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • Another example: temporary directories 30 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ...
  • 31. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Whoa,Whoa, Stop! • Another example: temporary directories 31 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ... What is this? • Not iteration • Not dataflow • Not concurrency • ????
  • 32. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • Under the covers 32 with obj: statements statements statements ... statements • If an object implements these methods it can monitor entry/exit to the code block obj.__enter__() obj.__exit__()
  • 33. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager • Implementation template 33 class Manager(object): def __enter__(self): return value def __exit__(self, exc_type, val, tb): if exc_type is None: return else: # Handle an exception (if you want) return True if handled else False • Use: with Manager() as value: statements statements
  • 34. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager Example • Automatically deleted temp directories 34 import tempfile import shutil class tempdir(object): def __enter__(self): self.dirname = tempfile.mkdtemp() return self.dirname def __exit__(self, exc, val, tb): shutil.rmtree(self.dirname) • Use: with tempdir() as dirname: ...
  • 35. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Alternate Formulation • @contextmanager is just a reformulation 35 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • It's the same code, glued together differently
  • 36. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • How does it work? 36 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • Think of "yield" as scissors • Cuts the function in half
  • 37. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • Each half maps to context manager methods 37 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname statements statements statements ... finally: shutil.rmtree(dirname) __enter__ __exit__ user statements ('with' block) • yield is the magic that makes it possible
  • 38. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • There is a wrapper class (Context Manager) 38 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): ... def __exit__(self, exc, val, tb): ... • And a decorator def contextmanager(func): def run(*args, **kwargs): return GeneratorCM(func(*args, **kwargs)) return run
  • 39. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • enter - Run the generator to the yield 39 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): return next(self.gen) def __exit__(self, exc, val, tb): ... • It runs a single "iteration" step • Returns the yielded value (if any)
  • 40. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • exit - Resumes the generator 40 class GeneratorCM(object): ... def __exit__(self, etype, val, tb): try: if etype is None: next(self.gen) else: self.gen.throw(etype, val, tb) raise RuntimeError("Generator didn't stop") except StopIteration: return True except: if sys.exc_info()[1] is not val: raise • Either resumes it normally or raises exception
  • 41. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Full Disclosure • Actual implementation is more complicated • There are some nasty corner cases • Exceptions with no associated value • StopIteration raised inside a with-block • Exceptions raised in context manager • Read source and see PEP-343 41
  • 42. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Discussion • Why start with this example? • A completely different use of yield • Being used to reformulate control-flow • It simplifies programming for others (easy definition of context managers) • Maybe there's more... (of course there is) 42
  • 43. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 3 43 Call me, maybe
  • 44. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 3 44 Call me, maybe
  • 45. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Async Processing • Consider the following execution model 45 def func(args): ... ... ... return result run_async(func, args) main thread • Examples: Run in separate process or thread, time delay, in response to event, etc.
  • 46. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example:Thread Pool 46 from concurrent.futures import ThreadPoolExecutor def func(x, y): 'Some function. Nothing too interesting' import time time.sleep(5) return x + y pool = ThreadPoolExecutor(max_workers=8) fut = pool.submit(func, 2, 3) r = fut.result() print('Got:', r) • Runs the function in a separate thread • Waits for a result
  • 47. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 47 >>> fut = pool.submit(func, 2, 3) >>> fut <Future at 0x1011e6cf8 state=running> >>> • Future - A result to be computed later • You can wait for the result to return >>> fut.result() 5 >>> • However, this blocks the caller
  • 48. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 48 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): result = fut.result() print('Got:', result) • Alternatively, you can register a callback • Triggered upon completion
  • 49. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exceptions 49 >>> fut = pool.submit(func, 2, 'Hello') >>> fut.result() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 395, in result return self.__get_result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result raise self._exception File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(*self.args, **self.kwargs) File "future2.py", line 6, in func return x + y TypeError: unsupported operand type(s) for +: 'int' and 'str' >>>
  • 50. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures w/Errors 50 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Error handling with callbacks • Exception propagates out of fut.result() method
  • 51. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 51 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Consider the structure of code using futures • Meditate on it... focus on the code. • This seems sort of familiar
  • 52. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Callback Hell? 52 • No, no, no.... keep focusing.
  • 53. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 53 def entry(): fut = pool.submit(func, 2, 3) fut.add_done_callback(exit) def exit(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • What if the function names are changed? • Wait! This is almost a context manager (yes)
  • 54. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Futures 54 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) run_inline_future(do_func) • Thought: Maybe you could do that yield trick • The extra callback function is eliminated • Now, just one "simple" function • Inspired by @contextmanager
  • 55. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com DéjàVu 55
  • 56. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com DéjàVu 56 • This twisted idea has been used before...
  • 57. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Preview 57 t = Task(gen) • There are two separate parts • Part 1:Wrapping generators with a "task" • Part 2: Implementing some runtime code run_inline_future(gen) • Forewarning: It will bend your mind a bit
  • 58. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 58 • Will continue to use threads for examples • Mainly because they're easy to work with • And I don't want to get sucked into an event loop • Don't dwell on it too much • Key thing: There is some background processing
  • 59. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 59 def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) • Problem: Stepping through a generator • Involves gluing callbacks and yields together def do_func(x, y): yield pool.submit(func, x, y) result = print('Got:', result) add_done_callback() cut enter exit
  • 60. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 60 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result)
  • 61. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 61 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Task class wraps around and represents a running generator.
  • 62. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 62 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Advance the generator to the next yield, sending in a value (if any)
  • 63. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 63 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Attach a callback to the produced Future
  • 64. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 64 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Collect result and send back into the generator
  • 65. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work? 65 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) t = Task(do_func(2, 3)) t.step() • Try it: • Output: Got: 5 • Yes, it works Note: must initiate first step of the task to get it to run
  • 66. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work? 66 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_many(n): while n > 0: result = yield pool.submit(func, n, n) print('Got:', result) n -= 1 t = Task(do_many(10)) t.step() • More advanced: multiple yields/looping • Yes, this works too.
  • 67. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 67 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
  • 68. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 68 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) send() or throw() depending on success
  • 69. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 69 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) Catch exceptions and pass to next step as appropriate
  • 70. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Error Example 70 def do_func(x, y): try: result = yield pool.submit(func, x, y) print('Got:', result) except Exception as e: print('Failed:', repr(e)) t = Task(do_func(2, 'Hello')) t.step() • Try it: • Output: Failed: TypeError("unsupported operand type(s) for +: 'int' and 'str'",) • Yep, that works too.
  • 71. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 71 • This whole thing is rather bizarre • Execution of the inlined future takes place all on its own (concurrently with other code) • The normal rules don't apply
  • 72. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Consider 72 def recursive(n): yield pool.submit(time.sleep, 0.001) print('Tick:', n) Task(recursive(n+1)).step() Task(recursive(0)).step() • Infinite recursion? • Output: Tick: 0 Tick: 1 Tick: 2 ... Tick: 1662773 Tick: 1662774 ...
  • 73. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 4 73 yield from yield from yield from yield from future (maybe) source: @UrsulaWJ
  • 74. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus 74 • Focus on the future • Not the past • Not now • Yes, the future. • No, really, the future. (but not the singularity)
  • 75. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus 75 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass ... generator must only produce Futures
  • 76. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 76 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • It's trying to delay the execution of a user- supplied inlined future until later.
  • 77. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 77 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • No Traceback (most recent call last): ... AttributeError: 'generator' object has no attribute 'add_done_callback'
  • 78. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 78 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • This is busted • gen is a generator, not a Future
  • 79. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt) 79 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • Idea: Just iterate the generator manually • Make it produce the required Futures
  • 80. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt) 80 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • No luck.The result gets lost somewhere Got: None • Hmmm.
  • 81. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt) 81 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5
  • 82. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt) 82 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5 manual running of generator with results (ugh!)
  • 83. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (4th Attempt) 83 def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • A better solution: yield from • 'yield from' - Runs the generator for you Got: 5 • And it works! (yay!) • Awesome
  • 84. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com PEP 380 • yield from gen - Delegate to a subgenerator 84 def generator(): ... yield value ... return result def func(): result = yield from generator() • Transfer control to other generators • Operations take place at the current yield • Far more powerful than you might think
  • 85. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Conundrum 85 def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen • "yield" and "yield from"? • Two different yields in the same function • Nobody will find that confusing (NOT!)
  • 86. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt) 86 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always!
  • 87. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt) 87 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always! • No. 'yield' and 'yield from' not interchangeable: Traceback (most recent call last): ... TypeError: 'Future' object is not iterable >>>
  • 88. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com ?????? 88 (Can it be made to work?)
  • 89. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Iterable Futures 89 def patch_future(cls): def __iter__(self): ! if not self.done(): yield self return self.result() cls.__iter__ = __iter__ from concurrent.futures import Future patch_future(Future) • A simple ingenious patch • It makes all Future instances iterable • They simply produce themselves and the result • It magically makes 'yield from' work!
  • 90. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com All Roads Lead to Future 90 yield self • Future is the only thing that actually yields • Everything else delegates using 'yield from' • Future terminates the chain generator generator generator Future yield from yield from yield from run() next(gen)
  • 91. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Decorator 91 import inspect def inlined_future(func): assert inspect.isgeneratorfunction(func) return func • Generators yielding futures is its own world • Probably a good idea to have some demarcation • Does nothing much at all, but serves as syntax @inlined_future def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen • Alerts others about what you're doing
  • 92. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Wrangling 92 t = Task(gen) t.step() • The "Task" object is just weird • No way to obtain a result • No way to join with it • Or do much of anything useful at all Task runs ???? t.step()
  • 93. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures 93 class Task(Future): def __init__(self, gen): !super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting
  • 94. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures 94 class Task(Future): def __init__(self, gen): !super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting A Task is a Future Set its result upon completion
  • 95. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 95 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) • Obtaining the result of task • So, you create a task that runs a generator producing Futures • The task is also a Future • Right. Got it.
  • 96. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 96 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) class Task(Future): ... def step(self, value=None, exc=None): try: ... except StopIteration as exc: self.set_result(exc.value) ...
  • 97. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Runners 97 def start_inline_future(fut): t = Task(fut) t.step() return t def run_inline_future(fut): t = start_inline_future(fut) return t.result() • You can make utility functions to hide details result = run_inline_future(do_func(2,3)) print('Got:', result) • Example: Run an inline future to completion • Example: Run inline futures in parallel t1 = start_inline_future(do_func(2, 3)) t2 = start_inline_future(do_func(4, 5)) result1 = t1.result() result2 = t2.result()
  • 98. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Step Back Slowly 98 • Built a generator-based task system for threads Tasks @inline_future run_inline_future() Threads pools concurrent.future Futures submit result • Execution of the future hidden in background • Note: that was on purpose (for now)
  • 99. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 99 • Ideas are the foundation asyncio coroutines Tasks @coroutine run_until_complete() Event Loop asyncio Futures result • In fact, it's almost exactly the same • Naturally, there are some details with event loop
  • 100. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Simple Example 100 import asyncio def func(x, y): return x + y @asyncio.coroutine def do_func(x, y): yield from asyncio.sleep(1) return func(x, y) loop = asyncio.get_event_loop() result = loop.run_until_complete(do_func(2,3)) print("Got:", result) • asyncio "hello world"
  • 101. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 101 import asyncio @asyncio.coroutine def echo_client(reader, writer): while True: line = yield from reader.readline() if not line: break resp = b'Got:' + line writer.write(resp) writer.close() loop = asyncio.get_event_loop() loop.run_until_complete( asyncio.start_server(echo_client, host='', port=25000) ) loop.run_forever() • asyncio - Echo Server
  • 102. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Be on the Lookout! 102
  • 103. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103 (source: globalpost.com)
  • 104. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 5 104 "Gil"
  • 105. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Python Threads 105 • Threads, what are they good for? • Answer: Nothing, that's what! • Damn you GIL!!
  • 106. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actually... 106 • Threads are great at doing nothing! time.sleep(2) # Do nothing for awhile data = sock.recv(nbytes) # Wait around for data data = f.read(nbytes) • In fact, great for I/O! • Mostly just sitting around
  • 107. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 107 • Threads are weak for computation • Global interpreter lock only allows 1 CPU • Multiple CPU-bound threads fight each other • Could be better http://www.dabeaz.com/GIL
  • 108. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Solution 108 • Naturally, we must reinvent the one thing that threads are good at • Namely, waiting around. • Event-loops, async, coroutines, green threads. • Think about it: These are focused on I/O (yes, I know there are other potential issues with threads, but work with me here)
  • 109. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 109 • Event-loops have their own issues • Don't bug me, I'm blocking right now (source: chicagotribune.com)
  • 110. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Standard Solution 110 • Delegate the work out to a process pool python python python python python main program CPU CPU CPU CPU • multiprocessing, concurrent.futures, etc.
  • 111. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 111 • Didn't we just do this with inlined futures? def fib(n): return 1 if n <= 2 else (fib(n-1) + fib(n-2)) @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result pool = ProcessPoolExecutor(4) result = run_inline_future(compute_fibs(35)) • It runs without crashing (let's ship it!)
  • 112. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 112 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t
  • 113. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 113 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 9.56s
  • 114. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 114 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) 9.56s • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 4.78s Inlined tasks running outside confines of the GIL?
  • 115. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Execution Model 115 • The way in which it works is a little odd @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result add this • Output: (2 Tasks) <_MainThread(MainThread, started 140735086636224)> <_MainThread(MainThread, started 140735086636224)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> ... ????
  • 116. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Process Pools 116 • Process pools involve a hidden result thread main thread python python CPU CPU submit result_thread • result thread reads returned values • Sets the result on the associated Future • Triggers the callback function (if any) results
  • 117. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Issue 117 • Our inlined future switches execution threads @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result main thread result thread • Switch occurs at the first yield • All future execution occurs in result thread • That could be a little weird (especially if it blocked)
  • 118. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Important Lesson 118 • If you're going to play with control flow, you must absolutely understand possible implications under the covers (i.e., switching threads across the yield statement).
  • 119. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Insight 119 • The yield is not implementation @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result • You can implement different execution models • You don't have to follow a formulaic rule
  • 120. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Thread Execution 120 • Variant: Run generator entirely in a single thread def run_inline_thread(gen): value = None exc = None while True: try: if exc: fut = gen.throw(exc) else: fut = gen.send(value) try: value = fut.result() exc = None except Exception as e: exc = e except StopIteration as exc: return exc.value • It just steps through... no callback function
  • 121. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 121 • Try it again with a thread pool (because why not?) @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result tpool = ThreadPoolExecutor(8) t1 = tpool.submit(run_inline_thread(compute_fibs(34))) t2 = tpool.submit(run_inline_thread(compute_fibs(34))) result1 = t1.result() result2 = t2.result()
  • 122. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 122 • Output: (2 Threads) <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> ... (works perfectly) 4.60s (a bit faster) • Processes, threads, and futures in perfect harmony • Uh... let's move along. Faster. Must go faster.
  • 123. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Big Idea 123 • You can mold and adapt generator execution • That yield statement: magic!
  • 124. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 6 124 Fake it until you make it
  • 125. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actors 125 • There is a striking similarity between coroutines and actors (i.e., the "actor" model) • Features of Actors • Receive messages • Send messages to other actors • Create new actors • No shared state (messages only) • Can coroutines serve as actors?
  • 126. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 126 • A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 idea: use generators to define a kind of "named" actor task
  • 127. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Attempt 1 127 _registry = { } def send(name, msg): _registry[name].send(msg) def actor(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) _registry[func.__name__] = gen return wrapper • Make a central coroutine registry and a decorator • Let's see if it works...
  • 128. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 128 • A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 • It seems to work (maybe) printer: 10 printer: 9 printer: 8 ... printer: 1
  • 129. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 129 • Recursive ping-pong (inspired by Stackless) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) ping pong
  • 130. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 130 ping 0 pong 1 Traceback (most recent call last): File "actor.py", line 36, in <module> send('ping', 0) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 24, in ping send('pong', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 31, in pong send('ping', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) ValueError: generator already executing • Alas, it does not work
  • 131. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Problems 131 • Important differences between actors/coroutines • Concurrent execution • Asynchronous message delivery • Although coroutines have a "send()", it's a normal method call • Synchronous • Involves the call-stack • Does not allow recursion/reentrancy
  • 132. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 1 132 class Actor(threading.Thread): def __init__(self, gen): super().__init__() self.daemon = True self.gen = gen self.mailbox = Queue() self.start() def send(self, msg): self.mailbox.put(msg) def run(self): while True: msg = self.mailbox.get() self.gen.send(msg) • Wrap the generator with a thread • Err...... no.
  • 133. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 2 133 _registry = { } _msg_queue = deque() def send(name, msg): _msg_queue.append((name, msg)) def run(): while _msg_queue: name, msg = _msg_queue.popleft() _registry[name].send(msg) • Write a tiny message scheduler • send() simply drops messages on a queue • run() executes as long as there are messages
  • 134. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 134 • Recursive ping-pong (reprise) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) run() ping pong
  • 135. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 135 ping 0 pong 1 ping 2 pong 3 ping 4 ping 5 ping 6 pong 7 ... ... forever • It works! • That's kind of amazing
  • 136. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Comments 136 • It's still kind of a fake actor • Lacking in true concurrency • Easily blocked • Maybe it's good enough? • I don't know • Key idea: you can bend space-time with yield
  • 137. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 7 137 A TerrifyingVisitor
  • 138. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Let's Write a Compiler 138 • Well, an extremely simple one anyways... • Evaluating mathematical expressions 2 + 3 * 4 - 5 • Why? • Because eval() is for the weak, that's why
  • 139. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 139 • Lexing : Make tokens 2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] • Parsing : Make a parse tree [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5)
  • 140. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 140 • Evaluation :Walk the parse tree PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5) 9 • It's almost too simple
  • 141. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 141 import re from collections import namedtuple tokens = [ r'(?P<NUM>d+)', r'(?P<PLUS>+)', r'(?P<MINUS>-)', r'(?P<TIMES>*)', r'(?P<DIVIDE>/)', r'(?P<WS>s+)', ] master_re = re.compile('|'.join(tokens)) Token = namedtuple('Token', ['type','value']) def tokenize(text): scan = master_re.scanner(text) return (Token(m.lastgroup, m.group()) for m in iter(scan.match, None) if m.lastgroup != 'WS')
  • 142. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 142 text = '2 + 3 * 4 - 5' for tok in tokenize(text): print(tok) Token(type='NUM', value='2') Token(type='PLUS', value='+') Token(type='NUM', value='3') Token(type='TIMES', value='*') Token(type='NUM', value='4') Token(type='MINUS', value='-') Token(type='NUM', value='5') • Example:
  • 143. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Parsing 143 • Must match the token stream against a grammar expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM • An expression is just a bunch of terms 2 + 3 * 4 - 5 term + term - term • A term is just one or more factors term term factor factor * factor (2) (3) (4)
  • 144. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse 144 expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM def expr(): ! term() ! while accept('PLUS','MINUS'): term() !print('Matched expr') def term(): ! factor() while accept('TIMES','DIVIDE'): factor() print('Matched term') def factor(): if accept('NUM'): print('Matched factor') else: raise SyntaxError() Encode the grammar as a collection of functions Each function steps through the rule
  • 145. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse 145 def parse(toks): lookahead, current = next(toks, None), None def accept(*toktypes): nonlocal lookahead, current if lookahead and lookahead.type in toktypes: current, lookahead = lookahead, next(toks, None) return True def expr(): term() while accept('PLUS','MINUS'): term() print('Matched expr') ... expr()
  • 146. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 146 class Node: _fields = [] def __init__(self, *args): for name, value in zip(self._fields, args): setattr(self, name, value) class BinOp(Node): _fields = ['op', 'left', 'right'] class Number(Node): _fields = ['value'] • Need some tree nodes for different things • Example: n1 = Number(3) n2 = Number(4) n3 = BinOp('*', n1, n2)
  • 147. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 147 def parse(toks): ... def expr(): left = term() while accept('PLUS','MINUS'): left = BinOp(current.value, left) left.right = term() return left def term(): left = factor() while accept('TIMES','DIVIDE'): left = BinOp(current.value, left) left.right = factor() return left def factor(): if accept('NUM'): return Number(int(current.value)) else: raise SyntaxError() return expr() Building nodes and hooking them together
  • 148. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Our Little Parser 148 text = '2 + 3*4 - 5' toks = tokenize(text) tree = parse(toks) • Story so far... BinOp('-', BinOp('+', Number(2), BinOp('*', Number(3), Number(4) ) ), Number(5) )
  • 149. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 149 class NodeVisitor: def visit(self, node): return getattr(self, 'visit_' + type(node).__name__)(node) • The "Visitor" pattern • Example: class MyVisitor(NodeVisitor): def visit_Number(self, node): print(node.value) def visit_BinOp(self, node): self.visit(node.left) self.visit(node.right) print(node.op) MyVisitor().visit(tree) 2 3 4 * + 5 - output
  • 150. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 150 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator 9
  • 151. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Digression 151 • Last 12 slides a whole graduate CS course • Plus at least one additional Python tutorial • Don't worry about it • Left as an exercise...
  • 152. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Death Spiral 152 # Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) • And it almost works... Traceback (most recent call last): File "compiler.py", line 100, in <module> val = Evaluator().visit(tree) File "compiler.py", line 63, in visit return getattr(self, 'visit_' + type(node).__name__)(node) File "compiler.py", line 80, in visit_BinOp leftval = self.visit(node.left) ... RuntimeError: maximum recursion depth exceeded while calling a Python object
  • 153. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 153 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator !%*@*^#^# Recursion (damn you to hell)
  • 154. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 154 + 0 + 1 + 2 + 3 + 4 ... + 999 0 + + + + 2 3 998 999 1 ... A deeply nested tree structure Blows up!
  • 155. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155 • The visitor pattern is bad idea • Better: Functional language with pattern matching and tail-call optimization I ToldYou So
  • 156. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com QUESTION 156 How do you NOT do something?
  • 157. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com QUESTION 157 How do you NOT do something? (yield?)
  • 158. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 158 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator Nope. Not doing that recursion.
  • 159. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 159 class NodeVisitor: def genvisit(self, node): result = getattr(self, 'visit_' + type(node).__name__)(node) if isinstance(result, types.GeneratorType): result = yield from result return result • Step 1:Wrap "visiting" with a generator • Thinking: No matter what the visit_() method produces, the result will be a generator • If already a generator, then just delegate to it
  • 160. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 160 >>> v = Evaluator() >>> n = Number(2) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 2 >>> • Example:A method that simply returns a value • Result: Carried as value in StopIteration
  • 161. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 161 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes (iteration) Again, note the return mechanism
  • 162. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 162 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes def visit_Number(self, node): return node.value Manually carrying out this method in the example
  • 163. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 163 class NodeVisitor: def visit(self, node): stack = [ self.genvisit(node) ] ! result = None while stack: try: node = stack[-1].send(result) stack.append(self.genvisit(node)) result = None except StopIteration as exc: ! stack.pop() result = exc.value ! return result • Step 2: Run depth-first traversal with a stack • Basically, a stack of running generators
  • 164. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Transcendence 164 # Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) print(val) • Does it work? • Yep 499500 • Yow!
  • 165. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 165 class Evaluator(NodeVisitor): def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': result = leftval + rightval ... return result • Each yield creates a new stack entry • Returned values (via StopIteration) get propagated as results generator generator generator stack yield yield return return
  • 166. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 166 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Nodes are visited and generators pushed onto a stack
  • 167. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 167 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Results propagate via StopIteration 12 (Final Result)
  • 168. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168 Final Words
  • 169. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Historical Perspective 169 • Generators seem to have started as a simple way to implement iteration (Python 2.3) • Took an interesting turn with support for coroutines (Python 2.5) • Taken to a whole new level with delegation support in PEP-380 (Python 3.3).
  • 170. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Control Flow Bending 170 • yield statement allows you to bend control-flow to adapt it to certain kinds of problems • Wrappers (context managers) • Futures/concurrency • Messaging • Recursion • Frankly, it blows my mind.
  • 171. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 171 • Inclusion of asyncio in standard library may be a game changer • To my knowledge, it's the only standard library module that uses coroutines/generator delegation in a significant manner • To really understand how it works, need to have your head wrapped around generators • Read the source for deep insight
  • 172. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Is it Proper? 172 • Are coroutines/generators a good idea or not? • Answer: I still don't know • Issue: Coroutines seem like they're "all in" • Fraught with potential mind-bending issues • Example:Will there be two standard libraries?
  • 173. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Two Libraries? 173 Python Standard Library Standard coroutine library (asyncio and friends) ????? • If two different worlds, do they interact? • If so, by what rules?
  • 174. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Personal Use 174 • My own code is dreadfully boring • Generators for iteration: Yes. • Everything else: Threads, recursion, etc. (sorry) • Nevertheless:There may be something to all of this advanced coroutine/generator business
  • 175. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Bit More Information 175
  • 176. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thanks! 176 • I hope you got some new ideas • Please feel free to contact me http://www.dabeaz.com • Also, I teach Python classes (shameless plug) @dabeaz (Twitter) • Special Thanks: Brian Curtin, Ken Izzo, George Kappel, Christian Long, Michael Prentiss,Vladimir Urazov, Guido van Rossum