python generator and asynico

Posted on |

python iterator

in python, iterator is any object that follows iterator protocol. which means should include method: __iter()__, next(), if no next element should raise StopIteration exception.

take an example, dict and list have implemented __iter()__ and __getitem__() methods, but not implemented next() method. so they are iterable but not iterator.

generator

to support delay operations, only return result when need, but not immediately return

  • generator funcion (with yield)

  • generator expression

Python generator in other languages is called coroutines

yield

yield is almost of return, but it pauses every step after executing the line with yield, and till call next() will continue from the next line of yield.

1
2
3
4
5
6
7
8
while True:
	sim.run(timeout, cb)
def cb():
    a = 1 
	yield print(a)
	a += 1

coroutines

example: 

1
2
3
4
5
6
7
8
def cor1(name):
        print("start cor1..name..", name)
        x = yield name
        print("send value", x)
cor_ =  cor1("zj")
print("next return", next(cor_))
print("send return", cor_.send(6))

to run coroutine, need first call next(), then send() is called. namely, if not call next() first, send() will wait, and never be called.

the thing is, when define a python generator/coroutine, it never will run; only through await(), next() call first, which then trigger the generator/coroutine start.

asyncio

asyncronized IO, event-driven coroutine, so users can add async/await to time-consuming IO.

  • event-loop

event loop will always run, track events and enqueue them, when idle dequeue event and call event-handling() to deal with it.

  • asyncio.Task

await

await only decorate async/coroutine, which is a waitable object, it works to hold the current coroutine(async func A) and wait the other coroutine(async func B) to the end.

1
2
3
4
5
6
7
8
9
async def funcB():
	return 1
async def funcA():
	result = await funcB()
        return result
run(funcA())

multi-task coroutines

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
loop.create_task()
run_until_complete() #block the thread untill coroutine completed
asyncio.sleep() #nonblock event-loop, with `await`, will return the control-priority to event-loop, at the end of sleep, control-priority will back to this coroutine 
asyncio.wait(), #nonblock event-loop, immeditialy return coroutine object, and add this corutine object to event-loop
``` 
the following example:  get the event-loop thread, add coroutine objct(task) in this event-loop, execute task till end.
```python
import asyncio
async def cor1(name):
        print("executing: ", name)
        await asyncio.sleep(1)
        print("executed: ", name)
loop = asyncio.get_event_loop()
tasks = [cor1("zj_" + str(i)) for i in range(3)]
wait_cor = asyncio.wait(tasks)
loop.run_until_complete(wait_cor)
loop.close()
``` 
### dynamically add coroutine
```shell 
loop.call_soon_threadsafe() # add coroutines sequencially  
asyncio.run_coroutine_threadsafe() #add coroutines async

add coroutine sequencially

in this sample, main_thread(_loop) will sequencely run from begining to end, during running, there are two coroutines registered, when thread-safe, these two coroutines will be executed.

the whole process actually looks like run in sequencialy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import asyncio
from threading import Thread
def start_loop(loop):
    asyncio.set_event_loop(loop)
    loop.run_forever()
def thread_(name):
    print("executing name:", name)
    return "return nam:" + name
_loop = asyncio.new_event_loop()
t = Thread(target=start_loop, args=(_loop,)) #is a thread or coroutine ?
t.start()
handle = _loop.call_soon_threadsafe(thread_, "zj")
handle.cancel()
_loop.call_soon_threadsafe(thread_, "zj2")
print("main thread non blocking...")
_loop.call_soon_threadsafe(thread_, "zj3")
print("main thread on going...")

add coroutines async

in this way, add/register async coroutine objects to the event-loop and execute the coroutines when thead-safe

1
2
3
4
5
6
7
8
9
10
11
future = asyncio.run_coroutine_threadsafe(thread_("zj"), _loop)
print(future.result())
asyncio.run_coroutine_threadsafe(thread_("zj2"), _loop)
print("main thread non blocking...")
asyncio.run_coroutine_threadsafe(thread_("zj3"), _loop)
print("main thread on going...")

async callback vs coroutines

compare callback and coroutines is hot topic in networking and web-js env.