Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
event loop loop forever: make lots of sock_connec at the same time #378
Comments
Catstyle
changed the title
|
Hey, thanks for the report! But please provide a reproducible sample code. I've tried several batches of 10k concurrent connections on the same server but cannot reproduce the issue. |
fantix
moved this from Triage Needed
to Question, Discussion & Others
in Fantix's Sorting Board
|
finally got some time server.py import asyncio
import socket
import uvloop
uvloop.install()
count = 0
backlog = 1024
async def connection_made(sock, loop):
global count
count += 1
print(f'{loop.time()}: {count}, {sock} connected')
await asyncio.sleep(1000)
def accept(sock):
loop = asyncio.get_running_loop()
for _ in range(backlog):
try:
conn, addr = sock.accept()
except (BlockingIOError, InterruptedError, ConnectionAbortedError):
return
conn.setblocking(False)
asyncio.create_task(connection_made(conn, loop))
async def main():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setblocking(False)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
sock.bind(('localhost', 56789))
sock.listen(backlog)
loop = asyncio.get_running_loop()
loop.add_reader(sock.fileno(), accept, sock)
await asyncio.sleep(1000)
if __name__ == '__main__':
asyncio.run(main())client.py import asyncio
import socket
import sys
import uvloop
uvloop.install()
count = 0
async def connect(addr, loop):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setblocking(False)
await loop.sock_connect(sock, addr)
return sock
async def wrapper(loop):
while 1:
try:
sock = await connect(('localhost', 56789), loop)
except BlockingIOError:
await asyncio.sleep(0)
else:
break
global count
count += 1
print(f'{loop.time()}: {count}, {sock} connected')
await asyncio.sleep(1000)
async def main(concurrency):
tasks = []
loop = asyncio.get_running_loop()
for x in range(concurrency):
tasks.append(asyncio.create_task(wrapper(loop)))
await asyncio.gather(*tasks)
if __name__ == '__main__':
# python client.py 1000
asyncio.run(main(int(sys.argv[1])))concurrency = 1000, works well concurrency = 5000, oops using vanilla asyncio, works well some kernel config |
fantix
moved this from Question, Discussion & Others
to Triage Needed
in Fantix's Sorting Board
fantix
added a commit
to fantix/uvloop
that referenced
this issue
Feb 8, 2021
CPython fixed the same issue in python/cpython#10419. Seems like under pressure, more write callbacks may happen before _remove_writer() is called, so we should check for done(). Fixes MagicStack#378
fantix
added a commit
to fantix/uvloop
that referenced
this issue
Feb 9, 2021
CPython fixed the same issue in python/cpython#10419. Seems like under pressure, more write callbacks may happen before _remove_writer() is called, so we should check for done(). Fixes MagicStack#378
fantix
added a commit
to fantix/uvloop
that referenced
this issue
Feb 9, 2021
Use _SyncSocketWriterFuture for cancellation, so that we could remove the writer directly in the callback. Fixes MagicStack#378 Closes MagicStack#389
fantix
added a commit
to fantix/uvloop
that referenced
this issue
Feb 9, 2021
Use _SyncSocketWriterFuture for cancellation, so that we could remove the writer directly in the callback. Fixes MagicStack#378 Closes MagicStack#389
Merged
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
PYTHONASYNCIODEBUGin env?: yesserver side: accept connections and do nothing, just close it after a few seconds, simulating heartbeat timeout
client side: connect to server side a lots in the same time, e.g. -c 5000
it succeeded the first time, failed the second time
i cannot even terminate it, it loop forever, all i can do is to
kill -9traceback logs:
after some checks, seems like the failed sock did not remove the writer correctly(or not in time?)
fut's done_callback is scheduled after the first time loop._sock_connect_cb invoked
at that time related writer is not removed and the socket fd is reused under heavy pressure?
then the same callback is invoked but for different fd?
checked asyncio, it check fut.done(), and uvloop check fut.cancelled()
The text was updated successfully, but these errors were encountered: