X Tutup
The Wayback Machine - https://web.archive.org/web/20200906154452/https://github.com/MagicStack/uvloop/issues/262/
Skip to content

/ uvloop

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Repeating calls to connect_write_pipe with sys.stdout as a stream lead to FileExistsError exception #262

Open
hh-h opened this issue Jul 31, 2019 · 7 comments
Open

Repeating calls to connect_write_pipe with sys.stdout as a stream lead to FileExistsError exception #262

hh-h opened this issue Jul 31, 2019 · 7 comments
Labels
help wanted need more info

Comments

@hh-h
Copy link

@hh-h hh-h commented Jul 31, 2019

  • uvloop version: 0.11.3, 0.12.2, 0.13.0rc1
  • Python version: 3.6, 3.7
  • Platform: linux, macos
  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: yes

Hello, this example doesn't work with uvloop.

import asyncio
import sys
import uvloop

uvloop.install()
loop = asyncio.get_event_loop()

async def main():
    transport, protocol = await loop.connect_write_pipe(
        asyncio.Protocol, sys.stdout
    )
    print("okay")

    transport2, protocol2 = await loop.connect_write_pipe(
        asyncio.Protocol, sys.stdout
    )
    print("okay")

loop.run_until_complete(main())
okay
Traceback (most recent call last):
  File "uvl.py", line 20, in <module>
    loop.run_until_complete(main())
  File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
  File "uvl.py", line 16, in main
    asyncio.Protocol, sys.stdout
  File "uvloop/loop.pyx", line 2703, in connect_write_pipe
  File "uvloop/loop.pyx", line 2698, in uvloop.loop.Loop.connect_write_pipe
  File "uvloop/handles/pipe.pyx", line 170, in uvloop.loop.WriteUnixTransport._open
  File "uvloop/handles/pipe.pyx", line 29, in uvloop.loop.__pipe_open
FileExistsError: [Errno 17] File exists

please, see discussion here b2wdigital/aiologger#42 (comment)
@1st1
Copy link
Member

@1st1 1st1 commented Aug 14, 2019

Interesting. Why do you think this is a good idea to connect two transports to the same fd?
@madkote
Copy link

@madkote madkote commented Sep 26, 2019 •
edited

@1st1 in case you log to stdout from different loggers, where both use stdout as stream handler.
@1st1
Copy link
Member

@1st1 1st1 commented Sep 26, 2019

Sure, but then you might have a race condition between these two protocols/transports not synchronizing their writes/reads to the FD. And if you want to do something like that anyways, the preferred solution would be probably to just os.dup() the FD for another protocol/transport pair, right?
@madkote
Copy link

@madkote madkote commented Sep 26, 2019

@1st1 Thx. I do fully agree regarding os.dup() - it can be even a good workaround for the library mentioned above.

But there might be other scenarios (which I am not aware right now), where asyncio would work, and uvoop would not. So the question is - how far uvloop want to be similar to asyncio in usage.
@WouldYouKindly
Copy link

@WouldYouKindly WouldYouKindly commented Sep 26, 2019

Just my two cents: we had cases when uvloop would raise FileNotFoundError, while the file was indeed there; also, sometimes writes would go to the wrong files. We did not manage to reliably reproduce it, and just ditched uvloop; asyncio works fine. It looked like uvloop mistook one file descriptor for another or something...

Non sure if it's relevant or not, and unfortunately cannot provide more info.
@1st1
Copy link
Member

@1st1 1st1 commented Sep 26, 2019

Yeah, this definitely needs to be investigated. If anyone wants to help I'd really appreciate that. I myself have no idea when I have time to debug this.
@1st1
Copy link
Member

@1st1 1st1 commented Oct 25, 2019

@hh-h The reason the original example does not work is because libuv requires every FD to be registered once. So in order to fix this in uvloop, uvloop itself will have to dup FDs behind the scenes. Which is a bit suboptimal, but I'll think about it.

@WouldYouKindly

Just my two cents: we had cases when uvloop would raise FileNotFoundError, while the file was indeed there; also, sometimes writes would go to the wrong files.

The only way I can imagine this happening is if you somehow close FDs without closing the transports/protocols that wrap them. In this case, uvloop might have old wrong transports wrapping "new" FDs. Or something else that's crazy like this.

What it really means is that you program might work in asyncio and not in uvloop now, but it's only a matter of time until it starts to break horribly in asyncio too. I might be wrong here, but I suggest you to better investigate what's really happening on your end. uvloop is used in production in enormous deployments and this is the first time I hear about "writing to wrong files". So if I were you I'd check just in case. Obviously I'd love to say something more definitive than just speculating, but I need a piece of code to reproduce the bug you describe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted need more info
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
@1st1 @madkote @hh-h @WouldYouKindly
You can’t perform that action at this time.
X Tutup