-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
asyncio support #95
Comments
Hi @wshayes, Thank you for liking python-arango. I am glad to hear that people are finding it useful. Unfortunately I am too busy right now to start a new project and my knowledge on asyncio (and async programming in general) is novice at best. I'm not sure when I will have time to invest in it. I will keep this issue open for any updates in the future. Best, |
I was about to put this request in when I found it was already in. I thought about it a bit and here is what I know. Most of the API would not have to change, in particular, you wouldn't have to make every def an async def. That is because you can make it so that in "async mode" the ordinary functions return a coroutine that returns the result instead of returning a result. Somebody can await that result in async code and it will be all good. The first step is making an HttpClient that uses aiohttp instead of requests; that should be easy, and the key is that we return a coroutine that returns the result. Another thing that needs to change are the Executors; I think we need to build a parallel set of Then there is the plumbing to make sure that when you using the async HttpClient you also get The main annoyance I see is that the async/await syntax is Python 3.5+ only. I think the clean solution is make an interface for the sync/asyncio implementations and then have a second package which I might work up the motivation to make a pull request. |
This will also be a very useful feature for me. @joowani, is this still something you plan to invest time in? |
Hi! I was on the need of an async driver for arangodb so I took yours @joowani and did a fork on async style! Feel free to comment it. https://github.com/bloodbare/aioarangodb Thanks for your amazing work, its been so easy to adapt to async world |
Hi @bloodbare, Wow this is amazing! I'll make sure to mention your project in python-arango readme. You should also send an email to ArangoDB team so they can have your driver on their website. Awesome work. |
@bloodbare's package looks amazing! Is there still a plan to support async io in the formal package? |
Hi! I was also looking for the asynchronous version and noticed that the version from @bloodbare had not been updated for over a year, so I made a fork (not a copy) and made it fully asynchronous. The current version (1.0.0) fully complient to python-arango 7.2.0. https://github.com/mirrorrim/aioarango @rennenc please take a look, I'll try to keep it up to date :) |
Thanks. I still think it would be great to have to adopted and maintained as part of the same repo. |
Hey I have no problem to keep it on my repo, I've not been updating because
we did not had the need, PRs are more than welcome.
R
Missatge de Rennen ***@***.***> del dia dl., 5 de jul. 2021 a
les 15:18:
… Thanks. I still think it would be great to have to adopted and maintained
as part of the same repo.
Nevertheless, will take a look.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADW4AKUW6UCNE4JEAJVJMTTWGWMDANCNFSM4GZY6DFA>
.
--
Ramon a.k.a bloodbare
|
Sorry, I didn't see any activity in your repository: 4 open issues and 1 pull request :( |
I see .. the PR notification where lost. I'll update it thanks for the ping!
Missatge de mirrorrim ***@***.***> del dia dl., 5 de jul.
2021 a les 17:09:
… Sorry, I didn't see any activity in your repository: 4 open issues and 1
pull request :(
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADW4APGHJBXR3ESJ6Q3MS3TWHDL5ANCNFSM4GZY6DFA>
.
--
Ramon a.k.a bloodbare
|
First thanks @joowani for all the efforts on the python-arango library it's really a well-written project. I was looking for an async solution and judging from the comments it seems like having an async version of this library will be beneficial. I see @mirrorrim and @bloodbare has already done some work to make this possible. @mirrorrim or @bloodbare have you reached out or would you consider making either of your repositories part of the ArangoDB Community? I think it would be cool to have this part of an official community package just to get more eyes on it and would be happy to contribute in terms of keeping it up to date with the python-arango package. Also would like to avoid creating another fork if possible. |
It would be great if @mirrorrim's work could be merged back into the main project as an optional API or it was somehow made an official community package as @alexvanzyl suggested. |
I agree with @incorvia, it would be really nice if we could get the async versions merged into the official community package and maintained as part of it in the future. I also want to highlight that at the moment this issue is closed and I think based on the discussion here it would be worth considering to reopen it and try to get forward with incorporating the work into the main community package, or what do you think? |
We would gladly welcome the aioarango package as a part of the arangodb-community org! I think merging it as part of the python-arango package might be beneficial but would need @joowani and @aMahanna to weigh in on that. If you want to bring the project over please ping me on Slack to discuss the details: https://join.slack.com/t/arangodb-community/shared_invite/zt-1b66mygms-j8TmOdXE7FojR5yA2Yg8kg (Chris.ArangoDB) |
I'm open to merging into python-arango, but it will nearly double the codebase (and double the effort to maintain). I'm not sure how much code reuse there could be between sync and async yet. It will probably require a non-trivial amount of work to refactor. |
This seems to be the main commit that was written to convert python-arango to use async.. perhaps a PR could be opened that uses this as a starting point. It doesn't look like it would double the code base even if async was conditionalized.. @mirrorrim any suggestions here since you did the work? |
In case you want some ideas for how to easily maintain the codebase with both the sync and async version, here's what we did for firedantic (shares many ideas with arangodantic that we also created, but for which we don't have a sync version).
I should highlight that the idea is from https://github.com/python-trio/unasync and the unasync.py we use is a modified version of https://github.com/encode/httpcore/blob/master/unasync.py. A suggestion for how to incorporate this same into python-arango would be to restructure the code a into directories for the sync/async versions, then ensure the async version is up to date and then create a modified version of the unasync.py so it's able to generate the sync version from the async one. Some special treatment likely needed for the client (httpx/requests), but that should be possible to do as well. Then after that, the main work for maintaining the library should be just a matter of updating the async version and ensure there's no issues with the generated sync version, so not a lot more work than maintaining just one code base. There might be better approaches (if you know any, please let me know as well), but this has served us really well for firedantic. |
What would be nice is to call a async version of db from the regular client instance. The aioarango library changed every method to async which might broadens the scope too much. It would probably be better to start with a limited set of functions that benefit the most of async operations such as reads and insertions. |
@joakimnordling |
Why not keep the async definitions truly async? |
Today I am using
https://aioarango.readthedocs.io/en/latest/
I also see there is
https://aioarangodb.readthedocs.io/en/latest/
which I have not done a real comparison so I don't feel a lot of reason
to push for changes.
My guess is that you could, however, use code generation to make both
sync and async stubs for the all the functions in the library and have a
sync-async library without undue duplication in a library.
…------ Original Message ------
From "Slava Shor" ***@***.***>
To "ArangoDB-Community/python-arango" ***@***.***>
Cc "Paul Houle" ***@***.***>; "Comment"
***@***.***>
Date 1/31/2023 4:20:21 PM
Subject Re: [ArangoDB-Community/python-arango] asyncio support (#95)
>I'm open to merging into python-arango, but it will nearly double the
>codebase (and double the effort to maintain). I'm not sure how much
>code reuse there could be between sync and async yet. It will probably
>require a non-trivial amount of work to refactor.
>
Why not keep the async definitions truly async?
—
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAUUCINV2VZ7ZRDFPKQQRGDWVF6ZLANCNFSM4GZY6DFA>.
You are receiving this because you commented.Message ID:
***@***.***>
|
As of today, neither libraries are actively developed nor maintained. |
If I run into some problem with aioarango I'll consider patching it.
Very big API wrappers like boto3 use code generation to build sync and
async apis from the same source. API wrapper code is highly structured
and much of the complexity is embodied in functions that marshall and
unmarshall arguments. Those functions are nice synchronous functions
that can be used in either environment.
Personally I code almost everything adb-related async even if it doesn't
need it just to keep the async API on my fingertips and for the
possibility I might want to move a bit or two of code into the async
world. Having almost the same API would make it be convenient for a
programmer to switch and with one code base the problem of maintaining
the sharable functions on both sides is solved.
…------ Original Message ------
From "Slava Shor" ***@***.***>
To "ArangoDB-Community/python-arango" ***@***.***>
Cc "Paul Houle" ***@***.***>; "Comment"
***@***.***>
Date 2/5/2023 4:20:17 AM
Subject Re: [ArangoDB-Community/python-arango] asyncio support (#95)
>Today I am using
>
>https://aioarango.readthedocs.io/en/latest/
>
>I also see there is
>
>https://aioarangodb.readthedocs.io/en/latest/
>
>which I have not done an actual comparison so I don't feel a lot of
>reason
>to push for changes.
>
As of today, neither libraries are actively developed nor maintained.
It's better to keep synchronous and async libraries separately, as it's
doubtful that both versions will be used in the same project. As well
one may find that in other DB drivers blocking and async versions are
being developed separately.
—
Reply to this email directly, view it on GitHub
<#95 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAUUCIOE7YTXIMCZNCLPVOLWV5WFDANCNFSM4GZY6DFA>.
You are receiving this because you commented.Message ID:
***@***.***>
|
I started a PR at Arangodantic. Maybe you want to have a look at it. I would be very happy about comments and remarks |
Hi @dasTholo, It's great to see proactive steps like yours, especially on such an important and highly requested feature. We recognize the increasing need for asyncio support and are actively pushing to add it to our roadmap. |
Welll... I have created a simple class, to help with this....
|
Hey @bencz, I like your class. The I would like to point out that we're about to start working on an asynchronous driver: python-arango-async. The time allocated for this is limited, but yes, it's finally happening. I can't give a clear estimation of when it would be ready, but I expect about 3 months. We plan to release it gradually, starting with a minimal release once we've covered the basic functionality, and then adding more incrementally. The new driver is going to follow the same python-arango interface for most classes. In the meantime, workarounds such as yours are a great way to "asynchronize" already existing code. Thanks for providing that example! |
I understand the limitation for python-arango-async delivery for production use. But this repo is being updated regulalry. can we have additional apis in this repo to support async to avoid differences in async repo? |
any news? |
Hi! The async repo is still work in progress. It has not been updated in a while because the time I can dedicate to that is unfortunately limited, and first priority has to be the currently existing driver, for which we had to work on a major release. |
Hi Joohwan,
We really like your arango library. I am curious what your plans are in regards to asyncio support. I saw you started another repo for it. I'd be interested in helping out where I could to move this forward.
Thanks!
The text was updated successfully, but these errors were encountered: