-
Notifications
You must be signed in to change notification settings - Fork 798
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python websocket doesn't get the right live data #139
Comments
Some more information would be helpful to diagnose what's going on. Do you see any error messages? Is the websocket still connected when you see the discrepancy? Does the delta grow over time? What data point (bid, ask, last price, etc) are you seeing the difference in? |
Yep the websocket is still connect, I don't see any error messages and the delta doesn't grow, it's just not match the live data. And I watch the 'last price'. I notice that my server CPU is heavily used (not related of the websocket but with other operations), I just try to lower the usage to see if I get delta. Hope it help you? |
So a couple of hours later the delta is now near 0, so when the CPU is overloaded the websocket get some delta with the live data... I don't know if it's logical and I don't know why... But the problem is solved |
Bad new it wasn't that... I still have delta.... So forget the CPU thing it wasn't the problem... Websocket last price: 7496 |
That script doesn't print the new value immediately when messages are received. You're effectively polling every 5 seconds - it's certainly possible for the last price to move 15 points in that amount of time. |
Thanks I will try with out the sleep of 5 seconds! And I will post an update here. |
@ryanfox I have the same issue with the python websocket. The orderbook data appears to come through fine, but when there is sudden movement in the market the data keeps coming through without an error, however it is obvious that the data delayed, by 60+ seconds (compared to the bitmex website on the same computer). For example, website price goes 7500>7530>7510, then 60 seconds later I see the same data come through on websocket. Is there a way to find out what the server time is when messages are sent out? Or more generally a way to detect whether this is delayed data or whether there is a problem on my end? Thanks |
Many of the websocket messages have a |
@ryanfox so I compared server timestamps for orderbook10 and position messages (I was previously using orderBookL2 updates but they don't contain timestamps) and it seems like whenever the exchange gets busy, the time lag between server time and when I get the message drifts out from the baseline value by >10 seconds, and later when the exchange settles down, the backlog is slowly cleared. What I'm confused by is where this backlog comes from? I don't think it's a problem with my internet connection because I have the website up simultaneously and the website does not appear delayed at all. This would also suggest it's not a server side issue, unless the website gets some special treatment over websocket apis? Is there something else within the library that could be causing this delay? EDIT: I should also add that I'm counting roughly 30-60 websocket msgs/sec on my connection, so with an average processing time of <1ms I am very confused as to why I'm seeing 10 secs delay on websocket vs website when exchange gets busy. |
You're saying in |
Yes. On further investigation, I have found that running the bitmex websocket by itself I do not encounter this issue. I see peak msgs/sec of around 230 and there is never a delay more than 1 second. The issue seems to stem from the fact that I run the bitmex websocket concurrently with other websockets. It seems that having 2 different websockets running reduces the speed of the bitmex websocket to handle messages, and thus they get queued up somehow.....but seeing as the ws.run_forever() is on a thread I don't understand what is causing this limitation. Any ideas would be appreciated! |
I run multi instances too! |
The web interface and python client are using the same API, but are separate connections - their requests may be served at different times. When the system is under load there may be latency between the trading engine and clients. It's a known performance issue - it's being worked on along with general performance improvements. |
To clarify my understanding of what you just said about same API/diff connections. Are you saying it's a known issue that the web interface could be faster than the websocket by >10 seconds when Bitmex is under heavy load? If there are 5000 websocket connections via API and 1000 via website, would you expect the website to be faster? |
The issue is that latency can accumulate between the engine and clients under server load.
The website might have less lag, or the python client might. They both use the same websocket API - from the server's perspective, they are both clients to serve. |
I am seeing the same problem, also on python using BitMEXWebsocket. I have not seen the website lag behind the python client, always the other way around. It is currently happening very often and very quickly --- within a few minutes of a restart with lags of a few minutes. A restart of the websocket clears the lag immediately. |
I am having the same issue. Delays of up to 10 seconds and restarting clears them. |
to echo above comments:
It seems like something somewhere is catching up over time, but I can't tell what it is (from my end). |
Just notice that the orderbook volumes for the first tick up/down are completely off at all times. |
I'm no expert, YMMV: The bitmex_websocket implementation relies on the websockets library. It is possible that this is a performance issue. See for example this discussion about lagging websocket collection from bitfinex. I think it's the computation speed, not the network connection because running the DeltaServer/nodejs implementation on the same computer instead faces no lagging issues. If you're sticking with python and don't need orderbookl2, remove it. It's by far the slowest/largest set of data, although you will have to remove/modify all references to orderbookl2 in the code to get it to run. Also, get_ticker uses the quote feed to get the best bid/ask and volumes but rounds the prices to the nearest dollar(!). Remove the rounding. |
I observe the same thing here. The NodeJS behavior is ok. |
@asiva thanks that is very useful to know. Have you tested whether the problem goes away/is less noticeable if you:
Regarding the delta server, (sorry if this is a noob question but I only know python) is it possible to run it and get the python script to fetch the latest data on demand locally? |
@scoshil Regarding the delta-server. Yes, works with python. Just follow the install procedure from the readme. It's very easy. It exposes a socket that returns JSON on your localhost. Check the curl examples. |
I'm trying to get the Node-js delta-server running. When I run: node index.js I get the following output:
The command I'm using in python is:
which returns a 500msg and the below output in the server:
Sorry if this is a beginner question but I don't really understand what this error means/how to fix it Additional info: |
@scoshil FYI delta-server struggles with orderbookL2 data on my computer. I haven't had the spare time to experiment with more/less resources etc. What version of nodejs did you install? I believe it has to be a version 6. I have node 6.14.2 and do not have the audit package installed by default. Perhaps try reinstalling? |
ah....I just went to nodejs.org and downloaded the latest version...it appears it's v8.11.3.....is that a similiar thing to python 2 vs python 3 where there is not necessarily backwards compatibility? EDIT: Just tried with Node v6.14.2 and getting same error... |
I have the same problem with BitMEXClient.getData. It gives me the same error as data.map is not a function. Here is the code: |
I just changed the index.js of this module and now it works. I am not sure if this change will affect other components in this module. It at least fixes the error now.
|
thanks @TomDwz ...I just tried your suggestion and now when I run node index.js the output is:
Also, this might be a stupid question but I am supposed to run >>node index.js in \official-ws\delta-server right? I ask because if I run >>node example.js in \official-ws\nodejs I get data flowing (but then the problem is I can't fetch that data when I run a request). EDIT: I decided to just make BitMEXClient.getData return 'out' instead of 'clone(out)' and now it seems to work. Am I correct in saying that if I fetch this data using python I won't run into any modification issues? Or should I still use .copy() in python? |
I think the best way is to check the clone function in your index.js again. You can do a "console.log(typeof data)" before the return statement and see if data is an object type or not. The reason I add the if statement to the function is because in my program the data parameter is an object and object doesn't have a function for .map(). And that causes the error. So I need to cast it to an array in order to use the .map() function. Hope you can find your error there too. |
this seems to be impacting XBTUSD. When I subscribe to XBTU18 there is no delay. The XBTUSD delay seems to grow with time, delayed even up to minutes. Restarting the program brings it back up to date, but then it starts to fall behind again. This impacts 'quotes' and 'trades'. Is there another way to request top level book every certain interval so that it gets around this lag? Instead of sending every event, send the most recent state every 1 second or something like that? |
blu - one thing to check is your code itself. if it's synchronous then the execution time of whatever you're doing with the data will cause the 'dropped' comms. you may well only be seeing it on xbtusd because the volume is so much greater, so more trades comes through. as soon as a message is received, pawn it off to another process and you'll see your throughput increase. |
Hi, I'm also annoyed by delaying python web socket library. Now, I'm trying to use Node.js library, and it doesn't seem to delay against web view. By python scripts, in first 2 or 3 minutes o.k. however, after that delay or lags become larger. |
Hi, I am having a similar-ish issue. I've been able to get the delta server setup, but for some reason am getting an error in the execution table. All other tables (including the authenticated ones) work fine. Has anyone run into this problem? The error readout I get when I try clicking on the execution table via the web interface is below:
|
scoshil, this seemed to work for me with delta-server:
import requests Replace /instrument with /orderBookL2 etc The market feed is now much better. |
@bluworld , PS: Depending on the index.js content I'm receiving two different errors:
|
@paltsev-p I have not set up private subscriptions yet, but I am getting this error when my python client uses requests to listen to the delta server. I get this error when using urllib or request module. It seems to occur from time to time. Seems like its an issue with the publishing server? Looking into this now but if anyone else has any ideas, that would be great. Anyone on the bitmex team here that can help? Traceback when using urllib: Traceback (most recent call last): Traceback when using Request: Traceback (most recent call last): |
have the same issue, I received absolute wrong data after long run (like after several hours), it appears ask1 price is less than bid1 price. I used pure websocket API, I am not using the delta server. |
@tsoonami have you found a solution? I have setup the delta server, but I get the same error as you whenever I'm trying to get authenticated tables... |
hi. I am 100% layman in coding. I contract a dev to wirte a bot for bitmex. I have several problems with quotes (my bot make several calls by minute and sometimes per second). the quotes aways have a delay of at least 2 seconds and i NEVER get xxxx.5 prices, only xxxx anyone got this problem with .5 prices??? |
after some time the websocket orderbook data drifts further and further from actual prices and becomes unusable. In fact, it can cause people to loose a lot of money. Is Bitmex team aware of this issue? If so, what is their suggested solution? |
I'm having the same issue. The problem appears to be on Bitmex side. The response is either
or
which is the valid blank response. The bitmex connector dies on the Does anyone have another solution? |
I have the same problem. It has more than 1000 $ divergence on (XBT/USD) after 24 hours! |
Oh lord, I just came from here: #310 |
@asiva It works perfectly when i remove the l2orderbook. Thanks! That might be why the nodejs server works fine as well, since it doesn't subscribe to the l2orderbook by default if I'm not mistaken. |
Doesn't work for me even if I remove l2orderbook. The price difference is obvious. |
this error still there |
Yep, the error is still here, but for those, who want to use Python + WebSocket and have have face with this awful bug, here's something, that you can do:
Docs: I've checked it and data was identical with the BitMEX web-interface. No delays at all. |
As identified by @k0rean-rand0m could be an issue with api-connectors Python websocket-client dependency. Might be better to subclass BitMEXWebsocket in bitmex_websocket.py to use something like asyncio, etc. |
@k0rean-rand0m Thanks for posting that code! It's definitely very helpful! Will private data like orders, fills, and positions be secure if transferred from bitmex to a wss client using code like you suggested? I assume it would be, due to the URI being "wws://*", but just wanted to confirm. I suppose I could always confirm using a network sniffer. But, figured it might be quicker to ask. Thanks again! P.S. A couple years ago I had issues connecting to HitBTC with the same symptoms described in this issue. I re-wrote the web-socket client a few times trying to fix it and eventually gave up. But, I do remember now that using the websockets library with asyncio worked without interruption. I had forgotten all about the websockets library. So cool that half a world a way you were there to point me in the right direction! Cheers mate! |
The delta-server also produces lags (not just the python websocket). I did quite a bit of testing and found these results: Lag duration varies
I created an endless loop polling the trade data (http://localhost:4444/trade?), selecting the last 10 seconds of data and calculating the volumes (in panda dataframes). Not applying any pause to the loop, I got about 10 polls/second. This was manageable during low volume periods. Comparing the timestamp of the latest trade with the UTC timestamp I got a delay of typically less than 1 second. However during periods of higher volumes (number of trades / second > 20) the lag time started to increase - the higher the number of trades, the faster the increase of the lag. Then I built a pause (say 0.1 s) into the loop. That yielded ca 5 polls / s. I found that there was far less lag. This loop could handle approx. 100 trades/s without building lag. It would increase to 2 seconds but quickly go back to below 1 s. Also, straining the delta server with polls from a different app that ran in parallel influenced the lag negatively. To conclude: The delta server needs "room to breath". It needs some processing power to load the data from bitmex. If you overload your server with too many polls it falls behind and you get a lag that increases in response to the data volume it needs to handle. So, I ended up using a lag measurer and dynamically adapting my polling rate to the lag. Works for me. |
BitMEX has a REST API, which is nice. If the client disconnects, wait for reconnect. Buffer the incoming data, while making a request to the REST API. Finally, filter the buffered data for missing trdMatchID from the REST API. |
I try to collect 'orderBookL2' data but while every 10 seconds it's about 5-10% of messages with error because can't find id with that price. It occurs because many messages with action "delete" came before "update" with the same id. I thought that it's because of missing "insert" and I use your approach with asyncio lib but then I realised that if I ignore "delete" messages all is ok. But messages of 'orderBookL2' have no attribute 'timestamp'. Maybe you know how to solve it? I can update 'size' to 0 instead deleting row but it can be wrong way...
|
I'have the same issue, data are delayed and I need a solution to get it in real time. I want to implement your proposal but I can't understand where I should put this piece of code in bitmex_websocket.py Could you or someone else please help me? Thanks in advance |
Much recommended https://github.com/bmoscon/cryptofeed |
I have same problem with binance websocket aggregate trades stream. When market volatility increase, time stamp of websocket received data deviates from actual time. I think it is because of client computer CPU speed. When there are many trades happening in volatile times, it becomes harder for client computer to handle so many "on message" interrupts and it queues them. After queue finishes, for example market volatility is reduced, it becomes normal. I think the solution is to use a faster CPU on client side. Someone said he uses threading to receive data, if CPU doesn't have extra threads, python extra threads will not behave like extra threads. |
Hi
I use the websocket with a python 3.6 script and I notice that after some couple of hours of running the data is not the real live value for example right now the live BTC is 7897 and I get 7922 with websocket (the script was launched 12 hours ago).
Here is my script:
Thanks for your feedback on this lag or bug !
The text was updated successfully, but these errors were encountered: