local transfer takes forever

[anarcat]

i ipfs added a file on my laptop. plugged it into the same gigabit switch as my workstation. running the daemon on both machines for a while.

then i ipfs get the given file on the workstation. again, same network segment. in around 30 minutes, only ~600MB were transfered, which is:

> 600Mibyte / (30 minute) à Kibyte/s

  (600 * mebibyte) / (30 * minute) = approx. 341.33333(kibibytes / s)

i.e. around 341KB/s or around 3mbps. this is a gigabit switch, i woudl have expected at least 10-20MB/s, that is 100-200mbps, with the disks being the bottleneck. looking at this, there is some other big elephant in the corridor blocking the way somewhere. :)

also, the transfer is bursty: it will transfer 7.5MB very fast (indeed around 10-20mbps according to iftop), then stop for a while (say 30-60s? hard to tell, i don't have the patience :)), then start again.

it feels like it's going through the whole "internet phonebook", calling up each person, asking about the weather and "oh by the way, do you know where i can get that 7MB block?" "ask your freaking roommate, dumbass", then goes around to the roommate for the block, and back to the phonebook again. :)

[anarcat]

note that i can of course ipfs connect between the two nodes (although I have no idea what that does, see #873 for that). ipfs ping between the two nodes is 3.72ms on average (where as regular ping is 0.425ms average).

[jbenet]

i.e. around 341KB/s or around 3mbps. this is a gigabit switch, i woudl have expected at least 10-20MB/s, that is 100-200mbps, with the disks being the bottleneck. looking at this, there is some other big elephant in the corridor blocking the way somewhere. :)

Yeah, this is really abysmal perf. I think your node is getting into a weird state. I just tried transferring 10GB and it took 12min to add (hitting awful leveldb compaction-- we're getting rid of leveldb for data blocks shortly, in favor of flat files, like git), then 15min to transfer elsewhere.

also, the transfer is bursty: it will transfer 7.5MB very fast (indeed around 10-20mbps according to iftop), then stop for a while (say 30-60s? hard to tell, i don't have the patience :)), then start again.

Hmmm i have seen this in occasion too-- wonder what it may be. reprovides @whyrusleeping ?

[anarcat]

and the command now completed:

anarcat@marcos:tmp$ time ipfs get [censored]
Saving file(s) to [censored]
3.19 GB 2h31m19s
2.82user 8.36system 2:31:19elapsed 0%CPU (0avgtext+0avgdata 13612maxresident)k
1360inputs+0outputs (11major+827minor)pagefaults 0swaps

2.5h is what i would have expected over the DSL, not the local network.

should i reset those nodes somehow? it's not like i have anything important in there right now...

[jbenet]

Hm, It's possible there's something funky with how the records are getting advertised to the dht.

[anarcat]

oh, and another thing to add here is that this is not specific to local transfers, but just shows that the problem is not with the bandwidth of the uplink. :)

[whyrusleeping]

this shouldnt be reprovides. those all happen in batched async, they dont block anything... Im gonna check out transfers myself in a minute

[whyrusleeping]

Can someone verify that this has been fixed? our transfers locally (same LAN) should be a smooth 3mbps (until we fix the stream issue we are stuck at 3mpbs)

[anarcat]

i'd be glad to. how do i do that, actually? it may need to wait for next week for me however.

[zorun]

I'm also seeing a very poor performance with large files.

For instance, the Sintel video at /ipfs/QmQv4YQNmRPuTTHs4AgBhKEFDdN7eQYeTbSmr8JVWVfury/sintel.mp4 seems to be widely available (and it downloads super fast using gateway.ipfs.io), but getting it via the local IPFS daemon is painfully slow. The first few MB load fast, then it stalls:

$ ipfs get QmQv4YQNmRPuTTHs4AgBhKEFDdN7eQYeTbSmr8JVWVfury/sintel.mp4
Saving file(s) to sintel.mp4
17.25 MB 

It progresses, but slowly: it seems to take between ~10 seconds and several minutes for each 256 KB block.

The DHT seems to know some nodes holding the data (note the error at the end, not sure if it is meaningful):

$ ipfs dht findprovs QmQv4YQNmRPuTTHs4AgBhKEFDdN7eQYeTbSmr8JVWVfury
QmaUcrt153FLdvJtfWXN7bGJgFcVBcc8Z1Z7bTAoqzyPHQ
QmSoLpPVmHKQ4XTPdz8tjDFgdeRFkpV8JgYq8JVJ69RrZm
QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64
QmXWAJ7Jic3r8XG4yiNLGbVRGqY3jgae814bw6w3G1WMXo
QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
QmSoLueR4xBeUbY9WZ9xGUUxunbKWcrNFTDAadQJmocnWm
QmZXxbfUdRYi578pectWLFNFv5USQrsXdYAGeCsMJ6X8Zt
QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
error: routing: not found

[whyrusleeping]

@zorun this is good info to have, thank you! A few questions, what version of the code are you running? A moderate performance improvement was merged last night that may affect this. The 10 second pauses to me feel like DHT issues.

The error: routing: not found at the end is normal, I should probably suppress it. FindProvider queries will run until they find 20 providers (i think its 20, i may be wrong) and return 'not found' if they dont find 20

[zorun]

I'm using version a97e9e7, so I think it has your fix.

I gc-ed the local repo and tried again getting the same file /ipfs/QmQv4YQNmRPuTTHs4AgBhKEFDdN7eQYeTbSmr8JVWVfury/sintel.mp4. This time, the 5 first MB went very fast, and then the rest went at a very stable rate of exactly 300 KB/s.

Thanks for the precision about the error message!

[Stebalien]

I believe this situation is much better now. It's still not perfect but I can get ~45MiB/s (~350mbps) on local host (I haven't tested over a real network but that shouldn't be the bottleneck.

[Stebalien]

Please reopen if you still experience this problem in the latest release (0.4.13).