problem on Linux Mint: copy to hubic folder creates a folder_segment

[D4void]

Hello

I am using hubicfuse on my Fedora since the beginning without any problem. Thanks for this, it 's just a great job.
I recently tested a backup script (which is based on hubicfuse) with another distrib; Linux Mint 17.1 (kernel 3.13.0-37)
I have problem here. I tested with the last commit and with hubicfuse v2.0
The same things happen.

If I copy a file to my hubic mount dir , ie: ~/Hubic
cp testfile.zip Hubic/default/Backuptest/test for example
I have a new directory created: Hubic/default/Backuptest/test_segments
If I go in it , I see my file copied ok.
Going to Hubic/default/Backuptest/test, I see my file too.

I don't have this problem on my fedora.
I don't know what can do this ?
Maybe a distro library not in the same version ?
Something I didn't set correctly but I don't see what ?
Or could it be a bug ?

Thanks for your help

[D4void]

A little precision after making another test:
when I do a ls of my mount dir, I don't see this created folder.
I see it on the web gui and with the hubic android app too.

[TurboGit]

A bug probably, but I suppose not in hubicfuse but maybe in a dependent lib not working properly. I would check the version of the different package/libs used by hubicfuse.

[maki-chan]

<folder>_segments are folders that hold segments of huge files that are uploaded in several splitted parts to hubic (default: files over 2G are splitted into 1G files)

[D4void]

ok but I copied just a few kb for my test.

[D4void]

I created a VM with a debian wheezy.
I compiled the last git version and I have a folder _segment too ...
File is 90 bytes.
Mint is based on debian and ubuntu.
I must not be the only one using Debian with hubifuse.
What's wrong ?

[TurboGit]

Nothing wrong. I also have a _segment directory. It is empty. Looks fine to me and I have never bothered with it.

[D4void]

you mean the default_segments at the Hubic file root ?
I have one too and yes , empty.

But do you have others one created when you copy a file in a folder (like described in my first post)?
In that case, this new segment folder created (and that I don't wan't to see in my files) is not empty and is containing the same file. If I delete this directory and I go in the target folder, I see the file I copied but it is empty. (because I deleted the segment folder)

And i don't have this with my fedora.

[TurboGit]

No I don't have segment files as described in your first post. Just the top level one:

$ cp hh.zip /mnt/hubic/default/
$ ls -al /mnt/hubic/default/
total 4
drwxr-xr-x 2 obry obry 0 May 4 17:28 .
drwxr-xr-x 2 obry obry 0 Jan 1 1970 ..
drwxr-xr-x 2 obry obry 0 Nov 14 2012 Documents
-rw-rw-rw- 1 obry obry 83 May 4 17:29 hh.zip
drwxr-xr-x 2 obry obry 0 Nov 17 2012 Images
drwxr-xr-x 2 obry obry 0 Nov 15 2012 Music
drwxr-xr-x 2 obry obry 0 Nov 14 2012 .ovh
drwxr-xr-x 2 obry obry 0 Nov 15 2012 scripts
drwxr-xr-x 2 obry obry 0 Nov 14 2012 Videos

And I'm on GNU/Debian sid (very close to Jessie released last week).

[D4void]

If I copy in default, no problem for me too because default_segments already exist I think.
Can you try to mkdir /mnt/hubic/default/Test
and cp hh.zip /mnt/hubic/default/Test
to see if you have a Test_segments created in /mnt/hubic/default/

I am really interested to see if it's coming from me or not. Thank you

I mounted Hubic in my home dir and not with root. Maybe it could be linked.

[TurboGit]

Sure, no segment created:

$ mkdir /mnt/hubic/default/Test
$ cp hh.zip /mnt/hubic/default/Test/
$ ls -al /mnt/hubic/default/
total 0
drwxr-xr-x 2 obry obry 0 May 4 17:56 .
drwxr-xr-x 2 obry obry 0 Jan 1 1970 ..
drwxr-xr-x 2 obry obry 0 Nov 14 2012 Documents
drwxr-xr-x 2 obry obry 0 Nov 17 2012 Images
drwxr-xr-x 2 obry obry 0 Nov 15 2012 Music
drwxr-xr-x 2 obry obry 0 Nov 14 2012 .ovh
drwxr-xr-x 2 obry obry 0 Nov 15 2012 scripts
drwxr-xr-x 2 obry obry 0 May 4 17:56 Test
drwxr-xr-x 2 obry obry 0 Nov 14 2012 Videos

BTW, I'm using hubicfuse compiled from Git master.

[D4void]

Thank you.
And can you check with your web browser or Android app ?
Because I don't see it too with ls on my mount dir.

[TurboGit]

I don't have a single _segments dir visible from my HubiC Android app. I don't use the Web interface.

[D4void]

ok thx.
So it's coming from me... I don't see what it can be.
Same things with Mint and Debian Wheezy (with git master compiled too)
I tried with package v2.0 on Mint too.
I don't think virtual machine can be the reason.

strange.
I will check compilation.

edit:

On Mint:
$ hubicfuse -V
FUSE library version: 2.9.2
fusermount version: 2.9.2
using FUSE kernel interface version 7.19
$ uname -a
Linux neutrinos 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:30:01 UTC 2014 i686 i686 i686 GNU/Linux

On my Fedora:
$ hubicfuse -V
FUSE library version: 2.9.3
fusermount version: 2.9.3
using FUSE kernel interface version 7.19
$ uname -a
Linux Pulsar 3.19.5-200.fc21.x86_64 #1 SMP Mon Apr 20 19:51:56 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[D4void]

This afternoon, I made some trace with -d option on linux mint and fedora.
I've done the same things (and I tested with options direct_io and big_writes, same things) :
hubicfuse Hubic/ -d -o noauto_cache,sync_read,direct_io,big_writes
mkdir Hubic/default/Test
cp dbtest.log Hubic/default/Test

We can see difference. On Mint, the directory Test_segments is created by a PUT.
I don't know fuse at all but I see http. I don't know if close connection code could tell something.
My only clue is Fuse library...
I will try tomorrow if I can "apt-get" the 2.9.3

@TurboGit I would like to know which version you have ? :)
I you tell me 2.9.2 I'm lost :p

*** MINT :
LOOKUP /default/Test/dbtest.log
getattr /default/Test/dbtest.log
unique: 193, error: -2 (No such file or directory), outsize: 16
unique: 194, opcode: CREATE (35), nodeid: 14, insize: 67, pid: 3928
create flags: 0x80c1 /default/Test/dbtest.log 0100644 umask=0022

GET /v1/AUTH_13543ec620463596fd27d4578b6e31db/default?format=xml&delimiter=/&prefix=Test_segments/ HTTP/1.1
< HTTP/1.1 200 OK
....

• Closing connection 20
  ....
  PUT /v1/AUTH_13543ec620463596fd27d4578b6e31db/default/Test_segments HTTP/1.1

*** FEDORA :

LOOKUP /default/Test/dbtest.log
getattr /default/Test/dbtest.log
unique: 8, error: -2 (No such file or directory), outsize: 16
unique: 9, opcode: CREATE (35), nodeid: 3, insize: 67, pid: 26057
create flags: 0x80c1 /default/Test/dbtest.log 0100640 umask=0027

GET /v1/AUTH_13543ec620463596fd27d4578b6e31db/default?format=xml&delimiter=/&prefix=Test_segments/ HTTP/1.1
< HTTP/1.1 200 OK
....

• Closing connection 5
  ....
  PUT /v1/AUTH_13543ec620463596fd27d4578b6e31db/default/Test/dbtest.log HTTP/1.1

[TurboGit]

On my GNU/Debian box I have 2.9.3:

$ apt-cache policy libfuse2
libfuse2:
Installed: 2.9.3-15+b1
Candidate: 2.9.3-15+b1

So you are not completely lost yet :)

[TurboGit]

BTW, do you have set segment_above in your .hubicfuse? If so what is the value? I see in the code that the segment are created only of above the given limit. Frankly, I don't see why it is created on your side if this is not set...

[D4void]

Hello
No I didn't set this in my .hubicfuse
I tested with libfuse 2.9.3 and I had the same problem. So I really don't
understand what happen. I even tested with debian Jessie. Same thing.
I am in holyday this week so I will continue next week.

I
Le 7 mai 2015 23:24, "Pascal Obry" [email protected] a écrit :

BTW, do you have set segment_above in your .hubicfuse? If so what is the
value? I see in the code that the segment are created only of above the
given limit. Frankly, I don't see why it is created on your side if this is
not set...

—
Reply to this email directly or view it on GitHub
#64 (comment).

[TurboGit]

I'm using GNU/Debian testing we it was Jessie for some time on my machine. No issue there. Something really fishy...

[D4void]

I tested again at home on a physical PC with a fresh Mint 17.1 installed and updated and this morning at work on a VM with Debian Jessie.
I removed all dependancies and reinstalled them

On the Debian:
apt-get remove curl libfuse-dev libcurl4-openssl-dev libxml2-dev libssl-dev libjson-c-dev libmagic-dev
apt-get remove libpcre3-dev libpcrecpp0 libselinux1-dev libsepol1-dev libssl-doc zlib1g-dev
apt-get remove libcurl3

apt-get install curl libfuse-dev libxml2-dev libjson-c-dev libmagic-dev
apt-get install libcurl4-openssl-dev
apt-get install libssl-dev

I compiled the source again.
By the way, I don't know why I have these "no" with ./configure because these files are provided by libfuse and libxml2 and they are present :
...
checking fuse.h usability... no
checking fuse.h presence... no
...
checking libxml/tree.h usability... no
checking libxml/tree.h presence... no

make is "ok" , no errors.

And always the same problem :( On the debian and on Mint

I tried to generate again the token for .hubicfuse, no change

I added segment_size=1073741824 and segment_above=2147483648 (the default values) in .hubicfuse, no change

I thought it could be my hubic account ? I created a new one, used hubic_token, no change, same problem.

No I am stuck. I really don't see! I am crazy or there is something I don't do correctly ?

Can we check lib version you have ?

apt-cache policy libfuse-dev libcurl4-openssl-dev libxml2-dev libjson-c-dev libmagic-dev libssl-dev
libfuse-dev:
Installé : 2.9.3-15+b1
libcurl4-openssl-dev:
Installé : 7.38.0-4+deb8u2
libxml2-dev:
Installé : 2.9.1+dfsg1-5
libjson-c-dev:
Installé : 0.11-4
libmagic-dev:
Installé : 1:5.22+15-2
libssl-dev:
Installé : 1.0.1k-3

3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) i686 GNU/Linux

[TurboGit]

My versions:

As I'm using unstable GNU/Debian branch my versions have probably moved recently, here it is:

$ apt-cache policy libfuse-dev libcurl4-openssl-dev libxml2-dev libjson-c-dev libmagic-dev libssl-dev
libfuse-dev:
Installed: 2.9.3-15+b1
libcurl4-openssl-dev:
Installed: 7.42.1-2
libxml2-dev:
Installed: 2.9.2+dfsg1-3
libjson-c-dev:
Installed: 0.11-4
libmagic-dev:
Installed: 1:5.22+15-2
libssl-dev:
Installed: 1.0.2a-1

I'm on a 64bits box:

4.0.0-1-amd64 #1 SMP Debian 4.0.2-1 (2015-05-11) x86_64 GNU/Linux

Could it be the issue here? An int going above the limit in the implementation and I don't do that because I'm using a 64bit box.

Worth looking at this... Maybe adding some debug trace in hubicfuse for the segment_* variables?

[D4void]

ok, I installed all the same lib from jessie testing.
Same problem.

I quickly installed a VM with Mint 64bits : and no problem !

So there is something when compiling hubicfuse on a 32 bits.

[TurboGit]

Ok, probably just an integer overflow as I said in my previous message. Now which one? Can you debug this on your 32bit box?

[D4void]

euh well i don't really see how to debug this :(
I didn't make C since school :)

IMO it must somewhere when there is a check if the directory exist , if we look the debug i put at the beginning (with No such file or directory and the "put" method that creates the _segment)

[LoloVTS]

Hello,

I'm facing same issue on a 32bits system.
This is the segment_above default value which is overlapping the long variable.
A workaround is to set segment_above to 2147483647

[TurboGit]

Thanks, that's indeed a good candidate for this issue. Just fixed on master. Can you test again on a 32bit system? Thanks.

[D4void]

Great thanks for help !
Tested and it works like a charm :)
For my culture, what is segment_above ?

[maki-chan]

segment_above tells the filesystem the size of a file in bytes at which it starts uploading the file in parts rather than in a whole. So if you have segment_above at 2147483647, every file with a size greater than that will be uploaded in parts with a size of segment_size bytes. On 64bit systems, the value can be higher and so I for example use it to upload files in one part with a size up to "4GB -1 byte"

[TurboGit]

Good to hear it is working on your side! Closing this issue then.

[TurboGit]

checking fuse.h usability... no
checking fuse.h presence... no
...
checking libxml/tree.h usability... no
checking libxml/tree.h presence... no

Should be fixed now.