Specifically, this is not dependent on last mod time, but instead a "version" field in each file's metadata that is incremented every time a file changes. I'm not really sure how that's going to help with concurrent (or conflicting) edits though.
Also what happens if one of the sync volumes becomes read only? Does it start deleting new things off the other volume? That happened to me with bit torrent sync and I lost new image files as a result.
I think the point is also "this is actually critical information that is important for end users of the system to understand, not esoteric knowledge only required by developers".
Overwrites unless the file is locked (ie. a .docx open in Word that two people are editing concurrently). If a file is locked you get multiple files with "(X's conflicted copy)" appended to their names
True, though I've found that very difficult to use in practice. I retrieved something through it once, but it wasn't pleasant. Especially with periodic autosaves, finding the last good version before an accidental overwrite requires trawling through pages and pages of old versions, with poor/nonexistent diff tools.
We use it all the time, with great success. We have one Dropbox (free account) for kids to keep their programming, drawing and other stuff in (one folder per kid). They lose their stuff all the time, in the most creative ways possible. Dropbox's 30-day version history can be a real lifesaver in those cases.
Although sometimes stuff is just gone and they're out of luck. I'm still not entirely sure what happens then, possibly a weird combination of everyone saving their project near closing time, the Internet connection maybe choking up a bit, and people shutting down their computer before everything is properly synced.
Conflicts are handled more elegantly than a simple overwrite. There are many ways to spin off a conflicted copy. This article covers the most common way: https://www.dropbox.com/help/36/en
Dropbox is pretty good about never losing data. It can be suggested that this is one of the most interesting or valuable parts of its technology. Its practical use of bandwidth is another.
Since we're listing alternatives here: I setup SeaFile (http://seafile.com/) a couple of months ago and I'm loving it so far. Mainly because it has client-side encryption and allows a single client to sync with different servers and selectively choose which 'libraries' (basically directories that are under sync) to use. Typical usecase is having a personal server for personal files and another one at the office for work-related stuff.
For now I run seafile inside a private network behind a vpn. But I'd rather find an alternative with a better security story. Something from a team that can justify their cryptographic constructs in detail.
I tried to use git annex assistant: I can get files to be backed-up on a server, but when doing sync between 2 different machines it has never worked (I've been doing the xmpp-setup dance multiple times, and it's especially painful due to having 2factor auth enabled on google)
I'll have to try using plain git annex from the commandline to try to understand what is going wrong (I already used it to clean some repositories that the assistant was refusing to delete)
PS: I tried it with 2 machines in the same LAN (both linux... though one of the 2 distros had a slightly older version of the git-annex package)
The default MAC algorithm to be applied on the filenames is HMACSHA1. A stronger one, for instance HMACSHA512, one can be chosen upon creation of the special remote with the option mac=HMACSHA512. The available MAC algorithms are HMACSHA1, HMACSHA224, HMACSHA256, HMACSHA384, and HMACSHA512.
I dont get it, why is deduping at a chunk level important, can't it just take it as a tradeoff and dedupe at the file level, or does that have the same problems? And why exactly is deduping important for a personal use case, bandwidth is relatively cheap.
In any case, while I want security in these systems to avoid mass survey style attacks, I assume anything I put over the wire to be compromised at least potentially. I keep the kind of files that need that level of encryption on encrypted flash drives.
If you are encrypting each file individually, you are still giving away the file names. Hence you use encrypted file containers. They benefit from block level deduplication. They can be large in size.
Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.
Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.
I'm not saying that one is better than other, just giving an example of where de-duping at block level is needed as @lugg asked.
Hashing doesn't solve the problem completely. You are still giving away information about file count, size & hierarchy.
Sure; I was just talking about the file names. You can avoid the hierarchy part by transmitting it separately from the files in a secure way, though.
Regarding selective sync : you'll have to organize the files logically into containers. I think even Dropbox doesn't let you sync files selectively, you have to have content separated into folders.
git-annex does. You can implement rules to decide what files should be synced, which should be dropped locally, etc.
For example, an MP3 player running it might have a rule such as
filename(*.mp3) or filename(*.ogg)
and it'll only sync those files, regardless of their directories.
I use seafile as well - I host it on my digital ocean instance and it works very well and things sync very quickly.
best part is that it does automatic, granular snapshots of my data - I've stopped using git for things like papers because I know it's backed up in seafile.
you mean you can use a friend you don't have to trust as a cross-backup system
I mainly mean it has client side encryption, the consequences of that I'd like to leave open since nobody really knows or can predict all possible vulnerabilities of all possible systems. But yes basically: your friend would have a very, very hard time doing anything useful with the data or the incoming packets.
none for the free version?
not sure what you mean, I host the thing myself (serisouly, 5$ a month for a DigitalOcean box: best way ever to spend money:) but the site seems to claim there is 1GB free storage if you try their service?
I'm curious as to why people have had such bad ownCloud experiences? I've had it set up on my servers for 12+ months now and am really enjoying the experience. Could someone please explain further?
Also happy owncloud user for over a year; though i mainly use it for calendar+contacts (both synced with both phone and emacs) and auto-backups of pictures I take with my phone.
Documents and such I tend to keep in various public or private git repos; habits die hard.
My issue with ownCloud is that it lacks partial (differential) updates. Change a bit in a file and have a whole one re-uploaded.
Also, security track combined with the implementation language scare me to the extent of not really willing to expose the whole thing to the Internet. Just being superstitious here.
I tried Owncloud for a while and was disappointed. Ended up using Syncdocs (http://syncdocs.com) to do end-to-end encryption to Google Drive. The main advantage of this was the versioning is done by Google and fast access from anywhere.
I'll be taking a look at Seafile now, looks promising.
> Ori is a distributed file system built for offline operation and empowers the user with control over synchronization operations and conflict resolution. We provide history through light weight snapshots and allow users to verify the history has not been tampered with. Through the use of replication instances can be resilient and recover damaged data from other nodes.
It seems well thought out, and competitive with many of the other approaches mentioned here. It uses Merkle trees (as does Git) that encompasses the file system structure and full history.
Academic software usually for publication.
After paper accepted, unless they could publish another paper using same software/concept/idea, this software will dis-continued.
"Replacement" is too strong a word here. P2P sync requires at least 2 peers to be online at once. For the simple case of syncing your work and home computers or sharing with coworkers that is not always a reliable assumption.
It's only a replacement for a centralized service like Dropbox if you have an always-connected peer (a de facto central server).
I don't understand why you're being downvoted because you make a valid point. It has no iOS or Android client, no cloud storage (unless you buy your own), no versioning and sync conflict behaviour is very different. Ergo, it's not a replacement.
That's kind of the whole point, isn't it? It's clearly meant to be a Dropbox replacement not for everybody, but for those who consider the lack of mandatory cloud storage a feature. So yeah, technically more a replacement for BitTorrent Sync than Dropbox, but not everybody knows about that.
Missing features can be added. Mobile clients by third party devs will surely crop up if/when this thing gets traction. The protocol is open, after all.
Not to be condescending, but what did you expect from a "roll your own" solution? Of course you have to host it yourself. This has been the case forever, OwnCloud being the biggest example right now.
That's fine, but call it what it is. I only objected to calling it a "replacement". There's a big functional difference between P2P and centralized architectures.
Syncthing is less purely peer-to-peer. It's easy enough set up a central server and have your devices only sync to that server because of the way connections are made.
Yes, absolutely. But that is a significantly greater requirement, each end user or workgroup running their own server. I'm not sure why I'm getting downvoted for pointing that out. It's not an apples-to-apples replacement.
This + a hosting provider would be more like an actual Dropbox replacement (e.g. owncloud.org links to a bunch of hosts).
That's not what I'm looking for. I do have a computer at home that is always on.
edit: Do people generally still turn off their home computers?
edit2: After thinking about it - what you seem to be describing is an alternative and not just a replacement. I want to replace the service that Dropbox offers with one of my own, I don't want to switch to an alternative Dropbox clone service.
Huge numbers of people only have a laptop at home which sleeps when they're at work. So the only sync service that's going to work for them is something with online storage as an intermediate step.
>>edit: Do people generally still turn off their home computers?
Yeah. My desktop computer eats up a lot of power even when idle. I do want the convenience of never turning it off, but I'd rather be environmentally friendly.
One of my favorite open source tools of this kind is unison. It works great. I set it up to go and I don't even notice it is there. It is quick, seems to have been around for a while and is packaged for most Linux distros. It has a GUI but you don't need it.
I can't really seem to find this information in the documentation.
Does this support delta/block-level sync for large files (e.g.: does mounting a 100 GB truecrypt container, modifying a file inside the container and unmounting it cause the entire 100 GB container to be uploaded)?
Does it utilize the native OS platform APIs for detecting file modification (e.g. inotify on linux) as opposed to scanning/polling large directories looking for modified date changes?
> Does this support delta/block-level sync for large files
Yes.
> Does it utilize the native OS platform APIs for detecting file modification
It uses polling.
Edit: inotify issue is here: https://github.com/calmh/syncthing/issues/9 (the author seems to want to implement this feature once go has a decent cross-platform file watching library)
EDIT: i realized i answered about btsync while question was about syncthing, sorry.
Regarding detecting file modifications: my friend had issues with his synology diskstation eating 100% cpu with btsync with around 50k files. With less files it was down to 20%. We never debugged it properly but it may be polling issues, but it could also have been out of ram-swapping issues (512MB ram was not enough).
No matter how much I tried, I always had crappy experience using OwnCloud (-> very slow to recognize new files and sync them across other computers). I tried it maybe 4-5 times in total, last time just couple of weeks ago.
Does it work fast for you? Do you use desktop client or primarily web GUI?
I tried it too, but ended up replacing it with something else. It was slow as a pig, was unable to handle datasync with failures and timeouts on any reasonable amount of files (a 2000 file user) and constantly made my CPU fan hit max.
My server was a Core i7 with 12GBs of RAM, SSD backing and a more or less dedicated NAS. I tried with both SQLite and MySQL backing. MySQL was better, but still unacceptable. If that's not enough to run it for a single user, I can't imagine what it's like trying to deploy it for a full family, even less an organization.
I love the idea behind owncloud, and I admit touting it myself in my own "honeymoon phase", but in hindsight I have no problem admitting that the execution is just too poor.
Same experience. I tried it on DO droplets (1, 2, 4 and then 16GB of RAM to see if I'm nuts), computer I'm working on (16GB RAM) and from a RPi (that was for fun though). Every single time it was painfully slow.
It took it about 1h to sync a folder with about 100 files, under 30MB total across 4 computers. Not to mention that minor modifications appeared only after 15 minutes (small text files).
BTSync worked great, but I wouldn't trust it with something sensitive.
Long story short, I went from an "all cloud" (Google + Dropbox) setup to a "all mine, no NSA" setup. Now I've decided that counter zealotry doesn't have that much for it, and I'll try to balance privacy with best of breed services, without trying to end up with all my eggs in one basket. Not doing that ensures I keep interop and portability in mind.
My end-result is calendar via Google (only solution I can get collaboration and invites working properly), files via Dropbox, mail with Fastmail (much better than Gmail) and contacts via Baikal on my own hosted server.
Fastmail is adding support for calendering and contacts via CalDAV and CardDAV, but I've yet to fully investigate that.
It's a moving target for me, but currently it's been stable for a few months, and I'm mostly happy.
Yeah, it was completely unusable for me (a few months ago). It ran a separate process for each file upload, taking a few seconds minimum and completely serial. I estimated something like a month and a half to do the initial sync of 100GB, with a mix of large and small files.
I once gave git-annex a serious try (a few weekends of all-day efforts), as a way of sharing my music library between personal machines. It seemed to fail pretty hard on the cross-platform part. I really wanted it to work, I just couldn't get it to do so.
It was great at first on a single Linux machine, but once I had to support Windows, things became much more complicated. Most of this I attribute to Windows - different filename character restrictions, no usable symlinks, git only kinda-sorta supported through cygwin. Setting up an SSH server under Windows was something I didn't want to do, and HTTP sync was pretty buggy (different hashing schemes between the two computers, for some reason).
If things have gotten better, or someone's written a good tutorial on having an actual decentralized repo shared between Windows and Linux machines, I'd be interested in giving it another go. Also might need to hack up a mobile app.
I haven't dug into Syncthing's protocol yet, but I also like the idea of being able to easily share a single sync'd folder or file with an outside person, but not having to point them at a full repository.
Can you explain to me why and how I would use git-annex for real time sync? From what I understand of it, it works well for backups and for managing media files, but I don't understand how it handles the use case of making two directories on two computers pretend to be one.
One thing I've never gotten about these "syncing" apps...
Lets say I install this software on my phone, my desktop, and my work computer. I have 100+ GB free on my work computer and my home desktop, but I only have 16GB on my phone. If I add 20GB worth of movies to my sync folder, its going to fill up my phone.
Dropbox has "selective sync", so individual PC's can choose to only sync a subset of folders from your account. Also, phones running Dropbox don't actually "sync" anything to the phone. You can choose to download individual files, or stream them straight from the Dropbox app, but it doesn't offer you any way to keep files on the phone up to date with whats in the "cloud".
I use dropsync on my phone. I have set up rules to sync screenshots to a different folder from photos, and periodically it will check a folder on dropbox and sync files from it to phone.
git-annex allows you to see all the files, without having the contents; then you can do "git-annex get <file>" to pull it, and when you're finished, you can do "git-annex drop <file>" to free up the space.
(It also allows you to do full automatic sync, or just automatic upload, etc. And different machines can have different configurations, e.g., RaspPi syncs everything with my VPS, but my Nexus tablet doesn't)
The two clients I have used (Dropbox and Bittorrent Sync) work on a selective basis - you can download a file on an one-off basis, or keep selected files synced, rather than the whole repository.
Dropbox, at least, lets you select which directories get synced per device, i.e. so your desktop and HTPC can share movies but your phone never sees any of them.
I just setup Sparkleshare (http://sparkleshare.org/) this weekend. I wanted something which wasn't Dropbox, and preferably open-source, and while Sparkleshare has a slightly clunky pair mechanism, it works beautifully.
Syncthing looks similar, and LAN sync'ing is a killer feature for those of us in offices with poor bandwidth.
Dropbox actually have a "Sync over LAN" checkbox. It just never seem to been implemented/enabled.
I actually just dropped Dropbox for btsync the other week and are very happy, it's iOS app even serves other clients with files over LAN.
Btsync however uses quite crappy crypto and is closed source etc etc.
Syncthing mentions they use TLS encryption, hopefully i can roll my own 8k keys.
I tried Sparkleshare last year aswell, but my experience was mixed. Also it relies on quite a different architecture (git) than a "versioning bittorrent protocol".
Looking forward to evaluating syncthing when time permits.
The last few times I've set up a new machine locally I've used Dropbox to pull over stuff. It usually moves at 40-50 MB/s (megabytes) which is certainly using my gigabit LAN.
Confirmed by [2], that Dropbox LAN sync indeed only works once the file reached their servers.
"Well, when you add a file to your computer's Dropbox, the file is then synced with Dropbox servers. Dropbox will then initiate the syncing process as soon as it determines a change has been made to the file. All linked computers and shared folders will then download any new version of the file. With LAN syncing, Dropbox will look for the new file on your Local Area Network first, bypassing the need to download the file from Dropbox servers, thus speeding up the syncing process considerably."
I never had much luck with Sparkleshare - in particular, I got burned by it a couple times when files I placed in the shared folder never got synchronized and I was not notified of this anywhere. Get to office, and oh crap my files weren't there..
I've been using Syncthing for some months now and it's working well for my use case of keeping laptop/desktop and home server files in sync. I had one occasion when I lost everything as I'd brought up syncthing on my server without the "sync" directory mounted. It happily deleted all files from my synced laptop as a result. That's now fixed, but it was a buttock clenching moment. Yay backups, and a third machine (desktop) which was suspended and thus out of sync, so still had my data.
Upstream developer is very friendly and attentive & seems happy to discuss new features and use cases.
Well, I can't be sure, but I think colloborative technology for rearranging threads on-the-fly might be slightly less complicated than monopoly-busting or time-travel.
That looks like a promising project in a space which definitely needs improvement. Owncloud and Sparkleshare are okay, but they are far from perfect, and there is large room for improvement.
How good is this at traversing firewalls? AFAIK, Dropbox will do some manner of HTTP trickery to allow syncing when behind overly-restrictive firewalls (so it just goes out the usually-provided web proxy), but the documentation here references forwarding ports + UPNP, so I'm guessing that doesn't apply here?
"Each node scans for changes every sixty seconds". There isn't any portable way to get notfications about file changes instead of polling? I know about jnotify in Java but well it's in java.
You may have picked this up elsewhere in the conversation, but they are waiting on Go to have good cross platform support in the standard library for file change notification. This was originally going to be part of 1.3, but it's been pushed off to the next version of Go.
I deal with a constant stream of these and want to have a distributed network - connected via the inet - that allows me to sync the drives in all locations.
i.e. I would like to setup a server in my home office, one in my co-founder's home office and another in my editor's home office.
Whenever my editor runs off a few hundred GB of data to a specific folder or to their drive, I would love for that to be auto-synced to both my server and that of my co-founder.
Will Syncthing allow me to do this easily and will it be appropriate for an application like that?
It will do what you describe, but since you are handling large files you should know that Syncthing currently doesn't do dedupe, so if you rename or move a file, or have multiple copies of it, it will sync the whole file again. Changes within a file that don't move are handled efficiently.
This looks very promising. But the documentation is not good. I have not managed to find a "1 minute" friendly overview of how it works. I mean, what data gets sent how where and why.
Indeed, the documentation leaves much to be desired. I wanted to give it a try but I couldn't figure out what IP it listens on for the sync protocol. It logs a message to console about the web interface listen addr:port but nothing about sync. Finally found it under 'Settings', hidden up in the corner.
"Then all current commercial services drop off, including SpiderOak, Bittorrent Sync and git-annex. This resulted in a clever combination of EncFS and dvcs-autosync. Because, in this day and age, you cannot trust any "cloud" provider with your unencrypted data."
The author was uninformed, though. git-annex has supported encryption for special remotes (including your own rsync.net) since the early beginning (the wiki page[1] has been there since 2011). And it's certainly open source.
A few months ago I looked into using Syncthing for my decentralized browser, Syncnet[0]. At that time it did not seem ready for primetime. Does anyone have a good feel for its maturity as of late? For example, is there an API? Syncthing looks very promising and I would love to integrate Syncnet with it.
I am very happy with the timemachine backup on my mac, but I have been looking for a good offsite backup solution so that I can trade storage with my family in case something happens to my house. This might finally be the right option. BT Sync seemed ok, but was more than I wanted my parents to try and setup.
Does this need to use the web gui? I tried setting this up on a digital ocean server and it spawns off a webserver on 8080 that I can't reach from my machine. Is there a "headless" mode for client-less servers?
Thanks
This helped me ( though I'm sure the ssh tunnel would've worked, I couldn't get it running right) -- I changed the web address to my ip:port, and could then log in, add my host, and then change it back to headless or localhost. Thanks!
It's closed-source; we can't fix things like the crappy crypto (shared key), and can't spot backdoors or see what it's actually doing without a lot of effort.
@damncabbage is right. I wanted to use it to integrate it into photo sharing application for mobile devices but I couldn't because it's closed source. This has a lot more promise.
Does somebody fund projects like this? Or is it just that the people in charge of them understand something about UI and marketing? Seems like a nice trend, if so.
Since I installed Bittorrent Sync, my need for such software stopped as it works really well and provides all I need from it.
I couldn't understand quite advantages and why would I replace BTSync, which BTW, works really well already and does all this nice things. Plus works on my Phone and Ipad and Nexus.
To clarify one thing, I have home server which obviously hosts BTSync repos with ample space. Ability to fine-grained share parts of it is invaluable.
So this may be as good a thread as any to ask for assistance:
My small business is looking for a combined file collaboration / file backup service that doesn't cost an excessive amount of money (we're a non-profit on a budget). Is there a good service for this? For example, Dropbox is mainly for sharing files, whereas Carbonite is mostly for backing up your computer. Is there a solution for both?
SpiderOak does both. Their interface is, IMO, not nearly as good as Dropbox (I've never used carbonite), but it does check both boxes (backup and sharing/sync).
We love to through around the words "alternative" and " replacement" it is neither until it is as easy to use/deploy as X for the average Joe. You know 90% of people out there who would't be able to out the words 'git', 'deploy' and 'heroku' in the right order as their eyes glaze over in confusion.
The reason is because most of the options; owncloud, btsync, seafile and syncthing dosn't upload to a unknown company server. It only shares the files between your devices. Its a huge pluss when you think about security. The downside is that a device have to be online to share or update the files.
I use BTSync and to avoid the problem i just upload the files to my server and display them under a subdomain.
http://pub.volpe.pw/pub/fox_gl.jpg
BtSync also offers a phone client, so to mimc the way Dropbox uploads pictures to dropbox was rather simple. I backed up a gallery on my phone and synced it to my server and symlinked it under the pub dir, as BTSync dosn't follow symlinks.
Once i had setup the subdomain it wasnt hard at all really. I only had a problem where nginx couldnt serve the files from my home dir (i preffer to have the syncing folder there), but it was solved quickly with google and SO.
I love that public directory feature! I ended up using seafile and just maintaining a copy of a public directory there using the command line client. Works like a charm, very easy to setup.
This looks very promising. And it's written in Go! The only major feature I think it's missing is file versioning.
I am curious, though: what do people use to get their files remotely? And what's the cheapest solution for hosting your own central server? Would a simple AWS instance work fine?
I really like the idea but one thing is stopping me: portability to iOS and Android devices, and mobile apps that work with Dropbox. Dropbox has a first-mover advantage.
This is mostly a problem for people like me who use both Android and iOS devices so alternatives need to support both platforms.
Yet another sync app is Pyd.io
The Web UI is super neat, and you can choose between various backends for storage. Pyd.io offers its own sync app which I found to be horribly slow. I'd suggest to use Pyd.io as a frontend and BtSync/Seafile/Syncthing as a backend.
In contrast to Syncthing, Hive2Hive is more a library, providing an API for any kind of program that requires file synchronization.
Another major difference is that Hive2Hive is written in Java and not in Go.
It would be quiet interesting to point out the differences between Hive2Hive and Syncthing... H2H uses for example a DHT as underlying peer-to-peer technology.
As soon as someone makes a Dropbox alternative that has no servers, has selective-sync to my iPhone, and is compatible with Keepass to store the database, then I'm there. I'd happily pay for it, too.
another alternative: http://www.filement.com/
I don't use it but friends do and are pretty happy with it.
From their home page:
- Combine devices and cloud services into a single interface.
- Transfer data between computers, smartphones, tablets and clouds.
- Manage and use data directly on the device or cloud it is stored.
Bittorrent Sync just overwrites files based on last mod time (terrible option). What does this do? Does it support backups? Versioning?