Let me tell you an anecdote about Owncloud. Once upon a time, I hosted an Owncloud instance for my family. I had a raid 5 system setup on an old Xserve with around 10 TB of usable space. I was really into learning about encryption at the time, so I figured it would be cool to encrypt all my families files on the server. I made sure that all my family members were properly educated about not forgetting their password, and then enabled the Encryption App[0] by following a tutorial. Everything went smoothly for about 6 months until I was updating Owncloud with a minor version upgrade, and then BAM; nobody (myself included) could decrypt any data on the server.
I posted around about it, trying to figure it, ripped apart the source code to the point where I found that manually decrypting the data was resulting in gibberish. All my backups were the encrypted data, so those didn't really help either.
I still don't know what happened to this day, but I've been eternally wary of personal fileservers ever since.
Replying to my own comment, because I'm out of the edit time-range and this kind of blew up.
If I had to point the discussion of this comment, I'd point it away from what I did wrong personally (I know I f*cked up in probably more than one way).
What I learned from this is that there are no situations in which I would personally host a file server of this feature level and control over my documents. I think "all in one" (dropbox/backup/encryption/sharing/file storage) personal solutions like Owncloud try to do too much, and end up with some seriously rough edges that can turn off very large portions of their user base.
I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"?
This is why you want client side encrypted clients who treat your NAS/Cloud Account/HDD as dumb storage. This way it's easy to immediately verify on your client computer that entire CRUD cycle works fine, and continues to work fine.
It also allows you to have simple redundancy, so if your NAS setup fucks up, the cloud account and external hard drive version is still there.
If anyone is interested in client side encryption, checkout my project securefs (https://github.com/netheril96/securefs). It is better than TrueCrypt as its encryption is randomized and authenticated, hence more secure, and it does not require preallocation of storage space.
I do something similar too, but I have around 1TB of data, so a full restoration will be painful. Remembering to plug in an external hard drive is irritating so I want to get a NAS that my laptop auto connects to, like time machine. What do you think?
Time Machine would work. Backblaze.com is an option as well ($5/month for unlimited storage, they'll backup an external drive) if you're looking for offsite backups.
> I guess the question I raise is, given my experience "can you prove to me that I can absolutely trust you with all my data"?
But the answer to that is "no". All of the time. No matter who you are asking it about. There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself.
>There is always the possibility of someone - no matter how much you trust them or pay them - screwing up your data. Including yourself.
Especially yourself. I trust a product with dedicated administrators who do nothing but keep people's data safe to do a much better job than i could ever do myself. I don't want that kind of responsibility when google and dropbox are willing to do it for me for a couple bucks a month.
Yeah, pretty much summarizes my fears with Owncloud. Not only that, but as mentioned before, I no longer have any desire to manage my own large scale storage (NAS/Raid, really anything), so that compounds my own fears with Owncloud.
I don't mean to belittle the severity of the bug you encountered -- but the root-cause of your problem was a lack of a backup. A large number of issues, some of them software bugs, many of them not, could lead to data loss.
No. The user should have a backup, but you can never blame a software problem on a lack of backup. That's like blaming a car-accident malfunctioning airbags.
The software screwed up and his backup strategy failed. The software didn't screw up because his backup strategy failed.
Yep, if your backup software doesn't have historical snapshots where you can jump back to a certain day, you are just going to write bad data as your backup and only have bad data to restore.
>>> Everything went smoothly for about 6 months until I was updating Owncloud with a minor version upgrade, and then BAM; nobody (myself included) could decrypt any data on the server.
> Do you back up the unencrypted data though? Regardless of the viability of the raid 5 system, the raid 5 wasn't what failed. It was the encryption.
> It wouldn't have even been possible to backup the encrypted data without getting all my family members passwords too.
You could have backed up the encrypted data (+owncloud metadata) in its encrypted form. Then, when you ran into the bug that corrupted the main copy of the encrypted data, you could have restored the old backup of the encrypted data and reverted to the last working version of Owncloud to access your data.
If you're backing up encrypted data, then yes, you need to (facilitate that your family) back up the keys to that data as well. Obviously not in the same place, as that negates the purpose of the encryption.
Backups are hard and you absolutely need to do them right, otherwise they're pointless.
The reason you need backups is exactly that you can't trust other people (or yourself) not to epically fuck things up. No matter how unambiguously totally someone else's fault it is, if your data is gone, it's gone.
If you want to rely on backups, you have to test them periodically and not overwrite older ones with newer ones. It sounds like the backups had probably never worked, or there was some necessary component that was not backed up.
Considering the number of problems we describe in backups
1) You need incremental backups, not just a live copy, so you can rollback.
2) You need redundancy both connected/disconnected and offsite.
3) You need to test restoring.
All these requirements... how do we not expect the user to screw up? This is like crypto, except every idiot knows "don't roll your own, you'll screw that up and leave a hole".
Same problem with OwnCloud itself - it requires too much configuration, as the OP explained.
Why are we doing all this garbage manually? Where's the automatic backup solution that provides differential incremental backups so you can rollback to various points, integrates with Google/Dropbox/whatever, integrates with OwnCloud, lets you plug in an external HDD for regular disconnected backups, etc.
Right, no arguments here. Backing up is really hard. Many backup systems are "push", and so you have to limit the production system's access so that it can't maliciously overwrite old backups.
From what you describe in the last paragraph, that sounds like a special Owncloud client. It needs decryption keys and it needs some automated tests for verifying the backup which vary based on your use case.
I've recently been testing git-annex with the webdav special remote, and it mostly solves the integrity piece but a) if you want encryption you lose the web UI, b) even if you don't want encryption, you still lose the web UI because there are no indexes, and c) it would be too high a bar for most people to set up.
So, what, everyone is cool with owncloud pushing encryption-breaking updates? What the fuck is that about?
The way apologists step up to the defense of stuff like this is appalling. The 'hard' stuff like this is supposed to be dealt with before the public release. Just because one has a backup doesn't mean that other programmers should force them to rely on it (whether they intended to or not... the point is that this kind of thing should never happen)
I am beginning to feel like we should start figuratively burning people at the stake for advertently or inadvertently destroying other peoples' data -- it is a sin about as cardinal as they get; and it seems many programmers are clearly not very scared of such an outcome. Wiping/bricking/rendering inaccessible your customers' data should be causing people to shit bricks, not go on the defensive.
Normals HATE technology bullshit like this. They just want it to work. Failures like these make the entire field look bad.
I don't think I've seen anyone say owncloud is not responsible. Yes, they fucked up, and royally.
But in the end, you can't trust them to not fuck up. You have to trust that they will make a mistake, so be prepared for it. Doing backups right is just really hard.
Which, I think, was alexggordon's point. That he no longer wanted to manage the data himself because it was a hard problem, better managed by teams of professionals who have the time to do things like verify backups because they can amortize the cost of that effort over tens of thousands of customers.
Maybe, but at the time of writing many of the responses here felt like the comment I originally replied to -- splitting hairs over pedantics and blaming the user for not testing his backups well enough. I still feel very strongly that he shouldn't have to
> I still feel very strongly that he shouldn't have to
As nice as that idea is, I don't think we have the tech yet to fully trust someone else with your data. You have to test it out that your backups work. Anyone who tells you otherwise is selling snake-oil at best.
That is not the lesson here. It is that a backup should not backup the encrypted data, but the unencrypted data, and have its own encryption. A simple rsync based bashscript can be fine for that, though I'd use borg.
Having 2 local copies with local monthly backup for a year and an off site backup (CrashPlan) is the secret. CrashPlan can go back in time for issues like this.
As a Systems Librarian I did the daily (31 days worth), weekly (52 weeks worth), monthly (48 months worth) back ups with 131 tapes was a pain but it always paid off. I just had three machines and needed to go to my tapes once a day.
It was a library with 500,000 items being used all the time. I had Librarians who were adding and purging our collection everyday full time and students and faculty checking in and out books etc. all the times.
Insider information all the Library Management Systems are totally broken and garbage (Open Source Evergreen is doing things better). I was required to reboot my Windows Server (Purchased before I was hired) everyday. I was told this was totally acceptable. The Oracle DB was solid but the rest of the stack was total and complete garbage that cost over $125,000 and service contracts of $30,000.
Edit: Off Site aka IT Office Fireproof Case was my monthly and previous weeks. Once a week I walked them over. In the library in a fire case were the daily, and the last week. Tape was it was 2007.
Every client would need to make its own backups, automatically. In such a situation it would be probably best to have two backups, those plus a backup of the encrypted data for the clients that fail to do the backups on their own. But you raise a good point: That's difficult to do.
I think it is unfair to extend this specific experience to all personal fileservers. And TBH, I think that encrypting the storage shouldn't be done on this level. Setting up the OS that runs owncloud to have encrypted data volumes is the better way. If encryption is needed at all for taht.
Transport encryption between the various devices and the fileserver is a must, however. But going beyond that is not something I would consider to be a good thing.
I like how OwnCloud was characterized on Bad Voltage (paraphrasing here): "It's as if someone a few years ago wrote down their definition of cloud and sent it back in time to 1996. Then a bunch of PHP developers in 1996 implemented it knowing nothing about what they were doing and then sent the code back to present day."
That's not to say I don't appreciate the fact that ownCloud exists, but I find it much harder to use, and the fact that it's written in PHP makes me not want to dig into it to fix.
I tried Owncloud a year or two ago, the feature set is pretty remarkable for an open source project. But between user error, software upgrades, or bugs, relying on it is just not viable. There's been too many horror stories of data loss and corruption to trust it.
I actually wish that Owncloud would focus on file syncing and not on apps. Their app API feels like the wrong level of abstraction, more like you're building wordpress plugins than actual apps. I appreciate that it makes it easy for devs to get started, but can result in some terrible security holes and messy APIs. Something like Sandstorm is a much better way to separate apps and content and prevent security issues.
Owncloud's syncing clients for mac/win/linux are descended from mirall and are pretty good. I used Owncloud server for a while then built myself a lightweight replacement in node that uses the same sync API. (webdav)
Did they ever fix the design of using a separate TCP connection per transfer, serially? It sure is fun to upload directories full of small files at 2 per second.
> You should have full control over your data. We help you achieve that: a safe home for all your data. Secure, under your control and developed in an open, transparent and trustworthy way. We are Nextcloud.
Why are we not in full control over our data with ownCloud anymore? What really changed that made the founder and top contributors leave ownCloud?
That is on everyone's mind, my guess would be that somehow some party within ownCloud was pushing for a bigger separation of features between the FOSS core and the enterprise product. This goes against the founders ideals and they left. Similar to OpenOffice vs LibreOffice, in fact I was personally expecting this to happen and was betting on the name LibreCloud ;)
<tinfoil_hat>
Their refusal to discuss reasons may indicate that a certain entity received a NSL or AWA+Gag to backdoor, and certain someones prefer to take their work to another entity not covered by said order.
</tinfoil_hat>
This is a statement for ownCloud customers to join Nextcloud. It's filled with direct comparisons with ownCloud without telling its name. It's worded to make ownCloud customers feel like it's not safe to use ownCloud anymore. But maybe that's my paranoid self misinterpreting it.
I beleive that's a statement about the core value of NextCloud, one that you can perhaps get from the OwnCloud product too, but not beeing a core value of OwnCloud Inc.
I think that's implied from the tagline "OwnCloud A safe home for all your data". And the Features page saying "Access, Sync and Share Your Data, Under Your Control!"
My question exactly, and also I was a little unsettled by how they went out of their way to not mention owncloud in the about page.
Why not just say "we're a fork of owncloud", instead they use forced marketing speak like "Nextcloud is a reboot of the most popular open source file sync and share solution", or "his previous project".
It would be interesting to hear a bit more background for this forking. What does Nextcloud try to achieve that was not possible within the ownCloud project? Was this a clash of personalities as one of the ownCloud founders is among those jumping the ship?
I sincerely hope that Nextcould embraces the C4 contract for community building. That would be IMHO the best way to create and grow a community that is truly open and will survive for a long time.
Excellent link, and thanks, but the branching thing is hardly touched on, other than a quick small paragraph that reads like "it's too hard for people to learn".
Jos Poortvliet, Owncloud's former community manager and now responsible for communication at Nextcloud calls it a "new start". Karlitschek and Poortvliet give as reason for the new founding "structural problems" and some "economic decisions" of the Owncloud company, that caused great dissatisfaction. They do not want to give further details. Nextcloud's business model should be focused on long-term viability and sustainability. Apparently the Owncloud company could not offer that in a satisfactory measure.
It sounds more like a (more) viable (or maybe, more profitable) business model went against the founders/tech-experts ideals and a collision occurred. I think it is very mature of Frank not to speak out on the issue while still in a probably somewhat emotional state about it. I'm curious what comes out when the dust settles, a statement from people left at ownCloud would be very nice and appropriate...
Based on what I heared from someone involved, it's actually the opposite. They don't want VC money to come in and loose theirs influence in the decision making.
There doesn't seem to be VC money involved with nextcloud. Spreed.com/me is funding the transition and stock will focus on being owned by employees it seems.
There's some telling comments in a thread on their forums[1]:
> So, the main thing we wanted is a sustainable company. A company that doesn't go away soon, that isn't forced by a bad funding situation to make stupid decisions, or to limit itself and not work with partners as much as possible.
> A similar fork? Well, if Frank and the rest of the engineers lose trust in Nextcloud, we won't have to fork as this time, we will make sure the company won't fully control trademark and code. No more CLA. Trademark in a foundation. So this is the first and last time a fork happens
>'why' is the question everybody has and I hope you understand I don't want to talk too much about that
Grow up, maybe, Jos?
This sounds to me like the usual and continuous splits inside OSS communities, but Jesus, I do miss the days when you had a detailed explanation as soon as somebody leaked the IRC logs!
You don't talk negative about other people, especially if you worked with them for a long time, and especially when it's not a matter of "they hid bodies of people they killed" but mere differences in opinion about how to do business. So no, this dirty laundry should not be aired in public.
On the other hand, that specific thing is the entire reason for the fork existing, its drive and motivation.
It could be anything. "We feel own-cloud isn't libre enough", "We don't feel this direction is profitable", "We feel owncloud was too open", "I personally disliked one of the people there any wanted to sink the ship", etc etc (note: I am being a bit tongue in cheek here)
If its a reason I agree with, then migration makes sense, if its a reason I don't agree with, then obviously migration might not be a good idea. But no one knows what the driving motivation of nextcloud is! So its a bit more worry-causing then reassuring
9 of the top 10 contributors in addition to the project founder and the lead security engineer are pushing this effort. Good foundation to fork and support users.
Let's hope this brings the transparency, openness and community awareness owncloud/nextcloud always needed.
They were 90% of the top contributors. The company had a lot more sales people. The devs don't control decisions of the company. Only the code and that's why I assume there was this fork.
Control can be taken and views/visions can be different. That's the great thing about open source. The community or even other participants can fork and continue their own vision. Projects shouldn't make a habit out of it so.
Given the amount of negative voices I hear each time OwnCloud is mentioned - and that includes German tech websites where this German company is mentioned more frequently - I don't know why the departure of the original developers should necessarily be a bad thing. The amount of negativity of previous OC users was unusual, too much to put it down with a "you can't satisfy everybody". I'm not even talking about the "OMG they use PHP" comments.
I must say does seem to be at least partially the fault of developers, since they likely had much more responsibility than upper-level managers since it was a startup founded by techies for the longest time.
So if the company manages to leave some of that baggage behind this may be quite alright.
Having worked for another German software-heavy company I must admit I understood all those complaints, because our own products were similar: Pretty good in principle, but way, WAY too little attention to details that matters to customers. With each release the number of really stupid annoying little bugs never seemed to decrease (talking about "my" ex company, I only know OC from forum comments - but OC users' complaints sound like the ones we deservedly got to hear a lot).
> since it was a startup founded by techies for the longest time
Well … we do have capable people who know how to properly design and develop products. We always work to make it easy to use for normal people and not only for techies. And I think the success proves that we did at least something right.
Disclaimer: I was the design lead for ownCloud and now moved on to Nextcloud.
I used OwnCloud for a while and never had any problems with it, but it seemed like overkill for my use. I didn't have multiple users and I wasn't using it to share files with anyone -- just syncing files between my own devices.
I started using Sandstorm[0] for my personal server, and there's an app for it called Davros[1], which implements the OwnCloud protocol. It doesn't really do a good job of advertising that fact (not even mentioned in its description), but you can use the OwnCloud clients with Davros. It's been satisfying my file-syncing needs with no problems. Just be aware that it lacks versioning.
I'm excited to see what's coming for NextCloud, though.
I feel sad for ownCloud and Nextcloud. ownCloud has a brand and this fork dilutes it. It will be on everyone's mind especially since the founder has left to create a competitor. For Nextcloud, unless they can explicitly state why this fork exists, it's hard to side with them. What kind of ideological differences are we talking about? Sure, you don't need to throw around personal accusations or finger point but it would help to know what motivated this.
On a related note, this is a risk which many platform companies take. If you opensource your project completely, then anyone can fork it and take it ahead if the forker is a big or well known person. I think we got really lucky with iojs and node.
It’s not a »risk which was taken«. ownCloud started out as a pure open source project and the company was formed later.
Having it under a free software license / open source is exactly so that the community can step in when there is a problem with the company.
Disclaimer: I am one of the people who left ownCloud for Nextcloud. Also one of the first contributors to the project when it was still very young, and the company didn’t exist yet.
Having contributed a tiny bit to ownCloud, I enjoyed working with the people that now left to start Nextcloud. I think they were the more technical minded people on the team at ownCloud, compared to the sales persons of which there seem to be an awful lot.
You are of course very welcome to contribute again – just check https://github.com/nextcloud or join us in IRC #nextcloud or #nextcloud-dev (freenode).
A year ago I reviewed most of the popular "Self-hosted 'Cloud'" packages and found they all had pretty ugly history (security vulnerabilities in particular) and the technologies used didn't inspire confidence in the developers (when data security and integrity are on the table, PHP developers wouldn't be my first choice, nor any choice).
Are there any decent tools out there today or is it still the same situation?
Good questions! Note, that I'm affilated with Nextcloud so obviously a bit biased.
Only because a project is very transparent about security vulnerabilities does not necessarily mean it's inherently insecure. In fact, at ownCloud we found all critical vulnerabilities ourself and also run a successful bug bounty program. (for Nextcloud we are also considering one)
While Seafile seems to work, the developers have been kinda sloppy both with licensing[1] and security[2]. The answers on both issues really make me question their integrity and ability to deliver a secure piece of software.
The impact is a different one though. In that scenario pointed by Hanno somebody needs to have access to the storage which already requires some kind of previous gained access. What could be done by an attacker then is to infect EXE files or so.
In the case of Seafile one could simply change passwords of any user etc.
But yes, crypto is hard and I agree that the way we did it at ownCloud is far away from the best way. :-)
The reason I prefer Owncloud to Seafile is that my files are stored as my files. I have a script that copies the photos I upload to Owncloud over to my NAS automatically.
Yes, there's FUSE, but I still prefer the way Owncloud handles my files.
I've been mucking around with Owncloud from the beginning and it's always been a pain (non-default configs never work as you expect, updating is a pain, installation is a pain, etc). Tried Syncthing and it was worse (I want central control, SyncTrayzor is obnoxious, doesn't "just work" as it's supposed to via upnp). Never heard of Seafile before but it looks way better, I'm going to give it a shot!
I've been using https://pydio.com for the last 6~8 months, and so far so good. I came over from owncloud. Admittedly, I had planned to use for my family of 3, but I'm really the only one using/syncing stuff. THe server side for pydio has been great, though their Windows client has had a few little issues. In all cases they were easy to fix, and ultimately caused no damage to server data (who knows it could even be a local Win/laptop issue). Though pydio is php-based, while seafile IIRC is python (I like both languages so doesn't matteer to me, but thought you might want to know ahead of time).
I'm curious to hear if Nextcloud is going to be rewriting the iOS mobile client from scratch, since it's GPL and Owncloud owns the copyrights so is the only one who can relicense for Apple.
Most likely this is what needs to happen sooner or later. For now the existing iOS app can be used of course since it can connect to any ownCloud/Nextcloud server. Or any WebDAV app, for that matter cause we support open standards.
We will make sure though that this problem does not arise again since there will be a Nextcloud foundation which is in full control of the copyrights.
Disclaimer: I am the former designer of ownCloud and moved on to Nextcloud.
I've been using ownCloud for quite some time now with a handful of users and many clients across Windows, Android, Linux, and OS X without any issues. This is running in a FreeBSD jail.
I too wish it wasn't written in PHP as I'd be inclined to add features, but I haven't had any of the issues mentioned in this thread.
Yeah, no shit! This is no surprise to me whatsoever. Owncloud has been stuck in a rut for a long time! I'm assuming that taking 90% of the top contributors means they'll still be taking along a hefty amount of the politics, but maybe with a fresh outlook this project will become competitive again.
Owncloud is NOT a backup solution, it is a remote access/sharing solution. At least one client syncing to an account on the Owncloud server should have an incremental backup solution. My personal setup is that my OSX clients backup to a Time Machine. Easy peasy. If I have a server-side failure, I keep motoring on, albeit without remote access/sharing until I get the server fixed. If I have a client-side failure, I can restore to a prior moment in time off the Time Machine or sync to a new client off of Owncloud depending on the nature of the failure.
I implemented Owncloud in my workplace, but we had one problem that made us uninstall it and create our own solution in Dart, the problem was that when you share a file it doesn't preserve the file path extructure and that was very important for us.
*structure, becuase every folder has meaning, if they share the document "law_2016.pdf" that is in the folder "/Enterprise_1/documents/laws/info" the folder structure has information about the context of the file in this case, they now is a document of law of the Enterprise 1.
I posted around about it, trying to figure it, ripped apart the source code to the point where I found that manually decrypting the data was resulting in gibberish. All my backups were the encrypted data, so those didn't really help either.
I still don't know what happened to this day, but I've been eternally wary of personal fileservers ever since.
[0] https://owncloud.org/blog/how-owncloud-uses-encryption-to-pr...