This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs.
Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.
- Under Linux and FreeBSD, bazil fuse offers a nice low dependency path to supporting FUSE. It is all pure Go code.
- Under Windows, you can use WinFSP, which is a rock solid usermode filesystem driver with FUSE support. Despite its name, you can use this with Cgofuse to get FUSE on Windows without CGo.
Combining the two above gets you pretty far already (though if you want to actually avoid a CGo dependency you do need to explicitly disable it.)
I think the next level would be to have a general filesystem interface (there are several options in the ecosystem already) that can be used as a common ground between different usermode filesystem and network filesystem server implementations. Then ideally, a magic Go library can exist that gives you the best possible setup with switchable backends depending on the platform and build configuration.
Right now Go itself is possibly standardizing a read-only filesystem interface, an extension of this with write support might serve as a decent jumping point for getting toward an idealized pluggable filesystem library.
It won't be enough for the likes of FUSE, though. Some extensions might be enough to bridge that gap, it's too early to tell. FUSE is pretty particular about things like rename(2) preserving node identity.
Datasaur looks awesome! Can't wait to try it out. Congrats on the launch!
Curious about data security and privacy? How do you guarantee privacy? Is there some cryptography or secure enclaves used? Some sets of documents (and email) are super high trust.
Guessing the on-prem version is probably safest route
Thanks - good to see so many people concerned about privacy here. I consider a privacy a top-level priority at Datasaur - all data is fully encrypted. Our employees will never be able to see or access any customer data. We already work with a bank and cleared their security bar :)
Hello! o/ We've responded to a number of points here.
You can check out our roadmap
> IPFS is unworkable... making it usable in the next few years
Absolutely not.
The OP brings up a lot of great, useful feedback for us, and we'll respond to it.
But the OP is also simply wrong in saying it's "not usable". There are millions of end users benefiting from IPFS, 100Ks of libp2p nodes, we see PBs/mo of traffic in the infrastructure that we run, and millions of daily requests to our gateway. Look to fully decentralized applications and systems like OpenBazaar, Textile, Dtube, and others.
Beyond that, we're well aware of the many shortcomings, and working on them. We're unfortunately spread thin across a lot of projects (IPFS, libp2p, filecoin, ipfs-cluster, etc), but each is seeing significant growth and improvement.
Decentralizing how we link, address, and move content is the basis of IPFS and Filecoin both. Both are solving different parts of moving our digital infrastructure to infrastructure that we control, that we govern, and that we are not held hostage by.
While we still have a long road to go w/ IPFS usability, it’s worth pointing to what works today -- there are millions of end users benefiting from IPFS, 100Ks of libp2p nodes, we see PBs/mo of traffic in the infrastructure that we run, and millions of daily requests to our gateway. Look to fully decentralized applications and systems like OpenBazaar, Textile, Dtube which use IPFS and paving the way. New technology -- especially new technology that seeks to build new platforms from the ground up -- is hard and extensive to build, and to polish. We’re working on it, and though nowhere close to where we want to be yet, there’s a lot of utility already provided.
Re Filecoin incentive structure -- this comes from recognizing that computers are not all the same -- there is huge utility to be had in dedicated infrastructure running 24/7, well maintained, high performance spread out around the world. P2P systems of the past have failed to match the reliability, uptime, and performance levels of centralized systems. Cryptocurrencies enable the creation of Open Services (like Bitcoin) that run public infrastructure w/ high uptime and reliability guarantees. With Filecoin, we aim to bring that to file storage and distribution. More here: https://www.youtube.com/watch?v=6h2WNxEV8q4
P2P systems historically have dealt with the problem altruistically, or with limited tit-for-tat, which both work in many cases, but have so far failed to work for large scale long-term resilient systems.
This is where Bitcoin managed to do something remarkable: achieve high uptimes typical of the best centralized systems, through a very clever, but still open and permissionless, economic incentive structure. Markets have been shown to be extremely useful to create a robust Open Services. More on these ideas here: https://www.youtube.com/watch?v=IfLIoOr4p0A -- we think this kind of thing is going to lead to extensive, global, public utilities run w/ internet-native money.
But it will take a while-- this stuff is extremely difficult to build right now-- it feels similar in nature to very early Web, or pre-unix systems. Lots of hand-rolled primitives, many with the capacity to cause serious failure (not very old cryptography, and complex security questions). Perhaps better programming languages will help us build these systems dramatically faster/easier. For now though, you can see the entire blockchain space wrestling with these problems.
This is great! We’ve been wanting something like this for a while-- and there are a bunch of utility in bringing ENS and IPFS together.
A lot of the problems you mention are known, and being worked on. Let me describe a bit more in each.
> Technically, IPFS work well for us, but we made sure to have a server seeding our website at all times (where we suffer similar problems like the author of the post describes).
The IPFS content model requires somebody with interest in the data to keep serving it. So for now, yes you need to keep some ipfs nodes with the content around. For example, we serve all our content (all our websites are distributed w/ ipfs) using ipfs-cluster, which connects to the gateways. We’re working on ipfs-cluster and filecoin as the ways to solve this. Ipfs-cluster for when you want to run your own infra (or a community gets together), and filecoin for when you want to hire someone else to serve it for you.
> Even then, we still had to make sure that the website is available in all main gateways, since most people don't run their own IPFS daemon. The strange part is that sometimes content is available in one gateway, but not in others. Or sometimes it's available in one gateway, but we can't get it on our local IPFS node.
This is a big problem that we’re working on right now. Many of our recent releases aim to fix / reduce this problem. The nature of this comes from content-routing scaling. We’ve detected this getting worse as the network grew orders of magnitude, and we’re working on fixing this right now.
> I understand that IPFS is not a blockchain, so I can't expect all the nodes to have the same content. But I do expect the main gateways to communicate more directly with each other.
Yes, definitely. We agree-- on it.
> Conceptually, IPSF websites are a bit like sending your website to someone via an unreliable slow mail; i.e., it's not that attractive. You can make it somehow more dynamic, using what they call IPNS (it allows you to update your content). But the result is so slow, that even the most devoted monk would lose his patience eventually.
IPNS key names are just not working well -- and getting worse with scaling. We’re working on fixing it, but lower prio than other more important problems. For now, we’ve been directing people to use DNS -- check out https://docs.ipfs.io/guides/concepts/dnslink/ -- these should be fast for you. But yeah, figure you know about this and may just be avoiding DNS in favor of more decentralized tools (ENS, IPNS key names).
Also check out https://dev.peerpad.net/ -- this is an experimental tool w/ dynamic content (give it 10-20s -- unfort takes a while to “get online” -- this is the dev pad). Once peers are connected you get a pad that’s distributable across ipfs nodes.
> A workaround is using a decentralized name system, like ENS.
> This works very well, but the result are still static websites. No comments or anything really interesting happening.
+1 for ENS. And, you could use ENS to point to something like the content hash of peerpad above (see the latest content via `ipfs dns dev.peerpad.net` or `dig TXT _dnslink.dev.peerpad.net` == /ipfs/QmWbsqqqG9YpNYDt5afp6HY8TrKMtCtdGUtUfgkS9fRYeH -- and this would be a _static_ html5 bundle that gives you a _dynamic_ local app. The hard part is backing up your content beyond your own browser-- pinning it to an ipfs-cluster or other tools, etc.
Anyway, the picture is clearly not there and usable yet, but the implications are big: you can get fully distributable p2p apps w/ fully dynamic content. (you could even encrypt the app bundles themselves, but this needs an extension to decrypt browser side) -- this feels like this: http://www.accademia.org/wp-content/uploads/2014/01/slaves-b... -- the form is emerging, but much work to be done.
> Those websites are censorship-resistance and very robust, you don't have to worry about ddos attacks. But then again, how many people worry about such things?
> That said, I still like the concept of IPFS. We are exploring a few options to add dynamic behavior to that now, where the dream is to mimic existing services in a decentralized way.
> Surely, they won't work as well, but the pro would be that it will be controlled by the users, and that they will be able to survive financially with no ads.
It takes a long while for all of this kind of tech to come together. Keep at it-- you make significant progress YoY, and it adds up. What we can do now in 2019 is vastly superior to what we could do in 2015. Big innovation leaps take significant time to develop -- but the good news is _a lot_ changes decade over decade:
Peerpad looks awesome, and is exactly in the direction of stuff we're interested in. Ironically, a friend sent me its link literally 5 minutes before I saw your comment. Great minds think alike.
We've been playing a bit adding a p2p network in the background of static ipfs website (using webRTC). That way if a website has enough visitors, they could mimic to each other many functions that normally a server does. I did similar things in previous projects I was involved in. But there it was native softwares, where here there are extra challenges caused by the (relatively) limitations of web technology.
We think it's great that people ask hard questions, and get involved. It's great to see others studying our work and we really appreciate the open discourse. There are a few things from this article I’d like to address. (Despite the length of this post...) these are quick comments, and not a proper in-depth response.
- (a) The article gets some things right and some things wrong -- there is good summarizing of several of our projects, and discussion of many difficult aspects in these projects. The article discusses many technological aspects in good depth, and highlights difficulties in building these systems, aligning incentives, and the trials of past projects. The article also has significant inaccuracies. For example, the sale figure -- which appears in the title and impacts the analysis -- is incorrect. We raised $205M -- officially here: https://protocol.ai/blog/filecoin-sale-completed/
- (b) The authors chose a provocative title. As some commenters have already pointed out, the conclusion is “[we] believe that it is not one.” Despite Betteridge’s law ( https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines ), many people who only read the headline will come to the opposite conclusion, and now we (not they) will have the burden of correcting those misunderstandings. Provocative titles, though they may drive imagination and clicks, can do a huge disservice to everyone in the space, and contribute to misinformation. Most people will only read the title, maybe the abstract, and use that to form and drive opinions. We choose titles of our research with diligence and care, and hope others do the same.
- (c) The article has a great technical overview of the Filecoin stack, and how it fits with IPFS and libp2p. This is a large structure with many pieces, and it is rare to see articles grasping how all the pieces fit together so well, and then explaining it cogently. In particular, it’s great to see this article diving deep and discussing advantages and disadvantages of low level technical structures (multihash, ipld, libp2p, and more). We modularized everything in the hope to generally improve peer-to-peer systems, and improve reusability. We hope these components will be useful to the author’s Tribler project (a network similar in goals to Filecoin), and we hope that we can also learn from and leverage solutions they have made.
- (d) many of the objectionable things described in this article are common in ICOs in general. Put another way, consider those claims also in terms of other significant token sales, such as Ethereum, Tezos, Polkadot, Blockstack, Cosmos, Golem. People were saying similar things about Ethereum when they did their sale in 2014. Perhaps worth doing a survey / analysis over all of them, comparing and contrasting the different things groups have done, how the ecosystem has improved, and suggest new directions.
- (e) It’s worth mentioning that the analysis gives a definition of a ponzi scheme, but their discussion does not map to that definition. Instead, the discussion centers on claims about future investor sentiment or speculation as the driver of value in the token, which is not the only way to establish value in token networks, and ignores the value of the services provided. That kind of analysis does not work for projects like Ethereum and other live and functioning crypto tokens. If the network is useful, and there is a way to generate or introduce value, by providing new or better services, and if the network can capture that value in the token itself (important step), then the tokens can hold value, based on the utility of the network as a service and not just or primarily speculation. Networks like Ethereum, Bitcoin, Zcash, and Filecoin aim to provide useful services, and much of the value stored in their tokens will be thanks to the utility of the networks.
Perhaps it’s worth pointing out that most crypto token projects are compared to ponzi schemes at some point :(
- (f) The article discusses the SAFT and assurances to investors, but does not discuss them in contexts of other token sales and ICOs. Most ICOs are structured as donations (not investments) to a project, with little to no legal recourse -- even though many people refer to these “donations” as being “investments”. In our case, we raised investment through an instrument (the SAFT) that is a direct liability to us, and gives investors greater guarantees on the completion of the project, or consequences otherwise. If we fail to deliver the network, we must return the proceeds of the token sale. Few token sales ever have such a clause. Our structure gives investors greater accountability, not less. The article discusses this in sec IV, but does not take into account that startups are similarly risky (i.e. that startup dissolution events return only remaining capital from the efforts), and does not mention how our structure improves on the ICO landscape in general.
- (g) We do share the legitimate concern that ICOs need stronger accountability, and some are structured in a way that leads to abuse. The community as a whole needs to raise the bar on accountability and ethical behavior. We have taken significant steps in this direction, not just in our sale but to improve the ecosystem -- the SAFT project, which was a gargantuan undertaking that many other networks are now using, is one example. Many other networks are introducing and improving structures. We believe token networks present a very important new way to form capital, with promising advantages to users, investors, and creators, but the space is still in its infancy, and significant changes are still ahead. Token sales have improved dramatically in the last three years, and we hope they continue to improve to find the right balance and protection of the interests of all parties involved with the network.
There are a ton of technology differences which the upcoming Filecoin v2 paper will address (very soon!).
On the conceptual layer there's a difference in that Sia and Storj seem to be a bit more geared towards consumers and file storage, while IPFS+Filecoin's aim is to be infrastructure in the commons.
