Video Version Pt. 1:
Video Version Pt. 2:
Pt 1: What is IPFS?
If you’re familiar with a video I did on the experimental platform DTube then you are probably at least slightly familiar with IPFS, which I intend to explain today. IPFS, similar to BitTorrent (which I would like to point out is legal unless used to pirate content), is a decentralized file sharing network. Unlike BitTorrent, which focuses on creating mini networks dedicated to sharing one file, however, IPFS is one large decentralized network which contains all devices running IPFS. Similarly, though IPFS shares some similarities in structure to BitTorrent with a few key differences, the goals of IPFS are also similar to BitTorrent with a few key differences. The name IPFS is an abbreviation of “InterPlanetary File System,” and the network is structured with the idea of building one large network servicing many clusters of devices.
If you have seen the TV show “The Expanse” you can probably picture the theoretical use case: if there were networks on a number of planets the communication within a particular planet could be fast and reliable, but communication from planet to planet may be slow and expensive. With this setup, however, a file would only need to enter a particular planet’s network once before always being available within the network and no longer be required to be sent from a centralized server on another planet.
Put in more contemporary terms, consider the idea of inter-continental communication. Say a file is made available on a server in the northeast United States. With the normal centralized internet if somebody in Sidney Australia wanted to access it the data would have to travel around the world taking extra time and putting stress on every piece of infrastructure along the way. With IPFS, however, after the first person in Sydney requested the file the second person in Sydney would get it from the first, the third would simultaneously get it in parts from the first and second, and so fourth.
Finally, I’ll give one more example, this one a bit more technical to better demonstrate IPFS better. Lets say there is three LANs (local area networks) each consisting of 100 devices with IPFS. Communication between devices within the same LAN is 200 MB/s. Now, those three LANs are part of one WAN (wide area network) and communication between devices that are part of the WAN but part of different LANs is 100 MB/s. Last, any device communicating with the internet (part of neither the local LAN or the WAN) is 25 MB/s. With IPFS the devices would first try to download from other local LAN devices, then if that fails they would try devices on the WAN, and if that still fails they would pull it from the internet; prioritizing the ideal route first. Additionally, since the file is then cached on one of the LANs if a device on a different LAN requests it the file will go over the WAN, not the internet, saving speed (plus now it’s on two LANs) and so fourth. As you can probably see, the concept sounds promising, so I will continue on with the explanation, moving on to how you can use the system yourself.
Each file on the IPFS network is identified with a hash, which is a string of text that is a mathematical derivative of the file. The beauty of hashes is that no matter where the file is coming from the integrity of the file is guaranteed because any alterations to the file will alter the hash (alterations to the file’s name will not change the hash, only alterations to the file’s content). Next, as mentioned in the previous section of this post, any device that has uploaded or requested the a file will store and redistribute the file. Each device will continue to host the file as long as the IPFS service is running until either the operator chooses to stop hosting the file or the file stops being requested for some time and is automatically removed from that specific device to optimize storage. To avoid a file from being automatically removed from your device you can ‘pin’ the file so that it always remains on it.
It is important that at least a device or two has any particular file pinned so that it is always available when requested. If you are attempting to host any of your own content I will delve deeper into that in the second part of this series.
Access & Clients
Now, with the basics out of the way we can finally start utilizing the network. Starting out here, you have three options: the IPFS client, web browsers, and gateways. Beginning with gateways, a gateway is a website which is attached to an IPFS node (for future reference, a node is simply any device running IPFS) which pulls the file content off of the IPFS network and provides it to your browser for viewing and/or downloading. IPFS.io and CloudFlare both have services like this and allow you to enter in the address followed by the hash to view the file in the web browser (example: https://ipfs.io/ipfs/-hash-). I am bringing up gateways first because they are the simplest and easiest way for you to get access to data on the IPFS network, and great to check out if this is the first time you are accessing IPFS. Additionally, they also make it super easy for access on mobile and for sharing access to others who do not have IPFS installed on their devices.
As the middle ground between a full fledged client and a gateway, you can also run IPFS in some browsers. Browsers like Opera and Brave come preinstalled with IPFS support, and plugins for Firefox, Chrome, and Chromium allow you to manually add IPFS support to those browsers as well. Using IPFS within your browser allows you to utilize the network structure benefits of IPFS, but does limit your capabilities in regards to uploading and/or ‘pinning’ content.
Finally, to fully leverage the capabilities of IPFS you can install the IPFS client. You can acquire the client at https://ipfs.io and install it on Windows, Mac, or some Linux distros. With this you will be able to acquire files based on their hash, host files in a temporary or permanent manner, and help build up the IPFS network. Furthermore, as somebody who has spend nearly the last decade experimenting with new and experimental software, I am very glad to tell you that the graphical client is super easy to understand – even for my dumb brain. Since you already know what pinning is, what hashes are, and the basics of the way the client will behave (such as clearing out files that are not pinned) you should be able to use IPFS now with very minimal experimentation.
Also, I should note that the IPFS client can be used to self host content such as videos and websites, which I will delve into in the next portion of this series.
With this coming to a closing I would like to talk about a few drawbacks of the IPFS system. One much more minor drawback is the fact that there are not a ton of people using IPFS. Now, of course with gateways and the like the average internet user could be accessing content on IPFS without even knowing what IPFS is, so that is a bit more of a minor issue.
The other, more notable of the two, is the issue of speed. As I said before, IPFS is experimental and is lacking full scale adoption. The biggest tangible effect of this is the speed at which files are located. Though regularly trafficked files on regularly trafficked nodes are extremely quick to be discovered, files towards the fringes of the network are a whole different story. If you place a file on your device and try to load it’s hash in a gateway it could take upwards of several minutes for the gateway to actually find the file (I’ve noticed that CloudFlare can often time out a few times, where the IPFS.io gateway does not time out but still takes a fair amount of time). Now, once it has been loaded by the gateway it will be cached for a fair amount of time so the next time you go to load it (unless it has been too long) it will be quick, and I will touch on this more in the next part of this series, but this is worth considering.
Closing (pt 1)
Well, I guess I hope that summarizes everything. I hope you consider checking out IPFS and playing around with it. It’s certainly a very promising new system that I expect has a good future ahead of it, and regardless I find playing around with this type of system to be fun.
Pt 2: Hosting
In the last portion of this series I spoke about the basics of IPFS, however, I also spoke about the next portion of the series which would cover hosting content within IPFS. Simply put, the internet is made up of files that are retrieved from a server and displayed in a web browser, so it shouldn’t be surprising to hear that a decentralized network meant to deliver files can be used to host content.
One of the biggest drawbacks, however, is that with IPFS there can be no interaction with a server like on the normal web, preventing services like contact forms, discussion boards, comments, and even easy edits to the web page. Now, there are some minor benefits to this (no web analytics, pop ups, or dynamic advertisements without being creative), but ultimately hosting content on IPFS will limit you to only a static web page. Though people can be pretty creative within these limitations, this tutorial will cover the bare bones of such a setup. From here I can break up this tutorial into three parts: creating content, hosting content, and assuring access to said content.
Creating Your Site/Content
Since you are likely already aware of how to upload or download content on the IPFS network, if you are going to use IPFS as a file sharing service you can simply skip this step as there will need to be little effort into getting your content ready once you have the file. Even videos can be uploaded strait onto the network and then downloaded or streamed by other users. That said, if you intend build a web page hosted on IPFS then continue on.
As I said before, you are going to be limited in what can actually implement in a web page on IPFS. Therefore, your web page creating project will look a lot like it would in the early 2000’s. Because Nvu was discontinued, I would recommend installing the HTML editor KompoZer to create a word document like site which contains everything within a single page (I told you this was going to be bare bones). Proofread it well, because it will be a pain to make changes, and get your HTML file ready. Alternatively, you could export a word document as HTML using MS Word or LibreOffice, though software dedicated to HTML (either KompoZer or your editor of choice) will probably be better. Finally, it would also be possible to just use a text document instead of an HTML document, but that would be really limiting your creative expression options, so again, I would just go with HTML.
If you want to have a website available, then you will need to know where it will reside (as it would be pointless to make a site that ends up becoming inaccessible). Your first option would be to try to host it yourself. This of course would require a device that is always online, so it’s certainly not for everyone, but it is worth considering.
Secondly, it may be possible to rely on the network itself to host your content. If it is accessed enough then it would be hosted by all the individual nodes accessing the content – especially if it is content that you expect others to pin specifically to assure that the content remains hosted. That said, for a lot of us we can’t assume that others will host it.
Third, you can always play a game of whack-a-mole with public gateways and the like. A week or so ago I used the free version of global upload (which will temporarily host a file but requires payment to actually pin it) to upload a text file that I then loaded in several gateways. I let it sit for a week or so and then checked it and it was still there, so if a file is small enough and you go out of your way to request it on a bunch of different gateways regularly it could probably just float around in the network. Furthermore, if it does get discarded on all the nodes at some point you can re-upload the file and request it to get things going again. You also may want to couple this along side self hosting if you were already planning on self hosting that way you can be sure that your files are quick to access.
Finally, there are also paid services that will host your content for you. While this may seem like we just went full circle back to regular paid internet hosts, for most people who would want to host content on IPFS this is probably the best. As I mentioned global upload in the previous option, they currently have paid options that would allow you to permanently pin content. Additionally, services like Pinata have both free and paid pinning services. Pinata’s free service allows 1GB of bandwidth, and their next plan up starts to charge you $0.15 per GB of bandwidth that you use. Not only are these very fair prices, but if you are hosting a simple web page than these are amazing! If a simple HTML website was 10 KB that would allow it to be accessed 10,000 times a month on Pinata, and furthermore since if it was accessed 10,000 times a month it would likely be distributed among the entire network instead of solely being on Pinata meaning you would get a whole lot of addition access as well.
For this experiment I am going to go with Pinata to host my IPFS website, using the free version as I do not need too much bandwidth. Hash: QmacQwiu9xsaBXoRpzA6Qjz48a52mF9be1EZ2Y7iUDeXDJ
With content created and uploaded, the final thing is to make sure your content is easily accessible. You should have your hash that can direct users to the file, however, ideally you want to have a clickable link that will work for users who may or may not know how to use IPFS. Also, some of these options may allow you to change the hash it directs to and thus modify content on your website. It is worth noting that IPFS does has a service called IPNS, which is essentially a hash that you can use to point to any hash of your choosing. This would be perfect for easily updating web content, however, because it is a step or two outside of the scope of this basic tutorial – and because IPFS’s speed may make IPNS practically unusable – I will be leaving it out of this guide. Anyway, I’ll cover three different methods of doing this below:
First, aside from sharing your hash on it’s own, the easiest way to have a clickable link that requires no extra understanding of IPFS is to simply share the link to a gateway that is open to your file. The only drawbacks are that the link is a bit long and you can’t change the hash without changing the link.
Secondly, if you want to share your content in standard web form you can use shortened URLs or redirect pages to get the user from a link to a gateway with your file’s hash on it. Services like Bit.ly allow you to shorten the link to a gateway and share the link with others. Similarly, if you already have a normal website you can create a redirect page (say example.com/ipfs.html), which when loaded will immediately redirect the user to the gateway page that is loading your file on IPFS. In either of these cases you can also generally change the link to direct people to a new IPFS hash if you need to change your page or content.
Finally, you can always use a domain, a subdomain service, or a subdomain of an existing domain that you own to direct a user to your IPFS website or content. In any of those cases you can set the domain to either redirect the user to an IPFS gateway service or use something like Cloud Flare's IPFS services to configure the domain to connect to IPFS content like it would on the normal web.
Closing Pt. 2
Well, I hope you found this interesting, useful, or both. Those links I provided as examples are real links that link to my new experimental IPFS website. Thanks for reading!