A storage protocol for the "seed-hunting"
BitTorrent might sounds new to many people. But when it comes to downloading "seeds", I'm sure many of you will recall the days when you used to "request seeds" from all over the internet to play games or watch movies. A seed file is an index file that contains metadata such as the location and size of the downloaded file, the address of the download server, and the address of the publisher. In simple terms, BitTorrent is a P2P downloading protocol that is much more efficient than the traditional method of downloading from a web server. To give a straightforward example, back in the days when I was still a student, it was common that we gathered around to copy the homework which was completed by one single classmate. In this way, each student has to do the copy work by themselves with only one hard copy available. The process cannot scale.
The common practice is to delegate students to different copy works, for instance some taking care of the multiple choice questions and the others responsible for filling in other questions. Such methodology is way more scalable and efficient.
The intuition adopted by BitTorrent is somewhat similar to the above practice. While downloading, the computer acts as a server and then transfers the proportion of file which has already been downloaded to other users. With that being said, we are downloading and uploading at the same time. And thus, where there are more users downloading this file, the more seeds will be available out there and consequently faster the synchronization will take place, as well as for the speed of the download.
The pioneer of decentralized storage: IPFS
Despite the significant increase in downloading efficiency, BitTorrent still has some problems. In order to optimize these problems, we can’t avoid mentioning IPFS, which many readers might be more familiar with. It is where those famous decentralized storage projects such as Arweave and Filecoin are built upon. IPFS, which translates as the InterPlanetary File System, is an Internet communication protocol that allows distributed storage and file sharing. To download using BitTorrent, you must use a seed file, which contains the addresses of all the content you download.
One of the significant advantages of IPFS is that it leverages a DAG data structure to store data, which means that IPFS stores and searches files using content-based rather than address-based mechanisms. This suggests that when we are searching for a file, we are only required to keep in mind of what it contains instead of where it is exactly located at. IPFS generates a unique hash for each file (e.g. QmSNssW5a9S3KVRCYMemjsTByrNNrtXFnxNYLfmDr9Vaan) and when the user needs to retrieve this file, they simply ask IPFS who has this hash (QmS...Vaan) and then can retrieve the file as they wished. Because hashes prevent repetitive storage, files with the same content will not be processed by IPFS, which optimizes storage and improves network performance as well.
The solution for dynamic storage: Ceramic
If you pay close attention so far, you will probably notice that IPFS has a huge flaw. In IPFS, once a file is stored, it cannot be modified on the system because calibration changes the hash and then the user cannot retrieve the file by using the original hash value. This is the pain point that IPFS has been criticized for: not an optimal solution for storing files that need to be updated or upgraded from time to time. Therefore, there is an urgent need for an efficient and decentralized solution for storing dynamic data.
The Solution of Dynamic Storage:
According to the description of Ceramic above, attentive readers may reveal a huge flaw in IPFS. Once stored, documents cannot be amended in IPFS considering that modifying the content of the file changes the hash value of the file and users cannot find the amended documents through the original hash value. This is a sore point that IPFS is criticized by everyone: Not good at storing files that need to be updated and upgraded at any time.
Hence, there is an urgent need for an efficient and decentralized solution for storing dynamic data. Fortunately, the exploration of this track has been set out. Buddies interested in Web 3.0, Social Fi, or DID must have heard the name of the project——Ceramic. Ceramic is a decentralized open source platform established for creating, hosting, and sharing data. Numerous DID and Social Graph projects are built based on it. As mentioned earlier, IPFS is doing well in storing static files, but it lacks the computing and state management capabilities and thus cannot realize more advanced database-like capabilities, such as variability, versioning, access control, and programmable logic. The occurrence of Ceramic provides a certain solution towards the problems above. ** **
Efficient Versioning
In Ceramic, each storage message can be represented as a superposale log (a log file on a computer recording the operation of a program), referred to as a Stream。Stream is similar to Git in concept. Git is an open source distributed versioning system that efficiently and quickly processes large and small project version management, which is the most popular version control software available today to store code, track revision history, merge code changes, revert to earlier versions of code, and so on. Git handles data like a "snapshot", similar to sharing documents and viewing historical versions with Google docs. Whenever you submit an update or save data status, it creates a snapshot of all the files at the time and saves the index of the snapshot. If documents are not amended,Git then will only save one link to previously stored files, rather than restore the document, which improves efficiency to a large extent.
In fact, IPFS can also adopt Git to store dynamic data. However, developers are required to create a new hash-log file to record the mapping table for each Git log and IPFS hash update and are required to keep synchronization dynamic manually, or use the IPNS naming system to keep dynamic updates. Such operations are time-consuming and inefficient. Ceramic uses a "superimposed log" approach, and StreamID does not change consistently with the content change. As a result, it can easily store modified or backtracked versions without changing hash values frequently. Besides, Ceramic builds a new layer on top of other storage protocols, indicating it is highly combinable. Users can choose where to store their data, including decentralized Arweave, Filecoin, centralized AWS, etc. by themselves and realize automated versioning via Ceramic. Also, given each Stream only stores log instead of data, a global ledger to globally synchronize data states is not necessary, in which case Ceramic's lateral scalability is rather high.
Convenient Identity Verification and Access Control
Apart from the advantages of versioning, Ceramic also provides very convenient identity verification and access control. When new data is added in the Stream, modifiers need to verify their identities or the data can not be modified. Different Streams can require different identity verification mechanisms, and Ceramic provides very powerful embedded identity verification mechanisms——DID. Specifically, it provides 3ID DID for end users, key DID for developers, NFT DID that supports verification with NFT, Safe DID that requires multi-identity verification for DAO and many others to ensure data security. Meanwhile, Ceramic empowers the Stream programmable logic. For instance, if Stream A changes status, then Stream B can be accessed and upgraded, etc. The emergence of Ceramic empowered the construction of Web 3.0 enormously. Nowadays, there are various DID as well as Web3.0 social platforms under development in Ceramic. Project like social graph middleware platform - CyberConnect, Web3.0 Twitter - Orbis, instant messaging platform - The Convo Space, etc. are worthing paying attention to. We are looking forward to the new possibilities that Ceramic's infrastructure could bring to the application layer.
END