To demo the possibilities of this file manager, I uploaded the frontend to the canister, so you can moderate the data on the canister from the files served by the canister. This is a bit meta, but it shows how easy it is to upload files to the canister and serve them.
Ofcourse this isnt the only way of using this file manager, you can also hook it up to a traditionally hosted frontend (asset canister) or a native app.
This file manager is hosted on the internet computer, a decentralized network that allows for secure and efficient hosting of content. The file manager is available to everyone through a canister that also hosts other content to interact with.
One of the key features of this file manager is the ability for users to easily upload files and directories and preview / download them. Once uploaded, the canister will serve the files and directories as static content. This means that users can share their files with others without having to worry about them being altered or modified by others.
To make the file management process even easier, a user-friendly interface is available on the canister for users to upload, delete, and download files and directories. This interface is also uploaded to the canister in a similar manner to the files and directories themselves.
It's important to note that users are in control of their own content when using this file manager. When a user is logged in, they can only moderate their own content. This means that users can only add files and directories to directories they are the owner of (excluding the root directory). Additionally, logged in users have a 10mb limit per upload attempt, which helps to prevent excessive usage and ensure efficient operation.
However, anonymous principals are also allowed to upload files and directories. They are limited to a 1mb upload limit, which helps to prevent excessive usage and ensure that resources are allocated fairly.
In summary, this file manager hosted on the internet computer provides a convenient and secure way for users to manage and share their files online. With its easy-to-use interface and strict usage limits, it ensures that everyone can benefit from this service without compromising its functionality.
One of the main goals of this project is to drive adoption for web 2.0 developers by giving them a way to easily migrate their frontend applications to the internet computer by providing a known FTP like interface. There is still a lot of work that needs to be done to improve this experience but the basis is set.
Next to that, this project also aims to provide a secure and easy way for users to manage and share their files for NFT projects and other usecases.
The project is still in its early stages, but I hope to continue working on it in the future, i have a lot of ideas and improvements that can be integrated. The plan is to opensource the code so developers can develop their own flavours of this file manager and model it any way they like.
With the knowledge that uploading 100kb of data takes about the same time as uploading 2mb, I devised a more efficient upload method that divides data into smaller chunks. To optimize performance, we established a limit of 1MB per chunk. If a file exceeds this limit, it is split into multiple chunks.
For instance, a file of 5mb will be split into 5 chunks of 1MB each. On the other hand, if you have 10 files of 0.1MB, each file will be divided into 1 chunk of 0.1MB, resulting in a total of 10 chunks.
By using this approach, we can upload up to 20 chunks of 0.1MB in a single 2MB call, which takes only 2-3 seconds to complete. This is much faster than uploading 20 chunks separately, which can take up to 40-60 seconds.
Our file upload process begins by processing all assets (files/directories) in the frontend and transforming them into a backend-understandable tree model. This model is then sent to our backend in a single call. In addition, we reserve the necessary chunks for each file and store the metadata for all assets during this initial step.
Once the backend processes this data, it returns a list of file data, each holding its own reserved chunk IDs. Then, the actual chunking of the files begins, with each chunk being tied to a reserved chunk ID that was fetched from the file. To group these chunks based on byte size, we store them as "chunks-with-ID" in memory, with a maximum of 2MB per group to avoid exceeding the ingress limit.
Next, we loop through all the chunks and assign them to a chunk group based on their byte size. Once this grouping process is complete, we can start uploading the chunks to the canister. We achieve this by sending one chunk group at a time to the canister and storing it in the correct location on the canister.
The initial reasoning behind creating a native application was to have direct access to the filesystem and queueing uploads, which would make it possible to sync folders and stop and start the upload/sync process. The advantage of a native app is that it can leverage the power of the underlying operating system, allowing for greater control over system resources and performance.
However, I later decided to switch to a frontend dapp, while still keeping the native app as a backup for reference purposes and to maybe pickup development again in the future.
While developing the web facing frontend, the canister backend code underwent some changes, which caused some functionality to break in the native app. Unfortunately, due to time constraints, I decided not to fix these issues at the moment. However, the native app should still be able to run and view the files.
In summary, I began this project by creating a native app using Tauri, React, TypeScript, and Rust. Later on, they decided to switch to a frontend dapp but kept the native app as a backup. Despite some functionality breaking in the native app due to changes in the canister backend code, the app is still able to run and view files. The use of Rust and Tauri also highlights the benefits of combining system programming languages with web technologies.
The backend canister for this file manager is written in Rust. This canister is responsible for managing various aspects of file storage, including file metadata, directories, chunks, and parent-child relationships.
When a user uploads a file or directory, the backend canister stores the metadata associated with that file or directory, such as its name, size, and creation date. It also manages the relationships between directories and their child files and subdirectories, ensuring that everything is organized and easy to navigate.
Because of the canister 2mb limitation for sending and receiving files, the frontend is responsible for chunking files into smaller, more manageable pieces. This process also helps to optimize file uploads and downloads / playback, as well as ensure that files can be easily stored and accessed on the canister.
Overall, the backend canister plays a critical role in ensuring the smooth functioning of the file manager. By handling the complex aspects of file storage and organization, it allows users to easily upload, manage, and access their files, making the file manager a useful and practical tool for anyone looking to manage their files and directories online.
The file manager's frontend is built using Create React App (React + Typescript), which makes setting it up fairly straightforward.
The top bar of the file manager contains a login button, which, when clicked, opens a new tab to the login page. Once you log in with your internet identity, you'll be redirected back to the file manager. If the file manager is currently processing files, it will display a loading indicator and indicate the progress of the upload process.
The file manager itself provides a number of useful features. If a file or directory is greyed-out, it means that you don't have moderation rights over it, as you're not the owner. However, you can still view the file and its details. If you do have moderation rights over a directory, you'll see a + button above the column where you can choose to upload multiple files, upload a directory, or create a new directory.
Clicking on a file or the ! icon of a directory opens a details modal with a file preview, where you'll also find options to moderate the content if available. When clicking on a directory, it opens and displays its contents in a new column on the right side.
Finally, at the bottom of the file manager, you'll see some metadata about the canister, such as the total bytes used by files, the number of cycles used on the canister, and the number of files and directories currently stored. These metrics provide useful insights into the overall health of the canister and its performance.
Next to the improvements that are mentioned in the repo's readme's there are some general improvements that can be made to the project.