Hi, Mathew here from Espresso.
Thanks @Schwartz10 for this fantastic analysis! I’m not fully aware of all the details and requirements Autark has expressed so please correct me if any of my assumptions are wrong.
IPFS pinning and data querying
I tend to agree with Daniel and @osarrouy that an IPFS pinning solution should be usable on its own, for 2 reasons:
Implementing a robust querying layer (regardless of it being fully decentralized or not) is no simple task. Handling concurrency, schema validation, data consistency while maintaining decent performance and high availability, all of this takes time. Yet the need for a simple IPFS pinning solution is increasing as more projects are being built on Aragon. Implementing a standalone IPFS service would quickly provide a solution to those needs.
Adding a querying API significantly increases the attack surface of the service. Considering that one of the requirements is to “Limit security vulnerabilities and expenses”, separating the query layer would probably be the best solution to meet this requirement.
Here above, I assumed that building the querying part as a 2nd layer would require no major extra work, but again correct me if I’m wrong.
I think we all agree that a completely decentralized solution would be ideal. However, I’m skeptical that it could be realistically achievable within the given timeframe.
Furthermore, I would argue that MongoDB is actually a perfectly fine solution for this in the near-term as it is not as centralized as it may sound. We could easily imagine for example a replica-set architecture where each of Aragon One, Aragon Black and Autark run its own MongoDB node. In combination with @Schwartz10 clever idea to store db snapshots on IPFS, this would be a fairly(albeit not completely) decentralized solution.
IPFS API and authentication
I feel like the Aragon IPFS service should be 100% compatible with the IPFS API
I completely agree with Olivier. It would be in the best interests of users as well as the devs who maintain the service to have a compatible solution. IPFS supports basic authentication but the client library also supports custom headers if you want to implement JWT authentication (or any complex authentication) without breaking the API.
Regarding JSON web tokens, I think they can be a great choice for APIs as they are lightweight, supported pretty much everywhere and can technically be stateless. However in practice they are quite often stateful, as we probably want a revoking mechanism of some kind. i.e. If the token gets lost/compromised or if you want to disable access to a user who previously had access.
Have you considered directly using a smart contract to validate authentication? What I have in mind would be something like this:
- Before sending the “pin request”, the library signs the CID with the user’s account.
- It then attaches the signature and user account to the request as an additional header, keeping 100% compatibility with IPFS.
- On the server side, a small service verifies that the signature is valid and that the smart contract contains the permission for this specific account. If yes, it forwards the request to IPFS. If not, it returns an error 403.
The smart contract could be a simple DAO where the IPFS access is managed by the ACL. This solution has its drawbacks however: the user would have to sign the CID every time he/she wants to pin data, and it would probably be harder to implement than JWT authentication.
Finally, regarding the Espresso Datastore, it solves slightly different problems than what this IPFS and data querying service is intending to solve so I’m not sure it would be an optimal solution for you. However there may be a few concepts and ideas that could be useful. I won’t be available tomorrow unfortunately but you can contact me on Telegram at mcormier and I will most certainly keep an eye on this thread