ESD Funding Request: On-chain Data Audit & North Star Metric Investigation - flour.works

Summary

We (flour.works) are a team of experienced web3 analysts.

We propose funding of 6000 USDC at delivery of

  1. on-chain data audit and report
  2. investigation and initial recommendations regarding Aragon’s North Star metric (Active Aragon DAOs)

Description

Proposal Rationale & Strategic Alignment

Per the Aragon Network DAO Charter, central to the operations of the DAO is the metric Active Aragon DAOs. This metric is used to judge the efficacy of Aragon Governance Proposals (i.e. will this proposal increase the number of Active Aragon DAOs). It is also used as a North Star for internal DAO teams, such as the Executive Sub-DAO (ESD), to guide funding and effort.

As the DAO continues to grow the Aragon network, it is imperative that everyone is aligned on the proper growth metric(s). This concept is not singular to Web3. Aligning on the most important moments and milestones for customers to experience (assessed via metrics) steers the whole organization towards building the right things at the right time.

We (flour.works) propose funding to perform an on-chain data audit. This “audit” will consist of a full review and decoding of all Aragon smart contracts to determine and document all available on-chain data. There purposes for this audit as this proposal is concerned are two-fold:

  1. evaluate the current Active Aragon DAOs metric and make recommendations
    • there are currently many different definitions of “active” floating, hence the need for this investigation/proposal
    • with a full understanding of available on-chain data, we can evaluate potential definitions for Aragon’s North Star metric
  2. provide a foundation for the Aragon Network DAO (AN DAO) community and flour.works to build better analytics for the benefit of Aragon Network
    • this proposal is intended to be just the first phase of building analytics that enable data-driven decisions within the Aragon Network - we (flour.works) intend to use the audit to identify analytics opportunities and submit a subsequent proposal to build
    • the audit will remain as an artifact for internal/community analytics members to use to build on behalf of Aragon

Limitations and risks

We are mitigating the downside risk of a waste of funds by focusing initially on an on-chain audit and investigation of data. Analytics projects can suffer from unclear outcomes or improperly scoped projects. Structuring it in this way provides flexibility for both AN DAO and flour.works to assess the efficacy of future analytics projects after an intense scoping phase, and before more funds are committed to more development.

Deliverables

  1. data audit
  • the audit will be delivered in the form of one or more documents/spreadsheets that maps all event data for each Aragon smart contract.
  • a data quality dashboard will be created ([example](https://dune.xyz/setlabs/Set-Protocol-V2-Dune-Data-Quality))
  • smart contract decoding
    • we will ensure that all Aragon smart contracts are properly decoded on Dune
  • north star metric investigation
    • a report highlighting potential data that could be used in defining Active Aragon DAOs
    • proof-of-concept metric queries (Dune)
    • an initial recommendation on Active Aragon DAOs metric definition

    Accomplishing these deliverables will require collaboration with the current Aragon team - specifically devs and analytics familiar with the Aragon smart contract system. We are assuming we will be able to access these teams for informational discussions.

    We estimate 1 month from project start to delivery.

    Team Information

    flour.works team

    Sid sidhemraj#8522 @sidhemraj
    Don Don ETH to 10k#3452 @dbustos20
    Anthony Anthony Bowman Index Coop#1559 @anthonybowman
    JD jdcook#7270 @jdcook
    Jack jackiepoo flour.works#0001 @emesrever

    We have a combined 1,393 public queries built and 589 stars given on Dune. This doesn’t include the many queries and dashboards built privately for teams across the web3 space. Previous projects include Index Coop, Set Protocol, Gamma, Gelato, Alchemist, and Galleon.

    Funding Information

    6000 USDC at delivery


    cc @daniel-ospina @fartunov @lee0007

    3 Likes

    Operating under the mandate of the Aragon Network DAO Charter we [ESD] are required to make funding decisions that help us increase the active number of Aragon DAO on the network, it is my understanding that all financial proposals are subject to this requirement as per the

    Aragon Governance Proposal Process S 5 (f) i 3 Description of why the author believes it will help to increase the number of Active Aragon DAOs

    For full transparency I actively sort out the help of this team because I currently do not have access to the data needed to assess whether ESD funding decisions are driving growth of our North Star Metric.

    As specialists in this area your expertise is evidenced imo by identifying the need to first “…determine and document all available on-chain data…” before more funds are committed.

    This approach will enable us to then identify and set objectives and key performance indicators that we can effectively measure. I also hope this process will help us to surface insights to inform data-driven decisions within the DAO and clarify our understanding of the North Star metric(s)

    Thank you for preparing this information for us. As per the S1 ESD funding allocation process your proposal will be considered at the next ESD general meeting scheduled in seven days.

    1 Like

    As part of the core product team, I am always eager in having more data points (qualitative and quantitative) to make sure our decisions are as based on evidence as possible.

    For this reason, we have dedicated resources to work on product analytics, including the generation of data dashboards (including Dune ones), as well as exploring new data sources (for example using thegraph.com to fetch information).

    Also, all of our contracts are verified on block explorers (meaning, they are decoded, but maybe not on Dune - which from my knowledge means just doing a request here: Dune Analytics).

    With this in mind, a few questions / remarks for the proposing team:

    • If the proposal focuses on Dune analytics information, and it supports Ethereum, Polygon, xDAI, BSC, and Optimism, how would you handle the fact that we have DAOs in Metis and Harmony (and in the future will have on Arbitrum) as well?
    • The core team is currently working on a new product that should our flagship solution in the mid-term. This creates a “timing” issue for this work, as most of our efforts should go towards this new product.
    • I totally agree that the “active DAO” concept is liquid nowadays - But this definition goes beyond mapping all events that contracts emmit - It is a strategic one that can (and should) impact the product development. What does the team think about that?
    • Our infrastructure is evolving beyond “regular on-chain” interactions - For example, proposals and votes happening on Vochain through Aragon Voice. How would you handle this?

    To be clear, having more people working on our data efforts is super welcome, but I want to make sure the efforts are put in the right place.

    On a side note, your site (flour.works) doesn’t seem to have proper redirection from HTTP to HTTPS, so the first time I accessed it appeared as an “unsafe” destination. This might be seen as you don’t have enough credibility.

    1 Like

    Tagging @mroldann here who runs Aragon’s analytics and is working with various other members from AN DAO on data initiatives.

    1 Like

    Hi!
    It’s great to see such a talented team wanting to help Aragon! I’m open to help/questions at patoR#7041

    I like the idea of having documented all our events as we don’t have anything like that yet. It will reduce the overhead whenever trying to get new data from our contracts.

    In order to better understand the scope of your proposed work (on top of what @ramon already asked):

    • Will you focus in Aragon Client or also include Aragon Govern?
    • Do you have and example of a data audit?
    • How familiar are you with Aragon codebase? How much help do you think is needed?

    Also, I am making public our internal dashboards in a separate post in order to share what we already have in place and what we are still lacking.

    @lee0007 - It’s my fault that this vital metric is not available to you so far. I’m still learning how to engage with the DAO and community while delivering operative demand

    2 Likes

    We are all learning here, that’s why its fun and why data is so important.

    1 Like

    As promised, sharing NSM dashboard here.
    Happy to talk about how to improve it!

    1 Like

    Hey everyone - thanks for the replies and questions. I will try to answer them here.

    Our intention was to focus on all smart contracts.

    The best public example is here, however this dash is a bit specific to Set’s needs, and so would Aragon’s. And this would not be the full deliverable - as written, we intend to provide written mapping of all of the event data for Aragon smart contracts.

    Not familiar currently, other than reading through the Aragon docs. I think a call or two with a dev and/or product manager that has full understanding of the smart contract system would go a long way.

    I don’t think this proposal is intended to re-vamp / solve all current analytics challenges. It is narrow in purpose for a reason. So yes, we would only focus on chains that are indexed on Dune to start - but as part of the data audit, we can investigate options we may have to target data outside of those chains (Metis & Harmony).

    Totally understand. I am aware this work is going on, but obviously am not in a place to judge the extent to which working with the current smart contract system is going to extend to your new product. My hope would be that it would be valuable and provide a foundation for analytics on the new product.

    Exactly. We intend to “evaluate potential definitions” for the NSM. Essentially, map out what data (on-chain data specifically) is available to use in this definition, and craft a set of potential definitions. We can also provide a recommendation, but ultimately the final decision regarding the NSM would fall on collaboration with those within AN DAO.

    This proposal focuses specifically on on-chain data. However, we have full flexibility to work with off-chain data that Aragon is generating. We just have no insight into what that data may be or look like at this point. But we can amend the proposal to also consider off-chain data that your products are generating if that is helpful.


    To level set, we intend to be flexible and transparent. We put this proposal forward with the understanding we had at the time regarding the needs AN DAO might have - but we are really just here to offer our analytics skillsets and expertise to help AN DAO wherever it is necessary. So if that means amending this proposal, we are happy to work with y’all on that.

    Thanks!

    2 Likes

    Thanks for the answers @jdcook

    What is still unclear to me is what will be that aggregated value upon delivery, on top of what we already have today (and maybe the actual problem is that is not super clear what we already have). To hopefully make this a little bit better, here are some resources that already exists:

    I want to reinforce that I am totally in favor of having more people looking into improving how we handle data and making sure more people have access to important KPIs - But would be great to have more clarity on how the deliverables of this proposal will move Aragon towards this.

    Maybe an option can be to split the proposal into a first “small” deliverable that brings this clarity, and then if the deliverable makes sense for the network as a whole, work on the larger milestone.

    Regards

    ESD General Meeting Monday 18 April UTC 1900: Funding approved for 6000 USDC at delivery of

    1. On-chain data audit and report
    2. Smart contract decoding
    3. North Star metric investigation

    Due to @daniel-ospina absence at today’s ESD General meeting, this proposal has majority support from ESD. Funding is approved on the basis that further questions and clarification may still be sought by ESD members and the community as we look to understand your work. Congrats @jdcook and Flour.DAO team we look forward to learning from this project

    cc @fartunov

    @lee0007 @fartunov
    Can you help me understand:

    • How this work was approved in lieu of Ramon’s unanswered questions, especially when it provides relevant NSM data from the core team?
    • What is the answer to Ramon’s first question of what is going to be provided above and beyond what we already have?

    In reviewing the current thread I am utterly confused what the basis for approving this work was.

    I’d also ask that when approving proposals that require the time of core team members that this is explicitly expressed in the proposal. It is completely reasonable in this scenario to expect core team time towards this task but I/they don’t know what that is and the core team suffers the same capacity constraints as members of the ESD

    1 Like

    All good @b3n you’ve raised several points

    I 100% agree that proposals requiring the time of core team members should be “explicitly expressed” as per this proposal

    Which returned the following positive and I understood supportive responses from product and analytics leads

    Here’s a summary of the proposed deliverables detailed in the proposal, can you indicate which of these we “already have”

    1. On-chain data audit and report
    2. Smart contract decoding [Dune]
    3. North Star metric investigation
    4. Subsequently an artifact for internal/community analytics members to use to build on behalf of Aragon…"

    Next, let’s note that the forum will never be the sole basis upon which decisions are made especially when in proactive communication with teams during the development of proposals and where proposal review is requested prior to posting to the forum. I am always happy to review and provide feedback on proposals prior to posting in forum. People only need to ask, this has always been part of my ESD role to date.

    We also have the ESD General Meetings and our professional experience as a basis for making decisions. I will share here my conversation with @ramon earlier today in response to his most recent question above and following the ESD majority (2/3 due to absence) funding decision.

    This IS the first small deliverable…It is aligning with AA that always proves the challenge hence this being 6k proposal from a world-class team that earns collectively more than that in a day. @mroldann shared the analytics dashboard…And while data studio looks great it is a retrospective metric, nothing there helps to inform future-facing [ESD] funding decisions.

    I am a data-driven and business performance professional. Data is the basis upon which I have and continue to manage - millions of dollars in performance marketing budgets over the past decade - and right now ESD and AN DAO simply do not have the data required to make data-driven decisions about the future of [funding] the network.

    We [ESD] have different questions to answer than the Product team and we are trying to align around the NorthStar and also operate with some autonomy, sans reliance on what I understand are already very stretched resources for analytics within AA.This is an effort to support analytics and be proactive, to seek answers ourselves, which could provide us all insight.

    Given Defi DAO are the largest reference audience for @Harry proposed GtM - [imo it would] be good to align with ecosystem talent beyond AA. Can you please help to empower the DAO in our need for data. Your support is mission-critical, your forum post will help to inform [proposed] work [&] I very specifically am not looking to rehash exisiting data points but to seek other [SMART] relevant metrics [as they relate to the the NSM]

    I’ll try to frame my need for data another way

    Given “active” number of DAO as currently measured using

    If these were the ESD success metrics then I would simply fund everything to generate proposals and increase “active” metrics for the network. But in my professional opinion that would amount to negligence, a breach of the duty of care entrusted to me as a member of the ESD for the strategic funding of network growth.

    And so I have sought the help of some of the most talented analytics minds I know to help surface, investigate and model data related to the NSM in search of a more refined measurement of funding success. The current process is simply too subjective. I want and we [the community] need data to help inform funding decisions

    1 Like

    Thanks, it’s clear now there is more context (and a direct but unseen response to Ramon’s questions) which is why i asked.

    Regarding this part: "

    Explicitly expressed to me would be X hours/days

    (A bit late to the conversation but) very happy to see this proposal pass!

    This is much needed, and the ‘active DAOs’ definition is pretty tricky. Even if it’s something we’ve been thinking about and wrestling with, a) having another (expert) perspective on this will no doubt improve the thinking, b) having a deliverable for it, within a short time frame, is way better than not having a metric!

    Kudos to @lee0007 and the Flour.works team for proposing a pragmatic and bitesize first step, I hope it can mark the beginning of a longer collaboration.

    (P.s. I’d love to join the kick-off of this initiative if possible - I am working on product marketing at Aragon and NSM helps defining which part of the market to pursue).

    3 Likes

    Glad we have the proposal through!

    Stoked to have some web3 native insights (Dune dashboard) where everyone can validate the network’s growth and health!

    A few notes:

    • Based on the original proposal, there is little to no work required from the AA/AL product teams to facilitate the delivery of the proposal. I have expressed my support based on that assumption?
    • As per the comments shared, ideally, we would want to see data across all Dune-supported EVM chains where Aragon is live. Is that the case? [clarifying question, won’t change me supporting the proposal]
    • With respect to “active DAOs,” - in addition to the ambiguity of what “active” means, overall the metric has many disadvantages. Ideally, we would want to see some exploration on whether a better metric can be derived based on available data and your experience and expertise with other projects [i.e. if everyone is measuring A and we chose B, we should have a great reason for doing so, or we should revert to also using A]

    @jdcook, does the above make sense?

    2 Likes

    @jdcook can you please respond. I understand you had a call with @ramon to redefine the scope. Could you please share some transparency

    After chatting with the Flour.works team, here is a list of points that would be great to have their help on top of what @mroldann already created.

    • Get all app (addresses and names) used by Aragon Client DAOs, and the most used methods/functions for each of them
    • Get all DAO names from Aragon Client
    • Some finance transactions are not triggering events and need to find out why. @mroldann will follow up on this below
    • Documentation for all contracts
    • Dune decoding

    Resources to help with all the work

    1 Like

    Hi there, completing what @ramon said.

    Currently, our Vault subgraph does not show complete information for some DAOs/tokens for both Balances and transactions .

    Example from Decentraland DAO

    • Left: balance according to subgraph
    • Right: real balance from Etherscan which coindices with Dune

    We would like help to find a way in Dune to fetch all DAO’s USD AUM evolution and transactions with queries that don’t timeout.

    1 Like

    Hey @fartunov - sorry for the delay. We had a good chat with Ramon and Martin - thanks @ramon for providing links to all the resources.

    We have been investigating the smart contract system and are working on documenting and mapping out events as per our first deliverable (referencing deliverables below).

    @fartunov to answer your questions here:

    It is our intent to focus on mainnet for now - the reason being is there is no new discovery we can make by extending to other chains. We would be happy to roll a future proposal to help with decoding and actual metric building on other chains once we have this foundational audit and investigation completed.

    We can definitely include that exploration in our North Star Metric investigation. Happy to ideate on some other potential NSM’s to consider other than “active DAOs” based on the available data and other metrics the industry is favoring.

    As per the list of things that @ramon and @mroldann have suggested (below) it would be great to have our help with - glad to see this!

    As I mentioned in our call with them, our goal is to be a resource - an extension of the analytics and product efforts already happening. So it is great to see the uncovering of some analytics needs right out of the gate. However, I want to be clear we will only be focusing on the 3 deliverables as a part of this proposal. We are hoping that these deliverables uncover other projects / tasks that we can help @ramon and @mroldann (or other teams at Aragon) tackle. We would plan on submitting another proposal for funding for these items outside of the scope of the current deliverables.

    3 Likes

    Thank you! The clarification is much appreciated!