Determining What Matters for NFT Valuation - Upshot

The third installment in our blog series exploring the use of machine learning to accurately price NFTs. Over the last few weeks, we’ve been diving deeper into how we use machine learning and crowd intelligence to generate automated and up-to-date NFT pricing, why machine learning models are important to achieve reliable valuations at scale, and much more.

You can read the previous entry on taking NFT pricing to the next level with machine learning here.

How Do We Determine Which Variables Matter for CryptoPunk Valuation?

At Upshot, we leverage machine learning models that ingest historical sales data, bids/asks, and NFT metadata to construct features based on this information to generate accurate, reliable pricing. Previously, we explained why machine learning models are necessary to robustly achieve reasonable valuations. However, ML models can be complex and difficult to understand.

So, how do we determine which variables are important in determining a price for an NFT? We use Shapley Values to uncover the importance of different variables used in our models. This helps us explain how a complex ML model arrived at its predicted NFT price, which is useful for developing the models further and understanding unexpected predictions.

A Shapley Value measures the extent to which a feature contributes to the performance of a model. We can also plot the direction and magnitude of the contribution for any feature. This is shown in the figure below for a sample feature in our CryptoPunks pricing model.

Looks Rare

We will focus on the feature “rarest_accessory,” which captures the commonality of the rarest accessory on the punk. This variable, among many others, is constructed through our iterative research and development process and contributes to determining the final valuation of a punk.

Blue corresponds to lower values of this variable, whereas red indicates higher values. Points towards the left denote that the impact of the variable on the NFT valuation is negative, whereas points towards the right denote that the impact is positive.

Overall, the effect is intuitive. If the rarest accessory of the punk is more common (red points), the predicted valuation will be lower (points towards the left). While the effect is intuitive we should not assign a causal interpretation to it. The influence of any particular variable will depend on how it interacts with other variables used in the prediction and how it may proxy for unobserved variables excluded from the analysis.

To see another example, the plot below shows the contribution of the “project.prevPrice” variable that captures the last sale price of any punk.

Last Sale

Again, the effect is intuitive. If the last sale price of a punk is high, then the predicted valuations for punks will be higher. However, this finding does not necessarily imply that high sale price events cause punk valuations to increase. It could also be that punk prices are simply trending upwards, implying that last sale prices will also be increasing and positively correlated with (but not causing) higher punk valuations. Nevertheless, the visualization helps us understand why our model may predict higher valuations for punks following a large sale event in the market.

Such analysis helps us intuitively understand potential drivers of NFT prices and identify the variables that really matter for each NFT project.

How does attribute rarity affect CryptoPunk prices?

One of the most important determinants of the price of a CryptoPunk is its “rarity”. We incorporate rarity into our machine learning models for pricing NFTs by constructing features based on the different attributes of a punk.

First, punks can be of different types, such as female, male, zombie, ape, and alien. The graph below shows the number of CryptoPunks (of the total 10,000) that are of a particular type.

5 Types of Punks

Male punks are the most common, whereas alien, ape, and zombie types are relatively rare. These types contribute to determining the value of a punk, with the rarer types typically selling for higher prices.

Second, we can go one level deeper to examine CryptoPunk traits, which correspond to the accessories worn by a punk. The graph below shows the distribution of the number of traits possessed by a punk.

1-7 Trait Punks

Most punks have 3 traits, or accessories. Very few have only 1 or more than 4. Diving even deeper, the graph below shows the distribution of each one of the accessories across all 10,000 punks.

So many earrings

Earrings and cigarettes are the most common, whereas the least common are beanies, chokers, and pilot helmets. We would expect that punks with relatively less common accessories would be valued higher and fetch higher sale prices. However, punks with a rare accessory may also possess several more common accessories. To account for this possibility, we construct rarity scores based on the combination of accessories worn by a punk.

The basic idea behind rarity scores involves calculating the probability of observing the set of accessories on a punk given how frequently these accessories occur across all punks. The higher this probability, the less rare the accessory set of the punk, yielding a lower rarity score. The lower this probability, the more rare the accessory set, yielding a higher rarity score.

We implement a fast approximation to these probabilities, as calculating them exactly given the large set of possible accessory combinations is computationally complex. In addition to calculating a first-order rarity score, we also calculate a second-order score which takes into account the probability of not just observing an accessory set but different pairwise combinations of accessories within this set.

How much do these constructed rarity variables contribute to our ML model’s pricing predictions? The figure below shows the Shapley Values (or the contribution to the model’s output) of the first-order “rarity_score” and the second-order “rarity_score_2nd_order” variables used by our model.

The graphs show that higher values of both variables (indicated by the red points) tend to lead to higher valuations (as they fall further to the right in the plots).

This approach to calculating rarities can be extended to arbitrary higher-orders of rarity and to other NFT projects beyond CryptoPunks where a fixed set of NFTs is released with a fixed distribution of rarities. Projects where new NFTs are released over time, possibly with new traits not seen before, require a more involved calculation of the rarity scores. Look forward to more on this in our future posts.

Stay tuned - next week we will wrap up the series by exploring NFT pricing in volatile and fast-moving markets and recap what we’ve learned about the use of machine learning to accurately price NFTs.

If you are interested in building new products enabled by near real-time NFT appraisals, please reach out on Discord or at [email protected].