Self-supervised learning: The dark matter of intelligence (2021)

164 points
1/20/1970
a year ago
by reqo

Comments


colordrops

We are giving more credit to humans than is due. We aren't capable of solving any arbitrary problem thrown our way. There are classes of problems we are good at solving due to us just being human and evolving to solve them, just as certain classes of problems are solveable by dogs. As referred to in this article, "common sense" is not some yet to be determined skill set, but rather the set of classes of problems humans are capable of solving that AI has not yet tackled.

Perhaps it's a matter of the AI connectome not being vast enough. Perhaps it's a deeper issue of AI architecture. Anyway it seems that that in this sense, "common sense" and "dark matter" are indeed analogous, in that it's just a place holder for yet to be understood (or at least emulated) phenomena. Perhaps that was intentional by the author, I can't tell.

a year ago

jewelry

Exactly. And this study did a good job exploring it philosophically. https://arxiv.org/abs/1911.01547

a year ago

dang

Discussed at the time:

Self-supervised learning: The dark matter of intelligence - https://news.ycombinator.com/item?id=26431507 - March 2021 (74 comments)

a year ago

3abiton

In light of LLMs it seems relevant.

a year ago

isaacfung

This article was written in 2021 when masked language models have been successfully applied in nlp(Bert, word2vec, glove, etc). However at the time, it was unclear how the same technique could be applied to vision tasks because unlike language which has a limited vocab, you can't explicitly assign a probability to every possible image. Since then researchers have already made significant progress with techniques like contrastive learning(simclr), self distillation (BYOL, DINO), masked image models, etc. A cookbook of self-supervised learning is a good source to learn more about this topic. https://arxiv.org/abs/2304.12210

a year ago

imjonse

SimCLR and others are older than 2021, BYOL is even mentioned in the blogpost. But your link indeed points to a more comprehensive overview.

a year ago

isaacfung

You are correct that SimCLR and BYOL were released one year earlier. Sorry I worded it poorly. By "at the time", I meant the period of time when masked language models just found success in NLP.

a year ago

thelastparadise

Especially in light of the releases of LLaMA and v2.

a year ago

p1esk

I have no idea what you mean. The article is mainly about dealing with uncertainty when trying to predict visual information. LLMs have no such problem.

a year ago

lukeinator42

LLMs are using self-supervised learning and the article talks about uncertainty when trying to predict missing information in the NLP domain and how it compares to the visual domain (and why it works so well for the NLP domain).

a year ago

[deleted]
a year ago

exogenous

[dead]

a year ago

jablongo

I took LeCun’s deep learning class at the Courant institute in 2016 and some of this stuff is directly lifted from those course materials, like the gif of contrastive learning in action.

a year ago

lukeplato

I recommend watching the Hinton talk on dark knowledge on YouTube

a year ago

killjoywashere

tl;dw?

a year ago

woadwarrior01

Model distillation.

a year ago

psuedo_uuh

Dark matter of intelligence. Do we think it’s going to prove that we live in a simulation? Ive always felt like the tech industry things too highly of ourselves and the naming of things hits at that. Artificial intelligence is a good example. Would using the word statistics in the name be more accurate with what’s going on?

a year ago

phreeza

(2021)

a year ago

[deleted]
a year ago

[deleted]
a year ago

soycello

[flagged]

a year ago