Inteligencia Artificial |HO| Google: "No tenemos foso (económico), ni tampoco lo tiene OpenAI"

Leak de un documento interno de un investigador de Google en el que dice (en relación a LLMs o Large Language Models): “No tenemos foso (económico), ni tampoco lo tiene OpenAI”. Básicamente el documento parece llegar a la conclusión de que el mejor camino para Google es abrazar el código abierto.

Las dos partes que han llamado más mi atención:

While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. This has profound implications for us:

We have no secret sauce. Our best hope is to learn from and collaborate with what others are doing outside Google. We should prioritize enabling 3P integrations.

People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should consider where our value add really is.

Giant models are slowing us down. In the long run, the best models are the ones

which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the <20B parameter regime.

Y la segunda, que insinúa que con LLMs va a ocurrir lo mismo que con text-to-image:

In many ways, this shouldn’t be a surprise to anyone. The current renaissance in open source LLMs comes hot on the heels of a renaissance in image generation. The similarities are not lost on the community, with many calling this the “Stable Diffusion moment” for LLMs.

In both cases, low-cost public involvement was enabled by a vastly cheaper mechanism for fine tuning called low rank adaptation, or LoRA, combined with a significant breakthrough in scale (latent diffusion for image synthesis, Chinchilla for LLMs). In both cases, access to a sufficiently high-quality model kicked off a flurry of ideas and iteration from individuals and institutions around the world. In both cases, this quickly outpaced the large players.

These contributions were pivotal in the image generation space, setting Stable Diffusion on a different path from Dall-E. Having an open model led to product integrations, marketplaces, user interfaces, and innovations that didn’t happen for Dall-E.

The effect was palpable: rapid domination in terms of cultural impact vs the OpenAI solution, which became increasingly irrelevant. Whether the same thing will happen for LLMs remains to be seen, but the broad structural elements are the same.

Que es lo que llevo pensando desde hace meses: “¿Qué ha pasado con DALL-E e Imagen de los que no leo nada desde el año pasado?”

¿Alternativa open source de ChatGPT-4 en unos meses? Ójala.