Google announced a new technology called LIMoE that it says represents a step toward reaching Google’s goal of an AI architecture called Pathways.

Pathways is an AI architecture that is a single model that can learn to do multiple tasks that are currently accomplished by employing multiple algorithms.

LIMoE is an acronym that stands for Learning Multiple Modalities with One Sparse Mixture-of-Experts Model. It’s a model that processes vision and text together.

While there are other architectures that to do similar things, the breakthrough is in the way the new model accomplishes these tasks, using a neural network technique called a Sparse Model.

The sparse model is described in a research paper from 2017 that introduced the Mixture-of-Experts layer (MoE) approach, in a research paper titled, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.

In 2021 Google announced a MoE model called GLaM: Efficient Scaling of Language Models with Mixture-of-Experts that was trained just on text.

The difference with LIMoE is that it works on text and images simultaneously.

The sparse model is different from the the “dense” models in that instead of devoting every part of the model to accomplishing a task, the sparse model assigns the task to various “experts” that specialize in a part of the task.

What this does is to lower the computational cost, making the model more efficient.

So, similar to how a brain sees a dog and know it’s a dog, that it’s a pug and that the pug displays a silver fawn color coat, this model can also view an image and accomplish the task in a similar way, by assigning computational tasks to different experts that specialize in the task of recognizing a dog, its breed, its color, etc.

The LIMoE model routes the problems to the “experts” specializing in a particular task, achieving similar or better results than current approaches to solving problems.

An interesting feature of the model is how some of the experts specialize mostly in processing images, others specialize mostly in processing text and some experts specialize in doing both.

Google’s description of how LIMoE works shows how there’s an expert on eyes, another for wheels, an expert for striped textures, solid textures, words, door handles, food & fruits, sea & sky, and an expert for plant images.

The announcement about the new algorithm describes these experts:

“There are also some clear qualitative patterns among the image experts — e.g., in most LIMoE models, there is an expert that processes all image patches that contain text. …one expert processes fauna and greenery, and another processes human hands.”

Experts that specialize in different parts of the problems provide the ability to scale and to accurately accomplish many different tasks but at a lower computational cost.

The research paper summarizes their findings:

• “We propose LIMoE, the first large-scale multimodal mixture of experts models.
• We demonstrate in detail how prior approaches to regularising mixture of experts models fall short for multimodal learning, and propose a new entropy-based regularisation scheme to stabilise training.
• We show that LIMoE generalises across architecture scales, with relative improvements in zero-shot ImageNet accuracy ranging from 7% to 13% over equivalent dense models.
• Scaled further, LIMoE-H/14 achieves 84.1% zeroshot ImageNet accuracy, comparable to SOTA contrastive models with per-modality backbones and pre-training.”

Matches State of the Art

There are many research papers published every month. But only a few are highlighted by Google.

Typically Google spotlights research because it accomplishes something new, in addition to attaining a state of the art.

LIMoE accomplishes this feat of attaining comparable results to today’s best algorithms but does it more efficiently.

The researchers highlight this advantage:

“On zero-shot image classification, LIMoE outperforms both comparable dense multimodal models and two-tower approaches.

The largest LIMoE achieves 84.1% zero-shot ImageNet accuracy, comparable to more expensive state-of-the-art models.

Sparsity enables LIMoE to scale up gracefully and learn to handle very different inputs, addressing the tension between being a jack-of-all-trades generalist and a master-of-one specialist.”

The successful outcomes of LIMoE led the researchers to observe that LIMoE could be a way forward for achieving a multimodal generalist model.

The researchers observed:

“We believe the ability to build a generalist model with specialist components, which can decide how different modalities or tasks should interact, will be key to creating truly multimodal multitask models which excel at everything they do.

LIMoE is a promising first step in that direction.”

Potential Shortcomings, Biases & Other Ethical Problems

There are shortcomings to this architecture that are not discussed in Google’s announcement but are mentioned in the research paper itself.

The research paper notes that, similar to other large-scale models, LIMoE may also introduce biases into the results.

The researchers state that they have not yet “explicitly” addressed the problems inherent in large scale models.

They write:

“The potential harms of large scale models…, contrastive models… and web-scale multimodal data… also carry over here, as LIMoE does not explicitly address them.”

The above statement makes a reference (in a footnote link) to a 2021 research paper called, On the Opportunities and Risks of Foundation Models (PDF here).

That research paper from 2021 warns how emergent AI technologies can cause negative societal impact such as:

“…inequity, misuse, economic and environmental impact, legal and ethical considerations.”

According to the cited paper, ethical problems can also arise from the tendency toward the homogenization of tasks, which can then introduce a point of failure that is then reproduced to other tasks that follow downstream.

The cautionary research paper states:

“The significance of foundation models can be summarized with two words: emergence and homogenization.

Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unanticipated consequences.

Homogenization indicates the consolidation of methodologies for building machine learning systems across a wide range of applications; it provides strong leverage towards many tasks but also creates single points of failure.”

One area of caution is in vision related AI.

The 2021 paper states that the ubiquity of cameras means that any advances in AI related to vision could carry a concomitant risk toward the technology being applied in an unanticipated manner which can have a “disruptive impact,” including with regard to privacy and surveillance.

Another cautionary warning related to advances in vision related AI is problems with accuracy and bias.

They note:

“There is a well-documented history of learned bias in computer vision models, resulting in lower accuracies and correlated errors for underrepresented groups, with consequently inappropriate and premature deployment to some real-world settings.”

The rest of the paper documents how AI technologies can learn existing biases and perpetuate inequities.

“Foundation models have the potential to yield inequitable outcomes: the treatment of people that is unjust, especially due to unequal distribution along lines that compound historical discrimination…. Like any AI system, foundation models can compound existing inequities by producing unfair outcomes, entrenching systems of power, and disproportionately distributing negative consequences of technology to those already marginalized…”

The LIMoE researchers noted that this particular model may be able to work around some of the biases against underrepresented groups because of the nature of how the experts specialize in certain things.

These kinds of negative outcomes are not theories, they are realities and have already negatively impacted lives in real-world applications such as unfair racial-based biases introduced by employment recruitment algorithms.

The authors of the LIMoE paper acknowledge those potential shortcomings in a short paragraph that serves as a cautionary caveat.

But they also note that there may be a potential to address some of the biases with this new approach.

They wrote:

“…the ability to scale models with experts that can specialize deeply may result in better performance on underrepresented groups.”

Lastly, a key attribute of this new technology that should be noted is that there is no explicit use stated for it.

It’s simply a technology that can process images and text in an efficient manner.

How it can be applied, if it ever is applied in this form or a future form, is never addressed.

And that’s an important factor that is raised by the cautionary paper (Opportunities and Risks of Foundation Models), calls attention to in that researchers create capabilities for AI without consideration for how they can be used and the impact they may have on issues like privacy and security.

“Foundation models are intermediary assets with no specified purpose before they are adapted; understanding their harms requires reasoning about both their properties and the role they play in building task-specific models.”

All of those caveats are left out of Google’s announcement article but are referenced in the PDF version of the research paper itself.

Pathways AI Architecture & LIMoE

Text, images, audio data are referred to as modalities, different kinds of data or task specialization, so to speak. Modalities can also mean spoken language and symbols.

So when you see the phrase “multimodal” or “modalities” in scientific articles and research papers, what they’re generally talking about is different kinds of data.

Google’s ultimate goal for AI is what it calls the Pathways Next-Generation AI Architecture.

Pathways represents a move away from machine learning models that do one thing really well (thus requiring thousands of them) to a single model that does everything really well.

Pathways (and LIMoE) is a multimodal approach to solving problems.

It’s described like this:

“People rely on multiple senses to perceive the world. That’s very different from how contemporary AI systems digest information.

Most of today’s models process just one modality of information at a time. They can take in text, or images or speech — but typically not all three at once.

Pathways could enable multimodal models that encompass vision, auditory, and language understanding simultaneously.”

What makes LIMoE important is that it is a multimodal architecture that is referred to by the researchers as an “…important step towards the Pathways vision…”

The researchers describe LIMoE a “step” because there is more work to be done, which includes exploring how this approach can work with modalities beyond just images and text.

This research paper and the accompanying summary article shows what direction Google’s AI research is going and how it is getting there.

Citations

Read Google’s Summary Article About LIMoE

LIMoE: Learning Multiple Modalities with One Sparse Mixture-of-Experts Model

Download and Read the LIMoE Research Paper

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (PDF)

Image by Shutterstock/SvetaZi