https://www.nature.com/articles/s41592-024-02353-z

The article begins by introducing the development of single-cell omics, which has given us unprecedented resources for studying cellular properties, including cell types and states and how they change during development, disease onset and treatment response. Despite the abundance of large datasets, current single-cell models are still unable to make full use of the information they contain.

The article then discusses the transformer architecture, a deep learning model that has become the architecture of choice for foundation models in other fields thanks to its ability to handle large-scale, heterogeneous datasets. The authors explore whether transformers could also trigger a similar shift in the field of single-cell modeling. The article describes the transformer architecture and its adaptation to single-cell data, provides a comprehensive review of existing applications of transformers in single-cell analysis, and critically discusses their potential in future single-cell biology.

The article also discusses the representation of single-cell data, including how non-sequential single-cell omics data can be converted into a format suitable as transformer input. It introduces the different components of the transformer architecture together with recently proposed adaptations specific to single-cell data. In addition, the article discusses applications of transformers on single-cell data, including tasks such as cell annotation and cell and gene representation learning, as well as single-cell modality prediction.

The article notes that although the application of transformers in single-cell omics has broad prospects, the field currently faces several limitations, including data noise, batch effects and sparsity. By examining these limitations and technical challenges, the authors aim to provide a structured outlook on future research directions at the intersection of machine learning and single-cell biology.

The article concludes by summarizing the potential of transformers in single-cell omics and setting out perspectives for future research, including the construction of a multimodal foundation model for a comprehensive understanding of cellular variation, and the modeling of cell states when perturbation data are taken into account. The authors stress that, despite the successful application of transformers in other fields, their application in single-cell omics still requires further evaluation and research.

1. Introduction

      The rise of single-cell omics has deepened our understanding of biological systems, giving us a detailed view of cellular heterogeneity and dynamics.

This includes new insights into cell types, cell states and their changes during development, disease onset and the response to treatment1-3. The data generated are large in volume and highly heterogeneous, presenting both opportunities and challenges. Current analysis methods cannot capture the variability in large-scale single-cell datasets spanning many samples, which has driven the development of novel computational strategies.

At the same time, machine learning has witnessed the remarkable impact of transformers, which were originally designed for natural language processing (NLP) tasks and have recently been used as the backbone of foundation models across domains4-6. A foundation model is a machine learning model trained by self-supervision on broad data; it can be efficiently adapted to a wide range of downstream tasks, usually with only a minimal amount of additional training6.

Transformer-based foundation models have been successfully applied in many fields, including computer vision7, speech processing8, time-series analysis9, and the modeling of categorical value sequences in genomics10 and proteomics11,12. The ability of transformers to exploit large-scale, heterogeneous datasets and to generalize across many tasks in different domains makes them a potential game changer for single-cell omics analysis (Fig. 1).

However, the data in all of these fields share one notable feature — a sequential structure that does not exist in raw single-cell omics data, which presents a unique challenge.

2. Overview of the transformer architecture

       The application of deep learning in single-cell biology is already widespread13–16.

Most deep learning models treat the input as a vector and iteratively apply a series of learnable layers, where each layer typically consists of a linear transformation followed by an element-wise nonlinear transformation.

In the context of single-cell genomics, the input vector may represent a cell, with the components of the vector corresponding to the RNA expression of individual genes. In single-cell applications, a common model architecture is the autoencoder, because it does not depend on data annotations, which are difficult to obtain. As shown in Fig. 2a, an autoencoder consists of an encoder and a decoder, and is trained to map the input to a low-dimensional latent representation that forms a bottleneck, from which the decoder attempts to faithfully reconstruct the input.

image-20241011111035128

Autoencoders trained on single-cell RNA counts can be used to reduce the dimensionality of cellular transcriptomic profiles. Because an autoencoder learns a single fixed set of parameters that is used for all inputs (that is, all cells), the bottleneck forces the model to learn feature extractors that capture the globally most relevant patterns while filtering out noise, ultimately yielding a meaningful latent representation of cellular variation.

       The transformer architecture, shown in Fig. 2, is a deep learning model that uses self-attention to process input data represented as a set of embeddings.

It borrows some ideas from the autoencoder, but also differs from it in essential ways. For example, like the autoencoder, the transformer as originally proposed contains an encoder and a decoder, and can also be trained without data annotations. Nevertheless, there are important differences between the two architectures.

First, unlike the autoencoder, the transformer uses the attention mechanism to adjust the processing of a given input feature according to all other input features in the data sample5. This allows the transformer, for example, to flexibly take into account the different gene interaction patterns being processed in different cell types.

Second, the transformer treats each input sample as a set of embeddings, which requires the cellular transcriptomic profile to be embedded into this format before it is passed to the transformer. For example, a cell can be represented by a set of gene embeddings that capture their respective RNA counts.

Third, the encoder and decoder components in the transformer differ in function and design from their counterparts in the autoencoder. In an autoencoder, the encoder maps the input to a bottleneck latent space, thereby performing dimensionality reduction (Fig. 2a), whereas the representation produced by the encoder in a transformer is not constrained by a bottleneck (Fig. 2b).

image-20241011111324868

Unlike its autoencoder counterpart, the transformer decoder processes the output of the transformer encoder and generates a sequence element by element. Transformer encoders and decoders are typically used independently; encoder-only models are often used to generate contextualized input embeddings for each gene. Such embeddings capture the properties of a given gene and its expression level in a given cell while taking into account the expression of other genes, which opens up possibilities for gene-level tasks, such as predicting gene dosage sensitivity in a specific cell17.

Decoder-only models and encoder–decoder transformers are used for sequence generation — for example, the simulation of transcriptomic cell profiles — but this requires representing RNA counts as a sequence. These significant differences compared with commonly used single-cell models underscore the need for a detailed description of single-cell transformers.

In this section, we review the components of the transformer architecture and recently proposed adaptations specific to single cells. For a glossary of terms, see Box 1.

image-20241011111547190

3. Representation of single-cell input data

       Transformers have had a remarkable impact on the processing of sequential data, for which they were originally designed5, such as natural language18, DNA10 or protein sequences19. Non-sequential single-cell omics data, however, present a challenge, because they require the data to be embedded into a format suitable for transformers.

A transformer treats each input sample as a set S composed of embeddings xi. The number of embeddings can vary between samples. In single-cell data, the input set S may represent the set of cells within a tissue or donor, where xi represents an individual cell. Alternatively, S may correspond to a single cell, with xi representing the properties of an individual gene, such as RNA counts.

This is analogous to tokenization in natural language processing, where text is split into subword units called “tokens,” and each unique subword has a corresponding embedding that is learned during training. Transformers are invariant to the order of input embeddings, which motivates the use of “positional encodings,” which are added to token embeddings to indicate the position of the token in the input.

xi is defined as the element-wise sum of the token embedding and a positional encoding of the same dimension5. Positional encodings can be computed with a fixed formula or learned, just like token embeddings. This encoding has been used in many transformer application areas, including natural language5, computer vision7 and time-series analysis9.

Notably, existing single-cell transformers differ mainly in the way they convert single-cell omics data into a form suitable as transformer input. The most commonly used approaches can be divided into three main categories (Fig. 3), as described below.

       Figure 3 analyzes three common approaches for converting the gene expression of a single cell into a format usable by transformers.

image-20241011111626542

Parts a–e show the process of converting the RNA counts of a single cell (a) into transformer input format; each approach uses additional embeddings that can be stored in a lookup table (b). These embeddings are typically learned during model training. All three approaches obtain the transformer input embedding through the element-wise summation of different embeddings.

Common processing approaches include:

●  Gene embeddings and positional encodings, where the order is determined by normalized RNA reads (c);

●  Gene embeddings together with embeddings of the bin corresponding to the expression level of each gene (d);

●  Embeddings obtained by projecting gene expression values together with positional encodings or gene embeddings (e). Parts f and g illustrate how some models define special tokens to encode additional information.

●  In part f, a special token, usually named CLS, can be used for cell (sample) representation. Such sample-level tokens constitute additional transformer input embeddings, which are concatenated with the input embeddings (denoted “concat”).

●  In part g, certain special token embeddings are also typically added to the input embeddings, similar to positional encodings, to encode additional gene (token)-level information. For example, such a token may indicate gene perturbation (denoted “perturb”).

Taken together, these approaches show how single-cell RNA count data can be converted into embedding representations that transformers can process, while the introduction of special tokens enhances the model’s ability to encode information at both the cell and gene levels.

For simplicity, we assume that the input sample S captures the transcriptomic profile of a cell. Note that other omics modalities have been parsed in a similar way20.

3-1: Ranking

The data are represented as a sequence of tokens similar to natural language input, which allows direct reuse of well-studied NLP transformer methods.

For example, in iSEEEK21, Geneformer17 and tGPT22, each gene is a token with a corresponding learnable embedding, and the order of gene tokens is determined by ranking normalized expression levels within the cell (Fig. 3c).

image-20241011112138010

Positional encodings are added to the token embeddings, just as in NLP. This approach reduces data resolution and leads to information loss23.

3-2: Value categorization

Each gene is represented by an embedding, and its RNA count in each cell undergoes value binning (Fig. 3d).

image-20241011112200963

Binning allows us to use methods designed for categorical data even when the raw input is continuous. As with ranking, it reduces data resolution. In value binning, we define contiguous value intervals. Each interval has a corresponding embedding, and xi is the sum of the gene embedding and the value bin embedding20,24. Intervals can be of equal size, as in the single-cell transformer scBERT24, or of adaptive size, as in scGPT20.

In the latter case, each interval represents an equal portion of all expressed genes in a single cell. These adaptively sized bins retain their meaning across sequencing batches, because the highest expression in a cell always corresponds to the highest expression bin.

In addition, the equal-sized bins of scBERT24 can, in some cases, cause almost all values to fall into a single bin, greatly reducing data resolution.

3-3: Value projection

Another approach uses a projection of the input data. Where the input data correspond to the transcriptomic profile of a cell represented as a vector of gene expression values, xi is the sum of two parts: a projection of the gene expression vector and a positional or gene embedding, where the projection is usually linear25 (Fig. 3e).

image-20241011112254837

The advantage of this solution over ranking and value categorization is that it does not necessarily reduce data resolution. However, it produces continuous embeddings, which differs from successful NLP transformers, and its effect on model performance remains unclear. The number of transformer input embeddings may or may not correspond to the number of genes in the input data. The number of output embeddings may correspond to the number of selected gene pathways, as in the model TOSICA25, a choice made to enhance model interpretability.

In addition, transformers can introduce an extra layer of information through special tokens. These tokens may represent perturbations applied to a given cell or gene20,26, species information26, data batch or data modality20.

Special token embeddings are either added to individual input embeddings xi for token-level information (Fig. 3f), or to the input set S for sample-level information (Fig. 3g). Special token embeddings are usually learnable, similar to gene or positional token embeddings.

image-20241011112327430

Where S represents a set of cells within a tissue and xi denotes an individual cell, xi can be modeled as the sum of a positional encoding and an embedding of the cell’s expression profile. The positional encoding is derived from the spatial coordinates of the cell, reflecting its position within the tissue structure. The embedding of the cell’s expression profile can be computed as the sum of learnable gene embeddings, each weighted by the expression level of the gene in that cell. This representation allows the modeling of cell–cell relationships, but limits the ability to analyze individual gene interactions27.

Given that single-cell transformers have only recently been introduced, few studies have examined the impact of encoding methods for non-sequential omics data. We expect the field to reach a consensus on this once best practices have been established.

4. Gene and cell attention

A transformer layer consists of two consecutive layers: an attention layer that transforms each element xi by “attending to” other elements in S, followed by a conventional feed-forward network (FFN) that processes elements xi individually. The attention mechanism is a key component of the transformer; it allows the transformer to capture dependencies between input embeddings and to generate contextualized embeddings5.

For details of the attention computation, see Box 2.

image-20241011112410699

In transformers for single-cell omics, where input embeddings correspond to genes, attention over genes may allow the transformer to capture underlying gene regulatory networks (GRNs). The so-called multi-head attention allows each attention layer to capture multiple facets of gene–gene relationships; for example, each attention head may capture a different cellular mechanism, such as the cell cycle, apoptosis, metabolism or DNA repair (Fig. 2f).

image-20241011112435189

Although in most current transformers the input embeddings correspond to genes17,20–22,24, modeling the interactions between cells within a tissue, where input embeddings correspond to cells27,28, has also been proposed. Attention over cells may exploit information on the spatial position of cells to encode intercellular relationships and capture cell–cell communication. Although modeling aspects of cell populations is promising, this review focuses mainly on gene-centric attention within single cells, because of its prevalence and more established status.

The attention variant described here is called self-attention. The output of attention is a transformation of the set of embeddings S', where the number of x'i is the same as in S, and the output embeddings usually have the same dimension as xi.

Figure 2d illustrates the computation of attention.

image-20241011112502830

x'i is a weighted sum of linear projections of all input elements xj in S. Importantly, the weights, which we call attention scores, depend on the elements in S themselves and are computed by the attention function. For simplicity, we described above what is called a single attention head, that is, an attention computation with one set of learnable parameters. Transformers use multi-head attention across multiple transformer layers (Fig. 2c)5.

image-20241011112521224

The computational complexity and memory requirements of the attention function grow quadratically with the number of input embeddings, owing to their pairwise interactions. This can become problematic when the number of input elements in S is large — for example, when the input embeddings correspond to more than a few thousand genes24. To address this, more efficient methods for computing attention have been developed, such as FlashAttention29, Performer30, sparse attention31 and iterative attention32.

Efficient attention computation and attention approximation, as well as general deep learning acceleration frameworks such as DeepSpeed33, remain active areas of research and development, which — from a computational standpoint — will likely allow future models to scale in both model capacity (number of model parameters) and input dimension (for example, number of genes).

Recent innovations — such as the DeepSpeed optimization library used in Geneformer17, efficient attention computation with FlashAttention29 used in scGPT20 and Cell2Sentence23, and attention approximation with Performer30 used in scBERT24 — have all been incorporated into various transformer models for single-cell applications.

The magnitude of attention scores can be used to infer the relative importance of input embeddings, essentially by acting as a weighting factor over value embeddings. However, the multiple layers and multiple attention heads in each layer require aggregation (for example, summation) of attention scores, which reduces interpretability. Attention scores and feature importance have been shown not to always correlate34, and therefore do not necessarily reliably identify the input elements responsible for the output.

5. Encoder and decoder

      The transformer was originally conceived as a combination of an encoder and a decoder, in which the transformer encoder and decoder are built by stacking multiple identical layers5.

The encoder (Box 3) processes the input data and generates latent representations, while the decoder outputs a sequence based on these representations.

image-20241011112617051

The encoder and decoder have slightly different architectures, both based on multiple layers of attention and FFN. The decoder differs from the encoder only in its attention components, using masked attention and encoder–decoder attention in place of the encoder’s attention. Masked attention constrains the attention mechanism so that each output can only be influenced by the preceding elements in the sequence (Fig. 4c,d).

image-20241011112641184

Encoder attention operates only on the input sequence, whereas encoder–decoder attention allows the decoder to attend to the output of the encoder as well as to its own input elements. This mechanism enables the decoder to align the output it generates with the input embeddings produced by the encoder.

To date, most single-cell transformers rely on an encoder-only architecture17,24,26, which enables masked language modeling (MLM) — an effective pre-training strategy that is described in detail in the next section.

tGPT22 and scMulan35 are exceptions that use decoder-only models. Decoder-only models allow the generation of conditional data — for example, transcriptomic profiles conditioned on cell type, donor age or sequencing technology35,36. Some models use variants of encoder- or decoder-based transformer layers. For instance, scGPT20 uses customized masked attention inspired by decoder attention masking, allowing autoregressive generation.

If sufficiently many annotated data are available, transformers can be trained directly on the task of interest. However, although single-cell datasets with millions of observations37 are readily available, cell annotations are usually limited and inconsistent across datasets38. This makes self-supervised learning (SSL) attractive for single-cell modeling through masked token or next token prediction (NTP), because it relies on the intrinsic structure of the data rather than on human annotations.

A very common self-supervised pre-training strategy is called MLM. The MLM strategy replaces a fraction of the input tokens with a MASK token embedding and predicts the masked tokens from the final embeddings (transformer output)4. MLM is used by many single-cell transformers17,20,24,26,39,40. Another form of pre-training used in transformer decoders and transformer encoder–decoder models is the NTP task. This task assumes that the input is a sequence, and the model is trained to predict the next token given the preceding tokens.

Although some models show improvements after self-supervised pre-training4, others achieve state-of-the-art results even without pre-training8. Similarly, some single-cell transformers make use of SSL pre-training17,20,24, as shown in Fig. 4a,b, whereas other models are trained in a purely supervised way on the task of interest25.

image-20241011112813846

Moreover, SSL pre-training has been shown not to improve performance on certain tasks41.

6. Applications of single-cell transformers

To date, transformers have been applied to a variety of single-cell tasks, such as cell annotation, cell and gene representation learning, and single-cell modality prediction under specific conditions or perturbations17,20,36.

       Figure 5 presents application cases of transformers in single-cell RNA sequencing (scRNA-seq).

image-20241011112846982

In part a, transformers can be used to predict the transcriptional outcome of genetic perturbations. One way to simulate a perturbation is to add a learnable embedding to the gene embedding that indicates the gene perturbation (labeled “Pert.”)20. G1–G4 denote embeddings corresponding to specific genes. The model outputs the gene embeddings altered by the perturbation.

In part b, single-cell transformers have been used to integrate the molecular profiles of cells across studies, tissues and even species 26,55. This demonstrates the broad applicability of transformers in handling single-cell data from different sources and of different types.

In part c, the output of the CLS token can be used for cell-level tasks, while the remaining outputs are used for gene-level tasks20. This shows that single-cell transformers can process data at different levels simultaneously, thereby informing both cell-level and gene-level analyses.

In part d, attention scores may capture gene–gene relationships within the cell, which can be used to infer gene regulatory networks (GRNs)20. This application of the attention mechanism reveals the potential of single-cell transformers for understanding complex intracellular interactions.

In summary, Figure 5 illustrates the application of single-cell transformers across areas such as predicting gene perturbation effects, cross-dataset integration, cell- and gene-level tasks, and the inference of gene regulatory networks, highlighting their versatility and potential in single-cell genomics research.

A list of selected single-cell transformers and their applications is provided in Table 1, and a comprehensive, categorized extended list is available at https://github.com/theislab/single-cell-transformer-papers.

image-20241011113037318

While some transformers have been developed for a single task — for example, cell type annotation or cross-modality prediction25,42 — others attempt to solve multiple tasks. The hope is that models trained on sufficiently diverse data (usually through SSL) will acquire foundational knowledge of cell biology, enabling them to perform well across a broad range of tasks related to cell function.

Technically, addressing different tasks is achieved by using the model in a zero-shot manner, or by adapting the final layers and fine-tuning the model for the selected task17,20.

6-1: Challenges in modeling single-cell data

Modeling single-cell data presents several major challenges that need to be considered. These challenges include43:

       Noise

Single-cell data are affected by various technical factors. For example, the data contain sequencing noise, such as dropout and cellular variation, that cannot be attributed to naturally occurring cell states, but are instead consequences of technical effects during processing, such as stress.

Arguably, observing more cells than is typical for a single-dataset model can improve the learning of general noise characteristics by transformer-based models. Some transformers also attempt to improve signal quality by preprocessing data with value categorization20,24.

       Batch effects

Data generated by different experiments, conditions or laboratories may differ systematically — so-called batch effects — which can confound the analysis. Some transformers attempt to explicitly correct batch effects using batch tokens20, while others rely on the model’s ability to generalize better after exposure to heterogeneous datasets17.

       Sparsity

Many single-cell readouts, such as single-cell assay for transposase-accessible chromatin with sequencing (scATAC-seq) and scRNA-seq44, are highly sparse, with most measured values equal to zero. Machine learning models need to take this into account to effectively avoid excessive computation.

Many single-cell transformers handle sparsity by processing only non-zero values17 or by grouping features by pathway25.

6-2: Gene representation

Transformers applied to single-cell data typically encode gene expression (or other gene-related omics modalities) as input embeddings, thereby producing gene-level representations. Such gene embeddings depend on the context provided by the expression of other genes in the cell, making them context-specific.

Context specificity is achieved by applying attention across genes (or other omics features) within a cell17,20. Context-specific embeddings can be used for a variety of tasks that have so far relied on fixed gene representations — for example, identifying functionally similar genes, predicting gene function, and predicting how gene function changes under different conditions (for instance, by comparing changes in gene embeddings after simulated treatments)17,20.

Contextualized gene embeddings can provide new insights by comparing similarity scores of such embeddings across cells. For example, gene embeddings from pre-trained scGPT20 were used to build similarity networks that group functionally related genes, identifying known pathways, including some not revealed by co-expression networks. In addition, fine-tuning the gene embeddings in Geneformer17 proved effective in predicting chromatin states and in distinguishing central from peripheral factors within gene networks in specific cell types.

Embeddings generated by transformers can also be used to inform other models that rely on predefined gene embeddings26,39,45, such as the perturbation response prediction model GEARS46. Likewise, although most current cross-species analyses use fixed gene orthology information (that is, gene identifiers (IDs) matched across species), predefined orthology mappings may lack functional gene information, relying only on protein sequence similarity and being context-independent47.

Therefore, multi-species transformers26 may provide improved context-specific orthology mappings.

6-3: Interactions between omics features

Omics features encompass the diverse data obtained from studies in various “omics” fields, such as genomics, epigenomics, transcriptomics, proteomics and metabolomics. These features include, but are not limited to, genes, transcripts, proteins, metabolites and accessible chromatin regions.

Interactions between omics features are essential for understanding the biology of cells and organisms. For example, paired transcriptomic and chromatin accessibility assays have been used to characterize interactions between genes48, which has important implications for disease prevention49.

Transformers introduce a novel way of examining multimodal interactions through the attention mechanism between omics features, producing a learnable graph of data relationships. Attention scores between cells and omics feature tokens can therefore be used to identify cell type marker genes, genes associated with specific cell phenotypes, and genes linked to biological processes, such as developmental regulators and genes associated with particular cell phenotypes22,25,50.

Similarly, attention can be used to predict interactions between omics features and to identify hub genes — genes that regulate or are regulated by a large number of other genes17,51. For example, gene attention scores in Geneformer17 concentrate on transcription factors and hub genes, revealing cellular regulatory mechanisms. Attention values are context-specific, so combining ATAC-seq and RNA-seq data may reveal context-specific (for example, cell-state-specific) gene regulation based on the expression of co-binding transcription factors and chromatin accessibility. This approach to model interpretability and biological insight discovery has been explored in TOSICA25.

TOSICA operates on pathway attention scores using pathway embeddings as cell representations, capturing cell trajectories and linking trajectory changes to specific pathways or regulons, thereby highlighting the regulatory networks that drive disease progression. In addition, scGPT20 uses gene attention scores not only to infer gene regulatory networks (GRNs), but also to analyze the impact of genetic perturbations on these networks, demonstrating the diverse insights that can be extracted from the attention scores of single-cell transformers.

Despite these successes, the utility of attention scores for interpretability is nuanced. As noted above, transformer attention scores do not correlate with feature importance in some cases34, which highlights the need for caution when generalizing their reliability as a tool for attributing feature importance.

6-4: Cell representation

High-quality representations of individual cells in a low-dimensional space are a key component of various downstream single-cell analyses52. At its core, this involves preserving biological variation, such as cell type and cell state, while minimizing technical confounders such as batch effects across datasets53.

However, when integrating data from multiple studies, tissues or even organisms, distinguishing unwanted batch effects from relevant covariates is challenging and often context-dependent54.

Transformers offer a promising approach to this problem through batch-unaware pre-training, which has been shown to be robust to certain batch effects. For example, Universal Cell Embeddings (UCE)55 and GeneCompass26 have been used to integrate the molecular profiles of cells at scale across studies, tissues and species, enabling UCE to transfer cell type annotations to data from species unseen by the model.

       Transformer-based embeddings differ from other techniques in several respects.

Unlike popular models based on variational autoencoders (VAEs)56, such as scVI13 and scArches57, or the recently proposed SCimilarity14, which explicitly learn low-dimensional embeddings, transformers do not by themselves produce low-dimensional cell embeddings through the commonly used pre-training tasks.

Instead, cell embeddings can be obtained by pooling the output embeddings of the transformer for an individual cell, which may correspond to contextualized gene embeddings17,45, or by introducing a special cell token (CLS) into the input that is trained on tasks capturing cell properties20,25.

Furthermore, although VAE-based models typically learn the influence of batch covariates explicitly by passing them along with the input features13, some transformers are claimed to be robust to batch-dependent technical artifacts despite not using batch covariate information explicitly17,26. In addition, some transformers can generalize to bulk data, producing meaningful bulk embeddings22,39. The flexibility of transformer input tokens facilitates the use of multimodal features for cell representation.

For example, scGPT20 treats individual features across omics layers as separate tokens, enabling paired and mosaic integration20, and has been shown to capture immune cell subsets that other state-of-the-art multi-omics integration methods failed to identify.

Other applications of transformer cell embeddings include identifying therapeutic targets by analyzing changes in cell embeddings after perturbation. For example, in Geneformer17, shifting the cell embeddings of cardiomyocytes from patients with hypertrophic or dilated cardiomyopathy toward embeddings corresponding to non-failing heart states suggested that inhibiting these genes might improve cardiomyocyte function.

The predicted effects have been validated experimentally, demonstrating the practical utility of single-cell transformers.

6-5: Single-cell modality generation

Transformers have been applied to a variety of tasks based on predicting gene expression or other omics modalities in individual cells. We refer to this class of tasks as single-cell modality generation. This includes predicting gene expression under perturbation — achieved by simulating the perturbation of a single input gene (for example, knockout or knockdown) or by introducing perturbation conditions (for example, exposing cells to a small molecule)17,20,58,59 — increasing read-depth predictions39, data imputation36, and cross-modality prediction, which uses one or more known modalities to predict a missing modality42,51.

For example, in a task where the state of control cells is given as input and the expression profile of perturbed cells is predicted, scGPT was fine-tuned on a subset of the Perturb-seq dataset, with special tokens indicating which genes were perturbed. The model was then shown to accurately predict gene expression responses to unseen perturbations.

In addition, generative transformers may simulate data directly, for example by specifying only the condition as input without any omics features36. Once such models are developed, they can be used for perturbation modeling and for predicting control datasets in settings where obtaining matched control tissue is challenging, such as studies involving invasive biopsies or phase I clinical trials.

6-6: Cell annotation

Many single-cell transformers are designed for single-cell annotation tasks, mainly using the transcriptomic profiles of cells. The primary focus of annotation is cell type prediction; for example, the scBERT24 and TOSICA25 models focus exclusively on this task.

General-purpose single-cell transformers are also commonly evaluated on cell type annotation17,20,26,40, and self-supervised pre-training through masked input modeling on large datasets has been shown to improve their classification performance17,20. Transformers show an encouraging ability to generalize to unseen datasets36, which is essential for annotating new datasets using reference datasets with consensus annotations. For example, TOSICA was shown to predict cell types effectively even on datasets unseen during training, overcoming batch effects25.

In addition, the UCE model has been used to transfer cell type annotations to unseen species55. However, non-deep-learning cell type predictors are often difficult to surpass. For example, in organ-specific cell type annotation, logistic regression typically performs comparably to more complex nonlinear predictors41,60,61.

In cross-organ cell type annotation, a non-transformer model, scTab, was shown to outperform both state-of-the-art non-deep-learning cell type predictors and scGPT62. Furthermore, although the abundance of data with cell type annotations facilitates evaluation on cell type annotation tasks, annotations are often inconsistent across datasets, and the reliability of cell type annotations has been questioned38.

Therefore, the cell type annotation task may not be a reliable indicator of model performance.

6-7: Spatial omics

       Transformers have also shown promise for spatial omics applications27,28,36.

In this study, we consider only models applied directly to spatial omics data, excluding those that process pathology slide images (PS: they leave these out, but those of us working on pathology might want to consider them). Nicheformer achieved promising results in predicting spatial neighborhood cell density and neighborhood composition.

In addition, it enables transfer learning between spatial and RNA sequencing analyses, allowing the prediction of the spatial context of dissociated cells based on their gene expression63. scGPT36, SpaFormer28 and CellPLM27 have proven effective in spatial transcriptomics gene expression imputation tasks, but only SpaFormer and CellPLM process multiple cells (rather than genes) simultaneously to model intercellular relationships.

These two models are also currently the only ones that include spatial information in the model input, encoding the spatial coordinates of cells through positional encodings. In addition, CellPLM performs well in denoising single-cell RNA sequencing data by leveraging the expression of similar cells in the same tissue. The rapid development of spatial transcriptomics64,65 and the ability of transformers to resolve spatial coordinates in other fields7 make the integration of these technologies a promising new area.

7. Large language models (LLMs) in single-cell analysis

       In addition to transformers trained on molecular data, there have been attempts to use transformers trained on text — that is, large language models (LLMs) — to model single-cell data.

We restrict the term LLM here to the context of modeling natural language, whereas some sources use the term to describe any transformer that processes sequential data. Some methods use language models directly, while others encode molecular data as text for further fine-tuning. The motivation for this approach comes from the success of LLMs such as GPT-4 (ref. 66), which is reported to be able to solve diverse and difficult tasks across a range of domains67.

After training on general language datasets, GPT-4 has shown promise in cell type annotation68 and various other biomedical tasks69 without task-specific training. In addition, embeddings from general-purpose LLMs have proven useful for certain single-cell tasks, including batch integration, cell type classification and gene property prediction70, suggesting that LLMs encode some single-cell knowledge.

Early attempts built LLM capabilities by converting single-cell omics data into text and then fine-tuning pre-trained language models23. For example, models that combine text with transcriptomic data, such as Cell2Sentence23, typically represent the transcriptomic profile of a cell as a sequence of genes ordered by expression level.

This is not unlike the ranking approach used for embedding transcriptomic data, but it allows additional textual cell descriptors to be added. Beyond integrating text with omics data, a specialized multimodal transformer pre-trained on general text and then trained on biomedical text, imaging and genomic data significantly outperformed its general-purpose counterpart on multiple biomedical tasks71.

These results demonstrate the promise of using general-purpose language models as a basis for specialized single-cell models. However, the challenge of effectively fusing textual data and text-centric models with single-cell omics data, which differ substantially from natural language, remains.

8. Foundation models

       “Foundation model” is a recently coined term describing an artificial intelligence (AI) system trained on large, diverse datasets to learn patterns without human annotation6.

Training on large unlabeled datasets allows them to adapt to a wide range of tasks using far less data than would be required to train a model from scratch. The transformer is the architecture of choice for foundation models. Examples include the natural language transformer architectures BERT4 and GPT-4 (ref. 66), Whisper8 for speech recognition and Vision Transformer7 for image recognition.

Recent work has explored the possibility of developing foundation models with alternative architectures, including graph neural networks (GNNs)72,73. GNNs have shown promise in applications such as modeling gene regulatory networks, molecular structures and intercellular communication in spatial transcriptomics74,75. However, they have not yet demonstrated the scalability that transformers achieve on large, diverse datasets.

When the relationships between entities are unknown, as is often the case in modeling biological data, transformers are appropriate. Although models combining the strengths of transformers and GNNs are under active development, the flexibility and efficient implementations of transformers make them the dominant architecture for developing foundation models6,72,73.

       A single-cell foundation model is a machine learning model trained on large amounts of data that can be efficiently adapted to a wide range of applications relevant to single-cell research.

Several recent single-cell models have been pre-trained on large datasets and claim to be foundation models, including the autoencoder-based SCimilarity14 and several transformers20,26,39. GeneCompass26 stands out with its 12-layer transformer architecture and more than 100 million parameters, using the largest pre-training dataset to date, containing over 120 million single-cell transcriptomes.

On the other hand, UCE55 is the largest single-cell transformer to date in terms of reported parameter count, with 650 million parameters spread across 33 layers. Even so, these models remain far smaller than NLP models. For example, PaLM has 112 layers and roughly 540 billion parameters76.

Despite the scale of single-cell transformers and their training datasets, independent benchmarks indicate that current single-cell foundation models are often outperformed by simpler, task-specific models that require substantially less training data and computational resources36,41,77.

9. Current limitations and evaluation of model capabilities

Although the promise of transformers in single-cell genomics is clear, the field is still at an early stage and faces a series of limitations. For example, the source of the sequential structure required for the direct application of transformers remains unclear. Current solutions, such as ranking genes to convert tabular data into sequences17,21 or ordering genes along chromosomal positions55, may be only preliminary.

To discern the added value of transformers in single-cell omics, rigorous evaluation is necessary. This will help answer several methodological questions, such as what the optimal data encoding, architecture and training procedures are, and evaluating the utility of transformers in biologically relevant single-cell applications that go beyond existing methods.

For example, although transformers may be trained on large datasets, it is critical to determine whether non-transformer-based methods, trained on comparable datasets with comparable computational resources, would produce similar or better results. Accordingly, several benchmarks of transformer models have been carried out36,41,77–79.

Preliminary results suggest that although single-cell transformers may generalize across datasets, state-of-the-art task-specific models as well as simpler models (such as logistic regression) often outperform them on tasks with fewer annotations, and the zero-shot performance of current transformers is open to question36,41,77.

Notably, although SCimilarity14 adopts an autoencoder architecture, it has shown promise for integrating diverse single-cell datasets after training on large datasets. Transformers therefore cannot yet be regarded as the method of choice for the future.

       Although the importance of model evaluation is self-evident, the design of existing benchmarks has several problems.

First, a common problem is insufficient evaluation of competing models. Comparisons often neglect hyperparameter tuning of competing models and overlook state-of-the-art models such as XGBoost80 and logistic regression81. All models that can exploit pre-training need to be pre-trained on the same datasets.

Second, many biological tasks, such as perturbation effect prediction, lack universally accepted metrics and benchmarks. Third, the lack of evaluation on large and diverse datasets can be a problem when assessing the performance of models that claim broad single-cell capabilities.

A large-scale benchmark for cell type classification has recently been proposed62. Fourth, dataset leakage — particularly in large single-cell models — is an emerging issue that needs attention.

Specifically, because models make use of large datasets during pre-training, it is difficult to ensure that the data used for model evaluation, or other highly similar datasets, have not previously been seen and memorized by the model. Finally, benchmarking itself is complicated by its resource intensity and by the lack of easily accessible datasets, code and model weights for training and evaluating models.

       Beyond models that directly model omics measurements, evaluating generative language models on single-cell tasks also poses unique challenges.

Specifically, because these models are trained on text generation, their outputs may not conform to the desired format. In addition, many state-of-the-art language models are not publicly available, and results obtained through the interfaces they provide may change over time23,68. Evaluation of LLMs on single-cell tasks therefore requires special adaptation.

At present, the disparities in how single-cell transformers are evaluated have created a confusing situation. The field would therefore benefit from comprehensive evaluation of the kind common in other fields. A transparent public leaderboard, open to contributions, would further increase the credibility of model comparisons83.

10. Conclusions and outlook

The following summarizes several topics for the future.

10-1: Toward a multimodal foundation model of cellular variation

       Ongoing work aims to build comprehensive reference atlases of all cells in humans and model organisms84–86.

Although the volume of single-cell data is increasing, the problem of jointly modeling cells across different studies, tissues, conditions and species remains unresolved. Currently, the main strategy for dataset integration is to create low-dimensional, batch-corrected cell embeddings for selected datasets of interest, usually limited to a single organ and species, and to rely on the learned representations for downstream analysis87,88.

Single-cell transformers offer an alternative approach — they have the capacity to be pre-trained on diverse datasets, potentially including all single-cell omics data generated across modalities. Jointly modeling omics data, including transcriptomics, epigenomics, the proteome and the spatial organization of cells, may enable transformers to accurately encode cell biology by integrating the different perspectives provided by each omics modality.

A multi-omics view of the cell is necessary for a fundamental understanding of the molecular hierarchy from genome to phenotype. For example, joint analysis of epigenomics and transcriptomics has provided new insights into the role of DNA methylation in cell differentiation89. Although current single-cell transformers integrate multiple modalities, they use only a small number of studies with readouts other than scRNA-seq.

Therefore, the development of large-scale multi-omics transformers remains an aspiration for future research20. Diverse inputs and pre-training techniques can capture biological phenomena at different scales, including the subcellular, cellular and tissue levels, making transformers a viable architecture for developing single-cell foundation models. Such multimodal foundation models may have the capacity to accurately model the gene regulatory structure of cells as well as truly multiscale tissue-level interactions.

However, multimodal foundation models pose major challenges in terms of computational resource requirements, because introducing more modalities may require longer input sequences. To mitigate the memory and computational demands of increased context length, models may adopt sparse data representation techniques — for example, considering only non-zero features or grouping raw omics features into meaningful units such as pathways.

In the broader machine learning community, an ongoing debate concerns whether a single multimodal model, broadly applicable across all domains and use cases, will exist in the future. Such a “general-purpose foundation model” would link all types of modalities, including text, images and audio, across domains — for example, general conversation, medical diagnosis and scientific experimentation — all through a natural language interface.

Set against this are foundation models for specific domains or modalities, trained on large but domain-specific datasets. Recent proposals for biomedical multimodal foundation models90 focus on the latter, and the first examples have also appeared in single-cell genomics and many other fields91–93.

Meanwhile, textual representations of cell profiles23 may help to develop general-purpose text foundation models spanning multiple scientific fields such as biology, medicine and chemistry.

10-2: Modeling the perturbation landscape

Recent perspectives on cell representation emphasize the need, in addition to describing cells using a snapshot of their molecular features, to capture how cells respond to perturbations58,94.

As perturbation datasets become more comprehensive, we expect future single-cell transformers to be pre-trained on rich perturbation screens in addition to the omics snapshots currently used, which will bring us closer to a holistic understanding of cell states and gene regulatory networks (GRNs)94.

If generative transformers can achieve this detailed understanding of single-cell data, they may become general-purpose single-cell data simulators. Such models could be used to synthesize the cellular omics data required for specific cell characteristics95.

10-3: Conclusions

The application of transformers in single-cell omics holds great potential, but caution is warranted until thorough evaluation has been carried out. It remains unclear whether transformers are the right architecture for modeling non-sequential omics data, and whether they outperform existing methods in the field. Unlike other machine learning models, however, transformers have demonstrated improvements across a broad range of tasks after pre-training on large-scale, diverse and unlabeled datasets.

Effective self-supervised pre-training makes it possible for them to benefit from increasingly available large-scale omics datasets, and potentially to be applicable to a wide range of data and downstream tasks. Although the aspiration of developing a general, foundational transformer encompassing all known single-cell biology is appealing, such a model is far from being realized. Despite the success of transformers across diverse modalities in other fields, their application to modeling single-cell omics data is still in its infancy. Combined with general work on interpreting learned models and their attention scores, it will be exciting to follow developments in this rapidly evolving field.

A categorized list of single-cell transformer papers can be found at https://github.com/theislab/single-cell-transformer-papers.