Transformer (machine learning model)

22 languages

- Article

- Talk

- Read

- Edit

- View history

Tools

From Wikipedia, the free encyclopedia

This article relies excessively on references to primary sources. Please improve this article by adding secondary or tertiary sources.

Find sources: "Transformer" machine learning model – news · newspapers · books · scholar · JSTOR (February 2023) (Learn how and when to remove this template message)

"Transformer architecture" redirects here. For the design of electrical transformers, see Transformer.

Part of a series on

Machine learning

and data mining

An illustration of a neural network with dark

show

Paradigms

show

Problems

show

Supervised learning

(classification • regression)

show

Clustering

show

Dimensionality reduction

show

Structured prediction

show

Anomaly detection

hide

Artificial neural network

- Autoencoder

- Cognitive computing

- Deep learning

- DeepDream

- Feedforward neural network

- Recurrent neural network

- LSTM

- GRU

- ESN

- reservoir computing

- Restricted Boltzmann machine

- GAN

- Diffusion model

- SOM

- Convolutional neural network

- U-Net

- Transformer

- Vision

- Spiking neural network

- Memtransistor

- Electrochemical RAM (ECRAM)

show

Reinforcement learning

show

Learning with humans

show

Model diagnostics

show

Mathematical foundations

show

Machine-learning venues

show

Related articles

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism.[1] The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM),[2] and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.[3] Input text is split into n-grams encoded as tokens and each token is converted into a vector via looking up from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism allowing the signal for key tokens to be amplified and less important tokens to be diminished. Though the transformer paper was published in 2017, the softmax-based attention mechanism was proposed earlier in 2014 by Bahdanau, Cho, and Bengio for machine translation,[4][5] and the Fast Weight Controller, similar to a transformer, was proposed in 1992 by Schmidhuber.[6][7][8]

This architecture is now used not only in natural language processing and computer vision,[9] but also in audio[10] and multi-modal processing. It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs)[11] and BERT[12] (Bidirectional Encoder Representations from Transformers).

Timeline[edit]

- In 1990, Elman network, using a recurrent network, encoded each word in a training set as a vector, called static "word embedding", and the whole vocabulary as a vector database, allowing it to perform such tasks as sequence-prediction that are beyond the power of a simple multilayer perceptron. Shortcoming of the static embeddings was that they didn't differentiate between multiple meanings of same-spelt words.[13]

- In 1992, the Fast Weight Controller was published by Jürgen Schmidhuber[6] It learns to answer queries by programming the attention weights of another neural network through outer products of key vectors and value vectors called FROM and TO. The Fast Weight Controller was later shown to be closely related to the Linear Transformer.[8][7][14] The terminology "learning internal spotlights of attention" was introduced in 1993.[15] An advantage of the fast linear transformers is that their computational complexity grows linearly with sequence length, while modern transformers scale quadratically.

- In 1993, the IBM alignment models were used for statistical machine translation.[16]

- In 1997, a precursor of large language model, using recurrent neural networks, such as long short-term memory, was proposed.

- In 2001, one-billion-word large text corpus, scraped from the Internet, referred to as "very very large" at the time, was used for word disambiguation.[17]

- In 2012, AlexNet demonstrated the effectiveness of large neural networks for image recognition, encouraging large artificial neural networks approach instead of older, statistical approaches.

- In 2014, a 380M-parameter seq2seq model using two LSTMs networks was proposed by Sutskever at al.[18]

- In 2014, gating proved to be useful in a 130M-parameter seq2seq model, which used a simplified gated recurrent units (GRUs). Bahdanau et al[19] showed that GRUs are neither better nor worse than gated LSTMs.[20][21]

- In 2014, improving the previous model by using an "additive" kind of attention mechanism in-between two LSTM networks was proposed by Bahdanau et al.[22] It was, however, not yet the parallelizable (scaled "dot product") kind of attention, later proposed in the 2017 transformer paper.

- In 2015, the relative performance of Global and Local (widowed) attention model architectures were assessed by Luong et al, a mixed attention architecture found to improve on the translations offered by Bahdanau's architecture, while the use of a local attention architecture reduced translation time. [23]

- In 2016, Google Translate gradually replaced the older statistical machine translation approach with the newer neural-networks-based approach that included a seq2seq model combined by LSTM and the "additive" kind of attention mechanism. They achieved a higher level of performance than the statistical approach, which took ten years to develop, in only nine months.[24][25]

- In 2017, the original (100M-sized) transformer model with a faster (parallelizable or decomposable) attention mechanism was proposed in the "Attention is all you need" paper. Because the model had difficulties converging, it was suggested that the learning rate should be linearly scaled up from 0 to maximal value for the first part of the training (i.e. 2% of the total number of training steps).[1]

- In 2018, in the ELMo paper, an entire sentence was processed before an embedding vector was assigning to each word in the sentence. A bi-directional LSTM was used to calculate such, deep contextualized embeddings for each word.

- In 2018, a bi-directional transformer has been used in (more than 1B-sized) BERT model.[26]

- In 2020, vision transformer[27] and speech-processing convolution-augmented transformer[28] outperformed recurrent neural networks, previously used for vision and speech.

- In 2020, difficulties with converging the original transformer were solved by normalizing layers before (instead of after) multiheaded attention by Xiong et al.[29]

- In 2023, uni-directional ("autoregressive") transformers were being used in the (more than 100B-sized) GPT-3 and other OpenAI GPT models.[30][31]

Predecessors[edit]

Before transformers, predecessors of attention mechanism were added to gated RNNs, such as LSTMs and gated recurrent units (GRUs), which processed datasets sequentially. Dependency on previous token computations prevented them from being able to parallelize the attention mechanism. In 1992, fast weight controller was proposed as an alternative to RNNs that can learn "internal spotlights of attention".[15][6] In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.

The performance of old models was enhanced by adding an attention mechanism, which allowed a model to access any preceding point along the sequence. The attention layer weighs all previous states according to a learned measure of relevance, providing relevant information about far-away tokens. This proved to be especially useful in language translation, where far-away context can be essential for the meaning of a word in a sentence. The state vector has been accessible only after the last English word was processed while, for example, translating it from French by a LSTM model. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved. If an attention mechanism is added, the decoder is given access to the state vectors of every input word, not just the last, and can learn attention weights that dictate how much to attend to each input state vector. The augmentation of seq2seq models with the attention mechanism was first implemented in the context of machine translation by Bahdanau, Cho, and Bengio in 2014.[4][5]

Decomposable attention[edit]

In 2016, highly parallelizable decomposable attention was successfully combined with a feedforward network.[32] This indicated that attention mechanisms were powerful in themselves and that sequential recurrent processing of data was not necessary to achieve the quality gains of RNNs with attention. Soon Jakob Uszkoreit from Google Research also proposed replacing RNNs with self-attention and started the effort to evaluate that idea.[33] Transformers, using an attention mechanism, processing all tokens simultaneously, calculated "soft" weights between them in successive layers. Since the attention mechanism only uses information about other tokens from lower layers, it can be computed for all tokens in parallel, which leads to improved training speed.

Training[edit]

Methods for stabilizing training[edit]

The plain transformer architecture had difficulty converging. In the original paper[33] the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of the training (usually recommended to be 2% of the total number of training steps), before decaying again.

A 2020 paper found that using layer normalization before (instead of after) multiheaded attention and feedforward layers stabilizes training, not requiring learning rate warmup.[29]

Pretrain-finetune[edit]

Transformers typically undergo self-supervised learning involving unsupervised pretraining followed by supervised fine-tuning. Pretraining is typically done on a larger dataset than fine-tuning, due to the limited availability of labeled training data. Tasks for pretraining and fine-tuning commonly include:

- language modeling[12]

- next-sentence prediction[12]

- question answering[3]

- reading comprehension

- sentiment analysis[1]

- paraphrasing[1]

Applications[edit]

The transformer has had great success in natural language processing (NLP), for example the tasks of machine translation and time series prediction. Many large language models such as GPT-2, GPT-3, GPT-4, Claude, BERT, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. These may include:

- machine translation

- document summarization

- document generation

- named entity recognition (NER)[34]

- biological sequence analysis

- writing computer code based on requirements expressed in natural language.

- video understanding.

In addition to the NLP applications, it has also been successful in other fields, such as computer vision, or the protein folding applications (such as AlphaFold).

Implementations[edit]

The transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch.

Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.[11]

Architecture[edit]

All transformers have the same primary components:

- Tokenizers, which convert text into tokens

- Embedding layers, which convert tokens into semantically meaningful representations

- Transformer layers, which carry out the reasoning capabilities, and consist of Attention and MLP layers

Transformer layers can be one of two types, encoder and decoder. In the original paper both of them were used, while later models included only one type of them. BERT is an example of encoder-only model; GPT are decoder-only models.

Input[edit]

The input text is parsed into tokens by a byte pair encoding tokenizer, and each token is converted into a vector via looking up from a word embedding table. Then, positional information of the token is added to the word embedding.

Encoder/decoder architecture[edit]

Like earlier seq2seq models, the original transformer model used an encoder/decoder architecture. The encoder consists of encoding layers that process the input tokens iteratively one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output as well as the decoder output's tokens so far.

The function of each encoder layer is to generate contextualized token representations, where each representation corresponds to a token that "mixes" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e., the tokens generated so far during inference time).[35] [36]

Both the encoder and decoder layers have a feed-forward neural network for additional processing of the outputs and contain residual connections and layer normalization steps.[36]

Scaled dot-product attention[edit]

The transformer building blocks are scaled dot-product attention units. For each attention unit, the transformer model learns three weight matrices: the query weights

{\displaystyle W_{Q}}, the key weights

{\displaystyle W_{K}}, and the value weights

{\displaystyle W_{V}}. For each token

i, the input token representation

x_{i} is multiplied with each of the three weight matrices to produce a query vector

{\displaystyle q_{i}=x_{i}W_{Q}}, a key vector

{\displaystyle k_{i}=x_{i}W_{K}}, and a value vector

{\displaystyle v_{i}=x_{i}W_{V}}. Attention weights are calculated using the query and key vectors: the attention weight

a_{ij} from token

i to token

j is the dot product between

q_{i} and

k_j. The attention weights are divided by the square root of the dimension of the key vectors,

{\displaystyle {\sqrt {d_{k}}}}, which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that

{\displaystyle W_{Q}} and

{\displaystyle W_{K}} are different matrices allows attention to be non-symmetric: if token

i attends to token

j (i.e.

{\displaystyle q_{i}\cdot k_{j}} is large), this does not necessarily mean that token

j will attend to token

i (i.e.

{\displaystyle q_{j}\cdot k_{i}} could be small). The output of the attention unit for token

i is the weighted sum of the value vectors of all tokens, weighted by

a_{ij}, the attention from token

i to each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices

Q,

K and

V are defined as the matrices where the

ith rows are vectors

q_{i},

k_{i}, and

v_{i} respectively. Then we can represent the attention as

{\displaystyle {\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\end{aligned}}}

where softmax is taken over the horizontal axis.

Multi-head attention[edit]

One set of

{\displaystyle \left(W_{Q},W_{K},W_{V}\right)} matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". In addition, the influence field representing relevance can become progressively dilated in successive layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects.[37] The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feed-forward neural network layers.

i, then we have

{\displaystyle {\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}}

where the matrix

X is the concatenation of word embeddings, and the matrices

{\displaystyle W_{i}^{Q},W_{i}^{K},W_{i}^{V}} are "projection matrices" owned by individual attention head

i, and

{\displaystyle W^{O}} is a final projection matrix owned by the whole multi-headed attention head.

Masked attention[edit]

M that is negative infinity at entries where the attention link must be cut, and zero at other places.

Encoder[edit]

Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weights their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders.

The first encoder takes positional information and embeddings of the input sequence as its input, rather than encodings. The positional information is necessary for the transformer to make use of the order of the sequence, because no other part of the transformer makes use of this.[33]

The encoder is bidirectional. Attention can be placed on tokens before and after the current token. Tokens are used instead of words to account for polysemy.

sinusoidal positional encoding with parameters

{\displaystyle N=10000,d=100}

Positional encoding[edit]

A positional encoding is a fixed-size vector representation that encapsulates the relative positions of tokens within a target sequence: it provides the transformer model with information about where the words are in the input sequence.

The positional encoding is defined as a function of type

{\displaystyle f:\mathbb {R} \to \mathbb {R} ^{d};d\in \mathbb {Z} ,d>0}, where

d is a positive even integer. The full positional encoding – as defined in the original paper – is given by the equation:

{\displaystyle (f(t)_{2k},f(t)_{2k+1})=(\sin(\theta ),\cos(\theta ))\quad \forall k\in \{0,1,\ldots ,d/2-1\}}

where

{\displaystyle \theta ={\frac {t}{r^{k}}},r=N^{2/d}}.

Here,

N is a free parameter that should be significantly larger than the biggest

k that would be input into the positional encoding function. In the original paper,[33] the authors chose

{\displaystyle N=10000}.

{\displaystyle f:\mathbb {R} \to \mathbb {C} ^{d/2}}

{\displaystyle f(t)=\left(e^{it/r^{k}}\right)_{k=0,1,\ldots ,{\frac {d}{2}}-1}}

where

{\displaystyle r=N^{2/d}}.

The main reason the authors chose this as the positional encoding function is that it allows one to perform shifts as linear transformations:

{\displaystyle f(t+\Delta t)=\mathrm {diag} (f(\Delta t))f(t)}

where

{\displaystyle \Delta t\in \mathbb {R} } is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations:

{\displaystyle \sum _{j}c_{j}f(t+\Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (f(\Delta t_{j}))\right)f(t)}

for any constants

c_{j}. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position".

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Decoder[edit]

Each decoder consists of three major components: a self-attention mechanism, an attention mechanism over the encodings, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder-decoder attention.[33][36]

Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow.[33] This allows for autoregressive text generation. For all attention heads, attention can't be placed on following tokens. The last decoder is followed by a final linear transformation and softmax layer, to produce the output probabilities over the vocabulary.

All members of OpenAI's GPT series have a decoder-only architecture.

Terminology[edit]

In large language model, the terminology is somewhat different than the terminology used in the original Transformer paper:[38]

- "encoder only": full encoder, full decoder.

- "encoder-decoder": full encoder, autoregressive decoder.

- "decoder only": autoregressive encoder, autoregressive decoder.

Here "autoregressive" means that a mask is inserted in the attention head to zero out all attention from one token to all tokens following it, as described in the "masked attention" section.

Subsequent work[edit]

Alternative functions[edit]

The original transformer uses ReLU activation function. Other activation functions were developed, such as SwiGLU.[39]

Alternative positional encodings[edit]

Transformers may use other positional encoding methods than sinusoidal.[40]

RoPE (rotary positional embedding),[41] is best explained by considering a list of 2-dimensional vectors

{\displaystyle [(x_{1}^{(1)},x_{1}^{(2)}),(x_{2}^{(1)},x_{2}^{(2)}),(x_{3}^{(1)},x_{3}^{(2)}),...]}. Now pick some angle

\theta . Then RoPE encoding is

{\displaystyle {\text{RoPE}}{\big (}x_{m}^{(1)},x_{m}^{(2)},m{\big )}={\begin{pmatrix}\cos m\theta &-\sin m\theta \\\sin m\theta &\cos m\theta \end{pmatrix}}{\begin{pmatrix}x_{m}^{(1)}\\x_{m}^{(2)}\\\end{pmatrix}}={\begin{pmatrix}x_{m}^{(1)}\cos m\theta -x_{m}^{(2)}\sin m\theta \\x_{m}^{(2)}\cos m\theta +x_{m}^{(1)}\sin m\theta \\\end{pmatrix}}}

For a list of

2n-dimensional vectors, a RoPE encoder is defined by a sequence of angles

{\displaystyle \theta ^{(1)},...,\theta ^{(n)}}. Then the RoPE encoding is applied to each pair of coordinates.

The benefit of RoPE is that the dot-product between two vectors depends on their relative location only:

{\displaystyle {\text{RoPE}}{\big (}x,m{\big )}^{T}{\text{RoPE}}{\big (}y,n{\big )}={\text{RoPE}}{\big (}x,m+k{\big )}^{T}{\text{RoPE}}{\big (}y,n+k{\big )}}

for any integer

k.

ALiBi (Attention with Linear Biases)[42] is not a replacement for the positional encoder on the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is

{\displaystyle {\begin{aligned}{\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}+sB\right)V\end{aligned}}}

Here,

s is a real number ("scalar"), and

B is the linear bias matrix defined by

{\displaystyle B={\begin{pmatrix}0&1&2&3&\cdots \\-1&0&1&2&\cdots \\-2&-1&0&1&\cdots \\-3&-2&-1&0&\cdots \\\vdots &\vdots &\vdots &\vdots &\ddots \\\end{pmatrix}}}

in other words,

{\displaystyle B_{i,j}=j-i}.

ALiBi allows pretraining on short context windows, then finetuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

Efficient implementation[edit]

FlashAttention[edit]

FlashAttention[43] is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It performs matrix multiplications in blocks, such that each block fits within the cache of a GPU, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow).

An improved version, FlashAttention-2,[44][45][46] was developed to cater to the rising demand for language models capable of handling longer context lengths. It offers enhancements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs (FP16/BF16), a 2x speed increase over the original FlashAttention.

Key advancements in FlashAttention-2 include the reduction of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head dimensions up to 256 and multi-query attention (MQA) and grouped-query attention (GQA).

Benchmarks revealed FlashAttention-2 to be up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware like H100 GPUs and new data types like FP8.

Multi-Query Attention[edit]

Multi-Query Attention changes the multiheaded attention mechanism.[47] Whereas normally,

{\displaystyle {\text{MultiheadedAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}({\text{Attention}}(XW_{i}^{Q},XW_{i}^{K},XW_{i}^{V}))W^{O}}

with Multi-Query Attention, there is just one

{\displaystyle W^{K},W^{V}}, thus:

{\displaystyle {\text{MultiQueryAttention}}(Q,K,V)={\text{Concat}}_{i\in [\#heads]}({\text{Attention}}(XW_{i}^{Q},XW^{K},XW^{V}))W^{O}}

This has a neutral effect on model quality and training speed, but increases inference speed.

Sub-quadratic transformers[edit]

Training transformer-based architectures can be expensive, especially for long inputs.[48] Alternative architectures include the Reformer (which reduces the computational load from

O(N^{2}) to

O(N\ln N)[48]), or models like ETC/BigBird (which can reduce it to

O(N))[49] where

N is the length of the sequence. This is done using locality-sensitive hashing and reversible layers.[50][51]

Ordinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers[52] reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.

Long Range Arena (2020)[53] is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

Random Feature Attention (2021)[54] uses Fourier random features:

{\displaystyle \varphi (x)={\frac {1}{\sqrt {D}}}[\cos \langle w_{1},x\rangle ,\sin \langle w_{1},x\rangle ,\cdots \cos \langle w_{D},x\rangle ,\sin \langle w_{D},x\rangle ]^{T}}

where

{\displaystyle w_{1},...,w_{D}} are independent samples from the normal distribution

{\displaystyle N(0,\sigma ^{2}I)}. This choice of parameters satisfy

{\displaystyle \mathbb {E} [\langle \varphi (x),\varphi (y)\rangle ]=e^{\frac {\|x-y\|^{2}}{2\sigma ^{2}}}}, or

{\displaystyle e^{\langle x,y\rangle /\sigma ^{2}}=\mathbb {E} [\langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle ]\approx \langle e^{\|x\|^{2}/2\sigma ^{2}}\varphi (x),e^{\|y\|^{2}/2\sigma ^{2}}\varphi (y)\rangle }

Consequently, the one-headed attention, with one query, can be written as

{\displaystyle {\text{Attention}}(q,K,V)={\text{softmax}}\left({\frac {qK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx {\frac {\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})v_{i}^{T}}{\varphi (q)^{T}\sum _{i}e^{\|k_{i}\|^{2}/2\sigma ^{2}}\varphi (k_{i})}}}

where

{\displaystyle \sigma =d_{K}^{1/4}}. Similarly for multiple queries, and for multiheaded attention.

{\displaystyle \varphi (k_{i})v_{i}^{T}} first, then multiply it with the query. In essence, we have managed to obtain a more precise version of

{\displaystyle {\text{Attention}}(Q,K,V)={\text{softmax}}\left({\frac {QK^{\mathrm {T} }}{\sqrt {d_{k}}}}\right)V\approx Q(K^{T}V/{\sqrt {d_{k}}})}

Performer (2022)[55] uses the same Random Feature Attention, but

{\displaystyle w_{1},...,w_{D}} are first independently sampled from the normal distribution

{\displaystyle N(0,\sigma ^{2}I)}, then they are Gram-Schmidt processed.

Multimodality[edit]

Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.

Vision transformers[27] adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like tokens in a standard transformer.

Conformer[28] and later Whisper[56] follow the same pattern for speech recognition, first turning the speech signal into a spectrogram, which is then treated like an image, i.e. broken down into a series of patches, turned into vectors and treated like tokens in a standard transformer.

Perceivers by Andrew Jaegle et al. (2021)[57][58] can learn from large amounts of heterogeneous data.

Regarding image outputs, Peebles et al introduced a diffusion transformer (DiT) which facilitates use of the transformer architecture for diffusion-based image production.[59] Also, Google released a transformer-centric image generator called "Muse" based on parallel decoding and masked generative transformer technology.[60] (Transformers played a less-central role with prior image-producing technologies,[61] albeit still a significant one.[62])

See also[edit]

- Perceiver – Machine learning algorithm for non-textual data

- BERT (language model) – Language model developed by Google

- GPT-3 – 2020 large language model

- GPT-4 – 2023 text-generating language model

- ChatGPT – AI chatbot developed by OpenAI

- Wu Dao – Chinese multimodal artificial intelligence program

- Vision transformer – Machine learning algorithm for vision processing

- BLOOM (language model) – Open-access multilingual language model

Jürgen Schmidhuber

10 languages

- Article

- Talk

- Read

- Edit

- View history

Tools

From Wikipedia, the free encyclopedia

Jürgen Schmidhuber

Schmidhuber speaking at the AI for GOOD Global Summit in 2017

Born

17 January 1963[1]

Munich,[1] West Germany

Alma mater

Technical University of Munich

Known for

Long short-term memory, Gödel machine, artificial curiosity, meta-learning

Scientific career

Fields

Artificial intelligence

Institutions

Dalle Molle Institute for Artificial Intelligence Research

Website

people

Jürgen Schmidhuber (born 17 January 1963)[1] is a German computer scientist noted for his work in the field of artificial intelligence, specifically artificial neural networks. He is a scientific director of the Dalle Molle Institute for Artificial Intelligence Research in Switzerland.[2] He is also director of the Artificial Intelligence Initiative and professor of the Computer Science program in the Computer, Electrical, and Mathematical Sciences and Engineering (CEMSE) division at the King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.[3]

He is best known for his foundational and highly-cited[4] work on long short-term memory (LSTM), a type of neural network architecture which went on to become the dominant technique for various natural language processing tasks in research and commercial applications in the 2010s.

Career[edit]

Schmidhuber completed his undergraduate (1987) and PhD (1991) studies at the Technical University of Munich in Munich, Germany.[1] His PhD advisors were Wilfried Brauer and Klaus Schulten.[5] He taught there from 2004 until 2009. From 2009,[6] until 2021, he was a professor of artificial intelligence at the Università della Svizzera Italiana in Lugano, Switzerland.[1]

He has served as the director of Dalle Molle Institute for Artificial Intelligence Research (IDSIA), a Swiss AI lab, since 1995.[1]

In 2014, Schmidhuber formed a company, Nnaisense, to work on commercial applications of artificial intelligence in fields such as finance, heavy industry and self-driving cars. Sepp Hochreiter, Jaan Tallinn, and Marcus Hutter are advisers to the company.[2] Sales were under US$11 million in 2016; however, Schmidhuber states that the current emphasis is on research and not revenue. Nnaisense raised its first round of capital funding in January 2017. Schmidhuber's overall goal is to create an all-purpose AI by training a single AI in sequence on a variety of narrow tasks.[7]

Research[edit]

This section relies excessively on references to primary sources. Please improve this section by adding secondary or tertiary sources.

Find sources: "Jürgen Schmidhuber" – news · newspapers · books · scholar · JSTOR (March 2023) (Learn how and when to remove this template message)

In the 1980s, backpropagation did not work well for deep learning with long credit assignment paths in artificial neural networks. To overcome this problem, Schmidhuber (1991) proposed a hierarchy of recurrent neural networks (RNNs) pre-trained one level at a time by self-supervised learning.[8] It uses predictive coding to learn internal representations at multiple self-organizing time scales. This can substantially facilitate downstream deep learning. The RNN hierarchy can be collapsed into a single RNN, by distilling a higher level chunker network into a lower level automatizer network.[8][9] In 1993, a chunker solved a deep learning task whose depth exceeded 1000.[10]

In 1991, Schmidhuber published adversarial neural networks that contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss.[11][12][13] The first network is a generative model that models a probability distribution over output patterns. The second network learns by gradient descent to predict the reactions of the environment to these patterns. This was called "artificial curiosity." In 2014, this principle was used in a generative adversarial network where the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. This can be used to create realistic deepfakes.

Schmidhuber supervised the 1991 diploma thesis of his student Sepp Hochreiter[14] and called it "one of the most important documents in the history of machine learning".[9] It not only tested the neural history compressor,[8] but also analyzed and overcame the vanishing gradient problem. This led to the deep learning method called long short-term memory (LSTM), a type of recurrent neural network. The name LSTM was introduced in a tech report (1995) leading to the most cited LSTM publication (1997), co-authored by Hochreiter and Schmidhuber.[15] The standard LSTM architecture which is used in almost all current applications was introduced in 2000 by Felix Gers, Schmidhuber, and Fred Cummins.[16] Today's "vanilla LSTM" using backpropagation through time was published with his student Alex Graves in 2005,[17][18] and its connectionist temporal classification (CTC) training algorithm[19] in 2006. CTC enabled end-to-end speech recognition with LSTM. By the 2010s, the LSTM became the dominant technique for a variety of natural language processing tasks including speech recognition and machine translation, and was widely implemented in commercial technologies such as Google Translate and Siri.[20] LSTM has become the most cited neural network of the 20th century.[9]

In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used LSTM principles to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks.[21][22] 7 months later, the ImageNet 2015 competition was won with an open-gated or gateless Highway network variant called Residual neural network.[23] This has become the most cited neural network of the 21st century.[9]

Since 2018, transformers have overtaken the LSTM as the dominant neural network architecture in natural language processing[24] through large language models such as ChatGPT. As early as 1992, Schmidhuber published an alternative to recurrent neural networks[25] which is now called a Transformer with linearized self-attention[26][27][9] (save for a normalization operator). It learns internal spotlights of attention:[28] a slow feedforward neural network learns by gradient descent to control the fast weights of another neural network through outer products of self-generated activation patterns FROM and TO (which are now called key and value for self-attention).[26] This fast weight attention mapping is applied to a query pattern.

In 2011, Schmidhuber's team at IDSIA with his postdoc Dan Ciresan also achieved dramatic speedups of convolutional neural networks (CNNs) on fast parallel computers called GPUs. An earlier CNN on GPU by Chellapilla et al. (2006) was 4 times faster than an equivalent implementation on CPU.[29] The deep CNN of Dan Ciresan et al. (2011) at IDSIA was already 60 times faster[30] and achieved the first superhuman performance in a computer vision contest in August 2011.[31] Between 15 May 2011 and 10 September 2012, their fast and deep CNNs won no fewer than four image competitions.[32][33] They also significantly improved on the best performance in the literature for multiple image databases.[34] The approach has become central to the field of computer vision.[33] It is based on CNN designs introduced much earlier by Yann LeCun et al. (1989)[35] who applied the backpropagation algorithm to a variant of Kunihiko Fukushima's original CNN architecture called neocognitron,[36] later modified by J. Weng's method called max-pooling.[37][33]

Credit disputes[edit]

Schmidhuber has controversially argued that he and other researchers have been denied adequate recognition for their contribution to the field of deep learning, in favour of Geoffrey Hinton, Yoshua Bengio and Yann LeCun, who shared the 2018 Turing Award for their work in deep learning.[2][20][38] He wrote a "scathing" 2015 article arguing that Hinton, Bengio and Lecun "heavily cite each other" but "fail to credit the pioneers of the field".[38] In a statement to the New York Times, Yann LeCun wrote that "Jürgen is manically obsessed with recognition and keeps claiming credit he doesn't deserve for many, many things... It causes him to systematically stand up at the end of every talk and claim credit for what was just presented, generally not in a justified manner."[2] Schmidhuber replied that LeCun did this "without any justification, without providing a single example,"[39] and published details of numerous priority disputes with Hinton, Bengio and LeCun.[40] Kory Mathewson suggested that Schmidhuber's accomplishments have been downplayed because of his personality.[20]

Recognition[edit]

Schmidhuber received the Helmholtz Award of the International Neural Network Society in 2013,[41] and the Neural Networks Pioneer Award of the IEEE Computational Intelligence Society in 2016[42] for "pioneering contributions to deep learning and neural networks."[1] He is a member of the European Academy of Sciences and Arts.[43][6]

He has been referred to as the "father of (modern) AI" or similar,[44][2][45][46][47][48][49][50][51][52][53][20] and also the "father of deep learning."[54][47] Schmidhuber himself, however, has called Alexey Grigorevich Ivakhnenko the "father of deep learning,"[55] and gives credit to many even earlier AI pioneers.[9]

Views[edit]

This section contains too many quotations for an encyclopedic entry. Please help improve the article by presenting facts as a neutrally worded summary with appropriate citations. Consider transferring direct quotations to Wikiquote or, for entire works, to Wikisource. (June 2023)

Schmidhuber states that "in 95% of all cases, AI research is really about our old motto, which is make human lives longer and healthier and easier."[51] He admits that "the same tools that are now being used to improve lives can be used by bad actors," but emphasizes that "they can also be used against the bad actors."[50]

He does not believe AI poses a "new quality of existential threat," and is more worried about the old nuclear warheads which can "wipe out human civilization within two hours, without any AI."[44] "A large nuclear warhead doesn’t need fancy face recognition to kill an individual. No, it simply wipes out an entire city with 10 million inhabitants."[44]

Since the 1970s, Schmidhuber wanted to create "intelligent machines that could learn and improve on their own and become smarter than him within his lifetime."[44] He differentiates between two types of AIs: AI tools directed by humans, in particular for improving healthcare, and more interesting AIs that "are setting their own goals," inventing their own experiments and learning from them, like curious scientists. He has worked on both types for decades,[44] and has predicted that scaled-up versions of AI scientists will eventually "go where most of the physical resources are, to build more and bigger AIs." Within "a few tens of billions of years, curious self-improving AIs will colonize the visible cosmos in a way that’s infeasible for humans. Those who don’t won’t have an impact."[44] He said: "don’t think of humans as the crown of creation. Instead, view human civilization as part of a much grander scheme, an important step (but not the last one) on the path of the universe from very simple initial conditions toward more and more unfathomable complexity. Now it seems ready to take its next step, a step comparable to the invention of life itself over 3.5 billion years ago."[44]

He strongly supports the open-source movement, and thinks it is going to "challenge whatever big-tech dominance there might be at the moment," also because AI keeps getting 100 times cheaper per decade.[44]