# Clone the repository and navigate to it
cd art_hug_04


# Run the setup task (installs Python 3.12.9 via pyenv and all dependencies)
task setup


# Run all examples
task run


# Run specific examples

# Demonstrates self-attention and multi-head attention
task run-attention-mechanism  

# Shows encoder-only, decoder-only, and encoder-decoder architectures
task run-modern-models        

def basic_tokenization_and_embedding():
    """Basic tokenization and embedding example from the article."""
    print_subsection("Basic Tokenization and Embedding")

    # 1. Choose a recent, efficient pre-trained model
    model_name = "roberta-base"  # Robust optimization of BERT
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name)

    # 2. Tokenize your input sentence
    sentence = "Transformers are amazing!"
    inputs = tokenizer(sentence, return_tensors="pt")
    print("Token IDs:", inputs["input_ids"])
    # Example output: tensor([[0, 44929, 32, 2770, 328, 2]])

    # 3. Pass tokens through the model to get embeddings
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state
        print("Embeddings shape:", embeddings.shape)
        # Example output: torch.Size([1, 6, 768])

    # 4. Convert token IDs back to tokens for visualization
    tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
    print(f"Tokens: {tokens}")
    # Output: ['<s>', 'Transform', 'ers', 'Ġare', 'Ġamazing', '!', '</s>']

```**What's happening here:**1.**Model Loading**: RoBERTa-base is loaded with its matching tokenizer
2.**Tokenization**: The sentence is split into subword tokens (notice "Transformers" becomes "Transform" + "ers")
3.**Special Tokens**: `<s>` and `</s>` mark the beginning and end of the sequence
4.**Embeddings**: Each token gets a 768-dimensional vector representation
5.**Context-Aware**: These embeddings already contain contextual information from the pre-trained model

The code serving as the practical demonstration of the first step in transformer processing. The function shows the foundational process that powers all transformer models - converting human language into machine-understandable numerical representations.

What this code demonstrates:

1.**Initialization**: It loads a modern pre-trained transformer model (RoBERTa) and its matching tokenizer
2.**Tokenization Process**: Shows how the sentence "Transformers are amazing!" gets broken into subword tokens, revealing how words like "Transformers" split into "Transform" + "ers"
3.**Embedding Generation**: Demonstrates how tokens are converted into rich 768-dimensional vectors that capture semantic meaning
4.**Visualization**: Converts token IDs back to readable tokens to show how the model internally represents text**Foundation for Understanding**: This code establishes the entry point for text into transformer models, showing how raw language becomes the numerical data that all later transformer operations work with**Contextual Bridge**: These embeddings form the input to the self-attention mechanisms explored in the next sections, connecting the dots between text input and contextual understanding**Practical Implementation**: It provides readers with executable code they can run immediately, making abstract concepts tangible**Modern Approach**: By using RoBERTa, it demonstrates current best practices rather than just theoretical concepts

The embeddings generated here become the raw material for all the sophisticated transformer operations covered later in the article. Self-attention mechanisms, which we'll explore next, operate on these very embeddings to understand relationships between words. This code bridges theory and practice, showing exactly how text enters the transformer ecosystem.

To run this example:

```bash
task run-attention-mechanism

```**Step-by-Step Breakdown:**1.**Model Selection**: Specify `'roberta-base'` for efficient, modern performance
2.**Tokenization**: The tokenizer splits text into tokens and assigns unique IDs
3.**Embedding Generation**: The model processes IDs and outputs `last_hidden_state`—embeddings for each token
4.**Shape Interpretation**: `(1, 6, 768)` means 1 sentence, 6 tokens, 768-dimensional embeddings

Try the code yourself. Watch raw text transform into numbers, then into context-rich vectors.**Key Takeaway:**This first step turns text into something models "understand." Next, we'll explore how attention lets transformers relate words and context—unlocking their true power.

For production efficiency, consider distilled models (`distilroberta-base`) or quantization techniques, both supported in Hugging Face.


## Looking Ahead

Next section breaks down transformer building blocks—tokens, embeddings, and positional encodings. We'll also introduce recent trends like multimodal transformers and efficient deployment. For transformer history and business impact, see Article 1. For tokenization deep-dive, jump to Article 5.

Ready to transform from passive user to confident architect? Let's dive in.


# Key Building Blocks of Transformers

Master transformers by understanding their core ingredients. Like essential spices in a chef's kitchen, these building blocks appear in every recipe. We'll break down each component step-by-step, with practical code and real-world examples, highlighting recent architectural advances.**Note:**The implementations for these concepts can be found in:

- `src/attention_mechanism.py` - Self-attention and multi-head attention demonstrations
- `src/positional_encoding.py` - Positional encoding implementations
- `src/transformer_blocks.py` - Complete transformer block examples
- `src/model_analysis.py` - Analysis and visualization utilities

```mermaid
stateDiagram-v2
  [*] --> Tokenization
  Tokenization --> Embedding: Convert to IDs
  Embedding --> PositionalEncoding: Add vectors
  PositionalEncoding --> Attention: Add position info
  Attention --> Normalization: Context mixing
  Normalization --> FeedForward: Stabilize
  FeedForward --> Output: Transform
  Output --> [*]

  style Tokenization fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
  style Embedding fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
  style PositionalEncoding fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
  style Attention fill:#c8e6c9,stroke:#43a047,stroke-width:1px,color:#333333
  style Normalization fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
  style FeedForward fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
  style Output fill:#c8e6c9,stroke:#43a047,stroke-width:1px,color:#333333

```\n\n**Step-by-Step Explanation:**\n- **Start**: Raw text enters the transformer pipeline
- **Tokenization**: Text splits into processable tokens
- **Embedding**: Tokens convert to high-dimensional vectors
- **Positional Encoding**: Position information adds to embeddings
- **Attention**: Self-attention mixes contextual information
-**Normalization**: Layer normalization stabilizes values
-**Feed-Forward**: Neural network transforms representations
-**Output**: Final contextualized representations emerge


## Tokens, Embeddings, and Position

Transformers only understand numbers, not raw text. The journey from sentence to tensor involves three crucial steps:

1.**Tokenization:**Break text into pieces called tokens. "Transformers are amazing!" becomes `['transform', 'ers', 'are', 'amazing', '!']`
2.**Embedding:**Map each token to a high-dimensional vector—a unique fingerprint learned from massive datasets
3.**Positional Encoding:**Add position information since transformers don't know word order by default—like giving each token a seat number


### Tokenizing and Embedding a Sentence

Now that we've covered the fundamental building blocks of transformers, let's examine how to implement tokenization and embedding in practice. The following code example demonstrates these concepts with real Python code that you can run and experiment with:

```python
from transformers import AutoTokenizer, AutoModel
import torch

sentence = "Transformers are amazing!"
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
inputs = tokenizer(sentence, return_tensors='pt')
print('Token IDs:', inputs['input_ids'])

model = AutoModel.from_pretrained('bert-base-uncased')
with torch.no_grad():
    outputs = model(**inputs)
    print('Embeddings shape:', outputs.last_hidden_state.shape)

```**What's happening:**- Tokenizer splits the sentence and converts tokens to IDs
- Model transforms IDs into embeddings (shape `[1, 6, 768]`: 1 batch, 6 tokens, 768 features per token)
- Positional encodings are handled automatically in Hugging Face models

Try your own sentence to see different tokenizations!**Note:**Classic transformers use absolute positional encodings (sinusoidal or learned). Many modern models—**Llama**,**DeBERTa**,**Mistral**,**DeepSeek**—use relative or rotary positional encodings (RoPE). These approaches boost generalization and performance on longer sequences. Hugging Face models handle this automatically.


### Understanding Rotary Positional Embedding (RoPE)

Rotary Positional Embedding (RoPE) represents a significant advancement in how transformers handle position information. Unlike traditional absolute positional encodings, RoPE applies a rotation transformation to the token embeddings that elegantly encodes relative position information.**Key advantages of RoPE:**-**Better generalization:**RoPE helps models generalize to sequence lengths beyond what they were trained on
-**Relative position awareness:**Captures relative distances between tokens more effectively than absolute positions
-**Mathematical elegance:**Uses rotation matrices in complex space to encode position while preserving vector norms
-**Improved long-range dependency learning:**Enhances the model's ability to connect information across distant parts of a sequence

RoPE has become the positional encoding method of choice for many modern transformer architectures including Llama, Mistral, and DeepSeek. It's particularly valuable for models intended to process variable or long sequences.

In implementation, RoPE rotates the Query and Key vectors in each attention head by angles dependent on token positions. This rotation causes dot-products between vectors to naturally encode their relative positions, all while maintaining the core self-attention mechanism's structure.

Together, tokenization, embedding, and positional encoding transform unstructured text into structured data. This unlocks classification, sentiment analysis, and information extraction.

Want more on tokenization? See Article 5 for tokenizer types and customization.


## Normalization and Residuals

Deep networks pack power but can lose or distort information across layers. Transformers use two tricks to keep learning on track:

-**Layer normalization:**Rescales layer outputs to zero mean and unit variance—keeping all orchestra instruments at the same volume
-**Residual connections:**Shortcut connections add input directly to output, letting information flow around obstacles and preventing vanishing gradients


### Residual Connection Example

Now let's see how self-attention works in code. The following implementation demonstrates the mathematics behind attention in a clear, step-by-step manner:

```python
import torch.nn as nn

class SimpleTransformerBlock(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.linear = nn.Linear(d_model, d_model)
        self.norm = nn.LayerNorm(d_model)
    def forward(self, x):
        return self.norm(x + self.linear(x))

```**How it works:**- Input `x` passes through a linear layer
- Output adds back to `x` (residual connection)
- LayerNorm keeps everything stable

This pattern repeats in every transformer layer, making deep models reliable and trainable.

Layer normalization remains standard in most transformers. Some large models (like**Llama 2**) use**RMSNorm**for efficiency and stability in very deep networks. Both are supported in Hugging Face and handled automatically.

In business, these features enable models to handle complex tasks—analyzing thousands of documents or running large-scale chatbots—without breaking down.

This code snippet demonstrates a simplified transformer block that implements the residual connection and normalization techniques described above. It's a foundational building block used in transformer architectures that helps maintain stable training in deep networks. The example shows how the input `x` is processed through a linear layer, then added back to the original input (residual connection), and finally normalized using LayerNorm. This pattern enables information to flow smoothly through the network while preventing gradient issues in deep architectures.

For neural network basics and vanishing gradient problems, see Article 2.


## Feed-Forward Networks

After attention, each token passes through its own mini neural network—the feed-forward block. Giving each word its own chef to add special seasoning after the whole meal has been tasted exemplifies this process.

Classic feed-forward networks have two linear layers with non-linear activation (like ReLU) between. This captures subtle, complex language patterns.


### Feed-Forward Network Block

Let's implement a feed-forward network in practice. The following code demonstrates how to build a standard FFN block as used in transformers:

```python
import torch.nn as nn

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model)
        )
    def forward(self, x):
        return self.net(x)

```**What happens:**- First linear layer expands dimension (e.g., 768 → 3072)
- ReLU adds non-linearity
- Second linear layer projects back to original size
- Every token processes independently—fast and scalable

Recent models use advanced variants—**Gated Linear Units (GLU)**,**SwiGLU**, or**Mixture-of-Experts (MoE)**layers. These boost expressiveness and efficiency in large models. Hugging Face handles the FFN choice based on your selected architecture.

This helps models distinguish praise from complaint in customer support, even with subtle language.

Stacking these blocks enables transformers to learn rich representations for tasks from entity extraction to creative generation.


## Recap & Next Steps

Let's review:

-**Tokenization**,**embedding**, and**positional encoding**convert text to structured tensors
-**Layer normalization**and**residuals**stabilize learning for deep models
-**Feed-forward networks**add expressive power for real-world tasks
- Modern models use relative/rotary positional encodings, advanced FFN variants, and sometimes RMSNorm for efficiency

You'll see these blocks repeatedly—whether fine-tuning models or building chatbots. For tokenization details, see Article 5. For neural network fundamentals, see Article 2. Ready for more? Next, we'll demystify attention—the transformer's secret sauce.


# Self-Attention and Multi-Head Attention Explained

```mermaid
flowchart TB
    subgraph "Self-Attention Mechanism"
        Input[Input Tokens]
        Q[Query Vectors]
        K[Key Vectors]
        V[Value Vectors]
        Scores[Attention Scores]
        Weights[Attention Weights]
        Output[Context-Aware Output]

        Input --> Q
        Input --> K
        Input --> V
        Q --> Scores
        K --> Scores
        Scores -->|Softmax| Weights
        Weights --> Output
        V --> Output
    end

    classDef default fill:#bbdefb,stroke:#1976d2,stroke-width:1px,color:#333333
    class Input,Q,K,V,Scores,Weights,Output,MH_Input,Head1,Head2,HeadN,Concat,MH_Output default

# From src/attention_mechanism.py
def demonstrate_self_attention():
    """Demonstrate self-attention calculation from the article."""
    print_subsection("Self-Attention Calculation")

    # Example dimensions for demonstration
    d_model = 64   # Model dimension (smaller for visualization)
    seq_len = 5    # Sequence length
    batch_size = 2 # Process 2 sequences at once

    # Generate random Query, Key, Value tensors
    # In real transformers, these come from linear projections
    q = torch.randn(batch_size, seq_len, d_model)
    k = torch.randn(batch_size, seq_len, d_model)
    v = torch.randn(batch_size, seq_len, d_model)

    # Step 1: Compute attention scores
    # Q @ K^T gives us a score for each query-key pair
    attn_scores = torch.matmul(q, k.transpose(-2, -1)) / (d_model**0.5)
    print(f"Attention scores shape: {attn_scores.shape}")
    # Output: torch.Size([2, 5, 5]) - each token attends to all others

    # Step 2: Apply softmax to get attention weights
    # This normalizes scores to probabilities that sum to 1
    attn_weights = torch.softmax(attn_scores, dim=-1)
    print(f"Attention weights shape: {attn_weights.shape}")

    # Step 3: Apply attention weights to values
    # This creates a weighted combination of value vectors
    output = torch.matmul(attn_weights, v)
    print(f"Output shape: {output.shape}")
    # Output: torch.Size([2, 5, 64]) - same shape as input

    # Visualize attention weights for first sequence
    plt.figure(figsize=(6, 5))
    sns.heatmap(attn_weights[0].numpy(), annot=True, fmt=".2f", cmap="Blues")
    plt.xlabel("Key Position")
    plt.ylabel("Query Position")
    plt.title("Self-Attention Weights")
    plt.show()

```**Step-by-step breakdown:**1.**Token Projection**: Each embedding projects to Q, K, V vectors (learned parameters in real models)
2.**Score Calculation**: Attention scores measure focus between tokens (scaled dot product for stability)
3.**Weight Normalization**: Softmax ensures each row sums to 1 (probabilities)
4.**Context Mixing**: Weighted sum creates new, context-rich token representations

Self-attention enables every token to gather context from the entire sequence—crucial for ambiguity and long-range dependencies.**Key Takeaways:**- Self-attention lets tokens focus on what's most relevant
- Query, Key, and Value vectors drive the mechanism
- Dot product and softmax are core attention operations
- Batch processing is standard in production models**Try it yourself:**Modify batch size, sequence length, or inputs and observe attention weight shifts.

👉**Note:**Modern transformers automatically apply memory-efficient algorithms (like FlashAttention) when supported by hardware and configuration, making large-scale training practical.


## Multi-Head Attention: Parallelizing Understanding

Single self-attention heads focus on one relationship type at a time. But language is complex! Multi-head attention runs several self-attention mechanisms in parallel—each with unique learned projections.

Each "head" specializes: one tracks grammar, another meaning, another sentiment. Combined outputs give models richer, multi-faceted understanding.

Here's multi-head attention schematic pseudocode (batch processing like real models):


### Multi-Head Attention Schematic (Batched, Pseudocode)

Let's implement multi-head attention to see how it works in practice. The following code from our codebase demonstrates how multiple attention heads operate in parallel, each capturing different aspects of the input:

```python

# For illustration only
multi_head_outputs = []
for head in range(num_heads):
    # Each head projects Q, K, V differently (learned weights)
    Q_h, K_h, V_h = project(Q, head), project(K, head), project(V, head)
    attn_scores_h = torch.matmul(Q_h, K_h.transpose(-2, -1)) / (d_k**0.5)
    attn_weights_h = torch.softmax(attn_scores_h, dim=-1)
    attn_output_h = torch.matmul(attn_weights_h, V_h)
    multi_head_outputs.append(attn_output_h)


# Concatenate outputs from all heads along the last dimension
concatenated = torch.cat(multi_head_outputs, dim=-1)


# Final linear transformation combines all heads
output = final_linear(concatenated)

```**What happens:**- Each head uses unique Q, K, V projections (different learned weights)
- Each computes self-attention independently and in parallel
- Outputs concatenate and pass through final linear layer mixing all head information

Thinking of multi-head attention as an expert panel—each sees data differently—leads to stronger, nuanced decisions through combined advice.**Key Takeaways:**- Multi-head attention captures multiple parallel relationships
- Each head focuses on different patterns or features
- Parallelism enables transformers to scale with complex data
- Modern libraries implement this efficiently with hardware optimizations

This code example is crucial to the article's explanation of multi-head attention, one of the core innovations in transformer architecture. It demonstrates how transformers process information in parallel through multiple attention mechanisms simultaneously. Here's what this code is illustrating:**Purpose in the Article:**This pseudocode demonstrates the practical implementation of multi-head attention, showing how transformers can analyze relationships between words from multiple perspectives simultaneously—a key advantage over previous architectures.**What the Code Demonstrates:**-**Parallel Processing:**Each attention head operates independently and in parallel, analyzing different aspects of the input data
-**Unique Projections:**Each head has its own learned projection matrices for Query, Key, and Value vectors, allowing it to specialize in different linguistic patterns
-**Core Attention Steps:**The same attention mechanism (dot product, scaling, softmax, weighted sum) is applied in each head
-**Information Integration:**The outputs from all heads are concatenated and processed through a final linear layer to combine the different perspectives

This implementation shows why transformers are so powerful—they don't just look at data from one angle, but simultaneously analyze multiple relationship types (grammar, semantics, entity relationships, etc.) and combine these insights for richer understanding.

The code aligns with the article's focus on making transformer internals accessible, showing both the conceptual aspects of multi-head attention and how it's implemented in practice.


## Visualizing Attention for Interpretability

Attention isn't just mathematical—it's a window into model "thinking." Visualizing attention weights reveals which words influence predictions.


### Attention Masking Patterns

Here's how we visualize different attention masking patterns from `src/attention_mechanism.py`:

To further demystify attention mechanisms, let's examine a practical visualization example from our codebase. This implementation demonstrates how different attention masking patterns are used in various transformer architectures:

```python
def attention_masking_demo():
    """Show different attention masking patterns."""
    print_subsection("Attention Masking Patterns")

    seq_len = 6

    # Different mask types used in transformers
    masks = {
        "No Mask (Encoder)": torch.ones(seq_len, seq_len),
        "Causal Mask (Decoder)": torch.tril(torch.ones(seq_len, seq_len)),
        "Random Mask": (torch.rand(seq_len, seq_len) > 0.3).float(),
    }

    fig, axes = plt.subplots(1, 3, figsize=(15, 4))

    for idx, (name, mask) in enumerate(masks.items()):
        ax = axes[idx]
        sns.heatmap(
            mask.numpy(),
            ax=ax,
            cmap="Blues",
            cbar=False,
            xticklabels=False,
            yticklabels=False,
        )
        ax.set_title(name)

    plt.tight_layout()
    plt.show()

    print("Mask types explained:")
    print("- No Mask: All positions can attend to all others (BERT)")
    print("- Causal Mask: Can only attend to previous positions (GPT)")
    print("- Random Mask: Used in some training techniques")

```**Masking Patterns Explained:**1.**No Mask (Encoder)**: Used in BERT and other encoder models. Every token can see every other token, enabling bidirectional understanding.
2.**Causal Mask (Decoder)**: Used in GPT and other decoder models. Tokens can only attend to previous tokens, ensuring autoregressive generation.
3.**Random Mask**: Used during training for models like BERT, where random tokens are masked and the model learns to predict them.

Tools like [BertViz](https://github.com/jessevig/bertviz) remain popular for interactive exploration. Hugging Face now offers integrated explainability tools like [`transformers-interpret`](https://github.com/cdpierse/transformers-interpret) and built-in visualization in model cards and Spaces.

This attention masking demo code provides a visual representation of different attention masking patterns used in transformer models, which is crucial for understanding how various transformer architectures process information differently. Here's how it connects to the broader article:

The attention masking patterns directly illustrate one of the fundamental differences between encoder and decoder architectures discussed throughout the article:

-**Bidirectional vs. Unidirectional Understanding:**The visualization clearly shows why encoders like BERT can understand context from both directions while decoders like GPT are limited to previous tokens only.
-**Architectural Foundations:**These masking patterns explain the core design choices that determine whether a transformer is better suited for understanding (encoders) or generation (decoders).

This example bridges the theoretical explanation of attention mechanisms with their practical implementation differences across model types. By visualizing these patterns, readers can:

1.**Understand Model Limitations:**See why GPT models sometimes struggle with coherence across long passages (they can only see previous tokens)
2.**Grasp Architecture Choices:**Connect the masking patterns to the model capabilities described in the "Encoder, Decoder, and Hybrid Architectures" section
3.**Appreciate Design Tradeoffs:**Recognize why different masking approaches are chosen based on the intended task

The visualization provides an intuitive complement to the mathematical explanations of attention in previous sections, making the abstract concept concrete through visual representation. This aids comprehension by showing exactly how information flows differently in various transformer architectures.

While attention mechanisms and architecture variants provide the foundation of transformer models, their practical implementation requires understanding various optimizations and performance considerations. Let's examine how these powerful models can be efficiently implemented in real-world applications.


### Example: RAG for Enhanced Understanding

Retrieval-Augmented Generation (RAG) represents a significant advancement in transformer applications, combining the strengths of language models with external knowledge retrieval. This section demonstrates how transformers can be enhanced by grounding their outputs in factual information, addressing one of the key limitations of traditional transformer models - their tendency to hallucinate or generate plausible but incorrect information.

RAG systems work by retrieving relevant documents from a knowledge base in response to a query, then providing those documents as context to a language model for generating an answer. This approach offers several advantages:

- Improved factual accuracy through grounding in source documents
- Dynamic knowledge updates without model retraining
- Greater transparency with citations to source information
- Enhanced performance on specialized domains with custom knowledge bases

The following examples demonstrate both simple and advanced RAG implementations, highlighting the practical aspects of integrating retrieval mechanisms with transformer-based generation.

Here's the complete RAG implementation from `src/rag_example.py` showing both FAISS and simple versions:

To implement these concepts, let's examine the RAG implementation from our codebase. This practical example demonstrates how transformers can use external knowledge sources to enhance their reasoning capabilities:

```python
def simple_rag_without_faiss():
    """A simpler version of RAG without FAISS dependency."""
    print("=== Simple RAG Example (without FAISS) ===\\n")

    # Documents representing our knowledge base
    docs = [
        "Transformers use self-attention to process sequences.",
        "BERT is an encoder-only transformer model.",
        "GPT is a decoder-only transformer model.",
        "T5 is an encoder-decoder transformer model.",
    ]

    query = "What architecture is BERT?"
    print(f"Query: '{query}'")

    # Simple keyword-based retrieval (no embeddings)
    print("\\nRetrieved documents (keyword matching):")
    retrieved_docs = []
    for doc in docs:
        if "BERT" in doc or "encoder" in doc.lower():
            retrieved_docs.append(doc)
            print(f"- {doc}")

    # Generate answer with retrieved context
    context = " ".join(retrieved_docs)
    prompt = f"Context: {context}\\nQuestion: {query}\\nAnswer:"

    print("\\nPrompt for generation:")
    print(prompt)

    # Use GPT-2 to generate an answer based on context
    generator = pipeline("text-generation", model="gpt2", max_new_tokens=30)
    result = generator(prompt, max_length=80, pad_token_id=50256)

    print("\\nGenerated answer:")
    print(result[0]["generated_text"].split("Answer:")[-1].strip())

def rag_example():
    """
    Full RAG example using FAISS for efficient similarity search.
    This demonstrates state-of-the-art retrieval-augmented generation.
    """
    if not FAISS_AVAILABLE:
        print("FAISS is required. Install with: pip install faiss-cpu")
        return

    print("=== RAG (Retrieval-Augmented Generation) Example ===\\n")

    # Step 1: Create embeddings for documents
    print("1. Creating document embeddings...")
    encoder = pipeline(
        "feature-extraction", model="sentence-transformers/all-MiniLM-L6-v2"
    )
    docs = [
        "Transformers use self-attention.",
        "BERT is encoder-only.",
        "GPT is decoder-only.",
    ]

    # Generate embeddings for each document
    doc_embeddings = []
    for doc in docs:
        # Get embeddings and average across tokens
        embedding = encoder(doc)[0]  # [num_tokens, embedding_dim]
        avg_embedding = np.mean(embedding, axis=0)
        doc_embeddings.append(avg_embedding)

    doc_embeddings = np.array(doc_embeddings).astype("float32")
    print(f"   Document embeddings shape: {doc_embeddings.shape}")

    # Step 2: Build FAISS index for fast similarity search
    print("\\n2. Building FAISS index...")
    embedding_dim = doc_embeddings.shape[1]  # 384 for all-MiniLM-L6-v2
    index = faiss.IndexFlatL2(embedding_dim)
    index.add(doc_embeddings)
    print(f"   Index built with {index.ntotal} documents")

    # Step 3: Retrieve relevant documents for a query
    query = "What architecture is BERT?"
    print(f"\\n3. Query: '{query}'")

    # Get query embedding
    query_result = encoder(query)[0]
    query_embedding = np.mean(query_result, axis=0).reshape(1, -1).astype("float32")

    # Search for similar documents
    k = 2  # Number of documents to retrieve
    distances, indices = index.search(query_embedding, k)

    print(f"\\n4. Retrieved documents (top {k}):")
    retrieved_docs = []
    for i, (idx, dist) in enumerate(zip(indices[0], distances[0])):
        doc = docs[idx]
        retrieved_docs.append(doc)
        print(f"   {i+1}. {doc} (distance: {dist:.4f})")

    # Step 4: Generate answer with context
    print("\\n5. Generating answer with context...")
    generator = pipeline("text-generation", model="gpt2", max_new_tokens=50)
    context = " ".join(retrieved_docs)
    prompt = f"Context: {context}\\nQuestion: {query}\\nAnswer:"

    answer = generator(
        prompt, max_length=100, num_return_sequences=1, pad_token_id=50256
    )
    answer_text = answer[0]["generated_text"].split("Answer:")[-1].strip()

    print("\\n6. Generated answer:")
    print(f"   {answer_text}")

```**Key Concepts in RAG:**1.**Document Embeddings**: Convert text documents into dense vectors that capture semantic meaning
2.**Similarity Search**: Use FAISS (Facebook AI Similarity Search) for efficient nearest neighbor retrieval
3.**Context Injection**: Provide retrieved documents as context to the language model
4.**Grounded Generation**: The model generates answers based on retrieved facts, reducing hallucination**Benefits of RAG:**- Updates knowledge without retraining
- Reduces hallucination by grounding in facts
- Perfect for enterprise search and QA systems
- Scales to millions of documents with FAISS

To run the RAG examples:

```bash

# Install optional dependencies
pip install sentence-transformers faiss-cpu


# Run the examples
task run  # This will include RAG examples if dependencies are installed

from transformers import BertTokenizer, BertModel
from bertviz import head_view


# Ensure you have the latest compatible versions:

# pip install transformers>=4.40.0 bertviz>=1.4.0

model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

sentence = "The cat sat on the mat."
inputs = tokenizer.encode(sentence, return_tensors='pt')
outputs = model(inputs)
attentions = outputs.attentions
head_view(attentions, tokenizer.convert_ids_to_tokens(inputs[0]))

```**Step-by-step:**1. Load BERT with attention outputs enabled
2. Tokenize your sentence
3. Run model and collect attention weights
4. Visualize with `head_view`—see how tokens attend across layers and heads

Why visualize attention?

- Debug unexpected outputs
- Explain predictions to stakeholders
- Spot biases or model weaknesses

Seeing attention patterns builds trust and improves models.**Try it yourself:**Visualize attention on your sentences. Notice pattern changes with context or phrasing. Explore Hugging Face's `transformers-interpret` or integrated visualization widgets in Spaces.**Key Takeaways:**- Attention maps show model token connections
- Visualization helps interpret, debug, and explain behavior
- Multiple tools available, including BertViz and Hugging Face libraries

This code demonstrates attention visualization in transformer models, connecting to the article's focus on transformer architecture and attention mechanisms. The visualization serves key functions:**Connection to Architecture:**Shows attention patterns at the core of transformer models, providing a tangible view of self-attention mechanisms.**Practical Implementation:**Offers executable code showing how to load a pre-trained BERT model, access attention weights, and visualize them.**Explainability:**Supports "demystifying" transformers by making their internal workings observable, showing how attention heads focus on token relationships.**Theory-Application Bridge:**Connects theoretical explanations with practical applications, demonstrating how to inspect model behavior for debugging and improvement.**Educational Value:**Provides an intuitive way to grasp abstract attention concepts, aligning with the article's educational purpose.

This code example bridges theoretical transformer components and advanced applications like RAG systems, helping readers progress from basic concepts to complex solutions.


# Encoder, Decoder, and Hybrid Architectures in Practice

```mermaid
classDiagram
    class TransformerArchitecture {
        +process_input()
        +generate_output()
    }

    class EncoderOnly {
        +encode(text): embeddings
        +classify(): labels
        +extract_features(): vectors
        Examples: DeBERTaV3, E5, MPNet
    }

    class DecoderOnly {
        +generate(prompt): text
        +complete(): continuation
        +chat(): response
        Examples: Llama 3, Mistral, DeepSeek
    }

    class EncoderDecoder {
        +encode(input): representation
        +decode(representation): output
        +transform(): new_sequence
        Examples: T5, UL2, BART
    }

    class RAG {
        +retrieve(query): documents
        +generate(context): answer
        Examples: Llama 3 + FAISS
    }

    TransformerArchitecture <|-- EncoderOnly
    TransformerArchitecture <|-- DecoderOnly
    TransformerArchitecture <|-- EncoderDecoder
    TransformerArchitecture <|-- RAG

    RAG --> DecoderOnly : uses
    RAG --> EncoderOnly : uses

```**Step-by-Step Explanation:**-**TransformerArchitecture**: Base class for all transformer types
-**EncoderOnly**: Understands and classifies text (DeBERTaV3, E5, MPNet)
-**DecoderOnly**: Generates new text (Llama 3, Mistral, DeepSeek)
-**EncoderDecoder**: Transforms input to output (T5, UL2, BART)
-**RAG**: Combines retrieval with generation for grounded answers
- Inheritance shows how architectures relate; RAG uses both encoder and decoder

Transformers aren't one-size-fits-all. Like picking the right kitchen tool, choosing the correct transformer architecture is essential for success. We'll break down the three core types—encoder-only, decoder-only, and encoder-decoder—using current models and best practices.


## Understanding Encoder-Only, Decoder-Only, and Encoder-Decoder Models

Meet the three main transformer architectures—each designed for different jobs, delivering state-of-the-art results:

-**Encoder-Only (e.g., DeBERTaV3, E5):**Expert reader that produces contextualized vectors. Ideal for classification, sentiment analysis, NER, and semantic search embeddings.
-**Decoder-Only (e.g., Llama 3, Mistral, DeepSeek):**Creative writer generating text one token at a time. Perfect for chatbots, story generation, code completion, and open-ended synthesis.
-**Encoder-Decoder (e.g., T5 v1.1, UL2, DeepSeek):**Think translator—encoder understands input, decoder generates new output. Best for translation, summarization, or question answering.

Let's see these differences with Hugging Face pipelines. Modern practice specifies both model and tokenizer explicitly:


### Comparing Modern Transformer Architectures with Hugging Face Pipelines

Having explored the foundational concepts of transformers and examined how retrieval-augmented generation works, let's turn our attention to the practical application of different transformer architectures. The following section provides a comprehensive comparison of encoder-only, decoder-only, and encoder-decoder architectures, demonstrating how each type excels at different tasks. By understanding these architectural differences, you'll be better equipped to select the right model for your specific use case.

Here's the complete implementation from `src/modern_models.py` showing how different architectures handle tasks:

```python
def modern_architecture_comparison():
    """
    Implement the modern architecture comparison from article.
    Uses DeBERTaV3, Llama 3/Mistral, and T5 v1.1 as mentioned.
    """
    print("=== Modern Transformer Architectures Comparison ===\\n")

    # 1. Encoder-only: DeBERTaV3 for sentiment analysis
    print("1. Encoder-Only Architecture (DeBERTaV3):")
    try:
        # Try to use DeBERTaV3 as mentioned in article
        classifier = pipeline(
            "sentiment-analysis",
            model="microsoft/deberta-v3-base",
            tokenizer="microsoft/deberta-v3-base",
        )
        result = classifier("Transformers are awesome!")
        print("   Model: microsoft/deberta-v3-base")
        print("   Task: Sentiment Analysis")
        print("   Input: 'Transformers are awesome!'")
        print(f"   Output: {result}\\n")
    except Exception:
        # Fallback strategy for models requiring authentication
        print("   Note: DeBERTaV3 might be large. Using DistilBERT as alternative.")
        classifier = pipeline("sentiment-analysis",
                            model="distilbert-base-uncased-finetuned-sst-2-english")
        result = classifier("Transformers are awesome!")
        print(f"   Output: {result}\\n")

    # 2. Decoder-only: Modern generative models
    print("2. Decoder-Only Architecture (Modern LLMs):")
    print("   Note: Llama 3 and Mistral require authentication/large downloads.")
    print("   Using smaller alternatives for demonstration:\\n")

    # GPT-2 as accessible decoder-only example
    print("   a) GPT-2 (Classic decoder-only):")
    generator = pipeline("text-generation", model="gpt2", max_new_tokens=20)
    prompt = "Once upon a time,"
    result = generator(prompt, max_length=30, num_return_sequences=1)
    print(f"      Input: '{prompt}'")
    print(f"      Output: {result[0]['generated_text']}\\n")

    # Show how to use Mistral (requires auth)
    print("   b) To use Mistral or Llama 3:")
    print("      ```python")
    print("      # Requires Hugging Face authentication")
    print("      from transformers import pipeline")
    print("      generator = pipeline('text-generation', ")
    print("                          model='mistralai/Mistral-7B-v0.1',")
    print("                          device_map='auto')")
    print("      result = generator('Once upon a time,')")
    print("      ```\\n")

    # 3. Encoder-decoder: T5 for summarization
    print("3. Encoder-Decoder Architecture (T5):")
    try:
        # Use T5-small as it's more manageable
        summarizer = pipeline("summarization", model="t5-small")
        long_text = (
            "Transformers have revolutionized natural language processing. "
            "By leveraging attention mechanisms, they achieve state-of-the-art results "
            "in a variety of tasks, including translation, summarization, and question "
            "answering. Their flexibility and scalability make them a popular choice for "
            "modern AI applications."
        )
        summary = summarizer(long_text, max_length=40, min_length=10, do_sample=False)
        print("   Model: t5-small")
        print("   Task: Summarization")
        print(f"   Input: '{long_text[:50]}...'")
        print(f"   Output: {summary[0]['summary_text']}")
    except Exception as e:
        print(f"   Error loading T5: {e}")

```**What's happening:**-**Encoder-only (DeBERTaV3):**Reads and classifies sentiment
-**Decoder-only (Llama 3):**Generates new text, continuing prompts
-**Encoder-decoder (T5 v1.1):**Translates by understanding input and generating output**Key recap:**- Encoder-only: understands and labels
- Decoder-only: generates new text
- Encoder-decoder: transforms input to output**Modern best practice:**For proprietary data QA, consider retrieval-augmented generation (RAG). RAG combines retrievers (FAISS, Elasticsearch) with generators (Llama 3, DeepSeek) for contextually grounded answers. See Article 18 for details.


## Choosing the Right Architecture for Your Task

Selecting the right transformer is like picking the best tool. Here's your up-to-date guide:**1. Understanding Tasks (classification, NER, search):**- Use encoder-only models (DeBERTaV3, E5, MPNet)
- Why: Excel at extracting meaning and relationships, efficient for large-scale inference**2. Text Generation Tasks (chatbots, story/code generation):**- Use decoder-only models (Llama 3, Mistral, DeepSeek)
- Why: Trained to predict next token, ideal for fluent, context-aware generation**3. Sequence-to-Sequence Tasks (translation, summarization):**- Use encoder-decoder models (T5 v1.1, UL2, DeepSeek)
- Why: Transform one sequence into another, handling understanding and generation**4. Retrieval-Augmented Generation (RAG) for Advanced QA:**- Use hybrid retriever (FAISS, Elasticsearch) + generator (Llama 3, DeepSeek)
- Why: References up-to-date or proprietary information, improving accuracy
- See Article 18 for hands-on examples

Consider these factors:

-**Scalability:**Encoder-only models process in parallel, efficient for classification/embedding
-**Latency:**Decoder-only and encoder-decoder generate token-by-token, slower for long outputs. Use ONNX export or vLLM for production
-**Model Size:**Larger models are accurate but resource-heavy. Consider distilled, quantized, or PEFT models
-**Efficient Fine-Tuning:**Use LoRA, QLoRA, or Hugging Face PEFT to adapt large models with minimal compute

Tip: Fine-tuning adapts pre-trained models to your data. Parameter-efficient fine-tuning (PEFT) like LoRA and QLoRA is standard for large models, dramatically reducing costs. See Article 12.

> 🚀 Production Tips (based on our examples):
> 
> -**Export Strategy**: ONNX for 2-3x speedup, vLLM for high-throughput serving
> -**Batching**: See `src/attention_mechanism.py:51-103` for batch processing patterns
> -**Model Fallbacks**: Implement fallbacks as in `src/modern_models.py:23-85`
> -**Memory Optimization**:
>     - Use `torch.no_grad()` for inference (see all our examples)
>     - Clear cache with `torch.cuda.empty_cache()` between runs
>     - Monitor with `torch.cuda.memory_summary()`
> -**Quantization**: INT8/INT4 for edge deployment, especially for decoder models**Quick Recap:**- Match architecture to task type
- Consider efficiency, speed, and resources
- Use modern fine-tuning techniques (LoRA, QLoRA)


## Case Studies: Matching Architectures to Real Applications

Real business scenarios showing how architecture choice impacts results:

-**Customer Support Chatbot:**- Need: Generate fluent, context-aware responses, reference knowledge bases
    - Use: Decoder-only (Llama 3 or DeepSeek), optionally with RAG, fine-tuned using LoRA
    - Why: Generates text, maintains conversation flow, cites documents
-**Legal Document Classification:**- Need: Categorize large legal text volumes
    - Use: Encoder-only (DeBERTaV3 or E5), fine-tuned with PEFT
    - Why: Excels at understanding complex input; efficient batch processing
-**Email Translation Service:**- Need: Translate emails for multinational teams
    - Use: Encoder-decoder (T5 v1.1 or DeepSeek), deployed with ONNX or vLLM
    - Why: Designed for sequence transformation; optimized for production

Let's try summarizing a business report with T5 v1.1:


### Summarization with T5 v1.1 (Encoder-Decoder Model)

With a solid understanding of transformer architectures and their applications, let's see how these models perform in real-world scenarios. The following example demonstrates how T5 v1.1, an encoder-decoder model, handles summarization tasks—transforming lengthy text into concise, meaningful summaries while preserving key information.

Now that we've explored the theory behind transformer architectures, let's examine a practical example of using T5 v1.1 for summarization tasks. This code demonstrates how an encoder-decoder model transforms a long passage into a concise summary while preserving key information—showing exactly how these models handle sequence-to-sequence tasks in real-world applications.

```python
from transformers import pipeline


# Load a summarization pipeline using T5 v1.1
summarizer = pipeline('summarization', model='google-t5/t5-v1_1-small', tokenizer='google-t5/t5-v1_1-small')

long_text = (
    "Transformers have revolutionized natural language processing. "
    "By leveraging attention mechanisms, they achieve state-of-the-art results "
    "in a variety of tasks, including translation, summarization, and question answering. "
    "Their flexibility and scalability make them a popular choice for modern AI applications."
)


# max_length: maximum tokens in summary; min_length: minimum tokens
summary = summarizer(long_text, max_length=40, min_length=10, do_sample=False)
print(summary)  # Outputs a concise summary

```**Step-by-step:**1. Load T5 v1.1 summarization pipeline (encoder-decoder)
2. Provide long input text (e.g., business report)
3. Encoder processes input; decoder generates concise summary
4. Result is distilled version for quick review**Tip:**`max_length` and `min_length` control summary token count.**Key Takeaway:**Right architecture and model make solutions faster, more accurate, easier to scale. Align choice with business goals—consider hybrid or optimized approaches for production.


## Summary and Next Steps

To recap:

-**Encoder-only:**Understanding and labeling (classification, NER, search)
-**Decoder-only:**Generating content (chatbots, writing, code)
-**Encoder-decoder:**Transforming sequences (translation, summarization)
-**RAG:**Combining retrieval and generation (enterprise search, knowledge QA)

Choosing correctly is strategic—get it right for smoother, faster, smarter AI projects.

Ready to practice? Next section offers hands-on exercises for selecting, fine-tuning (including LoRA/QLoRA), and deploying transformers. For adapting models to your data, see Articles 10 and 12.

Pause and reflect: Which architecture fits your next project? Try the code with your data to see differences. For production deployment, explore ONNX or vLLM export for fast inference (Article 8).


# Summary, Key Ideas, and Glossary

Congratulations on mastering transformer internals! This section is your quick-reference guide—reviewing how transformers work, why attention matters, and how to choose the right architecture using latest Hugging Face tools.


## 1. Transformers: Simple Parts, Big Results

Transformers combine modular components:

-**Tokenization:**Splits text into processable pieces. Modern models use Unigram, SentencePiece, or Tokenizer Fast for better multilingual support
-**Embeddings:**Converts tokens to meaning-capturing vectors
-**Positional Encoding:**Adds sequence order information
-**Layer Normalization & Residual Connections:**Stabilize training and information flow
-**Feed-Forward Networks:**Add expressive power per layer

These simple parts combine to solve complex tasks—translation, classification, summarization, and beyond.


### Tokenization and Embedding Recap

```python

# From src/attention_mechanism.py - demonstrates tokenization and embeddings

# Dependencies: transformers==4.45.0, torch==2.5.0 (see pyproject.toml)
from transformers import AutoTokenizer, AutoModel
import torch

sentence = "Transformers are revolutionizing AI."

# Using RoBERTa as implemented in our examples
tokenizer = AutoTokenizer.from_pretrained('roberta-base')
model = AutoModel.from_pretrained('roberta-base')

inputs = tokenizer(sentence, return_tensors='pt')
print('Token IDs:', inputs['input_ids'])

with torch.no_grad():
    outputs = model(**inputs)
    print('Embeddings shape:', outputs.last_hidden_state.shape)

# Note: For alternative models, see src/modern_models.py for implementation examples

import torch
import matplotlib.pyplot as plt
import seaborn as sns


# Example: fake attention weights for 5 tokens
attn_weights = torch.tensor([
    [0.4, 0.2, 0.1, 0.2, 0.1],
    [0.1, 0.5, 0.1, 0.2, 0.1],
    [0.1, 0.1, 0.6, 0.1, 0.1],
    [0.15, 0.15, 0.1, 0.5, 0.1],
    [0.1, 0.1, 0.1, 0.2, 0.5]
])
plt.figure(figsize=(6, 5))
sns.heatmap(attn_weights.numpy(), annot=True, cmap='Blues', cbar=False)
plt.xlabel('Attended Token')
plt.ylabel('Query Token')
plt.title('Example Self-Attention Map')
plt.show()


# For real model attention visualization, see BertViz or TransformerLens documentation.

# Specify the library version for reproducibility

# pip install transformers==4.41.0 torch
from transformers import pipeline


# Encoder-only: Sentiment analysis
classifier = pipeline('sentiment-analysis', model='bert-base-uncased')
print(classifier('Transformers are awesome!'))  # Output: [{'label': 'POSITIVE', ...}]


# Decoder-only: Text generation (try 'gpt2', 'llama-2-7b', or 'mistralai/Mistral-7B-v0.1')
generator = pipeline('text-generation', model='gpt2')
print(generator('Once upon a time,'))  # Output: [{'generated_text': ...}]


# Encoder-decoder: Translation (try 't5-small' or newer models)
translator = pipeline('translation_en_to_fr', model='t5-small')
print(translator('Transformers are amazing!'))  # Output: [{'translation_text': ...}]


# For state-of-the-art, swap in newer models from the Hugging Face Model Hub.

# Setup environment (Python 3.12.9 + dependencies)
task setup


# Run all examples
task run


# Run specific modules
task run-attention-mechanism  # Self-attention demos and visualizations
task run-modern-models       # Architecture comparisons


# Run tests
task test


# Format code
task format