CS6140 Machine Learning

HW7 - Word Vectors, Attention, Transformer

Make sure you check the syllabus for the due date. Please use the notations adopted in class.

Instructions. Submit your code (Jupyter notebook encouraged) together with a short report of results (attention-weight plots, translation examples, loss curves). Libraries (PyTorch, numpy, matplotlib) are allowed for data handling, plotting, and standard layers; where a problem says to complete a specific class (e.g. DotAttention, MultiheadAttention), that class's core logic must be your own. PROBLEM 1 has a from-scratch part (A) and a PyTorch part (B), same pattern as HW5's from-scratch/PyTorch pairs. (PROBLEM 1 moved here from HW5, since word/token embeddings are exactly what PROBLEMS 2–4's attention mechanisms consume as input.)

Requirements.

Fill in every TODO_STUDENT / #TODO in the class(es) named for each problem, and make sure the corresponding notebook runs top to bottom without errors.
Understand the whole notebook you run — including the provided/given code, not just the blanks you filled in — well enough to explain any part of it during office hours.

Datasets. Word vectors (Problem 1): 20 Newsgroups and the Text8 Wikipedia dataset (also on the course's MLdata folder). Machine Translation Spanish→English (Problems 2–4) using seq2seq models: data, dataloader, shared starter code (used by all three of those problems).

Starter code. In PROBLEMS 2–4 you can tune various parameters — data params: num_steps, sample_percent, batch_size, min_freq; model params: num_layers, num_blocks, num_hiddens, ffn_num_hiddens, num_heads, dropout, embed_size. Optional: try beam search (implemented, needs work) instead of greedy/argmax decoding; try an adaptive or decaying learning rate (not implemented). Answers to the (THEORY) problems may be written directly in a notebook markdown cell rather than a separate document, as long as they include any equation, plot, or illustration the answer depends on — not prose alone.

The PROBLEM 2–4 code can be computationally intensive, so consider: running on a local Nvidia GPU + CUDA, using Google Colab or similar, adjusting data size (sample_percent, batch size, etc. — on a Mac, try to make torch use MPS), or adjusting model params.

PROBLEM 1 — Word Vectors [60 points]

(A — 40 points, from scratch) Implement the word-embedding network discussed in class, following this tutorial: make positive word pairs $(i,j)$ relate to their frequency of co-occurrence; make negative pairs relate to 0 (downsample negatives for speed). Train the word vectors on the 20 Newsgroups dataset, and separately on the Text8 Wikipedia dataset.

What's tested: pick several non-trivial words (e.g. "China," "computer," "phone," "God," "Napoleon," "Catholic") and, for each, list the 15 most similar words by cosine-similarity of word vectors.

(B — 20 points, using PyTorch) Train the same network using PyTorch, and compare the resulting nearest-neighbor lists against your from-scratch part (A) result.

Library calls: torch.nn.

PROBLEM 2 Cross-attention (from Decoder to Encoder) [50 points]

(A) Complete the #TODO-s in [DotAttention] and in [Seq2SeqAttentionDecoder] before running the simple-attention notebook : implements dot-product cross-attention in a RNN Decoder with states that query the (key,val) from encoder output. Oberve the attention weights per query
* optional, no credit : try to substitute [DotProductAttention] with the [AdditiveAttention ] (already implemented) and recheck query-key weights
(B) Complete the #TODO-s in [MultiheadAttention] and in [MultiHeadSeq2SeqDecoder] before running the multihead-attention notebook : implements same mechanism as in (A) but with multi-head attention

PROBLEM 3 Self-attention (inside RNN) VS Cross-attention [50 points]

(A) Complete the #TODO-s in [SelfAttentionAugmentedEncoder] before running the encoder-self-attention-hybrid notebook : keeps the cross-attention from Pb2, and adds the self-attention inside the GRU-RNN encoder
(B) Complete the #TODO-s in class [SelfAttentiveGRUDecoder] before running the decoder-self-attention-hybrid notebook : keeps the cross-atention, and ads self-attention to both RNN-Encoder and RNN-Decoder

PROBLEM 4 Attention + Transformer [30 points]

(A) Complete the #TODO-s in [TransformerDecoderBlock] and [TransformerDecoder] before running the transformer-decoder-on-gru-encoder notebook : removes the RNN recurrence from the Decoder, using positional encoding to keep track of the order in the sequence
(B) optional no credit) Run the transformer notebook : removes recurrence(RNN) from both Encoder and Decoder : "attention is all you need".

PROBLEM 5 (THEORY) Why Multiple Attention Heads Instead of One Larger Head [25 points]

Suppose you replace your PROBLEM 2(B) multi-head attention with a single attention head of the same total dimensionality (so the total number of parameters is roughly unchanged), keeping everything else fixed. Performance drops noticeably on a task that requires simultaneously tracking several different kinds of relationships between words (e.g. both syntactic relationships like subject–verb agreement, and coreference relationships like a pronoun and the noun it refers to).

In a short written answer, explain, conceptually, why using multiple attention heads instead of one larger head can help in a case like this.

PROBLEM 6 (THEORY) Self-Attention vs. Cross-Attention [25 points]

Your PROBLEM 3 encoder/decoder uses self-attention (inside the RNN encoder and/or decoder) alongside the cross-attention from PROBLEM 2 (decoder queries attending to the encoder's key/value output). Explain the structural difference between self-attention and cross-attention (in terms of where Q, K, V come from), and explain why translation specifically needs both.