Ismail Ben Alla

Lab notes

Short, dated experiments: one question, one result, one lesson.

  1. Flash Attention was a clean drop-in win in my from-scratch GPT

    Question. Is wiring PyTorch's scaled_dot_product_attention into a hand-written GPT worth it versus my manual causal mask?

    Result. Swapping the manual masked attention for scaled_dot_product_attention (Flash) kept outputs identical while cutting memory and step time, with a manual fallback retained for older PyTorch.

    Lesson. Once you understand attention well enough to write it by hand, you've also earned the right to use the fast kernel without it being a black box.

    codetransformers, gpt, efficiency

  2. Three patch-embedding strategies, three different ViTs

    Question. Does it matter whether ViT patches are embedded via Conv2D, LayerNorm-on-flattened-patches, or a reshape?

    Result. Implemented all three in my ViT image classifier. The Conv2D embedding trained most stably; the reshape variant was the most transparent for teaching, the LayerNorm variant sat in between.

    Lesson. Tokenization is not a detail — the very first layer that turns pixels into tokens shapes everything the transformer can learn.

    codevit, vision, from-scratch

  3. Decoding strategy changed translation quality more than the model did

    Question. For English↔Darija, where does output quality actually come from — the trained weights or the decoding strategy?

    Result. Holding the model fixed, moving from greedy to beam search with length normalization, and tuning top-p, changed perceived quality more than a round of retraining.

    Lesson. At inference time the search matters as much as the model. Always sweep decoding before blaming the weights.

    codenlp, decoding, transformers

  4. Latency reshaped the whole architecture in real-time voice conversion

    Question. How much does a hard real-time latency budget change the design of a voice-conversion model?

    Result. Designing NatVC around a live-latency constraint ruled out lookahead-heavy blocks and pushed the whole pipeline toward streaming, causal, low-overhead components.

    Lesson. A latency budget is an architecture constraint, not an afterthought. It belongs at the top of the design, next to the loss.

    audio, voice-conversion, latency

  5. Building autograd from scratch demystified backprop for good

    Question. Do I actually understand backpropagation, or do I just trust loss.backward()?

    Result. Writing MyTorcha's DAG and chain-rule backward pass by hand made gradients concrete; a tiny numerical gradient check confirmed the hand-derived ones.

    Lesson. Backprop is just the chain rule applied over a graph. Implementing it once turns a black box into something you can debug.

    codebackprop, from-scratch, fundamentals

  6. Comparing Transformer, LSTM, and RNN on identical music data

    Question. Is a single well-tuned model or a fair three-way comparison the better way to learn sequence modeling?

    Result. In MusicGen, training all three on the same corpus with documented loss curves made the Transformer's advantage — and the RNN's limits — visible in a way one tuned model never could.

    Lesson. A controlled comparison is worth more than a leaderboard score. Fair baselines are where the understanding lives.

    codesequence-models, experiments, generative