3th Floor AI STAGE 0: FROM ZERO CAST 2026
3th Floor Studios · Salt Lake City, UT
FROM
ZERO
A language model started as a blank file, a single GPU,
and a stubborn question. This is every step, uncut.
2.04B
parameters
2
GPUs (V100 + TITAN)
45
days
STAGE 01 OF 11
BUILD THE
WORKBENCH
before any model exists, you need a place to build one
WHY THIS STEP EXISTS An AI model learns by doing the same math operation billions of times, and graphics cards were built to do exactly that, fast. Before any training starts you need the right tools installed. Think of it as setting up a workshop. You don't build the tools. You install them and make sure they talk to each other. The GPU driver tells Python "yes, the card is here." The Python packages handle the hard parts: math at scale, the model architecture, and a trick that lets a large model squeeze into limited memory. The code to the right shows the exact commands that set up this machine.
WHAT YOU NEED A graphics card with 16GB+ of video memory, we used a Tesla V100 32GB, the same hardware a cloud data center uses  ·  Linux (Ubuntu 20.04+) or Windows with WSL2  ·  Python 3.10+  ·  GPU drivers so Python can talk to the card (CUDA 12.1)  ·  500GB+ of free disk for training data and model files.
setup.sh# verify GPU is visible $ nvidia-smi +-----------------------------------+ | GPU 0 Tesla V100-SXM2-32GB | | Temp 31°C VRAM 0 / 32510 MiB | +-----------------------------------+ # install PyTorch with CUDA 12.1 $ pip install torch==2.1.0+cu121 torchvision torchaudio \ --index-url https://download.pytorch.org/whl/cu121 # install the model ecosystem $ pip install transformers==4.40.0 datasets tokenizers $ pip install peft trl bitsandbytes accelerate $ pip install sentencepiece safetensors # sanity check $ python -c "import torch; print(torch.cuda.is_available())" True
Watch for: A simulated terminal runs the actual install commands. nvidia-smi confirms the GPU is online, then pip downloads PyTorch and the model ecosystem. Every package shown is the real one you would run on your own machine.
STAGE 02 OF 11
FEED
THE BEAST
a model is a compression of everything it read: grammar, dead puppies, shakespeare, and whatever the internet decided to publish
WHY THIS STEP EXISTS The model learns by reading text and predicting the next word, over and over, billions of times. The quality and variety of what it reads is what determines what it knows. General models read the entire internet: books, Wikipedia, code, forums. Our model (3thfloor-2b) read a smaller, focused set: software testing textbooks, ISTQB material, the 3th Floor blog, and 8,000 curated Q&A pairs. Smaller, but the model knows exactly what it needs to know.
PUBLIC SOURCES (for a general model) Project Gutenberg: 70,000 public domain books  ·  Wikipedia: 6.7M articles  ·  StackExchange: 50M+ Q&A pairs  ·  The Pile: 800GB curated mix  ·  All available free via HuggingFace datasets.
collect.pyfrom datasets import load_dataset # Project Gutenberg (public domain books) gutenberg = load_dataset( "sedthh/gutenberg_english", split="train" ) # Wikipedia English, latest dump wiki = load_dataset( "wikimedia/wikipedia", "20231101.en", split="train" ) # StackExchange Q&A pairs stack = load_dataset( "HuggingFaceTB/smollm-corpus", "stackexchange", split="train" ) total = len(gutenberg)+len(wiki)+len(stack) print(f"Total: {total:,} documents") # Total: 10,284,847 documents
Watch for: Document cards fly in from four sources: books (gold), Wikipedia (green), StackExchange (blue), code (purple). The bar below fills to show how much of the final corpus each source contributes. The counters in the corner are live document and size totals as the download simulates completing.
STAGE 03 OF 11
TAKE OUT
THE TRASH
38% of everything we downloaded went straight in the bin
WHY THIS STEP EXISTS Raw internet text is filthy. HTML tags, navigation boilerplate, duplicate paragraphs copied across a thousand sites, encoding artifacts (é where é should be), text so short it teaches nothing. A model trained on garbage learns to produce garbage. Four filters run on every document: length, language, near-duplicate hash, and a boilerplate ratio score. If a document fails any filter, it's gone. No exceptions, no review. The brutality is the point.
cleanup.pyimport re, hashlib from datasets import Dataset def clean(text): text = re.sub(r'<[^>]+>', '', text) # strip HTML text = re.sub(r'http\S+', '', text) # remove URLs text = re.sub(r'\s+', ' ', text).strip() text = text.encode('ascii','ignore').decode() if len(text) < 100: return None # too short return text def dedup(docs): seen, out = set(), [] for doc in docs: h = hashlib.md5(doc[:500].encode()).hexdigest() if h not in seen: seen.add(h); out.append(doc) return out
3TH FLOOR VERSION Our domain corpus was hand-curated rather than scraped, so cleaning was less dramatic. We reviewed every source, generated synthetic Q&A pairs in the expected chat format, and discarded anything ambiguous about testing methodology. Quality over quantity: 8,198 curated pairs versus millions of noisy documents.
Watch for: The pipeline samples documents as they're scanned. You can see exactly what each filter is catching. HTML tags, encoding garbage, duplicates, spam, and boilerplate get flagged in red and collapse out. Survivors turn bright and get a green CLEAN badge. The live counter at the top shows the scan running at 1.4M documents per second.
STAGE 04 OF 11
FIND
THE WORDS
the model doesn't read letters. it reads integers. tokenization is the translation layer.
WHY THIS STEP EXISTS The model doesn't read text. It reads numbers, each one pointing to a chunk of text in a fixed dictionary. Before training you have to build that dictionary. The algorithm starts with individual letters (all 256 of them) and asks: what two characters appear next to each other most often? Merge them into one entry. Repeat 32,000 times. Common words end up as a single number. Unusual words get split into recognizable pieces. "tokenization" might become ["token", "ization"], 2 numbers instead of 13 letters. The smaller the token count, the faster the model can learn.
3TH FLOOR VERSION 3thfloor-2b fine-tuned an existing model, so it inherited that model's vocabulary (151,665 entries, already trained to read English). Building a dictionary from raw characters only happens when you're starting from absolute zero. That is exactly what this demo shows: no inherited vocabulary, no shortcuts, the full build.
train_tokenizer.pyfrom tokenizers import ByteLevelBPETokenizer tok = ByteLevelBPETokenizer() tok.train( files=["corpus_clean.txt"], vocab_size=32_000, min_frequency=2, special_tokens=[ "<|im_start|>", "<|im_end|>", "<pad>", "<unk>" ] ) tok.save_model("./tokenizer/") print(f"Vocab size: {tok.get_vocab_size():,}") # Vocab size: 32,000 # Training time: ~2 hours on CPU
Watch for: The sentence explodes into individual characters first, every single letter its own token. The merge table finds the most common character pairs and fuses them into one entry, 32,000 times. Watch the vocabulary counter climb from 256 raw characters to a full dictionary. When it finishes, those 52 characters have collapsed to 10 tokens. Then four examples show how the dictionary handles everything from common words to words it has never seen before.
STAGE 05 OF 11
DRAW THE
BLUEPRINT
a transformer is a stack of math. you decide how tall, how wide, and how far it sees.
WHY THIS STEP EXISTS Before any training happens, you decide the shape of the model. Think of it like choosing the size of a building before construction. The model is a stack of identical processing floors. On each floor, every word looks at every other word and decides what matters in context. That is what "attention" means here. Then each word updates its own meaning based on what it just noticed. More floors means deeper understanding. Wider floors means more nuance per word. A longer context window means the model can read more text at once. Every choice costs video memory. Slide the numbers too high and it won't fit on the card.
Layers (depth)12
how many times the model re-processes each token. More layers = deeper reasoning chains
Attention heads8
each head learns to notice a different kind of relationship: grammar, facts, tone, position
Model dimension1024
the size of the vector that represents each token. Think of it as how many notes the model can take per word
Context window2048
how many tokens the model can see at once. 2048 tokens is roughly 1,500 words
,
estimated parameters
VRAM REQUIRED (fp16)
, GB / 32 GB
ARCHITECTURE DIAGRAM
press LOCK IN to assemble
Watch for: Drag any slider and the parameter count and VRAM gauge update live. Set the architecture you want to build. Click Lock In and the diagram assembles block by block using YOUR specs: embedding first, then repeating attention and feedforward blocks, then the output head. The status bar shows your parameter count and confirms the lock.
STAGE 06 OF 11
SIZE THE
BUILD
how big do you want to go, and how much iron do you have?

Model Size

Parameters 2.04B
1B                                                             1T

Your GPU Setup

Total VRAM 56 GB
32 GB (1× V100)                                  1,000 GB

Pre-Training Time From Scratch

,
calculating...
Fine-tuning an existing model = weeks, not years. This shows the from-zero cost.

What That Means

Adjust the sliders above.
⚠ Model does not fit in VRAM. At GB VRAM you can hold roughly . Increase GPU VRAM or use extreme quantization tricks.
WHY 3TH FLOOR FINE-TUNED FOR CAST The pre-training run was running. We stopped it. CAST is August 3rd, and a fine-tune of Qwen2.5 on 8,198 curated Q&A pairs delivered a working, domain-accurate model in 45 days. The from-scratch build restarts after CAST, with no deadline attached: estimated ~150 days to completion.
Fine-tuning borrows the language foundation from a model that already read half the internet, then teaches your specialty on top. The from-scratch build constructs that foundation itself, exactly what this demo walks you through.
Same destination. Two different timelines. We chose the one that ships before the conference.
STAGE 07 OF 11
THE IGNITION
SWITCH
two hundred lines of python that start a 45-day fire
WHY THIS STEP EXISTS Everything before this was preparation. This is where learning actually happens. The script loads the model into the graphics card, loads the training data, and runs a loop: show the model a sentence, ask it to predict the next word, compare the guess to the real answer, measure how wrong it was, and nudge 2 billion numbers slightly in the right direction. Do that 768 times across all 8,198 training pairs. After 45 days on two GPUs running in parallel the model knows software testing. Too aggressive with the nudges and the math explodes. Too gentle and you wait forever. The script above is the entire recipe.
train.py (key sections)import os, torch from transformers import ( AutoModelForCausalLM, AutoTokenizer, TrainingArguments ) from peft import LoraConfig, get_peft_model from trl import SFTTrainer, SFTConfig # ── Config ──────────────────────── MODEL = "qwen/Qwen2.5-1.5B" DATA = "./dataset/training.jsonl" OUT = "./output/" RANK = 16 # LoRA rank SEQ_LEN = 2048 # context window EPOCHS = 3 # training passes LR = 1e-4 # ── Load model (4-bit) ──────────── model = AutoModelForCausalLM.from_pretrained( MODEL, load_in_4bit=True, device_map="auto", torch_dtype=torch.bfloat16 ) # ── LoRA adapter ────────────────── lora = LoraConfig( r=RANK, lora_alpha=RANK*2, target_modules=["q_proj","v_proj", "k_proj","o_proj"], lora_dropout=0.05, ) model = get_peft_model(model, lora) # ── Train ───────────────────────── trainer = SFTTrainer( model=model, train_dataset=dataset, args=SFTConfig( output_dir=OUT, num_train_epochs=EPOCHS, per_device_train_batch_size=1, gradient_accumulation_steps=32, learning_rate=LR, bf16=True, gradient_checkpointing=True, lr_scheduler_type="cosine", optim="paged_adamw_8bit", max_length=SEQ_LEN, ) ) trainer.train()
THE TRICKS THAT MAKE IT FIT A 2-billion-parameter model doesn't naturally fit on a 32GB card. Three tricks make it possible:

4-bit weights: instead of storing each number at full precision, we use 4 bits instead of 16. Half the memory, nearly the same result. Same idea as compressing an image. You lose a little, you keep the picture.

LoRA (the adapter trick): instead of retraining all 2 billion numbers, we freeze the original model and clip on tiny trainable layers at specific points. 99% of the model never changes. We train less than 1%. Same result, a fraction of the memory.

Memory checkpointing: during learning, the card needs to remember each step to calculate corrections. This trick forgets and recalculates as needed. 30% slower, but fits in 32GB.
Watch for: A terminal boots up and runs the actual training command. The script loads the 4-bit model into GPU memory, attaches the LoRA adapter (the small trainable layer on top), loads your 8,198 training pairs, then fires the first steps. You can see the loss on the first two steps, high because the model is still guessing. The real run continues on the next page.
STAGE 08 OF 11
45 DAYS ON
ONE GPU
loss is the model's confusion score. low is better. watch it fall.
WHAT YOU'RE WATCHING Loss measures how wrong the model was at predicting the next token. It starts high, the model is essentially guessing. Over thousands of steps it detects patterns: letters cluster into words, words follow grammar, grammar encodes meaning. Each vertical flag marks a threshold where the model's behavior visibly changed. The curve doesn't fall smoothly. You'll see spikes where it encountered unusual text and had to readjust. That's normal.
0
Step
,
Loss
0.00
Epoch
,
ETA (compressed)
GPU UTILIZATION (8 cores)
⏩ 5,000× COMPRESSED · real training: 45 days · V100-SXM2 32GB + TITAN RTX 24GB
Watch for: The loss curve draws in real time: 45 days of training compressed to about 10 seconds. Each point is one real training step. Watch the curve fall fast at first (the model finds obvious patterns like word boundaries and basic grammar), then slow as it works on harder things like staying on topic and knowing the domain. Six milestone flags pop up as the model crosses key thresholds. The GPU bars flicker the whole time. Both the V100 and the TITAN pinned at capacity for 45 straight days.
STAGE 09 OF 11
IS IT ANY
GOOD?
grading the model and shrinking it down to something you can actually run
DID IT ACTUALLY LEARN ANYTHING? We measure how surprised the model is by text it has never seen. Give it a real testing document. Ask it to predict each word. A model guessing randomly gets a score of 22,026. That is the worst possible result. A model that actually learned the subject scores in the low twenties or better. 3thfloor-2b scored 18.4 on held-out testing methodology text. Lower is better. The comparison boxes below show the same question given to the blank model on day 1 versus the trained model on day 45.
22,026
perplexity (lower = smarter)
Watch for: The perplexity gauge needle sweeps from 22,026 (random guessing) down to 18.4, the model's actual score on held-out testing text. Then two completion boxes appear side by side: the same prompt fed to the step-0 checkpoint (before any training) and the final checkpoint. The difference is what 45 days of gradient descent looks like.
SHRINK IT DOWN: EXPORT TO LAPTOP SIZE The trained model is 4.1GB, fine for a server but unusable on someone's laptop. A tool called llama.cpp compresses every number in the model from 16-bit to 4-bit precision. Same idea as converting a studio WAV file to an MP3. You lose a small amount of quality, you keep the song. Result: 1.3GB. 68% smaller. Nearly identical answers. Runs on a MacBook with 8GB RAM. That's the file on the USB drives at the back of the room.
export.sh# merge LoRA into base weights (CPU) $ python merge_adapter.py \ --base qwen2.5-1.5b \ --adapter ./output/adapter/ \ --out ./merged/ # convert to GGUF (llama.cpp) $ python convert_hf_to_gguf.py ./merged/ \ --outtype f16 \ --outfile 3thfloor-tester-f16.gguf # quantize to Q4_K_M $ ./llama-quantize \ 3thfloor-tester-f16.gguf \ 3thfloor-tester-3b-q4_k_m.gguf Q4_K_M
Watch for: The file size bar shrinks from 4.1 GB (float16 checkpoint) down to 1.3 GB (Q4_K_M GGUF). That 68% reduction is what makes the model runnable on a laptop. The format conversion happens via llama.cpp. That is the file on the USB drives at the back of the room.
grade the model first
STAGE 10 OF 11
TALK
TO IT
a language model is just a next-token picker. it makes that choice 500 times to write a paragraph.
HOW INFERENCE WORKS You pass a prompt → the model assigns probabilities to every token in its 32,000-word vocabulary → it samples one → appends it to the prompt → repeats. There is no plan, no outline, no thinking ahead. Just: given everything so far, what token is most likely next? Done 500 times. The fact that coherent text emerges from that process is the whole mystery.
SOMETHING REAL IS HAPPENING HERE While you watched the training loop, this browser tab built a tiny language model from scratch in JavaScript. It trained on 50KB of public domain text in about two seconds. It has roughly 8,000 parameters instead of 2 billion. Toggle to "tiny model" and type anything. What answers you is a genuine model that was just trained, in this tab, from the corpus you saw downloaded on page 2. It will be strange and half-grammatical. That's what two thousand parameters looks like. The pipeline you just walked through is identical, scaled up ten million times.
model training... click a prompt below or type your own
STAGE 11 OF 11
THE
BILL
what it actually cost. no rounding. no omissions.

Compute

Parameters0
GPUsV100-SXM2 32GB + TITAN RTX 24GB
GPU-hours1,080 h
Wall time45 days
Electricity~108 kWh
Est. cost~$13 (Utah rates)

Data

Training pairs0
DomainSoftware testing
SourcesISTQB · blog · SQA · synthetic
Tokens seen~147M
TokenizerQwen2 BPE (151,665)
Data collection1 week

Model

Final loss2.84
Architecture24L · 16H · d2048
Base modelQwen2.5-1.5B
Checkpoint (fp16)4.1 GB
GGUF Q4_K_M1.3 GB
Runs on8GB RAM laptop
"This model exists. It runs on a laptop. You built the same thing in ten minutes.
The difference is 53 days of gradient descent."
3th Floor Studios, Salt Lake City UT
Want to see more?
Help fund better hardware, more experiments, and more trained models.
SUPPORT ON KO-FI →
3thfloor.com/tester ↗