HR-Jev
GPT-2 for HR Outcome Prediction
A GPT-2 model that reads employee profiles and predicts four HR outcomes.
- 4 task heads on GPT-2
Built with
PyTorch
Hugging Face Transformers
pandas
TensorBoard
Gradio
pytest
Overview
HR data usually lives in tables, and tables are usually handled by classic tabular models. I wanted to test a different route: turn each employee record into a plain-English sentence and let a GPT-2 Transformer learn from it.
I wrote the GPT-2 architecture myself in PyTorch, loaded the pretrained GPT-2 weights into it, and added four task heads on top: attrition (binary), job level (5 classes), job satisfaction (4 classes), and monthly income (regression). For each task, the target attribute is removed from the sentence and the model predicts it from the final token's hidden state.
The project covers the full workflow: data cleaning and splitting, length-bucketed batching, TensorBoard logging, a pytest suite, saved test-metric reports, and an interactive Gradio app for trying out individual profiles.
The question
Can a GPT-2 Transformer learn tabular HR data if each employee record is rewritten as a plain-English sentence?
Key features
- Serializes each employee record into a fixed-order English sentence, such as "The employee is 30 years old, ..., travels rarely for work, ...".
- From-scratch GPT-2 backbone (causal self-attention, GELU feed-forward, LayerNorm) initialized from pretrained GPT-2 weights.
- Four task-specific heads, three for classification and one for regression, all trained jointly on one shared backbone.
- Data pipeline that cleans the data, label-encodes the targets, and saves reproducible 80/10/10 splits with per-split statistics.
- Training with AdamW, gradient clipping, and label smoothing, logged to TensorBoard and checkpointed periodically.
- Gradio app that loads a test sample or takes edited fields and predicts all four outcomes.
How it works
Clean and split
The raw HR table is normalized, constant columns are dropped, the targets are encoded, and the rows are split 80/10/10 with a fixed seed.
Records to sentences
Each record becomes one training sample per task: the target attribute is removed, and the remaining attributes are written as an English sentence and tokenized with the GPT-2 tokenizer.
Shared Transformer backbone
A GPT-2-style decoder, loaded with pretrained GPT-2 weights, encodes the sentence. Because attention is causal, the last token's hidden state summarizes the whole profile.
Task heads
The last-token vector goes to the head for that sample's task, either a classifier or a regressor. Classification uses label-smoothed cross-entropy and regression uses a scaled smooth-L1 loss.
Evaluate and serve
A tester script reports per-task accuracy, MAE, and MSE on the held-out split, and the Gradio app serves the best checkpoint.
Examples
Input
Age 30, business travel: rarely, department: research & development, overtime: no (other attributes omitted)
Output
The employee is 30 years old, ..., travels rarely for work, ..., works in the research & development department, ..., does not work overtime, ...
Tech stack
GPT-2 backbone, task heads, and training loop
GPT-2 tokenizer and pretrained weights
Clean, encode, and split the HR data
Track training losses and gradient statistics
Interactive prediction app
Tests for the tokenizer, batch loader, and model
What I learned
- Rewriting tables as text lets one language-model backbone serve several prediction tasks
Limitations
- Narrow HR dataset; some tasks (e.g. job satisfaction) remain hard to predict