You mention no supervised finetuning. May I ask why? I'm curious if you could get similar/better/worse results by just finetuning the LLM on your dataset rather than generating synthetic data, training a classifier and using GRPO?
In some other experiments, I’ve noticed that SFT can be very rigid and hard to generalize from, and later found this paper: https://arxiv.org/abs/2501.17161
Also I wanted to start from the instruct model, and wasn’t certain if it would be a good idea to do continued pretraining over top of it. Otherwise I’d need to create an instruct dataset for which the passages from the IPJ would be the answer… that also seemed a bit unnatural.
Perhaps there’s still some viable pathways in there, but I decided to see how the stupid simple thing worked out first and just go directly to RL. I think supervised fine tuning is feasible, but it’s not entirely straightforward what to train on and how much data to use if you want to follow up with RL.
Cool stuff in any case.