Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

agree, prefill is the weak spot right now. it goes through the same per-token path as decode, which is dumb for long prompts. The fix is on the list: during prefill we can batch the expert reads for the whole prompt per layer instead of per token, that amortizes the IO a lot. until that lands, honest answer is this is good for chat-length stuff, not for feeding it a 10k token document.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: