As we continue developing our software, we accumulate a growing amount of technical debt just to keep the system running. But I believe we are on the brink of an even larger issue. Cognitive debt.
Hope you enjoy this reading, all feedback is welcome.
Gotcha,
But this means we need to provide not only the same query (and be sure that the tokenizer is the same) but also provide the seed for the number generation. In this case we will have a deterministic outcome. (Unless we provide the list of numbers used, which for me feels wasteful)
But at this point none of the providers have this feature.
And I don’t think the open source have this also. But open source can be updated/changed.
Not that I see either looking at opensource inference engines. vLLM supports setting the seed so if you have that you do, and maybe it’s just as simple as harness setting and recording that. At least for replay this request type of operation but that does not show where in model each random numbers would have been applied.