Ornith-1.5 Server Quality and Real-Use Speed Benchmark
Ornith-1.5 Server Quality and Real-Use Speed Benchmark
We loaded Ornith-1.5-35B-A3B on an NVIDIA DGX Spark GB10 and measured quality and speed using an actual YouTube automation script-generation request. Instead of reloading the model for every trial as llama-cli would, we started one llama-server for each quantization model.
This was not a short synthetic sentence test. The measurements came from real production script prompts and the validator-retry process.
At a Glance
- Q5_K_M: passed the quality gate 2/2 within five attempts and generated at 64.6 tok/s—the best balance.
- Q6_K: slightly slower at 59.8 tok/s, but both fixtures passed in an average of three attempts.
- Q8_0: the largest file and the longest load time, but neither fixture passed the quality gate after five attempts.
- Higher bit count did not automatically guarantee structured-output quality or retry convergence.
- All three models loaded and ran successfully; there were no OOM or infrastructure errors.
Target
| Item | Value |
|---|---|
| Measured models | Ornith-1.5-35B-A3B-MTP Q5_K_M / Q6_K / Q8_0 |
| Real-use context | YouTube Shorts automation script generation |
| Evaluation target | Production quality-gate pass rate and speed on long real requests |
| Quality inputs | Science and History production fixtures |
| Output contract | JSON script, 4–6 scenes, 94–120 words, fact_refs, LTX video_prompt |
Environment
| Item | Setting |
|---|---|
| Hardware | NVIDIA DGX Spark GB10 |
| Memory | 121 GiB unified memory |
| Runtime | llama.cpp llama-server |
| Context | 65,536 tokens |
| KV cache | Q8_0 / Q8_0 |
| GPU offload | 999 layers |
| MTP | draft-mtp |
| Reasoning | off |
| Server request | OpenAI-compatible streaming JSON |
Each model server was loaded once and warmed up. Science and History quality requests and validator retries ran in the same server process. The process was separate from the operational Qwen server.
Method / Protocol
1. Use the same Science and History production prompts.
2. Fix temperature at 0.7 for the first attempt and 0.5 for retries.
3. Pass the previous validator error into the next request, up to five attempts.
4. Stream every request and record the time of the first output token.
5. Save the prompt and generation timings returned by the server together with MTP draft acceptance.
6. Do not add a separate synthetic prompt or short-token test for speed.
Therefore, the speed below is not the speed of a finished script alone. It is the practical speed of a request that processes a long prompt, JSON structure, and video-prompt generation under the real output contract.
Result
Quality results
| Model | File size | Server load | First pass | Pass within 5 | Average attempts |
|---|---|---|---|---|---|
| Ornith Q5_K_M | 23.61 GiB | 28.1 s | 0/2 | 2/2 | 4.0 |
| Ornith Q6_K | 27.20 GiB | 32.0 s | 0/2 | 2/2 | 3.0 |
| Ornith Q8_0 | 35.21 GiB | 44.1 s | 0/2 | 0/2 | 5.0 |
Q5_K_M passed on the third History attempt and fifth Science attempt. Q6_K passed on the second History attempt and fourth Science attempt.
Q8_0 ended with 123 words on History and 151 words plus missing fact_refs in the first scene on Science. Its sentence and scene structure did not collapse, but it failed to satisfy the production output contract completely.
Speed measured in real quality requests
| Model | Prompt avg. | Generation tokens avg. | Prompt processing | Generation | TTFT | Request time | MTP acceptance |
|---|---|---|---|---|---|---|---|
| Ornith Q5_K_M | 2,920 tokens | 1,654 tokens | 1,208 tok/s | 64.6 tok/s | 0.738 s | 26.4 s | 44.7% |
| Ornith Q6_K | 2,927 tokens | 1,641 tokens | 1,237 tok/s | 59.8 tok/s | 1.032 s | 28.4 s | 43.3% |
| Ornith Q8_0 | 2,922 tokens | 1,594 tokens | 1,176 tok/s | 54.4 tok/s | 0.836 s | 30.1 s | 46.0% |
Request time is the average time for the client to receive the response. Model loading was recorded separately and is not included in request time.
Comparison
Q5_K_M combined the smallest file size with the fastest generation. Q6_K was slightly slower, but converged to the quality gate with fewer retries. When the cost of repeated generation is included, Q6_K is also a practical quality-first candidate.
Q8_0 is easy to assume to be the quality ceiling, but this run did not support that assumption. File size and load time increased, but neither first-pass nor final pass rate improved.
The result shows why quantization should be evaluated with all of the following:
- Structured JSON compliance
- Word count and time-budget compliance
- Completeness of
fact_refs - Convergence under retry feedback
- Prompt-processing and generation speed
- Model load time and memory headroom
Interpretation
In this first measurement, Q5_K_M offered the best balance of speed, capacity, and final pass rate. Q6_K was a little slower but more efficient under retries, making it a quality-first operational candidate. Q8_0 has a lower priority for additional testing under the current YouTube script contract.
However, this used only two fixtures and one repetition per model. It cannot establish a final quantization ranking. Human semantic and factual review was not part of this validator.
Limitations and Next Tracks
- Only two fixtures: Science and History.
- One repetition per model.
- No reasoning on/off comparison.
- No direct MTP-off comparison.
- No coding, tool-call, or agent-task success measurement.
- No quality measurement after image, TTS, or video generation.
The next comparison should repeat Q5_K_M and Q6_K_M on the same fixtures several times, then measure reasoning-on and coding/tool-call lanes separately.