llama : fix command-r inference when omitting outputs

[compilade]

As reported in #6122 (comment), command-r was broken by #6122.

@Wuzzooy can you confirm whether this fixes the problem?

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3: 481 iterations 🚀

• Concurrent users: 8, duration: 10m
• HTTP request : avg=9814.5ms p(90)=28135.44ms fails=0, finish reason: stop=481 truncated=0
• Prompt processing (pp): avg=243.23tk/s p(90)=733tk/s total=190.52tk/s
• Token generation (tg): avg=98.03tk/s p(90)=271.21tk/s total=128.99tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=compilade/fix-command-r commit=64b7d858910e7056d36b113cb72512df274c34f1

Time series

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 481 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1711622819 --> 1711623453
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 638.83, 638.83, 638.83, 638.83, 638.83, 713.76, 713.76, 713.76, 713.76, 713.76, 711.92, 711.92, 711.92, 711.92, 711.92, 741.66, 741.66, 741.66, 741.66, 741.66, 754.32, 754.32, 754.32, 754.32, 754.32, 747.23, 747.23, 747.23, 747.23, 747.23, 741.09, 741.09, 741.09, 741.09, 741.09, 738.78, 738.78, 738.78, 738.78, 738.78, 738.64, 738.64, 738.64, 738.64, 738.64, 746.46, 746.46, 746.46, 746.46, 746.46, 751.47, 751.47, 751.47, 751.47, 751.47, 742.93, 742.93, 742.93, 742.93, 742.93, 767.55, 767.55, 767.55, 767.55, 767.55, 781.43, 781.43, 781.43, 781.43, 781.43, 747.85, 747.85, 747.85, 747.85, 747.85, 756.09, 756.09, 756.09, 756.09, 756.09, 754.91, 754.91, 754.91, 754.91, 754.91, 751.61, 751.61, 751.61, 751.61, 751.61, 762.95, 762.95, 762.95, 762.95, 762.95, 763.8, 763.8, 763.8, 763.8, 763.8, 758.25, 758.25, 758.25, 758.25, 758.25, 756.99, 756.99, 756.99, 756.99, 756.99, 755.76, 755.76, 755.76, 755.76, 755.76, 755.26, 755.26, 755.26, 755.26, 755.26, 755.91, 755.91, 755.91, 755.91, 755.91, 762.47, 762.47, 762.47, 762.47, 762.47, 759.86, 759.86, 759.86, 759.86, 759.86, 754.15, 754.15, 754.15, 754.15, 754.15, 755.01, 755.01, 755.01, 755.01, 755.01, 759.16, 759.16, 759.16, 759.16, 759.16, 757.99, 757.99, 757.99, 757.99, 757.99, 757.71, 757.71, 757.71, 757.71, 757.71, 757.11, 757.11, 757.11, 757.11, 757.11, 754.97, 754.97, 754.97, 754.97, 754.97, 753.98, 753.98, 753.98, 753.98, 753.98, 756.0, 756.0, 756.0, 756.0, 756.0, 760.6, 760.6, 760.6, 760.6, 760.6, 763.14, 763.14, 763.14, 763.14, 763.14, 766.21, 766.21, 766.21, 766.21, 766.21, 767.33, 767.33, 767.33, 767.33, 767.33, 764.98, 764.98, 764.98, 764.98, 764.98, 763.69, 763.69, 763.69, 763.69, 763.69, 758.83, 758.83, 758.83, 758.83, 758.83, 753.54, 753.54, 753.54, 753.54, 753.54, 756.75, 756.75, 756.75, 756.75, 756.75, 756.62, 756.62, 756.62, 756.62, 756.62, 750.87, 750.87, 750.87, 750.87, 750.87, 747.96, 747.96, 747.96, 747.96, 747.96, 744.07, 744.07, 744.07, 744.07, 744.07, 741.02, 741.02, 741.02, 741.02, 741.02, 739.47, 739.47, 739.47, 739.47, 739.47, 737.93, 737.93, 737.93, 737.93, 737.93, 740.28, 740.28, 740.28, 740.28, 740.28, 740.03, 740.03, 740.03, 740.03, 740.03, 732.73, 732.73, 732.73, 732.73, 732.73, 732.59, 732.59, 732.59, 732.59, 732.59, 734.96, 734.96, 734.96, 734.96, 734.96, 736.28, 736.28, 736.28, 736.28, 736.28, 735.37, 735.37, 735.37, 735.37, 735.37, 734.65, 734.65, 734.65, 734.65, 734.65, 735.73, 735.73, 735.73]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 481 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1711622819 --> 1711623453
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 28.4, 28.4, 28.4, 28.4, 28.4, 18.35, 18.35, 18.35, 18.35, 18.35, 18.32, 18.32, 18.32, 18.32, 18.32, 18.5, 18.5, 18.5, 18.5, 18.5, 19.01, 19.01, 19.01, 19.01, 19.01, 19.65, 19.65, 19.65, 19.65, 19.65, 20.4, 20.4, 20.4, 20.4, 20.4, 20.45, 20.45, 20.45, 20.45, 20.45, 20.65, 20.65, 20.65, 20.65, 20.65, 20.64, 20.64, 20.64, 20.64, 20.64, 20.55, 20.55, 20.55, 20.55, 20.55, 20.22, 20.22, 20.22, 20.22, 20.22, 19.97, 19.97, 19.97, 19.97, 19.97, 19.79, 19.79, 19.79, 19.79, 19.79, 19.39, 19.39, 19.39, 19.39, 19.39, 18.6, 18.6, 18.6, 18.6, 18.6, 18.49, 18.49, 18.49, 18.49, 18.49, 18.56, 18.56, 18.56, 18.56, 18.56, 18.67, 18.67, 18.67, 18.67, 18.67, 18.57, 18.57, 18.57, 18.57, 18.57, 18.46, 18.46, 18.46, 18.46, 18.46, 18.39, 18.39, 18.39, 18.39, 18.39, 18.29, 18.29, 18.29, 18.29, 18.29, 18.21, 18.21, 18.21, 18.21, 18.21, 18.26, 18.26, 18.26, 18.26, 18.26, 18.28, 18.28, 18.28, 18.28, 18.28, 18.19, 18.19, 18.19, 18.19, 18.19, 18.22, 18.22, 18.22, 18.22, 18.22, 18.31, 18.31, 18.31, 18.31, 18.31, 18.31, 18.31, 18.31, 18.31, 18.31, 18.26, 18.26, 18.26, 18.26, 18.26, 18.32, 18.32, 18.32, 18.32, 18.32, 18.46, 18.46, 18.46, 18.46, 18.46, 18.51, 18.51, 18.51, 18.51, 18.51, 18.61, 18.61, 18.61, 18.61, 18.61, 18.67, 18.67, 18.67, 18.67, 18.67, 18.64, 18.64, 18.64, 18.64, 18.64, 18.61, 18.61, 18.61, 18.61, 18.61, 18.58, 18.58, 18.58, 18.58, 18.58, 18.53, 18.53, 18.53, 18.53, 18.53, 18.43, 18.43, 18.43, 18.43, 18.43, 18.42, 18.42, 18.42, 18.42, 18.42, 18.42, 18.42, 18.42, 18.42, 18.42, 18.43, 18.43, 18.43, 18.43, 18.43, 18.46, 18.46, 18.46, 18.46, 18.46, 18.43, 18.43, 18.43, 18.43, 18.43, 18.38, 18.38, 18.38, 18.38, 18.38, 18.3, 18.3, 18.3, 18.3, 18.3, 17.99, 17.99, 17.99, 17.99, 17.99, 17.95, 17.95, 17.95, 17.95, 17.95, 17.79, 17.79, 17.79, 17.79, 17.79, 17.46, 17.46, 17.46, 17.46, 17.46, 17.29, 17.29, 17.29, 17.29, 17.29, 17.33, 17.33, 17.33, 17.33, 17.33, 17.37, 17.37, 17.37, 17.37, 17.37, 17.41, 17.41, 17.41, 17.41, 17.41, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.38, 17.38, 17.38, 17.38, 17.38, 17.4, 17.4, 17.4]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 481 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1711622819 --> 1711623453
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.28, 0.28, 0.28, 0.28, 0.28, 0.26, 0.26, 0.26, 0.26, 0.26, 0.17, 0.17, 0.17, 0.17, 0.17, 0.09, 0.09, 0.09, 0.09, 0.09, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.25, 0.25, 0.25, 0.25, 0.25, 0.28, 0.28, 0.28, 0.28, 0.28, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.14, 0.14, 0.14, 0.14, 0.14, 0.2, 0.2, 0.2, 0.2, 0.2, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.17, 0.27, 0.27, 0.27, 0.27, 0.27, 0.26, 0.26, 0.26, 0.26, 0.26, 0.3, 0.3, 0.3, 0.3, 0.3, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.15, 0.15, 0.15, 0.15, 0.15, 0.28, 0.28, 0.28, 0.28, 0.28, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.09, 0.09, 0.09, 0.09, 0.09, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.23, 0.23, 0.23, 0.23, 0.23, 0.27, 0.27, 0.27, 0.27, 0.27, 0.24, 0.24, 0.24, 0.24, 0.24, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.26, 0.26, 0.26, 0.26, 0.26, 0.38, 0.38, 0.38, 0.38, 0.38, 0.44, 0.44, 0.44, 0.44, 0.44, 0.53, 0.53, 0.53, 0.53, 0.53, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.46, 0.46, 0.46, 0.46, 0.46, 0.23, 0.23, 0.23, 0.23, 0.23, 0.1, 0.1, 0.1, 0.1, 0.1, 0.12, 0.12, 0.12, 0.12, 0.12, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 0.2, 0.2, 0.23, 0.23, 0.23, 0.23, 0.23, 0.27, 0.27, 0.27, 0.27, 0.27, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 481 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1711622819 --> 1711623453
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0]
                    

Loading

[Wuzzooy]

issue fixed