In theory, it shouldn’t matter with state-of-the-art models. But in practice, there are design advantages and use cases where it does matter.

What’s the Difference?

A system prompt is a fixed set of instructions or context provided by developers to guide the language model’s responses. It’s always included in the input, regardless of the user’s prompt. A user prompt is the actual query or input from the end-user interacting with the model.

What Does the Model See?

The model sees the whole context, including both system and user prompts, along with any other augmented information. There’s no difference in how the context is processed or tokenized.

The “Needle In A Haystack” Issue

Current state-of-the-art models like Claude V3 Opus have nearly eliminated this issue, achieving over 99% accuracy. They can carefully consider the entire context before planning ahead.

Why It Still Matters

Early transformer-based attention mechanisms paid more attention to the beginning and end of the context. Since the system prompt is always first, it got more weight and priority. The “lost-in-the-middle” issue was common and is still something to consider, especially with smaller quantized models.

Common System Prompt Uses

• Set the tone and provide initial guidance

• Add constraints and verification to the model’s output

• Increase resilience against attempts to break character

• For few-shot learning, examples can be included as part of the system prompt (except when user flexibility with examples is needed)

• The Needle In a Haystack Test

• Lost in the Middle: How Language Models Use Long Contexts