llm-engineering-prompts/upstream/trailofbits-awesome-ml-security/catalogue/prompt-engineering.md

title, task, lineage_type, upstream_source, upstream_sha, imported_at, prompt_class, upstream_changes, author, validated

title	task	lineage_type	upstream_source	upstream_sha	imported_at	prompt_class	upstream_changes	author	validated
Prompt Engineering		import	https://github.com/trailofbits/awesome-ml-security/blob/d7fe4fa9/prompt-engineering.md	d7fe4fa9	2026-06-26	catalogue	accepted	upstream	false

This is out-of-date.

This is a survey of recent prompt engineering research. Tips and tricks have been extracted from relevant works. Each of these techniques should be taken with a grain of salt as it may not generalize to the chosen task, model, or settings.

General advice

• Everything I'll forget about prompting LLMs

• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm 

  • "Use declarative and direct signifiers for tasks such as translate or rephrase this paragraph so that a 2nd grader can understand it.
  • Use few-shot demonstrations when the task requires a bespoke format, recognizing that few-shot examples may be interpreted holistically by the model rather than as independent samples.
  • Specify tasks using characters or characteristic situations as a proxy for an intention such as asking Gandhi or Nietzsche to solve a task. Here you are tapping into LLMs' sophisticated understanding of analogies.
  • Constrain the possible completion output using careful syntactic and lexical prompt formulations such as saying "Translate this French sentence to English" or by adding quotes around the French sentence.

• Encourage the model to break down problems into subproblems via step-by-step reasoning

• Prompt Engineering Tips and Tricks with GPT-3 

  • Make sure your inputs are grammatically correct and have good writing quality as LLMs tend to preserve stylistic consistency in their completions.
  • Rather than generating a list of N items, generate a single item N times. This avoids the language model getting stuck in a repetitive loop.
  • In order to improve output quality, generate many completions and then rank them heuristically."

• Reframing Instructional Prompts to GPTk's Language 

  • Use low-level patterns from other examples to make a given prompt easier to understand for an LLM.
  • Explicitly itemize instructions into bulleted lists. 
  • Turn negative statements such as don't create questions which are not to create questions which are.
  • When possible, break down a top-level task into different sub-tasks that can be executed in parallel or sequentially.
  • Avoid repeated and generic statements when trying to solve a very specific task

• How to get Codex to produce the code you want!

  • Provide the model with high-level task descriptions, high-level context, examples, and previous user input.  
  • Set the temperature to 0 if you want the same output each time. 
  • Use the stop sequence to stop Codex from generating variations of similar code. 
  • "Imagine that you already live in a timeline where the output you want exists. If you were using/quoting it in a blog post, what caption or context might you write for it?" (Twitter)

• Ask Me Anything: A simple strategy for prompting language models

  • Prioritize open-ended questions over restricted ones. 
  • Consider using AMA prompting, which combines collections of open-ended prompts with weak supervision. 

• Legal Prompting: Teaching a Language Model to Think Like a Lawyer

Few-Shot and Least-to-Most Prompting

• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

  • The order of the examples matter; Consider using a probing technique to identify the optimal order. 

• Calibrate Before Use: Improving Few-Shot Performance of Language Models 

  • Few-shot examples can have majority label bias, recency bias, or common token bias. You can use a calibration technique to overcome this. 

• Zero-Label Prompt Selection 

  • Consider using ZPS, a statistical prompt selection technique. 

• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models 

  • This is how you use least-to-most prompting:
    • The first stage is problem reduction. The prompt in this stage contains constant examples that demonstrate the reduction followed by the specific question to be reduced. 
    • The second stage is for problem-solving -- sequentially solve the generated subproblems from the first stage. The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions; (3) the question to be answered next.

• Compositional Semantic Parsing with Large Language Models

  • "We address these challenges with dynamic least to-most prompting, a generic refinement of least-to-most prompting that involves the following steps: (1) tree-structured decomposition of natural language inputs through LM-predicted syntactic parsing, (2) use the decomposition to dynamically select exemplars, and (3) linearize the decomposition tree and prompt the model to sequentially generate answers to subproblems."

• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing   

  • This is an older survey of prompt engineering techniques.

Chain-of-Thought Prompting

• Large Language Models are Zero-Shot Reasoners 

  • You can add the exact phrase "Let's think step by step" for chain-of-thought (CoT) prompting.  

• Chain of Thought Prompting Elicits Reasoning in Large Language Models  

  • You can write a sequence of questions to force the model to think through the problem step-by-step in what is known as handcrafted CoT. 

• Complexity-Based Prompting for Multi-Step Reasoning

  • You can use a complexity-based scheme to choose answers from prompts with higher reasoning complexity. More broadly, when using CoT, chains with more reasoning steps can perform better. 

• Measuring and Narrowing the Compositionality Gap in Language Models (Twitter thread)

  • For complex tasks, force the model to ask follow-up questions to arrive at the answer (add "Are follow-up questions needed here? Yes" to the prompt"). In addition, consider integrating a search engine.

Soft Prompting 

• Injecting World Knowledge into Language Models through Soft Prompts 

  • Using soft prompting, provide the LLM with an external memory with domain-specific knowledge by adding continuous vectors to the input sequence. 

• XPrompt: Exploring the Extreme of Prompt Tuning 

  • Consider learning and tuning soft prompts. 

• Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

  • Try using compositional soft prompting for compositional problems.

Automated Prompt Generation 

• Large Language Models Are Human-Level Prompt Engineers & AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts 

  • You can use an automated prompt generation technique that relies on gradient-based optimization. 

• How Can We Know What Language Models Know? 

  • You can use an automated prompt generation technique that relies on mining and paraphrasing.  

• Prefix-Tuning: Optimizing Continuous Prompts for Generation 

  • You can optimize continuous prefixes for the input instead of discrete trigger words in the prompt. 

• Automatic Chain of Thought Prompting in Large Language Models

  • You can sample a diverse set of questions to automatically generate reasoning chains.

Evaluation Research & Applications

• A Hazard Analysis Framework for Code Synthesis Large Language Models 
• Evaluating Large Language Models Trained on Code 
• Examining Zero-Shot Vulnerability Repair with Large Language Models 
• Security Implications of Large Language Model Code Assistants: A User Study
• Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions
• Pop Quiz! Can a Large Language Model Help With Reverse Engineering?   
• Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language 
• Red Teaming Language Models with Language Models & Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
• Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements 
• A Systematic Evaluation of Large Language Models of Code 
• Large Language Models Struggle to Learn Long-Tail Knowledge 
• Do Users Write More Insecure Code with AI Assistants?

Prompt Engineering Libraries

• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
• PromptChainer: Chaining Large Language Model Prompts through Visual Programming  
• microsoft/prompt-engine: A library for helping developers craft prompts for LLMs
• GPT with Python interpreter
• GPT with a browser (GitHub)

Fine-Tuning; Similar Techniques; and Other Experiments 

• Aligning Language Models to Follow Instructions
• Large Language Models Can Self-Improve
• Learning by Distilling Context
• Large Language Models with Controllable Working Memory 
• Prompt Injection: Parameterization of Fixed Inputs 
• Do Prompt-Based Models Really Understand the Meaning of their Prompts?
• Self-Programming Artificial Intelligence Using Code-Generating Language Models
• GitHub - semiosis/prompts: A free and open-source curation of prompts
• Machine Learning for Big Code and Naturalness