2026-06-26 18:56:21 +00:00
1 changed files with 150 additions and 0 deletions
@@ -0,0 +1,150 @@
 ---
 title: "Prompt Engineering"
 task: ""
 lineage_type: import
 upstream_source: https://github.com/trailofbits/awesome-ml-security/blob/d7fe4fa9/prompt-engineering.md
 upstream_sha: d7fe4fa9
 imported_at: 2026-06-26
 prompt_class: catalogue
 upstream_changes: accepted
 author: upstream
 validated: false
 ---
 **This is out-of-date.**
 This is a survey of recent prompt engineering research. Tips and tricks have been extracted from relevant works. Each of these techniques should be taken with a grain of salt as it may not generalize to the chosen task, model, or settings.
 ### General advice
 - [Everything I'll forget about prompting LLMs](https://olickel.com/everything-i-know-about-prompting-llms)
 - [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://arxiv.org/abs/2102.07350) 
    - "Use declarative and direct signifiers for tasks such as translate or rephrase this paragraph so that a 2nd grader can understand it.
    -  Use few-shot demonstrations when the task requires a bespoke format, recognizing that few-shot examples may be interpreted holistically by the model rather than as independent samples.
    - Specify tasks using characters or characteristic situations as a proxy for an intention such as asking Gandhi or Nietzsche to solve a task. Here you are tapping into LLMs' sophisticated understanding of analogies.
    - Constrain the possible completion output using careful syntactic and lexical prompt formulations such as saying "Translate this French sentence to English" or by adding quotes around the French sentence.
 - [Encourage the model to break down problems into subproblems via step-by-step reasoning](https://www.mihaileric.com/posts/a-complete-introduction-to-prompt-engineering/)
 - [Prompt Engineering Tips and Tricks with GPT-3](https://blog.andrewcantino.com/blog/2021/04/21/prompt-engineering-tips-and-tricks/) 
    - Make sure your inputs are grammatically correct and have good writing quality as LLMs tend to preserve stylistic consistency in their completions.
    - Rather than generating a list of N items, generate a single item N times. This avoids the language model getting stuck in a repetitive loop.
    - In order to improve output quality, [generate many completions and then rank them heuristically](https://www.mihaileric.com/posts/a-complete-introduction-to-prompt-engineering/)."
 - [Reframing Instructional Prompts to GPTk's Language](https://arxiv.org/abs/2109.07830) 
    - Use low-level patterns from other examples to make a given prompt easier to understand for an LLM.
    - Explicitly itemize instructions into bulleted lists. 
    - Turn negative statements such as don't create questions which are not to create questions which are.
    - When possible, break down a top-level task into different sub-tasks that can be executed in parallel or sequentially.
    - [Avoid repeated and generic statements when trying to solve a very specific task](https://www.mihaileric.com/posts/a-complete-introduction-to-prompt-engineering/)
 - [How to get Codex to produce the code you want!](https://microsoft.github.io/prompt-engineering/)
    - Provide the model with high-level task descriptions, high-level context, examples, and previous user input.  
    - Set the temperature to 0 if you want the same output each time. 
    -  Use the stop sequence to stop Codex from generating variations of similar code. 
    - "Imagine that you already live in a timeline where the output you want exists. If you were using/quoting it in a blog post, what caption or context might you write for it?" ([Twitter](https://twitter.com/davidad/status/1551143240065228800))
 - [Ask Me Anything: A simple strategy for prompting language models](https://arxiv.org/abs/2210.02441)
    - Prioritize open-ended questions over restricted ones. 
    - Consider using AMA prompting, which combines collections of open-ended prompts with weak supervision. 
 - [Legal Prompting: Teaching a Language Model to Think Like a Lawyer](https://arxiv.org/abs/2212.01326)
 ### Few-Shot and Least-to-Most Prompting
 - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://arxiv.org/abs/2104.08786)
    - The order of the examples matter; Consider using a probing technique to identify the optimal order. 
 - [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://arxiv.org/abs/2102.09690) 
    - Few-shot examples can have majority label bias, recency bias, or common token bias. You can use a calibration technique to overcome this. 
 - [Zero-Label Prompt Selection](https://arxiv.org/abs/2211.04668) 
    - Consider using ZPS, a statistical prompt selection technique. 
 -  [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625) 
    - This is how you use least-to-most prompting:
      - The first stage is problem reduction. The prompt in this stage contains constant examples that demonstrate the reduction followed by the specific question to be reduced. 
      - The second stage is for problem-solving -- sequentially solve the generated subproblems from the first stage. The prompt in this stage consists of three parts: (1) constant examples demonstrating how subproblems are solved; (2) a potentially empty list of previously answered subquestions and generated solutions; (3) the question to be answered next.
 - [Compositional Semantic Parsing with Large Language Models](https://arxiv.org/abs/2209.15003)
   - "We address these challenges with dynamic least to-most prompting, a generic refinement of least-to-most prompting that involves the following steps: (1) tree-structured decomposition of natural language inputs through LM-predicted syntactic parsing, (2) use the decomposition to dynamically select exemplars, and (3) linearize the decomposition tree and prompt the model to sequentially generate answers to subproblems."
 - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://arxiv.org/abs/2107.13586)   
    - This is an older survey of prompt engineering techniques.
 ### Chain-of-Thought Prompting
 - [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916) 
    - You can add the exact phrase "Let's think step by step" for chain-of-thought (CoT) prompting.  
 - [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)  
    -  You can write a sequence of questions to force the model to think through the problem step-by-step in what is known as handcrafted CoT. 
 - [Complexity-Based Prompting for Multi-Step Reasoning](https://arxiv.org/abs/2210.00720)
    - You can use a complexity-based scheme to choose answers from prompts with higher reasoning complexity. More broadly, when using CoT, chains with more reasoning steps can perform better. 
 - [Measuring and Narrowing the Compositionality Gap in Language Models](https://arxiv.org/abs/2210.03350) ([Twitter thread](https://twitter.com/ofirpress/status/1577302733383925762))
    - For complex tasks, force the model to ask follow-up questions to arrive at the answer (add "Are follow-up questions needed here? Yes" to the prompt"). In addition, consider integrating a search engine.
 ### Soft Prompting 
 - [Injecting World Knowledge into Language Models through Soft Prompts](https://arxiv.org/abs/2210.04726) 
    -  Using soft prompting, provide the LLM with an external memory with domain-specific knowledge by adding continuous vectors to the input sequence. 
 -  [XPrompt: Exploring the Extreme of Prompt Tuning](https://arxiv.org/abs/2210.04457) 
    - Consider learning and tuning soft prompts. 
 - [Learning to Compose Soft Prompts for Compositional Zero-Shot Learning](https://arxiv.org/abs/2204.03574)
    - Try using compositional soft prompting for compositional problems.
 ### Automated Prompt Generation 
 - [Large Language Models Are Human-Level Prompt Engineers](https://arxiv.org/abs/2211.01910) & [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://arxiv.org/abs/2010.15980) 
    - You can use an automated prompt generation technique that relies on gradient-based optimization. 
 - [How Can We Know What Language Models Know?](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00324/96460/How-Can-We-Know-What-Language-Models-Know) 
    - You can use an automated prompt generation technique that relies on mining and paraphrasing.  
 - [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://arxiv.org/abs/2101.00190) 
    - You can optimize continuous prefixes for the input instead of discrete trigger words in the prompt. 
 - [Automatic Chain of Thought Prompting in Large Language Models](https://arxiv.org/abs/2210.03493)
    - You can sample a diverse set of questions to automatically generate reasoning chains.
 ### Evaluation Research & Applications
 - [A Hazard Analysis Framework for Code Synthesis Large Language Models](https://arxiv.org/abs/2207.14157) 
 - [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374) 
 - [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://arxiv.org/abs/2112.02125) 
 - [Security Implications of Large Language Model Code Assistants: A User Study](https://arxiv.org/abs/2208.09727)
 - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)
 - [Pop Quiz! Can a Large Language Model Help With Reverse Engineering?](https://arxiv.org/abs/2202.01142)   
 - [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://arxiv.org/abs/2210.15157) 
 - [Red Teaming Language Models with Language Models](https://arxiv.org/abs/2202.03286) & [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://www.anthropic.com/red_teaming.pdf)
 - [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://arxiv.org/abs/2210.01970) 
 - [A Systematic Evaluation of Large Language Models of Code](https://arxiv.org/abs/2202.13169) 
 - [Large Language Models Struggle to Learn Long-Tail Knowledge](https://arxiv.org/abs/2211.08411) 
 - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/pdf/2211.03622.pdf)
 ### Prompt Engineering Libraries
 - [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://arxiv.org/abs/2202.01279#:~:text=PromptSource%3A%20An%20Integrated%20Development%20Environment%20and%20Repository%20for%20Natural%20Language%20Prompts,-Stephen%20H.&text=PromptSource%20is%20a%20system%20for,language%20input%20and%20target%20output)
 - [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://arxiv.org/abs/2208.07852)
 - [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://arxiv.org/abs/2203.06566)  
 - [microsoft/prompt-engine: A library for helping developers craft prompts for LLMs](https://github.com/microsoft/prompt-engine)
 - [GPT with Python interpreter](https://twitter.com/sergeykarayev/status/1569377881440276481?s=46&t=voSylmKII0grJoj8juIOYQ)
 - [GPT with a browser](https://twitter.com/sharifshameem/status/1405462642936799247) ([GitHub](https://github.com/nat/natbot))
 ### Fine-Tuning; Similar Techniques; and Other Experiments 
 - [Aligning Language Models to Follow Instructions](https://openai.com/blog/instruction-following/)
 - [Large Language Models Can Self-Improve](https://arxiv.org/abs/2210.11610)
 - [Learning by Distilling Context](https://arxiv.org/abs/2209.15189)
 - [Large Language Models with Controllable Working Memory](https://arxiv.org/abs/2211.05110) 
 - [Prompt Injection: Parameterization of Fixed Inputs](https://arxiv.org/abs/2206.11349) 
 - [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://arxiv.org/abs/2109.01247)
 - [Self-Programming Artificial Intelligence Using Code-Generating Language Models](https://openreview.net/forum?id=SKat5ZX5RET)
 - [GitHub - semiosis/prompts: A free and open-source curation of prompts](https://github.com/semiosis/prompts)
 - [Machine Learning for Big Code and Naturalness](https://ml4code.github.io/)