Lab Notes: AI Prompt Design

I’m fascinated by new technologies, and the change they might bring.

While I don’t use generative AI directly in my writing or in my consulting work—the intellectual property and privacy concerns are still too large for me—I do use it to test my ideas, argue with my premises, and summarize and filter learning material.

I’ve also built or modified dozens of Shortcuts and S-GPT-based GPT news bots, assistants, and advisors. I interact with them many times a day, every day, and I’ve been frustrated to find that most guides for getting the most out of them focus on use cases, not inputs.

Their advice is usually in the vein of “have it write your blog posts,” or “get it to teach you a new language,” instead of “here’s how to get it to check its work,” or “here’s how to get it to give you actually good ideas.”

So I’ve put together a short guide compiling what I’ve learned about getting the outputs you want from a machine you don’t control.

It’s in three parts:

1) Chain of Thought & Reasoning
2) Grounding & Accuracy
3) Creativity & Style

All examples are using ChatGPT 3.5 unless otherwise noted. To be as representative of the prompt’s utility as possible, responses are not cherry-picked. There may be one or two instances where I re-rolled a response, but in most cases these are the first responses from GPT.

Chain of Thought & Reasoning

Step-by-Step

To get better reasoning out of the LLM, get it to work out its process, step-by-step, before providing the output. You can do this in the first prompt (“zero-shot chain of thought”), or you can have the LLM output each step, one at a time.

Prompt with no chain of thought: “Write the outline for a blog post about the most overlooked content marketing strategies”

AI Response:

“Title: Unearthing Hidden Gems: The Most Overlooked Content Marketing Strategies

I. Introduction A. The significance of content marketing in today's digital landscape B. Commonly known content marketing strategies C. Setting the stage: Exploring lesser-known and underappreciated tactics

II. Leveraging User-Generated Content (UGC) A. Understanding the power of user-generated content B. Why UGC is often overlooked as a content marketing strategy C. How to encourage and curate user-generated content effectively D. Success stories of brands that capitalized on UGC

III. Nurturing a Community A. The impact of a strong community on brand loyalty and advocacy B. Why community building is frequently underestimated in content marketing C. Strategies to foster a thriving community around your brand D. Case studies of businesses that excelled in community-driven content marketing

...”

That’s remarkably bland and boring, isn’t it? And I feel like I’ve read it a thousand times before (heck, I’ve probably written it a hundred times myself).

So, let’s try this:

Prompt with a chain of thought: “Your task is to write the outline for a blog post about the most overlooked content marketing strategies. Don’t write the outline. Instead, tell me, step by step, how you’ll go about preparing the outline and what considerations you’ll be making. Remember, you do not have access to the internet.”

AI Response:

“Preparing the outline for a blog post about the most overlooked content marketing strategies requires a structured approach and careful consideration of various factors. Even without access to the internet, I can still use my existing knowledge to follow these steps:

1. Define the Objective: Start by understanding the purpose of the blog post. Is it to educate, inform, or persuade readers? Clarifying the objective helps in identifying the key points to include in the content.

2. Target Audience: Determine the target audience for the blog post. Are they small business owners, marketers, or content creators? Understanding the audience's knowledge level and interests will influence the tone and depth of the content.

3. Research Existing Knowledge: Recall my existing knowledge of content marketing strategies and recent trends to brainstorm overlooked strategies. This will involve thinking about strategies that may not receive as much attention as others but can still be highly effective.

...”

We’re off to a better start, considering our audience and having an objective. Let’s see where this takes us.

Follow-up prompt: “Now, run through these steps and provide me with the finished outline (if you cannot complete a step, move on to the next one)”

AI Response:

“Title: Unearthing the Hidden Gems: 5 Overlooked Content Marketing Strategies

Introduction
- Catchy opening hook
- Explain the importance of diversifying content marketing strategies
- Mention the goal of the blog post - to explore lesser-known but effective strategies

1. Niche Social Media Platforms
- Overview of mainstream platforms vs. niche platforms
- Advantages of using niche platforms for targeted audience reach
- Examples of niche platforms and their success stories
- How to identify the right niche platform for your business

2. User-Generated Content (UGC) Power
- The value of authenticity in content marketing
- Benefits of UGC in building trust and engagement
- Strategies to encourage and curate user-generated content
- Showcasing successful UGC campaigns from different industries

...”

While the first prompt provided similar (and in one case, identical) advice, the second prompt provided much clearer instruction on why it matters and how to write the content.

Imagine you are a writer assigned with following this outline:

I. Introduction
A. The significance of content marketing in today's digital landscape
B. Commonly known content marketing strategies
C. Setting the stage: Exploring lesser-known and underappreciated tactics

I have gotten briefs like this, and there’s so little to work with it feels arduous to even consider.

Now, try this one:

Introduction
- Catchy opening hook
- Explain the importance of diversifying content marketing strategies
- Mention the goal of the blog post - to explore lesser-known but effective strategies

This brief feels like it was written by a fellow writer, not a robot that’s read about writing.

Repeat the Prompt

LLMs can act like they suffer from recency bias, so having it repeat the prompt in its own words and consider the implications before outputting its response can often provide a more thorough and thoughtful response.

Prompt without the instruction to repeat: “I’m selling a new course on AI prompt design, but sales have slowed over the summer. What can I do to drum up extra interest on LinkedIn while acknowledging it’s a slow season?”

AI Response:

“To drum up extra interest in your new course on AI prompt design on LinkedIn during the slow season, you can take several proactive steps to engage your audience and increase visibility. Here are some strategies to consider:

1. Create Engaging Content: Share valuable and informative content related to AI prompt design. This could be in the form of short articles, infographics, or even videos. Make the content engaging and visually appealing to attract attention.

2. Offer Free Webinars or Workshops: Organize free webinars or workshops on AI prompt design. This can pique the interest of potential learners and give them a taste of what they can expect in the course.

3. Leverage Testimonials and Success Stories: If you already have some learners who have benefited from your course, ask them for testimonials or success stories. Share these on your LinkedIn page to build credibility and trust.

4. Run LinkedIn Polls and Surveys: Engage your LinkedIn audience by running polls and surveys related to AI prompt design. This will not only provide valuable insights but also help keep your audience engaged with your content.

...”

Note that it jumps straight to tactics, without considering any of the causes of the problem. If I ran off and tried these ideas, I’d be busy, but that’s probably about it.

Prompt with instruction to repeat and consider the implications: “I’m selling a new course on AI prompt design, but sales have slowed over the summer. What can I do to drum up extra interest on LinkedIn while acknowledging it’s a slow season? Before you respond, describe the problem I’m having back to me and provide any additional considerations”

AI Response:

“It seems you are facing a slowdown in sales for your new course on AI prompt design during the summer season. While this can be a common occurrence as people tend to be more relaxed and focused on vacations or other activities during the summer, it doesn't mean you can't take steps to drum up extra interest and potentially boost your sales. Here are some considerations to keep in mind:

1. Understand Your Target Audience: Analyze your target audience and their needs. Ensure that your course is addressing a problem or challenge that is relevant to them. By understanding their pain points, you can tailor your messaging and content to resonate with them.

2. Highlight the Benefits: Emphasize the unique selling points and benefits of your AI prompt design course. Explain how it can help individuals or businesses improve their AI capabilities, streamline processes, or save time and resources.

3. Create Engaging Content: Generate content related to AI, prompt design, and its applications that can capture the attention of your LinkedIn audience. This could be in the form of blog posts, infographics, short videos, or even sharing interesting industry news.

...”

Where the first prompt generated a list of thoughtless tactics, the prompt with the instruction to repeat and consider it generated a much more thoughtful output.

The first prompt gave me things to do, this gave me things to think about and then do.

Check Your Work

Not unlike humans, LLMs tend to be better editors than writers. So having it check its work and ensure that it followed the instruction can be a simple way to improve the output.

Let’s take the blog post outline from the first example and see if we can improve it.

[Response to blog post outline] “Are you sure these content marketing strategies are “overlooked”? Re-assess your outline to ensure it covers the right content.”

AI Response:

Apologies for the oversight. You're right; some of the strategies in the initial outline are commonly discussed rather than overlooked. Let me re-assess the outline to ensure it covers genuinely overlooked content marketing strategies:

Title: Unearthing the Hidden Opportunities: 5 Overlooked Content Marketing Strategies

Introduction
- Introduce the concept of overlooked content marketing strategies
- Highlight the significance of diversifying marketing approaches
- Mention the goal of the blog post - to explore lesser-known but effective strategies

1. Evergreen Content Re-Optimization
- Emphasize the potential of evergreen content
- Discuss the strategy of re-optimizing old but valuable content
- Explore methods to identify outdated content for improvement
- Provide a step-by-step guide for re-optimization
- Showcase the impact of this strategy on search rankings and organic traffic

2. Collaborative Content Marketing
- Explain the power of collaboration in content marketing
- Highlight the benefits of partnering with other businesses, content creators, or influencers
- Provide examples of successful collaborative content campaigns
- Offer tips on how to find and approach potential collaborators
- Discuss the mutual promotion and amplification advantages of collaboration

...”

Are these more “overlooked” than the first examples? How would you even measure that?

But what I love most is that the first prompts gave us outputs that would have added significantly more effort to the company implementing them (like spreading themselves thinly across niche social networks, or drumming up UGC).

These ideas, though, like re-purposing existing content or sharing efforts with other businesses, would be significantly more efficient. It’s as if, on its second attempt, it found tokes that more strongly related to the idea of being “overlooked” rather than simply rare.

2) Grounding & Accuracy

Examples

Giving LLMs specific “tokens”—examples—to base its response on can vastly improve its ability to provide a novel output. Without examples, it will tend to provide an average or “best practice” response.

Prompt without examples: “Come up with 3 ideas for LinkedIn posts to promote a course this summer on AI prompt design. Don’t provide the post text, just concepts and ideas”

AI Response:

“1. Visualize AI's Potential: Create a captivating LinkedIn post featuring stunning graphics or animations that showcase the vast possibilities of AI prompt design. Use eye-catching visuals to demonstrate how AI-powered systems can transform industries, enhance user experiences, and drive innovation. Mention the upcoming summer course as the gateway to unlocking these exciting opportunities in AI prompt design.

2. Testimonials from Industry Leaders: Craft a post that highlights success stories and testimonials from professionals who have already taken the AI prompt design course. Leverage their words to showcase the value and practical applications of the knowledge gained from the course. Include quotes, images, or short video clips from these industry leaders to add authenticity and credibility to the post.

...”

Okay, so it wants us to “use graphics” and get a testimonial. That’s not a lot to work with.

Let’s try:

Prompt with example: “Come up with 3 ideas for LinkedIn posts to promote a course this summer on AI prompt design. Don’t provide the post text, just concepts and ideas. Here’s an example idea: ‘A post about the course where every sentence is written by the AI in an extremely different style than the one before it, while maintaining a coherent narrative, demonstrating the power of skillful prompting’”

AI Response:

1. "Before and After: Unleashing the Magic of AI Prompting" - Showcase a side-by-side comparison of a poorly formulated AI prompt and its optimized version. Highlight how this simple tweak can transform the AI's output from mediocre to astonishing.

2. "AI Prompt Design: The Art of Precision" - Craft a visually engaging post that compares AI prompt design to an intricate puzzle. Showcase various puzzle pieces representing different components of a well-constructed prompt, emphasizing how putting them together with precision unlocks the true potential of AI-generated content.

...”

These ideas are not just better, they are far easier to act on. The first example gave us categories of ideas, the second one actually gave us ideas we could use.

Using Tools

LLMs are probabilistic, not deterministic, so if something has a “right” answer, have the LLMs use the appropriate tool for the job.

Note: This example was created using ChatGPT 4 and its built-in Code Interpreter.

Prompt without tool usage: “Most of my restaurant customers are happy, but I’m still getting negative reviews on Yelp. If customers who have a bad experience tell more people than customers who’ve had a good experience, what might be happening that’s causing a surge of bad reviews?”

AI Response:

“It's a common phenomenon that people are more likely to share a negative experience than a positive one. A single bad experience can lead to a negative review, while many good experiences may go unremarked. Here's what might be happening in your restaurant to cause a surge of bad reviews:

1. Inconsistency in Service or Food Quality: Even if most of your customers are happy, inconsistent experiences might lead to negative reviews. Regularly evaluate your staff's performance, the food quality, and the overall dining experience to ensure consistency.

2. High Expectations: If your restaurant has previously received positive reviews, new customers may come with higher expectations. Any deviation from those expectations, even if minor, might lead to disappointment and negative feedback.

...”

Okay, that’s a bit harsh. Apparently it’s all my fault.

Let’s see if a visualization would help clarify things:

Prompt with tool usage: “Most of my restaurant customers are happy, but I’m still getting negative reviews on Yelp. If customers who have a bad experience tell more people than customers who’ve had a good experience, visualize what might be happening that’s causing a surge of bad reviews”

AI Response:

It's common for people to share negative experiences more frequently than positive ones. This phenomenon is known as "negativity bias," and it can lead to a skewed perception of a business, product, or service. In the context of your restaurant, even if most customers are happy, those who have a bad experience might be more vocal about it, leading to a surge of bad reviews on platforms like Yelp.

Let's visualize this with a hypothetical scenario:
- Assume you have 100 customers in a given week.
- Let's say that 90% of them are happy with their experience, while 10% are not.
- Among the happy customers, only 20% decide to leave a positive review.
- Among the unhappy customers, 80% decide to leave a negative review.

...

The visualizations above provide a clear picture of what might be happening:

Customer Satisfaction (Pie Chart)
- Happy Customers (90%): A large majority of your customers are happy with their experience.
- Unhappy Customers (10%): A smaller portion are unhappy.

Reviews on Yelp (Bar Chart)

- Positive Reviews: Out of the happy customers, only 20% decide to leave a positive review, leading to fewer positive reviews on Yelp.
- Negative Reviews: Conversely, 80% of the unhappy customers decide to leave a negative review, causing a surge in negative reviews.

This phenomenon shows why even a small percentage of unhappy customers can have a disproportionately large impact on online reviews. It might be beneficial to encourage satisfied customers to leave reviews or to proactively address the concerns of unhappy customers to mitigate this effect. Strategies like offering incentives for reviews, following up with customers to ensure satisfaction, or implementing a feedback system within the restaurant could help balance the online review landscape.

Is this definitely what’s going on? Of course not, I don’t even own a restaurant. But it shows how using tools and not relying on the language model on its own can provide starkly different opinions.

Without the tool, the model told me my restaurant was bad. With the tool, it told me my restaurant was busy.

The implications of those beliefs would suggest remarkably different actions.

Non-Confidential Data

You may want to limit what you provide to the model for privacy and security reasons. But is there non-confidential data you could provide it that would improve its output without putting you at unnecessary risk?

Prompt without data: What are trends in the news right now that I should include in my weekly newsletter as an eyeglass repair business?

AI Response:

As an eyeglass repair business, it's essential to stay relevant and engage your customers by incorporating current trends in your weekly newsletter. While I don't have access to real-time data, here are some potential trends that were prevalent in 2021 and may still be relevant in 2023:

1. Sustainability and Eco-Friendly Practices: Consumers are increasingly concerned about the environmental impact of their purchases. If your business implements eco-friendly repair practices, uses sustainable materials, or promotes recycling, make sure to highlight this in your newsletter.

2. Remote Work and Eye Health: With the rise of remote work and prolonged screen time, people are experiencing more eye strain and digital eye fatigue. Share tips on how to protect their eyes while using digital devices and how regular eyeglass maintenance can enhance comfort during long working hours.

...”

Not great stuff, is it?

But with just a tiny bit of public, non-confidential, non-private data, you can get a much better output.

Here, I provided GPT with a handful of headlines from the public RSS feed of a popular tech website.

Prompt with data: “What are some trends in the news right now that might affect my business as an eyeglass repair service? Here is a list of news titles from Ars Technica that you can get inspiration from [copy/paste of the RSS feed]”

As an eyeglass repair service, there are a few trends in the news that might affect your business:

1. Australian Space Industry Cuts: The Australian government's decision to cut a $1.2 billion Earth science program has had a ripple effect on the country's commercial space industry. This has led to a loss of confidence in the sector and a reduction in investment, affecting companies across the board. As the space industry faces uncertainties in Australia, it may have an impact on companies that have business dealings or partnerships with Australian space-related ventures.

2. Trans-Atlantic Joint Venture for a New Space Station: Airbus and Voyager Space have established a joint venture to co-develop an Earth-orbiting research outpost called Starlab, aiming to field a successor to the International Space Station. While this may not directly impact your eyeglass repair service, it could signify potential opportunities for businesses in the space industry, including companies involved in supplying equipment or services to space missions and astronauts. If there's an increase in space-related activities, it could lead to a demand for specialized eyewear or repair services for astronauts or space equipment.

3. Meta's Ray-Ban Smart Glasses Struggle: Even people who purchased Meta's Ray-Ban "Stories" smart glasses don't seem to want to use them, as there's reportedly a low percentage of active users. While this specific trend may not be directly related to your eyeglass repair service, it indicates the importance of consumer demand and satisfaction in the wearable tech industry. As a repair service, it's essential to stay aware of consumer preferences and technology trends in the eyewear industry, especially if smart glasses become more popular in the future.

...”

In the race for novelty and genuine interest, this prompt beats the first one by a mile. Just double check its facts.

3) Creativity & Style

You Are

The LLM is looking for tokens that are related, like vectors in space, to the preceding tokens. Which means if you tell it who it “is,” it will look for tokens that are closer in “space” to ones used by that type of person/thing.

Prompt without a “character”: Write a 300-word blog post about the importance of creativity when writing about dry topics like B2B sales

AI Response:

Unleashing the Power of Creativity in B2B Sales Writing

In the world of B2B sales, writing about dry topics can seem like an insurmountable challenge. How do you engage your audience when discussing technical specifications, industry jargon, and complex processes? The answer lies in embracing creativity as a formidable tool to captivate your readers and drive meaningful connections.

At first glance, incorporating creativity into B2B sales writing may seem incongruent. After all, isn't this genre meant to be purely informative and objective? However, creativity should not be mistaken for exaggeration or fabrication; instead, it involves presenting information in a fresh, engaging, and imaginative manner. Here's why it's crucial:

1. Humanizing Complex Concepts: B2B sales often revolves around intricate technicalities that can quickly alienate readers. By infusing creativity, you can humanize these concepts, making them more relatable and accessible to a broader audience. Analogies, metaphors, and storytelling can help convey complex ideas in a way that resonates with your readers.

...”

You didn’t even finish reading that, did you? It was just too boring.

We can do better.

Prompt with “character”: You are a sharp-witted and funny copywriter. Write a 300-word blog post about the importance of creativity when writing about dry topics like B2B sales

AI Response:

Title: "Spice Up the Mundane: Unleashing Creativity in B2B Sales Writing"

Ah, B2B sales - the realm of excitement and spontaneity, said no one ever! Let's face it, writing about B2B sales can sometimes feel as thrilling as watching paint dry. But fear not, intrepid scribes of the corporate world, for there's a secret weapon at your disposal: creativity! Yes, that's right - the spark that can ignite even the dullest of topics and turn them into engaging, entertaining, and dare I say it, mildly thrilling pieces of prose.

Now, some may argue that creativity has no place in the world of serious business writing. "It's all about data and facts," they say, "Creativity is for poets and painters." Oh, how wrong they are! Creativity isn't just for the dreamers; it's a powerhouse tool that can cut through the monotony of B2B sales writing like a double-shot of espresso on a Monday morning.

Picture this: a world where every B2B sales piece is a carbon copy of its competitors. Dry, robotic, and as lifeless as a conference room with fluorescent lighting. Yawn! But with creativity, you can break the mold and inject life into those words. It's like giving your content a virtual makeover and turning it into the captivating belle of the ball.

...”

Genius? No. But it’s not boring, simply because I told it it wasn’t boring.

You Know

If you tell the LLM what it should know, it tends to “know” more about it. Again, because it’s looking for related tokens. So tell it it’s smart about something, and it will give you a smarter output. But don’t overdo it—if you tell it it knows something that can’t exist or doesn’t exist in reality, it’ll drfit into fantasy.

Prompt without knowledge: Draft a 200 word document discussing the risks of AI content and model collapse if LLMs are being trained on internet content that is increasingly full of AI-generated content

AI Response:

Title: Risks of AI Content and Model Collapse: A Growing Concern

Introduction:

As artificial intelligence (AI) technology continues to advance, language models like Large Language Models (LLMs) have shown remarkable capabilities in generating human-like content. However, there are mounting concerns over the risks associated with training LLMs on internet data that is increasingly saturated with AI-generated content. This document delves into the potential dangers of AI content and the risk of model collapse.

1. Proliferation of Misinformation:
AI-generated content may spread misinformation at an unprecedented scale. If LLMs are trained on such content, they may learn to produce convincing yet false narratives, leading to the dissemination of misleading information.

2. Loss of Human Creativity:
The overreliance on AI-generated content may discourage human creativity and innovation. Content creators might rely on AI-generated material, leading to a homogenization of ideas and reducing the diversity of thought.

...”

It’s fine, but there’s nothing novel or interesting in here. It honestly sounds like it was written by someone who doesn’t understand the topic and is just parroting what it’s heard.

Let’s see if this helps:

Prompt with knowledge: You are an expert in large language models and AI, as well as science communication and writing. You have a deep understanding of the implications of the technology, its uses, and its risks. Draft a 200 word document discussing the risks of AI content and model collapse if LLMs are being trained on internet content that is increasingly full of AI-generated content

AI Response:

Title: The Risks of AI Content and Model Collapse in an Era of Increasingly AI-Generated Internet Content

As artificial intelligence (AI) and large language models (LLMs) continue to advance, the proliferation of AI-generated content on the internet poses significant risks to the authenticity and reliability of information. The constant exposure to AI-generated texts, images, and videos threatens to erode the boundary between reality and artificial fabrication. This phenomenon, known as "model collapse," could lead to several critical implications for society.

Model collapse occurs when AI systems excessively rely on pre-existing patterns and data, resulting in a lack of diversity and creativity in their output. As LLMs are trained on vast quantities of internet content, increasingly filled with AI-generated material, the risk of such models becoming stagnant and unimaginative intensifies. Consequently, the credibility of AI-generated content diminishes, leading to a decline in public trust in the information presented.

Furthermore, the spread of AI-generated content might exacerbate the spread of misinformation and fake news. As LLMs ingest and regurgitate content without discerning its factual accuracy, false narratives could gain traction and influence public opinion. This erosion of truth and reliability poses a significant threat to the integrity of information available on the internet.

Look at that: One idea stacked on another in a logical flow, implying much greater grasp of the topic. The content is largely the same, but the first prompt wrote about the topic. The second one explained it.

Structured Thinking

LLMs are not great at humor or whimsy, but if you give it a format or constraint, it will have to use more creativity and tend to give you a more clever-sounding output.

Prompt with no structure: “write the last 3 sentences of a presentation about ai prompt design”

AI Response:

In conclusion, AI prompt design plays a pivotal role in shaping the capabilities and limitations of artificial intelligence systems. By crafting well-structured and ethical prompts, we can foster responsible AI development, ensuring these systems align with human values and respect user privacy. As we forge ahead in this rapidly evolving field, let us remember that thoughtful prompt design is the compass guiding us towards a future where AI augments human potential and enhances our lives in meaningful ways.

Not exactly riveting, is it?

I think we can do better:

Prompt with structure: “write the last 3 sentences of a presentation about ai prompt design: the first sentence should be a fact about the science of prompting, the second sentence should be a witty observation about the fact, and the third should be a surprisingly profound statement that wraps it all up”

AI Response:

Fact about the science of prompting: "The art of designing AI prompts involves understanding the delicate balance between specificity and generality."

Witty observation: "It's like crafting a riddle for a curious AI mind—just enough to engage, but not too much to confuse."

Surprisingly profound statement: "In the dance between prompt and response lies the potential to unlock the vast tapestry of human knowledge and creativity, revealing both the beauty of our ingenuity and the limitations of our understanding."

Couldn’t have said it better myself.