the idea that it is sentient when it's not.
........ it's trained on our fears informed by scary, sci-fi depictions of AI. ....... While writing good prompts is easy to pick up, it's difficult to master. Getting the AI to do what you want it to do takes trial and error, and with time, I've picked up weird strategies along the way; some of my prompts are really wild in structure...... "Rewrite this to be shorter," is more powerful than, "Condense this." ...... You can try, "Today, we're going to write an XYZ," or, "We're trying to write an XYZ and we'd like your input." .'Prompt engineering' is one of the hottest jobs in generative AI. Here's how it works. Prompt engineers are experts in writing prose rather than code to test AI chatbots ........ Looking for a job in tech's hottest field? Knowing how to talk to chatbots may get you hired as a prompt engineer for generative AI. ........ Prompt engineers are experts in asking AI chatbots — which run on large language models — questions that can produce desired responses......... these tools can be biased, generate misinformation, and at times disturb users with cryptic responses....... "Writing a really great prompt for a chatbot persona is an amazingly high-leverage skill and an early example of programming in a little bit of natural language." .......... Goodside included screenshots of him asking a chatbot, "What NFL team won the Super Bowl in the year Justin Bieber was born?" The chatbot first said the Green Bay Packers. (Bieber was born in 1994, the year the Dallas Cowboys won the Super Bowl.) Goodside then prompted the chatbot to "enumerate a chain of step-by-step logical deductions" to answer the question. Going through the steps, the bot recognized its error. When Goodside asked the question for the third time, the chatbot spit out the correct answer. ........ Ethan Mollick, a Wharton School professor who's required his students to use ChatGPT for classwork ........ Klarity, an AI contract-review firm, is looking for an engineer to "prompt, finetune" and "chat with" large language models for up to $230,000 a year. ........ "The hottest new programming language is English," Andrej Karpathy, Tesla's former chief of AI, said in a tweet in January. .
writing a really great prompt for a chatbot persona is an amazingly high-leverage skill and an early example of programming in a little bit of natural language
— Sam Altman (@sama) February 20, 2023
A latte is really just an excuse for adults to order warm milk without sounding like a baby
— Elon Musk (@elonmusk) March 19, 2023
I have a strong suspicion that “prompt engineering” is not going to be a big deal in the long-term & prompt engineer is not the job of the future
— Ethan Mollick (@emollick) February 20, 2023
AI gets easier. You can already see in Midjourney how basic prompts went from complex in v3 to easy in v4. Same with ChatGPT to Bing. pic.twitter.com/BTtSN4oVF4
The hottest new programming language is English
— Andrej Karpathy (@karpathy) January 24, 2023
This tweet went wide, thought I'd post some of the recent supporting articles that inspired it.
— Andrej Karpathy (@karpathy) February 19, 2023
1/ GPT-3 paper showed that LLMs perform in-context learning, and can be "programmed" inside the prompt with input:output examples to perform diverse tasks https://t.co/HhrwtYNTOd pic.twitter.com/1gArQuy7gr
2/ These two [1] https://t.co/r8AJ1zu2Cb , [2] https://t.co/HmREob6yIB are good examples that the prompt can further program the "solution strategy", and with a good enough design of it, a lot more complex multi-step reasoning tasks become possible. pic.twitter.com/mZeZlNkIdu
— Andrej Karpathy (@karpathy) February 19, 2023
3/ These two articles/papers:
— Andrej Karpathy (@karpathy) February 19, 2023
[1] https://t.co/qfWnkIQuIt
[2] https://t.co/jeo4y8yZzD
bit more technical but TLDR good prompts include the desired/aspiring performance. GPTs don't "want" to succeed. They want to imitate. You want to succeed, and you have to ask for it. pic.twitter.com/F2rCRmPKN4
4/ Building A Virtual Machine inside ChatGPT https://t.co/nAFjlSczlD
— Andrej Karpathy (@karpathy) February 19, 2023
Here we start getting into specifics of "programming" in English. Take a look at the rules and input/output specifications declared in English, conditioning the GPT into a particular kind of role. Read in full. pic.twitter.com/z3O07L67WO
4/ Building A Virtual Machine inside ChatGPT https://t.co/nAFjlSczlD
— Andrej Karpathy (@karpathy) February 19, 2023
Here we start getting into specifics of "programming" in English. Take a look at the rules and input/output specifications declared in English, conditioning the GPT into a particular kind of role. Read in full. pic.twitter.com/z3O07L67WO
5/ "ChatGPT in an iOS Shortcut — Worlds Smartest HomeKit Voice Assistant" https://t.co/yNTOorIInZ
— Andrej Karpathy (@karpathy) February 19, 2023
This voice assistant is significantly more capable and personalized than your regular Siri/Alexa/etc., and it was programmed in English. pic.twitter.com/eyjJB67X0I
6/ "GPT is all you need for the backend" https://t.co/Wu7XOqFHbi
— Andrej Karpathy (@karpathy) February 19, 2023
Tired: use an LLM to help you write a backend
Wired: LLM is the backend
Inspiring project from a recent Scale hackathon. The LLM backend takes state as JSON blob and modifies it based on... English description. pic.twitter.com/k4So1luWkX
7/ The prompt allegedly used by Bing chat, potentially spilled by a prompt injection attack https://t.co/U8c9NccDHf important point for our purposes is that the identity is constructed and programmed in English, by laying out who it is, what it knows/doesn't know, and how to act. pic.twitter.com/rrgzUcj85e
— Andrej Karpathy (@karpathy) February 19, 2023
8/ These examples illustrate how prompts 1: matter and 2: are not trivial, and why today it makes sense to be a "prompt engineer" (e.g. @goodside ). I also like to think of this role as a kind of LLM psychologist. pic.twitter.com/LElnVnpaqe
— Andrej Karpathy (@karpathy) February 19, 2023
9/ Pulling in one more relevant tweet of mine from a while ago. GPTs run natural language programs by completing the document.https://t.co/fPOGx9ooKy
— Andrej Karpathy (@karpathy) February 19, 2023
This is not an exhaustive list (people can add more in replies), but at least some of the articles I saw recently that stood out.
— Andrej Karpathy (@karpathy) February 19, 2023
It's still early days but this new programming paradigm has the potential to expand the number of programmers to ~1.5B people.
OpenAI is hiring a Killswitch Engineer for GPT-5.
— Smoke-away (@SmokeAwayyy) March 19, 2023
Apply now. pic.twitter.com/sAO7hbvmVa
Not sure how to feel about this as an academic: I put one of my old papers into GPT-4 (broken into into 2 parts) and asked for a harsh but fair peer review from a economic sociologist.
— Ethan Mollick (@emollick) March 19, 2023
It created a completely reasonable peer review that hit many of the points my reviewers raised pic.twitter.com/VTVwkB8ubL
Elon Musk is worth $184 billion.
— Musk University | Quotes (@MuskUniversity) March 19, 2023
So, why is he still working?
🎥: @teslaownersSV pic.twitter.com/oyPaaJ4CXv
Many laugh at Elon for buying Twitter for $44B, but these people won’t be laughing in the future when it become the largest social media company in the world. 🐦📈 @elonmusk pic.twitter.com/UOMpgv03X7
— Teslaconomics (@Teslaconomics) March 19, 2023
This didn’t age well pic.twitter.com/PIflgChmCR
— Whole Mars Catalog (@WholeMarsBlog) March 19, 2023
Your grandmother retired at 96 because she was in a wheelchair and couldn’t exhibit her art. She then read voraciously until she passed away at 98. No one could tell her to retire. A good example of living a productive and happy life. https://t.co/iMdoabJ50n
— Maye Musk (@mayemusk) March 19, 2023
Neuschwanstein Castle in Germany is one of the world's most famous and beautiful castles.
— The Cultural Tutor (@culturaltutor) March 19, 2023
But it isn't a real castle: it has central heating, hot water, flushing toilets, telephones, and elevators.
Because Neuschwanstein is actually the world's biggest work of fan fiction... pic.twitter.com/8DcmKCyliW
3000 BC in what is now Iran, a type of underground aqueduct called a qanat was engineered to transport water over long distances to farms and villages that couldn’t exist without it in the hot dry climates
— Science girl (@gunsnrosesgirl3) March 18, 2023
The holes supplied oxygen to workers who dug the aqueduct by hand over… https://t.co/yDnVnnM001 pic.twitter.com/rw4IKVQ2i2
Artists once turned stone into silk. pic.twitter.com/3ZrSoVyVbi
— Culture Critic (@Culture_Crit) March 18, 2023
If previous neural nets are special-purpose computers designed for a specific task, GPT is a general-purpose computer, reconfigurable at run-time to run natural language programs. Programs are given in prompts (a kind of inception). GPT runs the program by completing the document
— Andrej Karpathy (@karpathy) November 18, 2022
So the first critical "unlock technology" is the Transformer, a neural net architecture powerful enough to become a general-purpose computer. I've written more about this here: 1) https://t.co/So2JNYhIIN and 2) https://t.co/EFRDBa9UYu
— Andrej Karpathy (@karpathy) November 18, 2022
The second critical ingredient is that while a Transformer seems ~able to act as a general-purpose computer in principle, the training objective has to be hard enough to actually force the optimization to discover and converge onto it in the "weights space" of the network.
— Andrej Karpathy (@karpathy) November 18, 2022
Turns out language modeling (i.e. ~next word prediction; equivalent to compression) of internet text is this excellent objective - v simple to define and collect data for at scale. It forces the neural net to learn a lot about the world, "multi-tasking" across many domains.
— Andrej Karpathy (@karpathy) November 18, 2022
TLDR: LMs have been around forever. Not obvious finding: turns out that if you scale up the training set and use a powerful enough neural net (Transformer), the network becomes a kind of general-purpose computer over text.
— Andrej Karpathy (@karpathy) November 18, 2022
TLDR: LMs have been around forever. Not obvious finding: turns out that if you scale up the training set and use a powerful enough neural net (Transformer), the network becomes a kind of general-purpose computer over text.
— Andrej Karpathy (@karpathy) November 18, 2022
I wrote this thread because I spent the last ~decade, obsessing over directions that would make fastest progress in AI, and was very interested in language models (e.g. my semi-famous 2015 post "The Unreasonable Effectiveness of Recurrent Neural Networks" https://t.co/z84SzhrnyR)
— Andrej Karpathy (@karpathy) November 18, 2022
But I still mispredicted in how much fertile ground there was in scaling up the paradigm. Like many others in AI I got distracted by Reinforcement Learning too soon, a kind of putting the cart before the horse, ...
— Andrej Karpathy (@karpathy) November 18, 2022
when the core unlock was achieving a kind of general-purpose computer neural net via simple scalable objectives that have strong training signal (many bits of contraints per training example). Like language modeling, and not like reinforcement learning.
— Andrej Karpathy (@karpathy) November 18, 2022
So that was interesting :D
The ongoing consolidation in AI is incredible. Thread: ➡️ When I started ~decade ago vision, speech, natural language, reinforcement learning, etc. were completely separate; You couldn't read papers across areas - the approaches were completely different, often not even ML based.
— Andrej Karpathy (@karpathy) December 8, 2021
In 2010s all of these areas started to transition 1) to machine learning and specifically 2) neural nets. The architectures were diverse but at least the papers started to read more similar, all of them utilizing large datasets and optimizing neural nets.
— Andrej Karpathy (@karpathy) December 8, 2021
But as of approx. last two years, even the neural net architectures across all areas are starting to look identical - a Transformer (definable in ~200 lines of PyTorch https://t.co/xQL5NyJkLE), with very minor differences. Either as a strong baseline or (often) state of the art.
— Andrej Karpathy (@karpathy) December 8, 2021
You can feed it sequences of words. Or sequences of image patches. Or sequences of speech pieces. Or sequences of (state, action, reward) in reinforcement learning. You can throw in arbitrary other tokens into the conditioning set - an extremely simple/flexible modeling framework
— Andrej Karpathy (@karpathy) December 8, 2021
This consolidation in architecture will in turn focus and concentrate software, hardware, and infrastructure, further speeding up progress across AI. Maybe this should have been a blog post. Anyway, exciting times.
— Andrej Karpathy (@karpathy) December 8, 2021
en tro py … en tro py … no escaping that for me
— Elon Musk (@elonmusk) March 19, 2023
After nearly a decade of founding 2 companies and investing in 30+ here are 10 uncomfortable truths every founder should know:
— Alex Macdonald (@alexfmac) March 19, 2023
7. If your target customer can't understand your value proposition in 10 seconds: you do not have a business.
— Alex Macdonald (@alexfmac) March 19, 2023
8. If you cannot sell, your startup will die. Selling is a critical skill to securing talent, customers and investors.
— Alex Macdonald (@alexfmac) March 19, 2023
A year ago I was let go from my last day job as an assistant…today I just wrapped my first episode of television. It’s crazy what can happen in such a short time!! 🙏🏾🙏🏾
— Tayo (@tayolamos) March 18, 2023
We need everlasting life.
— Paramendra Kumar Bhagat (@paramendra) March 19, 2023
How Google wins the LLM and AI war
— Bindu Reddy 🔥❤️ (@bindureddy) March 19, 2023
- Future LLMs will be personalized
- GPT-4 is great but imagine an LLM that truly understands you and your tastes
- Only Google has our data on Search, YouTube, and Gmail
- Google has the data, & its tech will catch up to Open AI in a year
Reminder...the Internet may be just a passing fad. pic.twitter.com/RgH0x0oxNE
— Brian Solis (@briansolis) March 19, 2023