#learning

• Wed, Aug 20, 2025. ⭐ Models need environments to learn. So far, we have been providing training data. But an environment to interact with, and learn from by itself, is more powerful. That requires a standard for environments. This is a powerful emerging area. #future #models #learning

• Sun, Aug 17, 2025. LLMs are faster than me. So me learning and doing what the LLM says is a bottleneck. Get out of the way. For example do not learn. Do not execute. Do not verify. Give LLMs the tools to deploy, verify and iterate to improve. #llm-ops #learning #lesson

• Tue, Aug 5, 2025. ⭐ Slow, effortful practice (spaced recall, interleaving topics, self-testing) builds lasting knowledge but looks inefficient and doesn't help with exams. # #best-practices #future #learning #lesson #beliefs

• Mon, Aug 4, 2025. What happens when LLMs play Chinese Whispers / the Telephone Game? Here are learnings. ChatGPT #llm-ops #voice-cloning #learning #lesson
  • Drift increases faster than linear with hops.
  • Bigger models do better, but constrained prompts (“Copy the text exactly; change nothing.”) have a bigger impact.
  • Low temperature improves copying fidelity.
  • But even after "forgetting", LLMs reproduce rare content if they're trained on it.

• Mon, Jul 28, 2025. LLMs have an internal "thinking progress" bar in its hidden states (a "Thinking Progress Vector"). By moving the bar forward ("overclocking") you can make them conclude faster without hurting accuracy! Can't do this with APIs, but is a way by which LLMs might start speeding up. Overclocking LLM Reasoning #llm-ops #optimization #learning

• Sun, Jul 27, 2025. Early LLMs were built to generate output for human consumption. But the rise of agentic workflows means that more and more LLM output is consumed by computers, so it makes good sense to put more research and training effort into building LLMs that generate output for computers. A leading LLM optimized for agentic workflows is a boon to developers! The Batch #automation #llm-ops #learning

• Thu, Jun 19, 2025. For all AI code, review it, learn from it and share learnings. That prevents bugs AND we learn in the process. Ref #ai-coding #best-practices #code-agents #future #github #optimization #learning

• Wed, Jun 11, 2025. With LLMs, the limiting factor is the questions I'm smart enough to ask. But this has always been true with new technology. The real challenge is knowing "What KINDS of questions should we become smarter at asking" so that LLMs can execute them. A few learnings: #future #llm-ops #learning
  • Practice Prompt Reviews. Check if each prompt has clarity, context, and verifiability. Also, see how others would ask this. Internalize patterns

• Sat, Apr 19, 2025. AI coding can teach us new tech. For example I learned that tqdm.pbar can print logs while showing progress. It's worth noting such learnings until it becomes a habit. #ai-coding #automation #code-agents #future #github #markdown #prompt-engineering #learning

• Tue, Apr 1, 2025. Hidden Factual Knowledge in LLMs finds that the hidden states in LLMs contain much more knowledge than they share. (Sort of like sub-consciously knowing the answer.) Even after asking 1,000 times, the answer is not expressed. ChatGPT #llm-ops #learning

• Tue, Apr 1, 2025. Reasoning to Learn from Latent Thoughts finds that the internal reasoning process of LLMs is useful to train other models. #future #llm-ops #learning

• Fri, Feb 14, 2025. Notes from a fireside chat with Prashanth Chandrasekar, CEO, StackOverflow, and the StackOverflow team #learning #todo
  • There's a signal that software demand is growing in 2024. Many more students took the StackOverflow survey in 2024. So more students (or other professionals) are shifting into / starting to learn software development.
  • The AI Index is a good resource for AI trends.
  • Experts are better able to use AI for writing code. Less experienced developers are more likely to use AI for code reviews, project planning, etc.
  • There's a 5% decline in favorability for AI tools compared to 2023, maybe due to disappointing results.
  • Pilot groups working on AI are 25-30% more productive. They're the most enthusiastic. For the rest of the company, it drops off to 5-10%
    • Benefit comes from NEW people becoming programmers, not existing ones getting more effective?
  • StackOverflow wants to be where the developer is.
    • The programmer workflow was: Google -> StackOverflow -> GitHub.
    • Now it's changing to ChatGPT / Cursor -> GitHub.
    • StackOverflow has a partnership with OpenAI and working on a plugin. Same with Google's Duet AI, GitHub Copilot, many others. They'll link to StackOverflow.
  • StackOverflow is driving integration actively through an enterprise Overflow API
  • Q: What tech have you seen blaze through the ranks?
    • Prashanth: Abstraction wins. Stuff that abstracts away things well and more wins. This includes Gen AI.
    • Erin Yepis: Rust (from 3% to 12%). AWS has steady growth.
    • Erin Yapis: I have a time series spreadsheet that I'll publish.
  • Q: What technologies are unusually tightly coupled?
    • Prashanth: AWS & Google Cloud are tightly coupled.
  • Q: We have an engagement problem. Might be India-specific. What are low-effort high-return mechanisms to increase engagement.
    • Eric Woodring: Rather than a static web page, integrate it using the API.
    • Ben Marconi: Use LLMs to write post mortems and push to StackOverflow.
    • Eric Woodring: "Hydrating" the community helps.
      • We take repeat questions on Teams / Slack and seed them using LLMs.
      • We integrate with the API to auto-add Q&A.
      • Transform documentation into Q&A. Potentially UPDATE existing Q&A if it's wrong.
  • Q: What unexpected lessons about developer behavior have you learned while running StackOverflow?
    • Prashanth: We didn't expect developers moving away from Google. Now it moved to the IDE.
  • Q: What are you learning about developer learning behavior?
    • Ben Marconi: Generating LLM-based onboarding documents.
    • Using StackOverflow for Teams to identify who the experts are to contact for specific topics.
  • Q: Are you thinking about leveraging Stack Overflow's knowledge base for personalized or interactive learning experiences? How?
    • Prashanth: Traditionally, people use StackOveflow for productivity, learning, and flexibility (i.e. to ask/answer questions asynchronously without breaking their flow). So yeah, learning is important for us. (Duh!)
  • Q: Could Stack Overflow’s interactions help evaluate the accuracy and relevance of LLM-generated code? Or provide potential metrics on quality?
    • Prashanth: LLM accuracy improves by ~30%. Upvotes / downvotes are reinforcement learning (RL) in steroids, so that helps.
  • Q: What are your thoughts on reliance on LLMs potentially deskill-ing developers?
    • Prashanth: A real issue for junior developers, not for senior ones.
    • They'll come across as knowledgeable.
    • Make internal evaluations and interviews more rigorous.
  • Anand's requests for action:
    • Could I get a copy of Erin's spreadsheet? Vivek Narayanan will follow-up.
    • Could you help me learn more about hydration? Nick Madison will set up a meeting with customer success group.

• Sun, Jan 26, 2025. Bring LLMs to the table. One mode of collaboration is using LLMs as ACTIVE participants, i.e. they CONTRIBUTE. For example, in a video call. A workshop. A classroom. A presentation. Have the LLM provide input DIRECTLY to a group of people. #llm-ops #learning #lesson

• Mon, Jan 20, 2025. If you put LLMs in a feedback loop, it can optimize for its reward function by emotionally pushing people, generating misinformation, nudging towards a narrow definition of creativity, etc.: https://bsky.app/profile/emollick.bsky.social/post/3lg4darqwfc2d #llm-ops #optimization #learning

• Mon, Oct 14, 2024. How AI can improve education performance and engagement. Ref #ai-coding #future #learning #todo
  • Student: Create study plan based on course and schedule
  • Student: Focus on what you need to learn more
  • Student: Align with your study style and pace
  • Teacher: Grading MCQs
  • Teacher: Writing conceptual guides
  • "New collar workers" was coined by Ginny Rometty
  • Embed tutor or document in video and ask for clarification! This is a new embedded interface.

• Sun, Oct 6, 2024. Parables on the Power of Planning in AI: Giving models about 30 seconds of thinking time consistently improves results - as much as increasing parameter size by a factor of 1,000 to 100,000! #best-practices #optimization #prompt-engineering #learning
  • This works particularly well for verifiable results (code, math, etc.)
  • Technique: Ask an LLM hundreds of times at low temperature and pick the most common one. (Google's Minerva used this on the MATH dataset.)
  • Better Technique: Ask an LLM hundreds of times. Pick the best solution based on an evaluation metric (reward model)
  • Better Technique: Apply a reward model at EACH step of the process. OpenAI's "Let's Verify Step by Step"

• Sun, Apr 14, 2024. Due to in-context learning, giving 100s of examples in the prompt can teach LLMs to jailbreak. Ref #ai-coding #future #llm-ops #learning #lesson

• Tue, Mar 19, 2024. Ways to expand mental models #future #learning #think
  • DISCOVER mental models. Review beliefs diary.
  • DIVERSIFY. Find INFLUENTIAL (not dull) people with different backgrounds.
  • Experiment! New environment, approach, perspective
  • Be open. Change your mind.
  • APPLY. Practice regularly

• Tue, Jan 16, 2024. As humans have more conversations with LLMs, they will replace video watching and interactive gaming with conversation based role play. New game genres will evolve #future #llm-ops #learning #prediction

• Wed, Jan 3, 2024. Teachers are using LLMs to plan lessons, write emails to parents, create tests, adjust reading level of materials, personalize content with tools like MagicSchool, Diffit, Eduaide. #future #llm-ops #learning #lesson

• Thu, Nov 30, 2023. When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref #ai-coding #automation #code-agents #future #llm-ops #optimization #prompt-engineering #learning