Anyscale and Fireworks AI have the lowest error rates (5xx, 429) and rate limits across providers
Groq and Anthropic are among the highest, OpenAI is among the lowest, Google is in-between
OpenAI has lower error rates and lower latency than Azure
They have a ~35% cache hit rate
A few quick points supporting the mental model of "LLMs are aliens". #llm-ops
LLMs are clearly not machines. They give different answers each time.
LLMs are like humans: they exhibit human biases (e.g. guessing 42 or 37 often). But they fail in unusual ways. They can't count the "r"s in strawberry. They can go into an endless loop.
LLMs are a new form of intelligence. Thinking of them as aliens might minimize our confusions.
ChatGPT's Scheduled Tasks are pretty bad at fetching the latest news. Its use of search is poor. (I'm not sure if it actually searches.) I need to figure out other use cases for it. Possible options are: #chatgpt