The Data Stack Grows Up: Honest Evaluation, Agentic Loops, and the Real Cost of “Done

This week’s roundup has a theme running underneath it, whether the authors intended it or not: the gap between “it works” and “it’s actually correct” keeps getting wider as our tools get more powerful. Loading data isn’t the finish line, a 94% accuracy score can be a lie, and an agent that calls tools successfully isn’t the same as an agent that’s trustworthy. Alongside that thread, there’s a steady stream of practical tutorials on the plumbing of modern AI systems — structured output, UIs, dataframes, crawlers — that make up the day-to-day of building things that ship. Here’s what caught our attention.

Start with this reflection on dbt and “analysis-ready” data, which captures a lesson every junior analytics engineer learns the hard way: getting data into a warehouse is the easy 20%. The real work — modeling, testing, documenting, making data trustworthy enough for someone else to build a dashboard on — is the part nobody puts in the job posting. It’s a good reminder that “data pipeline” projects should be scoped with that asymmetry in mind.

On the LLM engineering side, this piece on structured output with local LLMs tackles a problem anyone who has tried to get JSON out of a 7B model reliably will recognize: the happy path is easy to demo and surprisingly easy to break in production. The most useful part is the failure-mode discussion — what to do when the constrained decoding still doesn’t save you from a semantically wrong answer.

For readers who want to go deeper than the usual attention-mechanism diagram, this piece reconstructing the Transformer from first principles is a refreshing change of pace. Rather than starting from the finished architecture and explaining Q, K, and V as givens, it asks why those particular design choices emerged at all — the kind of “derive it, don’t memorize it” approach that tends to stick better than another glossary of terms.

On the applied side, this walkthrough of putting a Streamlit front end on a stateful LangGraph agent is a solid template for anyone whose agent currently only exists as a notebook cell. The gap between a working agent loop and something a non-technical colleague can actually click through is bigger than it looks, and this is a practical map of that terrain.

The eternal Matplotlib vs. Plotly comparison won’t settle any arguments, but it’s a useful framing exercise: static, publication-ready plots versus interactive exploration are different jobs, not competing philosophies. If you’re still reaching for one tool by habit rather than by task, this is worth a skim.

The “Enterprise Document Intelligence” series continues to be one of the more specific and useful ongoing threads on RAG failure modes, and this entry on “listing questions” names a failure category that’s easy to overlook: questions whose correct answer is an exhaustive set of passages, not the single best-matching chunk. Most RAG pipelines are architecturally biased toward top-k retrieval and quietly fail exactly this kind of query — a good reminder to audit your eval set for “list all the…” style questions before you assume retrieval is “good enough.”

Its companion piece, on cross-reference resolution, tackles the equally common and equally annoying case where a document literally answers “see Section 7.2” and a naive RAG pipeline just… returns that. The fix — looping back to fetch the referenced context automatically — is a nice small illustration of why “agentic RAG” is more than a buzzword when your source documents are legal contracts or technical manuals riddled with internal references.

On the model-selection front, this piece on small language models and SmolLM3 makes a case that’s gaining momentum across the industry: a well-trained 3B model tuned to a narrow task will often match or beat a 70B general model at a fraction of the inference cost. As more teams move from “which frontier model should we use” to “which model can we afford to run at scale,” this kind of task-specific right-sizing is going to matter more than benchmark leaderboard chasing.

Few essays this month are as bluntly titled as “The Problem with pandas Isn’t Performance. It’s Cognitive Overhead”, and the argument holds up: Polars and DuckDB may be faster, but speed isn’t what makes pandas syntax exhausting to hold in your head. If your team’s pandas pain points are really about API sprawl and mutation semantics, a faster engine won’t fix that — a cleaner mental model will.

This roundup of five free courses on modern AI and LLMs is a handy bookmark for anyone building out a team’s learning path — covering generative AI at work, RAG and agentic app-building, fine-tuning, and the Hugging Face ecosystem. Free, structured curricula like this are worth pointing junior hires toward before throwing them straight into a codebase.

Perhaps the most important item in this batch is “My Fall-Detection Model Scored 94%, and It Was Lying to Me”. This is exactly the kind of honest post-mortem the field needs more of: a single evaluation-design choice — almost certainly some form of data leakage or non-stratified splitting — inflated results by 25 points on a system people might actually depend on to detect a real fall. In a domain where the cost of a false negative is someone lying on the floor, this is a sobering case study in why eval methodology deserves as much scrutiny as model architecture.

Back on the builder’s side, this guide to building a natural-language data agent is a fairly complete blueprint for the “ask your database a question in plain English” pattern that every analytics team is currently being asked to ship. The interesting parts are less about the LLM and more about the guardrails needed to keep a business user from accidentally asking for something the underlying SQL can’t safely express.

This piece on hybrid AI support architectures argues for blending RAG and fine-tuning rather than treating them as competing strategies — RAG for the ever-changing knowledge base, fine-tuning for tone, format, and domain reasoning patterns. It’s a sensible corrective to the tendency to pick one paradigm and force every use case through it.

For something more reflective, this monthly “lessons learned” post, including a candid note on the downside of conference travel, is a nice reminder that the human side of ML work — burnout, travel fatigue, time management — rarely makes it into technical writeups but shapes the work just as much.

This rundown of a “minimal AI engineer toolkit for 2026” is a useful gut-check for teams drowning in framework choice paralysis: six tools, chosen deliberately, beat twenty tools chosen by hype cycle. Worth comparing against your own stack to see what you’re carrying that you don’t actually need.

Debugging agents is its own emerging subdiscipline, and this walkthrough of building and debugging a minimal tool-calling agent makes a strong case for starting with a hand-rolled loop — real API calls, explicit validation, compact trace output — before reaching for a heavier agent framework. It’s much easier to debug a system you built yourself line by line than to debug someone else’s abstraction on top of an LLM’s non-determinism.

If your team is building or evaluating scraping infrastructure for RAG pipelines, this comparison of the best web crawling tools and APIs for 2026 is a useful reference point, particularly for teams that have outgrown a homegrown BeautifulSoup script but aren’t sure which managed crawling API actually produces clean enough output to feed a chunker without extra cleanup work.

For a genuinely unusual and worthwhile read, this analysis of the Kimi K3 technical report uses a 2.8-trillion-parameter open model’s own 47-page recipe as a lens on what “building a frontier model” now actually entails. The takeaway line — that surprisingly little of the effort is “the model” itself, and most of it is data, infrastructure, and evaluation — is a useful corrective for anyone still picturing frontier AI development as mostly an architecture problem.

Rounding out the theory side, this primer on semi-supervised learning is a solid refresher on a family of techniques that’s easy to forget about in an LLM-saturated news cycle, but still highly relevant anywhere labeled data is scarce and expensive — which, for most real-world problems, is most of the time.

Finally, this introduction to GitHub Agentic Workflows, now in public preview, is worth a look for any team curious about agentic automation baked directly into their existing CI/CD and repo tooling rather than bolted on as a separate product. Whether this becomes a genuinely useful layer or another workflow-YAML rabbit hole probably depends on how well GitHub scopes the permissions model — something worth watching as it moves out of preview.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *