Before the dawn of LLMs, I would often write little bits of software or small apps to make my life easier. Hardly a novel concept, programmers have been doing this for an age. It’s exhilarating to automate the boring or difficult parts of life!

Humans can finally extend their reach, their leverage, their power out past the bounds of space and time!

I built my entire career out of sharing my own methodologies of software engineering. I put code and ideas out into the world for free, other people liked them (many did not), and I landed myself a job where I got to focus a vast majority of my time on my passion projects.

And I can’t shake the feeling that this really is the end of an era. I built a community and a platform along with an open source framework. Through that community, through the world of open source, I can see that critical mass has been reached.

Funnily enough, LLMs are only one part of that equation. A big part, sure, but software has been trending easier since its inception. Small teams have been building wildly impressive software for many years now. To me LLMs just feel like the last chapter of a story that has been winding down for a while. We may still have many years ahead of us...but one way or another...

In the occult literature, conjuring is somehow both technical and magical. Your pentagram must be drawn just right. Your circle of salt mustn’t be broken. You must say your incantation just so, else you risk the wrath of a paranormal entity. You can’t give a demon too much information about yourself otherwise it will have power over you. Your instructions must be so clear that they cannot possibly be turned back on you by a malignant entity.

Starting to sound familiar?

I’ve found myself to be quite effective at the use of LLMs to build software, and the latest iterations of Claude and ChatGPT are only enhancing that fact. I am extremely skilled with my own suite of tools that I built for building these kinds of things. Because of that, I can give an LLM very clear instructions about what I want built and how I want it built.

The thing is that it really doesn’t feel like engineering any more. I feel instead that I’ve gotten really good at my spellwork. I’ve studied the ancient lore of “functional programming” and “abstraction”. My language of choice is called Elixir for christ’s sake. I’ve learned to draw very straight lines on my pentacles (the boundaries I give to the LLM). Place this here, place that there. I know where it will try to make naughty choices and how to head it off before the little machine demon can mess something up.

I’m starting to feel like a software conjurer, and honestly I have no idea how I feel about that.

There is a small but critical aspect missed in that line of thinking.

Right now, SOTA model API providers are cheaper and easier to use than rolling your own. So we use them. We’re getting a bunch of free tokens it would be stupid not to. But the fact that for ~$20k I can have all the hardware I need to roll my own LLM stack and “approximate” the benefits of what I currently get from Sonnet 4.1 is all that needs to be true for us to have a massive security blanket.

The cost of using a SOTA model is capped to some function of the time and money it would cost me to set up my own stack. If prices begin to rise, at some point, a $20k down payment for my own state of the art hardware and the time involved in setting up my own ML stack will become “cheaper”.

That side of the equation makes Nvidia just as much money, if not more money due to the lack of bulk purchasing, so it will always remain viable at some price point. The OSS models continue to advance on their own independent market and geopolitical forces.

I’m not going to waste my time setting up local models and OSS agents. I’m just going to benefit from the fact that SOTA model API providers margin can never exceed the amount of money it would take me to tell them to fuck off and roll my own.

But what about all of the cool features they have that are hard to get out of local and OSS models?

One other factor here that we’re seeing emerge is that so many LLM products differentiate themselves on their “harness”. i.e system prompts, agentic loops, stitching together disparate models. There are already tons of services that help you approach SOTA model performance with local models by combining different models and approximating similar behavior on weaker hardware.

The bottom line is that OpenAI and Anthropic and Gemini are saving you time and money, not providing you a service that you can’t get via similar means working directly with hardware providers. The moment they aren’t, you go straight to the source and work directly with Nvidia and the latest open model. Also, buy Nvidia stock 🤷‍♂️.

A note to the Elixir programmers who commonly say “let it crash”: This phrase gives outsiders and newcomers the wrong idea, and encourages bad habits for those who misinterpret it. If I had my way, we would stop saying it.

Elixir runs on the BEAM VM. If you’re familiar, skip to the next paragraph. Otherwise, some necessary context is that on the BEAM, all code is executed in a “process”. This can be thought of kind of like a green thread. Each process is a lightweight, share-nothing unit of concurrency. Processes communicate via message passing. When processes encounter an unhandled error, the process exits. We use supervisors (which are themselves supervised typically), to then restart any given crashed process.

When people say “let it crash”, they are referring to the fact that practically any exited process in your application will be subsequently restarted. Because of this, you can often be much less defensive around unexpected errors. You will see far fewer try/rescue, or matching on error states in Elixir code.

For those without the context of building Elixir applications, “let it crash” gives the impression of “jank”, of non-elegant code that does not consider user experience or possible failure states. A panic or crash in most languages is a catastrophic event, to be avoided at all costs.

Even for Elixir developers with experience, “let it crash” as a practice loses enough nuance so as to be reductive, and I’ve seen it actively impact the quality of code that comes across my desk.

First, we’ll talk about why you shouldn’t “let it crash”, and then we’ll talk about what you should do instead.

Processes are tied to real things

Open a socket connection, receiving or sending an HTTP request, opening a file, connecting to a database: all of these things are typically backed by a process. 99% of the time, that process is linked to the process that asked for that to happen. The most straightforward example of this is in the context of a web server communicating over a web socket, and if you’re familiar with it, Phoenix LiveView.

Lets imagine that we embrace the “let it crash” philosophy to the bone, and we write some code like this, handling a web socket message:

This code will crash the process in the following scenarios:

• the pattern match on the event name fails `”import_data”`

• the pattern match on the input params fails (there is no “data” key)

• `data` is not a list (technically, an enumerable of some kind)

• Any given item in data fails to be created or raises an error

In some cases, especially “invalid messages that should not be producible”, crashing is a good thing. For the first two, and maybe three things in the list above, hitting those cases implies a bug in your front end likely. Someone sending your process messages it was never designed to handle. Crashing may be the desired behavior.

BUT: There is a user on the other end of this. In the case of LiveView, the web socket is driving at least some part of their experience. Establishing a websocket takes time. What will the user experience be? Will they see a flash message about something going wrong? Will some of their UI state reset? None of these things are really acceptable. This pattern is all over, not just tied to LiveView. Do you want to close whatever file/database connection you have every time anything goes wrong? Or release some other expensive resource? What if you encounter thousands errors in quick succession? Will restarting be expensive?

Representing Failed States

Intentionally crashing on any possible error also means that you are not able to represent errored states to your users. For our above example, something like this might make more sense:

In this example, instead of just crashing and seeing some generic error, the user is given actionable information about what went wrong. We have represented the failed state.

With that said, in practice we take a middle-way. Specifically, in fully unexpected scenarios that cannot be remediated, like getting a malformed “import_data” command, it is often preferable to crash. We crash because the process is now in an unrecoverable bad state, and we don’t know how it got there. In this case, we want to remount the component into a good state. You still have to consider the UX of this scenario, and design your pages to gracefully remount.

Here we crash on unknown or garbled messages/input, but present validation errors to the user.

So, we’ve established that there are consequences to crashing, and that we should not default to the overly reductive “let it crash” mentality here. Does that mean that the “let it crash” people entirely wrong? Only kind of.

Let it heal

Remember when we talked about supervisors and restarting etc? The operative part of these things is not that processes are designed to crash, it is that they are designed to start. As in, when your application first boots up, you have to start up your process tree, and supervisors are at that point embedded with the information required to start a process if it crashes.

What this means for Elixir programmers is that we can let it crash when things are unrecoverable. What this does not mean for Elixir programmers is that all errors anywhere inside of a process should cause a crash.

The real magic of the BEAM is that for any given piece of code running in Elixir, there is another, higher level piece of code that knows how to handle errors that cannot be locally handled by that code.

You can’t write code that isn’t aware of the fact that something might go catastrophically wrong, because all of your code implicitly has a “how do I initialize myself” step that must be able to gather any requirements and “set the stage” for itself.

There is often still work that goes into designing these structures, but we are getting something for free or close-to-free that takes a crack team of experts or specialized frameworks to achieve in other languages.

While “let it heal” may not be as catchy, I think it better captures the BEAM’s true superpower: not that our processes can die, but that they can always come back to life.

In some ways, it feels like we’re living in the year 3000, but in other ways it’s like the stone ages. There are people succeeding past their wildest expectations with agentic tooling, and there are people trying and failing to find any success with these tools. OSS is about leveling the playing field, so let’s level it.

Right now, we have the ability to automate and share the vast majority of agentic development tools, but there is one major missing piece. Folks are using things like claude code, codex, gemini etc. which roll forward as new tools and best practices arise. We also use things like MCP servers to distribute tools. While some MCP servers are game changing, I have lots of gripes w/ MCP, but thats a topic for later. The underlying software and libraries that agents work with is also distributed via package managers. All of it can be iterated on and everyone can update to benefit.

However, the most crucial aspect of agentic development is currently left for everyone to figure out on their own! We can give our agents access to tools which they can use to populate their context window (i.e prompt), but the prompt that we actually start with affects everything down the line.

Whether or not you realize it, all of these agentic coding tools have their own, typically hefty, system prompt designed to make an agent better at operating within the framework and tools that they give it. Most folks realize that system prompts are both impossible to protect and also not a realistic moat, so many are published. However, as soon as you type into the chat box, you’re doing what many call “prompting”, but what I think of as “context management”.

Context Management

Ultimately, LLMs take some context, and produce some subsequent text. Its wild when you boil it down, but that is how it all works. Some special JSON text response formats indicate tool calls. “thinking” is just indicated by wrapping some text in <thinking> tags. Success with LLMs is figuring out this weird, difficult to predict and “squishy” concept of context management. While complex in practice, it really boils down to “what text in will produce desirable text out”.

While the concept of “vibe coding” is honestly ridiculous, it does nail one fascinating aspect of this whole thing, which is that context management, in my experience, really does involve intuition. Its not realistic to think about the mathematics or the vectors etc. It feels more like a soft skill, like a people problem, than a technical one.

As an OSS software engineer, a lot of my job is to lift up the community and engineers around me, and what I see currently is that they are being left behind. I’m not talking about LLM skeptics. I’m not looking to shove LLMs down anyone’s throat. I’m talking about less experienced engineers, or engineers who don’t have the time to discover and hone this new skill. They feel that there is some new state of the art out there, but that it is inaccessible to them because they haven’t figured out how to wield it.

Usage Rules - Breaking down the silos

The solution for this is frankly simple, and leans into the patterns we already have in place for software engineering. I’m calling it “Usage Rules”. The concept is simple, distributed with any given software package or tool, there is a `usage-rules.md` or and/or a `usage-rules/` folder with markdown files inside of it. These usage rules are not general documentation. It isn’t `llms.txt`. Instead, it is “what the authors of the tool think you should put into your agent’s context window to successfully use their package”.

For large frameworks, usage rules might be really big. For smaller/simpler tools, it might just be a few lines. For end users, even just a few lines can make a huge difference. Then we use tooling to synchronize those usage rules into the appropriate places (i.e `AGENTS.md`, `CLAUDE.md`) on the user’s file system, in such a way that it can be kept up to date over time. I’ve implemented this tooling for the Elixir ecosystem and I’d strongly encourage other languages, frameworks and library developers to follow suit, using the same file naming conventions.

In practice, we use it like so:

This synchronizes some builtin rules for Elixir & OTP, as well as any usage rules from your direct dependencies. There are lots of different ways to use usage rules. Perhaps you want some inlined into your AGENTS.md and some to link to the dependencies folder. Or perhaps you want to copy all rules into a folder and link there. Or perhaps you want to use CLAUDE.md style `@file/link`s. All available as options to `usage_rules.sync`!

All reports show effectively a night and day difference in folks’ experience with agents. The more off-the-beaten-path, or the more recently released what you’re working with is, the more effective thse tools are. Software changes a lot more often than LLM’s reindex the internet, so you have this problem even if you are using the most popular tech that exists.

On a fresh project, this yields a pre-loaded and useful AGENTS.md

What are the benefits?

There are numerous benefits, but lets go through some of the major ones:

• End users can get “free” context that helps their agents succeed with the tools they use.

• These can be collaborated on. When agents keep making the same mistakes, or not choosing the correct tools for the job, you can make a PR to the usage-rules.md!

• usage rules are updated with the project. When you upgrade your dependencies, simply re-sync your usage rules, and your agents are using the latest features, no matter when the last knowledge cut-off date is.

Taking it even further for Elixir

The Elixir usage_rules package provides tools that allow agents to be successful with no need for additional MCP servers. This is another critical component to making this stuff easy to use and accessible to everyone. We distribute two tasks, `mix usage_rules.search_docs` and `mix usage_rules.docs`, which allow the agent to search Elixir’s centralized documentation repository across all packages you are using in your app(or for specific packages). The built-in usage rules instruct the agent on how to run these tasks effectively.

Instead of making repeated shots in the dark, agents now have a fool-proof method of discovering their mistakes and what they don’t know! No need for an MCP server or any particular agentic tool. They can all call bash commands.

Rising Tides

I hope that these patterns and this type of tooling catches on, because rising tides raise all ships. We’ve been doing this for decades in OSS and new paradigms don’t change that prerogative.

To learn more, or to get started using it, see our docs at hexdocs.pm/usage_rules

I’m just going to dive in. Lets start with some guidelines

Guidelines

Pay attention

Step 1 is to realize that this stuff changes so ridiculously fast that you’re out of date the moment you use some new tool or pattern. There are two things to keep in mind:

Read the news

Stay up to date, especially with provider updates like Anthropic, Open AI, Gemini etc. They are playing tennis with who has the best coding model at this point, but for me Claude 4 has blown everything else out of the water currently.

For example: Claude 4 Best Practices

Stop reading the news

Don’t forget we’re here to get shit done. Spend some time refining your setup, keeping up with the world, sure. But don’t forget to act. A big part of working with LLMs is this weird new prompting/chat soft-skill. You have to hone it and refine it.

Don’t test the machine

I often did this myself starting out, which is shoving some half-assed “fix X bug” prompt at Claude to see if it would do what I wanted it to do. Protip: it’s not a person. It's a text machine, a word cannon, that takes text and makes text. We’re just learning how to use it in all these fancy ways. If you have a thought about things like:

• where it should probably look to find something

• what the root cause of the bug is

• an edge case it should consider

TELL it that stuff. Occasionally when working on tools for other developers to enhance their experience with LLMs I’ll try and prompt it like a doorknob, but when I’m actually using it to get shit done, it gets as much info as I can feed it.

Learn when not to use it

You’re going to have to build up instincts around when you have the kind of problem an LLM will help you with. No one can really do this for you, you just have to try it and see. It gets easier over time.

Tools

Zed

I personally much prefer zed over Claude Code, which is the other contender here. Their UI is excellent and has a whole bunch of whistles and bells that you’ll appreciate more once you figure things out. I still use Claude models, but I don’t use Claude Code.

Zed Pro

I don’t know how long this will last, but Zed’s pro plan gets past Anthropics rate limits. It costs quite a bit once you get past your allotted prompts. Personally, I value my time at a ridiculous premium, $500/mo is chump change for something that can make me 20-30% more effective. No hard numbers on how much more effective it is making me, but the potential is there. Some days it wastes more time than it saves, and sometimes it saves me 6 hours in one fell swoop. I’m learning to optimize for those latter cases.

MCP Servers

Personally, the only MCP servers I care to use currently are the Ash AI dev MCP server and Tidewave. Why do I need a GitHub MCP server when it can use `gh`? Besides, most of the time I don’t want it screwing around with that stuff. How I conduct myself on the internet is personal, not something I’m interested in delegating to a machine.

Rules Files/Context

Global Rules

I have two rules files that apply 100% of the time. You should have some too. They aren’t amazing, but they are, so far, getting the job done. The way that you apply these depends on what you’re using. Claude expects `CLAUDE.md`. Other things AFAIK are looking for `AGENTS.md`. In zed you manage them in a rules GUI, but it also looks for `AGENTS.md` too.

non-negotiables.md

guidelines.md

Usage Rules

Crowdsource your rules. Teach your agent where to find rules on how to use stuff (i.e docs). In Elixir, and with Ash Framework, we’re working on how this can work, and we’re pushing a pattern called usage-rules.md. I can run one command to create “breacrumbs” telling agents where to look for the rules of packages that I want it to use.

Situational Rules

In Claude Code, these are done via slash commands. With Zed, it’s under the agent three-dots button in the top right. Click on rules. You can select which rules always apply by marking them as default rules. Then, to use them, you type `@rule` and it will give you an autocomplete of your rules list.

I have three situational rules files currently, which map to workflows that I want the LLM to follow depending on what we’re doing.

feature-workflow.md

fix-workflow.md

task-workflow.md

Which brings us to:

Prompting

At the end of the day, all your rules just get put into the prompt/context that you send to the LLM. Some things to keep in mind:

LLMs do not KNOW anything

Make your LLM do research, and load that research up into the context window, token use be damned. Make it very clear that they are to research heavily.

Collaborate on a plan file

Have it build and collaborate on plan documents. When it’s made a plan, open it up, add a bunch of todo comments to the plan, and tell it to review (or just tell it what to change). This planning step is crucial because what you’re actually doing, is collaborating with the LLM on a better prompt for the LLM. The plan document performs so many useful functions that it is hard to count.

• Keeps the LLM on track

• The plan can be resumed at any time

• The plan can even be committed or shared as a trimmed down version of what you and the LLM did. Will humans read it? Probably not. If only we had a magic machine that could do a whole bunch of cool shit if given a bunch of relevant text…

• Acts as a clarified feedback mechanism for the work the agent is doing.

• When you have it load up things like “risks” and “unknowns” into the plan, you’re priming it to avoid mistakes but also you’re informing your *own* engineering process.

Do <this> like <that>

LLMs are amazing at “translation” and “reformatting”. So if you can point it to some code that is “kind of” like what you want right now, you’re going to get wildly better results.

Iterate on your rules

When you notice that the agent is doing something you don’t like often, or is repeatedly not doing something you want it do, edit your rules file. Bonus points for having the agent propose changes to the rules for you. Claude Code is good for this, zed is not since the rules don’t live in the codebase or in a place that its easy to point the agent at as far as I can tell.

Use Ash Framework

This is just a plug for my other tools, not really relevant to the above. If you’re interested in my other work, see:

• Ash Framework

• The Ash Framework Book

• The new Ash AI package for building AI enabled applications

Feedback

Like these ideas? Let me know?

Have some constructive feedback or ideas? Lets chat ❤️

Hate everything about this post? Make sure to smash that like and subscribe.

If LLMs turn out to be a fad and you’re reading this after they jump the shark, please accept this article as a historical artifact from a time when we entertained the idea that fancy autocomplete might eat the world.

Aside, for those who don’t know what Elixir is: it’s the best general purpose language in the world and if you haven’t heard of it wake up and smell the bacon.

Will “AI” force a convergence towards the tools it “knows” best?

The argument goes like so: Now that you have a bunch of folks who are “vibe coding”, they will just do whatever ChatGPT or Claude tell them to do. They will also be evaluating things almost solely against “how well did my ‘AI’ do with it” for making decisions going forward. For Elixir, this sounds like bad news bears. Here is a portion of a very long-winded response from ChatGPT when I asked it how I could “build a website like Amazon”.

Oh no! I in my ultimate wisdom know that this is nowhere close to the best stack for the job. But the noobs won’t know that at all! Let’s say instead they’ve heard of Elixir on a blog post, or they saw this article topping the hacker news charts 🤞 (hi mom!).

I’m in this unique situation where time-to-market happens to matter. Also what do you mean “learning curve”, you’re supposed to do the learning for me. Thanks Chat, you really helped me dodge a bullet here, guess I’ll stick with Node & Next.js, thanks! 🥰

Lets not beat a dead straw man

Mixing metaphors like a boss. While I’m sure the above scenario will play out countless times, at the end of the day this isn’t new. Theoretically this all goes one of two ways.

1. “AI” is able to overcome the fundamental issues with any tool, thus equalizing effectively all programming languages.

2. Vibe coders & new programmers working with LLMs hit the same walls that drove Elixir to exist in the first place.

In my opinion, both of these realities are “good news” in a way. In scenario 1, we’ve got magical mystery robots and they’ll just create the one programming language to rule them all and now we all have infinite automation in our pockets. Let’s talk about scenario 2, the one I find far more likely.

Hitting Walls

It’s no surprise that these things will mirror popular opinion. We can finagle our way there. This is a fresh prompt.

After giving me a bunch of sensible recommendations like using a CDN, horizontally scaling, using background jobs, migrating from Vercel to ECS, and scrolling past the recommendations to rewrite in Go and Rust, we finally found what we were looking for 🥹.

This brings us to a crucial realization: if LLMs can successfully guide someone toward Elixir when the use case demands it, then perhaps the real challenge isn't LLM bias toward mainstream tools - it's making sure LLMs can work effectively with our tools to enable developers to make that choice.

Alright, I’m up and running. Not sure how this would cost someone $10k on Vercel but I’m sure someone could find a way.

So let's test this theory. If I take Claude's advice and decide to explore Elixir, how well can it help me actually build something?

Not only does it work, but it does it effectively the same way that I personally would have done it if given the same task.

LLMs are quite good at translation. Where they often fall down with invention, this kind of thing is their bread and butter. It even kept it compatible with the node server I started with.

Plus some goodies:

Now, this doesn’t necessarily mean that non-mainstream technologies will be naturally surfaced over time, but, well…

If LLMs never get good enough to do things like troubleshoot esoteric technology and learn new things that weren’t in their training set, then they’ll fade into irrelevance anyway. So let’s extend this line of thinking out. If you think (like we probably all do) that your soup du jour is better than the other guy’s, then guess what? Either the LLMs will get “smart” enough to realize it too, or you’ll eat all the vibe coder’s lunch because they never figure it out.

Okay, so we’re all good then?

Yes and no. I know there are a lot of people who will hate what I’m about to say. People I respect, and whose opinions I value despite how incorrect they are on this front. “AI” is going to significantly change what it looks like to build software. I’m not worried about how I’m going to stack up against the “Giving into the vibes, embrace exponentials, and forget that code even exists” folks, but I am aware that there are significant new tools & variables at play here. Not factoring them in is how you become a dinosaur. Which, for what it's worth, is fully your prerogative.

With that in mind, if we want our tools to succeed in this space, we need to invest in it as well. Not exclusively, maybe even not significantly. But being relevant here is non-optional IMHO.

I’m the author of an application framework for Elixir called Ash Framework. This article is not about Ash, but my work on it and the support that I provide for it have given me some potentially unique insights. Over the last few months I’ve observed:

• a new item on the rubric for tools: the quality of answers from Claude & ChatGPT about it

• a huge increase in fully off the wall and weird questions driven by LLM hallucinations

• methodologies for working with LLM agents that are making me wish *more* of my users used them, not *less*

The new rubric

People are finding that the magic word smusher gives surprisingly good results for things like Typescript & Rust. All of a sudden, they have a new thing to factor in when making their decisions. FOMO is a powerful thing, so I think this plays a bigger factor than we’d like. Folks are worrying even more than ever about the future-proofing of their knowledge in the context of a major industry shift. If I invest my time to learn something that “AI” can’t help me with, will I be left behind? It’s a very real and important question, and I think to succeed these days, you have to alleviate the concern in some way.

The new headache

Fairly often these days I get questions about code that no one would ever have had a reason to think should work. These cases are clearly coming from folks asking LLMs how to do something, or using an agentic assistant. The first knee-jerk response here is to tell people that we don’t answer questions about “AI” generated code. And in some cases, if it’s clear that the person asking the question did zero research on their own, they’ll get a response like that. But what if there was another way?

Context is Key

Ultimately you should not be relying on an LLM as a source of “knowledge”. This is true effectively always, even for popular tools, because LLMs won’t “know” about recent changes, but is magnified when asking questions about things that are more esoteric.

The key to succeeding with LLMs is to treat them as if they are only capable of doing things like summarization and pattern transformation. Assume that, before extracting an “answer” to your question, you must first do something that would *place* that answer into its context window, making its job the summarization and/or reformatting of the information into some desired result/effect. Here are some real things you can do to this effect.

Research & Investigate

Here is a concrete example of using LLMs “badly”.

Looks plausible enough right? But as you can guess, it’s just flat out wrong. Let’s try again, but better this time. I’ll cheat a bit because I know where the docs that it needs are.

Not only is this correct, but it continues on to provide advice that would likely be easy to miss if skimming the docs. See the full answer here. All of a sudden, I’m not in the world of “please stop using LLMs to answer your questions”, I’m in the world of “please stop using LLMs badly to answer your questions”. I knew where to find this documentation, but what if I didn’t?

Have a look at another way of asking this question. First, I ask it to find the relevant docs. It does a great job. If you read the linked conversation, I then follow that up with the same question as before, and I get a correct answer without ever leaving Claude’s GUI.

Bonus - Use Tidewave

Tidewave is a server that speaks MCP (Model Context Protocol) that you can serve directly from your application. Think of it as giving your LLM a direct hotline to your running code - not just static documentation, but your actual living application.

Here's what this looks like in practice. Instead of asking Claude "How do I create a user in this app?" and getting a generic answer that might be outdated, Tidewave lets the LLM use tools like:

• project_eval - runs code snippets directly in your application context

• run_sql_query - executes database queries using your app's actual schema and tooling

• search_hex_docs - searches the documentation of your specific package versions

This means your agent might do things like search the hex docs of your dependencies to see how Ash works, use the `project_eval` tool to poke around in your running app, try creating a user using your app to see if it works, and then run a SQL query after the fact to make sure the user exists in the database. Wildly powerful stuff.

Make me an X like Y

This one is easy, and you can try it on your own apps today. Often when you end up building new features, they bear resemblance to one or more of the other things in your application. LLMs are very good at synthesis, summary, and categorization. So put them to work at it! A really excellent prompt to kick off an agent’s task often looks like this:

I'd like to make a feature to track user activity. Let's start with "last_login_at" and "last_seen_at". To start, we'll need a new plug in our router, kind of like @/path/to/similar/plug.ex, except instead of X it does Y. For the fields, take a look at `last_updated` for an example of the kind of field we want, except this one should be accepted as input when updating a user.

Crowdsource your context

I'm trying to push a new pattern for Elixir packages that I think could be game-changing. The concept is simple: any library can include a usage-rules.md file - essentially very terse documentation designed specifically for LLM context windows, explaining what to and what not to do when using the library.

We've done this for the main Ash packages, and the transformation is remarkable. We went from LLM agents being practically useless for Ash development to being able to generate idiomatic, production-ready code. The dichotomy of "meet my standards" versus "let the LLM do it" simply disappeared.

This isn't just about Ash or even Elixir. This is a template for how any technology community can turn LLMs from a threat into their most effective growth tool.

Evals: Training the Trainers

We can go even further. Instead of just helping LLMs use Elixir better, we can help train them to be better at Elixir in the first place. This is something we discussed at the contributors summit, so it is the product of many minds, not just mine.

The Elixir community could build evaluation datasets that test real-world scenarios: GenServer supervision trees, OTP fault tolerance patterns, Phoenix LiveView reactivity, writing Ecto queries, building applications with Ash. When companies like Anthropic and OpenAI train their models, they need diverse, high-quality benchmarks - and right now, they're mostly testing on Python and JavaScript.

If we get these evaluations to a degree of quality and scale, we get a bunch of benefits. We can determine which models are the best at writing Elixir. If done well enough, companies like OpenAI and Anthropic may even opt to use it to train their private models (something we should encourage in this scenario). You can find collections of benchmarks in various places, like: Hugging Face and GitHub. I see plenty of python stuff there, but not no Elixir 🤔.

Here's the kicker: Elixir's runtime makes it perfect for performance-based evals. We can build harnesses that don't just check if code compiles, but whether it actually handles concurrent load properly, recovers from crashes gracefully, or manages memory efficiently.

As models get better at Elixir through these evaluations, they become better advocates for it. We would be going beyond just making LLMs write better Elixir - we'd be making them recommend Elixir when it's the right choice.

If I can make LLMs a force to be reckoned with not only for Elixir but for the niche-within-a-niche that is Ash Framework, I think it's safe to say that LLMs are not, in-and-of-themselves, an existential threat to Elixir.

In fact, I think we're looking at this backwards. The real opportunity isn't surviving the LLM revolution - it's using it as a force multiplier. Better technologies have lost adoption battles because they had steeper learning curves or less accessible documentation, but maybe LLMs can flatten that learning curve if we do the work to make our tools competitive in this space.

The question isn't whether LLMs will change how we build software. They already have. The question is whether we'll shape that change or let it shape us.

I’m really proud of this book. Rebecca Le is an amazing writer and working with her has been extremely rewarding and fun!

In other big news, I’d like to congratulate her on joining the core team! She has earned it time and time again, even without her amazing work on the book. The book, which happens to be one of the most significant contributions made to the Ash ecosystem ever.

New Weekly Newsletter

We’ve launched the Ash Weekly Newsletter! Its the best place to stay up to date with what is happening in Ash news 😎

Ash Weekly

A weekly the goings on of the Ash Framework ecosystem.

By Zach Daniel

New Website

The AshHQ website has been completely revamped, both for a modernized look but also to make room for…

New Installer

We’ve got an awesome new installer experience available on the new website! It’s really flashy and fun to use, and something that we will continue to evolve over time. It acts both as an overview of all kinds of capabilities in Ash, but also to highlight the modularity of the ecosystem, and the way that any thing can be installed later. No need to decide up front what you want. (Still have some planned changes to make that clear in the installer)

Thank you

To those at Alembic and our technical reviewers thank you! To those who have already given us such great feedback on the book thank you! To those who encouraged us and helped us along the way, my wonderful wife and family, thank you.

I hope that you all enjoy the book. If you pick up a copy I’d love to hear your thoughts.

The word immutable is defined by Merriam-Webster as “not capable of or susceptible to change”. If nothing ever observes a change, it may as well have not happened anyway, so we will be discussing mutation as two discrete concepts: mutation, and the observation of mutation.

Here is a simple example of a mutable program in Javascript.

Here is a simple example of an immutable program in Elixir.

They look very similar. And in fact, they both have consistent behavior if the next line of the program was to print out the value of counter. We are guaranteed in both cases to see the value 1 printed. It would appear that the observation of variables behaves the same.

Rebinding vs Mutation

Before I explain the benefits of immutability, it’s important to disambiguate something that folks often initially believe is mutation.

In the Javascript example, our reference to the counter is mutated. However, in the Elixir example, what we’ve done is rebound the variable. No pointers or values have changed. In Javascript, primitive values are immutable, but a variable’s reference is not. In later examples we’ll see how Javascript allows mutating other values i.e objects, called reference values.

The first is an actual mutation of memory. The second is an allocation of new memory. The old memory is unaffected. One could argue that this is a form of mutation. This mutation, however, is syntactic.

This is a convenience that the language provides. When you rebind a variable, it is equivalent to creating a new variable, and altering references to that variable in the same scope to use that new name instead. Let’s look at what this means in practice.

In Javascript, we mutate the actual underlying memory. What this looks like in practice:

Notice how the value for counter is now 1. In Elixir, however, this behaves differently.

How can this be? We’ve clearly reassigned counter. The reason for this is that rebinding is equivalent to creating a new variable. Just like in Javascript, a variable is not available outside of the scope it was defined in. The Elixir code is effectively rewritten to this:

In this way, no reference to a given variable can ever truly be mutated in Elixir. You only ever create new bindings with new values. There is just a language level construct that allows you to reuse the same variable name.

Cool, so why immutability?

Now that you understand that rebinding isn’t mutation, you can see the fact that the value of a given variable will never change. This lends itself to well to understandable code. For example, in Javascript, objects are mutable, and are passed to functions by reference. Which means that calling functions can do “surprising stuff”:

After executing this code, profile has mutated. Before calling newScore, profile.score was 3. Afterwards, it is 7. This would be a pretty beginner mistake to make, but it is still something that you have to contend with. You will always have to consider using things like Object.freeze or Object.seal, or just trusting that your other code makes no mutations to the parameters that you pass in.

In Elixir, it is just not possible for that to happen. If I call a function, it can do all kinds of side effects, but it will never be able to affect the values of the variables that are currently in scope for me.

This is where observation of mutation comes into play. In the Javascript example, mutation can be observed effectively at any time. If I started an async function, for example, and passed in a mutable variable, it could change at any time.

Time: the most frustrating part of programming

And I don’t mean dealing with time zones. Although maybe that actually is the most frustrating part of programming.

What I mean is handling the passage of time. Let’s go back to our Javascript example, and introduce some concurrency. When concurrency comes into play, we can no longer think in terms of a discrete, causal series of events. Instead, we have to think of “state over time”.

The value of profile.score at the end of this program is entirely dependent on how long it takes to doSomethingElse(). That means that sequential runs of this program can produce different results. There are absolutely steps to mitigate this. For instance, you can await newScore. You can make doSomethingElse() an async function, and await them both using Promise.all().

But imagine that you had passed `profile` into both of those now-async functions? And each one modified it at a different time? Handling race conditions like this can be very difficult, and oftentimes immutable data structures are the solution. You can use things like locks and mutexes, but that is just as easy to get wrong as any other thing.

So, to summarize, mutation can occur at any time and can also be observed at any time in Javascript.

Elixir is not, by all definitions, completely immutable

In Elixir, everything runs in a process. It is a garbage collected language. Each process has its own stack, and its own heap. This garbage collection means that the memory location of a variable can technically change at any time. So from a pedantic standpoint, you could have two variables whose memory location swaps at some point while running your program. This is transparent to the programmer (which is good).

With this in mind, an argument could be made that anything stored in a process’s heap is mutable state. For example, let’s take a simple, GenServer-backed counter in Elixir.

When I increment this counter, new memory is allocated for the new number to live in. When I access this counter, I’m reading from that memory location. At some point, a garbage collection comes through, cleaning up the references to the old variables, and potentially moving the location of the new variable in memory.

So this process, while indirect, actually has all of the hallmarks of mutation! The same variable changing its location in memory in a way that is completely out of the developer’s control. Other variables being removed from memory or moved around at any time. SpoOoOoky!

So Elixir is mutable then? By some definitions, sure. In Elixir, values are immutable, but ultimately there is state, somewhere in our system. And it even changes over time! So what gives?

There are two secret sauces in the Elixir restaurant. They are:

• Function calling as the only way to observe mutating state

• Serialized access to mutating state

Function calling as the only way to observe mutating state

Something we have available to us in Elixir is the “process dictionary”. By many definitions, the process dictionary is mutable state. I would argue that, from the perspective of our program, it is not more or less mutable than any other thing. The reason for this, is that in Elixir, all mutating state requires calling a function to observe it.

Want to know the current time? You must call DateTime.now(). You can change the process dictionary with Process.put. You can even call a function that makes this change. Lets look at an example based on the Javascript “mutating profile” example from before:

Here we are “mutating” the process dictionary state. It was one thing, and now it is something else. However, we cannot observe this mutation without calling Process.get. The profile variable will always, unambiguously, equal %{score: 3}. The only way that could change is by reassigning the variable, and calling a function. For example:

Our variable values are always safe in Elixir 🥳

Serialized access to mutating state

This is the most important part of the whole equation. We’ve shown that we can accomplish mutating state in Elixir, albeit in a different way than you might have seen elsewhere. But what about the race condition example from before? We can’t really write an equivalent example to that in Elixir. Our concurrency primitive is to start another process, which has its own isolated process dictionary. There is no way to modify another process’s state.

So we’ll base this on our counter example, which is how you’d actually do This Kind Of Thing™ in Elixir. Each process handles messages serially, one-at-a-time. This means that, from the perspective of the counter, there is no such thing as a race condition! And callers, in order to interact with the counter, must call functions to send it a message, or to observe its changing state. Nothing the counter does can modify the state of the callers.

By forcing the mutation of state to be serialized through a process’s mailbox, and limiting the observation of mutating state to calling functions, our programs are more understandable, and less surprising.

What’s next?

There is a lot more to say on this topic, and I fully intend to do so! Understandability and predictability are not the only benefits of the concepts described above. Many of the benefits only even come into the picture when it’s time to scale, or to model graceful failure of our application components, and its those properties that I’ll be writing about next.

We won’t be going into the details of the what service or tool you use to actually send emails. The most common setup is using Swoosh with something like Postmark or Sendgrid. Phoenix sets you up with this out of the box, or you can look at the Swoosh docs to set it up yourself.

Here, we will be focused on how to architect your resources to get the guarantees you need from sending emails. We’ll start with the simple/least-powerful method and work our way up.

We’ll base this example on a real use case from ElixirForum (my answer to which the content of this post has been adapted from). We want to send users an email when someone comments on their post. Simple, right?

Before we dive in

In all of these examples I’m using “inline changes”. This is not actually best practice, you should prefer to extract your changes to modules that use Ash.Resource.Change, but inline functions are much simpler for demonstration.

It’s very important to keep in mind that CAP theorem limits us here. Specifically, your email sending service and your app together create a distributed system. There is no such thing as “exactly once”, where something in your application can be 100% guaranteed to only send one email. I’ll probably say something along these lines in a lot of my posts…but it bears repeating.

This means you are always choosing between one of two alternatives: at most once delivery, and at least once delivery. Especially when you get retries involved, at most once technically includes “maybe never”, and at least once technically includes “maybe a million times”. A lot of tooling around background jobs involve tools to help curtail this problem to a reasonable set of constraints.

I will detail how these guarantees apply to each of our strategies below.

After Action Hooks (at least once)

After action hooks happen in the transaction with the action. What this means is that you could send an email, and then an error could occur, and you’ll have sent an email for a comment that was not actually created. In some cases you may actually want that, but in this case it makes this strategy less-than-ideal.

Another ramification of this strategy is that errors sending emails could potentially prevent a comment’s creation. This is naturally not something we want interrupting our user’s experience.

The above implementation was simple, and could be good if you wanted to send an email perhaps notifying some owner of a resource every time someone even attempts to perform an action. Not in this use case though.

After Transaction Hooks (at most once)

After transaction hooks happen(you guessed it) after the transaction has been committed. It is also the only hook in Ash that is invoked both on success and failure. In the example below, we get better behavior in the case of errors than in the above example.

If something goes wrong, the user may not get an email. Additionally, if something goes wrong sending an email, the user might experience an errored request, but their comment will still have successfully been created. Better UX, but not perfect.

The above implementation was simple, just like the first one, and has some better properties.

It still has some undesirable properties. If something goes wrong sending an email, the caller could get an error, even though the transaction was submitted. You could handle this with defensive coding.

Even worse, however, is that this doesn’t “compose” with other actions calling it. For example, say you had something like an add_many_comments action, and that action called this action multiple times in a transaction. When calling that action, you’d get a warning printed that Ash is running an after_transaction hook in a transaction. This happens because the transaction was already started when this action was called.

Notifiers (at most once)

Ash has a system designed to solve exactly the problem described above. We call these “notifiers”. Specifically, their job is to do some post-transaction work, to notify other parts of your system (or users) of what has transpired. The most common notifier is Ash.Notifier.PubSub. However, writing your own is as simple as defining a notify/3 function!

Notifications are all automatically accumulated by Ash, and send after the final transaction has been committed. This is very useful for live systems! It prevents pub sub messages going out for data that has not yet been created or modified, and it gives you action side-effects that are at least once.

Now, our action is composable! Creating many comments in a single transactional action will send notifications after that transaction has closed. However, you still have to program defensively around errors, as errors in notifiers will still incur errors happening in your app.

For this reason, notifiers typically should just dispatch a message to something else running in your app to handle the work they are meant to do.

All of the above approaches have some flaws

The above approaches are relatively simple, but what about retries? The things that happen in an action are awaited by the user. Do you really want email sending code mixed in with your app code? Do you want users waiting on sending an email for your app to respond? Almost certainly not.

There is a better way, and that is with our wonderful friends Oban! Oban’s free features are more than enough to power this functionality, but their pro plan offers some awesome features. I highly advise checking it out, as it’s a staple tool in the Elixir ecosystem (this is not a sponsored statement).

With Oban (at least once/at most once)

Oban provides all the guarantees we want, and also allows our action to transactionally insert the “request” for an email to be sent. Anything going wrong with the email will not affect our response to the user, and if an error happens after inserting the job, the transaction is rolled back and no email will be sent.

With the above strategy, you’d implement and configure an Oban Worker by hand, and in that Oban worker you’d send your email. The way that you opt into at most once or at least once in this case is entirely dependent on your Oban configuration.

Using AshOban (at least once/at most once)

With this strategy, we leverage data to our advantage. Instead of making an Oban queue the authoritative representation of “emails that will be sent”, we use an Ash.Resource, configured with an `ash_oban` trigger.

With this setup, in my :create action I would create a NewCommentEmail instead of directly inserting the Oban job. The above case has “at most once” guarantees because it changes the state to sent, and then sends the email in an after_transaction hook. This means that the scheduler won’t ever reschedule this particular email, even if something goes wrong while sending.

If I want at least once guarantees, I can change that to a before_transaction hook. Easy-peasy.

There are other awesome benefits to this data-driven strategy of creating a NewCommentEmail row in a database table. Want to know how many emails have been sent ever, and how many are waiting to be sent?

Want to cancel a specific email from being sent? You can add a :canceled state and an action to set that new state. Even if the worker is in the queue to run, it always checks if the record still meets its schedule criteria before calling its update action!

Want to resend a specific email? You can add a `:resend` state, and update your trigger to account for that new state

Wrapping up

As you can see, the Oban-backed strategy is very powerful. However, the other strategies shown can be great starting points, or perhaps they cover the use case that you have well enough on their own.

These strategies apply to any kind of side effect. We’re just using emails here because that is such a common requirement. You should now be armed with the tools to navigate the various options in Ash for adding side effects to your actions.

What do you think?

This is the first hard-core detail-oriented writing I’ve done here on this Substack. I’ll be adapting this into a how-to guide in Ash’s documentation in the future. Do you like this kind of thing? Or do you prefer content on more high level concepts? Let me know in the comments!

Subscribe now

Let's start by answering the question “Where do Elixir developers hang out?”. You probably know about a few of these, but very few people are aware of just how many sub-communities there really are. This is certainly not an exhaustive list, and these are not in any particular order.

• Elixir Forum

• Stack Overflow

• Github Issues & Discussions

• Slack

  • Elixir Slack

  • EEF(Erlang Ecosystem Foundation) Slack

• Discord

  • Discord

  • Elixir-Tools Discord

  • Ash Framework Discord

  • Elixir Oasis

• #elixir on irc.libera.chat

• Social Networks

  • Twitter

  • Mastodon

  • Bluesky

• Telegram

  • Elixir Formula

  • Elixir World

• Reddit

With that information, I bet you know exactly where to go to participate in the community, so we can end the article here, right? Sure.

How did this happen?

People are on these different platforms for a reason. Twitter is banned in Brazil right now. Many workplaces don’t allow discord. People have personal reasons. Many don’t like the politics of twitter. Or the walled garden of Slack/Discord. Or simply want to be in a smaller community that they view as more like themselves, or perhaps more accepting of who they are.

So this conversation can’t be about “taking things away”, which is a natural first conclusion when you start thinking about this problem. Why don’t we just choose between Discord & Slack? Combine the communities? Because you’re going to alienate a large group of people regardless of how you do it.

Smushing all of these communities together is not a good idea even if we could. Even if everyone was willing, would one “mega-community” really make anything better? I find it doubtful. Many people have told me that they appreciate the Ash discord server in part because it is a much smaller group than “all Elixir developers”, cutting down the noise.

This is the “better”, in “for better or for worse”.

So what is the problem then?

Beginners can get lost

Where do I go for help?

• Google? Well, millions of questions have been answered in non-indexable places like Slack, Discord and IRC. So you won’t have nearly as much luck as you would here.

• Stack Overflow? Strangely unpopular among Elixir developers. But there is some good stuff here. You might find what you’re looking for, but its unlikely.

• Issue trackers? No! Those are for bug reports with proper reproductions and all the info necessary!

• Slack? What if the folks who know about the thing you’re asking about aren’t in Slack? Or they’re busy today?

• How about I search in Slack! Oh…only the last 10k messages are kept? I guess all that hard work from the community is gone forever then. None of the history is indexed by search engines either.

• IRC? Same problems.

• Discord? Same problems. You have unlimited search back here, which is awesome. For now. Who knows what changes discord will make?

• Right, Elixir Forum then. This is the actual best answer, but that doesn’t mean that it wouldn’t have been faster to get help elsewhere.

The optics aren’t great

As a community, we have to cut a difficult balance. A balance between considering the commercial aspects of what we do and the fun aspects. The Elixir community is a lot of fun to be a part of, and the fact that our direction is not set by corporate overlords is the reason that many of us are here. I, for one, believe it will play out better for our community in the long run.

With that said, a very large portion of the Elixir community is focused on working with Elixir professionally. We need a healthy job market. We need a healthy image from members that are not a part of our community, to increase adoption. We need things like case studies, success stories that act as a testament to the power of Elixir.

This stuff isn’t fun. Its annoying. Half the time it can even feel disingenuous. We are experts; We are aware of the challenges we faced building a thing. It wasn’t magic. We aren’t salespeople. But we have to sell it. At least, we do if we want Elixir to continue to be a viable career choice.

When someone asked if the Ash project was “dead”, I realized that for all the benefits I got from having my own discord server (custom roles, as many dedicated channels as I want, low signal/noise ratio for things we cared about) there was also one major drawback. I had created a silo.

I’ve always thought of Ash as something that could draw new developers to Elixir. This has proven to be true! But if you googled Ash Framework two years ago, you’d see a smattering of questions on Elixir Forum, a few of my conference talks, and it would not have been enough.

Creating a dedicated space in the Elixir Forum was the best thing we ever did for the Ash community. We still have our discord server, but we encourage questions to be posted in Elixir Forum. Some people are better at this than others. We aren’t forcing anything on anyone. We meet them where they are at. So all we had to do was make an avenue for participation in a more open community, and we got the benefits.

The Oban community ended up doing the same thing that we did, and just the other day, the Nerves community announced that they were doing the same. Huzzah! The silos are collapsing!

So what do we do?

No-one likes to think of the optics. I get that. And the truth is, you don’t have to. You do you.

However, if you’re part of the leadership of this community, or wish to make an active change, I highly implore you to consider the cost of these silos. They make the language look far less active and supported than it actually is. They make it hard for beginners to know where to get help, or to get the best help that they can.

We’re talking about this on Elixir Forum right now. People have a difficult time getting caught up on the Elixir news, because it happens on a social network they can’t access. Or because it happens in a silo they don’t know about.

Here are some positive impacts that can be made right now:

• If you’re posting awesome content on twitter, like this awesome video by José Valim on optimistic updates in LiveView, post it on the Elixir Forum too!

• Alternatively, post it anywhere you’ve got an RSS feed. Like Substack, or your blog.

• When in doubt, use Elixir Forum. It’s an open community. It has excellent SEO. You can subscribe to any part of it as an RSS feed. The posts are automatically sent to twitter even!

Elixir is a growing language. Lets help our newbies, and show the world the kind of traction that is really has!

Subscribe now

I occasionally work on our client projects at Alembic, something I actually really enjoy doing. Chris McCord talks about the benefits of using our own tools in anger, and I completely agree.

I’ll be focusing here on those days when my time isn’t spoken for by clients for the majority of the day. I’m also not focusing on the technical content of my day, but on the mindset and methods I use to Get Shit Done.

I have a problem

You’re going to see it soon enough, but the truth is that I have what is effectively a pathological level of concern for the members of the Ash community. My relationship with my work and how I balance it is something that I’m actively working on transforming. In fact, that is a part of why I’m writing this newsletter!

It helps me to write. It makes me feel like my output can be measured in more than bugs fixed and features announced. It gives me an avenue of expression to the users that I spend my days working on behalf of, and makes this all feel that much more like a conversation.

So write I shall!

Wake up

The first thing I do in the morning is look at my phone to see what issues/requests have come in overnight. Like before I get out of bed. This is probably the number one thing that I’d like to change. But every day is so wildly different depending on what sorts of things have come in from users on that day.

If there is a serious bug, then it takes priority over my feature work. If there are a lot of new people getting started that day, my morning will be filled with conversations helping them get started. If some of my clients have had questions or have requested PR reviews, then that has to happen too. It’s a very dynamic way to work, something that I both appreciate and that exhausts me.

Start the loop

I honestly don’t know if this is something that other people do, but I practically live my life in the loop. The loop is a simple concept, geared towards managing a firehose of incoming work, and helping an anxious programmer feel like they aren’t leaving anyone hanging.

I have a list of the kinds of things I have to do each day. Starting with work generating items at the top, and deep/engaging work at the bottom. For example, my list looks something like this:

• Check email, Discord, Slack, Reminders, ElixirForum, Bluesky, Twitter

• Fix serious bugs

• Answer questions

• Implement next item on roadmap

• Announce new features, or write new content

• Fix minor bugs

When I get to the bottom, I go back to the top. I’ll skip items if I’ve done them recently enough that new things shouldn’t have popped up. My priorities may change from day to day, but it doesn’t deviate much from this pattern.

5-minute rule

As I’m going, if anything comes up that would take less than five minutes to handle, I handle it on the spot, no hesitation. This is one of the main reasons community members get support as quickly as they do. They don’t get “filed away for when I have time”.

This is actually a commonly occurring rule in productivity systems, but I have one personal change to it that makes all the difference.

When something comes up that would take more than five minutes to handle, I find any part of it that will take five minutes or less to perform, and do that immediately. For an issue report, this might mean finding some clarifying question that I can ask the reporter that will help me when it comes back up in my loop. For a feature, or a project, this might mean “identify a few large themes in the request and write them down” or “re-summarize the effort in my own words.”

Taking on new work

For anything that takes more than five minutes, I will first comment to indicate that I’m aware of the item if necessary. Then, I’ll do one of two things:

• Mark it as unread. This is an escape valve for when I really just do not have time to consider the item at all.

• Decide when I’m going to consider the item again, and push the item off until that time. 9 times out of 10, this means I put a scheduled reminder into Apple Reminders.

When scheduling work, an important part of my mindset is that I’m not necessarily committing to do it at that time, only to consider doing it at that time. It can be a commitment, i.e in the case of time-sensitive items, but it does not have to be.

When a scheduled item comes up, I either handle it, or I push it to another time after applying the 5-minute rule. In this way, all the needles get pushed forward at least a little bit whenever they come back up in my loop.

Nothing gets lost

By operating this way, I ultimately do not miss even a single item that comes up as I’m working. If I ever load-shed incoming work, I do it consciously. I don’t miss notifications, emails (inbox zero is the way). If I ever do, it’s because I didn’t have something adequately hooked up to my loop (like I wasn’t watching a GitHub repository I should have been watching) and so it’s easy to rectify.

This means confidence for those interacting with or depending on me, and a feeling of control for myself.

Context switching is key

The way that I work necessitates switching modes often. I go through my loop hundreds of times a day. Sometimes stopping in the middle (at natural break points) of feature work to make sure that all the plates are still spinning. Everything I’ve read says that context switching is the death of productivity. Either I have a unique ability to context switch, or that isn’t exactly correct.

Close the loop

One thing I’ve found is that it can be very difficult to break out of this loop when it’s time to shut down. I’m so used to feeling confident that all my inboxes are cleared. I struggle often with putting my work down at the end of the day.

Something I’ve noticed that helps is setting a quit time at the start of the day, instead of trying to do it when you’re “done with your work.” So my loop these days has an item at the top that says “schedule shutdown time.”

And this is what I do, day in, day out

I honestly have no idea how unique this mental model for getting my work done is. Maybe it sounds wildly obvious to those reading it. Maybe everyone has a loop built in, and the fact that I’ve formalized my own loop a bit is weird. Maybe it takes 763 licks to get to the center of a Tootsie Pop. The world will never know.

It may have been a long time ago, but you signed up for a mailing list on the Ash Framework website. I never actually used that mailing list, but I very much intend to use this one.

So what are we doing here?

From here on, I’ll be communicating a bit more frequently, and in longer form than my usual tweet announcements. The Ash community continues to grow, and I think it will behoove me to share a bit more of my voice than a changelog, or simple feature announcements.

What will I be writing about, you ask? Why, anything and everything! While you may know me from my obsession with shipping code, I am, in fact, more than a feature-shipping machine. There are lots of things I’d like to talk to you all about. From Ash & Elixir, to things that I find personally interesting, to open source & ethics.

I’d like to hear from you, too! Click below to tell me about yourself, and what you’d like to get out of this news letter.

Tell me about yourself

Is that really it??

Not wanting to leave you all bored from a simple welcome email, let’s dive into our first topic! 🔥

Igniter

My latest project, and where a lot of my energy is going these days, is called Igniter. Igniter is a code generation and project patching framework. The level of DX that it provides for those writing and using code generators and installers in Elixir is, in my opinion, unparalleled.

Christian Alexander posted an awesome introductory video the other day. I was really pleased to see such a well made and informative introduction to igniter. Check it out, as well as his other content if you haven’t!

Why igniter?

Generators suck

It was high time to add generators and installers for our Ash packages. The setup was far too manual. With that said, I’ve always hated code generators as a general pattern. I’ve avoided them for years.

I was not willing to write a simple “copy templates to files” generator for new Ash projects. We needed to provide a first class experience, and build on a solid tool.

Some of the things that igniter does to provide a high quality experience:

• AST based patching of code

• Automatic discovery of installers

• Composable generators

• Diffs and confirmation of changes

DX is what Elixir needs

As someone who has been a member of the Elixir community for almost 10 years now, I know that the DX of Elixir is unparalleled. However, there are a class of things that are highly visible when new people join our community.

1. Our language server is lacking

2. Getting started is hard

Language Servers

This is not news to anyone, really, and I won’t harp on it here. The language server teams have worked hard at improving this problem, and I think that people are often far too hard on them. High quality language servers can be multi-year projects for multiple full time engineers. The most important news here is the formation of the official language server team. I was giddy when I heard the news!

Getting started

The main way that people enter into the Elixir ecosystem is via some kind of toolchain or framework. For instance, Phoenix, Ash, NX, Livebook, etc. I think we can drastically underestimate the amount of people who will bounce off of the Elixir ecosystem due to even small amounts of friction at the outset.

If we can give them a seamless getting started experience, with installers that allow them to incrementally adopt new parts of the Elixir ecosystem, with a simple, easy-to-remember CLI, then we can help smooth the way for these new folks to join our community and help bring Elixir out of its niche and put it in its rightful place.

So where is it headed?

By the time Igniter has become what I know that it can become, it will be like npm install on steroids. It will be like `brew install` but for installing libraries into your app. Your one stop shop. Additionally, it will overhaul existing generators to have a better UX, and allow for writing a whole new class of tasks with ease, like upgrade scripts and project transformers.

For one such example, see mix igniter.update_gettext, a magic one-liner to modify your project to resolve a warning from gettext.

Thats it folks!

Thank you all for reading, and I hope to continue to awe and amaze with more rambles for many years to come. Don’t forget to take the survey! I’d love to hear from you.

Happy hacking 😎

Tell me about yourself

It is my belief that, however hyperbolic, Declarative Design is the key to the future of software. I didn't invent it, nor do I count among the early pioneers of it. In fact, while this may be the first time you've thought about it specifically as "Declarative Design", you've almost certainly been exposed to it many times already.

If you've used SQL, HTML or CSS, you've worked with declarative languages. So instead of trying to pitch you on the concepts presented in Ash, I'd like to provide a small example of how you can add Declarative Design to your code-base today and benefit from it immediately.

Before we get started it is important to note that, if you are building any kind of software application at all, you are building an abstraction of a domain (a.k.a problem space, business use case). This single fact means that your system will be far more _internally consistent_ (each part of itself similar to the other) than it will be _consistent with external systems_. The high level constraints on your system inevitably shape every subsystem within it.

While "premature abstraction" can be a serious problem, “late abstraction” can be just as serious. Declarative Design is the method by which you side-step the problems presented by imperative styles of abstraction.

How do we side-step these problems? We extract the data, instead of abstracting the code.

Example

These examples are written in Elixir, but they are primarily pseudo-code/should be easily readable by any programmer. A bunch of code is left out, because the point is the structure/changes we make, not the minutia of each operation.

We'll start small, with a module that does some simple work of taking some data provided by an API and creating a corresponding record for that data.

We’ve shipped the above. Our service is creating data, everything is going swell. A few days later we get a call from a customer saying that his post creation requests are failing, and others are taking a very long time to complete.

So let’s create a function that will log the time each operation takes, and report any failures. Naturally we should do this for each of our operations.

Let’s follow the conventional wisdom of not abstracting too early, and take the obvious path.

Update shipped, it’s doing what it is supposed to, we're seeing metrics and log messages, and we were able to use that information to solve our customer's issue.

A few days later, we find out that there are some users that are creating bad data. We aren’t tracking who is making what changes, so let’s add some auditing code. We need to accept the user as an argument, and create an audit log with the result of each operation.

Alright, this is out in the world and with these audit logs we’ve found and banned the bad actors.

Our next requirement comes in. Apparently, we’d like to give the interns access to the system. They should be able to create posts, but not people or organizations.

Well, this should be pretty straightforward!

So what’s wrong with this code?

Honestly? Nothing. If this was your entire app, just these three functions, do not change this code. The problem is that the majority of our applications are considerably more operations like this. So with that lens, there are the three primary problems.

Duplication

This is the classic problem embodied with principles like DRY. We don’t want to repeat ourselves because that can make changing the way our system work very difficult. We’d have to change a lot of places, and there is risk that we will do it wrong in some of them. Or that we will forget to do it in some of them.

Hidden Divergence

The *real* problem, however, is the opportunities for hidden divergence. These operations are almost the same. But not quite! They all take a user argument, but they don’t all have rules that involve the user.

These opportunities for misunderstanding are very dangerous.

We’re leaving a lot on the table

You’ll see what I mean shortly. Ultimately, we could do more with this code, effectively for free. It just has to be written differently. Let’s take a look. I’m just going to skip to the end and show you what I’d suggest the “right answer” looks like.

So let's break down the benefits here. By separating this logic into two components, the "declaration", i.e the “what”, and the "engine", i.e the “how”. The amount of downstream good that comes from this kind of design is impossible for me to overstate.

1. Notice how legible the description is? I bet you could show that directly to a non technical person and they would understand "what" your code does, without having to know "how" it does it.

2. When you need to change something, you have a great starting point for that discussion. If you're changing your description, that means you're changing the domain. If you need to change the engine, you are changing the method.

3. Someone else can come along and rewrite the engine, leaving the description intact. This guiding force can allow for extremely easy refactors, potentially even in a different programming language entirely.

4. If I want to add a new operation, say `create_comment` it is effectively a triviality, and it can be configured with all the goodness we've built into our description.

5. This description is static, and can be used to power other things! You cold generate a markdown file, or a diagram from these descriptions. This is far more useful than you may realize, and we have all kinds of tools along these lines in Ash.

In closing

As you can see, we don't even really care that much about the procedural code anymore. We can discuss changes to the underlying behavior solely with the description.

This is your DSL, or "domain specific language". DSLs don't have to be confusing or magic. They don't have to use macros. A DSL could just be a map, or nested keyword lists like we used above.

Declarative Design can work at many levels across your system. It can be the driver behind complex system flows, and it can also help with abstracting and encapsulating code on a smaller scale.

Ultimately, in my experience, Declarative Design incurs very little cost or risk. It increases our leverage and flexibility, while keeping our code simple and understandable.

As of 2023, my methods have changed a bit. I still use bunpro, but have traded wanikani and kamesame for JPDB.

tl;dr - This essay is broken into four sections: 1) why I’m learning Japanese, 2) some basics about Japanese alphabets, 3) what makes it a hard language to learn, and 4) the tools I’m using to learn it. If you just want to know about that last bit feel free to scroll down to the bottom.

I've always wanted to learn another language. For years, I would pick up Duolingo and mess around with it for a few weeks before moving on to something else. Then, a little over a year ago, my best friend decided that he wanted to learn Japanese. While I’d love to say that being able to speak Japanese has been a lifelong dream, my motivation is slightly less pure. Specifically, I took the fact that Japanese is, by some accounts, the most difficult language an English speaker can attempt to learn, as a personal challenge.

This changed the shape of my learning journey. Now, instead of trying to optimize for short-term usage, I was headed down a path that would take years to complete. Knowing that was surprisingly _helpful_ with my motivation, as it changed my whole perspective around comprehension, ensuring the time took to learn new things didn't get me down. I was finally equipped for my language journey: I had motivation, a partner, and time, and that is all you need. Or so I thought.

If you're already familiar with the Japanese, you can probably skip this section.

I'll take a brief interlude to explain some foundations. There are three character sets, two of which are phonetic (like the English alphabet), meaning that they each have a specific sound. The first is called hiragana(ひらがな) and is the most basic/ubiquitous of the three. The second is called katakana(カタカナ) and generally has more specialized usage. For every hiragana, there is a corresponding katakana. They are most often used to represent "loan words" like バス(basu) for "bus" or パン(pan) for "bread" (from Portuguese). Having multiple phonetic alphabets sounded very strange to me at first, until I realized that English also has two (not counting cursive)! Look at each letter in "bard" compared to "BARD" to see what I mean.

The final character set, Kanji(漢字), is the one that most people have heard of. It is also one of the most daunting aspects of learning Japanese. There are roughly 50,000 Kanji, but the vast majority of them are no longer in use. The Japanese government maintains a set of 2,136 Kanji that must be taught at school, but most educated Japanese adults know somewhere from 3-4,000. If you work in a specialized field like medicine, or if you are an academic, that number can go as high as 6,000! (hearsay, can’t find a source on these numbers) I'll talk more about Kanji and their challenges a bit further on.

Grammatically, Japanese is about as different as you can get from English. The sentence structure is so different that you essentially have to think in reverse (some resources even teach the basics with Yoda). As you learn more advanced/casual grammar, you find out that things can go in pretty much whatever order you want. This does not help in the slightest. Prepositions (in, at, on) in English are post-positioned in Japanese and have little in common. Plurality is complicated; In English, "book" is unequivocally singular. In Japanese, 本(hon), isn't plural or singular, plurality is indicated by context, or by adding qualifying words. I could go on, but I say all this only to explain that learning Kanji isn't the only barrier involved in learning Japanese.

Back to the topic at hand

It became clear to me pretty quickly that Duolingo just wasn't going to cut it. The concepts didn't map well to the Japanese language and, in all honesty, I think Duolingo's framework just isn't challenging enough in general. This realization started my search for better tools. I had learned that multiple-choice/drag and drop questions are not an effective way to study languages. You need to be "producing" and "translating". Additionally, to learn Japanese you need to use tools that are tailored to Asian languages, or better yet the Japanese language itself. These two conditions knock out a lot of the most popular contenders like Duolingo, and Memrise. I'm not suggesting that those tools don't have value, or that they aren't a good place to start, but that there are more effective tools out there.

I'll spare you the whirlwind tour of every other tool that I tried and ultimately cast aside and skip ahead to the good stuff. What I ultimately discovered is that, at least for Japanese, you need to front-load Kanji and vocabulary. I've heard many people talk about this, and it is clear that they know it is true, but have trouble articulating _why_ it is true. Luckily, I think I'm at a point where I can explain the issue. Keep in mind, this is not meant to discourage, only to explain why trying to just dive into Japanese probably won't get you very far.

You can learn to "read" English, with its relatively simple phonetic alphabet in a matter of days. Your pronunciation will often be wrong, and you certainly won't know what all the words mean, but you can turn _symbols_ into _sounds_ from an early stage. What this means is that, generally speaking, trying to remember a word that you _see_ is the same as remembering a word that you _hear_. This same parallel isn't true for Japanese, because of Kanji. It typically takes _at least_ a year, and typically longer, for an adult to learn the Joyo Kanji (what Japanese children learn in school). If you're presented a word that contains a Kanji that you've never seen before, there is no good way to know how it sounds. There are some patterns/tricks you can use to make decent guesses but this method is not reliable. This prevents you from trying to remember if you've ever "heard" this new word before. Worst of all, if you can figure out the _meaning_ of this new word from context, you still won't know how to _say_ that word to someone else.

To put this in context, there is a site called NHK Easy News that presents a simplified version of Japanese news articles. I know 1000 kanji currently, and that represents ~90% of the Kanji that they use on NKH Easy News. That may sound like a lot but having to stop on every 10th word to look it up, and not being able to turn these unknown words into sounds to help you remember/let you access your auditory memory makes trying to read without a really good grasp of Kanji _extremely_ tiring. The thing I didn't mention yet: "Knowing" a kanji doesn't mean you know how to say it in every context. For example, the Kanji "人" in "人類", "一人", "芸人" is read differently for each (jin, ri, nin)! So I know 90% of the Kanji, but do I know 90% of all of the ways they use Kanji? Probably not.

Again, this wasn't meant to discourage you, but to show the merits of this approach.

My Tools

I'm still just barely intermediate at Japanese but I believe this "stack" is going to take me well into the realm of high-intermediate/advanced, and give me an excellent foundation to continue my learning from. The magic in all of these tools is that they use an SRS or "Spaced Repetition System", that is designed to quiz you on things that you've learned _just before_ you forget them. You'll be amazed at how much you can stuff into your brain using an SRS. _All_ of the tools I use are based around an SRS. Prioritize these tools in descending order. The workflow is simple: If you have _any_ pending reviews or lessons in a higher priority one, do those first. Get back to the other tools if/when you have time.

WaniKani

WaniKani is a system that teaches you most of the Joyo Kanji, and ~6000 vocabulary words in an order tailored to English speaking adults. Their website will explain much better than I could. Wanikani is very much a "point and shoot" solution and it works wonders. Do your lessons when you have time, and do your reviews when they come up, and your Kanji and vocabulary knowledge will skyrocket.

KameSame

WaniKani doesn't do everything. Specifically, it doesn't present you with the English translation of a vocabulary word and have you provide the Japanese word with the same meaning. Luckily, a tool was developed that integrates with WaniKani to provide exactly that. This takes your vocabulary learning to the next level. Just using WaniKani will set you up well for reading/listening, but with KameSame, you'll find yourself able to locate Japanese words on demand, which is (naturally) crucial for speaking.

Since I started using it, the creator has added a whole host of tools that will make it an extremely useful tool to have in your arsenal, even after you've completed WaniKani. This is one to keep your eye on. Unlike WaniKani, you add lessons at your own pace here with no real cap, so be careful not to get overzealous.

Bunpro

![[BunproExample.png]]

I said Kanji/Vocab _first_ but not _only_. Bunpro is an amazing tool for learning grammar via practice. I've never seen anything quite like it, but it embodies the SRS system even while presenting you with different variations of the grammar you are reviewing. In the same way that KameSame will amp up your word production, using Bunpro for a couple of months will have you expressing ideas and building sentences that aren't just "subject-object-verb".

Sources

https://www.fluentu.com/blog/japanese/japanese-sentence-structure-patterns/

https://www.sljfaq.org/afaq/how-many-kanji.html