Original interpretation: Putting Agent into ESP32, the easiest thing to avoid is not the performance pit, but the boundary illusion.

Copyright Statement and Disclaimer This article is an original interpretation based on “Show HN: OpenClaw-class agents on ESP32 (and the IDE that makes it possible)”. The copyright of the original text belongs to the original author and source. This article does not constitute an official translation and is only used for learning, research and discussion of opinions.

Original reference Show HN: OpenClaw-class agents on ESP32 (and the IDE that makes it possible): https://pycoclaw.com/

Beginning: The most dangerous illusion of edge agents is “Now that it’s running on the board, all that’s left is a matter of tuning.”

When Gong Chen successfully deployed Agent on ESP32 for the first time, the excitement was unforgettable to this day. It was last fall when his team took on a smart agriculture project, which required the deployment of dozens of sensor nodes in the greenhouse. Each node must be able to make local decisions and control irrigation autonomously. When they saw that little ESP32 development board, the Agent was actually running, reading sensor data, making decisions, and controlling relays—at that moment, the whole office was buzzing.

“It’s done!” the project leader patted him on the shoulder and said, “I knew you could handle it. Now that it’s running, the next step is optimization. Let’s try to put it into production by the end of the month!”

At that time, Gong Chen also felt that the most difficult hurdle had passed. Model compression is accomplished, inference latency is acceptable, and memory usage is also within control. Isn’t the rest just to adjust the parameters and optimize the response speed?

But in engineering systems, what is most likely to cost the team is this optimistic moment of “already running”.

Three months later, when Chen Gong stood in the customer’s greenhouse and faced a cluster of nodes that “inexplicably” stopped working, he truly understood: edge deployment and cloud deployment are not the same problem reduced in size, but another type of system boundary that has been completely rewritten. The network is no longer stable, the power supply is no longer reliable, the scene is no longer visible, rollback is no longer cheap, and troubleshooting is no longer readily available.

You think you are just lowering your capabilities, but in fact you are moving the fault conditions forward. In the cloud, if a service fails, you can know it immediately, log in immediately, restart it immediately, and roll back immediately. At the edge, if a node fails, you may have to drive two hours to the site to discover that the power supply contact is poor; you may have to manually flash the firmware, only to find that there is no network on site and cannot download the image; it may take you three days to investigate and discover that it is a boundary condition bug that is only triggered under a specific combination of temperature and humidity.

Therefore, what is really worthy of caution is not that the edge agent cannot do it, but that the team will continue to use the cloud method to understand it: as long as the main path is passed, the rest is optimization. This kind of understanding is almost certain to cause trouble in fringe scenarios.

Misunderstanding 1: Being able to run it on the device means that the solution has been proven.

Why do people think so

Chen Gong recalled his state of mind at the time. The feeling of “the board is really running” was indeed extremely strong. In the lab, they deployed the Agent to the ESP32, connected the sensor, ran the test script, and saw the LED lights flashing as expected, and the serial port outputting the correct decision results - all of this was too specific and visible to fool the team into thinking that the hardest thing was done.

This mentality has its rationality. In software development, “running” usually means that the core logic has been verified, and the rest is indeed perfected and optimized. Years of cloud development experience have given the team an intuition: as long as the main path is open, edge conditions can be gradually covered; as long as the basic functions are available, performance can be slowly tuned.

But edge deployments challenge this intuition. Because on the edge side, the distance between “running” and “available” is much larger than that on the cloud.

Why is this understanding wrong

On the edge side, running only proves the existence of the main path, but does not prove that the system can survive jitter, power outage, weak network, disconnection, resource competition and on-site misoperation. Really expensive failures often occur under these seemingly unmainstream conditions.

Chen Gong and his first customer gave them a painful lesson. For a system that ran well in the laboratory, half of the nodes went offline within three days after being deployed to the greenhouse. Investigations revealed that the WiFi signal in the greenhouse would attenuate as the plants grew taller - something that was impossible to find in the laboratory, which does not have growing plants. There are also some nodes that are in a “zombie” state: they are still running, but no longer report data or respond to control instructions. It was later discovered that it was caused by a memory leak, but in the cloud, this level of memory problems will be detected by monitoring and automatically restarted; at the edge, it means that you have to go to the site to check the equipment one by one.

A more insidious problem is a “normal-looking” glitch. There is a node where sensor data is always being reported, but the values ​​are obviously abnormal - the temperature is always 25 degrees and the humidity is always 60%, no matter how the actual environment changes. It took Chen Gong and the others a whole day to discover that it turned out that the sensor connection wire was pulled loose during on-site installation, and the fixed value was read due to poor contact. This kind of failure will not happen in the laboratory, because the connections in the laboratory are carefully inserted; but in the field, the installation method of workers may be completely different.

What is a more accurate understanding

If the equipment can run, the prototype is established; if the equipment can be managed, withdrawn, and restored on site, the plan is established. The former is a technical milestone, and the latter is a system milestone.

Chen Gong later set a new rule for the team: any edge deployment solution must complete the two milestones of “laboratory verification” and “field verification” before it can be considered usable. Laboratory verification focuses on functional correctness, and field verification focuses on system resilience. On-site verification includes but is not limited to:

• Run continuously for more than 72 hours in a real network environment
• Simulate behavior in network interruption, recovery, and jitter scenarios
• Simulate data integrity in power supply interruption and recovery scenarios
• Verify remote diagnostic and remote recovery capabilities
• Verify the reliability of firmware OTA upgrades under real network conditions
• Verify the operability of the on-site manual takeover process

Only when these are verified can the plan be said to be established.

Misunderstanding 2: Offline is only a network problem, not a system design problem

Why do people think so

Chen Gong admitted that in the early design, they also understood “offline” as a communication layer state: with or without network, connected or disconnected. So they designed a retry mechanism: when the network is disconnected, the data is cached locally and reported in batches when the network is restored.

This understanding is natural, because in cloud development, offline is indeed mainly a network problem. If communication between servers is interrupted, it can usually be restored by repairing the network; when the client goes offline, the user experience will be affected, but the core logic of the system can usually continue. Therefore, the team tends to use technical means such as reconnection, caching, and retry to “handle” offline.

Why is this understanding wrong

Offline at the edge is never just a network issue. It also means: you lose instant observation, you lose the convenience of remote takeover, and you lose the coordination ability that exists by default in the cloud. Many small problems that can be solved by “waiting and trying again” in cloud systems will gradually evolve into state drift, task replay, or inexplicability on the edge side.

Chen Gong and his colleagues realized this deeply during a livestock farm project. The site is in a remote mountainous area, and the network signal is intermittent. They deployed an offline caching strategy as originally planned: data is stored locally and reported when there is Internet access. But it soon became apparent that the definition of “networked” became complicated in edge scenarios.

First of all, “having a network” does not mean “being able to connect to the server”. Networks in mountainous areas often only have access to a local area network, or only to a specific operator’s services. The device thinks it is “online” but is actually unable to complete business communications. In this case, should the system process it offline or online? Cloud systems rarely need to consider this issue.

Second, local decisions and cloud policies during offline periods may conflict. The agent made decisions based on local rules during the offline period, but the cloud also made plans based on outdated data without knowing the offline decisions. Inconsistent status can lead to confusion when the two reconnect.

The most troublesome thing is the “I don’t know I’m offline” state. Chen Gong found that some devices will be in a “semi-online” state when the network is extremely unstable: they can send heartbeats, but cannot transmit data; they can receive instructions, but cannot fully execute them. In this state, the cloud thinks the device is normal, and the device thinks it is normal, but in fact the communication is incomplete.

What is a more accurate understanding

Offline is first and foremost a system mode, not an abnormal state. It requires you to design degradation paths, breakpoint resume strategies, and local minimum autonomy capabilities in advance instead of increasing the number of retries afterwards.

Chen Gong later redesigned the offline strategy. The core change is to change the binary state of “online/offline” to three system modes: “full-function mode/restricted mode/island mode”:

• Full-function mode: The network is good, real-time communication is possible, and the system runs according to complete logic
• Restricted mode: The network is unstable and only critical data can be transmitted. The system simplifies decision-making logic and prioritizes core functions.
• Island mode: completely offline, the system completely relies on local rules to run, only ensuring no damage, not optimal decision-making

Each mode has clear entry conditions, behavioral norms and exit strategies. The system automatically switches between modes based on network quality, communication success rate, task urgency and other factors. In this way, going offline is no longer “waiting for the network to recover”, but “continuing to run under restricted conditions”.

Misunderstanding 3: If you shrink the cloud architecture, you can get the edge architecture

Why do people think so

Chen Gong’s team did adopt the idea of ​​“cloud architecture reduction” in the early stages of architecture design. They have a ready-made cloud Agent framework with complete functions and elegant design. When migrating to the edge, the most labor-saving way seems to be to copy the original links and cut them as much as possible.

They retain the hierarchical structure of the analysis layer, reasoning layer, decision-making layer, and execution layer, but simplify the implementation of each layer. They retain the event-driven messaging model, but replace the cloud message bus with a local queue. They retained the concept of the configuration center, but replaced the cloud configuration service with local files.

The team feels that since the cloud has verified the rationality of the architecture, the edge only needs to reduce the resource version. This line of thinking looks reasonable on paper.

Why is this understanding wrong

The cloud defaults to being online, observable, rollable, and quickly patchable; the edge defaults to the opposite. When you shrink cloud thinking, you don’t get an edge system, but you get a cloud afterimage that is extremely fragile under edge conditions. Many designs in the cloud are reasonable because there is a whole set of infrastructure behind it that can be saved at any time; when it comes to the device side, these implicit supports have almost disappeared.

The first problem Chen Gong and others encountered in practice was observation. In cloud systems, logs are taken for granted—every request, every decision, and every exception is recorded in detail. After a problem occurs, you can query logs, analyze traces, and reproduce the scene. But on ESP32, the storage space is extremely limited and it is impossible to store a large number of logs. Moreover, when you need to troubleshoot the problem, the equipment may have been running on site for several days, and you cannot get the logs from that time.

They tried to solve it with “log sampling”, logging only key events. But it soon became apparent that in edge scenarios, the definition of critical events was difficult to determine in advance. In the cloud, you can define what a critical event is based on past experience; but at the edge, site conditions are ever-changing, and what you think is an edge situation may be the norm at a particular site.

Another example is rollback. In cloud systems, rollback is a standard operation: discover a problem, roll back to the previous version, and complete it within a few minutes. But at the edge, firmware flashing requires physical contact or relies on OTA mechanisms that can be unreliable. Moreover, rolling back means interrupting equipment operation, which may not be acceptable in some scenarios (such as ongoing irrigation decisions).

What is a more accurate understanding

Edge architecture should work backwards from on-site constraints rather than from cloud capabilities. You should first ask “What is the easiest way to break here?” and then “How should the main path be long?” The order cannot be reversed.

Chen Gong later summarized a design principle: every architectural decision must answer the three questions of “what to do when the network is interrupted”, “what to do when the storage is full”, and “what to do when the computing resources are insufficient”. If you can’t answer, or the answer is “This situation should not happen”, it means that this decision is not suitable for edge scenarios.

For example, in the cloud, they might use complex rules engines to make decisions; at the edge, they switch to simple lookup tables because rules engines behave unpredictably when resources are low. In the cloud, they might use JSON to transmit data; at the edge, they switch to a binary protocol because JSON parsing requires more memory. In the cloud, they may rely on cloud AI models; at the edge, they must keep a minimum available fallback model locally, even if its accuracy is not high.

These are not “downsizing” cloud architectures, but rather redesigns based on edge constraints.

Misunderstanding 4: A high local success rate means that the operation and maintenance costs will not be too high.

Why do people think so

Chen Gong admitted that they did underestimate the operation and maintenance costs in the early stages of the project. In the lab, teams can have direct access to equipment, quickly re-flash firmware, and repeat experiments. When local testing shows a success rate of over 95%, teams can easily mistake controllability in the experimental phase for controllability in the future field.

Their calculation at the time was that even a 5% failure rate could be solved with manual intervention, which should not be difficult. This estimation ignores the peculiarities of edge scenes.

Why is this understanding wrong

Once the device leaves your desktop, the operation and maintenance problem immediately becomes another type of problem: who discovers the fault, who recovers the status, who confirms whether the device is only temporarily offline, and who decides whether to send someone to the scene. The real cost of edge systems lies not in development but in maintenance.

Chen Gong and the others experienced the bitter experience at the first customer site. When a node fails, who first discovers the problem? Cloud systems have monitoring, alarms, and logs; for edge systems, you need to proactively discover them. Their original design relied on the device to proactively report status, but if the device crashed completely or the network was completely disconnected, it would not be able to report. So there is a “silent failure” - the device is offline, but no one knows.

They later added a heartbeat mechanism, but this created new problems. When the network is unstable, heartbeats may occasionally be lost, resulting in false alarms. A node may be marked as “failure” after the network jitters, triggering the operation and maintenance process. As a result, the operation and maintenance personnel rushed to the scene and found that the equipment was actually normal.

Even more troublesome are “ghost faults” - devices that appear to be normal (heartbeat is on, status is updated), but are actually functioning abnormally. Chen Gong and the others encountered this once. The device’s heartbeat and status reporting were normal, but the actual sensor readings had stopped updating. The reason is that the sensor driver enters a strange state, still running, but no longer reading real data. This kind of problem is almost impossible to happen in the cloud, because the monitoring granularity in the cloud is finer; but at the edge, limited resources prevent you from running complex self-checking logic on the device.

What is a more accurate understanding

The first priority of an edge agent is not “how flexible it is locally”, but “whether it can be picked up after being far away from developers.” In other words, operation and maintenance design must start at the same time as functional design and cannot be used as a make-up course after launch.

Chen Gong later redesigned the operation and maintenance architecture, with the core goals of “observability” and “recoverability”.

In terms of observability, they implemented minimal health self-checking on the device side and regularly verified whether key functional modules (sensor reading, network communication, storage writing) are normal. The self-test results are reported through heartbeats. If the heartbeat is lost or the self-test fails, the system will mark it as “suspected failure” and trigger the remote diagnosis process. Remote diagnostics will first try to collect more information through the existing connection to determine whether on-site intervention is really needed.

In terms of recoverability, they designed a multi-level recovery strategy:

• The first level is remote recovery: delivering repair scripts or configuration changes through OTA
• The second level is soft recovery: the device automatically restarts and attempts to recover
• The third level is hard recovery: on-site manual intervention and physical reset of the device

Each layer has clear trigger conditions and success rate statistics. Only upgrade to hard recovery if both remote recovery and soft recovery fail. This way, more than 90% of problems can be solved remotely, with only a few actually requiring on-site intervention.

If we still want to continue to differentiate, which dimensions should we really look at?

If you really want to judge whether the Agent solution on ESP32 is reliable, Chen Gong recommends looking at four dimensions.

The first dimension is whether the failure path has been designed. Power outages, restarts, network disconnections, and recovery after partial completion are not exceptions to edge scenarios, but are routine. Your system must account for these situations during the design phase rather than hoping they won’t happen. The checking criterion is simple: Can you draw the state transition diagram of the system? Does each state transition have clear trigger conditions and processing logic?

The second dimension is whether the on-site takeover is enforceable. When there is a problem with the system, can the team take over without relying on the original developer to be online? This means documentation needs to be complete, logs need to be readable, diagnostic tools need to be available, and recovery processes need to be standardized. Mr. Chen has seen too many systems where only the people who work on them can repair them. Once the people who work on them take a vacation, the system can only stare blankly.

The third dimension is whether the local state is sufficiently self-explanatory. What the edge device fears most is that it is “still alive, but no one knows what it is doing now.” The system should have a mechanism to expose internal status so that operation and maintenance personnel can determine whether the equipment is working normally, is temporarily busy, or has an error. Ideally, an operation and maintenance personnel who is not familiar with the details of the system should be able to determine the basic status of the equipment within a few minutes.

The fourth dimension is whether downgrading takes precedence over retries. Many edge crashes are not caused by insufficient retries, but because the system does not know when to admit that it should take a step back. A good edge system should have a clear downgrade strategy, and proactively simplify functions when resources are insufficient or conditions are poor, rather than forcefully supporting them.

A more reliable order of judgment

If Mr. Chen were asked to give a practical order, he would rank it like this:

The first step is to design the failure path and recovery path. Before you write any functional code, think about how the system might fail and how it might recover after a failure. Draw a state diagram and define the meaning and transition conditions of each state. After completing this step, your understanding of the system will be much deeper.

The second step is to define the minimum on-site connection area. Determine what minimum information needs to be exposed and what actions are required to allow non-developers to take over when a system problem occurs. Standardize this information and operations and create operation and maintenance manuals and tools.

The third step is to decide which functions are worth bringing to the device. Not all features need to be marginalized. Even if some functions can be implemented on the device, considering maintenance costs, it may be more cost-effective to place them in the cloud. Marginalization should be a decision based on cost-benefit analysis, not technical brilliance.

The fourth step is to discuss “can we do it smarter” at the end. When the foundation is solid, then consider optimizing performance, adding functions, and improving user experience. If the foundation is not strong, these optimizations will only increase the vulnerability of the system.

This sequence doesn’t look cool, but it’s closer to reality. Because the real value of the limbic system is not showing off skills, but living for a long time.

Conclusion: The most respectable thing about Edge Agent is not the moment it starts running on the board, but the moment it remains stable when you are not there

When Gong Chen now sees those demos of “running OpenClaw on ESP32”, his mentality is completely different. He will still appreciate the ingenuity of the technical implementation, but he knows that the real challenge begins after the demo is over.

He has no doubt that bringing Agent-like capabilities to the ESP32 is cool and inspiring. It will force us to rethink models, tools and computing boundaries, and even give rise to many new application scenarios.

But it would be a pity to regard this thing only as a “smaller can run” technical wonder. Because the real question raised by the edge side is actually more difficult: When the environment is uncontrollable, support is scarce, and recovery is expensive, are you still willing to admit that the system must be designed for failure instead of showing off skills for success?

Therefore, Chen Gong’s core judgment on edge Agent is that it is not a mini version of cloud Agent, but an independent knowledge about restriction, degradation and takeover. Whoever can accept this first will be more likely to turn it from a “running prototype” into a “viable system”.

Last month, Mr. Chen paid a return visit to the customer of the smart agriculture project. The nodes in the greenhouse have been running stably for eight months. During this period, they have experienced multiple network interruptions, a firmware upgrade, and two sensor replacements. However, the system has been handled as designed and has not caused business interruption.

The customer said: “The best thing about your system is that it requires me very little to worry about.”

Chen Gong knows that this is the biggest compliment of a marginal agent. It’s not “how many smart things it did”, but “it took care of itself when I wasn’t around”.

References and Acknowledgments

• Original text: Show HN: OpenClaw-class agents on ESP32 (and the IDE that makes it possible): https://pycoclaw.com/