sweetness.hmmz.org

Operon: Extreme Performance For Ansible

dw — Mon, 28 Oct 2019 12:30:00 +0000

I'm very excited to unveil Operon, a high performance replacement for Ansible® Engine, tailored for large installations and offered by subscription. Operon runs your existing playbooks, modules, plug-ins and third party tools without modification using an upgraded engine, dramatically increasing the practical number of nodes addressable in a single run, and potentially saving hours on every invocation.

Operon can be installed independently or side-by-side with Ansible Engine, enabling it to be gradually introduced to your existing projects or employed on a per-run basis.

Here is the runtime for 416 tasks of common.yml from DebOps 0.7.2 deployed via SSH:

Operon reduces runtime by around 60% compared to Ansible for a single node, but things really heat up for large runs. See how runtime scales using a 24 GiB, 8 core Xeon E5530 deploying to Google Cloud VMs over an 18 ms SSH connection:

Each run executed 416 tasks per node, including loop items. In the 1,024 node run, 490,496 tasks executed in 54 minutes, giving an average throughput of 151 tasks per second. Linear scaling is apparent, with just under 4x time increase moving from 256 to 1,024 nodes.

The 256 node Ansible run was cancelled following a lengthy period with no output, after many re-runs to iteratively reduce forks from 40 to 10, so Ansible would not exceed RAM. A 13 fork run may have succeeded, but further attempts were abandoned having consumed two days worth of compute time.

In the final run, Ansible completed 89% of tasks in 6h 13m prior to cancellation:

256 Nodes, DebOps common.yml

Operon deployed to all nodes in parallel for every run presented. Operon has imperceptible overhead executing 1,024 forks given 8 cores and cleanly scales to at least 6,144 given 24 cores. Had these results been recorded using 16 cores rather than 8, we expect the 1,024 node run would complete in 27 minutes rather than 54 minutes.

Memory usage is highly predictable and significantly decoupled from forks. With 256 forks, Operon uses 4x less RAM than Ansible uses for 10 forks, while consuming at least 15x less controller CPU time to achieve the same outcome.

This graph is crooked as the 64 node Ansible run executed with 40 forks, while the 256 node run executed with 10 forks. Ansible required 1.6 GiB per fork for the 256 node run, placing a severe restraint on achievable parallelism regardless of available RAM.

Operon is the progression of a design approach first debuted in Mitogen for Ansible. It inherits massive low-level efficiency improvements from that work, already depended on by thousands of users:

Beyond software

Performance is a secondary effect of a culture shift towards stronger user friendliness, compatibility and cost internalization. There is a lot to reveal here, but to offer a taste of what's planned, I'm pleased to announce a forwards-compatible playbook syntax guarantee, in addition to restoration of specific Ansible Engine constructs marked deprecated.

include:

- include: "i-will-always-work.yml"

"with" loops

- debug: msg={{item}}
  with_items: ["i", "will", "always", "work"]

"squash actions"

- apt:
    name: "{{item}}"
  with_items: ["i", "will",
               "always", "work"]

hyphens in group names

  $ cat hosts
  [i-will-always-work.us.mycorp.com]
  host1

hash merging

  # I will always work
  [defaults]
  hash_behaviour = merge

The Ansible 2.9-compatible syntax Operon ships will always be supported, and future syntax deprecations in Ansible Engine do not apply in Operon. Changes like these harm working configurations without improving capability, and are a major source of error-prone labour during upgrades.

Over time this guarantee will progressively extend to engine semantics and outwards.

How can I get this?

Operon is initially distributed with support from Network Genomics, backed by experience and dedication to service unavailable elsewhere. If your team are gridlocked by deployments or fatigued by years of breaking upgrades, consider requesting an evaluation, and don't hesitate to drop me an e-mail with any questions and concerns.

Software is always better in the open, so a public release will happen when some level of free support can be provided. Subscribe to the operon-announce mailing list to learn about future releases.

Will Operon help Windows performance?

Yes. If you're struggling with performance deploying to Windows, please get in touch.

Will Operon help network device performance?

Yes. Operon features an architectural redesign that extends far beyond the transport layer, and applying to all connection types equally.

Is Operon a fork of Ansible?

No. Operon is an incremental rewrite of the engine, a small component of around 60k code lines, of which around a quarter are replaced. Every Ansible installation includes around 715k lines, of which the vast majority is independently maintained by the wider Ansible community, just as Operon is.

Will Operon help improve Ansible Engine?

Yes. Operon is already promoting improvement within Ansible Engine, and since it remains an upstream, an incentive exists to contribute code upstream where practical.

Is Operon free software?

Yes. Operon is covered by same GPL license that covers Ansible, and you are free to make use of the code to the full extent of that license.

Does Operon break compatibility?

No. Operon does not break compatibility with the standard module collection, plug-in interfaces, or the surrounding Ansible ecosystem, and never plans to. Compatibility is a primary deliverable, including to keep pace with future improvements, and backwards compatibility such as improved playbook syntax stability.

I target only one node, what can Operon do for me?

Operon will help ensure the continued marketability of skills you have heavily invested in. It offers a powerful new flexibility that previously could not exist: your freedom to choose an engine. Whether you use it directly or not, you already benefit from Operon.

David

Mitogen v0.2.8 released

dw — Sun, 18 Aug 2019 21:45:59 +0100

Mitogen for Ansible v0.2.8 has been released. This version (finally) supports Ansible 2.8, comes with a supercharged replacement fetch module, and includes roughly 85% of what is needed to implemement fully asynchronous connect.

As usual a huge slew of fixes are included. This is a bumper release, running to over 20k lines of diff. Get it while it's hot, and as always, bug reports are welcome!

Threadless mode in Mitogen 0.3

dw — Sat, 16 Feb 2019 22:00:00 +0000

Mitogen has been explicitly multi-threaded since the design was first conceived. This choice is hard to regret, as it aligns well with the needs of operating systems like Windows, makes background tasks like proxying possible, and allows painless integration with existing programs where the user doesn't have to care how communication is implemented. Easy blocking APIs simply work as documented from any context, and magical timeouts, file transfers and routing happen in the background without effort.

The story has for the most part played out well, but as work on the Ansible extension revealed, this thread-centric worldview is more than somewhat idealized, and scenarios exist where background threads are not only problematic, but a serious hazard that works against us.

For that reason a new operating mode will hopefully soon be included, one where relatively minor structural restrictions are traded for no background thread at all. This article documents the reasoning behind threadless mode, and a strange set of circumstances that allow such a major feature to be supported with the same blocking API as exists today, and surprisingly minimal disruption to existing code.

Recap

Above is a rough view of Mitogen's process model, revealing a desirable symmetry as it currently exists. In the master program and replicated children, the user's code maintains full control of the main thread, with library communication requirements handled by a background thread using an identical implementation in every process.

Keeping the user in control of the main thread is important, as it possesses certain magical privileges. In Python it is the only thread from which signal handlers can be installed or executed, and on Linux some niche system interfaces require its participation.

When a method like remote_host.call(myfunc) is invoked, an outgoing message is constructed and enqueued with the Broker thread, and a callback handler is installed to cause any return value response message to be posted to another queue created especially to receive it. Meanwhile the thread that invoked Context.call(..) sleeps waiting for a message on the call's dedicated reply queue.

Latches

Those queues aren't simply Queue.Queue, but a custom reimplementation added early during Ansible extension development, as deficiencies in Python 2.x threading began to manifest. Python 2 permits the choice between up to 50 ms latency added to each Queue.get(), or for waits to execute with UNIX signals masked, thus preventing CTRL+C from interrupting the program. Given these options a reimplementation made plentiful sense.

The custom queue is called Latch, a name chosen simply because it was short and vaguely fitting. To say its existence is a great discomfort would be an understatement: reimplementing synchronization was never desired, even if just by leveraging OS facilities. True to tribal wisdom, the folly of Latch has been a vast time sink, costing many days hunting races and subtle misbehaviours, yet without it, good performance and usability is not possible on Python 2, and so it remains.

Due to this, when any thread blocks waiting for a result from a remote process, it always does so within Latch, a detail that will soon become important.

The Broker

Threading requirements are mostly due to Broker, a thread that has often changed role over time. Today its main function is to run an I/O multiplexer, like Twisted or asyncio. Except for some local file IO in master processes, broker thread code is asynchronous, regardless of whether it is communicating with a remote machine via an SSH subprocess or a local thread via a Latch.

When a user's thread is blocked on a reply queue, that thread isn't really blocked on a remote process - it is waiting for the broker thread to receive and decode any reply, then post it to the queue (or Latch) the thread is sleeping on.

Performance

Having a dedicated IO thread in a multi-threaded environment simplifies reasoning about communication, as events like unexpected disconnection always occur in a consistent location far from user code. But as is evident, it means every IO requires interaction of two threads in the local process, and when that communication is with a remote Mitogen process, a further two in the remote process.

It may come as no surprise that poor interaction with the OS scheduler often manifests, where load balancing pushes related communicating threads out across distinct cores, where their execution schedule bears no resemblance to the inherent lock-step communication pattern caused by the request-reply structure of RPCs, and between threads of the same process due to the Global Interpreter Lock. The range of undesirable effects defies simple description, it is sufficient to say that poor behaviour here can be disastrous.

To cope with this, the Ansible extension introduced CPU pinning. This feature locks related threads to one core, so that as a user thread enters a wait on the broker after sending it a message, the broker has much higher chance of being scheduled expediently, and for its use of shared resources (like the GIL) to be uncontended and exist in the cache of the CPU it runs on.

Runs of tests/bench/roundtrip.py with and without pinning.
Pinned?	Round-trip delay
No	960 usec	Average 848 usec ± 111 usec
	782 usec
	803 usec
Yes	198 usec	Average 197 usec ± 1 usec
	197 usec
	197 usec

It is hard to overstate the value of pinning, as revealed by the 20% speedup visible in this stress test, but enabling it is a double-edged sword, as the scheduler loses the freedom to migrate processes to balance load, and no general pinning strategy is possible that does not approach the complexity of an entirely new scheduler. As a simple example, if two uncooperative processes (such as Ansible and, say, a database server) were to pin their busiest workers to the same CPU, both will suffer disastrous contention for resources that a scheduler could alleviate if it were permitted.

While performance loss due to scheduling could be considered a scheduler bug, it could be argued that expecting consistently low latency lock-step communication between arbitrary threads is unreasonable, and so it is desirable that threading rather than scheduling be considered at fault, especially as one and not the other is within our control.

The desire is not to remove threading entirely, but instead provide an option to disable it where it makes sense. For example in Ansible, it is possible to almost halve the running threads if worker processes were switched to a threadless implementation, since there is no benefit in the otherwise single-threaded WorkerProcess from having a distinct broker thread.

UNIX fork()

In its UNIX manifestation, fork() is a defective abstraction surviving through symbolism and dogma, conceived at a time long predating the 1984 actualization of the problem it failed to solve. It has remained obsolete ever since. A full description of this exceeds any one paragraph, and an article in drafting since October already in excess of 8,000 words has not yet succeeded in fully capturing it.

For our purposes it is sufficient to know that, as when mixed with most UNIX facilities, mixing fork() with threads is extremely unsafe, but many UNIX programs presently rely on it, such as in Ansible's forking of per-task worker processes. For that reason in the Ansible extension, Mitogen cannot be permanently active in the top-level process, but only after fork within a "connection multiplexer" subprocess, and within the per-task workers.

In upcoming work, there is a renewed desire for a broker to be active in the top-level process, but this is extremely difficult while remaining compatible with Ansible's existing forking model. A threadless mode would be immediately helpful there.

Python 2.4

Another manifestation of fork() trouble comes in Python 2.4, where the youthful implementation makes no attempt to repair its threading state after fork, leading to incurable deadlocks across the board. For this reason when running on Python 2.4, the Ansible extension disables its internal use of fork for isolation of certain tasks, but it is not enough, as deadlocks while starting subprocesses are also possible.

A common idea would be to forget about Python 2.4 as it is too old, much as it is tempting to imagine HTTP 0.9 does not exist, but as in that case, Python is treated not just as a language runtime, but as an established network protocol that must be implemented in order to communicate with infrastructure that will continue to exist long into the future.

Implementation Approach

Recall it is not possible for a user thread to block without waiting on a Latch. With threadless mode, we can instead reinterpret the presence of a waiting Latch as the user's indication some network IO is pending, and since the user cannot become unblocked until that IO is complete, and has given up forward execution in favour of waiting, Latch.get() becomes the only location where the IO loop must run, and only until the Latch that caused it to run has some result posted to it by the previous iteration.

@mitogen.main(threadless=True)
def main(router):
    host1 = router.ssh(hostname='a.b.c')
    host2 = router.ssh(hostname='c.b.a')

    call1 = host1.call_async(os.system, 'hostname')
    call2 = host2.call_async(os.system, 'hostname')

    print call1.get().unpickle()
    print call2.get().unpickle()

In the example, after the (presently blocking) connection procedure completes, neither call_async() wakes any broker thread, as none exists. Instead they enqueue messages for the broker to run, but the broker implementation does not start execution until call1.get(), where get() is internally synchronized using Latch.

The broker loop ceases after a result becomes available for the Latch that is executing it, only to be restarted again for call2.get(), where it again runs until its result is available. In this way asynchronous execution progresses opportunistically, and only when the calling thread indicated it cannot progress until a result is available.

Owing to the inconvenient existence of Latch, an initial prototype was functional with only a 30 line change. In this way, an ugly and undesirable custom synchronization primitive has accidentally become the centrepiece of an important new feature.

Size Benefit

The intention is that threadless mode will become the new default in a future version. As it has much lower synchronization requirements, it becomes possible to move large pieces of code out of the bootstrap, including any relating to implementing the UNIX self-pipe trick, as required by Latch, and to wake the broker thread from user threads.

Instead this code can be moved to a new mitogen.threads module, where it can progressively upgrade an existing threadless mitogen.core, much like mitogen.parent already progressively upgrades it with an industrial-strength Poller as required.

Any code that can be removed from the bootstrap has an immediate benefit on cold start performance with large numbers of targets, as the bottleneck during cold start is often a restriction on bandwidth.

Performance Benefit

Threadless mode tallies in well with existing desires to lower latency and resource consumption, such as the plan to reduce context switches.

Runs of tests/bench/roundtrip.py with and without threadless
	Threaded+Pinned	Threadless
Average Round-trip Time	201 usec	131 usec (-34.82%)
Elapsed Time	4.220 sec	3.243 sec (-23.15%)
Context Switches	304,330	40,037 (-86.84%)
Instructions	10,663,813,051	8,876,096,105 (-16.76%)
Branches	2,146,781,967	1,784,930,498 (-15.85%)
Page Faults	6,412	17,529 (+173.37%)

Because no broker thread exists, no system calls are required to wake it when a message is enqueued, nor are any necessary to wake the user thread when a reply is received, nor any futex() calls due to one just-woke thread contending on a GIL that has not yet been released by a just-about-to-sleep peer. The effect across two communicating processes is a huge reduction in kernel/user mode switches, contributing to vastly reduced round-trip latency.

In the table an as-yet undiagnosed jump in page faults is visible. One possibility is that either the Python or C library allocator employs a different strategy in the absence of threads, the other is that a memory leak exists in the prototype.

Restrictions

Naturally this will place some restraints on execution. Transparent routing will no longer be quite so transparent, as it is not possible to execute a function call in a remote process that is also acting as a proxy to another process: proxying will not run while Dispatcher is busy executing the function call.

One simple solution is to start an additional child of the proxying process in which function calls will run, leaving its parent dedicated just to routing, i.e. exclusively dedicated to running what was previously the broker thread. It is expected this will require only a few lines of additional code to support in the Ansible extension.

For children of a threadless master, import statements will hang while the master is otherwise busy, but this is not much of a problem, since import statements usually happen once shortly after the first parent->child call, when the master will be waiting in a Latch.

For threadless children, no background thread exists to notice a parent has disconnected, and to ensure the process shuts down gracefully in case the main thread has hung. Some options are possible, including starting a subprocess for the task, or supporting SIGIO-based asynchronous IO, so the broker thread can run from the signal handler and notice the parent is gone.

Another restriction is that when threadless mode is enabled, Mitogen primitives cannot be used from multiple threads. After some consideration, while possible to support, it does not seem worth the complexity, and would prevent the aforementioned reduction of bootstrap code size.

Ongoing Work

Mitogen has quite an ugly concept of Services, added in a hurry during the initial Ansible extension development. Services represent a bundle of a callable method exposed to the network, a security policy determining who may call it, and an execution policy governing its concurrency requirements. Service execution always happens in a background thread pool, and is used to implement things like file transfer in the Ansible extension.

Despite heavy use, it has always been an ugly feature as it partially duplicates the normal parent->child function call mechanism. Looking at services from the perspective of threadless mode reveals some notion of a "threadless service", and how such a threadless service looks even more similar to a function call than previously.

It is possible that as part of the threadless work, the unification of function calls and services may finally happen, although no design for it is certain yet.

Summary

There are doubtlessly many edge cases left to discover, but threadless mode looks very doable, and promises to make Mitogen suitable in even more scenarios than before.

Until next time!

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-07-10: Mitogen released!

Mitogen v0.2.4 released

dw — Sun, 10 Feb 2019 23:59:59 +0000

Mitogen for Ansible ~~v0.2.4~~ v0.2.5 has been released. This version is noteworthy as it contains major refinements to the core libary and Ansible extension to improve its behaviour during larger Ansible runs.

Work on scalability is far from complete, as it progresses towards inclusion of a patch held back since last summer to introduce per-CPU multiplexers. The current idea is to exhaust profiling gains from a single process before landing it, as all single-CPU gains continue to apply in that case, and there is much less risk of inefficiency being hidden in noise created by multiple multiplexer processes.

Please kick the tires, and as always, bug reports are welcome!

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-07-10: Mitogen released!

Transmit optimisations in Mitogen 0.3

dw — Thu, 30 Aug 2018 19:12:00 +0100

An early goal for Mitogen was to make it simple to retrofit, avoiding any "opinionated" choices likely to cause needless or impossible changes in downstream code. Despite being internally asynchronous, a blocking and mostly thread-safe API is exposed, with management of the asynchrony punted to a thread, making integrating with a deployment script hopefully as easy as with a GUI.

Although rough edges remain due to this, such as struggles with subprocess reaping, based on experience working on Mitogen for Ansible and ignoring complexities unique to that environment, the design appears to mostly function as intended.

Mostly being operative as, due to the API choice, and despite gains already witnessed in the extension, some internals remain overly simplistic. Naturally as has been the lesson throughout, this of course means inefficient: horrifyingly, crying-in-the-shower inefficient.

While recently attacking some of Ansible's grosser naivities, now the excesses of continuous forking are gone, dirty laundry is again visible on Mitogen's side. This post describes one offender: message transmission and routing, how it looks today, why it is a tragedy, and how things will improve.

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-07-10: Mitogen released!
2018-08-27: A fork in the road for Mitogen

Overview

To recap, communication with a bootstrapped child is message-oriented to escape the limitations of stream-oriented IO. When an application makes a call, the sending thread enqueues a message with the broker thread, which is responsible for all IO, then sleeps waiting for the broker to deliver a reply.

This has many benefits: mutually ignorant threads can share a child without coordination, since a central broker exists behind the scenes. Errors can only occur on the broker thread, so handling is not spread throughout user code.

Message Transmission

Examining just the Mitogen aspects of transmission for an SSH-connected Ansible target, below are the rough steps repeated for every message in the stable branch.

Despite removing most system calls to fit things in one diagram, there is still plenty to absorb, and clearly many parts to what is conceptually a simple task. A component called Waker abstracts waking the broker thread. This implements a variant of the UNIX self-pipe trick, waking it by writing to a pipe it is sleeping on.

When the broker wakes, it calls waker's on_receive handler, causing any deferred functions to execute on its thread. Here the asynchronous half of the router runs, picking a stream to forward the message.

The stream responds by asking the broker to tell it when the SSH stream becomes writeable, which is implemented differently depending on OS, but in most cases it entails yet more system calls.

Since usually the SSH input buffer is empty, the broker immediately wakes again to call the stream's on_transmit handler, finally passing the message to SSH before marking the stream unwriteable again. At this point execution moves to SSH, for little than to read from a socket, do some crypto and write to another socket.

Better Message Transmission

In total transmission requires at least 2 task switches, 2 loop iterations, at least 5 reads/writes, and 2 poller reconfigurations.

While superficially logical, one problem is already obvious: transmitting always entails waking a thread, a nontrivial operation on UNIX. Another is the biggest performance bottleneck, the IO loop, is forced to iterate twice for every transmission, in part to cope with the possibility the SSH input buffer is full.

What if we were more optimistic: an error won't occur, and the SSH input buffer probably has space. Since we aren't expecting to cleanup a failure, there is no reason to involve the broker either. The new sequence:

Coordination is replaced with a lock, and the sending thread writes directly to SSH. We no longer check for writeability: simply try the write and if it fails, or buffered data exists, defer to the broker like before.

Now we have 1 task switch, 0 loop iterations, 2 lock operations, 3 reads/writes, and 0 poller reconfigurations, but still there is that unsightly task switch.

Even Better Message Transmission

The Ansible extension and new strategy work both offer something Ansible previously relied on SSH multiplexing to provide: a process where connection state persists during a run. As persistence is under our control, one final step becomes possible. Simply move SSH in-process:

Now we have 0 task switches, 0 loop iterations, 2 lock operations, 1 write, and 0 poller reconfigurations, or simply put, the minimum possible to support a threaded program communicating via SSH.

Some exciting possibilities emerge: passwords can be typed without allocating a PTY. Since usually Linux only supports 4,096 PTYs, this raises the scalability upper bound while reducing resource usage. Much better buffering is possible, eliminating Mitogen's own buffer, and optimally sizing SSH sockets to support file transfers.

Of course downsides exist: unlike libssh or libssh2, OpenSSH is part of a typical workflow, supports every authentication style, and it is common to stash configuration in ~/.ssh/config. Although libssh supports SSH configuration parsing, it's unclear how well it works in practice, and at least the author of ParallelSSH (and wrappers for both libraries) appears to have chosen libssh2 over it for reasons I'd like to discover.

Routing

For completeness, and since the diagrams exist already, here is routing between two SSH children from the context of their parent on the stable branch:

While internal switching is avoided, those nasty loop iterations are visible, as are the surrounding task switches. Optimistic sending benefits routing too:

Now the loop iterates once. Finally, with an in-process SSH client:

A single thread is woken, receives the message to be forwarded, delivers it, and sleeps all on one stack.

Summary

Complexity is fractal, but shying from it just leads to mediocre software. Both improvements exist as branches, and both will be supported by the Ansible extension in addition to the new work.

Until next time!

A fork in the road for Mitogen

dw — Mon, 27 Aug 2018 02:00:00 +0100

Mitogen for Ansible's original plan described facets of a scheme centered on features made possible by a rigorous single cohesive distributed program model, but of those facets, it quickly became clear that most users are really only interested in the big one: a much faster Ansible.

While I'd prefer feature work, this priority is fine: better performance usually entails enhancements that benefit the overall scheme, and improving people's lives in this manner is highly rewarding, so incentives remain aligned. It is impossible not to find renewed energy when faced with comments like this:

Enabling the mitogen plugin in ansible feels like switching from floppy to SSD
https://t.co/nCshkioX9h

Although feedback on the project has been very positive, the existing solution is sometimes not enough. Limitations in the extension and Ansible really bite, most often manifesting when running against many targets. In these scenarios, it is heartbreaking to see the work fail to help those who could benefit from it most, and that's what I'd like to talk about.

Controller-side Performance

Some time ago I began refactoring Ansible's linear strategy, aiming to get it to where controller-side enhancements might exist without adding more spaghetti, while becoming familiar with requirements for later features. To recap, the strategy plugin is responsible for almost every post-parsing task, including worker management. It is in many ways the beating heart at the core of every Ansible run.

After some months and one particularly enlightening conversation that work was resumed, eventually subsuming all of the remaining strategy support and result processing code, forming one huge refactor of a big chunk of upstream that has been gathering dust for almost a month.

The result exists today and is truly wonderful. It integrates Mitogen into the heart of Ansible without baking it in, introduces a carefully designed process model with strong persistence properties, eliminating most bottlenecks endured by the extension and vanilla Ansible, and provides an architectural basis for the next planned iteration of scalability work, Windows compatibility, some features mentioned, and quite a few that have been kept quiet.

With the new strategy it is possible to almost perfectly saturate an 8 vCPU machine given 100 targets, with minimal loss of speedup compared to single-target. Regarding single target, simple loops against localhost are up to 4x faster than the current stable extension.

While there are at least 2 obvious additional enhancements possible with this work, development reached a natural break in order to allow stablizing one piece of the puzzle at a time. Once this is done, it is clear exactly where to pick things up next.

Deep Cuts

There's just a small hitch: this work goes deep, entailing changes that, while so far would be possible as monkey-patches, are highly version-specific, and unlikely to remain monkey-patchable as the branch receives real-world usage. There must be a mechanism to ship unknown future patches to upstream code.

It was hoped it could land after Ansible 2.7, benefitting from related changes planned upstream, but they appear to have been delayed or abandoned, and so a situation exists where improvements cannot be shipped for at least another 4-6 months, assuming the related changes finally arrived in Ansible 2.8.

To the right is a rough approximation of components involved in executing a playbook. Those modified or replaced by the stable extension are green, yellow are replaced by the branch-in-waiting. Finally in orange are components affected by planned features and optimizations.

Although there are tens of thousands of lines of surrounding code, as should hopefully be clear, the number of untouched major components involved in a run has been dwindling fast. Put simply, the existing mechanism for delivering improvements is reaching its limit.

The F Word

Any seasoned developer, especially those familiar with the size of the Ansible code base, will hopefully understand the predicament. There is no problem delivering improvements today, assuming an unsupported one-off code dump was all anyone wanted, but that is never the case.

The problem lies in entering an unsustainable permanent marriage with a large project, not forgetting this outcome was an explicit non-goal from the start. Simultaneously over the months significant trust has been garnered to deliver these kinds of improvements, and abandoning one of the best yet would seem foolish.

Something of a many-variabled optimization process has recently come to an end, and I've found a solution that I am comfortable with. While making a release needs more time and may still not be definite, it seemed worth documenting at least some of the reasoning behind it before it comes.

Even though this outcome was undesirable, and although the solution in mind is not without restraint, it is still a cloud with many silver linings. For instance, new user configuration steps can be reduced to almost zero, core features can be added with minimal friction, and creative limitations are significantly uncapped.

What About The Extension?

The planned structure keeps the extension front-and-centre, so regardless of outcome it will continue to receive the majority of feature work and maintenance. It is definitely not going away.

With a third stable release looming, it's probably high time for a quick update. Many bugs were squashed since July, with stable work recently centered around problems with Ansible 2.6. This involved some changes to temporary file handling, and in the process, discovery of a huge missed optimization.

v0.2.3 will need only 2 roundtrips for each copy and template, or in terms of a 250ms transcontinental link, 10 seconds to copy 20 files vs. 30 seconds previously, or 2 minutes compared to vanilla's best configuration. This work is delayed somewhat as a new RPC chaining mechanism is added to better support all similar future changes, and identical situations likely to appear in similar tools.

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-03-28: Crowdfunding Mitogen: day 23
2018-04-20: Crowdfunding Mitogen: day 46
2018-05-23: Mitogen for Ansible status, 23 May
2018-07-10: Mitogen released!

Until next time!

Mitogen released!

dw — Tue, 10 Jul 2018 18:08:00 +0100

After 4 months development, a design phase stretching back 10 years and more than 1,300 commits, I am pleased to finally announce the first stable series of Mitogen and the Mitogen for Ansible extension.

Mitogen is a Python zero-deploy distributed programming library designed to drastically increase the functional capability of infrastructure software operating via SSH. Mitogen for Ansible is a drop-in replacement for Ansible's lower tiers, netting huge speed and efficiency improvements for common playbooks.

What's There

This initial series covers a widely compatible drop-in Ansible extension on Python 2.6, 2.7, and 3.6, a preview of the first value-added functionality for Ansible (Connection Delegation), and a freeze of the underlying library required to support it.

With the exception of some gotchas listed in the release notes, you should expect the Ansible extension to just work, and if it doesn't please let me know via GitHub.

Demo

Refer to the posts under Just Tuning In? below for a 1000 foot view of the direction this work is heading, but for an idea of how things are today, watch the first minute of this recording, demonstrating a loop-heavy configuration of Mitogen's tests executing against the local machine.

Installation

To install Mitogen for Ansible, just follow the 5 easy steps in the documentation. For non-Ansible users the library is available from PyPI via pip install mitogen. Introductory documentation for the library is very weak right now, it will improve over the course of the stable series.

Thanks to all the supporters!

Mitogen development in 2018 was sponsored by a fabulous group of individuals and businesses through a crowdfunding campaign launched in February. Thanks to everyone who participated by pledging, testing, writing bug reports, and helping with upfront planning. A huge special thanks to the primary sponsor:

Founded in 1976, CGI is one of the world’s largest IT and business consulting services firms, helping clients achieve their goals, including becoming customer-centric digital organizations.

For career opportunities, please visit cgi-group.co.uk/defence-and-intelligence-opportunities.

To directly apply to a UK team currently using Mitogen, contact us regarding Open Source Developer/DevOps opportunities.

What's next

Feature work will resume after most issues are ironed out of the stable series -- in particular I'm expecting more bugs around Python 3 and cross 2/3 interoperability. Once 0.2.x looks solid, one important goal is a complete and widely compatible Connection Delegation feature, including a rewrite of the fakessh component to support transparent use of the synchronize module.

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-03-28: Crowdfunding Mitogen: day 23
2018-04-20: Crowdfunding Mitogen: day 46
2018-05-23: Mitogen for Ansible status, 23 May

Until next time!

Mitogen for Ansible status, 23 May

dw — Wed, 23 May 2018 00:32:49 +0100

This is the third update on the status of developing Mitogen for Ansible.

Too long, didn’t read

A beta is coming soon! Aside from async tasks, the master branch is looking great. Since last update there have been many features and fixes, but with important forks in the road ahead, particularly around efficient support for many-host. Read on..

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-03-28: Crowdfunding Mitogen: day 23
2018-04-20: Crowdfunding Mitogen: day 46

Done: File Transfer

File transfer previously worked by constructing one RPC representing the complete file, which for large files resulted in an explosion in memory usage on each machine as the message was enqueued and transferred, with communication at each hop blocked until the message was delivered. This has required a rewrite since the original code was written, but a simple solution proved elusive.

Today file transfer is all but solved: files are streamed in 128KiB-sized messages, using a dedicated service that aggregates pending transfers by their most directly connected stream, serving one file at a time before progressing to the next transfer. An initial burst of 128KiB chunks is generated to fill a link with a 1MiB BDP, with further chunks sent as acknowledgements begin to arrive from the receiver. As an optimization, files 32KiB or smaller are still delivered in a single RPC, avoiding one roundtrip in a common scenario.

Compared to sftp(1) or scp(1), the new service has vastly lower setup overhead (1 RTT vs. 5) and far better safety properties, ensuring concurrent use of the API by unrelated ansible-playbook runs cannot create a situation where an inconsistent file may be observed by users, or a corrupt file is deployed with no indication a problem exists.

Since file transfer is implemented in terms of Mitogen's message bus, it is agnostic to Connection Delegation, allowing streaming file transfers between proxied targets regardless of how the connection is set up.

Some minor problems remain: the scheduler cannot detect a timed out transfer, risking a cascading hang when Connection Delegation is in use. This is not a regression compared to previously, as Ansible does not support this operation mode. In both cases during normal operation, the timeout will eventually be noticed when the underlying SSH connection times out.

Connection Delegation

Connection Delegation enables Ansible to use one or more intermediary machines to reach a target machine or container, with connections and code uploads deduplicated at each hop in the path. For an Ansible run against many containers on one target host, only one SSH connection to the target need exist, and module code need only be uploaded once on that connection.

While not yet complete, this feature exists today and works well, however some important functionality is still missing. Presently intermediary connection setup is single threaded, non-Python (i.e. Ansible) module uploads are duplicated, and the code to infer intermediary connection configurations using the APIs available in Ansible is.. hairy at best.

Fixing deduplication and single-threaded connection setup entails starting a service thread pool within each interpreter that will act as an intermediary. This requires some reworking of the nascent service framework, also making it easier to use for non-Ansible programs, and lays the groundwork for Topology-aware File Synchronization.

Custom module_utils

From the department of surprises, this one is a true classic. Ansible supports an undocumented (upstream docs patch) but nonetheless commonly used mechanism for bundling third party modules and overriding built-in support modules as part of the ZIP file deployed to the target. It implements this by virtualizing a core Ansible package namespace: ansible.module_utils, causing what Python finds there to vary on a per-task basis, and crucially, to have its implementation diverge entirely from the equivalent import in the Ansible controller process.

It is suffice to say I nearly lost my mind on discovering this "feature", not due to the functionality it provides, but the manner in which it opts to provide it. Rather than loading a core package namespace as a regular Python package using Mitogen's built-in mechanism, every Ansible module must undergo additional dependency scanning using its unique search path, and any dependencies found must correctly override existing loaded modules appearing in the target interpreter's namespace at runtime.

Given Mitogen's intended single-reusable-interpreter design, there is no way to support this without tempting strange behaviours appearing across tasks whose ansible.module_utils search path varies. While it is easy to arrange for ansible.module_utils.third_party_module to be installed, it is impossible to uninstall it while ensuring every reference to the previous implementation, including instances of every type defined by it, are extricated from the reusable interpreter post-execution, which is necessary if the next module to use the interpreter imports an entirely distinct implementation of ansible.module_utils.third_party_module.

Today, instead the interpreter forks when an extended or overridden module is found, and a custom importer is used to implement the overrides. This introduces an unavoidable inefficiency when the feature it in use, but it is still far better than always forking, or running the risk of varying module_utils search paths causing unfixable crashes.

Container Connections

To aid a common use-case for Connection Delegation, new connection types were added to support Linux containers and FreeBSD jails. It is now possible to run Ansible within a remote container reached via SSH, solving a common upstream feature request.

Presently although the container must have Python installed, matching Ansible's existing behaviour, it occurred to me that when the host machine has Python installed, there is no reason why Python needs to exist within the container. This would make a powerful feature made easy through Mitogen's design, and in a common use case, would support the ability to run auditing/compliance playbooks against app containers that were otherwise never customized for use with Ansible.

Su Become Method Support

Low-hanging fruit from the original crowdfunding plan. Now su(1) may be used for privilege escalation as easily as sudo(1).

Sudo/Su Connection Types

To support testing and somewhat uncommon use cases where a large number of user accounts may be targeted for parallel deployment on a small number of machines, there now exist explicit mitogen_sudo and mitogen_su connection types that, in combination with Connection Delegation, allow a single SSH connection to exist to a remote machine while exposing user accounts as individual (and therefore parallelizable) targets in Ansible inventory.

This sits somewhere between "hack" and "gorgeous", I really have no idea which, however it does make it simple to exploit Ansible's parallelism in certain setups, such as traditional web hosting where each customer exists as a UNIX account on a small number of machines.

Security

Unidirectional Routing exists and is always enabled for Ansible. This prohibits what was previously a new communication style available to targets, that, although ideally benign and potentially very powerful, fundamentally altered Ansible's security model and risked solution acceptance. It was possible for targets to send each other messages, and although permission checks occur on reception and thus should be harmless, represented the ability for otherwise air-gapped networks to be temporarily bridged for the duration of a run.

Secrets Masking

Mitogen supports new Blob() and Secret() string wrappers whose repr() contains a substitute for the actual value. These are employed in the Ansible extension, ensuring passwords and bulk file transfer data are no longer logged when verbose output is enabled. The types are preserved on deserialization, ensuring log messages generated by targets receive identical treatment.

User/misc bug fixes

Asynchronous Tasks (.. again, and again)

Ongoing work on the asynchronous task implementation has caused it to evolve once again, this time to make use of a new subtree detachment feature in the core library. The new approach is about 70% of what is needed for the final design, with one major hitch remaining.

Since an asynchronous task must outlive its parent, it must have a copy of every dependency needed by the module it will execute prior to disconnecting from the parent. This is exorbitantly fiddly work, interacting with many aspects including not least custom module_utils, and represents the last major obstacle in producing a functionally complete extension release.

Industrial grade multiplexing

Mitogen now supports swapping select(2) for epoll(7) or kqueue(2) depending on the host operating system, blasting through the maximum file descriptor limit of select(2), and ensuring this is no longer a hindrance for many-target runs. Children initially use the select(2) multiplexer (tiny and guaranteed available) until they become parents, when the implementation is transparently swapped for the real deal.

In future some interface tweaks are desirable to make full use of the new multiplexers: at least epoll(4) supports options that significantly reduce the system calls necessary to configure it. Although I have not measured a performance regression due to these calls, their presence is bothersome.

Many-Target Performance

Some expected growing pains appeared when real multiplexing was implemented. For testing I adopted a network of VMs running DebOps common.yml, with a quota for up to 500 targets, but so far, it is not possible to approach that without drowning in the kinks that start to appear. While some of these almost certainly lie on the Mitogen side, when profiling with only 40 targets enabled, inefficiencies in Mitogen are buried in the report by extreme inefficiencies present in Ansible itself.

Among the problems:

25% runtime wasted calling glob() (task setup stress test, not a real playbook)
10% runtime wasted enumerating template filters (stress test)
50% runtime wasted constructing task variables pre-fork (real run)
>50% runtime wasted recompiling templates (stress test)

And with that we reach a nexus: we have almost exhausted what can be accomplished working from the bottom-up, profiling on a micro scale is no longer sufficient to meet project goals, while fixing problems identified through profiling on a macro scale exceeds the project scope. Therefore, (lightning bolts, wild cackles), a new plan emerges..

Branching for a beta

With the exception of async tasks I consider the master branch to be in excellent health - for smaller target counts. For larger runs, wider-reaching work is necessary, but it does not make sense to disrupt the existing design due to it. Therefore master will be branched with the new branch kept open for fixes, not least the final pieces of async, while continuing work in parallel on a new increment.

Extension v2

Vanilla Ansible forks each time it executes a task, with the corresponding action plug-in gaining control of the main thread until completion, upon which all state aside from the task result is lost. When running under the extension, a connection multiplexer process is forked once at startup, and a separate broker thread exists in each forked task subprocess that connects back to the connection multiplexer process over a UNIX socket - necessary in the current design to have a persistent location to manage connections.

The new design comes in the form of a complete reworking of the Ansible linear strategy. Today's extension wraps Ansible's strategies while preserving their process and execution model. To implement the enhancements above sensibly, additional persistence is required and it becomes necessary to tackle a strategy implementation head-on.

The old desire for per-CPU connection multiplexers is incorporated, but moves those multiplexers back into Ansible, much like the pre-crowdfund extension. The top-level controller process gains a Mitogen broker thread with per-CPU forked children acting as connection multiplexers, and hosting service threads on which action plug-ins can sleep. Unlike vanilla Ansible, these processes exist for the duration of the run rather than per-task.

From the vantage point of only $ncpus processes, it is easy to fix template precompilation, plug-in path caching, connection caching, target<->worker affinity, and ensuring task variable generation is parallelized. Some sizeable obstacles exist, not least:

Liberal shared data structure mutation in the task executor that must be fixed to handle threading, mostly contained to PlayContext.
Preserving the existing callback plug-in model. Callbacks must always fire in the top-level process.
Synchronization or serialization overhead, pick one. Either the strategy logic runs duplicate in each child (requiring coordination with the top-level process), or it runs once in the parent, and configuration must be serialized for every task.

Can't this be done upstream?

It should, but I've experimented and there simply isn't time. If >1 week is reasonable to add missing documentation, there is no hope real patches will land before full-time work must conclude. For upstreaming to happen the onus lies with the 20+ strong permanent team, it's simply not possible to commit unbounded time to land even trivial changes, a far cry from occasional patches to a privately controlled repository.

At least 16k words have been spent since conversations started around September 2017, and while they bore some fruit over time, few actionable outcomes have resulted, and the detectable levels of team-originated engagement regarding the work has been minimal. There is no expectation of fireworks, however it may be helpful to realize after 3 months no evidence exists of any member testing the code and experiencing success or failure, let alone a report of such.

It's sufficient to say after so long I find this increasingly troublesome, and while I cannot hope to understand internal priorities, as an outside contributor funded by end users, soliciting engagement on a well-documented enhancement that in some scenarios nets an order of magnitude performance improvement to a commercial product, some rather basic questions come to mind.

Code Quality

There is a final uneasy aspect to upstreaming, and it is that of being left with the task of cleaning up, with no guarantee the mess won't simply return. Some of this code is in an abject (253 LOC, 37 locals) state (279 LOC, 24 locals) of sin (306 LOC, 38 locals), for 2018 and in a product less than 72 months old, that has been funded almost since inception. While I have begun refactoring the strategy plug-in within the confines of the Mitogen repository, responsibility for benefitting from that work in mainline rests with others.

Until next time!

Crowfunding Mitogen: day 46

dw — Fri, 20 Apr 2018 18:41:31 +0100

This is the second update on the status of developing the Mitogen extension for Ansible, only 2 weeks late!

Too long, didn’t read

Gearing up to remove the scary warning labels and release a beta! Running a little behind, but not terribly. Every major risk is solved except file transfer, which should be addressed this week.

23 days, 257 commits, 186 files changed, 7292 insertions(+), 1503 deletions(-)

Just tuning in?

2017-09-15: Mitogen, an infrastructure code baseline that sucks less
2018-03-06: Quadrupling Ansible performance with Mitogen
2018-03-28: Crowdfunding Mitogen: day 23

Started: Python 3 Support

A very rough branch exists for this, and I’m landing volleys of fixes when I have downtime between bigger pieces of work. Ideally this should have been ready for the end of April, but it may take a few weeks more.

I originally hoped to have a clear board before starting this, instead it is being interwoven as busywork when I need a break from whatever else I’m working on.

Done: multiplexer throughput

The situation has improved massively. Hybrid TTY/socketpair mode is a thing and as promised it significantly helps, but just not quite as much as I hoped.

Today on a 2011-era Macbook Pro Mitogen can pump an SSH client/daemon at around 13MB/sec, whereas scp in the same configuration hits closer to 19MB/sec. In the case of SSH, moving beyond this is not possible without a patched SSH installation, since SSH hard-wires its buffer sizes around 16KB, with no ability to override them at runtime.

With multiple SSH connections that 13MB should cleanly multiply up, since every connection can be served in a single IO loop iteration.

A bunch of related performance fixes were landed, including removal of yet another special case for handling deferred function calls, only taking locks when necessary, and reducing the frequency of the stream implementations modifying the status of their descriptors' readability/writeability.

As we’re in the ballpark of existing tools, I’m no longer considering this as much of a priority as before. There is definitely more low-hanging fruit, but out-of-the-box behaviour should no longer raise eyebrows.

Done: task isolation

As before, by default each script is compiled once, however it is now re-executed in a spotless namespace prior to each invocation, working around any globals/class variable sharing issues that may be present. The cost of this is negligible, on the order of 100 usec.

When this is insufficient, a mitogen_task_isolation=fork per-task variable exists to allow explicitly forcing a particular module to run in a new process. Enabling this by default causes something on the order of a 33% slowdown, which is much better than expected, but still not good enough to enable forking by default.

Aside from building up a blacklist of modules that should always be forked, task isolation is pretty much all done, with just a few performance regressions remaining to fix in the forking case.

Done: exotic module support

Every style of Ansible module is supported aside from the prehistorical “module replacer” type. That means today all of these work and are covered by automated tests:

Built-in new-style Python scripts
User-supplied new-style Python scripts
Ancient key=value style input scripts
Statically linked Go programs
Perl scripts

Python module support was updated to remove the monkey-patching in use before. Instead, sys.stdin, sys.stdout and sys.stderr are redirected to StringIO objects, allowing a much larger variety of custom user scripts to be run in-process even when they don’t use the new-style Ansible module APIs.

Done: free strategy support

The "free" strategy can now be used by specifying ANSIBLE_STRATEGY=mitogen_free. The mitogen strategy is now an alias of mitogen_linear.

Done: temporary file handling

This should be identical to Ansible’s handling in all cases.

Done: interpreter recycling

An upper bound exists to prevent a remote machine from being spammed with thousands of Python interpreters, which was previously possible when e.g. using a with_items loop that templatized become_user.

Once 20 interpreters exist, the extension shuts down the most recently created interpreter before starting a new one. This strategy isn’t perfect, but it should suffice to avoid raised eyebrows in most common cases for the time being.

Done: precise standard IO emulation

Ansible’s complex semantics for when it does/does not merge stdout and stderr during module runs are respected in every case, including emulation of extraneous \r characters. This may seem like a tiny and pointless nit, however it is almost certainly the difference between a tested real-world playbook succeeding under the extension or breaking horribly.

Done: async tasks

We’re on the third iteration of asynchronous tasks, and I really don’t want to waste any more time on it. The new implementation works a lot more like Ansible’s existing implementaion, for as much as that implementation can be said to “work” at all.

Done: better error messages

Connection errors no longer crash with an inscrutible stack trace, but trigger Ansible’s internal error handling by raising the right exception types.

Mitogen’s logging integration with the Ansible display framework is much improved, and errors and warnings correctly show up on the console in red without having to specify -vvv.

Still more work to do on this when internal RPCs fail, but that’s less likely to be triggered than a connection error.

New debugging mode

An “emergency” debugging mode has been added, in the form of MITOGEN_DUMP_THREAD_STACKS=1. When this is present, every interpreter will dump the stack of every thread into the logging framework every 5 seconds, allowing hangs to be more easily diagnosed directly from the controller machine’s logs.

While adding this, it struck me that there is a really sweet piece of functionality missing here that would be easy to add – an interactive debugger. This might turn up in the form of an in-process web server allowing viewing the full context hierarchy, and running code snippets against remotely executing stacks, much like Werkzeug’s interactive debugger.

Performance regressions

In addition to simply not being my focus recently, a lot of the new functionality has introduced import statements that impact code running in the target, and so performance has likely slipped a little from the original posted benchmarks, most likely during run startup in the presence of a high latency network.

I will be back to investigate these problems (and fix those for which no investigation is required – the module loader!) once all remaining functionality is stable.

File Transfer

This seemingly simple function has required the greatest deal of thought out of every issues I’ve encountered so far. The initial problem relates to flow control, and the absense of any natural mechanism to block a producer (file server) while intermediary pipe buffers (i.e. the SSH connection) are filled.

Even when flow control exists, an additional problem arises since with Mitogen there is no guarantee that one SSH connection = one target machine, especially once connection delegation is implemented. Some kind of bandwidth sharing mechanism must also exist, without poorly reimplementing the entirety of TCP/IP in a Python script.

For the initial release I have settled on basic design that should ensure the available bandwidth is fully utilized, with each upload target having its file data served on a first-come-first-served basis.

When any file transfer is active, one of the service threads in the associated connection multiplexer process (the same ones used for setting up connections) will be dedicated to a long-running loop that monitors every connected stream’s transmit queue size, enqueuing additional file chunks as the queue drains.

Files are served one-at-a-time to make it more likely that if a run is interrupted, rather than having every partial file transfer thrown away, at least a few targets will have received the full file, allowing that copy to be skipped when the play is restarted.

The initial implementation will almost certainly be replaced eventually, but this basic design should be sufficient for what is needed today, and should continue to suffice when connection delegation is implemented.

Testing / CI

The smattering of unit and integration tests that exist are running and passing under Travis CI. In preparation for a release, master is considered always-healthy and my development has moved to a new dmw branch.

I’m taking a “mostly top down” approach to testing, written in the form of Ansible playbooks, as this gives the widest degree of coverage, ensuring that high level Ansible behaviour is matched with/without the extension installed. For each new test written, the result must pass under regular Ansible in addition to Ansible with the extension.

“Bottom up” type tests are written as needs arise, usually when Ansible’s user interface doesn’t sufficiently expose whatever is being tested.

Also visible in Travis is a debops_common target: this is running all 255 tasks from DebOps common.yml against a Docker instance. It’s the first in what should be 4-5 similar DebOps jobs, deploying real software with the final extension.

I have begun exploring integrating the extension with Ansible’s own integration tests, but it looks likely this is too large a job for Travis. Work here is ongoing.

Security

A few items have been chipped off the list.

Message source verification was audited everywhere, and is covered by automated tests.
All internal message handlers specify a policy indicating what kind of participants are allowed to deliver messages to them.
As above, but for mitogen.service. A service cannot be exposed without attaching an access policy to it.

Notably absent is unidirectional routing mode. I will make time to finish that shortly.

User bug fixes

Poor refactoring broke select EINTR handling
SSH password was being supplied as the sudo password
Acquiring a controlling TTY was fixed on FreeBSD

Summary

Super busy, slightly behind! Until next time..

Crowdfunding Mitogen: day 23

dw — Wed, 28 Mar 2018 08:22:33 +0100

This is the first in what I hope will be at least a bi-weekly series to keep backers up to date on the current state of delivering the Mitogen extension for Ansible. I’m trying to use every second I have wisely until every major time risk is taken care of, so please forgive the knowledge-dump style of this post :)

Haven't been following?

Introduction to Mitogen (September 2017)
Introduction to the Mitogen extension for Ansible (March 2018)

Too long, didn’t read

Well ahead of time. Some exciting new stuff popped up, none of it intractably scary.

Funding Update

I have some fabulous news on funding: in addition to what was already public on Kickstarter, significant additional funding has become available, enough that I should be able to dedicate full time to the project for at least another 10 weeks!

Naturally this has some fantastic implications, including making it significantly likely that I’ll be able to implement Topology-aware File Synchronization.

Python 3 Support

I could not commit to this due to worrying Python 3 would become a huge and destablizing time sink, ruining any chance of delivering more immediately useful functionality.

The missing piece (exception syntax) to support from Python 2.4 all the way to 3.x has been found - it came via an extraordinarily fruitful IRC chat with the Ansible guys, and was originally implemented in Ansible itself by Marius Gedminas. With this last piece of the puzzle, the only bugs left to worry about are renamed imports and the usual bytes/str battles. Both are trivial to address with strong tests - something already due for the coming weeks. It now seems almost guaranteed Python 3 will be completed as part of this work, although I am still holding off on a 100% commitment until more pressing concerns are addressed.

New Risk: multiplexer throughput

Some truly insane performance bugs have been found and fixed already, particularly around the stress caused by delivering huge single messages, however during that work a new issue was found: IO multiplexer throughput truly sucks for many small messages.

This doesn’t impact things much except in one area: file transfer. While I haven’t implemented a final solution for file transfer yet, as part of that I will need to address what (for now) seems a hard single-thread performance limit: Mitogen’s current IO loop cannot push more than ~300MiB/sec in 128KiB-sized chunks, or to put it another way, best case 3MiB/sec given 100 targets.

Single thread performance: the obvious solution is sharding the multiplexer across multiple processes, and already that was likely required for completing the multithreaded connect work. This is a straightforward change that promises to comfortably saturate a Gigabit Ethernet port using a 2011 era Macbook while leaving plenty of room for components further up (Ansible) and down (ssh) the stack.

TTY layer: I’ve already implemented some fixes for this (increase buffer sizes, reduce loop iterations), but found some ugly new problems as a result: the TTY layer in every major UNIX has, at best, around a 4KiB buffer, forcing many syscalls and loop iterations, and it seems on no OS is this buffer tunable. Fear not, there is already a kick-ass solution for this too.

This problem should disappear entirely by the time real file transfer support is implemented - today the extension is still delivering files as a single large message. The blocker to fixing that is a missing flow control mechanism to prevent saturation of the message queue, which requires a little research. This hopefully isn’t going to be a huge amount of work, and I’ve already got a bunch of no-brainer yet hacky ways to fix it.

New risk: task isolation

It was only a matter of time, but the first isolation-related bug was found, due to a class variable in a built-in Ansible module that persists some state across invocations of the module’s main() function. I’d been expecting something of this sort, so already had ideas for solving it when it came up, and really it was quite a surprise that only one such bug was reported out of all those reports from initial testers.

The obvious solution is forking a child for each task by default, however as always the devil is in the details, and in many intractable ways forking actually introduces state sharing problems far deadlier than those it promises to solve, in addition to introducing a huge (3ms on Xeon) penalty that is needless in most cases. Basically forking is absolute hell to get right - even for a tiny 2 kLOC library written almost entirely by one author who wrote his first fork() call somewhere in the region of 20 years ago, and I’m certain this is liable to become a support nightmare.

The most valuable de facto protection afforded by fork - memory safety, is pretty redundant in an almost perfectly memory safe language like Python, that’s why the language is so popular at all.

Meanwhile forking is needed anyway for robust implementation of asynchronous tasks, so while implementing it would never have been wasted work, it is not obvious to me that forking could or should ever become the default mode. It amounts to a very ripe field for impossible to spot bugs of much harder classes than the simple solution of running everything in a single process, where we only need to care about version conflicts, crap monkey patches, needlessly global variables and memory/resource leaks.

I’m still exploring the solution space for this one, current thinking is maybe (maybe! this is totally greenfield) something like:

Built-in list of fixups for ridiculously easy to repair bugs, like the yum_repository example above.
Whitelist for in-process execution any module known (and manually audited) to be perfectly safe. Common with_items modules like lineinfile easily fit in this class.
Whitelist for in-process safe but nonetheless leaky modules, such as the buggy yum_repository module above that simply needs its bytecode re-executed (100usec) to paper over the bug. Can’t decide whether to keep this mode or not - or simply merge it with the above mode.
Default to forking (3ms - max 333 with_items/sec) for all unknown bespoke (user) modules and built-in modules of dubious quality, with a mitogen_task_isolation variable permitting the mode to be overridden by the user on a per-task basis. “Oh that one loop is eating 45 minutes? Try it with mitogen_task_isolation=none”

All the Mitogen-side forking bits are implemented already, and I’m deferring the Ansible-side bits to be done simultaneous to supporting exotic module types, since that whole chunk of code needs a rewrite and no point in rewriting it twice.

Meanwhile whatever the outcome of this work, be assured you will always have your cake and eat it - this project is all about fixing performance, not regressing it. I hope this entire topic becomes a tiny implementation detail in the coming weeks.

On the testing front I was absolutely overjoyed to discover DebOps by way of a Mitogen bug report. This deserves a whole article on its own, meanwhile it represents what is likely to be a huge piece of the testing puzzle.

Multithreaded connect

A big chunk is already implemented in order to fix an unrelated bug! The default pool size has 16 threads in one process, so there will only be a minor performance penalty for the first task to run when the number of targets exceeds 16. Meanwhile, the queue size is adjustable via an environment variable. I’ll tidy this up later.

Even though it basically already exists, I’m not yet focused on making multithreaded connect work - including analysing the various performance weirdness that appears when running Mitogen against multiple targets. These definitely exist, I just haven’t made time yet to determine whether it’s an Ansible-side scaling issue or a Mitogen-side issue. Stay tuned and don’t worry! Multi-target runs are already zippy, and I’m certain any issues found can be addressed.

Security

At least a full day will be dedicated to nothing but coming up with new attack scenarios, meanwhile I’m feeling pretty good about security already. The fabulous Alex Willmer has been busily inventing new cPickle attack scenarios, and some of them are absolutely fantastically scary! He’s sitting on at least one exciting new attack that represents a no-brainer decider on the viability of keeping cPickle or replacing it.

Serialization aside, I’ve been busy comparing Ansible’s existing security model to what the extension provides today, and have at least identified unidirectional routing mode as a must-have for delivering the extension. Regarding that, it is possible to have a single playbook safely target 2 otherwise completely partitioned networks. Today with Mitogen, one network could route messages towards workers in the other network using the controller as a bridge. While this should be harmless (given existing security mitigations), it still introduces a scary capability for an attacker that shouldn’t exist.

Some more security bugs I’m fixing here

Deferring Windows support

Really screwed up on planning here - turns out Ansible on Windows does not use Python whatsoever, and so implementing the support in Mitogen would mean increasing the installation requirements for Windows targets. That’s stupid, it violates Ansible’s zero-install design and was explicitly a non-goal from the get go.

Meanwhile WinRM has extremely poor options for bidirectional IO, and likely viable Mitogen support for Windows will include introducing a, say, SSL-encrypted reversion connection from the target machine in order to get efficient IO.

I will shortly be polling everyone who has pledged towards the project, and if nobody speaks up to save Windows, it’s being pushed to the back of the queue.

A big, big thanks, once again!

It goes without saying but none of this work has been a lone effort, starting from planning, article review, funding, testing, and an endless series of suggestions, questions and recommendations coming from so many people. Thanks to everyone, whether you contributed a single $1 or a single typo bug report.

Summary

Super busy, but also super on target! Until next time..

Kickstarting free software: one week later

dw — Tue, 13 Mar 2018 14:20:28 +0000

It’s been an incredibly intense first week crowdfunding the Mitogen extension for Ansible involving far more effort than anticipated, where I have worked almost flat out from waking until the early hours just to ensure any queries are answered thoroughly. I cannot complain, because it has been so much fun that I’d change almost nothing of the experience, and already the campaign has reached 46% from the exposure it received.

As a recap Mitogen is a library for writing distributed programs that require zero deployment, with the prototype extension implementing an architectural change that vastly improves Ansible’s performance in common scenarios, laying a framework to extend this advantage far beyond simple overhead reduction.

Initial testers

A great deal of work has simply been staying on top of bug reports and ensuring experiences with the prototype are solid – for each report from one tester, we can assume 10 more hit the same bug but did not or could not report it.

Of the many reports received, I have addressed almost all of them promptly. Some fabulous bugs have been found and fixed along with one report via Reddit of a performance improvement so fantastical that it exceeds even my most contrived overhead-heavy example:

"With mitogen my playbook runtime went from 45 minutes to just under 3 minutes. Awesome work!"

This is a common theme – anywhere with_items appears, Mitogen has the most profound impact. The obvious reason is that during loops the same module is executed repeatedly, and after one iteration is guaranteed to be compiled and ready on the target.

So many lessons!

Developing the campaign from a thought exercise one idle Sunday evening into an actually practical project has taken a lot of work – far more than I anticipated, and at almost every step I have learned something novel. This is all reuseable knowledge for anyone attempting a similar project in future, and I will write it up as time permits.

Regardless of outcomes the campaign has already proven one very exciting result: real users will stake real money towards something as seemingly mundane as free infrastructure, and I think that’s beyond amazing. In a world content to throw millions of dollars at junk ICOs almost weekly, crowdfunding free software seems to me a practice that should happen far more often.

Thank you

I wish to thank everyone for the support shown thus far, and I’d encourage you to consider tapping that Ansible user you know on the shoulder to let them know about the project. For those working close to infrastructure consulting, please consider using the final week to corner your boss regarding associating your company logo with a sexy project that promises to receive many eyeballs over the coming years.

Thanks for reading!

David.

Quadrupling Ansible performance with Mitogen

dw — Tue, 06 Mar 2018 12:58:09 +0000

[tl;dr: the Mitogen extension for Ansible exists today and it’s as awesome as I promised, but I want to push things much further. If you value free time, this project needs your support]

Allegedly on site as a developer, two summers ago I found myself in a situation you are no doubt familiar with, where despite preferences unrelated problems inevitably gravitate towards whoever can deal with them. Following an exhausting day spent watching a dog-slow Ansible job fail repeatedly, one evening I dusted off a personal aid to help me relax: an ancient, perpetually unfinished hobby project whose sole function until then had simply been to remind me things can always improve.

Something of a miracle had struck by the early hours of next morning, as almost every outstanding issue had been solved, and to my disbelief the code ran reliably. 18 months later and for the first time in living memory, I am excited to report delivery of that project, one of sufficient complexity as to have warranted extreme persistence - in this case from concept to implementation, over more than a decade.

The miracle? It comes in the form of Mitogen - a tiny Python library you won’t have heard of, but I hope as an Ansible user you will soon eternally be glad for, on discovering ansible-playbook now completes in very reasonable time even in the face of deeply unreasonable operating conditions.

Mitogen is a library for writing distributed programs that require zero deployment, specifically designed to fit the needs of infrastructure software like Ansible. Without upfront configuration it supports any UNIX machine featuring an installed Python interpreter, which is to say almost all of them. While the concept is hard to explain - even to fellow engineers, its value is easy to grasp:

This trace shows two Ansible runs of a basic 100-step playbook over a 1 ms latency network against a single target host. The first run employs SSH pipelining, Ansible’s current most optimal configuration, where it consumes almost 4.5 Mbytes network bandwidth in a running time of 59 secs.

The second uses the prototype Mitogen extension for Ansible, with a far more reasonable 90 Kbytes consumed in 8.1 secs. An unmodified playbook executes over 7 times faster while consuming 50x less bandwidth.

Less than half the CPU time was consumed on the host machine, meaning that by one metric it should handle at least twice as many targets. Crucially no changes were required to the target machine, including new software or nasty on-disk caches to contend with.

While only pure overhead is measured above, the benefits very much extend to real-world scenarios. See the documentation and issue #85 (4.2x time, 3.1x CPU) for examples.

How is this possible?

Mitogen is perhaps most easily described as a kind of network-capable fork() on steroids. It allows programs to establish lazily-loaded duplicates on remote hosts, without requiring any upfront remote disk writes, and to communicate with those copies once they exist. The copies can in turn recursively split to produce further children - with bidirectional message routing between every copy handled automatically.

In the context of Ansible, unlike with SSH pipelining where up to one SSH invocation, sudo invocation and script compilation are required for every playbook step, and with all scripts re-uploaded for each step, with Mitogen only one of each exists per target for the duration of the playbook run, with all code cached in RAM between steps. Absolutely everything is reused, saving 300-800 ms on every step.

The extension represents around a week’s work, replaces hundreds of lines of horrid shell-related code in Ansible, and is already at the point where on one real-world playbook, Ansible is only 2% slower than equivalent SSH commands. Presently connection establishment is single-threaded, so the prototype is only good for a few hosts, but rest assured this limitation’s days are numbered.

Not just a speed up, a paradigm shift you’ll adore

If this seems impressive and couldn’t be improved upon, prepare for some deep shocks. You can think of the extension not just as a performance improvement, but something of a surreptitious beachhead from which I intend to thoroughly assault your sense of reality.

This performance is a side effect of a far more interesting property: Ansible is no longer running on just the host machine, but temporarily distributed throughout the target network for the duration of the run, with bidirectional communication between all pieces, and you won’t believe the crazy functionality this enables.

What if I told you it were possible not only to eliminate that final 2%, but turn it sharply negative, while simultaneously reducing resource consumption? “Surely Ansible can’t execute faster than equivalent raw SSH commands?” You bet it can! And if you care about such things, this could be yours by Autumn. Read on..

Pushing brains into the ether, no evil agents required

As I teased last year, Ansible takes its name from a faster-than-light communication device from science fiction, yet despite these improvements it is still fundamentally bound by the speed with which information physically propagates. Pull and agent-based tooling is strongly advantageous here: control flow occurs at the same point as the measurements necessary to inform that flow, and no penalty is incurred for traversing the network.

Today, reducing latency in Ansible means running it within the target network, or in pull mode, where the playbook is stored on the target alongside for example, secrets for decrypting any vaults, and the hairy mechanics required to keep that in sync and executing when appropriate. This is a far cry from the simplicity of tapping ansible-playbook live.yml on your laptop, and so it is an option of last resort.

What would be amazing is some hybrid where we could have the performance and scaleability benefits of pull, combined with the stateless simplicity of push, without introducing dedicated hosts or permanent caches and agents running on the target machines, that amount to persistent intermediate state and introduce huge headaches of their own, all without sacrificing the fabulous ability to shut everything down with a simple CTRL+C.

The opening volley: connection delegation

As a first step to exploiting previously impossible functionality, I will enhance the extension to support delegating connection establishment to a machine on the target network, avoiding the cost of establishing hundreds of SSH connections over a low throughput, high latency network link.

Unlike with SSH proxying, this has the huge benefit of caching and serving Ansible code from RAM on the intermediary, avoiding uploading approximatey 50KiB of code for every playbook step, and ensuring those cached responses are delivered over the low latency LAN fabric on the target network. For 100 target machines, this replaces the transmission of 5 Mbytes of data for every playbook step with on the order of kilobytes worth of tiny remote procedure calls.

All the Mitogen-side infrastructure for this exists today, and is already used to implement become support. It could be flipped on with a few lines of code in the Ansible extension, but there are a few more importer bugs to fix before it’ll work perfectly.

Finally as a reminder, since Mitogen operates recursively delegation also operates recursively, with code caching and connection establishment happening at each hop. Not only is this useful for navigating slow links and complicated firewall setups, as we’ll see, it enables some exciting new scenarios.

Asynchronous Connect

Ansible is intended to manage many machines simultaneously, and while the extension’s improvements presently work well for single-machine playbooks, that is all but a niche application for many users.

Having the newfound ability to delegate connection establishment to an intermediary on the target network, far away from our laptop’s high latency 3G connection, and with the ability to further sub-delegate from that intermediary, we can implement a divide and conquer strategy, forming a large tree comprising the final network of target machines for the playbook run, with responsibility for caching and connection multiplexing evenly divided across the tree, neatly avoiding single resource bottlenecks.

I will rewrite Mitogen’s connection establishment to be asynchronous: creation of many downstream connections can be scheduled in parallel, with the ability to enqueue commands prior to completion, including recursive commands that would cause those connections to in turn be used as intermediaries.

The cost of establishing connections should become only the cost of code upload (~50KiB) and the latency of a single SSH connection per tree layer, as connections at each layer occur in parallel. For an imaginary 1,700 node cluster split into quarters of 17 racks and 25 nodes per rack, connection via a 300 ms 3G network should complete in well under 15 seconds.

Topology-aware file synchronization

So you have a playbook on your laptop deploying a Django application via the synchronize module, to 100 Ubuntu machines running in a datacentre 300 ms away. Each run of the playbook entails a groan followed by a long walk, as a 3.8 second rsync run is invoked 100 times via your 3G connection, just to synchronize a 3 Mbyte asset the design team won’t stop tweaking. Not only are there 6 minutes of roundtrips buried in those invocations, but that puny 3G connection is forced to send a total of 300 Mbytes toward the target network.

What is the point of continually re-sending that file to the same set of machines in some far-off network? What if it could be uploaded exactly once, then automatically cached and redistributed within the target network, producing exactly one upload per layer in the hierarchy:

Why stop at delegating connection establishment and module caching? Now we have a partial copy of Ansible within the network, nothing prevents implementing all kinds of smarts. Here is another feature that is a cinch to build once bidirectional communication exists between topology-aware code, which the prototype extension already provides today.

Generalized forwarding

After a brutal 4 hour meeting involving 10 executives our hero Bob, Senior Disaster Architect III, emerges bloodstained yet victorious against the tyrannical security team, as his backends can talk with impunity to the entire Internet just so apt-get can reach packages.debian.org for the 15 seconds Bob’s daily Ansible CI job requires.

That evening, having regaled his giddy betrothed (HR Coordinator II) with heroic story of war, Bob catches a brief yet chilling glimmer of doubt for all that transpired. “Was there another way?” he sleepily ponders, before succumbing to a cosier battle waged by those fatigued and heavy eyelids. Suddenly aware again, Bob emerges bathed in a mysterious utopian dreamscape where CI jobs executed infinitely quickly, war and poverty did not exist, and the impossible had always been possible.

Building on Mitogen’s message routing, forwarding all kinds of pipes and network sockets becomes trivial, including schemes that would allow exposing a transient, locked down HTTP proxy to Bob’s apt-get invocation only for as long as necessary, all with a few lines of YAML in a playbook.

While this is already possible with SSH forwarding, the hand-configuration involved is messy, and becomes extremely hairy when the target of the forward is not the host machine. My initial goal is to support forwarding of UNIX and TCP sockets, as they cover all use cases I have in mind. Speaking of which..

Topology-aware Git pull

Another common security fail seen in Ansible playbooks is to call Git directly from target machines, including granting those machines access to a Git server. This is a horrid violation: even read-only access implies the machine needs permanent firewall rules that shouldn’t exist, just for the scant moments a pull is in progress. Granting backends access to a site as complex as GitHub.com, you may as well abandon all outbound firewalling, as this is enough for even the puniest script kiddy to exfiltrate a production database.

What if Git could run with the permissions of the local Ansible user, on the user’s own machine, and be served efficiently to the target machines only for the duration of the push, faster than 100 machines talking to GitHub.com, and only to the single read-only repository intended?

Building on generalized forwarding, topology-aware Git repeats all the caching and single-upload tricks of file synchronization, but this time implementing the Git protocol between each node.

In the scheme I will implement, a single round-trip is necessary for git-fetch-pack to pull just the changed objects from the laptop over the high latency 3G link, before propagating at LAN speeds throughout the target network, with git-ls-remote output delivered as part of the message that initiates the pull. Not only is the result more efficient than a normal git-pull, but backends no longer require network access to Git.

The final word: Inversion of control

Remember we talked about making Ansible run faster than equivalent SSH commands? Well, today Ansible requires one network round-trip per playbook step, so just like SSH, it must pay the penalty for every round-trip unless something gives, and that something is the partial delegation of control to the target machine itself.

With inversion of control, the role of ansible-playbook simply becomes that of shipping code and selective chunks of data to target machines, where those machines can execute and make control decisions without necessitating a conversation with the master after each step, just to figure out what to execute next.

Ansible has all the framework to enable implementing this today, by significantly extending the prototype extension’s existing strategy plug-in, and teaching it how to automatically send and wait on batches of tasks, rather than on single tasks at a time.

Aside from improved performance, the semantics of the existing linear strategy will be preserved, and playbooks need not be changed to cope: on the target machine tasks will not suddenly begin running concurrently, or in any order different to previously.

App-level connection persistence

As a final battle against latency during playbook development and debugging, I will support detaching the connection tree from ansible-playbook on exit, and teach the extension to reuse it at startup. This will reduce the overhead of repeat runs, especially against many targets, to the order of hundreds of milliseconds, as no new SSH connections, module compilations or code uploads are required.

Connection persistence opens the floodgates for adding sweet new tooling, although I’m not sure how desirable it is to expose an implementation detail like this forever, while also extending the interface provided by Ansible itself. As a simple example, we could provide an ansible-ssh tool that reuses the connection tree along with Ansible’s tunnelling, delegation, dynamic inventory and authentication configuration to forward a pipe to a remote shell.

The cost of slow tooling

Ansible has over 28,500 stars on GitHub, representing just those users who have a GitHub account and ever thought to star it, and appears to grow by 150 stars per week. Around London the going rate to hire one user is $100/hour, and conservatively, we could expect that user is trotting out a 15 minute run of ansible-playbook live.yml at least once per week.

We can expect that if Ansible is running merely twice as slowly as necessary, 7.5 minutes of that run is lost productivity, and across those 28,500 users, the economic cost is in the region of $356,250 per invocation or $17,100,000 per year. In reality the average user is running Ansible far more often, including thousands of times per minute under various CI systems worldwide, and those runs often last far longer than 15 minutes, but I’d recommend that mental guesstimation is left as an exercise to readers who are already blind drunk.

The future is beautiful if you want it to be

My name is David, and nothing jinxes my day quite like slow tooling. I have poured easily 500 hours in some form into this project over a decade and on my own time. The project has now reached an inflection point where the fun part is over, the science is done and the effect is real, and only a small, highly predictable set of milestones remain to deliver what I hope you agree is a much brighter future.

Before reading I doubt you would have believed it possible to provide the features described without a complex infrastructure running in the target network, now I hope you’ll join me in disproving one final impossibility.

While everything here will exist in time, it cannot exist in 2018 without your support, and that’s why I’d like to try something crazy, that would allow me to devote myself to delivering a vastly improved daily routine for thousands of people just like you and me.

You may have guessed already: I want you to crowdfund awesome tooling.

What value would you place on an extra productive hour every working week? In the UK that’s an easy question: it’s around $4,800 per year. And what risk is there to contributing $100 to an already proven component? I hope you’ll agree this too is a no-brainer, both for you and your employer.

To encourage success I’m offering a unique permanent placement of your brand on the GitHub repository and documentation. Funds will be returned if the minimum goal cannot be reached, however just 3 weeks are sufficient to ensure a well tested extension, with my full attention given to every bug, ready to save many hours right on time to enjoy the early sunlight of Spring.

Totalling much less than the economic damage caused by a single run of today’s Ansible, the grand plan is divided into incrementally related stretch goals. I cannot imagine this will achieve full funding, but if it does, as a finale I’ll deliver a feature built on Ansible that you never dreamed possible.

What will that be? Pledge today if you’d like to find out.

Combating obsolescence in our beloved tools

As a modern area deployment tooling is exposed to the ebb and flow of the software industry far more than typical, and unexpected disruption happens continuously. Without ongoing evolution, exposure to buggy and unfamiliar new tooling is all but guaranteed, with benefits barely justifying the cost of their integration. As we know all too well, rational ideas like cost/benefit rarely win the hearts of buzzword-hungry and youthful infrastructure teams, so counterarguments must be presented another way.

As a recent example there is growing love for mgmt, which is designed from the outset as an agent-based reactive distributed system, much as Mitogen nudges Ansible towards. However unlike mgmt, Ansible preserves its zero-install and agentless nature, while laying a sound framework for significantly more exciting features. If that alone does not win loyalty, we’re at least guaranteed that every migration-triggering new feature implemented in such systems can be headed off with minimal effort, long into the foreseeable future.

David.

Much ado about latency: Mitogen and the BFG9000 of import hooks

dw — Tue, 13 Feb 2018 21:35:32 +0000

After a long winter break from recreational programming, over the past days I finally built up steam and broke a chunk of new ground on Mitogen, this time growing its puny module forwarder into a bona fide beast, ready to handle almost any network condition and user code thrown at it.

No adversary is a match for the BFG

Recap

Mitogen is a library for executing parts of a Python program in a remote context, primarily over sudo and SSH connections, and establishing bidirectional communication with those parts. Targeting infrastructure applications, it requires no upfront configuration of target machines, aside from an SSH daemon and Python 2.x interpreter, which is the default for almost every Linux machine found on any conceivable network.

The target need not possess a writeable filesystem, code is loaded dynamically on demand, and execution occurs entirely from RAM.

How Import Works

To implement dynamic loading, child Python processes (“contexts”) have a PEP-302 import hook installed that causes attempts to import modules unavailable locally to automatically be served over the network connection to the parent process. For example, in a script like:

        import mitogen
        import requests

        def get_url(url):
            return requests.get(url).text

        @mitogen.main()
        def main(router):
            host = router.ssh(hostname='k3')
            print host.call(get_url, 'https://www.google.com/')

If the requests package is missing on the host k3, it will automatically be copied and imported in RAM, without requiring upfront configuration, or causing or requiring writes to the remote filesystem.

Kathmandu to Paris via 3G: serious business

So far, so good. Just one hitch

While the loader has served well over the library’s prototypical life (which in real time, is approaching 12 years!), it has always placed severe limits on the structure of the loaded code, as each additional source file introduced one network round-trip to serve it.

Given a relatively small dependency such as Kenneth Reitz' popular Requests package, comprising 17 submodules, this means 17 additional network round-trips. While that may not mean much over a typical local area network segment where roundtrips are measured in microseconds, it quickly multiplies over even modest wide-area networks, where infrastructure tooling is commonly deployed.

For a library like Requests, 17 round-trips amounts to 340ms latency over a reasonably local 20ms link, which is comfortably within the realms of acceptable, however over common radio and international links of 200ms or more, already this adds at least 3.4 seconds to the startup cost of any Mitogen program, time wasted doing nothing but waiting on the network.

Sadly, Requests is hardly even the biggest dependency Mitogen can expect to encounter. For testing I chose django.db.models as a representative baseline: heavily integrated with all of Django, it transitively imports over 160 modules across numerous subpackages. That means on an international link, over 30 seconds of startup latency spent on one dependency.

It is worth note that Django is not something I’d expect to see in a typical Mitogen program, it’s simply an extraordinarily worst-case target worth hitting. If Mitogen can handle django.db.models, it should cope with pretty much anything.

Combining evils, over an admittedly better-than-average Nepali mobile data network, and an international link to my ~~IRC box~~ mail server in Paris, django.db.models takes almost 60 seconds to load with the old design.

In the real world, this one-file-per-roundtrip characteristic means the current approach sucks almost as much as Ansible does, which calls into doubt my goal of implementing an Ansible-trumping Ansible connection plug-in. Clearly something must give!

50.35 seconds and hundreds of roundtrips spent transferring django.db.models from Kathmandu to Paris via 3G. Despite a fast link, throughput averages 13KiB/sec and never exceeds 45KiB/sec. Well over half of the 989 frames sent are wasted on signalling (Y=0)

Trying harder

Over the years I discarded many approaches for handling this latency nightmare:

Having the user explicitly configure a module list to deliver upfront to new contexts, which sucks and is plainly unmaintainable.
Installing a PEP-302 hook in the master in order to observe the import graph, which would be technically exciting, but likely to suck horribly due to fragility and inevitable interference with real PEP-302 hooks, such as py2exe.
Observing the import graph caused by a function call in a single context, then using it to preload modules in additional contexts. This seems workable, except the benefit would only be felt by multiple-child Mitogen programs. Single child programs would continue to pay the latency tax.
Variants of 2 and 3, except caching the result as intermediate state in the master’s filesystem. Ignoring the fact persistent intermediate state is always evil (a topic for later!), that would require weird and imperfect invalidation rules, which means performance would suck during development and prototyping, and bugs are possible where state gets silently wedged and previously working programs inexplicably slow down.

Finally last year I settled on using static analysis, and restricting preloading at package boundaries. When a dependency is detected in a package external to the one being requested, it is not preloaded until the child has demonstrated, by requesting the top-level package module from its parent, that the child lacks all of the submodules contained by it.

This seems like a good rule: preloading can occur aggressively within a package, but must otherwise wait for a child to signal a package as missing before preemptively wasting time and bandwidth delivering code the child never needed.

As a final safeguard, preloading is restricted to only modules the master itself loaded. It is not sufficient for an import statement to exist: surrounding conditional logic must have caused the module to be loaded by the master. In this manner the semantics of platform, version-specific and lazy imports are roughly preserved.

Syntax tree hell

Quite predictably, after attempting to approach the problem with regexes, I threw my hands up on realizing a single regex may not handle every possible import statement:

import a
import a as b
from a import b
from a import b as c
from a import (b, c, d)

I gleefully thought I’d finally found a use for the compiler and ast modules, and these were the obvious alternative to avoiding the rats nest of multiple regexes. Not quite. You see, across Python releases the grammar has changed, and in lock-step so have the representations exported by the compiler and ast modules.

Adding insult to injury: neither module is supported through every interesting Python version. I have seen Python 2.4 deployed commercially as recently as summer 2016, and therefore consider it mandatory for the kind of library I want on my toolbelt. To support antique and chic Python alike, it was necessary to implement both approaches and select one at runtime. Many might see this is an opportunity to drop 2.4, but “just upgrade lol” is never a good answer while maintaining long shelf-life systems, and should never be a a barrier to applying a trusted Swiss Army Knife.

After some busy days last September, I had a working scanner built around syntax trees, except for a tiny problem: it was ridiculously slow. Parsing the 8KiB mitogen.core module took 12ms on my laptop, which multiplied up is over a second of CPU burnt scanning dependencies for a package like Django. If memory serves, reality was closer to 3 seconds: far exceeding the latency saved while talking to a machine on a LAN.

Sometimes hacking bytecode make perfect sense

I couldn’t stop groaning the day I abandoned ASTs. As is often true when following software industry best practice, we are left holding a decomposing trout that, while technically fulfilling its role, stinks horribly, costs all involved a fortune to support and causes pains worse than those it was intended to relieve. Still hoping to avoid regexes, I went digging for precedent elsewhere in tools dealing with the same problem.

That’s when I discovered the strange and unloved modulefinder buried in the standard library, a forgotten relic from a bygone era, seductively deposited there as a belated Christmas gift to all, on a gloomy New Year’s Eve 2002 by Guido’s own brother. Diving in, I was shocked and mesmerized to find dependencies synthesized by recompiling each module and extracting IMPORT_FROM opcodes from the compiled bytecode. Reimplementing a variant, I was overjoyed to discover django.db.models transitive dependencies enumerated in under 350ms on my laptop. A workable solution!

The solution has some further crazy results: IMPORT_FROM has barely changed since the Python 2.4 days, right through to Python 3.x. The same approach works everywhere, including PyPy, which uses the same format, which makes this more portable than the ast and compiler modules!

Coping with concurrency

Now a mechanism exists to enumerate dependencies, we need a mode of delivery. The approach used is simplistic, and (as seen later), will likely require future improvement.

On receiving a GET_MODULE message from a child, a parent (don’t forget, Mitogen operates recursively!) first tries to satisfy the request from its own cache, before forwarding it upwards towards the master. The master sends LOAD_MODULE messages for all dependencies known to be missing from the child before sending a final message containing the module that was actually requested. Since contexts always cache unsolicited LOAD_MODULE messages from upstream, by the time the message arrives for the requested module, many dependencies should be in RAM and no further network roundtrips requesting them are required.

Meanwhile for each stream connected to any parent, a set of module names ever delivered on that stream are recorded. Each parent is allowed to ignore any GET_MODULE for which a corresponding LOAD_MODULE has already been sent, preventing a race between in-flight requests causing the same module to ever be sent twice.

This places the onus on downstream contexts to ensure the single LOAD_MODULE message received for each distinct module always reaches every interested party. In short, GET_MODULE messages must be deduplicated and synchronized not only for any arriving from a context’s children, but also from its own threads.

Some further gory details are in the docs

Pretty pictures

And finally the result. For my test script, the total number of roundtrips dropped from 166 to 13, one of which is for the script itself, and 3 negative requests for extension modules that cannot be transferred. That leaves, bugs aside, 9 roundtrips to transfer the most obscene dependency I could think of.

One more look at the library’s network profile. Over the same connection as previously, the situation has improved immensely:

570 packets sending Django 7,231km over 3G in 16.47 seconds. Throughput averages 38.6KiB/sec and peaks at 2.7MiB/sec

Not only is performance up, but the number of frames transmitted has dropped by 42%. That’s a 42% fewer changes of connection hang due to crappy WiFi!

One final detail is visible: around the 10 second mark, a tall column of frames is sent with progressively increasing size, almost in the same instant. This is not some bug, it is Path MTU Discovery (PMTUD) in action. PMTUD is a mechanism by which IP subprotocols can learn the maximum frame size tolerated by the path between communicating peers, which in turn maximizes link efficiency by minimizing bandwidth wasted on headers. The size is ramped up until either loss occurs or an intermediary signals error via ICMP.

Just like the network path, PMTUD is dynamic and must restart on any signal indicating network conditions have changed. Comparing this graph with the previous, we see one final improvement as a result of providing the network layer enough data to do its job: PMTUD appears restart much less frequently, and the stream is pegged at the true path MTU for much longer.

Futures

Aside from simple fixes to reduce wasted roundtrips for extension modules that can’t be imported, and optional imports of top-level packages that don’t exist on the master, there are two major niggles remaining in how import works today.

The first is an irritating source of latency present in deep trees: currently it is impossible for intermediary nodes satisfying GET_MODULE requests for children to streamily send preloaded modules towards a child until the final LOAD_MODULE arrives at the intermediary for the module actually requested by the child. That means preloading is artificially serialized at each layer in the tree, when a better design would allow it to progress concurrent to the LOAD_MODULE messages still in-flight from the master.

This will present itself when doing multi-machine hops where links between the machines are slow or suffer high latency. It will also be important to fix before handling hundreds to thousands of children, such as should become practical once asynchronous connect() is implemented.

There are various approaches to tweaking the design so that concurrency is restored, but I would like to let the paint dry a little on the new implementation before destablizing it again.

The second major issue is almost certainly a bug waiting to be discovered, but I’m out of energy to attack it right now. It relates to complex situations where many children have different functions invoked in them, from a complex set of overlapping packages. In such cases, it is possible that a LOAD_MODULE for an unrelated GET_MODULE prematurely delivers the final module from another import, before it has had all requisite modules preloaded into the child.

To fix that, the library must ensure the tree of dependencies for all module requests are sent downstream depth-first, i.e. it is never possible for any module to appear in a LOAD_MODULE before all of its dependencies have first.

Finally there are latency sources buried elsewhere in the library, including at least 2 needless roundtrips during connection setup. Fighting latency is an endless war, but with module loading working efficiently, the most important battle is over.

Mitogen, an infrastructure code baseline that sucks less

dw — Fri, 15 Sep 2017 15:36:52 +0100

After many years of occasional commitment, I'm finally getting close to a solid implementation of a module I've been wishing existed for over a decade: given a remote machine and an SSH connection, just magically make Python code run on that machine, with no hacks involving error-prone shell snippets, temporary files, or hugely restrictive single use request-response shell pipelines, and suchlike.

I'm borrowing some biology terminology and calling it Mitogen, as that's pretty much what the library does. Apply some to your program, and it magically becomes able to recursively split into self-replicating parts, with bidirectional communication and message routing between all the pieces, without any external assistance beyond an SSH client and/or sudo installation.

Mitogen's goal is straightforward: make it childsplay to run Python code on remote machines, eventually regardless of connection method, without being forced to leave the rich and error-resistant joy that is a pure-Python environment. My target users would be applications like Ansible, Salt, Fabric and similar who (through no fault of their own) are universally forced to resort to obscene hacks in their implementations to affect a similar result. Mitogen may also be of interest to would-be authors of pure Python Internet worms, although support for autonomous child contexts is currently (and intentionally) absent.

Because I want this tool to be useful to infrastructure folk, Mitogen does not require free disk space on the remote machines, or even a writeable filesystem -- everything is done entirely in RAM, making it possible to run your infrastructure code against a damaged machine, for example to implement a repair process. Newly spawned Python interpreters have import hooks and logging handlers configured so that everything is fetched or forwarded over the network, and the only disk accesses necessary are those required to start a remote interpreter.

Recursion

Mitogen can be used recursively: newly started child contexts can in turn be used to run portions of itself to start children-of-children, with message routing between all contexts handled automatically. Recursion is used to allow first SSHing to a machine before sudoing to a new account, all with the user's Python code retaining full control of each new context, and executing code in them transparently, as easily as if no SSH or sudo connection were involved at all. The master context is able to control and manipulate children created in this way as easily as if they were directly connected, the API remains the same.

Currently there exists just two connection methods: ssh and sudo, with the sudo support able to cope with typing passwords interactively, and crap configurations that have requiretty enabled.

I am explicitly planning to support Windows, either via WMI, psexec, or Powershell Remoting. As for other more exotic connection methods, I might eventually implement bootstrap over an IPMI serial console connection if for nothing else then as a demonstrator of how far this approach can be taken, but the ability to use the same code to manage a machine with or without a functional networking configuration would be in itself a very powerful feature.

This looks a bit like X. Isn't this just X?

Mitogen is far from the first Python library to support remote bootstrapping, but it may be the first to specifically target infrastructure code, minimal networking footprint, read-only filesystems, stdio and logging redirection, cross-child communication, and recursive operation. Notable similar packages include Pyro and py.execnet.

This looks a bit like Fabric. Isn't this just Fabric?

Fabric's API feels kinda similar to what Mitogen offers, but it fundamentally operates in terms of chunks of shell snippets to implement all its functionality. You can't easily (at least, as far as I know) trick Fabric into running your Python code remotely, or for that matter recursively across subsequent sudo and SSH connections, and arrange for that code to communicate bidirectionally with code running in the local process and autonomously between any spawned children.

Mitogen internally reuses this support for bidirectional communication to implement some pretty exciting functionality:

SSH Client Emulation

So your program has an elaborate series of tunnels setup, and it's running code all over the place. You hit a problem, and suddenly feel the temptation to drop back to raw shell and SSH again: "I just need to sync some files!", you tell yourself, before loudly groaning on realizing the spaghetti of duplicated tunnel configurations that would be required to get rsync running the same way as your program. What's more, you realize that you can't even use rsync, because you're relying on Mitogen's ability to run code over sudo with requiretty enabled, and you can't even directly log into that target account.

Not a problem: Mitogen supports running local commands with a modified environment that causes their attempt to use SSH to run remote command lines to be redirected into Mitogen, and tunnelled over your program's existing tunnels. No duplicate configuration, no wasted SSH connections, no 3-way handshake latency.

The primary goal of the SSH emulator to simplify porting existing infrastructure scripts away from shell, including those already written in Python. As a first concrete target for Mitogen, I aim to retrofit it to Ansible as a connection plug-in, where this functionality becomes necessary to support e.g. Ansible's synchronize module.

Compared To Ansible

To understand the value of Mitogen, a short comparison against Ansible may be useful. I created an Ansible playbook talking to a VMWare Fusion Ubuntu machine, with SSH pipelining enabled (the current best performance mode in Ansible). The playbook simply executes /bin/true with become: true and discards the result 100 times.

I then created an equivalent script written against Mitogen, using its SSH and sudo functionality, and finally a trivial change to the Mitogen variant that executes the control loop on the target machine. In terms of architecture, the first Mitogen script is closer to a fair comparison to Ansible's control flow, but the latter is a good example of the kind of intelligence Mitogen enables that would be messy, if not close to impossible with Ansible's existing architecture.

[Side note: this is comparing performance characteristics only, in particular I am not advocating writing code against Mitogen directly! It's possible, but you get none of the ease of use that a tool like Ansible provides. On saying that, though, a Mitogen-enabled tool composed of tens of modules would have similar performance to the numbers below, just a slightly increased base cost due to initial module upload]

Method	Bytes A→B	Bytes B→A	Packets A→B	Packets B→A	Duration (ms)
Ansible default	5,001,352	486,500	8,864	4,460	55,065
Ansible pipelining	4,562,905	178,622	4,282	2,033	25,643
Mitogen local loop	45,847	17,982	247	135	1,245
Mitogen remote loop	22,511	5,766	51	39	784

The first and most obvious property of Ansible is that it uses a metric crap-ton of bandwidth, averaging 45kb of data for each run of /bin/true. In comparison, the raw command line "ssh host /bin/true" generates only 4.7kb and 311ms, including SSH connection setup and teardown.

Bandwidth aside, CPU alone cannot account for runtime duration, clearly significant roundtrips are involved, generating sufficient latency to become visible on an in-memory connection to a local VM. Why is that? Things are about to get real ugly, and I'm already starting to feel myself getting depressed. Remember those obscene hacks I mentioned earlier? Well, buckle your seatbelt Dorothy, because Kansas is going bye-bye..

The Ugly

[Side note: the name Ansible is borrowed from Ender's Game, where it refers to a faster-than-light communication technology. Giggles]

When you write some code in Ansible, like shell: /bin/true, you are telling Ansible (in most cases) that you want to execute a module named shell.py on the target machine, passing /bin/true as its argument.

So far, so logical. But how is Ansible actually running shell.py? "Simple", by default (no pipelining) it looks like this:

First it scans shell.py for every module dependency,
then it adds the module and all dependents into an in-memory ZIP file, alongside a file containing the module's serialized arguments,
then it base64-encodes this ZIP file and mixes it into a templatized self-extracting Python script (module_common.py),
then it writes the templatized script to the local filesystem, where it can be accessed by sftp,
then it uploads the script to the target machine:
1. first it runs a fairly simple bash snippet over SSH to find the user's home directory,
2. then it runs a bigger bash snippet to create a temporary directory in the user's home directory in which to write the templatized script,
3. then it starts an sftp session and uses it to write the templatized script to the new temporary directory,
then it runs another snippet over SSH to mark the script executable,
then it wraps a snippet to execute the templatized script using an obscene layer of quoting (16 quotes!!!) and passes it to sudo,
finally the templatized script runs:
1. first it creates yet another temporary directory on the target machine, this time using the tempfile module,
2. then it writes a base64-decoded copy of the embedded ZIP file as ansible_modlib.zip into that directory,
3. then it opens the newly written ZIP file using the zipfile module and extracts the module to be executed into the same temporary directory, named like ansible_mod_<modname>.py,
4. then it opens the newly written ZIP file in append mode and writes a custom sitecustomize.py module into it, causing the ZIP file to be written to disk for a second time on this machine, and a third time in total,
5. then it uses the subprocess module to execute the extracted script, with PYTHONPATH set to cause Python's ZIP importer to search for additional dependent modules inside the extracted-and-modified ZIP file,
6. then it uses the shutil module to delete the second temporary directory,
then the shell snippet that executed the templatized script is used to run rm -rf over the first temporary directory.

When pipelining is disabled, which is the default, and required for cases where sudo has requiretty enabled, these steps (and their associated network roundtrips) recur for every single playbook step. And now you know why Ansible makes execution over a local 1Gbit LAN feel like it's communicating with a host on Mars.

Need a breath? Don't worry, things are about to get better. Here are some pretty graphs to look at while you're recovering..

The Ugly (from your network's perspective)

This shows Ansible's pipelining mode, constantly reuploading the same huge data part and awaiting a response for each run. Be sure to note the sequence numbers (transmit byte count) and the scale of the time axis:

Now for Mitogen, demonstrating vastly more conservative use of the network:

The SSH connection setup is clearly visible in this graph, accounting for about the first 300ms on the time axis. Additional excessive roundtrips are visible as Mitogen waits for its command-line to signal successful first stage bootstrap before uploading the main implementation, and 2 subsequent roundtrips first to fetch mitogen.sudo module followed by the mitogen.master module. Eliminating module import roundtrips like these will probably be an ongoing battle, but there is a clean 80% solution that would apply in this specific case I just haven't gotten around to implementing yet.

The fine curve representing repeated executions of /bin/true is also visible: each bump in the curve is equivalent to Ansible's huge data uploads from earlier, but since Mitogen caches code in RAM remotely, unlike Ansible it doesn't need to reupload everything for each call, or start a new Python process, or rewrite a ZIP file on disk, or .. etc.

Finally one last graph, showing Mitogen with the execution loop moved to the remote machine. All the latency induced by repeatedly invoking /bin/true from the local machine has disappeared.

The Less Ugly

Ansible's pipelining mode is much better, and somewhat resembles Mitogen's own bootstrap process. Here the templatized initial script is fed directly into the target Python interpreter, however they immediately deviate since Ansible starts by extracting the embedded ZIP file per step 8 above, and discarding all the code it uploaded once the playbook step completes, with no effort made to preserve either the Python processes spawned, or the significant amount of uploaded module code for each step.

Pipelining mode is a huge improvement, however it still suffers from making use of the SSH stdio pipeline only once (which was expensive to setup, even with multiplexing enabled), the destination Python interpreter only once (usually ~100ms+ per invocation), and as mentioned repeatedly, no caching of code in the target, not even on disk.

When Mitogen is executing your Python function:

it executes SSH with a single Python command-line,
then it waits for that command-line to report "EC0" on stdout,
then it writes a copy of itself over the SSH pipe,
1. meanwhile the remote Python interpreter forks into two processes,
2. the first re-execs itself to clear the huge Python command-line passed over SSH, and resets argv[0] to something descriptive,
3. the second signals "EC0" and waits for the parent context to send 7KiB worth of Mitogen source, which it decompresses and feeds to the first before exitting,
4. the Mitogen source reconfigures the Python module importer, stdio, and logging framework to point back into itself, then starts a private multiplexer thread,
5. the main thread writes "EC1" then sleeps waiting for CALL_FUNCTION messages,
6. meanwhile the multiplexer routes messages between this context's main thread, the parent, and any child contexts, and waits for something to trigger shutdown.
then it waits for the remote process to report "EC1",
then it writes a CALL_FUNCTION message which includes the target module, class, and function name and parameters,
1. the slave receives the CALL_FUNCTION message and begins execution, satisfying in-RAM module imports using the connection to the parent context as necessary.

On subsequent invocations of your Python function, or other functions from the same module, only steps 3.6, 5, and 5.1 are necessary.

This all sounds fine and dandy, but how can I use it?

I'm working on it! For now my goal is to implement enough functionality so that Mitogen can be made to work with Ansible's process model. The first problem is that Ansible runs playbooks using multiple local processes, and has no subprocess<->host affinity, so it is not immediately possible to cache Mitogen's state for a host. I have a solid plan for solving that, but it's not yet implemented.

There are a huge variety of things I haven't started yet, but will eventually be needed for more complex setups:

Getting Started Documentation: it's missing.
Asynchronous connect(): so large numbers of contexts can be spawned in reasonable time. For, say, 3 tiers targeting a 1,500 node network connecting in 30 seconds or so: a per-rack tier connecting to 38-42 end nodes, a per-quadrant tier connecting to 10 or so racks, a single box in the datacentre tier for access to a management LAN, reducing latency and caching uploaded modules within a datacenter's network, and the top-level tier which is the master program itself.
Better Bootstrap, Module Caching And Prefetching: currently Mitogen is wasting network roundtrips in various places. This makes me lose sleep.
General Robustness: no doubt with real-world use, many edge cases, crashes, hangs, races and suchlike will be be discovered. Of those, I'm most concerned with ensuring the master process never hangs with CTRL+C or SIGTERM, and in the case of master disconnect, orphaned contexts completely shut down 100% of the time, even if their main thread has hung.
Better Connection Types: it should at least support SSH connection setup over a transparently forwarded TCP connection (e.g. via a bastion host), so that key material never leaves the master machine. Additionally I haven't even started on Windows support yet.
Security Audit: currently the package is using cPickle with a highly restrictive class whitelist. I still think it should be possible to use this safely, but I'm not yet satisfied this is true. I'd also like it to optionally use JSON if the target Python version is modern enough. Additionally some design tweaks are needed to ensure a compromised slave cannot use Mitogen to cross-infect neighbouring nodes.
Richer Primitives: I've spent so much effort keeping the core of Mitogen compact that overall design has suffered, and while almost anything is possible using the base code, often it involves scrobbling around in the internal plumbing to get things working. Specifically I'd like to make it possible to pass Context handles as RPC parameters, and generalise the fakessh code so that it can handle other kinds of forwarding (e.g. TCP connections, additional UNIX pipe scenarios).
Tests. The big one: I've only started to think about tests recently as the design has settled, but so much system-level trickery is employed, always spread out across at least 2 processes, that an effective test strategy is so far elusive. Logical tests don't capture any of the complex OS/IO ordering behaviour, and while typical integration tests would capture that, they are too coarse to rely on for catching new bugs quickly and with strong specificity.

Why are you writing about this now?

If you read this far, there's a good chance you either work in infrastructure tooling, or were so badly burned by your experience there that you moved into management. Either way, you might be the person who could help me spend more time on this project. Perhaps you are on a 10-person team with a budget, where 30% of the man-hours are being wasted on Ansible's connection latency? If so, you should definitely drop me an e-mail.

The problem with projects like this is that it is almost impossible to justify commercially, it is much closer to research than product, and nobody ever wants to pay for that. However, that phase is over, the base implementation looks clean and feels increasingly solid, my development tasks are becoming increasingly target-driven, and I'd love the privilege to polish up what I have, to make contemporary devops tooling a significantly less depressing experience for everyone involved.

If you merely made it to the bottom of the article because you're interested or have related ideas, please drop me an e-mail. It's not quite ready for the prime time, but things work more than sufficiently that early experiementation is probably welcome at this point.

Meanwhile I will continue aiming to make it suitable for use with Ansible, or perhaps a gentle fork of Ansible, since its internal layering isn't the greatest.