Benchmarking#

Traveler includes a repeatable benchmark based on the full MTC example model. It measures two Traveler implementations and two ActivitySim implementations of the same modeling sequence:

Load and annotate households, persons, land use, and selected OMX skims.
Run a five-alternative auto-ownership model.
Run work-location choice with tour-mode logsums from the same 21-mode utility specification.

The benchmark is useful for evaluating runtime and memory, but it is not a claim that Traveler already replaces the complete ActivitySim platform. The model sequence here is intentionally much narrower than a production activity-based model.

Set up the benchmark#

Traveler and ActivitySim use separate environments. ActivitySim 1.5.1 requires an older Python and NumPy stack, so the benchmark creates a locked Python 3.11 environment containing the released ActivitySim 1.5.1 and sharrow 2.15.0 packages:

uv run python benchmarking/mtc_full.py setup-activitysim

Download and conversion time is deliberately excluded. A normal benchmark run first prepares the MTC inputs, warms the JAX and sharrow compilation caches, and only then launches each measured implementation in a fresh process:

uv run python benchmarking/mtc_full.py run --engine all --repeat 3

--engine all runs these four implementations:

traveler evaluates work-location logsums for all 1,454 destination zones. This not how ActivitySim is normally configured, and doing so is generally considered to be computationally infeasible in ActivitySim. It is included here as proof that this is now possible in Traveler.
traveler-sampled uses a 30-zone sample-logsum-simulate workflow.
activitysim uses the same 30-zone workflow in one process, with sharrow required for expression evaluation.
activitysim-multiprocess uses ActivitySim’s native pipeline multiprocessing, shared skims, and the same model specifications. The tuned default is ten single-threaded worker processes on an 18-core Apple Silicon workstation.

For a quick functional check, reduce the population and chunk sizes. The number of ActivitySim processes cannot exceed the number of households:

uv run python benchmarking/mtc_full.py run \
  --engine all --households 20 \
  --traveler-chunk-size 20 \
  --traveler-sampled-chunk-size 20 \
  --activitysim-chunk-size 20 \
  --activitysim-processes 10 \
  --repeat 1

Preparation can also be performed separately. This is useful when several measured runs will use the same population and chunk shapes:

uv run python benchmarking/mtc_full.py prepare \
  --engine all --households 500000 \
  --traveler-chunk-size 500 \
  --traveler-sampled-chunk-size 2500 \
  --activitysim-chunk-size 10000000 \
  --activitysim-processes 10

uv run python benchmarking/mtc_full.py run \
  --engine all --households 500000 \
  --traveler-chunk-size 500 \
  --traveler-sampled-chunk-size 2500 \
  --activitysim-chunk-size 10000000 \
  --activitysim-processes 10 \
  --repeat 3 --skip-prepare

Do not use --skip-prepare after changing a population or chunk shape unless you intend to include compilation in the measurement. Timestamped JSON reports are written under benchmarking/results/. They contain individual phase timings, whole-process wall time, configuration, record counts, output checks, and sampled peak memory for the worker process and its descendants.

Full-scale results#

The following results were collected using uv run poe benchmark on an 18-core Apple Silicon workstation with 128 GiB of RAM. Each implementation selected 500,000 households from the full MTC inputs. The runner completed an unmeasured preparation pass and then ran each engine three times in fresh processes. Compilation caches were warm and data download was not measured.

Wall times are medians, with the observed three-run range in parentheses. Peak process-tree RSS is also the median of the three sampled peaks.

Implementation	Work-location method	Processes	Median wall time (range)	Median peak process-tree RSS
Traveler	All 1,454 zones	1	35.84 s (34.66–35.85)	5.16 GiB
Traveler sampled	30-zone sample	1	7.11 s (6.99–7.32)	3.06 GiB
ActivitySim + sharrow	30-zone sample	1	44.40 s (42.94–44.43)	19.94 GiB
ActivitySim + sharrow, multiprocess	30-zone sample	10	34.51 s (33.38–34.55)	34.79 GiB

The ranges show modest run-to-run variation, but three repetitions are still a small sample. Repeated, interleaved runs on an otherwise idle machine remain important when drawing precise performance conclusions.

Using the medians, native multiprocessing reduced ActivitySim wall time by about 22 percent. Sampled Traveler was about 6.2 times as fast as single-process ActivitySim and 4.9 times as fast as multiprocess ActivitySim. All-zone Traveler was about 4 percent slower than multiprocess ActivitySim and used 19 percent less wall time than single-process ActivitySim, even though it calculated logsums for every destination instead of only 30 sampled zones.

The RSS figures need special care. They are the sum of resident memory reported for the root process and all descendants. Shared pages can therefore be counted once for each ActivitySim worker. Unique set size was unavailable on the test machine, so 34.79 GiB should not be read as 34.79 GiB of distinct physical memory. The single-process figures are easier to compare: sampled Traveler’s 3.06 GiB median RSS was about 85 percent below ActivitySim’s 19.94 GiB.

In what ways is this apples-to-apples?#

The benchmark controls several important sources of variation:

All implementations use the same released MTC household, person, land-use, and OMX inputs, and load the same subset of skim matrices.
The auto-ownership, work-location, and tour-mode utility equations were translated directly between the Traveler and ActivitySim representations.
The three sampled implementations use the same sample size of 30 zones and the standard sample-of-alternatives correction.
ActivitySim is configured to require sharrow rather than falling back to ordinary expression evaluation.
Download, input conversion, and one-time compilation are outside the measured region. Each recorded repetition otherwise starts in a fresh process, and process startup and input loading are included.
Single- and multiprocess ActivitySim produced identical household and person counts, auto-ownership counts, and aggregate work-location output checks in the full-scale run.

For the core sample-logsum-simulate sequence, traveler-sampled versus either ActivitySim implementation is therefore a reasonable engineering comparison. It measures the cost of expressing and executing nearly the same numerical work through the two platforms.

In what ways is it not apples-to-apples?#

The results should not be generalized without these qualifications:

All-zone Traveler does much more logsum work than the other three implementations. This is an intentional handicap, not an equal-algorithm comparison. Sampled Traveler is the appropriate Traveler reference for the ActivitySim tests.
The platforms select 500,000 households using their own deterministic sampling machinery. Traveler loaded 1,315,472 persons and ActivitySim loaded 1,315,695, a difference of 223 people (about 0.017 percent). Single- and multiprocess ActivitySim use exactly the same sampled population.
ActivitySim brings mature, general-purpose workflow infrastructure to this small sequence, including table management, random-number channels, pipeline apportioning, checkpoint storage, and coalescing. Traveler’s benchmark path is narrower and does not yet provide all equivalent production services.
Conversely, the ActivitySim multiprocess path was optimized for this model: skims are shared, numerical libraries are limited to one thread per worker, unused shadow-pricing buffers and tables are omitted, and redundant intermediate checkpoints are disabled. Ten workers was selected after full-scale tests on this machine; a different computer may have a different optimum.
Warm-cache measurement reflects repeated production runs more than a first run on a new model specification. JAX and sharrow have different compilation behavior, so cold-start timing is a separate question.
Only one model sequence, dataset, operating system, and hardware platform is represented. The reported pass used three repetitions and fixed, implementation-specific chunk sizes. It reduces sensitivity to one unusually fast or slow run, but remains evidence rather than a universal ranking.
The benchmark does not include most of a production ActivitySim model: accessibility, school location, tour generation and scheduling, stop and trip models, shadow pricing, summaries, tracing, estimation, restartability, and many validation and reporting tasks are outside its scope.

Outlook#

The results support a conservative conclusion: Traveler has a credible path to materially faster and lower-memory execution for the dense numerical core of an activity-based model. They do not establish that a future full Traveler platform will preserve the roughly five-to-six-fold runtime advantage seen in this small sampled workflow.

Several architectural features make continued gains plausible:

Compiled and fused array programs. JAX/XLA can compile a complete utility, probability, logsum, and choice kernel and fuse adjacent array operations. This reduces Python dispatch and avoids repeatedly materializing temporary pandas objects.
Regular, compact data. JaxTable, JaxDataset, encoded categoricals, and explicitly selected dtypes can keep frequently used model data in compact arrays. Polars can handle columnar preparation without forcing the numerical core through object-heavy tables.
Fewer representation changes. Keeping model state in JAX-compatible arrays across consecutive steps can avoid dataframe joins, index rebuilding, and conversions at every model boundary. Zero-copy interchange should be used where a boundary is unavoidable.
Native accelerator and parallel execution. The same numerical kernels can target multicore CPUs and, where model size and transfer costs justify it, GPUs or other JAX accelerators. Longer term, independent chunks can be sharded without duplicating the entire model state in conventional worker processes.
Purpose-built persistence. Checkpoints can store only changed arrays and metadata at deliberate boundaries instead of serializing a general table pipeline after many individual components. This can reduce both I/O and peak memory while retaining restartability.

Building the missing platform features will consume some of today’s advantage. Production-quality reproducibility, validation, checkpointing, tracing, estimation, scheduling constraints, shadow pricing, diagnostics, and clear error handling all have real costs. Some models are irregular or data-dependent and will not map neatly to large static JAX kernels. JIT compilation, shape specialization, device transfers, and accelerator memory also require careful management.

ActivitySim and sharrow will continue to improve as well. They already provide a broad, proven modeling system, and newer data backends, reduced pipeline I/O, and better parallel scheduling can narrow the gap. The relevant future test is therefore not today’s small Traveler prototype against a static ActivitySim release, but two complete platforms running the same production model with the same correctness, reproducibility, and operational requirements.

Even after allowing for those costs, it is reasonable to expect Traveler to retain a meaningful advantage in computation-heavy choice and logsum models, and likely a substantial memory advantage when a single compact array state replaces replicated dataframe-oriented worker state. End-to-end gains for a complete model will probably be smaller than the kernel-level gains and will need to be demonstrated component by component as Traveler’s feature coverage grows.