Data Validation

In this demo, we showcase some data validation tools. We’ll start by loading the MTC mini data for testing.

import jax.numpy as jnp
import pytest

import mtc
import traveler as tv

store = mtc.mini()
store

<Store with keys households, persons, skims, land_use, time_periods, tours>

store.table_types()

{'households': mtc.tables.households.Households,
 'tours': mtc.tables.tours.Tours,
 'persons': mtc.tables.persons.Persons,
 'land_use': mtc.tables.landuse.LandUse}

lu = store.land_use

lu.RESACRE

Array([ 1.     ,  1.     ,  1.     ,  1.     ,  1.     ,  7.     ,
       13.     ,  8.33042,  9.79332, 10.37666, 12.58862,  1.     ,
        1.     ,  1.     ,  3.     , 13.73247, 23.84518,  2.09941,
       14.19946, 15.35359, 15.18943,  9.     ,  4.413  ,  2.     ,
        4.     ], dtype=float32)

lu.info()

<JaxTable id_col=TAZ>
- DISTRICT             (25,) int8
- SD                   (25,) int8
- TOTHH                (25,) int32
- TOTPOP               (25,) int32
- TOTACRE              (25,) float32
- RESACRE              (25,) float32
- CIACRE               (25,) float32
- TOTEMP               (25,) int32
- AGE0519              (25,) int32
- RETEMPN              (25,) int32
- FPSEMPN              (25,) int32
- HEREMPN              (25,) int32
- OTHEMPN              (25,) int32
- AGREMPN              (25,) int32
- MWTEMPN              (25,) int32
- PRKCST               (25,) float32
- OPRKCST              (25,) float32
- area_type            (25,) int32
- HSENROLL             (25,) float32
- COLLFTE              (25,) float32
- COLLPTE              (25,) float32
- TOPOLOGY             (25,) int32
- TERMINAL             (25,) float32
- TAZ                  (25,) int32
- county_id            (25,) int8

Let’s validate the contents of the “tours” table. Before we validate this data, we can see what’s in the table using the info method.

store.tours.info()

<JaxTable id_col=tour_id>
- person_id            (20000,) int32
- tour_id              (20000,) int32
- otaz                 (20000,) categorical: (25 categories)
- dtaz                 (20000,) categorical: (25 categories)
- out_period           (20000,) categorical: EA, AM, MD, ... (5 categories)
- in_period            (20000,) categorical: EA, AM, MD, ... (5 categories)
- _prng_key_           (20000,) key<fry>
- _person_idx_         (20000,) int32
- _household_idx_      (20000,) int32
- household_id         (20000,) int32

We can validate the contents of this table using the “validate” method.

store.tours.validate()

This method has no return value, but it will raise a ValidationError if there is a problem.
If we want to see the details of the validation process, we can turn on verbose output.

store.tours.validate(verbose=2)

Field 'tour_id' validated as <class 'numpy.int32'>.
Field 'person_id' validated as <class 'numpy.int32'>.
Field 'household_id' validated as <class 'numpy.int32'>.
Field '_household_idx_' validated as <class 'numpy.int32'>.
Field '_person_idx_' validated as <class 'numpy.int32'>.
Skipping validation of field 'is_joint' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'is_atwork_subtour' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'hov2_available' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'work_tour_is_SOV' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'terminal_time' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'number_of_participants' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.
Skipping validation of field 'daily_parking_cost' in table 'tours' because it derives from 'tour_preprocessor' which is not yet completed.

You’ll notice that some of the fields in this table were skipped. That’s because they are not “raw” data, but they are expected to come from some model component that has not yet been run. We can tell the validation tool which steps have been run by passing it a sources argument, and then the validation of fields created by those steps will not be skipped.

with pytest.raises(tv.errors.ValidationError) as e:
    store.tours.validate(sources=["tour_preprocessor"])
print(e.value)

Table validation failed:
Field 'is_joint' is missing in table 'tours'.
Field 'is_atwork_subtour' is missing in table 'tours'.
Field 'hov2_available' is missing in table 'tours'.
Field 'work_tour_is_SOV' is missing in table 'tours'.
Field 'terminal_time' is missing in table 'tours'.
Field 'number_of_participants' is missing in table 'tours'.
Field 'daily_parking_cost' is missing in table 'tours'.

If we manually add the required fields to the table, we can emulate what the step would do, and pass validation, but only if they are the correct dtypes.

store = store.assign_on(
    "tours",
    is_joint=jnp.asarray(0.0),
    is_atwork_subtour=jnp.asarray(False),
    daily_parking_cost=jnp.asarray(15.0),
    hov2_available=True,
    work_tour_is_SOV=False,
    terminal_time=jnp.float32(2.0),
    number_of_participants=jnp.int8(1),
)

with pytest.raises(tv.errors.ValidationError) as e:
    store.tours.validate(sources=["tour_preprocessor"])
print(e.value)

Table validation failed:
Field 'is_joint' in table 'tours' has type float32, expected <class 'bool'>.

Let’s try again, using the correct dtypes.

store = store.assign_on(
    "tours",
    is_joint=jnp.asarray(False),
)

store.tours.validate(sources=["tour_preprocessor"])