Appendix C-2
U.S. Data Quality Control Tables

Sample Definitions

In order to assess quality of the data arising from the U.S. component of this Study Phase, the same three hierarchical sample definitions used in the Canadian component were used to construct summary tables. These were required to manage the analysis of the very large volume of data recorded by the AP+ system.

No records excluded sample (2,265,248 total records among 12 drivers)

At least 30 mph sample (2,013,942 total records among 12 drivers)

Cleaned analysis sample (1,935,577 total records among 9 drivers)

No records excluded

The first sample included all raw data and was constructed for comparison purposes only. A limited number of appendix C-2 tables are based on the all records sample.

All records recorded at speeds of at least 30 mph

The second sample definition eliminates records in which speed was recorded at less than 30 mph. There were two reasons for this exclusion. The primary reason was that the study was designed to examine the effects of the fatigue management intervention in highway driving and 30 mph was the definition adopted by the study team. The second reason was that experience from the Canadian Study Phase indicated that most records with artifact data were eliminated by restricting attention to records recorded when vehicle speed was at least 30 mph. Data Quality Table 1 provides the numbers (%) of records recorded with speeds of at least 30 mph, less than 30 mph but greater than 1 mph, and equal to 1 mph. The value of 1 mph is velocity record by the AP+ system when the truck was standing still.

Clean analysis sample

The "cleaned analysis" sample is constructed by:

Excluding records with mph<30

See explanation above for rationale.

Excluding records with durations >30 sec

In the Canadian Study Phase it was observed that almost all problematic records remaining after excluding records with speeds less than 30 miles were associated with very long record duration. A careful evaluation indicated that by excluding all records with durations greater than 30 seconds, nearly no valid records would be excluded while filtering out long duration records indicative of problematic data. This same filter was applied to the current U.S. dataset.

Hard code variable value deletions (within records)

Careful examination of driver specific distributions of outcome variables recorded by the AP+ system was used to identify additional artifacts and problematic data among the remaining records with speeds at least 30 mph and with record durations no greater than 30 seconds. Hard code variable value deletions within records were made on a case by case basis after careful evaluation. Reasons for exclusions of these records were documented. The following hard code variable value deletions were made.

X and Y acceleration values were set to missing for the first 4 drivers as specified by Pierre Pommarel of AP+ (see discussion below).

X and Y acceleration for driver 42 during the FEEDBACK condition were all equal to 1.27. Therefore, X and Y acceleration values were set to missing under both the NO FEEDBACK and FEEDBACK conditions for this driver in the AP+ cleaned analysis dataset.

The lane tracking variable was set to missing if lane tracking coincidence was less than 50% (see discussion below - this was the rule used in FMT Canada).

S teering w heel and f ront w heel m ovements measurements were hard coded to missing for some drivers: s teering w heel m ovement values were set to missing for drivers 36 and 42. Front w heel m ovement values were set to missing under the FEEDBACK condition for the FEEDBACK legs of drivers 35 and 42. Then, for symmetry, the front wheel movement values under the NO FEEDBACK condition for these drivers were also set to missing.

Hard code driver deletions

In addition to hard code variable value deletions, all data from specific drivers were excluded from the cleaned analysis sample. This occurred when AP+ recorded data was only available under one of the two conditions or because there was insufficient data under one of the two conditions to permit meaningful comparisons. Thus, the cleaned analysis sample was defined on the basis of the subset of drivers with sufficient data under both conditions (FEEDBACK and NO FEEDBACK), restricting attention to records recorded at speeds of at least 30 mph, with record durations no more than 30 seconds and after excluding additional data found to be invalid following careful examination of driver specific distributions.

On this basis drivers 33, 34, and 39 were excluded from the cleaned analysis sample for AP+ outcomes. Additional detail regarding reasons for excluding these drivers is provided in Section 2.2 below. Also, as discussed below, drivers 35 and 36 were considered candidates for deletion. After data cleaning, driver 35 only had 8.4 hours under the FEEDBACK condition while driver 36 had only 8.6 hours. Nonetheless, a decision was made to retain these drivers since there was at least 8 hours of recording and light of the small sample size.

Details of Quality Assessment Analysis

More record durations ≥ 3 seconds (Data Quality Tables 2 and 3)

Before excluding any records, we previously found in the Canada Study Phase that the average percentages of records with >3 second durations were 6.2% and 10.5% in the NO FEEDBACK and FEEDBACK conditions, respectively (Data Quality Table 2). In the cleaned analysis sample (Data Quality Table 3) these average percentages became 7.3% and 7.6%, respectively.

In contrast, with no records excluded, in the U.S. Study Phase, the average percentages of records with >3 second durations were 13.9% and 15.3%, respectively, in the NO FEEDBACK and FEEDBACK conditions (Data Quality Table 2). In the cleaned analysis sample (Data Quality Table 3) these average percentages became 16.5% and 18.0%, respectively.

There were no apparent reasons to explain why the numbers of longer duration records increased by a factor of 2.5 in the U.S. Study Phase compared to the Canadian Study Phase. Long duration records are due to time intervals when there are no changes in any recorded parameters. An increase in the percentages of longer duration records could occur if fewer parameters are being recorded or if sensors and less sensitive to change or for some other unknown reason. No additional action was taken on the basis of this observation.

Sum of Record Durations (Data Quality Table 4 and summary table below )

Data Quality Table 4 excludes records with velocities less than 30 mph. In the Canadian Study Phase, restricting attention to records with durations of at least equal to 30 mph resulted in the exclusion of almost all long duration records. However, this is not the case for the U.S. Study Phase. The following comments were received from Pierre Pommarel of AP+ on Thursday 6/12/2003 2:19 PM.

Here are some notes concerning the data from the first two weeks of the experiment (no feedback):

* For ALL drivers, ignore the X and Y accelerometer values

* driver 031 (truck # 432-1312) and driver 032 (truck #432-1200) seems to have coherent data

* driver 033 (truck # 432-1263) and driver 034 (truck #432-1261) have an incomplete set of data, coming from the fact that both AP+ units were in "frozen" state. I think that driver 033 got data that can be processed. However, driver 034 has almost no data.

We did find extremely long duration records for driver 33 at speeds >30 mph (see Appendix 2) that artificially inflated the follow-up duration reported for drivers in Data Quality Table 4. The following table also excludes records with durations ≥30 seconds in addition to excluding records with < 30 mph. After this exclusion, Pierre's observation is confirmed and we see that there is very little data for driver 33.

The following table can be compared to Pierre's overall summary of ATA-FMT data collected distributed by way of e-mail on Wed 7/16/2003 1:07 PM. Selected sections are paraphrased below.

(The following grids contain text which is paraphrased from Pierre Pommarel's (of AP+) e-mail of Wednesday, July 16, 2003, 1:07 PM.)

Drivers 31 to 34: NO FEEDBACK condition

Drivers 31 to 34: With feedback

Before Hermitage intervention (First week)

After Hermitage intervention (Second week)

Drivers 35 to 38: NO FEEDBACK condition

Drivers 35 to 38: With FEEDBACK condition

Based on the sum of record durations for records when speed was at least 30 miles per hour and the record duration was no greater than 30 seconds, the following summary conclusions were made about each driver:

• Driver 31 had more than 80 hours under both NO FEEDBACK and FEEDBACK conditions, confirming Pierre's report that this driver has a complete set of data. There was sufficient data for inclusion of driver 31 into the analysis final cleaned analysis dataset.
• Driver 32 had approximately 80 NO FEEDBACK hours but only approximately 40 FEEDBACK hours, consistent with the incorrect memo card insertion during the second week. There was sufficient data for inclusion of driver 32 into the analysis final cleaned analysis dataset.
• Driver 33 had very few hours during both conditions. Pierre indicated that there was complete data collected during the second week of FEEDBACK but we were unable to confirm this since total follow-up duration at mph ³ 30 was estimated to be only 5.3 hours. Driver 33 did not appear to have sufficient cumulative record durations for inclusion into the final cleaned analysis dataset. Therefore, data from driver 33 was excluded.
• Driver 34 had no NO FEEDBACK hours. There are 22.9 FEEDBACK hours. Driver 34 did not have sufficient record durations under both experimental conditions for inclusion into the final cleaned analysis dataset.
• Driver 35 had more than 80 NO FEEDBACK hours. However, this driver had 8.4 FEEDBACK hours after excluding records with speeds <30 mph or record durations ³3 seconds. This finding is consistent with Pierre's report indicating only 2 days of recorded data due to memo card insertion error. Although there was only 8.4 hours of valid recorded data under the FEEDBACK condition, driver 35 was not excluded from analyses because doing so would have reduced the sample size from 9 to 8.
• Driver 36 was found to have a relatively small amount of valid follow-up data under both conditions (19.1 NO FEEDBACK hours and 8.6 FEEDBACK hours) consistent with Pierre's report that the memo card was not inserted. However, it is unclear how there can be any data at all if the memo card was not inserted? Nonetheless, since there was at least 8 hours of recorded data under both conditions, this driver was retained in the analysis sample.
• Driver 37 had approximately 40 hours under both conditions. Therefore, we were unable to confirm that Pierre report that this driver had a complete set of data under both conditions. However, there did appear to be sufficient data for inclusion of driver 37 into the analysis final cleaned analysis dataset.
• Driver 38 had approximately 80 hours under both NO FEEDBACK and FEEDBACK conditions, confirming Pierre's report that this driver has a complete set of data. There was sufficient data for inclusion of driver 38 into the final analysis dataset.
• Driver 39 was found to have very few hours during either condition, with only four hours under the NO FEEDBACK condition and less than one hour under the FEEDBACK condition. Driver 39 did not have sufficient record duration for inclusion into the final cleaned analysis dataset.
• Driver 40 had approximately 29 and 75 hours under NO FEEDBACK and FEEDBACK conditions respectively. There was sufficient data for inclusion of driver 40 into the final cleaned analysis dataset.
• Driver 41 had approximately 45 and 51 hours under NO FEEDBACK and FEEDBACK conditions respectively. There was sufficient data for inclusion of driver 41 into the final cleaned analysis dataset.
• Driver 42 had approximately 69 and 26 hours under NO FEEDBACK and FEEDBACK conditions respectively. There was sufficient data for inclusion of driver 42 into the final cleaned analysis dataset.

Filtering of zero duration and zero velocity records

During quality assessment procedures, we observed that the total numbers of records prior to any exclusions differed between that indicated in Data Quality Table 1 and Data Quality Table 2. The difference is that Data Quality Table 1 includes records with 0 second durations. Although the title of Table 2 indicates that there were no exclusions, in fact, 0 second duration records were excluded from the outset in analyses dealing with record durations as was done in the Canadian Study Phase. There were 22,155 0 second duration records (2,265,248 minU.S. 22,155 = 2,243,093). Upon closer inspection, we find that there are actually 2,243,095 records counted in Data Quality Table 2. The two record discrepancy relates to two records with mph=0 (driver 36 and 41, each in the NO FEEDBACK condition). These records were also excluded since the speed standing still is recorded at 1 mph. Thus, 0 mph records were defined as invalid and excluded during an initial data cleaning filter and so these two records do not appear in Data Quality Table 1

The following exhibit provides the frequency distributions of record durations without excluding 0 second records at the outset.

Poor lane tracking confidence (Data Quality Tables 9 and 10)

As with Canada Study Phase, if lane tracking confidence <50%, lane tracking offset (and, consequently, lateral distance as well) was set to missing (Data Quality Tables 11 - 14). For consistency with rules agreed upon for the Canadian Study Phase, the SafeTRAC Driver Alertness Summery values were not set to missing (Data Quality Tables 15 - 18). Since this outcome measure is computed using a proprietary scoring algorithm it was not clear whether poor lane tracking confidence (i.e., <50%) implied that the SafeTRAC Driver Alertness Summery should also set to missing.

Inconsistencies in steering wheel movements metric (Data Quality Tables 18-20)

In the Canada Study Phase, average mean and median steering wheel values during both the NO FEEDBACK and FEEDBACK conditions were around 11. In contrast, for the current U.S. Study Phase, these values are both around 54. Clearly there was either a change in metric or some other problem resulting in qualitatively different distributions for steering wheel movements between the two studies. As a consequence, steering wheel movement data are not poolable between studies. There is no apparent reason for this difference in distributions.

In addition, inspection of the steering wheel data for drivers 36 and 42 indicated that these distributions were qualitatively different compared to steering wheel data from the other drivers. Driver 36 had mean and median values on the order of more than 180 under both conditions. Driver 42 had a mean value of less than six and a median of zero in the NO FEEDBACK condition, and mean and median values of zero in the FEEDBACK condition. Therefore, steering wheel movements were hard coded to missing for drivers 36 and 42. Driver 39 also had distributions of steering wheel movements similar to driver 42, but driver 42 was already excluded from all analyses due to insufficient total follow-up. The following special 'AD' versions of Data Quality Tables 19 and 20 are presented without excluding any drivers. These tables provide the distributions of steering wheel movements for all drivers based on the cleaned analysis sample and document the reason why steering wheel movement data were excluded for drivers 36 and 39, even though these drivers are included in the analysis dataset.

Poor lane tracking confidence (Data Quality Tables 9 and 10)

As with Canada Study Phase, if lane tracking confidence <50%, lane tracking offset (and, consequently, lateral distance as well) was set to missing (Data Quality Tables 11 - 14). For consistency with rules agreed upon for the Canadian Study Phase, the SafeTRAC Driver Alertness Summery values were not set to missing (Data Quality Tables 15 - 18). Since this outcome measure is computed using a proprietary scoring algorithm it was not clear whether poor lane tracking confidence (i.e., <50%) implied that the SafeTRAC Driver Alertness Summery should also set to missing.

Inconsistencies in steering wheel movements metric (Data Quality Tables 18-20)

In the Canada Study Phase, average mean and median steering wheel values during both the NO FEEDBACK and FEEDBACK conditions were around 11. In contrast, for the current U.S. Study Phase, these values are both around 54. Clearly there was either a change in metric or some other problem resulting in qualitatively different distributions for steering wheel movements between the two studies. As a consequence, steering wheel movement data are not poolable between studies. There is no apparent reason for this difference in distributions.

In addition, inspection of the steering wheel data for drivers 36 and 42 indicated that these distributions were qualitatively different compared to steering wheel data from the other drivers. Driver 36 had mean and median values on the order of more than 180 under both conditions. Driver 42 had a mean value of less than six and a median of zero in the NO FEEDBACK condition, and mean and median values of zero in the FEEDBACK condition. Therefore, steering wheel movements were hard coded to missing for drivers 36 and 42. Driver 39 also had distributions of steering wheel movements similar to driver 42, but driver 42 was already excluded from all analyses due to insufficient total follow-up. The following special 'AD' versions of Data Quality Tables 19 and 20 are presented without excluding any drivers. These tables provide the distributions of steering wheel movements for all drivers based on the cleaned analysis sample and document the reason why steering wheel movement data were excluded for drivers 36 and 39, even though these drivers are included in the analysis dataset.

Data Quality Table 19AD: Cleaned Analysis Sample (All Drivers)
NO FEEDBACK Steering Wheel Movements Distribution

Data Quality Table 20AD: Cleaned Analysis Sample (All Drivers)
FEEDBACK Steering Wheel Movements Distribution

Data Quality Table 23AD: Cleaned Analysis Sample (All Drivers)
FEEDBACK Wheel Movements Distribution

Ambient light (Data Quality Table 24)

The median ambient light values for all drivers under both conditions is 0, confirming that most driving was at night. In comparison, in the Canada Study Phase, the median values were typically around 150.

Drivers 32, 35, 41, and 42 appeared to have appreciable amounts of non-night driving in the NO FEEDBACK condition. In the FEEDBACK condition this was true for only drivers 32 and 35.

The validity of this observation was validated by comparison with Data Quality Table 29. Drivers 32, 35, and 42 have AP+ On/off sensor "Day Light" values equal to 1 in relatively high numbers confirming the results from the ambient light meter. In contrast, the AP+ On/off sensor for "Day Light" in the NO FEEDBACK condition for driver 42 has no values equal to 1, even though driver 42 appeared to have appreciable amounts of non-night driving in the NO FEEDBACK condition on the basis of the ambient light values. There is no apparent reason for this discrepancy.

During the FEEDBACK condition, driver 32 had many more records with the AP+ On/off sensor for "Day Light" equal to 1 compared to any other driver and driver 35 had the second most number of records.

Of the 227 total PVT trials during the NO FEEDBACK condition, 49 (21.6%) were performed during the day or evening. Most of these were from drivers 39, 40, 41, and 42. Of the 209 total PVT trials during the FEEDBACK condition, 35 (16.7%) were performed during the day or evening and 31 of these were from these same four drivers.

Raw data metric for speed (Data Quality Table 30)

AP+ velocity data obtained during the Canadian Study Phase was provided in km/hr. There was no information provided to U.S. that the nature of the AP+ recording would change. However, inspection of preliminary versions of Data Quality Table 30 clearly indicated that that velocity was now being provided in mph rather than km\hr. Our programming initially applied a translation to the velocity data to convert km/hr to mph. The initial version of Data Quality Table 30 indicated that the average mean and median speeds under both conditions were about 38 mph with maximum values of roughly 44 mph.

We found these values to be obviously incorrect. Since 1 kilometer/hour = 0.6213712 mile/hour (mph) and 1 mile/hour (mph) = 1.609344 kilometer/hour these values can be 'untransformed' by multiplying by 1.61 (38 by 1.61 is 62, and 44 by 1.61 is 70.8). These values are very similar to average and maximum values for velocity observed in the Canada Study Phase. Therefore, the translation subroutine was removed for the U.S. data.

Engine Rotation (Data Quality Table 31)

The engine rotation values for driver 42 were all equal to 0 during the FEEDBACK condition. Therefore, all engine rotation data during both the NO FEEDBACK and FEEDBACK conditions were set to missing in the AP+ cleaned analysis dataset. After this exclusion, the distributions of engine rotation values looked very similar between studies.

X and Y acceleration (Data Quality Tables 32 and 33)

The following comments were from Pierre Pommarel of AP+ on Thursday, June 12, 2003, 2:19 PM.

Here are some notes concerning the data from the first two weeks of the experiment (no feedback):

* For ALL drivers, ignore the X and Y accelerometer values

* driver 031 (truck # 432-1312) and driver 032 (truck #432-1200) seems to have coherent data

* driver 033 (truck # 432-1263) and driver 034 (truck #432-1261) have an incomplete set of data, coming from the fact that both AP+ units were in "frozen" state. I think that driver 033 got data that can be processed. However, driver 034 has almost no data.

We interpreted the reference to 'ALL drivers' to mean drivers 31, 32, 33, and 34 only. We hard coded X and Y acceleration data for these 4 drivers to missing as reflected in Tables 32 and 33. Our data analysis confirmed that driver 34 had no U.S.ble AP+ data under the NO FEEDBACK condition. In contrast, driver 33 appeared to have substantial amount of NO FEEDBACK data and a full complement of FEEDBACK data (see Data Quality Table 4).

In further data quality procedures it was discovered that the values of X and Y acceleration for driver 42 during the FEEDBACK condition were all equal to 1.27. Therefore, X and Y acceleration values were set to missing under both the NO FEEDBACK and FEEDBACK conditions for this driver in the AP+ cleaned analysis dataset.

DQ Table 1: No Records Excluded Speed Categories (mph)

DQ Table 2: No Records Excluded Record Duration Category (seconds)

DQ Table 3: Cleaned Analysis Sample Record Duration Category (seconds)

Data Quality Table 4: No Exclusions and >=30 mph Sum of Record Durations (hours)

DQ Table 5: Cleaned Analysis Sample NF Record Duration Distributions (seconds)

DQ Table 6: Cleaned Analysis Sample FB Record Duration Distributions (seconds)

DQ Table 7: No Records Excluded SafeTRAC Lane Tracking (Offset) Confidence Category

DQ Table 8: Cleaned Analysis Sample SafeTRAC Lane Tracking (Offset) Confidence Category

DQ Table 9: Cleaned Analysis Sample NF SafeTRAC Lane Tracking (Offset) Confidence Category

DQ Table 10: Cleaned Analysis Sample FB SafeTRAC Lane Tracking (Offset) Confidence Category

Data Quality Table 11: Cleaned Analysis Sample SafeTRAC Vehicle Offset (unweighted by record duration)

Data Quality Table 12: Cleaned Analysis Sample Lateral Distance (2*Vehicle Offset-250) (unweighted by record duration)

Data Quality Table 13: Clean Analysis Sample NF Percentiles of Lateral Distance (2*Vehicle Offset-250)

Data Quality Table 14: Clean Analysis Sample FB Percentiles of Lateral Distance (2*Vehicle Offset-250)"

Data Quality Table 15: Cleaned Analysis Sample SafeTRAC Driver Alertness Summary (unweighted by record duration)

Data Quality Table 16: Cleaned Analysis Sample NF SafeTRAC Driver Alertness Distribution (unweighted by record duration)

Data Quality Table 17: Cleaned Analysis Sample FB SafeTRAC Driver Alertness Distribution (unweighted by record duration)

Data Quality Table 18: Cleaned Analysis Sample Steering Wheel Movements Summary (unweighted by record duration)

Data Quality Table 19: Cleaned Analysis Sample NF Steering Wheel Movements Distribution

Data Quality Table 20: Cleaned Analysis Sample FB Steering Wheel Movements Distribution

Data Quality Table 21: Cleaned Analysis Sample Wheel Movements Summary (unweighted by record duration)

Data Quality Table 22: Cleaned Analysis Sample NF Wheel Movements Distribution

Data Quality Table 23: Cleaned Analysis Sample FB Wheel Movements Distribution

Data Quality Table 24: Clean Analysis Sample Ambient Light (unweighted by record duration)

Data Quality Table 25: Cleaned Analysis Sample PERCLOS Camera Summary (where daylight=0) (unweighted by record duration)

Data Quality Table 26: Cleaned Analysis Sample NF PERCLOS Camera (where daylight=0) Distribution (unweighted by duration)

Data Quality Table 27: Cleaned Analysis Sample FB PERCLOS Camera (where daylight=0) Distribution (unweighted by duration)

Data Quality Table 28: No Records Excluded APP On/Off Sensor Status (Numbers of Records)

Data Quality Table 28: No Records Excluded (Continued) APP On/Off Sensor Status (Numbers of Records)

Data Quality Table 29: Cleaned Analysis Sample APP On/Off Sensor Status (Numbers of Records)

Data Quality Table 29: Cleaned Analysis Sample (Continued) APP On/Off Sensor Status (Numbers of Records)

Data Quality Table 30: Cleaned Analysis Sample Vehicle Speed (converted to mph) (unweighted by record duration)

Data Quality Table 31: Cleaned Analysis Sample Engine Rotation (unweighted by record duration)

Data Quality Table 32: Cleaned Analysis Sample "X" Longitudinal Acceleration (unweighted by record duration)

Data Quality Table 33: Cleaned Analysis Sample "Y" Lateral Acceleration (unweighted by record duration)

Data Quality Table 34: Safe Track Events by Driver and Condition