PlaceLab “Intensive Activity” Test Dataset 1 (PLIA1)

Please read the PlaceLab Data overview document before continuing.

Note: We have substantially changed the format used for storing PlaceLab data. We are leaving this dataset online because some researchers have used it. However, those new to the PlaceLab should select a more recent dataset to familiarize themselves with the data and facility capabilities. 

Overview  of PLIA1

This sample PlaceLab dataset was recorded on Friday March 4, 2005 from 9 AM to 12 noon with a volunteer from our research team who was familiar with the PlaceLab, but not a creator of the core technical infrastructure.

The researcher was asked to perform a set of common household activities during the four-hour period that included the following: preparing two recipes, doing a load of dishes, cleaning the kitchen, doing at least two loads of laundry, making the bed, and light cleaning around the apartment. The volunteer determined the sequence, pace, and concurrency of these activities and also integrated additional household tasks. Our intent was to have a short test dataset of a manageable size that could be easily placed on the web without concerns about anonymity. We wanted this test dataset, however, to show a variety of activity types and activate as many sensors as possible, but in a natural way. In addition to the activities above, the researcher searches for items, struggles to use an appliance, talks on the phone, answers email, and performs other everyday tasks. The researcher wore two mobile accelerometers (one on the left thigh and the other on the right wrist) and a Polar M32 wireless heart rate monitor. The mobile sensors are called MITes and will be described in a publication available shortly.

The dataset includes four hours of fully annotated video. The annotation was done using ProCoderDV, a professional annotation application. All four hours were annotated with behavior, using descriptors for body posture, type of activity, location, and social context. The annotation ontology is described in more detail later. Additionally, one hour was annotated with all instances of interaction with objects or furnishings that, according to raters, should have produced certain switch or motion-based sensor activations.

The format of this dataset is consistent with those used for other data collection periods, although the exact number and position of sensors may change.

Purpose

This dataset is intended primarily as a sample that researchers can explore in order to familiarize themselves with the type of data the PlaceLab is capable of generating. Researchers seriously interested in using PlaceLab datasets, including those from non-researcher volunteers staying in the facility for extended time periods, should contact us. The PlaceLab can be reconfigured to collect other types of sensor data as well.

IMPORTANT! More recent datasets have higher quality data and the formats have changed. For example, the portable wireless sensors have been dramatically improved since this sample dataset was collected.

Images from this document or raw data from this dataset is not intended for use in news articles or other publications without the explicit consent of a PlaceLab researcher.

Directory Structure

Size: 4.11 GB total (258 MB without Video and Audio)

The root directory of the data is named PlaceLab-03-04-2005, indicating the start date of the data collection. We refer to this as [root] below. This directory contains four subdirectories:

With the sensorData and videoData directories, the data is structured by date. In this case, all data was collected on a single day, so the subdirectory is 04Mar2005. Below that directory is one directory for every hour of the day in military time (i.e., 09 – 12 for this dataset). More detail on each data type follows below.

The data can be found here: http://web.mit.edu/cron/group/house_n/data/PlaceLab/PlaceLab-03-04-2005. Note that each of the directories is zipped for convenient downloading except for the (extremely large) videoData directory.

PlaceLab Sensor Data Overview

All raw sensor data except video and audio is located in the [root]/sensorData directory.  In this dataset, there are 11 different types of sensors whose data is recorded. Listed by type of sensor, they are:

Type of Sensor

Description

Filename

Wired switch

Detects on/off and open/closed events, such as doors being opened/closed and knobs being turned using switches built into the infrastructure.

1WireSwitch.dat

Wired humidity

Measures relative humidity in various locations using a wired sensor.

1WireHumidity.dat

Wired pressure

Measures barometric pressure in the living room using a wired sensor.

1WirePressure.dat

Wired light

Measures degree of illumination in several areas (especially those without cameras) in order to detect if lights are on.

1WireIllumination.dat

Wired temperature

Measures ambient temperature at floor and ceiling around the apartment.

1WireTemperature.dat

Wired gas

Measures amount of gas flow of the hot water heater and the stovetop burners.

1WireGasFlow.dat

Wired current

Measures amount of current flow in 43 electrical circuits around the apartment.

1WireCurrent.dat

Wired water

Measures amount of water flow for hot or cold faucets and toilets.

1WireWaterFlow.dat

Wireless static

Measures movement of the object to which it is attached and wirelessly sends data to receivers scattered around the apartment.

MITesStatic.dat and MITesStaticAlive.dat (stores “alive” pings received from sensors to verify they are active even when not moving).

Wireless heart rate

Measures heart rate of the person wearing a Polar wireless heart rate chest strap.

MITesHeartRate.bin

Wireless on-body

Measures limb movement of the person wearing the sensor using custom, wireless 3-axis 0-10G accelerometers. Receivers scattered around the apartment collect data.

MITesOnBodyPL16.bin

MITesOnBodyPL17.bin

MITesOnBodyPL18.bin

MITesOnBodyPL19.bin

MITesOnBodyPL20.bin

MITesOnBodyPL21.bin

Areas of visual motion from cameras

Indicates which of 18 camera views were selected (and saved as one of 4 quadrants) at any given moment. Also indicates which audio stream is selected. Can be useful for determining location of a PlaceLab resident.

VideoAudioActive.dat

Areas of visual motion from cameras

Indicates the amount of motion detected in all 18 camera views. Can be useful for determining location of a PlaceLab resident.

VideoDifferencer.dat

Warnings – any unusual data received by the data archiving computer (e.g. due to a faulty sensor). Used by PlaceLab team for debugging.

Warnings.dat and Unknown.dat

Audio and visual data is also saved and located in the [root]/sensorData directory.

All sensor data (with the exception of video, heart rate, and on-body data, discussed more below) is stored as text in .dat files with a separate file for each type of sensor.  Each file represents an hour of sensor activity.  Each line of text within one of these files contains information about the sensor state at the time indicated by a time stamp.

Wired sensors are directly connected to a TINI board in one of the cabinet units in the PlaceLab.  Refer to [link to paper] for more information about how they are wired.  Data from these wired sensors is stored in files named 1Wire*.dat, where * is the type of sensor.  The format is as follows:

Timestamp(hr:min:s:ms)   Type_number   ID_number   Value   IP_address

The first value is the time at which the information was recorded.  The second value indicates the sensor type.  The third value is the ID number of the particular sensor that sent the data at the given time – each sensor has a unique ID.  The Value field contains a measurement that is in units specific to what the sensor is meant to detect.  The final value on each row represents the last byte of the four byte (32 bit) IP address of TINI board to which the sensor is connected. TINI boards are scattered throughout the apartment. Figure 1 below shows the location of each TINI board.

Figure 1: TINI Board Locations with Identifiers

Wired Switches

The switches are sensors that allow communication through the standard Dallas Semiconductor 1-wire protocol. Specifically, the switches use a DS2406P Dual Addressable Switch plus 1kb Memory IC and were obtained from MAXIM for $4.65(US) apiece.  Documentation for the sensors and ordering information can be found on the MAXIM website [http://www.maxim-ic.com/quick_view2.cfm/qv_pk/2907]. The sensors detect whether a movable object, such as a cabinet door or drawer, is in an open or closed state. Since the IC is dual addressable two switches can be attached to one sensor board. They send an integer value of zero or two for open, and a value of one or three for closed, depending to which of the two sensor board locations a switch is wired.

There were 85 switches (53 different sensor boards) active during this data collection period.  The location of each switch sensor during this experiment is indicated on Figure 2. The description of each sensor can be found in the sensor ID file ([root]\sensorLocationInformation\sensorlist.csv).  For example, in that file the row,

0900000022A09C12Z, 1, 1WireSwitch, 69, Living room window right, 3, 95

indicates that switch sensor with ID 0900000022A09C12Z is located at pixel position (3, 95) on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  This sensor is on the right side living room window.  The sensor ID can be looked up on the images in [root]\sensorLocations\ims\. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

Figure 2: Switch Sensor Locations

  Switch sensors are sampled at different frequencies based on if there are other sensor types connected to the TINI board. If no other sensor type (e.g. only switch sensors) are connected to a TINI board then the sampling frequency is approximately 1.1 seconds. This frequency is approximate due to variance that can result from the number of different switch sensor boards connected to a specific TINI board. If other sensors are connected to the same TINI board, then the switch sensor sampling frequency depends on the number of other sensors and the individual sampling frequencies of these other sensors. The non-switch sensors are sampled at a slower sampling rate in order to avoid causing delays in the sampling of the switch sensors. Thus the sampling strategy is as follows. Once a minute all sensors attached to a TINI board are sampled. If the sampling of all devices attached to the TINI board (including switch sensors) takes less than 60 seconds then in the remaining seconds of the minute cycle only the switch sensors are polled (see Figure 3). Future upgrades to the PlaceLab may allow for a higher sampling rate.

Figure 3: Switch Sensor Sampling Strategy

The switch data is stored in [root]\sensorData\[date]\[hour]\1WireSwitch.dat. Data in each 1WireSwitch.dat file looks like,

            09:00:06:921 1 A4000000223AA112 3.0 62

            09:00:07:250 1 1500000022CFF412 2.0 66

            09:00:18:046 1 350000002233C912 3.0 66

            .

            .

            .

            09:01:33:843 1 3700000022B4B912 3.0 52

            09:01:34:921 1 AA00000022B46A12 0.0 52

            09:01:37:531 1 3700000022B4B912 2.0 52

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (1 for switches), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\sensorData\09\1WireSwitch.dat file.  Therefore, the first line of the example indicates that switch sensor 3700000022B4B912 located on the front entrance door (see Figure 2) returned a value of 3.0 on March 4, 2005 at 9:01 AM (and 33.843 seconds), indicating the door was opened.  The next time data from that same sensor was recorded was at 9:01 AM (and 37.531 seconds) with a value of 2.0, indicating the door was closed.

Humidity Sensors

The humidity sensors are sensors that allow communication through the standard Dallas Semiconductor 1-wire protocol. Specifically, the humidity sensors use a Honeywell HIH-3610, a popular relative humidity to DC voltage sensor. The Honeywell HIH-3610-002 were ordered from Arrow Electronics [http://www.arrow.com/] for $23.72(US) apiece. Documentation for the Honeywell HIH-3610 can be found on the Honeywell website [http://content.honeywell.com/sensing/prodinfo/humiditymoisture/009012_2.pdf]. Additionally, the humidity sensors use a DS2438Z Smart Battery Monitor IC and were obtained from MAXIM for $1.97(US) apiece.  Documentation for the sensors and ordering information can be found on the MAXIM website [http://www.maxim-ic.com/quick_view2.cfm/qv_pk/2919]. The sensors measure humidity in RH (relative humidity) between 0% and 100% with a maximum precision of approximately 1/3 %. 

There were 10 humidity sensors active during this data collection period. The location of each humidity sensor during this experiment is indicated on Figure 4. The description of each sensor can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  For example, in that file the row,

0D0000003F607126, 2, 1WireHumidity, 69, Living room bookshelf attic, 44, 40

indicates that humidity sensor with ID 0D0000003F607126 is located at position 44, 40 on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  This sensor is located at the top of the bookshelf to the right of the living room window.  The sensor ID can be looked up on the images in [root]\sensorLocations\ims\. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

Figure 4: Humidity Sensor Locations

Humidity data is stored in [root]\sensorData\[date]\[hour]\1WireHumidity.dat.  The humidity sensors are sampled at approximately once per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WireHumidity.dat file looks like,

09:00:06:093 2 120000004C002126 16.87749512483423 54

09:00:09:593 2 040000004C095D26 14.55548760372063 68

09:00:12:625 2 1E0000003F607F26 12.06580303329285 51

.

.

.

09:01:00:953 2 040000004C095D26 14.55646746086851 68

09:01:01:015 2 120000004C002126 16.87296477167182 54

09:01:02:796 2 1E0000003F607F26 11.99264929718915 51

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (2 for humidity), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\sensorData\04Mar2005\09\1WireHumidity.dat file.  Therefore, the first line of the example indicates that humidity sensor 040000004C095D26 located above the microwave (see Figure 4) returned a value of 14.55548760372063% RH on March 4, 2005 at 9:00 AM (and 9.593 seconds).  The next time data from that same sensor was recorded was at 9:01 AM (and 0.953 seconds) with a value of 14.55646746086851% RH. (Most of that precision is noise but shown here for clarity!)

 

Barometric Pressure Sensors

The barometric pressure sensors are sensors that allow communication through the standard Dallas Semiconductor 1-wire protocol. Specifically, the pressure sensors use a Motorola MPX4115A, an integrated silicon pressure sensor. The Motorola MPXA4115A6U-ND was ordered from Digikey [http://www.digikey.com/] for $17.57(US) apiece. Documentation for the MPX4115A can be found on the Digikey website [http://rocky.digikey.com/WebLib/Motorola/Web%20Data/MPX4115A,%20MPXA4115A%20SERIES.pdf]. Additionally, the pressure sensors use DS2438Z Smart Battery Monitor ICs that were obtained from MAXIM for $1.97(US) apiece.  Documentation for the sensors and ordering information can be found on the MAXIM website [http://www.maxim-ic.com/quick_view2.cfm/qv_pk/2919]. The sensors measure barometric pressure in kiloPascals between 15 kPa and 115 kPa with a maximum precision of 1/ (4.59) kPa.

There was a single pressure sensor used during this experiment.  Its location in indicated on Figure 5. The description of each sensor can be found in the sensor ID file ([root]\sensorLocationInformation\ sensorlist.csv).  This sensor is located at the top of the bookshelf to the right of the living room window.

Figure 5: Pressure Sensor Location

Pressure data is stored in [root]\sensorData\[date]\[hour]\1WirePressure.dat.  The pressure sensor is sampled at approximately once per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WirePressure.dat file looks like,

     09:00:49:953 3 4A0000004C097626 102.7777777777777 69

      09:01:44:906 3 4A0000004C097626 102.7777777777777 69

      .

      .

      .

      09:58:50:515 3 4A0000004C097626 102.7777777777777 69

      09:59:45:390 3 4A0000004C097626 102.7777777777777 69

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (3 for pressure), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\sensorData\04Mar2005\09\1WirePressure.dat file.  Therefore, the first line of the example indicates that the pressure sensor, with ID 040000004C095D26, returned a value of 102.7777777777777 on March 4, 2005 at 9:00 AM (and 49.953 seconds).  The next time data from that same sensor was recorded was at 9:01 AM (and 44.906 seconds) with the same value. As with many of the other sensors, most of the fractional precision shown here is  noise or results from division operations (e.g. when converting voltage to pressure).

Light Sensors

The light sensors are sensors that allow communication through the standard Dallas Semiconductor 1-wire protocol. Specifically, the light sensors use a Vishay silicon PIN photodiode (model# BPW34), high speed and high sensitive PIN photodiode in a miniature flat plastic package. The Vishay photodiode was ordered from Arrow Electronics [http://www.arrow.com/] for $0.57(US) apiece. Documentation for the Vishay photodiode can be found on the Vishay website [http http://www.vishay.com/docs/81521/81521.pdf]. Additionally, the light sensors use a DS2438Z Smart Battery Monitor IC and were obtained from MAXIM for $1.97(US) apiece.  Documentation for the sensors and ordering information can be found on the MAXIM website [http://www.maxim-ic.com/quick_view2.cfm/qv_pk/2919]. The sensors measure irradiance (mW/cm2) in between [0.020 to 1.2].

There were 6 light sensors active during this data collection period placed in the following locations:

ID

Location

030000004BFDA026

Master bath ceiling light shelf above shower  facing mirror

150000004C1DAD26

Bedroom above wardrobe

5A0000004C0EA126

Bedroom above side closet

680000004C1DA126

Powder room above toilet facing mirror

D30000004C005226

Bedroom ceiling light shelf

F10000004C0B8B26

Living room television screen

 The location of each light sensor during this experiment is indicated on Figure 6. The description of each sensor can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  For example, in that file the row,

030000004BFDA026, 4, 1WireIllumination, 55, Master bath sink attic, 786, 224

indicates that light sensor with ID 030000004BFDA026 is located at position 786, 224 on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  This sensor is located near the ceiling above the sink in the master bathroom.  The sensor ID can be looked up on the images in [root]\sensorLocations\ims\. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

Figure 6: Light Sensor Locations

Illumination data is stored in [root]\sensorData\[date]\[hour]\1Wireillumination.dat.  The light sensors are sampled at approximately once per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WireIllumination.dat file looks like,

     09:00:00:359 4 680000004C1DA126 4.8828125 56

      09:00:03:968 4 5A0000004C0EA126 2.44140625 62

      09:00:06:921 4 F10000004C0B8B26 0.0 54

      .

      .

      .

      09:01:00:140 4 5A0000004C0EA126 2.44140625 62

      09:01:01:906 4 F10000004C0B8B26 -4.8828125 54

      09:01:02:484 4 030000004BFDA026 4.8828125 55

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (4 for light), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\ sensorData\04Mar2005\09\1WireHumidity.dat file.  Therefore, the first line of the example indicates that light sensor F10000004C0B8B26 located on the television screen in the living room (see Figure 6) returned a value of 0.0 on March 4, 2005 at 9:00 AM (and 6.921 seconds).  The next time data from that same sensor was recorded was at 9:01 AM (and 1.906 seconds) with a value of –4.8828125. It should be noted that the negative value is a result of near complete dark light conditions and the photodiodes inaccuracy at these very low light conditions.

Temperature Sensors

The temperature sensors are sensors that allow communication through the standard Dallas Semiconductor 1-wire protocol. The temperature sensor used is the DS18S20 and was obtained from MAXIM for $2.57(US) apiece.  Documentation for the sensors and ordering information can be found on the MAXIM website [http://pdfserv.maxim-ic.com/en/ds/DS18S20.pdf]. The sensors measure temperature in degrees Celsius from -55 to +125 with a maximum precision of  ±0.5 degree in the range –10 to +85 Celsius.

 

There were 36 temperature sensors active during this data collection period.  Floor sensors are usually located 25 centimeters off the ground and “ceiling” sensors are normally located 200 centimeters from the ground (which is usually 22 centimeters from the ceiling). The location of each temperature sensor during this experiment is indicated on Figure7. The description of each sensor can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  For example, in that file the row,

750008003AFC4310, 5, 1WireTemperature, 55, Master bath over sink left upper, 786, 212

indicates that temperature sensor with ID 750008003AFC4310 is located at position 786, 212 on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  This sensor is on the ceiling as indicated by the description of the cabinet (“over sink left upper” is a high cabinet).  The sensor ID can be looked up on the images in [root]\sensorLocations\ims\. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

Figure 7: Temperature Sensor Locations

Temperature data is stored in [root]\sensorData\[date]\[hour]\1WireTemperature.dat.  Temperature sensors are sampled at approximately 1 per min in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WireTemperature.dat file looks like,

          09:00:12:890 5 750008003AFC4310 21.5 55

            09:00:13:828 5 EA0008003AE29410 18.5 63

            09:00:15:265 5 1E0008003AD70A10 25.0 63

            .

            .

            .

            09:01:01:093 5 9E0008003AFF6D10 19.5 66

            09:01:01:171 5 750008003AFC4310 21.5 55

            09:01:10:515 5 EA0008003AE29410 18.5 63

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (5 for temperature), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\sensorData\04Mar2005\09\1WireTemperature.dat file.  Therefore, the first line of the example indicates that temperature sensor 750008003AFC4310 located in the ceiling above the sink in the master bathroom (see Figure 7) returned a value of 21.5 degrees C on May 4, 2005 at 9:00 AM and (12.890 seconds).  The next time data from that same sensor was recorded was at 9:01 AM and (1.171 seconds) with a value of 21.5 degrees C.

As one might expect, the temperature values at the floor are typically lower than temperature readings at the ceiling.  Some sensors are located near cool drafts (e.g. near the glass door) and tend to output lower temperatures.

Gas Flow Sensors

The gas flow sensors are mass flow meters and were obtained from Omega (Model# FMA 1700) at [~550USD].  Documentation for the sensors can be found [http://omega.com/manuals/manualpdf/M1680.pdf].  The sensors measure gas flow in gallons per minute between 0.5 and 5 Gallons Per Minute (GPM) with a maximum precision of ±1.5% full scale. The output of the sensor is a linear voltage between 0-5VCD for full output scale and the response time is 800ms. The sensor is powered by a +12V power supply.

There was a single gas flow sensor used during this experiment.  Its location in indicated on Figure 8. The description of each sensor can be found in the sensor ID file ([root]\sensorLocationInformation\ sensorlist.csv).  It is attached to the gas line leading to the kitchen stove.

Figure 8: Gas Sensor Location

Gas flow data is stored in [root]\sensorData\[date]\[hour]\1WireGasFlow.dat.  The gas flow sensor is sampled at approximately once per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WireGasFlow.dat file looks like,

     09:00:08:250 6 5B00000053C01E26 0.05 68

      09:00:59:609 6 5B00000053C01E26 0.05 68

      .

      .

.

      09:28:56:750 6 5B00000053C01E26 0.05 68

      09:29:47:515 6 5B00000053C01E26 0.99 68

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms),  sensor type_number (6 for gas), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\sensorData\04Mar2005\09\1WireGasFlow.dat file.  Therefore, the first line of the example indicates that the gas flow sensor, with ID 5B00000053C01E26, associated with the kitchen stove, returned a value of 0.05 GPM on March 4, 2005 at 9:00 AM (and 8.250 seconds).  The next time data from that same sensor was recorded was at 9:00 AM (and 59.609 seconds) with the same value.

Current Sensors

The current sensors are split-core current transformers and were obtained from CR Magnetics (Model# CR3110-3000) at $12.25 per device.  Documentation for the sensors can be found [http://www.crmagnetics.com/pdf/3110.pdf].  The sensor is attached to a generic 1-wire sensor board using the DS2438Z Smart Battery Monitor as an ADC. The sensors board measure current flow in amperes between 0 and 20 amps with a maximum precision of  ±7% full scale. The sensor board generates a linear output voltage from 0-10.23VDC for a full output scale. The sensor board is powered by a +12V power supply.

There were 37 current sensors active during this data collection period. All the current sensors are located in a utility box near the main circuit breaker in the apartment. The location of outlets and devices for each circuit is indicated on Figure 9 and can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  Note that the colors in Figure 9 are significant and indicate which outlets/devices are associated with which labels.

In the sensorList.csv file the row,

1A00000053B37626, 7, 1WireCurrent, 0, Living Room Lighting, 30, 520

indicates that current sensor with ID 1A00000053B37626 is located at position 30, 520 on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  This sensor is attached to the circuit associated with the living room lighting.  The sensor ID can be looked up on the images in [root]\sensorLocations\ims\. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

Figure 9: Current Sensor Locations subdivided by light and power.

Current flow data is stored in [root]\sensorData\[date]\[hour]\1WireCurrent.dat.  The current sensors are sampled at approximately once per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat (in seconds) depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Data in each 1WireCurrent.dat file looks like,

     09:00:01:046 7 A600000054001D26 0.0 67

      09:00:02:203 7 D300000054005D26 3.42 67

      09:00:03:359 7 5400000053B35D26 0.0 67

      .

.

      .

      09:01:06:140 7 A600000054001D26 0.0 67

      09:01:07:015 7 D300000054005D26 0.62 67

      09:01:07:890 7 5400000053B35D26 0.0 67

This data type follows the standard 1-wire sensor format described above: Timestamp(hr:min:s:ms), sensor type_number (7 for current), ID_number of 1-wire sensor, value returned by sensor, and last two digits of the IP address of the computer (i.e., the TINI board) that sent the data.  This example is found in the [root]\ sensorData\04Mar2005\09\1WireCurrent.dat file.  Therefore, the first line of the example indicates that current sensor D300000054005D26 associated with the hot water heater located in the bedroom   (see Figure 9) returned a value of 3.42 amperes on March 4, 2005 at 9:00 AM (and 2.203 seconds).  The next time data from that same sensor was recorded was at 9:01 AM (and 7.015 seconds) with a value of 0.62 amperes.

Water Flow Sensors

Water flow sensors have been installed on hot and cold water supply lines at fixtures in the PlaceLab kitchen, laundry closet, powder room, and master bathroom. Paddle-wheel flow meters from Omega Engineering, Inc. (Model# FPR133) are used. Details on these meters can be found at http://www.omega.com/manuals/manualpdf/M1982_d.pdf. The model chosen measures water flow in gallons per minute between 0.5 and 5 gpm with an accuracy of +/- 7%.

There were 14 water flow sensors active during this data collection period. A flow sensor is located on each cold and hot water outlet. These are kitchen sink hot, kitchen sink cold, kitchen dishwasher, master bath sink hot, master bath sink cold, mater bath toilet, master bath shower hot, master bath shower cold, powder room sink hot, powder room sink cold, powder room toilet, clothes washer hot, clothes washer cold.. The location of each water flow sensor during this experiment is indicated on Figure 10. The description of each sensor can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  For example, refer to the following:

1300000053D19026, 8, 1WireWaterFlow, 0, Kitchen Sink Cold, 314, 101

This entry indicates that the sensor with ID 1300000053D19026 is of type ‘8’ (1WireWaterFlow). It is attached to an undesignated TINI board (i.e., 0), and is installed on the cold water supply to the kitchen sink. This sensor is located at position 314, 101 on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.

Water flow data is stored in [root]\sensorData\[date]\[hour]\1WireWaterFlow.dat.  The water flow sensors are sampled approximately 5 times per minute in this data set.  All analog 1-wire sensors are sampled infrequently to avoid causing delays in sampling of switch sensors.  The sampling rate can vary somewhat depending upon activations of other nearby sensors.  Future upgrades to the PlaceLab may allow for a higher sampling rate.

Figure 10: Water Flow Locations

Data recorded from the water flow sensors follows the standard 1-wire sensor format described previously:

Timestamp(hr:min:s:ms)     Type_number     ID_number      Value      IP_address

Below is a sample of data from the WireWaterFlow.dat file for 9:00am on 4 March 2005:

     09:00:02:281 8 9A00000053D0E426 0.02 55

      09:00:03:984 8 7200000053D35C26 0.02 55

      .

      .

      .

      09:00:18:046 8 9A00000053D0E426 0.01 55

      09:00:19:625 8 7200000053D35C26 0.01 55

      09:00:20:796 8 5B00000053C56326 0.01 55

All entries in this file begin with a Timestamp indicating when the value was received by the data logger. Type_number ‘8’ designates a flow sensor and will not vary in this document. The 16-digit hexadecimal ID_number can be used to identify the location of this sensory using the [root]\sensorLocations\sensorlist.csv spreadsheet. To locate the image that shows the sensor of interest, use the ImageMap.csv file located in the [root]\sensorLocations\ims\ directory.

In the example shown here, the sensor (9A00000053D0E426) shown on the first line is located on the hot water pipe for the master bathroom sink. At the time of report, it returned a value of 0.02 GPM. The next time data from that same sensor reported (15.765 seconds later) it returned a value of 0.01 GPM.

One problem with using the flow sensors in the PlaceLab is that there is a tendency for the paddlewheels to flow backwards sometimes even when the water is on. This problem is due to a flow restrictor on all of the faucet nozzles and unequal pressure on the hot water and cold water lines. The backflow problem occurs when both the hot water and cold water for a single faucet are opened all the way or nearly all the way. Restriction of water flow as a result of the flow restrictor on the faucet nozzle causes pressure to build and water to backflow through the hot water line. The back flow is not substantial, but it causes the reading of the hot water flow to be at its minimum value event though the hot water is on. Incidentally, this reading is near accurate since the temperature of the water is cooler than if the hot and cold water flow were set equal for lower individual rates of flow.

Wireless Object Movement Sensors

The wireless object movement sensors (MITes) are sensors designed to be attached to any movable physical object. Fundamentally, MITes consist of one two-axis 2G accelerometer and an RF transceiver. A MITes is a simple sensor used to measure acceleration and broadcast a unique ID whenever a dynamic movement causes the measured acceleration to surpass a specified sensitivity threshold. The MITes have been dramatically improved since this dataset was collected.

There were 124 MITes sensors active during this data collection period.  The MITes are attached to a wide variety of movable objects throughout the PlaceLab. These objects include items such as cabinet and entry doors, drawers, chairs, toilet seat covers, couch cushions, soap dispensers, etc. The location of each MITes sensor during this experiment is indicated on Figure 11. The description of each sensor can be found in the sensor ID file ([root]\sensorLocations\sensorlist.csv).  For example, the following entry,

91, 11, MITesStatic, 0, kitchen - microwave door, 460, 130

indicates that MITes sensor with ID 91 is located at position (460,130) on the PlaceLab plan diagram in [root]\sensorLocations\plan.jpg.  As indicated this sensor has been placed on the door of the kitchen microwave. 

Figure 11: Static MITes Locations

The MITes sensor data is stored in [root]\sensorData\[date]\[hour]\MITesStatic.dat. 

The data transmitted by the individual MITes is collected by one or more MITes RF receivers located throughout the PlaceLab. In total there are six receivers placed throughout the PlaceLab to ensure maximum RF coverage. The following table lists the locations of the six receivers and the identifier of the PC to which each receiver is attached. 

Placelab Machine

Physical Location

Placelab 16 (PL16)

Kitchen island

Placelab 17 (PL17)

Living Room

Placelab 18 (PL18)

Bedroom

Placelab 19 (PL19)

Office

Placelab 20 (PL20)

Hallway Bathroom (Powder Room)

Placelab 21 (PL21)

Master Bathroom

Data in each MITesStatic.dat file looks like,

09:01:35:156 11 274 21 0 18.0

09:01:35:671 11 274 21 1 7.0

09:01:37:515 11 95 16 3 43.0

This data type follows the format:

Timestamp(hr:min:s:ms),  sensor type_number (11 for MITes), ID_number of sensor, last two digits of the IP address of the computer (i.e., the TINI board) that sent the data, resend ID, and the value at which the sensor was activated.  This example is found in the [root]\sensorData\04Mar2005\09\MITesStatic.dat file.  Therefore, the first line of the example indicates that MITes sensor 274 located on door to the master bedroom (see Figure 11) was activated with an acceleration value of 18.0 g on March 4, 2005 at 9:01 AM (and 35.156 seconds). This value represents the magnitude of the difference between the current acceleration value and a running average filter over the previous three acceleration samples.  This value is measured in units of g (also known as g-force or g-load). g is a non-SI unit of acceleration which is defined as 9.80665 m/(s*s), approximately the acceleration due to Earth’s gravitation. 

As can be seen in the previous table, each receiver is attached to a different processing machine (PC). The MITes data collection architecture is distributed by having an individual computer process the data for each receiver and then forward this data to a central routing machine (PL16). This central routing machine then forwards the pre-processed data on to a data recording machine (PL15). Figure 12 describes this architecture. This distribution is to allow for future complex, real-time processing of both the data being transmitted by the wireless object movement MITes and the higher resolution data being transmitted by the Mobil Accelerometer MITes and the Heart Rate MITes (see below) while not disrupting the collection of other sensor data such as the wired sensors, video, etc. Artifacts of this distributed architecture and the wireless point-to-point communication between the wireless MITes transmitter and the receiver lead to the need for some rudimentary pre-processing before the collected data is saved. The next few paragraphs present why this pre-processing is needed and how it is done.

Figure 12: MITes Receiver Architecture

Wireless transmission is inherently lossy. In the case of the MITes, two factors lead to data loss: Receiver Channel Multiplexing and Environmental Conditions.

The first factor is due to the nature of the RF receiver and how it listens to multiple channels via multiplexing. Intuitively, time spent listening to one channel means time is not spent listening to all other channels transmitting data. The receiver must switch between channels stopping to listen to each individual channel for a short period of time while ignoring all other channels for this period of time. This means that in order to listen to multiple channels the receiver must ignore all but one channel per time slice, and thus risk losing data on all other channels.

The second factor leading to data is a result of corruption of data during transmission. This corruption is due to other WIFI signals and other environmental electromagnetic noise, signal reflections, etc. A paper is currently in progress to fully document the performance of the MITes under various conditions.

Obviously it is vital that a single object activation (defined as the dynamic movement of an object which causes the accelerometer of the associated MITes to surpass some threshold) is not missed due to data loss or corruption. In order to overcome this problem, each time an object activation occurs, the associated MITes broadcasts its data for that specific activation six times via six identical data packets (see figure 13). The duplicate sends ensure that an activation signal is not missed due to packet loss or corruption.

 

Figure 13: MITes Transmission Scheme Example

As mentioned earlier there are some artifacts associated with using wireless transmission. One of these artifacts is that a receiver may receive anywhere from 1 (84% data loss) to 6 (0% data loss) identical packets for a single object activation. Activity recognition algorithms and memory constraints make it undesirable to save more than one of these packets per activation. Thus, some data filtering is done prior to saving the activation data. The distributed architecture of the MITes data collection (see figure 11) enables this filtering to be done at two levels – both at the individual processing unit level (Filter Serial Link data on PL 16 –PL21) and at the central router level (Filter UDP Link data on PL16). The same filtering technique is used at both levels. See figure X for a high-level example of the data filtering.  

 The filtering method used takes advantage 1.) a resendID associated with each object activation 2.) the fact that each MITes has a unique ID and 3.) the timing of MITes transmissions. Keep in mind that the overall goal is to ensure that only one data element is recorded per object activation. Every time an object activation takes place the MITes related to that object activation broadcasts six identical data packets encapsulating the data element associated with the object activation. One of the header elements in each of these packets is the unique ID associated with the particular MITes. Another header element is the Transmission ID. The Transmission ID is the same for all six data packets associated with a singular object activation. Transmission IDs loop from 0 to 6 inclusive. These numbers increment with each successive object activation.

The individual processing unit level filtering (That done on the individual PCs with a receiver attached) works by storing the most recently received Transmission ID associated with a particular MITes. If a packet is received with the stored Transmission ID, that packet is considered to be a duplicate and is ignored. If a Transmission ID different from the stored Transmission ID is received by a particular MITes, then this packet is forwarded on to the central routing machine and that new Transmission ID is stored. Only one Transmission ID is stored at a time. Since we have point-to-point transmission, this scheme ensures that no duplicate packets are sent to the central routing machine. Figure 14 outlines this filtering scheme.

Figure 14: Filtering Performed on Each Local Processing Unit

One problem that still needs to be overcome is the case where a receiver receives a packet with Transmission ID = 0 and then does not receive any packets associated with Transmission IDs 1 through 6. That is all packets associated with Transmission IDs 1 through 6 are either corrupted or lost. This means that the receiver completely missed 6 object activations. Further, since for every object activation six identical packets are broadcast, 36 data consecutive data packets were lost or corrupted. This is rare, but could be possible. What is the consequence of this amount of packet loss in terms of filtering? It means that the next set of six packets associated with an object activation will have a Transmission ID = 0, the same Transmission ID currently being stored for this particular MITes. Following the above filtering scheme, this new object activation would be ignored. An addition to our filtering algorithm is needed. The addition needed is a timer that clears the Transmission ID being stored if it has not changed in 900ms. Why does this fix the problem mentioned? Luckily the individual MITes will broadcast six packets per activation regardless if they are received by the receiver. Further, due to hardware transmission delays and battery life considerations, a MITes will take at least 300 milliseconds between object activation broadcasts. This means that there are 300 milliseconds between the broadcast of the last of the six packets with Transmission ID = n and the broadcast of the first packet with Transmission ID = n+1. As a result, there is at least 2.1 seconds before a MITes uses the same Transmission ID for two different object activations (it actually takes more time due to the overhead of sending each packet). Thus, if the Transmission ID is cleared after 900ms, there is no way that the filtering algorithm will ignore two separate object activations with the same Transmission ID. Figure 15 provides a graphical example of this algorithmic addition and the case which it is meant to address.

Figure 15: Example of why timeout case is needed

The above filtering method also works at the central routing machine level. The filtering is needed since the radiation patterns of the six receivers in the PlaceLab have overlap. As a result, it is possible that more than one receiver receives a single object activation. Each receiver that receives this activation will perform the individual processing unit filtering and forward one packet associated with the object activation to the central routing machine. This means that the central routing machine could recive multiple packets from the individual processing unit layer associated with the same object activation. This is similar to the way the receivers attached to the individual processing units can receive redundant packets and thus the same filtering scheme is used. Figure 16 represents why the same filtering scheme will work at this higher level.  

Figure 16: Global Filtering Example

MITes have a battery power source. In order to automatically detect if a MITes has died or malfunctioned, each MITes sensor broadcasts a signal indicating that it is operational. This signal is broadcast approximately once an hour.  However, due to internal clock issues at low power levels, the broadcast period can range from 45 minutes to about 75 minutes. This is a trade off for conserving battery power. The MITesStaticAlive.dat file keeps track of this “alive signal” information.

Data in each MITesStaticAlive.dat file looks like,

09:00:20:906 40 337 0

09:00:27:843 40 282 0

The first three values are the same as in the format described above.  The last value is irrelevant and can be ignored.  This example is found in the [root]\sensorData\ 04Mar2005\09\MITesStaticAlive.dat file.  Therefore, the first line of the example indicates that MITes sensor 337 located on the coffee table in the living room (see Figure X) sent an “alive” signal on March 4, 2005 at 9:00 AM (and 20.906 seconds).

Audio/Video Activity 

Nine infrared cameras, 9 color cameras, and 19 microphones are distributed throughout the apartment in cabinet components and above working surfaces, such as the office desk and kitchen counters. Eighteen computers use image-processing algorithms to select the 4 video streams and 1 audio stream that may best capture an occupant’s behavior, as inferred from motion activity and the camera layout in the environment. The file VideoDifferencer.dat contains information about the amount of activity each camera is recording and is in the following format:

Timestamp (hr:min:s:ms)      Type_number      Computer_ID      diff      Value1     Value2

All entries in this file begin with a Timestamp that is generated when the log entry is written to disk. The Type_number designation is a numeric code indicating that the data included in this file refers to video differencing values;  this code is always equal to ‘30’ and used only for routing the data across the network. Computer_ID is the identification number of the computer to which the particular camera is attached. Use the figure below to identify the location of the camera indicated by the Computer_ID.

In the data file, the string diff remains constant throughout the file and can be ignored. The next 2 values represent the mean number of pixels (Value1) that changed per frame in the camera image over a certain number of frames (Value2). A small value (e.g. < 30) is typically no motion or noise. A large value (e.g. > 200) indicates a large amount of motion in the view at the timestamp time. Each of the 18 computers processing video sends this video information approximately every 2 seconds. This data can be used to determine which regions of the home had the greatest visual activity at any time.

For example, at 10:48AM on 4 March 2005, the following VideoDifferencer.dat values were recorded.  Note the cluster of activity in the region represented by video streams V16, V14, V25, V19.  Based on the mapping of these cameras, this suggests activity occurring in the kitchen near the corridor.

10:48:09:781 30 V11 diff 1 11

10:48:09:937 30 V12 diff 0 12

10:48:09:953 30 V20 diff 0 12

10:48:10:109 30 V26 diff 0 12

10:48:10:140 30 V28 diff 1 12

10:48:10:171 30 V29 diff 1 12

10:48:10:390 30 V21 diff 1 12

10:48:10:468 30 V23 diff 1 12

10:48:10:515 30 V22 diff 34 12

10:48:10:546 30 V18 diff 2 12

10:48:10:640 30 V13 diff 0 12

10:48:10:765 30 V17 diff 0 12

10:48:10:843 30 V24 diff 4 12

10:48:10:859 30 V16 diff 282 12

10:48:10:937 30 V14 diff 1289 12

10:48:11:046 30 V25 diff 280 12

10:48:11:265 30 V19 diff 434 12

10:48:11:718 30 V27 diff 0 12

Each .avi video file produced by the PlaceLab contains the four video streams and single audio stream that best capture the occupant’s behavior at a particular time. The audio stream to be recorded is selected from the 19 microphone locations shown below according to the video stream indicating the most movement at a given time. A lookup table that maps each camera to a specific microphone is used to make the selection.

Camera

Mic.

Camera

Mic.

Camera

Mic.

Camera

Mic.

V11

A11

V16

A19

V21

A16

V26

A24

V12

A13

V17

A28

V22

A24

V27

A21

V13

A18

V18

A19

V23

A20

V28

A15

V14

A28

V19

A19

V24

A24

V29

A12

V15

A13

V20

A15

V25

A21

A record of the audio and video streams selected is stored in the file VideoAudioActive.dat. Each line in this file is in one of the 3 following formats:

Timestamp(hr:min:s:ms)    31    View      Value1     Value2

Timestamp(hr:min:s:ms)    31    Audio     Value

Timestamp(hr:min:s:ms)    31    Summary     V    Value    Percent . . . A    Value    Percent

As in the AudioDifferencer.dat files, each line of VideoAudioActive.dat begins with a Timestamp and a Type_number designation, which in this case is always ’31’.

In lines with a ‘View’ keyword, Value 1 is the numeric label (from 1 through 4) for a quadrant of the multiplexed video image. Value 2 is the camera view presently selected for that quadrant. This line is generated every 2.5 minutes for each quadrant, and at any time when Value 2 changes.  A sample video frame is shown below with the quadrant labels and camera IDs indicated.

Quadrant: 4                Camera: V21

Quadrant: 3                Camera: V28

Quadrant: 2                Camera: V16

Quadrant: 1          Camera: V14

In an ‘Audio’ line, Value is the ID number of the microphone that is currently selected. An audio line is automatically added to the file whenever a video quadrant is reported.

The ‘Summary’ lines report what cameras and microphones have been selected and for what percentage of time they have been selected since the last summary. Following the ‘V’ in the summary, Value is the ID of one of the cameras that has been and Percent is the percentage of the time since the last summary the camera view has remained selected. A minimum of four values will appear here. Following the ‘A’, Value is the ID of the microphone selected and Percent is the percentage of the time since the last summary that this microphone has been selected. At least one microphone will be reported, but there may be several of these value-percent pairs following ‘A’.

Shown below is a sample of VideoAudioActive.dat from 3:15am on 4 March 2005.  Because there was no activity in the PlaceLab at this time, the values provided did not change during the 2 minutes reflected here.

03:15:53:201 31 View 2 13

03:15:53:201 31 Audio 13

03:15:54:936 31 View 1 14

03:15:54:936 31 Audio 13

03:15:55:311 31 View 3 12

03:15:55:311 31 Audio 13

03:18:03:201 31 View 4 21

03:18:03:201 31 Audio 13

03:18:03:201 31 Summary V 12 100.0 13 100.0 14 100.0 21 100.0 A 13 100.0

Mobile Accelerometer Data

More information on the wireless accelerometers, known as mobile MITes, can be obtained by contacting House_n researchers. A paper is in progress. Mobile MITes stream real-time, 3-axis, +/- 10G accelerometer data.

The on-body sensor data is stored in binary file format in [root]\sensorData\[date]\[hour]\MITesOnBodyPLxx.bin, where xx is the ID number of the computer to which the receiver that recorded the signal is connected. The following table shows the receiver location and the associated computer number:

Placelab Machine Name / Number

Physical Location

Placelab 16 / 16

Kitchen island

Placelab 17 / 17

Living Room

Placelab 18 / 18

Bedroom

Placelab 19 / 19

Office

Placelab 20 / 20

Hallway Bathroom (Powder Room)

Placelab 21 / 21

Master Bathroom

Since up to 200 samples per second are collected from the wireless mobile devices throughout the day for multiple days, the size of the data file is very large. In order to minimize the file sizes the mobile MITes data is stored in a binary format. Each line of data is contained in eight 2-byte segments with the high byte first (2-byte Java shorts for each line item). The following is the standard line representation of a single line of data and an exemplar, decoded entry:

Representation:

Timestamp(hr:min:s:ms)  Channel_Number   Computer_ID   X_Acc Y_Acc Z_Acc

Exemplar Decoded Line:

12:15:35:456 12 16 455 574 489

Example Encoded Line (Hexadecimal):

00 0C 00 0F 00 23  01 C8 00 0C 00 10 01 C7 02 3E 01 E9

The first four two-byte segments represent the time, as with the previous sensors. The next two-byte represents a channel number, which is unique for each mobile MITes and can be used to matched a sensor to a part of the body. This is accomplished by the fact that the each mobile MITes has a physical label with a part of the body on it (e.g. “Wrist”, “Thigh”, etc.). This labeling helps ensure that the same MITes is placed on the same body part in the same orientation across several days or weeks. The last 3 bytes are measures of the relative acceleration in the x, y, and z axes, respectively. The range of each axis is [0 1023]. The units are in fractions of a g (20/1024, or approximately 0.02). To convert an acceleration value to the actual number of g’s, subtract 511 then multiply by 20/1024. The rest values of the accelerometers are dependent on the rotational orientation of the accelerometers.

In this study, the volunteer was wearing two mobile MITes. The first, programmed to channel 12, was located on the participant’s right wrist (dominant hand). The MITes was oriented such that the battery was toward the participant’s body and the ‘+’ sign on the battery holder was closer to the participant’s fingers. The second mobile MITes was placed on the participants left thigh. Again, the MITes was oriented so that the battery holder was toward the participant’s body and the ‘+’ sign was closer to the participant’s feet. 

(towards fingers/feet)

Battery holder

In PlaceLab studies, volunteers can wear up to 5 MITes simultaneously.

Wireless Heart Rate Data  

Another sensor the volunteer wore was a wireless heart rate monitor. The wireless heart rate monitor is an extension of the MITes interfaced with coded Polar heartbeat receiver units. The receiver units receive heartbeat signals from the wearable Polar M32 band.  The MITes, in turn, receive decoded pulses from the receiver, which correspond to heartbeats. The MITes device converts the beats in to a BPM value and transmits the value to a MITes receiver. The Polar M32 band is available from Polar and its distributors, while the receiver unit is available directly from Polar (Part#: 2380157).

The wireless heart rate data is also stored in binary file format in [root]\sensorData\[date]\[hour]\MITesHeartRate.bin. Each piece of data is contained in eight two-byte segments (two bytes for each sub-item). For the heart rate data, the first four two-byte segments represent the Timestamp(hr:min:s:ms)  (each of the hour, minute, second, and millisecond values are one byte, high byte first). The remaining four two-byte segments are the type number of the sensor (always 12 for heart rate), followed by the sensor ID number for the heart rate monitor (206 for this dataset), the computer ID, and finally the heart rate in beats per minute.

Representation:

Timestamp(hr:min:s:ms)   Type_number   ID_number   Computer_ID   Value

Example Encoded Line (Hexadecimal):

00 0C 00 3B 00 3B  03 E7 00 0C 00 CE 00 10 00 5A

Example line:

12 59 59 999 12 206 16 90 [endline char]

Translation:

Time: 12:59:59:999

Type: 12 (HR)

Sensor ID: 206

Computer_ID: 16  (as in Placelab16)

Value: 90 (BPM)

There is some startup time required by the HR sensor to lock onto the signal from the strap.  Once the signal is locked on, the HR Sensor will generally not lose it.  If the signal from the strap is lost while the HR sensor is in the middle of processing a bpm value, a sporadic value may result when the signal is reacquired.  The BPM value is sent as soon as it is calculated which is at most once a second and can take longer if the heart rate drops below 60 BPM.  A constant 1/s rate should not be assumed.

When using the strap, fluctuations in a person’s heart beat will be noticeable in the output of the HR sensor.  Loss of signal from the strap can also cause slight fluctuations but these are generally rare.  Applying a median and/or averaging filter on received values is recommended to remove fluctuations and outrageous values.

Like all static MITes operating in high channels (i.e. 60), the HR Sensors are somewhat susceptible to packet loss due to WI-FI interference. Fortunately the loss of packets does not affect the BPM value that is sent—only the number of values that are actually received.

Warnings

The final two files in the [root]\sensorData\[date]\[hour] directory are Warnings.dat and Unknown.dat. The first file contains warnings from sensors. These warnings are defined by the designer of a particular sensor and serve as alerts for custom events, such as out-of-range readings that the sensor designer would like to have reported in the hourly status report.

Every sensor in the PlaceLab is registered in the file [root]\sensorLocationInformation\sensorlist.csv. Any data received by the PlaceLab systems from sensors that are not registered in this file  is recorded into Unknown.dat

For example, a researcher could accidentally plug a new switch sensor into the 1-Wire network, and not update the sensorlist.csv file. At this point, if the associated TINI  board  is restarted, the new sensor starts generation data that gets sent to the control server. The server however has no information about this sensor, its type, or where it is located. The data from this sensor will be saved into Unknown.dat

Data and Audio/Video Synchronization

In order for all sensor recordings to temporally correspond to the video data, some time synchronization strategy is needed. This synchronization is done by recording synchronization timestamps for every keyframe of video in a separate text file in real time as the video signal is being digitized. These synchronization timestamps are stored in files named 00.txt, 01.txt along with 00.avi, 01.avi in the videoDate/[date] directory. These file names correspond to the hour of the day in 24-hour military time.

The timestamp files have records with three space separated fields, with each record representing an individual keyframe in the corresponding hour of video. The format is:

Key_Frame_Number  Key_Frame_Base_Time  Synchornization_Time

An example of the data contained in each file is as follows:


21014  3597.39  59.59.296
21015  3597.59  59.59.468
21016  3597.72  59.59.656
21017  3597.92  59.59.828
21018  3598.12  00.00.00 

This shows the last five lines of a timestamp file for an hour of video consisting of 21018 keyframes.  The last (timestamp) field is 00:00:00 which means the digitizing of the file ended exactly on the hour, but the video Key_Frame_Base_Time (middle field) is about 2 seconds short of 3600 seconds. This means that if no synchronization was used the sensor data would have been out of sync with video by about 2 seconds at the end of this hour.

The reason that the Synchronization_Time applies to all sensors is this is the system time of a central machine that is responsible for time stamping and writing to file all sensor data. This means between sensor time is synchronized and only the elapse of keyframes differs from this system time. This is why synchronization is needed.


Thus, to map from a timestamp in a sensor data file to a frame of video, locate the nearest timestamp in the last field of the .txt file for the hour - the corresponding keyframe in the video will take you to the video that corresponds to the sensor event.

Further, the annotation data (see next section) recorded by the ProCoderDV annotation tool is indexed based on the video Key_Frame_Base_Time, i.e., the middle field in the timestamp file. To map from a ProCoderDV annotation timestamp to a sensor firing, it is necessary to locate the corresponding Key_Frame_Base_Time and find the sensor firing closest to the Synchronization Time. This is the inverse of mapping a senor activation to Key_Frame_Base_Time.

For Example, suppose that the annotation file has the following line:


"00:04:23.26",,"standing still (state)","kitchen","alone",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

first convert the time to seconds (in this case, 263.26) and find the timestamp file record where the Key_Frame_Base_Time (middle field) is closest to 263.26. The Synchronization_Time (last field) on that record will match the sensor data timestamp at the time denoted in the annotation file.

Note: Currently there is an approximately 3 second latency between time stamped video and sensor data due to the compression process and occasional dropped frames during digitization. That is, the Synchronization_Time and the timestamp in individual sensor files differ by a constant 3 seconds. 

Annotation Data

Annotation data provides “ground truth” information about what is really happening in the PlaceLab dataset. The annotation data can be used to validate algorithms or search the data. We begin by describing the annotation data located in [root]/annotationData. Annotations were created by manual labeling using a commercial program called ProCoderDV. More information on this software can be found here: http://mingus.kc.vanderbilt.edu/pcdv/. ProCoderDV was created to make it easy to annotate relatively short segments of video with mutually exclusive activities, and it is not optimized for this annotation task. However, relative to other annotation tools it is reasonably robust and affordable. House_n is currently improving/testing a custom-designed annotation tool optimized for PlaceLab datasets. Eventually that tool will be added to this website.

Annotation filenames follow the general format of:

[exercise ID]_[date]_[annotation method]_[annotator initials].pdv

Exercise ID includes the toolset (PlaceLab = PL, Portable Kit = PK), the study code (Intensive Activity = IA), and the session number (###).  The date and time of the annotated hour are included in the name using the format "YYYYMMDDHH." The annotation method refers to the set of annotation labels and annotation instructions used by the annotator to code the hour and corresponds to the name of the associated ".cod"  file.  The annotator initials can be two to four characters and are provided as a way of distinguishing hours that have been repeatedly annotated to calculate reliability metrics.

For the example dataset, the hour of March 4, 2005, 9:00am has been coded using version 3 of the activity description annotation  codes by an MIT student with initials Y.N.; the associated annotation  file is therefore named "PLIA004_2005030409_AD03_YN.pdv"; the associated code file that was used in ProcoderDV to annotate this file is named  "AD03.cod"

There are 2 types of files associated with ProcoderDV: code (.cod) files and .pdv files. The software takes as input a code file and video file and produces a pdv file. The code file is an ASCII text document that contains definitions of the annotation codes to be used by the application.  The first line of a code file, when viewed as text, contains a number (n), which is the number of code types listed in that file. The next n lines contain comma-separated lists with information about each code. The first element is a single character that represents the particular code (used for single key press coding). The next element is the nominal label for the code, and, the last is an integer label used to group related codes.  Any of these fields may be blank. The final set of non-empty lines contain the names of the categories to which the previous codes belong. The order of the categories maps to the integer labels used to group the codes, i.e. the first category correlates to the number 0, the second to the number 1, etc.

Example from actual code file:

104

"[","start",0

"]","stop",0

"e","bending","1"

"-","kneeling","1"

"y","lying down (action)","1"

. . .

"r","bathroom","2"

"b","bedroom","2"

"d","dining area","2"

. . .

"1","dressing",4

"2","drinking",4

"3","drying dishes",4

"4","drying laundry",4

"5","drying laundry background",4

"6","dusting",4

. . .

"startstop"

"body posture"

"room location"

"social context"

"activity"

The ProCoderDV .pdv file is an ASCII text document that, when viewed as text, contains a multi-line header that includes information such as date, time, location, exercise, additional notes, the filenames of the associated video and code files, and the start and stop times of the video segment to be annotated. Each of the next two non-empty lines contains a number. The first is the number of rows of annotations included in the file; the second number specifies the number of fields in each row (this is held constant at 50). Each row is a comma-separated list, as in the code files. The fields map to the categories that are listed at the end of the given code file. For example, the IntensiveActivity code file (AD03) has the categories: startstop, body position, room location, social context, and activity. As a result, only the first 6 column fields in the rows of the corresponding pdv files are utilized and they correspond to the 6 categories.

Example .pdv file:

EVENTS

Researcher

3-04-05

11:00 am

PlaceLab

PLIA004

VH

<notes>

</notes>

..\videoData\04Mar2005\11.avi

.\AD03.cod

00:00:00.00

00:59:58.80

 233

 50

 

"00:00:00.00","start","bending","bedroom","alone","folding laundry",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:22.88",,"sitting down (action)",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:23.64",,"sitting (state)",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:41.10","stop","","","","folding laundry","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","",

"00:00:41.77",,"standing up (action)",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:41.97",,"walking",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:45.38",,,"hallway",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:47.27",,,"kitchen",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:50.67","start",,,,"putting things away",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:54.71","stop",,,,"putting things away",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:00:58.85","start","standing still (state)",,,"hand-washing dishes",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:22.61","stop",,,,"hand-washing dishes",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:28.66","start",,,,"cleaning miscellaneous",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:31.89",,"walking",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:35.56",,"standing still (state)",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:52.21",,"walking",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:52.96","stop",,,,"cleaning miscellaneous",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:55.54","start","standing still (state)",,,"dishwashing miscellaneous",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

"00:03:57.90",,"walking",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Start/stop labels, in the second column, are associated with activity labels appearing on the same line.  Each start-activity line must precede a corresponding stop-activity line within a particular file (“start…folding laundry” must have a “stop…folding laundry”).  Different activities may, however, overlap - one does not need to be stopped before another is started – to represent multitasking.  In contrast, the body posture, social context, and room location categories are mutually exclusive; only one instance from each of these categories applies to the occupant at a given time. Labels from these categories therefore do not need associated start-stop labels. Each instance usually appears in a separate annotation row, but they may be combined if more than one become applicable at the same time; in these cases, the start-stop labels apply only to the activity category and not to the mutually exclusive categories.

Each hour of the data was manually annotated using an ontology of 89 activity, body posture, and room/context codes that were developed for PlaceLab data. Definitions for the activity description ontology are provided in a Microsoft Word document at [root]/annotationData/AD03_ontology.doc.  These definitions describe visual cues the observer should note in determining the appropriate activity label. Of particular note are the “background” activity labels, which describe activities that may not be receiving active attention from the participant at any given time, yet that describe the major activities in which they are participating.  For example, TV watching is in the background as long as the television is on, even when the participant is not actively watching.  The equivalent non-background activity may co-occur with the background activity when the participant actively attends to the task. For example, “watching TV background” begins when the participant turns on the TV, “watching TV,” the non-background label, can describe sections of time within the longer background time period when the participant is actively watching:

start…. watching TV background                     [participant turns on TV]

start… watching TV                                         [participant sitting in front of TV, watching]

stop… watching TV                                         [participant goes to other room]

stop… watching TV background                      [participant turns off TV]

Each hour should begin with start labels for all foreground and background activities and labels for body posture, social context, and room location. The end of the annotated hour should include end labels for all activities in progress.

For the sample dataset, an additional annotation task was completed using the SV01.cod (sensor verification, version 1) code file. The annotator was instructed to label expected start-stop data signals from the static MITes based on observed movement of objects in the PlaceLab environment. For example, if the refrigerator door was opened, the annotator marked the start and stop of the open swing and selects the label that corresponds to the sensor location. These annotations are used to validate the data collected from the sensors. The .pdv and .cod files for this task are included in the same folder (e.g., [root]/annotationData/SV01.cod)

Example .pdv file for sensor verification:

EVENTS
Researcher
3-04-05
9:00 am
PlaceLab
PLIA004
EO
< notes>

< /notes>
..\videoData\04Mar2005\11.avi
.\SV01.cod

55
50

"00:00:22.72","start","bedroom -
bed",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:22.72","start","bedroom - bed - left
pillow","indirect",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:22.72","start","bedroom - bed - right
pillow","indirect",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:23.28","stop","bedroom -
bed",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:23.28","stop","bedroom - bed - left
pillow",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:23.28","stop","bedroom - bed - right
pillow",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:30.17","start","bedroom -
bed",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:00:41.45","stop","bedroom -
bed",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:20.94","start","kitchen - coffee
machine",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:25.94","stop","kitchen - coffee
machine",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:34.98","start","kitchen - coffee
machine",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:36.00","stop","kitchen - coffee
machine",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:42.33","start","kitchen - coffee
machine",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"00:03:48.10","stop","kitchen - coffee

The format of this activity file is similar to the one presented above. The main difference is the increased detail with respect to specific objects being moved in lieu of focusing on specific activities. The “indirect” column denotes that an object was moved by indirect means. In the example above, a sitting on the bed near a pillow but not on the pillow will indirectly move the pillow.

Trained (undergraduate) annotators required about 2 hours of effort for each hour of video annotated using ProCoderDV.  In order to determine inter-rater reliability, we calculated Cohen's Kappa for an hour independently annotated by two raters. Five out of 12 of the body postures and four out of 12 of the activities labeled for the hour had satisfactory reliability scores (Kappa > 0.7). Body postures with extended state (e.g., standing, sitting, walking) received higher agreement between annotators than fast-action body postures (e.g., sitting down action, turning/pivoting). Activities that involved single appliances (e.g., ironing, vacuuming, using telephone) received the highest agreement between annotators. Based on these results, we are revising our annotation training procedures and ontology. It is important to note that the hour used to produce the above inter-reliability scores was not one of the hours included in this sample dataset. Inter-reliability scores for the provided dataset will eventually be calculated and added to this website.