A Brief History of Big Data (in Federal Transportation)

A Brief History of Big Data (in Federal Transportation)

August 03, 2016  | Thomas Grogan

Guest op-ed by Thomas Grogan, Senior Economist, HDR, Inc.

Big Data is big.

Big enough to generate 3.5 billion search results each day.

Big enough to process over 25 million transit trips daily, to find patterns in ridership to improve safety, service and reduce costs.

Big enough to potentially eliminate six billion metric tons of GHG pollution by reducing inefficient vehicle operations and travel demand.

Big Data can seem complicated because it means many things to many people. But it is not new. At its core, Big Data is still the collection, analysis, and communication of information. What has changed over time is the size, speed, and variety of data which is collected and communicated. However, the challenge lies in using innovative methods to analyze the data and gather meaningful insights which can be used to inform decisions makers.

The Federal Government has been collecting data in increasing size and scale since the first U.S. decennial census in 1790. The methods were rudimentary by today’s standards: the law required that every household be visited; that completed census schedules be posted in public places; and that “the aggregate amount of each description of persons” for every district be transmitted to the president.

Even at a time of great uncertainty in our nation’s future, the government understood the value of quantitatively capturing a moment in time in the country. The total cost was $44,000, or approximately 0.55% of Federal spending at the time (assuming $8 million in nominal terms, based on various estimates). As the nation grew, population data alone would not be enough to provide decision makers with the tools to solve the problem faced by citizens. The next step would be to collect data for how the population moved.

The first mail-out census in 1960 was also the first time that transportation information was formally collected in the decennial census. Under the “Employment Status and Work Experience” section, the multiple choice question was asked “How did this person get to work last week?” This data began to inform transportation planning products used by the Federal Highway Administration. The concentrated effort by the Federal government to collect transportation data that began as one observational data series 56 years ago has grown exponentially in scope, type, and volume of data, as a result of technological progress. And this collected information influences programmatic spending, determines compliance with laws, and guides private sector investment and operational decisions.

In June 2016 alone, the Department of Transportation updated or made available 9 data sets, ranging from numerical to geographic across different modes and types (employment, freight, traffic, performance, maps). Much of this information can be collected and reported automatically by devices; it does not require mailing paper surveys or interviewing people in person. Unlike the data from the first Census, which was only accessible at a single geographical location in a printed document, these data sets are accessible by anyone in the country with a strong internet connection and the right tools and skills to sort through them.

In many cases, real-time information from transit agencies is made developer-friendly in the form of APIs (application program interfaces). When you go online or to an app to track a bus route or order a car service, you’re accessing Big Data, right at your fingertips. A current challenge for agencies and organizations is processing this data on the back-end to gain the appropriate insights for long run transportation policies.

Today, the Census collects much more than the original question asked, and the government established programs with dedicated resources just for transportation data. DOT inherited decades-old data collection programs on the railroad and aviation sectors when they absorbed the functions of the Interstate Commerce Commission and the Civil Aeronautics Board via deregulation. Some might argue that DOT established its own “Big Data” program with the creation of the Bureau of Transportation Statistics (BTS) under ISTEA in 1991. Soon after, the advances in IT technology helped expand the Department’s data and research programs at a rapidly increasing pace.

With the creation of the Research and Innovative Technology Administration (RITA) through the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU), BTS was moved to this administration in 2005. Currently, the Office of the Assistant Secretary for Research and Technology (OST-R) oversees 8 multi-modal research programs, including BTS.

Other organizational developments that are occurring across all agencies include the growth in Chief Innovation and Chief Data Officers (CDO); US DOT appointed its first CDO in July 2014. Perhaps as important as the number of programs are the strategic initiatives the OST-R is spearheading.

The extremely popular “Smart Cities Challenge” by DOT was primarily championed by the OST-R. Making the knowledge and lessons learned of the “Smart Cities Challenge” available to everyone for replication is an important component of this program. This makes the implications for structured data programs and analysis all the more important as the transportation industry embraces Big Data.

This increase in Big Data isn’t limited to just transportation. The future of urban areas depends on accessing Big Data from a wide range of industries including energy, education, health, employment, environment, and housing, as well as transportation. For these “Digital Cities” to become a sustainable reality, we must learn to analyze these different sources of data in conjunction with each other.

The views and opinions expressed in this article are those of Thomas Grogan and do not express those of HDR Inc. or of the Eno Center for Transportation.

Share

Related Articles

What We Can Learn from Transportation Tech Companies About Empowering Emerging Leaders

What We Can Learn from Transportation Tech Companies About Empowering Emerging Leaders

Transportation technology companies like Uber or Bird have been in the spotlight as investors push to define a path to profitability. While...

Webinar: Transportation and Cybersecurity: Next Steps to Secure Our Future

Webinar: Transportation and Cybersecurity: Next Steps to Secure Our Future

Successful cyber attacks on our transportation systems have a debilitating effect on the economy and public safety. Systems including oil...

Mandatory Cybersecurity Incident Reporting, Better Protection Measures for Critical Infrastructures Pushed at Homeland Security Committee Hearing

Mandatory Cybersecurity Incident Reporting, Better Protection Measures for Critical Infrastructures Pushed at Homeland Security Committee Hearing

This past Tuesday, the House Homeland Security Committee convened a hearing to discuss cybersecurity mandates on nationally significant...

Webinar: Leveraging Transit Data: Best Practices for Data Management and Sharing

Webinar: Leveraging Transit Data: Best Practices for Data Management and Sharing

Transit agencies collect a wealth of data on their systems and how riders use and experience them. As transit ridership begins to recover...

Guest Op-Ed: Bus Lanes in the Car Capital: Lessons and Challenges from Los Angeles

Guest Op-Ed: Bus Lanes in the Car Capital: Lessons and Challenges from Los Angeles

This month, the Eno Center for Transportation released a new report titled “A Budding Model: Los Angeles’s Flower Street Bus Lane,”...

Eno Releases New Mobility on Demand Reports with Data Insights and Recommendations

Eno Releases New Mobility on Demand Reports with Data Insights and Recommendations

Even before the COVID-19 pandemic, public transit in the United States was going through a very dynamic period. The FTA's MOD Sandbox...

Webinar: Introducing Platform-Based Approaches to Public Transit

Webinar: Introducing Platform-Based Approaches to Public Transit

Platform technology is changing the way we consume products and services in many aspects of our lives. Netflix and Spotify have changed the...

Census Releases Full 2019 ACS Data, Including Commute Patterns

Census Releases Full 2019 ACS Data, Including Commute Patterns

The Census Bureau this week released the full dataset for the 2019 American Community Survey, including 1-year and 5-year aggregate data...

MOD Fare Integration for Transit: A Case Study in the Los Angeles and Puget Sound Regions

MOD Fare Integration for Transit: A Case Study in the Los Angeles and Puget Sound Regions

MOD Fare Integration for Transit is part of our research report series examining the FTA Sandbox Program in the Los Angeles and Puget Sound...

Webinar: MOD Fare Integration for Transit

Webinar: MOD Fare Integration for Transit

This webinar is an overview of MOD Fare Integration for Transit: A Case Study in the Los Angeles and Puget Sound Regions, which is the...

Road to Recovery Webinar: Rebalancing the Load: How Road User Charging Can Influence Traveler Behavior

Road to Recovery Webinar: Rebalancing the Load: How Road User Charging Can Influence Traveler Behavior

Congestion pricing can be a valuable tool for regions to address congestion and meet goals for sustainability and equity. Intelligent...

Webinar: Automated Vehicle Technology, Public Policy, and BMW's Level 3 AV System

Webinar: Automated Vehicle Technology, Public Policy, and BMW's Level 3 AV System

While much of the transportation-related news has been focused on COVID-19 recovery, automated vehicle technologies are quietly progressing...

Be Part of the Conversation
Sign up to receive news, events, publications, and course notifications.
No thanks