Filtering Large Datasets

What is it?

Here are a series of examples to show you how to work with large sets of data.

Why should I learn it?

These examples will show you how you filter large datasets with user interaction.

What can I do with it?

Feel free to modify and customize any of these scripts for your own purposes.

How long will it take?

You already know all the JavaScript you need to be able to apply these concepts. These examples will allow you to apply them to larger datasets.

Getting Started

This series of exercises and examples will allow you to see how to apply interactive techniques when working with large datasets. This will allow you to read data and provide ways for users to explore it.

The data used in these exercises was found on the City of Austin Data Portal, specifically the Austin Fire Department's Fire Incidents from the first half of 2016. This dataset includes more than 1400 incidents. The data itself can be found in the file data.json. Download it and place it in the folder where you save the following examples. The data itself started as the spreadsheet from the data portal. Useful columns for incident number, month, battalion, status, priority, unit, district, latitude and longitude were selected. A CSV to JSON converter was used to get the file into the appropriate format and line breaks were removed using the "Remove Line Breaks" feature in Text Wrangler. If your data includes apostrophes or quotes in a text item, those lines must be escaped in order for them to be read correctly by the script.

\' or \"

Note the use of the script tag to link to the dataset and the line to parse the JSON for use in the script.

<script src="data.json"></script>

data = JSON.parse(data);

Filter with a Dropdown

This example allows the user to choose a selection from a dropdown and see the incidents associated with that particular month. Note that in all these examples, the script begins by clearing out the DOM element on each form action to prevent continued appending of the results.

Filter with a Dropdown

Filter with Text Input

This example shows how to create a text search within a column of data. The script searches for a text input and matches it with a substring in the "Unit" column, starting at the beginning of the string.

Filter with Text Input

Filter Within String

This example looks to match a text input within a string. This example uses the JavaScript includes() method to identify whether a string is included in an item.

Filter Within String

Filter With Map

Now that you can filter the results, you may need to use them in a chart or map. The AFD Fire Incidents data includes latitude and longitude for the addresses, and you might want to show these on a map based on the selected month.

Recall that the data for a Google Chart must be in an Array Table rather than JSON format. This can also be accomplished by using theCSV to JSON converter. The data in this format can be found in the file data2.json

Filter with Map

Now You Try

Remember the JSON data from yesterday's exercise about housing data? Here is a more extensive version with most of Austin's zip codes, transposed so that the first key is zipcode (as opposed to measure). I have also included the column that includes the percentage below poverty in each zipcode.

  1. Have the entire json section print to a table.
  2. Now add a dropdown so that you can also have selections for below 10% poverty level and 10% or more poverty level. You should have three items in your dropdown, one for all, one for below and one for above. What logic do you have to add to get the script to evaluate the poverty level?
  3. Add text input field in a second form that lets the user input her own zip to see the results.

This is a very challenging exercise. Focus on each step and then think about the logic necessary to get the desired result. There may be multiple ways to solve these problems.

Filter Data Exercise

Moving On

All these exercises demonstrate practical use of the features we have discussed, this time working with a large dataset. Apply these techniques to manipulate your own datasets. 

© 2016 CodeActually ·