# Efficient way to store and search through large data set?

**URL:** <https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250>\
**Category:** Questions about Thunkable X\
**Created:** [September 10, 2020, 12:40am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250 "2020-09-10T00:40:58Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![secondaryaccount](https://sea1.discourse-cdn.com/flex015/user_avatar/community.thunkable.com/secondaryaccount/32/91441_2.png) [@secondaryaccount](https://community.thunkable.com/u/secondaryaccount)\
**Post date:** [September 12, 2020, 5:56am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/21 "2020-09-12T05:56:35Z")

</div>

This is what I realized tonight. I don’t need to get the list frequently. I can get the data once and parse it into a Google spreadsheet or even a list in the app. Or even sort it. And then each time I want to search for the closest sensor, I can just quickly look through the parsed data without getting all 19,000 records every time the screen refreshes.

---

<div class="post-metadata">

**Author:** ![tatiang](https://sea1.discourse-cdn.com/flex015/user_avatar/community.thunkable.com/tatiang/32/55482_2.png) [@tatiang](https://community.thunkable.com/u/tatiang)\
**Post date:** [September 12, 2020, 6:33am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/22 "2020-09-12T06:33:31Z")

</div>

This is kinda cool but it doesn’t work with data sets as large as mine:

> **[How to import JSON data to a Google Sheet - Databox Help Desk](https://help.databox.com/article/393-how-to-import-json-data-to-a-google-sheet)**
>
> The Google Sheets integration makes it easy for you to visualize data you have stored in Google Sheets.  It has quickly become one of Databox's most popular int

---

<div class="post-metadata">

**Author:** ![actech](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@actech](https://community.thunkable.com/u/actech)\
**Post date:** [September 12, 2020, 7:16am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/23 "2020-09-12T07:16:06Z")

</div>

The problem is that blocks don’t work with this amount of data and you can’t even save it to the stored variable to speed up reuse.

---

<div class="post-metadata">

**Author:** ![tatiang](https://sea1.discourse-cdn.com/flex015/user_avatar/community.thunkable.com/tatiang/32/55482_2.png) [@tatiang](https://community.thunkable.com/u/tatiang)\
**Post date:** [September 13, 2020, 3:38am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/24 "2020-09-13T03:38:14Z")

</div>

I think I’m in business! I tried a bunch of JSON to CSV converters online and most can’t handle large data sets. But this one can:

[https://www.convertcsv.com/json-to-csv.htm](https://www.convertcsv.com/json-to-csv.htm)

I entered the API url for the entire sensor data set and downloaded the converter csv file. I opened that csv file in Google Sheets and it’s all there! 20,000 rows. Phew.

---

<div class="post-metadata">

**Author:** ![tatiang](https://sea1.discourse-cdn.com/flex015/user_avatar/community.thunkable.com/tatiang/32/55482_2.png) [@tatiang](https://community.thunkable.com/u/tatiang)\
**Post date:** [September 19, 2020, 3:11am UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/25 "2020-09-19T03:11:12Z")

</div>

@actech The new PurpleAir API ([https://api.purpleair.com](https://api.purpleair.com)) has a lat/lon bounding box as part of the API call. And you can retrieve selected fields only rather than the entire dataset. This is gonna make my location search SO much faster. Woo hoo!

---

<div class="post-metadata">

**Author:** ![ioannis](https://sea1.discourse-cdn.com/flex015/user_avatar/community.thunkable.com/ioannis/32/146956_2.png) [@ioannis](https://community.thunkable.com/u/ioannis)\
**Post date:** [November 8, 2024, 12:23pm UTC](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250/26 "2024-11-08T12:23:54Z")

</div>

Closed on November 8, 2024, 12:23pm UTC



[Previous page](https://community.thunkable.com/t/efficient-way-to-store-and-search-through-large-data-set/805250.md?page=1)
