What is the best way to save large amount of data persistently on the device?

Dear all,
I’m developing a wallet of identity card photos with thunkable and I would kindly ask you an help to find the best way to save data persistently on the device ( consider at least 20 Mbyte of base64 encoded photos ).

Since the app manages ID card photos (ex. Driver license, health insurance card, etc.) I don’t want to save them on the cloud (no airtable or gdrive is possible here) because of the privacy, so the app storage is my only possible choice.

What I did since now is to save a thunkable list of photos (encoded in base64) into a stored variable, this works fine but has a strict limit of 6Mbyte on Android, indeed if you exceed this limit the operating system throws the following exception:

10-14 12:43:29.267 13766 14508 W unknown:ReactNative: android.database.sqlite.SQLiteFullException: database or disk is full (code 13 SQLITE_FULL[13])

I also considered and implemented resizing and compression of photos but I still need at least 20Mbyte or so in order to save at least 20 photos with an acceptable quality.

I hope that my explanation was clear and that someone can help with this problem.

In the meantime I will try to experiment alternative solutions and I will post here if I can find something interesting.

Have a nice day and thank you all in advance for your help.

UPDATE 22.10.24 After a lot of testing on different devices (see the list below) I can say that on Android a stored variable can successfully manage a file (in this case a photo in base 64) with a maximum size of 999 Kbyte.

Vendor Model
Samsung S10
Samsung S7
Samsung A21
Huawei P20 Pro
Huawei P30 Pro
Xiaomi Redmi 9
Xiaomi Poco M2 PRO
Google Pixel 6
Google Pixel 7
One Plus 7T
Oppo A15
Vivo Y50
1 Like

If you know in advance you need 20Mb then you could just user 4 stored variables instead of just one.

1 Like

Thank you, this is good! Do you know if there are also other existing data structure like local DB could help in this case?

That kind of structure is a bit more easier to manage in iterations or cycles.

Anyway, using multiple stored variables could be a very effective solution.

1 Like

If I would attempt something like this I would use a local table to store the image name/title and other info and to also store the variable name used for the stored variable. This way I can use as many variable as I want.

I will use the dynamic variables
image

2 Likes

This is a great solution!

Thank you so much, I didn’t know about dinamically named variables.

Have nice day.

1 Like

Dear @muneer and Community,
do you think there can be another possible way to solve the problem?

After a lot of tests, I found that your solution works very well on iOS (indeed, you can persistently store hundreds of Mbyte on the device) but not in Android where the limit is 1 Mb* (see below).

Indeed, if you exceed 1Mb of data the SQLiteFullException still raises, it does not matter if you use one or multiple stored variables:

10-28 08:37:41.964  3271 16625 E SQLiteLog: (13) statement aborts at 18: [INSERT OR REPLACE INTO catalystLocalStorage VALUES (?, ?);] database or disk is full
10-28 08:37:41.964  3271 16625 E SQLiteLog: (1) Force to rollback the active transaction caused by the special error (13), during 'INSERT OR REPLACE INTO catalystLocalStorage VALUES (?, ?);'
10-28 08:37:41.968  3271 16625 W unknown:ReactNative: database or disk is full (code 13 SQLITE_FULL[13])
10-28 08:37:41.968  3271 16625 W unknown:ReactNative: android.database.sqlite.SQLiteFullException: database or disk is full (code 13 SQLITE_FULL[13])

*Notice that 1 Mbyte is the limit that works well on all the Android devices I tested until now and listed below.

Vendor Model
Samsung S10
Samsung S7
Samsung A21
Huawei P20 Pro
Huawei P30 Pro
Xiaomi Redmi 9
Xiaomi Poco M2 PRO
Google Pixel 6
Google Pixel 7
One Plus 7T
Oppo A15
Vivo Y50

Many thanks in advance for your help.
D

1 Like

That is odd. The default size is 6MB but that default can be changed by setting AsyncStorage_db_size_in_MB = 6 in Gradle.properties file. Because we do not have access to this file it is not an option here.

I will do some testing and come back to you on this. Just to see if it is a limitation of Thunkable or something to do with Android.

2 Likes

Thank you @muneer.

Actually, 1 Mbyte is the maximum limit that I’m sure is working on all the devices that I mentioned before, but I also found substantial differences between vendors, for example, on Samsung S10 I was able to manage at least 3Mbyte but, with Huwaei P30, the limit was 1 Mbyte (the same was for Google Pixel and others).

I also prepared a demo app for this test where you can digit how many dynamic variables you want to create, then you click the button “start” and the app initializes each of them with two base 64 of roughly 1Mbyte downloaded from an API.

You can see the app here.

As I said, on iPhone, it looks like there is no limit on how many variables you can create and initializes with the demo app (I was able to store more then 200Mbyte of data on the device), but, on Android, the limit was 1 or 2, depending on the specific vendor and model.

Thank you for your help.
regards,
D