Page MenuHomeFeedback Tracker

Json File Length Limitation
Reviewed, UrgentPublic

Description

I recently ran into the issue that some data stored about players on my server was lost. I saved an array to json and after the restart I'm loading it back in to continue using it. But the Problem is that the Json Output is Limited to 2^16 characters (notepad says 65534 bytes, but it's close). Why do you have this strange limit ? Please change it so we can store more information formatted in json in our files without loosing all the data, because the result is no longer valid json that DayZ is unable to load

Details

Severity
Tweak
Resolution
Open
Reproducibility
Always
Operating System
Windows 7
Category
General
Steps To Reproduce
modded class MissionServer {
	
	void MissionServer() {
		TIntArray testArray = new TIntArray();
		for (int i = 0; i < 100000; i++) {
			testArray.Insert(i);
		}
		FileSerializer fs = new FileSerializer();
		if (fs.Open("$profile:test.bin", FileMode.WRITE))
		{
			Print("Opened File to write");
			fs.Write(testArray);
			fs.Close();
		} else {
			Print("Error Writing File");
		}
	}
}

As you will be able to see the: Json is cut of at index 6663 of the array

Related Objects

Event Timeline

LBmaster created this task.Jan 7 2020, 9:46 PM
LBmaster added a comment.EditedJan 7 2020, 9:48 PM

And does the FileSerializer has the same limitations ? Otherwise I would run into the same issues soon. Looks like the file the FileSerializer creates has an even lower limit set to 32768 (2^15)

LBmaster edited Steps To Reproduce. (Show Details)Jan 7 2020, 10:25 PM
LBmaster edited Steps To Reproduce. (Show Details)
Geez changed the task status from New to Reviewed.Jan 9 2020, 2:38 PM
Geez added a subscriber: Geez.

Hello LBmaster.
There is no limit in JsonSerializer. However, there is a character limit when using the debug Print() feature.
Regards,
Geez

LBmaster added a comment.EditedJan 9 2020, 7:13 PM

Thanks. Would have been a very strange limit just cutting off the Json output. Any ETA when it will be resolved ? And just one thing to note: the FileSerializer does not seem to have a limit too, if you are still looking into that. It was just a coincidene, that the size of my testfile was around 32768 bytes.

I believe he was meaning the JsonFileLoader, which DOES have a limitation on the JsonSaveFile function of 64kb. However the FileSerializer does not seem to have this limitation, which is correct (so far in my tests), it just creates a file that cannot be edited by admins/server owners which is less than ideal.

Thats correct. I thought I wrote JsonFileLoader somewhere, but apparently not ^^. But that was the thing I meant

Geez added a comment.Jan 23 2020, 4:05 PM

Hello everyone.
If you use the following code, it should show the full information. This is only a workaround until the issue gets fixed.

void test2()
	{
		TIntArray testArray = new TIntArray();
		for (int i = 0; i < 100000; i++) {
			testArray.Insert(i);
		}
		string textOut;
		JsonSerializer js = new JsonSerializer();
		bool ok = js.WriteToString(testArray, true, textOut);	
		Print(textOut);
		FileSerializer fs = new FileSerializer();
		if (fs.Open("d:\\test.txt", FileMode.WRITE))
		{
			Print("Opened File to write");
			fs.Write(textOut);
			fs.Close();
		} else {
			Print("Error Writing File");
		}
	}

Something I just discovered ... JsonSaveFile is creating your JSON file but it is also converting all tabs therein to spaces (1 to 4). As almost every line in your JSON file starts with at least one tab the additional character count can add up rather quickly (e.g. 1000 lines starting with a single tab is 1000 tabs that are converted into 4000 spaces). This means if you read a file IN and then save it back OUT the file size actually increases proportionally by the number of tabs replaced. If your original JSON file (with tabs) is riding just under the 64 kb limit and/or has a ton of nested tab levels ... then your file when saved back out could exceed this 64kb limit and you end up with a truncated file :(

I discovered this issue when one of my Airdrop-Upgraded subscribers reported parsing errors and server crashes after digging into the issue and examining his before & after files (my mod examines the file after it is loaded, makes any necessary corrections and/or updates, and then saves the updated data back to the original file.