I recently ran into the issue that some data stored about players on my server was lost. I saved an array to json and after the restart I'm loading it back in to continue using it. But the Problem is that the Json Output is Limited to 2^16 characters (notepad says 65534 bytes, but it's close). Why do you have this strange limit ? Please change it so we can store more information formatted in json in our files without loosing all the data, because the result is no longer valid json that DayZ is unable to load
Description
Description
Details
Details
- Severity
- Tweak
- Resolution
- Open
- Reproducibility
- Always
- Operating System
- Windows 7
- Category
- General
Steps To Reproduce
modded class MissionServer {
void MissionServer() {
TIntArray testArray = new TIntArray();
for (int i = 0; i < 100000; i++) {
testArray.Insert(i);
}
FileSerializer fs = new FileSerializer();
if (fs.Open("$profile:test.bin", FileMode.WRITE))
{
Print("Opened File to write");
fs.Write(testArray);
fs.Close();
} else {
Print("Error Writing File");
}
}
}As you will be able to see the: Json is cut of at index 6663 of the array
Related Objects
Related Objects
- Mentioned In
- T139337: EXP: 1.03 JSON File size
Event Timeline
Comment Actions
And does the FileSerializer has the same limitations ? Otherwise I would run into the same issues soon. Looks like the file the FileSerializer creates has an even lower limit set to 32768 (2^15)
Comment Actions
Hello LBmaster.
There is no limit in JsonSerializer. However, there is a character limit when using the debug Print() feature.
Regards,
Geez
Comment Actions
Thanks. Would have been a very strange limit just cutting off the Json output. Any ETA when it will be resolved ? And just one thing to note: the FileSerializer does not seem to have a limit too, if you are still looking into that. It was just a coincidene, that the size of my testfile was around 32768 bytes.
Comment Actions
I believe he was meaning the JsonFileLoader, which DOES have a limitation on the JsonSaveFile function of 64kb. However the FileSerializer does not seem to have this limitation, which is correct (so far in my tests), it just creates a file that cannot be edited by admins/server owners which is less than ideal.
Comment Actions
Thats correct. I thought I wrote JsonFileLoader somewhere, but apparently not ^^. But that was the thing I meant
Comment Actions
Hello everyone.
If you use the following code, it should show the full information. This is only a workaround until the issue gets fixed.
void test2()
{
TIntArray testArray = new TIntArray();
for (int i = 0; i < 100000; i++) {
testArray.Insert(i);
}
string textOut;
JsonSerializer js = new JsonSerializer();
bool ok = js.WriteToString(testArray, true, textOut);
Print(textOut);
FileSerializer fs = new FileSerializer();
if (fs.Open("d:\\test.txt", FileMode.WRITE))
{
Print("Opened File to write");
fs.Write(textOut);
fs.Close();
} else {
Print("Error Writing File");
}
}Comment Actions
Something I just discovered ... JsonSaveFile is creating your JSON file but it is also converting all tabs therein to spaces (1 to 4). As almost every line in your JSON file starts with at least one tab the additional character count can add up rather quickly (e.g. 1000 lines starting with a single tab is 1000 tabs that are converted into 4000 spaces). This means if you read a file IN and then save it back OUT the file size actually increases proportionally by the number of tabs replaced. If your original JSON file (with tabs) is riding just under the 64 kb limit and/or has a ton of nested tab levels ... then your file when saved back out could exceed this 64kb limit and you end up with a truncated file :(
I discovered this issue when one of my Airdrop-Upgraded subscribers reported parsing errors and server crashes after digging into the issue and examining his before & after files (my mod examines the file after it is loaded, makes any necessary corrections and/or updates, and then saves the updated data back to the original file.
Comment Actions
Since nobody seems to care about it, I made my own Json Saver / Reader class. Feel free to use it in your Projects. I've successfully saved the TestArray from above which was a 1064kb file. This is how you use it to save files:
EDIT: DON'T use it ^^ you will see below why
TIntArray testArray = new TIntArray();
for (int i = 0; i < 100000; i++) {
testArray.Insert(i);
}
Print("Saving Json ...");
JsonSaver<TIntArray>.WriteToFile("$profile:/testArray.json", testArray);This is the whole class. Put it in 3_Game
class JsonSaver<Class T> {
static ref JsonSerializer js = new JsonSerializer();
static bool WriteToFile(string file, T data) {
string jsonOut = "";
bool ok = js.WriteToString(data, true, jsonOut);
if (!ok)
return false;
FileHandle handle = OpenFile(file, FileMode.WRITE);
if (handle == 0)
return false;
for (int i = 0; i < jsonOut.Length();) {
int end = Math.Min(i + 60000, jsonOut.Length());
int length = end - i;
string sub = jsonOut.Substring(i, length);
FPrint(handle, sub);
i += 60000;
}
CloseFile(handle);
return true;
}
static bool ReadFromFile(string file, out T data) {
if (!FileExist(file))
return false;
string content,line,error;
FileHandle handle = OpenFile(file, FileMode.READ);
if (handle == 0)
return false;
while (FGets(handle, line) >= 0) {
content += line;
}
CloseFile(handle);
if(!js.ReadFromString(data, content, error))
Error(error);
return true;
}
}Comment Actions
Okay this is getting even more "interesting"
I've noticed, that Doing this:
modded class MissionServer {
void MissionServer() {
TIntArray testArray = new TIntArray();
for (int i = 0; i < 100000; i++) {
testArray.Insert(i);
}
Print("Saving Json ...");
JsonSaver<TIntArray>.WriteToFile("$profile:/testArray.json", testArray);
}
}would cause the server to freeze at some random position in the code (for me while loading Trader Objects) removing the Code will make it run fine again. If I don't put 100.000 items in the list, but 1.000.000, the server will crash completly instead of just freezing. DK, what's wrong there, but this definitely needs some attention. I've also checked if this occours while saving the json file, but that's not the case. The file is saved correctly and all my Prints were executed. When setting it to 10.000, the Server will start, but then eventualy crash. It's random every time.
This was one run:
SCRIPT : Opening File for writing Json SCRIPT : Opened File Handle. SCRIPT : Starting to write File... SCRIPT : Writing Chars: 0 to 60000 -> 60000 SCRIPT : Writing Chars: 60000 to 98892 -> 38892 SCRIPT : Closed json File SCRIPT : Group System Loading Configs...
crash
This was another run:
SCRIPT : Opening File for writing Json SCRIPT : Opened File Handle. SCRIPT : Starting to write File... SCRIPT : Writing Chars: 0 to 60000 -> 60000
crash
and another one:
SCRIPT : Opening File for writing Json SCRIPT : Opened File Handle. SCRIPT : Starting to write File... SCRIPT : Writing Chars: 0 to 60000 -> 60000 SCRIPT : Writing Chars: 60000 to 98892 -> 38892 SCRIPT : Closed json File SCRIPT : [1:21] ERROR: Missing ';' at the end of the line SCRIPT : string str = '[TRADER] DEBUG START' SCRIPT : string str = '[TRADER] READING OBJECT TYPE ENTRY..' SCRIPT : string str = '[TRADER] READING OBJECT POSITION ENTRY..' SCRIPT : AnimSoundObjectBuilderBank: Invalid sound set "generalgrunt_SoundVoice_Char_SoundSet". SCRIPT : AnimSoundObjectBuilderBank: Invalid sound set "greathelmet_generalgrunt_SoundVoice_Char_SoundSet". SCRIPT : AnimSoundObjectBuilderBank: Invalid sound set "Gasmask_generalgrunt_SoundVoice_Char_SoundSet". SCRIPT : AnimSoundObjectBuilderBank: Invalid sound set "Motohelmet_generalgrunt_SoundVoice_Char_SoundSet". SCRIPT : AnimSoundObjectBuilderBank: Invalid sound set "Gag_generalgrunt_SoundVoice_Char_SoundSet". SCRIPT : string str = '[TRADER] SPAWNED OBJECT 'SurvivorF_Irena' AT '<3699.270020, 402.130005, 5967.910156>'' SCRIPT : string str = '[TRADER] Object was a Man..' SCRIPT : string str = '[TRADER] READING OBJECT ORIENTATION ENTRY..' SCRIPT : string str = '[TRADER] OBJECT ORIENTATION = '<-13.234265, 0.000000, 0.000000>'' SCRIPT : string str = '[TRADER] READING OBJECT ATTACHMENT ENTRY..' SCRIPT : On Inventory Enter: BoxCerealCrunchin Player: SurvivorBase<c2ae0020> false SCRIPT : string str = '[TRADER] 'BoxCerealCrunchin' WAS ATTACHED' SCRIPT : string str = '[TRADER] READING OBJECT ATTACHMENT ENTRY..'
nothing changed. There must be some memory beeing overwritten when converting the Object into json. When staying under the limit of 60000 characters, everything seems to be fine
Comment Actions
While doing some more tests, I've noticed, that also with a json String length < 60.000 I would get server crashes. It seems to have to do something with the string.Substring() method.
static bool WriteToFile(string file, T data) {
JsonSerializer js = new JsonSerializer();
string jsonOut;
bool ok = js.WriteToString(data, true, jsonOut);
Print("Length: " + jsonOut.Length());
Print("Json: " + jsonOut);
if (!ok)
return false;
Print("Opening File for writing Json");
FileHandle handle = OpenFile(file, FileMode.WRITE);
Print("Opened File Handle.");
if (handle == 0)
return false;
Print("Starting to write File...");
for (int i = 0; i < jsonOut.Length();) {
int end = Math.Min(i + 60000, jsonOut.Length());
int length = end - i;
Print("Writing Chars: " + i + " to " + end + " -> " + length);
string sub = jsonOut.Substring(i, length);
Print("Sub finished");
Print("Sub: " + sub.Length());
FPrint(handle, sub);
Print("FPrint Worked");
i += 60000;
}
CloseFile(handle);
Print("Closed json File");
return true;
}and this was the output sometimes (sometimes it crashed later in the program)
SCRIPT : Opening File for writing Json SCRIPT : Opened File Handle. SCRIPT : Starting to write File... SCRIPT : Writing Chars: 0 to 58892 -> 58892