Loading Json file is slow + fix
Feedback, NormalPublic
Actions

Description

I noticed trying to load bigger Json files (100kb+) will result in a huge lag spike. Trying to load files bigger than 1mb can take several minutes
I looked at the JsonFileLoader class ant noticed a huge performance issue in loading files (saving is fine)

The problem ist, that each line is read through FGet and concatenated to the output string, which is a problem for two reasons:

A lot of small IO operations, which are unnecessarry, because the whole file needs to be loaded anyways
A lot of string concatenations, which are especially expensive, because of c++ internally copying the whole string each time, stressing the CPU and Ram

Details

Severity: Major
Resolution: Open
Reproducibility: Always
Operating System: Windows 11 x64
Category: General

Steps To Reproduce

Here is how to fix:

replace the

string fileContent;
string lineContent;
while (FGets(handle, lineContent) >= 0)
{
	fileContent += lineContent;
}

with

string fileContent;
ReadFile(handle, fileContent, 2000000000);

Another suggestion, because this only works with Json files, because there are no nullterminators in the file:
Add a function, which can just read a whole file into memory / into a string and not only until it finds a nullterminator

Just a small comparison between the two versions:
A 6mb file takes around 5 minutes to load with the old version. With the new version is was 6ms! Please fix it, it's a super simple fix

Event Timeline

LBmaster created this task.Mar 12 2024, 8:39 AM

Geez changed the task status from New to Feedback.Mar 12 2024, 10:49 AM

Loading Json file is slow + fixFeedback, NormalPublicActions

Description

Details

Event Timeline

Loading Json file is slow + fix
Feedback, NormalPublic
Actions