Page MenuHomeFeedback Tracker

Huge overhead for array type data.
Reviewed, WishlistPublic

Description

Arrays have a huge impact on network traffic. Transmitting same data as a single field array and as a simple variable results in a huge difference in data sent/recieved. More details in repro section. {F22520}

Details

Legacy ID
276865728
Severity
None
Resolution
Open
Reproducibility
Always
Category
Engine
Steps To Reproduce

We used following tests and #monitor.

someVariable = "Some shitty value";

for "_i" from 0 to 999 step 1 do
{

publicVariable "someVariable";

};

In this case we got following result:

Server load: FPS 50, memory 287 MB, out: 15 Kbps in: 340 Kbps

Which is not really good, but is acceptable.
Now if we transmit same data as an array member:

someVariable = ["Some shitty value"]

We get following:

Server load: FPS 50, memory 287 MB, out: 10 Kbps in: 1730 Kbps

It is 5 times more data, which makes it impossible to use big arrays to sync data between clients.
For example we currently have a shopping array which much more complex than what we used here and it is broadcasted to all players affected. Which results in the following:

Server load: FPS 50, memory 287 MB, out: 30 Kbps in: 7911 Kbps
Server load: FPS 50, memory 287 MB, out: 26 Kbps in: 15674 Kbps
Server load: FPS 50, memory 287 MB, out: 16 Kbps in: 15830 Kbps
Server load: FPS 50, memory 287 MB, out: 24 Kbps in: 15319 Kbps

...
40 times more
...

Server load: FPS 50, memory 287 MB, out: 13 Kbps in: 515 Kbps

This needs to be fixed.

Additional Information

We made some tests on Arma 2 free client (because there is no packed data encryption).
Here is screenshot with comparison of array data and single variable
http://screencast.com/t/ymRdAwoX
First is just an empty array,
2nd is array as in example,
3rd is array with 2 strings
4th is string varriable.

text file from screenshot is attached.

Event Timeline

edxp edited Steps To Reproduce. (Show Details)Oct 13 2013, 3:21 PM
edxp edited Additional Information. (Show Details)
edxp set Category to Engine.
edxp set Reproducibility to Always.
edxp set Severity to None.
edxp set Resolution to Open.
edxp set Legacy ID to 276865728.May 7 2016, 5:08 PM
edxp edited a custom field.

confirmed. this needs to be optimised

I noticed the same thing in A2, arrays are sent as serialized data structures.

I think it's because Arma arrays support different data types for each item in the array. Eg: ["abc", 123, [67,46], player]. And nested arrays compound the problem further.

So every item in the array needs to have it's type defined when sent, even if the array contains all the same types, like an array of integers. Eg: [31,41,45,1,0]

So this would have to be sent as ("pseudo-data"):
[integer:31, integer:41, integer:45, integer:1, integer:0]
instead of:
{integer array: [31,41,45,1,0]}

Your screenshot even indicates the data type names themselves are stored as whole strings. It's almost like unoptimised xml.

Yep. It's more usefull to store data types as identifiers with one byte of data instead of using strings.

The second thing - I think there's something wrong in ARRAY description/header inside of the serialized net packet. One data type descriptor per one element should be.

This thing is critical when your server holds > 50-100 players and you're using BIS_fnc_MP or same system to call remote functions passing arguments to them.

gdscei added a subscriber: gdscei.May 7 2016, 5:08 PM

My proposed solution would be type-specific arrays, similiar to C# for example.

Example:

myStringArray = array("String", ["1", "2", "3"]);
where first param is type, second the actual array. With this solution you would also need to be able to make custom types though, maybe in the config.

Other solution would be to use numbers to determine type instead of full string in the packets. I can't imagine this being any more painful to deal with engine-side... Probably actually easier than strings.

maybe it is time to add parseArray method? http://feedback.arma3.com/view.php?id=16568

This way you can broadcast arrays as strings

_arr = [1, "2", [3]];
var = str _arr;
publicVariable "var";

_arr = parseArray var; //[1, "2", [3]];

it will be just workaround but not fixing and will not fix all kind of cases of using arrays on network. For example client-to-server data packet:

MYPREFIX_packet = [player, cursorTarget, 123, 321, true, false, "some other stuff", ["nested", "array"]];
publicVariableServer "MYPREFIX_packet";

There's a problem while deserializing player and cursorTarget, because if you send to server "player" and "cursorTarget, it'll cause issues after deserializing back via parseArray. player and cursorTarget don't exist on the server side.

The problem should be fixed on the engine level, not on scripts.

ruebe added a subscriber: ruebe.May 7 2016, 5:08 PM
ruebe added a comment.Jul 29 2014, 3:34 PM

There is *a lot* of room left for improvements, to say kindly.
Baffles me that these things get transmitted in plain text? Nothing is compressed at all?

You'd assume this would be the perfect use case for aggressive (static/hybrid) dictionary compression, given server and clients share a fair amount of code anyways.

Seems fixed, could anyone confirm this?

Ok. I've used our algorythm again:

// =====================================
someVariable = "Some shitty value";

for "_i" from 0 to 999 step 1 do
{

publicVariable "someVariable";

};
// =====================================

Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 0 Kbps
Server load: FPS 50, memory 227 MB, out: 1 Kbps in: 189 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 108 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 0 Kbps

It is better than it was in initial ticket.

// =====================================
someVariable = ["Some shitty value"];

for "_i" from 0 to 999 step 1 do
{

publicVariable "someVariable";

};
// =====================================

Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 0 Kbps
Server load: FPS 50, memory 227 MB, out: 1 Kbps in: 304 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 349 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 350 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 366 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 296 Kbps
Server load: FPS 50, memory 227 MB, out: 0 Kbps in: 0 Kbps

The behaviour was changed. It seems the speed (amount of data and/or their size in single packet) of transfering via pV has been changed but actualy we almost have same amount of data for arrays: 304 + 349 + 350 + 366 + 296 = 1 665 kbps.

As you can see the pV data transfer speed was reduced, but amount of the data almost was not changed. Dedicated server was running on the same machine with a client.

PS Was tested on Dedicated Server with simpliest mission possible on Virtual Reality island with a single playable slot available and Dev Console enabled.

arma3.exe

File version: 1.26.126.789
Product version: 1.26.0.126789

arma3server.exe

File version: 1.26.126.789
Product version: 1.26.0.126789

Paratus added a subscriber: Paratus.May 7 2016, 5:08 PM

"maybe it is time to add parseArray method? http://feedback.arma3.com/view.php?id=16568 [^]"

Could you not use call compile to do that?

_blah = call compile "[33,44,55]"

Client
PREFIX_send = format ["[33, 44, 55, %1]", player];
fail, error in expression
PREFIX_send = "[33, 44, 55, player]"; // fail, undefined var player

publicVariableServer "PREFIX_send";

Server
_blah = call compile PREFIX_send;
fail

Normal way
Client
PREFIX_send = [33, 44, 55, player];
publicVariableServer "PREFIX_send";

Server
PREFIX_send is ready to be used

George_ added a subscriber: George_.May 7 2016, 5:08 PM

This should be optimized already, the normal generic serialization should not be used for simple types including arrays.

Last related change has been done in revision 130744. Can someone please test if this issue is solved and we can close it?