Page MenuHomeFeedback Tracker

Nitrado API Timeout
Assigned, UrgentPublic

Description

Hey guys! Hope you had an amazing holiday!

I'm opening this for a major problem we're experiencing on console.
Given the absence of modding for DayZ Console, all of us rely on the Nitrado API to set bots helping us managing our servers.
Player locations & interactions, bans, whitelist, priority, cfg, all is handled through the Nitrado API, meaning all we can do as server owners to manage our servers depends on it.

Here's the problem: since the last week of December, maybe even before that, Nitrado changed their service providers, and since then their API has been unresponsive, working approx 6 hours per day with odd intervals that sometimes had 12+ hours gaps.
The error coming back is an connect ETIMEDOUT API pipe failure.

While we could of course open a Nitrado support ticket and talk with the usual random first level person, it can be confirmed by any server owner reading this that that goes nowhere.
Given that they're our exclusive server host option, it's getting quite ridiculous that we can't even handle our servers and all we get is "apologies we're working on it" every so often.

Hence me asking your help here: can we please have you ask for all of us here to ask to them to give us access under the main dayzxb_missions / dayzps_missions for read/write symlinks to config, serverDZ.cfg, ban.txt, whitelist.txt, priority.txt so we can access those with a much simpler and sturdier protocol like FTP which we already have for dayzOffline.chernarusplus and that fails only in case the connection to the server itself is unavailable?
It would be as simple as having our mirror mission set this way

dayzxb_missions
|_dayzOffline.chernarusplus
|_config
|_serverDZ.cfg
|_ban.txt
|_whitelist.txt
|_priority.txt

Please, it's our only option to run things on console, and it's getting exasperating for us.

Details

Severity
Major
Resolution
Open
Reproducibility
Random
Operating System
Windows 7
Category
General

Event Timeline

Geez changed the task status from New to Need More Info.Jan 8 2024, 11:40 AM
Geez added a subscriber: Geez.

Hello TheDevilsKnock.
Can you please send us the names of the problematic servers where the issue manifests and also the Nitrado support ticket ID from the mentioned ticket?

TheDevilsKnock added a comment.EditedJan 8 2024, 12:15 PM

HI Geez!

Can you please send us the names of the problematic servers where the issue manifests

It's not server specific.
It's a global Nitrado Rest API global downtime. 500 class response from all endpoints. Affected all services that use the Nitrapi as intermediate to get/post server data.
Last one happening between 22:50 till 00:40, the one previous one was yesterday between 2:50 till 3:20, CET time.
It's probably getting mitigated, but it still continues.

and also the Nitrado support ticket ID from the mentioned ticket?

Sadly I do not have one, but I can send one in right away.
Edit:
Shared plenty of times on their official Discord. Issue started December 23rd


During the whole period, FTP was functioning with no issues. Hence why me asking to please consider the above request.

Geez added a comment.Jan 8 2024, 12:44 PM

Please send us a name of at least one affected server so we can try this on our end.

Please send us a name of at least one affected server so we can try this on our end.

Added mines in this private ticket.
https://feedback.bistudio.com/T178276

There you go with the ticket.
A technical question about API downtime asked from the official support is asked to be redirected on discord where no people working in their technical department are, because being a configuration?


It is impossible to work with this.

Geez changed the task status from Need More Info to Assigned.Jan 9 2024, 2:43 PM

Thank you, we have notified Nitrado.

Geez added a comment.Jan 9 2024, 4:33 PM

They are aware of the problem and are actively working on a fix.

Kamikaze added a subscriber: Kamikaze.EditedJan 29 2024, 10:40 PM

Hello,

It would be great if we could upload files(bans/whitelist) using FTP because the API still gives errors (like 500/502/504) from time to time, although not as often as before. Having the option to upload files this way would be a good backup plan for when other things go wrong. I know it's a big request, but I've been running servers for over 7 years, and dealing with troublemakers has always been a challenge. The recent issues with the Nitrado API have only made it harder. I hope you can consider this feedback.

Geez added a comment.Jan 30 2024, 1:03 PM

@TheDevilsKnock

Any new developments on your end?

@TheDevilsKnock

Any new developments on your end?

Right now we're kinda back to the norm where logs start populating ~15 minutes after a server start and update every ~7 minutes. Those are the current issues:

  • Nitrado added a new layer of undisclosed rate limits, where subsequent requests (where of course async ones are) may trigger a temporary IP ban [average 1 hour].
  • For some reasons, without notice they deleted pretty much all API tokens.
  • Beside requester specific problem, API requests can randomly return with empty response.
  • Post requests may return a 200 response (OK one), but may not execute the action on server (adding to white/priority/ban lists as example)

As it goes with Nitrado, all those problems happen whenever there is a major event. Them releasing Ark and then Palworld matched perfectly in timing. Game updates are the same.

Geez added a comment.Feb 15 2024, 4:31 PM

Hello again.
Nitrado had to implement additional protection layer on the API which could cause some malfunction. In order to resolve these issues, please contact Nitrado support directly.

After talking to the Nitrado Support, for everyone else wondering: The Rate Limiting supports 15k req/h, after that you get the rate limit response. The support can up the rate limit for specific IPs, you need to tell them the expected reqs/h and your server ip.