After BLAST Pro Series came to an end, we stopped by the RFRSH Entertainment offices to sit down with CEO and founder Nikolaj Nyholm for an interview about his company's first foray into event organizing.

The 42-year-old talked about day one's technical issues, which caused the first two rounds of the round robin to be postponed to day two, at length and explained what steps they took to prevent those issues from happening later on during the tournament.

Nikolaj Nyholm talked day one issues

Nyholm then discussed day two, evaluated the event and the their ideas as a whole, and shared some of RFRSH's plans for 2018 and 2019:

Could you go through the issues on day one and the decisions that you had to make as the day progressed?

To take things from the start, most of the players came in already on Tuesday, flying in from Oakland. All the teams had been there barring North, Astralis obviously had to go home a little bit early from that event, so they had spent some days at home. We had three of the teams here in the RFRSH office, where we're sitting now, in our rooms, practicing. Wednesday evening we put the practice rooms at the hotels and not before Thursday did we actually get access to the arena. Wednesday evening there was a Queen concert, they needed to disassemble everything and the stage is starting to be built on Thursday morning.

Given the fact that we have the format of having two rounds of round robin on Friday, we wanted to do that in a smaller space. We had initially been thinking about doing it out of the arena and we're still considering that for the future, we obviously need to assess how to do it best, do we go full on studio somewhere, do we rent a movie theater with room for a thousand fans, or do we do it as a smaller part of an arena experience. But anyhow, we were set up on the third floor of the arena, so not in, which of course some of the people who saw the initial part of the stream could see we weren't in the main arena. We could actually see the arena from there, but we weren't out there on the seats, they were still building on the floor. We had set up the six player booths there along with the admin facilities for Graham ["messioso" Pitt] and his team of admins.

What happens initially is: after about two rounds we started seeing CPU spikes on the server, which of course results in lag amongst the players. We go through first one server, we go through the backup server, we actually don't put any games live there, but when we had teams log in for sort of a warmup-type practice, we started seeing the same CPU spikes happening again. We use a third server, this time PGL's server, same issues, and finally we decide that we're going to play it without production on FACEIT servers. What happens there, the teams connect, get started, no issues whatsoever, and suddenly, after about 1-4 rounds depending on what match was happening, everything crashes at once. The players' connections to the FACEIT servers simply disappear. At that time, the admin connection to the servers are still up and running, so it has absolutely nothing to do with the FACEIT servers, they were running as they were supposed to and we're of course super, super happy with them being able to take this on. So the only thing that we can conclude at that point is that we have some sort of connectivity issues.

For those who are technically interested, the players are on the same physical switch as the admins but on a different VLAN. For security reasons we had set up a separate VLAN so that we can control the players' access to the rest of the network but also the players' access — given the fact we're suddenly playing on public internet servers — to other sites. And of course the admins need full access, so they were set up on a separate VLAN, and that is what crashed. The VLAN with the admins is still operational. What exactly happened there we're still diagnosing. Whether this is some poor connection somewhere, whether it's a hardware issue, the logs have been submitted to Cisco for review right now, but the problem is we actually can't fully test what happened back then. We had to remove all computers, all equipment, all cabling given the fact this was the area through which fans were going to move half a day later, Saturday morning.

We decide that we're going to push the first matches to Saturday morning instead of having the players hang around, hang around, hang around, waiting for something to happen, giving us sufficient time to fully test the setup in the arena, make certain that we also get additional back-up hardware on the networking side, and really continue to test what is happening and remove all the risks that might happen anywhere on the network. So we send all the players home back to the hotel, which is fortunately just around the corner from the arena, for them to get some sleep.

RFRSH filled the arena in their first stab at event organizing

I'm guessing you didn't get much sleep afterwards.

I absolutely got no sleep whatsoever, I slept a couple of hours tonight and I'm very grateful for that, but no, out of the respect for the players and especially for the fans, I think it is far from an ideal start to any tournament. And even worse when you're new, even worse when you're doing something a new way, obviously the fact that we are suddenly playing three games simultaneously puts additional pressure on servers, network, everything. It solves certain things, like not having players switch computers at all, of course the other tournaments could do that as well. One of the things which there were no issues with whatsoever were computer issues, driver issues, etc., which the players often complain about in other circumstances when they have to use computers that have been used by other players and they swap SSDs, there are still going to be some hardware issues.

So what we do is we go down to stress test everything with all 30 player computers running in the arena, which are by the way not the same PCs that were used earlier. For this event we had a total of 80 PCs, two times 30 for the players and then an additional 20 for a combination of backup and a few admin PCs. For us the most important thing during that Friday evening to Saturday morning were two elements, diagnose issues and figure out some of the things we were concerned about. First and foremost, initially we believed that — and this is also why we sent out two different messages — these were server issues. We concluded at the end once we had cycled through the three sets of servers and the FACEIT servers that they were indeed not server issues but that they were network issues. None of the networking hardware from upstairs came downstairs, entirely separate networks, never touched each other. We got additional backup hardware, we were going to use the hardware from upstairs as backup hardware for downstairs in the arena. That risk was removed.

We went through all the risk factors on our servers, I think many server admins probably know that GOTV can be somewhat of a hog in different situations, and given the fact that we even had to run multiple GOTV instances to give different delay for the different operators. For example, our betting operator needed access to the data faster than what is being sent by GOTV to some of our Twitch partners. We removed that, meaning that the betting operator — of course we lose that money, but then at least we removed that as an additional risk. They could no longer get access to data faster than other providers. We looked a lot at the demo recording into memory, which we also felt was a risk and we actually decided and determined not to do the demo recordings were we to run our local servers.

Finally, we stress tested the network with CS running on all 30 computers all connected to the server, all connected to external servers, meaning private servers at FACEIT, and in the end we decided for the tournament to run with the FACEIT servers. Of course we're grateful for FACEIT for providing us, that was always meant as the backup, but we actually decided to use that as our primary servers. And the reason also being that we had sub-10 miliseconds pings to the FACEIT servers all along. PGL, who were doing the in-game production, felt confident that that was more than sufficient for them, obviously our own network had sub-milisecond pings, it would be even better, but this was the one which we felt was certainly not going to give us any issues whatsoever on the server side.

That's basically what happened, I have to say that it wasn't until 4:45 Saturday morning that I felt confident when we had diagnosed ourselves out of every issue that we felt could happen. From about 5 AM we sat down and put together the exact plan of primary servers, secondary servers, tertiary servers, and together with Graham I sat down and did the plan for Saturday on backup solutions and how exactly we're going to operate it. On the servers on our own network we actually removed the demo recording, we informed our betting partner about the change in their ability to get data and so forth. The first players, NiP, showed up at about 9 AM on Saturday morning to start setting up while the cleaners and carpenters and so forth were still finishing the last things on the stage.

Do you feel happy with how day two went overall?

Absolutely, there was nothing from day one that happened on day two, so there were no server or networking issues whatsoever. We're fully aware that there were some small glitches in the in-game production which affected the stream, this is all related to HUD and so forth. I have to give PGL credit for the fact that they also spent time on Friday evening diagnosing server and networking issues, which they should probably have been spending on finishing the last things for the actual production on Saturday. So that is the spill-over effect, other spill-over effects were the crew that goes home later, Graham cycled his admins and his officials to sleep during the night so that we didn't have a zombie crew on Saturday.

This is our first event, big or small, ever. The only thing we've ever done before is that we have a company CS team here which plays in a company league in Denmark and once in a while they stream for fun. So literally this was our first event and we knew there were going to be things around the format, we knew there were going to be things about the production, I have to say we did not anticipate the Friday type of issues around the production, we did anticipate the Saturday type of issues around the production. We're going through everything, the team is sitting out here already now, going through everything for the next event as long as it's as fresh as possible in our minds.

So you're taking steps to prevent things like that from happening again.

Absolutely. One of the open mistakes which I will flatly admit that we made was that we did not ask for all 30 players to come to the venue and test things out beforehand. What they can expect in the future is that they have to show up, we have to simulate probably about half a map and we have to simulate about half a map on FACEIT servers also, to make certain everything is in order. You can script a lot of testing, which is always going to be a bit different when we suddenly have a different FACEIT client log in from somewhere else. I have to admit, there was one server-side issue on Saturday, which had to do with the FACEIT Anti-Cheat client. It was during the first match, that was the only thing that we saw server-side on Saturday, but that was addressed within 30 seconds.

Like you said, this was your first event, your first attempt into the Counter-Strike scene, what would you say are your biggest takeaways from the first BLAST, what are the biggest things you'll take with you into the future?

We have a lot of production side things which we want to improve. It's interesting that actually quite a lot of Riot people were in the audience and one of the comments from them, which I think was also correct, was that the brightness was too low in some portions in-game. Smaller things like that through to what cameras do we use at what points. I was personally unhappy that we didn't see the player run-ins on the big screens, they were actually on the small screens, but these are minor things which the TV production needs to work through.

On the higher level, there's stuff like: Should we be casting game B and game C from the arena? I think if you're in the arena and you're watching all three streams at once but you might be primarily looking at B, it's not really an issue that there is sound coming from the audience with regards to A. But if you're on the stream — these are some of the things I've seen on Twitter — you're listening to someone cast the B stream but the audience reaction is from the A stream. That is very out of place, so while it was very cool to have all three casting desks on the scene, I think that's a question mark whether we need to pull them back into a studio for casting.

I think we need to do a frank and honest assessment of Friday. I have to say, we did have a discussion of if we should take Friday as linear matches instead of three matches at the same time. We decided to go for the three at the same time as a general rehearsal for Saturday on the technical side. And I'm bloody happy we did, because otherwise we might not have encountered the issues, and if those had happened on Saturday, it would have been even even more detrimental than it was on Friday. We wouldn't have time to fix it, so at least we had the night between Friday and Saturday.

BLAST will return in 2018 with three to four events, Nyholm promises

These are some of the things, we're going to sit down, all the talent feedback is coming back through Anders, who's going to be sitting down with everyone later this week or early next week. I'm reading as much as possible on Twitter, I know there are some people who are very unhappy about Friday and that's built over to Saturday, but I take everything as constructive feedback, including for example the thing I mentioned before, where we cast I think is a super legitimate question. But I have to say that we're quite determined to at least on Saturday keep the three-matches-at-the-same-time type of setup, even if some fans are disappointed that they can't watch all the matches. I saw someone, I think it was lurppis, write a couple of days ago, the great thing is you get to watch the best match at any given time. And that for me as something between a core fan and a casual fan, that is a great experience. I haven't played anything serious since Half-Life, so it's a very long time ago, but on the other hand I'm not somebody who showed up just yesterday.

I'm going to sit down and watch our different partners' production, I read a lot of positive comments about DR, the national broadcaster here in Denmark. I know my parents watched it, I'm going to have dinner with them tonight and ask them about it. I spent a lot of time sitting down in the audience, just walking into empty seats and seeing fan reactions. I can read the Twitter comments later, but I can only see the fan reactions during the action and also ask them questions. Some of them were really wondering "Who's this weird guy?", we Danes are not always good at interacting with each other, "why is this guy coming and asking me questions about this?" But I only told them afterwards that we were simply collecting feedback on the event. I'm super pleased with the overall sentiment, the feedback that we've been getting from players and fans, and we know we can improve, we can always improve. We're not going to be arrogant about this, we know that we're doing things differently and that might not be to everyone's taste. We're at least trying to keep true to the game and then of course you can disagree in how we do our actual production.

Considering the risk you took, how very different BLAST was to anything else, do you feel happy with the event as a whole?

I have to say from a general product perspective, yes. For execution on Friday we get an absolute zero, we may even get a minus for dragging it out so long. Maybe that decision should have been made earlier. It's a decision that ultimately I took and after the decision was made I conferred with the coaches and IGLs of different teams and they concurred also on how to execute things on Saturday. On the product, I think we're trending the right way.

Can you share anything about what is coming up, what your plans are for 2018?

I can say as much as that we have an option on Copenhagen in November, we have an option on the arena. We're going to start a dialogue with them, we're going to start a dialogue with the different parties that were involved in the Copenhagen production, because of course we need the finances to make it work. Both the partners who were in this production, so sponsors, and the ones that decided to come and look and make certain that we could really execute on the first one. We have a meeting with DR lined up about their role in next year, I think they can play an even bigger role on some parts.

And then I can say as much as that the distribution of teams and the executive producer was sitting with two cities already at 11:30 PM, putting down the plan for 2018. We had officials from 14 cities globally visiting this event, all with the intent to work with us in 2018 or 2019, and who were really using the event to evaluate the product. At the end of the day, they knew everything that was going to happen, and then evaluate our execution of our product.

So, yes, three to four events next year.

I think one of the things we are also requesting from the cities is multi-year commitment. I have over 20 emails in my inbox this morning from people asking if we're going to be back in November and when the pre-sales are opening. We wanted to be something annually recurring, I think it's super important for the perception of esports, the continuity, the regularity. That's something that everyone is hungry for, the players, they want to know their schedules well in advance, they don't really know today, the fans do, they want to know what teams are coming. If we just look at our ticket sales, and of course we're looking at the billing address, we can't determine if people have traveled here for the event, or if they were in Copenhagen anyhow, or if they have a foreign card and live here. But we had credit cards from well over 26 nations being used at ticketmaster.

The CEO thanked the players for their understanding (Photo courtesy of RFRSH)

What was the furthest nation, out of interest?

It's Australia, but, actually, what I think I'm proudest about is knowing that people who had Thanksgiving in the U. S. traveled to Copenhagen to watch BLAST Pro. That would be the equivalent of most Europeans traveling on Christmas to watch a tournament in the U. S. or in South Africa...

If you have anything else, a final statement or a message to the fans?

I want to give an outright apology for Friday. I'm also sorry that I didn't get up on the stream or something to explain the situation more in depth, I have to say that when I got up on that chair to announce to the players that we're going to postpone until Saturday, I had about a million different options going through my head of what had actually been happening. Right there, it was first of all a decision to move it to Saturday and then we spent all of the night, so I should have probably been talking with you on Saturday morning and given you the story at that point.

But at the same time, no stone fell from my heart until the first games were live in the morning without an audience, and then finally that the games were live at 2 PM in the afternoon. Then I want to thank all the people who have been sending us constructive comments, I want to thank the players who have certainly been there for support. Getting a hug from kennyS after telling him that he wasn't going to play on Friday evening and telling me "look, it's okay, we understand, we've been through this before." And then I think I want to thank Semmler and his crew on the desk for paddling, paddling, paddling, and paddling more. If I could make a paddle with gold leaf, which is what the trophy was made out of, that would be what Semmler received last night. I think it's an absolutely amazing group of people around this, my colleagues have been sweating their asses off to make this happen and I'm grateful to them as well.