The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
āThe superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)ā
Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasnāt happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didnāt think about grabbing someone elseās radio. That may have been a lucky mistake though, as Iām not sure what all of those compressors turning at at the same time would have done.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most Iād the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
The IT person for McKinsey & Company said water came within feet of destroying all of the data for the firm and its clients. That led them to stand up a second DC.
That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.
> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
I remembered going to our data center in Long Island City and labeling all drives and pulling them from the servers and putting them in a box and took them home.
One of the best decisions I made around that time was installing two backup Towerstream lines. When everyone else was out for weeks we were up and running, full operation.
> āThey had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,ā Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
I'm surprised they located their datacenter in NYC.
The NYSE, NASDAQ, and most other exchanges and brokerages run in datacenters in Cartaret, NJ and Mahwah, NJ. I can't think of a good reason why anyone would put a datacenter in Manhattan.
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
I had the dubious honor of rolling servers on office chairs and hundreds of drives from an EMC box in a sterilite to get it upstate to a satellite office.
There was about $5M of gear in my wifeās grossly overloaded minivan. The bean counters tried to deny my mileage because I was using a personal vehicle without prior auth. lol. My boss was ready to kill.
>The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Sandy power disruption always reminds me of this photo - burned into my brain - https://archive.is/DoZsr
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
You'd generally not want to have the sprinklers go off while things were energized, but there could be a mechanism to monitor flow, and only cut power if sprinklers were actually sending out water. But even if this was present, your first priority would be for the sprinklers to not go off as there would be a lot of clean-up / drying needed after the fact.
Unless you have a mist system (NFPA 750), sprinklers put out a lot of water.
I love this story and that they kept the wabi-sabi :-)
Missing NYC and this reminds me of NYI back in the day (still there perhaps) and operating datacenters in the heart of Manhattan before things changed.
I like that they're keeping the metal close to the traders actually and I'd be totally aligned with that decision. I am surprised however that they haven't shifted to Halon fire extinguishers or something that is safe(r) for a data center.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
While CRACs won't recirculate oxygen, they can move air past sprinklers faster than they can handle which can delay operation, typically 5 fps. Although that's only if you are using sprinklers, clean agent is a different matter.
He had to deal with so much absolute bullshit to get the diesel generator I mentioned in the story done, and then Hurricane Sandy destroyed it instantly a few years later. Felt bad for him.
Sprinkler heads on fire suppression systems go off locally - if there's a fire, then only the sprinkler heads in the vicinity will go off. The alternative is to let the fire spread and destroy the entire facility.
But in addition, this particular story takes place in a skyscraper in Manhattan, with the data center in question occupying a portion of one floor, with office space surrounding it. So letting a fire burn would create significant risk to human life as well.
Often only one sprinkler head would discharge, so the damage might be localized within that ~12' radius to some degree. It's not a deluge system like many movies make out sprinkler systems to be where every head discharges simultaneously.
The DC in question was in the middle of a 33+ story office building, so the the worst case is "entire building catches fire and people die", not "equipment destroyed".
I heard of an inert gas fire fighting equipment discharging without a fire in a data center. The pressure wave created by the initial discharge ruined all the hard disks and caused replacement of all the equipment even though there was no fire (or water) as the conditions meant no more warranty
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if itās elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Oh, this reminds me a story from the time when I worked at a university. Poland, middle of harsh winter. The day before, there was a fire system inspection. On the day of event I was paged - our datacenter got flooded with water sprinklers (WTF why there are water sprinklers in DC!?), then the system opened all the doors and windows. Shit got frozen. The whole building was like an ice-skating rink. In the end only machines on top of the racks got damaged. Turned out that inspection was done by some relative of one of the professors and he fucked up something in fire system. It was a fun event haha
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
The real answer is probably lost in the sands of time but even at 200 people we still had different teams with overlapping responsibilities. It's possible I was the only person reachable and close enough after hours, and while I didn't know where the CRAC unit controls were I did know how to rip power cords out.
That's how the server room at my school worked. They said to get out of there if the alarm went off because the room would fill with some inert gas and you'd suffocate.
The gas is used first, then water as a fallback. I'm not sure if sprinklers are required by code in all data centers, but they would be in an office building setup like Jane Street's.
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
First, the guy probably wouldn't have keycard access to whatever room contained such a switch. Second, it's highly unlikely that the system would be able to cool the room fast enough to avoid hitting the temperature limit.
Incidentally, we actually had a fire in the office building I worked in long ago. The HVAC didn't shut down, and smoke started to come out of random vents throughout the building. It was chaos. And yet, plenty of workers wanted to finish and send that email they were working on, commit code and push it to the remote server, etc. What sticks with me most is the one guy, a very senior manager, running around yelling at people to get off their computers and get out of the building, because no job is ever worth your life.
They'd be shutdown by a dry contact that may variously be coming from the VESDA (Very Early Smoke Detection Apparatus), or an internal smoke detector, or a duct mounted smoke detector. Not as simple as flipping an obvious switch on a front panel.
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
Iām surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isnāt this mandatory, even? Especially in a datacenter in an office building?
The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
āThe superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)ā
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasnāt happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didnāt think about grabbing someone elseās radio. That may have been a lucky mistake though, as Iām not sure what all of those compressors turning at at the same time would have done.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most Iād the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
> 55 gallon drums of diesel fuel up 17 flights of stairs
Having moved 55 gallon drums around with a hand cart I don't think anyone is carrying them up stairs. They weigh 400 pounds!
The IT person for McKinsey & Company said water came within feet of destroying all of the data for the firm and its clients. That led them to stand up a second DC.
My colleague Brett has a Hurricane Sandy story about Jane Street as well
https://x.com/BrettHarrison/status/2117636364304892206
Which in turn reminds me of the "Interdictor" blogger who did the same during Hurricane Katrina https://en.wikipedia.org/wiki/Interdictor_(blog)
That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.
Why the hell did they park the generators on the 17th floor where people have to hand carry diesel fuel is the real question.
> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
Diesel is ~7lbs/gallon. That is truly superhuman damn.
I wonder how well these "superhuman folks" were paid
I remembered going to our data center in Long Island City and labeling all drives and pulling them from the servers and putting them in a box and took them home.
One of the best decisions I made around that time was installing two backup Towerstream lines. When everyone else was out for weeks we were up and running, full operation.
The ars technica article says:
> āThey had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,ā Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
Podcast version - https://soundcloud.com/stack-exchange/stack-overflow-podcast...
I'm surprised they located their datacenter in NYC.
The NYSE, NASDAQ, and most other exchanges and brokerages run in datacenters in Cartaret, NJ and Mahwah, NJ. I can't think of a good reason why anyone would put a datacenter in Manhattan.
This is the actual NYSE datacenter https://maps.app.goo.gl/Da1E6Je9eJAGA1RK8
This is the actual NASDAQ datacenter https://maps.app.goo.gl/ZS9VJcZmf8VtvWEd6
yep
sandy was a wreck
the only time i remember executing on DR plans (which went well!)
(i became a consultant by the time covid hit, so i didn't get to do the whole covid scramble every sysadmin did)
https://web.archive.org/web/20121103050116/http://searchdata...
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
I had the dubious honor of rolling servers on office chairs and hundreds of drives from an EMC box in a sterilite to get it upstate to a satellite office.
There was about $5M of gear in my wifeās grossly overloaded minivan. The bean counters tried to deny my mileage because I was using a personal vehicle without prior auth. lol. My boss was ready to kill.
Not fun but educational.
>The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Sandy power disruption always reminds me of this photo - burned into my brain - https://archive.is/DoZsr
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
did you at least ask for pay increase?
Also a perfect way to ruin the ability to work remotely. If you live so close, why don't you come every day for 8 hours? /s
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
The 2020 US NEC seems to have provisions for an EPO, but I'm not sure when it was added if it was present during the time of OP's incident:
* https://www.f-t.com/post/emergency-power-off-in-data-centers
EPO disconnects for generators was around earlier than the above:
* https://www.mgiepss.com/blog/emergency-power-off-epo
You'd generally not want to have the sprinklers go off while things were energized, but there could be a mechanism to monitor flow, and only cut power if sprinklers were actually sending out water. But even if this was present, your first priority would be for the sprinklers to not go off as there would be a lot of clean-up / drying needed after the fact.
Unless you have a mist system (NFPA 750), sprinklers put out a lot of water.
I love this story and that they kept the wabi-sabi :-)
Missing NYC and this reminds me of NYI back in the day (still there perhaps) and operating datacenters in the heart of Manhattan before things changed.
I like that they're keeping the metal close to the traders actually and I'd be totally aligned with that decision. I am surprised however that they haven't shifted to Halon fire extinguishers or something that is safe(r) for a data center.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
While CRACs won't recirculate oxygen, they can move air past sprinklers faster than they can handle which can delay operation, typically 5 fps. Although that's only if you are using sprinklers, clean agent is a different matter.
The episode of their podcast where Ron explores the facilities side of their data centers and offices is worth a listen too
https://signalsandthreads.com/the-thermodynamics-of-trading/
Daniel's as solid a dude as they come.
He had to deal with so much absolute bullshit to get the diesel generator I mentioned in the story done, and then Hurricane Sandy destroyed it instantly a few years later. Felt bad for him.
Are there internet connected fire alarms that can be hacked to destroy the contents of some data centers? Like the European freezers?
I did not know that data centers with sprinklers exist. I guess this is a very stupid idea in the first place.
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
Sprinkler heads on fire suppression systems go off locally - if there's a fire, then only the sprinkler heads in the vicinity will go off. The alternative is to let the fire spread and destroy the entire facility.
But in addition, this particular story takes place in a skyscraper in Manhattan, with the data center in question occupying a portion of one floor, with office space surrounding it. So letting a fire burn would create significant risk to human life as well.
Often only one sprinkler head would discharge, so the damage might be localized within that ~12' radius to some degree. It's not a deluge system like many movies make out sprinkler systems to be where every head discharges simultaneously.
The DC in question was in the middle of a 33+ story office building, so the the worst case is "entire building catches fire and people die", not "equipment destroyed".
I heard of an inert gas fire fighting equipment discharging without a fire in a data center. The pressure wave created by the initial discharge ruined all the hard disks and caused replacement of all the equipment even though there was no fire (or water) as the conditions meant no more warranty
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if itās elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Oh, this reminds me a story from the time when I worked at a university. Poland, middle of harsh winter. The day before, there was a fire system inspection. On the day of event I was paged - our datacenter got flooded with water sprinklers (WTF why there are water sprinklers in DC!?), then the system opened all the doors and windows. Shit got frozen. The whole building was like an ice-skating rink. In the end only machines on top of the racks got damaged. Turned out that inspection was done by some relative of one of the professors and he fucked up something in fire system. It was a fun event haha
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
> They could have diverged their compute to another datacenter surely.
The latency impact would be huge
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
The real answer is probably lost in the sands of time but even at 200 people we still had different teams with overlapping responsibilities. It's possible I was the only person reachable and close enough after hours, and while I didn't know where the CRAC unit controls were I did know how to rip power cords out.
I thought these data centers have a way to pump in a gas that prevents fires? Remember touring a data center at some point that has this ability.
That's how the server room at my school worked. They said to get out of there if the alarm went off because the room would fill with some inert gas and you'd suffocate.
They stopped using it many places because the risk to people was greater than the equipment.
The gas is used first, then water as a fallback. I'm not sure if sprinklers are required by code in all data centers, but they would be in an office building setup like Jane Street's.
Even though this happened in 2012, it's interesting that Jane Street is doubling down on compute and renting even more capacity from neoclouds. They are planning to expand 10x the amount of GPUs and are locking in tons of compute. On top of the $500M+ per MW of compute that they are paying Anthropic.
This is a trend in the wider algo trading market, not specific to Jane street
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
Close, it was Janus Mutual Funds, 26 years ago.
https://youtu.be/EYJLeuUjMlU?is=lzmEC-Q_bJ93FTgG
TIL: wabi-sabi. Iāve been trying to find a way to express the beauty of imperfection in the age of AI, and this fits perfectly.
Itās extremely common and cliche for people to use that term for imperfect stuff these days.
God dangit Bobby
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
good to see you make front mbac; glad you're still holding it down!!!!
remember sandy? that was crazy
How is it the only datacenter?
33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street
It's Jane Street's only datacenter. not the only one in NYC. As they explain, they chose to specially locate their datacenter.
The headline is confusing. Itās Jane Streetās only data center in NYC, not NYCās only data center.
This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then
Jane Streetās only NYC data center. Not the only data center in general.
Work nearby, it looks just as evil in person.
> the smoke alarm is wired to shut down the CRAC units
Why not just turn the CRAC back on?
> you donāt trust the automation unless thereās a big red-painted, mechanically connected lever you can pull to turn it off
Seems like this is one of those cases where you want to override the automation.
First, the guy probably wouldn't have keycard access to whatever room contained such a switch. Second, it's highly unlikely that the system would be able to cool the room fast enough to avoid hitting the temperature limit.
Incidentally, we actually had a fire in the office building I worked in long ago. The HVAC didn't shut down, and smoke started to come out of random vents throughout the building. It was chaos. And yet, plenty of workers wanted to finish and send that email they were working on, commit code and push it to the remote server, etc. What sticks with me most is the one guy, a very senior manager, running around yelling at people to get off their computers and get out of the building, because no job is ever worth your life.
Building fires are no joke. Get the F out.
They'd be shutdown by a dry contact that may variously be coming from the VESDA (Very Early Smoke Detection Apparatus), or an internal smoke detector, or a duct mounted smoke detector. Not as simple as flipping an obvious switch on a front panel.
I did not know Jane Street had a datacenter in NYC. Must of had it a long time ago.
Wow crazy I didn't know they had such simple systems
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
Sounds like a good idea to hack in and blow up the infrastructure of the hedge funds. How you gonna hedge this suckers
Almost too close to 'Pool on the roof must have a leak' to be real ;)
The EPO is in the place I put that thing that time.
Ah the apocryphal VAX story/scenario is alive and well
Iām surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isnāt this mandatory, even? Especially in a datacenter in an office building?
I am shocked to hear they do not use foam.
Related :) - https://news.ycombinator.com/item?id=50434310
He earned his bonus that year.
Man, I wish MOC were still alive. Iād love to know his take on this story.
Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.
[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane
Likey codes or building management wouldn't let them make the necessary modifications to safely install a Halon system
If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....
So fire suppression and data centers don't mix too well
Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.
Or CO2 but that's a life or death situation if there's any people still inside
Probably not safe to use on a floor that has people on it.
> Everyoneās heard of how awesome Jane Street is.
I guess Iām the last one to hear it.
Alarm firmware wasn't programmed in OCaml
They just need to hire some full stack people!
should be drowning tbh, less market manipulation
why don't they have a dry sprinkler system?
Datacenters on the coast never made sense to me.
Reminds me of DirectNic's datacenter flooding during Katrina (New Orleans), we all kept posting on Somethingawful until they finally shut it down.
> a false smoke alarm, a datacenter with no cooling, and a sprint to pull every power cord before the sprinklers went off
I smell AI