Introduction
Introduction Statistics Contact Development Disclaimer Help
_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
Visit Hacker News on the Web
COMMENT PAGE FOR:
OpenFreeMap survived 100k requests per second
cube00 wrote 22 hours 33 min ago:
It's really surprising that no CDNs or cloud storage providers offer
even the single PMTiles file in some sort of shared library customers
can use.
I guess they'd all rather their customers each upload the 120GB file
and then charge them all individually.
If they're crafty they'll have their storage configured so there's only
one actual copy on the underlying infra so every other shadow copy is
pure profit.
arend321 wrote 1 day ago:
I'm using OpenFreeMap commercially, fantastic and stable service.
cube00 wrote 22 hours 43 min ago:
Hopefully you've been able to contribute back financially if you're
making money from this service.
arend321 wrote 22 hours 15 min ago:
For sure, monthly sponsor.
CommanderData wrote 1 day ago:
I love sites like wplace can still go viral and blow up, in a age of an
increasingly centralised web. Woot
ivanjermakov wrote 1 day ago:
> Nice idea, interesting project, next time please contact me before.
I understand that my popular service might bring your less popular one
to the halt, but please configure it on your end so I know
_programmatically_ what its capabilities are.
I host no API without rate-limiting. Additionally, clearly listing
usage limits might be a good idea.
Aurornis wrote 22 hours 20 min ago:
> I understand that my popular service might bring your less popular
one to the halt, but please configure it on your end so I know
_programmatically_ what its capabilities are.
Quite entitled expectations for someone using a free and open service
to underpin their project.
The requests were coming from distributed clients, not a central API
gateway that could respond to rate limiting requests
> I host no API without rate-limiting. Additionally, clearly listing
usage limits might be a good idea.
Again, this wasn’t a central, well-behaved client hitting the API
from a couple of IPs or with a known API key.
They calculate that per every 1 user of the wlive.place website, they
were getting 1500 requests. This implies a lot of botting and
scripting.
This is basically load testing the web site at DDoS scale.
zamadatix wrote 18 hours 36 min ago:
> The requests were coming from distributed clients, not a central
API gateway that could respond to rate limiting requests
The block was done based on URL origin rather than client/token,
why wouldn't a rate limiter solution consider the same? For this
case (a site which uses the API) it would work perfectly fine.
Especially since the bots don't even care about the information
from this API so non-site based bots aren't even going to bother to
pull the OpenFreeMap tiles.
ivanjermakov wrote 22 hours 10 min ago:
Ugh, then I agree. This way it's indistinguishable from DDoS
attack.
Aeolun wrote 1 day ago:
I think it’s reasonable to assume that a free service is not going
to deal gracefully with your 100k rps hug of death. The fact that it
actually did is an exception, not the rule.
If you are hitting anything free with more than 10rps (temporarily)
you are an taking advantage in my opinion.
wiradikusuma wrote 1 day ago:
"Wplace.live happened. Out of the blue, a new collaborative drawing
website appeared, built from scratch using OpenFreeMap." -- as a
founder, you know you're working on the wrong thing when there's a "fun
project" getting daily traffic more than what you'd get in a lifetime
:)
parhamn wrote 1 day ago:
Since cloudflare is already sponsoring it, I do wonder how much of this
type of service can be implemented all on cloudflare. Their stack could
be great for tile serving.
PUSH_AX wrote 1 day ago:
You can run containers now on CF, I think this was one of the last
barriers that might have prevented a lot of software being migrated
without a re-architecting.
Most stuff could run there now.
leobuskin wrote 1 day ago:
I’m also surprised to see nginx and hetzner in this project. Why
not entirely Cloudflare: workers, R2, and cache
conradfr wrote 1 day ago:
You can get cheap dedicated server on Hetzner with unlimited
bandwidth, would the cost be similar with CF?
bravesoul2 wrote 1 day ago:
429 is your friend... but well done for handling the load!
Ericson2314 wrote 1 day ago:
Oh wow, TIL there is finally a simple way to actually view
OpenStreetMap! Gosh, that's overdue. Glad it's done though!
drewda wrote 21 hours 30 min ago:
The OSM Foundation has been serving raster tiles for years and years
(that's what's visible by default on the slippy map at
www.openstreetmap.org): [1] After on and off experimentation by
various contributors, OSMF just released vector tiles as well:
[1]: https://wiki.openstreetmap.org/wiki/OpenStreetMap_Carto
[2]: https://operations.osmfoundation.org/policies/vector/
bspammer wrote 1 day ago:
What was wrong with the main site? Genuine question
[1]: https://www.openstreetmap.org
bravesoul2 wrote 1 day ago:
... And then it became 1M rps!
biker142541 wrote 1 day ago:
Curious how this would have compared to a static pmtiles file being
read directly by maplibre. I’ve had good luck with virtually equal
latency to served tiles when consuming pmtiles via range requests on
Bunnycdn.
nielsole wrote 1 day ago:
[1]: https://github.com/hyperknot/openfreemap?tab=readme-ov-file#...
biker142541 wrote 1 day ago:
Interesting, I should benchmark this. I have only used Bunnycdn so
far and latency seemed similar to most tile providers like Maptiler
and others (but a very limited test). This was using the full
planet pmtiles file.
Bunnycdn also makes it easy to prevent downloading the entire file,
either in case you care about anyone using it or just want to
prevent surprise downloads for those exploring network tab.
biker142541 wrote 1 day ago:
Quick benchmark of pmtiles directly in maplibre vs served tiles,
both via Bunnycdn and 5 areas sampled using same style.
Total impact on page end to end load time: 39ms longer with
cached range requests from pmtiles than cached tiles.
Individual requests are comparable in the 20-35ms range, so the
slight extra time seems to be from the additional round trip for
range headers (makes sense).
nielsole wrote 12 hours 54 min ago:
What I haven't understood is why not write a program that
serves the pmtiles individually on the server? Might not be as
performant (but easily hit 1k RPS) but wouldn't require
mounting another filesystem which practically rules out a
containerised environment.
hyperknot wrote 1 day ago:
Yes, wplace could solve their whole need by a single, custom-built
static pmtile. No need to serve 150 GB of OSM data for their
use-case.
hoppp wrote 1 day ago:
Cool... You did well to ban them.
Its a ddos attack, lucky you dont have to pay for the brandwidth, then
its a denial of wallet
Starlevel004 wrote 1 day ago:
> I believe what is happening is that those images are being drawn by
some script-kiddies. If I understand correctly, the website limited
everyone to 1 pixel per 30 seconds, so I guess everyone was just
scripting Puppeteer/Chromium to start a new browser, click a pixel, and
close the browser, possibly with IP address rotation, but maybe that
wasn't even needed.
I think you perhaps underestimate just how big of a thing this became
basically overnight. I mentioned a drawing over my house to a few
people and literally everyone instantly knew what I meant without even
saying the website. People love /r/place style things every few years,
and this having such a big canvas and being on a world map means that
there is a lot of space for everyone to draw literally where they live.
indrora wrote 21 hours 8 min ago:
It's also way more than 1px/30s -- Its like 20px/30s and you have a
"tank" of them, which you can expand to however big you want.
Placing pixels gives you points, which you can turn into more pixels
or a bigger bag of pixels over time. I've seen people who have done
enough pixel pushing that they get 3-4K pixels at a time.
yifanl wrote 22 hours 15 min ago:
They have the user count from the dev, 2 million daily users
shouldn't be generating billions of requests unless a good portion of
them are botting.
zamadatix wrote 18 hours 40 min ago:
Why not? This is tile requests right, not login requests or
something, so shouldn't a single user be expected to consume a few
thousand zipping around the map while looking at drawings overlaid?
I'm sure there is some botting, it's basically guaranteed, but I
wouldn't be surprised if nearly half the traffic was "legitimate".
The bots don't normally need to reload (or even load) the map tiles
anyways.
zahlman wrote 22 hours 23 min ago:
> I think you perhaps underestimate just how big of a thing this
became basically overnight. I mentioned a drawing over my house to a
few people and literally everyone instantly knew what I meant without
even saying the website.
On the other hand, this is the first I've heard of this thing.
johnisgood wrote 21 hours 38 min ago:
I have known about this kind of pixel drawing but it was on empty
canvas.
Aurornis wrote 22 hours 33 min ago:
> I think you perhaps underestimate just how big of a thing this
became basically overnight.
They don’t need to estimate because in the article they talked to
the site and got their traffic numbers: An estimated 2 million users.
That’s 1500 requests per user, which implies a lot of scripting is
going on.
andai wrote 1 day ago:
From the screenshot I wanted to say, couldn't this be done on a single
VPS? Seemed over engineered to me. Then I realized the silly pixels are
on top of a map of the entire earth. Dang!
I'm curious what the peak req/s is like. I think it might be just
barely within the range supported by benchmark-friendly web servers.
Unless there's some kind of order of magnitude slowdowns due to the
nature of the application.
Edit: Looks like about 64 pixels per km (4096 per km^2). At full color
uncompressed that's about 8TB to cover the entire earth (thinking
long-term!). 10TB box is €20/month from Hetzner. You'd definitely
want some caching though ;)
Edit 2: wplace uses 1000x1000 px pngs for the drawing layer. The
drawings load instantly, while the map itself is currently very laggy,
and some chunks permanently missing.
TylerE wrote 1 day ago:
"€20/month from Hetzner" is great until you actually need it to be
up and working when you need it.
motorest wrote 1 day ago:
> "€20/month from Hetzner" is great until you actually need it to
be up and working when you need it.
I managed a few Hetzner cloud instances, and some report perfect
uptime for over a year. The ones that don't, I was the root cause.
What exactly leads you to make this sort of claim? Do you actually
have any data or are you just running your mouth off?
slacktivism123 wrote 1 day ago:
>What exactly leads you to make this sort of claim? [1] [2] [3]
>are you just running your mouth off?
Don't be snarky. Edit out swipes.
[1]: https://news.ycombinator.com/item?id=29651993
[2]: https://news.ycombinator.com/item?id=42365295
[3]: https://news.ycombinator.com/item?id=44038591
[4]: https://news.ycombinator.com/newsguidelines.html
eitland wrote 22 hours 34 min ago:
I’ve just checked all three, and it appears to be rather more
nuanced than you make it out to be.
Aeolun wrote 1 day ago:
This all seems to be Hetzner Cloud offerings?
immibis wrote 1 day ago:
IME Hetzner's not unreliable. I don't think you could serve 100k
requests per second on a single VPS though. (And with dedicated,
you're on the hook for redundancy yourself, same as any dedicated.)
cyberpunk wrote 1 day ago:
They’re unreliable as soon as you have to deal with their
support who have the technical knowledge of a brick.
And as soon as you have to do ant business / deal with the german
side of the business expect everything to slow down to 2 weeks
for response which will still be incorrect.
They are simply not worth the hassle. Go with a competent host.
celsoazevedo wrote 1 day ago:
Ideally, we'd all be using very good and competent hosting
companies, but they're not ideal for free/low revenue projects.
It's better to have some down time than having to shut down the
service because you can't afford to keep it running.
I think that in both cases here (OpenFreeMap and wplace),
Hetzner/OVH/Scaleway is the way to go. Depending on what we're
doing, the cost savings can even allow us to have redundancy at
another cheap provider just in case something goes wrong.
Aeolun wrote 1 day ago:
> They’re unreliable as soon as you have to deal with their
support who have the technical knowledge of a brick.
Since I never have to, that’s perfect isn’t it? If you need
support from Hetzner you are using the wrong host.
ranger_danger wrote 15 hours 43 min ago:
Simply running "ipfs daemon" is enough to get hetzner to
threaten canceling your service.
They send nasty letters with pcap dumps of your "hacking"
attempts and it doesn't matter how wrong you think they are.
cyberpunk wrote 22 hours 48 min ago:
So they've never decided to blackhole your prod traffic 2
days after migrating to them? lucky.
Aeolun wrote 17 hours 24 min ago:
I don’t think I’ve ever migrated anywhere and not kep
the previous thing running for a month unless I wasn’t
worried about exactly that thing happening (to be fair, not
prod traffic blackhole, but similar effect)
perching_aix wrote 1 day ago:
Haven't worked with Cloudflare yet first hand, and I'm not familiar
with web map tech. But if the site really is pretty much just serving
lots of static files, why is Hetzner in the loop? Wouldn't fully
migrating to Cloudflare Pages be possible?
internetter wrote 1 day ago:
The tiles need to be rendered. Yes frequent tiles can be cached but
you already have a cache… it’s Cloudflare. Theoretically you
could port the tileserver to Cloudflare pages but then you’d need
to… port it… and it probably wouldn’t be cheaper
ohdeargodno wrote 1 day ago:
Noone renders tiles on servers anymore. The vast majority of
services have moved on to sending out vector tiles.
hoppp wrote 1 day ago:
It would cost a lot. Hetzner is hardware and them you can hammer
it, free bandwidth. You get a very good server for cheap.
cloudflare would be pay per request, a hefty sum if ddos happens
perching_aix wrote 1 day ago:
Did a quick cost calc (with the help of gpt5, so might be wrong)
when I read their comment about Pages not being suitable for this
many files.
They say they're receiving $500/mo in donos and that it's
currently just enough to cover their infra costs. Given 300
million 70 KB files, R2 + high cache hit ratio would work out to
about $300 in storage-months + request costs, or $600/mo with
Cache Reserve and then they'd always hit cache if I understand
the project right: meaning the costs shouldn't blow up beyond
that, and that request count would essentially just not matter.
cuu508 wrote 1 day ago:
Factor in that you also need resources to generate and upload
the tiles weekly.
hoppp wrote 1 day ago:
Yea but the cost is not a fixed monthly sum and things can go
wrong as we can see from the blog post.
An accident could bankrupt the dev.
A dedicated server will always cost the same so you always know
how much you pay.
It will cost 40 Euro/month to have 6 cores/12 threads,64gb of
ram and 1Tb of ssd.
Dirt cheap compared to any other alternative
hyperknot wrote 1 day ago:
They are actually static files. There is just too many of them,
about 300 million. You cannot put that in Pages.
jonathanlydall wrote 1 day ago:
Is CloudFlare’s R2 an option for you?
piperswe wrote 10 hours 8 min ago:
That, or enabling Cache Reserve to automatically have a global
cache in R2 instead of only caching on the PoP-level
(disclaimer/source: I am a CF employee)
perching_aix wrote 1 day ago:
Oh interesting, okay. For some reason I had the impression that the
tiles were static and rendered offline.
ch33zer wrote 1 day ago:
Since the limit you ran into was number of open files could you just
raise that limit? I get blocking the spammy traffic but theoretically
could you have handled more if that limit was upped?
hyperknot wrote 1 day ago:
I've just written my question to the nginx community forum, after a
lengthy debugging session with multiple LLMs. Right now, I believe it
was the combination of multi_accept + open_file_cache >
worker_rlimit_nofile. [1] Also, the servers were doing 200 Mbps, so I
couldn't have kept up _much_ longer, no matter the limits.
[1]: https://community.nginx.org/t/too-many-open-files-at-1000-re...
justinclift wrote 1 day ago:
> so I couldn't have kept up _much_ longer, no matter the limits.
Why would that kind of rate cause a problem over time?
toast0 wrote 1 day ago:
I'm pretty sure your open file cache is way too large. If you're
doing 1k/sec, and you cache file descriptors for 60 minutes,
assuming those are all unique, that's asking for 3 million FDs to
be cached, when you've only got 1 million available. I've never
used nginx or open_file_cache[1], but I would tune it way down and
see if you even notice a difference in performance in normal
operation. Maybe 10k files, 60s timeout.
> Also, the servers were doing 200 Mbps, so I couldn't have kept up
_much_ longer, no matter the limits.
For cost reasons or system overload?
If system overload ... What kind of storage? Are you monitoring
disk i/o? What kind of CPU do you have in your system? I used to
push almost 10GBps with https on dual E5-2690 [2], but it was a
larger file. 2690s were high end, but something more modern will
have much better AES acceleration and should do better than 200
Mbps almost regardless of what it is. [1] to be honest, I'm not
sure I understand the intent of open_file_cache... Opening files is
usually not that expensive; maybe at hundreds of thousands of rps
or if you have a very complex filesystem. PS don't put tens of
thousands of files in a directory. Everything works better if you
take your ten thousand files and put one hundred files into each of
one hundred directories. You can experiment to see what works best
with your load, but a tree where you've got N layers of M
directories and the last layer has M files is a good plan, 64 <= M
<= 256. The goal is keeping the directories compact so searching
and editing is effective.
[2]
[1]: https://www.intel.com/content/www/us/en/products/sku/64596...
CoolCold wrote 5 hours 10 min ago:
> [1] to be honest, I'm not sure I understand the intent of
open_file_cache... Opening files is usually not that expensive
I may have a hint here - remember, that Nginx was created in the
times of dialup was a thing yet and having single Pentium 3
server was a norm (I believe I've seen myself that wwwXXX
machines in the Rambler DCs over that time).
So my a bit educated guess here, that saving every syscal was
sorta ultimate goal and it was more efficient in terms of at
least latency by that times. You may take a look how Nginx parses
http methods (GET/POST) to save operations.
Myself I don't remember seeing large benefits of using
open_file_cache, but I likely never did a proper perf test here.
Say ensure use of sendfile/buffers/TLS termination made much more
influence for me on modern (10-15 years old) HW.
Aeolun wrote 1 day ago:
If you do 200Mbps on a hetzner server after cloudflare caching,
you are going to run out of traffic pretty rapidly. The limit is
20TB / month (which you’d reach in roughly 9 days).
CoolCold wrote 5 hours 20 min ago:
You are probably talking about VMs - those do have traffic
limits. Servers, on the other side, with default 1Gbit NICs
doesn't (let's say until you consume 80%+ of bandwidth for
months)
Quoting:
> Traffic
>All root servers have a dedicated 1 GBit uplink by default and
with it unlimited traffic. Inclusive monthly traffic for
servers with 10G uplink is 20TB. There is no bandwidth
limitation. We will charge € 1 ($1.20)/TB for overusage.
MaKey wrote 20 hours 56 min ago:
Small addition: That limit applies to Hetzner Cloud servers,
their dedicated servers have unlimited traffic.
Aeolun wrote 17 hours 22 min ago:
Depends on your connection I think. Mine do 1Gbit/sec but
have a 20TB limit. The 100Mbit ones are unlimited (last I
checked)
johnisgood wrote 21 hours 14 min ago:
One would think services like these do not have to rely on
online services and have their own rack of servers. Or is this
so alien these days?
ndriscoll wrote 1 day ago:
One thing that might work for you is to actually make the empty
tile file, and hard link it everywhere it needs to be. Then you
don't need to special case it at runtime, but instead at generation
time.
NVMe disks are incredibly fast and 1k rps is not a lot (IIRC my
n100 seems to be capable of ~40k if not for the 1 Gbit NIC
bottlenecking). I'd try benchmarking without the tuning options
you've got. Like do you actually get 40k concurrent connections
from cloudflare? If you have connections to your upstream kept
alive (so no constant slow starts), ideally you have numCores
workers and they each do one thing at a time, and that's enough to
max out your NIC. You only add concurrency if latency prevents you
from maxing bandwidth.
hyperknot wrote 1 day ago:
Yes, that's a good idea. But we are talking about 90+% of the
titles being empty (I might be wrong on that), that's a lot of
hard links. I think the nginx config just need to be fixed, I
hope I'll receive some help on their forum.
ndriscoll wrote 1 day ago:
You could also try turning off the file descriptor cache. Keep
in mind that nvme ssds can do ~30-50k random reads/second with
no concurrency, or at least hundreds of thousands with
concurrency, so even if every request hit disk 10 times it
should be fine. There's also kernel caching which I think
includes some of what you'd get from nginx's metadata cache?
willsmith72 wrote 1 day ago:
so 96% availability = "survived" now?
but interesting write-up. If I were a consumer of OpenFreeMap, I would
be concerned that such an availability drop was only detected by user
reports
timmg wrote 1 day ago:
96% during a unique event. I think you would typically consider long
term in a stat like that.
Assuming it was close to 100% the rest of the year, that works out to
99.97% over 12 months.
ndriscoll wrote 1 day ago:
If I were a consumer of a free service from someone who will not take
your money to offer support or an SLA (i.e. is not trying to run a
business), I would assume there's little to no monitoring at all.
rtaylorgarlock wrote 1 day ago:
Is it always/only 'laziness' (derogatory, i know) when caching isn't
implemented by a site like wplace.live ? Why wouldn't they save
openfreemap all the traffic when a caching server on their side
presumably could serve tiles almost as fast or faster than openfreemap?
toast0 wrote 1 day ago:
Why should they when openfreemap is behind a CDN and their home page
says things like:
> Using our public instance is completely free: there are no limits
on the number of map views or requests. There’s no registration, no
user database, no API keys, and no cookies. We aim to cover the
running costs of our public instance through donations.
> Is commercial usage allowed?
> Yes.
IMHO, reading this and then just using it, makes a lot of sense.
Yeah, you could put a cache infront of their CDN, but why, when they
said it's all good, no limits, for free?
I might wonder a bit, if I knew the bandwidth it was using, but I
might be busy with other stuff if my site went unexpectedly viral.
radu_floricica wrote 22 hours 8 min ago:
Politeness? Also speed.
But it depends on the project architecture. If the tiles are needed
only client-side, then there's really little reason to cache things
on _my_ server. That would imply I'm better at caching
openstreetmap tiles than... openstreetmap. Plus you're just making
the system needlessly more complicated.
And there's little reason for openstreet map to be upset, since
it's not like _I_ am making 2 million requests - there are 2
million separate users of osm+. It's aligned to their incentive of
having more people use osm and derived works. All is well.
Aeolun wrote 1 day ago:
I think, when you read that, you should be reassured that nobody is
going to suddenly tell you to pay, and then still implement caching
on your own side to preserve the free offering for everyone else.
Seriously, whose first thought on reading that is “oh great, I
can exploit this”.
naniwaduni wrote 1 day ago:
You don't need to be thinking "I can't exploit this" when you can
just stop thinking about it.
hyperknot wrote 1 day ago:
We are talking about an insane amount of data here. It was 56 Gbit/s
(or 56 x 1 Gbit servers 100% saturated!).
This is not something a "caching server" could handle. We are talking
on the order of CDN networks, like Cloudflare, to be able to handle
this.
Sesse__ wrote 1 day ago:
> We are talking about an insane amount of data here. It was 56
Gbit/s. This is not something a "caching server" could handle.
You are not talking about an insane amount of data if it's 56
Gbit/s. Of course a caching server could handle that.
Source: Has written servers that saturated 40gig (with TLS) on an
old quadcore.
bigstrat2003 wrote 1 day ago:
I realize that what constitutes "insane" is a subjective
judgement. But, uh... I most certainly would call 56 Gbps insane.
Which is not to say that hardware which handles it doesn't exist.
It might not even be especially insane hardware. But that is a
pretty insane data rate in my book.
hyperknot wrote 1 day ago:
OK, technically there might exist such server, I guess Netflix
and friends are using those. But we are talking about a community
supported, free service here. Hetzner servers are my only
options, because of their unmetered bandwidth.
Sesse__ wrote 1 day ago:
It really depends on the size of the active set. If it fits
into RAM of whatever server you are using, then it's not a
problem at all, even with completely off-the-shelf hardware and
software. Slap two 40gig NICs in it, install Varnish or
whatever and you're good to go. (This is, of course, assuming
that you have someone willing to pay for the bandwidth out to
your users!)
If you need to go to disk to serve large parts of it, it's a
different beast. But then again, Netflix was doing 800gig
already three years ago (in large part from disk) and they are
handicapping themselves by choosing an OS where they need to do
significant amounts of the scaling work themselves.
hyperknot wrote 1 day ago:
I'm sure the server hardware is not a problem. The full
dataset is 150 GB and the server has 64 GB RAM, most of which
will be never requested. So I'm sure that the used tiles
would actually get served from OS cache. If not, it's on a
RAID 0 NVME SSD, connected locally.
What I've been referring to is the fact that even unlimited 1
Gbps connections can be quite expensive, now try to find a
2x40 gig connection for a reasonable money. That one user
generated 200 TB in 24 hours! I have no idea about bandwidth
pricing, but I bet it ain't cheap to serve that.
Sesse__ wrote 1 day ago:
Well, “bandwidth is expensive” is a true claim, but
it's also a very different claim from “a [normal] caching
server couldn't handle 56 Gbit/sec”…?
Aeolun wrote 1 day ago:
56 Gbit/sec costs you about €590/day even on Hetzner.
hyperknot wrote 1 day ago:
You are correct. I was putting "a caching server on their
side" in the context of their side being a single dev
hobby project running on a VPS, exploding on the weekend.
I agree that these servers do exist and some companies do
pay for this bandwidth as part of their normal
operations.
ndriscoll wrote 1 day ago:
I'd be somewhat surprised if nginx couldn't saturate a 10Gbit link
with an n150 serving static files, so I'd expect 6x $200 minipcs to
handle it. I'd think the expensive part would be the
hosting/connection.
wyager wrote 1 day ago:
> or 56 x 1 Gbit servers 100% saturated
Presumably a caching server would be 10GbE, 40GbE, or 100GbE
56Gbit/sec of pre-generated data is definitely something that you
can handle from 1 or 2 decent servers, assuming each request
doesn't generate a huge number of random disk reads or something
markerz wrote 1 day ago:
It looks like a fun website, not a for-profit website. The
expectations and focus of fun websites is more to just get it working
than to handle the scale. It sounds like their user base exploded
overnight, doubling every 14 hours or so. It also sounds like it’s
other a solo dev or a small group based on the maintainers wording.
VladVladikoff wrote 1 day ago:
I actually have a direct answer for this: priorities.
I run a fairly popular auction website and we have map tiles via
stadia maps. We spend about $80/month on this service for our volume.
We definitely could get this cost down to a lower tier by caching the
tiles and serving them from our proxy. However we simply haven’t
yet had the time to work on this, as there is always some other task
which is higher priority.
latchkey wrote 1 day ago:
Like reading and commenting on HN articles! ;-)
jspiner wrote 1 day ago:
The cache hit rate is amazing. Is there something you implemented
specifically for this?
hyperknot wrote 1 day ago:
Yes, I designed the whole path structure / location blocks with
caching in mind. Here is the generated nginx.conf, if you are
interested:
[1]: https://github.com/hyperknot/openfreemap/blob/main/docs/asse...
eggbrain wrote 1 day ago:
Limiting by referrer seems strange — if you know a normal user makes
10-20 requests (let’s assume per minute), can’t you just rate limit
requests to 100 requests per minute per IP (5x the average load) and
still block the majority of these cases?
Or, if it’s just a few bad actors, block based on JA4/JA3
fingerprint?
toast0 wrote 1 day ago:
Limiting by referrer is probably the right first step. (And changing
the front page text)
You want to track usage by the site, not the person, because you can
ask a site to change usage patterns in a way you can't really ask a
site's users. Maybe a per IP limit makes sense too, but you wouldn't
want them low enough that it would be effective for something like
this.
hyperknot wrote 1 day ago:
What if one user really wants to browse around the world and explore
the map. I remember spending half an hour in Google Earth desktop,
just exploring around interesting places.
I think referer based limits are better, this way I can ask high
users to please choose self-hosting instead of the public instance.
feverzsj wrote 1 day ago:
So, OFM was hit by another Million Dollar Homepage for kids.
charcircuit wrote 1 day ago:
>Nice idea, interesting project, next time please contact me before.
It's impossible to predict that one's project may go viral.
>As a single user, you broke the service for everyone.
Or you did by not having a high enough fd limit. Blaming sites when
using it too much when you advertise there is no limit is not cool.
It's not like wplace themselves were maliciously hammering the API.
rikafurude21 wrote 1 day ago:
the funny part is that his service didnt break- cloudflares cache
caught 99% of the requests. just wanted to feel powerful and break
the latest viral trend.
010101010101 wrote 1 day ago:
Do you expect him just to let the service remain broken or to scale
up to infinite cost to himself on this volunteer project? He worked
with the project author to find a solution that works for both and
does not degrade service for every other user, under literally no
obligation to do anything at all.
This isn’t Anthropic deciding to throttle users paying hundreds of
dollars a month for a subscription. Constructive criticism is one
thing, but entitlement to something run by an individual volunteer
for free is absurd.
toast0 wrote 1 day ago:
The project page kind of suggests he might scale up to infinite
cost...
> Financially, the plan is to keep renting servers until they cover
the bandwidth. I believe it can be self-sustainable if enough
people subscribe to the support plans.
Especially since he said Cloudflare is providing the CDN for
free... Yes, running the origins costs money, but in most cases,
default fd limits are low, and you can push them a lot higher. At
some point you'll run into i/o limits, but I think the I/O at the
origin seems pretty managable if my napkin math was right.
If the files are all tiny, and the fd limit is the actual
bottleneck, there's ways to make that work better too. IMHO, it
doesn't make sense to accept a inbound connection if you can't get
a fd to read a file for it, so better to limit the concurrent
connections and let connections sit in the listen queue and have a
short keepalive time out to make sure you're not wasting your fds
on idle connections. With no other knowledge, I'd put the
connection limit at half the FD limit, assuming the origin server
is dedicated for this and serves static files exclusively. But, to
be honest, if I set up something like this, I probably wouldn't
have thought about FD limits until they got hit, so no big deal ...
hopefully whatever I used to monitor would include available fds by
default and I'd have noticed, but it's not a default output
everywhere.
charcircuit wrote 1 day ago:
We are talking about hosting a fixed amount of static files. This
should be a solved problem. This is nothing like running large AI
models for people.
010101010101 wrote 1 day ago:
The nature of the service is completely irrelevant.
charcircuit wrote 1 day ago:
Running a no limit service for free definitely depends on the
marginal cost of serving a single request.
columb wrote 1 day ago:
You are so entitled... Because of you most nice things have "no
limits but...". Not cool stress testing someone's infrastructure. Not
cool.
The author of this post is more than understanding, tried to fix it
and offered a solution even after blocking them. On a free service.
Show us what you have done.
charcircuit wrote 1 day ago:
>You are so entitled
That's how agreements work. If someone says they will sell a
hamburger for $5, and another person pays $5 for a hamburger, then
they are entitled to a hamburger.
>On a free service.
It's up to the owner to price the service. Being overwhelmed by
traffic when there are no limits is not a problem limited only to
free services.
austhrow743 wrote 1 day ago:
Hamburger situation is not comparable. It’s a trade.
This is just someone being not very specific in a text file on
their computer. I have many such notes, some of them publicly
viewable.
eszed wrote 1 day ago:
Sure, and if you bulk-order 5k hamburgers the restaurant will
honor the price, but they'll also tell you "we're going to need
some notice to handle that much product". Perfect analogy,
really. This guy handled the situation perfectly, imo.
charcircuit wrote 1 day ago:
Except in this case the restauraut would have been able to
handle the 5k orders if they didn't arbitrarily have their
workers work with their hands tied behind their back. And
instead of untieing their workers and appreciating the
realization they were accidently bottlenecking themselves they
blame the nearby event who caused a spike in foot traffic.
Publicly attacking your users instead of celebrating their
success and your new learnings is not what I would call
handling it perfectly. I think going for a halo effect strategy
where you celebrate how people are using your platform to
accomplish their goals will help people understand how what is
being done is valuable and want people to adopt it or
financially support it. On the other hand attacking people who
use your platform publicly can make people apprehensive in
using it fearing that they will be criticized too.
perching_aix wrote 1 day ago:
> Do you offer support and SLA guarantees?
>
> At the moment, I don’t offer SLA guarantees or personalized
support.
From the website.
LoganDark wrote 1 day ago:
> I believe what is happening is that those images are being drawn by
some script-kiddies.
Oh absolutely not. I've seen so many autistic people literally just
nolifing and also collaborating on huge arts on wplace. It is
absolutely not just script kiddies.
> 3 billion requests / 2 million users is an average of 1,500 req/user.
A normal user might make 10-20 requests when loading a map, so these
are extremely high, scripted use cases.
I don't know about that either. Users don't just load a map, they look
all around the place to search for and see a bunch of the art others
have made. I don't know how many requests is typical for "exploring a
map for hours on end" but I imagine a lot of people are doing just
that.
I wouldn't completely discount automation but these usage patterns seem
by far not impossible. Especially since wplace didn't expect sudden
popularity so they may not have optimized their traffic patterns as
much as they could have.
Karliss wrote 1 day ago:
Just scrolled around a little bit 2-3minutes with network monitor
open. That already resulted in 500requests, 5MB transferred (after
filtering by vector tile data). Not sure how many of those got cached
by browser with no actual requests, cached by browser exchanging only
headers or cached by cloudflare. I am guessing that the typical 10-20
requests/user case is for embedded map fragment like those commonly
found in contact page where most users don't scroll at all or at most
slightly zoom out to better see rest of city.
nemomarx wrote 1 day ago:
There are some user scripts to overlay templates on the map and
coordinate working together, but I can't imagine that increases the
load much. What might is that wplace has been struggling under the
load and you have to refresh to see your pixels placed or any changes
and that could be causing more calls an hour maybe?
fnord77 wrote 1 day ago:
sounds like they survived 1,000 reqs/sec and the cloudflare CDN
survived 99,000 reqs/sec
v5v3 wrote 1 day ago:
The article mentions Cloudflare, so how much of this was cached by
them?
alessandroberna wrote 1 day ago:
99.38%
do_anh_tu wrote 1 day ago:
Do you even read the article?
jwilk wrote 1 day ago:
From the HN Guidelines < [1] >:
> Please don't comment on whether someone read an article. "Did you
even read the article? It mentions that" can be shortened to "The
article mentions that".
[1]: https://news.ycombinator.com/newsguidelines.html
RandomBacon wrote 1 day ago:
That guideline is decent I guess.
I am disappointed that they edited another guideline for the
worse:
> Please don't comment about the voting on comments. It never
does any good, and it makes boring reading.
It used to just say, don't complain about voting.
If the number of votes are so taboo, why do they even show us the
number or user karma (and have a top list)?
RandomBacon wrote 1 day ago:
We can't even talk about the guidelines?
keketi wrote 1 day ago:
Are you new? Nobody actually reads the articles.
LorenDB wrote 1 day ago:
False. I almost never upvote an article without reading it, and
half of those upvotes are because I already read something
similar recently that gave me the same information.
eszed wrote 1 day ago:
I'll submit in the second case (already read something similar)
that, properly speaking, we should read both, and upvote (or
submit, if not already here) the better of the articles.
Not that, you know, I often take the time to do that, either -
but it would improve the site and the discussions if we all
did.
colinbartlett wrote 1 day ago:
Thank you for this breakdown and for this level of transparency. We
have been thinking of moving from MapTiler to OpenFreeMap for
StatusGator's outage maps.
hyperknot wrote 1 day ago:
Feel free to migrate. If you ever worry about High Availability,
self-hosting is always an option. But I'm working hard on making the
public instance as reliable as possible.
<- back to front page
You are viewing proxied material from codevoid.de. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.