 |
[HN Gopher] Exa-d: How to store the web in S3 |
|
___________________________________________________________________ |
|
|
|
Exa-d: How to store the web in S3 |
|
|
|
exa-d is our internal data processing framework that stores the web |
|
in S3. It helps deal with the complexity of data at (web) scale |
|
using specific design decisions like declarative typed dependencies |
|
and enabling sparse updates. |
|
|
|
Author : willbryk |
|
Score : 30 points |
|
Date : 2026-01-14 01:13 UTC (6 hours ago) |
|
|
 |
web link (exa.ai) |
 |
w3m dump (exa.ai) |
|
|
|
| swyx wrote: |
|
| hi will! super nicely written, nice look under the hood of your |
|
| processing. as an orchestration guy i always wondered why |
|
| everyone seems to converge on using Ray, and as a secondary |
|
| thought, how well is Anyscale capturing the Ray market. |
|
| |
|
| if i were doing what you do i might set up a lot of rate |
|
| limits/anomaly detection in case some weird unintended |
|
| invalidation causes a weird spike in your dependency graphs. is |
|
| there good practice there for anomaly detection other than "setup |
|
| a bunhc of dashboards and be on call"? |
|
|
|
___________________________________________________________________ |
|
(page generated 2026-01-14 08:00 UTC) |
