#[1]Facebook Engineering » Feed [2]Facebook Engineering » Comments Feed
  [3]alternate [4]alternate [5]alternate

  fbpx

  [6]Skip to content

[7]Facebook Engineering

  Search this site ____________________ (BUTTON) [icon-search.svg]
  (BUTTON)
    * [8]Open Source
         + [9]Open Source
         + [10]Facebook Open Source
    * [11]Platforms
         + [12]Android
         + [13]iOS
         + [14]Web
    * [15]Infrastructure Systems
         + [16]Core Data
         + [17]Data Infrastructure
         + [18]DevInfra
         + [19]Production Engineering
         + [20]Security
    * [21]Physical Infrastructure
         + [22]Connectivity
         + [23]Data Center Engineering
         + [24]Networking & Traffic
    * [25]Video Engineering & AR/VR
         + [26]Video Engineering
         + [27]Virtual Reality
         + [28]Research Publications
    * [29]Artificial Intelligence
         + [30]ML Applications
         + [31]AI Research
         + [32]Research Publications
    * [33]Watch Videos

  ____________________ (BUTTON)

  POSTED ON AUGUST 11, 2021 TO [34]Open Source, [35]Production
  Engineering

  Open-sourcing a more precise time appliance
  Open sourcing a more precise time appliance
  By [36]Ahmad Byagowi, [37]Oleg Obleukhov
  (BUTTON)
  (BUTTON)
  (BUTTON)
    * Facebook engineers have built and open-sourced an Open Compute Time
      Appliance, an important component of the modern timing
      infrastructure.
    * To make this possible, we came up with the Time Card — a PCI
      Express (PCIe) card that can turn almost any commodity server into
      a time appliance.
    * With the help of the OCP community, we established the [38]Open
      Compute Time Appliance Project and open-sourced every aspect of the
      [39]Open Time Server.

  In March 2020, we announced that we were in the process of switching
  over the servers in our data centers (together with our consumer
  products) to [40]a new timekeeping service based on the Network Time
  Protocol (NTP). The new service, built in-house and later open-sourced,
  was more scalable and improved the accuracy of timekeeping in the
  Facebook infrastructure from 10 milliseconds to 100 microseconds. More
  accurate time keeping enables more advanced infrastructure management
  across our data centers, as well as faster performance of distributed
  databases.

  The new NTP-based time architecture uses a [41]Stratum 1 — an important
  component that is directly linked to an authoritative source of time,
  such as a global navigation satellite system (GNSS) or a cesium clock.

  Figure showing Stratums 0 through 3. Stratum 1 is an important
  component which is a device directly linked to an authoritative source
  of time, such as a Global Navigation Satellite System or a Cesium
  clock.

  Many companies rely on public NTP pools such as time.facebook.com to
  act as their Stratum 1. However, this approach has its drawbacks. These
  pools add dependency on internet connectivity and can impact overall
  security and reliability of the system. For instance, if connectivity
  is lost or an external service is down, it can result in outages or
  drift in timing for the dependent system.

  To remove these dependencies, we’ve built a new dedicated piece of
  hardware called Time Appliance, which consists of a GNSS receiver and a
  miniaturized atomic clock (MAC). Users of time appliances can keep
  accurate time, even in the event of GNSS connectivity loss. While
  building our Time Appliance, we also invented a Time Card, a PCIe card
  that can turn any commodity server into a time appliance.

Why do we need a new time device?

  Off-the-shelf time appliances have their own benefits. They work right
  out of the box and because many of these devices have been on the
  market for decades, they are battle-tested and generally stable enough
  to work without supervision for a long time.

  However, these solutions also come with trade-offs:
    * In most cases, they are outdated and often vulnerable to software
      security concerns. Feature requests and security fixes may take
      months or even years to implement.
    * These devices come with closed source software, which makes
      configuring and monitoring them limited and challenging. While
      configuration is done manually via a proprietary CLI or Web UI,
      monitoring often uses SNMP, a protocol that was not designed for
      this purpose.
    * They include proprietary hardware that is not user-serviceable.
      When a single component breaks, there is no easy way to replace it.
      You have to either ship it to the vendor for repair or buy an
      entire new appliance.
    * Since off-the-shelf devices are made in low quantities, they come
      with a higher markup and can become very costly to operate over
      time. The high cost associated with off-the-shelf devices create
      limitations for many in the industry. An open source version would
      open the door to broader applications.

  Until now, companies have had to accept these trade-offs and work
  within the constraints described above. We decided it was time to try
  something different, so we took a serious look at what it would take to
  build our new Time Appliance — specifically, one using the x86
  architecture.

Prototyping the Time Appliance

  Here’s a block diagram of what we envisioned:

  Block diagram of the time appliance prototype
  It all starts from a GNSS receiver that provides the time of day (ToD)
  as well as the one pulse per second (PPS). When the receiver is backed
  by a high-stability oscillator (e.g., an atomic clock or an
  oven-controlled crystal oscillator), it can provide time that is
  nanosecond-accurate. The time is delivered across the network via an
  off-the-shelf network card which supports PPS in/out and hardware time
  stamping of packets, such as the NVIDIA Mellanox ConnectX-6 Dx used in
  our initial appliance.

  The output of the GPS disciplined oscillator (GPSDO) was fed into the
  EXT time-stamping of the ConnectX-6 Dx network card. In addition, the
  GNSS receiver provides the ToD via a serial port and a popular GPS
  reporting protocol called NMEA. Using ts2phc tool allowed us to
  synchronize the physical hardware clock of the network interface
  controller (NIC) down to a couple of tens of nanoseconds, as shown
  below:

  block of code

  Our prototype gave us confidence that building such an appliance was
  possible. However, there was a lot of room for improvement.

  To increase the reliability of the system, we divided it into two major
  parts: payload and delivery. The payload is the precision time that is
  essentially an interpolation system driven by a local oscillator to
  create nanoseconds of time measurement between consecutive PPS signals
  received by the GNSS receiver. We considered putting the GNSS receiver,
  the high-stability local oscillator, and the necessary processing logic
  into a PCIe form factor, and we called it the Time Card.

  Block diagram of the time card functionality

  Here is the sketch of the Time Card we initially envisioned on a
  napkin:

  Original design for the time appliance

  We used an onboard MAC, a multiband GNSS receiver, and a
  field-programmable gate array (FPGA) to implement the time engine. The
  time engine’s job is to interpolate in nanoseconds the granularity
  required between consecutive PPS signals. The GNSS receiver also
  provides a ToD in addition to a 1 PPS signal. In the event of the loss
  of GNSS reception, the time engine relies on the ongoing
  synchronization of the atomic clock based on an average ensemble of the
  consecutive PPS pulses.

  The time engine consists of a set of processing blocks implemented on
  the FPGA of the Time Card. These processing blocks include various
  filtering, synchronization, error checking, time-stamping, and
  PCIe-related subsystems to allow the Time Card to perform as a system
  peripheral that provides precision time for the open time server.

  It should be noted that the accuracy of a GNSS receiver is within tens
  of nanoseconds, while the required ongoing synchronization
  (calibration) of the MAC is within 10 picoseconds (1,000 times more
  accurate).

  At first, this sounds impossible. However, the GNSS system provides
  timing based on continuous communication with standard time. This
  ability allows the GNSS onboard clock to be constantly synchronized
  with a source of time provided to its constellation, giving it
  virtually no long-term drifting error. Therefore, the MAC’s calibration
  is performed via a comparison of a MAC-driven counter and the
  GNSS-provided PPS pulse. Taking more time for the comparison allows us
  to achieve a higher precision of calibration for the MAC. Of course,
  this is with the consideration that the MAC is a linear time invariant
  system.

  In this block diagram, you can see a 10 MHz signal from the rubidium
  clock entering the time engine. The clock signal feeds into a digital
  clock module and a digital PLL, resulting in a 125 MHz frequency. The
  125 MHz feeds into the ToD unit.

  In this block diagram, you can see a 10 MHz signal from the rubidium
  clock entering the time engine. This clock signal can be replaced by a
  10 MHz SMA input. The clock signal feeds into a digital clock module
  and a digital PLL (12.5x resulted from 25 up and divided by 2),
  resulting in a 125 MHz frequency. The 125 MHz (8-nanosecond periods)
  feeds into the ToD unit.

  Block diagram showing the filtering process

  The ToD unit associates the 8-nanosecond increments in digital values
  of 0b000001 since the LSB (least significant bit) is associated to 250
  picoseconds (driven from 32 bits of subsecond accuracy on the gPTP).

  On the other hand, the PPS signal coming filtered from the GNSS is used
  to snapshot the result of the increments. If the 125 MHz is accurate,
  the accumulated increments should result in exactly 1-second intervals.
  However, in reality, there is always a mismatch between the accumulated
  value and a theoretical 1-second interval.

  The values can be adjusted using an internal PI (proportional and
  integral) control loop. The adjustment can be done by either altering
  the 0b000001 value by steps of 250 picoseconds or fine-tuning the 12.5x
  PPL. In addition, further (more finely tuned) adjustments can be
  applied by steering the rubidium oscillator.

  The longer a GNSS isn’t available, the more time accuracy is lost. The
  rate of the time accuracy deterioration is called holdover. Usually,
  holdover is described as a timeframe for accuracy and how long it takes
  to exceed it. For example, the holdover of a MAC is within 1
  microsecond for 24 hours. This means that after 24 hours, the time
  accuracy is nondeterministic but accurate within 1 microsecond.

  As an alternative approach, we are counting on the new generation of
  chip-scale and miniaturized atomic clocks with their capability to
  receive PPS inputs. This allows the time engine of the Time Card to
  hand off the ultraprecision syntonization of the high-stability
  oscillator to the component rather than use digital resources to reach
  the target.

  As a general principle, the more accurate the tuning, the better the
  holdover performance that can be achieved. In terms of delivery, using
  a NIC with precision timing ensures that network packets receive very
  accurate time stamps, which is critical for keeping the time precise as
  it is shared with other servers across the network. Such a NIC can also
  receive a PPS signal directly from the Time Card.

  After conceptualizing the idea and various implementation iterations,
  we were able to put together a prototype.

  Image of the time card prototype

The Time Appliance in action

  The Time Card allows any x86 machine with a NIC capable of hardware
  time-stamping to be turned into a time appliance. This system is
  agnostic to whether it runs for NTP, PTP, SyncE, or any other time
  synchronization protocol, since the accuracy and stability provided by
  the Time Card is sufficient for almost any system.

  Image showing the components to build a time appliance
  The beauty of using PCIe cards is that the setup can be assembled even
  on a home PC, as long as it has enough PCIe slots available.

  The next step would be to install Linux. The Time Card driver is
  included in Linux kernel [42]5.15 or newer. Or, it can be built from
  the[43] OCP GitHub repository on kernel 5.12 or newer.

  The driver will expose several devices, including the PHC clock, GNSS,
  PPS, and atomic clock serial:
$ ls -l /sys/class/timecard/ocp0/
lrwxrwxrwx. 1 root    0 Aug  3 19:49 device -> ../../../0000:04:00.0/
-r--r--r--. 1 root 4096 Aug  3 19:49 gnss_sync
lrwxrwxrwx. 1 root    0 Aug  3 19:49 i2c -> ../../xiic-i2c.1024/i2c-2/
lrwxrwxrwx. 1 root    0 Aug  3 19:49 pps -> ../../../../../virtual/pps/pps1/
lrwxrwxrwx. 1 root    0 Aug  3 19:49 ptp -> ../../ptp/ptp2/
lrwxrwxrwx. 1 root    0 Aug  3 19:49 ttyGNSS -> ../../tty/ttyS7/
lrwxrwxrwx. 1 root    0 Aug  3 19:49 ttyMAC -> ../../tty/ttyS8/

  The driver also allows us to monitor the Time Card, the GNSS receiver,
  and the atomic clock status and flash a new FPGA bitstream using the
  devlink cli.

  The only thing left to do is to configure the NTP and/or PTP server to
  use the Time Card as a reference clock. To configure chrony, one simply
  needs to specify refclock attribute:
$ grep refclock /etc/chrony.conf
refclock PHC /dev/ptp2 tai poll 0 trust

  And enjoy a very precise and stable NTP Stratum 1 server:
$ chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#* PHC0                          0   0   377     1     +4ns[   +4ns] +/-
36ns

  For the PTP server (for example, [44]ptp4u) one will first need to
  synchronize Time Card PHC with the NIC PHC. This can be easily done by
  using the phc2sys tool which will sync the clock values with the high
  precision usually staying within single digits of nanoseconds:
$ phc2sys -s /dev/ptp2 -c eth0 -O 0 -m

  For greater precision, it’s recommended to connect the Time Card and
  the NIC to the same CPU PCIe lane. For greater precision, one can
  connect the PPS output of the Time Card to the PPS input of the NIC.

  To validate and confirm the precision, we’ve used an external
  validation device called Calnex Sentinel connected to the same network
  via several switches and an independent GNSS antenna. It can perform
  PPS testing as well as NTP and/or PTP protocols:

  On this graph, the blue line represents NTP measurement results. The
  precision stays within ±40 microseconds throughout the 48-hour
  measurement interval. The orange line represents PTP measurement
  results. The offset is practically 0 ranging within nanoseconds range.

  The blue line represents NTP measurement results. The precision stays
  within ±40 microseconds throughout the 48-hour measurement interval.

  The orange line represents PTP measurement results. The offset is
  practically 0 ranging within nanoseconds range.

  Indeed, when we compare 1 PPS between Time Card output and the internal
  reference of the Calnex Sentinel, we see that the combined error ranges
  within ±200 nanoseconds:

  when we compare 1 PPS between Time Card output and the internal
  reference of the Calnex Sentinel, we see that the combined error ranges
  within ±200 nanoseconds


  But what’s even more important is that these measurements demonstrate
  stability of the Time Appliance outputs.

  In the event of the GNSS signal loss, we need to make sure the time
  drift (aka holdover) of the atomic-backed Time Card stays within 1
  microsecond per 24 hours. Here is a graph showing the holdover of the
  atomic clock (SA.53s) over a 24-hour interval. As you can see, the PPS
  drift stays within 300 nanoseconds, which is within the atomic clock
  spec.

  Here is a graph showing the holdover of the atomic clock (SA.53s) over
  a 24-hour interval. As you can see, the PPS drift stays within 300
  nanoseconds, which is within the atomic clock spec.

  The modular design of the Time Card allows the swap of the atomic clock
  with an oven-controlled crystal oscillator (OCXO) or a temperature
  compensated crystal oscillator (TCXO) for a budget solution with the
  compromise on the holdover capabilities.

Open-sourcing the design of the Time Appliance

  Building a device that is very precise, inexpensive, and free from
  vendor lock was an achievement on its own. But we wanted to have a
  greater impact on the industry. We wanted to truly set it free and make
  it open and affordable for everyone, from a research scientist to a
  large cloud data center. That’s why we engaged with the Open Compute
  Project (OCP) to create a brand-new [45]Time Appliance Project (TAP).
  Under the OCP umbrella, we open-sourced at the Time Appliance Project
  [46]GitHub repository, including the specs, schematics, mechanics, BOM,
  and the source code. Now, as long as printing the PCB and soldering
  tiny components does not sound scary, anyone can build their own Time
  Card for a fraction of the cost of a regular time appliance. We also
  worked with several vendors such as [47]Orolia who will be building and
  selling time cards, and NVIDIA who are selling the precision
  timing-capable ConnectX-6 Dx (and the precision timing-capable
  BlueField-2 DPU).

  We published an Open Time Server spec at [48]www.opentimeserver.com,
  which explains in great detail how to combine the hardware (Time Card,
  Network Card, and a commodity server) and the software (OS driver, NTP,
  and/or PTP server) to build the Time Appliance. Building an appliance
  based on this spec will give full control to the engineers maintaining
  the device, improving monitoring, configuration, management, and
  security.

  The Time Appliance is an important step in the journey to improve the
  timing infrastructure for everyone, but there is more to be done. We
  will continue to work on other elements, including improving the
  precision and accuracy of the synchronization of our own servers, and
  we intend to continue sharing this work with the Open Compute
  community.

  IFRAME:
  [49]https://www.facebook.com/plugins/like.php?href=https://engineering.
  fb.com/2021/08/11/open-source/time-appliance/&width=450&layout=standard
  &action=like&size=small&show_faces=false&share=true&height=35&appId=250
  208035192

Read More in Open Source

  [50]View All

  Open sourcing Winterfell: A STARK prover and verifier

  AUG 4, 2021
  [51]Open sourcing Winterfell: A STARK prover and verifier

  Full Sharded Data Parallel hero

  JUL 15, 2021
  [52]Fully Sharded Data Parallel: faster AI training with fewer GPUs

  Kats — a one-stop shop for time series analysis

  JUN 21, 2021
  [53]Meet Kats — a one-stop shop for time series analysis

  Rust Facebook Rustacean

  APR 29, 2021
  [54]A brief history of Rust at Facebook

  hsthrift - Open-sourcing Thrift for Haskell

  FEB 5, 2021
  [55]Open-sourcing Thrift for Haskell

  2Africa subsea cable: connectivity innovations, faster apps, and
  progress toward net zero

  DEC 30, 2020
  [56]2020 year in review: Connectivity innovations, faster apps, and
  progress toward net zero

Related Posts
    __________________________________________________________________

  Mar 18, 2020

Building a more accurate time service at Facebook scale

  Jun 21, 2021

Meet Kats — a one-stop shop for time series analysis

  Aug 04, 2021

Open sourcing Winterfell: A STARK prover and verifier

Related Positions
    __________________________________________________________________

    * Applied Research Manager (Mobile Vision Zurich)
      ZURICH, SWITZERLAND
    * Software Engineer - Technical Lead VR Shell Input (Text Input)
      SEATTLE, US
    * AR Calling Platform Engineer
      BURLINGAME, US
    * AR Calling Platform Engineer
      REMOTE, US
    * Software Engineer, Machine Learning
      LONDON, UNITED KINGDOM

  See All Jobs

Available Positions
    __________________________________________________________________

    * Applied Research Manager (Mobile Vision Zurich)
      ZURICH, SWITZERLAND
    * Software Engineer - Technical Lead VR Shell Input (Text Input)
      SEATTLE, US
    * AR Calling Platform Engineer
      BURLINGAME, US
    * AR Calling Platform Engineer
      REMOTE, US
    * Software Engineer, Machine Learning
      LONDON, UNITED KINGDOM

  See All Jobs

Stay Connected

    * footer-fb-engineering
      Facebook Engineering
      [57](BUTTON) Like
    * footer-fbopensource
      @fbOpenSource
      (BUTTON) Follow
    * footer-research
      Facebook Research
      [58](BUTTON) Like
    * footer-developers
      Facebook Developers
      [59](BUTTON) Like
    * footer-rss
      RSS
      [60](BUTTON) Subscribe

Open Source

  Facebook believes in building community through open source technology.
  Explore our latest projects in Artificial Intelligence, Data
  Infrastructure, Development Tools, Front End, Languages, Platforms,
  Security, Virtual Reality, and more.
    * android
      ANDROID
    * ios
      iOS
    * web
      WEB
    * backend
      BACKEND
    * hardware
      HARDWARE

  (BUTTON) Learn More

  Facebook © 2021
    * [61]About
    * [62]Careers
    * [63]Privacy
    * [64]Cookies
    * [65]Terms
    * [66]Help

  To help personalize content, tailor and measure ads, and provide a
  safer experience, we use cookies. By clicking or navigating the site,
  you agree to allow our collection of information on and off Facebook
  through cookies. Learn more, including about available controls:
  [67]Cookies Policy
  (BUTTON) I Agree

References

  Visible links
  1. https://engineering.fb.com/feed/
  2. https://engineering.fb.com/comments/feed/
  3. https://engineering.fb.com/wp-json/wp/v2/posts/17988
  4. https://engineering.fb.com/wp-json/oembed/1.0/embed?url=https://engineering.fb.com/2021/08/11/open-source/time-appliance/
  5. https://engineering.fb.com/wp-json/oembed/1.0/embed?url=https://engineering.fb.com/2021/08/11/open-source/time-appliance/&format=xml
  6. https://engineering.fb.com/2021/08/11/open-source/time-appliance/#content
  7. https://engineering.fb.com/
  8. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
  9. https://engineering.fb.com/category/open-source/
 10. https://opensource.facebook.com/
 11. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
 12. https://engineering.fb.com/category/android/
 13. https://engineering.fb.com/category/ios/
 14. https://engineering.fb.com/category/web/
 15. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
 16. https://engineering.fb.com/category/core-data/
 17. https://engineering.fb.com/category/data-infrastructure/
 18. https://engineering.fb.com/category/developer-tools/
 19. https://engineering.fb.com/category/production-engineering/
 20. https://engineering.fb.com/category/security/
 21. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
 22. https://engineering.fb.com/category/connectivity/
 23. https://engineering.fb.com/category/data-center-engineering/
 24. https://engineering.fb.com/category/networking-traffic/
 25. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
 26. https://engineering.fb.com/category/video-engineering/
 27. https://engineering.fb.com/category/virtual-reality/
 28. https://research.fb.com/publications/?cat=11
 29. https://engineering.fb.com/2021/08/11/open-source/time-appliance/
 30. https://engineering.fb.com/category/ml-applications/
 31. https://engineering.fb.com/category/ai-research/
 32. https://research.fb.com/publications/?cat=13
 33. https://engineering.fb.com/videos
 34. https://engineering.fb.com/category/open-source/
 35. https://engineering.fb.com/category/production-engineering/
 36. https://engineering.fb.com/author/ahmad-byagowi/
 37. https://engineering.fb.com/author/oleg-obleukhov/
 38. http://ocptap.com/
 39. http://opentimeserver.com/
 40. https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/
 41. https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/
 42. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=773bda96492153e11d21eb63ac814669b51fc701
 43. https://github.com/opencomputeproject/Time-Appliance-Project/tree/master/Time-Card/DRV
 44. https://github.com/facebookincubator/ptp
 45. http://www.ocptap.com/
 46. https://github.com/opencomputeproject/Time-Appliance-Project
 47. https://www.orolia.com/about-the-atomic-reference-time-card-art-card/
 48. http://www.opentimeserver.com/
 49. https://www.facebook.com/plugins/like.php?href=https://engineering.fb.com/2021/08/11/open-source/time-appliance/&width=450&layout=standard&action=like&size=small&show_faces=false&share=true&height=35&appId=250208035192
 50. https://engineering.fb.com/category/open-source/
 51. https://engineering.fb.com/2021/08/04/open-source/winterfell/
 52. https://engineering.fb.com/2021/07/15/open-source/fsdp/
 53. https://engineering.fb.com/2021/06/21/open-source/kats/
 54. https://engineering.fb.com/2021/04/29/developer-tools/rust/
 55. https://engineering.fb.com/2021/02/05/open-source/hsthrift/
 56. https://engineering.fb.com/2020/12/30/connectivity/2020-year-in-review/
 57. https://www.facebook.com/Engineering/
 58. https://research.fb.com/
 59. https://developers.facebook.com/
 60. https://code.facebook.com/posts/rss/
 61. https://www.facebook.com/facebook
 62. https://www.facebook.com/careers
 63. https://www.facebook.com/about/privacy
 64. https://www.facebook.com/help/cookies
 65. https://www.facebook.com/policies
 66. https://www.facebook.com/help
 67. https://www.facebook.com/policies/cookies/

  Hidden links:
 69. https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/
 70. https://engineering.fb.com/2021/06/21/open-source/kats/
 71. https://engineering.fb.com/2021/08/04/open-source/winterfell/
 72. https://www.facebook.com/careers/v2/jobs/947806502396127/
 73. https://www.facebook.com/careers/v2/jobs/147575507242479/
 74. https://www.facebook.com/careers/v2/jobs/272972701136499/
 75. https://www.facebook.com/careers/v2/jobs/272972701136499/
 76. https://www.facebook.com/careers/v2/jobs/253856829167430/
 77. https://www.facebook.com/careers/
 78. https://www.facebook.com/careers/v2/jobs/947806502396127/
 79. https://www.facebook.com/careers/v2/jobs/147575507242479/
 80. https://www.facebook.com/careers/v2/jobs/272972701136499/
 81. https://www.facebook.com/careers/v2/jobs/272972701136499/
 82. https://www.facebook.com/careers/v2/jobs/253856829167430/
 83. https://www.facebook.com/careers/

You are viewing proxied material from 1436.ninja. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.