 |
people.gnome.org_federico.atom.xml - sfeed_tests - sfeed tests and RSS and Atom… |
 |
git clone git://git.codemadness.org/sfeed_tests |
 |
Log |
|
Files |
|
Refs |
|
README |
|
LICENSE |
|
--- |
|
people.gnome.org_federico.atom.xml (1402111B) |
|
--- |
|
1 <?xml version="1.0" encoding="utf-8"?> |
|
2 <feed xmlns="http://www.w3.org/2005/Atom"><title>Federico's Blog</title>… |
|
3 terms. Technically there are a lot of incomplete moving parts; |
|
4 socially there is a lot of missing documentation to be written, a lot |
|
5 of miscommunication and mismatched expectations.</p> |
|
6 <p>The following is a brief and incomplete, but hopefully encourag… |
|
7 summary …</p></summary><content type="html"><p>"Themes in … |
|
8 terms. Technically there are a lot of incomplete moving parts; |
|
9 socially there is a lot of missing documentation to be written, a lot |
|
10 of miscommunication and mismatched expectations.</p> |
|
11 <p>The following is a brief and incomplete, but hopefully encourag… |
|
12 summary of the status of themes in GNOME. I want to give you an |
|
13 overall picture of the status of things, and more importantly, an idea |
|
14 of how you can help. This is not a problem that can be solved by a |
|
15 small team of platform developers.</p> |
|
16 <p>I wish to thank Alexander Mikhaylenko for providing most of the |
|
17 knowledge in this post.</p> |
|
18 <h1>Frame of reference</h1> |
|
19 <p>First, I urge you to read Cassidy James Blaede's comprehensiv… |
|
20 Need for a FreeDesktop Dark Style |
|
21 Preference</a>". |
|
22 That gives an excellent, well-researched introduction to the "dark |
|
23 style" problem, the status quo on other platforms, and exploratory |
|
24 plans for GNOME and Elementary from 2019.</p> |
|
25 <p>Go ahead, read it. It's very good.</p> |
|
26 <p>There is also a <a href="https://www.youtube.com/watch?v=gi_… |
|
27 research</a> |
|
28 if you prefer to watch a video.</p> |
|
29 <p>Two key take-aways from this: First, about this being a |
|
30 <strong>preference</strong>, not a system-enforced setting:&… |
|
31 <blockquote> |
|
32 <p>I’m explicitly using the language “Dark Style Preference”… |
|
33 reason! As you’ll read further on, it’s important that this is |
|
34 treated as a user “preference,” not an explicit “mode” or |
|
35 strictly-enforced “setting.” It’s also not a “theme” in the se… |
|
36 that it just swaps out some assets, but is a way for the OS to |
|
37 support a user expressing a preference, and apps to respond to that |
|
38 preference.</p> |
|
39 </blockquote> |
|
40 <p>Second, about the <strong>accessibility</strong> im… |
|
41 <blockquote> |
|
42 <p>Clearly there’s an accessibility and usability angle here. An… |
|
43 with other accessibility efforts, it’s important to not relegate a |
|
44 dark style preference to a buried “Universal Access” or |
|
45 “Accessibility” feature, as that makes it less discoverable, less |
|
46 tested, and less likely to be used by folks who could greatly |
|
47 benefit, but don’t consider themselves “disabled.”</p> |
|
48 </blockquote> |
|
49 <h1>Libadwaita and the rest of the ecosystem</h1> |
|
50 <p>Read the <a href="https://discourse.gnome.org/t/libadwaita-1… |
|
51 roadmap</a>; |
|
52 it is very short, but links to very interesting issues on gitlab.</p&… |
|
53 <p>For example, this merge request is for an <a href="https://g… |
|
54 and high-contrast |
|
55 preferences</a>. |
|
56 It has links to pending work in other parts of the platform: libhandy, |
|
57 gsettings schemas, portals so that containerized applications can |
|
58 query those preferences.</p> |
|
59 <p>As far as I understand it, applications that just use GTK3 or l… |
|
60 can opt in to supporting the dark style preference — it is opt-in |
|
61 because doing that unconditionally in GTK/libhandy right now would |
|
62 break existing applications.. If your app uses libadwaita, it is |
|
63 assumed that you have opted into supporting that preference, since |
|
64 libadwaita's widgets already make that assumption, and it is not |
|
65 API-stable yet — so it can make that assumption from the beginning.<… |
|
66 <p>There is discussion of the accessibility implications in <a … |
|
67 mockups</a>.</p> |
|
68 <h1>CSS parity across implementations</h1> |
|
69 <p>In GNOME we have three implementations of CSS:</p> |
|
70 <ul> |
|
71 <li> |
|
72 <p>librsvg uses servo's engine for CSS selector matching, and micr… |
|
73 </li> |
|
74 <li> |
|
75 <p>GTK has its own CSS parser and processor.</p> |
|
76 </li> |
|
77 <li> |
|
78 <p>Gnome-shell uses an embedded version of libcroco for parsing, b… |
|
79 does most of the selector matching and cascading with gnome-shell's |
|
80 own Shell Toolkit code.</p> |
|
81 </li> |
|
82 </ul> |
|
83 <p>None of those implementations supports <code>@media</c… |
|
84 properties with <code>var()</code>. That is, unlike in the … |
|
85 applications cannot have this in their CSS:</p> |
|
86 <div class="highlight"><pre><span></span><cod… |
|
87 <span class="c">/* styles for dark style */</span> |
|
88 <span class="p">}</span> |
|
89 |
|
90 <span class="p">@</span><span class="k">media</span… |
|
91 <span class="c">/* styles for light style */</span> |
|
92 <span class="p">}</span> |
|
93 </code></pre></div> |
|
94 |
|
95 <p>Or even declaring colors in a civilized fashion:</p> |
|
96 <div class="highlight"><pre><span></span><cod… |
|
97 <span class="nv">--main-bg-color</span><span class="p"&… |
|
98 <span class="p">}</span> |
|
99 |
|
100 <span class="nt">some_widget</span> <span class="p">{&… |
|
101 <span class="k">background-color</span><span class="p"&… |
|
102 <span class="p">}</span> |
|
103 </code></pre></div> |
|
104 |
|
105 <p>Or combining the two:</p> |
|
106 <div class="highlight"><pre><span></span><cod… |
|
107 <span class="p">:</span><span class="nd">root</sp… |
|
108 <span class="nv">--main-bg-color</span><span class="p… |
|
109 <span class="nv">--main-fg-color</span><span class="p… |
|
110 <span class="p">}</span> |
|
111 <span class="p">}</span> |
|
112 |
|
113 <span class="p">@</span><span class="k">media</span… |
|
114 <span class="p">:</span><span class="nd">root</sp… |
|
115 <span class="nv">--main-bg-color</span><span class="p… |
|
116 <span class="nv">--main-fg-color</span><span class="p… |
|
117 <span class="p">}</span> |
|
118 <span class="p">}</span> |
|
119 |
|
120 <span class="nt">some_widget</span> <span class="p">{&… |
|
121 <span class="k">background-color</span><span class="p"&… |
|
122 <span class="p">}</span> |
|
123 </code></pre></div> |
|
124 |
|
125 <p>Boom. I think this would remove some workarounds we have right… |
|
126 <ul> |
|
127 <li> |
|
128 <p>Just like GTK, libadwaita generates four variants of the system… |
|
129 stylesheet using scss (regular, dark, high-contrast, |
|
130 high-contrast-dark). This would be obviated with <a href="https://… |
|
131 queries</a> |
|
132 for <code>prefers-color-scheme</code>, <code>prefers… |
|
133 in the web platform.</p> |
|
134 </li> |
|
135 <li> |
|
136 <p>GTK has a custom <code>@define-color</code> keyword… |
|
137 nor librsvg support that. This would be obviated with <a href="htt… |
|
138 properties</a> - |
|
139 the <code>var()</code> mechanism. (I don't know if some "… |
|
140 would be better done as |
|
141 <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/env()">… |
|
142 but none of the three implementations support that, either.)</p> |
|
143 </li> |
|
144 </ul> |
|
145 <h1>Accent colors</h1> |
|
146 <p>They are currently implemented with GTK's <code>@define-c… |
|
147 not ideal if the colors have to trickle down from GTK to SVG icons, |
|
148 since librsvg doesn't do <code>@define-color</code> - it wou… |
|
149 <a href="https://gitlab.gnome.org/GNOME/librsvg/-/issues/459"><… |
|
150 instead</a>.</p> |
|
151 <p>Of course, gnome-shell's libcroco doesn't do <code>@defin… |
|
152 <p>Look for <code>@accent_color</code>, <code>@a… |
|
153 stylesheet</a>, |
|
154 or better yet, <strong>write documentation!</strong></p&g… |
|
155 <p>The default style:</p> |
|
156 <p><img alt="Default blue style" src="https://people.gnome.org/… |
|
157 <p>Accent color set to orange (e.g. tweak it in GTK's CSS inspecto… |
|
158 <p><img alt="Orange accents for widgets" src="https://people.gn… |
|
159 <div class="highlight"><pre><span></span><cod… |
|
160 <span class="p">@</span><span class="k">define-color&l… |
|
161 |
|
162 <span class="c">/* background+text pair */</span> |
|
163 <span class="p">@</span><span class="k">define-color&l… |
|
164 <span class="p">@</span><span class="k">define-color&l… |
|
165 </code></pre></div> |
|
166 |
|
167 <h1>Custom widgets</h1> |
|
168 <p>Again, your app's custom stylesheet for its custom widgets can … |
|
169 colors defined through <code>@define-color</code> from the s… |
|
170 <h1>Recoloring styles</h1> |
|
171 <p>You will be able to do this after it gets merged into the main … |
|
172 e.g. recolor everything to sepia:</p> |
|
173 <p><img alt="Adwaita recolored to sepia" src="https://people.gn… |
|
174 <div class="highlight"><pre><span></span><cod… |
|
175 <span class="p">@</span><span class="k">define-color&l… |
|
176 |
|
177 <span class="p">@</span><span class="k">define-color&l… |
|
178 <span class="p">@</span><span class="k">define-color&l… |
|
179 |
|
180 <span class="p">@</span><span class="k">define-color&l… |
|
181 |
|
182 <span class="p">@</span><span class="k">define-color&l… |
|
183 <span class="p">@</span><span class="k">define-color&l… |
|
184 </code></pre></div> |
|
185 |
|
186 <p>Of course <code>shade()</code> is not web-platform … |
|
187 it, or redo it by implementing <a href="https://developer.mozilla.org… |
|
188 function</a> for |
|
189 color values.</p> |
|
190 <h1>Recoloring icons</h1> |
|
191 <p>Currently GTK takes some defined colors and <a href="https:/… |
|
192 inject into SVG for |
|
193 icons</a>. |
|
194 This has <a href="https://gitlab.gnome.org/GNOME/gtk/-/issues/2314"&g… |
|
195 problems</a>.</p> |
|
196 <p>There is also some discussion about <a href="https://gitlab.… |
|
197 icons</a> across |
|
198 desktop environments.</p> |
|
199 <h1>How you can help</h1> |
|
200 <p>Implement support for <a href="https://developer.mozilla.org… |
|
201 queries</a> |
|
202 in our three CSS implementations (librsvg, gnome-shell, GTK). Decide |
|
203 how CSS media features like <code>prefers-color-scheme</code>… |
|
204 <code>prefers-contrast</code>, <code>inverted-colors&l… |
|
205 themes and accessibility, and decide if we should use them for |
|
206 familiarity with the web platform, or if we need media features with |
|
207 different names.</p> |
|
208 <p>Implement support for <a href="https://developer.mozilla.org… |
|
209 <code>var()</code></a> |
|
210 in our three CSS implementations. Decide if we should replace the |
|
211 current <code>@define-color</code> with that (note that <… |
|
212 in GTK, but not in librsvg or gnome-shell).</p> |
|
213 <p>See the <a href="https://discourse.gnome.org/t/libadwaita-1-… |
|
214 roadmap</a> |
|
215 and help out!</p> |
|
216 <p>Port applications to use the proposed APIs for querying the dar… |
|
217 preference. There are a bunch of hacky ways of doing it right now; |
|
218 they need to be migrated to the new system.</p> |
|
219 <p>Personally I would love help with finishing to <a href="http… |
|
220 styles to |
|
221 Rust</a> - |
|
222 this is part of unifying librsvg's and gnome-shell's CSS machinery.</… |
|
223 Mark Wielaard. In 2019 I started maintaining an <a href="https://gi… |
|
224 repository in GitLab</a>, with the intention of updating |
|
225 the build system and starting a Rust port of bzip2. Unfortunately I |
|
226 have left this project slip by.</p> |
|
227 <p>The new maintainer of the <a href="https://gitlab.com/bzip2/… |
|
228 Mark Wielaard. In 2019 I started maintaining an <a href="https://gi… |
|
229 repository in GitLab</a>, with the intention of updating |
|
230 the build system and starting a Rust port of bzip2. Unfortunately I |
|
231 have left this project slip by.</p> |
|
232 <p>The new maintainer of the <a href="https://gitlab.com/bzip2/… |
|
233 Bzip2 is Micah Snyder. Thanks, Micah, for picking it up!</p></co… |
|
234 code</a>. Around the same time, Linux distributions |
|
235 started shipping the first versions of Firefox that also required |
|
236 Rust. I unashamedly wanted to ride that wave: distros would <em>h… |
|
237 integrate a new language in their build infrastructure, or they would �… |
|
238 code</a>. Around the same time, Linux distributions |
|
239 started shipping the first versions of Firefox that also required |
|
240 Rust. I unashamedly wanted to ride that wave: distros would <em>h… |
|
241 integrate a new language in their build infrastructure, or they would |
|
242 be left without Firefox. I was hoping that having a working Rust |
|
243 toolchain would make it easier for the rustified librsvg to get into |
|
244 distros.</p> |
|
245 <p>Two years after that, <a href="https://lwn.net/Articles/7713… |
|
246 that this made it hard or impossible to build librsvg (and all the |
|
247 software that depends on it, which is A Lot) on all the architectures |
|
248 that Debian builds on — specifically, on things like HP PA-RISC or |
|
249 Alpha, which <a href="https://www.debian.org/ports/">even Debian m… |
|
250 <p>Recently there was a similar kerfuffle, this time from <a hr… |
|
251 Gentoo</a>, specifically about how Python's cryptography |
|
252 package now requires Rust. So, it doesn't build for platforms that |
|
253 Rust/LLVM don't support, like hppa, alpha, and Itanium. It also |
|
254 doesn't build for platforms for which there are no Rust packages from |
|
255 Gentoo yet (mips, s390x, riscv among them).</p> |
|
256 <h2>Memories of discontinued architectures</h2> |
|
257 <p>Let me reminisce about a couple of discontinued architectures. … |
|
258 reading <a href="https://en.wikipedia.org/wiki/DEC_Alpha">Wikipedi… |
|
259 developed in 2001, and HP, who purchased Compaq, who purchased DEC, |
|
260 stopped selling Alpha systems in 2007. Notably, Compaq phased out the |
|
261 Alpha in favor of the Itanium, which stopped being developed in 2017.<… |
|
262 <p>I <em>used</em> an Alpha machine in 1997-1998, back… |
|
263 <a href="https://twitter.com/migueldeicaza/">Miguel</a> kind… |
|
264 where he worked, and the computer lab there got an Alpha box to let |
|
265 the scientists run mathematical models on a machine with really fast |
|
266 floating-point. This was a time when people actually regularly ssh'ed |
|
267 into machines to run X11 applications remotely — in their case, I |
|
268 think it was Matlab and Mathematica. Good times.</p> |
|
269 <p>The Alpha had fast floating point, much faster than Intel x86 C… |
|
270 and I was delighted to do graphics work on it. That was the first |
|
271 64-bit machine I used, and it let me learn how to fix code that only |
|
272 assumed 32 bits. It had a really picky floating-point unit. Whereas |
|
273 x86 would happily throw you a NaN if you used uninitialized memory as |
|
274 floats, the Alpha would properly fault and crash the program. I fixed |
|
275 so many bugs thanks to that!</p> |
|
276 <p>I also have fond memories of the 32-bit SPARC |
|
277 boxes at the University and their flat-screen fixed-frequency CRT |
|
278 displays, but you know, I haven't <em>seen</em> one of those… |
|
279 1998. Because I was doing graphics work, I used the single SPARC |
|
280 machine in the computer lab at the Institute that had 24-bit graphics, |
|
281 with a humongous 21" CRT display. PCs at the time still had 8-bit video |
|
282 cards and shitty little monitors.</p> |
|
283 <p>At about the same time that the Institute got its Alpha, it als… |
|
284 one of the first 64-bit UltraSPARCs from Sun — a very expensive |
|
285 machine definitely not targeted to hobbyists. I think it had two CPUs! |
|
286 Multicore did not exist!</p> |
|
287 <p>I think I saw a single Itanium machine in my life, probably aro… |
|
288 2002-2005. The Ximian/Novell office in Mexico City got one, for QA |
|
289 purposes — an incredibly loud and unstable machine. I don't think we |
|
290 ever did any actual development on that box; it was a "can you |
|
291 reproduce this bug there" kind of thing. I think Ximian/Novell had a |
|
292 contract with HP to test the distro there, I don't remember.</p> |
|
293 <h2>Unsupported architectures at the LLVM level</h2> |
|
294 <p>Platforms like the Alpha and Itanium that Rust/LLVM don't suppo… |
|
295 those platforms are dead in the water. The compiler cannot target |
|
296 them, as Rust generates machine code via LLVM, and LLVM doesn't |
|
297 support them.</p> |
|
298 <p>I don't know why distributions maintained by volunteers give |
|
299 themselves the responsibility to keep their software running on |
|
300 platforms that have not been manufactured for years, and that were |
|
301 never even hobbyist machines.</p> |
|
302 <p>I read the other day, and now I regret not keeping the link, so… |
|
303 like this: don't assume that your hobby computing entitles you to free |
|
304 labor on the part of compiler writers, software maintainers, and |
|
305 distro volunteers. (If someone helps me find the source, I'll happily |
|
306 link to it and quote it properly.)</p> |
|
307 <h2>Non-tier-1 platforms and "$distro does not build Rust there ye… |
|
308 <p>I think people are discovering these once again:</p> |
|
309 <ul> |
|
310 <li> |
|
311 <p>Writing and supporting a compiler for a certain architecture ta… |
|
312 </li> |
|
313 <li> |
|
314 <p>Supporting a distro for a certain architecture takes Real Work.… |
|
315 </li> |
|
316 <li> |
|
317 <p>Fixing software to work on a certain architecture takes Real Wo… |
|
318 </li> |
|
319 </ul> |
|
320 <p>Rust divides its support for different platforms into <a hre… |
|
321 from tier 1, the most supported, to tier 3, the least supported. Or, |
|
322 I should say, <em>taken care of</em>, which is a combination… |
|
323 actually have the hardware in question, and whether the general CI and |
|
324 build tooling is prepared to deal with them as effectively as it does |
|
325 for tier 1 platforms.</p> |
|
326 <p>In other words: there are more people capable of paying attenti… |
|
327 testing things on, x86_64 PCs than there are for |
|
328 <code>sparc-unknown-linux-gnu</code>.</p> |
|
329 <h2>Some anecdotes from Suse</h2> |
|
330 <p>At Suse we actually support IBM's s390x big iron; those mainfr… |
|
331 Suse Linux Enterprise Server. You have to pay a lot of money to get a |
|
332 machine like that and support for it. It's a room-sized beast that |
|
333 requires professional babysitting.</p> |
|
334 <p>When librsvg and Firefox started getting rustified, there was of |
|
335 course concern about getting Rust to work properly on the s390x. |
|
336 I worked sporadically with the people who made the distro work there, |
|
337 and who had to deal with building Rust and Firefox on it (librsvg |
|
338 was a non-issue after getting Rust and Firefox to work).</p> |
|
339 <p>I think all the LLVM work for the s390x was done at IBM. There… |
|
340 probably a couple of miscompilations that affected Firefox; they got fix… |
|
341 <p>One would expect bugs in software for IBM mainframes to be fixe… |
|
342 IBM or its contractors, not by volunteers maintaining a distro in |
|
343 their spare time.</p> |
|
344 <p>Giving computing time on mainframes to volunteers in distros co… |
|
345 like a good samaritan move, or a trap to extract free labor from |
|
346 unsuspecting people.</p> |
|
347 <h3>Endianness bugs</h3> |
|
348 <p>Firefox's problems on the s390x were more around big-endian bug… |
|
349 anything. You see, all the common architectures these days (x86_64 |
|
350 and arm64) are little-endian. However, s390x is <a href="https://en.… |
|
351 which means that all multi-byte numbers in memory are stored backwards |
|
352 from what most software expects.</p> |
|
353 <p>It is not a problem to write software that assumes little-endia… |
|
354 big-endian all the time, but it takes a little care to write software |
|
355 that works on either.</p> |
|
356 <p>Most of the software that volunteers and paid people write assu… |
|
357 little-endian CPUs, because that is likely what they are targeting. |
|
358 It is a pain in the ass to encounter code that works incorrectly on |
|
359 big-endian — a pain because <em>knowing where to look</em>… |
|
360 bugs is tricky, and <em>fixing existing code</em> to work wi… |
|
361 endianness can be either very simple, or a major adventure in |
|
362 refactoring and testing.</p> |
|
363 <p>Two cases in point:</p> |
|
364 <p><strong>Firefox.</strong> When Suse started dealin… |
|
365 s390x, there were endianness bugs in the graphics code in Firefox that |
|
366 deals with pixel formats. Whether pixels get stored in memory as |
|
367 ARGB/ABGR/RGBA/etc. is a platform-specific thing, and is generally a |
|
368 combination of the graphics hardware for that platform, plus the |
|
369 actual CPU architecture. At that time, it looked like the C++ code in |
|
370 Firefox that deals with pixels had been rewritten/refactored, and had |
|
371 lost big-endian support along the way. I don't know the current |
|
372 status (not a single big-endian CPU in my vincinity), but I haven't |
|
373 seen related bugs come in the Suse bug tracker? Maybe it's fixed now?&l… |
|
374 <p><strong>Librsvg</strong> had <a href="https://gi… |
|
375 big-endian</a>. One was in the old code for SVG |
|
376 filter effects that was written in C; <a href="https://gitlab.gnome.o… |
|
377 initial port to Rust inherited the same bug (think of a line-by-line |
|
378 port, althought it wasn't exactly like that), but it got fixed when my |
|
379 Summer of Code intern Ivan Molodetskikh refactored the code to have a |
|
380 <code>Pixel</code> abstraction that works for little-endian … |
|
381 wraps Cairo's funky requirements.</p> |
|
382 <p>The other endian-related bug in librsvg was when <a href="ht… |
|
383 masks</a>. Again, a little refactoring with that <code>Pixe… |
|
384 abstraction fixed it.</p> |
|
385 <p>I knew that the original C code for SVG filter effects didn't w… |
|
386 big-endian. But even back then, at Suse we never got |
|
387 reports of it producing incorrect results on the s390x... maybe people d… |
|
388 their mainframes to run <code>rsvg-convert</code>? I was ho… |
|
389 Rust of that code would automatically fix that bug, and it kind of |
|
390 happened that way through Ivan's careful work.</p> |
|
391 <p>And the code for masks? There were two bugs reported with that… |
|
392 root cause: <a href="https://gitlab.gnome.org/GNOME/librsvg/-/issues/… |
|
393 Debian</a> as a |
|
394 failure in librsvg's test suite (yay, it caught that bug!), and one |
|
395 from <a href="https://gitlab.gnome.org/GNOME/librsvg/-/issues/322">… |
|
396 G4</a> with a MATE |
|
397 desktop and seeing incorrectly-rendered SVG icons.</p> |
|
398 <p>And you know what? I am <strong>delighted</strong>… |
|
399 those lovely machines alive. A laptop that doesn't get warm enough to |
|
400 burn your thighs, what a concept. A perfectly serviceable 32-bit |
|
401 laptop with a maximum of about 1 GB of RAM and a 40 GB hard drive (it |
|
402 didn't have HDMI!)... But you know, it's the same kind of delight I |
|
403 feel when people talk about doing film photography on a |
|
404 <a href="https://en.wikipedia.org/wiki/Rollei_35">Rollei 35</a… |
|
405 nostalgia for hardware of days past, and a lot of mixed feelings about |
|
406 not throwing out working things and creating more trash.</p> |
|
407 <p>As a graphics programmer I feel the responsibility to write cod… |
|
408 works on little-endian and big-endian, but you know, it's not exactly |
|
409 an everyday concern anymore. The last big-endian machine I used on an |
|
410 everyday basis was the SPARCs in the university, more than 20 years |
|
411 ago.</p> |
|
412 <h3>Who gets paid to fix this?</h3> |
|
413 <p>That's the question. Suse got paid to support Firefox on the s… |
|
414 suppose IBM has an interest in fixing LLVM there; both actually have |
|
415 people and hardware and money to that effect.</p> |
|
416 <p>Within Suse, I am by default responsible for keeping librsvg |
|
417 working for the s390x as well — it gets built as part of the distro, |
|
418 after all. I have never gotten an endianness bug report from the Suse |
|
419 side of things.</p> |
|
420 <p>Which leads me to suspect that, probably similar to Debian and … |
|
421 we <em>build</em> a lot of software because it's in the buil… |
|
422 don't <em>run</em> it to its fullest extent. Do people run … |
|
423 s390x virtual machines? Maybe? Did they not notice endianness bugs |
|
424 <strong>because they were not in the code path that most GNOME ico… |
|
425 actually use</strong>? Who knows!</p> |
|
426 <p>I'm thankful to Simon from the Debian bug for pointing out |
|
427 the failure in librsvg's test case for masks, and to Mingcong for |
|
428 actually showing a screenshot of a MATE desktop running on a PPC |
|
429 PowerBook. Those were useful things for them to do.</p> |
|
430 <p>Also — they were kind about it. It was a pleasure to interac… |
|
431 <p>The librsvg 2.40.x series is the last "C only" version of the l… |
|
432 it was deprecated in 2017.</p> |
|
433 <p>During the port to Rust, I rewrote the path parser to be |
|
434 spec-compliant, and …</p></summary><content type="html"><p>… |
|
435 <p>The librsvg 2.40.x series is the last "C only" version of the l… |
|
436 it was deprecated in 2017.</p> |
|
437 <p>During the port to Rust, I rewrote the path parser to be |
|
438 spec-compliant, and fixed a few cases that the C version did not |
|
439 handle. One of this cases is for compact Arc data.</p> |
|
440 <p>The <a href="https://www.w3.org/TR/SVG11/paths.html#PathData… |
|
441 one to remove whitespace between numbers if the next number starts |
|
442 with a sign. For example, <code>23-45</code> gets parsed as… |
|
443 -45</code>.</p> |
|
444 <p>In addition, the arguments of the Arc commands have two flags i… |
|
445 middle of a bunch of numbers. The flags can be <code>0</code&g… |
|
446 may be no whitespace between the flags and the next number. For |
|
447 example, <code>A1.98 1.98 0 0015 13.96</code> gets parsed as… |
|
448 13.96</code> — note the two <code>0 0</code> flags b… |
|
449 <p>Librsvg 2.40.x does not parse this correctly. |
|
450 Adwaita-icon-theme-3.36, and possibly earlier versions, uses minimized |
|
451 SVG files with compressed whitespace, and will not render correctly |
|
452 with the C-only version of librsvg.</p> |
|
453 <p>This is <code>help-contents-symbolic.svg</code> ren… |
|
454 <p><img alt="icon rendered incorrectly" src="https://people.gno… |
|
455 <p>And this is <code>help-contents-symbolic.svg</code>… |
|
456 <p><img alt="icon rendered correctly" src="https://people.gnome… |
|
457 <p>This is not the only icon with compact Arc commands; there are … |
|
458 others that will also be mis-rendered in 2.40.x.</p> |
|
459 <p>I don't know when Adwaita started using SVGs with compressed |
|
460 whitespace; probably it didn't when librsvg 2.40.x was the latest |
|
461 version, or everyone would have noticed mis-rendered icons.</p> |
|
462 <p><strong>Background:</strong> Someone recently filed… |
|
463 <a href="https://gitlab.gnome.org/GNOME/librsvg/-/issues/654">bug&… |
|
464 memory unsafety in librsvg 2.40.x's path parser, which mysteriously |
|
465 enough only manifests itself in big-endian platforms. I wouldn't be |
|
466 surprised if this had latent bugs on little-endian as well.</p> |
|
467 <p>Please use at least librsvg 2.48.x; any earlier versions are n… |
|
468 supported. Generally I keep an eye on the last two stable release |
|
469 sets (2.48.x and 2.50.x as of this writing), but only commit fixes to |
|
470 the latest stable series (2.50.x currently).</p></content><categor… |
|
471 <h2>Changes to continuous integration</h2> |
|
472 <p>Some days ago, <a href="https://gitlab.gnome.org/GNOME/librs… |
|
473 scripts</a> to be much faster. A complete pipeline used to take |
|
474 about 90 minutes to run, now it takes about 15 minutes on average.<… |
|
475 <p><img alt="Graph with pipeline timings, which shrink drastica… |
|
476 <p>The <a href="https://gitlab.gnome.org/GNOME/librsvg/-/merge_… |
|
477 <h2>Changes to continuous integration</h2> |
|
478 <p>Some days ago, <a href="https://gitlab.gnome.org/GNOME/librs… |
|
479 scripts</a> to be much faster. A complete pipeline used to take |
|
480 about 90 minutes to run, now it takes about 15 minutes on average.<… |
|
481 <p><img alt="Graph with pipeline timings, which shrink drastica… |
|
482 <p>The <a href="https://gitlab.gnome.org/GNOME/librsvg/-/merge_… |
|
483 <ul> |
|
484 <li> |
|
485 <p>Move the <code>cargo check</code> stage to the begi… |
|
486 "does this even have a chance of compiling?".</p> |
|
487 </li> |
|
488 <li> |
|
489 <p>Have the code style and formatting tests, <code>cargo cli… |
|
490 fmt</code>, run in parallel with the unit tests. These lints ca… |
|
491 but they are easy to fix after one is finished modifying the main |
|
492 code.</p> |
|
493 </li> |
|
494 <li> |
|
495 <p>Run the unit tests and the smoke tests in debug mode, so they c… |
|
496 quickly.</p> |
|
497 </li> |
|
498 <li> |
|
499 <p>Run the complete integration test suite in release mode. This … |
|
500 longer to compile, but there are some slow tests that benefit a lot |
|
501 from faster execution.</p> |
|
502 </li> |
|
503 <li> |
|
504 <p>Move the release tests until the end, and only run them once a … |
|
505 — or whenever, by hand. These take a good amount of time to run, |
|
506 because they do a full <code>make distcheck</code> and aut… |
|
507 then, now these tests take 30-40 minutes, instead of the 90 from |
|
508 before.</p> |
|
509 </li> |
|
510 <li> |
|
511 <p>Between each stage of the pipeline, don't cache what doesn't he… |
|
512 reduce compilation time. It seems that keeping around a big cache, |
|
513 with the whole build <code>target</code>, between each pip… |
|
514 worse than not having one at all.</p> |
|
515 </li> |
|
516 </ul> |
|
517 <p><img alt="Complete pipeline with all the stages" src="https:… |
|
518 <h2>Test suite in Rust</h2> |
|
519 <p>Beteen Sven Neumann, Dunja Lalic, and myself we have finally … |
|
520 the test suite to Rust</a>: all of librsvg's tests are |
|
521 now in Rust, except for the C API tests. We had to do a few things:<… |
|
522 <ul> |
|
523 <li> |
|
524 <p>Review the old tests and remove some obsolete ones.</p> |
|
525 </li> |
|
526 <li> |
|
527 <p>Port each of the test modules to Rust. They are small, but eac… |
|
528 has special little things — test for crashes in the XML loading |
|
529 code, test for crashes during rendering, test the library's security |
|
530 limits.</p> |
|
531 </li> |
|
532 <li> |
|
533 <p>Fix the small tests that come as part of the documentation.<… |
|
534 </li> |
|
535 <li> |
|
536 <p>Untangle the reference tests and port them to Rust.</p> |
|
537 </li> |
|
538 <li> |
|
539 <p>Move little chunks of code around so the unit tests and integra… |
|
540 tests can share utilities to compare images, compute file paths for |
|
541 test fixtures, etc.</p> |
|
542 </li> |
|
543 </ul> |
|
544 <p>The most complicated thing to port was the reference tests. Th… |
|
545 the most important ones; each test loads an SVG document, renders it, |
|
546 and compares the result to a reference PNG image. There are some |
|
547 complications in the tests; they have to create a special |
|
548 configuration for Fontconfig and Pango, so as to have reproducible |
|
549 font rendering. The pango-rs bindings do not cover this part of |
|
550 Pango, so we had to do some things by hand.</p> |
|
551 <p>Anyway, <a href="https://gitlab.gnome.org/GNOME/librsvg/-/tr… |
|
552 now the tests run automatically in parallel, across all CPU cores, so |
|
553 we save on total testing time.</p> |
|
554 <h2>What's next: cargo-c and publish to crates.io</h2> |
|
555 <p>We want to be able to <a href="https://gitlab.gnome.org/GNOM… |
|
556 normal crate; this implies being able to compile, test, and publish |
|
557 entirely from Cargo. The compilation and testing part is done.</p> |
|
558 <p>Now, we have to reorganize the code so it can be published to |
|
559 crates.io. Librsvg comes in three parts, <code>rsvg_internals<… |
|
560 implementation of the library, <code>librsvg</code> with the… |
|
561 and <code>librsvg_crate</code> with the Rust API. However, … |
|
562 API to crates.io, it would be more convenient to have a single crate |
|
563 instead of one with the internals and one with the API.</p> |
|
564 <p>The next step is thus to reorganize the code:</p> |
|
565 <ul> |
|
566 <li> |
|
567 <p>Make it possible to implement the C API as a compile-time optio… |
|
568 top of the normal Rust code. We want to <a href="https://gitlab.gn… |
|
569 compile the traditional shared library <code>librsvg.so</code… |
|
570 depending on C tools for compiling and linking.</p> |
|
571 </li> |
|
572 <li> |
|
573 <p>Combine <code>rsvg_internals</code> and <code>… |
|
574 publish them together. Crates.io has a 10 MB limit per crate; now |
|
575 that the test suite lives in a separate <code>tests</code>… |
|
576 shouldn't be a problem.</p> |
|
577 </li> |
|
578 <li> |
|
579 <p>I would like to <a href="https://gitlab.gnome.org/GNOME/libr… |
|
580 publishing the Rust API; right now they expose some implementation |
|
581 details that are of no interest to callers of the library.</p> |
|
582 </li> |
|
583 </ul> |
|
584 <h2>What remains to be ported to Rust?</h2> |
|
585 <p>Only two things, which amount to less than 900 lines of C code… |
|
586 <ul> |
|
587 <li> |
|
588 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/-/issues/534… |
|
589 convert SVG to PNG and other formats. Fortunately, Sven Neumann |
|
590 wrote some <a href="https://gitlab.gnome.org/GNOME/librsvg/-/blob/m… |
|
591 as it is like an API that we need to keep stable: if we change the |
|
592 command-line options or the program's behavior, we would break |
|
593 everyone's scripts.</p> |
|
594 </li> |
|
595 <li> |
|
596 <p>The gdk-pixbuf module for loading SVG. Alberto Ruiz <a hre… |
|
597 porting it to Rust</a>. The generic part of this code |
|
598 could later serve to wrap other Rust image codecs and plug them to |
|
599 gdk-pixbuf.</p> |
|
600 </li> |
|
601 </ul></content><category term="misc"></category><category term="li… |
|
602 in the December 2020 / March 2021 round:</p> |
|
603 <ul> |
|
604 <li> |
|
605 <p><strong>Revamp the text engine</strong> - Do you kn… |
|
606 layout? Can you read a right-to-left language, or do you write in a |
|
607 language that requires complex shaping? Would you like to implement �… |
|
608 in the December 2020 / March 2021 round:</p> |
|
609 <ul> |
|
610 <li> |
|
611 <p><strong>Revamp the text engine</strong> - Do you kn… |
|
612 layout? Can you read a right-to-left language, or do you write in a |
|
613 language that requires complex shaping? Would you like to implement |
|
614 the <a href="https://www.w3.org/TR/SVG2/text.html">SVG 2 text sp… |
|
615 base</a>? This project requires someone who can write Rust |
|
616 comfortably; it will require reading and refactoring some |
|
617 existing code. You don't need to be an expert in exotic lifetimes |
|
618 and trait bounds and such; the code doesn't use them.</p> |
|
619 </li> |
|
620 <li> |
|
621 <p><strong>Implement SVG2/CSS3 features</strong> - Are… |
|
622 features</a> in Inkscape, and would like to add support for them |
|
623 in librsvg? Would you like to do small changes to many parts of the |
|
624 code to implement small features, one at a time? Do you like |
|
625 test-driven development? This project requires someone who can |
|
626 write Rust code at a medium level; you'll learn a lot by |
|
627 cutting&amp;pasting from existing code and refactoring things to |
|
628 implement SVG2 features.</p> |
|
629 </li> |
|
630 </ul> |
|
631 <p><strong>Important:</strong> Outreachy's December 20… |
|
632 for students in the Southern hemisphere</a>. People in the |
|
633 Northern hemisphere can wait until the 2021 mid-year round.</p> |
|
634 <p>You can see <a href="https://www.outreachy.org/apply/project… |
|
635 <strong>The deadline for initial contributions and project applica… |
|
636 October 31, 2020 at 16:00 UTC.</strong></p></content><catego… |
|
637 time on C libraries) express concerns along the following lines:</p&g… |
|
638 <ol> |
|
639 <li>Compiled Rust code doesn't have a stable ABI (application bina… |
|
640 <li>So, we can't have shared libraries in the traditional fashion … |
|
641 Linux distributions.</li> |
|
642 <li>Also Rust bundles …</li></ol></summary><content … |
|
643 time on C libraries) express concerns along the following lines:</p&g… |
|
644 <ol> |
|
645 <li>Compiled Rust code doesn't have a stable ABI (application bina… |
|
646 <li>So, we can't have shared libraries in the traditional fashion … |
|
647 Linux distributions.</li> |
|
648 <li>Also Rust bundles its entire standard library with every binar… |
|
649 </ol> |
|
650 <p>These are extremely valid concerns to be addressed by people li… |
|
651 myself who propose that chunks of infrastructural libraries |
|
652 should be done in Rust.</p> |
|
653 <p>So, let's begin.</p> |
|
654 <p>The first part of this article is a super-quick introduction to… |
|
655 libraries and how Linux distributions use them. If you already know |
|
656 those things, feel free to skip to the "<a href="#rust_does_not_have_… |
|
657 ABI</a>" section.</p> |
|
658 <h2>How do distributions use shared libraries?</h2> |
|
659 <p>If several programs run at the same time and use the same share… |
|
660 (say, <code>libgtk-3.so</code>), the operating system can lo… |
|
661 the library in memory and share the read-only parts of the code/data |
|
662 through the magic of virtual memory.</p> |
|
663 <p><em>In theory</em>, if a library gets a bugfix but … |
|
664 interface, one can just recompile the library, stick the new <code>… |
|
665 <code>/usr/lib</code> or whatever, and be done with it. Pro… |
|
666 the library do not need to be recompiled.</p> |
|
667 <p>If libraries limit their public interface to a plain C ABI |
|
668 (application binary interface), they are relatively easy to consume |
|
669 from other programming languages. Those languages don't have to deal |
|
670 with name mangling of C++ symbols, exception handlers, constructors, |
|
671 and all that complexity. Pretty much every language has some form of |
|
672 C FFI (foreign function interface), which roughly means "call C |
|
673 functions without too much trouble".</p> |
|
674 <p>For the purposes of a library, what's an |
|
675 <a href="https://en.wikipedia.org/wiki/Application_binary_interface"&… |
|
676 Wikipedia says, "An ABI defines how data structures or computational |
|
677 routines are accessed in machine code [...] A common aspect of an |
|
678 ABI is the calling convention", which means that to call a function in |
|
679 machine code you need to frob the call and stack pointers, pass some |
|
680 function arguments in registers or push some others to the stack, etc. |
|
681 Really low-level stuff. Each machine architecture or operating system |
|
682 usually defines a C standard ABI.</p> |
|
683 <p>For libraries, we commonly understand an ABI to mean the machin… |
|
684 implications of their programming interface. Which functions are |
|
685 available as public symbols in the <code>.so</code> file? T… |
|
686 values do C enum values correspond, so that they can be passed to |
|
687 those functions? What is the exact order and type of arguments that |
|
688 the functions take? What are the struct sizes, and the order and |
|
689 types and padding of the fields that those functions take? Does one |
|
690 pass arguments in CPU registers or on the stack? Does the caller or |
|
691 the callee clean up the stack after a function call?</p> |
|
692 <h2>Bug fixes and security fixes</h2> |
|
693 <p>Linux distributions generally try <em>really hard</em&… |
|
694 version of each shared library installed in the system: a single |
|
695 <code>libjpeg.so</code>, a single <code>libpng.so</… |
|
696 <p>This is helpful when there needs to be an update to fix a bug, |
|
697 security-related or not: users can just download the updated package |
|
698 for the library, which when installed will just stick in a new <code&… |
|
699 in the right place, and the calling software won't need to be updated.&l… |
|
700 <p>This is possible only if the bug <em>really only changes … |
|
701 code</em> without changing behavior or interface. If a bug fix re… |
|
702 part of the public API or ABI to change, then you are screwed; all |
|
703 calling software needs to be recompiled. "Irresponsible" library |
|
704 authors either learn really fast when distros complain loudly about |
|
705 this sort of change, or they don't learn and get forever marked by |
|
706 distros as "that irresponsible library" which always requires special |
|
707 handling in order not to break other software.</p> |
|
708 <p>Sidenote: sometimes it's more complicated. Poppler (the PDF |
|
709 rendering library) ships at least two stable APIs, one Glib-based in |
|
710 C, and one Qt-based in C++. However, some software like texlive uses |
|
711 Poppler's internals library directly, which of course does not have a |
|
712 stable API, and thus texlive breaks frequently as Poppler evolves. |
|
713 Someone should extend the public, stable API so that texlive doesn't |
|
714 have to use the library's internals!</p> |
|
715 <h2>Bundled libraries</h2> |
|
716 <p>Sometimes it is not irresponsible authors of libraries, but rat… |
|
717 that people who use the libraries find out that over time the behavior |
|
718 of the library changes subtly, maybe without breaking the API or ABI, |
|
719 and they are better off bundling a specific version of the library |
|
720 with their software. That version is what they test their software |
|
721 against, and they try to learn its quirks.</p> |
|
722 <p>Distros inevitably complain about this, and either patch the ca… |
|
723 software by hand to force it to use the system's shared library, or |
|
724 succeed in getting patches accepted by the software so that they have |
|
725 a <code>--use-system-libjpeg</code> option or similar.</p… |
|
726 <p>This doesn't work very well if the bundled version of the libra… |
|
727 extra patches that are not in a distro's usual patches. Or |
|
728 vice-versa; it may actually work better to use the distro's version of |
|
729 the library, if it has extra fixes that the bundled library doesn't. |
|
730 Who knows! It's a case-by-case situation.</p> |
|
731 <h2 id="rust_does_not_have_a_stable_abi">Rust does not have a stab… |
|
732 <p>By default indeed it doesn't, because the compiler team wants t… |
|
733 the freedom to change the data layout and Rust-to-Rust calling |
|
734 conventions, often for performance reasons, at any time. For example, |
|
735 it is not guaranteed that struct fields will be laid out in memory in |
|
736 the same order as they are written in the code:</p> |
|
737 <div class="highlight"><pre><span></span><cod… |
|
738 <span class="w"> </span><span class="n">bar</spa… |
|
739 <span class="w"> </span><span class="n">baz</spa… |
|
740 <span class="w"> </span><span class="n">beep</sp… |
|
741 <span class="w"> </span><span class="n">qux</spa… |
|
742 <span class="p">}</span><span class="w"></span> |
|
743 </code></pre></div> |
|
744 |
|
745 <p>The compiler is free to rearrange the struct fields in memory a… |
|
746 sees fit. Maybe it decides to put the two <code>bool</code>… |
|
747 other to save on inter-field padding due to alignment requirements; |
|
748 maybe it does static analysis or profile-guided optimizations and |
|
749 picks an optmal ordering.</p> |
|
750 <p>But we can override this! Let's look at data layout first, and… |
|
751 calling conventions.</p> |
|
752 <h3>Data layout for C versus Rust</h3> |
|
753 <p>The following is the same struct as above, but with an extra &l… |
|
754 <div class="highlight"><pre><span></span><cod… |
|
755 <span class="k">struct</span> <span class="nc">Foo<… |
|
756 <span class="w"> </span><span class="n">bar</spa… |
|
757 <span class="w"> </span><span class="n">baz</spa… |
|
758 <span class="w"> </span><span class="n">beep</sp… |
|
759 <span class="w"> </span><span class="n">qux</spa… |
|
760 <span class="p">}</span><span class="w"></span> |
|
761 </code></pre></div> |
|
762 |
|
763 <p>With that attribute, the struct will be laid out just as this C… |
|
764 <div class="highlight"><pre><span></span><cod… |
|
765 <span class="cp">#include</span> <span class="cpf">&am… |
|
766 |
|
767 <span class="k">struct</span> <span class="nc">Foo<… |
|
768 <span class="kt">bool</span> <span class="n">bar&l… |
|
769 <span class="kt">double</span> <span class="n">baz… |
|
770 <span class="kt">bool</span> <span class="n">beep&… |
|
771 <span class="kt">int32_t</span> <span class="n">qu… |
|
772 <span class="p">}</span> |
|
773 </code></pre></div> |
|
774 |
|
775 <p>(Aside: it is unfortunate that <a href="https://people.gnome… |
|
776 but that's because <code>gboolean</code> predates C99, and c… |
|
777 20 years ago are <em>too new</em> to use. (Aside aside: sin… |
|
778 other post, Rust's repr(C) for bool is actually defined as C99's bool; |
|
779 it's no longer undefined.))</p> |
|
780 <p>Even Rust's data-carrying enums can be laid out in a manner fri… |
|
781 to C and C++:</p> |
|
782 <div class="highlight"><pre><span></span><cod… |
|
783 <span class="k">enum</span> <span class="nc">MyEnum<… |
|
784 <span class="w"> </span><span class="n">A</span&… |
|
785 <span class="w"> </span><span class="n">B</span&… |
|
786 <span class="p">}</span><span class="w"></span> |
|
787 </code></pre></div> |
|
788 |
|
789 <p>This means, use C layout, and a <code>u8</code> for… |
|
790 will be laid out like this:</p> |
|
791 <div class="highlight"><pre><span></span><cod… |
|
792 <span class="cp">#include</span> <span class="cpf">&am… |
|
793 |
|
794 <span class="k">enum</span> <span class="n">MyEnumTag&… |
|
795 <span class="n">A</span><span class="p">,</… |
|
796 <span class="n">B</span> |
|
797 <span class="p">};</span> |
|
798 |
|
799 <span class="k">typedef</span> <span class="kt">uint32… |
|
800 |
|
801 <span class="k">typedef</span> <span class="k">struct&… |
|
802 <span class="kt">float</span> <span class="n">… |
|
803 <span class="kt">bool</span> <span class="n">y… |
|
804 <span class="p">}</span> <span class="n">MyEnumPayload… |
|
805 |
|
806 <span class="k">typedef</span> <span class="k">union&l… |
|
807 <span class="n">MyEnumPayloadA</span> <span class… |
|
808 <span class="n">MyEnumPayloadB</span> <span class… |
|
809 <span class="p">}</span> <span class="n">MyEnumPayload… |
|
810 |
|
811 <span class="k">typedef</span> <span class="k">struct&… |
|
812 <span class="kt">uint8_t</span> <span class="n"&g… |
|
813 <span class="n">MyEnumPayload</span> <span class=… |
|
814 <span class="p">}</span> <span class="n">MyEnum</sp… |
|
815 </code></pre></div> |
|
816 |
|
817 <p>The gory details of data layout are in the <a href="https://… |
|
818 Rustonomicon</a> and |
|
819 the <a href="https://rust-lang.github.io/unsafe-code-guidelines/intro… |
|
820 Guidelines</a>.</p> |
|
821 <h3>Calling conventions</h3> |
|
822 <p>An ABI's calling conventions detail things like how to call fun… |
|
823 in machine code, and how to lay out function arguments in registers or |
|
824 the stack. <a href="https://en.wikipedia.org/wiki/X86_calling_conven… |
|
825 conventions</a> |
|
826 has a good cheat-sheet, useful when you are looking at assembly code |
|
827 and registers in a low-level debugger.</p> |
|
828 <p>I've already written about how it is possible to write Rust cod… |
|
829 export functions callable from C; one uses the <code>extern "C"<… |
|
830 function definition and a <code>#[no_mangle]</code> attribut… |
|
831 name pristine. This is how librsvg is able to have the following:</p… |
|
832 <div class="highlight"><pre><span></span><cod… |
|
833 <span class="k">pub</span><span class="w"> </span&g… |
|
834 <span class="w"> </span><span class="n">filename<… |
|
835 <span class="w"> </span><span class="n">error</s… |
|
836 <span class="p">)</span><span class="w"> </span>… |
|
837 <span class="w"> </span><span class="c1">// ...<… |
|
838 <span class="p">}</span><span class="w"></span> |
|
839 </code></pre></div> |
|
840 |
|
841 <p>Which compiles to what a C compiler would produce for this:<… |
|
842 <div class="highlight"><pre><span></span><cod… |
|
843 </code></pre></div> |
|
844 |
|
845 <p>(Aside: librsvg <a href="https://gitlab.gnome.org/GNOME/libr… |
|
846 stubs</a> that just |
|
847 call the Rust-exported functions, but there is now <a href="https://g… |
|
848 directly from |
|
849 Rust</a> which I |
|
850 just haven't had time to investigate. Help is appreciated!)</p> |
|
851 <h3>Summary of ABI so far</h3> |
|
852 <p>It is <em>one's decision</em> to export a stable C … |
|
853 There is some awkwardness in how types are laid out in C, because the |
|
854 Rust type system is richer, but things can be made to work well with a |
|
855 little thought. Certainly no more thought than the burden of |
|
856 designing and maintaining a stable API/ABI in plain C.</p> |
|
857 <p>I'll fold the second concern into here — "we can't have shared |
|
858 libraries in traditional distro fashion". Yes, we can, API/ABI-wise, |
|
859 but read on.</p> |
|
860 <h2>Rust bundles its entire standard library with Rust-built .so's… |
|
861 <p>I.e. it statically links all the Rust dependencies. This produ… |
|
862 large .so:</p> |
|
863 <ul> |
|
864 <li>librsvg-2.so (version 2.40.21, C only) - 1408840 bytes</li… |
|
865 <li>librsvg-2.so (version 2.49.3, Rust only) - 9899120 bytes</… |
|
866 </ul> |
|
867 <p>Holy crap! What's all that?</p> |
|
868 <p>(And I'm cheating: this is both with link-time optimization tur… |
|
869 and by running <code>strip(1)</code> on the .so. If you jus… |
|
870 it will be bigger.)</p> |
|
871 <p>This has Rust's standard library statically linked (or at least… |
|
872 bits of that librsvg actually uses), plus all the Rust dependencies |
|
873 (cssparser, selectors, nalgebra, glib-rs, cairo-rs, locale_config, |
|
874 rayon, xml5ever, and an assload of crates). I could explain why each |
|
875 one is needed:</p> |
|
876 <ul> |
|
877 <li>cssparser - librsvg needs to parse CSS.</li> |
|
878 <li>selectors - librsvg needs to run the CSS selector matching |
|
879 algorithm.</li> |
|
880 <li>nalgebra - the code for SVG filter effects uses vectors and |
|
881 matrices.</li> |
|
882 <li>glib-rs, cairo-rs - draw to Cairo and export GObject types.<… |
|
883 <li>locale_config - so that localized SVG files can work.</li&g… |
|
884 <li>rayon - so filters can use all your CPU cores instead of proce… |
|
885 one pixel at a time.</li> |
|
886 <li>Etcetera. SVG is big and requires a lot of helper code!</l… |
|
887 </ul> |
|
888 <p>Is this a problem?</p> |
|
889 <p>Or more exactly, why does this happen, and why do people percei… |
|
890 as a problem?</p> |
|
891 <h3>Stable APIs/ABIs and distros</h3> |
|
892 <p>Many Linux distributions have worked <em>really hard</… |
|
893 there is a single copy of "system libraries" in an installation. |
|
894 There is Just One Copy of <code>/usr/lib/libc.so</code>, <… |
|
895 etc., and packages are compiled with special options to tell them to |
|
896 really use the sytem libraries instead of their bundled versions, or |
|
897 patched to do so if they don't provide build-time options for that.</… |
|
898 <p>In a way, this works well for distros:</p> |
|
899 <ul> |
|
900 <li> |
|
901 <p>A bug in a library can be fixed in a single place, and all |
|
902 applications that use it get the fix automatically.</p> |
|
903 </li> |
|
904 <li> |
|
905 <p>A security bug can be patched in a single place, and in theory |
|
906 applications don't need to be audited further.</p> |
|
907 </li> |
|
908 </ul> |
|
909 <p>If you maintain a library that is shipped in Linux distros, and… |
|
910 break the ABI, you'll get complaints from distros very quickly.</p> |
|
911 <p>This is good because it creates responsible maintainers for lib… |
|
912 that can be depended on. It's how Inkscape/GIMP can have a stable |
|
913 toolkit to be written in.</p> |
|
914 <p>This is bad because it encourages stagnation in the long term. … |
|
915 how we get a horrible, unsafe, error-prone API in libjpeg that can |
|
916 never ever be improved because it would requires changes in tons of |
|
917 software; it's why <code>gboolean</code> is still a 32-bit &… |
|
918 twenty-something years, even though everything else close to C has |
|
919 decided that booleans are 1 byte. It's how Inkscape/GIMP take many |
|
920 years to move from GTK2 to GTK3 (okay, that's lack of paid developers |
|
921 to do the grunt work, but it is enabled by having forever-stable APIs).&… |
|
922 <p>However, a long-term stable API/ABI has a <strong>lot of … |
|
923 the Windows API is the crown jewels; it is why people can rely on glib |
|
924 and glibc to not break their code for many years and take them for grant… |
|
925 <h3>But we only have a single stable ABI anyway</h3> |
|
926 <p>And that is the C ABI. Even C++ libraries have trouble with th… |
|
927 people sometimes write the internals of a library in C++ for |
|
928 convenience, but export a stable C API/ABI from it.</p> |
|
929 <p>High level languages like Python have <em>real trouble<… |
|
930 precisely because of ABI issues.</p> |
|
931 <h3>Actually, in GNOME we have gone further than that</h3> |
|
932 <p>In GNOME we have constructed a sweet little universe where <… |
|
933 Introspection</a> is |
|
934 basically a C ABI with a ton of machine-generated annotations to make |
|
935 it friendly to language bindings.</p> |
|
936 <p>Still, we rely on a C ABI underneath. See <a href="https://… |
|
937 thread on advancing the C ABI from Rust</a> for |
|
938 lots of food for thought.</p> |
|
939 <h3>Single copies of libraries with a C ABI</h3> |
|
940 <p>Okay, let's go back to this. What price do we pay for single c… |
|
941 of libraries that, by necessity, must export a C ABI?</p> |
|
942 <ul> |
|
943 <li> |
|
944 <p>Code that can be conveniently called from C, maybe from C++, and |
|
945 moderately to very inconvently from ANYTHING ELSE. With most new |
|
946 application code being written definitely not in C, maybe we should |
|
947 reconsider our priorities here.</p> |
|
948 </li> |
|
949 <li> |
|
950 <p>No language facilities like generics or field visibility, which… |
|
951 not even "modern language" features. Even C++ templates get |
|
952 compiled and statically linked into the calling code, because |
|
953 there's no way to pass information like the size of <code>T</… |
|
954 <code>Array&lt;T&gt;</code> across a C ABI. You w… |
|
955 public and some private? You are out of luck.</p> |
|
956 </li> |
|
957 <li> |
|
958 <p>No knowledge of data ownership except by careful reading of the… |
|
959 function's documentation. Does the function free its arguments? |
|
960 How - with <code>free()</code> or <code>g_free()<… |
|
961 caller just lend it a reference? Can the data be copied bit-by-bit |
|
962 or must a special function be called to make a copy? |
|
963 GObject-Introspection carries this information in its annotations, |
|
964 while the C ABI has no idea and just ships raw pointers around.</p&… |
|
965 </li> |
|
966 </ul> |
|
967 <p>More food for thought note: <a href="https://twitter.com/hsi… |
|
968 thread</a> says |
|
969 this about the C++ ABI: "Also, the ABI matters for whether the actual |
|
970 level of practicality of complying with LGPL matches the level of |
|
971 practicality intended years ago when some project picked LGPL as its |
|
972 license. Of course, the standard does not talk about LGPL, either. |
|
973 LGPL has rather different implications for Rust and Go than it does |
|
974 for C and Java. It was obviously written with C in mind."</p> |
|
975 <h2>Monomorphization and template bloat</h2> |
|
976 <p>While C++ had the problem of "lots of template code in header f… |
|
977 Rust has the problem that <a href="https://pingcap.com/blog/generics-… |
|
978 of compiled |
|
979 code</a>. |
|
980 There are tricks to avoid this and they are all the decision of the |
|
981 library/crate author. Both share the root cause that templated or |
|
982 generic code must be recompiled for every specific use, and thus |
|
983 cannot live in a shared library.</p> |
|
984 <p>Also, see this wonderful <a href="https://thume.ca/2019/07/1… |
|
985 generics</a>, |
|
986 and think that a plain C ABI means we have NOTHING of the sort.</p> |
|
987 <p>Also, see <a href="https://gankra.github.io/blah/swift-abi/"… |
|
988 Couldn't</a> for more food for |
|
989 thought. This is extremely roughly equivalent to GObject's boxed |
|
990 types; callers keep values on the heap but know the type layout via |
|
991 annotation magic, while the library's actual implementation |
|
992 is free to have the values on the stack or wherever for its own use.<… |
|
993 <h2>Should all libraries export APIs with generics and exotic type… |
|
994 <p>No!</p> |
|
995 <p>You probably want something like a low-level array of values, |
|
996 <code>Vec&lt;T&gt;</code>, to be inlined everywhere … |
|
997 type of the vector's elements. Element accesses can be inlined to a |
|
998 single machine instruction in the best case.</p> |
|
999 <p>But not everything requires this absolute raw performance with |
|
1000 everything inlined everywhere. It is fine to pass references or |
|
1001 pointers to things and do dynamic dispatch from a vtable if you are |
|
1002 not in a super-tight loop, as we love to do in the GObject world.</p&… |
|
1003 <h2>Library sizes</h2> |
|
1004 <p>I don't have a good answer to librsvg's compiled size. If gnom… |
|
1005 merges my branch to rustify the CSS code, it will also grow its binary |
|
1006 size by quite a bit.</p> |
|
1007 <p>It is my intention to have a Rust crate that both librsvg and |
|
1008 gnome-shell share for their CSS styling needs, but right now I have no |
|
1009 idea if this would be a shared library or just a normal Rust crate. |
|
1010 Maybe it's possible to have a very general CSS library, and the |
|
1011 application registers which properties it can parse and how? Is it |
|
1012 possible to do this as a shared library without essentially |
|
1013 reinventing libcroco? I don't know yet. We'll see.</p> |
|
1014 <h2>A metaphor which I haven't fully explored</h2> |
|
1015 <p>If every application or end-user package is kind of like a livi… |
|
1016 organism, with its own cycles and behaviors and organs (dependent |
|
1017 libraries) that make it possible...</p> |
|
1018 <p>Why do distros expect all the living organisms on your machine … |
|
1019 share The World's Single Lungs Service, and The World's Single Stomach |
|
1020 Service, and The World's Single Liver Service?</p> |
|
1021 <p>You know, instead of letting every organism have its own slight… |
|
1022 different version of those organs, customized for it? We humans know |
|
1023 how to do vaccination campaigns and everything; maybe we need better |
|
1024 tools to apply bug fixes where they are needed?</p> |
|
1025 <p>I know this metaphor is extremely imperfect and not how things … |
|
1026 software, but it makes me wonder.</p></content><category term="mis… |
|
1027 <p>The GNOME Foundation's <a href="https://mail.gnome.org/archi… |
|
1028 up, and we are looking for candidates. Of the 7 directors, we are |
|
1029 replacing 4, and the 3 remaining positions remain for another year. |
|
1030 You …</p></summary><content type="html"><p>I forgot to wri… |
|
1031 <p>The GNOME Foundation's <a href="https://mail.gnome.org/archi… |
|
1032 up, and we are looking for candidates. Of the 7 directors, we are |
|
1033 replacing 4, and the 3 remaining positions remain for another year. |
|
1034 You could be one of those four.</p> |
|
1035 <p>I would like it very much if there were candidates and director… |
|
1036 fall outside the box of "white male programmer"; it is unfortunate |
|
1037 that for the current Board we ended up with all dudes. GNOME has a |
|
1038 <a href="https://wiki.gnome.org/Foundation/CodeOfConduct">Code of … |
|
1039 <p>Allan Day wrote a <a href="https://blogs.gnome.org/aday/2020… |
|
1040 year</a>. We are moving from a model where the Board does a |
|
1041 little bit of everything, to one with a more strategic role — now that |
|
1042 the Foundation has full-time employees, they take care of most of the |
|
1043 executive work.</p> |
|
1044 <p><strong>The call-for-candidates is open until May 29, so … |
|
1045 <ul> |
|
1046 <li><a href="https://blogs.gnome.org/aday/2020/05/26/gnome-foun… |
|
1047 <li><a href="https://mail.gnome.org/archives/foundation-announc… |
|
1048 </ul></content><category term="misc"></category><category term="gn… |
|
1049 that by now has a lot of accumulated crap. It is such an old configurat… |
|
1050 it didn't even use the modern convention of <code>~/.emacs.d/init.… |
|
1051 like a newer Emacs …</p></summary><content type="html"><p>… |
|
1052 that by now has a lot of accumulated crap. It is such an old configurat… |
|
1053 it didn't even use the modern convention of <code>~/.emacs.d/init.… |
|
1054 like a newer Emacs version will allow <code>.config/emacs</code… |
|
1055 standard... at last).</p> |
|
1056 <p>I have wanted to change my Emacs configuration for some time, a… |
|
1057 the pretty and modern toys.</p> |
|
1058 <p>The things that matter the most to me:</p> |
|
1059 <ul> |
|
1060 <li>Not have a random dumpster in <code>~/.emacs</code>… |
|
1061 <li>Pretty colors.</li> |
|
1062 <li>Magit.</li> |
|
1063 <li>Rust-mode or whatever the new thing is for rust-analyzer and t… |
|
1064 </ul> |
|
1065 <p>After looking at several examples of configurations that mentio… |
|
1066 as a unified way of loading packages and configuring them, I found <a… |
|
1067 configuration</a> which is extremely well |
|
1068 documented. The author does literate programming with org-mode and elis… |
|
1069 something which I'm casually interested in, but not just now — but tha… |
|
1070 everything ends up very well explained and easy to read.</p> |
|
1071 <p>I extracted bits of that configuration and ended up with the fo… |
|
1072 <h2>Everything in <code>~/.emacs/init.el</code> and wi… |
|
1073 <div class="highlight"><pre><span></span><cod… |
|
1074 |
|
1075 <span class="p">(</span><span class="nb">require</s… |
|
1076 |
|
1077 <span class="p">(</span><span class="k">setq</span&… |
|
1078 <span class="o">&#39;</span><span class="p">… |
|
1079 <span class="p">(</span><span class="s">&q… |
|
1080 <span class="p">(</span><span class="s">&q… |
|
1081 |
|
1082 <span class="p">(</span><span class="nv">package-initi… |
|
1083 <span class="c1">;(package-refresh-contents)</span> |
|
1084 |
|
1085 <span class="c1">;; Use-package for civilized configuration</sp… |
|
1086 |
|
1087 <span class="p">(</span><span class="nb">unless</sp… |
|
1088 <span class="p">(</span><span class="nv">package-ins… |
|
1089 <span class="p">(</span><span class="nb">require</s… |
|
1090 |
|
1091 <span class="p">(</span><span class="k">setq</span&… |
|
1092 </code></pre></div> |
|
1093 |
|
1094 <h2><code>~/.emacs.d/custom.el</code> for <code>… |
|
1095 <div class="highlight"><pre><span></span><cod… |
|
1096 <span class="c1">;; my init files.</span> |
|
1097 |
|
1098 <span class="p">(</span><span class="k">setq</span&… |
|
1099 <span class="p">(</span><span class="nf">load</span… |
|
1100 </code></pre></div> |
|
1101 |
|
1102 <h2>Which-key to get hints when typing command prefixes</h2> |
|
1103 <div class="highlight"><pre><span></span><cod… |
|
1104 |
|
1105 <span class="p">(</span><span class="nb">use-package&l… |
|
1106 <span class="nb">:diminish</span> |
|
1107 <span class="nb">:config</span> |
|
1108 <span class="p">(</span><span class="nv">which-key-m… |
|
1109 <span class="p">(</span><span class="nv">which-key-s… |
|
1110 <span class="p">(</span><span class="k">setq</spa… |
|
1111 </code></pre></div> |
|
1112 |
|
1113 <h2>Don't pollute the modeline with common modes</h2> |
|
1114 <div class="highlight"><pre><span></span><cod… |
|
1115 |
|
1116 <span class="p">(</span><span class="nb">use-package&l… |
|
1117 <span class="nb">:defer</span> <span class="mi">5<… |
|
1118 <span class="nb">:config</span> |
|
1119 <span class="p">(</span><span class="nv">diminish<… |
|
1120 </code></pre></div> |
|
1121 |
|
1122 <h2>Magit to use git in a civilized fashion</h2> |
|
1123 <div class="highlight"><pre><span></span><cod… |
|
1124 |
|
1125 <span class="p">(</span><span class="nb">use-package&l… |
|
1126 <span class="nb">:config</span> |
|
1127 <span class="p">(</span><span class="nv">global-set-… |
|
1128 </code></pre></div> |
|
1129 |
|
1130 <h2>Move between windows with Shift-arrows</h2> |
|
1131 <div class="highlight"><pre><span></span><cod… |
|
1132 <span class="p">(</span><span class="nv">windmove-defa… |
|
1133 </code></pre></div> |
|
1134 |
|
1135 <h2>Pretty colors</h2> |
|
1136 <p>I was using <code>solarized-dark</code> but I like … |
|
1137 <div class="highlight"><pre><span></span><cod… |
|
1138 |
|
1139 <span class="p">(</span><span class="nb">use-package&l… |
|
1140 <span class="nb">:config</span> |
|
1141 <span class="p">(</span><span class="nv">custom-them… |
|
1142 <span class="o">&#39;</span><span class="p">(&l… |
|
1143 <span class="o">&#39;</span><span class="p">(&l… |
|
1144 </code></pre></div> |
|
1145 |
|
1146 <h2>Nyan cat instead of scrollbars</h2> |
|
1147 <div class="highlight"><pre><span></span><cod… |
|
1148 <span class="c1">;; scroll-bar-mode is turned off in custom.el<… |
|
1149 |
|
1150 <span class="p">(</span><span class="nb">use-package&l… |
|
1151 <span class="nb">:config</span> |
|
1152 <span class="p">(</span><span class="nv">nyan-mode&l… |
|
1153 </code></pre></div> |
|
1154 |
|
1155 <h2>Move buffers to adjacent windows</h2> |
|
1156 <div class="highlight"><pre><span></span><cod… |
|
1157 |
|
1158 <span class="p">(</span><span class="nb">use-package&l… |
|
1159 <span class="nb">:config</span> |
|
1160 <span class="p">(</span><span class="nv">global-set-… |
|
1161 <span class="p">(</span><span class="nv">global-set-… |
|
1162 <span class="p">(</span><span class="nv">global-set-… |
|
1163 <span class="p">(</span><span class="nv">global-set-… |
|
1164 </code></pre></div> |
|
1165 |
|
1166 <h2>Change buffer names for files with the same name</h2> |
|
1167 <div class="highlight"><pre><span></span><cod… |
|
1168 <span class="p">(</span><span class="nb">require</s… |
|
1169 <span class="p">(</span><span class="k">setq</span&… |
|
1170 <span class="nv">uniquify-buffer-name-style</span> <… |
|
1171 </code></pre></div> |
|
1172 |
|
1173 <h2>Helm to auto-complete in grand style</h2> |
|
1174 <div class="highlight"><pre><span></span><cod… |
|
1175 <span class="nb">:diminish</span> |
|
1176 <span class="nb">:init</span> <span class="p">(</s… |
|
1177 <span class="nb">:bind</span> <span class="p">((</… |
|
1178 <span class="p">(</span><span class="s">&q… |
|
1179 <span class="p">(</span><span class="s">&q… |
|
1180 <span class="p">(</span><span class="s">&q… |
|
1181 <span class="p">(</span><span class="s">&q… |
|
1182 <span class="p">(</span><span class="s">&q… |
|
1183 <span class="p">(</span><span class="s">&q… |
|
1184 <span class="c1">;; Look at what was cut recently &amp… |
|
1185 <span class="p">(</span><span class="s">&q… |
|
1186 |
|
1187 <span class="nb">:map</span> <span class="nv">… |
|
1188 <span class="c1">;; We can list ‘actions’ on the curre… |
|
1189 <span class="p">(</span><span class="s">&q… |
|
1190 <span class="c1">;; Let&#39;s keep tab-completetion an… |
|
1191 <span class="p">(</span><span class="s">&q… |
|
1192 <span class="p">(</span><span class="s">&q… |
|
1193 </code></pre></div> |
|
1194 |
|
1195 <h2>Ripgrep to search in grand style</h2> |
|
1196 <div class="highlight"><pre><span></span><cod… |
|
1197 |
|
1198 <span class="p">(</span><span class="nb">use-package&l… |
|
1199 <span class="nb">:config</span> |
|
1200 <span class="p">(</span><span class="nv">global-set-… |
|
1201 <span class="p">(</span><span class="nv">global-set-… |
|
1202 |
|
1203 <span class="p">(</span><span class="nb">use-package&l… |
|
1204 </code></pre></div> |
|
1205 |
|
1206 <h2>Rust mode and Language Server</h2> |
|
1207 <p>Now that RLS is in the process of being deprecated, it's gettin… |
|
1208 with rust-analyzer. Also, rust-mode goes away in favor of rustic.</p… |
|
1209 <div class="highlight"><pre><span></span><cod… |
|
1210 |
|
1211 <span class="p">(</span><span class="nb">use-package&l… |
|
1212 |
|
1213 <span class="p">(</span><span class="nb">use-package&l… |
|
1214 |
|
1215 <span class="p">(</span><span class="nb">use-package&l… |
|
1216 |
|
1217 <span class="p">(</span><span class="nb">use-package&l… |
|
1218 <span class="nb">:config</span> |
|
1219 <span class="p">(</span><span class="nf">define-key&… |
|
1220 </code></pre></div> |
|
1221 |
|
1222 <h2>Performatively not get distracted</h2> |
|
1223 <div class="highlight"><pre><span></span><cod… |
|
1224 |
|
1225 <span class="p">(</span><span class="n">setq</span&… |
|
1226 <span class="w"> </span><span class="p">(</spa… |
|
1227 <span class="w"> </span><span class="o">&… |
|
1228 |
|
1229 <span class="p">(</span><span class="n">defun</span… |
|
1230 <span class="w"> </span><span class="p">(</span>… |
|
1231 <span class="w"> </span><span class="p">(</span&… |
|
1232 <span class="w"> </span><span class="s">&… |
|
1233 <span class="w"> </span><span class="s">&… |
|
1234 <span class="w"> </span><span class="s">&… |
|
1235 <span class="w"> </span><span class="n">statu… |
|
1236 </code></pre></div> |
|
1237 |
|
1238 <h2>Stuff from custom.el</h2> |
|
1239 <p>The interesting bits here are making LSP work; everything else … |
|
1240 <div class="highlight"><pre><span></span><cod… |
|
1241 <span class="c1">;; custom-set-variables was added by Custom.<… |
|
1242 <span class="c1">;; If you edit it by hand, you could mess it up,… |
|
1243 <span class="c1">;; Your init file should contain only one such i… |
|
1244 <span class="c1">;; If there is more than one, they won&#39;t… |
|
1245 <span class="o">&#39;</span><span class="p">(<… |
|
1246 <span class="o">&#39;</span><span class="p">(<… |
|
1247 <span class="p">(</span><span class="k">quote</s… |
|
1248 <span class="p">(</span><span class="s">&quot;… |
|
1249 <span class="o">&#39;</span><span class="p">(<… |
|
1250 <span class="o">&#39;</span><span class="p">(<… |
|
1251 <span class="o">&#39;</span><span class="p">(<… |
|
1252 <span class="o">&#39;</span><span class="p">(<… |
|
1253 <span class="o">&#39;</span><span class="p">(<… |
|
1254 <span class="o">&#39;</span><span class="p">(<… |
|
1255 <span class="o">&#39;</span><span class="p">(<… |
|
1256 <span class="o">&#39;</span><span class="p">(<… |
|
1257 <span class="o">&#39;</span><span class="p">(<… |
|
1258 <span class="o">&#39;</span><span class="p">(<… |
|
1259 <span class="o">&#39;</span><span class="p">(<… |
|
1260 <span class="o">&#39;</span><span class="p">(<… |
|
1261 <span class="o">&#39;</span><span class="p">(<… |
|
1262 <span class="p">(</span><span class="k">quote</s… |
|
1263 <span class="p">(</span><span class="nv">helm-lsp&… |
|
1264 <span class="o">&#39;</span><span class="p">(<… |
|
1265 <span class="o">&#39;</span><span class="p">(<… |
|
1266 <span class="o">&#39;</span><span class="p">(<… |
|
1267 <span class="o">&#39;</span><span class="p">(<… |
|
1268 <span class="p">(</span><span class="nv">custom-set-fa… |
|
1269 <span class="c1">;; custom-set-faces was added by Custom.</spa… |
|
1270 <span class="c1">;; If you edit it by hand, you could mess it up,… |
|
1271 <span class="c1">;; Your init file should contain only one such i… |
|
1272 <span class="c1">;; If there is more than one, they won&#39;t… |
|
1273 <span class="p">)</span> |
|
1274 </code></pre></div> |
|
1275 |
|
1276 <h2>Results</h2> |
|
1277 <p>I am very happy with rustic / rust-analyzer and the Language Se… |
|
1278 documentation on each thing when one moves the cursor around code is som… |
|
1279 that I never thought would work well in Emacs. I haven't decided if I l… |
|
1280 lsp-rust-analyzer-inlay-hints-mode</code> or if it drives me nuts;… |
|
1281 names of function arguments and inferred types among the code. I suppos… |
|
1282 turn it off and on as needed.</p> |
|
1283 <p>Some days ago, before using helm, I had projectile-mode to work… |
|
1284 checkouts and I was quite liking it. I haven't found how to configure |
|
1285 helm-projectile to work; I'll have to keep experimenting.</p></con… |
|
1286 map extracted from OpenStreetMap.</p> |
|
1287 <p>According to <a href="https://valgrind.org/docs/manual/ms-ma… |
|
1288 the following point during the execution of rsvg-convert. I pasted |
|
1289 only the part that refers to Bézier paths:</p> |
|
1290 <div class="highlight"><pre><span></span><cod… |
|
1291 <span class="nt">n</span> <span class="nt"&g… |
|
1292 map extracted from OpenStreetMap.</p> |
|
1293 <p>According to <a href="https://valgrind.org/docs/manual/ms-ma… |
|
1294 the following point during the execution of rsvg-convert. I pasted |
|
1295 only the part that refers to Bézier paths:</p> |
|
1296 <div class="highlight"><pre><span></span><cod… |
|
1297 <span class="nt">n</span> <span class="nt"&g… |
|
1298 <span class="nt">---------------------------------------------… |
|
1299 <span class="nt">1</span> <span class="nt">33</s… |
|
1300 <span class="nt">2</span> <span class="nt">-</spa… |
|
1301 <span class="o">|</span> <span class="nt">-</sp… |
|
1302 <span class="o">|</span> <span class="nt">-</… |
|
1303 <span class="o">|</span> <span class="nt">-<… |
|
1304 <span class="o">|</span> <span class="nt">-&… |
|
1305 <span class="o">|</span> <span class="nt">… |
|
1306 <span class="o">|</span> <span class="nt"&g… |
|
1307 <span class="o">|</span> <span class="nt"… |
|
1308 <span class="o">|</span> <span class="n… |
|
1309 <span class="o">|</span> <span class=… |
|
1310 <span class="nt">3</span> <span class="o">|</span… |
|
1311 <span class="o">|</span> |
|
1312 <span class="nt">4</span> <span class="nt">-</spa… |
|
1313 <span class="o">|</span> <span class="nt">-</sp… |
|
1314 <span class="o">|</span> <span class="nt">-</… |
|
1315 <span class="o">|</span> <span class="nt">-<… |
|
1316 <span class="o">|</span> <span class="nt">-&… |
|
1317 <span class="o">|</span> <span class="nt">… |
|
1318 <span class="nt">5</span> <span class="o">|</span… |
|
1319 </code></pre></div> |
|
1320 |
|
1321 <p>Line 1 has the totals, and we see that at that point the progr… |
|
1322 1,329,943,212 bytes on the heap.</p> |
|
1323 <p>Lines 3 and 5 give us a hint that <code>into_path</… |
|
1324 the function that converts a temporary/mutable <code>PathBuilder&l… |
|
1325 permanent/immutable <code>Path</code>.</p> |
|
1326 <p>Lines 2 and 4 indicate that the arrays of <code>PathCo… |
|
1327 inside those immutable <code>Path</code>s, use 24.88% + 3.… |
|
1328 program's memory; between both they use |
|
1329 352,523,448 + 50,990,328 = 403,513,776 bytes.</p> |
|
1330 <p>That is about 400 MB of <code>PathCommand</code>. … |
|
1331 <h2>What is in a PathCommand?</h2> |
|
1332 <p>A <code>Path</code> is a list of commands similar t… |
|
1333 in SVG to draw Bézier paths. It is a flat array of <code>PathCom… |
|
1334 <div class="highlight"><pre><span></span><cod… |
|
1335 <span class="w"> </span><span class="n">path_comman… |
|
1336 <span class="p">}</span><span class="w"></span> |
|
1337 |
|
1338 <span class="k">pub</span><span class="w"> </span&g… |
|
1339 <span class="w"> </span><span class="n">MoveTo</… |
|
1340 <span class="w"> </span><span class="n">LineTo</… |
|
1341 <span class="w"> </span><span class="n">CurveTo<… |
|
1342 <span class="w"> </span><span class="n">Arc</spa… |
|
1343 <span class="w"> </span><span class="n">ClosePath&l… |
|
1344 <span class="p">}</span><span class="w"></span> |
|
1345 </code></pre></div> |
|
1346 |
|
1347 <p>Let's see the variants of <code>PathCommand</code>:… |
|
1348 <ul> |
|
1349 <li><code>MoveTo</code>: 2 double-precision floating-p… |
|
1350 <li><code>LineTo</code>: same.</li> |
|
1351 <li><code>CurveTo</code>: 6 double-precision floating-… |
|
1352 <li><code>EllipticalArc</code>: 7 double-precision flo… |
|
1353 flags (see below).</li> |
|
1354 <li><code>ClosePath</code>: no extra data.</li> |
|
1355 </ul> |
|
1356 <p>These variants vary a lot in terms of size, and each element of… |
|
1357 <code>Path.path_commands</code> array occupies the maximum o… |
|
1358 (i.e. <code>sizeof::&lt;EllipticalArc&gt;</code>).&l… |
|
1359 <h2>A more compact representation</h2> |
|
1360 <p>Ideally, each command in the array would only occupy as much sp… |
|
1361 it needs.</p> |
|
1362 <p>We can represent a <code>Path</code> in a different… |
|
1363 <ul> |
|
1364 <li>A very compact array of commands without coordinates.</li&g… |
|
1365 <li>An array with coordinates only.</li> |
|
1366 </ul> |
|
1367 <p>That is, the following:</p> |
|
1368 <div class="highlight"><pre><span></span><cod… |
|
1369 <span class="w"> </span><span class="n">commands<… |
|
1370 <span class="w"> </span><span class="n">coords</… |
|
1371 <span class="p">}</span><span class="w"></span> |
|
1372 </code></pre></div> |
|
1373 |
|
1374 <p>The <code>coords</code> array is obvious; it is jus… |
|
1375 coordinates in the <code>Path</code> in the order in which t… |
|
1376 <p>And the <code>commands</code> array?</p> |
|
1377 <h3>PackedCommand</h3> |
|
1378 <p>We saw above that the biggest variant in <code>PathComman… |
|
1379 <code>Arc(EllipticalArc)</code>. Let's look inside it:</… |
|
1380 <div class="highlight"><pre><span></span><cod… |
|
1381 <span class="w"> </span><span class="k">pub</spa… |
|
1382 <span class="w"> </span><span class="k">pub</spa… |
|
1383 <span class="w"> </span><span class="k">pub</spa… |
|
1384 <span class="w"> </span><span class="k">pub</spa… |
|
1385 <span class="w"> </span><span class="k">pub</spa… |
|
1386 <span class="w"> </span><span class="k">pub</spa… |
|
1387 <span class="p">}</span><span class="w"></span> |
|
1388 </code></pre></div> |
|
1389 |
|
1390 <p>There are 7 <code>f64</code> floating-point number… |
|
1391 <code>large_arc</code> and <code>sweep</code>, a… |
|
1392 with two variants, with pretty names instead of just <code>true<… |
|
1393 <code>false</code>).</p> |
|
1394 <p>Thus, we have 7 doubles and two flags. Between the two flag… |
|
1395 are 4 possibilities.</p> |
|
1396 <p>Since no other <code>PathCommand</code> variant has… |
|
1397 following enum, which fits in a single byte:</p> |
|
1398 <div class="highlight"><pre><span></span><cod… |
|
1399 <span class="k">enum</span> <span class="nc">PackedCom… |
|
1400 <span class="w"> </span><span class="n">MoveTo</… |
|
1401 <span class="w"> </span><span class="n">LineTo</… |
|
1402 <span class="w"> </span><span class="n">CurveTo<… |
|
1403 <span class="w"> </span><span class="n">ArcSmallNeg… |
|
1404 <span class="w"> </span><span class="n">ArcSmallPos… |
|
1405 <span class="w"> </span><span class="n">ArcLargeNeg… |
|
1406 <span class="w"> </span><span class="n">ArcLargePos… |
|
1407 <span class="w"> </span><span class="n">ClosePath&l… |
|
1408 <span class="p">}</span><span class="w"></span> |
|
1409 </code></pre></div> |
|
1410 |
|
1411 <p>That is, simple values for <code>MoveTo</code>/etc.… |
|
1412 the different types of <code>Arc</code>.</p> |
|
1413 <h2>Packing a PathCommand into a PackedCommand</h2> |
|
1414 <p>In order to pack the array of <code>PathCommand</code&… |
|
1415 many coordinates each of its variants will produce:</p> |
|
1416 <div class="highlight"><pre><span></span><cod… |
|
1417 <span class="w"> </span><span class="k">fn</span… |
|
1418 <span class="w"> </span><span class="k">match&l… |
|
1419 <span class="w"> </span><span class="n">Pat… |
|
1420 <span class="w"> </span><span class="n">Pat… |
|
1421 <span class="w"> </span><span class="n">Pat… |
|
1422 <span class="w"> </span><span class="n">Pat… |
|
1423 <span class="w"> </span><span class="n">Pat… |
|
1424 <span class="w"> </span><span class="p">}</s… |
|
1425 <span class="w"> </span><span class="p">}</span&… |
|
1426 <span class="p">}</span><span class="w"></span> |
|
1427 </code></pre></div> |
|
1428 |
|
1429 <p>Then, we need to convert each <code>PathCommand</code&… |
|
1430 write its coordinates into an array:</p> |
|
1431 <div class="highlight"><pre><span></span><cod… |
|
1432 <span class="w"> </span><span class="k">fn</span… |
|
1433 <span class="w"> </span><span class="k">match&l… |
|
1434 <span class="w"> </span><span class="n">Pat… |
|
1435 <span class="w"> </span><span class="n">… |
|
1436 <span class="w"> </span><span class="n">… |
|
1437 <span class="w"> </span><span class="n">… |
|
1438 <span class="w"> </span><span class="p">}&l… |
|
1439 |
|
1440 <span class="w"> </span><span class="c1">//… |
|
1441 |
|
1442 <span class="w"> </span><span class="n">Pat… |
|
1443 <span class="w"> </span><span class="p">}</s… |
|
1444 <span class="w"> </span><span class="p">}</span&… |
|
1445 <span class="p">}</span><span class="w"></span> |
|
1446 </code></pre></div> |
|
1447 |
|
1448 <p>Let's look at that <code>to_packed_and_coords</code>… |
|
1449 <div class="highlight"><pre><span></span><cod… |
|
1450 <span class="w"> </span><span class="k">fn</span… |
|
1451 <span class="w"> </span><span class="n">coords&… |
|
1452 <span class="w"> </span><span class="n">coords&… |
|
1453 <span class="w"> </span><span class="n">coords&… |
|
1454 <span class="w"> </span><span class="n">coords&… |
|
1455 <span class="w"> </span><span class="n">coords&… |
|
1456 <span class="w"> </span><span class="n">coords&… |
|
1457 <span class="w"> </span><span class="n">coords&… |
|
1458 |
|
1459 <span class="w"> </span><span class="k">match&l… |
|
1460 <span class="w"> </span><span class="p">(&l… |
|
1461 <span class="w"> </span><span class="p">(&l… |
|
1462 <span class="w"> </span><span class="p">(&l… |
|
1463 <span class="w"> </span><span class="p">(&l… |
|
1464 <span class="w"> </span><span class="p">}</s… |
|
1465 <span class="w"> </span><span class="p">}</span&… |
|
1466 <span class="p">}</span><span class="w"></span> |
|
1467 </code></pre></div> |
|
1468 |
|
1469 <h2>Creating the compact Path</h2> |
|
1470 <p>Let's look at <code>PathBuilder::into_path</code> l… |
|
1471 <div class="highlight"><pre><span></span><cod… |
|
1472 <span class="w"> </span><span class="k">pub</spa… |
|
1473 <span class="w"> </span><span class="kd">let<… |
|
1474 <span class="w"> </span><span class="kd">let<… |
|
1475 <span class="w"> </span><span class="p">.&l… |
|
1476 <span class="w"> </span><span class="p">.&l… |
|
1477 <span class="w"> </span><span class="p">.&l… |
|
1478 </code></pre></div> |
|
1479 |
|
1480 <p>First we compute the total number of coordinates using <code… |
|
1481 each command <code>cmd</code> its <code>num_coordinate… |
|
1482 accumulator.</p> |
|
1483 <p>Now we know how much memory to allocate:</p> |
|
1484 <div class="highlight"><pre><span></span><cod… |
|
1485 <span class="w"> </span><span class="kd">let<… |
|
1486 </code></pre></div> |
|
1487 |
|
1488 <p>We use <code>Vec::with_capacity</code> to allocate … |
|
1489 need for the <code>packed_commands</code>; adding elements w… |
|
1490 <code>realloc()</code>, since we already know how many eleme… |
|
1491 <p>We use the <code>vec!</code> macro to create an arr… |
|
1492 <code>num_coords</code> times; that macro uses <code>w… |
|
1493 array we will use to store the coordinates for all the commands.</p&g… |
|
1494 <div class="highlight"><pre><span></span><cod… |
|
1495 </code></pre></div> |
|
1496 |
|
1497 <p>We get a mutable slice out of the whole array of coordinates.&l… |
|
1498 <div class="highlight"><pre><span></span><cod… |
|
1499 <span class="w"> </span><span class="kd">le… |
|
1500 <span class="w"> </span><span class="n">pac… |
|
1501 <span class="w"> </span><span class="n">coo… |
|
1502 <span class="w"> </span><span class="p">}</s… |
|
1503 </code></pre></div> |
|
1504 |
|
1505 <p>For each command, we see how many coordinates it will generate … |
|
1506 put that number in <code>n</code>. We get a mutable sub-sli… |
|
1507 <code>coords_slice</code> with only that number of elements,… |
|
1508 <code>to_packed</code> for each command.</p> |
|
1509 <p>At the end of each iteration we move the mutable slice to where… |
|
1510 next command's coordinates will go.</p> |
|
1511 <div class="highlight"><pre><span></span><cod… |
|
1512 <span class="w"> </span><span class="n">com… |
|
1513 <span class="w"> </span><span class="n">coo… |
|
1514 <span class="w"> </span><span class="p">}</s… |
|
1515 <span class="w"> </span><span class="p">}</span&… |
|
1516 </code></pre></div> |
|
1517 |
|
1518 <p>At the end, we create the final and immutable <code>Path&… |
|
1519 each array <code>into_boxed_slice</code> like the last time.… |
|
1520 the two arrays, the one with <code>PackedCommand</code>s and… |
|
1521 coordinates, occupy the minimum space they need.</p> |
|
1522 <h2>An iterator for Path</h2> |
|
1523 <p>This is all very well, but we also want it to be easy to iterat… |
|
1524 that compact representation; the <code>PathCommand</code> en… |
|
1525 beginning are very convenient to use and that's what the rest of the |
|
1526 code already uses. Let's make an iterator that unpacks what is inside |
|
1527 a <code>Path</code> and produces a <code>PathCommand&l… |
|
1528 <div class="highlight"><pre><span></span><cod… |
|
1529 <span class="w"> </span><span class="n">commands<… |
|
1530 <span class="w"> </span><span class="n">coords</… |
|
1531 <span class="p">}</span><span class="w"></span> |
|
1532 </code></pre></div> |
|
1533 |
|
1534 <p>We need an iterator over the array of <code>PackedCommand… |
|
1535 each command. However, to get elements of <code>coords</code&g… |
|
1536 use a slice of <code>f64</code> instead of an iterator.</… |
|
1537 <p>Let's look at the implementation of the iterator:</p> |
|
1538 <div class="highlight"><pre><span></span><cod… |
|
1539 <span class="w"> </span><span class="k">type</sp… |
|
1540 |
|
1541 <span class="w"> </span><span class="k">fn</span… |
|
1542 <span class="w"> </span><span class="k">if</… |
|
1543 <span class="w"> </span><span class="kd">le… |
|
1544 <span class="w"> </span><span class="kd">le… |
|
1545 <span class="w"> </span><span class="bp">se… |
|
1546 <span class="w"> </span><span class="nb">So… |
|
1547 <span class="w"> </span><span class="p">}</s… |
|
1548 <span class="w"> </span><span class="nb">No… |
|
1549 <span class="w"> </span><span class="p">}</s… |
|
1550 <span class="w"> </span><span class="p">}</span&… |
|
1551 <span class="p">}</span><span class="w"></span> |
|
1552 </code></pre></div> |
|
1553 |
|
1554 <p>Since we want each iteration to produce a <code>PathComma… |
|
1555 as having the associated <code>type Item = PathCommand</cod… |
|
1556 <p>If the <code>self.commands</code> iterator has anot… |
|
1557 another <code>PackedCommand</code> available.</p> |
|
1558 <p>We call <code>PathCommand::from_packed</code> with … |
|
1559 unpack a command and its coordinates. We see how many coordinates the |
|
1560 command consumed and re-slice <code>self.coords</code> accor… |
|
1561 commands, so that it now points to the coordinates for the next |
|
1562 command.</p> |
|
1563 <p>We return <code>Some(cmd)</code> if there was an el… |
|
1564 iterator is empty.</p> |
|
1565 <p>The implementation of <code>from_packed</code> is o… |
|
1566 bit from it:</p> |
|
1567 <div class="highlight"><pre><span></span><cod… |
|
1568 <span class="w"> </span><span class="k">fn</span… |
|
1569 <span class="w"> </span><span class="k">match&l… |
|
1570 <span class="w"> </span><span class="n">Pac… |
|
1571 <span class="w"> </span><span class="kd"&g… |
|
1572 <span class="w"> </span><span class="kd"&g… |
|
1573 <span class="w"> </span><span class="n">… |
|
1574 <span class="w"> </span><span class="p">}&l… |
|
1575 |
|
1576 <span class="w"> </span><span class="c1">//… |
|
1577 |
|
1578 <span class="w"> </span><span class="n">Pac… |
|
1579 <span class="w"> </span><span class="n">… |
|
1580 <span class="w"> </span><span class="n">… |
|
1581 <span class="w"> </span><span class="n">… |
|
1582 <span class="w"> </span><span class="p">)),… |
|
1583 |
|
1584 <span class="w"> </span><span class="n">Pac… |
|
1585 |
|
1586 <span class="w"> </span><span class="n">Pac… |
|
1587 |
|
1588 <span class="w"> </span><span class="n">Pac… |
|
1589 <span class="w"> </span><span class="p">}</s… |
|
1590 <span class="w"> </span><span class="p">}</span&… |
|
1591 <span class="p">}</span><span class="w"></span> |
|
1592 </code></pre></div> |
|
1593 |
|
1594 <h2>Results</h2> |
|
1595 <p>Before the changes (this is the same Massif heading as above):&… |
|
1596 <div class="highlight"><pre><span></span><cod… |
|
1597 <span class="c"> n time(i) total(B) useful</s… |
|
1598 <span class="nb">-------------------------------------------------… |
|
1599 <span class="c"> 33 24</span><span class="nt">,</sp… |
|
1600 <span class="c"> ^^^^^^^^^^^… |
|
1601 <span class="c"> boo<… |
|
1602 </code></pre></div> |
|
1603 |
|
1604 <p>After:</p> |
|
1605 <div class="highlight"><pre><span></span><cod… |
|
1606 <span class="c"> n time(i) total(B) useful</s… |
|
1607 <span class="nb">-------------------------------------------------… |
|
1608 <span class="c"> 28 26</span><span class="nt">,</sp… |
|
1609 <span class="c"> ^^^^^^^^^^^… |
|
1610 <span class="c"> oh yeah&… |
|
1611 </code></pre></div> |
|
1612 |
|
1613 <p>We went from using 1,329,943,212 bytes down to 1,023,147,907 … |
|
1614 that is, we knocked it down by 300 MB.</p> |
|
1615 <p>However, that is for the whole program. Above we saw that <… |
|
1616 occupies 403,513,776 bytes; how about now?</p> |
|
1617 <div class="highlight"><pre><span></span><cod… |
|
1618 <span class="o">|</span> <span class="o">-&gt;<… |
|
1619 <span class="o">|</span> <span class="o">-&gt;&l… |
|
1620 <span class="o">|</span> <span class="o">-&gt;… |
|
1621 <span class="o">|</span> <span class="o">-&g… |
|
1622 <span class="o">|</span> <span class="o">-&… |
|
1623 </code></pre></div> |
|
1624 |
|
1625 <p>Perfect. We went from occupying 403,513,776 bytes to just |
|
1626 81,525,328 bytes. Instead of <code>Path</code> data amount… |
|
1627 heap, it is just 7.45%.</p> |
|
1628 <p>I think we can stop worrying about <code>Path</code>… |
|
1629 this turned out without having to use <code>unsafe</code>.&l… |
|
1630 <h2>References</h2> |
|
1631 <ul> |
|
1632 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/e9… |
|
1633 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/cb… |
|
1634 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/b1… |
|
1635 </ul></content><category term="misc"></category><category term="li… |
|
1636 OpenStreetMap, and it has about 600,000 elements. Most of them are |
|
1637 <code>&lt;path&gt;</code>, that is, specifications f… |
|
1638 <p>A <code>&lt;path&gt;</code> can look like t… |
|
1639 <div class="highlight"><pre><span></span><cod… |
|
1640 </code></pre></div> |
|
1641 |
|
1642 <p>The …</p></summary><content type="html"><p>We got… |
|
1643 OpenStreetMap, and it has about 600,000 elements. Most of them are |
|
1644 <code>&lt;path&gt;</code>, that is, specifications f… |
|
1645 <p>A <code>&lt;path&gt;</code> can look like t… |
|
1646 <div class="highlight"><pre><span></span><cod… |
|
1647 </code></pre></div> |
|
1648 |
|
1649 <p>The <code>d</code> attribute contains a <a href=… |
|
1650 create a Bézier path, very similar to PostScript's operators. Librsvg |
|
1651 has the following to represent those commands:</p> |
|
1652 <div class="highlight"><pre><span></span><cod… |
|
1653 <span class="w"> </span><span class="n">MoveTo</… |
|
1654 <span class="w"> </span><span class="n">LineTo</… |
|
1655 <span class="w"> </span><span class="n">CurveTo<… |
|
1656 <span class="w"> </span><span class="n">Arc</spa… |
|
1657 <span class="w"> </span><span class="n">ClosePath&l… |
|
1658 <span class="p">}</span><span class="w"></span> |
|
1659 </code></pre></div> |
|
1660 |
|
1661 <p>Those commands get stored in an array, a <code>Vec</co… |
|
1662 <div class="highlight"><pre><span></span><cod… |
|
1663 <span class="w"> </span><span class="n">path_comman… |
|
1664 <span class="p">}</span><span class="w"></span> |
|
1665 </code></pre></div> |
|
1666 |
|
1667 <p>Librsvg translates each of the commands inside a <code>&a… |
|
1668 into a <code>PathCommand</code> and pushes it into the <c… |
|
1669 <code>PathBuilder</code>. When it is done parsing the attri… |
|
1670 <code>PathBuilder</code> remains as the final version of the… |
|
1671 <p>To let a <code>Vec</code> grow efficiently as items… |
|
1672 it, Rust makes the <code>Vec</code> grow by powers of 2. … |
|
1673 the <em>capacity</em> of the <code>Vec</code> is… |
|
1674 twice its capacity. That way there are only O(log₂n) calls to |
|
1675 <code>realloc()</code>, where <code>n</code> is … |
|
1676 <p>However, this means that once we are done adding items to the &… |
|
1677 there may still be some free space in it: <em>the capacity exceed… |
|
1678 length of the array</em>. The invariant is that |
|
1679 <code>vec.capacity() &gt;= vec.len()</code>.</p> |
|
1680 <p>First I wanted to shrink the <code>PathBuilder</code&g… |
|
1681 capacity in the end.</p> |
|
1682 <h2>First step: convert to Box&lt;[T]&gt;</h2> |
|
1683 <p>A "boxed slice" is a contiguous array in the heap, that cannot … |
|
1684 shrink. That is, it has no extra capacity, only a length.</p> |
|
1685 <p><code>Vec</code> has a method <a href="https://d… |
|
1686 eactly that: it consumes the vector and converts it into a boxed |
|
1687 slice without extra capacity. In its innards, it does a <code>rea… |
|
1688 on the <code>Vec</code>'s buffer to match its length.</p&… |
|
1689 <p>Let's see the numbers that Massif reports:</p> |
|
1690 <div class="highlight"><pre><span></span><cod… |
|
1691 <span class="c"> n time(i) total(B) useful</s… |
|
1692 <span class="nb">-------------------------------------------------… |
|
1693 <span class="c"> 23 22</span><span class="nt">,</sp… |
|
1694 <span class="c"> ^^^^^^^^^^^… |
|
1695 <span class="c"> before&… |
|
1696 |
|
1697 <span class="c"> 30 22</span><span class="nt">,</sp… |
|
1698 <span class="c"> ^^^^^^^^^^^… |
|
1699 <span class="c"> after&l… |
|
1700 </code></pre></div> |
|
1701 |
|
1702 <p>That is, we went from using 1,493,746,540 bytes on the heap to… |
|
1703 1,329,943,324 bytes. Simply removing extra capacity from the path |
|
1704 commands saves about 159 MB for this particular file.</p> |
|
1705 <h2>Second step: make the allocator do less work</h2> |
|
1706 <p>However, the <code>extra-heap</code> column in that… |
|
1707 like: there are 223,637,748 bytes in <code>malloc()</code>… |
|
1708 space in the heap.</p> |
|
1709 <p>I suppose that so many calls to <code>realloc()</code&… |
|
1710 fragmented.</p> |
|
1711 <p>It would be good to be able to read most of the <code>&am… |
|
1712 temporary buffers that don't need so many calls to <code>realloc()… |
|
1713 that in the end get copied to exact-sized buffers, without extra |
|
1714 capacity.</p> |
|
1715 <p>We can do just that with the <a href="https://docs.rs/smallv… |
|
1716 same API as <code>Vec</code>, but it can store small arrays … |
|
1717 stack, without an extra heap allocation. Once the capacity is full, |
|
1718 the stack buffer "spills" into a heap buffer automatically.</p> |
|
1719 <p>Most of the <code>d</code> attributes in the huge f… |
|
1720 fewer than 32 commands. That is, if we use the following:</p> |
|
1721 <div class="highlight"><pre><span></span><cod… |
|
1722 <span class="w"> </span><span class="n">path_comman… |
|
1723 <span class="p">}</span><span class="w"></span> |
|
1724 </code></pre></div> |
|
1725 |
|
1726 <p>We are saying that there can be up to 32 items in the <code… |
|
1727 without causing a heap allocation; once that is exceeded, it will work |
|
1728 like a normal <code>Vec</code>.</p> |
|
1729 <p>At the end we still do <code>into_boxed_slice</code>… |
|
1730 independent heap allocation with an exact size.</p> |
|
1731 <p>This reduces the <code>extra-heap</code> quite a bi… |
|
1732 <div class="highlight"><pre><span></span><cod… |
|
1733 <span class="c"> n time(i) total(B) useful</s… |
|
1734 <span class="nb">-------------------------------------------------… |
|
1735 <span class="c"> 33 24</span><span class="nt">,</sp… |
|
1736 <span class="c"> … |
|
1737 </code></pre></div> |
|
1738 |
|
1739 <p>Also, the total bytes shrink from 1,553,581,072 to |
|
1740 1,416,831,176 — we have a smaller heap because there is not so much |
|
1741 work for the allocator, and there are a lot fewer temporary blocks |
|
1742 when parsing the <code>d</code> attributes.</p> |
|
1743 <h2>Making the code prettier</h2> |
|
1744 <p>I put in the following:</p> |
|
1745 <div class="highlight"><pre><span></span><cod… |
|
1746 <span class="k">pub</span><span class="w"> </span&g… |
|
1747 <span class="w"> </span><span class="n">path_comman… |
|
1748 <span class="p">}</span><span class="w"></span> |
|
1749 |
|
1750 <span class="sd">/// This one is immutable</span> |
|
1751 <span class="k">pub</span><span class="w"> </span&g… |
|
1752 <span class="w"> </span><span class="n">path_comman… |
|
1753 <span class="p">}</span><span class="w"></span> |
|
1754 |
|
1755 <span class="k">impl</span><span class="w"> </span&… |
|
1756 <span class="w"> </span><span class="sd">/// Consum… |
|
1757 <span class="w"> </span><span class="k">pub</spa… |
|
1758 <span class="w"> </span><span class="n">Path<… |
|
1759 <span class="w"> </span><span class="n">pat… |
|
1760 <span class="w"> </span><span class="p">}</s… |
|
1761 <span class="w"> </span><span class="p">}</span&… |
|
1762 <span class="p">}</span><span class="w"></span> |
|
1763 </code></pre></div> |
|
1764 |
|
1765 <p>With that, <code>PathBuilder</code> is just a tempo… |
|
1766 immutable <code>Path</code> once we are done feeding it. &l… |
|
1767 slice of the exact size, without any extra capacity.</p> |
|
1768 <h2>Next steps</h2> |
|
1769 <p>All the coordinates in librsvg are stored as <code>f64<… |
|
1770 floating point numbers. The SVG/CSS spec says that single-precision |
|
1771 floats are enough, and that 64-bit floats should be used only for |
|
1772 geometric transformations.</p> |
|
1773 <p>I'm a bit scared to make that change; I'll have to look closely… |
|
1774 results of the test suite to see if rendered files change very much. |
|
1775 I suppose even big maps require only as much precision as <code>f3… |
|
1776 after all, that is what OpenStreetMap uses.</p> |
|
1777 <h2>References</h2> |
|
1778 <ul> |
|
1779 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/fc… |
|
1780 Box&lt;[T]&gt;</a></li> |
|
1781 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/ee… |
|
1782 </ul></content><category term="misc"></category><category term="li… |
|
1783 librsvg's DOM nodes smaller in memory. Since that time, there have |
|
1784 been some changes to the code; that is why in this post some of the |
|
1785 type names are different from last time's.</p> |
|
1786 <p>Every SVG element is represented with …</p></summary><c… |
|
1787 librsvg's DOM nodes smaller in memory. Since that time, there have |
|
1788 been some changes to the code; that is why in this post some of the |
|
1789 type names are different from last time's.</p> |
|
1790 <p>Every SVG element is represented with this struct:</p> |
|
1791 <div class="highlight"><pre><span></span><cod… |
|
1792 <span class="w"> </span><span class="n">element_typ… |
|
1793 <span class="w"> </span><span class="n">element_nam… |
|
1794 <span class="w"> </span><span class="n">id</span… |
|
1795 <span class="w"> </span><span class="n">class</s… |
|
1796 <span class="w"> </span><span class="n">specified_v… |
|
1797 <span class="w"> </span><span class="n">important_s… |
|
1798 <span class="w"> </span><span class="n">result</… |
|
1799 <span class="w"> </span><span class="n">transform&l… |
|
1800 <span class="w"> </span><span class="n">values</… |
|
1801 <span class="w"> </span><span class="n">cond</sp… |
|
1802 <span class="w"> </span><span class="n">style_attr&… |
|
1803 <span class="w"> </span><span class="n">element_imp… |
|
1804 <span class="p">}</span><span class="w"></span> |
|
1805 </code></pre></div> |
|
1806 |
|
1807 <p>The two biggest fields are the ones with types <code>Spec… |
|
1808 <code>ComputedValues</code>. These are the sizes of the who… |
|
1809 and those two types:</p> |
|
1810 <div class="highlight"><pre><span></span><cod… |
|
1811 sizeof SpecifiedValues: 824 |
|
1812 sizeof ComputedValues: 704 |
|
1813 </code></pre></div> |
|
1814 |
|
1815 <p>In this post, we'll reduce the size of <code>SpecifiedVal… |
|
1816 <h2>What is SpecifiedValues?</h2> |
|
1817 <p>If we have an element like this:</p> |
|
1818 <div class="highlight"><pre><span></span><cod… |
|
1819 </code></pre></div> |
|
1820 |
|
1821 <p>The values of the style properties <code>stroke-width<… |
|
1822 stored in a <code>SpecifiedValues</code> struct. This struc… |
|
1823 fields, one for each possible style property:</p> |
|
1824 <div class="highlight"><pre><span></span><cod… |
|
1825 <span class="w"> </span><span class="n">baseline_sh… |
|
1826 <span class="w"> </span><span class="n">clip_path&l… |
|
1827 <span class="w"> </span><span class="n">clip_rule&l… |
|
1828 <span class="w"> </span><span class="sd">/// ...<… |
|
1829 <span class="w"> </span><span class="n">stroke</… |
|
1830 <span class="w"> </span><span class="n">stroke_widt… |
|
1831 <span class="w"> </span><span class="sd">/// ...<… |
|
1832 <span class="p">}</span><span class="w"></span> |
|
1833 </code></pre></div> |
|
1834 |
|
1835 <p>Each field is a <code>SpecifiedValue&lt;T&gt;<… |
|
1836 CSS/SVG, a style property can be unspecified, or it can have an |
|
1837 <code>inherit</code> value to force the property to be copie… |
|
1838 parent, or it can actually have a specified value. Librsvg represents |
|
1839 these as follows:</p> |
|
1840 <div class="highlight"><pre><span></span><cod… |
|
1841 <span class="k">where</span><span class="w"></span&… |
|
1842 <span class="w"> </span><span class="n">T</span&… |
|
1843 <span class="p">{</span><span class="w"></span> |
|
1844 <span class="w"> </span><span class="n">Unspecified… |
|
1845 <span class="w"> </span><span class="n">Inherit<… |
|
1846 <span class="w"> </span><span class="n">Specified&l… |
|
1847 <span class="p">}</span><span class="w"></span> |
|
1848 </code></pre></div> |
|
1849 |
|
1850 <p>Now, <code>SpecifiedValues</code> has a bunch of fi… |
|
1851 one for each of the style properties that librsvg supports. That is |
|
1852 why <code>SpecifiedValues</code> has a size of 824 bytes; i… |
|
1853 sub-structure within <code>Element</code>, and it would be g… |
|
1854 size.</p> |
|
1855 <h2>Not all properties are specified</h2> |
|
1856 <p>Let's go back to the chunk of SVG from above:</p> |
|
1857 <div class="highlight"><pre><span></span><cod… |
|
1858 </code></pre></div> |
|
1859 |
|
1860 <p>Here we only have two specified properties, so the <code>… |
|
1861 <code>stroke</code> fields of <code>SpecifiedValues<… |
|
1862 <code>SpecifiedValue::Specified(something)</code> and all th… |
|
1863 be left as <code>SpecifiedValue::Unspecified</code>.</p&g… |
|
1864 <p>It would be good to store only complete values for the properti… |
|
1865 are specified, and just a small flag for unset properties.</p> |
|
1866 <h2>Another way to represent the set of properties</h2> |
|
1867 <p>Since there is a maximum of 47 properties per element (or more… |
|
1868 librsvg adds support for extra ones), we can have a small array of |
|
1869 47 bytes. Each byte contains the index within another array that |
|
1870 contains only the values of specified properties, or a sentinel value |
|
1871 for properties that are unset.</p> |
|
1872 <p>First, I made an enum that fits in a <code>u8</code>… |
|
1873 the sentinel value, which also gives us the total number of |
|
1874 properties. The <code>#[repr(u8)]</code> guarantees that th… |
|
1875 byte.</p> |
|
1876 <div class="highlight"><pre><span></span><cod… |
|
1877 <span class="k">enum</span> <span class="nc">PropertyI… |
|
1878 <span class="w"> </span><span class="n">BaselineShi… |
|
1879 <span class="w"> </span><span class="n">ClipPath<… |
|
1880 <span class="w"> </span><span class="n">ClipRule<… |
|
1881 <span class="w"> </span><span class="n">Color</s… |
|
1882 <span class="w"> </span><span class="c1">// ...<… |
|
1883 <span class="w"> </span><span class="n">WritingMode… |
|
1884 <span class="w"> </span><span class="n">XmlLang<… |
|
1885 <span class="w"> </span><span class="n">XmlSpace<… |
|
1886 <span class="w"> </span><span class="n">UnsetProper… |
|
1887 <span class="p">}</span><span class="w"></span> |
|
1888 </code></pre></div> |
|
1889 |
|
1890 <p>Also, since before these changes there was the following monste… |
|
1891 represent "which property is this" plus the property's value:</p> |
|
1892 <div class="highlight"><pre><span></span><cod… |
|
1893 <span class="w"> </span><span class="n">BaselineShi… |
|
1894 <span class="w"> </span><span class="n">ClipPath<… |
|
1895 <span class="w"> </span><span class="n">ClipRule<… |
|
1896 <span class="w"> </span><span class="n">Color</s… |
|
1897 <span class="w"> </span><span class="c1">// ...<… |
|
1898 <span class="p">}</span><span class="w"></span> |
|
1899 </code></pre></div> |
|
1900 |
|
1901 <p>I changed the definition of <code>SpecifiedValues</cod… |
|
1902 to store which properties are specified, and another only with the |
|
1903 values for the properties that are actually specified:</p> |
|
1904 <div class="highlight"><pre><span></span><cod… |
|
1905 <span class="w"> </span><span class="n">indices<… |
|
1906 <span class="w"> </span><span class="n">props</s… |
|
1907 <span class="p">}</span><span class="w"></span> |
|
1908 </code></pre></div> |
|
1909 |
|
1910 <p>There is a thing that is awkward in Rust, or which I haven't fo… |
|
1911 to solve in a nicer way: given a <code>ParsedProperty</code>… |
|
1912 corresponding <code>PropertyId</code> for its discriminant. … |
|
1913 thing:</p> |
|
1914 <div class="highlight"><pre><span></span><cod… |
|
1915 <span class="w"> </span><span class="k">fn</span… |
|
1916 <span class="w"> </span><span class="k">use<… |
|
1917 |
|
1918 <span class="w"> </span><span class="k">match&l… |
|
1919 <span class="w"> </span><span class="n">Bas… |
|
1920 <span class="w"> </span><span class="n">Cli… |
|
1921 <span class="w"> </span><span class="n">Cli… |
|
1922 <span class="w"> </span><span class="n">Col… |
|
1923 <span class="w"> </span><span class="c1">//… |
|
1924 <span class="w"> </span><span class="p">}</s… |
|
1925 <span class="w"> </span><span class="p">}</span&… |
|
1926 <span class="p">}</span><span class="w"></span> |
|
1927 </code></pre></div> |
|
1928 |
|
1929 <h2>Initialization</h2> |
|
1930 <p>First, we want to initialize an empty <code>SpecifiedValu… |
|
1931 element of the the <code>indices</code> array is set to the … |
|
1932 means that the corresponding property is not set:</p> |
|
1933 <div class="highlight"><pre><span></span><cod… |
|
1934 <span class="w"> </span><span class="k">fn</span… |
|
1935 <span class="w"> </span><span class="n">Specifi… |
|
1936 <span class="w"> </span><span class="n">ind… |
|
1937 <span class="w"> </span><span class="n">pro… |
|
1938 <span class="w"> </span><span class="p">}</s… |
|
1939 <span class="w"> </span><span class="p">}</span&… |
|
1940 <span class="p">}</span><span class="w"></span> |
|
1941 </code></pre></div> |
|
1942 |
|
1943 <p>That sets the <code>indices</code> field to an arra… |
|
1944 <code>PropertyId::UnsetProperty</code> sentinel value. Also… |
|
1945 is empty; it hasn't even had a block of memory allocated for it yet. |
|
1946 That way, SVG elements without style properties don't use any extra |
|
1947 memory.</p> |
|
1948 <h2>Which properties are specified and what are their indices?<… |
|
1949 <p>Second, we want a function that will give us the index in <c… |
|
1950 some property, or that will tell us if the property has not been set |
|
1951 yet:</p> |
|
1952 <div class="highlight"><pre><span></span><cod… |
|
1953 <span class="w"> </span><span class="k">fn</span… |
|
1954 <span class="w"> </span><span class="kd">let<… |
|
1955 |
|
1956 <span class="w"> </span><span class="k">if</… |
|
1957 <span class="w"> </span><span class="nb">No… |
|
1958 <span class="w"> </span><span class="p">}</s… |
|
1959 <span class="w"> </span><span class="nb">So… |
|
1960 <span class="w"> </span><span class="p">}</s… |
|
1961 <span class="w"> </span><span class="p">}</span&… |
|
1962 <span class="p">}</span><span class="w"></span> |
|
1963 </code></pre></div> |
|
1964 |
|
1965 <p>(If someone passes <code>id = PropertyId::UnsetProperty&l… |
|
1966 to <code>indices</code> will panic, which is what we want, s… |
|
1967 valid property id.)</p> |
|
1968 <h2>Change a property's value</h2> |
|
1969 <p>Third, we want to set the value of a property that has not been… |
|
1970 or change the value of one that was already specified:</p> |
|
1971 <div class="highlight"><pre><span></span><cod… |
|
1972 <span class="w"> </span><span class="k">fn</span… |
|
1973 <span class="w"> </span><span class="kd">let<… |
|
1974 |
|
1975 <span class="w"> </span><span class="k">if</… |
|
1976 <span class="w"> </span><span class="bp">se… |
|
1977 <span class="w"> </span><span class="p">}</s… |
|
1978 <span class="w"> </span><span class="bp">se… |
|
1979 <span class="w"> </span><span class="kd">le… |
|
1980 <span class="w"> </span><span class="bp">se… |
|
1981 <span class="w"> </span><span class="p">}</s… |
|
1982 <span class="w"> </span><span class="p">}</span&… |
|
1983 <span class="p">}</span><span class="w"></span> |
|
1984 </code></pre></div> |
|
1985 |
|
1986 <p>In the first case in the <code>if</code>, the prope… |
|
1987 just replace its value. In the second case, the property was not set; |
|
1988 we add it to the <code>props</code> array and store its resu… |
|
1989 <code>indices</code>.</p> |
|
1990 <h2>Results</h2> |
|
1991 <p>Before:</p> |
|
1992 <div class="highlight"><pre><span></span><cod… |
|
1993 sizeof SpecifiedValues: 824 |
|
1994 </code></pre></div> |
|
1995 |
|
1996 <p>After:</p> |
|
1997 <div class="highlight"><pre><span></span><cod… |
|
1998 sizeof SpecifiedValues: 72 |
|
1999 </code></pre></div> |
|
2000 |
|
2001 <p>The pathological file <a href="https://gitlab.gnome.org/GNOM… |
|
2002 463,412,720 bytes in memory before these changes. After the changes, |
|
2003 it uses 314,526,136 bytes.</p> |
|
2004 <p>I also measured memory consumption for a normal file, in this c… |
|
2005 <a href="https://gitlab.gnome.org/Teams/Design/icon-development-kit/-… |
|
2006 uses 17 MB; the new version only 13 MB.</p> |
|
2007 <h2>How to keep fine-tuning this</h2> |
|
2008 <p>For now, I am satisfied with <code>SpecifiedValues</co… |
|
2009 still be made smaller:</p> |
|
2010 <ul> |
|
2011 <li> |
|
2012 <p>The crate <a href="https://lib.rs/crates/tagged-box">tagg… |
|
2013 an enum-of-boxes, and codifies the enum's discriminant into the |
|
2014 box's pointer. This way each variant occupies the minimum possible |
|
2015 memory, although in a separately-allocated block, and the container |
|
2016 itself uses only a pointer. I am not sure if this is worth it; each |
|
2017 <code>ParsedProperty</code> is 64 bytes, but the flat arr… |
|
2018 Vec&lt;ParsedProperty&gt;</code> is very appealing in a … |
|
2019 I have not checked the sizes of each individual property to see if |
|
2020 they vary a lot among them.</p> |
|
2021 </li> |
|
2022 <li> |
|
2023 <p>Look for a crate that lets us have the properties in a single m… |
|
2024 block, a kind of arena with variable types. This can be implemented |
|
2025 with a bit of <code>unsafe</code>, but one has to be caref… |
|
2026 of different types.</p> |
|
2027 </li> |
|
2028 <li> |
|
2029 <p>The crate <a href="https://lib.rs/crates/enum_set2">enum_… |
|
2030 compact bit array. If we changed the representation of |
|
2031 <code>SpecifiedValue</code>, this would reduce the <cod… |
|
2032 minimum.</p> |
|
2033 </li> |
|
2034 </ul> |
|
2035 <p>If someone wants to dedicate some time to implement and measure… |
|
2036 I would be very grateful.</p> |
|
2037 <h2>Next steps</h2> |
|
2038 <p>According to Massif, the next thing is to keep making <code&… |
|
2039 smaller. The next thing to shrink is <code>ComputedValues</cod… |
|
2040 route is to do exactly the same as I did for <code>SpecifiedValues… |
|
2041 not sure if it would be better to try to <a href="https://gitlab.gnom… |
|
2042 structs</a> between elements.</p></content><category term="m… |
|
2043 willing to mentor the following project for librsvg.</p> |
|
2044 <h2>Project: Revamp the text engine in librsvg</h2> |
|
2045 <p>Librsvg supports only a few features of the <a href="https:/… |
|
2046 specification</a>. It requires |
|
2047 extra features to be really useful:</p> |
|
2048 <ul> |
|
2049 <li> |
|
2050 <p><strong>Proper bidirectional support …</strong>&l… |
|
2051 willing to mentor the following project for librsvg.</p> |
|
2052 <h2>Project: Revamp the text engine in librsvg</h2> |
|
2053 <p>Librsvg supports only a few features of the <a href="https:/… |
|
2054 specification</a>. It requires |
|
2055 extra features to be really useful:</p> |
|
2056 <ul> |
|
2057 <li> |
|
2058 <p><strong>Proper bidirectional support.</strong> Lib… |
|
2059 <code>unicode-bidi</code> properties for text elements, amon… |
|
2060 very rudimentary fashion. It just translates those properties to |
|
2061 Pango terminology and asks <code>PangoLayout</code> to lay o… |
|
2062 really wants finer control of that, for which...</p> |
|
2063 </li> |
|
2064 <li> |
|
2065 <p>... ideally you would make librsvg <strong>use Harfbuzz d… |
|
2066 wrapper that is close to its level of operation. Pango is a bit too high |
|
2067 level for the needs of SVG.</p> |
|
2068 </li> |
|
2069 <li> |
|
2070 <p>Manual layout of text glyphs. After a text engine like Harfbuzz |
|
2071 does the shaping, librsvg would need to lay out the produced glyphs in |
|
2072 the way of the SVG attributes <code>dx, dy, x, y</code>, etc… |
|
2073 specification has the algorithms for this.</p> |
|
2074 </li> |
|
2075 <li> |
|
2076 <p>The cherry on top: text-on-a-path. Again, the spec has the det… |
|
2077 You would make Wikimedia content creators very happy with this!</p> |
|
2078 </li> |
|
2079 </ul> |
|
2080 <p><strong>Requirements:</strong> Rust for programming… |
|
2081 Unicode concepts and text layout. Familiarity with Cairo and Harfbuzz |
|
2082 would help a lot. Preference will be given to people who can write a |
|
2083 right-to-left human language, <strong>or</strong> a language… |
|
2084 shaping.</p> |
|
2085 <p><a href="https://wiki.gnome.org/Outreach/SummerOfCode/Studen… |
|
2086 use cases, which is basically rendering icons. But for SVG files with |
|
2087 thousands of elements, it could do a lot better.</p> |
|
2088 <h2>Memory consumption in the DOM</h2> |
|
2089 <p>Librsvg shares some common problems with web browsers: it must |
|
2090 construct a …</p></summary><content type="html"><p>Librsvg… |
|
2091 use cases, which is basically rendering icons. But for SVG files with |
|
2092 thousands of elements, it could do a lot better.</p> |
|
2093 <h2>Memory consumption in the DOM</h2> |
|
2094 <p>Librsvg shares some common problems with web browsers: it must |
|
2095 construct a DOM tree in memory with SVG elements, and keep a bunch of |
|
2096 information for each of the tree's nodes. For example, each SVG |
|
2097 element may have an <code>id</code> attribute, or a <code… |
|
2098 transformation matrix; etc.</p> |
|
2099 <p>Apart from the tree node metadata (pointers to sibling and pare… |
|
2100 nodes), each node has this:</p> |
|
2101 <div class="highlight"><pre><span></span><cod… |
|
2102 <span class="k">pub</span><span class="w"> </span&g… |
|
2103 <span class="w"> </span><span class="n">node_type&l… |
|
2104 <span class="w"> </span><span class="n">element_nam… |
|
2105 <span class="w"> </span><span class="n">id</span… |
|
2106 <span class="w"> </span><span class="n">class</s… |
|
2107 <span class="w"> </span><span class="n">specified_v… |
|
2108 <span class="w"> </span><span class="n">important_s… |
|
2109 <span class="w"> </span><span class="n">result</… |
|
2110 <span class="w"> </span><span class="n">transform&l… |
|
2111 <span class="w"> </span><span class="n">values</… |
|
2112 <span class="w"> </span><span class="n">cond</sp… |
|
2113 <span class="w"> </span><span class="n">style_attr&… |
|
2114 |
|
2115 <span class="w"> </span><span class="n">node_impl&l… |
|
2116 <span class="p">}</span><span class="w"></span> |
|
2117 </code></pre></div> |
|
2118 |
|
2119 <p>On a 64-bit box, that <code>NodeData</code> struct … |
|
2120 are the <code>SpecifiedValues</code> (824 bytes) and <c… |
|
2121 <p>Librsvg represents <em>all</em> tree nodes with tha… |
|
2122 like this:</p> |
|
2123 <div class="highlight"><pre><span></span><cod… |
|
2124 <span class="nt">&lt;rect</span> <span class="na"&g… |
|
2125 <span class="nt">&lt;path</span> <span class="na"&g… |
|
2126 <span class="nt">&lt;text</span> <span class="na"&g… |
|
2127 <span class="c">&lt;!-- etc --&gt;</span> |
|
2128 <span class="nt">&lt;/svg&gt;</span> |
|
2129 </code></pre></div> |
|
2130 |
|
2131 <p>There are 4 elements in that file. However, there are also tr… |
|
2132 for the XML text nodes, that is, the whitespace between tags and the |
|
2133 "<code>Hello</code>" inside the <code>&lt;text&… |
|
2134 <p>The contents of each of those text nodes is tiny (a newline and… |
|
2135 a couple of spaces), but each node still takes up at least 1808 bytes |
|
2136 from the <code>NodeData</code> struct, plus the size of the … |
|
2137 <p>Let's refactor this to make it easier to remove that overhead.&… |
|
2138 <h2>First step: separate text nodes from element nodes</h2> |
|
2139 <p>Internally, librsvg represents XML text nodes with a <code&g… |
|
2140 which is basically a string with some extra stuff. All the concrete |
|
2141 structs for tree node types must implement a trait called <code>No… |
|
2142 and <code>NodeChars</code> is no exception:</p> |
|
2143 <div class="highlight"><pre><span></span><cod… |
|
2144 <span class="w"> </span><span class="c1">// a string… |
|
2145 <span class="p">}</span><span class="w"></span> |
|
2146 |
|
2147 <span class="k">impl</span><span class="w"> </span&… |
|
2148 <span class="w"> </span><span class="c1">// a mostly… |
|
2149 <span class="p">}</span><span class="w"></span> |
|
2150 </code></pre></div> |
|
2151 |
|
2152 <p>You don't see it in the definition of <code>NodeData</… |
|
2153 section, but for a text node, the <code>NodeData.node_impl</cod… |
|
2154 point to a heap-allocated <code>NodeChars</code> (it can do … |
|
2155 <code>NodeChars</code> implements <code>NodeTrait</… |
|
2156 Box&lt;dyn NodeTrait&gt;</code>).</p> |
|
2157 <p>First, I turned the <code>NodeData</code> struct in… |
|
2158 and moved all of its previous fields to an <code>Element</code&… |
|
2159 <div class="highlight"><pre><span></span><cod… |
|
2160 <span class="k">pub</span><span class="w"> </span&g… |
|
2161 <span class="w"> </span><span class="n">Element<… |
|
2162 <span class="w"> </span><span class="n">Text</sp… |
|
2163 <span class="p">}</span><span class="w"></span> |
|
2164 |
|
2165 <span class="c1">// This is the old struct with a different name&l… |
|
2166 <span class="k">pub</span><span class="w"> </span&g… |
|
2167 <span class="w"> </span><span class="n">node_type&l… |
|
2168 <span class="w"> </span><span class="n">element_nam… |
|
2169 <span class="w"> </span><span class="n">id</span… |
|
2170 <span class="w"> </span><span class="n">class</s… |
|
2171 <span class="w"> </span><span class="n">specified_v… |
|
2172 <span class="w"> </span><span class="n">important_s… |
|
2173 <span class="w"> </span><span class="n">result</… |
|
2174 <span class="w"> </span><span class="n">transform&l… |
|
2175 <span class="w"> </span><span class="n">values</… |
|
2176 <span class="w"> </span><span class="n">cond</sp… |
|
2177 <span class="w"> </span><span class="n">style_attr&… |
|
2178 <span class="w"> </span><span class="n">node_impl&l… |
|
2179 <span class="p">}</span><span class="w"></span> |
|
2180 </code></pre></div> |
|
2181 |
|
2182 <p>The size of a Rust enum is the maximum of the sizes of its vari… |
|
2183 plus a little extra for the discriminant (you can think of a C struct |
|
2184 with an int for the discriminant, and a union of variants).</p> |
|
2185 <p>The code <a href="https://gitlab.gnome.org/GNOME/librsvg/-/c… |
|
2186 changes</a> |
|
2187 to split <code>NodeData</code> in this way, by adding access… |
|
2188 functions to each of the <code>Element</code> or <code>… |
|
2189 is one of those refactors where you can just change the declaration, |
|
2190 and walk down the compiler's errors to make each case use the accesors |
|
2191 instead of whatever was done before.</p> |
|
2192 <h2>Second step: move the Element variant to a separate allocation… |
|
2193 <p>Now, <a href="https://gitlab.gnome.org/GNOME/librsvg/-/commi… |
|
2194 this</a>:</p> |
|
2195 <div class="highlight"><pre><span></span><cod… |
|
2196 <span class="w"> </span><span class="n">Element<… |
|
2197 <span class="w"> </span><span class="n">Text</sp… |
|
2198 <span class="p">}</span><span class="w"></span> |
|
2199 </code></pre></div> |
|
2200 |
|
2201 <p>That way, the <code>Element</code> variant is the s… |
|
2202 pointer to the heap-allocated <code>Box</code>), and the <… |
|
2203 as <code>NodeChars</code> as usual.</p> |
|
2204 <p>This means that <code>Element</code> nodes are just… |
|
2205 extra pointer, plus an extra heap allocation.</p> |
|
2206 <p>However, the <code>Text</code> nodes get a lot smal… |
|
2207 <ul> |
|
2208 <li>Before: <code>sizeof::&lt;NodeData&gt;() = 1808&… |
|
2209 <li>After: <code>sizeof::&lt;NodeData&gt;() = 72<… |
|
2210 </ul> |
|
2211 <p>By making the <code>Element</code> variant a lot sm… |
|
2212 which is just a pointer), it has no extra overhead on the <code>Te… |
|
2213 variant.</p> |
|
2214 <p>This means that in the SVG file, all the whitespace between XML |
|
2215 elements now takes a lot less memory.</p> |
|
2216 <h2>Some numbers from a pathological file</h2> |
|
2217 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/issues/42"&g… |
|
2218 an SVG file that is just a <code>&lt;use&gt;</code> … |
|
2219 per line:</p> |
|
2220 <div class="highlight"><pre><span></span><cod… |
|
2221 <span class="nt">&lt;defs&gt;</span> |
|
2222 <span class="nt">&lt;symbol</span> <span class="n… |
|
2223 <span class="c">&lt;!-- a few elements here --&gt;&l… |
|
2224 <span class="nt">&lt;/symbol&gt;</span> |
|
2225 <span class="nt">&lt;/defs&gt;</span> |
|
2226 |
|
2227 <span class="nt">&lt;use</span> <span class="na">… |
|
2228 <span class="nt">&lt;use</span> <span class="na">… |
|
2229 <span class="nt">&lt;use</span> <span class="na">… |
|
2230 <span class="c">&lt;!-- about 196,000 similar lines --&g… |
|
2231 <span class="nt">&lt;/svg&gt;</span> |
|
2232 </code></pre></div> |
|
2233 |
|
2234 <p>So we have around 196,000 elements. According to <a href="h… |
|
2235 tool</a>, this makes <code>rsvg-convert</code> allocat… |
|
2236 old version, versus 463,412,720 bytes in the new version, or about 60% |
|
2237 of the space.</p> |
|
2238 <h2>Next steps</h2> |
|
2239 <p>There is a lot of repetition in the text nodes of a typical SVG… |
|
2240 For example, in that pathological file above, most of the whitespace is |
|
2241 identical: between each element there is a newline and two spaces. |
|
2242 Instead of having thousands of little allocations, all with the same |
|
2243 string, there could be a pool of shared strings. Files with "real" |
|
2244 indentation could get benefits from sharing the whitespace-only text |
|
2245 nodes.</p> |
|
2246 <p>Real browser engines are very careful to share the style structs |
|
2247 across elements if possible. Look for "style struct sharing" in |
|
2248 <a href="https://hacks.mozilla.org/2017/08/inside-a-super-fast-css-en… |
|
2249 going to take some good work in librsvg, but we can get there |
|
2250 gradually.</p> |
|
2251 <h2>References</h2> |
|
2252 <ul> |
|
2253 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/merge_req… |
|
2254 refactoring</a></li> |
|
2255 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/74… |
|
2256 Text</a></li> |
|
2257 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/-/commit/83… |
|
2258 smaller</a>. |
|
2259 The commit message has parts of the massif log with all the |
|
2260 interesting numbers.</li> |
|
2261 <li><a href="https://gitlab.gnome.org/GNOME/librsvg/issues/528"… |
|
2262 consumption</a></li> |
|
2263 </ul></content><category term="misc"></category><category term="li… |
|
2264 to other languages through <a href="https://people.gnome.org/~federic… |
|
2265 <p>You could call this a use of the <a href="https://en.wikiped… |
|
2266 <a href="https://gnome.pages.gitlab.gnome.org/librsvg/doc/rsvg_intern… |
|
2267 implementation of librsvg, and exports an …</p></summary><conten… |
|
2268 to other languages through <a href="https://people.gnome.org/~federic… |
|
2269 <p>You could call this a use of the <a href="https://en.wikiped… |
|
2270 <a href="https://gnome.pages.gitlab.gnome.org/librsvg/doc/rsvg_intern… |
|
2271 implementation of librsvg, and exports an interface with many knobs |
|
2272 that are not exposed from the public APIs. The knobs are to allow for |
|
2273 the variations in each of those APIs.</p> |
|
2274 <p>This post is about some interesting things that have come up du… |
|
2275 the creation/separation of those public APIs, and the implications of |
|
2276 having an internals library that implements both.</p> |
|
2277 <h2>Initial code organization</h2> |
|
2278 <p>When librsvg was being ported to Rust, it just had an <code&… |
|
2279 crate that compiled as a <code>staticlib</code> to a <cod… |
|
2280 later linked into the final <code>librsvg.so</code>.</p&g… |
|
2281 <p>Eventually the code got to the point where it was feasible to p… |
|
2282 toplevel C API to Rust. This was relatively easy to do, since |
|
2283 everything else underneath was already in Rust. At that point I |
|
2284 became interested <a href="https://people.gnome.org/~federico/blog/a-… |
|
2285 first to port the test suite to Rust and be able to run tests in |
|
2286 parallel, and then to actually have a public API in Rust with more |
|
2287 modern idioms than the historical, GObject-based API in C.</p> |
|
2288 <p>Version <a href="https://gitlab.gnome.org/GNOME/librsvg/-/ta… |
|
2289 a C API.</p> |
|
2290 <p>Most of the C API of librsvg is in the <code>RsvgHandle&l… |
|
2291 <code>RsvgHandle</code> gets loaded with SVG data from a fil… |
|
2292 then gets rendered to a Cairo context. The naming of Rust source |
|
2293 files more or less matched the C source files, so where there was |
|
2294 <code>rsvg-handle.c</code> initially, later we had <code&… |
|
2295 part of that code.</p> |
|
2296 <p>So, <code>handle.rs</code> had the Rust internals o… |
|
2297 a bunch of <code>extern "C"</code> functions callable from C… |
|
2298 this function in the public C API:</p> |
|
2299 <div class="highlight"><pre><span></span><cod… |
|
2300 <span class="n">GFile</span>… |
|
2301 </code></pre></div> |
|
2302 |
|
2303 <p>The corresponding Rust implementation <a href="https://gitla… |
|
2304 <div class="highlight"><pre><span></span><cod… |
|
2305 <span class="k">pub</span><span class="w"> </span&g… |
|
2306 <span class="w"> </span><span class="n">raw_handle&… |
|
2307 <span class="w"> </span><span class="n">raw_gfile&l… |
|
2308 <span class="p">)</span><span class="w"> </span>… |
|
2309 <span class="w"> </span><span class="kd">let</sp… |
|
2310 |
|
2311 <span class="w"> </span><span class="fm">assert!<… |
|
2312 <span class="w"> </span><span class="kd">let</sp… |
|
2313 |
|
2314 <span class="w"> </span><span class="n">rhandle<… |
|
2315 <span class="p">}</span><span class="w"></span> |
|
2316 </code></pre></div> |
|
2317 |
|
2318 <ol> |
|
2319 <li>Get the Rust struct corresponding to the C GObject.</li> |
|
2320 <li>Check the arguments.</li> |
|
2321 <li>Convert from C GObject reference to Rust reference.</li> |
|
2322 <li>Call the actual implementation of <code>set_base_gfile&l… |
|
2323 struct.</li> |
|
2324 </ol> |
|
2325 <p>You can see that this function takes in arguments with C types,… |
|
2326 converts them to Rust types. It's basically just glue between the C |
|
2327 code and the actual implementation.</p> |
|
2328 <p>Then, the actual implementation of <code>set_base_gfile&l… |
|
2329 this</a>:</p> |
|
2330 <div class="highlight"><pre><span></span><cod… |
|
2331 <span class="w"> </span><span class="k">fn</span… |
|
2332 <span class="w"> </span><span class="k">if</… |
|
2333 <span class="w"> </span><span class="bp">se… |
|
2334 <span class="w"> </span><span class="p">}</s… |
|
2335 <span class="w"> </span><span class="n">rsv… |
|
2336 <span class="w"> </span><span class="p">}</s… |
|
2337 <span class="w"> </span><span class="p">}</span&… |
|
2338 <span class="p">}</span><span class="w"></span> |
|
2339 </code></pre></div> |
|
2340 |
|
2341 <p>This is an actual method for a Rust <code>Handle</code… |
|
2342 types as arguments — no conversions are necessary here. However, |
|
2343 there is a pesky call to <code>rsvg_g_warning</code>, about … |
|
2344 <p>I found it cleanest, although not the shortest code, to structu… |
|
2345 things like this:</p> |
|
2346 <ul> |
|
2347 <li> |
|
2348 <p>C code: bunch of stub functions where <code>rsvg_blah<… |
|
2349 corresponding <code>rsvg_rust_blah</code>.</p> |
|
2350 </li> |
|
2351 <li> |
|
2352 <p>Toplevel Rust code: bunch of <code>#[no_mangle] unsafe ex… |
|
2353 convert from C argument types to Rust types, and call safe Rust |
|
2354 functions — for librsvg, these happened to be methods for a struct. |
|
2355 Before returning, the toplevel functions convert Rust return values |
|
2356 to C return values, and do things like converting the <code>Err(… |
|
2357 <code>Result&lt;&gt;</code> into a <code>GEr… |
|
2358 C API required.</p> |
|
2359 </li> |
|
2360 </ul> |
|
2361 <p>In the very first versions of the code where the public API was |
|
2362 implemented in Rust, the <code>extern "C"</code> functions a… |
|
2363 their implementation. However, after some refactoring, it turned out |
|
2364 to be cleaner to leave those functions just with the task of |
|
2365 converting C to Rust types and vice-versa, and put the actual |
|
2366 implementation in very Rust-y code. This made it easier to keep the |
|
2367 <code>unsafe</code> conversion code (unsafe because it deals… |
|
2368 coming from C) only in the toplevel functions.</p> |
|
2369 <h2>Growing out a Rust API</h2> |
|
2370 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/commit/cd848… |
|
2371 started. That commit just created a <a href="https://doc.rust-lang.o… |
|
2372 with two crates; the <code>rsvg_internals</code> crate that … |
|
2373 <code>librsvg_crate</code> with the public Rust API.</p&g… |
|
2374 <p>The commits over the subsequent couple of months are of intense |
|
2375 refactoring:</p> |
|
2376 <ul> |
|
2377 <li> |
|
2378 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/commit/d01f7… |
|
2379 functions to a separate <code>c_api.rs</code> source file.… |
|
2380 <code>handle.rs</code> with only the safe Rust implementat… |
|
2381 API, and <code>c_api.rs</code> with the unsafe entry point… |
|
2382 convert argument types, return values, and errors.</p> |
|
2383 </li> |
|
2384 <li> |
|
2385 <p>The API primitives get expanded to allow for a public Rust API … |
|
2386 is "hard to misuse" unlike the C API, which needs to be called |
|
2387 in a certain order.</p> |
|
2388 </li> |
|
2389 </ul> |
|
2390 <h2>Needing to call a C macro</h2> |
|
2391 <p>However, there was a little problem. The Rust code cannot call |
|
2392 <a href="https://developer.gnome.org/glib/stable/glib-Message-Logging… |
|
2393 stderr or uses structured logging. Librsvg used that to signal |
|
2394 conditions where something went (recoverably) wrong, but there was no |
|
2395 way to return a proper error code to the caller — it's mainly used as |
|
2396 a debugging aid.</p> |
|
2397 <p>This is what the <code>rsvg_internals</code> used t… |
|
2398 <p>First, the C code exports a function that just calls the macro:… |
|
2399 <div class="highlight"><pre><span></span><cod… |
|
2400 <span class="cm"> * since glib-rs doesn&#39;t bind the g_log f… |
|
2401 <span class="cm"> */</span> |
|
2402 <span class="kt">void</span> |
|
2403 <span class="nf">rsvg_g_warning_from_c</span><span class=… |
|
2404 <span class="p">{</span> |
|
2405 <span class="n">g_warning</span> <span class="p">(… |
|
2406 <span class="p">}</span> |
|
2407 </code></pre></div> |
|
2408 |
|
2409 <p>Second, the Rust code binds that function to be callable from R… |
|
2410 <div class="highlight"><pre><span></span><cod… |
|
2411 <span class="w"> </span><span class="k">extern</… |
|
2412 <span class="w"> </span><span class="k">fn</… |
|
2413 <span class="w"> </span><span class="p">}</span&… |
|
2414 |
|
2415 <span class="w"> </span><span class="k">unsafe</… |
|
2416 <span class="w"> </span><span class="n">rsvg_g_… |
|
2417 <span class="w"> </span><span class="p">}</span&… |
|
2418 <span class="p">}</span><span class="w"></span> |
|
2419 </code></pre></div> |
|
2420 |
|
2421 <p>However! Since the standalone <code>librsvg_crate</co… |
|
2422 code from the public <code>librsvg.so</code>, the helper <… |
|
2423 is not available!</p> |
|
2424 <h3>A configuration feature for the internals library</h3> |
|
2425 <p>And yet! Those warnings are only meaningful for the C API, whi… |
|
2426 not able to return error codes from all situations. However, the Rust |
|
2427 API <em>is</em> able to do that, and so doesn't need the war… |
|
2428 stderr. My first solution was to add a build-time option for whether |
|
2429 the <code>rsvg_internals</code> library is being build for t… |
|
2430 the Rust one.</p> |
|
2431 <p>In case we are building for the C library, the code calls |
|
2432 <code>rsvg_g_warning_from_c</code> as usual.</p> |
|
2433 <p>But in case we are building for the Rust library, that code is a |
|
2434 no-op.</p> |
|
2435 <p>This is the <a href="https://gitlab.gnome.org/GNOME/librsvg/… |
|
2436 <div class="highlight"><pre><span></span><cod… |
|
2437 <span class="c1"># Enables calling g_warning() when built as part … |
|
2438 <span class="n">c-library</span> <span class="o">=<… |
|
2439 </code></pre></div> |
|
2440 |
|
2441 <p>And <a href="https://gitlab.gnome.org/GNOME/librsvg/blob/4df… |
|
2442 <div class="highlight"><pre><span></span><cod… |
|
2443 <span class="k">pub</span><span class="w"> </span&g… |
|
2444 <span class="w"> </span><span class="k">unsafe</… |
|
2445 <span class="w"> </span><span class="k">extern&… |
|
2446 <span class="w"> </span><span class="k">fn&… |
|
2447 <span class="w"> </span><span class="p">}</s… |
|
2448 |
|
2449 <span class="w"> </span><span class="n">rsvg_g_… |
|
2450 <span class="w"> </span><span class="p">}</span&… |
|
2451 <span class="p">}</span><span class="w"></span> |
|
2452 |
|
2453 <span class="cp">#[cfg(not(feature = </span><span class="… |
|
2454 <span class="k">pub</span><span class="w"> </span&g… |
|
2455 <span class="w"> </span><span class="c1">// The onl… |
|
2456 <span class="w"> </span><span class="c1">// are cal… |
|
2457 <span class="w"> </span><span class="c1">// meaning… |
|
2458 <span class="w"> </span><span class="c1">// g_warni… |
|
2459 <span class="p">}</span><span class="w"></span> |
|
2460 </code></pre></div> |
|
2461 |
|
2462 <p>The first function is the one that is compiled when the <cod… |
|
2463 feature is enabled; this happens when building <code>rsvg_internal… |
|
2464 link into <code>librsvg.so</code>.</p> |
|
2465 <p>The second function does nothing; it is what is compiled when |
|
2466 <code>rsvg_internals</code> is being used just from the <… |
|
2467 with the Rust API.</p> |
|
2468 <p>While this worked well, it meant that <strong>the interna… |
|
2469 built twice</strong> on each compilation run of the whole librsvg … |
|
2470 once for <code>librsvg.so</code>, and once for <code>l… |
|
2471 <h2>Making programming errors a <code>g_critical</code>… |
|
2472 <p>While <code>g_warning()</code> means "something wen… |
|
2473 continue", <code>g_critical()</code> means "there is a progr… |
|
2474 historical reasons Glib does not abort when <code>g_critical()<… |
|
2475 except by setting <a href="https://developer.gnome.org/glib/stable/gl… |
|
2476 running a development version of Glib.</p> |
|
2477 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/commit/9d26e… |
|
2478 C API was called out of order, by using a similar |
|
2479 <code>rsvg_g_critical_from_c()</code> wrapper for a C macro.… |
|
2480 <h2>Separating the C-callable code into yet another crate</h2&g… |
|
2481 <p>To recapitulate, at that point we had this:</p> |
|
2482 <div class="highlight"><pre><span></span><cod… |
|
2483 <span class="o">|</span> <span class="nv">Cargo</s… |
|
2484 <span class="o">|</span> |
|
2485 <span class="o">+-</span> <span class="nv">rsvg_intern… |
|
2486 <span class="o">|</span> <span class="o">|</span&g… |
|
2487 <span class="o">|</span> <span class="o">+-</span&… |
|
2488 <span class="o">|</span> <span class="nv">c_api&… |
|
2489 <span class="o">|</span> <span class="nv">handle… |
|
2490 <span class="o">|</span> <span class="o">*</s… |
|
2491 <span class="o">|</span> |
|
2492 <span class="o">+-</span> <span class="nv">librsvg<… |
|
2493 <span class="o">|</span> <span class="o">*</span… |
|
2494 <span class="o">|</span> <span class="nv">rsvg</… |
|
2495 <span class="o">|</span> |
|
2496 <span class="o">+-</span> <span class="nv">librsvg_cra… |
|
2497 <span class="o">|</span> <span class="nv">Cargo<… |
|
2498 <span class="o">+-</span> <span class="nv">src</… |
|
2499 <span class="o">|</span> <span class="nv">lib<… |
|
2500 <span class="o">+-</span> <span class="nv">tests<… |
|
2501 <span class="o">*</span>.<span class="nv">rs&l… |
|
2502 </code></pre></div> |
|
2503 |
|
2504 <p>At this point <code>c_api.rs</code> with all the &l… |
|
2505 place. That code is only relevant to <code>librsvg.so</code>… |
|
2506 —, not to the Rust API in <code>librsvg_crate</code>.</… |
|
2507 <p>I started moving the C API glue to a separate <a href="https… |
|
2508 along with the C stubs:</p> |
|
2509 <div class="highlight"><pre><span></span><cod… |
|
2510 | *.c - stub functions that call into Rust |
|
2511 | rsvg-base.c - contains rsvg_g_warning_from_c() among others |
|
2512 | Cargo.toml |
|
2513 | c_api.rs - what we had before |
|
2514 </code></pre></div> |
|
2515 |
|
2516 <p>This made the dependencies look like the following:</p> |
|
2517 <div class="highlight"><pre><span></span><cod… |
|
2518 ^ ^ |
|
2519 | \ |
|
2520 | \ |
|
2521 librsvg_crate librsvg_c_api |
|
2522 (Rust API) ^ |
|
2523 | |
|
2524 librsvg.so |
|
2525 (C API) |
|
2526 </code></pre></div> |
|
2527 |
|
2528 <p>And also, this made it possible to <a href="https://gitlab.g… |
|
2529 for <code>rsvg_internals</code>, since the code that calls |
|
2530 <code>rsvg_g_warning_from_c</code> now lives in <code>… |
|
2531 <p>With that, <code>rsvg_internals</code> is compiled … |
|
2532 <p>This also helped clean up some code in the internals library. |
|
2533 Deprecated functions that render SVGs directly to <code>GdkPixbuf&… |
|
2534 in <code>librsvg_c_api</code> and don't clutter the <code… |
|
2535 All the GObject boilerplate is there as well now; <code>rsvg_inter… |
|
2536 mostly safe code except for the glue to libxml2.</p> |
|
2537 <h2>Summary</h2> |
|
2538 <p>It was useful to move all the code that dealt with incoming C t… |
|
2539 our outgoing C return values and errors, into the same place, and |
|
2540 separate it from the "pure Rust" code.</p> |
|
2541 <p>This took gradual refactoring and was not done in a single step… |
|
2542 it left the resulting Rust code rather nice and clean.</p> |
|
2543 <p>When we added a new public Rust API, we had to shuffle some code |
|
2544 around that could only be linked in the context of a C library.</p> |
|
2545 <p>Compile-time configuration features are useful (like <code&g… |
|
2546 world), but they do cause double compilation if you need a C-internals |
|
2547 and a Rust-internals library from the same code.</p> |
|
2548 <p>Having proper error reporting throughout the Rust code is a lot… |
|
2549 work, but pretty much invaluable. The glue code to C can then convert |
|
2550 and expose those errors as needed.</p> |
|
2551 <p>If you need both C and Rust APIs into the same code base, you m… |
|
2552 up naturally using a facade pattern for each. It helps to gradually |
|
2553 refactor the internals to be as "pure idiomatic Rust" as possible, |
|
2554 while letting API idiosyncrasies bubble up to each individual facade.<… |
|
2555 <a href="https://blog.ometer.com/2006/10/14/text-layout-that-works-pr… |
|
2556 a CSS-aware canvas from around 2006. It uses libcroco to parse CSS, |
|
2557 and implements selector matching by hand in C.</p> |
|
2558 <p>This code is getting rather dated, and libcroco is unmaintained… |
|
2559 <p>I've been reading the code for |
|
2560 <a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66fc5c07/src… |
|
2561 and |
|
2562 <a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66fc5c07/src… |
|
2563 and it …</p></summary><content type="html"><p>Gnome-shell … |
|
2564 <a href="https://blog.ometer.com/2006/10/14/text-layout-that-works-pr… |
|
2565 a CSS-aware canvas from around 2006. It uses libcroco to parse CSS, |
|
2566 and implements selector matching by hand in C.</p> |
|
2567 <p>This code is getting rather dated, and libcroco is unmaintained… |
|
2568 <p>I've been reading the code for |
|
2569 <a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66fc5c07/src… |
|
2570 and |
|
2571 <a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66fc5c07/src… |
|
2572 and it looks very feasible to port it gradually to Rust, by using the |
|
2573 same crates that librsvg uses, and eventually removing libcroco |
|
2574 altogether: <strong>gnome-shell is the last module that uses libc… |
|
2575 distro packages</strong>.</p> |
|
2576 <h2>Strategy</h2> |
|
2577 <p><code>StTheme</code> and <code>StThemeNode<… |
|
2578 keep them in memory. The values of individual properties are just |
|
2579 tokenized and kept around as a linked list of <code>CRTerm</cod… |
|
2580 represents a single token.</p> |
|
2581 <p>Later, the drawing code uses functions like |
|
2582 <code>st_theme_node_lookup_color(node, "property_name")</code&g… |
|
2583 <code>st_theme_node_lookup_length()</code> to query the vari… |
|
2584 it needs. It is <em>then</em> that the type of each propert… |
|
2585 determined: prior to that step, property values are just tokenized, |
|
2586 not parsed into usable values.</p> |
|
2587 <p>I am going to start by porting the individual parsers to Rust, … |
|
2588 to what Paolo and I did for librsvg. It turns out that there's some |
|
2589 code we can share.</p> |
|
2590 <p>So far I have the <a href="https://gitlab.gnome.org/federico… |
|
2591 colors</a> |
|
2592 implemented in Rust. This <a href="https://gitlab.gnome.org/federico… |
|
2593 code</a> |
|
2594 from the C parsers, and replaces it with a little Rust code, since the |
|
2595 cssparser crate can already parse CSS colors with alpha with no extra |
|
2596 work — libcroco didn't support alpha.</p> |
|
2597 <p>As a bonus, this supports <code>hsl()</code> colors… |
|
2598 out of the box!</p> |
|
2599 <p>After all the parsers are done, the next step would be to conve… |
|
2600 representation of complete stylesheets into pure Rust code.</p> |
|
2601 <h2>What can we expect?</h2> |
|
2602 <p><strong>A well-maintained CSS stack.</strong> Fire… |
|
2603 crates in question, so librsvg and gnome-shell should get maintenance |
|
2604 of a robust CSS stack "for free", for the foreseeable future.</p> |
|
2605 <p><strong>Speed.</strong> Caveat: I have no profile … |
|
2606 know how much time it spends doing CSS parsing and cascading, but it |
|
2607 looks like the Rust version has a good chance of being more efficient.&l… |
|
2608 <p>The <a href="https://docs.rs/selectors/">selectors crate&… |
|
2609 interesting |
|
2610 <a href="https://hacks.mozilla.org/2017/08/inside-a-super-fast-css-en… |
|
2611 from Mozilla Servo, and it is also now used in Firefox. It supports |
|
2612 doing selector matching using Bloom filters, and can also avoid |
|
2613 re-cascading child nodes if a change to a parent would not cause its |
|
2614 children to change.</p> |
|
2615 <p>All the parsing is done with zero-copy parsers thanks to Rust's… |
|
2616 slices; without so many <code>malloc()</code> calls in the p… |
|
2617 the parsing stage should really fly.</p> |
|
2618 <p><strong>More CSS features.</strong> The selectors c… |
|
2619 basically all kinds of selectors as defined by recent CSS specs; one |
|
2620 just has to provide the correct hooks into the calling code's |
|
2621 representation of the DOM tree. The kind of matching that <code>S… |
|
2622 can do is somewhat limited; the rustification should make it match |
|
2623 much more closely to what people expect from CSS engines in web |
|
2624 browsers.</p> |
|
2625 <p><strong>A well-defined model of property inheritance.<… |
|
2626 model for CSS property inheritance is a bit ad-hoc and inconsistent. |
|
2627 I haven't quite tested it, but from looking at the code, it seems that |
|
2628 not all properties get inherited in the same way. I hope to move it |
|
2629 to something closer to what librsvg already does, which should make it |
|
2630 match people's expectations from the web.</p> |
|
2631 <h2>In the meantime</h2> |
|
2632 <p>I have a merge request ready to simply move the libcroco source… |
|
2633 directly inside gnome-shell's source tree. This should let distros |
|
2634 remove their libcroco package as soon as possible. That MR does not |
|
2635 require Rust yet.</p> |
|
2636 <p>My playground is here:</p> |
|
2637 <ul> |
|
2638 <li><a href="https://gitlab.gnome.org/federico/gnome-shell/comm… |
|
2639 styles</a></li> |
|
2640 <li><a href="https://gitlab.gnome.org/federico/stylish">Styl… |
|
2641 library that will implement gnome-shell's styling code.</li> |
|
2642 </ul> |
|
2643 <p>This does not compile yet! I'll plug things together tomorrow.… |
|
2644 <p>(Oh, yes, the project to redo Firefox's CSS stack in Rust used … |
|
2645 called Stylo. I'm calling this Stylish, as in Styles for the Shell.)<… |
|
2646 or <code>6px</code>. Sometimes the unit is a <strong>… |
|
2647 says that lengths with percentage units should be resolved with |
|
2648 respect to a certain rectangle. For example, consider this circle |
|
2649 element:</p> |
|
2650 <div class="highlight"><pre><span></span><cod… |
|
2651 or <code>6px</code>. Sometimes the unit is a <strong>… |
|
2652 says that lengths with percentage units should be resolved with |
|
2653 respect to a certain rectangle. For example, consider this circle |
|
2654 element:</p> |
|
2655 <div class="highlight"><pre><span></span><cod… |
|
2656 </code></pre></div> |
|
2657 |
|
2658 <p>This means, draw a solid black circle whose center is at 50% of… |
|
2659 width and 75% of the height of the current viewport. The circle |
|
2660 should have a 4-pixel radius.</p> |
|
2661 <p>The process of converting that kind of units into absolute pixe… |
|
2662 the final drawing is called <strong>normalization</strong>. … |
|
2663 units sometimes need to be normalized with respect to the current |
|
2664 viewport (a local coordinate system), or with respect to the size of |
|
2665 another object (e.g. when a clipping path is used to cut the current |
|
2666 shape in half).</p> |
|
2667 <p>One detail about normalization is that it can be with respect t… |
|
2668 horizontal dimension of the current viewport, the vertical dimension, |
|
2669 or both. Keep this in mind: at normalization time, we need to be able |
|
2670 to distinguish between those three modes.</p> |
|
2671 <h2>The original C version</h2> |
|
2672 <p>I have <a href="https://people.gnome.org/~federico/news-2016… |
|
2673 following is a small summary.</p> |
|
2674 <p>The original C code had <a href="https://gitlab.gnome.org/GN… |
|
2675 <div class="highlight"><pre><span></span><cod… |
|
2676 <span class="kt">double</span> <span class="n">len… |
|
2677 <span class="kt">char</span> <span class="n">facto… |
|
2678 <span class="p">}</span> <span class="n">RsvgLength<… |
|
2679 </code></pre></div> |
|
2680 |
|
2681 <p>The <a href="https://gitlab.gnome.org/GNOME/librsvg/blob/ef7… |
|
2682 character depending on the length's unit: <code>'p'</code> … |
|
2683 for inches, etc., and <code>'\0'</code> for the default unit… |
|
2684 <p>Along with that, the <a href="https://gitlab.gnome.org/GNOME… |
|
2685 the direction (horizontal, vertical, both) to which the length in |
|
2686 question refers. It did this by taking another character as an |
|
2687 argument to the normalization function:</p> |
|
2688 <div class="highlight"><pre><span></span><cod… |
|
2689 <span class="nf">_rsvg_css_normalize_length</span> <span … |
|
2690 <span class="p">{</span> |
|
2691 <span class="k">if</span> <span class="p">(</sp… |
|
2692 <span class="k">return</span> <span class="n">… |
|
2693 <span class="k">else</span> <span class="k">if<… |
|
2694 <span class="k">if</span> <span class="p">(<… |
|
2695 <span class="k">return</span> <span class="n"… |
|
2696 <span class="k">if</span> <span class="p">(<… |
|
2697 <span class="k">return</span> <span class="n"… |
|
2698 <span class="k">if</span> <span class="p">(<… |
|
2699 <span class="k">return</span> <span class="n"… |
|
2700 <span class… |
|
2701 <span class="p">}</span> <span class="k">else</… |
|
2702 <span class="p">}</span> |
|
2703 </code></pre></div> |
|
2704 |
|
2705 <p><a href="https://people.gnome.org/~federico/news-2016-11.htm… |
|
2706 how the directions are identified at normalization time. The function |
|
2707 above expects one of <code>'h'/'v'/'o'</code> for horizontal… |
|
2708 two places in the code passed the wrong character.</p> |
|
2709 <h2>Making the C version cleaner</h2> |
|
2710 <p>Before converting that code to Rust, I <a href="https://gitl… |
|
2711 characters</a> and made the code use proper enums to |
|
2712 identify a length's units.</p> |
|
2713 <div class="highlight"><pre><span></span><cod… |
|
2714 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2715 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2716 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2717 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2718 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2719 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2720 <span class="o">+</span> <span class="n">LENGTH_UNI… |
|
2721 <span class="o">+</span><span class="p">}</span>… |
|
2722 <span class="o">+</span> |
|
2723 <span class="k">typedef</span> <span class="k">struct… |
|
2724 <span class="kt">double</span> <span class="n">le… |
|
2725 <span class="o">-</span> <span class="kt">char</… |
|
2726 <span class="o">+</span> <span class="n">LengthUnit… |
|
2727 <span class="p">}</span> <span class="n">RsvgLength&l… |
|
2728 </code></pre></div> |
|
2729 |
|
2730 <p>Then, <a href="https://gitlab.gnome.org/GNOME/librsvg/commit… |
|
2731 direction in which to normalize as an enum instead of a char.</p> |
|
2732 <div class="highlight"><pre><span></span><cod… |
|
2733 <span class="o">+</span> <span class="n">LENGTH_DIR… |
|
2734 <span class="o">+</span> <span class="n">LENGTH_DIR… |
|
2735 <span class="o">+</span> <span class="n">LENGTH_DIR… |
|
2736 <span class="o">+</span><span class="p">}</span>… |
|
2737 |
|
2738 <span class="kt">double</span> |
|
2739 <span class="o">-</span><span class="n">_rsvg_css_norm… |
|
2740 <span class="o">+</span><span class="n">_rsvg_css_norm… |
|
2741 </code></pre></div> |
|
2742 |
|
2743 <h2>Making the C version easier to get right</h2> |
|
2744 <p>While doing the last change above, I found a place in the code … |
|
2745 used the wrong direction by mistake, probably due to a cut&amp;paste |
|
2746 error. Part of the problem here is that the code was specifying the |
|
2747 direction at normalization time.</p> |
|
2748 <p>I decided to change it so that <a href="https://gitlab.gnome… |
|
2749 direction since initialization</a>, so that subsequent |
|
2750 code wouldn't have to worry about that. Hopefully, initializing a |
|
2751 <code>width</code> field should make it obvious that it need… |
|
2752 <code>LENGTH_DIR_HORIZONTAL</code>.</p> |
|
2753 <div class="highlight"><pre><span></span><cod… |
|
2754 <span class="kt">double</span> <span class="n">le… |
|
2755 <span class="n">LengthUnit</span> <span class="n">… |
|
2756 <span class="o">+</span> <span class="n">LengthDir&… |
|
2757 <span class="p">}</span> <span class="n">RsvgLength&l… |
|
2758 </code></pre></div> |
|
2759 |
|
2760 <p>That is, so that instead of</p> |
|
2761 <div class="highlight"><pre><span></span><cod… |
|
2762 <span class="n">foo</span><span class="p">.</span… |
|
2763 |
|
2764 <span class="p">...</span> |
|
2765 |
|
2766 <span class="cm">/* at rendering time */</span> |
|
2767 <span class="kt">double</span> <span class="n">final… |
|
2768 </code></pre></div> |
|
2769 |
|
2770 <p>we would instead do this:</p> |
|
2771 <div class="highlight"><pre><span></span><cod… |
|
2772 <span class="n">foo</span><span class="p">.</span… |
|
2773 |
|
2774 <span class="p">...</span> |
|
2775 |
|
2776 <span class="cm">/* at rendering time */</span> |
|
2777 <span class="kt">double</span> <span class="n">final… |
|
2778 </code></pre></div> |
|
2779 |
|
2780 <p>This made the drawing code, which deals with a lot of coordinat… |
|
2781 the same time, a lot less noisy.</p> |
|
2782 <h2>Initial port to Rust</h2> |
|
2783 <p>To recap, this was the state of the structs after the initial |
|
2784 refactoring in C:</p> |
|
2785 <div class="highlight"><pre><span></span><cod… |
|
2786 <span class="n">LENGTH_UNIT_DEFAULT</span><span class… |
|
2787 <span class="n">LENGTH_UNIT_PERCENT</span><span class… |
|
2788 <span class="n">LENGTH_UNIT_FONT_EM</span><span class… |
|
2789 <span class="n">LENGTH_UNIT_FONT_EX</span><span class… |
|
2790 <span class="n">LENGTH_UNIT_INCH</span><span class="p… |
|
2791 <span class="n">LENGTH_UNIT_RELATIVE_LARGER</span><sp… |
|
2792 <span class="n">LENGTH_UNIT_RELATIVE_SMALLER</span> |
|
2793 <span class="p">}</span> <span class="n">LengthUnit<… |
|
2794 |
|
2795 <span class="k">typedef</span> <span class="k">enum<… |
|
2796 <span class="n">LENGTH_DIR_HORIZONTAL</span><span cla… |
|
2797 <span class="n">LENGTH_DIR_VERTICAL</span><span class… |
|
2798 <span class="n">LENGTH_DIR_BOTH</span> |
|
2799 <span class="p">}</span> <span class="n">LengthDir<… |
|
2800 |
|
2801 <span class="k">typedef</span> <span class="k">struct&… |
|
2802 <span class="kt">double</span> <span class="n">len… |
|
2803 <span class="n">LengthUnit</span> <span class="n">… |
|
2804 <span class="n">LengthDir</span> <span class="n">d… |
|
2805 <span class="p">}</span> <span class="n">RsvgLength<… |
|
2806 </code></pre></div> |
|
2807 |
|
2808 <p>This <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/… |
|
2809 <div class="highlight"><pre><span></span><cod… |
|
2810 <span class="w"> </span><span class="nb">Default<… |
|
2811 <span class="w"> </span><span class="n">Percent<… |
|
2812 <span class="w"> </span><span class="n">FontEm</… |
|
2813 <span class="w"> </span><span class="n">FontEx</… |
|
2814 <span class="w"> </span><span class="n">Inch</sp… |
|
2815 <span class="w"> </span><span class="n">RelativeLar… |
|
2816 <span class="w"> </span><span class="n">RelativeSma… |
|
2817 <span class="p">}</span><span class="w"></span> |
|
2818 |
|
2819 <span class="k">pub</span><span class="w"> </span&g… |
|
2820 <span class="w"> </span><span class="n">Horizontal&… |
|
2821 <span class="w"> </span><span class="n">Vertical<… |
|
2822 <span class="w"> </span><span class="n">Both</sp… |
|
2823 <span class="p">}</span><span class="w"></span> |
|
2824 |
|
2825 <span class="k">pub</span><span class="w"> </span&g… |
|
2826 <span class="w"> </span><span class="n">length</… |
|
2827 <span class="w"> </span><span class="n">unit</sp… |
|
2828 <span class="w"> </span><span class="n">dir</spa… |
|
2829 <span class="p">}</span><span class="w"></span> |
|
2830 </code></pre></div> |
|
2831 |
|
2832 <p>It <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/82… |
|
2833 direction and produced an <code>RsvgLength</code>:</p> |
|
2834 <div class="highlight"><pre><span></span><cod… |
|
2835 <span class="w"> </span><span class="k">pub</spa… |
|
2836 <span class="p">}</span><span class="w"></span> |
|
2837 </code></pre></div> |
|
2838 |
|
2839 <p>(This was <a href="https://gitlab.gnome.org/GNOME/librsvg/co… |
|
2840 C code did very little error checking!)</p> |
|
2841 <h2>The initial Parse trait</h2> |
|
2842 <p>It was at that point that it seemed convenient to <a href="h… |
|
2843 trait</a>, which all CSS value types would implement to |
|
2844 parse themselves from string.</p> |
|
2845 <p>However, parsing an <code>RsvgLength</code> also ne… |
|
2846 the <code>LengthDir</code>. My initial version of the <c… |
|
2847 associated called <code>Data</code>, through which one could… |
|
2848 of data during parsing/initialization:</p> |
|
2849 <div class="highlight"><pre><span></span><cod… |
|
2850 <span class="w"> </span><span class="k">type</sp… |
|
2851 <span class="w"> </span><span class="k">type</sp… |
|
2852 |
|
2853 <span class="w"> </span><span class="k">fn</span… |
|
2854 <span class="p">}</span><span class="w"></span> |
|
2855 </code></pre></div> |
|
2856 |
|
2857 <p>This was explicitly to be able to <a href="https://gitlab.gn… |
|
2858 <code>RsvgLength</code>:</p> |
|
2859 <div class="highlight"><pre><span></span><cod… |
|
2860 <span class="w"> </span><span class="k">type</sp… |
|
2861 <span class="w"> </span><span class="k">type</sp… |
|
2862 |
|
2863 <span class="w"> </span><span class="k">fn</span… |
|
2864 <span class="p">}</span><span class="w"></span> |
|
2865 </code></pre></div> |
|
2866 |
|
2867 <p>This was okay for lengths, but <em>very noisy</em> … |
|
2868 didn't require an extra bit of data. In the rest of the code, the |
|
2869 helper type was <code>Data = ()</code> and there was a pair … |
|
2870 in every place that <code>parse()</code> was called.</p&g… |
|
2871 <h2>Removing the helper Data type</h2> |
|
2872 <h3>Introducing one type per direction</h3> |
|
2873 <p>Over a year later, that <code>()</code> bit of data… |
|
2874 nuts. I started refactoring the <code>Length</code> module … |
|
2875 <p>First, I <a href="https://gitlab.gnome.org/GNOME/librsvg/com… |
|
2876 their direction at the same time:</p> |
|
2877 <div class="highlight"><pre><span></span><cod… |
|
2878 <span class="k">pub</span><span class="w"> </span&g… |
|
2879 <span class="k">pub</span><span class="w"> </span&g… |
|
2880 </code></pre></div> |
|
2881 |
|
2882 <p>This was <a href="https://gitlab.gnome.org/GNOME/librsvg/com… |
|
2883 needed to know the relevant <code>LengthDir</code>.</p> |
|
2884 <p>Now, for example, the declaration for the <code>&lt;c… |
|
2885 like this:</p> |
|
2886 <div class="highlight"><pre><span></span><cod… |
|
2887 <span class="w"> </span><span class="n">cx</span… |
|
2888 <span class="w"> </span><span class="n">cy</span… |
|
2889 <span class="w"> </span><span class="n">r</span&… |
|
2890 <span class="p">}</span><span class="w"></span> |
|
2891 </code></pre></div> |
|
2892 |
|
2893 <p>(Ignore the <code>Cell</code> everywhere; we got ri… |
|
2894 <h3>Removing the <code>dir</code> field</h3> |
|
2895 <p>Since now the information about the length's direction is embod… |
|
2896 the <code>LengthHorizontal/LengthVertical/LengthBoth</code> … |
|
2897 possible to <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/78… |
|
2898 <code>Length</code> struct.</p> |
|
2899 <div class="highlight"><pre><span></span><cod… |
|
2900 <span class="w"> </span><span class="n">length<… |
|
2901 <span class="w"> </span><span class="n">unit</s… |
|
2902 <span class="o">-</span><span class="w"> </span&… |
|
2903 <span class="w"> </span><span class="p">}</span>… |
|
2904 |
|
2905 <span class="o">+</span><span class="k">pub</span&g… |
|
2906 <span class="o">+</span><span class="k">pub</span&g… |
|
2907 <span class="o">+</span><span class="k">pub</span&g… |
|
2908 <span class="o">+</span><span class="w"></span> |
|
2909 <span class="o">+</span><span class="n">define_length_… |
|
2910 <span class="o">+</span><span class="n">define_length_… |
|
2911 <span class="o">+</span><span class="n">define_length_… |
|
2912 </code></pre></div> |
|
2913 |
|
2914 <p>Note the use of a <code>define_length_type!</code> … |
|
2915 those three newtypes.</p> |
|
2916 <h3>Removing the <code>Data</code> associated type<… |
|
2917 <p>And finally, this made it possible to <a href="https://gitla… |
|
2918 type</a> from the <code>Parse</code> trait.</p> |
|
2919 <div class="highlight"><pre><span></span><cod… |
|
2920 <span class="o">-</span><span class="w"> </span&… |
|
2921 <span class="w"> </span><span class="k">type</s… |
|
2922 |
|
2923 <span class="o">-</span><span class="w"> </span&… |
|
2924 <span class="o">+</span><span class="w"> </span&… |
|
2925 <span class="w"> </span><span class="p">}</span>… |
|
2926 </code></pre></div> |
|
2927 |
|
2928 <p>The resulting mega-commit removed a bunch of stray parentheses … |
|
2929 from all calls to <code>parse()</code>, and the code ended u… |
|
2930 read.</p> |
|
2931 <h2>Removing the newtypes</h2> |
|
2932 <p>This was fine for a while. Recently, however, I figured out th… |
|
2933 would be possible to embody the information for a length's direction |
|
2934 in a different way.</p> |
|
2935 <p>But to get there, I first needed a temporary refactor.</p> |
|
2936 <h3>Replacing the macro with a trait with a default implementation… |
|
2937 <p>Deep in the guts of <code>length.rs</code>, the key… |
|
2938 different based on <code>LengthDir</code> is its <code>… |
|
2939 <div class="highlight"><pre><span></span><cod… |
|
2940 <span class="w"> </span><span class="n">Horizontal&… |
|
2941 <span class="w"> </span><span class="n">Vertical<… |
|
2942 <span class="w"> </span><span class="n">Both</sp… |
|
2943 <span class="p">}</span><span class="w"></span> |
|
2944 |
|
2945 <span class="k">impl</span><span class="w"> </span&… |
|
2946 <span class="w"> </span><span class="k">fn</span… |
|
2947 <span class="w"> </span><span class="k">match&l… |
|
2948 <span class="w"> </span><span class="n">Len… |
|
2949 <span class="w"> </span><span class="n">Len… |
|
2950 <span class="w"> </span><span class="n">Len… |
|
2951 <span class="w"> </span><span class="p">}</s… |
|
2952 <span class="w"> </span><span class="p">}</span&… |
|
2953 <span class="p">}</span><span class="w"></span> |
|
2954 </code></pre></div> |
|
2955 |
|
2956 <p>That method gets passed, for example, the <code>width/hei… |
|
2957 current viewport for the <code>x/y</code> arguments. The me… |
|
2958 to use the width, height, or a combination of both.</p> |
|
2959 <p>And of course, the interesting part of the <code>define_l… |
|
2960 was to generate code for calling <code>LengthDir::Horizontal::scal… |
|
2961 appropriate depending on the <code>LengthDir</code> in quest… |
|
2962 <p>First I made a trait called <code>Orientation</code>… |
|
2963 method, and three zero-sized types that implement that trait. Note |
|
2964 how each of these three implementations corresponds to one of the |
|
2965 <code>match</code> arms above:</p> |
|
2966 <div class="highlight"><pre><span></span><cod… |
|
2967 <span class="w"> </span><span class="k">fn</span… |
|
2968 <span class="p">}</span><span class="w"></span> |
|
2969 |
|
2970 <span class="k">pub</span><span class="w"> </span&g… |
|
2971 <span class="k">pub</span><span class="w"> </span&g… |
|
2972 <span class="k">pub</span><span class="w"> </span&g… |
|
2973 |
|
2974 <span class="k">impl</span><span class="w"> </span&… |
|
2975 <span class="w"> </span><span class="k">fn</span… |
|
2976 <span class="w"> </span><span class="n">x</s… |
|
2977 <span class="w"> </span><span class="p">}</span&… |
|
2978 <span class="p">}</span><span class="w"></span> |
|
2979 |
|
2980 <span class="k">impl</span><span class="w"> </span&… |
|
2981 <span class="w"> </span><span class="k">fn</span… |
|
2982 <span class="w"> </span><span class="n">y</s… |
|
2983 <span class="w"> </span><span class="p">}</span&… |
|
2984 <span class="p">}</span><span class="w"></span> |
|
2985 |
|
2986 <span class="k">impl</span><span class="w"> </span&… |
|
2987 <span class="w"> </span><span class="k">fn</span… |
|
2988 <span class="w"> </span><span class="n">viewpor… |
|
2989 <span class="w"> </span><span class="p">}</span&… |
|
2990 <span class="p">}</span><span class="w"></span> |
|
2991 </code></pre></div> |
|
2992 |
|
2993 <p>Now most of the contents of the <code>define_length_type!… |
|
2994 the <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/21a40f4ef8… |
|
2995 <code>LengthTrait</code></a>. Crucially, this trait h… |
|
2996 <code>Orientation</code> associated type, <strong>whic… |
|
2997 Orientation trait</strong>:</p> |
|
2998 <div class="highlight"><pre><span></span><cod… |
|
2999 <span class="w"> </span><span class="k">type</sp… |
|
3000 |
|
3001 <span class="w"> </span><span class="o">..</span… |
|
3002 |
|
3003 <span class="w"> </span><span class="k">fn</span… |
|
3004 <span class="w"> </span><span class="k">match&l… |
|
3005 <span class="w"> </span><span class="n">Len… |
|
3006 |
|
3007 <span class="w"> </span><span class="n">Len… |
|
3008 <span class="w"> </span><span class="bp"&g… |
|
3009 <span class="w"> </span><span class="o">… |
|
3010 <span class="w"> </span><span class="p">}&l… |
|
3011 |
|
3012 <span class="w"> </span><span class="o">..&… |
|
3013 <span class="w"> </span><span class="p">}</span&… |
|
3014 <span class="p">}</span><span class="w"></span> |
|
3015 </code></pre></div> |
|
3016 |
|
3017 <p>Note that the incantation is |
|
3018 <code>&lt;Self::Orientation&gt;::scaling_factor(...)</c… |
|
3019 associated type.</p> |
|
3020 <p>Now the <code>define_length_type!</code> macro is s… |
|
3021 interesting part being just this:</p> |
|
3022 <div class="highlight"><pre><span></span><cod… |
|
3023 <span class="w"> </span><span class="p">{</span&… |
|
3024 <span class="w"> </span><span class="k">pub<… |
|
3025 |
|
3026 <span class="w"> </span><span class="k">impl<… |
|
3027 <span class="w"> </span><span class="k">typ… |
|
3028 <span class="w"> </span><span class="p">}</s… |
|
3029 <span class="w"> </span><span class="p">}</span&… |
|
3030 <span class="p">}</span><span class="w"></span> |
|
3031 |
|
3032 <span class="n">define_length_type</span><span class="o"&… |
|
3033 <span class="n">define_length_type</span><span class="o"&… |
|
3034 <span class="n">define_length_type</span><span class="o"&… |
|
3035 </code></pre></div> |
|
3036 |
|
3037 <p>We moved from having three newtypes of length-with-LengthDir to… |
|
3038 newtypes with dir-as-associated-type.</p> |
|
3039 <h3>Removing the newtypes and the macro</h3> |
|
3040 <p>After that temporary refactoring, we had the <code>Orient… |
|
3041 the three zero-sized types <code>Horizontal</code>, <code… |
|
3042 <p>I figured out that one can use <a href="https://doc.rust-lan… |
|
3043 around the type that <code>Length</code> needs to normalize … |
|
3044 using an associated type in an extra <code>LengthTrait</code>… |
|
3045 <div class="highlight"><pre><span></span><cod… |
|
3046 <span class="w"> </span><span class="k">pub</spa… |
|
3047 <span class="w"> </span><span class="k">pub</spa… |
|
3048 <span class="w"> </span><span class="n">orientation… |
|
3049 <span class="p">}</span><span class="w"></span> |
|
3050 |
|
3051 <span class="k">impl</span><span class="o">&lt;<… |
|
3052 <span class="w"> </span><span class="k">pub</spa… |
|
3053 <span class="w"> </span><span class="k">match&l… |
|
3054 <span class="w"> </span><span class="n">Len… |
|
3055 |
|
3056 <span class="w"> </span><span class="n">Len… |
|
3057 <span class="w"> </span><span class="bp"&g… |
|
3058 <span class="w"> </span><span class="o… |
|
3059 <span class="w"> </span><span class="p">}&l… |
|
3060 |
|
3061 <span class="w"> </span><span class="o">..&… |
|
3062 <span class="w"> </span><span class="p">}</s… |
|
3063 <span class="w"> </span><span class="p">}</span&… |
|
3064 <span class="p">}</span><span class="w"></span> |
|
3065 </code></pre></div> |
|
3066 |
|
3067 <p>Now the incantation is <code>&lt;O as Orientation&… |
|
3068 the method on the generic type; it is no longer an associated type in |
|
3069 a trait.</p> |
|
3070 <p>With that, users of lengths look like this; here, our <code&… |
|
3071 element from before:</p> |
|
3072 <div class="highlight"><pre><span></span><cod… |
|
3073 <span class="w"> </span><span class="n">cx</span… |
|
3074 <span class="w"> </span><span class="n">cy</span… |
|
3075 <span class="w"> </span><span class="n">r</span&… |
|
3076 <span class="p">}</span><span class="w"></span> |
|
3077 </code></pre></div> |
|
3078 |
|
3079 <p>I'm very happy with the readability of all the code now. I use… |
|
3080 think of <code>PhantomData</code> as a way to deal with <… |
|
3081 C</a>, but it turns out that it is also useful to keep a generic |
|
3082 type around should one need it.</p> |
|
3083 <p>The final <code>Length</code> struct is this:</p… |
|
3084 <div class="highlight"><pre><span></span><cod… |
|
3085 <span class="w"> </span><span class="k">pub</spa… |
|
3086 <span class="w"> </span><span class="k">pub</spa… |
|
3087 <span class="w"> </span><span class="n">orientation… |
|
3088 <span class="p">}</span><span class="w"></span> |
|
3089 </code></pre></div> |
|
3090 |
|
3091 <p>And it only takes up as much space as its <code>length<… |
|
3092 <code>PhantomData</code> is zero-sized after all.</p> |
|
3093 <p>(Later, we renamed <code>Orientation</code> to <… |
|
3094 structure remained the same.)</p> |
|
3095 <h2>Summary</h2> |
|
3096 <p>Over a couple of years, librsvg's type that represents CSS leng… |
|
3097 went from a C representation along the lines of "all data in the world |
|
3098 is an int", to a Rust representation that uses some interesting type |
|
3099 trickery:</p> |
|
3100 <ul> |
|
3101 <li> |
|
3102 <p>C struct with <code>char</code> for units.</p> |
|
3103 </li> |
|
3104 <li> |
|
3105 <p>C struct with a <code>LengthUnits</code> enum.</… |
|
3106 </li> |
|
3107 <li> |
|
3108 <p>C struct without an embodied direction; each place that needs to |
|
3109 normalize needs to get the orientation right.</p> |
|
3110 </li> |
|
3111 <li> |
|
3112 <p>C struct with a built-in direction as an extra field, done at |
|
3113 initialization time.</p> |
|
3114 </li> |
|
3115 <li> |
|
3116 <p>Same struct but in Rust.</p> |
|
3117 </li> |
|
3118 <li> |
|
3119 <p>An ugly but workable <code>Parse</code> trait so th… |
|
3120 at parse/initialization time.</p> |
|
3121 </li> |
|
3122 <li> |
|
3123 <p>Three newtypes <code>LengthHorizontal</code>, <c… |
|
3124 <code>LengthBoth</code> with a common core. A cleaned-up … |
|
3125 macro to generate those newtypes.</p> |
|
3126 </li> |
|
3127 <li> |
|
3128 <p>Replace the <code>LengthDir</code> enum with an <… |
|
3129 trait, and three zero-sized types <code>Horizontal/Vertical/Both… |
|
3130 implement the trait.</p> |
|
3131 </li> |
|
3132 <li> |
|
3133 <p>Replace most of the macro with a helper trait <code>Lengt… |
|
3134 an <code>Orientation</code> associated type.</p> |
|
3135 </li> |
|
3136 <li> |
|
3137 <p>Replace the helper trait with a single <code>Length&l… |
|
3138 type, which puts the orientation as a generic parameter. The macro |
|
3139 disappears and there is a single implementation for everything.</p&… |
|
3140 </li> |
|
3141 </ul> |
|
3142 <p>Refactoring never ends!</p></content><category term="misc… |
|
3143 refactoring</a>, |
|
3144 librsvg now does all CSS parsing and matching in Rust, <strong>wit… |
|
3145 libcroco</strong>. In addition, the CSS engine comes from Mozilla … |
|
3146 it should be able to handle much more complex CSS than librsvg ever |
|
3147 could before.</p> |
|
3148 <p>This is the story of …</p></summary><content type="html… |
|
3149 refactoring</a>, |
|
3150 librsvg now does all CSS parsing and matching in Rust, <strong>wit… |
|
3151 libcroco</strong>. In addition, the CSS engine comes from Mozilla … |
|
3152 it should be able to handle much more complex CSS than librsvg ever |
|
3153 could before.</p> |
|
3154 <p>This is the story of CSS support in librsvg.</p> |
|
3155 <h2>Introduction</h2> |
|
3156 <p>The <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/c… |
|
3157 parsing</a> |
|
3158 in librsvg dates from 2002. It was as minimal as possible, written to |
|
3159 support a small subset of what was then |
|
3160 <a href="https://www.w3.org/TR/1998/REC-CSS2-19980512/">CSS2</a… |
|
3161 <p>Librsvg handled CSS stylesheets more "piecing them apart" than |
|
3162 "parsing them". You know, when <code>g_strsplit()</code> is… |
|
3163 The basic parsing algorithm was to turn a stylesheet like this:</p> |
|
3164 <div class="highlight"><pre><span></span><cod… |
|
3165 |
|
3166 <span class="p">.</span><span class="nc">classname<… |
|
3167 <span class="n">fill</span><span class="p">:</s… |
|
3168 <span class="n">stroke-width</span><span class="p">… |
|
3169 <span class="p">}</span> |
|
3170 </code></pre></div> |
|
3171 |
|
3172 <p>Into a hash table whose keys are strings like <code>rect&… |
|
3173 and whose values are everything inside curly braces.</p> |
|
3174 <p>The selector matching phase was equally simple. The code only … |
|
3175 a few possible match types as follows. If it wanted to match a |
|
3176 certain kind of CSS selector, it would say, "what would this selector |
|
3177 look like in CSS syntax", it would make up a string with that syntax, |
|
3178 and compare it to the key strings it had stored in the hash table from |
|
3179 above.</p> |
|
3180 <p>So, to match an <strong>element name selector</strong&… |
|
3181 element-&gt;name)</code>, obtain something like <code>re… |
|
3182 table had such a key.</p> |
|
3183 <p>To match a <strong>class selector</strong>, it woul… |
|
3184 element-&gt;class)</code>, obtain something like <code>.… |
|
3185 in the hash table.</p> |
|
3186 <p>This scheme supported only a few combinations. It handled <… |
|
3187 <code>.class</code>, <code>tag.class</code>, and… |
|
3188 This was enough to support very simple stylesheets.</p> |
|
3189 <p>The value corresponding to each key in the hash table was the s… |
|
3190 between curly braces in the stylesheet, so the second rule from the |
|
3191 example above would contain <code>fill: green; stroke-width: 4;<… |
|
3192 librsvg decided that an SVG element matched that CSS rule, it would |
|
3193 re-parse the string with the CSS properties and apply them to the |
|
3194 element's style.</p> |
|
3195 <p>I'm amazed that so little code was enough to deal with a good n… |
|
3196 of SVG files with stylesheets. I suspect that this was due to a few |
|
3197 things:</p> |
|
3198 <ul> |
|
3199 <li> |
|
3200 <p>While people were using complex CSS in HTML all the time, it was |
|
3201 less common for SVG...</p> |
|
3202 </li> |
|
3203 <li> |
|
3204 <p>... because CSS2 was somewhat new, and the SVG spec was still b… |
|
3205 written...</p> |
|
3206 </li> |
|
3207 <li> |
|
3208 <p>... and SVGs created with illustration programs don't really use |
|
3209 stylesheets; they include the full style information inside each |
|
3210 element instead of symbolically referencing it from a stylesheet.</… |
|
3211 </li> |
|
3212 </ul> |
|
3213 <p>From the kinds of bugs that librsvg has gotten around "CSS supp… |
|
3214 too limited", it feels like SVGs which use CSS features are either |
|
3215 hand-written, or machine-generated from custom programs like data |
|
3216 plotting software. Illustration programs tend to list all style |
|
3217 properties explicitly in each SVG element, and don't use CSS.</p> |
|
3218 <h2>Libcroco appears</h2> |
|
3219 <p>The first commit to <a href="https://gitlab.gnome.org/GNOME/… |
|
3220 libcroco</a> |
|
3221 was to do CSS parsing, from March 2003.</p> |
|
3222 <p>At the same time, libcroco was introducing code to do CSS match… |
|
3223 However, this code never got used in librsvg; it still kept its simple |
|
3224 string-based matcher. Maybe libcroco's API was not ready?</p> |
|
3225 <p>Libcroco fell out of maintainership around the first half of 20… |
|
3226 volunteers have kept fixing it since then.</p> |
|
3227 <h2>Problems with librsvg's string matcher for CSS</h2> |
|
3228 <p>The C implementation of CSS matching in librsvg remained basica… |
|
3229 untouched until 2018, when Paolo Borelli and I started porting the |
|
3230 surrounding code to Rust.</p> |
|
3231 <p>I had a lot of trouble figuring out the concepts from the code.… |
|
3232 didn't know all the <a href="https://gnome.pages.gitlab.gnome.org/lib… |
|
3233 implementations</a>, |
|
3234 and librsvg didn't use it, either.</p> |
|
3235 <p>I think that librsvg's code suffered from what the refactoring |
|
3236 literature calls <a href="https://refactoring.guru/smells/primitive-o… |
|
3237 obsession</strong></a>. |
|
3238 Instead of having a parsed representation of CSS selectors, librsvg |
|
3239 just stored a stringified version of them. So, a selector like |
|
3240 <code>rect#classname</code> really was stored with a string … |
|
3241 an actual decomposition into structs.</p> |
|
3242 <p>Moreover, things were misnamed. This is the field that stored |
|
3243 stylesheet data inside an RsvgHandle:</p> |
|
3244 <div class="highlight"><pre><span></span><cod… |
|
3245 </code></pre></div> |
|
3246 |
|
3247 <p>From just looking at the field declaration, this doesn't tell me |
|
3248 anything about what kind of data is stored there. One has to grep the |
|
3249 source code for where that field is used:</p> |
|
3250 <div class="highlight"><pre><span></span><cod… |
|
3251 <span class="nf">rsvg_css_define_style</span> <span class… |
|
3252 <span class="k">const</span> <span… |
|
3253 <span class="k">const</span> <span… |
|
3254 <span class="k">const</span> <span… |
|
3255 <span class="n">gboolean</span> <s… |
|
3256 <span class="p">{</span> |
|
3257 <span class="n">GHashTable</span> <span class="o">… |
|
3258 |
|
3259 <span class="n">styles</span> <span class="o">=<… |
|
3260 </code></pre></div> |
|
3261 |
|
3262 <p>Okay, it looks up a <code>selector</code> by name i… |
|
3263 gives back... another hash table <code>styles</code>? What'… |
|
3264 <div class="highlight"><pre><span></span><cod… |
|
3265 <span class="n">g_strdup</span>… |
|
3266 <span class="n">style_value_data_new&… |
|
3267 </code></pre></div> |
|
3268 |
|
3269 <p>Another string key called <code>style_name</code>, … |
|
3270 <code>StyleValueData</code>; what's in it?</p> |
|
3271 <div class="highlight"><pre><span></span><cod… |
|
3272 <span class="n">gchar</span> <span class="o">*<… |
|
3273 <span class="n">gboolean</span> <span class="n">im… |
|
3274 <span class="p">}</span> <span class="n">StyleValueDat… |
|
3275 </code></pre></div> |
|
3276 |
|
3277 <p>The <code>value</code> is another string. Strings … |
|
3278 <p>At the time, I didn't really figure out what each level of nest… |
|
3279 tables was supposed to mean. I didn't understand why we handled style |
|
3280 properties in a completely different part of the code, and yet this |
|
3281 part had a <code>css_props</code> field that didn't seem to … |
|
3282 <p>It took a while to realize that <code>css_props</code&… |
|
3283 storing a mapping of selector names to properties; it was storing a |
|
3284 mapping of selector names to <strong>declaration lists</strong&… |
|
3285 property/value pairs.</p> |
|
3286 <p>So, when I started <a href="https://gitlab.gnome.org/GNOME/l… |
|
3287 Rust</a>, |
|
3288 I started to create real types with for each concept.</p> |
|
3289 <div class="highlight"><pre><span></span><cod… |
|
3290 <span class="k">type</span> <span class="nc">Declarati… |
|
3291 |
|
3292 <span class="k">pub</span><span class="w"> </span&g… |
|
3293 <span class="w"> </span><span class="n">selectors_t… |
|
3294 <span class="p">}</span><span class="w"></span> |
|
3295 </code></pre></div> |
|
3296 |
|
3297 <p>Even though the keys of those HashMaps are still strings, becau… |
|
3298 librsvg didn't have a better way to represent their corresponding |
|
3299 concepts, at least those declarations let one see what the hell is |
|
3300 being stored without grepping the rest of the code. This is a part of |
|
3301 the code that I didn't really touch very much, so it was nice to have |
|
3302 that reminder.</p> |
|
3303 <p>The <a href="https://gitlab.gnome.org/GNOME/librsvg/commit/1… |
|
3304 Rust</a> |
|
3305 kept the same algorithm as the C code, the one that created strings |
|
3306 with <code>element.class</code> and compared them to the sto… |
|
3307 Ugly, but it still worked in the same limited fashion.</p> |
|
3308 <h2>Rustifying the CSS parsers</h2> |
|
3309 <p>It turns out that CSS parsing is divided in two parts. One can… |
|
3310 <code>style</code> attribute inside an element, for example&… |
|
3311 <div class="highlight"><pre><span></span><cod… |
|
3312 <span class="na">style=</span><span class="s">&a… |
|
3313 </code></pre></div> |
|
3314 |
|
3315 <p>This is a plain declaration list which is not associated to any |
|
3316 selectors, and which is applied directly to just the element in which it |
|
3317 appears.</p> |
|
3318 <p>Then, there is the <code>&lt;style&gt;</code&g… |
|
3319 <div class="highlight"><pre><span></span><cod… |
|
3320 rect { |
|
3321 fill: green; |
|
3322 stroke: magenta; |
|
3323 stroke-width: 4; |
|
3324 } |
|
3325 <span class="nt">&lt;/style&gt;</span> |
|
3326 </code></pre></div> |
|
3327 |
|
3328 <p>This means that all <code>&lt;rect&gt;</code&g… |
|
3329 <p>I started to look for existing Rust crates to parse and handle … |
|
3330 data. The <a href="https://docs.rs/cssparser/">cssparser</a>… |
|
3331 <a href="https://docs.rs/selectors/">selectors</a> crates co… |
|
3332 I thought they should do a pretty good job of things.</p> |
|
3333 <p>And they do! Except that they are not a drop-in replacement for |
|
3334 anything. They are what gets used in Mozilla's Servo browser engine, |
|
3335 so they are optimized to hell, and the code can be pretty intimidating.&… |
|
3336 <p>Out of the box, cssparser provides a CSS tokenizer, but it does |
|
3337 not know how to handle any properties/values in particular. One must |
|
3338 use the tokenizer to implement a parser for each kind of CSS property |
|
3339 one wants to support — Servo has mountains of code for all of HTML's |
|
3340 style properties, and librsvg had to provide a smaller mountain of code |
|
3341 for SVG style properties.</p> |
|
3342 <p>Thus started the big task of porting librsvg's string-based par… |
|
3343 for CSS properties into ones based on cssparser tokens. Cssparser |
|
3344 provides a <code>Parser</code> struct, which extracts tokens… |
|
3345 stream. Out of this, librsvg defines a <code>Parse</code> t… |
|
3346 things:</p> |
|
3347 <div class="highlight"><pre><span></span><cod… |
|
3348 |
|
3349 <span class="k">pub</span><span class="w"> </span&g… |
|
3350 <span class="w"> </span><span class="k">type</sp… |
|
3351 |
|
3352 <span class="w"> </span><span class="k">fn</span… |
|
3353 <span class="p">}</span><span class="w"></span> |
|
3354 </code></pre></div> |
|
3355 |
|
3356 <p>What's with those two default lifetimes in <code>Parser&a… |
|
3357 Cssparser tries very hard to be a zero-copy tokenizer. One of the |
|
3358 lifetimes refers to the input string which is wrapped in a |
|
3359 <code>Tokenizer</code>, which is wrapped in a <code>Pa… |
|
3360 is for the <code>ParserInput</code> itself.</p> |
|
3361 <p>In the actual implementation of that trait, the <code>Err… |
|
3362 the lifetime that refers to the input string. For example, there is a |
|
3363 <code>BasicParseErrorKind::UnexpectedToken(Token&lt;'i&gt;… |
|
3364 when there is an unexpected token. And to avoid copying the substring |
|
3365 into the error, one returns a slice reference into the original |
|
3366 string, thus the lifetime.</p> |
|
3367 <p>I was more of a Rust newbie back then, and it was very hard to … |
|
3368 sense of how cssparser was meant to be used.</p> |
|
3369 <p>The process was more or less this:</p> |
|
3370 <ul> |
|
3371 <li> |
|
3372 <p>Port the C parsers to Rust; implement types for each CSS proper… |
|
3373 </li> |
|
3374 <li> |
|
3375 <p>Port the <code>&amp;str</code>-based parsers in… |
|
3376 </li> |
|
3377 <li> |
|
3378 <p>Fix the error handling scheme to match what cssparser's high-le… |
|
3379 traits expect.</p> |
|
3380 </li> |
|
3381 </ul> |
|
3382 <p>This last point was... hard. Again, I wasn't comfortable enoug… |
|
3383 Rust lifetimes and nested generics; in the end it was all right.</p&g… |
|
3384 <h2>Moving declaration lists to Rust</h2> |
|
3385 <p>With the individual parsers for CSS properties done, and with t… |
|
3386 already using a different type for each property, the next thing was |
|
3387 to implement cssparser's traits to parse declaration lists.</p> |
|
3388 <p>Again, a declaration list looks like this:</p> |
|
3389 <div class="highlight"><pre><span></span><cod… |
|
3390 <span class="nt">stroke-width</span><span class="o">:&… |
|
3391 </code></pre></div> |
|
3392 |
|
3393 <p>It's essentially a key/value list.</p> |
|
3394 <p>The trait that cssparser wants us to implement is this:</p&g… |
|
3395 <div class="highlight"><pre><span></span><cod… |
|
3396 <span class="w"> </span><span class="k">type</sp… |
|
3397 <span class="w"> </span><span class="k">type</sp… |
|
3398 |
|
3399 <span class="w"> </span><span class="k">fn</span… |
|
3400 <span class="w"> </span><span class="o">&am… |
|
3401 <span class="w"> </span><span class="n">name<… |
|
3402 <span class="w"> </span><span class="n">input&l… |
|
3403 <span class="w"> </span><span class="p">)</span&… |
|
3404 <span class="p">}</span><span class="w"></span> |
|
3405 </code></pre></div> |
|
3406 |
|
3407 <p>That is, define a type for a <code>Declaration</code&g… |
|
3408 <code>parse_value()</code> method that takes a <code>n… |
|
3409 a <code>Declaration</code> or an error.</p> |
|
3410 <p>What this <em>really</em> means is that the type yo… |
|
3411 <code>Declaration</code> needs to be able to represent all t… |
|
3412 that you care about. Thus, a struct plus a big enum like this:</p> |
|
3413 <div class="highlight"><pre><span></span><cod… |
|
3414 <span class="w"> </span><span class="k">pub</spa… |
|
3415 <span class="w"> </span><span class="k">pub</spa… |
|
3416 <span class="w"> </span><span class="k">pub</spa… |
|
3417 <span class="p">}</span><span class="w"></span> |
|
3418 |
|
3419 <span class="k">pub</span><span class="w"> </span&g… |
|
3420 <span class="w"> </span><span class="n">BaselineShi… |
|
3421 <span class="w"> </span><span class="n">ClipPath<… |
|
3422 <span class="w"> </span><span class="n">ClipRule<… |
|
3423 <span class="w"> </span><span class="n">Color</s… |
|
3424 <span class="w"> </span><span class="n">ColorInterp… |
|
3425 <span class="w"> </span><span class="n">Direction&l… |
|
3426 <span class="w"> </span><span class="o">..</span… |
|
3427 <span class="p">}</span><span class="w"></span> |
|
3428 </code></pre></div> |
|
3429 |
|
3430 <p>This gives us declaration lists (the stuff inside curly braces … |
|
3431 CSS stylesheet), but it doesn't give us qualified rules, which are |
|
3432 composed of selector names plus a declaration list.</p> |
|
3433 <h2>Refactoring towards real CSS concepts</h2> |
|
3434 <p>Paolo Borelli has been steadily refactoring librsvg and fixing … |
|
3435 like the primitive obsession I mentioned above. We now have real |
|
3436 concepts like a Document, Stylesheet, QualifiedRule, Rule, AtRule.</p… |
|
3437 <p>This refactoring took a long time, because it involved redoing … |
|
3438 loading code and its interaction with the CSS parser a few times.</p&… |
|
3439 <h2>Implementing traits from the selectors crate</h2> |
|
3440 <p>The <a href="https://docs.rs/selectors">selectors</a&g… |
|
3441 contains Servo's code for parsing CSS selectors and doing matching. |
|
3442 However, it is <em>extremely</em> generic. Using it involve… |
|
3443 good number of concepts.</p> |
|
3444 <p>For example, this <code>SelectorImpl</code> trait h… |
|
3445 collection of types that refer to your implementation of an element |
|
3446 tree. How do you represent an attribute/value? How do you represent |
|
3447 an identifier? How do you represent a namespace and a local name?</p… |
|
3448 <div class="highlight"><pre><span></span><cod… |
|
3449 <span class="w"> </span><span class="k">type</sp… |
|
3450 <span class="w"> </span><span class="k">type</sp… |
|
3451 <span class="w"> </span><span class="k">type</sp… |
|
3452 <span class="w"> </span><span class="k">type</sp… |
|
3453 <span class="w"> </span><span class="k">type</sp… |
|
3454 <span class="w"> </span><span class="k">type</sp… |
|
3455 <span class="w"> </span><span class="k">type</sp… |
|
3456 <span class="w"> </span><span class="k">type</sp… |
|
3457 <span class="w"> </span><span class="k">type</sp… |
|
3458 <span class="w"> </span><span class="k">type</sp… |
|
3459 <span class="w"> </span><span class="k">type</sp… |
|
3460 <span class="w"> </span><span class="k">type</sp… |
|
3461 <span class="p">}</span><span class="w"></span> |
|
3462 </code></pre></div> |
|
3463 |
|
3464 <p>A lot of those can be <code>String</code>, but Serv… |
|
3465 I ended up using the <a href="https://docs.rs/markup5ever"><cod… |
|
3466 interning framework for markup and XML concepts like a <code>Local… |
|
3467 <code>Namespace</code>, etc. This reduces memory consumptio… |
|
3468 storing string copies of element names everywhere, one just stores |
|
3469 tokens for interned strings.</p> |
|
3470 <p>(In the meantime I had to implement support for XML namespaces,… |
|
3471 the selectors code really wants, but which librsvg never supported.)<… |
|
3472 <p>Then, the selectors crate wants you to say how your code implem… |
|
3473 element tree. It has a monster trait <code>Element</code>:&… |
|
3474 <div class="highlight"><pre><span></span><cod… |
|
3475 <span class="w"> </span><span class="k">type</sp… |
|
3476 |
|
3477 <span class="w"> </span><span class="k">fn</span… |
|
3478 |
|
3479 <span class="w"> </span><span class="k">fn</span… |
|
3480 |
|
3481 <span class="w"> </span><span class="k">fn</span… |
|
3482 |
|
3483 <span class="w"> </span><span class="o">..</span… |
|
3484 |
|
3485 <span class="w"> </span><span class="k">fn</span… |
|
3486 <span class="w"> </span><span class="k">fn</span… |
|
3487 |
|
3488 <span class="w"> </span><span class="k">fn</span… |
|
3489 <span class="w"> </span><span class="o">&am… |
|
3490 <span class="w"> </span><span class="n">local_n… |
|
3491 <span class="w"> </span><span class="p">)</span&… |
|
3492 |
|
3493 <span class="w"> </span><span class="k">fn</span… |
|
3494 <span class="w"> </span><span class="o">&am… |
|
3495 <span class="w"> </span><span class="n">id</… |
|
3496 <span class="w"> </span><span class="n">case_se… |
|
3497 <span class="w"> </span><span class="p">)</span&… |
|
3498 |
|
3499 <span class="w"> </span><span class="o">..</span… |
|
3500 <span class="p">}</span><span class="w"></span> |
|
3501 </code></pre></div> |
|
3502 |
|
3503 <p>That is, when you provide an implementation of <code>Elem… |
|
3504 <code>SelectorImpl</code>, the selectors crate will know how… |
|
3505 element tree and ask it questions like, "does this element have the id |
|
3506 <code>#foo</code>?"; "does this element have the name <co… |
|
3507 sense in the end, but it is quite intimidating when you are not 100% |
|
3508 comfortable with webs of traits and associated types and generics with |
|
3509 a bunch of trait bounds!</p> |
|
3510 <p>I tried implementing that trait twice in the last year, and fai… |
|
3511 It turns out that its API <a href="https://github.com/servo/servo/iss… |
|
3512 fix</a> that landed last |
|
3513 June, but I didn't notice until a couple of weeks ago.</p> |
|
3514 <h2>So?</h2> |
|
3515 <p>Two days ago, Paolo and I committed the <a href="https://git… |
|
3516 completely replace |
|
3517 libcroco</a>.</p> |
|
3518 <p>And, after implementing CSS <a href="https://gitlab.gnome.or… |
|
3519 have real CSS concepts and a good pipeline for the CSS cascade), a |
|
3520 bunch of very old bugs started falling down |
|
3521 (<a href="https://gitlab.gnome.org/GNOME/librsvg/issues/336">1<… |
|
3522 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues/466">2</… |
|
3523 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues/428">3</… |
|
3524 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues/167">4</… |
|
3525 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues/79">5</a… |
|
3526 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues/441">6</… |
|
3527 <p>Now it is going to be easy to implement things like <a href=… |
|
3528 application specify a user |
|
3529 stylesheet</a>. In |
|
3530 particular, this should let GTK remove the <a href="https://gitlab.gn… |
|
3531 hack</a> |
|
3532 it has to recolor SVG icons while using librsvg indirectly.</p> |
|
3533 <h2>Conclusion</h2> |
|
3534 <p>This will appear in librsvg 2.47.1 — that version will no lon… |
|
3535 require libcroco.</p> |
|
3536 <p>As far as I know, the only module that still depends on libcroc… |
|
3537 GNOME or otherwise) is <strong>gnome-shell</strong>. It use… |
|
3538 and get the basic structure of selectors so it can implement matching |
|
3539 by hand.</p> |
|
3540 <p>Gnome-shell has some code which looks awfully similar to what l… |
|
3541 had when it was written in C:</p> |
|
3542 <ul> |
|
3543 <li> |
|
3544 <p><a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66f… |
|
3545 has the high-level CSS stylesheet parser and the selector matching cod… |
|
3546 </li> |
|
3547 <li> |
|
3548 <p><a href="https://gitlab.gnome.org/GNOME/gnome-shell/blob/66f… |
|
3549 has the low-level CSS property parsers.</p> |
|
3550 </li> |
|
3551 </ul> |
|
3552 <p>... and it turns out that those files come all the way from |
|
3553 <a href="https://blog.ometer.com/2006/10/14/text-layout-that-works-pr… |
|
3554 the CSS-aware canvas that Mugshot used! Mugshot was a circa-2006 |
|
3555 pre-Facebook aggregator for social media data like blogs, Flickr |
|
3556 pictures, etc. HippoCanvas also got used in Sugar, the GUI for |
|
3557 One Laptop Per Child. Yes, our code is <em>that</em> old… |
|
3558 <p>Libcroco is unmaintained, and has outstanding CVEs. I would be… |
|
3559 happy to assist someone in porting gnome-shell's CSS code to Rust :)<… |
|
3560 see if the gripes from my <a href="https://people.gnome.org/~federico… |
|
3561 things where it is not obvious how to proceed, I've started taking |
|
3562 more detailed notes in a <a href="https://gitlab.gnome.org/federico/g… |
|
3563 <p>Today I was looking at which <a href="https://gitlab.gnome.o… |
|
3564 see if the gripes from my <a href="https://people.gnome.org/~federico… |
|
3565 things where it is not obvious how to proceed, I've started taking |
|
3566 more detailed notes in a <a href="https://gitlab.gnome.org/federico/g… |
|
3567 <p>Today I was looking at which <a href="https://gitlab.gnome.o… |
|
3568 implemented by third parties, that is, which external projects provide |
|
3569 their own image codecs pluggable into gdk-pixbuf.</p> |
|
3570 <p>And there are not that many!</p> |
|
3571 <p>The only four that I found are <strong>libheif, libopenra… |
|
3572 librsvg</strong> (this last one, of course).</p> |
|
3573 <p><em>Update 2019/Sep/12</em> - Added <strong>a… |
|
3574 <p>All of those use the gdk-pixbuf module API in a remarkably simi… |
|
3575 fashion. Did they cut&amp;paste each other's code? Did they do the |
|
3576 simplest thing that didn't crash in gdk-pixbuf's checks for buggy |
|
3577 loaders, which happens to be exactly what they do? Who knows! Either |
|
3578 way, this makes future API changes in the modules a lot easier, since |
|
3579 they all do the same right now.</p> |
|
3580 <p>I'm trying to decide between these:</p> |
|
3581 <ul> |
|
3582 <li> |
|
3583 <p>Keep modules as they are; find a way to sandbox them from gdk-p… |
|
3584 itself. This is hard because the API is "chatty"; modules and |
|
3585 calling code go back and forth peeking at each other's structures.<… |
|
3586 </li> |
|
3587 <li> |
|
3588 <p>Decide that third-party modules are only useful for thumbnailer… |
|
3589 modify them to <em>be</em> thumbnailers instead of generic… |
|
3590 modules. This would mean that those formats would stop working |
|
3591 automatically in gdk-pixbuf based viewers like EOG.</p> |
|
3592 </li> |
|
3593 <li> |
|
3594 <p>Have "blessed" codecs inside gdk-pixbuf which are not modules so |
|
3595 their no longer have API/ABI stability constraints. Keep |
|
3596 third-party modules separate. Sandbox the internal ones with a |
|
3597 non-chatty API.</p> |
|
3598 </li> |
|
3599 <li> |
|
3600 <p>If all third-party modules work indeed as <a href="https://g… |
|
3601 module API can be simplified quite a lot since no third-party |
|
3602 modules implement animations or saving. If so, simplify the module |
|
3603 API and the gdk-pixbuf internals rather drastically.</p> |
|
3604 </li> |
|
3605 </ul> |
|
3606 <p>Do you know any other image formats which provide <a href="h… |
|
3607 modules</a>? <a href="mailto:[email protected]">Mail me, p… |
|
3608 vulnerability</a> in the code that processes <code>.desktop&… |
|
3609 <code>.directory</code> files, through which an attacker cou… |
|
3610 file that causes shell command execution (<a href="https://paper.seeb… |
|
3611 full disclosure, where KDE didn't even get a chance of fixing the …<… |
|
3612 vulnerability</a> in the code that processes <code>.desktop&… |
|
3613 <code>.directory</code> files, through which an attacker cou… |
|
3614 file that causes shell command execution (<a href="https://paper.seeb… |
|
3615 full disclosure, where KDE didn't even get a chance of fixing the bug |
|
3616 before it was published.</p> |
|
3617 <p>There are many protocols for <a href="https://en.wikipedia.o… |
|
3618 coordinated, responsible fashion</a>, but the gist of them is this… |
|
3619 <ol> |
|
3620 <li> |
|
3621 <p>Someone finds a vulnerability in some software through studying |
|
3622 some code, or some other mechanism.</p> |
|
3623 </li> |
|
3624 <li> |
|
3625 <p>They report the vulnerability to the software's author through … |
|
3626 private channel. For free softare in particular, researchers can |
|
3627 use <a href="https://oss-security.openwall.org/wiki/disclosure/res… |
|
3628 researchers</a>, which includes <strong>notifying the |
|
3629 author/maintainer and distros and security groups.</strong> Fr… |
|
3630 software projects can <a href="https://alexgaynor.net/2013/oct/19/… |
|
3631 </li> |
|
3632 <li> |
|
3633 <p>The author and reporter agree on a deadline for releasing a pub… |
|
3634 report of the vulnerability, or in semi-automated systems like |
|
3635 Google Zero, a deadline is automatically established.</p> |
|
3636 </li> |
|
3637 <li> |
|
3638 <p>The author works on fixing the vulnerability.</p> |
|
3639 </li> |
|
3640 <li> |
|
3641 <p>The deadline is reached; the patch has been publically released, |
|
3642 the appropriate people have been notified, systems have been |
|
3643 patched. If there is no patch, the author and reporter can agree |
|
3644 on postponing the date, or the reporter can publish the |
|
3645 vulnerability report, thus creating public pressure for a fix.</p&… |
|
3646 </li> |
|
3647 </ol> |
|
3648 <p>The steps above gloss over many practicalities and issues from … |
|
3649 real world, but the idea is basically this: the author or maintainer |
|
3650 of the software is given a chance to fix a security bug before |
|
3651 information on the vulnerability is released to the hostile world. |
|
3652 The idea is to <strong>keep harm from being done</strong> by… |
|
3653 unpatched vulnerabilities until there is a fix for them (... or until |
|
3654 the deadline expires).</p> |
|
3655 <h2>What happened instead</h2> |
|
3656 <p>Around the beginning of July, the reporter <a href="https://… |
|
3657 bugs in KDE</a>.</p> |
|
3658 <p>On <a href="https://twitter.com/zer0pwn/status/1156312405472… |
|
3659 <p>On August 3, he <a href="https://twitter.com/zer0pwn/status/… |
|
3660 vulnerability.</p> |
|
3661 <p>On August 4, he <a href="https://twitter.com/zer0pwn/status/… |
|
3662 <p>KDE is left with having to patch this in emergency mode. On Au… |
|
3663 KDE releases a <a href="https://kde.org/info/security/advisory-201908… |
|
3664 <ul> |
|
3665 <li> |
|
3666 <p>Description of exactly what causes the vulnerability.</p> |
|
3667 </li> |
|
3668 <li> |
|
3669 <p>Description of how it was solved.</p> |
|
3670 </li> |
|
3671 <li> |
|
3672 <p>Instructions on what to do for users of various versions of KDE |
|
3673 libraries.</p> |
|
3674 </li> |
|
3675 <li> |
|
3676 <p>Links to easy-to-cherry-pick patches for distro vendors.</p&… |
|
3677 </li> |
|
3678 </ul> |
|
3679 <p>Now, distro vendors are, in turn, in emergency mode, as they mu… |
|
3680 apply the patch, run it through QA, release their own advisories, |
|
3681 etc.</p> |
|
3682 <h2>What if this had been done with coordinated disclosure?</h2… |
|
3683 <p>The bug would have been fixed, probably in the same way, <em… |
|
3684 not be in emergency mode</em>. <a href="https://kde.org/info/s… |
|
3685 <blockquote> |
|
3686 <p>Thanks to Dominik Penner for finding and documenting this issue… |
|
3687 have contacted us before making the issue public) and to David Faure for… |
|
3688 </blockquote> |
|
3689 <p>This is an extremely gracious way of thanking the reporter.<… |
|
3690 <h2>I am not an infosec person...</h2> |
|
3691 <p>... but some behaviors in the infosec sphere are deeply uncomfo… |
|
3692 to me. I don't like it when security "research" is hard to tell from |
|
3693 vandalism. "Excuse me, you left your car door unlocked" vs. "Hey |
|
3694 everyone, this car is unlocked, have at it".</p> |
|
3695 <p>I don't know the details of the discourse in the infosec sphere… |
|
3696 full disclosure against irresponsible vendors of proprietary software or |
|
3697 services. However, <strong>KDE is free software</strong>! … |
|
3698 an asshole to them.</p></content><category term="misc"></category>… |
|
3699 <ul> |
|
3700 <li> |
|
3701 <p>From a constructor, calling a method on a partially-constructed |
|
3702 object is dangerous.</p> |
|
3703 </li> |
|
3704 <li> |
|
3705 <p>A constructor needs to set up "not quite initialized" values in… |
|
3706 instance struct until a construct-time property is set.</p> |
|
3707 </li> |
|
3708 <li> |
|
3709 <p>You actually need to override <code>GObjectClass::constru… |
|
3710 <ul> |
|
3711 <li> |
|
3712 <p>From a constructor, calling a method on a partially-constructed |
|
3713 object is dangerous.</p> |
|
3714 </li> |
|
3715 <li> |
|
3716 <p>A constructor needs to set up "not quite initialized" values in… |
|
3717 instance struct until a construct-time property is set.</p> |
|
3718 </li> |
|
3719 <li> |
|
3720 <p>You actually need to override <code>GObjectClass::constru… |
|
3721 <code>::constructor</code>?) to take care of construct-onl… |
|
3722 need to be considered together, not individually.</p> |
|
3723 </li> |
|
3724 <li> |
|
3725 <p>Constructors can't report an error, unless you derive from |
|
3726 <code>GInitable</code>, which is not in gobject, but in gi… |
|
3727 why does <em>that</em> force the constructor to take a <… |
|
3728 </li> |
|
3729 <li> |
|
3730 <p>You need more than one constructor, but that needs to be done w… |
|
3731 helper functions.</p> |
|
3732 </li> |
|
3733 </ul> |
|
3734 <p>This article, <a href="https://matklad.github.io/2019/07/16/… |
|
3735 Constructors</a>, |
|
3736 explains all of these problems very well. It is not centered on |
|
3737 GObject, but rather on constructors in object-oriented languages in |
|
3738 general.</p> |
|
3739 <p>(Spoiler: Rust does not have constructors or partially-initiali… |
|
3740 these problems don't really exist there.)</p> |
|
3741 <p>(Addendum: <em>that</em> makes it somewhat awkward … |
|
3742 C to Rust, but librsvg was able to solve it nicely with |
|
3743 <code>&lt;buzzword&gt;</code><a href="http://clif… |
|
3744 very nice tutorial on <a href="https://nora.codes/tutorial/speedy-des… |
|
3745 Rust</a>. |
|
3746 It covers prototyping a dice roller app with Glade, writing the code with |
|
3747 Rust and the gtk-rs bindings, and integrating the app into the desktop w… |
|
3748 a <code>.desktop</code> file.</p></content><category t… |
|
3749 the rsvg-view-3 program.</strong></p> |
|
3750 <h2>History of rsvg-view</h2> |
|
3751 <p>Rsvg-view <a href="https://gitlab.gnome.org/GNOME/librsvg/bl… |
|
3752 out</a> |
|
3753 as a 71-line C program to aid development of librsvg. It would just |
|
3754 render an SVG file to a pixbuf, stick that pixbuf in a <code>GtkIm… |
|
3755 widget …</p></summary><content type="html"><p>I am prepari… |
|
3756 the rsvg-view-3 program.</strong></p> |
|
3757 <h2>History of rsvg-view</h2> |
|
3758 <p>Rsvg-view <a href="https://gitlab.gnome.org/GNOME/librsvg/bl… |
|
3759 out</a> |
|
3760 as a 71-line C program to aid development of librsvg. It would just |
|
3761 render an SVG file to a pixbuf, stick that pixbuf in a <code>GtkIm… |
|
3762 widget, and show a window with that.</p> |
|
3763 <p>Over time, it slowly acquired most of the command-line options … |
|
3764 <code>rsvg-convert</code> supports. And I suppose, as a way… |
|
3765 Cairo-ification of librsvg, it also got the ability to print SVG files |
|
3766 to a <code>GtkPrintContext</code>. At last count, it was a … |
|
3767 that is not really the best code in the world.</p> |
|
3768 <h2>What makes rsvg-view awkward?</h2> |
|
3769 <p>Rsvg-view requires GTK. But GTK requires librsvg, indirectly, … |
|
3770 gdk-pixbuf! There is not a hard circular dependency because GTK goes, |
|
3771 "gdk-pixbuf, load me this SVG file" without knowing how it will be |
|
3772 loaded. In turn, gdk-pixbuf initializes the SVG loader provided by |
|
3773 librsvg, and that loader reads/renders the SVG file.</p> |
|
3774 <p>Ideally librsvg would only depend on gdk-pixbuf, so it would be… |
|
3775 to provide the SVG loader.</p> |
|
3776 <p>The rsvg-view source code still has a few calls to GTK function… |
|
3777 are now deprecated. The program emits GTK warnings during normal use.&l… |
|
3778 <p>Rsvg-view is... not a very good SVG viewer. It doesn't even st… |
|
3779 with the window scaled properly to the SVG's dimensions! If used for |
|
3780 quick testing during development, it cannot even aid in viewing the |
|
3781 transparent background regions which the SVG does not cover. It just |
|
3782 sticks a lousy custom widget inside a <code>GtkScrolledWindow</… |
|
3783 not have the conventional niceties to view images like zooming with |
|
3784 the scroll wheel.</p> |
|
3785 <p><a href="https://wiki.gnome.org/Apps/EyeOfGnome/">EOG<… |
|
3786 invest effort in making it pleasant to use.</p> |
|
3787 <h2>Removal of rsvg-view</h2> |
|
3788 <p>So, the next version of librsvg will not provide the <code&g… |
|
3789 binary. Please update your packages accordingly. Distros may be able to |
|
3790 move the compilation of librsvg to a more sensible place in the |
|
3791 platform stack, now that it doesn't depend on GTK being available.</p… |
|
3792 <p>What can you use instead? Any other image viewer. <a href=… |
|
3793 there are dozens of other good viewers, too.</p></content><categor… |
|
3794 slight change of plans since <a href="https://people.gnome.org/~feder… |
|
3795 <ul> |
|
3796 <li> |
|
3797 <p>The 1.0.x series is in strict maintenance mode and will not cha… |
|
3798 build systems. <strong>This is targeted towards embedded use<… |
|
3799 projects which already embed the …</p></li></ul></… |
|
3800 slight change of plans since <a href="https://people.gnome.org/~feder… |
|
3801 <ul> |
|
3802 <li> |
|
3803 <p>The 1.0.x series is in strict maintenance mode and will not cha… |
|
3804 build systems. <strong>This is targeted towards embedded use<… |
|
3805 projects which already embed the bzip2-1.0.6 sources and undoubtedly |
|
3806 patch the build system. Right now this series, and the tagged 1.0.7 |
|
3807 release, live in the <a href="https://sourceware.org/git/?p=bzip2.g… |
|
3808 </li> |
|
3809 <li> |
|
3810 <p>The 1.1.x series has Meson and CMake build systems, and a coupl… |
|
3811 extra changes to modernize the C code but which were not fit for the |
|
3812 1.0.7 release. <strong>This is targeted towards operating system |
|
3813 distributions</strong>. This lives in the master branch of the … |
|
3814 repository for bzip2</a>.</p> |
|
3815 </li> |
|
3816 </ul> |
|
3817 <p><strong>Distros and embedded users</strong> should … |
|
3818 immediately. The patches they already have for the bzip2's |
|
3819 traditional build system should still apply. The release includes bug |
|
3820 fixes and security fixes that have accumulated over the years, |
|
3821 including the new <a href="https://gitlab.com/federicomenaquintero/bz… |
|
3822 <p>Once 1.1.0 is released, distributions should be able to remove … |
|
3823 patches to the build system and just start using Meson or CMake. You |
|
3824 may want to monitor the <a href="https://gitlab.com/federicomenaquint… |
|
3825 appreciated fixing the issues there so we can make the first release |
|
3826 with the new build systems!</p></content><category term="misc"></c… |
|
3827 <p>I am preparing a bzip2-1.0.7 release. You can see the <a hr… |
|
3828 notes</a>, which should be of interest:</p> |
|
3829 <ul> |
|
3830 <li> |
|
3831 <p>Many historical patches from various distributions are integrat… |
|
3832 now.</p> |
|
3833 </li> |
|
3834 <li> |
|
3835 <p>We have a new fix for the just-published <a href="https://cv… |
|
3836 <p>I am preparing a bzip2-1.0.7 release. You can see the <a hr… |
|
3837 notes</a>, which should be of interest:</p> |
|
3838 <ul> |
|
3839 <li> |
|
3840 <p>Many historical patches from various distributions are integrat… |
|
3841 now.</p> |
|
3842 </li> |
|
3843 <li> |
|
3844 <p>We have a new fix for the just-published <a href="https://cv… |
|
3845 Albert Astals Cid.</p> |
|
3846 </li> |
|
3847 <li> |
|
3848 <p><strong>Bzip2 has moved to Meson</strong> for its p… |
|
3849 courtesy of Dylan Baker. For special situations, a CMake build |
|
3850 system is also provided, courtesy of Micah Snyder.</p> |
|
3851 </li> |
|
3852 </ul> |
|
3853 <h2>What's with the soname?</h2> |
|
3854 <p>From bzip2-1.0.1 (from the year 2000), until bzip2-1.0.6 (from … |
|
3855 release tarballs came with a special |
|
3856 <a href="https://gitlab.com/federicomenaquintero/bzip2/blob/962d60610… |
|
3857 instead of a static one.</p> |
|
3858 <p>This never used libtool or anything; it specified linker flags … |
|
3859 hand. Various distributions either patched this special makefile, or |
|
3860 replaced it by another one, or outright replaced the complete build |
|
3861 system for a different one.</p> |
|
3862 <p>Some things to note:</p> |
|
3863 <ul> |
|
3864 <li> |
|
3865 <p>This hand-written <a href="https://gitlab.com/federicomenaqu… |
|
3866 line like <code>$(CC) -shared -Wl,-soname -Wl,libbz2.so.1.0 -o |
|
3867 libbz2.so.1.0.6</code>. This means, make the DT_SONAME field in… |
|
3868 ELF file be <code>libbz2.so.1.0</code> (note the two digit… |
|
3869 the filename of the shared library be <code>libbz2.so.1.0.6</… |
|
3870 </li> |
|
3871 <li> |
|
3872 <p>Fedora patched the soname in a patch called "saneso" to just be |
|
3873 <code>libbz2.so.1</code>.</p> |
|
3874 </li> |
|
3875 <li> |
|
3876 <p>Stanislav Brabec, from openSUSE, <a href="https://gitlab.com… |
|
3877 makefiles with autotools</a>, which meant using libtool. It |
|
3878 has this interesting note:</p> |
|
3879 </li> |
|
3880 </ul> |
|
3881 <blockquote> |
|
3882 <p>Incompatible changes:</p> |
|
3883 <p>soname change. Libtool has no support for two parts soname suff… |
|
3884 libbz2.so.1.0). It must be a single number (e. g. libbz2.so.1). That is |
|
3885 why soname must change. But I see not a big problem with it. Several |
|
3886 distributions already use the new number instead of the non-standard |
|
3887 number from Makefile-libbz2_so.</p> |
|
3888 </blockquote> |
|
3889 <p>(In fact, if I do <code>objdump -x /usr/lib64/*.so | grep… |
|
3890 that most libraries have single-digit sonames.)</p> |
|
3891 <p>In my experience, both Fedora and openSUSE are very strict, and |
|
3892 correct, about obscure things like library sonames.</p> |
|
3893 <p>With the switch to Meson, bzip2 no longer uses libtool. It wil… |
|
3894 a single-digit soname — this is not in the <code>meson.build<… |
|
3895 expect it to be there within the next couple of days.</p> |
|
3896 <p>I don't know what distros which decided to preserve the <cod… |
|
3897 will need to do; maybe they will need to patch <code>meson.build&l… |
|
3898 own.</p> |
|
3899 <p>Fortunately, <strong>the API/ABI are still exactly the sa… |
|
3900 preserve the old soname which your distro was using and linking libbz2 |
|
3901 will probably keep working as usual.</p> |
|
3902 <p>(This is a C-only release as usual. The Rust branch is still |
|
3903 experimental.)</p></content><category term="misc"></category><cate… |
|
3904 <p><a href="https://gitlab.com/federicomenaquintero/bzip2/blob/… |
|
3905 numbers. This table is used by <a href="https://gitlab.com/federicom… |
|
3906 bzlib_private.h</a>:</p> |
|
3907 <div class="highlight"><pre><span></span><cod… |
|
3908 |
|
3909 <span class="cp">#define BZ_RAND_DECLS \&… |
|
3910 <span class="cp"> Int32 rNToGo; \&… |
|
3911 <span class="cp"> Int32 rTPos \&… |
|
3912 |
|
3913 <span class="cp">#define BZ_RAND_INIT_MASK \&… |
|
3914 <span class="cp"> s-&gt;rNToGo = 0; … |
|
3915 <span class="cp"> s-&gt;rTPos = 0 … |
|
3916 |
|
3917 <span class="cp">#define BZ_RAND_MASK ((s- …</span></co… |
|
3918 <p><a href="https://gitlab.com/federicomenaquintero/bzip2/blob/… |
|
3919 numbers. This table is used by <a href="https://gitlab.com/federicom… |
|
3920 bzlib_private.h</a>:</p> |
|
3921 <div class="highlight"><pre><span></span><cod… |
|
3922 |
|
3923 <span class="cp">#define BZ_RAND_DECLS \&… |
|
3924 <span class="cp"> Int32 rNToGo; \&… |
|
3925 <span class="cp"> Int32 rTPos \&… |
|
3926 |
|
3927 <span class="cp">#define BZ_RAND_INIT_MASK \&… |
|
3928 <span class="cp"> s-&gt;rNToGo = 0; … |
|
3929 <span class="cp"> s-&gt;rTPos = 0 … |
|
3930 |
|
3931 <span class="cp">#define BZ_RAND_MASK ((s-&gt;rNToGo == 1) ? 1… |
|
3932 |
|
3933 <span class="cp">#define BZ_RAND_UPD_MASK \&… |
|
3934 <span class="cp"> if (s-&gt;rNToGo == 0) { … |
|
3935 <span class="cp"> s-&gt;rNToGo = BZ2_rNums[s-&gt;rTPo… |
|
3936 <span class="cp"> s-&gt;rTPos++; … |
|
3937 <span class="cp"> if (s-&gt;rTPos == 512) s-&gt;rTPos… |
|
3938 <span class="cp"> } \&… |
|
3939 <span class="cp"> s-&gt;rNToGo--;</span> |
|
3940 </code></pre></div> |
|
3941 |
|
3942 <p>Here, <code>BZ_RAND_DECLS</code> is used to declare… |
|
3943 <code>rTPos</code>, into two structs (<a href="https://gi… |
|
3944 <div class="highlight"><pre><span></span><cod… |
|
3945 <span class="p">...</span> |
|
3946 <span class="n">Bool</span> <span class="n">blo… |
|
3947 <span class="n">BZ_RAND_DECLS</span> |
|
3948 <span class="p">...</span> |
|
3949 <span class="p">}</span> <span class="n">DState</sp… |
|
3950 </code></pre></div> |
|
3951 |
|
3952 <p>Then, the code that needs to initialize those fields calls |
|
3953 <code>BZ_RAND_INIT_MASK</code>, which expands into code to s… |
|
3954 zero.</p> |
|
3955 <p>At several points in the code, <code>BZ_RAND_UPD_MASK<… |
|
3956 expands into code that updates the randomization state, or something |
|
3957 like that, and uses <code>BZ_RAND_MASK</code> to get a usefu… |
|
3958 randomization state.</p> |
|
3959 <p>I have no idea yet what the state is about, but let's port it |
|
3960 directly.</p> |
|
3961 <h2>Give things a name</h2> |
|
3962 <p>It's interesting to see that <strong>no code except for t… |
|
3963 the fields <code>rNToGo</code> and <code>rTPos</cod… |
|
3964 <code>BZ_RAND_DECLS</code>. So, let's make up a <strong&… |
|
3965 Since I have no better name for it, I shall call it just |
|
3966 <a href="https://gitlab.com/federicomenaquintero/bzip2/commit/7bd2dc3… |
|
3967 and replaced the macro-which-creates-struct-fields with a |
|
3968 <code>RandState</code>-typed field:</p> |
|
3969 <div class="highlight"><pre><span></span><cod… |
|
3970 - Int32 rNToGo; \ |
|
3971 - Int32 rTPos \ |
|
3972 +typedef struct { |
|
3973 + Int32 rNToGo; |
|
3974 + Int32 rTPos; |
|
3975 +} RandState; |
|
3976 |
|
3977 ... |
|
3978 |
|
3979 - BZ_RAND_DECLS; |
|
3980 + RandState rand; |
|
3981 </code></pre></div> |
|
3982 |
|
3983 <p>Since the fields now live inside a sub-struct, I changed the ot… |
|
3984 macros to use <code>s-&gt;rand.rNToGo</code> instead of … |
|
3985 for the other field.</p> |
|
3986 <h2>Turn macros into functions</h2> |
|
3987 <p>Now, three commits (<a href="https://gitlab.com/federicomena… |
|
3988 macros <code>BZ_RAND_INIT_MASK</code>, <code>BZ_RAND_M… |
|
3989 into functions.</p> |
|
3990 <p>And now that the functions live in the same C source file as the |
|
3991 lookup table they reference, <a href="https://gitlab.com/federicomena… |
|
3992 avoid having it as read/write unshared data in the linked binary.</p&… |
|
3993 <p>Premature optimization concern: doesn't de-inlining those macr… |
|
3994 cause performance problems? At first, we will get the added overhead |
|
3995 from a function call. When the whole code is ported to Rust, the Rust |
|
3996 compiler will probably be able to figure out that those tiny functions |
|
3997 can be inlined (or we can <code>#[inline]</code> them by han… |
|
3998 or if we have more hubris than faith in LLVM).</p> |
|
3999 <h2>Port functions and table to Rust</h2> |
|
4000 <p>The functions are so tiny, and the table so cut-and-pasteable, … |
|
4001 it's easy to <a href="https://gitlab.com/federicomenaquintero/bzip2/c… |
|
4002 <div class="highlight"><pre><span></span><cod… |
|
4003 <span class="k">pub</span><span class="w"> </span&g… |
|
4004 <span class="w"> </span><span class="n">RandState&l… |
|
4005 <span class="w"> </span><span class="n">rNToGo&… |
|
4006 <span class="w"> </span><span class="n">rTPos&l… |
|
4007 <span class="w"> </span><span class="p">}</span&… |
|
4008 <span class="p">}</span><span class="w"></span> |
|
4009 |
|
4010 <span class="cp">#[no_mangle]</span><span class="w">&l… |
|
4011 <span class="k">pub</span><span class="w"> </span&g… |
|
4012 <span class="w"> </span><span class="k">if</span… |
|
4013 <span class="w"> </span><span class="mi">1</… |
|
4014 <span class="w"> </span><span class="p">}</span&… |
|
4015 <span class="w"> </span><span class="mi">0</… |
|
4016 <span class="w"> </span><span class="p">}</span&… |
|
4017 <span class="p">}</span><span class="w"></span> |
|
4018 |
|
4019 <span class="cp">#[no_mangle]</span><span class="w">&l… |
|
4020 <span class="k">pub</span><span class="w"> </span&g… |
|
4021 <span class="w"> </span><span class="k">if</span… |
|
4022 <span class="w"> </span><span class="n">r</s… |
|
4023 <span class="w"> </span><span class="n">r</s… |
|
4024 <span class="w"> </span><span class="k">if</… |
|
4025 <span class="w"> </span><span class="n">r&l… |
|
4026 <span class="w"> </span><span class="p">}</s… |
|
4027 <span class="w"> </span><span class="p">}</span&… |
|
4028 <span class="w"> </span><span class="n">r</span&… |
|
4029 <span class="p">}</span><span class="w"></span> |
|
4030 </code></pre></div> |
|
4031 |
|
4032 <p>Also, we define the <code>RandState</code> type as … |
|
4033 <a href="https://gitlab.com/federicomenaquintero/bzip2/blob/dc01d95a7… |
|
4034 as the C struct. <strong>This is what allows us to have a <cod… |
|
4035 the C struct</strong>, while in reality the C code doesn't access … |
|
4036 directly; it is just used as a struct field.</p> |
|
4037 <div class="highlight"><pre><span></span><cod… |
|
4038 <span class="cp">#[repr(C)]</span><span class="w"><… |
|
4039 <span class="k">pub</span><span class="w"> </span&g… |
|
4040 <span class="w"> </span><span class="n">rNToGo</… |
|
4041 <span class="w"> </span><span class="n">rTPos</s… |
|
4042 <span class="p">}</span><span class="w"></span> |
|
4043 </code></pre></div> |
|
4044 |
|
4045 <p><a href="https://gitlab.com/federicomenaquintero/bzip2/commi… |
|
4046 declarations in <code>bzlib_private.h</code>. With those fu… |
|
4047 ported to Rust, we can remove <code>randtable.c</code>. Yay… |
|
4048 <h2>A few cleanups</h2> |
|
4049 <p>After moving to another house one throws away useless boxes; we… |
|
4050 to do some cleanup in the Rust code after the initial port, too.</p&g… |
|
4051 <p>Rust prefers snake_case fields rather than camelCase ones, and I |
|
4052 agree. I <a href="https://gitlab.com/federicomenaquintero/bzip2/comm… |
|
4053 <p>Then, I discovered that the <code>EState</code> str… |
|
4054 fields for the randomization state. I just <a href="https://gitlab.c… |
|
4055 <h2>Exegesis</h2> |
|
4056 <p>What is that randomization state all about?</p> |
|
4057 <p>And why does <code>DState</code> (the struct used d… |
|
4058 randomization state, but <code>EState</code> (used during co… |
|
4059 need it?</p> |
|
4060 <p>I found <a href="https://gitlab.com/federicomenaquintero/bzi… |
|
4061 <div class="highlight"><pre><span></span><cod… |
|
4062 <span class="cm"> Now a single bit indicating (non-)random… |
|
4063 <span class="cm"> As of version 0.9.5, we use a better sor… |
|
4064 <span class="cm"> which makes randomisation unnecessary. … |
|
4065 <span class="cm"> the randomised bit to &#39;no&#3… |
|
4066 <span class="cm"> still needs to be able to handle randomi… |
|
4067 <span class="cm"> so as to maintain backwards compatibilit… |
|
4068 <span class="cm"> older versions of bzip2.</span> |
|
4069 <span class="cm"> --*/</span> |
|
4070 <span class="n">bsW</span><span class="p">(</… |
|
4071 </code></pre></div> |
|
4072 |
|
4073 <p>Okay! So <em>compression</em> no longer uses rando… |
|
4074 <em>decompression</em> has to support files which were compr… |
|
4075 randomization. Here, <code>bsW(s,1,0)</code> always writes … |
|
4076 <p>However, the decompression code <a href="https://gitlab.com/… |
|
4077 bit</a> from the file so that it can see whether it is |
|
4078 dealing with an old-format file:</p> |
|
4079 <div class="highlight"><pre><span></span><cod… |
|
4080 </code></pre></div> |
|
4081 |
|
4082 <p>Later in the code, this <code>s-&gt;blockRandomised&l… |
|
4083 the bit is on, the code calls <code>BZ2_rand_update_mask()</cod… |
|
4084 appropriate. If one is using files compressed with Bzip2 0.9.5 or |
|
4085 later, those randomization functions are not even called.</p> |
|
4086 <p>Talk about preserving compatibility with the past.</p> |
|
4087 <h2>Explanation, or building my headcanon</h2> |
|
4088 <p>Bzip2's compression starts by running a <a href="https://en.… |
|
4089 Transform</a> on a block of data to compress, which is a wonderful |
|
4090 algorithm that I'm trying to fully understand. Part of the BWT |
|
4091 involves sorting all the string rotations of the block in question.</… |
|
4092 <p>Per <a href="https://gitlab.com/federicomenaquintero/bzip2/b… |
|
4093 randomization helper to make sorting perform well in extreme cases, |
|
4094 but not-so-old versions fixed this.</p> |
|
4095 <p>This explains why the decompression struct <code>DState&l… |
|
4096 <code>blockRandomised</code> bit, but the compression struct… |
|
4097 need one. The fields that the original macro was pasting into |
|
4098 <code>EState</code> were just a vestige from 1999, which is … |
|
4099 released.</p></content><category term="misc"></category><category … |
|
4100 <p>Perhaps the most exciting thing is that Dylan Baker made a mer… |
|
4101 request to add <a href="https://gitlab.com/federicomenaquintero/bzip2… |
|
4102 merged now into the master branch.</p> |
|
4103 <p>The current status is this:</p> |
|
4104 <ul> |
|
4105 <li>Both Meson and Autotools are …</li></ul></summar… |
|
4106 <p>Perhaps the most exciting thing is that Dylan Baker made a mer… |
|
4107 request to add <a href="https://gitlab.com/federicomenaquintero/bzip2… |
|
4108 merged now into the master branch.</p> |
|
4109 <p>The current status is this:</p> |
|
4110 <ul> |
|
4111 <li>Both Meson and Autotools are supported.</li> |
|
4112 <li>We have CI runs for both build systems.</li> |
|
4113 </ul> |
|
4114 <h2>A plea for help: add CI runners for other platforms!</h2> |
|
4115 <p>Do you use *BSD / Windows / Solaris / etc. and know how to make |
|
4116 Gitlab's CI work for them?</p> |
|
4117 <p>The only runners we have now for bzip2 are for well-known Linux |
|
4118 distros. I would really like to keep bzip2 working on non-Linux |
|
4119 platforms. If you know how to make Gitlab CI runners for other |
|
4120 systems, <a href="https://gitlab.com/federicomenaquintero/bzip2/merge… |
|
4121 <h2>Why two build systems?</h2> |
|
4122 <p>Mainly uncertainty on my part. I haven't used Meson extensivel… |
|
4123 people tell me that it works better than Autotools out of the box for |
|
4124 Windows.</p> |
|
4125 <p>Bzip2 runs on all sorts of ancient systems, and I don't know wh… |
|
4126 Meson or Autotools will be a better fit for them. Time will tell. |
|
4127 Hopefully in the future we can have only a single supported build |
|
4128 system for bzip2.</p></content><category term="misc"></category><c… |
|
4129 branch</a>, which means that if you had a previous clone of this |
|
4130 repository, you'll have to re-fetch it and rebase any changes you may |
|
4131 have on top.</p> |
|
4132 <p>I apologize for the inconvenience!</p> |
|
4133 <p>But I have a good excuse: Julian …</p></summary><conte… |
|
4134 branch</a>, which means that if you had a previous clone of this |
|
4135 repository, you'll have to re-fetch it and rebase any changes you may |
|
4136 have on top.</p> |
|
4137 <p>I apologize for the inconvenience!</p> |
|
4138 <p>But I have a good excuse: Julian Seward pointed me to a <a… |
|
4139 at sourceware</a> where Mark Wielaard reconstructed a commit |
|
4140 history for bzip2, based on the historical tarballs starting from |
|
4141 bzip2-0.1. Bzip2 was never maintained under revision control, so the |
|
4142 reconstructed repository should be used mostly for historical |
|
4143 reference (go look for <code>bzip2.exe</code> in the initial… |
|
4144 <p>I have rebased all the post-1.0.6 commits on top of Mark's repo… |
|
4145 this is what is in the <a href="https://gitlab.com/federicomenaquinte… |
|
4146 <p>There is a new <a href="https://gitlab.com/federicomenaquint… |
|
4147 where I will do the gradual port to Rust.</p> |
|
4148 <p>I foresee no other force-pushes to the master branch in the fut… |
|
4149 Apologies again if this disrupts your workflow.</p> |
|
4150 <p><strong>Update:</strong> <a href="https://gitlab… |
|
4151 weave the histories together, I'll do another force-push, the very |
|
4152 last one, I promise. If you send merge requests, I'll rebase them |
|
4153 myself if that happens.</p></content><category term="misc"></categ… |
|
4154 and <a href="http://valgrind.org/">Valgrind</a> fame. Julia… |
|
4155 to cede the maintainership of <a href="https://sourceware.org/bzip2/"… |
|
4156 to me.</p> |
|
4157 <p>Bzip2 has not had a release since 2010. In the meantime, Linux |
|
4158 distros have accumulated a number of bug/security fixes for it …</p… |
|
4159 and <a href="http://valgrind.org/">Valgrind</a> fame. Julia… |
|
4160 to cede the maintainership of <a href="https://sourceware.org/bzip2/"… |
|
4161 to me.</p> |
|
4162 <p>Bzip2 has not had a release since 2010. In the meantime, Linux |
|
4163 distros have accumulated a number of bug/security fixes for it. |
|
4164 Seemingly every distributor of bzip2 patches its build system. The |
|
4165 documentation generation step is a bit creaky. There is no source |
|
4166 control repository, nor bug tracker. I hope to fix these things |
|
4167 gradually.</p> |
|
4168 <p>This is the new <a href="https://gitlab.com/federicomenaquin… |
|
4169 <p>Ways in which you can immediately help by submitting merge requ… |
|
4170 <ul> |
|
4171 <li> |
|
4172 <p>Look at the <a href="https://gitlab.com/federicomenaquintero… |
|
4173 version number.</p> |
|
4174 </li> |
|
4175 <li> |
|
4176 <p>Create a basic <a href="https://gitlab.com/help/ci/README.md… |
|
4177 builds the code and runs the tests.</p> |
|
4178 </li> |
|
4179 <li> |
|
4180 <p>Test the autotools setup, courtesy of Stanislav Brabec, and imp… |
|
4181 it as you see fit.</p> |
|
4182 </li> |
|
4183 </ul> |
|
4184 <p>The <a href="https://people.gnome.org/~federico/blog/bzip2-i… |
|
4185 until the Autotools setup settles down.</p> |
|
4186 <p>I hope to have a 1.0.7 release soon, but this really needs <… |
|
4187 help. Let's revive this awesome little project.</p></content><cat… |
|
4188 <a href="https://sourceware.org/bzip2/">bzip2/bzlib</a> to R… |
|
4189 serve to refresh bzip2, which had its last release in 2010 and has |
|
4190 been nominally unmaintained for years.</p> |
|
4191 <p>I hope to make several posts detailing how this port is done �… |
|
4192 <a href="https://sourceware.org/bzip2/">bzip2/bzlib</a> to R… |
|
4193 serve to refresh bzip2, which had its last release in 2010 and has |
|
4194 been nominally unmaintained for years.</p> |
|
4195 <p>I hope to make several posts detailing how this port is done. … |
|
4196 post, I'll talk about setting up a Rust infrastructure for bzip2 and |
|
4197 my experiments in replacing the C code that does CRC32 computations.<… |
|
4198 <h2>Super-quick summary of how librsvg was ported to Rust</h2&g… |
|
4199 <ul> |
|
4200 <li> |
|
4201 <p>Add the necessary autotools infrastructure to build a Rust |
|
4202 sub-library that gets linked into the main public library.</p> |
|
4203 </li> |
|
4204 <li> |
|
4205 <p>Port bit by bit to Rust. Add unit tests as appropriate. Refac… |
|
4206 endlessly.</p> |
|
4207 </li> |
|
4208 <li> |
|
4209 <p><strong>MAINTAIN THE PUBLIC API/ABI AT ALL COSTS</stro… |
|
4210 notice that the library is being rewritten under their feet.</p> |
|
4211 </li> |
|
4212 </ul> |
|
4213 <p>I have no idea of how bzip2 works internally, but I do know how… |
|
4214 maintain ABIs, so let's get started.</p> |
|
4215 <h2>Bzip2's source tree</h2> |
|
4216 <p>As a very small project that just builds a library and couple of |
|
4217 executables, bzip2 was structured with all the source files directly |
|
4218 under a toplevel directory.</p> |
|
4219 <p>The only tests in there are three reference files that get comp… |
|
4220 then uncompressed, and then compared to the original ones.</p> |
|
4221 <p>As the rustification proceeds, I'll move the files around to be… |
|
4222 places. The scheme from librsvg worked well in this respect, so I'll |
|
4223 probably be copying many of the techniques and organization from |
|
4224 there.</p> |
|
4225 <h2>Deciding what to port first</h2> |
|
4226 <p>I looked a bit at the bzip2 sources, and the code to do CRC32 |
|
4227 computations seemed isolated enough from the rest of the code to port |
|
4228 easily.</p> |
|
4229 <p>The CRC32 code was arranged like this. First, a lookup table in |
|
4230 <code>crc32table.c</code>:</p> |
|
4231 <div class="highlight"><pre><span></span><cod… |
|
4232 <span class="mh">0x00000000L</span><span class="p">… |
|
4233 <span class="mh">0x130476dcL</span><span class="p">… |
|
4234 <span class="p">...</span> |
|
4235 <span class="p">}</span> |
|
4236 </code></pre></div> |
|
4237 |
|
4238 <p>And then, three macros in <code>bzlib_private.h</code&… |
|
4239 CRC32 code in the library:</p> |
|
4240 <div class="highlight"><pre><span></span><cod… |
|
4241 |
|
4242 <span class="cp">#define BZ_INITIALISE_CRC(crcVar) \&… |
|
4243 <span class="cp">{ \&… |
|
4244 <span class="cp"> crcVar = 0xffffffffL; \&… |
|
4245 <span class="cp">}</span> |
|
4246 |
|
4247 <span class="cp">#define BZ_FINALISE_CRC(crcVar) \&… |
|
4248 <span class="cp">{ \&… |
|
4249 <span class="cp"> crcVar = ~(crcVar); \&… |
|
4250 <span class="cp">}</span> |
|
4251 |
|
4252 <span class="cp">#define BZ_UPDATE_CRC(crcVar,cha) \&… |
|
4253 <span class="cp">{ \&… |
|
4254 <span class="cp"> crcVar = (crcVar &lt;&lt; 8) ^ … |
|
4255 <span class="cp"> BZ2_crc32Table[(crcVar &gt;&g… |
|
4256 <span class="cp"> ((UChar)cha)]; \&… |
|
4257 <span class="cp">}</span> |
|
4258 </code></pre></div> |
|
4259 |
|
4260 <p>Initially I wanted to just remove this code and replace it with… |
|
4261 the existing Rust crates to do CRC32 computations, but first I needed |
|
4262 to know which variant of CRC32 this is.</p> |
|
4263 <h2>Preparing the CRC32 port so it will not break</h2> |
|
4264 <p>I needed to set up tests for the CRC32 code so the replacement … |
|
4265 would compute exactly the same values as the original:</p> |
|
4266 <ul> |
|
4267 <li><a href="https://gitlab.com/federicomenaquintero/bzip2/comm… |
|
4268 crc32.c</a> - |
|
4269 that file is going to hold all the CRC32 code, not only the lookup tab… |
|
4270 <li><a href="https://gitlab.com/federicomenaquintero/bzip2/comm… |
|
4271 functions</a> - |
|
4272 so I can move them to Rust and have the C code call them.</li> |
|
4273 </ul> |
|
4274 <p>Then I needed a test that computed the CRC32 values of several |
|
4275 strings, so I could capture the results and make them part of the |
|
4276 test.</p> |
|
4277 <div class="highlight"><pre><span></span><cod… |
|
4278 <span class="k">static</span> <span class="k">const<… |
|
4279 <span class="k">static</span> <span class="k">const<… |
|
4280 <span class="k">static</span> <span class="k">const<… |
|
4281 |
|
4282 <span class="kt">int</span> |
|
4283 <span class="nf">main</span> <span class="p">(</spa… |
|
4284 <span class="p">{</span> |
|
4285 <span class="n">printf</span> <span class="p">(<… |
|
4286 <span class="n">printf</span> <span class="p">(<… |
|
4287 <span class="n">printf</span> <span class="p">(<… |
|
4288 <span class="n">printf</span> <span class="p">(<… |
|
4289 <span class="c1">// ...</span> |
|
4290 <span class="p">}</span> |
|
4291 </code></pre></div> |
|
4292 |
|
4293 <p>This computes the CRC32 values of some strings using the origin… |
|
4294 algorithm, and prints their results. Then I could cut&amp;paste tho… |
|
4295 results, and turn the <code>printf</code> into <code>a… |
|
4296 test.</p> |
|
4297 <div class="highlight"><pre><span></span><cod… |
|
4298 <span class="nf">main</span> <span class="p">(</spa… |
|
4299 <span class="p">{</span> |
|
4300 <span class="n">assert</span> <span class="p">(<… |
|
4301 <span class="n">assert</span> <span class="p">(<… |
|
4302 <span class="n">assert</span> <span class="p">(<… |
|
4303 <span class="n">assert</span> <span class="p">(<… |
|
4304 <span class="c1">// ...</span> |
|
4305 <span class="p">}</span> |
|
4306 </code></pre></div> |
|
4307 |
|
4308 <h2>Setting up a Rust infrastructure for bzip2</h2> |
|
4309 <p>Two things made this reasonably easy:</p> |
|
4310 <ul> |
|
4311 <li>A patch from Stanislav Brabec, from Suse, to <a href="https… |
|
4312 framework to bzip2</a></li> |
|
4313 <li>The existing <a href="https://people.gnome.org/~federico/bl… |
|
4314 </ul> |
|
4315 <p>I.e. "copy and paste from somewhere that I know works well". |
|
4316 Wonderful!</p> |
|
4317 <p>This is the <a href="https://gitlab.com/federicomenaquintero… |
|
4318 bzip2</a>. It does the following:</p> |
|
4319 <ol> |
|
4320 <li>Create a Cargo workspace (a <code>Cargo.toml</code>… |
|
4321 single member, a <code>bzlib_rust</code> directory where … |
|
4322 of the code will live.</li> |
|
4323 <li>Create <code>bzlib_rust/Cargo.toml</code> and <… |
|
4324 sources. This will generate a <code>staticlib</code> for… |
|
4325 can be linked into the main <code>libbz2.la</code>.</l… |
|
4326 <li>Puts in automake hooks so that <code>make clean</code… |
|
4327 do what you expect for the Rust part.</li> |
|
4328 </ol> |
|
4329 <p>As a side benefit, librsvg's Autotools+Rust infrastructure alre… |
|
4330 handled things like cross-compilation correctly, so I have high hopes |
|
4331 that this will be good enough for bzip2.</p> |
|
4332 <h2>Can I use a Rust crate for CRC32?</h2> |
|
4333 <p>There are <a href="https://crates.io/search?q=crc&amp;so… |
|
4334 hoping especially to be able to use <a href="https://github.com/srijs… |
|
4335 SIMD-accelerated.</p> |
|
4336 <p>I wrote a Rust version of the "CRC me a buffer" test from above… |
|
4337 if crc32fast produced the same values as the C code, and of course it |
|
4338 didn't. Eventually, after <a href="https://mstdn.mx/@federicomena/10… |
|
4339 out</a> what variant of CRC32 is being used in the original |
|
4340 code.</p> |
|
4341 <p>It turns out that this is directly doable in Rust with the <… |
|
4342 of the <code>crc</code> crate</a>. This crate lets on… |
|
4343 polynomial and the mode of computation; there are <a href="https://gi… |
|
4344 CRC32</a> and I wasn't fully aware of them.</p> |
|
4345 <p>The magic incantation is this:</p> |
|
4346 <div class="highlight"><pre><span></span><cod… |
|
4347 </code></pre></div> |
|
4348 |
|
4349 <p>With that, <a href="https://gitlab.com/federicomenaquintero/… |
|
4350 test</a> |
|
4351 produces the same values as the C code. Yay!</p> |
|
4352 <h2>But it can't be that easy</h2> |
|
4353 <p>Bzlib stores its internal state in the <a href="https://gitl… |
|
4354 struct</a>, |
|
4355 defined in |
|
4356 <a href="https://gitlab.com/federicomenaquintero/bzip2/blob/60be65f9/… |
|
4357 <p>That struct stores several running CRC32 computations, and the … |
|
4358 for each one of those is a single <code>UInt32</code> value.… |
|
4359 just replace those struct fields with something that comes from Rust, |
|
4360 since the C code does not know the size of a <code>crc32::Digest&l… |
|
4361 Rust.</p> |
|
4362 <p>The normal way to do this (say, like in librsvg) would be to tu… |
|
4363 <code>UInt32 some_crc</code> into <code>void *some_crc… |
|
4364 Rust side, with whatever size it needs.</p> |
|
4365 <p><strong>However!</strong></p> |
|
4366 <p>It turns out that bzlib lets the caller <a href="https://git… |
|
4367 allocator</a> |
|
4368 so that bzlib doesn't use plain <code>malloc()</code> by def… |
|
4369 <p>Rust lets one define a <a href="https://doc.rust-lang.org/al… |
|
4370 However, bzlib's concept of a custom allocator includes a bit of |
|
4371 context:</p> |
|
4372 <div class="highlight"><pre><span></span><cod… |
|
4373 <span class="c1">// ...</span> |
|
4374 |
|
4375 <span class="kt">void</span> <span class="o">*<… |
|
4376 <span class="kt">void</span> <span class="p">(<… |
|
4377 <span class="kt">void</span> <span class="o">*<… |
|
4378 <span class="p">}</span> <span class="n">bz_stream<… |
|
4379 </code></pre></div> |
|
4380 |
|
4381 <p>The caller sets up <code>bzalloc/bzfree</code> call… |
|
4382 context for the allocator. However, Rust's <code>GlobalAlloc</… |
|
4383 compilation time, and we can't pass that context in a good, |
|
4384 thread-safe fashion to it.</p> |
|
4385 <h2>Who uses the bzlib custom allocator, anyway?</h2> |
|
4386 <p>If one sets <code>bzalloc/bzfree</code> to <code… |
|
4387 plain <code>malloc()/free()</code> by default. Most softwar… |
|
4388 <p>I am looking in <a href="https://codesearch.debian.net/searc… |
|
4389 gets set, hoping that I can figure out if that software really needs a |
|
4390 custom allocator, or if they are just dressing up <code>malloc()&l… |
|
4391 logging code or similar (ImageMagick seems to do this; Python seems to |
|
4392 have a genuine concern about the Global Interpreter Lock). Debian's |
|
4393 codesearch is a fantastic tool!</p> |
|
4394 <h2>The first rustified code</h2> |
|
4395 <p>I <a href="https://gitlab.com/federicomenaquintero/bzip2/com… |
|
4396 table</a> |
|
4397 and fixed it up for Rust's syntax, and also ported the CRC32 |
|
4398 computation functions. I gave them the same names as the original C |
|
4399 ones, and exported them, e.g.</p> |
|
4400 <div class="highlight"><pre><span></span><cod… |
|
4401 <span class="w"> </span><span class="mh">0x00000000&… |
|
4402 <span class="w"> </span><span class="o">..</span&… |
|
4403 <span class="p">};</span><span class="w"></span> |
|
4404 |
|
4405 <span class="cp">#[no_mangle]</span><span class="w">&l… |
|
4406 <span class="k">pub</span><span class="w"> </span&g… |
|
4407 <span class="w"> </span><span class="o">*</span&… |
|
4408 <span class="p">}</span><span class="w"></span> |
|
4409 </code></pre></div> |
|
4410 |
|
4411 <p>This is a straight port of the C code. Rust is very strict abo… |
|
4412 integer sizes, and arrays can only be indexed with a <code>usize&l… |
|
4413 random integer — hence the explicit conversions.</p> |
|
4414 <p>And with this, <a href="https://gitlab.com/federicomenaquint… |
|
4415 linkage</a>, |
|
4416 the tests pass!</p> |
|
4417 <p>First pass at rustifying CRC32: <strong>done</strong&g… |
|
4418 <h2>But that does one byte at a time</h2> |
|
4419 <p>Indeed; the original C code to do CRC32 only handled one byte a… |
|
4420 time. If I replace this with a SIMD-enabled Rust crate, it will want |
|
4421 to process whole buffers at once. I hope the code in bzlib can be |
|
4422 refactored to do that. We'll see!</p> |
|
4423 <h2>How to use an existing Rust crate for this</h2> |
|
4424 <p>I just found out that one does not in fact need to use a comple… |
|
4425 <code>crc32::Digest</code> to do equivalent computations; on… |
|
4426 <a href="https://docs.rs/crc/1.8.1/crc/crc32/fn.update.html">crc32… |
|
4427 by hand and maintain a single <code>u32</code> state, just l… |
|
4428 <code>UInt32</code> from the C code.</p> |
|
4429 <p>So, I may not need to mess around with a custom allocator just … |
|
4430 Stay tuned.</p> |
|
4431 <p>In the meantime, I've <a href="https://github.com/srijs/rust… |
|
4432 crc32fast</a> to make |
|
4433 it possible to use a custom polynomial and order and still get the |
|
4434 benefits of SIMD.</p></content><category term="misc"></category><c… |
|
4435 instantiate a GObject and then change its state via method calls. |
|
4436 Sometimes this is expected and desired; a <code>GtkCheckButton<… |
|
4437 certainly can change its internal state from pressed to not pressed, |
|
4438 for example.</p> |
|
4439 <p>Other times, objects are mutable while they are being …</p… |
|
4440 instantiate a GObject and then change its state via method calls. |
|
4441 Sometimes this is expected and desired; a <code>GtkCheckButton<… |
|
4442 certainly can change its internal state from pressed to not pressed, |
|
4443 for example.</p> |
|
4444 <p>Other times, objects are mutable while they are being "assemble… |
|
4445 "configured", and only yield a final immutable result until later. |
|
4446 This is the case for <code>RsvgHandle</code> from librsvg.&l… |
|
4447 <p>Please bear with me while I write about the history of the |
|
4448 <code>RsvgHandle</code> API and why it ended up with differe… |
|
4449 same thing.</p> |
|
4450 <h2>The traditional RsvgHandle API</h2> |
|
4451 <p>The final purpose of an <code>RsvgHandle</code> is … |
|
4452 loaded in memory. Once it is loaded, the SVG document does not |
|
4453 change, as librsvg does not support animation or creating/removing SVG |
|
4454 elements; it is a static renderer.</p> |
|
4455 <p>However, before an <code>RsvgHandle</code> achieves… |
|
4456 to be loaded first. Loading can be done in two ways:</p> |
|
4457 <ul> |
|
4458 <li>The historical/deprecated way, using the <a href="https://d… |
|
4459 <code>rsvg_handle_close()</code> APIs. Plenty of code in … |
|
4460 <code>write/close</code> idiom before GLib got a good abst… |
|
4461 streams; you can see another example in <a href="https://developer.… |
|
4462 The idea is that applications do this:</li> |
|
4463 </ul> |
|
4464 <div class="highlight"><pre><span></span><cod… |
|
4465 <span class="n">handle</span> <span class="o">=</sp… |
|
4466 |
|
4467 <span class="k">while</span> <span class="p">(</spa… |
|
4468 <span class="n">rsvg_handle_write</span><span class="p… |
|
4469 <span class="p">}</span> |
|
4470 |
|
4471 <span class="n">rsvg_handle_close</span> <span class="p"&… |
|
4472 |
|
4473 <span class="o">//</span> <span class="n">now</span… |
|
4474 |
|
4475 <span class="n">rsvg_handle_render</span> <span class="p"… |
|
4476 </code></pre></div> |
|
4477 |
|
4478 <ul> |
|
4479 <li>The streaming way, with <a href="https://developer.gnome.or… |
|
4480 which takes a <a href="https://developer.gnome.org/gio/unstable/GIn… |
|
4481 which take a <a href="https://developer.gnome.org/gio/unstable/GFil… |
|
4482 </ul> |
|
4483 <div class="highlight"><pre><span></span><cod… |
|
4484 <span class="n">stream</span> <span class="o">=</sp… |
|
4485 <span class="n">handle</span> <span class="o">=</sp… |
|
4486 |
|
4487 <span class="n">rsvg_handle_read_stream_sync</span> <span… |
|
4488 |
|
4489 <span class="o">//</span> <span class="n">now</span… |
|
4490 |
|
4491 <span class="n">rsvg_handle_render</span> <span class="p"… |
|
4492 </code></pre></div> |
|
4493 |
|
4494 <h2>A bit of history</h2> |
|
4495 <p>Let's consider a few of <code>RsvgHandle</code>'s f… |
|
4496 <p><strong>Constructors:</strong></p> |
|
4497 <ul> |
|
4498 <li><code>rsvg_handle_new()</code></li> |
|
4499 <li><code>rsvg_handle_new_with_flags()</code></li&g… |
|
4500 </ul> |
|
4501 <p><strong>Configure the handle for loading:</strong>&… |
|
4502 <ul> |
|
4503 <li><code>rsvg_handle_set_base_uri()</code></li> |
|
4504 <li><code>rsvg_handle_set_base_gfile()</code></li&g… |
|
4505 </ul> |
|
4506 <p><strong>Deprecated loading API:</strong></p> |
|
4507 <ul> |
|
4508 <li><code>rsvg_handle_write()</code></li> |
|
4509 <li><code>rsvg_handle_close()</code></li> |
|
4510 </ul> |
|
4511 <p><strong>Streaming API:</strong></p> |
|
4512 <ul> |
|
4513 <li><code>rsvg_handle_read_stream_sync()</code></li… |
|
4514 </ul> |
|
4515 <p>When librsvg first acquired the concept of an <code>RsvgH… |
|
4516 had <code>rsvg_handle_new()</code> with no arguments. About… |
|
4517 got <code>rsvg_handle_new_with_flags()</code> to allow more … |
|
4518 another 2 years to actually add some usable flags — the first one was |
|
4519 to configure the parsing limits in the underlying calls to libxml2.</… |
|
4520 <p>About 3 years after <code>RsvgHandle</code> appear… |
|
4521 <code>rsvg_handle_set_base_uri()</code> to configure the "ba… |
|
4522 relative references in the SVG document get resolved. For example, if |
|
4523 you are reading <code>/foo/bar.svg</code> and it contains an… |
|
4524 xlink:ref="smiley.png"/&gt;</code>, then librsvg needs to be a… |
|
4525 the path <code>/foo/smiley.png</code> and that is done relat… |
|
4526 (The base URI is implicit when reading from a specific SVG file, but |
|
4527 it needs to be provided when reading from an arbitrary stream that may |
|
4528 not even come from a file.)</p> |
|
4529 <p>Initially <code>RsvgHandle</code> had the <code&… |
|
4530 it got the streaming functions once GIO appeared. Eventually the |
|
4531 streaming API would be the preferred one, instead of just being a |
|
4532 convenience for those brave new apps that started using GIO.</p> |
|
4533 <p>A summary of librsvg's API may be something like:</p> |
|
4534 <ul> |
|
4535 <li> |
|
4536 <p>librsvg gets written initially; it doesn't even have an |
|
4537 <code>RsvgHandle</code>, and just provides a single functi… |
|
4538 <code>FILE *</code> and renders it to a <code>GdkPix… |
|
4539 </li> |
|
4540 <li> |
|
4541 <p>That gets replaced with <code>RsvgHandle</code>, it… |
|
4542 constructor, and the <code>write/close</code> API to feed … |
|
4543 progressively.</p> |
|
4544 </li> |
|
4545 <li> |
|
4546 <p>GIO appears, we get the first widespread streaming APIs in GNOM… |
|
4547 and <code>RsvgHandle</code> gets the ability to read from … |
|
4548 </li> |
|
4549 <li> |
|
4550 <p><code>RsvgHandle</code> gets <code>rsvg_handl… |
|
4551 may want to configure extra stuff for libxml2.</p> |
|
4552 </li> |
|
4553 <li> |
|
4554 <p>When Cairo appears and librsvg is ported to it, <code>Rsv… |
|
4555 extra flag so that SVGs rendered to PDF can embed image data |
|
4556 efficiently.</p> |
|
4557 </li> |
|
4558 </ul> |
|
4559 <p>It's a convoluted history, but <code>git log -- rsvg.h<… |
|
4560 <h2>Where is the mutability?</h2> |
|
4561 <p>An <code>RsvgHandle</code> gets created, with flags… |
|
4562 doesn't know if it will be given data with the <code>write/close&l… |
|
4563 with the streaming API. Also, someone may call <code>set_base_uri… |
|
4564 it. So, the handle must remain mutable while it is being populated |
|
4565 with data. After that, it can say, "no more changes, I'm done".</p&g… |
|
4566 <p>In C, this doesn't even have a name. Everything is mutable by … |
|
4567 all the time. This monster was the private data of <code>RsvgHand… |
|
4568 before it got ported to Rust:</p> |
|
4569 <div class="highlight"><pre><span></span><cod… |
|
4570 <span class="c1">// set during construction</span> |
|
4571 <span class="n">RsvgHandleFlags</span> <span class="n… |
|
4572 |
|
4573 <span class="c1">// GObject-ism</span> |
|
4574 <span class="n">gboolean</span> <span class="n">is… |
|
4575 |
|
4576 <span class="c1">// Extra crap for a deprecated API</span&g… |
|
4577 <span class="n">RsvgSizeFunc</span> <span class="n"&g… |
|
4578 <span class="n">gpointer</span> <span class="n">us… |
|
4579 <span class="n">GDestroyNotify</span> <span class="n"… |
|
4580 |
|
4581 <span class="c1">// Data only used while parsing an SVG</sp… |
|
4582 <span class="n">RsvgHandleState</span> <span class="n… |
|
4583 <span class="n">RsvgDefs</span> <span class="o">*&… |
|
4584 <span class="n">guint</span> <span class="n">nest_… |
|
4585 <span class="n">RsvgNode</span> <span class="o">*&… |
|
4586 <span class="n">RsvgNode</span> <span class="o">*&… |
|
4587 <span class="n">GHashTable</span> <span class="o">… |
|
4588 <span class="n">RsvgSaxHandler</span> <span class="o"… |
|
4589 <span class="kt">int</span> <span class="n">handle… |
|
4590 <span class="n">GHashTable</span> <span class="o">… |
|
4591 <span class="n">xmlParserCtxtPtr</span> <span class="… |
|
4592 <span class="n">GError</span> <span class="o">**&l… |
|
4593 <span class="n">GCancellable</span> <span class="o"&g… |
|
4594 <span class="n">GInputStream</span> <span class="o"&g… |
|
4595 |
|
4596 <span class="c1">// Data only used while rendering</span> |
|
4597 <span class="kt">double</span> <span class="n">dpi… |
|
4598 <span class="kt">double</span> <span class="n">dpi… |
|
4599 |
|
4600 <span class="c1">// The famous base URI, set before loading<… |
|
4601 <span class="n">gchar</span> <span class="o">*<… |
|
4602 <span class="n">GFile</span> <span class="o">*<… |
|
4603 |
|
4604 <span class="c1">// Some internal stuff</span> |
|
4605 <span class="n">gboolean</span> <span class="n">in… |
|
4606 <span class="n">gboolean</span> <span class="n">is… |
|
4607 <span class="p">};</span> |
|
4608 </code></pre></div> |
|
4609 |
|
4610 <p>"Single responsibility principle"? This is a horror show. That |
|
4611 <code>RsvgHandlePrivate</code> struct has all of these:</… |
|
4612 <ul> |
|
4613 <li>Data only settable during construction (flags)</li> |
|
4614 <li>Data set after construction, but which may only be set before |
|
4615 loading (base URI)</li> |
|
4616 <li>Highly mutable data used only during the loading stage: state |
|
4617 machines, XML parsers, a stack of XML elements, CSS properties...</… |
|
4618 <li>The DPI (dots per inch) values only used during rendering.<… |
|
4619 <li>Assorted fields used at various stages of the handle's life.&l… |
|
4620 </ul> |
|
4621 <p>It took a lot of refactoring to get the code to a point where i… |
|
4622 clear that an <code>RsvgHandle</code> in fact has distinct s… |
|
4623 lifetime, and that some of that data should only live during a |
|
4624 particular stage. Before, everything seemed a jumble of fields, used |
|
4625 at various unclear points in the code (for the struct listing above, |
|
4626 I've grouped related fields together — they were somewhat shuffled in |
|
4627 the original code!).</p> |
|
4628 <h2>What would a better separation look like?</h2> |
|
4629 <p>In the <a href="https://gitlab.gnome.org/GNOME/librsvg/">… |
|
4630 <div class="highlight"><pre><span></span><cod… |
|
4631 <span class="sd">/// from the C API.</span> |
|
4632 <span class="k">pub</span><span class="w"> </span&g… |
|
4633 <span class="w"> </span><span class="n">dpi</spa… |
|
4634 <span class="w"> </span><span class="n">load_flags&… |
|
4635 |
|
4636 <span class="w"> </span><span class="n">base_url<… |
|
4637 <span class="w"> </span><span class="c1">// needed … |
|
4638 <span class="w"> </span><span class="n">base_url_cs… |
|
4639 |
|
4640 <span class="w"> </span><span class="n">size_callba… |
|
4641 <span class="w"> </span><span class="n">is_testing&… |
|
4642 <span class="w"> </span><span class="n">load_state&… |
|
4643 <span class="p">}</span><span class="w"></span> |
|
4644 </code></pre></div> |
|
4645 |
|
4646 <p>Internally, that <code>CHandle</code> struct is now… |
|
4647 public <code>RsvgHandle</code> object. Note that all of <… |
|
4648 <code>Cell&lt;&gt;</code> or <code>RefCell&… |
|
4649 allow for "interior mutability" in the <code>CHandle</code> … |
|
4650 modified after intialization.</p> |
|
4651 <p>The last field's cell, <code>load_state</code>, con… |
|
4652 <div class="highlight"><pre><span></span><cod… |
|
4653 <span class="w"> </span><span class="n">Start</s… |
|
4654 |
|
4655 <span class="w"> </span><span class="c1">// Being l… |
|
4656 <span class="w"> </span><span class="n">Loading<… |
|
4657 |
|
4658 <span class="w"> </span><span class="c1">// Fully l… |
|
4659 <span class="w"> </span><span class="n">ClosedOk<… |
|
4660 |
|
4661 <span class="w"> </span><span class="n">ClosedError… |
|
4662 <span class="p">}</span><span class="w"></span> |
|
4663 </code></pre></div> |
|
4664 |
|
4665 <p>A <code>CHandle</code> starts in the <code>St… |
|
4666 will be loaded with a stream, or with the legacy write/close API.</p&… |
|
4667 <p>If the caller uses the write/close API, the handle moves to the |
|
4668 <code>Loading</code> state, which has a <code>buffer&l… |
|
4669 being fed to it.</p> |
|
4670 <p>But if the caller uses the stream API, the handle tries to pars… |
|
4671 SVG document from the stream, and it moves either to the <code>Clo… |
|
4672 state, or to the <code>ClosedError</code> state if there is … |
|
4673 <p>Correspondingly, when using the write/close API, when the caller |
|
4674 finally calls <code>rsvg_handle_close()</code>, the handle c… |
|
4675 the <code>buffer</code>, parses it, and also gets either int… |
|
4676 <code>ClosedError</code> state.</p> |
|
4677 <p>If you look at the variant <code>ClosedOk { handle: Handl… |
|
4678 a fully loaded <code>Handle</code> inside, which right now i… |
|
4679 around a reference-counted <code>Svg</code> object:</p> |
|
4680 <div class="highlight"><pre><span></span><cod… |
|
4681 <span class="w"> </span><span class="n">svg</spa… |
|
4682 <span class="p">}</span><span class="w"></span> |
|
4683 </code></pre></div> |
|
4684 |
|
4685 <p>The reason why <code>LoadState::ClosedOk</code> doe… |
|
4686 directly, and instead wraps it with a <code>Handle</code>, i… |
|
4687 the first pass at refactoring. Also, <code>Handle</code> co… |
|
4688 API-level logic which I'm not completely sure makes sense as a |
|
4689 lower-level <code>Svg</code> object. We'll see.</p> |
|
4690 <h2>Couldn't you move more of <code>CHandle</code>'s f… |
|
4691 <p>Sort of, kind of, but the public API still lets one do things l… |
|
4692 call <code>rsvg_handle_get_base_uri()</code> after the handl… |
|
4693 even though its result will be of little value. So, the fields that |
|
4694 hold the <code>base_uri</code> information are kept in the l… |
|
4695 <code>CHandle</code>, not in the individual variants of <… |
|
4696 <h2>How does this look from the Rust API?</h2> |
|
4697 <p><code>CHandle</code> implements the public C API of… |
|
4698 <code>Handle</code> implements the basic "load from stream",… |
|
4699 an SVG element", and "render to a Cairo context" functionality.</p> |
|
4700 <p>This basic functionality gets exported in a cleaner way through… |
|
4701 <a href="https://gitlab.gnome.org/GNOME/librsvg/blob/master/librsvg_c… |
|
4702 interior mutability in there at all; that API uses a builder pattern |
|
4703 to gradually configure an SVG loader, which returns a fully loaded |
|
4704 <code>SvgHandle</code>, out of which you can create a <co… |
|
4705 <p>In fact, it may be possible to refactor all of this a bit and |
|
4706 implement <code>CHandle</code> directly in terms of the new … |
|
4707 use <code>CHandle</code> as the "holding space" while the SV… |
|
4708 configured, and later turned into a fully loaded <code>SvgHandle&l… |
|
4709 internally.</p> |
|
4710 <h2>Conclusion</h2> |
|
4711 <p>The C version of <code>RsvgHandle</code>'s private … |
|
4712 of fields. Without knowing the code, it was hard to know that they |
|
4713 belonged in groups, and each group corresponded roughtly to a stage in |
|
4714 the handle's lifetime.</p> |
|
4715 <p>It took plenty of refactoring to get the fields split up cleanl… |
|
4716 librsvg's internals. The process of refactoring <code>RsvgHandle&… |
|
4717 and ensuring that the various states of a handle are consistent, in |
|
4718 fact exposed a few bugs where the state was not being checked |
|
4719 appropriately. The public C API remains the same as always, but has |
|
4720 better internal checks now.</p> |
|
4721 <p>GObject APIs tend to allow for a lot of mutability via methods … |
|
4722 change the internal state of objects. For <code>RsvgHandle</co… |
|
4723 to change this into a single <code>CHandle</code> that maint… |
|
4724 in a contained fashion, and later translates it internally into an |
|
4725 immutable <code>Handle</code> that represents a fully-loaded… |
|
4726 scheme ties in well with the new Rust API for librsvg, which keeps |
|
4727 everything immutable after creation.</p></content><category term="… |
|
4728 librsvg's main library, I wanted to start porting the high-level test |
|
4729 suite to Rust. This is mainly to be able to run tests in parallel, |
|
4730 which <code>cargo test</code> does automatically in order to… |
|
4731 However, this meant that librsvg needed …</p></summary><content … |
|
4732 librsvg's main library, I wanted to start porting the high-level test |
|
4733 suite to Rust. This is mainly to be able to run tests in parallel, |
|
4734 which <code>cargo test</code> does automatically in order to… |
|
4735 However, this meant that librsvg needed a Rust API that would exercise |
|
4736 the same code paths as the C entry points.</p> |
|
4737 <p>At the same time, I wanted the Rust API to make it impossible to |
|
4738 misuse the library. From the viewpoint of the C API, an <code>Rsv… |
|
4739 has different stages:</p> |
|
4740 <ul> |
|
4741 <li>Just initialized</li> |
|
4742 <li>Loading</li> |
|
4743 <li>Loaded, or in an error state after a failed load</li> |
|
4744 <li>Ready to render</li> |
|
4745 </ul> |
|
4746 <p>To ensure consistency, the public API checks that you cannot re… |
|
4747 <code>RsvgHandle</code> that is not completely loaded yet, o… |
|
4748 in a loading error. But wouldn't it be nice if it were impossible to |
|
4749 call the API functions in the wrong order?</p> |
|
4750 <p>This is exactly what <a href="https://gitlab.gnome.org/GNOME… |
|
4751 to which you give a filename or a stream, and it will return a |
|
4752 fully-loaded <code>SvgHandle</code> or an error. Then, you … |
|
4753 <code>CairoRenderer</code> if you have an <code>SvgHan… |
|
4754 <p>For historical reasons, the C API in librsvg is not perfectly |
|
4755 consistent. For example, some functions which return an error will |
|
4756 actually return a proper <a href="https://developer.gnome.org/glib/st… |
|
4757 return a <code>gboolean</code> with no further explanation o… |
|
4758 In contrast, all the Rust API functions that can fail will actually |
|
4759 return a <a href="https://doc.rust-lang.org/std/result/index.html">… |
|
4760 error value. In the Rust API, there is no "wrong order" in which the |
|
4761 various API functions and methods can be called; it tries to do the |
|
4762 whole "make invalid states unrepresentable".</p> |
|
4763 <p>To implement <a href="https://gitlab.gnome.org/GNOME/librsvg… |
|
4764 <a href="https://gitlab.gnome.org/GNOME/librsvg/blob/master/rsvg_inte… |
|
4765 realize that librsvg could be a lot easier to use. The C API has |
|
4766 always forced you to call it in this fashion:</p> |
|
4767 <ol> |
|
4768 <li>Ask the SVG for its dimensions, or how big it is.</li> |
|
4769 <li>Based on that, scale your Cairo context to the size you actual… |
|
4770 want.</li> |
|
4771 <li>Render the SVG to that context's current transformation matrix… |
|
4772 </ol> |
|
4773 <p>But first, (1) gives you inadequate information because |
|
4774 <code>rsvg_handle_get_dimensions()</code> returns <a href… |
|
4775 structure</a> with <code>int</code> fields for the wid… |
|
4776 height. The API is similar to gdk-pixbuf's in that it always wants to |
|
4777 think in whole pixels. However, an SVG is not necessarily |
|
4778 integer-sized.</p> |
|
4779 <p>Then, (2) forces you to calculate some geometry in almost all c… |
|
4780 as most apps want to render SVG content scaled proportionally to a |
|
4781 certain size. This is not hard to do, but it's an inconvenience.</p&… |
|
4782 <h2>SVG dimensions</h2> |
|
4783 <p>Let's look at (1) again. The question, "how big is the SVG" is… |
|
4784 meaningless when we consider that SVGs <strong>can be scaled to an… |
|
4785 that's the whole point of them!</p> |
|
4786 <p>When you ask <code>RsvgHandle</code> how big it is,… |
|
4787 you and whisper in your ear, "how big do you want it to be?".</p> |
|
4788 <p>And that's the thing. The HTML/CSS/SVG model is that one embeds |
|
4789 content into <strong>viewports</strong> of a given size. Th… |
|
4790 responsible for scaling the content to fit into that viewport.</p> |
|
4791 <p>In the end, what we want is a rendering function that takes a C… |
|
4792 context and a Rectangle for a viewport, and that's it. The function |
|
4793 should take care of fitting the SVG's contents within that viewport.<… |
|
4794 <p>There is now <a href="https://gitlab.gnome.org/GNOME/librsvg… |
|
4795 the end, programs should just have to load their SVG handle, and |
|
4796 directly ask it to render at whatever size they need, instead of doing |
|
4797 the size computations by hand.</p> |
|
4798 <h2>When will this be available?</h2> |
|
4799 <p>I'm in the middle of a <a href="https://gitlab.gnome.org/fed… |
|
4800 to make this <em>viewport</em> concept really work. So far … |
|
4801 <ul> |
|
4802 <li> |
|
4803 <p>Defining APIs that take a viewport.</p> |
|
4804 </li> |
|
4805 <li> |
|
4806 <p>Refactoring all the geometry computation to support the semanti… |
|
4807 C API, plus the new <code>with_viewport</code> semantics.&… |
|
4808 </li> |
|
4809 <li> |
|
4810 <p>Fixing the code that kept track of an internal offset for all |
|
4811 temporary images.</p> |
|
4812 </li> |
|
4813 <li> |
|
4814 <p>Refactoring all the code that mucks around with the Cairo conte… |
|
4815 affine transformation matrix, which is a big mutable mess.</p> |
|
4816 </li> |
|
4817 <li> |
|
4818 <p>Tests, examples, documentation.</p> |
|
4819 </li> |
|
4820 </ul> |
|
4821 <p>I want to make the Rust API available for the 2.46 release, whi… |
|
4822 hopefully not too far off. It should be ready for the next GNOME |
|
4823 release. In the meantime, you can check out the open bugs for the |
|
4824 <a href="https://gitlab.gnome.org/GNOME/librsvg/milestones/20">2.4… |
|
4825 for the first 3.33 tarballs is approximately one month from now!</str… |
|
4826 with Rust and Cargo is Cargo's use of build scripts, i.e. the |
|
4827 <code>build.rs</code> that many Rust programs use for doing … |
|
4828 main build. This post is about my exploration of what <code>build… |
|
4829 with Rust and Cargo is Cargo's use of build scripts, i.e. the |
|
4830 <code>build.rs</code> that many Rust programs use for doing … |
|
4831 main build. This post is about my exploration of what <code>build… |
|
4832 does.</p> |
|
4833 <p>Thanks to <a href="https://nirbheek.in/">Nirbheek Chauhan… |
|
4834 and additions to a draft of this article!</p> |
|
4835 <p>TL;DR: <code>build.rs</code> is pretty ad-hoc and s… |
|
4836 compared to Meson's very nice, high-level patterns for build-time |
|
4837 things.</p> |
|
4838 <p>I have the intuition that giving names to the things that are |
|
4839 usually done in <code>build.rs</code> scripts, and creating … |
|
4840 them, can make it easier later to implement those abstractions in |
|
4841 terms of Meson. Maybe we can eliminate <code>build.rs</code>… |
|
4842 Maybe Cargo can acquire higher-level concepts that plug well to Meson?&l… |
|
4843 <p>(That is... I think we can refactor our way out of this mess.)&… |
|
4844 <h2>What does <code>build.rs</code> do?</h2> |
|
4845 <p>The first paragraph in the <a href="https://doc.rust-lang.or… |
|
4846 scripts</a> tells us this:</p> |
|
4847 <blockquote> |
|
4848 <p>Some packages need to compile third-party non-Rust code, for ex… |
|
4849 C libraries. Other packages need to link to C libraries which can |
|
4850 either be located on the system or possibly need to be built from |
|
4851 source. Others still need facilities for functionality such as code |
|
4852 generation before building (think parser generators).</p> |
|
4853 </blockquote> |
|
4854 <p>That is,</p> |
|
4855 <ul> |
|
4856 <li> |
|
4857 <p>Compiling third-party non-Rust code. For example, maybe there … |
|
4858 C sub-library that the Rust crate needs.</p> |
|
4859 </li> |
|
4860 <li> |
|
4861 <p>Link to C libraries... located on the system... or built from |
|
4862 source. For example, in <a href="https://gtk-rs.org">gtk-rs<… |
|
4863 <code>libgtk-3.so</code>, <code>libcairo.so</code… |
|
4864 those libraries with <code>pkg-config</code>.</p> |
|
4865 </li> |
|
4866 <li> |
|
4867 <p>Code generation. In the C world this could be generating a par… |
|
4868 with <code>yacc</code>; in the Rust world there are many u… |
|
4869 code that is later used in your actual program.</p> |
|
4870 </li> |
|
4871 </ul> |
|
4872 <p>In the next sections I'll look briefly at each of these cases, … |
|
4873 a different order.</p> |
|
4874 <h2>Code generation</h2> |
|
4875 <p>Here is an example, in <a href="https://gitlab.gnome.org/GNO… |
|
4876 for a couple of things that get autogenerated before compiling the |
|
4877 main library:</p> |
|
4878 <ul> |
|
4879 <li>A perfect hash function (PHF) of attributes and CSS property n… |
|
4880 <li>A pair of lookup tables for SRGB linearization and un-lineariz… |
|
4881 </ul> |
|
4882 <p>For example, this is <code>main()</code> in <cod… |
|
4883 <div class="highlight"><pre><span></span><cod… |
|
4884 <span class="w"> </span><span class="n">generate_ph… |
|
4885 <span class="w"> </span><span class="n">generate_sr… |
|
4886 <span class="p">}</span><span class="w"></span> |
|
4887 </code></pre></div> |
|
4888 |
|
4889 <p>And this is the first few lines of of the first function:</p… |
|
4890 <div class="highlight"><pre><span></span><cod… |
|
4891 <span class="w"> </span><span class="kd">let</sp… |
|
4892 <span class="w"> </span><span class="kd">let</sp… |
|
4893 |
|
4894 <span class="w"> </span><span class="fm">writeln!&l… |
|
4895 |
|
4896 <span class="w"> </span><span class="c1">// ... etc… |
|
4897 <span class="p">}</span><span class="w"></span> |
|
4898 </code></pre></div> |
|
4899 |
|
4900 <p>Generate a path like <code>$OUT_DIR/attributes-codegen.rs… |
|
4901 with that name, a <code>BufWriter</code> for the file, and s… |
|
4902 to it.</p> |
|
4903 <p>Similarly, the second function:</p> |
|
4904 <div class="highlight"><pre><span></span><cod… |
|
4905 <span class="w"> </span><span class="kd">let</sp… |
|
4906 <span class="w"> </span><span class="kd">let</sp… |
|
4907 |
|
4908 <span class="w"> </span><span class="kd">let</sp… |
|
4909 <span class="w"> </span><span class="kd">let</sp… |
|
4910 |
|
4911 <span class="w"> </span><span class="c1">// ...<… |
|
4912 |
|
4913 <span class="w"> </span><span class="n">print_table… |
|
4914 <span class="w"> </span><span class="n">print_table… |
|
4915 <span class="p">}</span><span class="w"></span> |
|
4916 </code></pre></div> |
|
4917 |
|
4918 <p>Compute two lookup tables, create a file named |
|
4919 <code>$OUT_DIR/srgb-codegen.rs</code>, and write the lookup … |
|
4920 <p>Later in the actual librsvg code, the generated files get inclu… |
|
4921 into the source code using the <code>include!</code> macro. … |
|
4922 where <a href="https://gitlab.gnome.org/GNOME/librsvg/blob/a5c8a9ca/r… |
|
4923 <div class="highlight"><pre><span></span><cod… |
|
4924 |
|
4925 <span class="k">extern</span><span class="w"> </spa… |
|
4926 |
|
4927 <span class="c1">// the generated file has the declaration for enu… |
|
4928 <span class="fm">include!</span><span class="p">(</… |
|
4929 </code></pre></div> |
|
4930 |
|
4931 <p>One thing to note here is that the generated source files |
|
4932 (<code>attributes-codegen.rs</code>, <code>srgb-codege… |
|
4933 directory that Cargo creates for the compilation artifacts. The files &… |
|
4934 not</strong> get put into the original source directories with the… |
|
4935 the library's code; the idea is to keep the source directories |
|
4936 pristine.</p> |
|
4937 <p>At least in those terms, Meson and Cargo agree that source dire… |
|
4938 should be kept clean of autogenerated files.</p> |
|
4939 <p>The <a href="https://doc.rust-lang.org/cargo/reference/build… |
|
4940 <blockquote> |
|
4941 <p>In general, build scripts should not modify any files outside of |
|
4942 OUT_DIR. It may seem fine on the first blush, but it does cause |
|
4943 problems when you use such crate as a dependency, because there's an |
|
4944 implicit invariant that sources in .cargo/registry should be |
|
4945 immutable. cargo won't allow such scripts when packaging.</p> |
|
4946 </blockquote> |
|
4947 <p>Now, some things to note here:</p> |
|
4948 <ul> |
|
4949 <li> |
|
4950 <p>Both the <code>build.rs</code> program and the actu… |
|
4951 the <code>$OUT_DIR</code> environment variable for the loc… |
|
4952 generated sources.</p> |
|
4953 </li> |
|
4954 <li> |
|
4955 <p>The Cargo docs say that if the code generator needs input files… |
|
4956 can look for them based on its current directory, which will be the |
|
4957 toplevel of your source package (i.e. your toplevel <code>Cargo.… |
|
4958 </li> |
|
4959 </ul> |
|
4960 <p><strong>Meson hates this scheme of things</strong>.… |
|
4961 systematic about where it finds input files and sources, and where |
|
4962 things like code generators are allowed to place their output.</p> |
|
4963 <p>The way Meson communicates these paths to code generators is via |
|
4964 command-line arguments to <a href="https://mesonbuild.com/Reference-m… |
|
4965 <a href="https://github.com/mesonbuild/meson/blob/master/test%20cases… |
|
4966 documentation:</p> |
|
4967 <div class="highlight"><pre><span></span><cod… |
|
4968 |
|
4969 <span class="n">outputs</span><span class="w"> </sp… |
|
4970 <span class="w"> </span><span class="k">output</sp… |
|
4971 <span class="w"> </span><span class="n">command</s… |
|
4972 <span class="w"> </span><span class="p">...</span&… |
|
4973 <span class="p">)</span><span class="w"></span> |
|
4974 </code></pre></div> |
|
4975 |
|
4976 <p>This defines a target named <code>'generated'</code>… |
|
4977 <code>generator.py</code> program to output two files, <c… |
|
4978 Python program will get called with <code>@OUTDIR@</code> as… |
|
4979 argument; in effect, meson will call |
|
4980 <code>/full/path/to/generator.py @OUTDIR@</code> explicitly… |
|
4981 passed through environment variables.</p> |
|
4982 <p>If this looks similar to what Cargo does above with <code>… |
|
4983 because it <strong>is</strong> similar. It's just that <… |
|
4984 the concept of generating code at build time (Meson's name for this is |
|
4985 a <strong>custom target</strong>), and provides a mechanism … |
|
4986 the generator, which files it is expected to generate, and how to call |
|
4987 the program with appropriate arguments to put files in the right |
|
4988 place.</p> |
|
4989 <p>In contrast, Cargo assumes that all of that information can be |
|
4990 inferred from an environment variable.</p> |
|
4991 <p>In addition, if the custom target takes other files as input (s… |
|
4992 it can call <code>yacc my-grammar.y</code>), the <code>… |
|
4993 take an <code>input:</code> argument. This way, Meson can a… |
|
4994 those input files, so that the appropriate things will be rebuilt if |
|
4995 the input files change.</p> |
|
4996 <p>Now, Cargo could very well provide a small utility crate that b… |
|
4997 scripts could use to figure out all that information. Meson would |
|
4998 tell Cargo to use its scheme of things, and pass it down to build |
|
4999 scripts via that utility crate. I.e. to have</p> |
|
5000 <div class="highlight"><pre><span></span><cod… |
|
5001 |
|
5002 <span class="k">extern</span><span class="w"> </spa… |
|
5003 |
|
5004 <span class="kd">let</span><span class="w"> </span&… |
|
5005 <span class="c1">// ^^^^^^^^^^^^^^^^^^^^^^^^^^… |
|
5006 |
|
5007 <span class="kd">let</span><span class="w"> </span&… |
|
5008 |
|
5009 <span class="c1">// let the build system know about generated depe… |
|
5010 <span class="n">cargo_high_level</span>::<span class="n"&… |
|
5011 </code></pre></div> |
|
5012 |
|
5013 <p>A similar mechanism could be used for the way Meson likes to pa… |
|
5014 command-line arguments to the programs that deal with custom targets.<… |
|
5015 <h2>Linking to C libraries on the system</h2> |
|
5016 <p>Some Rust crates need to link to lower-level C libraries that a… |
|
5017 do the work. For example, in <a href="https://gtk-rs.org">gtk-rs&… |
|
5018 crates called <code>gtk</code>, <code>gdk</code>… |
|
5019 called <code>gtk-sys</code>, <code>gdk-sys</code>… |
|
5020 just direct wrappers on top of the C functions of the respective |
|
5021 system libraries: <code>gtk-sys</code> makes almost every f… |
|
5022 <code>libgtk-3.so</code> available as a Rust-callable functi… |
|
5023 <p>System libraries sometimes live in a well-known part of the fil… |
|
5024 (<code>/usr/lib64</code>, for example); other times, like in… |
|
5025 they could be anywhere. To find that location plus other related |
|
5026 metadata (include paths for C header files, library version), many |
|
5027 system libraries use <a href="https://www.freedesktop.org/wiki/Softwa… |
|
5028 level, one can run <code>pkg-config</code> on the command li… |
|
5029 scripts, to query some things about libraries. For example:</p> |
|
5030 <div class="highlight"><pre><span></span><cod… |
|
5031 $ <span class="nv">pkg</span><span class="o">-</spa… |
|
5032 <span class="mi">3</span>.<span class="mi">24</span… |
|
5033 |
|
5034 # <span class="nv">what</span> <span class="nv">compil… |
|
5035 $ <span class="nv">pkg</span><span class="o">-</spa… |
|
5036 <span class="o">-</span><span class="nv">pthread</s… |
|
5037 |
|
5038 # <span class="nv">and</span> <span class="nv">which&l… |
|
5039 $ <span class="nv">pkg</span><span class="o">-</spa… |
|
5040 <span class="o">-</span><span class="nv">lgtk</span… |
|
5041 </code></pre></div> |
|
5042 |
|
5043 <p>There is a <a href="https://docs.rs/pkg-config/0.3.14/pkg_co… |
|
5044 use to call this, and communicate that information to Cargo. The |
|
5045 example in the crate's documentation is for asking pkg-config for the |
|
5046 <code>foo</code> package, with version at least <code>… |
|
5047 <div class="highlight"><pre><span></span><cod… |
|
5048 |
|
5049 <span class="k">fn</span> <span class="nf">main</sp… |
|
5050 <span class="w"> </span><span class="n">pkg_config&… |
|
5051 <span class="p">}</span><span class="w"></span> |
|
5052 </code></pre></div> |
|
5053 |
|
5054 <p>And the documentation says,</p> |
|
5055 <blockquote> |
|
5056 <p>After running pkg-config all appropriate Cargo metadata will be |
|
5057 printed on stdout if the search was successful.</p> |
|
5058 </blockquote> |
|
5059 <p>Wait, what?</p> |
|
5060 <p>Indeed, printing specially-formated stuff on stdout is how <… |
|
5061 scripts communicate back to Cargo about their findings. To quote <a … |
|
5062 on build scripts</a>; the following is talking |
|
5063 about the stdout of <code>build.rs</code>:</p> |
|
5064 <blockquote> |
|
5065 <p>Any line that starts with cargo: is interpreted directly by |
|
5066 Cargo. This line must be of the form cargo:key=value, like the |
|
5067 examples below:</p> |
|
5068 </blockquote> |
|
5069 <div class="highlight"><pre><span></span><cod… |
|
5070 <span class="nv">cargo</span>:<span class="nv">rustc&l… |
|
5071 <span class="nv">cargo</span>:<span class="nv">rustc&l… |
|
5072 <span class="nv">cargo</span>:<span class="nv">rustc&l… |
|
5073 <span class="nv">cargo</span>:<span class="nv">rustc&l… |
|
5074 # <span class="nv">arbitrary</span> <span class="nv">u… |
|
5075 <span class="nv">cargo</span>:<span class="nv">root<… |
|
5076 <span class="nv">cargo</span>:<span class="nv">libdir&… |
|
5077 <span class="nv">cargo</span>:<span class="k">include&… |
|
5078 </code></pre></div> |
|
5079 |
|
5080 <p>One can use the stdout of a <code>build.rs</code> p… |
|
5081 command-line options for <code>rustc</code>, or set environm… |
|
5082 or add library paths, or specific libraries.</p> |
|
5083 <p><strong>Meson hates this scheme of things</strong>.… |
|
5084 do the pkg-config calls itself, and then pass that information down to |
|
5085 Cargo, you guessed it, via command-line options or something |
|
5086 well-defined like that. Again, the example <code>cargo_high_level… |
|
5087 proposed above could be used to communicate this information from |
|
5088 Meson to Cargo scripts. Meson also doesn't like this because it would |
|
5089 prefer to know about <code>pkg-config</code>-based libraries… |
|
5090 fashion, without having to run a random script like <code>build.rs… |
|
5091 <h2>Building C code from Rust</h2> |
|
5092 <p>Finally, some Rust crates build a bit of C code and then link t… |
|
5093 into the compiled Rust code. I have no experience with that, but |
|
5094 the respective build scripts generally use the <a href="https://docs.… |
|
5095 call a C compiler and pass options to it conveniently. I suppose |
|
5096 Meson would prefer to do this instead, or at least to have a |
|
5097 high-level way of passing down information to Cargo.</p> |
|
5098 <p>In effect, Meson has to be in charge of picking the C compiler. |
|
5099 Having the thing-to-be-built pick on its own has caused big problems |
|
5100 in the past: GObject-Introspection made the same mistake years ago |
|
5101 when it decided to use distutils to detect the C compiler; gtk-doc did |
|
5102 as well. When those tools are used, we still deal with problems with |
|
5103 cross-compilation and when the system has more than one C compiler in |
|
5104 it.</p> |
|
5105 <h2>Snarky comments about the Unix philosophy</h2> |
|
5106 <p>If part of the Unix philosophy is that shit can be glued togeth… |
|
5107 environment variables and stringly-typed stdout... it's a pretty bad |
|
5108 philosophy. All the cases above boil down to having a well-defined, |
|
5109 more or less strongly-typed way to pass information between programs |
|
5110 instead of shaking proverbial tree of the filesystem and the |
|
5111 environment and seeing if something usable falls down.</p> |
|
5112 <h2>Would we really have to modify all <code>build.rs</co… |
|
5113 <p>Probably. Why not? Meson already has a lot of very well-struc… |
|
5114 knowledge of how to deal with multi-platform compilation and |
|
5115 installation. Re-creating this knowledge in ad-hoc ways in <code>… |
|
5116 is not very pleasant or maintainable.</p> |
|
5117 <h3>Related work</h3> |
|
5118 <ul> |
|
5119 <li> |
|
5120 <p><a href="https://internals.rust-lang.org/t/external-dependen… |
|
5121 thread on using a declarative format to specify external dependencies<… |
|
5122 </li> |
|
5123 <li> |
|
5124 <p><a href="https://github.com/joshtriplett/metadeps">Run pk… |
|
5125 </li> |
|
5126 </ul></content><category term="misc"></category><category term="ru… |
|
5127 <p><img alt="Librsvg authors by lines of code by year" src="htt… |
|
5128 <p>Authors by percentage of lines of code, each year:</p> |
|
5129 <p><img alt="Librsvg authors by percentage of lines of code by … |
|
5130 <p>Which lines of code remain each year?</p> |
|
5131 <p><img alt="Lines of code that remain each year" src="https://… |
|
5132 <p>The shitty thing about a gradual rewrite is that a few people e… |
|
5133 "owning" all the lines of source code. Hopefully this post is a little … |
|
5134 <p><img alt="Librsvg authors by lines of code by year" src="htt… |
|
5135 <p>Authors by percentage of lines of code, each year:</p> |
|
5136 <p><img alt="Librsvg authors by percentage of lines of code by … |
|
5137 <p>Which lines of code remain each year?</p> |
|
5138 <p><img alt="Lines of code that remain each year" src="https://… |
|
5139 <p>The shitty thing about a gradual rewrite is that a few people e… |
|
5140 "owning" all the lines of source code. Hopefully this post is a little |
|
5141 acknowledgment of the people that made librsvg possible.</p> |
|
5142 <p>The charts are made with the incredible tool |
|
5143 <a href="https://github.com/erikbern/git-of-theseus">git-of-theseu… |
|
5144 to <a href="https://mastodon.art/@norwin">@[email protected]<… |
|
5145 up! Its README also points to a |
|
5146 <a href="https://github.com/src-d/hercules">Hercules</a> plo… |
|
5147 graphs. You know, for if you needed something to keep your computer |
|
5148 busy during the weekend.</p></content><category term="misc"></cate… |
|
5149 Rust now</a>. </p> |
|
5150 <p>Today I finished porting the GObject boilerplate for the main |
|
5151 <code>RsvgHandle</code> object into Rust. This means that t… |
|
5152 calls things like <code>g_type_register_static()</code>, nor… |
|
5153 <code>rsvg_handle_class_init()</code> and such; all those ar… |
|
5154 Rust now</a>. </p> |
|
5155 <p>Today I finished porting the GObject boilerplate for the main |
|
5156 <code>RsvgHandle</code> object into Rust. This means that t… |
|
5157 calls things like <code>g_type_register_static()</code>, nor… |
|
5158 <code>rsvg_handle_class_init()</code> and such; all those ar… |
|
5159 is this done?</p> |
|
5160 <h2>The life-changing magic of glib::subclass</h2> |
|
5161 <p><a href="https://coaxion.net/blog/">Sebastian Dröge</… |
|
5162 utilities to make it possible to subclass GObjects in Rust, with |
|
5163 little or no unsafe code. This <a href="https://github.com/gtk-rs/gl… |
|
5164 <a href="https://github.com/gtk-rs/glib">glib-rs</a>, the Ru… |
|
5165 <p>Librsvg now uses the subclassing functionality in glib-rs, whic… |
|
5166 care of some things automatically:</p> |
|
5167 <ul> |
|
5168 <li>Registering your GObject types at runtime.</li> |
|
5169 <li>Creating safe traits on which you can implement <code>cl… |
|
5170 <code>instance_init</code>, <code>set_property</c… |
|
5171 GObject paraphernalia.</li> |
|
5172 </ul> |
|
5173 <p>Check this out:</p> |
|
5174 <div class="highlight"><pre><span></span><cod… |
|
5175 |
|
5176 <span class="k">impl</span><span class="w"> </span&… |
|
5177 <span class="w"> </span><span class="k">const</s… |
|
5178 |
|
5179 <span class="w"> </span><span class="k">type</sp… |
|
5180 |
|
5181 <span class="w"> </span><span class="k">type</sp… |
|
5182 <span class="w"> </span><span class="k">type</sp… |
|
5183 |
|
5184 <span class="w"> </span><span class="n">glib_object… |
|
5185 |
|
5186 <span class="w"> </span><span class="k">fn</span… |
|
5187 <span class="w"> </span><span class="n">klass&l… |
|
5188 <span class="w"> </span><span class="p">}</span&… |
|
5189 |
|
5190 <span class="w"> </span><span class="k">fn</span… |
|
5191 <span class="w"> </span><span class="n">Handle&… |
|
5192 <span class="w"> </span><span class="p">}</span&… |
|
5193 <span class="p">}</span><span class="w"></span> |
|
5194 </code></pre></div> |
|
5195 |
|
5196 <p>In the <code>impl</code> line, <code>Handle&l… |
|
5197 to be <code>RsvgHandlePrivate</code> in the C code.</p> |
|
5198 <p>The following lines say this:</p> |
|
5199 <ul> |
|
5200 <li> |
|
5201 <p><code>const NAME: &amp;'static str = "RsvgHandle";<… |
|
5202 for GType's perusal.</p> |
|
5203 </li> |
|
5204 <li> |
|
5205 <p><code>type ParentType = glib::Object;</code> - Pare… |
|
5206 </li> |
|
5207 <li> |
|
5208 <p><code>type Instance</code>, <code>type Class&… |
|
5209 equivalent to GObject's class and instance structs.</p> |
|
5210 </li> |
|
5211 <li> |
|
5212 <p><code>glib_object_subclass!();</code> - All the boi… |
|
5213 automatically.</p> |
|
5214 </li> |
|
5215 <li> |
|
5216 <p><code>fn class_init</code> - Should be familiar to … |
|
5217 GObjects!</p> |
|
5218 </li> |
|
5219 </ul> |
|
5220 <p>And then, a couple of the property declarations:</p> |
|
5221 <div class="highlight"><pre><span></span><cod… |
|
5222 <span class="w"> </span><span class="n">subclass<… |
|
5223 <span class="w"> </span><span class="n">ParamSp… |
|
5224 <span class="w"> </span><span class="n">nam… |
|
5225 <span class="w"> </span><span class="s">&am… |
|
5226 <span class="w"> </span><span class="s">&am… |
|
5227 <span class="w"> </span><span class="n">Han… |
|
5228 <span class="w"> </span><span class="mi">0&… |
|
5229 <span class="w"> </span><span class="n">Par… |
|
5230 <span class="w"> </span><span class="p">)</s… |
|
5231 <span class="w"> </span><span class="p">}),</spa… |
|
5232 <span class="w"> </span><span class="n">subclass<… |
|
5233 <span class="w"> </span><span class="n">ParamSp… |
|
5234 <span class="w"> </span><span class="n">nam… |
|
5235 <span class="w"> </span><span class="s">&am… |
|
5236 <span class="w"> </span><span class="s">&am… |
|
5237 <span class="w"> </span><span class="mf">0.… |
|
5238 <span class="w"> </span><span class="kt">f6… |
|
5239 <span class="w"> </span><span class="mf">0.… |
|
5240 <span class="w"> </span><span class="n">Par… |
|
5241 <span class="w"> </span><span class="p">)</s… |
|
5242 <span class="w"> </span><span class="p">}),</spa… |
|
5243 <span class="w"> </span><span class="c1">// ... etc… |
|
5244 <span class="p">];</span><span class="w"></span> |
|
5245 </code></pre></div> |
|
5246 |
|
5247 <p>This is quite similar to the way C code usually registers prope… |
|
5248 for new GObject subclasses.</p> |
|
5249 <p>The moment at which a new GObject subclass gets registered agai… |
|
5250 GType system is in the <code>foo_get_type()</code> call. Th… |
|
5251 librsvg for that:</p> |
|
5252 <div class="highlight"><pre><span></span><cod… |
|
5253 |
|
5254 <span class="n">GType</span> |
|
5255 <span class="nf">rsvg_handle_get_type</span> <span class=… |
|
5256 <span class="p">{</span> |
|
5257 <span class="k">return</span> <span class="n">rsvg… |
|
5258 <span class="p">}</span> |
|
5259 </code></pre></div> |
|
5260 |
|
5261 <p>And the Rust function that actually implements this:</p> |
|
5262 <div class="highlight"><pre><span></span><cod… |
|
5263 <span class="k">pub</span><span class="w"> </span&g… |
|
5264 <span class="w"> </span><span class="n">Handle</… |
|
5265 <span class="p">}</span><span class="w"></span> |
|
5266 </code></pre></div> |
|
5267 |
|
5268 <p>Here, <code>Handle::get_type()</code> gets implemen… |
|
5269 Sebastian's <a href="https://github.com/gtk-rs/glib/tree/master/src/s… |
|
5270 the parent class from the <code>impl ObjectSubclass for Handle<… |
|
5271 above, and calls <code>g_type_register_static()</code> inter… |
|
5272 <p>I can confirm now that implementing GObjects in Rust in this wa… |
|
5273 exposing them to C, really works and is actually quite pleasant to |
|
5274 do. <a href="https://gitlab.gnome.org/federico/librsvg/blob/subclass… |
|
5275 <h2>Further work</h2> |
|
5276 <p>There is some auto-generated C code to register librsvg's error… |
|
5277 and a flags type against GType; I'll move those to Rust over the next |
|
5278 few days.</p> |
|
5279 <p>Then, I think I'll try to actually remove all of the library's … |
|
5280 points from the C code and implement them in Rust. Right now each C |
|
5281 function is really just a single call to a Rust function, so this |
|
5282 should be trivial-ish to do.</p> |
|
5283 <p>I'm waiting for a glib-rs release, the first one that will have… |
|
5284 <code>glib::subclass</code> code in it, before merging all o… |
|
5285 librsvg's master branch.</p> |
|
5286 <h2>A new Rust API for librsvg?</h2> |
|
5287 <p>Finally, this got me thinking about what to do about the Rust b… |
|
5288 to librsvg itself. The <a href="https://github.com/selaux/rsvg-rs"&g… |
|
5289 machinery to generate the binding: it reads the <a href="https://peo… |
|
5290 Introspection</a> data from <code>Rsvg.gir</code> and … |
|
5291 for it.</p> |
|
5292 <p>However, the resulting API is mostly identical to the C API. T… |
|
5293 an <code>rsvg::Handle</code> with the same methods as the on… |
|
5294 <code>RsvgHandle</code>... and that API is not particularly … |
|
5295 <p>At some point I had an unfinished branch to <a href="https:/… |
|
5296 librsvg</a>. The intention was that librsvg's build procedure |
|
5297 would first build <code>librsvg.so</code> itself, then gener… |
|
5298 usual, and <strong>then</strong> generate rsvg-rs from that.… |
|
5299 fucking with Autotools, and didn't finish integrating the projects.</… |
|
5300 <p>Rsvg-rs is an <em>okay</em> Rust API for using libr… |
|
5301 perfectly well from the <a href="https://github.com/selaux/rsvg-rs"&g… |
|
5302 that all the functionality of librsvg is in Rust, I would like to take |
|
5303 this opportunity to experiment with a better API for loading and |
|
5304 rendering SVGs from Rust. This may make it more clear how to refactor |
|
5305 the toplevel of the library. Maybe the <code>librsvg</code>… |
|
5306 its own Rust crate for public consumption, in addition to the usual |
|
5307 <code>librsvg.so</code> and <code>Rsvg.gir</code>… |
|
5308 ABI.</p></content><category term="misc"></category><category term=… |
|
5309 Rust code. Paolo Borelli's and Carlos Martín Nieto's latest commits |
|
5310 made it possible.</p> |
|
5311 <p>What does "almost 100% Rust code" mean here?</p> |
|
5312 <ul> |
|
5313 <li> |
|
5314 <p>The C code no longer has struct fields that refer to the librar… |
|
5315 real work. The only field …</p></li></ul></summar… |
|
5316 Rust code. Paolo Borelli's and Carlos Martín Nieto's latest commits |
|
5317 made it possible.</p> |
|
5318 <p>What does "almost 100% Rust code" mean here?</p> |
|
5319 <ul> |
|
5320 <li> |
|
5321 <p>The C code no longer has struct fields that refer to the librar… |
|
5322 real work. The only field in <code>RsvgHandlePrivate</code&g… |
|
5323 pointer to a Rust-side structure. All the rest of the library's |
|
5324 data lives in Rust structs.</p> |
|
5325 </li> |
|
5326 <li> |
|
5327 <p>The public API is implemented in C, but it is just stubs that |
|
5328 immediately call into Rust functions. For example:</p> |
|
5329 </li> |
|
5330 </ul> |
|
5331 <div class="highlight"><pre><span></span><cod… |
|
5332 <span class="nf">rsvg_handle_render_cairo_sub</span> <spa… |
|
5333 <span class="p">{</span> |
|
5334 <span class="n">g_return_val_if_fail</span> <span cla… |
|
5335 <span class="n">g_return_val_if_fail</span> <span cla… |
|
5336 |
|
5337 <span class="k">return</span> <span class="n">rsvg… |
|
5338 <span class="p">}</span> |
|
5339 </code></pre></div> |
|
5340 |
|
5341 <ul> |
|
5342 <li> |
|
5343 <p>The GObject boilerplate and supporting code is still in C: |
|
5344 <code>rsvg_handle_class_init</code> and <code>set_pr… |
|
5345 </li> |
|
5346 <li> |
|
5347 <p>All the high-level tests are still done in C.</p> |
|
5348 </li> |
|
5349 <li> |
|
5350 <p>The gdk-pixbuf loader for SVG files is done in C.</p> |
|
5351 </li> |
|
5352 </ul> |
|
5353 <p>Someone posted a <a href="https://www.reddit.com/r/rust/comm… |
|
5354 comparing lines of code in each language vs. time.</p> |
|
5355 <h2>Rustifying the remaining C code</h2> |
|
5356 <p>There is only a handful of very small functions from the public… |
|
5357 still implemented in C, and I am converting them one by one to Rust. |
|
5358 These are just helper functions built on top of other public API that |
|
5359 does the real work.</p> |
|
5360 <p>Converting the gdk-pixbuf loader to Rust seems like writing a l… |
|
5361 glue code for the loadable module; the actual loading is just a couple |
|
5362 of calls to librsvg's API.</p> |
|
5363 <h3>Rsvg-rs in rsvg?</h3> |
|
5364 <p>Converting the tests to Rust... ideally this would use the <… |
|
5365 bindings; for example, it is what I already use for <a href="https://… |
|
5366 benchmarking program for librsvg.</p> |
|
5367 <p>I have an <a href="https://gitlab.gnome.org/federico/librsvg… |
|
5368 into librsvg's own repository. This is because...</p> |
|
5369 <ol> |
|
5370 <li>Librsvg builds its library, <code>librsvg.so</code>… |
|
5371 <li>Gobject-introspection runs on <code>librsvg.so</code&… |
|
5372 produces <code>librsvg.gir</code></li> |
|
5373 <li>Rsvg-rs's build system calls <a href="https://github.com/gt… |
|
5374 Rust binding's code.</li> |
|
5375 </ol> |
|
5376 <p>As you can imagine, doing all of this with Autotools is... rath… |
|
5377 convoluted. It gives me a lot of anxiety to think that there is also |
|
5378 an <a href="https://gitlab.gnome.org/GNOME/librsvg/commits/wip/meson"… |
|
5379 <em>probably</em> doing the .so→.gir→rs chain would be e… |
|
5380 knows. Help in this area is <strong>much</strong> appreciat… |
|
5381 <h3>An alternative?</h3> |
|
5382 <p>Rustified tests could, of course, call the C API of librsvg by … |
|
5383 in <code>unsafe</code> code. This may not be idiomatic, but… |
|
5384 be done relatively quickly.</p> |
|
5385 <h2>Future work</h2> |
|
5386 <p>There are two options to get rid of all the C code in the libra… |
|
5387 just leave C header files for public consumption:</p> |
|
5388 <ol> |
|
5389 <li> |
|
5390 <p>Do the GObject implementation in Rust, using Sebastian Dröge'… |
|
5391 from GStreamer to do this easily.</p> |
|
5392 </li> |
|
5393 <li> |
|
5394 <p>Work on making <a href="https://gitlab.gnome.org/federico/gn… |
|
5395 API directly, and in an ABI-compatible fashion to what there is |
|
5396 right now.</p> |
|
5397 </li> |
|
5398 </ol> |
|
5399 <p>The second case will probably build upon the first one, since o… |
|
5400 my plans for gnome-class is to make it generate code that uses |
|
5401 Sebastian's, instead of generating all the GObject boilerplate by |
|
5402 hand.</p></content><category term="misc"></category><category term… |
|
5403 that doxed her. Coraline is doing extremely valuable work with the |
|
5404 <a href="https://www.contributor-covenant.org/">Contributor Covena… |
|
5405 projects have <a href="https://www.contributor-covenant.org/adopters"… |
|
5406 <p>Coraline has been working for years in making free software, and |
|
5407 computer technology …</p></summary><content type="html"><p>… |
|
5408 that doxed her. Coraline is doing extremely valuable work with the |
|
5409 <a href="https://www.contributor-covenant.org/">Contributor Covena… |
|
5410 projects have <a href="https://www.contributor-covenant.org/adopters"… |
|
5411 <p>Coraline has been working for years in making free software, and |
|
5412 computer technology circles in general, a welcome place for |
|
5413 underrepresented groups.</p> |
|
5414 <p>I hope Coraline stays safe and strong. You can <a href="htt… |
|
5415 on Patreon</a>.</p></content><category term="misc"></categor… |
|
5416 year's GUADEC. Sorry, here it is!</p> |
|
5417 <p><a href="https://people.gnome.org/~federico/blog/docs/fmq-re… |
|
5418 <p>You can also get the <a href="https://people.gnome.org/~fede… |
|
5419 released under a <a href="https://creativecommons.org/licenses/by-sa/… |
|
5420 <p>This is the <a href="http://videos.guadec.org/2018/GUADEC%20… |
|
5421 <p><strong><em>Update Dec/06:</em></strong>… |
|
5422 year's GUADEC. Sorry, here it is!</p> |
|
5423 <p><a href="https://people.gnome.org/~federico/blog/docs/fmq-re… |
|
5424 <p>You can also get the <a href="https://people.gnome.org/~fede… |
|
5425 released under a <a href="https://creativecommons.org/licenses/by-sa/… |
|
5426 <p>This is the <a href="http://videos.guadec.org/2018/GUADEC%20… |
|
5427 <p><strong><em>Update Dec/06:</em></strong>… |
|
5428 pointers</a>; I've updated the example code in the presentation to |
|
5429 match <a href="https://gitlab.gnome.org/GNOME/librsvg/merge_requests/… |
|
5430 request has an interesting conversation on FFI esoterica, too.</p>… |
|
5431 to Rust, I went into a digression that has to do with the way librsvg |
|
5432 decides which files are allowed to be referenced from within an SVG.<… |
|
5433 <h1>Resource references in SVG</h1> |
|
5434 <p>SVG files can reference other files …</p></summary><con… |
|
5435 to Rust, I went into a digression that has to do with the way librsvg |
|
5436 decides which files are allowed to be referenced from within an SVG.<… |
|
5437 <h1>Resource references in SVG</h1> |
|
5438 <p>SVG files can reference other files, i.e. they are not |
|
5439 self-contained. For example, there can be an element like <code>&… |
|
5440 xlink:href="foo.png"&gt;</code>, or one can request that a sub… |
|
5441 another SVG be included with <code>&lt;use xlink:href="seconda… |
|
5442 Finally, there is the <code>xi:include</code> mechanism to i… |
|
5443 or XML into another XML file.</p> |
|
5444 <p>Since librsvg is sometimes used to render untrusted files that … |
|
5445 the internet, it needs to be careful not to allow those files to |
|
5446 reference any random resource on the filesystem. We don't want |
|
5447 something like |
|
5448 <code>&lt;text&gt;&lt;xi:include href="/etc/passwd" pa… |
|
5449 or something equally nefarious that would exfiltrate a random file |
|
5450 into the rendered output.</p> |
|
5451 <p>Also, want to catch malicious SVGs that want to "phone home" by |
|
5452 referencing a network resource like |
|
5453 <code>&lt;image xlink:href="http://evil.com/pingback.jpg"&… |
|
5454 <p>So, librsvg is careful to have a single place where it can load |
|
5455 secondary resources, and first it validates the resource's URL to see |
|
5456 if it is allowed.</p> |
|
5457 <p>The actual validation rules are not very important for this |
|
5458 discussion; they are something like "no absolute URLs allowed" (so you |
|
5459 can't request <code>/etc/passwd</code>, "only siblings or (g… |
|
5460 siblings allowed" (so <code>foo.svg</code> can request <c… |
|
5461 <code>subdir/bar.svg</code>, but not <code>../../bar.s… |
|
5462 <h1>The code</h1> |
|
5463 <p>There was a central function <code>rsvg_io_acquire_stream… |
|
5464 URL as a string. The code assumed that that URL had been first |
|
5465 validated with a function called <code>allow_load(url)</code>… |
|
5466 structure guaranteed that all the places that may acquire a stream |
|
5467 would actually go through <code>allow_load()</code> first, t… |
|
5468 code in Rust made it possible to actually make it impossible to |
|
5469 acquire a disallowed URL.</p> |
|
5470 <p>Before:</p> |
|
5471 <div class="highlight"><pre><span></span><cod… |
|
5472 |
|
5473 <span class="k">pub</span><span class="w"> </span&g… |
|
5474 |
|
5475 <span class="k">pub</span><span class="w"> </span&g… |
|
5476 <span class="w"> </span><span class="k">if</span… |
|
5477 <span class="w"> </span><span class="n">acquire… |
|
5478 <span class="w"> </span><span class="p">}</span&… |
|
5479 <span class="w"> </span><span class="nb">Err<… |
|
5480 <span class="w"> </span><span class="p">}</span&… |
|
5481 <span class="p">}</span><span class="w"></span> |
|
5482 </code></pre></div> |
|
5483 |
|
5484 <p>The refactored code now has an <code>AllowedUrl</code&… |
|
5485 URL, plus the promise that it <strong>has</strong> gone thro… |
|
5486 <ul> |
|
5487 <li>The URL has been run through a URL well-formedness parser.<… |
|
5488 <li>The resource is allowed to be loaded following librsvg's rules… |
|
5489 </ul> |
|
5490 <div class="highlight"><pre><span></span><cod… |
|
5491 |
|
5492 <span class="k">impl</span><span class="w"> </span&… |
|
5493 <span class="w"> </span><span class="k">pub</spa… |
|
5494 <span class="w"> </span><span class="kd">let<… |
|
5495 |
|
5496 <span class="w"> </span><span class="k">if</… |
|
5497 <span class="w"> </span><span class="nb">Ok… |
|
5498 <span class="w"> </span><span class="p">}</s… |
|
5499 <span class="w"> </span><span class="nb">Er… |
|
5500 <span class="w"> </span><span class="p">}</s… |
|
5501 <span class="w"> </span><span class="p">}</span&… |
|
5502 <span class="p">}</span><span class="w"></span> |
|
5503 |
|
5504 <span class="c1">// new prototype</span> |
|
5505 <span class="k">pub</span><span class="w"> </span&g… |
|
5506 </code></pre></div> |
|
5507 |
|
5508 <p>This forces callers to validate the URLs as soon as possible, r… |
|
5509 after they get them from the SVG file. Now it is not possible to |
|
5510 request a stream unless the URL has been validated first.</p> |
|
5511 <h1>Plain URIs vs. fragment identifiers</h1> |
|
5512 <p>Some of the elements in SVG that reference other data require f… |
|
5513 files:</p> |
|
5514 <div class="highlight"><pre><span></span><cod… |
|
5515 </code></pre></div> |
|
5516 |
|
5517 <p>And some others, that reference particular elements in secondar… |
|
5518 require a fragment ID:</p> |
|
5519 <div class="highlight"><pre><span></span><cod… |
|
5520 </code></pre></div> |
|
5521 |
|
5522 <p>And finally, the <code>feImage</code> element, used… |
|
5523 a filter effects pipeline, allows either:</p> |
|
5524 <div class="highlight"><pre><span></span><cod… |
|
5525 <span class="o">&lt;</span><span class="n">feImage… |
|
5526 |
|
5527 <span class="o">&lt;!--</span> <span class="n">wil… |
|
5528 <span class="o">&lt;</span><span class="n">feImage… |
|
5529 </code></pre></div> |
|
5530 |
|
5531 <p>So, I introduced a general <code>Href</code> parser… |
|
5532 <div class="highlight"><pre><span></span><cod… |
|
5533 <span class="w"> </span><span class="n">PlainUri<… |
|
5534 <span class="w"> </span><span class="n">WithFragmen… |
|
5535 <span class="p">}</span><span class="w"></span> |
|
5536 |
|
5537 <span class="sd">/// Optional URI, mandatory fragment id</span&… |
|
5538 <span class="k">pub</span><span class="w"> </span&g… |
|
5539 </code></pre></div> |
|
5540 |
|
5541 <p>The parts of the code that absolutely require a fragment id now… |
|
5542 <code>Fragment</code>. Parts which require a <code>Pl… |
|
5543 <p>The next step is making those structs contain an <code>Al… |
|
5544 directly, instead of just strings, so that for callers, obtaining a |
|
5545 fully validated name is a one-step operation.</p> |
|
5546 <p>In general, the code is moving towards a scheme where all file … |
|
5547 done at loading time. Right now, some of those external references |
|
5548 get resolved at rendering time, which is somewhat awkward (for |
|
5549 example, at rendering time the caller has no chance to use a |
|
5550 <code>GCancellable</code> to cancel loading). This refactor… |
|
5551 validation is leaving the code in a very nice state.</p></content>… |
|
5552 in Thessaloniki, Greece. This is the beautiful city that will host |
|
5553 next year's GUADEC, but fortunately GUADEC will be in summertime!</p&… |
|
5554 <p>We held the hackfest at the <a href="http://coho.gr/">CoH… |
|
5555 office between the …</p></summary><content type="html"><p>… |
|
5556 in Thessaloniki, Greece. This is the beautiful city that will host |
|
5557 next year's GUADEC, but fortunately GUADEC will be in summertime!</p&… |
|
5558 <p>We held the hackfest at the <a href="http://coho.gr/">CoH… |
|
5559 office between the University and the sea.</p> |
|
5560 <p>Every such hackfest I am overwhelmed by the kind hackers who wo… |
|
5561 [gnome-class], the code generator for GObject implementations in |
|
5562 Rust.</p> |
|
5563 <p>Mredlek has been working on generalizing the code generators in |
|
5564 gnome-class, so that we can have the following from the same run:</p&… |
|
5565 <ul> |
|
5566 <li> |
|
5567 <p>Rust code generation, for the GObject implementations themselve… |
|
5568 Thanks to mredlek, this is much cleaner than it was before; now both |
|
5569 classes and interfaces share the same code for most of the |
|
5570 boilerplate.</p> |
|
5571 </li> |
|
5572 <li> |
|
5573 <p>GObject Introspection (<code>.gir</code>) generatio… |
|
5574 can be generated automatically.</p> |
|
5575 </li> |
|
5576 <li> |
|
5577 <p>C header files (<code>.h</code>), so the generated … |
|
5578 C code as usual.</p> |
|
5579 </li> |
|
5580 </ul> |
|
5581 <p>So far, Rust and GIR work; C header files are not generated yet… |
|
5582 <p>Mredlek is a new contributor to gnome-class, but unfortunately … |
|
5583 able to attend the hackfest. Not only did he rewrite the gnome-class |
|
5584 parser using the new version of <a href="https://docs.rs/syn/0.15.22/… |
|
5585 passing owned types to GObject methods, such as <code>String</c… |
|
5586 <code>Variant</code>. But the biggest thing is probably tha… |
|
5587 lot easier to debug the generated Rust source; see <a href="https://f… |
|
5588 on debugging</a> for details.</p> |
|
5589 <p>Speaking of which, thanks to Jordan Petridis for making the |
|
5590 documentation be published automatically from Gitlab's Continuous |
|
5591 Integration pipelines.</p> |
|
5592 <p>Alex Crichton kindly refactored our error propagation code, and… |
|
5593 wrote docs on it</a>! Along with Jordan, they updated the |
|
5594 code for the Rust 2018 edition, and generally wrangled the build |
|
5595 process to conform with the lastest Rust nightlies. Alex also made |
|
5596 code generation a lot faster, by offloading auto-indentation to an |
|
5597 external <code>rustfmt</code> process, instead of using it a… |
|
5598 <code>rustfmt</code> crate meant that the compiler had a lot… |
|
5599 During the whole hackfest, Alex was very helpful with Rust questions |
|
5600 in general. While my strategy to see what the compiler does is to |
|
5601 examine the disassembly in gdb, his strategy seems to be to look at |
|
5602 the LLVM intermediate representation instead... OMG.</p> |
|
5603 <h1>And we can derive very simple GtkWidgets now!</h1> |
|
5604 <p>Saving the best for last... Antoni Boucher, the author of <a… |
|
5605 been working on making it possible to derive from <code>gtk::Widge… |
|
5606 <a href="https://gitlab.gnome.org/federico/gnome-class/merge_requests… |
|
5607 deriving from <code>gtk::DrawingArea</code> from Rust with v… |
|
5608 <p>Normally, the <a href="https://gtk-rs.org/">gtk-rs</a&… |
|
5609 for GObject, which really is a type hierarchy defined at runtime. The |
|
5610 static binding really wants to know what is a subclass of what: it |
|
5611 needs to know in advance that <code>Button</code>'s hierarch… |
|
5612 Container → Widget → Object</code>, plus all the <code>G… |
|
5613 by any of those classes. Antoni has been working on making |
|
5614 gnome-class extract that information automatically from GIR files, so |
|
5615 that the gtk-rs macros that define new types will get all the |
|
5616 necessary information.</p> |
|
5617 <h1>Future work</h1> |
|
5618 <p>There are still <a href="https://gitlab.gnome.org/GNOME/gtk/… |
|
5619 from deriving, say, from <code>gtk::Container</code>, but ho… |
|
5620 resolved soon.</p> |
|
5621 <p>Sebastian Dröge has been refactoring his Rust tools to create … |
|
5622 subclasses with very idiomatic and refined Rust code. This is now at |
|
5623 a state where gnome-class itself could generate that sort of code, |
|
5624 instead of generating all the boilerplate from scratch. So, we'll |
|
5625 start doing that, and integrating the necessary bits into gtk-rs as |
|
5626 well.</p> |
|
5627 <p>Finally, during the last day I took a little break from gnome-c… |
|
5628 work on librsvg. Julian Sparber has been updating the code to use new |
|
5629 bindings in cairo-rs, and is also adding a <a href="https://gitlab.gn… |
|
5630 fetch an SVG element's geometry precisely.</p> |
|
5631 <h1>Thessaloniki</h1> |
|
5632 <p>Oh, boy, I wish the weather had been warmer. The city looks |
|
5633 delightful to walk around, especially in the narrow streets on the |
|
5634 hills. Can't wait to see it in summer during GUADEC.</p> |
|
5635 <h1>Thanks</h1> |
|
5636 <p>Finally, thanks to <a href="http://coho.gr/">CoHo</a&g… |
|
5637 Foundation for sponsoring my travel and accomodation. And to |
|
5638 <a href="https://www.centricular.com/">Centricular</a> for t… |
|
5639 <p>Special thanks to Jordan Petridis for being on top of everything |
|
5640 build-wise all the time.</p> |
|
5641 <p><img alt="Sponsored by the GNOME Foundation" src="https://pe… |
|
5642 from C to Rust. For many technical reasons, the library still uses |
|
5643 libxml2, GNOME's historic XML parsing library, but some of the |
|
5644 callbacks to handle XML events like <code>start_element</code&g… |
|
5645 <code>characters</code>, are now implemented in Rust. This … |
|
5646 from C to Rust. For many technical reasons, the library still uses |
|
5647 libxml2, GNOME's historic XML parsing library, but some of the |
|
5648 callbacks to handle XML events like <code>start_element</code&g… |
|
5649 <code>characters</code>, are now implemented in Rust. This … |
|
5650 running into all the cases where the original C code in librsvg failed |
|
5651 to handle errors properly; Rust really makes it obvious when that |
|
5652 happens.</p> |
|
5653 <p>In this post I want to talk a bit about propagating errors. Yo… |
|
5654 a function, it returns an error, and then what?</p> |
|
5655 <h2>What can fail?</h2> |
|
5656 <p>It turns out that this question is highly context-dependent. L… |
|
5657 say a program is starting up and tries to read a configuration file. |
|
5658 What could go wrong?</p> |
|
5659 <ul> |
|
5660 <li> |
|
5661 <p>The file doesn't exist. Maybe it is the very first time the pr… |
|
5662 is run, and so there <em>isn't</em> a configuration file a… |
|
5663 program provide a default configuration in this case? Or does it |
|
5664 absolutely need a pre-written configuration file to be somewhere?</… |
|
5665 </li> |
|
5666 <li> |
|
5667 <p>The file can't be parsed. Should the program warn the user and |
|
5668 exit, or should it revert to a default configuration (should it |
|
5669 overwrite the file with valid, default values)? <em>Can</em&… |
|
5670 the program warn the user, or is it a user-less program that at best |
|
5671 can just shout into the void of a server-side log file?</p> |
|
5672 </li> |
|
5673 <li> |
|
5674 <p>The file can be parsed, but the values are invalid. Same quest… |
|
5675 as the case above.</p> |
|
5676 </li> |
|
5677 <li> |
|
5678 <p>Etcetera.</p> |
|
5679 </li> |
|
5680 </ul> |
|
5681 <p>At each stage, the code will probably see very low-level errors… |
|
5682 not found", "I/O error", "parsing failed", "value is out of range"). |
|
5683 What the code decides to do, or what it is able to do at any |
|
5684 particular stage, depends both on the semantics you want from the |
|
5685 program, and from the code structure itself.</p> |
|
5686 <h2>Structuring the problem</h2> |
|
5687 <p>This is an easy, but very coarse way of handling things:</p&… |
|
5688 <div class="highlight"><pre><span></span><cod… |
|
5689 <span class="nf">read_configuration</span> <span class="p… |
|
5690 <span class="p">{</span> |
|
5691 <span class="cm">/* open the file */</span> |
|
5692 |
|
5693 <span class="cm">/* parse it */</span> |
|
5694 |
|
5695 <span class="cm">/* set global variables to the configuration … |
|
5696 |
|
5697 <span class="cm">/* return true if success, or false if failur… |
|
5698 <span class="p">}</span> |
|
5699 </code></pre></div> |
|
5700 |
|
5701 <p>What is bad about this? Let's see:</p> |
|
5702 <ul> |
|
5703 <li> |
|
5704 <p>The calling code just gets a success/failure condition. In the… |
|
5705 of failure, it doesn't get to know why things failed.</p> |
|
5706 </li> |
|
5707 <li> |
|
5708 <p>If the function sets global variables with configuration values… |
|
5709 they get read... and something goes wrong and the function returns |
|
5710 an error... the caller ends up possibly in an inconsistent state, |
|
5711 with a set of configuration variables that are only halfway-set.</p… |
|
5712 </li> |
|
5713 <li> |
|
5714 <p>If the function finds parse errors, well, do you really want to… |
|
5715 UI code from inside it? The caller might be a better place to make |
|
5716 that decision.</p> |
|
5717 </li> |
|
5718 </ul> |
|
5719 <h2>A slightly better structure</h2> |
|
5720 <p>Let's add an enumeration to indicate the possible errors, and a |
|
5721 structure of configuration values.</p> |
|
5722 <div class="highlight"><pre><span></span><cod… |
|
5723 <span class="w"> </span><span class="n">ConfigFileD… |
|
5724 <span class="w"> </span><span class="n">ParseError&… |
|
5725 <span class="w"> </span><span class="n">ValueError&… |
|
5726 <span class="p">}</span><span class="w"></span> |
|
5727 |
|
5728 <span class="k">struct</span> <span class="nc">ConfigV… |
|
5729 <span class="w"> </span><span class="c1">// a bunch… |
|
5730 <span class="p">}</span><span class="w"></span> |
|
5731 |
|
5732 <span class="k">fn</span> <span class="nf">read_config… |
|
5733 <span class="w"> </span><span class="c1">// open th… |
|
5734 |
|
5735 <span class="w"> </span><span class="c1">// parse t… |
|
5736 |
|
5737 <span class="w"> </span><span class="c1">// validat… |
|
5738 |
|
5739 <span class="w"> </span><span class="c1">// if ever… |
|
5740 <span class="p">}</span><span class="w"></span> |
|
5741 </code></pre></div> |
|
5742 |
|
5743 <p>This is better, in that the caller decides what to do with the |
|
5744 validated <code>ConfigValues</code>: maybe it can just copy… |
|
5745 program's global variables for configuration.</p> |
|
5746 <p>However, this scheme doesn't give the caller all the informatio… |
|
5747 would like to present a really good error message. For example, the |
|
5748 caller will get to know if there is a parse error, but it doesn't know |
|
5749 specifically what failed during parsing. Similarly, it will just get |
|
5750 to know if there was an invalid value, but not which one.</p> |
|
5751 <h2>Ah, so the problem is fractal</h2> |
|
5752 <p>We could have new structs to represent the little errors, and t… |
|
5753 make them part of the original error enum:</p> |
|
5754 <div class="highlight"><pre><span></span><cod… |
|
5755 <span class="w"> </span><span class="n">line</sp… |
|
5756 <span class="w"> </span><span class="n">column</… |
|
5757 <span class="w"> </span><span class="n">error_reaso… |
|
5758 <span class="p">}</span><span class="w"></span> |
|
5759 |
|
5760 <span class="k">struct</span> <span class="nc">ValueEr… |
|
5761 <span class="w"> </span><span class="n">config_key&… |
|
5762 <span class="w"> </span><span class="n">error_reaso… |
|
5763 <span class="p">}</span><span class="w"></span> |
|
5764 |
|
5765 <span class="k">enum</span> <span class="nc">ConfigErr… |
|
5766 <span class="w"> </span><span class="n">ConfigFileD… |
|
5767 <span class="w"> </span><span class="n">ParseError&… |
|
5768 <span class="w"> </span><span class="n">ValueError&… |
|
5769 <span class="p">}</span><span class="w"></span> |
|
5770 </code></pre></div> |
|
5771 |
|
5772 <p>Is that enough? It depends.</p> |
|
5773 <p>The <code>ParseError</code> and <code>ValueEr… |
|
5774 <code>error_reason</code> fields, which are strings. Presum… |
|
5775 a <code>ParseError</code> with <code>error_reason = "u… |
|
5776 <code>ValueError</code> with <code>error_reason = "can… |
|
5777 <p>One problem with this is that if the low-level errors come with… |
|
5778 messages in English, then the caller has to know how to localize them |
|
5779 to the user's language. Also, if they don't have a machine-readable |
|
5780 error code, then the calling code may not have enough information to |
|
5781 decide what do do with the error.</p> |
|
5782 <p>Let's say we had a <code>ParseErrorKind</code> enum… |
|
5783 <code>UnexpectedToken</code>, <code>EndOfFile</code… |
|
5784 calling code know the <em>reason</em> for the error. Also, … |
|
5785 <code>gimme_localized_error_message()</code> method for that… |
|
5786 error.</p> |
|
5787 <div class="highlight"><pre><span></span><cod… |
|
5788 <span class="w"> </span><span class="n">UnexpectedT… |
|
5789 <span class="w"> </span><span class="n">EndOfFile&l… |
|
5790 <span class="w"> </span><span class="n">MissingComm… |
|
5791 <span class="w"> </span><span class="c1">// ... etc… |
|
5792 <span class="p">}</span><span class="w"></span> |
|
5793 |
|
5794 <span class="k">struct</span> <span class="nc">ParseEr… |
|
5795 <span class="w"> </span><span class="n">line</sp… |
|
5796 <span class="w"> </span><span class="n">column</… |
|
5797 <span class="w"> </span><span class="n">kind</sp… |
|
5798 <span class="p">}</span><span class="w"></span> |
|
5799 </code></pre></div> |
|
5800 |
|
5801 <p>How can we expand this? Maybe the <code>ParseErrorKind::… |
|
5802 variant wants to contain data that indicates <em>which</em> … |
|
5803 was wrong, so it would be <code>UnexpectedToken(String)</code&g… |
|
5804 similar.</p> |
|
5805 <p>But is <em>that</em> useful to the calling code? F… |
|
5806 which is reading a configuration file... it probably only needs to |
|
5807 know if it could parse the file, but maybe it doesn't really need any |
|
5808 additional details on the reason for the parse error, other than |
|
5809 having something useful to present to the user. Whether it is |
|
5810 appropriate to burden the user with the actual details... does the app |
|
5811 expect to make it the user's job to fix broken configuration files? |
|
5812 Yes for a web server, where the user is a sysadmin; probably not for a |
|
5813 random end-user graphical app, where people shouldn't need to write |
|
5814 configuration files by hand in the first place (should <em>those&l… |
|
5815 "Details" section in the error message window? I don't know!).</p> |
|
5816 <p>Maybe the low-level parsing/validation code <em>can</e… |
|
5817 errors. But how can we propagate them to something more useful to the |
|
5818 upper layers of the code?</p> |
|
5819 <h2>Translation and propagation</h2> |
|
5820 <p>Maybe our original <code>read_configuration()</code>… |
|
5821 low-level errors into high-level ones:</p> |
|
5822 <div class="highlight"><pre><span></span><cod… |
|
5823 <span class="w"> </span><span class="c1">// open fi… |
|
5824 |
|
5825 <span class="w"> </span><span class="k">if</span… |
|
5826 <span class="w"> </span><span class="k">return&… |
|
5827 <span class="w"> </span><span class="p">}</span&… |
|
5828 |
|
5829 <span class="w"> </span><span class="kd">let</sp… |
|
5830 |
|
5831 <span class="w"> </span><span class="c1">// parse f… |
|
5832 |
|
5833 <span class="w"> </span><span class="kd">let</sp… |
|
5834 |
|
5835 <span class="w"> </span><span class="c1">// validat… |
|
5836 |
|
5837 <span class="w"> </span><span class="kd">let</sp… |
|
5838 |
|
5839 <span class="w"> </span><span class="c1">// yay!<… |
|
5840 <span class="w"> </span><span class="nb">Ok</spa… |
|
5841 <span class="p">}</span><span class="w"></span> |
|
5842 </code></pre></div> |
|
5843 |
|
5844 <p>Etcetera. It is up to each part of the code to decide what do … |
|
5845 lower-level errors. Can it recover from them? Should it fail the |
|
5846 whole operation and return a higher-level error? Should it warn the |
|
5847 user right there?</p> |
|
5848 <h2>Language facilities</h2> |
|
5849 <p>C makes it really easy to ignore errors, and pretty hard to pre… |
|
5850 detailed errors like the above. One could mimic what Rust is actually |
|
5851 doing with a collection of <code>union</code> and <code&g… |
|
5852 gets very awkward very fast.</p> |
|
5853 <p>Rust provides these facilities at the language level, and the i… |
|
5854 around <code>Result</code> and error handling are very nice … |
|
5855 even crates like <a href="https://boats.gitlab.io/failure/intro.html"… |
|
5856 automating error translation, propagation, and conversion to strings |
|
5857 for presenting to users.</p> |
|
5858 <h2>Infinite details</h2> |
|
5859 <p>I've been recommending <a href="http://joeduffyblog.com/2016… |
|
5860 comes into a discussion of error handling in programming languages. |
|
5861 It's a long, detailed, but very enlightening read on recoverable |
|
5862 vs. unrecoverable errors, simple error codes vs. exceptions |
|
5863 vs. monadic results, the performance/reliability/ease of use of each |
|
5864 model... Definitely worth a read.</p></content><category term="mis… |
|
5865 gdk-pixbuf's history. There is some talk about replacing it with |
|
5866 something newer; hopefully this history will show some things that |
|
5867 worked, some that didn't, and why.</p> |
|
5868 <h2>The beginnings</h2> |
|
5869 <p>Gdk-pixbuf started as a replacement for Imlib, the image …<… |
|
5870 gdk-pixbuf's history. There is some talk about replacing it with |
|
5871 something newer; hopefully this history will show some things that |
|
5872 worked, some that didn't, and why.</p> |
|
5873 <h2>The beginnings</h2> |
|
5874 <p>Gdk-pixbuf started as a replacement for Imlib, the image loadin… |
|
5875 rendering library that GNOME used in its earliest versions. Imlib |
|
5876 came from the Enlightenment project; it provided an easy API around |
|
5877 the idiosyncratic libungif, libjpeg, libpng, etc., and it maintained |
|
5878 decoded images in memory with a uniform representation. Imlib also |
|
5879 worked as an image cache for the Enlightenment window manager, which |
|
5880 made memory management very inconvenient for GNOME.</p> |
|
5881 <p>Imlib worked well as a "just load me an image" library. It sho… |
|
5882 that a small, uniform API to load various image formats into a common |
|
5883 representation was desirable. And in those days, hiding all the |
|
5884 complexities of displaying images in X was very important indeed.</p&… |
|
5885 <h2>The initial API</h2> |
|
5886 <p>Gdk-pixbuf replaced Imlib, and added two important features: |
|
5887 reference counting for image data, and support for an alpha channel.<… |
|
5888 <p>Gdk-pixbuf appeared with support for RGB(A) images. And althou… |
|
5889 theory it was possible to grow the API to support other |
|
5890 representations, <a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/b… |
|
5891 <code>GDK_COLORSPACE_RGB</code>, and the <a href="https:/… |
|
5892 functions <a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/blob/369… |
|
5893 channel was done with a <code>gboolean</code> argument in co… |
|
5894 single <code>GDK_COLORSPACE_RGB</code> value; we didn't have… |
|
5895 <a href="https://gitlab.freedesktop.org/cairo/cairo/blob/201791a5/src… |
|
5896 enum values.</p> |
|
5897 <p>While all the code in gdk-pixbuf carefully checks that those |
|
5898 conditions are met — RGBA at 8 bits per channel —, some applications |
|
5899 inadvertently assume that <em>that</em> is the only possible… |
|
5900 into trouble really fast if gdk-pixbuf ever started returning pixbufs |
|
5901 with different color spaces or depths.</p> |
|
5902 <p>One can still see the battle between bilevel-alpha |
|
5903 vs. continuous-alpha in <a href="https://gitlab.gnome.org/GNOME/gdk-p… |
|
5904 <div class="highlight"><pre><span></span><cod… |
|
5905 <span class="p">{</span> |
|
5906 <span class="n">GDK_PIXBUF_ALPHA_BILEVEL</span><s… |
|
5907 <span class="n">GDK_PIXBUF_ALPHA_FULL</span> |
|
5908 <span class="p">}</span> <span class="n">GdkPixbufAlph… |
|
5909 </code></pre></div> |
|
5910 |
|
5911 <p>Fortunately, only the "<a href="https://gitlab.gnome.org/GNO… |
|
5912 drawable</a>" functions take values of this type: before the Xre… |
|
5913 days, it was a Big Deal to draw an image with alpha to an X window, |
|
5914 and applications often opted to use a bitmask instead, even if they |
|
5915 had jagged edges as a result.</p> |
|
5916 <h2>Pixel formats</h2> |
|
5917 <p>The only pixel format that ever got implemented was unpremultip… |
|
5918 RGBA on all platforms. Back then I didn't understand <a href="https:… |
|
5919 alpha</a>! Also, the GIMP followed that scheme, and copying |
|
5920 it seemed like the easiest thing.</p> |
|
5921 <p>After gdk-pixbuf, libart also copied that pixel format, I think… |
|
5922 <p>But later we got Cairo, Pixman, and all the Xrender stack. The… |
|
5923 prefer premultiplied ARGB. Moreover, Cairo prefers it if each pixel |
|
5924 is actually a 32-bit value, with the ARGB values inside it in |
|
5925 platform-endian order. So if you look at a memory dump, a Cairo pixel |
|
5926 looks like BGRA on a little-endian box, while it looks like ARGB on a |
|
5927 big-endian box.</p> |
|
5928 <p>Every time we paint a <code>GdkPixbuf</code> to a &… |
|
5929 conversion from unpremultiplied RGBA to premultiplied, platform-endian |
|
5930 ARGB. I talked a bit about this in <a href="https://people.gnome.org… |
|
5931 copies in GNOME</a>.</p> |
|
5932 <h2>The loading API</h2> |
|
5933 <p>The public loading API in gdk-pixbuf, and its relationship to l… |
|
5934 plug-ins, evolved in interesting ways.</p> |
|
5935 <p>At first the public API and loaders only implemented <code&g… |
|
5936 you gave the library a <code>FILE *</code> and it gave you b… |
|
5937 Back then we didn't have a robust MIME sniffing framework in the form |
|
5938 of a library, so gdk-pixbuf got its own. This lives in the |
|
5939 mostly-obsolete <a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/bl… |
|
5940 even has its own <a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/b… |
|
5941 Nowadays we do most MIME sniffing with GIO.</p> |
|
5942 <p>After the intial <code>load_from_file</code> API...… |
|
5943 loading first, and animation support aftewards.</p> |
|
5944 <h2>Progressive loading</h2> |
|
5945 <p>This where the calling program feeds chunks of bytes to the lib… |
|
5946 and at the end a fully-formed <code>GdkPixbuf</code> comes o… |
|
5947 a single "read a whole file" operation.</p> |
|
5948 <p>We conflated this with a way to get <a href="https://gitlab.… |
|
5949 modified</a> as the data gets parsed. I think we wanted to suppor… |
|
5950 case of a web browser, which downloads images slowly over the network, |
|
5951 and gradually displays them as they are downloaded. In 1998, images |
|
5952 downloading slowly over the network was a real concern!</p> |
|
5953 <p>It took a lot of very careful work to convert the image loaders… |
|
5954 parsed a whole file at a time, into loaders that could maintain some |
|
5955 state between each time that they got handed an extra bit of buffer.<… |
|
5956 <p>It also sounded easy to implement the progressive updating API … |
|
5957 simply emitting a signal that said, "this rectangular area got updated |
|
5958 from the last read". It could handle the case of reading whole |
|
5959 scanlines, or a few pixels, or even area-based updates for progressive |
|
5960 JPEGs and PNGs.</p> |
|
5961 <p>The internal API for the image format loaders still keeps a |
|
5962 distinction between the "load a whole file" API and the "load an image |
|
5963 in chunks". Not all loaders got redone to simply just use the second |
|
5964 one: <code>io-jpeg.c</code> <a href="https://gitlab.gnom… |
|
5965 corresponding libjpeg functions. I think it could remove that code |
|
5966 and use the progressive loading functions instead.</p> |
|
5967 <h2>Animations</h2> |
|
5968 <p>Animations: we followed the GIF model for animations, in which… |
|
5969 frame overlays the previous one, and there's a delay set between each |
|
5970 frame. This is not a video file; it's a hacky flipbook.</p> |
|
5971 <p>However, animations presented the problem that the whole gdk-pi… |
|
5972 API was meant for static images, and now we needed to support |
|
5973 multi-frame images as well.</p> |
|
5974 <p>We defined the "correct" way to use the gdk-pixbuf library as to |
|
5975 actually try to load an animation, and then see if it is a |
|
5976 single-frame image, in which case you can just get a <code>GdkPixb… |
|
5977 the only frame and use it.</p> |
|
5978 <p>Or, if you got an animation, that would be a <a href="https:… |
|
5979 object, from which you could ask for an iterator to get each frame as |
|
5980 a separate <code>GdkPixbuf</code>.</p> |
|
5981 <p>However, the progressive updating API never got extended to rea… |
|
5982 support animations. So, we have awkward functions like |
|
5983 <code>gdk_pixbuf_animation_iter_on_currently_loading_frame()</c… |
|
5984 <h2>Necessary accretion</h2> |
|
5985 <p>Gdk-pixbuf got support for saving just a few formats: JPEG, PN… |
|
5986 TIFF, ICO, and some of the formats that are implemented with the |
|
5987 Windows-native loaders.</p> |
|
5988 <p>Over time gdk-pixbuf got support for preserving some metadata-i… |
|
5989 chunks from formats that provide it: DPI, color profiles, image |
|
5990 comments, hotspots for cursors/icons...</p> |
|
5991 <p>While an image is being loaded with the progressive loaders, th… |
|
5992 a clunky way to specify that one doesn't want the actual size of the |
|
5993 image, but another size instead. The loader can handle that situation |
|
5994 itself, hopefully if an image format actually embeds different sizes |
|
5995 in it. Or if not, the main loading code will rescale the full loaded |
|
5996 image into the size specified by the application.</p> |
|
5997 <h2>Historical cruft</h2> |
|
5998 <p><a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/blob/3693… |
|
5999 funky encoding. Nowadays it's just easier to directly store a PNG or |
|
6000 JPEG or whatever in a <code>GResource</code>.</p> |
|
6001 <p><a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/blob/3693… |
|
6002 Hopefully mostly unused now, but there's a good number of mostly old, |
|
6003 third-party software that still uses gdk-pixbuf as an image loader and |
|
6004 renderer to X drawables.</p> |
|
6005 <p><a href="https://gitlab.gnome.org/GNOME/gdk-pixbuf/blob/3693… |
|
6006 scaling functions, which the original versions of EOG used for the |
|
6007 core of the image viewer. Nowadays Cairo is the preferred way of |
|
6008 doing this, since it not only does scaling, but general affine |
|
6009 transformations as well. Did you know that |
|
6010 <code>gdk_pixbuf_composite_color</code> takes 17 arguments,… |
|
6011 an image with alpha on top of a checkerboard? Yes, that used to be |
|
6012 the core of EOG.</p> |
|
6013 <h2>Debatable historical cruft</h2> |
|
6014 <p><code>gdk_pixbuf_get_pixels()</code>. This lets th… |
|
6015 pixels of a loaded pixbuf, and modify them. Gdk-pixbuf just did not |
|
6016 have a concept of immutability.</p> |
|
6017 <p>Back in GNOME 1.x / 2.x, when it was fashionable to put icons b… |
|
6018 menu items, or in toolbar buttons, applications would load their icon |
|
6019 images, and modify them in various ways before setting them onto the |
|
6020 corresponding widgets. Some things they did: load a colorful icon, |
|
6021 desaturate it for "insensitive" command buttons or menu items, or |
|
6022 simulate desaturation by compositing a 1x1-pixel checkerboard on the |
|
6023 icon image. Or lighten the icon and set it as the "prelight" one onto |
|
6024 widgets.</p> |
|
6025 <p>The concept of "decode an image and just give me the pixels" is… |
|
6026 course useful. Image viewers, image processing programs, and all |
|
6027 those, of course need this functionality.</p> |
|
6028 <p>However, these days GTK would prefer to have a way to decode an… |
|
6029 and ship it as fast as possible ot the GPU, without intermediaries. |
|
6030 There is all sorts of awkward machinery in the GTK widgets that |
|
6031 can consume either an icon from an icon theme, or a user-supplied |
|
6032 image, or one of the various schemes for providing icons that GTK has |
|
6033 acquired over the years.</p> |
|
6034 <p>It is interesting to note that <code>gdk_pixbuf_get_pixel… |
|
6035 pretty much since the beginning, but it was only until much later that |
|
6036 we got <code>gdk_pixbuf_get_pixels_with_length()</code>, the… |
|
6037 *</code> buffer and also its length" function, so that calling cod… |
|
6038 chance of actually checking for buffer overruns. (... and it is one |
|
6039 of the broken "give me a length" functions that returns a <code>gu… |
|
6040 rather than a <code>gsize</code>. There is a better |
|
6041 <code>gdk_pixbuf_get_byte_length()</code> which actually ret… |
|
6042 though.)</p> |
|
6043 <h2>Problems with mutable pixbufs</h2> |
|
6044 <p>The main problem is that as things are right now, we have no |
|
6045 flexibility in changing the internal representation of image data to |
|
6046 make it better for current idioms: GPU-specific pixel formats may not |
|
6047 be unpremultiplied RGBA data.</p> |
|
6048 <p>We have no API to say, "this pixbuf has been modified", akin to |
|
6049 <code>cairo_surface_mark_dirty()</code>: once an applicatio… |
|
6050 <code>gdk_pixbuf_get_pixels()</code>, gdk-pixbuf or GTK have… |
|
6051 data <em>will</em> be changed and they have to re-run the pi… |
|
6052 the image to the GPU (format conversions? caching? creating a |
|
6053 texture?).</p> |
|
6054 <p>Also, ever since the beginnings of the gdk-pixbuf API, we had a… |
|
6055 create pixbufs from arbitrary user-supplied RGBA buffers: the |
|
6056 <code>gdk_pixbuf_new_from_data</code> functions. One proble… |
|
6057 that memory management of the buffer is up to the calling application, |
|
6058 so the resulting pixbuf isn't free to handle those resources as it |
|
6059 pleases.</p> |
|
6060 <p>A relatively recent addition is <code>gdk_pixbuf_new_from… |
|
6061 takes a <code>GBytes</code> buffer instead of a random <c… |
|
6062 is created that way, it is <em>assumed</em> to be immutable,… |
|
6063 is basically a shared reference into a byte buffer, and it's just |
|
6064 easier to think of it as immutable. (Nothing in C actually enforces |
|
6065 immutability, but the API indicates that convention.)</p> |
|
6066 <p>Internally, <code>GdkPixbuf</code> actually prefers… |
|
6067 <code>GBytes</code>. It will <a href="https://gitlab.gno… |
|
6068 something calls the old <code>gdk_pixbuf_get_pixels()</code>… |
|
6069 that will just take ownership of the internal buffer from the |
|
6070 <code>GBytes</code> (if the <code>GBytes</code> … |
|
6071 case, it will copy the buffer from the <code>GBytes</code> a… |
|
6072 of that copy. In either case, when the pixbuf downgrades itself to |
|
6073 pixels, it is assumed that the calling application will modify the |
|
6074 pixel data.</p> |
|
6075 <h2>What would immutable pixbufs look like?</h2> |
|
6076 <p>I mentioned this a bit in "<a href="https://people.gnome.org… |
|
6077 loaders in gdk-pixbuf would create immutable pixbufs, with an internal |
|
6078 representation that is friendly to GPUs. In the proposed scheme, that |
|
6079 internal representation would be a Cairo image surface; it can be |
|
6080 something else if GTK/GDK eventually prefer a different way of |
|
6081 shipping image data into the toolkit.</p> |
|
6082 <p>Those pixbufs would be immutable. In true C fashion we can cal… |
|
6083 undefined behavior to change the pixel data (say, an app could request |
|
6084 <code>gimme_the_cairo_surface</code> and tweak it, but that … |
|
6085 supported).</p> |
|
6086 <p>I think we could also have a "just give me the pixels" API, and… |
|
6087 "create a pixbuf from these pixels" one, but those would be one-time |
|
6088 conversions at the edge of the API. Internally, the pixel data that |
|
6089 actually lives inside a <code>GdkPixbuf</code> would remain … |
|
6090 preferred representation, which is not necessarily what the |
|
6091 application sees.</p> |
|
6092 <h2>What worked well</h2> |
|
6093 <p>A small API to load multiple image formats, and paint the images |
|
6094 easily to the screen, while handling most of the X awkwardness |
|
6095 semi-automatically, was very useful!</p> |
|
6096 <p>A way to get and modify pixel data: applications clearly like d… |
|
6097 this. We can formalize it as an application-side thing only, and keep |
|
6098 the internal representation immutable and in a format that can evolve |
|
6099 according to the needs of the internal API.</p> |
|
6100 <p>Pluggable loaders, up to a point. Gdk-pixbuf doesn't support a… |
|
6101 image formats in the world out of the box, but it is relatively easy |
|
6102 for third-parties to provide loaders that, once installed, are |
|
6103 automatically usable for all applications.</p> |
|
6104 <h2>What didn't work well</h2> |
|
6105 <p>Having effectively two pixel formats supported, and nothing els… |
|
6106 gdk-pixbuf does packed RGB and unpremultiplied RGBA, and that's it. |
|
6107 This isn't completely terrible: applications which really want to |
|
6108 know about indexed or grayscale images, or high bit-depth ones, are |
|
6109 <em>probably</em> specialized enough that they can afford to… |
|
6110 custom loaders with all the functionality they need.</p> |
|
6111 <p>Pluggable loaders, up to a point. While it is relatively easy … |
|
6112 create third-party loaders, installation is awkward from a system's |
|
6113 perspective: one has to run the script to regenerate the loader cache, |
|
6114 there are more shared libraries running around, and the loaders are |
|
6115 not sandboxed by default.</p> |
|
6116 <p>I'm not sure if it's worthwhile to let any application read "an… |
|
6117 image format if gdk-pixbuf supports it. If your word processor lets |
|
6118 you paste an image into the document... do you want it to use |
|
6119 gdk-pixbuf's limited view of things and include a high bit-depth image |
|
6120 with its probably inadequate conversions? Or would you rather do some |
|
6121 processing by hand to ensure that the image looks as good as it can, |
|
6122 in the format that your word processor actually supports? I don't |
|
6123 know.</p> |
|
6124 <p>The API for animations is very awkward. We don't even support |
|
6125 APNG... but honestly I don't recall actually seeing one of those in |
|
6126 the wild.</p> |
|
6127 <p>The progressive loading API is awkward. The "feed some bytes i… |
|
6128 loader" part is mostly okay; the "notify me about changes to the pixel |
|
6129 data" is questionable nowadays. Web browsers don't use it; they |
|
6130 implement their own loaders. Even EOG doesn't use it.</p> |
|
6131 <p>I think most code that actually connects to <code>GdkPixb… |
|
6132 signals only uses the <code>size-prepared</code> signal — … |
|
6133 emitted soon after reading the image headers, when the loader gets to |
|
6134 know the dimensions of the image. Apps sometimes use this to say, |
|
6135 "this image is W*H pixels in size", but don't actually decode the |
|
6136 rest of the image.</p> |
|
6137 <p>The gdk-pixbuf model of static images, or GIF animations, doesn… |
|
6138 well for multi-page TIFFs. I'm not sure if this is actualy a problem. |
|
6139 Again, applications with actual needs for multi-page TIFFs are |
|
6140 probably specialized enough that they will want a full-featured TIFF |
|
6141 loader of their own.</p> |
|
6142 <h2>Awkward architectures</h2> |
|
6143 <h3>Thumbnailers</h3> |
|
6144 <p>The thumbnailing system has slowly been moving towards a model … |
|
6145 we actually have thumbnailers specific to each file format, instead of |
|
6146 just assuming that we can dump any image into a gdk-pixbuf loader.</p… |
|
6147 <p>If we take this all the way, we would be able to remove some we… |
|
6148 code in, for example, the JPEG pixbuf loader. Right now it supports |
|
6149 loading images at a size that the calling code requests, not only at |
|
6150 the "natural" size of the JPEG. The thumbnailer can say, "I want to |
|
6151 load this JPEG at 128x128 pixels" or whatever, and <em>in theory&l… |
|
6152 JPEG loader will do the minimal amount of work required to do that. |
|
6153 It's not 100% clear to me if this is actually working as intended, or |
|
6154 if we downscale the whole image anyway.</p> |
|
6155 <p>We had a distinction between in-process and out-of-process |
|
6156 thumbnailers, and it had to do with the way pixbuf loaders are used; |
|
6157 I'm not sure if they are all out-of-process and sandboxed now.</p> |
|
6158 <h3>Non-raster data</h3> |
|
6159 <p>There is a gdk-pixbuf loader for SVG images which uses librsvg |
|
6160 internally, but only in a very basic way: it simply loads the SVG at |
|
6161 its preferred size. Librsvg jumps through some hoops to compute a |
|
6162 "preferred size" for SVGs, as not all of them actually indicate one. |
|
6163 The SVG model would rather have the renderer say that the SVG is to be |
|
6164 inserted into a rectangle of certain width/height, and |
|
6165 scaled/positioned inside the rectangle according to some other |
|
6166 parameters (i.e. like one would put it inside an HTML document, with a |
|
6167 <code>preserveAspectRatio</code> attribute and all that). G… |
|
6168 historically operated with a different model, one of "load me an |
|
6169 image, I'll scale it to whatever size, and paint it".</p> |
|
6170 <p>This gdk-pixbuf loader for SVG files gets used for the SVG |
|
6171 thumbnailer, or more accurately, the "throw random images into a |
|
6172 gdk-pixbuf loader" thumbnailer. It may be better/cleaner to have a |
|
6173 specific thumbnailer for SVGs instead.</p> |
|
6174 <p>Even EOG, our by-default image viewer, doesn't use the gdk-pixb… |
|
6175 loader for SVGs: it actually special-cases them and uses librsvg |
|
6176 directly, to be able to load an SVG once and re-render it at different |
|
6177 sizes if one changes the zoom factor, for example.</p> |
|
6178 <p>GTK reads its SVG icons... without using librsvg... by assuming… |
|
6179 librsvg installed its gdk-pixbuf loader, so it loads them as any |
|
6180 normal raster image. This kind of dirty, but I can't quite pinpoint |
|
6181 why. I'm sure it would be convenient for icon themes to ship a single |
|
6182 SVG with tons of icons, and some metadata on their <code>id</co… |
|
6183 could pick them out of the SVG file with <code>rsvg_render_cairo_s… |
|
6184 something. Right now icon theme authors are responsible for splitting |
|
6185 out those huge SVGs into many little ones, one for each icon, and I |
|
6186 don't think that's their favorite thing in the world to do :)</p> |
|
6187 <h3>Exotic raster data</h3> |
|
6188 <p>High bit-depth images... would you expect EOG to be able to loa… |
|
6189 Certainly; maybe not with all the fancy conversions from a real RAW |
|
6190 photo editor. But maybe this can be done as EOG-specific plugins, |
|
6191 rather than as low in the platform as the gdk-pixbuf loaders?</p> |
|
6192 <p>(Same thing for thumbnailing high bit-depth images: the loadin… |
|
6193 should just provide its own thumbnailer program for those.)</p> |
|
6194 <h3>Non-image metadata</h3> |
|
6195 <p>The <code>gdk_pixbuf_set_option</code> / <code&g… |
|
6196 functions is so that pixbuf loaders can set key/value pairs of strings |
|
6197 onto a pixbuf. Loaders use this for <code>comment</code> bl… |
|
6198 for color calibration, or DPI information for images that have it, or |
|
6199 EXIF data from photos. It is up to applications to actually use this |
|
6200 information.</p> |
|
6201 <p>It's a bit uncomfortable that gdk-pixbuf makes no promises abou… |
|
6202 kind of raster data it gives to the caller: right now it is raw |
|
6203 RGB(A) data that is not gamma-corrected nor in any particular color |
|
6204 space. It is up to the caller to see if the pixbuf has an ICC profile |
|
6205 attached to it as an <code>option</code>. Effectively, this… |
|
6206 applications don't know if they are getting SRGB, or linear RGB, or |
|
6207 what... unless they specifically care to look.</p> |
|
6208 <p>The gdk-pixbuf API could probably make promises: if you call &… |
|
6209 function</em> you will get SRGB data; if you call <em>this o… |
|
6210 you'll get the raw RGBA data and we'll tell you its |
|
6211 colorspace/gamma/etc.</p> |
|
6212 <p>The various <code>set_option</code> / <code>g… |
|
6213 gdk-pixbuf <em>saving</em> code (up to now we have just talk… |
|
6214 loaders). I don't know enough about how applications use the saving |
|
6215 code in gdk-pixbuf... the thumbnailers use it to save PNGs or JPEGs, |
|
6216 but other apps? No idea.</p> |
|
6217 <h2>What I would like to see</h2> |
|
6218 <p><strong>Immutable pixbufs in a useful format.</strong&… |
|
6219 this</a> in a merge request; the internal code is now ready |
|
6220 to take in different internal representations of pixel data. My goal |
|
6221 is to make Cairo image surfaces the preferred, immutable, internal |
|
6222 representation. This would give us a |
|
6223 <code>gdk_pixbuf_get_cairo_surface()</code>, which pretty mu… |
|
6224 needs one reimplements by hand.</p> |
|
6225 <p><strong>Find places that assume mutable pixbufs.</stro… |
|
6226 mutable pixbufs, I think we would need to audit applications and |
|
6227 libraries to find places that cause <code>GdkPixbuf</code> s… |
|
6228 into mutable ones: basically, find callers of |
|
6229 <code>gdk_pixbuf_get_pixels()</code> and related functions, … |
|
6230 reimplement them differently. Maybe they don't need to tint icons by |
|
6231 hand anymore? Maybe they <em>don't need icons</em> anymore,… |
|
6232 changing UI paradigms? Maybe they are using gdk-pixbuf as an image |
|
6233 loader only?</p> |
|
6234 <p><strong>Reconsider the loading-updates API.</strong>… |
|
6235 <code>GdkPixbufLoader::area-updated</code> signal at all? D… |
|
6236 if we just... not emit it, or just emit it once at the end of the |
|
6237 loading process? (Caveat: keeping it unchanged more or less means |
|
6238 that "immutable pixbufs" as loaded by gdk-pixbuf actually mutate while |
|
6239 being loaded, and this mutation is exposed to applications.)</p> |
|
6240 <p><strong>Sandboxed loaders.</strong> While these da… |
|
6241 progressive feed-it-bytes API, sandboxed loaders would maybe prefer a |
|
6242 read-a-whole-file approach. I don't know enough about memfd or how |
|
6243 sandboxes pass data around to know how either would work.</p> |
|
6244 <p><strong>Move loaders to Rust.</strong> Yes, really… |
|
6245 security-sensitive, and while we <em>do</em> need to sandbox… |
|
6246 certainly be better to do them in a memory-safe language. There are |
|
6247 already pure Rust-based image loaders: <a href="https://crates.io/cra… |
|
6248 <a href="https://crates.io/crates/png">PNG</a>, <a href="… |
|
6249 I have no idea how featureful they are. We can certainly try them |
|
6250 with gdk-pixbuf's own suite of test images. We can modify them to add |
|
6251 hooks for things like a <code>size-prepared</code> notificat… |
|
6252 already have a way to read "just the image headers".</p> |
|
6253 <p>Rust makes it very easy to plug in <a href="https://crates.i… |
|
6254 <a href="https://crates.io/crates/afl">fuzz testing</a>, and… |
|
6255 perfect for improving the loaders.</p> |
|
6256 <p>I started <a href="https://gitlab.gnome.org/federico/gdk-pix… |
|
6257 loaders</a> some months ago, but there's nothing useful |
|
6258 yet. One mismatch between gdk-pixbuf's model for loaders, and the |
|
6259 existing Rust codecs, is that Rust codecs generally take something |
|
6260 that implements the <code>Read</code> trait: a blocking API … |
|
6261 abstract sources; it's a pull API. The gdk-pixbuf model is a push |
|
6262 API: the calling code creates a loader object, and then pushes bytes |
|
6263 into it. The gdk-pixbuf convenience functions that take a |
|
6264 <code>GInputStream</code> basically do this:</p> |
|
6265 <div class="highlight"><pre><span></span><cod… |
|
6266 |
|
6267 <span class="k">while</span> <span class="p">(</spa… |
|
6268 <span class="n">n_read</span> <span class="o">=<… |
|
6269 <span class="n">gdk_pixbuf_loader_write</span><span c… |
|
6270 <span class="p">}</span> |
|
6271 |
|
6272 <span class="n">gdk_pixbuf_loader_close</span> <span clas… |
|
6273 </code></pre></div> |
|
6274 |
|
6275 <p>However, this cannot be flipped around easily. We could probab… |
|
6276 a second thread (easy, safe to do in Rust) to make the reader/decoder |
|
6277 thread block while the main thread pushes bytes into it.</p> |
|
6278 <p>Also, I don't know how the Rust bindings for GIO present things… |
|
6279 <code>GInputStream</code> and friends, with our nice async c… |
|
6280 that.</p> |
|
6281 <p><strong>Deprecate animations?</strong> Move that c… |
|
6282 at memes in it? Do any "real apps" actually use GIF animations for |
|
6283 their UI?</p> |
|
6284 <p><strong>Formalize promises around returned color profiles… |
|
6285 mentioned above: have an "easy API" that returns SRGB, and a "raw API" |
|
6286 that returns the ARGB data from the image, plus info on its ICC |
|
6287 profile, gamma, or any other info needed to turn this into a |
|
6288 "good enough to be universal" representation. (I <em>think</em… |
|
6289 Apple APIs that pass colors around do so with an ICC profile attached, |
|
6290 which seems... pretty much necessary for correctness.)</p> |
|
6291 <p><strong>Remove the internal MIME-sniffing machinery.</… |
|
6292 <p><strong>Deprecate the crufty/old APIs in gdk-pixbuf.</… |
|
6293 Scaling/transformation, compositing, <code>GdkPixdata</code>, |
|
6294 <code>gdk-pixbuf-csource</code>, all those. Pixel crunching… |
|
6295 Cairo; the others are better done with <code>GResource</code>… |
|
6296 <p><strong>Figure out if we want blessed codecs; fix thumbna… |
|
6297 loaders statically, unconditionally. Exotic formats can go in their |
|
6298 own custom thumbnailers. Figure out if we want sandboxed loaders for |
|
6299 everything, or just for user-side images (not ones read from the |
|
6300 trusted system installation).</p> |
|
6301 <p><strong>Have GTK4 communicate clearly about its drawing m… |
|
6302 are having a disconnect between the GUI chrome, which is CSS/GPU |
|
6303 friendly, and graphical content generated by applications, which by |
|
6304 default right now is done via Cairo. And having Cairo as a to-screen |
|
6305 and to-printer API is certainly very convenient! You Wouldn't Print a |
|
6306 GUI, but certainly you would print a displayed document.</p> |
|
6307 <p>It would also be useful for GTK4 to actually define what its pr… |
|
6308 image format is if it wants to ship it to the GPU with as little work |
|
6309 as possible. Maybe it's a Cairo image surface? Maybe something else?&l… |
|
6310 <h2>Conclusion</h2> |
|
6311 <p>We seem to change imaging models every ten years or so. Xlib, … |
|
6312 Xrender with Cairo, then GPUs and CSS-based drawing for widgets. |
|
6313 We've gone from trusted data on your local machine, to potentially malic… |
|
6314 rains from the Internet. Gdk-pixbuf has spanned all of these periods |
|
6315 so far, and it is due for a big change.</p></content><category ter… |
|
6316 because it was leaking all the SVG nodes — has been interesting.</p… |
|
6317 <p><em>Memory leaks in Rust? Isn't it supposed to prevent t… |
|
6318 <p>Well, yeah, but the leaks were caused by the C side of things, … |
|
6319 <code>unsafe</code> code in Rust, which …</p></summa… |
|
6320 because it was leaking all the SVG nodes — has been interesting.</p… |
|
6321 <p><em>Memory leaks in Rust? Isn't it supposed to prevent t… |
|
6322 <p>Well, yeah, but the leaks were caused by the C side of things, … |
|
6323 <code>unsafe</code> code in Rust, which does not prevent lea… |
|
6324 <p><a href="https://gitlab.gnome.org/federico/librsvg/commit/29… |
|
6325 function implemented in Rust, which returns a newly-acquired reference |
|
6326 to an SVG node. The old code simply got a pointer to the node, |
|
6327 without acquiring a reference. The new code was forgetting to |
|
6328 <code>rsvg_node_unref()</code>. No biggie.</p> |
|
6329 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/commit/2d3dd… |
|
6330 was apparently calling all the functions to unref nodes as |
|
6331 appropriate, and even calling the <code>rsvg_tree_free()</code&… |
|
6332 end; this is the "free the whole SVG tree" function.</p> |
|
6333 <p>There are these types:</p> |
|
6334 <div class="highlight"><pre><span></span><cod… |
|
6335 <span class="k">pub</span><span class="w"> </span&g… |
|
6336 |
|
6337 <span class="c1">// This is the real structure we care about</s… |
|
6338 <span class="k">pub</span><span class="w"> </span&g… |
|
6339 <span class="w"> </span><span class="c1">// This is… |
|
6340 <span class="w"> </span><span class="k">pub</spa… |
|
6341 <span class="w"> </span><span class="o">..</span… |
|
6342 <span class="p">}</span><span class="w"></span> |
|
6343 </code></pre></div> |
|
6344 |
|
6345 <p><code>Tree</code> is the real struct that holds the… |
|
6346 other data. Each node is an <code>Rc&lt;Node&gt;</code… |
|
6347 leaked (... and all the children, recursively) because its reference |
|
6348 count never went down from 1.</p> |
|
6349 <p><code>RsvgTree</code> is just an empty type. The c… |
|
6350 <code>*const Tree</code> as <code>*const RsvgTree</… |
|
6351 the C code.</p> |
|
6352 <p>The <code>rsvg_tree_free()</code> function, callabl… |
|
6353 <div class="highlight"><pre><span></span><cod… |
|
6354 <span class="k">pub</span><span class="w"> </span&g… |
|
6355 <span class="w"> </span><span class="k">if</span… |
|
6356 <span class="w"> </span><span class="kd">let<… |
|
6357 <span class="w"> </span><span cla… |
|
6358 <span class="w"> </span><span class="p">}</span&… |
|
6359 <span class="p">}</span><span class="w"></span> |
|
6360 </code></pre></div> |
|
6361 |
|
6362 <p>When we call <code>Box::from_raw()</code> on a <… |
|
6363 a <code>Box&lt;RsvgTree&gt;</code>... which is a box… |
|
6364 frees zero memory when the box gets dropped.</p> |
|
6365 <p>The code was missing this cast:</p> |
|
6366 <div class="highlight"><pre><span></span><cod… |
|
6367 <span class="w"> </span><… |
|
6368 <span class="w"> </span><span class="kd">let</sp… |
|
6369 </code></pre></div> |
|
6370 |
|
6371 <p>So, <code>tree as *mut Tree</code> gives us a value… |
|
6372 <code>Box::from_raw()</code> to return a <code>Box&… |
|
6373 Dropping the box will drop the <code>Tree</code>, reduce the… |
|
6374 on the root node, and free all the nodes recursively.</p> |
|
6375 <h2>Monitoring an <code>Rc&lt;T&gt;</code>'s r… |
|
6376 <p>So, how does one set a gdb watchpoint on the reference count?&l… |
|
6377 <p>First I set a breakpoint on a function which I knew would get p… |
|
6378 the <code>Rc&lt;Node&gt;</code> I care about:</p&… |
|
6379 <div class="highlight"><pre><span></span><cod… |
|
6380 Breakpoint 3 at 0x7ffff71f3aaa: file rsvg_internals/src/structure.rs, li… |
|
6381 |
|
6382 (gdb) c |
|
6383 Continuing. |
|
6384 |
|
6385 Thread 1 &quot;rsvg-convert&quot; hit Breakpoint 3, &lt;rsvg… |
|
6386 |
|
6387 (gdb) p node |
|
6388 $5 = (alloc::rc::Rc&lt;rsvg_internals::node::Node&gt; *) 0x64c890 |
|
6389 </code></pre></div> |
|
6390 |
|
6391 <p>Okay, <code>node</code> is a reference to an <co… |
|
6392 <div class="highlight"><pre><span></span><cod… |
|
6393 $6 = {ptr = {pointer = {__0 = 0x625800}}, phantom = {&lt;No data fie… |
|
6394 </code></pre></div> |
|
6395 |
|
6396 <p>Why, a <code>pointer</code> to the actual contents … |
|
6397 again:</p> |
|
6398 <div class="highlight"><pre><span></span><cod… |
|
6399 $9 = {strong = {value = {value = 3}}, weak = {value = {value = 1}}, ...… |
|
6400 </code></pre></div> |
|
6401 |
|
6402 <p>Aha! There are the <code>strong</code> and <cod… |
|
6403 watchpoint on the strong reference count:</p> |
|
6404 <div class="highlight"><pre><span></span><cod… |
|
6405 (gdb) watch *$ptr |
|
6406 Hardware watchpoint 4: *$ptr |
|
6407 </code></pre></div> |
|
6408 |
|
6409 <p>Continue running the program until the reference count changes:… |
|
6410 <div class="highlight"><pre><span></span><cod… |
|
6411 <span class="nv">Thread</span> <span class="mi">1</… |
|
6412 |
|
6413 <span class="nv">Old</span> <span class="nv">value<… |
|
6414 <span class="nv">New</span> <span class="nv">value<… |
|
6415 </code></pre></div> |
|
6416 |
|
6417 <p>At this point I can print a stack trace and see if it makes sen… |
|
6418 check that the refs/unrefs are matched, etc.</p> |
|
6419 <p>TL;DR: dig into the <code>Rc&lt;T&gt;</code>… |
|
6420 watch it. It's wrapped in several layers of Rust-y types; <code>N… |
|
6421 pointers, an <code>RcBox</code> for the actual container of … |
|
6422 object it's wrapping, and <code>Cell</code>s for the refcoun… |
|
6423 until you reach the refcount values and they are there.</p> |
|
6424 <h2>So, how did I find the missing cast?</h2> |
|
6425 <p>Using that gdb recipe, I watched the reference count of the top… |
|
6426 SVG node change until the program exited. When the program |
|
6427 terminated, the reference count was 1 — it should have dropped to 0 if |
|
6428 there was no memory leak.</p> |
|
6429 <p>The last place where the toplevel node loses a reference is in |
|
6430 <code>rsvg_tree_free()</code>. I ran the program again and … |
|
6431 function was being called; it <em>was</em> being called corr… |
|
6432 that the problem must lie in that function. After a little |
|
6433 head-scratching, I found the missing cast. Other functions of the |
|
6434 form <code>rsvg_tree_whatever()</code> had that cast, but &l… |
|
6435 missing it.</p> |
|
6436 <p>I think Rust now has better facilities to tag structs that are … |
|
6437 as raw pointers to <code>extern</code> code, to avoid this k… |
|
6438 casting. We'll see.</p> |
|
6439 <p>In the meantime, apologies for the buggy releases!</p></c… |
|
6440 for librsvg.</p> |
|
6441 <p>A popular way to add logging to Rust code is to use the <a h… |
|
6442 This lets you sprinkle simple messages in your code:</p> |
|
6443 <div class="highlight"><pre><span></span><cod… |
|
6444 <span class="n">debug</span><span class="o">!</span… |
|
6445 </code></pre></div> |
|
6446 |
|
6447 <p>However, the <a href="https://crates.io/crates/log">log �… |
|
6448 for librsvg.</p> |
|
6449 <p>A popular way to add logging to Rust code is to use the <a h… |
|
6450 This lets you sprinkle simple messages in your code:</p> |
|
6451 <div class="highlight"><pre><span></span><cod… |
|
6452 <span class="n">debug</span><span class="o">!</span… |
|
6453 </code></pre></div> |
|
6454 |
|
6455 <p>However, the <a href="https://crates.io/crates/log">log&l… |
|
6456 messages do not get emitted anywhere. The calling code has to set up |
|
6457 a logger. Crates like <a href="https://crates.io/crates/env_logger"&… |
|
6458 program initialization, that gets configured through an environment |
|
6459 variable.</p> |
|
6460 <p>And this is a problem for librsvg: we are <em>not</em… |
|
6461 initialization! Librsvg is a library; it doesn't have a <code>mai… |
|
6462 function. And since most of the calling code is not Rust, we can't |
|
6463 assume that they can call code that can initialize the logging |
|
6464 framework.</p> |
|
6465 <h2>Why not use glib's logging stuff?</h2> |
|
6466 <p>Currently this is a bit clunky to use from Rust, since glib's |
|
6467 structured logging functions are not bound yet in <a href="http://gtk… |
|
6468 would be good to bind them and get this over with.</p> |
|
6469 <h2>What user experience do we want?</h2> |
|
6470 <p>In the past, what has worked well for me to do logging from lib… |
|
6471 is to allow the user to set an environment variable to control the |
|
6472 logging, or to drop a log configuration file in their $HOME. The |
|
6473 former works well when the user is in control of running the program |
|
6474 that will print the logs; the latter is useful when the user is not |
|
6475 directly in control, like for gnome-shell, which gets launched through |
|
6476 a lot of magic during session startup.</p> |
|
6477 <p>For librsvg, it's probably enough to just use an environment |
|
6478 variable. Set <code>RSVG_LOG=parse_errors</code>, run your … |
|
6479 useful output. <a href="https://makezine.com/2016/06/10/push-button-… |
|
6480 <h2>Other options in Rust?</h2> |
|
6481 <p>There is a <a href="https://crates.io/crates/slog">slog&l… |
|
6482 context-less macros which depend on a single global logger, it |
|
6483 provides logging macros to which you pass a logger object.</p> |
|
6484 <p>For librsvg, this means that the basic <code>RsvgHandle&l… |
|
6485 own logger, based on an environment variable or whatever, and pass it |
|
6486 around to all its child functions for when they need to log something.&l… |
|
6487 <p>Slog supports structured logging, and seems to have some fancy … |
|
6488 modes. We'll see.</p></content><category term="misc"></category><… |
|
6489 on in librsvg right now:</p> |
|
6490 <ol> |
|
6491 <li> |
|
6492 <p>Paolo Borelli finished porting all the CSS properties to Rust. |
|
6493 What was once a gigantic <code>RsvgState</code> struct in… |
|
6494 along with all the janky C code to parse individual properties …<… |
|
6495 on in librsvg right now:</p> |
|
6496 <ol> |
|
6497 <li> |
|
6498 <p>Paolo Borelli finished porting all the CSS properties to Rust. |
|
6499 What was once a gigantic <code>RsvgState</code> struct in… |
|
6500 along with all the janky C code to parse individual properties. |
|
6501 The process of porting <code>RsvgState</code> to Rust has… |
|
6502 about two months ago</a>, and has involved many multi-commit |
|
6503 merge requests and refactorings. This is a tremendous amount of |
|
6504 really good work! The result is all in Rust now in a <code>Sta… |
|
6505 struct, which is opaque from C's viewpoint. The only places in C |
|
6506 that still require accessors to the <code>State</code> ar… |
|
6507 effects code. Which brings me to...</p> |
|
6508 </li> |
|
6509 <li> |
|
6510 <p>Ivan Molodetskikh, my Summer of Code student, submitted his <… |
|
6511 merge request</a> and it's merged to master now. This ports |
|
6512 the bookkeeping infrastructure for SVG filters to Rust, and also |
|
6513 the <code>feOffset</code> filter is ported now. Right no… |
|
6514 anything fancy to iterate over the pixels of Cairo image surfaces; |
|
6515 that will come later. I am very happy that filters, which were a |
|
6516 huge barrier, are now starting to get chipped away into nicer code.&l… |
|
6517 </li> |
|
6518 <li> |
|
6519 <p>I have started to move librsvg's old representation of CSS |
|
6520 properties into something that can really represent properties that |
|
6521 are not specified, or explicitly set to <code>inherit</code&… |
|
6522 element's parent, or set to a normal value. Librsvg never had a |
|
6523 representation of property values that actually matched the SVG/CSS |
|
6524 specs; it just knew whether a property was specified or not for an |
|
6525 element. This worked fine for properties which the spec mandates |
|
6526 that they should inherit automatically, but those that <em>don'… |
|
6527 were handled through special hacks. The new code makes this a lot |
|
6528 cleaner. It should also make it easier to copy Servo's idioms for |
|
6529 property inheritance.</p> |
|
6530 </li> |
|
6531 </ol></content><category term="misc"></category><category term="li… |
|
6532 over the years.</p> |
|
6533 <h1>In ye olden days</h1> |
|
6534 <p>In the context of GIMP/GNOME, the only thing that knew how to d… |
|
6535 images to X11 windows (doing palette mapping for 256-color graphics |
|
6536 cards and dithering if necessary) was the …</p></summary><conten… |
|
6537 over the years.</p> |
|
6538 <h1>In ye olden days</h1> |
|
6539 <p>In the context of GIMP/GNOME, the only thing that knew how to d… |
|
6540 images to X11 windows (doing palette mapping for 256-color graphics |
|
6541 cards and dithering if necessary) was the GIMP. Later, when GTK+ was |
|
6542 written, it exported a <code>GtkPreview</code> widget, which… |
|
6543 image buffer supplied by the application and render it to an X window |
|
6544 — this was what GIMP plug-ins could use in their user interface to |
|
6545 show, well, previews of what they were about to do with the user's |
|
6546 images. Later we got some obscure magic in a <code>GdkColorContex… |
|
6547 object, which helped allocate X11 colors for the X drawing primitives. |
|
6548 In turn, <code>GdkColorContext</code> came from the port tha… |
|
6549 XmHTML's color context object (and for those that remember, XmHTML |
|
6550 became the first version of GtkHtml; later it was rewritten as a port |
|
6551 of KDE's HTML widget). Thankfully all that stuff is gone now; we can |
|
6552 now assume that video cards are 24-bit RGB or better everywhere, and |
|
6553 there is no need to worry about limited color palettes and color |
|
6554 allocation.</p> |
|
6555 <p>Later, we started using the Imlib library, from the Enlightenme… |
|
6556 project, as an easy API to load images — the APIs from libungif, |
|
6557 libjpeg, libpng, etc. were not something one really wanted to use |
|
6558 directly — and also to keep images in memory with a uniform |
|
6559 representation. Unfortunately, Imlib's memory management was |
|
6560 peculiar, as it was tied to Enlightenment's model for caching and |
|
6561 rendering loaded/scaled images.</p> |
|
6562 <p>A bunch of people worked to write GdkPixbuf: it kept Imlib's c… |
|
6563 of a unified representation for image data, and an easy API to load |
|
6564 various image formats. It added support for an alpha channel (we only |
|
6565 had 1-bit masks before), and it put memory management in the hands of |
|
6566 the calling application, in the form of reference counting. GdkPixbuf |
|
6567 obtained some high-quality scaling functions, mainly for use by Eye Of |
|
6568 Gnome (our image viewer) and by applications that just needed scaling |
|
6569 instead of arbitrary transformations.</p> |
|
6570 <p>Later, we got libart, the first library in GNOME to do antialia… |
|
6571 vector rendering and affine transformations. Libart was more or less |
|
6572 compatible with GdkPixbuf: they both had the same internal |
|
6573 representation for pixel data, but one had to pass the |
|
6574 pixels/width/height/rowstride around by hand.</p> |
|
6575 <h1>Mea culpa</h1> |
|
6576 <p>Back then I didn't understand <a href="https://keithp.com/~k… |
|
6577 which is now ubiquitous. The GIMP made the decision to use |
|
6578 non-premultiplied alpha when it introduced layers with transparency, |
|
6579 probably to "avoid losing data" from transparent pixels. GdkPixbuf |
|
6580 follows the same scheme.</p> |
|
6581 <p>(Now that the GIMP uses GEGL for its internal representation of |
|
6582 images... I have no idea what it does with respect to alpha.)</p> |
|
6583 <h1>Cairo and afterwards</h1> |
|
6584 <p>Some time after the libart days, we got Cairo and pixman. Cair… |
|
6585 different representation of images than GdkPixbuf's, and it supported |
|
6586 more pixel formats and color models.</p> |
|
6587 <p>GTK2 got patched to use Cairo in the toplevel API. We still ha… |
|
6588 dichotomy between Cairo's image surfaces, which are ARGB premultiplied |
|
6589 data in memory, and GdkPixbufs, which are RGBA non-premultiplied. |
|
6590 There are utilities in GTK+ to do these translations, but they are |
|
6591 inconvenient: every time a program loads an image with GdkPixbuf's |
|
6592 easy API, a translation has to happen from non-premul RGBA to premul |
|
6593 ARGB.</p> |
|
6594 <p>Having two formats means that we inevitably do translations bac… |
|
6595 forth of practically the same data. For example, when one embeds a |
|
6596 JPEG inside an SVG, librsvg will read that JPEG using GdkPixbuf, |
|
6597 translate it to Cairo's representation, composite it with Cairo onto |
|
6598 the final result, and finally translate the whole thing back to a |
|
6599 GdkPixbuf... if someone uses librsvg's legacy APIs to output pixbufs |
|
6600 instead of rendering directly to a Cairo surface.</p> |
|
6601 <p>Who uses that legacy API? GTK+, of course! GTK+ loads scalabl… |
|
6602 icons with GdkPixbuf's loader API, which dynamically links librsvg at |
|
6603 runtime: in effect, GTK+ doesn't use librsvg directly. And the SVG |
|
6604 pixbuf loader uses the "gimme a pixbuf" API in librsvg.</p> |
|
6605 <h1>GPUs</h1> |
|
6606 <p>Then, we got GPUs everywhere. Each GPU has its own preferred p… |
|
6607 format. Image data has to be copied to the GPU at some point. |
|
6608 Cairo's ARGB needs to be translated to the GPU's preferred format and |
|
6609 alignment.</p> |
|
6610 <h1>Summary so far</h1> |
|
6611 <ul> |
|
6612 <li> |
|
6613 <p>Libraries that load images from standard formats have different |
|
6614 output formats. Generally they can be coaxed into spitting ARGB or |
|
6615 RGBA, but we don't expect them to support any random representation |
|
6616 that a GPU may want.</p> |
|
6617 </li> |
|
6618 <li> |
|
6619 <p>GdkPixbuf uses non-premultiplied RGBA data, always in that orde… |
|
6620 </li> |
|
6621 <li> |
|
6622 <p>Cairo uses premultiplied ARGB in platform-endian 32-bit chunks:… |
|
6623 each pixel is 0xaarrggbb, then the bytes are shuffled around |
|
6624 depending on whether the platform is little-endian or big-endian.</… |
|
6625 </li> |
|
6626 <li> |
|
6627 <p>Cairo internally uses a subset of the formats supported by pixm… |
|
6628 </li> |
|
6629 <li> |
|
6630 <p>GPUs use whatever they damn well please.</p> |
|
6631 </li> |
|
6632 <li> |
|
6633 <p>Hilarity ensues.</p> |
|
6634 </li> |
|
6635 </ul> |
|
6636 <h1>What would we like to do?</h1> |
|
6637 <p>We would like to reduce the number of translations between image |
|
6638 formats along the loading-processing-display pipeline. Here is a |
|
6639 plan:</p> |
|
6640 <ul> |
|
6641 <li> |
|
6642 <p>Make sure Cairo/pixman support the image formats that GPUs gene… |
|
6643 prefer. Have them do the necessary conversions if the rest of the |
|
6644 program passes an unsupported format. Ensure that a Cairo image |
|
6645 surface can be created with the GPU's preferred format.</p> |
|
6646 </li> |
|
6647 <li> |
|
6648 <p>Make GdkPixbuf just be a wrapper around a Cairo image surface. |
|
6649 <code>GdkPixbuf</code> is already an opaque structure, and… |
|
6650 to copy pixel data in case the calling code requests it, or wants to |
|
6651 turn a pixbuf from immutable to mutable.</p> |
|
6652 </li> |
|
6653 <li> |
|
6654 <p>Provide GdkPixbuf APIs that deal with Cairo image surfaces. For |
|
6655 example, deprecate <code>gdk_pixbuf_new()</code> and |
|
6656 <code>gdk_pixbuf_new_from_data()</code>, in favor of a new |
|
6657 <code>gdk_pixbuf_new_from_cairo_image_surface()</code>. I… |
|
6658 <code>gdk_pixbuf_get_pixels()</code> and related functions… |
|
6659 <code>gdk_pixbuf_get_cairo_image_surface()</code>. Mark t… |
|
6660 data" functions as highly discouraged, and only for use really by |
|
6661 applications that want to use GdkPixbuf as an image loader and |
|
6662 little else.</p> |
|
6663 </li> |
|
6664 <li> |
|
6665 <p>Remove calls in GTK+ that cause image conversions; make them use |
|
6666 Cairo image surfaces directly, from GdkTexture up.</p> |
|
6667 </li> |
|
6668 <li> |
|
6669 <p>Audit applications to remove calls that cause image conversions. |
|
6670 Generally, look for where they use GdkPixbuf's deprecated APIs and |
|
6671 update them.</p> |
|
6672 </li> |
|
6673 </ul> |
|
6674 <h1>Is this really a performance problem?</h1> |
|
6675 <p>This is in the "<a href="https://people.gnome.org/~federico/… |
|
6676 issues. All those conversions are not really slow (they don't make up |
|
6677 for the biggest part of profiles), but they are nevertheless things |
|
6678 that we could avoid doing. We may get some speedups, but it's |
|
6679 probably more interesting to look at things like power consumption.</… |
|
6680 <p>Right now I'm seeing this as a cool, minor optimization, but mo… |
|
6681 <strong>a way to gradually modernize our image API</strong>.… |
|
6682 <p>We seem to change imaging models every N years (X11 -&gt; l… |
|
6683 -&gt; Cairo -&gt; render trees in GPUs -&gt; ???). It is ve… |
|
6684 applications to use different APIs. In the meantime, we can provide a |
|
6685 more linear path for image data, instead of doing unnecessary |
|
6686 conversions everywhere.</p> |
|
6687 <h1>Code</h1> |
|
6688 <p>I have a <a href="https://gitlab.gnome.org/federico/gdk-pixb… |
|
6689 gdk-pixbuf</a>, |
|
6690 which I'll be working on this week. Meanwhile, you may be interested |
|
6691 in the ongoing <a href="https://wiki.gnome.org/Hackfests/Performance2… |
|
6692 generation for GObject interfaces. This is so that you can do</p> |
|
6693 <div class="highlight"><pre><span></span><cod… |
|
6694 <span class="w"> </span><span class="n">interface&l… |
|
6695 <span class="w"> </span><span class="kr">virtua… |
|
6696 <span class="w"> </span><span class="p">}</span&… |
|
6697 <span class="p">}</span><span class="w"></span> |
|
6698 </code></pre></div> |
|
6699 |
|
6700 <p>and it will generate the appropriate <code>FooIface</c… |
|
6701 with the C versions of interfaces.</p> |
|
6702 <p>It turns …</p></summary><content type="html"><p>T… |
|
6703 generation for GObject interfaces. This is so that you can do</p> |
|
6704 <div class="highlight"><pre><span></span><cod… |
|
6705 <span class="w"> </span><span class="n">interface&l… |
|
6706 <span class="w"> </span><span class="kr">virtua… |
|
6707 <span class="w"> </span><span class="p">}</span&… |
|
6708 <span class="p">}</span><span class="w"></span> |
|
6709 </code></pre></div> |
|
6710 |
|
6711 <p>and it will generate the appropriate <code>FooIface</c… |
|
6712 with the C versions of interfaces.</p> |
|
6713 <p>It turns out that this can share a lot of code from the existin… |
|
6714 generator for classes: both classes and interfaces are "just virtual |
|
6715 method tables", plus signals and properties, and classes can actually |
|
6716 have per-instance fields and such. I started refactoring the code |
|
6717 generator to allow this.</p> |
|
6718 <p>I also took a second look at how to present good error messages… |
|
6719 the <code>syn</code> crate encounters a parse error. I need… |
|
6720 and experiment with this carefully.</p> |
|
6721 <h2>Back home</h2> |
|
6722 <p>I'm back home now, jetlagged but very happy that gnome-class is… |
|
6723 more advanced a state than it was before the hackfest. I'm <strong&g… |
|
6724 thankful</strong> that practically everyone worked on it!</p> |
|
6725 <p>Also, <strong>thanks</strong> to Alberto and Natali… |
|
6726 apartment and showing me around Madrid, all while wrangling their |
|
6727 adorable baby Mario. We had a lovely time on Saturday, and ate |
|
6728 excellent food downtown.</p> |
|
6729 <p><img alt="Sponsored by the GNOME Foundation" src="https://pe… |
|
6730 <p><img alt="Hosted by OpenShine" src="https://www.openshine.co… |
|
6731 <p>Philippe <a href="https://gitlab.gnome.org/federico/gnome-cl… |
|
6732 <p>Alberto made the syntax for per-instance <a href="https://gi… |
|
6733 more ergonomic, and then made that code <a href="https://gitlab.gnome… |
|
6734 <p>Martin improved our <a href="https://gitlab.gnome.org/federi… |
|
6735 <code>snake_case</code> for code generation.</p> |
|
6736 <p>Daniel added initial support for <a href="https://gitlab.gno… |
|
6737 This is not finished yet …</p></summary><content type="html"><… |
|
6738 <p>Philippe <a href="https://gitlab.gnome.org/federico/gnome-cl… |
|
6739 <p>Alberto made the syntax for per-instance <a href="https://gi… |
|
6740 more ergonomic, and then made that code <a href="https://gitlab.gnome… |
|
6741 <p>Martin improved our <a href="https://gitlab.gnome.org/federi… |
|
6742 <code>snake_case</code> for code generation.</p> |
|
6743 <p>Daniel added initial support for <a href="https://gitlab.gno… |
|
6744 This is not finished yet, but the initial parser and code generation |
|
6745 is done.</p> |
|
6746 <p>Guillaume turned <a href="https://github.com/gtk-rs/gir">… |
|
6747 <a href="http://gtk-rs.org/">gtk-rs</a>, from a binary into … |
|
6748 us have all the GObject Introspection information for parent classes |
|
6749 at compilation time.</p> |
|
6750 <p>Antoni has been working on a tricky problem. <a href="https… |
|
6751 bitfields</a> do not get reconstructed correctly from the |
|
6752 GObject Introspection information — <a href="https://github.com/rus… |
|
6753 bitfields yet</a>. This has two implications. |
|
6754 First, we lose some of the original struct fields in the generated |
|
6755 bindings. Second, the sizes of the generated structs are not the |
|
6756 same as the original C structs, so <code>g_type_register_static()&… |
|
6757 complains that one is trying to register an invalid class.</p> |
|
6758 <p>Yesterday we got as far as reading the <a href="http://refsp… |
|
6759 ABI manuals to see what the hell C compilers are supposed to do for |
|
6760 laying out structs with bitfields. Most likely, we will have a |
|
6761 temporary fix in <a href="https://github.com/gtk-rs/gir">gir</a… |
|
6762 structs with the same layout as the C ones, with padding in place of |
|
6763 the space for bitfields. Later we can remove this when rustc gets |
|
6764 support for C bitfields.</p> |
|
6765 <p>I've been working on support for GObject interfaces. The basic |
|
6766 parsing is done; I'm about to refactor the code generation so I can |
|
6767 reuse the parts that fill vtables from classes.</p> |
|
6768 <p>Yesterday we went to the Madrid Rust Meetup, a regular meeting … |
|
6769 rustaceans here. Martin talked about WebRender; I talked about |
|
6770 refactoring C to port it to Rust, and then Alex talked about Rust's |
|
6771 plans for 2018. Fun times.</p> |
|
6772 <p><img alt="Sponsored by the GNOME Foundation" src="https://pe… |
|
6773 <p><img alt="Hosted by OpenShine" src="https://www.openshine.co… |
|
6774 <p>[arm]: </p></content><category term="misc"></category><ca… |
|
6775 The <a href="https://www.openshine.com/">OpenShine</a> folks… |
|
6776 seventh floor of a building by the <a href="https://www.openstreetmap… |
|
6777 roundabout</a>.</p> |
|
6778 <p>I am very, very thankful that this time everyone seems to be wo… |
|
6779 on developing <a href="https://gitlab.gnome.org/federico/gnome-class"… |
|
6780 The <a href="https://www.openshine.com/">OpenShine</a> folks… |
|
6781 seventh floor of a building by the <a href="https://www.openstreetmap… |
|
6782 roundabout</a>.</p> |
|
6783 <p>I am very, very thankful that this time everyone seems to be wo… |
|
6784 on developing <a href="https://gitlab.gnome.org/federico/gnome-class"… |
|
6785 more brainpower is definitely welcome — all the indirection, type |
|
6786 conversion, GObject obscurity, and procedural macro shenanigans |
|
6787 definitely take a toll on oneself.</p> |
|
6788 <h1>Gnome-class internals</h1> |
|
6789 <p><img alt="Gnome-class internals on the whiteboard" src="http… |
|
6790 <p>I explained how gnome-class works to the rest of the hackfest |
|
6791 attendees. I've been writing a document on <a href="https://federico… |
|
6792 internals</a>, so the whiteboard was a whirlwind tour through |
|
6793 it.</p> |
|
6794 <h1>Error messages from the compiler</h1> |
|
6795 <p>Antoni Boucher, the author of <a href="http://relm.ml/">r… |
|
6796 asynchronous widgets with an Elm-like model), explained to me how relm |
|
6797 manages to present good error messages from the Rust compiler, when |
|
6798 the user's code has mistakes. Right now this is in a very bad state |
|
6799 in gnome-class: user errors within the invocation of the procedural |
|
6800 macro get shown by the compiler as errors <em>at</em> the ma… |
|
6801 don't get line number information that is meaningful.</p> |
|
6802 <p>For a large part of the day we tried to refactor bits of gnome-… |
|
6803 to do something similar. It is very slightly better now, but this |
|
6804 really requires me to sit down calmly, at home, and to fully |
|
6805 understand how relm does it and what changes are needed in the <a hre… |
|
6806 parser crate to make it easy to present good errors.</p> |
|
6807 <p>I think I'll continue this work at home, as there is a lot of s… |
|
6808 code to understand: the combinator parsers in <a href="https://githu… |
|
6809 handling scheme in <a href="http://relm.ml/">relm</a>, and t… |
|
6810 <h1>Further work during the hackfest</h1> |
|
6811 <p>Other people working on gnome-class are adding support for GObj… |
|
6812 properties, inheritance from non-Rust classes, and improving the |
|
6813 ergonomics of class-private structures.</p> |
|
6814 <p>I think I'll stop working on error messages for now, and focus … |
|
6815 on either supporting GTypeInterfaces, or completing support for type |
|
6816 conversions for methods and signals.</p> |
|
6817 <h1>Other happenings in Rust</h1> |
|
6818 <p>Paolo Borelli has been <a href="https://gitlab.gnome.org/GNO… |
|
6819 This is the big structure that holds all the CSS state for SVG |
|
6820 elements. This is very meticulous work, and I'm thankful that Paolo |
|
6821 is paying good attention to it. Soon we will have all the style |
|
6822 machinery for librsvg in Rust, which will make it easier to use the |
|
6823 <a href="https://crates.io/crates/selectors">selectors crate from … |
|
6824 latter is unmaintained.</p> |
|
6825 <h1>Food</h1> |
|
6826 <p><img alt="Food in Madrid" src="https://people.gnome.org/~fed… |
|
6827 <p>Ah, Spanish food. We have been enjoying cheese, jamón, tortil… |
|
6828 pimientos, oxtail stews, natillas, café con leche...</p> |
|
6829 <h1>Thanks</h1> |
|
6830 <p>Thanks to <a href="https://www.openshine.com/">OpenShine&… |
|
6831 Foundation for sponsoring my travel. And thanks for Alberto Ruiz for |
|
6832 putting me up in his house!</p> |
|
6833 <p><img alt="Sponsored by the GNOME Foundation" src="https://pe… |
|
6834 properties from C to Rust. Many properties have symbolic values:</p&… |
|
6835 <div class="highlight"><pre><span></span><cod… |
|
6836 |
|
6837 stroke-linecap: butt | round | square | inherit |
|
6838 |
|
6839 fill-rule: nonzero | evenodd | inherit |
|
6840 </code></pre></div> |
|
6841 |
|
6842 <p><code>StrokeLinejoin</code> is the first property t… |
|
6843 write a …</p></summary><content type="html"><p>I have star… |
|
6844 properties from C to Rust. Many properties have symbolic values:</p&… |
|
6845 <div class="highlight"><pre><span></span><cod… |
|
6846 |
|
6847 stroke-linecap: butt | round | square | inherit |
|
6848 |
|
6849 fill-rule: nonzero | evenodd | inherit |
|
6850 </code></pre></div> |
|
6851 |
|
6852 <p><code>StrokeLinejoin</code> is the first property t… |
|
6853 write a little bunch of machinery to allow CSS properties to be kept |
|
6854 in Rust-space instead of the main C structure that holds them |
|
6855 (upcoming blog post about that). But for now, I just want to show how |
|
6856 this boiled down to a macro after refactoring.</p> |
|
6857 <h1>First cut at the code</h1> |
|
6858 <p>The <code>stroke-linejoin</code> property can have … |
|
6859 <code>bevel</code>, or <code>inherit</code>. He… |
|
6860 and the conventional machinery which librsvg uses to parse property valu… |
|
6861 <div class="highlight"><pre><span></span><cod… |
|
6862 <span class="k">pub</span><span class="w"> </span&g… |
|
6863 <span class="w"> </span><span class="n">Miter</s… |
|
6864 <span class="w"> </span><span class="n">Round</s… |
|
6865 <span class="w"> </span><span class="n">Bevel</s… |
|
6866 <span class="w"> </span><span class="n">Inherit<… |
|
6867 <span class="p">}</span><span class="w"></span> |
|
6868 |
|
6869 <span class="k">impl</span><span class="w"> </span&… |
|
6870 <span class="w"> </span><span class="k">type</sp… |
|
6871 <span class="w"> </span><span class="k">type</sp… |
|
6872 |
|
6873 <span class="w"> </span><span class="k">fn</span… |
|
6874 <span class="w"> </span><span class="k">match&l… |
|
6875 <span class="w"> </span><span class="s">&am… |
|
6876 <span class="w"> </span><span class="s">&am… |
|
6877 <span class="w"> </span><span class="s">&am… |
|
6878 <span class="w"> </span><span class="s">&am… |
|
6879 <span class="w"> </span><span class="n">_&l… |
|
6880 <span class="w"> </span><span class="p">}</s… |
|
6881 <span class="w"> </span><span class="p">}</span&… |
|
6882 <span class="p">}</span><span class="w"></span> |
|
6883 </code></pre></div> |
|
6884 |
|
6885 <p>We <code>match</code> the allowed string values and… |
|
6886 big deal, right?</p> |
|
6887 <p>Properties also have a default value. For example, the SVG spe… |
|
6888 that if a shape doesn't have a <code>stroke-linejoin</code> … |
|
6889 it will use <code>miter</code> by default. Let's implement … |
|
6890 <div class="highlight"><pre><span></span><cod… |
|
6891 <span class="w"> </span><span class="k">fn</span… |
|
6892 <span class="w"> </span><span class="n">StrokeL… |
|
6893 <span class="w"> </span><span class="p">}</span&… |
|
6894 <span class="p">}</span><span class="w"></span> |
|
6895 </code></pre></div> |
|
6896 |
|
6897 <p>So far, we have three things:</p> |
|
6898 <ul> |
|
6899 <li>An enum definition for the property's possible values.</li&… |
|
6900 <li><code>impl Parse</code> so we can parse the proper… |
|
6901 <li><code>impl Default</code> so the property knows it… |
|
6902 </ul> |
|
6903 <h1>Where things got repetitive</h1> |
|
6904 <p>The next property I ported was <code>stroke-linecap</c… |
|
6905 following values:</p> |
|
6906 <div class="highlight"><pre><span></span><cod… |
|
6907 <span class="k">pub</span><span class="w"> </span&g… |
|
6908 <span class="w"> </span><span class="n">Butt</sp… |
|
6909 <span class="w"> </span><span class="n">Round</s… |
|
6910 <span class="w"> </span><span class="n">Square</… |
|
6911 <span class="w"> </span><span class="n">Inherit<… |
|
6912 <span class="p">}</span><span class="w"></span> |
|
6913 </code></pre></div> |
|
6914 |
|
6915 <p>This is similar in shape to the <code>StrokeLinejoin</… |
|
6916 it's just different names.</p> |
|
6917 <p>The parsing has exactly the same shape, and just different valu… |
|
6918 <div class="highlight"><pre><span></span><cod… |
|
6919 <span class="w"> </span><span class="k">type</sp… |
|
6920 <span class="w"> </span><span class="k">type</sp… |
|
6921 |
|
6922 <span class="w"> </span><span class="k">fn</span… |
|
6923 <span class="w"> </span><span class="k">match&l… |
|
6924 <span class="w"> </span><span class="s">&am… |
|
6925 <span class="w"> </span><span class="s">&am… |
|
6926 <span class="w"> </span><span class="s">&am… |
|
6927 <span class="w"> </span><span class="s">&am… |
|
6928 |
|
6929 <span class="w"> </span><span class="n">_&l… |
|
6930 <span class="w"> </span><span class="p">}</s… |
|
6931 <span class="w"> </span><span class="p">}</span&… |
|
6932 <span class="p">}</span><span class="w"></span> |
|
6933 </code></pre></div> |
|
6934 |
|
6935 <p>Same thing with the default:</p> |
|
6936 <div class="highlight"><pre><span></span><cod… |
|
6937 <span class="w"> </span><span class="k">fn</span… |
|
6938 <span class="w"> </span><span class="n">StrokeL… |
|
6939 <span class="w"> </span><span class="p">}</span&… |
|
6940 <span class="p">}</span><span class="w"></span> |
|
6941 </code></pre></div> |
|
6942 |
|
6943 <p>Yes, the SVG spec has</p> |
|
6944 <div class="highlight"><pre><span></span><cod… |
|
6945 </code></pre></div> |
|
6946 |
|
6947 <p>somewhere in it, much to the delight of the 12-year old in me.&… |
|
6948 <h1>Refactoring to a macro</h1> |
|
6949 <p>Here I wanted to define a <code>make_ident_property!()<… |
|
6950 get invoked like this:</p> |
|
6951 <div class="highlight"><pre><span></span><cod… |
|
6952 <span class="w"> </span><span class="n">StrokeLinej… |
|
6953 <span class="w"> </span><span class="n">default<… |
|
6954 |
|
6955 <span class="w"> </span><span class="s">&quot;m… |
|
6956 <span class="w"> </span><span class="s">&quot;r… |
|
6957 <span class="w"> </span><span class="s">&quot;b… |
|
6958 <span class="w"> </span><span class="s">&quot;i… |
|
6959 <span class="p">);</span><span class="w"></span> |
|
6960 </code></pre></div> |
|
6961 |
|
6962 <p>It's called <code>make_ident_property</code> becaus… |
|
6963 definition from simple string identifiers. It has the name of the |
|
6964 property (<code>StrokeLinejoin</code>), a <code>defaul… |
|
6965 elements, one for each possible value.</p> |
|
6966 <p>In Rust-speak, the macro's basic pattern is like this:</p> |
|
6967 <div class="highlight"><pre><span></span><cod… |
|
6968 <span class="w"> </span><span class="p">(</span&… |
|
6969 <span class="w"> </span><span class="n">default<… |
|
6970 <span class="w"> </span><span class="cp">$($str_pr… |
|
6971 <span class="w"> </span><span class="p">)</span&… |
|
6972 <span class="w"> </span><span class="o">..</… |
|
6973 <span class="w"> </span><span class="p">};</span… |
|
6974 <span class="p">}</span><span class="w"></span> |
|
6975 </code></pre></div> |
|
6976 |
|
6977 <p>Let's dissect that pattern:</p> |
|
6978 <div class="highlight"><pre><span></span><cod… |
|
6979 <span class="w"> </span><span class="p">(</span&… |
|
6980 <span class="c1">// ^^^^^^^^^^^^ will match an identifier and pu… |
|
6981 |
|
6982 <span class="w"> </span><span class="n">default<… |
|
6983 <span class="c1">// ^^^^^^^^^^^^^^^ will match an ident… |
|
6984 <span class="c1">// ^^^^^^^^ arbitrary text</span> |
|
6985 |
|
6986 <span class="w"> </span><span class="cp">$($str_pr… |
|
6987 <span class="w"> </span><span class… |
|
6988 <span class="c1">// ^^ start of repetition ^^ end … |
|
6989 |
|
6990 <span class="w"> </span><span class="p">)</span&… |
|
6991 <span class="w"> </span><span class="o">..</… |
|
6992 <span class="w"> </span><span class="p">};</span… |
|
6993 <span class="p">}</span><span class="w"></span> |
|
6994 </code></pre></div> |
|
6995 |
|
6996 <p>For example, saying "<code>$foo: ident</code>" in a… |
|
6997 compiler will expect an identifier, and bind it to <code>$foo</… |
|
6998 macro's definition.</p> |
|
6999 <p>Similarly, an <code>expr</code> means that the comp… |
|
7000 look for an expression — in this case, we want one of the string |
|
7001 values.</p> |
|
7002 <p>In a macro pattern, anything that is not a binding is just arbi… |
|
7003 text which must appear in the macro's invocation. This is how we can |
|
7004 create a little syntax of our own within the macro: the "<code>de… |
|
7005 part, and the "<code>=&gt;</code>" inside each string/sy… |
|
7006 <p>Finally, macro patterns allow repetition. Anything within <… |
|
7007 indicates repetition. Here, <code>$(...)+</code> indicates … |
|
7008 compiler must match one or more of the repeating elements.</p> |
|
7009 <p>I pasted the duplicated code, and substituted the actual symbol… |
|
7010 for the macro's bindings:</p> |
|
7011 <div class="highlight"><pre><span></span><cod… |
|
7012 <span class="w"> </span><span class="p">(</span&… |
|
7013 <span class="w"> </span><span class="n">default<… |
|
7014 <span class="w"> </span><span class="cp">$($str_pr… |
|
7015 <span class="w"> </span><span class="p">)</span&… |
|
7016 <span class="w"> </span><span class="cp">#[deri… |
|
7017 <span class="w"> </span><span class="k">pub<… |
|
7018 <span class="w"> </span><span class="cp">$(… |
|
7019 <span class="c1">// ^^^^^^^^^^^^^ this is how we invoke a… |
|
7020 |
|
7021 <span class="w"> </span><span class="p">}</s… |
|
7022 |
|
7023 <span class="w"> </span><span class="k">impl<… |
|
7024 <span class="w"> </span><span class="k">fn&… |
|
7025 <span class="w"> </span><span class="cp"&g… |
|
7026 <span class="c1">// ^^^^^^^^^^^^^^^ construct an enum… |
|
7027 |
|
7028 <span class="w"> </span><span class="p">}&l… |
|
7029 <span class="w"> </span><span class="p">}</s… |
|
7030 |
|
7031 <span class="w"> </span><span class="k">impl<… |
|
7032 <span class="w"> </span><span class="k">typ… |
|
7033 <span class="w"> </span><span class="k">typ… |
|
7034 |
|
7035 <span class="w"> </span><span class="k">fn&… |
|
7036 <span class="w"> </span><span class="k">… |
|
7037 <span class="w"> </span><span class="c… |
|
7038 <span class="c1">// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^… |
|
7039 |
|
7040 <span class="w"> </span><span class="n… |
|
7041 <span class="w"> </span><span class="p">… |
|
7042 <span class="w"> </span><span class="p">}&l… |
|
7043 <span class="w"> </span><span class="p">}</s… |
|
7044 <span class="w"> </span><span class="p">};</span… |
|
7045 <span class="p">}</span><span class="w"></span> |
|
7046 </code></pre></div> |
|
7047 |
|
7048 <h1>Getting rid of duplicated code</h1> |
|
7049 <p>Now we have a macro that we can call to define new properties. |
|
7050 Librsvg now has this, which is much more readable than all the code |
|
7051 written by hand:</p> |
|
7052 <div class="highlight"><pre><span></span><cod… |
|
7053 <span class="w"> </span><span class="n">StrokeLinej… |
|
7054 <span class="w"> </span><span class="n">default<… |
|
7055 |
|
7056 <span class="w"> </span><span class="s">&quot;m… |
|
7057 <span class="w"> </span><span class="s">&quot;r… |
|
7058 <span class="w"> </span><span class="s">&quot;b… |
|
7059 <span class="w"> </span><span class="s">&quot;i… |
|
7060 <span class="p">);</span><span class="w"></span> |
|
7061 |
|
7062 <span class="n">make_ident_property</span><span class="o"… |
|
7063 <span class="w"> </span><span class="n">StrokeLinec… |
|
7064 <span class="w"> </span><span class="n">default<… |
|
7065 |
|
7066 <span class="w"> </span><span class="s">&quot;b… |
|
7067 <span class="w"> </span><span class="s">&quot;r… |
|
7068 <span class="w"> </span><span class="s">&quot;s… |
|
7069 <span class="w"> </span><span class="s">&quot;i… |
|
7070 <span class="p">);</span><span class="w"></span> |
|
7071 |
|
7072 <span class="n">make_ident_property</span><span class="o"… |
|
7073 <span class="w"> </span><span class="n">FillRule<… |
|
7074 <span class="w"> </span><span class="n">default<… |
|
7075 |
|
7076 <span class="w"> </span><span class="s">&quot;n… |
|
7077 <span class="w"> </span><span class="s">&quot;e… |
|
7078 <span class="w"> </span><span class="s">&quot;i… |
|
7079 <span class="p">);</span><span class="w"></span> |
|
7080 </code></pre></div> |
|
7081 |
|
7082 <p>Etcetera. It's now easy to port similar symbol-based propertie… |
|
7083 C to Rust.</p> |
|
7084 <p>Eventually I'll need to refactor all the crap that deals with |
|
7085 inheritable properties, but that's for another time.</p> |
|
7086 <h1>Conclusion and references</h1> |
|
7087 <p>Rust macros are very powerful to refactor repetitive code like … |
|
7088 <p><a href="https://doc.rust-lang.org/book/second-edition/appen… |
|
7089 has an introductory appendix to macros, and <a href="https://danielke… |
|
7090 Macros</a> is a |
|
7091 fantastic resource that really dives into what you can do.</p></co… |
|
7092 push some commits, the CI pipelines build the code and presumably run |
|
7093 the test suite, and later you can know if this succeeded of failed.</… |
|
7094 <p>But by the time something fails, the broken code is already in … |
|
7095 public repository.</p> |
|
7096 <p>The …</p></summary><content type="html"><p>Gitlab… |
|
7097 push some commits, the CI pipelines build the code and presumably run |
|
7098 the test suite, and later you can know if this succeeded of failed.</… |
|
7099 <p>But by the time something fails, the broken code is already in … |
|
7100 public repository.</p> |
|
7101 <p>The Rust community uses Bors, a bot that prevents this from hap… |
|
7102 <ul> |
|
7103 <li> |
|
7104 <p>You push some commits and submit a merge request.</p> |
|
7105 </li> |
|
7106 <li> |
|
7107 <p>A human looks at your merge request; they may tell you to make |
|
7108 changes, or they may tell Bors that your request is approved for |
|
7109 merging.</p> |
|
7110 </li> |
|
7111 <li> |
|
7112 <p>Bors looks for approved merge requests. It merges each into a |
|
7113 <em>temporary branch</em> and waits for the CI pipeline to… |
|
7114 CI passes, Bors automatically merges to master. If CI fails, Bors |
|
7115 annotates the merge request with the failure, <strong>and the ma… |
|
7116 repository stays working</strong>.</p> |
|
7117 </li> |
|
7118 </ul> |
|
7119 <p>Bors also tells you if the mainline has moved forward and there… |
|
7120 merge conflict. In that case you need to do a rebase yourself; the |
|
7121 repository stays working in the meantime.</p> |
|
7122 <p>This leads to a very fair, very transparent process for contrib… |
|
7123 and for maintainers. For all the details, watch <a href="https://www… |
|
7124 presentation on Rust's community |
|
7125 automation</a> |
|
7126 (<a href="http://edunham.net/2016/09/27/rust_s_community_automation.h… |
|
7127 <p>For a description of where Bors came from, read <a href="htt… |
|
7128 blog</a>.</p> |
|
7129 <p><a href="https://github.com/graydon/bors">Bors</a> … |
|
7130 <a href="https://github.com/servo/homu">Homu</a> and it is w… |
|
7131 use currently. However, Homu depends on Github.</p> |
|
7132 <p>I just found out that there is a <a href="https://github.com… |
|
7133 Gitlab</a>. Would anyone care |
|
7134 to set it up?</p> |
|
7135 <p><strong>Update:</strong> <a href="https://a.weir… |
|
7136 <a href="https://octodon.social/@graydon/99719514193737493">people… |
|
7137 suggested porting <a href="https://bors.tech/">Bors-ng</a> t… |
|
7138 <a href="https://a.weirder.earth/@bb010g/99719537971696863">for sc… |
|
7139 reasons</a>.</p></content><category term="misc"></category><… |
|
7140 Summer, both for <a href="https://www.outreachy.org/">Outreachy<… |
|
7141 <h1>Librsvg projects</h1> |
|
7142 <p><strong><em>Project:</em></strong> <… |
|
7143 <p>Currently librsvg implements SVG filter effects in C. These ar… |
|
7144 image processing filters like Gaussian blur, matrix convolution, |
|
7145 Porter-Duff alpha …</p></summary><content type="html"><p>I… |
|
7146 Summer, both for <a href="https://www.outreachy.org/">Outreachy<… |
|
7147 <h1>Librsvg projects</h1> |
|
7148 <p><strong><em>Project:</em></strong> <… |
|
7149 <p>Currently librsvg implements SVG filter effects in C. These ar… |
|
7150 image processing filters like Gaussian blur, matrix convolution, |
|
7151 Porter-Duff alpha compositing, etc.</p> |
|
7152 <p>There are <a href="https://gitlab.gnome.org/GNOME/librsvg/mi… |
|
7153 <ul> |
|
7154 <li> |
|
7155 <p>Split the single <code>rsvg-filter.c</code> into mu… |
|
7156 easier to port each one individually.</p> |
|
7157 </li> |
|
7158 <li> |
|
7159 <p>Figure out the common infrasctructure: <code>RsvgFilter&l… |
|
7160 <code>RsvgFilterPrimitive</code>. All the filter use thes… |
|
7161 intermediate results when processing SVG elements.</p> |
|
7162 </li> |
|
7163 <li> |
|
7164 <p>Experiment with the correct Rust abstractions to process images |
|
7165 pixel-by-pixel. We would like to omit per-pixel bounds checks on |
|
7166 array accesses. The <a href="https://crates.io/crates/image">im… |
|
7167 traits for pixels. WebKit's implementation of SVG filters also has |
|
7168 interesting abstractions for things like the need for a sliding |
|
7169 window with edge handling for Gaussian blurs.</p> |
|
7170 </li> |
|
7171 <li> |
|
7172 <p>Ensure that our current filters code is actually working. Not … |
|
7173 of the official SVG test suite's tests are in place right now for |
|
7174 the filter effects; it is likely that some of our implementation is |
|
7175 broken.</p> |
|
7176 </li> |
|
7177 </ul> |
|
7178 <p>For this project, it will be especially helpful to have a little |
|
7179 background in image processing. You don't need to be an expert; just |
|
7180 to have done some pixel crunching at some point. You need to be able |
|
7181 to read C and write Rust.</p> |
|
7182 <p><strong><em>Project:</em></strong> <… |
|
7183 <p>Librsvg uses an very simplistic algorithm for CSS cascading. I… |
|
7184 libcroco to parse CSS style data; libcroco is unmaintained and rather |
|
7185 prone to exploits. I want to use Servo's selectors crate to do the |
|
7186 cascading; we already use the rust-cssparser crate as a tokenizer for |
|
7187 basic CSS properties.</p> |
|
7188 <ul> |
|
7189 <li> |
|
7190 <p>For each node in its DOM tree, librsvg's <code>Node</c… |
|
7191 <code>Vec&lt;&gt;</code> of children. <a href=… |
|
7192 sibling and the first/last children instead</a>. This is the |
|
7193 data structure that rust-selectors prefers. The Kuchiki crate has |
|
7194 an example implementation; borrowing some patterns from there could |
|
7195 also help us simplify our reference counting for nodes.</p> |
|
7196 </li> |
|
7197 <li> |
|
7198 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/issues/223"&… |
|
7199 big <code>RsvgState</code> struct which holds the CSS stat… |
|
7200 is easy to port this to Rust; it's more interesting to gradually |
|
7201 move it to a scheme like Servo's, with a distinction between |
|
7202 specified/computed/used values for each CSS property.</p> |
|
7203 </li> |
|
7204 </ul> |
|
7205 <p>For this project, it will be helpful to know a bit of how CSS w… |
|
7206 Definitely be comfortable with Rust concepts like ownership and |
|
7207 borrowing. You don't need to be an expert, but if you are going |
|
7208 through the "fighting the borrow checker" stage, you'll have a harder |
|
7209 time with this. Or it may be what lets you grow out of it! You need |
|
7210 to be able to read C and write Rust.</p> |
|
7211 <p><strong><em>Bugs for newcomers:</em></stro… |
|
7212 to librsvg</a>. Some of these are in the Rust part, some |
|
7213 in the C part, some in both &mdash; take your pick!</p> |
|
7214 <h1>Projects for gnome-class</h1> |
|
7215 <p><a href="https://gitlab.gnome.org/federico/gnome-class">G… |
|
7216 GObject implementations in Rust. Or at least that's the intention |
|
7217 &mdash; the project is in early development. The code is so new that |
|
7218 <a href="https://gitlab.gnome.org/federico/gnome-class/issues">pra… |
|
7219 nature.</p> |
|
7220 <p>Gnome-class works like a little compiler. This is from one of … |
|
7221 examples; note the call to <code>gobject_gen!</code> in ther… |
|
7222 <div class="highlight"><pre><span></span><cod… |
|
7223 <span class="w"> </span><span class="n">val</spa… |
|
7224 <span class="p">}</span><span class="w"></span> |
|
7225 |
|
7226 <span class="k">impl</span><span class="w"> </span&… |
|
7227 <span class="w"> </span><span class="k">fn</span… |
|
7228 <span class="w"> </span><span class="n">Signale… |
|
7229 <span class="w"> </span><span class="n">val… |
|
7230 <span class="w"> </span><span class="p">}</s… |
|
7231 <span class="w"> </span><span class="p">}</span&… |
|
7232 <span class="p">}</span><span class="w"></span> |
|
7233 |
|
7234 <span class="n">gobject_gen</span><span class="o">!<… |
|
7235 <span class="w"> </span><span class="n">class</s… |
|
7236 <span class="w"> </span><span class="k">type<… |
|
7237 <span class="w"> </span><span class="p">}</span&… |
|
7238 |
|
7239 <span class="w"> </span><span class="k">impl</sp… |
|
7240 <span class="w"> </span><span class="n">signal&… |
|
7241 |
|
7242 <span class="w"> </span><span class="k">fn</… |
|
7243 <span class="w"> </span><span class="kd">le… |
|
7244 <span class="w"> </span><span class="n">pri… |
|
7245 <span class="w"> </span><span class="bp">se… |
|
7246 <span class="w"> </span><span class="p">}</s… |
|
7247 <span class="w"> </span><span class="p">}</span&… |
|
7248 <span class="p">}</span><span class="w"></span> |
|
7249 </code></pre></div> |
|
7250 |
|
7251 <p>Gnome-class implements this <code>gobject_gen!</code&g… |
|
7252 <ol> |
|
7253 <li> |
|
7254 <p>First we parse the code inside the macro using the <code>… |
|
7255 This is a crate that lets you parse Rust source code from the |
|
7256 <code>TokenStream</code> that the compiler hands to implemen… |
|
7257 macros. You give a <code>TokenStream</code> to <code>… |
|
7258 structs that represent function definitions, <code>impl</code&g… |
|
7259 expressions, etc. From this parsing stage we build an Abstract Syntax |
|
7260 Tree (AST) that closely matches the structure of the code that the |
|
7261 user wrote.</p> |
|
7262 </li> |
|
7263 <li> |
|
7264 <p>Second, we take the AST and convert it to higher-level concepts, |
|
7265 while verifying that the code is semantically valid. For example, we |
|
7266 build up a <code>Class</code> structure for each defined GOb… |
|
7267 annotate it with the methods and signals that the user defined for it. |
|
7268 This stage is the High-level Internal Representation (HIR).</p> |
|
7269 </li> |
|
7270 <li> |
|
7271 <p>Third, we generate Rust code from the validated HIR. For each |
|
7272 class, we write out the boilerplate needed to register it against the |
|
7273 GObject type system. For each virtual method we write a trampoline to |
|
7274 let the C code call into the Rust implementation, and then write out |
|
7275 the actual Rust impl that the user wrote. For each signal, we |
|
7276 register it against the GObjectClass, and write the appropriate |
|
7277 trampolines both to invoke the signal's default handler and any Rust |
|
7278 callbacks for signal handlers.</p> |
|
7279 </li> |
|
7280 </ol> |
|
7281 <p>For this project, you definitely need to have written GObject c… |
|
7282 C in the past. You don't need to know the GObject internals; just |
|
7283 know that there are things like type registration, signal creation, |
|
7284 argument marshalling, etc.</p> |
|
7285 <p>You don't need to know about compiler internals.</p> |
|
7286 <p>You don't need to have written Rust procedural macros; you can … |
|
7287 as you go. The code has enough infrastructure right now that you can |
|
7288 cut&amp;paste useful bits to get started with new features. You sho… |
|
7289 definitely be comfortable with the Rust borrow checker and simple |
|
7290 lifetimes &mdash; again, you can cut&amp;paste useful code alrea… |
|
7291 I'm happy to help with those.</p> |
|
7292 <p>This project demands a little patience. Working on the impleme… |
|
7293 of procedural macros is not the smoothest experience right now (one |
|
7294 needs to examine generated code carefully, and play some tricks with |
|
7295 the compiler to debug things), but it's getting better very fast.</p&… |
|
7296 <h1>How to apply as an intern</h1> |
|
7297 <p><a href="https://www.outreachy.org/apply/">Details for Ou… |
|
7298 <p><a href="https://wiki.gnome.org/Outreach/SummerOfCode/Studen… |
|
7299 in GNOME, and in particular, it's what librsvg uses to render all |
|
7300 SVGs.</p> |
|
7301 <p>My immediate problem with Cairo is that it explodes when called… |
|
7302 floating-point coordinates that fall outside the range that its |
|
7303 internal fixed-point numbers can …</p></summary><content type="h… |
|
7304 in GNOME, and in particular, it's what librsvg uses to render all |
|
7305 SVGs.</p> |
|
7306 <p>My immediate problem with Cairo is that it explodes when called… |
|
7307 floating-point coordinates that fall outside the range that its |
|
7308 internal fixed-point numbers can represent. There is no validation of |
|
7309 incoming data, so the polygon intersector ends up with data that makes |
|
7310 no sense, and it crashes.</p> |
|
7311 <p>I've been studying how Cairo converts from floating-point to its |
|
7312 fixed-point representation, and it's a nifty little algorithm. So I |
|
7313 thought, no problem, I'll add validation, see how to represent the |
|
7314 error state internally in Cairo, and see if clients are happy with |
|
7315 getting back a <code>cairo_t</code> in an error state.</p… |
|
7316 <p>Cairo has a very thorough test suite... <strong><em>… |
|
7317 is documented to be very hard to pass fully for all rendering |
|
7318 backends. This is understandable, as there may be bugs in X servers |
|
7319 or OpenGL implementations and such. But for the basic, software-only, |
|
7320 in-memory image backend, Cairo should 100% pass its test suite all the |
|
7321 time. This is not the case right now; in my tree, for all the tests |
|
7322 of the image backend I get</p> |
|
7323 <div class="highlight"><pre><span></span><cod… |
|
7324 </code></pre></div> |
|
7325 |
|
7326 <p>I have been looking at test failures to see what needs fixing. … |
|
7327 reference images just need to be regenerated: there have been minor |
|
7328 changes in font rendering that broke the reference tests. Some |
|
7329 others have small differences in rendering gradients - not noticeable |
|
7330 by eye, just by diff tools.</p> |
|
7331 <p>But some tests, I have no idea what changed that made them brea… |
|
7332 <p>Cairo's git repository is accessible through [cgit.freedesktop.… |
|
7333 As far as I know there is no continuous integration infrastructure to |
|
7334 ensure that tests keep passing.</p> |
|
7335 <h1>Adding minimal continuous testing</h1> |
|
7336 <p>I've set up a <a href="https://gitlab.com/federicomenaquinte… |
|
7337 already has a <a href="https://bugs.freedesktop.org/show_bug.cgi?id=1… |
|
7338 invalid <code>free()</code></a>, and some regenerated … |
|
7339 <p>The repository <a href="https://gitlab.com/federicomenaquint… |
|
7340 pipeline</a> on every commit. The test artifacts can then be |
|
7341 downloaded when the test suite fails. Right now it is only testing |
|
7342 the image backend, for in-memory software rendering.</p> |
|
7343 <h1>Initial bugs</h1> |
|
7344 <p>I've started reporting <a href="https://gitlab.com/federicom… |
|
7345 tests that fail. These should really be in Cairo's Bugzilla, but for |
|
7346 now Gitlab makes it much easier to include test images directly in the |
|
7347 bug descriptions, so that they are easier to browse. Read on.</p> |
|
7348 <h1>Would you like to help?</h1> |
|
7349 <p>A lot of projects use Cairo. We owe it to ourselves to have a … |
|
7350 with a test suite that doesn't break. Getting to that point requires |
|
7351 several things:</p> |
|
7352 <ul> |
|
7353 <li>Fixing current failures in the image backend.</li> |
|
7354 <li>Setting up the CI infrastructure to be able to test other back… |
|
7355 <li>Fixing failures in the other backends.</li> |
|
7356 </ul> |
|
7357 <p>If you have experience with Cairo, please take a look at the &l… |
|
7358 You can see the <a href="https://gitlab.com/federicomenaquintero/cair… |
|
7359 suite in the same fashion on your machine.</p> |
|
7360 <p>I think we can make use of modern infrastructure like gitlab and |
|
7361 continuous integration to improve Cairo quickly. Currently it suffers |
|
7362 from lack of attention and hostile tools. Help us out if you can!</p… |
|
7363 version 0.12. <code>syn</code> is a somewhat esoteric crate… |
|
7364 parse Rust code... from a stream of tokens... from within the |
|
7365 implementation of a procedural macro. Gnome-class implements a |
|
7366 mini-language inside your own Rust …</p></summary><content type=… |
|
7367 version 0.12. <code>syn</code> is a somewhat esoteric crate… |
|
7368 parse Rust code... from a stream of tokens... from within the |
|
7369 implementation of a procedural macro. Gnome-class implements a |
|
7370 mini-language inside your own Rust code, and so it needs to parse |
|
7371 Rust!</p> |
|
7372 <p>The API of <code>syn</code> has changed <em>a… |
|
7373 ass — but the new API seems on the road to stabilization, and is nicer |
|
7374 indeed.</p> |
|
7375 <p>Here is a quick list of things I had to change in gnome-class to |
|
7376 upgrade its version of <code>syn</code>.</p> |
|
7377 <p>There is no <code>extern crate synom</code> anymore… |
|
7378 <div class="highlight"><pre><span></span><cod… |
|
7379 </code></pre></div> |
|
7380 |
|
7381 <p><code>SynomBuffer</code> is now <code>TokenBu… |
|
7382 <div class="highlight"><pre><span></span><cod… |
|
7383 </code></pre></div> |
|
7384 |
|
7385 <p><code>PResult</code>, the result of <code>Syn… |
|
7386 arguments reversed:</p> |
|
7387 <div class="highlight"><pre><span></span><cod… |
|
7388 <span class="o">+</span><span class="w"> </span>… |
|
7389 |
|
7390 <span class="c1">// therefore:</span> |
|
7391 |
|
7392 <span class="k">impl</span><span class="w"> </span&… |
|
7393 |
|
7394 <span class="kd">let</span><span class="w"> </span&… |
|
7395 </code></pre></div> |
|
7396 |
|
7397 <p>The language tokens like <code>synom::tokens::Amp</cod… |
|
7398 <code>synom::tokens::Type</code>, are easier to use now. Th… |
|
7399 macro which you can use in type definitions, instead of having to |
|
7400 remember the particular name of each token type:</p> |
|
7401 <div class="highlight"><pre><span></span><cod… |
|
7402 |
|
7403 <span class="n">synom</span>::<span class="n">tokens&l… |
|
7404 </code></pre></div> |
|
7405 |
|
7406 <p>And for the corresponding values when matching:</p> |
|
7407 <div class="highlight"><pre><span></span><cod… |
|
7408 |
|
7409 <span class="n">syn</span><span class="o">!</span&g… |
|
7410 </code></pre></div> |
|
7411 |
|
7412 <p>And to instantiate them for quoting/spanning:</p> |
|
7413 <div class="highlight"><pre><span></span><cod… |
|
7414 <span class="o">+</span><span class="w"> </span… |
|
7415 </code></pre></div> |
|
7416 |
|
7417 <p>(OK, that one wasn't nicer after all.)</p> |
|
7418 <p>To the get string for an <code>Ident</code>:</p&… |
|
7419 <div class="highlight"><pre><span></span><cod… |
|
7420 </code></pre></div> |
|
7421 |
|
7422 <p>There is no <code>Delimited</code> anymore; instead… |
|
7423 struct. My diff has this:</p> |
|
7424 <div class="highlight"><pre><span></span><cod… |
|
7425 + inputs: parens!(syn!(Punctuated&lt;MyThing, Token!(,)&gt;)) &… |
|
7426 </code></pre></div> |
|
7427 |
|
7428 <p>There is no <code>syn::Mutability</code> anymore; n… |
|
7429 basically</p> |
|
7430 <div class="highlight"><pre><span></span><cod… |
|
7431 </code></pre></div> |
|
7432 |
|
7433 <p>which I guess lets you refer to the span of the original <co… |
|
7434 if you need.</p> |
|
7435 <p>Some things changed names:</p> |
|
7436 <div class="highlight"><pre><span></span><cod… |
|
7437 |
|
7438 <span class="n">PatIdent</span><span class="w"> </s… |
|
7439 <span class="w"> </span><span class="n">mode</sp… |
|
7440 <span class="w"> </s… |
|
7441 <span class="w"> </span><span class="n">ident</s… |
|
7442 <span class="w"> </span><span class="n">subpat</… |
|
7443 <span class="w"> </span><span class="n">at_token<… |
|
7444 <span class="p">}</span><span class="w"></span> |
|
7445 |
|
7446 <span class="n">TypeParen</span><span class="p">.</… |
|
7447 </code></pre></div> |
|
7448 |
|
7449 <p>(I don't know everything that changed names; gnome-class doesn'… |
|
7450 all the syn types yet; these are just the ones I've run into.)</p> |
|
7451 <p>This new <code>syn</code> is much better at acknowl… |
|
7452 macro hygiene. The <a href="https://github.com/dtolnay/syn/tree/mast… |
|
7453 it shows how to properly span generated code vs. original code, so |
|
7454 compiler error messages are nice. I <a href="https://github.com/rust… |
|
7455 macro hygiene</a> at some point.</p></content><category term… |
|
7456 librsvg's continous integration (CI) pipeline. Take a look at this |
|
7457 beauty:</p> |
|
7458 <p><img alt="Continuous integration pipeline" src="https://peop… |
|
7459 <p>On every push, we run the <strong>Test</strong> sta… |
|
7460 on a Fedora container that runs "<code>make check</code>" an… |
|
7461 test suite passes.</p> |
|
7462 <p>We have a …</p></summary><content type="html"><p>… |
|
7463 librsvg's continous integration (CI) pipeline. Take a look at this |
|
7464 beauty:</p> |
|
7465 <p><img alt="Continuous integration pipeline" src="https://peop… |
|
7466 <p>On every push, we run the <strong>Test</strong> sta… |
|
7467 on a Fedora container that runs "<code>make check</code>" an… |
|
7468 test suite passes.</p> |
|
7469 <p>We have a <strong>Lint</strong> stage which can be … |
|
7470 clippy</code> to get Rust lints (check the style of Rust idioms), … |
|
7471 fmt</code> to check indentation and code style and such.</p> |
|
7472 <p>We have a <strong>Distro_test</strong> stage which … |
|
7473 weekly, using Gitlab's <em>Schedules</em> feature, to check … |
|
7474 pass on three major Linux distros. Recently we had trouble with |
|
7475 different rendering due to differences in Freetype versions, which |
|
7476 broke the tests (<em>ahem, likely because </em><em>I&l… |
|
7477 Freetype in a while and distros were already using a newer one</em>… |
|
7478 distro tests are intended to catch that.</p> |
|
7479 <p>Finally, we have a <strong>Rustc_test</strong> stag… |
|
7480 librsvg depends on have different minimum versions for the Rust |
|
7481 compiler. These tests are intended to show when updating a dependency |
|
7482 changes the minimum Rust version on which librsvg would compile. We |
|
7483 don't have a policy yet for "how far from $newest" we should always |
|
7484 work on, and it would be good to get input from distros on this. I |
|
7485 think these Rust tests will be scheduled weekly as well.</p> |
|
7486 <p>Jordan has been experimenting with the pipeline's stages and the |
|
7487 distro-specific idiosyncrasies for each build. This pipeline depends |
|
7488 on some <a href="https://gitlab.com/alatiera/librsvg-oci-images">c… |
|
7489 librsvg's dependencies installed. These images are built weekly in |
|
7490 <code>gitlab.com</code>, so every week <code>gitlab.gn… |
|
7491 librsvg's CI pipelines. Once image registries are enabled in |
|
7492 <code>gitlab.gnome.org</code>, we should be able to regenera… |
|
7493 images locally without depending on an external service.</p> |
|
7494 <p>With the pre-built images, and caching of Rust artifacts, Jorda… |
|
7495 able to <strong>reduce the time for the "test on every commit" bui… |
|
7496 around 20 minutes, to little under 4 minutes in the current |
|
7497 iteration. This will get even faster if the builds start using ccache |
|
7498 and parallel builds from GNU make.</p> |
|
7499 <p>Currently we have a problem in that <a href="https://gitlab.… |
|
7500 builds</a>, and haven't had a chance to investigate the root |
|
7501 cause. Hopefully we can add 32-bit jobs to the CI pipeline to catch |
|
7502 this breakage as soon as possible.</p> |
|
7503 <p>Having all these container images built for the CI infrastructu… |
|
7504 means that it will be easy for people to <strong>set up a developm… |
|
7505 environment</strong> for librsvg, even though we have <a href="… |
|
7506 now</a> thanks to Jordan. I haven't investigated setting up a |
|
7507 Flatpak-based environment; this would be nice to have as well.</p>… |
|
7508 <code>rsvg-rs</code> is the Rust binding to librsvg. Like t… |
|
7509 it gets generated from a pre-built <a href="https://people.gnome.org/… |
|
7510 <p>It would be nice for librsvg to provide the Rust binding by its… |
|
7511 that librsvg's own internal tools can be …</p></summary><content… |
|
7512 <code>rsvg-rs</code> is the Rust binding to librsvg. Like t… |
|
7513 it gets generated from a pre-built <a href="https://people.gnome.org/… |
|
7514 <p>It would be nice for librsvg to provide the Rust binding by its… |
|
7515 that librsvg's own internal tools can be implemented in Rust — |
|
7516 currently all the tests are done in C, as are the <code>rsvg-conve… |
|
7517 <code>rsvg-view-3(1)</code> programs.</p> |
|
7518 <p>There are some implications for how <code>rsvg-rs</cod… |
|
7519 For librsvg's internal consumption, the binding can be built from the |
|
7520 <code>Rsvg-2.0.gir</code> file that gets built out of the ma… |
|
7521 for public consumption of <code>rsvg-rs</code>, when it is b… |
|
7522 crate and built by Cargo, that <code>Rsvg-2.0.gir</code> nee… |
|
7523 built and available: it wouldn't be appropriate for Cargo to build |
|
7524 librsvg and the <code>.gir</code> file itself.</p> |
|
7525 <p>If this sort of thing interests you, <a href="https://gitlab… |
|
7526 seems like it would be easier to just port some major parts from C to |
|
7527 Rust than to just add accessors for them. Also, more and more of the |
|
7528 meat of the library is in Rust now.</p> |
|
7529 <p>I'm …</p></summary><content type="html"><p>Librsv… |
|
7530 seems like it would be easier to just port some major parts from C to |
|
7531 Rust than to just add accessors for them. Also, more and more of the |
|
7532 meat of the library is in Rust now.</p> |
|
7533 <p>I'm switching back and forth a lot between C and Rust these day… |
|
7534 C feels very, very primitive these days.</p> |
|
7535 <h1>A sort of elegy to C</h1> |
|
7536 <p>I fell in love with the C language about 24 years ago. I learn… |
|
7537 basics of it by reading a Spanish translation of <a href="https://en.… |
|
7538 Language by K&amp;R</a> second edition. I had been using Turb… |
|
7539 before in a reasonably low-level fashion, with pointers and manual |
|
7540 memory allocation, and C felt refreshing and empowering.</p> |
|
7541 <p>K&amp;R is a great book for its <em>style of writing&… |
|
7542 programming. This little book even taught you how to implement a |
|
7543 simple <code>malloc()</code>/<code>free()</code>… |
|
7544 low-level constructs that seemed part of the language could be |
|
7545 implemented in the language itself!</p> |
|
7546 <p>I got good at C over the following years. It is a small langua… |
|
7547 with a small standard library. It was probably the perfect language |
|
7548 to implement Unix kernels in 20,000 lines of code or so.</p> |
|
7549 <p>The GIMP and GTK+ taught me how to do fancy object orientation … |
|
7550 GNOME taught me how to maintain large-scale software in C. 20,000 |
|
7551 lines of C code started to seem like a project one could more or less |
|
7552 fully understand in a few weeks.</p> |
|
7553 <p>But our code bases are not that small anymore. Our software no… |
|
7554 <em>huge</em> expectations on the features that are availabl… |
|
7555 language's standard library.</p> |
|
7556 <h2>Some good experiences with C</h2> |
|
7557 <p>Reading the POV-Ray code source code for the first time and lea… |
|
7558 how to do object orientation and inheritance in C.</p> |
|
7559 <p>Reading the GTK+ source code and learning a C style that was le… |
|
7560 maintainable, and clean.</p> |
|
7561 <p>Reading SIOD's source code, then the early Guile sources, and s… |
|
7562 how a Scheme interpreter can be written in C.</p> |
|
7563 <p>Writing the initial versions of Eye of Gnome and fine-tuning the |
|
7564 microtile rendering.</p> |
|
7565 <h2>Some bad experiences with C</h2> |
|
7566 <p>In the Evolution team, when everything was crashing. We had to… |
|
7567 Solaris machine just to be able to buy Purify; there was no Valgrind |
|
7568 back then.</p> |
|
7569 <p>Debugging gnome-vfs threading deadlocks.</p> |
|
7570 <p>Debugging Mesa and getting nowhere.</p> |
|
7571 <p>Taking over the intial versions of Nautilus-share and seeing th… |
|
7572 never <code>free()</code>d anything.</p> |
|
7573 <p>Trying to refactor code where I had no idea about the memory |
|
7574 management strategy.</p> |
|
7575 <p>Trying to turn code into a library when it is full of global va… |
|
7576 and no functions are <code>static</code>.</p> |
|
7577 <p>But anyway — let's get on with things in Rust I miss in C.<… |
|
7578 <h1>Automatic resource management</h1> |
|
7579 <p>One of the first blog posts I read about Rust was "<a href="… |
|
7580 having to close a socket</a>". Rust borrows C++'s ideas about |
|
7581 <a href="http://wiki.c2.com/?ResourceAcquisitionIsInitialization">… |
|
7582 adds in the single-ownership principle for values, and gives you |
|
7583 automatic, deterministic resource management in a very neat package.<… |
|
7584 <ul> |
|
7585 <li> |
|
7586 <p>Automatic: you don't <code>free()</code> by hand. … |
|
7587 files get closed, mutexes get unlocked when they go out of scope. |
|
7588 If you are wrapping an external resource, you just implement the |
|
7589 <a href="https://doc.rust-lang.org/book/second-edition/ch15-03-drop… |
|
7590 like part of the language since you don't have to babysit its |
|
7591 lifetime by hand.</p> |
|
7592 </li> |
|
7593 <li> |
|
7594 <p>Deterministic: resources get created (memory allocated, initial… |
|
7595 files opened, etc.), and they get destroyed when they go out of |
|
7596 scope. There is no garbage collection: things really get terminated |
|
7597 when you close a brace. You start to see your program's data |
|
7598 lifetimes as a tree of function calls.</p> |
|
7599 </li> |
|
7600 </ul> |
|
7601 <p>After forgetting to free/close/destroy C objects all the time, … |
|
7602 worse, figuring out where code that I didn't write forgot to do those |
|
7603 things (or did them <em>twice</em>, incorrectly)... I don't … |
|
7604 again.</p> |
|
7605 <h1>Generics</h1> |
|
7606 <p><code>Vec&lt;T&gt;</code> really is a vecto… |
|
7607 It's not an array of pointers to individually allocated objects. It |
|
7608 gets compiled <em>specifically</em> to code that can only ha… |
|
7609 type <code>T</code>.</p> |
|
7610 <p>After writing many janky macros in C to do similar things... I … |
|
7611 want to do it again.</p> |
|
7612 <h1>Traits are not just interfaces</h1> |
|
7613 <p><a href="https://doc.rust-lang.org/book/second-edition/ch17-… |
|
7614 has traits, which at first seem like Java interfaces — an easy way to |
|
7615 do dynamic dispatch, so that if an object implements <code>Drawabl… |
|
7616 you can assume it has a <code>draw()</code> method.</p> |
|
7617 <p>However, traits are more powerful than that.</p> |
|
7618 <h2>Associated types</h2> |
|
7619 <p><a href="https://doc.rust-lang.org/book/second-edition/ch19-… |
|
7620 provies the <code>Iterator</code> trait which you can implem… |
|
7621 <div class="highlight"><pre><span></span><cod… |
|
7622 <span class="w"> </span><span class="k">type</sp… |
|
7623 <span class="w"> </span><span class="k">fn</span… |
|
7624 <span class="p">}</span><span class="w"></span> |
|
7625 </code></pre></div> |
|
7626 |
|
7627 <p>This means that whenever you implement <code>Iterator<… |
|
7628 object, you also have to specify an <code>Item</code> type f… |
|
7629 will be produced. If you call <code>next()</code> and there… |
|
7630 you'll get back a <code>Some(YourElementType)</code>. When … |
|
7631 out of items, it will return <code>None</code>.</p> |
|
7632 <p>Associated types can refer to <em>other</em> traits… |
|
7633 <p>For example, in Rust, you can use <code>for</code> … |
|
7634 implements the <code>IntoIterator</code> trait:</p> |
|
7635 <div class="highlight"><pre><span></span><cod… |
|
7636 <span class="w"> </span><span class="sd">/// The ty… |
|
7637 <span class="w"> </span><span class="k">type</sp… |
|
7638 |
|
7639 <span class="w"> </span><span class="sd">/// Which … |
|
7640 <span class="w"> </span><span class="k">type</sp… |
|
7641 |
|
7642 <span class="w"> </span><span class="k">fn</span… |
|
7643 <span class="p">}</span><span class="w"></span> |
|
7644 </code></pre></div> |
|
7645 |
|
7646 <p>When implementing this trait, you must provide both the type of… |
|
7647 <code>Item</code> which your iterator will produce, and <… |
|
7648 type that implements <code>Iterator</code> and that holds yo… |
|
7649 <p>This way you can build webs of types that refer to each other. … |
|
7650 can have a trait that says, "I can do foo and bar, but only if you |
|
7651 give me a type that can do this and that".</p> |
|
7652 <h1>Slices</h1> |
|
7653 <p>I already posted about <a href="https://people.gnome.org/~fe… |
|
7654 how this is a pain in the ass once you get used to having them.</p> |
|
7655 <h1>Modern tooling for dependency management</h1> |
|
7656 <p>Instead of </p> |
|
7657 <ul> |
|
7658 <li>Having to invoke <code>pkg-config</code> by hand o… |
|
7659 <li>Wrangling include paths for header files...</li> |
|
7660 <li>... and library files.</li> |
|
7661 <li>And basically depending on the user to ensure that the correct |
|
7662 versions of libraries are installed,</li> |
|
7663 </ul> |
|
7664 <p>You write a <code>Cargo.toml</code> file which list… |
|
7665 your dependencies. These get downloaded from a well-known location, |
|
7666 or from elsewhere if you specify.</p> |
|
7667 <p>You don't have to fight dependencies. It just works when you &… |
|
7668 <h1>Tests</h1> |
|
7669 <p>C makes it very hard to have unit tests for several reasons:<… |
|
7670 <ul> |
|
7671 <li> |
|
7672 <p>Internal functions are often <code>static</code>. … |
|
7673 called outside of the source file that defined them. A test program |
|
7674 either has to <code>#include</code> the source file where … |
|
7675 live, or use <code>#ifdef</code>s to remove the <code&g… |
|
7676 </li> |
|
7677 <li> |
|
7678 <p>You have to write Makefile-related hackery to link the test pro… |
|
7679 to only part of your code's dependencies, or to only part of the |
|
7680 rest of your code.</p> |
|
7681 </li> |
|
7682 <li> |
|
7683 <p>You have to pick a testing framework. You have to register tes… |
|
7684 against the testing framework. You have to <em>learn</em>… |
|
7685 framework.</p> |
|
7686 </li> |
|
7687 </ul> |
|
7688 <p>In Rust you write</p> |
|
7689 <div class="highlight"><pre><span></span><cod… |
|
7690 <span class="k">fn</span> <span class="nf">test_that_f… |
|
7691 <span class="w"> </span><span class="fm">assert!<… |
|
7692 <span class="p">}</span><span class="w"></span> |
|
7693 </code></pre></div> |
|
7694 |
|
7695 <p>anywhere in your program or library, and when you type <code… |
|
7696 IT JUST FUCKING WORKS. That code only gets linked into the |
|
7697 test binary. You don't have to compile anything twice by hand, or |
|
7698 write Makefile hackery, or figure out how to extract internal |
|
7699 functions for testing.</p> |
|
7700 <p>This is a very killer feature for me.</p> |
|
7701 <h1>Documentation, with tests</h1> |
|
7702 <p>Rust generates documentation from comments in Markdown syntax. … |
|
7703 in the docs <em>gets run as tests</em>. You can illustrate … |
|
7704 used <em>and</em> test it at the same time:</p> |
|
7705 <div class="highlight"><pre><span></span><cod… |
|
7706 <span class="sd">///</span> |
|
7707 <span class="sd">/// ```</span> |
|
7708 <span class="sd">/// assert_eq!(multiply_by_two(5), 10);</span&… |
|
7709 <span class="sd">/// ```</span> |
|
7710 <span class="k">fn</span> <span class="nf">multiply_by… |
|
7711 <span class="w"> </span><span class="n">x</span&… |
|
7712 <span class="p">}</span><span class="w"></span> |
|
7713 </code></pre></div> |
|
7714 |
|
7715 <p>Your example code <em>gets run as tests</em> to ens… |
|
7716 documentation stays up to date with the actual code.</p> |
|
7717 <p><strong>Update 2018/Feb/23:</strong> QuietMisdreavu… |
|
7718 doctests into runnable code |
|
7719 internally</a>. |
|
7720 This is high-grade magic and thoroughly interesting.</p> |
|
7721 <h1>Hygienic macros</h1> |
|
7722 <p>Rust has hygienic macros that avoid all of C's problems with th… |
|
7723 macros that inadvertently shadow identifiers in the code. You don't |
|
7724 need to write macros where every symbol has to be in parentheses for |
|
7725 <code>max(5 + 3, 4)</code> to work correctly.</p> |
|
7726 <h1>No automatic coercions</h1> |
|
7727 <p>All the bugs in C that result from inadvertently converting an … |
|
7728 to a <code>short</code> or <code>char</code> or … |
|
7729 to explicitly convert.</p> |
|
7730 <h1>No integer overflow</h1> |
|
7731 <p>Enough said.</p> |
|
7732 <h1>Generally, no undefined behavior in safe Rust</h1> |
|
7733 <p>In Rust, it is considered a bug in the language if something wr… |
|
7734 in "safe Rust" (what you would be allowed to write outside <code>u… |
|
7735 blocks) results in undefined behavior. You can shift-right a negative |
|
7736 integer and it will do exactly what you expect.</p> |
|
7737 <h1>Pattern matching</h1> |
|
7738 <p>You know how <code>gcc</code> warns you if you <… |
|
7739 handle all values? That's like a little baby.</p> |
|
7740 <p>Rust has <a href="https://doc.rust-lang.org/book/second-edit… |
|
7741 trick for enums inside a <code>match()</code> expression. I… |
|
7742 destructuring so you can return multiple values from a function:</p&g… |
|
7743 <div class="highlight"><pre><span></span><cod… |
|
7744 <span class="w"> </span><span class="k">pub</spa… |
|
7745 <span class="p">}</span><span class="w"></span> |
|
7746 |
|
7747 <span class="kd">let</span><span class="w"> </span&… |
|
7748 <span class="kd">let</span><span class="w"> </span&… |
|
7749 </code></pre></div> |
|
7750 |
|
7751 <p>You can <code>match()</code> on strings. YOU CAN M… |
|
7752 <div class="highlight"><pre><span></span><cod… |
|
7753 |
|
7754 <span class="k">match</span><span class="w"> </span… |
|
7755 <span class="w"> </span><span class="s">&quot;r… |
|
7756 <span class="w"> </span><span class="s">&quot;g… |
|
7757 <span class="w"> </span><span class="n">_</span&… |
|
7758 <span class="p">}</span><span class="w"></span> |
|
7759 </code></pre></div> |
|
7760 |
|
7761 <p>You know how this is illegible?</p> |
|
7762 <div class="highlight"><pre><span></span><cod… |
|
7763 </code></pre></div> |
|
7764 |
|
7765 <p>How about this instead, with pattern matching on function argum… |
|
7766 <div class="highlight"><pre><span></span><cod… |
|
7767 <span class="k">pub</span><span class="w"> </span&g… |
|
7768 <span class="k">pub</span><span class="w"> </span&g… |
|
7769 |
|
7770 <span class="k">fn</span> <span class="nf">my_func<… |
|
7771 <span class="w"> </span><span class="n">Frob… |
|
7772 <span class="w"> </span><span class="n">Bazi… |
|
7773 <span class="w"> </span><span class="k">if</span… |
|
7774 <span class="w"> </span><span class="o">..</… |
|
7775 <span class="w"> </span><span class="p">}</span&… |
|
7776 |
|
7777 <span class="w"> </span><span class="k">if</span… |
|
7778 <span class="w"> </span><span class="o">..</… |
|
7779 <span class="w"> </span><span class="p">}</span&… |
|
7780 <span class="p">}</span><span class="w"></span> |
|
7781 |
|
7782 <span class="o">..</span><span class="p">.</span>… |
|
7783 |
|
7784 <span class="n">my_func</span><span class="p">(</sp… |
|
7785 </code></pre></div> |
|
7786 |
|
7787 <h1>Standard, useful error handling</h1> |
|
7788 <p>I've talked at length about this. No more returning a boolean … |
|
7789 extra explanation for an error, no ignoring errors inadvertently, no |
|
7790 exception handling with nonlocal jumps.</p> |
|
7791 <h1>#[derive(Debug)]</h1> |
|
7792 <p>If you write a new type (say, a struct with a ton of fields), y… |
|
7793 <code>#[derive(Debug)]</code> and Rust will know how to auto… |
|
7794 type's contents for debug output. You no longer have to write a |
|
7795 special function that you must call in gdb by hand just to examine a |
|
7796 custom type.</p> |
|
7797 <h1>Closures</h1> |
|
7798 <p>No more passing function pointers and a <code>user_data&l… |
|
7799 <h1>Conclusion</h1> |
|
7800 <p>I haven't done the "<a href="https://doc.rust-lang.org/book/… |
|
7801 compiler is able to prevent data races in threaded code. I imagine it |
|
7802 being a game-changer for people who write concurrent code on an |
|
7803 everyday basis.</p> |
|
7804 <p>C is an old language with primitive constructs and primitive to… |
|
7805 It was a good language for small uniprocessor Unix kernels that ran in |
|
7806 trusted, academic environments. It's no longer a good language for |
|
7807 the software of today.</p> |
|
7808 <p>Rust is not easy to learn, but I think it is completely worth i… |
|
7809 It's hard because it demands a lot from your understanding of the code |
|
7810 you want to write. I think it's one of those languages that make you |
|
7811 a better programmer and that let you tackle more ambitious problems.<… |
|
7812 a program that actually has a <code>main()</code> function.&… |
|
7813 <p>My experience with Rust so far has been threefold:</p> |
|
7814 <ul> |
|
7815 <li> |
|
7816 <p>Porting chunks of C to Rust for librsvg - this is all work on |
|
7817 librsvg's internals and no users are exposed …</p></li>&… |
|
7818 a program that actually has a <code>main()</code> function.&… |
|
7819 <p>My experience with Rust so far has been threefold:</p> |
|
7820 <ul> |
|
7821 <li> |
|
7822 <p>Porting chunks of C to Rust for librsvg - this is all work on |
|
7823 librsvg's internals and no users are exposed to it directly.</p> |
|
7824 </li> |
|
7825 <li> |
|
7826 <p>Working on <a href="https://github.com/nikomatsakis/gnome-cl… |
|
7827 to generate GObject boilerplate from Rust. This feels like working |
|
7828 on the edge of the exotic; it is something that runs <em>in</… |
|
7829 compiler and spits code on behalf of the programmer.</p> |
|
7830 </li> |
|
7831 <li> |
|
7832 <p>A few patches to the <a href="http://gtk-rs.org">gtk-rs&l… |
|
7833 internals, or something that feels library-like.</p> |
|
7834 </li> |
|
7835 </ul> |
|
7836 <p>But other than toy programs to test things, I haven't written a |
|
7837 stand-alone tool until <a href="https://people.gnome.org/~federico/bl… |
|
7838 to just <em>run the thing</em> instead of waiting for other … |
|
7839 code to use it!</p> |
|
7840 <h1>Parsing command-line arguments</h1> |
|
7841 <p>There are quite a few Rust crates ("libraries") to parse comman… |
|
7842 arguments. I read about <a href="https://docs.rs/structopt-derive/0.… |
|
7843 blog</a>; structopt lets you define a <code>struct</code&… |
|
7844 your command-line options, and then you annotate the fields in that |
|
7845 <code>struct</code> to indicate how they should be parsed fr… |
|
7846 It works via Rust's procedural macros. Internally it generates stuff |
|
7847 for the <a href="https://docs.rs/clap/2.29.2/clap/">clap</a>… |
|
7848 command-line options.</p> |
|
7849 <p>And it is quite pleasant! This is basically all I needed to do… |
|
7850 <div class="highlight"><pre><span></span><cod… |
|
7851 <span class="cp">#[structopt(name = </span><span class="s… |
|
7852 <span class="k">struct</span> <span class="nc">Opt<… |
|
7853 <span class="w"> </span><span class="cp">#[structop… |
|
7854 <span class="cp"> long = </span><span cla… |
|
7855 <span class="cp"> help = </span><span cla… |
|
7856 <span class="cp"> default_value = </span><… |
|
7857 <span class="w"> </span><span class="n">sleep_secs&… |
|
7858 |
|
7859 <span class="w"> </span><span class="cp">#[structop… |
|
7860 <span class="cp"> long = </span><span cla… |
|
7861 <span class="cp"> help = </span><span cla… |
|
7862 <span class="cp"> default_value = </span><… |
|
7863 <span class="w"> </span><span class="n">num_parse&l… |
|
7864 |
|
7865 <span class="w"> </span><span class="cp">#[structop… |
|
7866 <span class="cp"> long = </span><span cla… |
|
7867 <span class="cp"> help = </span><span cla… |
|
7868 <span class="cp"> default_value = </span><… |
|
7869 <span class="w"> </span><span class="n">num_render&… |
|
7870 |
|
7871 <span class="w"> </span><span class="cp">#[structop… |
|
7872 <span class="cp"> help = </span><span clas… |
|
7873 <span class="w"> </span><span class="n">render_to_p… |
|
7874 |
|
7875 <span class="w"> </span><span class="cp">#[structop… |
|
7876 <span class="cp"> parse(from_os_str))]</span>… |
|
7877 <span class="w"> </span><span class="n">inputs</… |
|
7878 <span class="p">}</span><span class="w"></span> |
|
7879 |
|
7880 <span class="k">fn</span> <span class="nf">main</sp… |
|
7881 <span class="w"> </span><span class="kd">let</sp… |
|
7882 |
|
7883 <span class="w"> </span><span class="k">if</span… |
|
7884 <span class="w"> </span><span class="fm">eprint… |
|
7885 <span class="w"> </span><span class="n">process… |
|
7886 <span class="w"> </span><span class="p">}</span&… |
|
7887 |
|
7888 <span class="w"> </span><span class="o">..</span… |
|
7889 <span class="p">}</span><span class="w"></span> |
|
7890 </code></pre></div> |
|
7891 |
|
7892 <p>Each field in the <code>Opt</code> struct above cor… |
|
7893 argument; each field has annotations for <code>structopt</code&… |
|
7894 appropriate code to parse each option. For example, the |
|
7895 <code>render_to_pixbuf</code> field has a long option name c… |
|
7896 that field will be set to <code>true</code> if the <code&… |
|
7897 to rsvg-bench.</p> |
|
7898 <h1>Handling errors</h1> |
|
7899 <p>Command-line programs generally have the luxury of being able t… |
|
7900 exit as soon as they encounter an error.</p> |
|
7901 <p>In C this is a bit cumbersome since you need to deal with <e… |
|
7902 place that may return an error, find out what to print, and call |
|
7903 <code>exit(1)</code> by hand or something. If you miss a si… |
|
7904 error is returned, your program will keep running with an inconsistent |
|
7905 state.</p> |
|
7906 <p>In languages with exception handling, it's a bit easier - a sma… |
|
7907 script can just let exceptions be thrown wherever, and if it catches |
|
7908 them at the toplevel, it can just print the exception and abort |
|
7909 gracefully. However, these nonlocal jumps make me uncomfortable; I |
|
7910 think <a href="http://joeduffyblog.com/2016/02/07/the-error-model/"&g… |
|
7911 <p>Rust makes this easy: it forces you to handle every call that m… |
|
7912 return an error, but it lets you bubble errors up easily, or handle |
|
7913 them in-place, or translate them to a higher-level error.</p> |
|
7914 <p>In the Rust world the [<code>failure</code>] crate … |
|
7915 as a convenient, modern way to handle errors.</p> |
|
7916 <p>In rsvg-bench, errors can come from several places:</p> |
|
7917 <ul> |
|
7918 <li> |
|
7919 <p>I/O errors when reading files and directories.</p> |
|
7920 </li> |
|
7921 <li> |
|
7922 <p>Errors from librsvg's parsing stage; you get a <a href="http… |
|
7923 </li> |
|
7924 <li> |
|
7925 <p>Errors from the rendering stage. This can be a Cairo error (a |
|
7926 <a href="https://www.cairographics.org/manual/cairo-Error-handling.… |
|
7927 render" from librsvg's old convenience api in C. Don't you hate it |
|
7928 when C code just gives up and returns NULL or a boolean false, |
|
7929 without any further details on <em>what</em> went wrong?&l… |
|
7930 </li> |
|
7931 </ul> |
|
7932 <p>For rsvg-bench, I just needed to be able to represent Cairo err… |
|
7933 generic rendering errors. Everything else, like an <code>io::Erro… |
|
7934 automatically wrapped by the <code>failure</code> crate's me… |
|
7935 needed to do this:</p> |
|
7936 <div class="highlight"><pre><span></span><cod… |
|
7937 <span class="cp">#[macro_use]</span><span class="w">&l… |
|
7938 <span class="k">extern</span><span class="w"> </spa… |
|
7939 |
|
7940 <span class="cp">#[derive(Debug, Fail)]</span><span class… |
|
7941 <span class="k">enum</span> <span class="nc">Processin… |
|
7942 <span class="w"> </span><span class="cp">#[fail(dis… |
|
7943 <span class="w"> </span><span class="n">CairoError&… |
|
7944 <span class="w"> </span><span class="n">status&… |
|
7945 <span class="w"> </span><span class="p">},</span… |
|
7946 |
|
7947 <span class="w"> </span><span class="cp">#[fail(dis… |
|
7948 <span class="w"> </span><span class="n">RenderingEr… |
|
7949 <span class="p">}</span><span class="w"></span> |
|
7950 </code></pre></div> |
|
7951 |
|
7952 <p>Whenever the code gets a Cairo error, I can translate it to a |
|
7953 <code>ProcessingError::CairoError</code> and bubble it up:&l… |
|
7954 <div class="highlight"><pre><span></span><cod… |
|
7955 <span class="w"> </span><span class="kd">let</sp… |
|
7956 <span class="w"> </span><span class="kd">let</sp… |
|
7957 <span class="w"> <… |
|
7958 <span class="w"> <… |
|
7959 <span class="w"> </span><span class="p">.</s… |
|
7960 |
|
7961 <span class="w"> </span><span class="o">..</span… |
|
7962 <span class="p">}</span><span class="w"></span> |
|
7963 </code></pre></div> |
|
7964 |
|
7965 <p>And when librsvg returns a "couldn't render" error, I translate… |
|
7966 to a <code>ProcessingError::RenderingError</code>:</p> |
|
7967 <div class="highlight"><pre><span></span><cod… |
|
7968 <span class="w"> </span><span class="o">..</span… |
|
7969 |
|
7970 <span class="w"> </span><span class="kd">let</sp… |
|
7971 |
|
7972 <span class="w"> </span><span class="k">if</span… |
|
7973 <span class="w"> </span><span class="nb">Ok<… |
|
7974 <span class="w"> </span><span class="p">}</span&… |
|
7975 <span class="w"> </span><span class="nb">Err<… |
|
7976 <span class="w"> </span><span class="p">}</span&… |
|
7977 <span class="p">}</span><span class="w"></span> |
|
7978 </code></pre></div> |
|
7979 |
|
7980 <p>Here, the <code>Ok()</code> case of the <code>… |
|
7981 it's just <code>()</code>, as the generated images are not s… |
|
7982 are just rendered to get some timings, not to be saved or anything.</… |
|
7983 <h1>Up to where do errors bubble?</h1> |
|
7984 <p>This is the "do everything" function:</p> |
|
7985 <div class="highlight"><pre><span></span><cod… |
|
7986 <span class="w"> </span><span class="o">..</span… |
|
7987 |
|
7988 <span class="w"> </span><span class="k">for</spa… |
|
7989 <span class="w"> </span><span class="n">process… |
|
7990 <span class="w"> </span><span class="p">}</span&… |
|
7991 |
|
7992 <span class="w"> </span><span class="nb">Ok</spa… |
|
7993 <span class="p">}</span><span class="w"></span> |
|
7994 </code></pre></div> |
|
7995 |
|
7996 <p>For each path passed in the command line, process it. The prog… |
|
7997 sees if the path corresponds to a directory, and it will scan it |
|
7998 recursively. Or if the path is an SVG file, the program will load the |
|
7999 file and render it.</p> |
|
8000 <p>Finally, <code>main()</code> just has this:</p&g… |
|
8001 <div class="highlight"><pre><span></span><cod… |
|
8002 <span class="w"> </span><span class="kd">let</sp… |
|
8003 |
|
8004 <span class="w"> </span><span class="o">..</span… |
|
8005 |
|
8006 <span class="w"> </span><span class="k">match</s… |
|
8007 <span class="w"> </span><span class="nb">Ok<… |
|
8008 <span class="w"> </span><span class="nb">Err<… |
|
8009 <span class="w"> </span><span class="fm">ep… |
|
8010 <span class="w"> </span><span class="n">pro… |
|
8011 <span class="w"> </span><span class="p">}</s… |
|
8012 <span class="w"> </span><span class="p">}</span&… |
|
8013 <span class="p">}</span><span class="w"></span> |
|
8014 </code></pre></div> |
|
8015 |
|
8016 <p>I.e. process command line arguments, run the whole thing, and p… |
|
8017 error if there was one.</p> |
|
8018 <p>I really appreciate that most places that can return an error a… |
|
8019 put a <code>?</code> for the error to bubble up. This is mu… |
|
8020 in C, where every call must have an <code>if (something_bad_happen… |
|
8021 deal_with_it; }</code> after it... and Rust won't let me get away … |
|
8022 ignoring an error, but it makes it easy to actually deal with it properl… |
|
8023 <h1>Reading an SVG file quickly</h1> |
|
8024 <p>Why, just <code>mmap()</code> it and feed it to lib… |
|
8025 This is easy in Rust:</p> |
|
8026 <div class="highlight"><pre><span></span><cod… |
|
8027 <span class="w"> </span><span class="kd">let</sp… |
|
8028 <span class="w"> </span><span class="kd">let</sp… |
|
8029 |
|
8030 <span class="w"> </span><span class="kd">let</sp… |
|
8031 |
|
8032 <span class="w"> </span><span class="kd">let</sp… |
|
8033 <span class="w"> </span><span class="o">..</span… |
|
8034 <span class="p">}</span><span class="w"></span> |
|
8035 </code></pre></div> |
|
8036 |
|
8037 <p>Many things can go wrong here:</p> |
|
8038 <ul> |
|
8039 <li><code>File::open()</code> can return an io::Error.… |
|
8040 <li><code>MmapOptions::map()</code> can return an io::… |
|
8041 system call, or from the <code>fstat(2)</code> to read the… |
|
8042 it.</li> |
|
8043 <li><code>rsvg::Handle::new_from_data()</code> can ret… |
|
8044 file.</li> |
|
8045 </ul> |
|
8046 <p>The little <code>?</code> characters after each cal… |
|
8047 mean, just give me back the result, or convert the error to a |
|
8048 <code>failure::Error</code> that can be examined later. Thi… |
|
8049 legible to me.</p> |
|
8050 <h1>Summary</h1> |
|
8051 <p>Writing command-line programs in Rust is fun! It's nice to have |
|
8052 neurotically-safe scripts that one can trust in the future.</p> |
|
8053 <p><a href="https://gitlab.gnome.org/federico/rsvg-bench">Rs… |
|
8054 compared to 2.40.20: SVGs with many <a href="https://www.w3.org/TR/SV… |
|
8055 attributes would slow it down. It was fixed in 2.42.1. We changed |
|
8056 from using a <a href="https://github.com/lalrpop/lalrpop/issues/269"&… |
|
8057 called, to <a href="https://github.com/servo/rust-cssparser">one �… |
|
8058 compared to 2.40.20: SVGs with many <a href="https://www.w3.org/TR/SV… |
|
8059 attributes would slow it down. It was fixed in 2.42.1. We changed |
|
8060 from using a <a href="https://github.com/lalrpop/lalrpop/issues/269"&… |
|
8061 called, to <a href="https://github.com/servo/rust-cssparser">one t… |
|
8062 parsing.</p> |
|
8063 <p>When I rewrote librsvg's parser for the <code>transform&l… |
|
8064 to Rust, I was just <a href="https://people.gnome.org/~federico/news-… |
|
8065 I chose <a href="https://github.com/lalrpop/lalrpop">lalrpop</a… |
|
8066 It generates big, fast parsers, like what you would need for a |
|
8067 compiler — but it compiles the tokenizer's regexes each time you call |
|
8068 the parser. This is not a problem for a compiler, where you basically |
|
8069 call the parser only once, but in librsvg, we may call it thousands of |
|
8070 times for an SVG file with thousands of objects with <code>transfo… |
|
8071 attributes.</p> |
|
8072 <p>So, for 2.42.1 I rewrote that parser using |
|
8073 <a href="https://github.com/servo/rust-cssparser">rust-cssparser&l… |
|
8074 parse CSS data; it's a simple tokenizer with an API that knows about |
|
8075 CSS's particular constructs. This is exactly the kind of data that |
|
8076 librsvg cares about. Today all of librsvg's internal parsers work |
|
8077 using rust-cssparser, or they are so simple that they can be done with |
|
8078 Rust's normal functions to split strings and such.</p> |
|
8079 <h1>Getting good timings</h1> |
|
8080 <p>Librsvg ships with <code>rsvg-convert</code>, a com… |
|
8081 render an SVG file and write the output to a PNG. While it would be |
|
8082 possible to get timings for SVG rendering by timing how long |
|
8083 <code>rsvg-convert</code> takes to run, it's a bit clunky fo… |
|
8084 startup adds noise to the timings, and it only handles one file at a |
|
8085 time.</p> |
|
8086 <p>So, I've written <a href="https://gitlab.gnome.org/federico/… |
|
8087 librsvg. I wanted a tool that:</p> |
|
8088 <ul> |
|
8089 <li> |
|
8090 <p>Is able to process many SVG images with a single command. For |
|
8091 example, this lets us answer a question like, "how long does version |
|
8092 N of librsvg take to render a directory full of SVG icons?" — which |
|
8093 is important for the performance of an application chooser.</p> |
|
8094 </li> |
|
8095 <li> |
|
8096 <p>Is able to <em>repeatedly</em> process SVG files, f… |
|
8097 SVG 1000 times in a row". This is useful to get accurate timings, |
|
8098 as a single render may only take a few microseconds and may be hard |
|
8099 to measure. It also helps with running profilers, as they will be |
|
8100 able to get more useful samples if the SVG rendering process runs |
|
8101 repeatedly for a long time.</p> |
|
8102 </li> |
|
8103 <li> |
|
8104 <p>Exercises librsvg's major code paths for parsing and rendering |
|
8105 separately. For example, librsvg uses different parts of the XML |
|
8106 parser depending on whether it is being pushed data, vs. being asked |
|
8107 to pull data from a stream. Also, we may only want to benchmark the |
|
8108 parser but not the renderer; or we may want to parse SVGs only once |
|
8109 but render them many times after that.</p> |
|
8110 </li> |
|
8111 <li> |
|
8112 <p>Is aware of librsvg's peculiarities, such as the extra pass to |
|
8113 convert a Cairo image surface to a GdkPixbuf when one uses the |
|
8114 convenience function <code>rsvg_handle_get_pixbuf()</code>… |
|
8115 </li> |
|
8116 </ul> |
|
8117 <p>Currently rsvg-bench supports all of that.</p> |
|
8118 <h1>An initial benchmark</h1> |
|
8119 <p>I ran this</p> |
|
8120 <p><code>/usr/bin/time rsvg-bench -p 1 -r 1 /usr/share/icons… |
|
8121 <p>to cause every SVG icon in <code>/usr/share/icons</cod… |
|
8122 rendered once (i.e. just render every file sequentially). I did this |
|
8123 for librsvg 2.40.20 (C only), and 2.42.{0, 1, 2} (C and Rust). There |
|
8124 are 5522 SVG files in there. The timings look like this:</p> |
|
8125 <table> |
|
8126 <thead> |
|
8127 <tr> |
|
8128 <th>version</th> |
|
8129 <th>time (sec)</th> |
|
8130 </tr> |
|
8131 </thead> |
|
8132 <tbody> |
|
8133 <tr> |
|
8134 <td>2.40.20</td> |
|
8135 <td>95.54</td> |
|
8136 </tr> |
|
8137 <tr> |
|
8138 <td>2.42.0</td> |
|
8139 <td>209.50</td> |
|
8140 </tr> |
|
8141 <tr> |
|
8142 <td>2.42.1</td> |
|
8143 <td>97.18</td> |
|
8144 </tr> |
|
8145 <tr> |
|
8146 <td>2.42.2</td> |
|
8147 <td>95.89</td> |
|
8148 </tr> |
|
8149 </tbody> |
|
8150 </table> |
|
8151 <p><img alt="Bar chart of timings" src="https://people.gnome.or… |
|
8152 <p>So, 2.42.0 was over twice as slow as the C-only version, due to… |
|
8153 parsing problems. But now, 2.42.2 is practically just as fast as the |
|
8154 C only version. What made this possible?</p> |
|
8155 <ul> |
|
8156 <li>2.40.20 - the old C-only version</li> |
|
8157 <li>2.42.0 - C + Rust, with a lalrpop parser for the <code>t… |
|
8158 <li>2.42.1 - Servo's cssparser for the <code>transform</c… |
|
8159 <li>2.42.2 - removed most C-to-Rust string copies during parsing&l… |
|
8160 </ul> |
|
8161 <p>I have started taking profiles of rsvg-bench runs with sysprof,… |
|
8162 there are some improvements worth making. Expect news soon!</p> |
|
8163 <p><a href="https://gitlab.gnome.org/federico/rsvg-bench">Rs… |
|
8164 preparation for the 2.42.1 release?</p> |
|
8165 <p>I have prepared a list of bugs which I'd like to be fixed in the |
|
8166 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues?milestone_titl… |
|
8167 I'm already working …</p></summary><content type="html"><p>… |
|
8168 preparation for the 2.42.1 release?</p> |
|
8169 <p>I have prepared a list of bugs which I'd like to be fixed in the |
|
8170 <a href="https://gitlab.gnome.org/GNOME/librsvg/issues?milestone_titl… |
|
8171 I'm already working on them.</p> |
|
8172 <p>There are two other bugs which I'd love someone to look at. Ne… |
|
8173 of these requires deep knowledge of librsvg, just some debugging and |
|
8174 code-writing:</p> |
|
8175 <ul> |
|
8176 <li> |
|
8177 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/issues/141"&… |
|
8178 the wrong fill: it's an image of a builder's trowel, and the inside |
|
8179 is filled black instead of with a nice gradient. This is the only |
|
8180 place in librsvg where a <code>cairo_surface_t</code> is c… |
|
8181 <code>GdkPixbuf</code>; this involves unpremultiplying the… |
|
8182 Maybe the relevant function is buggy?</p> |
|
8183 </li> |
|
8184 <li> |
|
8185 <p><a href="https://gitlab.gnome.org/GNOME/librsvg/issues/136"&… |
|
8186 parsed incorrectly. It is a list of CSS length values, separated by |
|
8187 commas or spaces. Currently librsvg uses a shitty parser based on |
|
8188 <code>g_strsplit()</code> only for commas; it doesn't allo… |
|
8189 space-separated list. Then, it uses <code>g_ascii_strtod()</… |
|
8190 plain numbers; it doesn't support CSS lengths generically. This |
|
8191 parser needs to be rewritten in Rust; we already have machinery |
|
8192 there to parse CSS length values properly.</p> |
|
8193 </li> |
|
8194 </ul> |
|
8195 <p>Feel free to <a href="[email protected]">contact me</… |
|
8196 bugs themselves, if you would like to work on them. I'll happily |
|
8197 guide you through the code :)</p></content><category term="misc"><… |
|
8198 Continuous Integration (CI) enabled for projects there. After every |
|
8199 commit, the CI machinery can build the project, run the tests, and |
|
8200 tell you if something goes wrong.</p> |
|
8201 <p><a href="https://mail.gnome.org/archives/desktop-devel-list/… |
|
8202 week" mail to …</p></summary><content type="html"><p>One n… |
|
8203 Continuous Integration (CI) enabled for projects there. After every |
|
8204 commit, the CI machinery can build the project, run the tests, and |
|
8205 tell you if something goes wrong.</p> |
|
8206 <p><a href="https://mail.gnome.org/archives/desktop-devel-list/… |
|
8207 week" mail to desktop-devel-list, and a link to how Nautilus |
|
8208 implements CI in Gitlab. It turns out that it's reasonably easy to |
|
8209 set up: you just create a <a href="https://docs.gitlab.com/ce/ci/yam… |
|
8210 toplevel of your project, and that has the configuration for what to |
|
8211 run on every commit.</p> |
|
8212 <p>Of course instead of reading the manual, I copied-and-pasted th… |
|
8213 from Nautilus and just changed some things in it. <a href="https://g… |
|
8214 linter</a> so you can at least check the syntax before pushing a |
|
8215 full job.</p> |
|
8216 <p>Then I read <a href="https://mail.gnome.org/archives/desktop… |
|
8217 builds its CI jobs on both Fedora and Ubuntu... and then the |
|
8218 realization hit me:</p> |
|
8219 <p><em>This lets me CI librsvg on multiple distros at once.&… |
|
8220 trouble with slight differences in fontconfig/freetype in the past, |
|
8221 and this would let me catch them early.</p> |
|
8222 <p>However, people on IRC advised against this, as <strong>w… |
|
8223 hardware</strong> to run CI on a large scale.</p> |
|
8224 <p>Linux distros have a vested interest in getting code out of gno… |
|
8225 that works well. Surely they can give us some hardware?</p></cont… |
|
8226 weeks since <a href="https://people.gnome.org/~federico/blog/librsvg-… |
|
8227 already received and merged <a href="https://gitlab.gnome.org/GNOME/l… |
|
8228 weird that Github uses "pull request" and Everyone(tm) knows the PR |
|
8229 acronym, but Gitlab uses "merge request"?)</p> |
|
8230 <h1>Notifications …</h1></summary><content type="html"><… |
|
8231 weeks since <a href="https://people.gnome.org/~federico/blog/librsvg-… |
|
8232 already received and merged <a href="https://gitlab.gnome.org/GNOME/l… |
|
8233 weird that Github uses "pull request" and Everyone(tm) knows the PR |
|
8234 acronym, but Gitlab uses "merge request"?)</p> |
|
8235 <h1>Notifications about merge requests</h1> |
|
8236 <p>One thing to note if your GNOME project has moved to Gitlab: &… |
|
8237 want to get notified of incoming merge requests</strong>, you need |
|
8238 to tell Gitlab that you want to "<strong>Watch</strong>" tha… |
|
8239 using one of the default notification settings. <a href="https://git… |
|
8240 Soriano</a> for making me aware of this.</p> |
|
8241 <h1>Notifications from Github's mirror</h1> |
|
8242 <p>The <a href="https://github.com/GNOME/">github</a> … |
|
8243 requests are <a href="https://wiki.gnome.org/Sysadmin/GitHub">auto… |
|
8244 is no way to notify the upstream maintainers when someone creates a |
|
8245 pull request in the mirror (this is super-unfriendly by default, but |
|
8246 at least submitters get notified that their PR would not be looked at |
|
8247 by anyone, by default).</p> |
|
8248 <p>If you have a Github account, you can Watch the project in ques… |
|
8249 get notified — the bot will close the pull request, but you will get |
|
8250 notified, and then you can check it by hand, review it as appropriate, |
|
8251 or redirect the submitter to gitlab.gnome.org instead.</p></conten… |
|
8252 <strong>last release</strong> in the 2.40.x series, which i… |
|
8253 immediately.</p> |
|
8254 <p>People and distros are <strong>strongly encouraged</st… |
|
8255 <a href="https://ftp.gnome.org/pub/GNOME/sources/librsvg/2.41/">li… |
|
8256 implemented in a …</p></summary><content type="html"><p>To… |
|
8257 <strong>last release</strong> in the 2.40.x series, which i… |
|
8258 immediately.</p> |
|
8259 <p>People and distros are <strong>strongly encouraged</st… |
|
8260 <a href="https://ftp.gnome.org/pub/GNOME/sources/librsvg/2.41/">li… |
|
8261 implemented in a mixture of C and Rust. It is 100% API and ABI |
|
8262 compatible with 2.40.x, so it is a drop-in replacement for it. If you |
|
8263 or your distro can compile Firefox 57, you can probably build |
|
8264 librsvg-2.41.x without problems.</p> |
|
8265 <h1>Some statistics</h1> |
|
8266 <p>Here are a few runs of <a href="https://github.com/cgag/loc"… |
|
8267 when run on librsvg. The output is trimmed by hand to only include C |
|
8268 and Rust files.</p> |
|
8269 <div class="highlight"><pre><span></span><cod… |
|
8270 <span class="nb">-------------------------------------------------… |
|
8271 <span class="c"> Language Files Lines Blank Comment … |
|
8272 <span class="nb">-------------------------------------------------… |
|
8273 <span class="c"> C 41 20972 3438 2100 1… |
|
8274 <span class="c"> C/C</span><span class="nb">++</spa… |
|
8275 </code></pre></div> |
|
8276 |
|
8277 <div class="highlight"><pre><span></span><cod… |
|
8278 <span class="nb">-------------------------------------------------… |
|
8279 <span class="c"> Language Files Lines Blank Comment … |
|
8280 <span class="nb">-------------------------------------------------… |
|
8281 <span class="c"> C 34 17253 3024 1892 1… |
|
8282 <span class="c"> C/C</span><span class="nb">++</spa… |
|
8283 <span class="c"> Rust 38 11254 1873 675 … |
|
8284 </code></pre></div> |
|
8285 |
|
8286 <div class="highlight"><pre><span></span><cod… |
|
8287 <span class="c">just &quot;real source code&quot;:</spa… |
|
8288 <span class="nb">-------------------------------------------------… |
|
8289 <span class="c"> Language Files Lines Blank Comment … |
|
8290 <span class="nb">-------------------------------------------------… |
|
8291 <span class="c"> C 34 17253 3024 1892 1… |
|
8292 <span class="c"> C/C</span><span class="nb">++</spa… |
|
8293 <span class="c"> Rust 38 9340 1513 610 … |
|
8294 </code></pre></div> |
|
8295 |
|
8296 <h2>Summary</h2> |
|
8297 <p>Not counting blank lines nor comments:</p> |
|
8298 <ul> |
|
8299 <li> |
|
8300 <p>The C-only version has 16734 lines of C code.</p> |
|
8301 </li> |
|
8302 <li> |
|
8303 <p>The C-only version has <strong>no unit tests</strong&g… |
|
8304 </li> |
|
8305 <li> |
|
8306 <p>The Rust-and-C version has 13539 lines of C code, 7217 lines of… |
|
8307 code, and 1489 lines of unit tests in Rust.</p> |
|
8308 </li> |
|
8309 </ul> |
|
8310 <p>As for the integration tests:</p> |
|
8311 <ul> |
|
8312 <li> |
|
8313 <p>The C-only version has 64 integration tests.</p> |
|
8314 </li> |
|
8315 <li> |
|
8316 <p>The Rust-and-C version has 130 integration tests.</p> |
|
8317 </li> |
|
8318 </ul> |
|
8319 <p>The Rust-and-C version supports a few more SVG features, and it… |
|
8320 LOT more robust and spec-compliant with the SVG features that were |
|
8321 supported in the C-only version.</p> |
|
8322 <p>The C sources in librsvg are shrinking steadily. It would be |
|
8323 incredibly awesome if someone could run some <code>git filter-bran… |
|
8324 with the <a href="https://github.com/cgag/loc"><code>loc<… |
|
8325 lines vs. commits over time.</p></content><category term="misc"></… |
|
8326 access it <a href="https://gitlab.gnome.org/GNOME/librsvg">here<… |
|
8327 <p>Gitlab allows workflows similar to Github: you can create an a… |
|
8328 there, fork the librsvg repository, file bug reports, create merge |
|
8329 requests... Hopefully this will make it nicer for contributors.</p&g… |
|
8330 <p>In the meantime, feel free to <a href="https://gitlab.gnome.… |
|
8331 access it <a href="https://gitlab.gnome.org/GNOME/librsvg">here<… |
|
8332 <p>Gitlab allows workflows similar to Github: you can create an a… |
|
8333 there, fork the librsvg repository, file bug reports, create merge |
|
8334 requests... Hopefully this will make it nicer for contributors.</p&g… |
|
8335 <p>In the meantime, feel free to <a href="https://gitlab.gnome.… |
|
8336 <p>This is a huge improvement for GNOME's development infrastructu… |
|
8337 Thanks to Carlos Soriano, Andrea Veri, Philip Chimento, Alberto Ruiz, |
|
8338 and all the people that made the move to Gitlab possible.</p></con… |
|
8339 C code that implements SVG's <code>&lt;text&gt;</code&g… |
|
8340 been replacing the little parsers in librsvg with Rust code.</p> |
|
8341 <p>And these days, the lack of string slices in C is bothering me … |
|
8342 lot</em>.</p> |
|
8343 <h1>What …</h1></summary><content type="html"><p>Por… |
|
8344 C code that implements SVG's <code>&lt;text&gt;</code&g… |
|
8345 been replacing the little parsers in librsvg with Rust code.</p> |
|
8346 <p>And these days, the lack of string slices in C is bothering me … |
|
8347 lot</em>.</p> |
|
8348 <h1>What if...</h1> |
|
8349 <p>It feels like it should be easy to just write something like<… |
|
8350 <div class="highlight"><pre><span></span><cod… |
|
8351 <span class="k">const</span> <span class="kt">char… |
|
8352 <span class="kt">size_t</span> <span class="n">len… |
|
8353 <span class="p">}</span> <span class="n">StringSlice&l… |
|
8354 </code></pre></div> |
|
8355 |
|
8356 <p>And then a whole family of functions. The starting point, wher… |
|
8357 slice a whole string:</p> |
|
8358 <div class="highlight"><pre><span></span><cod… |
|
8359 <span class="nf">make_slice_from_string</span> <span clas… |
|
8360 <span class="p">{</span> |
|
8361 <span class="n">StringSlice</span> <span class="n">… |
|
8362 |
|
8363 <span class="n">assert</span> <span class="p">(<… |
|
8364 |
|
8365 <span class="n">slice</span><span class="p">.</… |
|
8366 <span class="n">slice</span><span class="p">.</… |
|
8367 <span class="k">return</span> <span class="n">slic… |
|
8368 <span class="p">}</span> |
|
8369 </code></pre></div> |
|
8370 |
|
8371 <p>But that wouldn't keep track of the lifetime of the original st… |
|
8372 Okay, this is C, so you are used to keeping track of that yourself.</… |
|
8373 <p>Onwards. Substrings?</p> |
|
8374 <div class="highlight"><pre><span></span><cod… |
|
8375 <span class="nf">make_sub_slice</span><span class="p">… |
|
8376 <span class="p">{</span> |
|
8377 <span class="n">StringSlice</span> <span class="n">… |
|
8378 |
|
8379 <span class="n">assert</span> <span class="p">(<… |
|
8380 <span class="n">assert</span> <span class="p">(<… |
|
8381 <span class="cm">/* The su… |
|
8382 <span class="n">sub</span><span class="p">.</sp… |
|
8383 <span class="n">sub</span><span class="p">.</sp… |
|
8384 <span class="k">return</span> <span class="n">sub&… |
|
8385 <span class="p">}</span> |
|
8386 </code></pre></div> |
|
8387 |
|
8388 <p>Then you could write a million wrappers for <code>g_strsp… |
|
8389 friends, or equivalents to them, to give you slices instead of C |
|
8390 strings. But then:</p> |
|
8391 <ul> |
|
8392 <li> |
|
8393 <p>You have to keep track of lifetimes yourself.</p> |
|
8394 </li> |
|
8395 <li> |
|
8396 <p>You have to wrap every function that returns a plain "<code&… |
|
8397 </li> |
|
8398 <li> |
|
8399 <p>... and every function that takes a plain "<code>char *&… |
|
8400 without a length parameter, because...</p> |
|
8401 </li> |
|
8402 <li> |
|
8403 <p>You <strong>CANNOT</strong> take <code>slice.… |
|
8404 expects a plain "<code>char *</code>", because your slice… |
|
8405 nul terminator (the <code>'\0</code> byte at the end of a … |
|
8406 what kills the whole plan.</p> |
|
8407 </li> |
|
8408 </ul> |
|
8409 <p>Even if you had a helper library that implements C string slices |
|
8410 like that, you would have a mismatch every time you needed to call a C |
|
8411 function that expects a conventional C string in the form of a |
|
8412 "<code>char *</code>". <em>You need to put a nul ter… |
|
8413 only have a slice, you need to <em>allocate memory</em>, cop… |
|
8414 it, and slap a 0 byte at the end. <em>Then</em> you can pas… |
|
8415 function that expects a normal C string.</p> |
|
8416 <p>There is hacky C code that needs to pass a substring to another |
|
8417 function, so it <em>overwrites the byte after the substring with a… |
|
8418 passes the substring, and <em>overwrites the byte back</em>.… |
|
8419 horrible, and doesn't work with strings that live in read-only |
|
8420 memory. But that's the best that C lets you do.</p> |
|
8421 <p>I'm very happy with string slices in Rust, which work exactly l… |
|
8422 <code>StringSlice</code> above, but <code>&amp;str… |
|
8423 everything knows how to handle it.</p> |
|
8424 <p>The <code>glib-rs</code> crate has conversion trait… |
|
8425 slices into C, and vice-versa. We alredy saw some of those in the |
|
8426 blog post about <a href="https://people.gnome.org/~federico/blog/how-… |
|
8427 <h1>Sizes of things</h1> |
|
8428 <p>Rust uses <code>usize</code> to specify the size of… |
|
8429 integer; 32 bits on 32-bit machines, and 64 bits on 64-bit machines; |
|
8430 it's like C's <code>size_t</code>.</p> |
|
8431 <p>In the Glib/C world, we have an assortment of types to represen… |
|
8432 sizes of things:</p> |
|
8433 <ul> |
|
8434 <li> |
|
8435 <p><code>gsize</code>, the same as <code>size_t&… |
|
8436 </li> |
|
8437 <li> |
|
8438 <p><code>gssize</code>, a signed integer of the same s… |
|
8439 okay if used to represent a negative offset, and <em>really funk… |
|
8440 the Glib functions like |
|
8441 <code>g_string_new_len (const char *str, gssize len)</co… |
|
8442 means "call <code>strlen(str)</code> for me because I'm to… |
|
8443 length myself".</p> |
|
8444 </li> |
|
8445 <li> |
|
8446 <p><code>int</code> - broken, as in libxml2, but we ca… |
|
8447 64-bit machines, an <code>int</code> to specify a length m… |
|
8448 objects bigger than 2 GB.</p> |
|
8449 </li> |
|
8450 <li> |
|
8451 <p><code>long</code> - marginally better than <code… |
|
8452 of actually being the same size as <code>size_t</code>, bu… |
|
8453 Probably okay for negative offsets; problematic for sizes which |
|
8454 should really be unsigned.</p> |
|
8455 </li> |
|
8456 <li> |
|
8457 <p>etc.</p> |
|
8458 </li> |
|
8459 </ul> |
|
8460 <p>I'm not sure how old <code>size_t</code> is in the … |
|
8461 can't have been there since the beginning of time &mdash; otherwise |
|
8462 people wouldn't have been using <code>int</code> to specify … |
|
8463 called "<a href="http://www.greaterthancode.com/podcast/054-code-hosp… |
|
8464 <a href="http://www.nadiaodunayo.com/">Nadia Odunayo</a>. … |
|
8465 <p>Nadia talks about thinking of how to make people comfortable in… |
|
8466 code and in your team/organization/etc., and does it in terms of |
|
8467 thinking about host/guest relationships. Have you ever …</p></s… |
|
8468 called "<a href="http://www.greaterthancode.com/podcast/054-code-hosp… |
|
8469 <a href="http://www.nadiaodunayo.com/">Nadia Odunayo</a>. … |
|
8470 <p>Nadia talks about thinking of how to make people comfortable in… |
|
8471 code and in your team/organization/etc., and does it in terms of |
|
8472 thinking about host/guest relationships. Have you ever stayed in an |
|
8473 AirBnB where the host carefully prepares some "welcome instructions" |
|
8474 for you, or puts little notes in their apartment to orient/guide you, |
|
8475 or gives you basic guidance around their city's transportation system? |
|
8476 We can think in similar ways of how to make people comfortable with |
|
8477 code bases.</p> |
|
8478 <p>This of course hit me on so many levels, because in the past I'… |
|
8479 written about analogies between software and urbanism/architecture. |
|
8480 <a href="https://people.gnome.org/~federico/docs/software-with-qwan/i… |
|
8481 talks about Christopher Alexander's architecture/urbanism patterns in |
|
8482 the context of software, based on <a href="http://dreamsongs.com/">… |
|
8483 and <a href="http://zeta.math.utsa.edu/~yxk833/">Nikos Salingaros&… |
|
8484 process. <a href="https://people.gnome.org/~federico/blog/legacy-sys… |
|
8485 how GNOME evolved parts of its user-visible software, and makes an |
|
8486 analogy with cities that evolve over time instead of being torn down |
|
8487 and rebuilt, based on urbanism ideas by Jane Jacobs, and |
|
8488 architecture/construction ideas by Stewart Brand.</p> |
|
8489 <p>I definitely intend to do some thinking on Nadia's ideas for Co… |
|
8490 Hospitality and try to connect them with this.</p> |
|
8491 <p>In the meantime, I've just rewritten the <a href="https://gi… |
|
8492 gnome-class</a> to make it suitable as an |
|
8493 introduction to hacking there.</p></content><category term="misc">… |
|
8494 Hackfest</a>, kindly hosted at the <a href="https://kinvolk.io/… |
|
8495 This is in a <em>great</em> location, half a block away from… |
|
8496 Tor</a> station, right at the entrance of the trendy Kreuzberg |
|
8497 neighborhood — full of interesting people, incredible graffitti, and |
|
8498 good …</p></summary><content type="html"><p>Last weekend I… |
|
8499 Hackfest</a>, kindly hosted at the <a href="https://kinvolk.io/… |
|
8500 This is in a <em>great</em> location, half a block away from… |
|
8501 Tor</a> station, right at the entrance of the trendy Kreuzberg |
|
8502 neighborhood — full of interesting people, incredible graffitti, and |
|
8503 good, diverse food.</p> |
|
8504 <p><a href="https://people.gnome.org/~federico/blog/images/kott… |
|
8505 <h1>My goals for the hackfest</h1> |
|
8506 <p>Over the past weeks I had been converting <a href="https://g… |
|
8507 from the old <a href="https://github.com/nikomatsakis/lalrpop/">la… |
|
8508 Macros framework for Rust, or <code>proc-macro2</code> for s… |
|
8509 parser for the gnome-class mini-language needs to be rewritten from |
|
8510 being specified in a lalrpop grammar, to using Rust's <a href="https:… |
|
8511 crate.</p> |
|
8512 <p>Syn is a parser for Rust source code, written as a set of <a… |
|
8513 combinator parser macros. For gnome-class we want to extend the Rust |
|
8514 language with a few conveniences to be able to specify GObject |
|
8515 classes/subclasses, methods, signals, properties, interfaces, and all |
|
8516 the goodies that GObject Introspection would expect.</p> |
|
8517 <p>During the hackfest, <a href="https://github.com/alexcrichto… |
|
8518 kindly took over my baby steps in compiler writing and made everything |
|
8519 much more functional. It was invaluable to have him there to reason |
|
8520 about macro hygiene (we <em>are</em> generating an unhygieni… |
|
8521 in the quoting system, and general Rust-iness of the whole thing.</p&… |
|
8522 <p>I was also able to talk to <a href="https://coaxion.net/blog… |
|
8523 writing GObjects in Rust by hand, for GStreamer, and what sort of |
|
8524 things gnome-class could make easier. Sebastian knows GObject very |
|
8525 well, and has been doing awesome work in making it easy to derive |
|
8526 GObjects by hand in Rust, without lots of boilerplate — something with |
|
8527 which gnome-class can certainly help.</p> |
|
8528 <p>I was also looking forward to talking again with <a href="ht… |
|
8529 Gomez</a>, one of the maintainers of <a href="http://gtk-rs.org… |
|
8530 does so much work in the Rust ecosystem that I can't believe he has |
|
8531 time for it all.</p> |
|
8532 <p><img alt="Graffitti heads" src="https://people.gnome.org/~fe… |
|
8533 <h1>Extend the Rust language for GObject? Like Vala?</h1> |
|
8534 <p>Yeah, pretty much.</p> |
|
8535 <p>Except that instead of a wholly new language, we use Rust as-is… |
|
8536 we just add syntactic constructs that make it easy to write GObjects |
|
8537 without boilerplate. For example, this works right now:</p> |
|
8538 <div class="highlight"><pre><span></span><cod… |
|
8539 |
|
8540 <span class="k">extern</span><span class="w"> </spa… |
|
8541 |
|
8542 <span class="cp">#[macro_use]</span><span class="w">&l… |
|
8543 <span class="k">extern</span><span class="w"> </spa… |
|
8544 <span class="k">use</span><span class="w"> </span&g… |
|
8545 |
|
8546 <span class="n">gobject_gen</span><span class="o">!<… |
|
8547 <span class="w"> </span><span class="c1">// Derives… |
|
8548 <span class="w"> </span><span class="n">class</s… |
|
8549 <span class="w"> </span><span class="p">}</span&… |
|
8550 |
|
8551 <span class="w"> </span><span class="k">impl</sp… |
|
8552 <span class="w"> </span><span class="c1">// non… |
|
8553 <span class="w"> </span><span class="k">pub<… |
|
8554 <span class="w"> </span><span class="mi">1&… |
|
8555 <span class="w"> </span><span class="p">}</s… |
|
8556 |
|
8557 <span class="w"> </span><span class="kr">virtua… |
|
8558 <span class="w"> </span><span class="mi">1&… |
|
8559 <span class="w"> </span><span class="p">}</s… |
|
8560 <span class="w"> </span><span class="p">}</span&… |
|
8561 |
|
8562 <span class="w"> </span><span class="c1">// Inherit… |
|
8563 <span class="w"> </span><span class="n">class</s… |
|
8564 <span class="w"> </span><span class="p">}</span&… |
|
8565 |
|
8566 <span class="w"> </span><span class="k">impl</sp… |
|
8567 <span class="w"> </span><span class="c1">// ove… |
|
8568 <span class="w"> </span><span class="c1">// may… |
|
8569 <span class="w"> </span><span class="kr">virtua… |
|
8570 <span class="w"> </span><span class="mi">2&… |
|
8571 <span class="w"> </span><span class="p">}</s… |
|
8572 <span class="w"> </span><span class="p">}</span&… |
|
8573 <span class="p">}</span><span class="w"></span> |
|
8574 |
|
8575 <span class="cp">#[test]</span><span class="w"></sp… |
|
8576 <span class="k">fn</span> <span class="nf">test</sp… |
|
8577 <span class="w"> </span><span class="kd">let</sp… |
|
8578 <span class="w"> </span><span class="kd">let</sp… |
|
8579 |
|
8580 <span class="w"> </span><span class="fm">assert!<… |
|
8581 <span class="w"> </span><span class="fm">assert!<… |
|
8582 <span class="w"> </span><span class="fm">assert!<… |
|
8583 <span class="w"> </span><span class="fm">assert!<… |
|
8584 <span class="p">}</span><span class="w"></span> |
|
8585 </code></pre></div> |
|
8586 |
|
8587 <p>This generates a little boatload of <a href="https://people.… |
|
8588 including a good number of <code>unsafe</code> calls to GObj… |
|
8589 like <code>g_type_register_static_simple()</code>. It also … |
|
8590 traits and paraphernalia that Glib-rs would create for the Rust |
|
8591 binding of a normal GObject written in C.</p> |
|
8592 <p>The idea is that from the outside world, your generated GObject |
|
8593 classes are indistinguishable from GObjects implemented in C.</p> |
|
8594 <p><em>The idea is to write GObject libraries in a better la… |
|
8595 which can then be consumed from language bindings.</em></p> |
|
8596 <h2>Current status of gnome-class</h2> |
|
8597 <p>Up to about two weeks before the hackfest, the syntax for this |
|
8598 mini-language was totally ad-hoc and limited. After a very productive |
|
8599 <a href="https://mail.gnome.org/archives/rust-list/2017-October/msg00… |
|
8600 syntax that definitely looks more Rust-like. It is also easier to |
|
8601 implement, since the Rust parser in syn can be mostly reused as-is, or |
|
8602 pruned down for the parts where we only support GObject-like methods, |
|
8603 and not all the Rust bells and whistles (generics, lifetimes, trait |
|
8604 bounds).</p> |
|
8605 <p>Gnome-class supports deriving classes directly from the basic G… |
|
8606 or from other GObject subclasses in the style of <a href="https://git… |
|
8607 <p>You can define virtual and non-virtual methods. You can overri… |
|
8608 virtual methods from your superclasses.</p> |
|
8609 <p>Not all argument types are supported. In the end we should sup… |
|
8610 argument types which are convertible from Rust to C types. We need to |
|
8611 finish figuring out the annotations for ownership transfer of |
|
8612 references.</p> |
|
8613 <p>We don't support GObject signals yet; I think that's my next ta… |
|
8614 <p>We don't support GObject properties yet.</p> |
|
8615 <p>We don't support defining new GType interfaces yet, but it is p… |
|
8616 It should be easy to support implementing existing interfaces, as it |
|
8617 is pretty much the same as implementing a subclass.</p> |
|
8618 <p>The best way to see what works right now is probably to <a h… |
|
8619 examples</a>, which also work as tests.</p> |
|
8620 <h1>Digression on macro hygiene</h1> |
|
8621 <p>Rust macros are <em>hygienic</em>, unlike C macros … |
|
8622 textual substitution. That is, names declared inside Rust macros will |
|
8623 not clash with names in the calling code.</p> |
|
8624 <p>One peculiar thing about gnome-class is that the user gives us … |
|
8625 names, like a class name <code>Foo</code> and some things in… |
|
8626 method name <code>bar</code>, and a signal <code>baz&l… |
|
8627 From there we want to generate a bunch of boilerplate for GObject |
|
8628 registration and implementaiton. Some of the generated names in that |
|
8629 boilerplate would be</p> |
|
8630 <div class="highlight"><pre><span></span><cod… |
|
8631 <span class="n">FooClass</span> <span class="o"&g… |
|
8632 <span class="n">Foo</span><span class="p">::</span&… |
|
8633 <span class="n">Foo</span><span class="p">::</span&… |
|
8634 <span class="n">Foo</span><span class="p">::</span&… |
|
8635 <span class="n">foo_bar</span><span class="p">()</s… |
|
8636 <span class="n">foo_get_type</span><span class="p">()&… |
|
8637 </code></pre></div> |
|
8638 |
|
8639 <p>However, if we want to actually generate those names inside our |
|
8640 gnome-class macro <em>and make them visible to the caller</em&g… |
|
8641 so <em>unhygienically</em>. Alex started started a <a hr… |
|
8642 on macro hygiene</a>, so expect some news in the Rust world |
|
8643 soon.</p> |
|
8644 <p>TL;DR: there is a difference between a <em>code generator… |
|
8645 gnome-class mostly intends to be, and a <em>macro system</em>… |
|
8646 an aid in typing repetitive code.</p> |
|
8647 <p><img alt="Fuck wars" src="https://people.gnome.org/~federico… |
|
8648 <h1>People for whom to to be thankful</h1> |
|
8649 <p>During the hackfest, <a href="http://blog.nirbheek.in/">N… |
|
8650 from Autotools to the Meson build system, and dealing with Rust |
|
8651 peculiarities along the way. This is exactly what I needed! Thanks, |
|
8652 Nirbheek!</p> |
|
8653 <p><a href="https://coaxion.net/blog/">Sebastian</a> a… |
|
8654 internals and how to use them from the Rust side.</p> |
|
8655 <p><a href="http://zee-nix.blogspot.com/">Zeeshan</a> … |
|
8656 ramen, Greek, excellent pizza... My stomach is definitely thankful.</… |
|
8657 <h1>Berlin</h1> |
|
8658 <p>I love Berlin. It is a cosmopolitan, progressive, LGBTQ-friend… |
|
8659 city, with lots of things to do, vast distances to be traveled, with |
|
8660 good public transport and bike lanes, diverse food to be eaten along |
|
8661 the way...</p> |
|
8662 <p>But damnit, it's also cold at this time of the year. I don't t… |
|
8663 the weather was ever above 10°C while we were there, and mostly in a |
|
8664 constant state of not-quite-rain. This is much different from the Berlin |
|
8665 in the summer that I knew!</p> |
|
8666 <p><a href="https://people.gnome.org/~federico/blog/images/kimc… |
|
8667 <p>This is my third time visiting Berlin. The first one was durin… |
|
8668 Desktop Summit in 2011, and the second one was when my family and I |
|
8669 visited the city two years ago. It is a city that I would definitely |
|
8670 like to know better.</p> |
|
8671 <h1>Thanks to the GNOME Foundation...</h1> |
|
8672 <p>... for sponsoring my travel and accomodation during the hackfe… |
|
8673 <p><img alt="Sponsored by the GNOME Foundation" src="https://pe… |
|
8674 Emacs pop up a desktop-wide notification when a compilation finishes, |
|
8675 i.e. after "<code>M-x compile</code>" is done. Let's see if… |
|
8676 wasting time in the web when I launch a compilation.</p> |
|
8677 <div class="highlight"><pre><span></span><cod… |
|
8678 <span class="p">(</span><span class="nb">append … |
|
8679 Emacs pop up a desktop-wide notification when a compilation finishes, |
|
8680 i.e. after "<code>M-x compile</code>" is done. Let's see if… |
|
8681 wasting time in the web when I launch a compilation.</p> |
|
8682 <div class="highlight"><pre><span></span><cod… |
|
8683 <span class="p">(</span><span class="nb">append&… |
|
8684 <span class="o">&#39;</span><span class="p"… |
|
8685 |
|
8686 <span class="p">(</span><span class="nb">defun</spa… |
|
8687 <span class="p">(</span><span class="nv">call-proces… |
|
8688 <span class="s">&quot;-t&quot;</span> <sp… |
|
8689 <span class="s">&quot;-i&quot;</span> <sp… |
|
8690 <span class="s">&quot;Compilation finished in Emacs&am… |
|
8691 <span class="nv">status</span><span class="p">… |
|
8692 </code></pre></div></content><category term="misc"></c… |
|
8693 <p>Now let's get on and see how glib-rs handles boxed types.</p… |
|
8694 <h1>Boxed types?</h1> |
|
8695 <p>Let's say you are given a sealed cardboard box with <em>s… |
|
8696 can't know what's inside. You can just pass it on to someone else …&l… |
|
8697 <p>Now let's get on and see how glib-rs handles boxed types.</p… |
|
8698 <h1>Boxed types?</h1> |
|
8699 <p>Let's say you are given a sealed cardboard box with <em>s… |
|
8700 can't know what's inside. You can just pass it on to someone else, or |
|
8701 burn it. And since computers are magic duplication machines, you may |
|
8702 want to copy the box and its contents... and maybe some day you will |
|
8703 get around to opening it.</p> |
|
8704 <p>That's a boxed type. You get a pointer to <em>something&… |
|
8705 what's inside. You can just pass it on to someone else, burn it — I |
|
8706 mean, free it — or since computers are magic, copy the pointer and |
|
8707 whatever it points to.</p> |
|
8708 <p>That's exactly the API for boxed types.</p> |
|
8709 <div class="highlight"><pre><span></span><cod… |
|
8710 <span class="k">typedef</span> <span class="kt">void&l… |
|
8711 |
|
8712 <span class="n">GType</span> <span class="nf">g_boxed_… |
|
8713 <span class="n">GBoxedCopyFunc… |
|
8714 <span class="n">GBoxedFreeFunc… |
|
8715 </code></pre></div> |
|
8716 |
|
8717 <h2>Simple copying, simple freeing</h2> |
|
8718 <p>Imagine you have a color...</p> |
|
8719 <div class="highlight"><pre><span></span><cod… |
|
8720 <span class="n">guchar</span> <span class="n">r<… |
|
8721 <span class="n">guchar</span> <span class="n">g<… |
|
8722 <span class="n">guchar</span> <span class="n">b<… |
|
8723 <span class="p">}</span> <span class="n">Color</spa… |
|
8724 </code></pre></div> |
|
8725 |
|
8726 <p>If you had a pointer to a Color, how would you copy it? Easy:&… |
|
8727 <div class="highlight"><pre><span></span><cod… |
|
8728 <span class="p">{</span> |
|
8729 <span class="n">Color</span> <span class="o">*<… |
|
8730 <span class="o">*</span><span class="n">b</span… |
|
8731 <span class="k">return</span> <span class="n">b<… |
|
8732 <span class="p">}</span> |
|
8733 </code></pre></div> |
|
8734 |
|
8735 <p>That is, allocate a new <code>Color</code>, and ess… |
|
8736 contents.</p> |
|
8737 <p>And to free it? A simple <code>g_free()</code> wor… |
|
8738 things that need to be freed individually.</p> |
|
8739 <h2>Complex copying, complex freeing</h2> |
|
8740 <p>And if we had a color with a name?</p> |
|
8741 <div class="highlight"><pre><span></span><cod… |
|
8742 <span class="n">guchar</span> <span class="n">r<… |
|
8743 <span class="n">guchar</span> <span class="n">g<… |
|
8744 <span class="n">guchar</span> <span class="n">b<… |
|
8745 <span class="kt">char</span> <span class="o">*<… |
|
8746 <span class="p">}</span> <span class="n">ColorWithName… |
|
8747 </code></pre></div> |
|
8748 |
|
8749 <p>We can't just <code>*a = *b</code> here, as we actu… |
|
8750 <code>name</code>. Okay:</p> |
|
8751 <div class="highlight"><pre><span></span><cod… |
|
8752 <span class="p">{</span> |
|
8753 <span class="n">ColorWithName</span> <span class="o"&… |
|
8754 <span class="n">b</span><span class="o">-&gt;&… |
|
8755 <span class="n">b</span><span class="o">-&gt;&… |
|
8756 <span class="n">b</span><span class="o">-&gt;&… |
|
8757 <span class="n">b</span><span class="o">-&gt;&… |
|
8758 <span class="k">return</span> <span class="n">b<… |
|
8759 <span class="p">}</span> |
|
8760 </code></pre></div> |
|
8761 |
|
8762 <p>The corresponding <code>free_color_with_name()</code&g… |
|
8763 <code>g_free(b)</code>, of course.</p> |
|
8764 <h1>Glib-rs and boxed types</h1> |
|
8765 <p>Let's look at this by parts. First, a <a href="https://gith… |
|
8766 trait</a> to define the basic API to manage the |
|
8767 memory of boxed types. This is what defines the <code>copy</co… |
|
8768 functions, like above.</p> |
|
8769 <div class="highlight"><pre><span></span><cod… |
|
8770 <span class="w"> </span><span class="k">unsafe</… |
|
8771 <span class="w"> </span><span class="k">unsafe</… |
|
8772 <span class="p">}</span><span class="w"></span> |
|
8773 </code></pre></div> |
|
8774 |
|
8775 <p>Second, the <a href="https://github.com/gtk-rs/glib/blob/f0a… |
|
8776 <div class="highlight"><pre><span></span><cod… |
|
8777 <span class="w"> </span><span class="n">inner</s… |
|
8778 <span class="w"> </span><span class="n">_dummy</… |
|
8779 <span class="p">}</span><span class="w"></span> |
|
8780 </code></pre></div> |
|
8781 |
|
8782 <p>This struct is generic over <code>T</code>, the act… |
|
8783 wrapping, and <code>MM</code>, something which must implemen… |
|
8784 <code>BoxedMemoryManager</code> trait.</p> |
|
8785 <p>Inside, it stores <code>inner</code>, an <code&g… |
|
8786 The <code>_dummy: PhantomData&lt;MM&gt;</code> is a… |
|
8787 struct doesn't actually store a memory manager, it acts as if it does |
|
8788 — it does not concern us here.</p> |
|
8789 <h2>The <em>actual</em> representation of boxed data&l… |
|
8790 <p>Let's look at that <a href="https://github.com/gtk-rs/glib/b… |
|
8791 <div class="highlight"><pre><span></span><cod… |
|
8792 <span class="w"> </span><span class="n">Native</… |
|
8793 <span class="w"> </span><span class="n">ForeignOwne… |
|
8794 <span class="w"> </span><span class="n">ForeignBorr… |
|
8795 <span class="p">}</span><span class="w"></span> |
|
8796 </code></pre></div> |
|
8797 |
|
8798 <p>We have three cases:</p> |
|
8799 <ul> |
|
8800 <li> |
|
8801 <p><code>Native(Box&lt;T&gt;)</code> - this bo… |
|
8802 know everything about it!</p> |
|
8803 </li> |
|
8804 <li> |
|
8805 <p><code>ForeignOwned(*mut T)</code> - this boxed valu… |
|
8806 we own it now. We will have to free it when we are done with it.</… |
|
8807 </li> |
|
8808 <li> |
|
8809 <p><code>ForeignBorrowed(*mut T)</code> - this boxed v… |
|
8810 outside, but we are just borrowing it temporarily: we <strong>do… |
|
8811 free it when we are done with it.</p> |
|
8812 </li> |
|
8813 </ul> |
|
8814 <p>For example, if we look at the <a href="https://github.com/g… |
|
8815 trait</a> for the <code>Boxed</code> struct, we will i… |
|
8816 calls the <code>BoxedMemoryManager::free()</code> <strong… |
|
8817 <code>ForeignOwned</code> value:</p> |
|
8818 <div class="highlight"><pre><span></span><cod… |
|
8819 <span class="w"> </span><span class="k">fn</span… |
|
8820 <span class="w"> </span><span class="k">unsafe&… |
|
8821 <span class="w"> </span><span class="k">if&… |
|
8822 <span class="w"> </span><span class="n">… |
|
8823 <span class="w"> </span><span class="p">}&l… |
|
8824 <span class="w"> </span><span class="p">}</s… |
|
8825 <span class="w"> </span><span class="p">}</span&… |
|
8826 <span class="p">}</span><span class="w"></span> |
|
8827 </code></pre></div> |
|
8828 |
|
8829 <p>If we had a <code>Native(Box&lt;T&gt;)</code&g… |
|
8830 and Rust knows how to <code>Drop</code> its own <code>… |
|
8831 allocated in the heap).</p> |
|
8832 <p>But for external resources, we must tell Rust how to manage the… |
|
8833 Again: in the case where the Rust side owns the reference to the |
|
8834 external boxed data, we have a <code>ForeignOwned</code> and… |
|
8835 <code>free()</code>ing it; in the case where the Rust side i… |
|
8836 data temporarily, we have a <code>ForeignBorrowed</code> and… |
|
8837 we are done.</p> |
|
8838 <h2>Copying</h2> |
|
8839 <p>When do we have to copy a boxed value? For example, when we tr… |
|
8840 from Rust to Glib with full transfer of ownership, i.e. the |
|
8841 <a href="https://github.com/gtk-rs/glib/blob/f0a2aae96162fd628fb0e3ee… |
|
8842 before</a>. This is how that trait method is |
|
8843 implemented for <code>Boxed</code>:</p> |
|
8844 <div class="highlight"><pre><span></span><cod… |
|
8845 <span class="w"> </span><span class="k">fn</span… |
|
8846 <span class="w"> </span><span class="k">use<… |
|
8847 <span class="w"> </span><span class="kd">let<… |
|
8848 <span class="w"> </span><span class="n">Nat… |
|
8849 <span class="w"> </span><span class="n">For… |
|
8850 <span class="w"> </span><span class="p">};</… |
|
8851 <span class="w"> </span><span class="k">unsafe&… |
|
8852 <span class="w"> </span><span class="p">}</span&… |
|
8853 <span class="p">}</span><span class="w"></span> |
|
8854 </code></pre></div> |
|
8855 |
|
8856 <p>See the <code>MM:copy(ptr)</code> in the last line?… |
|
8857 happens. The lines above just get the appropriate pointer to the data |
|
8858 data from the <code>AnyBox</code> and cast it.</p> |
|
8859 <p>There is extra boilerplate in <code>boxed.rs</code>… |
|
8860 mostly a bunch of trait implementations to copy the boxed data at the |
|
8861 appropriate times (e.g. the <code>FromGlibPtrNone</code> tra… |
|
8862 implementation of the <code>Deref</code> trait to get to the… |
|
8863 / AnyBox</code> easily, etc. The trait implementations are there … |
|
8864 make it as convenient as possible to handle <code>Boxed</code&g… |
|
8865 <h2>Who implements BoxedMemoryManager?</h2> |
|
8866 <p>Up to now, we have seen things like the implementation of <c… |
|
8867 <code>Boxed</code>, which uses <code>BoxedMemoryManage… |
|
8868 implementation of <code>ToGlibPtr</code> which uses <code… |
|
8869 <p>But those are just the trait's "abstract" methods, so to speak.… |
|
8870 actually implements them?</p> |
|
8871 <p>Glib-rs has a general-purpose macro to wrap Glib types. It can… |
|
8872 boxed types, shared pointer types, and GObjects. For now we will just |
|
8873 look at boxed types.</p> |
|
8874 <p>Glib-rs comes with a macro, <a href="http://gtk-rs.org/docs/… |
|
8875 different ways. You can use it to automatically write the boilerplate |
|
8876 for a boxed type like this:</p> |
|
8877 <div class="highlight"><pre><span></span><cod… |
|
8878 <span class="w"> </span><span class="k">pub</spa… |
|
8879 |
|
8880 <span class="w"> </span><span class="k">match</s… |
|
8881 <span class="w"> </span><span class="n">copy<… |
|
8882 <span class="w"> </span><span class="n">free<… |
|
8883 <span class="w"> </span><span class="n">get_typ… |
|
8884 <span class="w"> </span><span class="p">}</span&… |
|
8885 <span class="p">}</span><span class="w"></span> |
|
8886 </code></pre></div> |
|
8887 |
|
8888 <p>This expands to an internal |
|
8889 <a href="https://github.com/gtk-rs/glib/blob/f0a2aae96162fd628fb0e3ee… |
|
8890 things. We will only look at particularly interesting bits.</p> |
|
8891 <p>First, the macro creates a newtype around a tuple with 1) the a… |
|
8892 data type you want to box, and 2) a memory manager. In the example |
|
8893 above, the newtype would be called <code>Color</code>, and i… |
|
8894 <code>ffi:Color</code> (say, a C struct).</p> |
|
8895 <div class="highlight"><pre><span></span><cod… |
|
8896 </code></pre></div> |
|
8897 |
|
8898 <p>Aha! And that <code>MemoryManager</code>? The mac… |
|
8899 type:</p> |
|
8900 <div class="highlight"><pre><span></span><cod… |
|
8901 </code></pre></div> |
|
8902 |
|
8903 <p>Then it <a href="https://github.com/gtk-rs/glib/blob/f0a2aae… |
|
8904 <code>MemoryManager</code> struct:</p> |
|
8905 <div class="highlight"><pre><span></span><cod… |
|
8906 <span class="w"> </span><span class="cp">#[… |
|
8907 <span class="w"> </span><span class="k">uns… |
|
8908 <span class="w"> </span><span class="cp"&g… |
|
8909 <span class="w"> </span><span class="p">}&l… |
|
8910 |
|
8911 <span class="w"> </span><span class="cp">#[… |
|
8912 <span class="w"> </span><span class="k">uns… |
|
8913 <span class="w"> </span><span class="cp"&g… |
|
8914 <span class="w"> </span><span class="p">}&l… |
|
8915 <span class="w"> </span><span class="p">}</s… |
|
8916 </code></pre></div> |
|
8917 |
|
8918 <p>There! <em>This</em> is where the <code>copy… |
|
8919 on the bits of code with which you invoked the macro. In the call to |
|
8920 <code>glib_wrapper!()</code> we had this:</p> |
|
8921 <div class="highlight"><pre><span></span><cod… |
|
8922 <span class="w"> </span><span class="n">free<… |
|
8923 </code></pre></div> |
|
8924 |
|
8925 <p>In the impl aboe, the <code>$copy_expr</code> will … |
|
8926 <code>ffi::color_copy(mut_override(ptr))</code> and <code… |
|
8927 <code>ffi::color_free(ptr)</code>, which defines our impleme… |
|
8928 manager for our <code>Color</code> boxed type.</p> |
|
8929 <h2>Zero-sized what?</h2> |
|
8930 <p>Within the macro's definition, let's look again at the definiti… |
|
8931 our boxed type and the memory manager object that actually implements |
|
8932 the <code>BoxedMemoryManager</code> trait. Here is what the… |
|
8933 to with our <code>Color</code> example:</p> |
|
8934 <div class="highlight"><pre><span></span><cod… |
|
8935 |
|
8936 <span class="w"> </span><span class="k">pub<… |
|
8937 |
|
8938 <span class="w"> </span><span class="k">impl<… |
|
8939 <span class="w"> </span><span class="k">uns… |
|
8940 <span class="w"> </span><span class="k">uns… |
|
8941 <span class="w"> </span><span class="p">}</s… |
|
8942 </code></pre></div> |
|
8943 |
|
8944 <p>Here, <code>MemoryManager</code> is a zero-sized ty… |
|
8945 take up any space</strong> in the <code>Color</code> t… |
|
8946 in the heap, it is really as if it contained an <code>ffi::Color&l… |
|
8947 C struct we are wrapping) <em>and nothing else</em>.</p&g… |
|
8948 <p>All the knowledge about how to copy/free <code>ffi::Color… |
|
8949 the compiler</strong> thanks to the trait implementation. When th… |
|
8950 expands all the macros and monomorphizes all the generic functions, |
|
8951 the calls to <code>ffi::color_copy()</code> and <code>… |
|
8952 inlined at the appropriate spots</strong>. There is no need to ha… |
|
8953 auxiliary structures taking up space in the heap, just to store |
|
8954 function pointers to the copy/free functions, or anything like that.<… |
|
8955 <h1>Next up</h1> |
|
8956 <p>You may have seen that our example call to <code>glib_wra… |
|
8957 passed in a <code>ffi::color_get_type()</code> function. We… |
|
8958 how glib-rs wraps Glib's <code>GType</code>, <code>GVa… |
|
8959 getting closer and closer to being able to wrap <code>GObject</… |
|
8960 <p>Stay tuned!</p></content><category term="misc"></category… |
|
8961 blog</a>, so they may be a bit hard to find from this new blog. |
|
8962 Here is a list of those posts, just so they are easier to find:</p> |
|
8963 <ul> |
|
8964 <li><a href="https://people.gnome.org/~federico/news-2016-10.ht… |
|
8965 <li><a href="https://people.gnome.org/~federico/news-2016-10.ht… |
|
8966 blog</a>, so they may be a bit hard to find from this new blog. |
|
8967 Here is a list of those posts, just so they are easier to find:</p> |
|
8968 <ul> |
|
8969 <li><a href="https://people.gnome.org/~federico/news-2016-10.ht… |
|
8970 <li><a href="https://people.gnome.org/~federico/news-2016-10.ht… |
|
8971 <li><a href="https://people.gnome.org/~federico/news-2016-11.ht… |
|
8972 <li><a href="https://people.gnome.org/~federico/news-2016-11.ht… |
|
8973 <li><a href="https://people.gnome.org/~federico/news-2016-11.ht… |
|
8974 <li><a href="https://people.gnome.org/~federico/news-2016-11.ht… |
|
8975 <li><a href="https://people.gnome.org/~federico/news-2017-01.ht… |
|
8976 <li><a href="https://people.gnome.org/~federico/news-2017-02.ht… |
|
8977 <li><a href="https://people.gnome.org/~federico/news-2017-02.ht… |
|
8978 <li><a href="https://people.gnome.org/~federico/news-2017-02.ht… |
|
8979 <li><a href="https://people.gnome.org/~federico/news-2017-02.ht… |
|
8980 <li><a href="https://people.gnome.org/~federico/news-2017-04.ht… |
|
8981 </ul> |
|
8982 <p>Within this new blog, you can look for articles with the <a … |
|
8983 a detour and look at <a href="https://wiki.gnome.org/Projects/GObject… |
|
8984 seem like an obscure part of the GNOME platform, it is an absolutely |
|
8985 vital part of it: it is what lets people write GNOME applications in |
|
8986 any language.</p> |
|
8987 <p>Let's start with a …</p></summary><content type="html">… |
|
8988 a detour and look at <a href="https://wiki.gnome.org/Projects/GObject… |
|
8989 seem like an obscure part of the GNOME platform, it is an absolutely |
|
8990 vital part of it: it is what lets people write GNOME applications in |
|
8991 any language.</p> |
|
8992 <p>Let's start with a bit of history.</p> |
|
8993 <h1>Brief history of language bindings in GNOME</h1> |
|
8994 <p>When we started GNOME in 1997, we didn't want to write <em&g… |
|
8995 C. We had some inspiration from elsewhere.</p> |
|
8996 <h2>Prehistory: GIMP and the Procedural Database</h2> |
|
8997 <p>There was already good precedent for software written in a comb… |
|
8998 programming languages. Emacs, the flagship text editor of the |
|
8999 GNU project, was written with a relatively small core in C, and the |
|
9000 majority of the program in Emacs Lisp.</p> |
|
9001 <p>In similar fashion, we were very influenced by the design of th… |
|
9002 which was very innovative at that time. The GIMP has a large core |
|
9003 written in C. However, it supports plug-ins or <em>scripts</em… |
|
9004 variety of languages. Initially the only scripting language available |
|
9005 for the GIMP was Scheme.</p> |
|
9006 <p>The GIMP's plug-ins and scripts run as separate processes, so |
|
9007 they don't have immediate access to the data of the image being |
|
9008 edited, or to the core functions of the program like "paint with a |
|
9009 brush at this location". To let plug-ins and scripts access these |
|
9010 data and these functions, the GIMP has what it calls a |
|
9011 Procedural Database (PDB). This is a |
|
9012 list of functions that the core program or plug-ins wish to export. |
|
9013 For example, there are functions like <code>gimp-scale-image</c… |
|
9014 <code>gimp-move-layer</code>. Once these functions are regi… |
|
9015 PDB, any part of the program or plug-ins can call them. Scripts are |
|
9016 often written to automate common tasks — for example, when one wants |
|
9017 to adjust the contrast of photos and scale them in bulk. Scripts can |
|
9018 call functions in the PDB easily, irrespective of the programming |
|
9019 language they are written in.</p> |
|
9020 <p>We wanted to write GNOME's core libraries in C, and write a sim… |
|
9021 Procedural Database to allow those libraries to be called from any |
|
9022 programming language. Eventually it turned out that a PDB was not |
|
9023 necessary, and there were better ways to go about enabling different |
|
9024 programming languages.</p> |
|
9025 <h2>Enabling sane memory management</h2> |
|
9026 <p>GTK+ started out with a very simple scheme for memory managemen… |
|
9027 container owned its child widgets, and so on recursively. When you |
|
9028 freed a container, it would be responsible for freeing its children.<… |
|
9029 <p>However, consider what happens when a widget needs to hold a re… |
|
9030 to another widget that is not one of its children. For example, a |
|
9031 GtkLabel with an underlined mnemonic ("_N_ame:") needs to have a |
|
9032 reference to the GtkEntry that should be focused when you press |
|
9033 Alt-N. In the very earliest versions of GTK+, how to do this was |
|
9034 undefined: C programmers were already used to having shared pointers |
|
9035 everywhere, and they were used to being responsible for managing their |
|
9036 memory.</p> |
|
9037 <p>Of course, this was prone to bugs. If you have something like&… |
|
9038 <div class="highlight"><pre><span></span><cod… |
|
9039 <span class="n">GtkWidget</span> <span class="n">p… |
|
9040 |
|
9041 <span class="kt">char</span> <span class="o">*<… |
|
9042 <span class="n">GtkWidget</span> <span class="o">*… |
|
9043 <span class="p">}</span> <span class="n">GtkLabel</… |
|
9044 </code></pre></div> |
|
9045 |
|
9046 <p>then if you are writing the destructor, you may simply want to&… |
|
9047 <div class="highlight"><pre><span></span><cod… |
|
9048 <span class="nf">gtk_label_free</span> <span class="p">… |
|
9049 <span class="p">{</span> |
|
9050 <span class="n">g_free</span> <span class="p">(<… |
|
9051 <span class="n">gtk_widget_free</span> <span class="p… |
|
9052 |
|
9053 <span class="n">free_parent_instance</span> <span cla… |
|
9054 <span class="p">}</span> |
|
9055 </code></pre></div> |
|
9056 |
|
9057 <p>Say you have a GtkBox with the label and its associated GtkEntr… |
|
9058 Then, freeing the GtkBox would recursively free the label with that |
|
9059 <code>gtk_label_free()</code>, and then the entry with its o… |
|
9060 the time the entry gets freed, the line |
|
9061 <code>gtk_widget_free (widget_to_focus)</code> has already … |
|
9062 we get a double-free bug!</p> |
|
9063 <p>Madness!</p> |
|
9064 <p>That is, we had no idea what we were doing. Or rather, our |
|
9065 understanding of widgets had not evolved to the point of acknowledging |
|
9066 that a widget tree is not a simply tree, but rather a |
|
9067 directed graph of container-child relationships, plus |
|
9068 random-widget-to-random-widget relationships. And of course, other |
|
9069 parts of the program <em>which are not even widget implementations… |
|
9070 need to keep references to widgets and free them or not as |
|
9071 appropriate.</p> |
|
9072 <p>I think Marius Vollmer was the first person to start formalizing |
|
9073 this. He came from the world of GNU Guile, a Scheme interpreter, and |
|
9074 so he already knew how garbage collection and seas of shared |
|
9075 references ought to work.</p> |
|
9076 <p>Marius implemented reference-counting for GTK+ — that's where |
|
9077 <code>gtk_object_ref()</code> and <code>gtk_object_unr… |
|
9078 got moved to the base <code>GObject</code> class, so we now … |
|
9079 and <code>g_object_unref()</code> and a host of functions to… |
|
9080 references, notification of destruction, and all the things required |
|
9081 to keep garbage collectors happy.</p> |
|
9082 <h2>The first language bindings</h2> |
|
9083 <p>The very first language bindings were written by hand. The GTK… |
|
9084 was small, and it seemed feasible to take</p> |
|
9085 <div class="highlight"><pre><span></span><cod… |
|
9086 <span class="kt">void</span> <span class="nf">gtk_widg… |
|
9087 |
|
9088 <span class="kt">void</span> <span class="nf">gtk_cont… |
|
9089 <span class="kt">void</span> <span class="nf">gtk_cont… |
|
9090 </code></pre></div> |
|
9091 |
|
9092 <p>and just wrap those functions in various languages, by hand, on… |
|
9093 as-needed basis.</p> |
|
9094 <p>Of course, there is a lot of duplication when doing things that… |
|
9095 As the C API grows, one needs to do more and more manual work to |
|
9096 keep up with it.</p> |
|
9097 <p>Also, C structs with public fields are problematic. If we had&… |
|
9098 <div class="highlight"><pre><span></span><cod… |
|
9099 <span class="n">guchar</span> <span class="n">r<… |
|
9100 <span class="n">guchar</span> <span class="n">g<… |
|
9101 <span class="n">guchar</span> <span class="n">b<… |
|
9102 <span class="p">}</span> <span class="n">GdkColor</… |
|
9103 </code></pre></div> |
|
9104 |
|
9105 <p>and we <em>expect</em> program code to fill in a &l… |
|
9106 pass it to a drawing function like</p> |
|
9107 <div class="highlight"><pre><span></span><cod… |
|
9108 </code></pre></div> |
|
9109 |
|
9110 <p>then it is no problem to do that in C:</p> |
|
9111 <div class="highlight"><pre><span></span><cod… |
|
9112 |
|
9113 <span class="n">gdk_set_foreground_color</span> <span cla… |
|
9114 </code></pre></div> |
|
9115 |
|
9116 <p>But to do that in a high level language? You don't have access… |
|
9117 struct fields! And back then, libffi wasn't generally available.</p&… |
|
9118 <p>Authors of language bindings had to write some glue code, in C,… |
|
9119 hand, to let people access a C struct and then pass it on to GTK+. |
|
9120 For example, for Python, they would need to write something like</p&g… |
|
9121 <div class="highlight"><pre><span></span><cod… |
|
9122 <span class="nf">make_wrapped_gdk_color</span> <span clas… |
|
9123 <span class="p">{</span> |
|
9124 <span class="n">GdkColor</span> <span class="o">*&… |
|
9125 <span class="n">PyObject</span> <span class="o">*&… |
|
9126 |
|
9127 <span class="n">g_color</span> <span class="o">=&l… |
|
9128 <span class="cm">/* ... fill in g_color-&gt;r, g, b from t… |
|
9129 |
|
9130 <span class="n">py_color</span> <span class="o">=&… |
|
9131 <span class="k">return</span> <span class="n">py_c… |
|
9132 <span class="p">}</span> |
|
9133 </code></pre></div> |
|
9134 |
|
9135 <p>Writing that by hand is an incredible amount of drudgery.</p… |
|
9136 <p>What language bindings needed was a <em>description</e… |
|
9137 machine-readable format, so that the glue code could be written by a |
|
9138 code generator.</p> |
|
9139 <h2>The first API descriptions</h2> |
|
9140 <p>I don't remember if it was the GNU Guile people, or the PyGTK p… |
|
9141 who started to write descriptions of the GNOME API by hand. For ease |
|
9142 of parsing, it was done in a Scheme-like dialect. A description may |
|
9143 look like</p> |
|
9144 <div class="highlight"><pre><span></span><cod… |
|
9145 <span class="c1">;;; void gtk_widget_show (GtkWidget *widge… |
|
9146 <span class="p">(</span><span class="nf">method… |
|
9147 <span class="p">(</span><span class="nf"&g… |
|
9148 <span class="p">(</span><span class="nf"&g… |
|
9149 |
|
9150 <span class="c1">;;; void gtk_widget_hide (GtkWidget *widge… |
|
9151 <span class="p">(</span><span class="nf">method… |
|
9152 <span class="p">(</span><span class="nf"&g… |
|
9153 <span class="p">(</span><span class="nf"&g… |
|
9154 |
|
9155 <span class="p">(</span><span class="nf">class</spa… |
|
9156 <span class="c1">;;; void gtk_container_add (GtkContainer *… |
|
9157 <span class="p">(</span><span class="nf">method… |
|
9158 <span class="p">(</span><span class="nf"&g… |
|
9159 <span class="p">(</span><span class="nf"&g… |
|
9160 |
|
9161 <span class="p">(</span><span class="nf">struct</sp… |
|
9162 <span class="p">(</span><span class="nf">field… |
|
9163 <span class="p">(</span><span class="nf">field… |
|
9164 <span class="p">(</span><span class="nf">field… |
|
9165 </code></pre></div> |
|
9166 |
|
9167 <p>Again, writing those descriptions by hand (and keeping up with … |
|
9168 API) was a lot of work, but the glue code to implement the binding |
|
9169 could be done mostly automatically. The generated code may need |
|
9170 subsequent tweaks by hand to deal with details that the Scheme-like |
|
9171 descriptions didn't contemplate, but it was better than writing |
|
9172 <em>everything</em> by hand.</p> |
|
9173 <h2 id="type-system">Glib gets a real type system</h2> |
|
9174 <p>Tim Janik took over the parts of Glib that implement |
|
9175 objects/signals/types, and added a lot of things to create a good type |
|
9176 system for C. This is where things like <code>GType</code>,… |
|
9177 fundamental types come from.</p> |
|
9178 <p>For example, a <code>GType</code> is an identifier … |
|
9179 type plus, well, a value of that type. You can ask a <code>GValue… |
|
9180 you an int? are you a GObject?".</p> |
|
9181 <p>You can register new types: for example, there would be code i… |
|
9182 that registers a new <code>GType</code> for <code>GdkC… |
|
9183 "are you a color?".</p> |
|
9184 <p>Registering a type involves telling the GObject system things l… |
|
9185 to copy values of that type, and how to free them. For <code>GdkC… |
|
9186 this may be just <code>g_new() / g_free()</code>; for refere… |
|
9187 it may be <code>g_object_ref() / g_object_unref()</code>.<… |
|
9188 <h3>Objects can be queried about some of their properties</h3&g… |
|
9189 <p>A widget can tell you when you press a mouse button mouse on it… |
|
9190 will emit the <code>button-press-event</code> signal. When … |
|
9191 implementation registers this signal, it calls something like</p> |
|
9192 <div class="highlight"><pre><span></span><cod… |
|
9193 <span class="n">gtk_widget_get_type</span><span c… |
|
9194 <span class="p">...</span> |
|
9195 <span class="n">G_TYPE_BOOLEAN</span><span class=… |
|
9196 <span class="mi">1</span><span class="p">,<… |
|
9197 <span class="n">GDK_TYPE_EVENT</span><span class=… |
|
9198 </code></pre></div> |
|
9199 |
|
9200 <p>This tells GObject that <code>GtkWidget</code> will… |
|
9201 <code>button-press-event</code>, with a return type of <c… |
|
9202 a single argument of type <code>GDK_TYPE_EVENT</code>. This… |
|
9203 appropriate marshalling of arguments when the signal is emitted.</p&g… |
|
9204 <p>But also! <em>You can query the signal for its argument … |
|
9205 run <code>g_signal_query()</code>, which will then tell you … |
|
9206 the signal: its name, return type, argument types, etc. A language |
|
9207 binding could run <code>g_signal_query()</code> <em>an… |
|
9208 signal automatically</em> to the Scheme-like description language.… |
|
9209 then generate the binding from <em>that</em>.</p> |
|
9210 <h2>Not all of an object's properties can be queried</h2> |
|
9211 <p>Unfortunately, although GObject signals and properties <em&g… |
|
9212 queried, methods can't be. C doesn't have classes with methods, and GOb… |
|
9213 not really have any provisions to implement them. </p> |
|
9214 <p>Conventionally, for a static method one would just do</p> |
|
9215 <div class="highlight"><pre><span></span><cod… |
|
9216 <span class="nf">gtk_widget_set_flags</span> <span class=… |
|
9217 <span class="p">{</span> |
|
9218 <span class="cm">/* modify a struct field within &quot;wid… |
|
9219 <span class="cm">/* repaint or something */</span> |
|
9220 <span class="p">}</span> |
|
9221 </code></pre></div> |
|
9222 |
|
9223 <p>And for a virtual method one would put a function pointer in th… |
|
9224 structure, and provide a convenient way to call it:</p> |
|
9225 <div class="highlight"><pre><span></span><cod… |
|
9226 <span class="n">GtkObjectClass</span> <span class="n"… |
|
9227 |
|
9228 <span class="kt">void</span> <span class="p">(<… |
|
9229 <span class="p">}</span> <span class="n">GtkWidgetClas… |
|
9230 |
|
9231 <span class="kt">void</span> |
|
9232 <span class="nf">gtk_widget_draw</span> <span class="p"&g… |
|
9233 <span class="p">{</span> |
|
9234 <span class="n">GtkWidgetClass</span> <span class="o"… |
|
9235 |
|
9236 <span class="p">(</span><span class="o">*</span… |
|
9237 <span class="p">}</span> |
|
9238 </code></pre></div> |
|
9239 |
|
9240 <p>And GObject has no idea about this method — there is no way t… |
|
9241 it; it just exists in C-space.</p> |
|
9242 <p>Now, historically, GTK+'s header files have been written in a &… |
|
9243 consistent style. It is quite possible to write a tool that will take |
|
9244 a header file like</p> |
|
9245 <div class="highlight"><pre><span></span><cod… |
|
9246 <span class="k">typedef</span> <span class="k">struct&… |
|
9247 <span class="n">GtkObject</span> <span class="n">p… |
|
9248 |
|
9249 <span class="kt">void</span> <span class="p">(<… |
|
9250 <span class="p">}</span> <span class="n">GtkWidgetClas… |
|
9251 |
|
9252 <span class="kt">void</span> <span class="nf">gtk_widg… |
|
9253 <span class="kt">void</span> <span class="nf">gtk_widg… |
|
9254 </code></pre></div> |
|
9255 |
|
9256 <p>and parse it, even if it is with a simple parser that does not |
|
9257 completely understand the C language, and have heuristics like</p> |
|
9258 <ul> |
|
9259 <li> |
|
9260 <p>Is there a <code>class_name_foo()</code> function p… |
|
9261 corresponding <code>foo</code> field in the <code>Cl… |
|
9262 static method.</p> |
|
9263 </li> |
|
9264 <li> |
|
9265 <p>Is there a <code>class_name_bar()</code> function w… |
|
9266 <code>Class</code> structure? It's probably a virtual met… |
|
9267 </li> |
|
9268 <li> |
|
9269 <p>Etc.</p> |
|
9270 </li> |
|
9271 </ul> |
|
9272 <p>And in fact, that's what we had. C header files would get pars… |
|
9273 with those heuristics, and the Scheme-like description files would get |
|
9274 generated.</p> |
|
9275 <h2>Scheme-like descriptions get reused, kind of</h2> |
|
9276 <p>Language binding authors started reusing the Scheme-like |
|
9277 descriptions. Sometimes they would cannibalize the descriptions from |
|
9278 PyGTK, or Guile (again, I don't remember where the canonical version |
|
9279 was maintained) and use them as they were.</p> |
|
9280 <p>Other times they would copy the files, modify them by hand some… |
|
9281 and <em>then</em> use them to generate their language bindin… |
|
9282 <h2>C being hostile</h2> |
|
9283 <p>From just reading/parsing a C function prototype, you cannot kn… |
|
9284 certain things. If one function argument is of type <code>Foo *&l… |
|
9285 <ul> |
|
9286 <li> |
|
9287 <p>the function gets a pointer to something which it should not mo… |
|
9288 ("in" parameter)</p> |
|
9289 </li> |
|
9290 <li> |
|
9291 <p>the function gets a pointer to uninitialized data which it will… |
|
9292 ("out" parameter)</p> |
|
9293 </li> |
|
9294 <li> |
|
9295 <p>the function gets a pointer to initialized data which it will u… |
|
9296 and modify ("inout" parameter)</p> |
|
9297 </li> |
|
9298 <li> |
|
9299 <p>the function will copy that pointer and hold a reference to the |
|
9300 pointed data, and not free it when it's done</p> |
|
9301 </li> |
|
9302 <li> |
|
9303 <p>the function will take over the ownership of the pointed data, … |
|
9304 free it when it's done</p> |
|
9305 </li> |
|
9306 <li> |
|
9307 <p>etc.</p> |
|
9308 </li> |
|
9309 </ul> |
|
9310 <p>Sometimes people would include these annotations in the Scheme-… |
|
9311 description language. But wouldn't it be better if those annotations |
|
9312 came <em>from the C code itself</em>?</p> |
|
9313 <h1>GObject Introspection appears</h1> |
|
9314 <p>For GNOME 3, we wanted a unified solution for language bindings… |
|
9315 <ul> |
|
9316 <li> |
|
9317 <p>Have a single way to extract the machine-readable descriptions … |
|
9318 the C API.</p> |
|
9319 </li> |
|
9320 <li> |
|
9321 <p>Have every language binding be automatically generated from tho… |
|
9322 descriptions.</p> |
|
9323 </li> |
|
9324 <li> |
|
9325 <p>In the descriptions, have <em>all</em> the informat… |
|
9326 generate a correct language binding...</p> |
|
9327 </li> |
|
9328 <li> |
|
9329 <p>... including documentation.</p> |
|
9330 </li> |
|
9331 </ul> |
|
9332 <p>We had to do a lot of work to accomplish this. For example:<… |
|
9333 <ul> |
|
9334 <li> |
|
9335 <p>Remove C-isms from the public API. Varargs functions, those th… |
|
9336 have <code>foo (int x, ...)</code>, can't be easily desc… |
|
9337 other languages. Instead, have something like |
|
9338 <code>foov (int x, int num_args, GValue *args_array)</c… |
|
9339 consumed by other languages.</p> |
|
9340 </li> |
|
9341 <li> |
|
9342 <p>Add <em>annotations</em> throughout the code so tha… |
|
9343 can know about in/out/inout arguments, and whether pointer arguments |
|
9344 are borrowed references or a full transfership of ownership.</p> |
|
9345 </li> |
|
9346 <li> |
|
9347 <p>Take the in-line documentation comments and store them as part … |
|
9348 the machine-readable description of the API.</p> |
|
9349 </li> |
|
9350 <li> |
|
9351 <p>When compiling a library, automatically do all the things like |
|
9352 <code>g_signal_query()</code> and spit out machine-readabl… |
|
9353 those parts of the API.</p> |
|
9354 </li> |
|
9355 </ul> |
|
9356 <p>So, GObject Introspection is all of those things.</p> |
|
9357 <h2>Annotations</h2> |
|
9358 <p>If you have looked at the C code for a GNOME library, you may h… |
|
9359 seen something like this:</p> |
|
9360 <div class="highlight"><pre><span></span><cod… |
|
9361 <span class="cm"> * gtk_widget_get_parent:</span> |
|
9362 <span class="cm"> * @widget: a #GtkWidget</span> |
|
9363 <span class="cm"> *</span> |
|
9364 <span class="cm"> * Returns the parent container of @widget.</s… |
|
9365 <span class="cm"> *</span> |
|
9366 <span class="cm"> * Returns: (transfer none) (nullable): the paren… |
|
9367 <span class="cm"> **/</span> |
|
9368 <span class="n">GtkWidget</span> <span class="o">*<… |
|
9369 <span class="nf">gtk_widget_get_parent</span> <span class… |
|
9370 <span class="p">{</span> |
|
9371 <span class="p">...</span> |
|
9372 <span class="p">}</span> |
|
9373 </code></pre></div> |
|
9374 |
|
9375 <p>See that "<code>(transfer none) (nullable)</code>" … |
|
9376 The <code>(transfer none)</code> means that the return value… |
|
9377 ownership does <em>not</em> get transferred to the caller, i… |
|
9378 retains ownership. Finally, the <code>(nullable)</code> ind… |
|
9379 function can return <code>NULL</code>, when the widget has n… |
|
9380 <p>A language binding will then use this information as follows:&l… |
|
9381 <ul> |
|
9382 <li> |
|
9383 <p>It will not <code>unref()</code> the parent widget … |
|
9384 </li> |
|
9385 <li> |
|
9386 <p>It will deal with a <code>NULL</code> pointer in a … |
|
9387 assuming that references are not null.</p> |
|
9388 </li> |
|
9389 </ul> |
|
9390 <p>Every now and then someone discovers a public function which is |
|
9391 lacking an annotation of that sort — for GNOME's purposes this is a |
|
9392 bug; fortunately, it is easy to add that annotation to the C sources |
|
9393 and regenerate the machine-readable descriptions.</p> |
|
9394 <h2>Machine-readable descriptions, or repository files</h2> |
|
9395 <p>So, what do those machine-readable descriptions actually look l… |
|
9396 They moved away from a Scheme-like language and got turned into XML, |
|
9397 because early XXIst century.</p> |
|
9398 <p>The machine-readable descriptions are called <em>GObject … |
|
9399 Repository files</em>, or GIR for short.</p> |
|
9400 <p>Let's look at some parts of <code>Gtk-3.0.gir</code>… |
|
9401 <code>/usr/share/gir-1.0/Gtk-3.0.gir</code>.</p> |
|
9402 <div class="highlight"><pre><span></span><cod… |
|
9403 |
|
9404 <span class="nt">&lt;namespace</span> <span class="… |
|
9405 <span class="na">version=</span><span class=… |
|
9406 <span class="na">shared-library=</span><span… |
|
9407 <span class="na">c:identifier-prefixes=</span>&… |
|
9408 <span class="na">c:symbol-prefixes=</span><s… |
|
9409 </code></pre></div> |
|
9410 |
|
9411 <p>For the toplevel "<code>Gtk</code>" namespace, this… |
|
9412 called. All identifiers have "<code>Gtk</code>" or "<cod… |
|
9413 <h3>A class with methods and a signal</h3> |
|
9414 <p>Let's look at the description for <code>GtkEntry</code… |
|
9415 <div class="highlight"><pre><span></span><cod… |
|
9416 <span class="na">c:symbol-prefix=</span><span … |
|
9417 <span class="na">c:type=</span><span class="s"… |
|
9418 <span class="na">parent=</span><span class="s"… |
|
9419 <span class="na">glib:type-name=</span><span c… |
|
9420 <span class="na">glib:get-type=</span><span cl… |
|
9421 <span class="na">glib:type-struct=</span><span… |
|
9422 |
|
9423 <span class="nt">&lt;doc</span> <span class="na… |
|
9424 widget. A fairly large set of key bindings are supported |
|
9425 by default. If the entered text is longer than the allocation |
|
9426 ... |
|
9427 <span class="nt">&lt;/doc&gt;</span> |
|
9428 </code></pre></div> |
|
9429 |
|
9430 <p>This is the start of the description for <code>GtkEntry&l… |
|
9431 that everything is prefixed with "<code>Gtk</code>", so the … |
|
9432 "<code>Entry</code>". Its parent class is <code>Widge… |
|
9433 registers it against the GObject type system is <code>gtk_entry_ge… |
|
9434 <p>Also, there are the toplevel documentation comments for the <… |
|
9435 class.</p> |
|
9436 <p>Onwards!</p> |
|
9437 <div class="highlight"><pre><span></span><cod… |
|
9438 <span class="nt">&lt;implements</span> <span cl… |
|
9439 <span class="nt">&lt;implements</span> <span cl… |
|
9440 <span class="nt">&lt;implements</span> <span cl… |
|
9441 </code></pre></div> |
|
9442 |
|
9443 <p>GObject classes can implement various interfaces; this is the l… |
|
9444 that <code>GtkEntry</code> supports.</p> |
|
9445 <p>Next, let's look at a single method:</p> |
|
9446 <div class="highlight"><pre><span></span><cod… |
|
9447 <span class="nt">&lt;doc</span> <span class="… |
|
9448 |
|
9449 <span class="nt">&lt;return-value</span> <spa… |
|
9450 <span class="nt">&lt;type</span> <span clas… |
|
9451 <span class="nt">&lt;/return-value&gt;</span> |
|
9452 |
|
9453 <span class="nt">&lt;parameters&gt;</span> |
|
9454 <span class="nt">&lt;instance-parameter</span>… |
|
9455 <span class="nt">&lt;type</span> <span cl… |
|
9456 <span class="nt">&lt;/instance-parameter&gt;<… |
|
9457 <span class="nt">&lt;/parameters&gt;</span> |
|
9458 <span class="nt">&lt;/method&gt;</span> |
|
9459 </code></pre></div> |
|
9460 |
|
9461 <p>The method <code>get_text</code> and its correspond… |
|
9462 value is an UTF-8 encoded string, and ownership of the memory for that |
|
9463 string is not transferred to the caller.</p> |
|
9464 <p>The method takes a single parameter which is the <code>en… |
|
9465 <p>Now, let's look at a signal:</p> |
|
9466 <div class="highlight"><pre><span></span><cod… |
|
9467 <span class="nt">&lt;doc</span> <span class="… |
|
9468 the Enter key. ...<span class="nt">&lt;/doc&gt;</span&g… |
|
9469 |
|
9470 <span class="nt">&lt;return-value</span> <spa… |
|
9471 <span class="nt">&lt;type</span> <span clas… |
|
9472 <span class="nt">&lt;/return-value&gt;</span> |
|
9473 <span class="nt">&lt;/glib:signal&gt;</span> |
|
9474 |
|
9475 <span class="nt">&lt;/class&gt;</span> |
|
9476 </code></pre></div> |
|
9477 |
|
9478 <p>The "<code>activate</code>" signal takes no argumen… |
|
9479 type <code>void</code>, i.e. no return value.</p> |
|
9480 <h3>A struct with public fields</h3> |
|
9481 <p>The following comes from <code>Gdk-3.0.gir</code>; … |
|
9482 <code>GdkRectangle</code>.</p> |
|
9483 <div class="highlight"><pre><span></span><cod… |
|
9484 <span class="na">c:type=</span><span class="s… |
|
9485 <span class="na">glib:type-name=</span><span … |
|
9486 <span class="na">glib:get-type=</span><span c… |
|
9487 <span class="na">c:symbol-prefix=</span><span… |
|
9488 |
|
9489 <span class="nt">&lt;field</span> <span class="… |
|
9490 <span class="nt">&lt;type</span> <span class=… |
|
9491 <span class="nt">&lt;/field&gt;</span> |
|
9492 <span class="nt">&lt;field</span> <span class="… |
|
9493 <span class="nt">&lt;type</span> <span class=… |
|
9494 <span class="nt">&lt;/field&gt;</span> |
|
9495 <span class="nt">&lt;field</span> <span class="… |
|
9496 <span class="nt">&lt;type</span> <span class=… |
|
9497 <span class="nt">&lt;/field&gt;</span> |
|
9498 <span class="nt">&lt;field</span> <span class="… |
|
9499 <span class="nt">&lt;type</span> <span class=… |
|
9500 <span class="nt">&lt;/field&gt;</span> |
|
9501 |
|
9502 <span class="nt">&lt;/record&gt;</span> |
|
9503 </code></pre></div> |
|
9504 |
|
9505 <p>So that's the <code>x/y/width/height</code> fields … |
|
9506 order as they are defined in the C code.</p> |
|
9507 <p>And so on. The idea is for the whole API exported by a GObject |
|
9508 library to be describable by that format. If something can't be |
|
9509 described, it's a bug in the library, or a bug in the format.</p> |
|
9510 <h1>Making language bindings start up quickly: typelib files</h… |
|
9511 <p>As we saw, the GIR files are the XML descriptions of GObject AP… |
|
9512 Dynamic languages like Python would prefer to generate the language |
|
9513 binding on the fly, as needed, instead of pre-generating a huge |
|
9514 binding.</p> |
|
9515 <p>However, GTK+ is a big API: <code>Gtk-3.0.gir</code>… |
|
9516 all of that just to be able to generate <code>gtk_widget_show()<… |
|
9517 would be too slow. Also, there are GTK+'s dependencies: Atk, Gdk, |
|
9518 Cairo, etc. You don't want to parse <em>everything</em> jus… |
|
9519 <p>So, we have an extra step that compiles the GIR files down to b… |
|
9520 <code>.typelib</code> files. For example, |
|
9521 <code>/usr/lib64/girepository-1.0/Gtk-3.0.typelib</code> is … |
|
9522 machine. Those files get <code>mmap()</code>ed for fast acc… |
|
9523 shared between processes.</p> |
|
9524 <h2>How dynamic language bindings use typelib files</h2> |
|
9525 <p>GObject Introspection comes with a library that language binding |
|
9526 implementors can use to consume those <code>.typelib</code> … |
|
9527 <code>libgirepository</code> library has functions like "lis… |
|
9528 available in this namespace", or "call this function with these |
|
9529 values for arguments, and give me back the return value here".</p> |
|
9530 <p>Internally, <code>libgirepository</code> uses <c… |
|
9531 functions in the dynamically-linked libraries.</p> |
|
9532 <p>So, when you write <code>foo.py</code> and do</p… |
|
9533 <div class="highlight"><pre><span></span><cod… |
|
9534 <span class="n">gi</span><span class="o">.</span>… |
|
9535 <span class="kn">from</span> <span class="nn">gi.repos… |
|
9536 <span class="n">win</span> <span class="o">=</span&… |
|
9537 </code></pre></div> |
|
9538 |
|
9539 <p>what happens is that <code>pygobject</code> calls &… |
|
9540 the <code>.typelib</code>, and sees that the constructor for… |
|
9541 function called <code>gtk_window_new()</code>. After seeing… |
|
9542 wants to be called, it calls the function using <code>libffi</c… |
|
9543 result with a <code>PyObject</code>, and that's what you get… |
|
9544 <h1>Static languages</h1> |
|
9545 <p>A static language like Rust prefers to have the whole language … |
|
9546 pre-generated. This is what the various crates in <a href="https://g… |
|
9547 do.</p> |
|
9548 <p><a href="https://github.com/gtk-rs/gir/tree/master/src">T… |
|
9549 and does two things:</p> |
|
9550 <ul> |
|
9551 <li> |
|
9552 <p>Reconstructs the C function prototypes and C struct declaration… |
|
9553 but in a way Rust can understand them. This gets output to the <a hr… |
|
9554 crate</a>.</p> |
|
9555 </li> |
|
9556 <li> |
|
9557 <p>Creates idiomatic Rust code for the language binding. This gets |
|
9558 output to the various crates; for example, <a href="https://github.co… |
|
9559 </li> |
|
9560 </ul> |
|
9561 <p>When reconstructing the C structs and prototypes, we get stuff … |
|
9562 <div class="highlight"><pre><span></span><cod… |
|
9563 <span class="k">pub</span><span class="w"> </span&g… |
|
9564 <span class="w"> </span><span class="k">pub</spa… |
|
9565 <span class="w"> </span><span class="k">pub</spa… |
|
9566 <span class="p">}</span><span class="w"></span> |
|
9567 |
|
9568 <span class="k">extern</span><span class="w"> </spa… |
|
9569 <span class="w"> </span><span class="k">pub</spa… |
|
9570 <span class="p">}</span><span class="w"></span> |
|
9571 </code></pre></div> |
|
9572 |
|
9573 <p>And the idiomatic bindings? Stay tuned!</p></content><ca… |
|
9574 Apart from all the Rust goodness, and the large number of bug fixes, I |
|
9575 am very happy with the way the build system works these days. I've |
|
9576 found it invaluable to have good examples of Autotools incantations to �… |
|
9577 Apart from all the Rust goodness, and the large number of bug fixes, I |
|
9578 am very happy with the way the build system works these days. I've |
|
9579 found it invaluable to have good examples of Autotools incantations to |
|
9580 copy&amp;paste, so hopefully this will be useful to someone else.<… |
|
9581 <p>There are some subtleties that a "good" autotools setup demands… |
|
9582 so far I think librsvg is doing well:</p> |
|
9583 <ul> |
|
9584 <li> |
|
9585 <p>The <code>configure</code> script checks for <co… |
|
9586 </li> |
|
9587 <li> |
|
9588 <p>"<code>make distcheck</code>" works. This means th… |
|
9589 performed with <code>builddir != srcdir</code>, and also… |
|
9590 the available tests and they all pass.</p> |
|
9591 </li> |
|
9592 <li> |
|
9593 <p>The <code>rsvg_internals</code> library is built wi… |
|
9594 <code>Makefile.am</code> calls <code>cargo build<… |
|
9595 able to handle debug and release builds.</p> |
|
9596 </li> |
|
9597 <li> |
|
9598 <p>"<code>make clean</code>" cleans up the Rust build … |
|
9599 </li> |
|
9600 <li> |
|
9601 <p>If you change a <code>.rs</code> file and type <… |
|
9602 gets rebuilt.</p> |
|
9603 </li> |
|
9604 <li> |
|
9605 <p>Etcetera. I think librsvg feels like a normal autotool'ed libr… |
|
9606 Let's see how this is done.</p> |
|
9607 </li> |
|
9608 </ul> |
|
9609 <h1>Librsvg's basic autotools setup</h1> |
|
9610 <p>Librsvg started out with a fairly traditional autotools setup w… |
|
9611 <a href="https://git.gnome.org/browse/librsvg/tree/configure.ac?h=2.4… |
|
9612 historical reasons the <code>.[ch]</code> source files live … |
|
9613 <code>librsvg/</code> directory, not in a <code>src<… |
|
9614 that.</p> |
|
9615 <div class="highlight"><pre><span></span><cod… |
|
9616 ├ configure.ac |
|
9617 ├ Makefile.am |
|
9618 ├ *.[ch] |
|
9619 ├ src/ |
|
9620 ├ doc/ |
|
9621 ├ tests/ |
|
9622 └ win32/ |
|
9623 </code></pre></div> |
|
9624 |
|
9625 <h1>Adding Rust to the build</h1> |
|
9626 <p>The Rust source code lives in <a href="https://git.gnome.org… |
|
9627 where <a href="https://git.gnome.org/browse/librsvg/tree/rust/Cargo.t… |
|
9628 <code>src</code> subdirectory with the <code>*.rs</… |
|
9629 <div class="highlight"><pre><span></span><cod… |
|
9630 ├ configure.ac |
|
9631 ├ Makefile.am |
|
9632 ├ *.[ch] |
|
9633 ├ src/ |
|
9634 ├ rust/ &lt;--- this is new! |
|
9635 │ ├ Cargo.toml |
|
9636 │ └ src/ |
|
9637 ├ doc/ |
|
9638 ├ tests/ |
|
9639 └ win32/ |
|
9640 </code></pre></div> |
|
9641 |
|
9642 <h2>Detecting the presence of <code>cargo</code> and &… |
|
9643 <p>This goes in <code>configure.ac</code>:</p> |
|
9644 <div class="highlight"><pre><span></span><cod… |
|
9645 AS_IF<span class="o">(</span><span class="nb">test<… |
|
9646 AC_MSG_ERROR<span class="o">([</span>cargo is required. … |
|
9647 <span class="o">)</span> |
|
9648 AC_CHECK_PROG<span class="o">(</span>RUSTC, <span class="… |
|
9649 AS_IF<span class="o">(</span><span class="nb">test<… |
|
9650 AC_MSG_ERROR<span class="o">([</span>rustc is required. … |
|
9651 <span class="o">)</span> |
|
9652 </code></pre></div> |
|
9653 |
|
9654 <p>These two try to execute <code>cargo</code> and <… |
|
9655 with an error message if they are not present.</p> |
|
9656 <h2>Supporting debug or release mode for the Rust build</h2> |
|
9657 <p>One can call cargo like "<code>cargo build --release</… |
|
9658 optimizations, or normally like just "<code>cargo build</code&g… |
|
9659 debug information. That is, the latter is the default: if you don't |
|
9660 pass any options, cargo does a debug build.</p> |
|
9661 <p>Autotools and C compilers normally work a bit differently; one … |
|
9662 call the configure script like "<code>CFLAGS='-g -O0' ./configure&… |
|
9663 debug build, or "<code>CFLAGS='-O2 -fomit-frame-pointer' ./configu… |
|
9664 a release build.</p> |
|
9665 <p>Linux distros already have all the infrastructure to pass the |
|
9666 appropriate <code>CFLAGS</code> to <code>configure<… |
|
9667 appropriate flag to Cargo. My main requirement for this was:</p> |
|
9668 <ul> |
|
9669 <li>Distros shouldn't have to substantially change their RPM specf… |
|
9670 (or whatever) to accomodate the Rust build.</li> |
|
9671 <li>I assume that distros will want to make release builds by defa… |
|
9672 <li>I as a developer am comfortable with passing extra options to … |
|
9673 debug builds on my machine.</li> |
|
9674 </ul> |
|
9675 <p>The scheme in librsvg lets you run "<code>configure --ena… |
|
9676 make it call a plain <code>cargo build</code>, or a plain "&… |
|
9677 it use <code>cargo build --release</code> instead. The <… |
|
9678 usual through an environment variable. This way, distros don't have |
|
9679 to change their packaging to keep on making release builds as usual.<… |
|
9680 <p>This goes in <code>configure.ac</code>:</p> |
|
9681 <div class="highlight"><pre><span></span><cod… |
|
9682 dnl we build <span class="k">in</span> public release mode. |
|
9683 |
|
9684 AC_ARG_ENABLE<span class="o">(</span>debug, |
|
9685 AC_HELP_STRING<span class="o">([</span>--enabl… |
|
9686 <span class="o">[</span>Build R… |
|
9687 <span class="o">[</span><span class="nv">… |
|
9688 <span class="o">[</span><span class="nv">… |
|
9689 |
|
9690 AC_MSG_CHECKING<span class="o">(</span>whether to build Rust… |
|
9691 <span class="k">if</span> <span class="nb">test</sp… |
|
9692 AC_MSG_RESULT<span class="o">(</span>yes<span class="… |
|
9693 <span class="nv">RUST_TARGET_SUBDIR</span><span class… |
|
9694 <span class="k">else</span> |
|
9695 AC_MSG_RESULT<span class="o">(</span>no<span class="o… |
|
9696 <span class="nv">RUST_TARGET_SUBDIR</span><span class… |
|
9697 <span class="k">fi</span> |
|
9698 AM_CONDITIONAL<span class="o">([</span>DEBUG_RELEASE<span… |
|
9699 |
|
9700 AC_SUBST<span class="o">([</span>RUST_TARGET_SUBDIR<span … |
|
9701 </code></pre></div> |
|
9702 |
|
9703 <p>This defines an Automake conditional called <code>DEBUG_R… |
|
9704 will use in <code>Makefile.am</code> later.</p> |
|
9705 <p>It also causes <code>@RUST_TARGET_SUBDIR@</code> to… |
|
9706 with either <code>debug</code> or <code>release</co… |
|
9707 <h2>Adding Rust source files</h2> |
|
9708 <p>The <code>librsvg/rust/src</code> directory has all… |
|
9709 tracks their dependencies and whether they need to be rebuilt if one cha… |
|
9710 However, since that directory is not tracked by <code>make</cod… |
|
9711 rebuild things if a Rust source file changes! So, we need to tell our |
|
9712 <code>Makefile.am</code> about those files:</p> |
|
9713 <div class="highlight"><pre><span></span><cod… |
|
9714 rust/build.rs <span class="se">\</span> |
|
9715 rust/Cargo.toml <span class="se">\</span> |
|
9716 rust/src/aspect_ratio.rs <span class="se">\</span> |
|
9717 rust/src/bbox.rs <span class="se">\</span> |
|
9718 rust/src/cnode.rs <span class="se">\</span> |
|
9719 rust/src/color.rs <span class="se">\</span> |
|
9720 ... |
|
9721 |
|
9722 <span class="nv">RUST_EXTRA</span> <span class="o">=&l… |
|
9723 rust/Cargo.lock |
|
9724 |
|
9725 <span class="nv">EXTRA_DIST</span> <span class="o">+=&… |
|
9726 </code></pre></div> |
|
9727 |
|
9728 <p>It's a bit unfortunate that the change tracking is duplicated i… |
|
9729 <code>Makefile</code>, but we are already used to listing al… |
|
9730 in there, anyway.</p> |
|
9731 <p>Most notably, the <code>rust</code> subdirectory is… |
|
9732 in <code>Makefile.am</code>, since there is no <code>r… |
|
9733 <h2>Cargo release or debug build?</h2> |
|
9734 <div class="highlight"><pre><span></span><cod… |
|
9735 <span class="nv">CARGO_RELEASE_ARGS</span><span class="o"… |
|
9736 <span class="cp">else</span> |
|
9737 <span class="nv">CARGO_RELEASE_ARGS</span><span class="o"… |
|
9738 <span class="cp">endif</span> |
|
9739 </code></pre></div> |
|
9740 |
|
9741 <p>We will call <code>cargo build</code> with that arg… |
|
9742 <h2>Verbose or quiet build?</h2> |
|
9743 <p>Librsvg uses <code>AM_SILENT_RULES([yes])</code> in… |
|
9744 you just run "<code>make</code>" for a quiet build, or "<… |
|
9745 full command lines passed to the compiler. Cargo supports something |
|
9746 similar, so let's add it to <code>Makefile.am</code>:</p&… |
|
9747 <div class="highlight"><pre><span></span><cod… |
|
9748 <span class="nv">cargo_verbose_</span> <span class="o">… |
|
9749 <span class="nv">cargo_verbose_0</span> <span class="o"&g… |
|
9750 <span class="nv">cargo_verbose_1</span> <span class="o"&g… |
|
9751 </code></pre></div> |
|
9752 |
|
9753 <p>This expands the <code>V</code> variable to empty, … |
|
9754 expanding <em>that</em> gives us the final command-line argu… |
|
9755 <code>CARGO_VERBOSE</code> variable.</p> |
|
9756 <h2>What's the filename of the library we are building?</h2> |
|
9757 <div class="highlight"><pre><span></span><cod… |
|
9758 </code></pre></div> |
|
9759 |
|
9760 <p>Remember our <code>@RUST_TARGET_SUBDIR@</code> from… |
|
9761 plain "<code>cargo build</code>", it will put the binaries in |
|
9762 <code>rust/target/debug</code>. But if you call "<code&g… |
|
9763 will put the binaries in <code>rust/target/release</code>.&l… |
|
9764 <p>With the bit above, the <code>RUST_LIB</code> varia… |
|
9765 for the built library. The <code>@abs_top_builddir@</code> … |
|
9766 the build directory is not the same as the source directory.</p> |
|
9767 <h2>Okay, so how do we call <code>cargo</code>?</h2… |
|
9768 <div class="highlight"><pre><span></span><cod… |
|
9769 <span class="nb">cd</span> <span class="k">$(</… |
|
9770 <span class="nv">CARGO_TARGET_DIR</span><span class="… |
|
9771 </code></pre></div> |
|
9772 |
|
9773 <p>We make the funky library filename depend on <code>$(RUST… |
|
9774 That's what will cause <code>make</code> to rebuild the Rust… |
|
9775 the Rust source files changes.</p> |
|
9776 <p>We override the <code>CARGO_TARGET_DIR</code> with … |
|
9777 call <code>cargo build</code> with the correct arguments.<… |
|
9778 <h2>Linking into the main C library</h2> |
|
9779 <div class="highlight"><pre><span></span><cod… |
|
9780 <span class="k">$(</span>LIBRSVG_LIBS<span class=… |
|
9781 <span class="k">$(</span>LIBM<span class="k">)… |
|
9782 <span class="k">$(</span>RUST_LIB<span class="k"&… |
|
9783 </code></pre></div> |
|
9784 |
|
9785 <p>This expands our <code>$(RUST_LIB)</code> from abov… |
|
9786 with librsvg's other dependencies.</p> |
|
9787 <h2><code>make check</code></h2> |
|
9788 <p>This is our hook so that <code>make check</code> wi… |
|
9789 <div class="highlight"><pre><span></span><cod… |
|
9790 <span class="nb">cd</span> <span class="k">$(&… |
|
9791 <span class="nv">CARGO_TARGET_DIR</span><span cla… |
|
9792 </code></pre></div> |
|
9793 |
|
9794 <h2><code>make clean</code></h2> |
|
9795 <p>Same thing for <code>make clean</code> and <code… |
|
9796 <div class="highlight"><pre><span></span><cod… |
|
9797 <span class="nb">cd</span> <span class="k">$(&… |
|
9798 <span class="nv">CARGO_TARGET_DIR</span><span cla… |
|
9799 </code></pre></div> |
|
9800 |
|
9801 <h1>Vendoring dependencies</h1> |
|
9802 <p>Linux distros probably want Rust packages to come bundled with … |
|
9803 dependencies, so that they can replace them later with newer/patched |
|
9804 versions.</p> |
|
9805 <p>Here is a hook so that <code>make dist</code> will … |
|
9806 run before making the tarball. That command will creates a |
|
9807 <code>rust/vendor</code> directory with a copy of all the Ru… |
|
9808 librsvg depends on.</p> |
|
9809 <div class="highlight"><pre><span></span><cod… |
|
9810 |
|
9811 <span class="nf">dist-hook</span><span class="o">:<… |
|
9812 <span class="o">(</span><span class="nb">cd</sp… |
|
9813 cargo vendor -q <span class="o">&amp;&amp;</span>… |
|
9814 mkdir .cargo <span class="o">&amp;&amp;</span> &… |
|
9815 cp cargo-vendor-config .cargo/config<span class="o">)</span… |
|
9816 </code></pre></div> |
|
9817 |
|
9818 <p>The tarball needs to have a <code>rust/.cargo/config</… |
|
9819 the vendored sources (i.e. the embedded dependencies), but we don't |
|
9820 want <em>that</em> in our development source tree. Instead,… |
|
9821 from a <a href="https://git.gnome.org/browse/librsvg/tree/rust/cargo-… |
|
9822 source tree:</p> |
|
9823 <div class="highlight"><pre><span></span><cod… |
|
9824 <span class="c1">#</span> |
|
9825 <span class="c1"># In the distributed tarball, this file should en… |
|
9826 <span class="c1"># rust/.cargo/config</span> |
|
9827 |
|
9828 <span class="k">[source.crates-io]</span> |
|
9829 <span class="n">registry</span> <span class="o">=</… |
|
9830 <span class="n">replace-with</span> <span class="o">=&… |
|
9831 |
|
9832 <span class="k">[source.vendored-sources]</span> |
|
9833 <span class="n">directory</span> <span class="o">=<… |
|
9834 </code></pre></div> |
|
9835 |
|
9836 <h1>One last thing</h1> |
|
9837 <p>If you put this in your <code>Cargo.toml</code>, re… |
|
9838 smaller. This turns on link-time optimizations (LTO), which removes |
|
9839 unused functions from the binary.</p> |
|
9840 <div class="highlight"><pre><span></span><cod… |
|
9841 <span class="n">lto</span> <span class="o">=</span&… |
|
9842 </code></pre></div> |
|
9843 |
|
9844 <h1>Summary and thanks</h1> |
|
9845 <p>I think the above is some good boilerplate that you can put in … |
|
9846 <code>configure.ac</code> / <code>Makefile.am</code… |
|
9847 your C code. It handles <code>make</code>-y things like <… |
|
9848 check</code>; debug and release builds; verbose and quiet builds; |
|
9849 <code>builddir != srcdir</code>; all the goodies.</p> |
|
9850 <p>I think the only thing I'm missing is to check for the <code… |
|
9851 binary. I'm not sure how to only check for that if I'm the one making |
|
9852 tarballs... maybe an <code>--enable-maintainer-mode</code> f… |
|
9853 <p>This would definitely not have been possible without prior work. |
|
9854 Thanks to everyone who figured out Autotools before me, so I could |
|
9855 cut&amp;paste your goodies:</p> |
|
9856 <p><em>Update 2017/Nov/11:</em> Fixed the initializati… |
|
9857 to Tobias Mueller for catching this.</p> |
|
9858 <ul> |
|
9859 <li><a href="https://www.figuiere.net/hub/blog/?2016/10/07/862-… |
|
9860 <li><a href="http://lukenukem.co.nz/gsoc/2017/05/17/gso_2.html"… |
|
9861 <li><a href="https://github.com/endlessm/ostree/commit/9169268c… |
|
9862 <li><a href="https://blog.ometer.com/2017/01/10/dear-package-ma… |
|
9863 </ul></content><category term="misc"></category><category term="li… |
|
9864 <p>In the <a href="https://people.gnome.org/~federico/blog/how-… |
|
9865 the <a href="http://gtk-rs.org/docs/glib/translate/trait.FromGlib.htm… |
|
9866 code convert from/to Glib's simple types, like to convert from a Glib |
|
9867 <code>gboolean</code> to a Rust <code>bool</code>… |
|
9868 <p>In the <a href="https://people.gnome.org/~federico/blog/how-… |
|
9869 the <a href="http://gtk-rs.org/docs/glib/translate/trait.FromGlib.htm… |
|
9870 code convert from/to Glib's simple types, like to convert from a Glib |
|
9871 <code>gboolean</code> to a Rust <code>bool</code>… |
|
9872 needs of strings; since they are passed by reference and are not |
|
9873 copied as simple values, we can use |
|
9874 <a href="http://gtk-rs.org/docs/glib/translate/trait.FromGlibPtrNone.… |
|
9875 <a href="http://gtk-rs.org/docs/glib/translate/trait.FromGlibPtrFull.… |
|
9876 <em>ownership transfer</em> we want, none for "just make it … |
|
9877 using a borrowed reference", or full for "I'll take over the data and |
|
9878 free it when I'm done". Going the other way around, we can use |
|
9879 <a href="http://gtk-rs.org/docs/glib/translate/trait.ToGlibPtr.html"&… |
|
9880 Glib.</p> |
|
9881 <p>In this part, we'll see the tools that glib-rs provides to do |
|
9882 conversions of more complex data types. We'll look at two cases:</p&… |
|
9883 <ul> |
|
9884 <li> |
|
9885 <p><a href="#null-term-string-array">Passing null-terminated… |
|
9886 from Glib to Rust</p> |
|
9887 </li> |
|
9888 <li> |
|
9889 <p><a href="#passing-glists">Passing <code>GList</c… |
|
9890 </li> |
|
9891 </ul> |
|
9892 <p>And one final case just in passing:</p> |
|
9893 <ul> |
|
9894 <li><a href="#passing-containers-to-glib">Passing containers… |
|
9895 </ul> |
|
9896 <h1>Passing arrays from Glib to Rust</h1> |
|
9897 <p>We'll look at the case for transferring null-terminated arrays … |
|
9898 strings, since it's an interesting one. There are other traits to |
|
9899 convert from Glib arrays whose length is known, not implied with a |
|
9900 NULL element, but for now we'll only look at arrays of strings.</p> |
|
9901 <h2 id="null-term-string-array">Null-terminated arrays of strings&… |
|
9902 <p>Look at this function for <code>GtkAboutDialog</code&g… |
|
9903 <div class="highlight"><pre><span></span><cod… |
|
9904 <span class="cm"> * gtk_about_dialog_add_credit_section:</span&… |
|
9905 <span class="cm"> * @about: A #GtkAboutDialog</span> |
|
9906 <span class="cm"> * @section_name: The name of the section</spa… |
|
9907 <span class="cm"> * @people: (array zero-terminated=1): The people… |
|
9908 <span class="cm"> * ...</span> |
|
9909 <span class="cm"> */</span> |
|
9910 <span class="kt">void</span> |
|
9911 <span class="n">gtk_about_dialog_add_credit_section</span> &… |
|
9912 <span class="k">const</spa… |
|
9913 <span class="k">const</spa… |
|
9914 </code></pre></div> |
|
9915 |
|
9916 <p>You would use this like</p> |
|
9917 <div class="highlight"><pre><span></span><cod… |
|
9918 <span class="s">&quot;Alice &lt;[email protected]&… |
|
9919 <span class="s">&quot;Bob &lt;[email protected]&gt;&… |
|
9920 <span class="s">&quot;Clara &lt;[email protected]&… |
|
9921 <span class="nb">NULL</span> |
|
9922 <span class="p">};</span> |
|
9923 |
|
9924 <span class="n">gtk_about_dialog_add_credit_section</span> &… |
|
9925 </code></pre></div> |
|
9926 |
|
9927 <p>The function expects an array of <code>gchar *</code&g… |
|
9928 a NULL. Instead of passing an explicit length for the array, it's |
|
9929 done implicitly by requiring a NULL pointer after the last element. |
|
9930 The gtk-doc annotation says <code>(array zero-terminated=1)</co… |
|
9931 generate information for the GObject-Introspection Repository (GIR), |
|
9932 this is what comes out:</p> |
|
9933 <table class="highlighttable"><tr><td class="linenos">… |
|
9934 <span class="normal"> 2</span> |
|
9935 <span class="normal"> 3</span> |
|
9936 <span class="normal"> 4</span> |
|
9937 <span class="normal"> 5</span> |
|
9938 <span class="normal"> 6</span> |
|
9939 <span class="normal"> 7</span> |
|
9940 <span class="normal"> 8</span> |
|
9941 <span class="normal"> 9</span> |
|
9942 <span class="normal">10</span></pre></div></t… |
|
9943 <span class="na">c:identifier=</span><span class=… |
|
9944 <span class="na">version=</span><span class="s"&g… |
|
9945 .. |
|
9946 <span class="nt">&lt;parameter</span> <span class… |
|
9947 <span class="nt">&lt;doc</span> <span class="na… |
|
9948 <span class="nt">&lt;array</span> <span class="… |
|
9949 <span class="nt">&lt;type</span> <span class=… |
|
9950 <span class="nt">&lt;/array&gt;</span> |
|
9951 <span class="nt">&lt;/parameter&gt;</span> |
|
9952 </code></pre></div> |
|
9953 </td></tr></table> |
|
9954 <p>You can see the <code>transfer-ownership="none"</code&… |
|
9955 that the function will not take ownership of the passed array; it will |
|
9956 make its own copy instead. By convention, GIR assumes that arrays of |
|
9957 strings are NULL-terminated, so there is no special annotation for |
|
9958 that here. If we were implementing this function in Rust, how would we |
|
9959 read that C array of UTF-8 strings and turn it into a Rust |
|
9960 <code>Vec&lt;String&gt;</code> or something? Easy:&… |
|
9961 <div class="highlight"><pre><span></span><cod… |
|
9962 <span class="kd">let</span><span class="w"> </span&… |
|
9963 <span class="c1">// rust_translators is a Vec&lt;String&gt… |
|
9964 </code></pre></div> |
|
9965 |
|
9966 <p>Let's look at how this bad boy is implemented.</p> |
|
9967 <h3>First stage: <code>impl FromGlibPtrContainer for Vec&… |
|
9968 <p>We want to go from a "<code>*mut *mut c_char</code&g… |
|
9969 to a <code>Vec&lt;String&gt;</code>. Indeed, there … |
|
9970 <code>FromGlibPtrContainer</code> trait for <code>Vec&… |
|
9971 <a href="https://github.com/gtk-rs/glib/blob/e46aa7f27cc74f3cdcb54d94… |
|
9972 <div class="highlight"><pre><span></span><cod… |
|
9973 <span class="w"> </span><span class="k">unsafe</… |
|
9974 <span class="w"> </span><span class="n">FromGli… |
|
9975 <span class="w"> </span><span class="p">}</span&… |
|
9976 </code></pre></div> |
|
9977 |
|
9978 <p>So... that <code>from_glib_none()</code> will retur… |
|
9979 want. Let's look at the first few lines of <a href="https://github.c… |
|
9980 <table class="highlighttable"><tr><td class="linenos">… |
|
9981 <span class="normal">2</span> |
|
9982 <span class="normal">3</span> |
|
9983 <span class="normal">4</span></pre></div></td… |
|
9984 <span class="w"> </span><span class="k">unsafe&… |
|
9985 <span class="w"> </span><span class="n">Fro… |
|
9986 <span class="w"> </span><span class="p">}</s… |
|
9987 </code></pre></div> |
|
9988 </td></tr></table> |
|
9989 <p>Aha! This is inside a <a href="https://github.com/gtk-rs/gl… |
|
9990 It's done like that so the same trait can be implemented for <code>… |
|
9991 <code>mut</code> pointers to <code>c_char</code>… |
|
9992 <p>See the call to <code>c_ptr_array_len()</code> in l… |
|
9993 out where the NULL pointer is at the end of the array: it figures out |
|
9994 the array's length. </p> |
|
9995 <h3>Second stage: <code>impl FromGlibContainerAsVec::from_gl… |
|
9996 <p>Now that the length of the array is known, the implementation c… |
|
9997 <a href="https://github.com/gtk-rs/glib/blob/e46aa7f27cc74f3cdcb54d94… |
|
9998 <table class="highlighttable"><tr><td class="linenos">… |
|
9999 <span class="normal"> 2</span> |
|
10000 <span class="normal"> 3</span> |
|
10001 <span class="normal"> 4</span> |
|
10002 <span class="normal"> 5</span> |
|
10003 <span class="normal"> 6</span> |
|
10004 <span class="normal"> 7</span> |
|
10005 <span class="normal"> 8</span> |
|
10006 <span class="normal"> 9</span> |
|
10007 <span class="normal">10</span> |
|
10008 <span class="normal">11</span> |
|
10009 <span class="normal">12</span></pre></div></t… |
|
10010 <span class="w"> </span><span class="k">unsafe&… |
|
10011 <span class="w"> </span><span class="k">if&… |
|
10012 <span class="w"> </span><span class="k">… |
|
10013 <span class="w"> </span><span class="p">}&l… |
|
10014 |
|
10015 <span class="w"> </span><span class="kd">le… |
|
10016 <span class="w"> </span><span class="k">for… |
|
10017 <span class="w"> </span><span class="n">… |
|
10018 <span class="w"> </span><span class="p">}&l… |
|
10019 <span class="w"> </span><span class="n">res… |
|
10020 <span class="w"> </span><span class="p">}</s… |
|
10021 </code></pre></div> |
|
10022 </td></tr></table> |
|
10023 <p>Lines 3/4: If the number of elements is zero, or the array is N… |
|
10024 return an empty <code>Vec</code>.</p> |
|
10025 <p>Line 7: Allocate a <code>Vec</code> of suitable siz… |
|
10026 <p>Lines 8/9: For each of the pointers in the C array, call |
|
10027 <code>from_glib_none()</code> to convert it from a <code&… |
|
10028 like we saw in <a href="https://people.gnome.org/~federico/blog/how-g… |
|
10029 <p>Done! We started with a <code>*mut *mut c_char</code&… |
|
10030 c_char</code> and ended up with a <code>Vec&lt;String&am… |
|
10031 <h1 id="passing-glists">Passing <code>GList</code>s to… |
|
10032 <p>Some functions don't give you an array; they give you a <cod… |
|
10033 <code>GSList</code>. There is an implementation of |
|
10034 <code>FromGlibPtrArrayContainerAsVec</code> <a href="http… |
|
10035 <code>GList</code></a>:</p> |
|
10036 <div class="highlight"><pre><span></span><cod… |
|
10037 <span class="k">where</span><span class="w"> </span… |
|
10038 |
|
10039 <span class="w"> </span><span class="k">unsafe</… |
|
10040 <span class="w"> </span><span class="kd">let<… |
|
10041 <span class="w"> </span><span class="n">FromGli… |
|
10042 <span class="w"> </span><span class="p">}</span&… |
|
10043 </code></pre></div> |
|
10044 |
|
10045 <p>The <code>impl</code> declaration is pretty horribl… |
|
10046 method: <code>from_glib_none_as_vec()</code> takes in a <… |
|
10047 <code>g_list_length()</code> on it, and finally calls |
|
10048 <code>FromGlibContainer::from_glib_none_num()</code> with th… |
|
10049 <h3>I have a Glib container and its length</h3> |
|
10050 <p>In turn, that <code>from_glib_none_num()</code> goe… |
|
10051 <div class="highlight"><pre><span></span><cod… |
|
10052 <span class="w"> </span><span class="k">unsafe</… |
|
10053 <span class="w"> </span><span class="n">FromGli… |
|
10054 <span class="w"> </span><span class="p">}</span&… |
|
10055 </code></pre></div> |
|
10056 |
|
10057 <p>Okay, getting closer to the actual implementation.</p> |
|
10058 <h3>Give me a vector already</h3> |
|
10059 <p>Finally, we get to the function that <a href="https://github… |
|
10060 <table class="highlighttable"><tr><td class="linenos">… |
|
10061 <span class="normal"> 2</span> |
|
10062 <span class="normal"> 3</span> |
|
10063 <span class="normal"> 4</span> |
|
10064 <span class="normal"> 5</span> |
|
10065 <span class="normal"> 6</span> |
|
10066 <span class="normal"> 7</span> |
|
10067 <span class="normal"> 8</span> |
|
10068 <span class="normal"> 9</span> |
|
10069 <span class="normal">10</span> |
|
10070 <span class="normal">11</span> |
|
10071 <span class="normal">12</span> |
|
10072 <span class="normal">13</span> |
|
10073 <span class="normal">14</span> |
|
10074 <span class="normal">15</span> |
|
10075 <span class="normal">16</span> |
|
10076 <span class="normal">17</span></pre></div></t… |
|
10077 <span class="k">where</span><span class="w"> </span… |
|
10078 |
|
10079 <span class="w"> </span><span class="k">unsafe</… |
|
10080 <span class="w"> </span><span class="k">if</… |
|
10081 <span class="w"> </span><span class="k">ret… |
|
10082 <span class="w"> </span><span class="p">}</s… |
|
10083 <span class="w"> </span><span class="kd">let<… |
|
10084 <span class="w"> </span><span class="k">for<… |
|
10085 <span class="w"> </span><span class="kd">le… |
|
10086 <span class="w"> </span><span class="k">if&… |
|
10087 <span class="w"> </span><span class="n">… |
|
10088 <span class="w"> </span><span class="p">}&l… |
|
10089 <span class="w"> </span><span class="n">ptr… |
|
10090 <span class="w"> </span><span class="p">}</s… |
|
10091 <span class="w"> </span><span class="n">res<… |
|
10092 <span class="w"> </span><span class="p">}</span&… |
|
10093 </code></pre></div> |
|
10094 </td></tr></table> |
|
10095 <p>Again, ignore the horrible <code>impl</code> declar… |
|
10096 <code>from_glib_none_num_as_vec()</code>.</p> |
|
10097 <p>Line 4: that function takes in a <code>ptr</code> t… |
|
10098 the list's length, which we already computed above.</p> |
|
10099 <p>Line 5: Return an empty vector if we have an empty list.</p&… |
|
10100 <p>Line 8: Allocate a vector of suitable capacity.</p> |
|
10101 <p>Line 9: For each element, convert it with <code>from_glib… |
|
10102 it to the array.</p> |
|
10103 <p>Line 14: Walk to the next element in the list.</p> |
|
10104 <h1 id="passing-containers-to-glib">Passing containers from Rust t… |
|
10105 <p>This post is getting a bit long, so I'll just mention this brie… |
|
10106 There is a trait <code>ToGlibContainerFromSlice</code> that … |
|
10107 and can convert it to various Glib types.</p> |
|
10108 <ul> |
|
10109 <li> |
|
10110 <p>To <a href="https://github.com/gtk-rs/glib/blob/e46aa7f27cc7… |
|
10111 methods like <code>to_glib_none_from_slice()</code> and |
|
10112 <code>to_glib_full_from_slice()</code></p> |
|
10113 </li> |
|
10114 <li> |
|
10115 <p>To an <a href="https://github.com/gtk-rs/glib/blob/e46aa7f27… |
|
10116 between <code>to_glib_none_from_slice()</code>, which give… |
|
10117 we saw <a href="https://people.gnome.org/~federico/blog/how-glib-rs… |
|
10118 <code>to_glib_full_from_slice()</code>, which gives you ba… |
|
10119 array with copied items. Finally, <code>to_glib_container_from_… |
|
10120 gives you back a <code>g_malloc()</code>ed array of <em… |
|
10121 than plain values themselves. Which function you choose depends on |
|
10122 which C API you want to call.</p> |
|
10123 </li> |
|
10124 </ul> |
|
10125 <p>I hope this post gives you enough practice to be able to "follo… |
|
10126 traits" for each of those if you want to look at the implementations.<… |
|
10127 <h1>Next up</h1> |
|
10128 <p>Passing boxed types, like public structs.</p> |
|
10129 <p>Passing reference-counted types.</p> |
|
10130 <p>How glib-rs wraps GObjects.</p></content><category term="… |
|
10131 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10132 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10133 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10134 </ul> |
|
10135 <p>During the <a href="https://wiki.gnome.org/Hackfests/Rust201… |
|
10136 started the implementation of <a href="http://smallcultfollowing.com/… |
|
10137 that will let people implement new GObject classes in …</p></sum… |
|
10138 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10139 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10140 <li><a href="https://people.gnome.org/~federico/blog/how-glib-r… |
|
10141 </ul> |
|
10142 <p>During the <a href="https://wiki.gnome.org/Hackfests/Rust201… |
|
10143 started the implementation of <a href="http://smallcultfollowing.com/… |
|
10144 that will let people implement new GObject classes in Rust and export |
|
10145 them to the world. Currently, if you want to write a new GObject |
|
10146 (e.g. a new widget) and put it in a library so that it can be used |
|
10147 from language bindings via GObject-Introspection, you have to do it in |
|
10148 C. It would be nice to be able to do this in a safe language like |
|
10149 Rust.</p> |
|
10150 <h1>How would it be done by hand?</h1> |
|
10151 <p>In a C implementation of a new GObject subclass, one calls thin… |
|
10152 <code>g_type_register_static()</code> and <code>g_sign… |
|
10153 careful to specify the correct <code>GType</code> for each v… |
|
10154 super-careful about everything, as C demands.</p> |
|
10155 <p>In Rust, one <em>can</em> in fact do exactly the sa… |
|
10156 same, low-level GObject and GType functions. You can use |
|
10157 <code>#[repr(C)]</code>] for the instance and class structs … |
|
10158 allocate for you, and which you then fill in.</p> |
|
10159 <p>You can see an example of this in gst-plugins-rs. This is wher… |
|
10160 GObject, in Rust, by calling Glib functions by |
|
10161 hand: <a href="https://github.com/sdroege/gst-plugin-rs/blob/782fe5dc… |
|
10162 <a href="https://github.com/sdroege/gst-plugin-rs/blob/782fe5dcc93dbd… |
|
10163 <h1>How would it be done by a machine?</h1> |
|
10164 <p>That's what Niko's gnome-class is about. During the hackfest it |
|
10165 got to the point of being able to generate the code to create a new |
|
10166 GObject subclass, register it, and export functions for methods. The |
|
10167 syntax is not finalized yet, but it looks something like this:</p> |
|
10168 <div class="highlight"><pre><span></span><cod… |
|
10169 <span class="w"> </span><span class="n">class</s… |
|
10170 <span class="w"> </span><span class="k">struct&… |
|
10171 <span class="w"> </span><span class="n">val… |
|
10172 <span class="w"> </span><span class="p">}</s… |
|
10173 |
|
10174 <span class="w"> </span><span class="n">signal&… |
|
10175 |
|
10176 <span class="w"> </span><span class="k">fn</… |
|
10177 <span class="w"> </span><span class="kd">le… |
|
10178 <span class="w"> </span><span class="n">pri… |
|
10179 <span class="w"> </span><span class="c1">//… |
|
10180 <span class="w"> </span><span class="p">}</s… |
|
10181 |
|
10182 <span class="w"> </span><span class="k">fn</… |
|
10183 <span class="w"> </span><span class="kd">le… |
|
10184 <span class="w"> </span><span class="n">pri… |
|
10185 <span class="w"> </span><span class="p">}</s… |
|
10186 <span class="w"> </span><span class="p">}</span&… |
|
10187 <span class="p">}</span><span class="w"></span> |
|
10188 </code></pre></div> |
|
10189 |
|
10190 <p>I started adding support for declaring GObject signals — main… |
|
10191 able to parse them from what goes inside <code>gobject_gen!()</… |
|
10192 being able to call <code>g_signal_newv()</code> at the appro… |
|
10193 the <code>class_init()</code> implementation.</p> |
|
10194 <h1>Types in signals</h1> |
|
10195 <p>Creating a signal for a GObject class is basically like specify… |
|
10196 function prototype: the object will invoke a callback function with |
|
10197 certain arguments and return value when the signal is emitted. For |
|
10198 example, this is how <code>GtkButton</code> registers its &l… |
|
10199 signal</a>:</p> |
|
10200 <div class="highlight"><pre><span></span><cod… |
|
10201 <span class="n">g_signal_new</span> <span class="p"&g… |
|
10202 <span class="p">...</span> |
|
10203 <span class="n">G_TYPE_BOOLEAN</span><s… |
|
10204 <span class="mi">1</span><span class="p… |
|
10205 <span class="n">GDK_TYPE_EVENT</span><s… |
|
10206 </code></pre></div> |
|
10207 |
|
10208 <p><code>g_signal_new()</code> creates the signal and … |
|
10209 integer. Later, when the object wants to emit the signal, it uses |
|
10210 that signal id like this:</p> |
|
10211 <div class="highlight"><pre><span></span><cod… |
|
10212 <span class="n">gboolean</span> <span class="n">return… |
|
10213 |
|
10214 <span class="n">g_signal_emit</span> <span class="p">(… |
|
10215 </code></pre></div> |
|
10216 |
|
10217 <p>In the nice <code>gobject_gen!()</code> macro, if I… |
|
10218 declaration like</p> |
|
10219 <div class="highlight"><pre><span></span><cod… |
|
10220 </code></pre></div> |
|
10221 |
|
10222 <p>then I will need to be able to translate the type names for |
|
10223 <code>ButtonPressEvent</code> and <code>bool</code&… |
|
10224 understand: I need the <a href="https://developer.gnome.org/gobject/… |
|
10225 types like <code>gboolean</code> get constants like <a hr… |
|
10226 that are defined at runtime, like <code>GDK_TYPE_EVENT</code>… |
|
10227 generated at runtime, too, when one registers the type with |
|
10228 <code>g_type_register_*()</code>.</p> |
|
10229 <table> |
|
10230 <thead> |
|
10231 <tr> |
|
10232 <th align="left">Rust type</th> |
|
10233 <th align="left">GType</th> |
|
10234 </tr> |
|
10235 </thead> |
|
10236 <tbody> |
|
10237 <tr> |
|
10238 <td align="left">i32</td> |
|
10239 <td align="left">G_TYPE_INT</td> |
|
10240 </tr> |
|
10241 <tr> |
|
10242 <td align="left">u32</td> |
|
10243 <td align="left">G_TYPE_UINT</td> |
|
10244 </tr> |
|
10245 <tr> |
|
10246 <td align="left">bool</td> |
|
10247 <td align="left">G_TYPE_BOOLEAN</td> |
|
10248 </tr> |
|
10249 <tr> |
|
10250 <td align="left">etc.</td> |
|
10251 <td align="left">etc.</td> |
|
10252 </tr> |
|
10253 </tbody> |
|
10254 </table> |
|
10255 <h1>Glib types in Rust</h1> |
|
10256 <p>How does <a href="http://gtk-rs.org/docs/glib/">glib-rs&l… |
|
10257 types?</p> |
|
10258 <h2>Going from Glib to Rust</h2> |
|
10259 <p>First we need a way to convert Glib's types to Rust, and vice-v… |
|
10260 There is a trait to convert simple Glib types into Rust types:</p> |
|
10261 <div class="highlight"><pre><span></span><cod… |
|
10262 <span class="w"> </span><span class="k">fn</span… |
|
10263 <span class="p">}</span><span class="w"></span> |
|
10264 </code></pre></div> |
|
10265 |
|
10266 <p>This means, if I have a <code>T</code> which is a G… |
|
10267 you a <code>from_glib()</code> function which will convert i… |
|
10268 which is <code>Sized</code>, i.e. a type whose size is known… |
|
10269 <p>For example, this is how it is implemented for booleans:</p&… |
|
10270 <div class="highlight"><pre><span></span><cod… |
|
10271 <span class="w"> </span><span class="cp">#[inline]&… |
|
10272 <span class="w"> </span><span class="k">fn</span… |
|
10273 <span class="w"> </span><span class="o">!</s… |
|
10274 <span class="w"> </span><span class="p">}</span&… |
|
10275 <span class="p">}</span><span class="w"></span> |
|
10276 </code></pre></div> |
|
10277 |
|
10278 <p>and you use it like this:</p> |
|
10279 <div class="highlight"><pre><span></span><cod… |
|
10280 |
|
10281 <span class="kd">let</span><span class="w"> </span&… |
|
10282 </code></pre></div> |
|
10283 |
|
10284 <p>Booleans in glib and Rust have <a href="https://people.gnome… |
|
10285 different values. Glib's booleans use the C convention: 0 is false |
|
10286 and anything else is true, while in Rust booleans are strictly <code&… |
|
10287 or <code>true</code>, and the size is undefined (with the cu… |
|
10288 one byte).</p> |
|
10289 <h2>Going from Rust to Glib</h2> |
|
10290 <p>And to go the other way around, from a Rust <code>bool<… |
|
10291 There is this trait:</p> |
|
10292 <div class="highlight"><pre><span></span><cod… |
|
10293 <span class="w"> </span><span class="k">type</sp… |
|
10294 |
|
10295 <span class="w"> </span><span class="k">fn</span… |
|
10296 <span class="p">}</span><span class="w"></span> |
|
10297 </code></pre></div> |
|
10298 |
|
10299 <p>This means, if you have a Rust type that maps to a corresponding |
|
10300 <code>GlibType</code>, this will give you a <code>to_g… |
|
10301 conversion.</p> |
|
10302 <p>This is the implementation for booleans:</p> |
|
10303 <div class="highlight"><pre><span></span><cod… |
|
10304 <span class="w"> </span><span class="k">type</sp… |
|
10305 |
|
10306 <span class="w"> </span><span class="cp">#[inline]&… |
|
10307 <span class="w"> </span><span class="k">fn</span… |
|
10308 <span class="w"> </span><span class="k">if</… |
|
10309 <span class="w"> </span><span class="p">}</span&… |
|
10310 <span class="p">}</span><span class="w"></span> |
|
10311 </code></pre></div> |
|
10312 |
|
10313 <p>And it is used like this:</p> |
|
10314 <div class="highlight"><pre><span></span><cod… |
|
10315 |
|
10316 <span class="n">g_some_function_that_takes_gboolean</span>&l… |
|
10317 </code></pre></div> |
|
10318 |
|
10319 <p>(If you are thinking "a function call to marshal a boolean" —… |
|
10320 the functions are inlined, and the optimizer basically compiles them |
|
10321 down to nothing.)</p> |
|
10322 <h2 id="pointer-types">Pointer types - from Glib to Rust</h2> |
|
10323 <p>That's all very nice for simple types like booleans and ints. |
|
10324 Pointers to other objects are slightly more complicated.</p> |
|
10325 <p>GObject-Introspection allows one to specify how pointer argumen… |
|
10326 functions are handled by using a <a href="https://developer.gnome.org… |
|
10327 <h3><code>(transfer none)</code></h3> |
|
10328 <p>For example, if you call <code>gtk_window_set_title(windo… |
|
10329 would expect the function to make its own copy of the <code>"Hello… |
|
10330 string. In Rust terms, you would be passing it a simple <em>borro… |
|
10331 reference</em>. GObject-Introspection (we'll abbreviate it as GI)… |
|
10332 this <code>GI_TRANSFER_NOTHING</code>, and it's specified by… |
|
10333 <code>(transfer none)</code> in the documentation strings f… |
|
10334 or return values.</p> |
|
10335 <p>The corresponding trait to bring in pointers from Glib to Rust, |
|
10336 without taking ownership, is this. It's <code>unsafe</code>… |
|
10337 used to de-reference pointers that come from the wild west:</p> |
|
10338 <div class="highlight"><pre><span></span><cod… |
|
10339 <span class="w"> </span><span class="k">unsafe</… |
|
10340 <span class="p">}</span><span class="w"></span> |
|
10341 </code></pre></div> |
|
10342 |
|
10343 <p>And you use it via this generic function:</p> |
|
10344 <div class="highlight"><pre><span></span><cod… |
|
10345 <span class="k">pub</span><span class="w"> </span&g… |
|
10346 <span class="w"> </span><span class="n">FromGlibPtr… |
|
10347 <span class="p">}</span><span class="w"></span> |
|
10348 </code></pre></div> |
|
10349 |
|
10350 <p>Let's look at how this works. Here is the <code>FromGlib… |
|
10351 implemented for strings.</p> |
|
10352 <table class="highlighttable"><tr><td class="linenos">… |
|
10353 <span class="normal">2</span> |
|
10354 <span class="normal">3</span> |
|
10355 <span class="normal">4</span> |
|
10356 <span class="normal">5</span> |
|
10357 <span class="normal">6</span> |
|
10358 <span class="normal">7</span></pre></div></td… |
|
10359 <span class="w"> </span><span class="cp">#[inline]&… |
|
10360 <span class="w"> </span><span class="k">unsafe</… |
|
10361 <span class="w"> </span><span class="fm">assert… |
|
10362 <span class="w"> </span><span class="nb">String… |
|
10363 <span class="w"> </span><span class="p">}</span&… |
|
10364 <span class="p">}</span><span class="w"></span> |
|
10365 </code></pre></div> |
|
10366 </td></tr></table> |
|
10367 <p>Line 1: given a pointer to a <code>c_char</code>, t… |
|
10368 <p>Line 4: check for NULL pointers</p> |
|
10369 <p>Line 5: Use the CStr to wrap the C |
|
10370 <code>ptr</code>, <a href="https://people.gnome.org/~fede… |
|
10371 copy the string for us.</p> |
|
10372 <p>Unfortunately, there's a copy involved in the last step. It ma… |
|
10373 possible to use <a href="https://doc.rust-lang.org/std/borrow/enum.Co… |
|
10374 the <code>char*</code> from Glib is indeed valid UTF-8.</… |
|
10375 <h3><code>(transfer full)</code></h3> |
|
10376 <p>And how about transferring ownership of the pointed-to value? … |
|
10377 is this trait:</p> |
|
10378 <div class="highlight"><pre><span></span><cod… |
|
10379 <span class="w"> </span><span class="k">unsafe</… |
|
10380 <span class="p">}</span><span class="w"></span> |
|
10381 </code></pre></div> |
|
10382 |
|
10383 <p>And the implementation for strings is as follows. In Glib's sc… |
|
10384 things, "transferring ownership of a string" means that the recipient |
|
10385 of the string must eventually <code>g_free()</code> it.</… |
|
10386 <table class="highlighttable"><tr><td class="linenos">… |
|
10387 <span class="normal">2</span> |
|
10388 <span class="normal">3</span> |
|
10389 <span class="normal">4</span> |
|
10390 <span class="normal">5</span> |
|
10391 <span class="normal">6</span> |
|
10392 <span class="normal">7</span> |
|
10393 <span class="normal">8</span></pre></div></td… |
|
10394 <span class="w"> </span><span class="cp">#[inline]&… |
|
10395 <span class="w"> </span><span class="k">unsafe</… |
|
10396 <span class="w"> </span><span class="kd">let<… |
|
10397 <span class="w"> </span><span class="n">glib_ff… |
|
10398 <span class="w"> </span><span class="n">res<… |
|
10399 <span class="w"> </span><span class="p">}</span&… |
|
10400 <span class="p">}</span><span class="w"></span> |
|
10401 </code></pre></div> |
|
10402 </td></tr></table> |
|
10403 <p>Line 1: given a pointer to a <code>c_char</code>, t… |
|
10404 <p>Line 4: Do the conversion with <code>from_glib_none()<… |
|
10405 saw before, put it in <code>res</code>.</p> |
|
10406 <p>Line 5: Call <code>g_free()</code> on the original … |
|
10407 <p>Line 6: Return the <code>res</code>, a Rust string … |
|
10408 <h2>Pointer types - from Rust to Glib</h2> |
|
10409 <p>Consider the case where you want to pass a <code>String&l… |
|
10410 that takes a <code>*const c_char</code> — in C parlance, a… |
|
10411 Glib function acquiring ownership of the string. For example, assume |
|
10412 that the C version of <code>gtk_window_set_title()</code> is… |
|
10413 module. You may want to call it like this:</p> |
|
10414 <div class="highlight"><pre><span></span><cod… |
|
10415 <span class="w"> </span><span class="n">gtk_ffi<… |
|
10416 <span class="p">}</span><span class="w"></span> |
|
10417 </code></pre></div> |
|
10418 |
|
10419 <p>Now, what would that <code>make_c_string_from_rust_string… |
|
10420 <ul> |
|
10421 <li> |
|
10422 <p><strong>We have:</strong> a Rust <code>String… |
|
10423 </li> |
|
10424 <li> |
|
10425 <p><strong>We want:</strong> a <code>*const char… |
|
10426 </li> |
|
10427 </ul> |
|
10428 <p>So, let's write this:</p> |
|
10429 <table class="highlighttable"><tr><td class="linenos">… |
|
10430 <span class="normal">2</span> |
|
10431 <span class="normal">3</span> |
|
10432 <span class="normal">4</span> |
|
10433 <span class="normal">5</span></pre></div></td… |
|
10434 <span class="w"> </span><span class="kd">let</sp… |
|
10435 <span class="w"> </span><span class="kd">let</sp… |
|
10436 <span class="w"> </span><span class="n">ptr</spa… |
|
10437 <span class="p">}</span><span class="w"></span> |
|
10438 </code></pre></div> |
|
10439 </td></tr></table> |
|
10440 <p>Line 1: Take in a <code>&amp;String</code>; ret… |
|
10441 <p>Line 2: Build a <a href="https://people.gnome.org/~federico/… |
|
10442 allocates a byte buffer with space for a nul terminator, and copies |
|
10443 the string's bytes. We <code>unwrap()</code> for this simpl… |
|
10444 <code>CString::new()</code> will return an error if the <… |
|
10445 characters in the middle of the string, which C doesn't understand.</… |
|
10446 <p>Line 3: Call <code>into_raw()</code> to get a point… |
|
10447 cast it to a <code>*const c_char</code>. <em>We'll ne… |
|
10448 <p>But this kind of sucks, because we the have to use this functio… |
|
10449 the pointer to a C function, and then reconstitute the <code>CStri… |
|
10450 can free the byte buffer:</p> |
|
10451 <div class="highlight"><pre><span></span><cod… |
|
10452 <span class="k">unsafe</span><span class="w"> </spa… |
|
10453 <span class="kd">let</span><span class="w"> </span&… |
|
10454 </code></pre></div> |
|
10455 |
|
10456 <p>The solution that Glib-rs provides for this is very Rusty, and … |
|
10457 elegant.</p> |
|
10458 <h3>Stashes</h3> |
|
10459 <p>We want:</p> |
|
10460 <ul> |
|
10461 <li>A temporary place to put a piece of data</li> |
|
10462 <li>A pointer to that buffer</li> |
|
10463 <li>Automatic memory management for both of those</li> |
|
10464 </ul> |
|
10465 <p>Glib-rs defines a <code>Stash</code> for this:</… |
|
10466 <table class="highlighttable"><tr><td class="linenos">… |
|
10467 <span class="normal">2</span> |
|
10468 <span class="normal">3</span> |
|
10469 <span class="normal">4</span> |
|
10470 <span class="normal">5</span> |
|
10471 <span class="normal">6</span></pre></div></td… |
|
10472 <span class="w"> </span><span class="n"&g… |
|
10473 <span class="w"> </span><span class="n"&g… |
|
10474 <span class="w"> </span><span class="k">pub</spa… |
|
10475 <span class="w"> </span><span class="k">pub</spa… |
|
10476 <span class="p">);</span><span class="w"></span> |
|
10477 </code></pre></div> |
|
10478 </td></tr></table> |
|
10479 <p>... and the piece of data must be of of the <em>associate… |
|
10480 <code>ToGlibPtr::Storage</code>, which we will see shortly.&… |
|
10481 <p>This struct <code>Stash</code> goes along with the … |
|
10482 <div class="highlight"><pre><span></span><cod… |
|
10483 <span class="w"> </span><span class="k">type</sp… |
|
10484 |
|
10485 <span class="w"> </span><span class="k">fn</span… |
|
10486 <span class="w"> … |
|
10487 <span class="p">}</span><span class="w"></span> |
|
10488 </code></pre></div> |
|
10489 |
|
10490 <p>Let's unpack this by looking at the implementation of the "tran… |
|
10491 String to a C function while keeping ownership":</p> |
|
10492 <table class="highlighttable"><tr><td class="linenos">… |
|
10493 <span class="normal">2</span> |
|
10494 <span class="normal">3</span> |
|
10495 <span class="normal">4</span> |
|
10496 <span class="normal">5</span> |
|
10497 <span class="normal">6</span> |
|
10498 <span class="normal">7</span> |
|
10499 <span class="normal">8</span> |
|
10500 <span class="normal">9</span></pre></div></td… |
|
10501 <span class="w"> </span><span class="k">type</sp… |
|
10502 |
|
10503 <span class="w"> </span><span class="cp">#[inline]&… |
|
10504 <span class="w"> </span><span class="k">fn</span… |
|
10505 <span class="w"> </span><span class="kd">let<… |
|
10506 <span class="w"> </span><span class="n">Stash&l… |
|
10507 <span class="w"> </span><span class="p">}</span&… |
|
10508 <span class="p">}</span><span class="w"></span> |
|
10509 </code></pre></div> |
|
10510 </td></tr></table> |
|
10511 <p>Line 1: We implement <code>ToGlibPtr&lt;'a *const c_c… |
|
10512 declaring the lifetime <code>'a</code> for the <code>S… |
|
10513 <p>Line 2: Our temporary storage is a <code>CString</code… |
|
10514 <p>Line 6: Make a CString like before.</p> |
|
10515 <p>Line 7: Create the <code>Stash</code> with a pointe… |
|
10516 and the CString itself.</p> |
|
10517 <h3><code>(transfer none)</code></h3> |
|
10518 <p>Now, we can use "<code>.0</code>" to extract the fi… |
|
10519 which is precisely the pointer we want to a byte buffer:</p> |
|
10520 <div class="highlight"><pre><span></span><cod… |
|
10521 <span class="k">unsafe</span><span class="w"> </spa… |
|
10522 </code></pre></div> |
|
10523 |
|
10524 <p>Now Rust knows that the temporary buffer inside the <code>… |
|
10525 <code>my_string</code>, and it will free it automatically wh… |
|
10526 out of scope. If we can accept the <code>.to_glib_none().0</co… |
|
10527 for "lending" pointers to C, this works perfectly.</p> |
|
10528 <h3 id="ptr-transfer-full"><code>(transfer full)</code&g… |
|
10529 <p>And for transferring ownership to the C function? The <code… |
|
10530 trait has another method:</p> |
|
10531 <div class="highlight"><pre><span></span><cod… |
|
10532 <span class="w"> </span><span class="o">..</span… |
|
10533 |
|
10534 <span class="w"> </span><span class="k">fn</span… |
|
10535 <span class="p">}</span><span class="w"></span> |
|
10536 </code></pre></div> |
|
10537 |
|
10538 <p>And here is the implementation for strings:</p> |
|
10539 <div class="highlight"><pre><span></span><cod… |
|
10540 <span class="w"> </span><span class="k">fn</span… |
|
10541 <span class="w"> </span><span class="k">unsafe&… |
|
10542 <span class="w"> </span><span class="n">gli… |
|
10543 <span class="w"> </span><s… |
|
10544 <span class="w"> </span><span class="k">… |
|
10545 <span class="w"> </span><span class="p">}</s… |
|
10546 <span class="w"> </span><span class="p">}</span&… |
|
10547 </code></pre></div> |
|
10548 |
|
10549 <p>We basically <code>g_strndup()</code> the Rust stri… |
|
10550 buffer <em>and</em> its <code>len()</code>, and … |
|
10551 code will be responsible for <code>g_free()</code>ing the C-… |
|
10552 <h1>Next up</h1> |
|
10553 <p>Transferring lists and arrays. Stay tuned!</p></content>… |
|
10554 in several ways. In this post I'll show several things related to the |
|
10555 process of building strings, from bytes in memory, or from a file, or |
|
10556 from <code>char *</code> things passed from C.</p> |
|
10557 <h1>Strings in Rust</h1> |
|
10558 <p>The easiest way to build …</p></summary><content type="… |
|
10559 in several ways. In this post I'll show several things related to the |
|
10560 process of building strings, from bytes in memory, or from a file, or |
|
10561 from <code>char *</code> things passed from C.</p> |
|
10562 <h1>Strings in Rust</h1> |
|
10563 <p>The easiest way to build a string is to do it directly at compi… |
|
10564 time:</p> |
|
10565 <div class="highlight"><pre><span></span><cod… |
|
10566 </code></pre></div> |
|
10567 |
|
10568 <p>In Rust, strings are UTF-8. Here, the compiler checks our stri… |
|
10569 literal is valid UTF-8. If we try to be sneaky and insert an |
|
10570 invalid character...</p> |
|
10571 <div class="highlight"><pre><span></span><cod… |
|
10572 </code></pre></div> |
|
10573 |
|
10574 <p>We get a compiler error:</p> |
|
10575 <div class="highlight"><pre><span></span><cod… |
|
10576 --&gt; foo.rs:2:30 |
|
10577 | |
|
10578 2 | let my_string = &quot;Hello \xf0&quot;; |
|
10579 | ^^ |
|
10580 </code></pre></div> |
|
10581 |
|
10582 <p>Rust strings know their length, unlike C strings. They <em&… |
|
10583 a nul character in the middle, because they don't need a nul |
|
10584 terminator at the end.</p> |
|
10585 <div class="highlight"><pre><span></span><cod… |
|
10586 <span class="fm">println!</span><span class="p">(</… |
|
10587 </code></pre></div> |
|
10588 |
|
10589 <p>The output is what you expect:</p> |
|
10590 <div class="highlight"><pre><span></span><cod… |
|
10591 00000000 48 65 6c 6c 6f 20 00 20 7a 65 72 6f 0a |Hello . zer… |
|
10592 0000000d ^ note the nul char here |
|
10593 $ |
|
10594 </code></pre></div> |
|
10595 |
|
10596 <p>So, to summarize, in Rust:</p> |
|
10597 <ul> |
|
10598 <li>Strings are encoded in UTF-8</li> |
|
10599 <li>Strings know their length</li> |
|
10600 <li>Strings can have nul chars in the middle</li> |
|
10601 </ul> |
|
10602 <p>This is a bit different from C:</p> |
|
10603 <ul> |
|
10604 <li>Strings don't exist!</li> |
|
10605 </ul> |
|
10606 <p>Okay, just kidding. In C:</p> |
|
10607 <ul> |
|
10608 <li>A lot of software has standardized on UTF-8.</li> |
|
10609 <li>Strings don't know their length - a <code>char *</cod… |
|
10610 beginning of the string.</li> |
|
10611 <li>Strings conventionally have a nul terminator, that is, a zero … |
|
10612 that marks the end of the string. Therefore, you can't have nul |
|
10613 characters in the middle of strings.</li> |
|
10614 </ul> |
|
10615 <h1>Building a string from bytes</h1> |
|
10616 <p>Let's say you have an array of bytes and want to make a string … |
|
10617 them. Rust won't let you just cast the array, like C would. First |
|
10618 you need to do UTF-8 validation. For example:</p> |
|
10619 <table class="highlighttable"><tr><td class="linenos">… |
|
10620 <span class="normal"> 2</span> |
|
10621 <span class="normal"> 3</span> |
|
10622 <span class="normal"> 4</span> |
|
10623 <span class="normal"> 5</span> |
|
10624 <span class="normal"> 6</span> |
|
10625 <span class="normal"> 7</span> |
|
10626 <span class="normal"> 8</span> |
|
10627 <span class="normal"> 9</span> |
|
10628 <span class="normal">10</span> |
|
10629 <span class="normal">11</span> |
|
10630 <span class="normal">12</span></pre></div></t… |
|
10631 <span class="w"> </span><span class="kd">let</sp… |
|
10632 <span class="w"> </span><span class="k">match</s… |
|
10633 <span class="w"> </span><span class="nb">Ok<… |
|
10634 <span class="w"> </span><span class="nb">Err<… |
|
10635 <span class="w"> </span><span class="p">}</span&… |
|
10636 <span class="p">}</span><span class="w"></span> |
|
10637 |
|
10638 <span class="k">fn</span> <span class="nf">main</sp… |
|
10639 <span class="w"> </span><span class="n">convert_and… |
|
10640 <span class="w"> </span><span class="n">convert_and… |
|
10641 <span class="p">}</span><span class="w"></span> |
|
10642 </code></pre></div> |
|
10643 </td></tr></table> |
|
10644 <p>In lines 10 and 11, we call <code>convert_and_print()<… |
|
10645 arrays of bytes; the first one is valid UTF-8, and the second one |
|
10646 isn't.</p> |
|
10647 <p>Line 2 calls <a href="https://doc.rust-lang.org/std/string/s… |
|
10648 i.e. something with a success value or an error. In lines 3-5 we |
|
10649 unpack this <code>Result</code>. If it's <code>Ok<… |
|
10650 which has been validated for UTF-8. Otherwise, we print the debug |
|
10651 representation of the error.</p> |
|
10652 <p>The program prints the following:</p> |
|
10653 <div class="highlight"><pre><span></span><cod… |
|
10654 Hello |
|
10655 FromUtf8Error { bytes: [72, 101, 240, 108, 108, 111], error: Utf8Error {… |
|
10656 </code></pre></div> |
|
10657 |
|
10658 <p>Here, in the error case, the <a href="https://doc.rust-lang.… |
|
10659 are UTF-8 and are <code>valid_up_to</code> index 2; that is … |
|
10660 index. We also get some extra information which lets the program know |
|
10661 if the problematic sequence was incomplete and truncated at the end of |
|
10662 the byte array, or if it's complete and in the middle.</p> |
|
10663 <p>And for a "just make this printable, pls" API? We can |
|
10664 use <a href="https://doc.rust-lang.org/std/string/struct.String.html#… |
|
10665 sequences with <code>U+FFFD REPLACEMENT CHARACTER</code>:<… |
|
10666 <div class="highlight"><pre><span></span><cod… |
|
10667 <span class="w"> </span><span class="kd">let</sp… |
|
10668 <span class="w"> </span><span class="fm">println!&l… |
|
10669 <span class="p">}</span><span class="w"></span> |
|
10670 |
|
10671 <span class="k">fn</span> <span class="nf">main</sp… |
|
10672 <span class="w"> </span><span class="n">convert_and… |
|
10673 <span class="w"> </span><span class="n">convert_and… |
|
10674 <span class="p">}</span><span class="w"></span> |
|
10675 </code></pre></div> |
|
10676 |
|
10677 <p>This prints the following:</p> |
|
10678 <div class="highlight"><pre><span></span><cod… |
|
10679 Hello |
|
10680 He�llo |
|
10681 </code></pre></div> |
|
10682 |
|
10683 <h1>Reading from files into strings</h1> |
|
10684 <p>Now, let's assume you want to read chunks of a file and put the… |
|
10685 strings. Let's go from the low-level parts up to the high level "read |
|
10686 a line" API.</p> |
|
10687 <h2>Single bytes and single UTF-8 characters</h2> |
|
10688 <p>When you open a <a href="https://doc.rust-lang.org/std/fs/st… |
|
10689 <a href="https://doc.rust-lang.org/std/io/trait.Read.html"><cod… |
|
10690 it can also give you back an iterator over <em>bytes</em>, o… |
|
10691 over UTF-8 <em>characters</em>.</p> |
|
10692 <p>The <a href="https://doc.rust-lang.org/std/io/trait.Read.htm… |
|
10693 whose <code>next()</code> method returns <code>Result&… |
|
10694 the iterator for its next item, that <code>Result</code> mea… |
|
10695 out of it successfully, or an I/O error.</p> |
|
10696 <p>In contrast, the <a href="https://doc.rust-lang.org/std/io/t… |
|
10697 a <a href="https://doc.rust-lang.org/std/io/struct.Chars.html"><… |
|
10698 <code>Result&lt;char, CharsError&gt;</code>, not &l… |
|
10699 extended <a href="https://doc.rust-lang.org/std/io/enum.CharsError.ht… |
|
10700 when <code>next()</code> tries to read the next UTF-8 sequen… |
|
10701 the file has invalid data. <code>CharsError</code> also has… |
|
10702 I/O errors.</p> |
|
10703 <h2>Reading lines</h2> |
|
10704 <p>While you could build a UTF-8 string one character at a time, t… |
|
10705 are more efficient ways to do it.</p> |
|
10706 <p>You can create a <a href="https://doc.rust-lang.org/std/io/s… |
|
10707 that implements the <a href="https://doc.rust-lang.org/std/io/trait.R… |
|
10708 convenient <a href="https://doc.rust-lang.org/std/io/trait.BufRead.ht… |
|
10709 String and it returns a <code>Result&lt;usize, io::Error&… |
|
10710 number of bytes read, or an error.</p> |
|
10711 <p>That method is declared in the <a href="https://doc.rust-lan… |
|
10712 implements. Why the separation? Because other concrete structs also |
|
10713 implement <code>BufRead</code>, such as <a href="https://… |
|
10714 you use a vector of bytes like an I/O <code>Read</code> or &… |
|
10715 implementation, similar to <a href="https://developer.gnome.org/gio/s… |
|
10716 <p>If you prefer an iterator rather than the <code>read_line… |
|
10717 <code>BufRead</code> also gives you a <a href="https://do… |
|
10718 a <a href="https://doc.rust-lang.org/std/io/struct.Lines.html"><… |
|
10719 <p>In both cases — the <code>read_line()</code> meth… |
|
10720 error that you can get back can be of <a href="https://doc.rust-lang.… |
|
10721 which indicates that there was an invalid UTF-8 sequence in the line |
|
10722 to be read. It can also be a normal I/O error, of course.</p> |
|
10723 <h1>Summary so far</h1> |
|
10724 <p>There is no way to build a <code>String</code>, or … |
|
10725 UTF-8 data. All the methods that let you turn bytes into string-like |
|
10726 things perform validation, and return a <code>Result</code> … |
|
10727 your bytes validated correctly.</p> |
|
10728 <p>The exceptions are in the <code>unsafe</code> metho… |
|
10729 like <a href="https://doc.rust-lang.org/std/string/struct.String.html… |
|
10730 them if you are <em>absolutely sure</em> that your bytes wer… |
|
10731 UTF-8 beforehand.</p> |
|
10732 <p>There is no way to bring in data from a file (or anything file-… |
|
10733 that implements the <a href="https://doc.rust-lang.org/std/io/trait.R… |
|
10734 without going through functions that do UTF-8 validation. There is |
|
10735 not an unsafe "read a line" API without validation — you would have to |
|
10736 build one yourself, but the I/O hit is probably going to be slower than |
|
10737 validating data in memory, anyway, so you may as well validate.</p> |
|
10738 <h1>C strings and Rust</h1> |
|
10739 <p>For unfortunate historical reasons, C flings around <code>… |
|
10740 different things. In the context of Glib, it can mean</p> |
|
10741 <ul> |
|
10742 <li>A valid, nul-terminated UTF-8 sequence of bytes (a "normal str… |
|
10743 <li>A nul-terminated file path, which has no meaningful encoding&l… |
|
10744 <li>A nul-terminated sequence of bytes, not validated as UTF-8.<… |
|
10745 </ul> |
|
10746 <p>What a particular <code>char *</code> means depends… |
|
10747 <h2>Bringing a string from C to Rust</h2> |
|
10748 <p>From Rust's viewpoint, getting a raw <code>char *</cod… |
|
10749 c_char</code>" in Rust parlance) means that it gets a pointer to a… |
|
10750 unknown length.</p> |
|
10751 <p>Now, that may not be entirely accurate:</p> |
|
10752 <ul> |
|
10753 <li>You may indeed only have a pointer to a buffer of unknown leng… |
|
10754 <li>You may have a pointer to a buffer, and also know its length |
|
10755 (i.e. the offset at which the nul terminator is)</li> |
|
10756 </ul> |
|
10757 <p>The Rust standard library provides a <a href="https://doc.ru… |
|
10758 "I have a pointer to an array of bytes, and I know its length, and I |
|
10759 know the last byte is a nul".</p> |
|
10760 <p><code>CStr</code> provides an <a href="https://d… |
|
10761 raw pointer, and walks the memory to which it points until it finds a |
|
10762 nul byte. You <em>must</em> give it a valid pointer, and yo… |
|
10763 guarantee that there is a nul terminator, or <code>CStr</code&g… |
|
10764 the end of your process' address space looking for one.</p> |
|
10765 <p>Alternatively, if you know the length of your byte array, and y… |
|
10766 that it has a nul byte at the end, you can |
|
10767 call <a href="https://doc.rust-lang.org/std/ffi/struct.CStr.html#meth… |
|
10768 the function will check that a) the last byte in that slice is indeed |
|
10769 a nul, and b) there are no nul bytes in the middle.</p> |
|
10770 <p>The unsafe version of this last function |
|
10771 is <a href="https://doc.rust-lang.org/std/ffi/struct.CStr.html#method… |
|
10772 an <code>&amp;[u8]</code> slice, but <em>you</e… |
|
10773 and that there are no nul bytes in the middle.</p> |
|
10774 <p><em>I really like that the Rust documentation tells you w… |
|
10775 are not "instantaneous" and must instead walks arrays, like to do |
|
10776 validation or to look for the nul terminator above.</em></p> |
|
10777 <h2>Turning a CStr into a string-like</h2> |
|
10778 <p>Now, the above indicates that a <code>CStr</code> i… |
|
10779 bytes. We have no idea what the bytes inside look like; we just know |
|
10780 that they don't contain any other nul bytes.</p> |
|
10781 <p>There is a <a href="https://doc.rust-lang.org/std/ffi/struct… |
|
10782 <code>Result&lt;&amp;str, Utf8Error&gt;</code>.… |
|
10783 of bytes. If the array is valid, the function just returns a slice of |
|
10784 the validated bytes minus the nul terminator (i.e. just what you |
|
10785 expect for a Rust string slice). Otherwise, it returns an <code>U… |
|
10786 with the details like we discussed before.</p> |
|
10787 <p>There is also <a href="https://doc.rust-lang.org/std/ffi/str… |
|
10788 replacement of invalid UTF-8 sequences like we discussed before.</p&g… |
|
10789 <h1>Conclusion</h1> |
|
10790 <p>Strings in Rust are UTF-8 encoded, they know their length, and … |
|
10791 can have nul bytes in the middle.</p> |
|
10792 <p>To build a string from raw bytes, you must go through functions… |
|
10793 do UTF-8 validation and tell you if it failed. There are unsafe |
|
10794 functions that let you skip validation, but then of course you are on |
|
10795 your own.</p> |
|
10796 <p>The low-level functions which read data from files operate on b… |
|
10797 On top of those, there are convenience functions to read validated |
|
10798 UTF-8 characters, lines, etc. All of these tell you when there was |
|
10799 invalid UTF-8 or an I/O error.</p> |
|
10800 <p>Rust lets you wrap a raw <code>char *</code> that y… |
|
10801 that can later be validated and turned into a string. Anything that |
|
10802 manipulates a raw pointer is <code>unsafe</code>; this inclu… |
|
10803 pointer into a C string abstraction" API, and the "build me an array |
|
10804 of bytes from this raw pointer" API. Later, you can validate <em>… |
|
10805 as UTF-8 and build real Rust strings — or know if the validation |
|
10806 failed.</p> |
|
10807 <p>Rust builds these little "corridors" through the API so that il… |
|
10808 states are unrepresentable.</p></content><category term="misc"></c… |
|
10809 called |
|
10810 <a href="https://people.gnome.org/~federico/blog/docs/fmq-porting-c-t… |
|
10811 This is the PDF file; be sure to scroll past the full-page |
|
10812 presentation pages until you reach the speaker's notes, especially for |
|
10813 the code sections!</p> |
|
10814 <p><a href="https://people.gnome.org/~federico/blog/docs/fmq-po… |
|
10815 <p>You can also get the …</p></summary><content type="html… |
|
10816 called |
|
10817 <a href="https://people.gnome.org/~federico/blog/docs/fmq-porting-c-t… |
|
10818 This is the PDF file; be sure to scroll past the full-page |
|
10819 presentation pages until you reach the speaker's notes, especially for |
|
10820 the code sections!</p> |
|
10821 <p><a href="https://people.gnome.org/~federico/blog/docs/fmq-po… |
|
10822 <p>You can also get the <a href="https://people.gnome.org/~fede… |
|
10823 released under a <a href="https://creativecommons.org/licenses/by-sa/… |
|
10824 <p>For the presentation, my daughter Luciana made some drawings of |
|
10825 Ferris, the Rust mascot, also released under the same license:</p> |
|
10826 <p><a href="https://people.gnome.org/~federico/blog/docs/ferris… |
|
10827 <a href="https://people.gnome.org/~federico/blog/docs/ferris-2.png"&g… |
|
10828 <a href="https://people.gnome.org/~federico/blog/docs/ferris-3.png"&g… |
|
10829 <a href="https://people.gnome.org/~federico/blog/docs/ferris-4.png"&g… |
|
10830 been some API breaks (!!!) in the unstable libraries that it uses |
|
10831 since the last time I locked them. This post is about an interesting |
|
10832 case of API breakage.</p> |
|
10833 <p><a href="https://github.com/servo/rust-cssparser">rust-cs… |
|
10834 CSS. Well, more …</p></summary><content type="html"><p>Ye… |
|
10835 been some API breaks (!!!) in the unstable libraries that it uses |
|
10836 since the last time I locked them. This post is about an interesting |
|
10837 case of API breakage.</p> |
|
10838 <p><a href="https://github.com/servo/rust-cssparser">rust-cs… |
|
10839 CSS. Well, more like <em>tokenizing</em> CSS: you give it a… |
|
10840 gives you back tokens, and you are supposed to compose CSS selector |
|
10841 information or other CSS values from the tokens.</p> |
|
10842 <p>Librsvg uses rust-cssparser now for most of the micro-languages… |
|
10843 SVG's attribute values, instead of its old, fragile C parsers. I hope |
|
10844 to be able to use it in conjunction with Servo's <a href="https://git… |
|
10845 crate to fully parse CSS data and replace <a href="https://git.gnome.… |
|
10846 <p>A few months ago, rust-cssparser's API looked more or less like… |
|
10847 following. This is the old representation of a <code>Token</co… |
|
10848 <div class="highlight"><pre><span></span><cod… |
|
10849 <span class="w"> </span><span class="c1">// an iden… |
|
10850 <span class="w"> </span><span class="n">Ident</s… |
|
10851 |
|
10852 <span class="w"> </span><span class="c1">// a plain… |
|
10853 <span class="w"> </span><span class="n">Number</… |
|
10854 |
|
10855 <span class="w"> </span><span class="c1">// a perce… |
|
10856 <span class="w"> </span><span class="n">Percentage&… |
|
10857 |
|
10858 <span class="w"> </span><span class="n">WhiteSpace&… |
|
10859 <span class="w"> </span><span class="n">Comma</s… |
|
10860 |
|
10861 <span class="w"> </span><span class="o">..</span… |
|
10862 <span class="p">}</span><span class="w"></span> |
|
10863 </code></pre></div> |
|
10864 |
|
10865 <p>That is, a <code>Token</code> can be an <code>… |
|
10866 <code>Number</code>, a <code>Percentage</code>, … |
|
10867 <p>On top of that is the old API for a <code>Parser</code… |
|
10868 a string and then it gives you back tokens:</p> |
|
10869 <div class="highlight"><pre><span></span><cod… |
|
10870 <span class="w"> </span><span class="k">pub</spa… |
|
10871 |
|
10872 <span class="w"> </span><span class="k">pub</spa… |
|
10873 |
|
10874 <span class="w"> </span><span class="o">..</span… |
|
10875 <span class="p">}</span><span class="w"></span> |
|
10876 </code></pre></div> |
|
10877 |
|
10878 <p>This means the following. You create the parser out of a strin… |
|
10879 with <code>new()</code>. You can then extract a <code>… |
|
10880 sucessfully, or with an empty error value. The parser uses a lifetime |
|
10881 <code>'i</code> on the string from which it is constructed: … |
|
10882 return identifiers, for example, could return sub-string slices that |
|
10883 come from the original string, and the parser has to be marked with a |
|
10884 lifetime so that it does not outlive its underlying string.</p> |
|
10885 <p>A few commits later, rust-cssparser got changed to return detai… |
|
10886 error values, so that instead of <code>()</code> you get a a… |
|
10887 with sub-cases like <code>UnexpectedToken</code> or <code… |
|
10888 <p>After the changes to the error values for results, I didn't pay… |
|
10889 attention to rust-cssparser for while. Yesterday I wanted to update |
|
10890 librsvg to use the newest rust-cssparser, and had some interesting |
|
10891 problems.</p> |
|
10892 <p>First, <code>Parser::new()</code> was changed from … |
|
10893 taking a <code>ParserInput</code> struct. This is an implem… |
|
10894 lets the parser cache the last token it saw. Not a big deal:</p> |
|
10895 <div class="highlight"><pre><span></span><cod… |
|
10896 <span class="kd">let</span><span class="w"> </span&… |
|
10897 |
|
10898 <span class="c1">// you now construct it like</span> |
|
10899 <span class="kd">let</span><span class="w"> </span&… |
|
10900 <span class="kd">let</span><span class="w"> </span&… |
|
10901 </code></pre></div> |
|
10902 |
|
10903 <p>I am not completely sure why this is exposed to the public API,… |
|
10904 Rust won't allow you to have two mutable references to a |
|
10905 <code>ParserInput</code>, and the only consumer of a (mutabl… |
|
10906 the <code>Parser</code>, anyway.</p> |
|
10907 <p>However, the <code>parser.next()</code> function ch… |
|
10908 <div class="highlight"><pre><span></span><cod… |
|
10909 <span class="k">pub</span><span class="w"> </span&g… |
|
10910 |
|
10911 <span class="c1">// new version</span> |
|
10912 <span class="k">pub</span><span class="w"> </span&g… |
|
10913 <span class="c1">// note this bad boy here -------^</span> |
|
10914 </code></pre></div> |
|
10915 |
|
10916 <p>The successful <code>Result</code> from <code>… |
|
10917 <code>Token</code>, not a plain <code>Token</code&g… |
|
10918 giving you a borrowed reference to its internally-cached token.</p> |
|
10919 <p>My parsing functions for the old API looked similar to the |
|
10920 following. This is a function that parses a string into an angle; it |
|
10921 can look like <code>"45deg"</code> or <code>"1.5rad"&l… |
|
10922 <table class="highlighttable"><tr><td class="linenos">… |
|
10923 <span class="normal"> 2</span> |
|
10924 <span class="normal"> 3</span> |
|
10925 <span class="normal"> 4</span> |
|
10926 <span class="normal"> 5</span> |
|
10927 <span class="normal"> 6</span> |
|
10928 <span class="normal"> 7</span> |
|
10929 <span class="normal"> 8</span> |
|
10930 <span class="normal"> 9</span> |
|
10931 <span class="normal">10</span> |
|
10932 <span class="normal">11</span> |
|
10933 <span class="normal">12</span> |
|
10934 <span class="normal">13</span> |
|
10935 <span class="normal">14</span> |
|
10936 <span class="normal">15</span> |
|
10937 <span class="normal">16</span> |
|
10938 <span class="normal">17</span> |
|
10939 <span class="normal">18</span> |
|
10940 <span class="normal">19</span> |
|
10941 <span class="normal">20</span> |
|
10942 <span class="normal">21</span> |
|
10943 <span class="normal">22</span> |
|
10944 <span class="normal">23</span> |
|
10945 <span class="normal">24</span> |
|
10946 <span class="normal">25</span> |
|
10947 <span class="normal">26</span></pre></div></t… |
|
10948 <span class="w"> </span><span class="kd">let</sp… |
|
10949 |
|
10950 <span class="w"> </span><span class="kd">let</sp… |
|
10951 <span class="w"> </span><span class="p">.</s… |
|
10952 |
|
10953 <span class="w"> </span><span class="k">match</s… |
|
10954 <span class="w"> </span><span class="n">Token&l… |
|
10955 |
|
10956 <span class="w"> </span><span class="n">Token&l… |
|
10957 <span class="w"> </span><span class="kd">le… |
|
10958 |
|
10959 <span class="w"> </span><span class="k">mat… |
|
10960 <span class="w"> </span><span class="s">… |
|
10961 <span class="w"> </span><span class="s">… |
|
10962 <span class="w"> </span><span class="s">… |
|
10963 <span class="w"> </span><span class="n">… |
|
10964 <span class="w"> </span><span class="p">}&l… |
|
10965 <span class="w"> </span><span class="p">},</… |
|
10966 |
|
10967 <span class="w"> </span><span class="n">_</s… |
|
10968 <span class="w"> </span><span class="p">}.</span… |
|
10969 <span class="w"> </span><span class="n">… |
|
10970 <span class="w"> </span><span class="p">… |
|
10971 <span class="w"> </span><span class="p">… |
|
10972 <span class="p">}</span><span class="w"></span> |
|
10973 </code></pre></div> |
|
10974 </td></tr></table> |
|
10975 <p>This is a bit ugly, but it was the first version that passed the |
|
10976 tests. Lines 4 and 5 mean, "get the first token or return an error". |
|
10977 Line 17 means, "anything except <code>deg</code>, <code&g… |
|
10978 causes the <code>match</code> expression to generate an erro… |
|
10979 feeling very proud of using <code>and_then()</code> in line … |
|
10980 <code>parser.expect_exhausted()</code>, to ensure that the p… |
|
10981 any more tokens after the angle/units.</p> |
|
10982 <p>However, in the new version of rust-cssparser, Parser.next() gi… |
|
10983 back a <code>Result</code> with a <code>&amp;Token… |
|
10984 token —, while the old version returned a plain <code>Token</… |
|
10985 I thought, I'm just going to de-reference the value in the <code>m… |
|
10986 be done with it:</p> |
|
10987 <div class="highlight"><pre><span></span><cod… |
|
10988 <span class="w"> </span><span class="p">.</s… |
|
10989 |
|
10990 <span class="w"> </span><span class="k">match</s… |
|
10991 <span class="w"> </span><span class="c1">// ^ de… |
|
10992 <span class="w"> </span><span class="n">Token&l… |
|
10993 |
|
10994 <span class="w"> </span><span class="n">Token&l… |
|
10995 <span class="w"> </span><span class="c1">// … |
|
10996 </code></pre></div> |
|
10997 |
|
10998 <p>The compiler complained elsewhere. The whole function now look… |
|
10999 this:</p> |
|
11000 <table class="highlighttable"><tr><td class="linenos">… |
|
11001 <span class="normal"> 2</span> |
|
11002 <span class="normal"> 3</span> |
|
11003 <span class="normal"> 4</span> |
|
11004 <span class="normal"> 5</span> |
|
11005 <span class="normal"> 6</span> |
|
11006 <span class="normal"> 7</span> |
|
11007 <span class="normal"> 8</span> |
|
11008 <span class="normal"> 9</span> |
|
11009 <span class="normal">10</span> |
|
11010 <span class="normal">11</span> |
|
11011 <span class="normal">12</span> |
|
11012 <span class="normal">13</span></pre></div></t… |
|
11013 <span class="w"> </span><span class="kd">let</sp… |
|
11014 |
|
11015 <span class="w"> </span><span class="kd">let</sp… |
|
11016 <span class="w"> </span><span class="p">.</s… |
|
11017 |
|
11018 <span class="w"> </span><span class="k">match</s… |
|
11019 <span class="w"> </span><span class="c1">// ...… |
|
11020 <span class="w"> </span><span class="p">}.</span… |
|
11021 <span class="w"> </span><span class="n">… |
|
11022 <span class="w"> </span><span class="p">… |
|
11023 <span class="w"> </span><span class="p">… |
|
11024 <span class="p">}</span><span class="w"></span> |
|
11025 </code></pre></div> |
|
11026 </td></tr></table> |
|
11027 <p>But in line 4, <code>token</code> is now a referenc… |
|
11028 inside <code>parser</code>, and <code>parser</code&… |
|
11029 compiler didn't like that line 10 (the call to |
|
11030 <code>parser.expect_exhausted()</code>) was trying to borrow… |
|
11031 again.</p> |
|
11032 <p>I played a bit with creating a temporary scope around the assig… |
|
11033 to <code>token</code> so that it would only borrow <code&… |
|
11034 scope. Things ended up like this, without the call to <code>and_t… |
|
11035 after the <code>match</code>:</p> |
|
11036 <table class="highlighttable"><tr><td class="linenos">… |
|
11037 <span class="normal"> 2</span> |
|
11038 <span class="normal"> 3</span> |
|
11039 <span class="normal"> 4</span> |
|
11040 <span class="normal"> 5</span> |
|
11041 <span class="normal"> 6</span> |
|
11042 <span class="normal"> 7</span> |
|
11043 <span class="normal"> 8</span> |
|
11044 <span class="normal"> 9</span> |
|
11045 <span class="normal">10</span> |
|
11046 <span class="normal">11</span> |
|
11047 <span class="normal">12</span> |
|
11048 <span class="normal">13</span> |
|
11049 <span class="normal">14</span> |
|
11050 <span class="normal">15</span> |
|
11051 <span class="normal">16</span> |
|
11052 <span class="normal">17</span> |
|
11053 <span class="normal">18</span> |
|
11054 <span class="normal">19</span> |
|
11055 <span class="normal">20</span> |
|
11056 <span class="normal">21</span> |
|
11057 <span class="normal">22</span> |
|
11058 <span class="normal">23</span> |
|
11059 <span class="normal">24</span> |
|
11060 <span class="normal">25</span> |
|
11061 <span class="normal">26</span> |
|
11062 <span class="normal">27</span> |
|
11063 <span class="normal">28</span> |
|
11064 <span class="normal">29</span> |
|
11065 <span class="normal">30</span></pre></div></t… |
|
11066 <span class="w"> </span><span class="kd">let</sp… |
|
11067 <span class="w"> </span><span class="kd">let</sp… |
|
11068 |
|
11069 <span class="w"> </span><span class="kd">let</sp… |
|
11070 <span class="w"> </span><span class="kd">let<… |
|
11071 <span class="w"> </span><span class="p">.&l… |
|
11072 |
|
11073 <span class="w"> </span><span class="k">match&l… |
|
11074 <span class="w"> </span><span class="n">Tok… |
|
11075 |
|
11076 <span class="w"> </span><span class="n">Tok… |
|
11077 <span class="w"> </span><span class="kd"&g… |
|
11078 |
|
11079 <span class="w"> </span><span class="k">… |
|
11080 <span class="w"> </span><span class="s… |
|
11081 <span class="w"> </span><span class="s… |
|
11082 <span class="w"> </span><span class="s… |
|
11083 <span class="w"> </span><span class="n… |
|
11084 <span class="w"> </span><span class="p">… |
|
11085 <span class="w"> </span><span class="p">},&… |
|
11086 |
|
11087 <span class="w"> </span><span class="n">_&l… |
|
11088 <span class="w"> </span><span class="p">}</s… |
|
11089 <span class="w"> </span><span class="p">};</span… |
|
11090 |
|
11091 <span class="w"> </span><span class="n">parser</… |
|
11092 |
|
11093 <span class="w"> </span><span class="nb">Ok</spa… |
|
11094 <span class="p">}</span><span class="w"></span> |
|
11095 </code></pre></div> |
|
11096 </td></tr></table> |
|
11097 <p>Lines 5 through 25 are basically</p> |
|
11098 <div class="highlight"><pre><span></span><cod… |
|
11099 <span class="w"> </span><span class="c1">// par… |
|
11100 <span class="w"> </span><span class="p">};</span… |
|
11101 </code></pre></div> |
|
11102 |
|
11103 <p>And after <em>that</em> is done, I test for <cod… |
|
11104 There is no chaining of results with helper functions; instead it's |
|
11105 just going through each token linearly.</p> |
|
11106 <p>The API break was annoying to deal with, but fortunately the ca… |
|
11107 code ended up cleaner, and I didn't have to change anything in the |
|
11108 tests. I hope rust-cssparser can stabilize its API for consumers that |
|
11109 are not Servo.</p></content><category term="misc"></category><cate… |
|
11110 but never blogged about it. Shame on me!</em></p> |
|
11111 <p>I wrote an article, <a href="https://recompilermag.com/issue… |
|
11112 Recompiler magazine. Is GNOME, now at 20 years old, legacy software? |
|
11113 Is it different from mainframe software …</p></summary><content … |
|
11114 but never blogged about it. Shame on me!</em></p> |
|
11115 <p>I wrote an article, <a href="https://recompilermag.com/issue… |
|
11116 Recompiler magazine. Is GNOME, now at 20 years old, legacy software? |
|
11117 Is it different from mainframe software because "everyone" can change |
|
11118 it? Does long-lived software have the same patterns of change as |
|
11119 cities and physical artifacts? Can we learn from the building trades |
|
11120 and urbanism for maintaining software in the long term? <em>Could… |
|
11121 turn legacy software into a good legacy?</em></p> |
|
11122 <p>You can read the article <a href="https://recompilermag.com/… |
|
11123 <p>Also, let me take this opportunity to recommend <a href="htt… |
|
11124 magazine. It is the most enjoyable technical publication I read. |
|
11125 Their <a href="https://recompilermag.com/podcast/">podcast</a&g… |
|
11126 <p><strong>Update 2017/06/10</strong> - Spanish versio… |
|
11127 the Alt-Tab behavior in gnome-shell.</p> |
|
11128 <p>The default is to have <code>Alt-Tab</code> switch … |
|
11129 current workspace. One can use <code>Alt-backtick</code> (o… |
|
11130 have above Tab) to switch between windows in the …</p></summary>… |
|
11131 the Alt-Tab behavior in gnome-shell.</p> |
|
11132 <p>The default is to have <code>Alt-Tab</code> switch … |
|
11133 current workspace. One can use <code>Alt-backtick</code> (o… |
|
11134 have above Tab) to switch between windows in the current application.<… |
|
11135 <p>I prefer a Windows-like setup, where <code>Alt-Tab</co… |
|
11136 windows in the current workspace, regardless of the application to |
|
11137 which they belong.</p> |
|
11138 <p>Many moons ago there was a gnome-shell extension to change this |
|
11139 behavior, but these days (GNOME 3.24) it can be done without |
|
11140 extensions. It is a bit convoluted.</p> |
|
11141 <h1>With the GUI</h1> |
|
11142 <p>If you are using X instead of Wayland, this works:</p> |
|
11143 <ol> |
|
11144 <li> |
|
11145 <p>Unset the <strong>Switch applications</strong> comm… |
|
11146 <code>gnome-control-center</code>, go to <em>Keyboa… |
|
11147 applications</em> command. Click on it, and hit <code>Ba… |
|
11148 dialog that prompts you for the keyboard shortcut. Click on the |
|
11149 <em>Set</em> button.</p> |
|
11150 </li> |
|
11151 <li> |
|
11152 <p>Set the <strong>Switch windows</strong> command. W… |
|
11153 <em>Keyboard</em> settings, find the <em>Switch win… |
|
11154 it, and hit <code>Alt-Tab</code>. Click <em>Set<… |
|
11155 </li> |
|
11156 </ol> |
|
11157 <p>That should be all you need, unless you are in Wayland. In tha… |
|
11158 you need to do it on the command line.</p> |
|
11159 <h1>With the command line, or in Wayland</h1> |
|
11160 <p>The kind people on <a href="irc://irc.gnome.org/#gnome-hacke… |
|
11161 3.24, changing <code>Alt-Tab</code> doesn't work on Wayland … |
|
11162 because the compositor captures the <code>Alt-Tab</code> key… |
|
11163 inside the dialog that prompts you for a keyboard shortcut. In that |
|
11164 case, you have to change the configuration keys directly instead of |
|
11165 using the GUI:</p> |
|
11166 <div class="highlight"><pre><span></span><cod… |
|
11167 gsettings <span class="nb">set</span> org.gnome.desktop.wm.k… |
|
11168 gsettings <span class="nb">set</span> org.gnome.desktop.wm.k… |
|
11169 gsettings <span class="nb">set</span> org.gnome.desktop.wm.k… |
|
11170 </code></pre></div> |
|
11171 |
|
11172 <p>Of course the above also works in X, too.</p> |
|
11173 <h1>Changing windows across all workspaces</h1> |
|
11174 <p>If you'd like to switch between windows in all workspaces, rath… |
|
11175 in the current workspace, find the <code>org.gnome.shell.window-sw… |
|
11176 current-workspace-only</code> GSettings key and change it. You ca… |
|
11177 in <code>dconf-editor</code>, or on the command line with<… |
|
11178 <div class="highlight"><pre><span></span><cod… |
|
11179 </code></pre></div></content><category term="misc"></c… |
|
11180 standard library when you open a file. I wanted to learn how Rust |
|
11181 handles system calls and <code>errno</code>, and all the lit… |
|
11182 POSIX API. This is what I learned!</p> |
|
11183 <h1>The C side of …</h1></summary><content type="html"><… |
|
11184 standard library when you open a file. I wanted to learn how Rust |
|
11185 handles system calls and <code>errno</code>, and all the lit… |
|
11186 POSIX API. This is what I learned!</p> |
|
11187 <h1>The C side of things</h1> |
|
11188 <p>When you open a file, or create a socket, or do anything else t… |
|
11189 returns an object that can be accessed like a file, you get a <em>… |
|
11190 descriptor</em> in the form of an <code>int</code>.<… |
|
11191 <div class="highlight"><pre><span></span><cod… |
|
11192 <span class="cm"> * -1 in case of error.</span> |
|
11193 <span class="cm"> */</span> |
|
11194 <span class="kt">int</span> <span class="nf">open</… |
|
11195 <span class="kt">int</span> <span class="nf">socket<… |
|
11196 </code></pre></div> |
|
11197 |
|
11198 <p>You get a nonnegative integer in case of success, or -1 in case… |
|
11199 error. If there's an error, you look at <code>errno</code>,… |
|
11200 integer error code. </p> |
|
11201 <div class="highlight"><pre><span></span><cod… |
|
11202 |
|
11203 <span class="nl">retry_open</span><span class="p">:<… |
|
11204 <span class="n">fd</span> <span class="o">=</span&g… |
|
11205 <span class="k">if</span> <span class="p">(</span&g… |
|
11206 <span class="k">if</span> <span class="p">(</sp… |
|
11207 <span class="cm">/* File doesn&#39;t exist */</span… |
|
11208 <span class="p">}</span> <span class="k">else</… |
|
11209 <span class="p">...</span> |
|
11210 <span class="p">}</span> <span class="k">else</… |
|
11211 <span class="k">goto</span> <span class="n">re… |
|
11212 <span class="p">}</span> |
|
11213 <span class="p">}</span> |
|
11214 </code></pre></div> |
|
11215 |
|
11216 <p>Many system calls can return <code>EINTR</code>, wh… |
|
11217 call", which means that <em>something</em> interrupted the k… |
|
11218 was doing your system call and it returned control to userspace, with |
|
11219 the syscall unfinished. For example, your process may have received a |
|
11220 Unix signal (e.g. you send it <code>SIGSTOP</code> by pressi… |
|
11221 terminal, or you resized the terminal and your process got a |
|
11222 <code>SIGWINCH</code>). Most of the time <code>EINTR&… |
|
11223 retry the operation: if you Control-Z a program to suspend it, and |
|
11224 then <code>fg</code> to continue it again; and if the progra… |
|
11225 of <code>open()</code>ing a file, you would expect it to con… |
|
11226 point and to actually open the file. Software that doesn't check for |
|
11227 <code>EINTR</code> can fail in very subtle ways!</p> |
|
11228 <p>Once you have an open file descriptor, you can read from it:<… |
|
11229 <div class="highlight"><pre><span></span><cod… |
|
11230 <span class="nf">read_five_bytes</span> <span class="p"&g… |
|
11231 <span class="p">{</span> |
|
11232 <span class="kt">ssize_t</span> <span class="n">re… |
|
11233 |
|
11234 <span class="nl">retry</span><span class="p">:<… |
|
11235 <span class="n">result</span> <span class="o">=<… |
|
11236 <span class="k">if</span> <span class="p">(</sp… |
|
11237 <span class="k">if</span> <span class="p">(<… |
|
11238 <span class="k">goto</span> <span class="n"&g… |
|
11239 <span class="p">}</span> <span class="k">else&… |
|
11240 <span class="k">return</span> <span class="mi… |
|
11241 <span class="p">}</span> |
|
11242 <span class="p">}</span> <span class="k">else</… |
|
11243 <span class="k">return</span> <span class="n">… |
|
11244 <span class="p">}</span> |
|
11245 <span class="p">}</span> |
|
11246 </code></pre></div> |
|
11247 |
|
11248 <p>... and one has to remember that if <code>read()</code… |
|
11249 were at the end-of-file; if it returns less than the number of bytes |
|
11250 requested it means we were close to the end of file; if this is a |
|
11251 nonblocking socket and it returns <code>EWOULDBLOCK</code> o… |
|
11252 must decide to retry the operation or actually wait and try again |
|
11253 later.</p> |
|
11254 <p>There is a lot of buggy software written in C that tries to use… |
|
11255 POSIX API directly, and gets these subtleties wrong. Most programs |
|
11256 written in high-level languages use the I/O facilities provided by |
|
11257 their language, which hopefully make things easier.</p> |
|
11258 <h1>I/O in Rust</h1> |
|
11259 <p>Rust makes <a href="https://doc.rust-lang.org/book/first-edi… |
|
11260 ignore an error, the code <em>looks</em> like it is ignoring… |
|
11261 (e.g. you can grep for <code>unwrap()</code> and find lazy c… |
|
11262 code actually <em>looks better</em> if it doesn't ignore the… |
|
11263 properly propagates it upstream (e.g. you can use the <code>?</… |
|
11264 propagate errors to the calling function).</p> |
|
11265 <p>I keep recommending <a href="http://joeduffyblog.com/2016/02… |
|
11266 discusses POSIX-like error codes vs. exceptions vs. more modern |
|
11267 approaches like Haskell's and Rust's - definitely worth studying over |
|
11268 a few of days (also, see Miguel's valiant effort to <a href="https://… |
|
11269 from exceptions for I/O errors</a>).</p> |
|
11270 <p>So, what happens when one opens a file in Rust, from the toplev… |
|
11271 down to the system calls? Let's go down the rabbit hole.</p> |
|
11272 <p>You can open a file like this:</p> |
|
11273 <div class="highlight"><pre><span></span><cod… |
|
11274 |
|
11275 <span class="k">fn</span> <span class="nf">main</sp… |
|
11276 <span class="w"> </span><span class="kd">let</sp… |
|
11277 <span class="w"> </span><span class="o">..</span… |
|
11278 <span class="p">}</span><span class="w"></span> |
|
11279 </code></pre></div> |
|
11280 |
|
11281 <p>This does <em>not</em> give you a raw file descript… |
|
11282 <code>io::Result&lt;fs::File, io::Error&gt;</code>, … |
|
11283 you actually got back a File that you can operate on, or an error.</p… |
|
11284 <p>Let's look at the <a href="https://github.com/rust-lang/rust… |
|
11285 <div class="highlight"><pre><span></span><cod… |
|
11286 <span class="w"> </span><span class="k">pub</spa… |
|
11287 <span class="w"> </span><span class="n">OpenOpt… |
|
11288 <span class="w"> </span><span class="p">}</span&… |
|
11289 |
|
11290 <span class="w"> </span><span class="k">pub</spa… |
|
11291 <span class="w"> </span><span class="n">OpenOpt… |
|
11292 <span class="w"> </span><span class="p">}</span&… |
|
11293 <span class="w"> </span><span class="o">..</span… |
|
11294 <span class="p">}</span><span class="w"></span> |
|
11295 </code></pre></div> |
|
11296 |
|
11297 <p>Here, <code>OpenOptions</code> is an auxiliary stru… |
|
11298 pattern. Instead of passing bitflags for the various |
|
11299 <code>O_CREATE/O_APPEND/etc.</code> flags from the <code&… |
|
11300 builds a struct with the desired options, and finally calls <code>… |
|
11301 on it.</p> |
|
11302 <p>So, let's look at the <a href="https://github.com/rust-lang/… |
|
11303 <div class="highlight"><pre><span></span><cod… |
|
11304 <span class="w"> </span><span class="bp">self&l… |
|
11305 <span class="w"> </span><span class="p">}</span&… |
|
11306 |
|
11307 <span class="w"> </span><span class="k">fn</span… |
|
11308 <span class="w"> </span><span class="kd">let<… |
|
11309 <span class="w"> </span><span class="nb">Ok<… |
|
11310 <span class="w"> </span><span class="p">}</span&… |
|
11311 </code></pre></div> |
|
11312 |
|
11313 <p>See that <code>fs_imp::File::open()</code>? That's… |
|
11314 platform-specific wrapper for opening files. Let's look |
|
11315 at <a href="https://github.com/rust-lang/rust/blob/3f8b93693da78c2cfe… |
|
11316 <div class="highlight"><pre><span></span><cod… |
|
11317 <span class="w"> </span><span class="kd">let<… |
|
11318 <span class="w"> </span><span class="n">File<… |
|
11319 <span class="w"> </span><span class="p">}</span&… |
|
11320 </code></pre></div> |
|
11321 |
|
11322 <p>The first line, <code>let path = cstr(path)?</code>… |
|
11323 into a nul-terminated C string. The second line calls the following:<… |
|
11324 <div class="highlight"><pre><span></span><cod… |
|
11325 <span class="w"> </span><span class="kd">let<… |
|
11326 <span class="w"> </span><span class="n… |
|
11327 <span class="w"> </span><span class="n… |
|
11328 <span class="w"> </span><span class="p… |
|
11329 <span class="w"> </span><span class="kd">let<… |
|
11330 <span class="w"> </span><span class="n">ope… |
|
11331 <span class="w"> </span><span class="p">})</… |
|
11332 <span class="w"> </span><span class="kd">let<… |
|
11333 |
|
11334 <span class="w"> </span><span class="o">..</… |
|
11335 |
|
11336 <span class="w"> </span><span class="nb">Ok<… |
|
11337 <span class="w"> </span><span class="p">}</span&… |
|
11338 </code></pre></div> |
|
11339 |
|
11340 <p>Here, <code>let flags = ...</code> converts the <… |
|
11341 beginning to an int with bit flags.</p> |
|
11342 <p>Then, it does <code>let fd = cvt_r (LAMBDA)</code>,… |
|
11343 calls the actual <code>open64()</code> from libc (a Rust wra… |
|
11344 libc): it returns a file descriptor, or -1 on error. Why is this |
|
11345 done in a lambda? Let's look at <a href="https://github.com/rust-lan… |
|
11346 <div class="highlight"><pre><span></span><cod… |
|
11347 <span class="w"> </span><span class="k">where</s… |
|
11348 <span class="w"> </span><span class="n">F<… |
|
11349 <span class="p">{</span><span class="w"></span> |
|
11350 <span class="w"> </span><span class="k">loop</sp… |
|
11351 <span class="w"> </span><span class="k">match&l… |
|
11352 <span class="w"> </span><span class="nb">Er… |
|
11353 <span class="w"> </span><span class="n">oth… |
|
11354 <span class="w"> </span><span class="p">}</s… |
|
11355 <span class="w"> </span><span class="p">}</span&… |
|
11356 <span class="p">}</span><span class="w"></span> |
|
11357 </code></pre></div> |
|
11358 |
|
11359 <p>Okay! Here <code>f</code> is the lambda that calls… |
|
11360 it in a loop and translates the POSIX-like result into something |
|
11361 friendly to Rust. This loop is where it handles <code>EINTR</c… |
|
11362 translated into <code>ErrorKind::Interrupted</code>. I supp… |
|
11363 for <code>convert_retry()</code>? Let's look at |
|
11364 the <a href="https://github.com/rust-lang/rust/blob/3f8b93693da78c2cf… |
|
11365 <div class="highlight"><pre><span></span><cod… |
|
11366 <span class="w"> </span><span class="k">if</span… |
|
11367 <span class="w"> </span><span class="nb">Err<… |
|
11368 <span class="w"> </span><span class="p">}</span&… |
|
11369 <span class="w"> </span><span class="nb">Ok<… |
|
11370 <span class="w"> </span><span class="p">}</span&… |
|
11371 <span class="p">}</span><span class="w"></span> |
|
11372 </code></pre></div> |
|
11373 |
|
11374 <p>(The <code>IsMinusOne</code> shenanigans are just a… |
|
11375 multiple integer types without a lot of <code>as</code> cast… |
|
11376 <p>The above means, if the POSIX-like result was -1, return an <… |
|
11377 the last error returned by the operating system. That should surely |
|
11378 be <code>errno</code> internally, correct? Let's look at |
|
11379 the <a href="https://github.com/rust-lang/rust/blob/3f8b93693da78c2cf… |
|
11380 <div class="highlight"><pre><span></span><cod… |
|
11381 <span class="w"> </span><span class="n">Error&l… |
|
11382 <span class="w"> </span><span class="p">}</span&… |
|
11383 </code></pre></div> |
|
11384 |
|
11385 <p>We don't need to look at <code>Error::from_raw_os_error()… |
|
11386 conversion function from an <code>errno</code> value into a … |
|
11387 However, let's look at <a href="https://github.com/rust-lang/rust/blo… |
|
11388 <div class="highlight"><pre><span></span><cod… |
|
11389 <span class="w"> </span><span class="k">unsafe</… |
|
11390 <span class="w"> </span><span class="p">(</s… |
|
11391 <span class="w"> </span><span class="p">}</span&… |
|
11392 <span class="p">}</span><span class="w"></span> |
|
11393 </code></pre></div> |
|
11394 |
|
11395 <p>Here, <code>errno_location()</code> is an <code&… |
|
11396 (or whatever C library your Unix uses). It returns a pointer to the |
|
11397 actual int which is the <code>errno</code> thread-local vari… |
|
11398 code can't use libc's global variables directly, there needs to be a |
|
11399 way to get their addresses via function calls - that's what |
|
11400 <code>errno_location()</code> is for.</p> |
|
11401 <h2>And on Windows?</h2> |
|
11402 <p>Remember the internal <code>File.open()</code>? Th… |
|
11403 like <a href="https://github.com/rust-lang/rust/blob/3f8b93693da78c2c… |
|
11404 <div class="highlight"><pre><span></span><cod… |
|
11405 <span class="w"> </span><span class="kd">let<… |
|
11406 <span class="w"> </span><span class="kd">let<… |
|
11407 <span class="w"> </span><span class="n">c&l… |
|
11408 <span class="w"> </span><span c… |
|
11409 <span class="w"> </span><span c… |
|
11410 <span class="w"> </span><span c… |
|
11411 <span class="w"> </span><span c… |
|
11412 <span class="w"> </span><span c… |
|
11413 <span class="w"> </span><span c… |
|
11414 <span class="w"> </span><span class="p">};</… |
|
11415 <span class="w"> </span><span class="k">if</… |
|
11416 <span class="w"> </span><span class="nb">Er… |
|
11417 <span class="w"> </span><span class="p">}</s… |
|
11418 <span class="w"> </span><span class="nb">Ok… |
|
11419 <span class="w"> </span><span class="p">}</s… |
|
11420 <span class="w"> </span><span class="p">}</span&… |
|
11421 </code></pre></div> |
|
11422 |
|
11423 <p><code>CreateFileW()</code> is the Windows API funct… |
|
11424 conversion of error codes inside <code>Error::last_os_error()</… |
|
11425 analogously - it calls <code>GetLastError()</code> from the … |
|
11426 converts it.</p> |
|
11427 <h2>Can we not call C libraries?</h2> |
|
11428 <p>The Rust/Unix code above depends on the system's libc for <c… |
|
11429 <code>errno</code>, which are entirely C constructs. Libc i… |
|
11430 the system calls. There are efforts to make the Rust standard library |
|
11431 <em>not</em> use libc and use syscalls directly.</p> |
|
11432 <p>As an example, you can look at |
|
11433 the <a href="https://github.com/rust-lang/rust/blob/3f8b93693da78c2cf… |
|
11434 system kernel entirely written in Rust. Fun times!</p> |
|
11435 <p><strong>Update:</strong> If you want to see what a … |
|
11436 like, <a href="https://github.com/japaric/steed">take a look at st… |
|
11437 without C dependencies.</p> |
|
11438 <h1>Conclusion</h1> |
|
11439 <p>Rust is very meticulous about error handling, but it succeeds in |
|
11440 making it pleasant to read. I/O functions give you back an |
|
11441 <code>io::Result&lt;&gt;</code>, which you piece apa… |
|
11442 error.</p> |
|
11443 <p>Internally, and for each platform it supports, the Rust standard |
|
11444 library translates <code>errno</code> from libc into an <… |
|
11445 enum. The standard library also automatically handles Unix-isms like |
|
11446 retrying operations on <code>EINTR</code>.</p> |
|
11447 <p>I've been enjoying reading the <a href="https://github.com/r… |
|
11448 has taught me many Rust-isms, and it's nice to see how the |
|
11449 hairy/historical libc constructs are translated into clean Rust |
|
11450 idioms. I hope this little trip down the rabbit hole for the |
|
11451 <code>open(2)</code> system call lets you look in other inte… |
|
11452 an |
|
11453 <a href="https://people.gnome.org/~federico/misc/activity-log.el">… |
|
11454 Back then, I seemed to write multiple short blog entries in a day |
|
11455 rather than longer articles (<em>doing Mastodon before it was cool… |
|
11456 But my blogging patterns have changed. I've been wanting to add …<… |
|
11457 an |
|
11458 <a href="https://people.gnome.org/~federico/misc/activity-log.el">… |
|
11459 Back then, I seemed to write multiple short blog entries in a day |
|
11460 rather than longer articles (<em>doing Mastodon before it was cool… |
|
11461 But my blogging patterns have changed. I've been wanting to add some |
|
11462 more features to the script: moving to a page-per-post model, support |
|
11463 for draft articles, tags, and syntax highlighting for code excerpts...&l… |
|
11464 <p>This is a wheel that I do not find worth reinventing these days. |
|
11465 After <a href="https://mastodon.social/@federicomena/8360985">aski… |
|
11466 generators (thanks to everyone who replied!), I've decided to give |
|
11467 <a href="https://blog.getpelican.com/">Pelican</a> a try. I… |
|
11468 documentation" is high on my list of things to look for when shopping |
|
11469 for tools, and Pelican's docs are nice from the start.</p> |
|
11470 <p>The old blog is still available <a href="https://people.gnom… |
|
11471 <p>If you find broken links, or stuff that doesn't work correctly … |
|
11472 |
