Meeting 2014 09 22

Agenda

• work week update (jack)
• Pending changes to build/test infra (larsberg)
• Note: looking for a new HTTP library - https://github.com/servo/servo/wiki/HTTP-library-requirements (jdm)
• libgreen upstream removal plan (larsberg)
• html5ever status (kmc)
• cargo issues (jack)
  • what issues remain
    • Running tests in dependencies
    • Sharing target dir with ports/cef
  • what needs addressing now?
• pointers-to-pointers - JSRef, Atom, ... (zwarich)
• hi-pri / blocking rust issues (larsberg)

Attending

• mrobinson, jdm, zwarich, SimonSapin, Manishearth, brson, larsberg, mbrubeck, cgaebel, azita, jack

Work week

• jack: Mountain View (SF space was not available). Looks like ebalint +1 from U.Szeged (Hungary). No word from Samsung OSG or Samsung HQ. Possibly Junyoung & Ryan like last time. Unknown from OSG. Logistics for room and food are set up. The hotel (Avante) is an easy walk from the office and the Moz offices has Monday and Friday catering. We have catering Wed. The other two, at least there are food trucks on Thursdays. Tuesday we will have to figure out. Registration link should be out - do it today if you haven't already. Questions?

Pending changes to build/test infra

• larsberg: Travis works great on small projects, but larger number of PRs doesn't let it keep up. Jack and I want to resurrect buildbot on much higher-powered machines. Look for bors to reappear over next several days. Travis was good experimented but we're going to put it behind us. Feedback desired; the current time to land patches is unacceptable, so if you disagree talk to Jack or me.
• jack: Build cycle times should be as low as possible without spending lots of money on hardware. Should be under 10 minutes after this change as a ballpark. We can spin up multiple machines to do builds in parallel like Travis. We won't have delays due to amazon spot-bidding, so the goal here is not only to fix frustrations of unreliable builds but we can make them really fast, too. First step to get it working, then some optimizations for speed, and the cadence should be must higher.
• brson: Switching to AWS?
• jack: No, takes too long to spin up. Leasing machine in MacStadium (physical mac minis) and some nodes in Linode (cheaper, so can leave on all the time instead of spot bidding).
• kmc: Will we still have easy try builds?
• jack: Not in this first iteration, but we want to bring them back. Right now the goal is to make the CI loop work.
• Manishearth: Do like Rust team, use Travis for try?
• kmc: It's been broken for weeks...
• jack: Not that nobody cares about it, but uncertain why it's broken and Travis unresponsive.
• kmc: Nobody cares enough to put in the effort, which is considerable, is my point.
• zwarich: Even if someone on Rust team wanted to put in full time to figure it out...
• jack: Fails about 50% of the time.
• SimonSapin: 10 minutes for the builds or also runnign the testsuites?
• jack: Hopefully all of it. That will be form several places - waiting on spinning up machines in travis, waiting long time for builds, waiting long time for tests... on my local machine build is 5 minutes end to end. We won't break anything while we do this, hopefully. We'll get bors up and running and let everyone know when to start testing it.

New HTTP library

• jdm: Just a heads-up that there's a bunch of activity in the HTTP client space. We've put together a wiki page for our requirements: https://github.com/servo/servo/wiki/HTTP-library-requirements
• jdm: We will keep adding to that. We have at least group of devs (http://hyperium.github.io/hyper/hyper/index.html who are very interested in helping us, so we can hopefully move off of rust-http onto something that is being actively maintained. If you have thoughts, please put them in the wiki page and talk with me.
• brson: Which project are you enthusiastic about?
• jdm: hyper. John Reem & seanmonstar, I believe.

libgreen removal

• larsberg: As we know, libgreen is being deprecated. I talked with Alex and Aaron about how this will be staged and what it means for Servo. First step they are going to remove librustuv (I/O for green threads). Then they will split out all the libgreen functionality into a crate. So our first step is to remove any I/O we are doing on green threads. Second step, start linking to this new libgreen external crate. We have some ongoing discussions about the right move in the long term for lightweight threading.
• zwarich: I forgot to reply to the email thread about this, but I think someone made an assertion that our green-threads that do I/O are all singletons. I think that's incorrect; in a lot of cases we have one resource thread per pipeline. Are we going to change that?
• larsberg: Are they per constellation or per pipeline?
• kmc: They should be global because we want to share caches and such.
• jdm: They're global (per constellation).
• larsberg: We create them when we create the constellation. In some brave new world where Servo supports multiple windows/tabs we'd need to change them to be shared across constellations.
• jack: But we will need to be able to load multiple images concurrently.
• kmc: At some point we may want to write our own concurrent loader with its own async I/O. Not urgent right now, though.
• jack: Is there a timeline for the libgreen changes?
• larsberg: "Soon."
• jack: But not this week, from what I understand.

HTML5ever

• kmc: Have a branch that builds and passes most tests in Servo. Next step is to find some people willing to review ~7k LOC.
• zwarich: Can I see it before signing up?
• kmc: On github (https://github.com/kmcallister/html5ever). Will move it to Servo org at some point.
• larsberg: We have projects which are nominally upstream, but in reality upstream is original author. How should we be reviewing this? Often review is "did we update submodule pointer".
• jack: Is 7k all of html5ever and servo integration?
• kmc: It's html5ever and tests, etc. Integration into Servo is comparable to current hubbub bindings. A few hundred lines.
• jdm: I'll do the integration part
• kmc: SImilar to jsmanaged code.
• simonsapin: We've landed big chunks in the past without review, for new libraries like this.
• kmc: I'd appreciate the review. It's a big chunk of code and there's a lot going on. But that's up to the team.
• jack: Our macro masters are SimonSapin and Manish. Maybe one of them can look at it?
• kmc: I have a 400 line procedural macro. That's the scariest part. And some smaller stuff that isn't too bad.
• jdm: ms2ger has volunteered!
• SimonSapin: Playing with a macro once = you're a master?
• jack: Well, you've done more than I have.
• zwarich: Wouldn't it be fitting if you could review your own macro code, since it generates the code in the first place?
• jack: Awesome! This is exciting. I can't wait to see hubbub go away.
• kmc: I'm excited, too. Few people in #rust have been interested in using it and my flatmate is intersted in using it from Python.
• jack: In the CTO presentation, I used html5ever as one of my points where we could replace something inside of gecko with something that servo has produced. Since it's more substantial than libjpeg, it's more interesting. This will probably result in some meetings with the Gecko team, but it does have to wait on a stable Rust compiler.
• kmc: The architecture is similar to Gecko's. Character encodings will be an issue. I also don't implemnet View Source that aren't in a spec but are part of a real browser. Also need the document.write and async loading support.
• jdm: ms2ger asked about off-thread parsing?
• kmc: Parser is parametrized over it. My plan is to support on-thread parsing with the nodes just raw pointers, but in off-thread parsing, you can use message sending between tasks and serialize everything in the traits for tree builders. That's also work that has not been done yet. I don't think it will be very challenging and it's similar to how Gecko already does things.
• jack: We can test that support locally.

Cargo issues

• jack: Have we surfaced all of the cargo issues? Running tests and dependencies independently are the big ones. All task dependencies should not consume the output or we'll get a warning. So we'll get them on rust-azure... if you do 'mach bootstrap cargo --force' then you'll get to see them now.
• https://github.com/servo/servo/issues/is:open 2853
• https://github.com/rust-lang/cargo/issues/482
• jack: Sharing target dirs we're working on a workaround for it.
• SimonSapin: I have a PR with a workaround for it, but alex does not want it.
• mbrubeck: I put the link in with the tracking bug and the sharing target dirs one (which is the main backend part for most of our problems). The work to be done is fairly simple, but it's a matter of getting agreement on the interfaces and design. I can help with the implementation, too, if needed.
• jack: I'll chat with Alex and find out where these are on his radar. If we can't get them up, we can do it ourselves. I think there's also some features support that we need for conditional per-platform compilation (to remove the glfw strangeness). The main thing is: are there any more cargo issues? Also, which are the most important? Tests & dependencies is the top.
• jdm: Debug vs. opt mozjs is really really important: https://github.com/rust-lang/cargo/issues/601
• jack: Arbitrary custom profiles?
• SimonSapin: Could it be a feature? https://github.com/rust-lang/cargo/issues/385
• jack: I don't think so. I think features pull in dependencies conditionally, and jdm wants different build flags. Skia and SM have special debug flags, but we don't do them normally.
• mbrubeck: Features set a CFG flag that you can check in Rust code, but doesn't help for native dependencies.
• jack: Maybe a shell environment variable would be sufficient.
• kmc: I'll need it for html5ever, too. Planning to maintain versions for rust master and servo, using CFG directives ala rustc's bootstrapping support.
• jack: Won't work for syntax changes?
• kmc: Syntax doesn't change that much in ways that matter before cfg directives are applied. It works okay for the Rust bootstrap. I've been asking people what the best practices are for maintaining a rust library at several compiler versions, and there aren't people who do that today.
• jack: Speaking of branches, all of the submodules we own we keep pegged to the servo working version unless an external community member comes along. I'd like to move to a system where the submodules have travis yaml, track rust nightly, and servo is just behind a bit. Maybe it will get us some more users.
• kmc: A friend of mine wants to write a console text-mode browser using some of servo's parts, so there are a lot of projects that could take advantage of it if we were on master.
• simonsapin: I do it on cssparser and url. I usually don't even need a branch; just point cargo.lock back a bit. It's just if we want to make changes before upgrading Rust that we need a branch.
• jack: Rust is stable enough that this should be doable. We only have two projects that don't point at upstream master: glfw and rust-http. I suspect all of these submodules should be pretty easy. So, if you're landing changes to submodules, you might want to put a travis yaml in there and get it going.
• SimonSapin: Another issue is that sometimes when downloading a nightly, I get another version, and it's kind of random. CDN is caching strange versions and not updating them. https://github.com/rust-lang/rust/issues/16649
• jack: For cargo?
• SimonSapin: Cargo or Rust.
• jack: Better naming scheme for nightlies? Can we put the name in there?
• brson: Archived nightlies?
• jack: We download rust-target-triple.nightly.tar.gz. Sometimes get a bad one.
• brson: We have an idea, but we haven't implemented it yet. We could also have an archive with a date, but neither are done. If you need them,we could start. I think fixing the issue with the single nightly out of sync is something we want for 1.0.
• jack: Sounds sufficient for me.
• SimonSapin: It just means that if we want a cargo update, we have to wait a few days. Because we have our own Rust snapshots in Servo - we don't use nightly.
• jack: The CI stuff downloads the current cargo nightly every build. So if we want to roll out a new Cargo build, we have to wait a few days for the caches to invalidate. Not a huge problem.

Rust nightly issue: https://github.com/rust-lang/rust/issues/14431

Pointers to pointers

• zwarich: I've been working, based on discussions with nrc, on smart pointers and inheritence to never take another pointer byref and pass it. Previously, DOM bindings were on JSRef<T> and all were &self. That makes no sense becaues JSRef is already a pointer, so why take another pointer to it? I got rid of most, but two cases. One needs a Deref trait change. The other is that for trait objects they only work if the self param is &self or a Box<self>, but doesn't work for just JSRef<T>. nrc would like to have traits on T where all the methods where self is JSRef<self>. So just like Box, you can have methods on JSRef, and that's where we want to go. If there's any reason you know that would not be compatible, please let us know. This also plays into the struct inheritance debate that will go down tomorrow, because of vtable plus JSRef<T> access requirement. It's assumed the ones that are trait-based will be able to be extended via deref to get the vtable ptr and then again to the methods that take a JSRef<T>. Also trying to extend this to Atom because we do a lot of &Atom. I want to do the same thing for RC and Arc, though we don't use them much in Servo. But I want to fix it in the standard library.
• kmc: In passing, part of the html5ever part changes our namespace enum to string-cached Atoms, so there will be a lot more of them in Servo. When I was writing the parser, I didn't think hard about these issues with Atom and was planning to go back and do micro-optimizations. I also have a pie-in-the-sky Rust feature that would let the implementation of RC<T> detect when a cloned value will not outlive the original and skip the refcount then, so you could pass RC<T> by value and get the efficiency gains.
• zwarich: My approach is that these should be encodable in the Rust language. Separate RCRef type that has a lifetime param anchored to the known RC and it's freely duplicable. The RCRef gives you the information (Beyond just &T) is that the thing pointed to by the pointer has an RC header before it. So, if you want to make a new RC, you can do it from an RCRef, but can't from an &T because you don't know if there's an RC header. That's my approach and I think it should work, and fortunately doesn't require language support.
• kmc: Sounds similar to RefCell, so should work out fine.
• jack: When we pass a pointer to a pointer, does the optimizer fix that up?
• zwarich: The only way it could is if things get inlined, but with virtual methods, they're never going to get inlined. So, only in those cases, which won't happen for JSRef a great deal, but I didn't go dump the LLVM IR to confirm that.
• kmc: Custom smart pointers for self is a feature that will probably be really useful.
• zwarich: If we decide on an inheritance proposal but we don't have this feature, then we've made the rust team do a ton of work and it wouldn't be clear if we could directly use it. We should make this especially clear at the meeting tomorrow.

Rust high-priority issues

• larsberg: Rustdoc infinite loop. Is that something I should push on? Is it blocking us?
• SimonsSapin: For now we've just disabled docs for libstyle. IT would be nice, but it's not blocking per se.
• larsberg: Ok. It looks like people are poking at it.

More Cargo stuff

• larsberg: Has anyone noticed that deleting libscript and doing a cargo build doesn't work?
• mbrubeck: Yet another thing that cargo#482 would get us (I think).
• SimonSapin: Seen something similar with documentation. If you delete docs, not everything gets rebuilt.
• jack: I have an issue filed; there's no incremental way to clean a dependency, and that's on a todo list for this week along with cargo update + revision and adding package ids to more commands. https://github.com/rust-lang/cargo/issues/573
• SimonSapin: Can I haz cargo clean-doc?
• jack: It should, but I'm not sure if it's on the list. Take a look at the issue to see which ones he's adding. If he forgets one of these, my understanding is the work should be done so it won't be hard to add.
• zwarich: Doesn't detect when dom bindings need rebuilding.
• mbrubeck: I believe that's fixed; I sent an email last week when the fix landed.
• zwarich: When we get warnings and errors, you don't get full path to the profile.
• jack: That may be fixed. Previously, Cargo captured all output and re-colorized it for printing. That might be why we were losing filename. Now it should be spitting out same output for rustc. We should make sure something's filed if it's not fixed.
• zwarich: Will I hit the caching thing if I get a new cargo?
• jack: You shouldn't, just travis.
• SimonSapin: Sometimes it affects Travis, sometimes me...
• jack: Maybe because you're in Europe. I haven't seen this problem; use mach cargo --version to double check.