Network unavailable #170
Comments
|
Hi @psvensson, thanks for raising this. Unfortunately, TruffleSqueak's implementation of the SocketPlugin is currently incomplete and still has a few issues, so it is not surprising that some networking tasks fail. The problem is that we're emulating what the SocketPlugin does using Java NIO and that's not always working. Ultimately, it would be nice to implement the SocketPlugin based on a POSIX backend, similar to GraalPython (#39 (comment)). |
|
Hi Fabio, thanks for the quick reply. OK, but then it is a known issue and you also know where it can be fixed. I know a bit of Smalltalk and a bit of Java, but not much about GraalVM or indeed truffleSqueak. I will probably fail but will see what I can do :) I do not know Python so would start to see if I could find a good enough solution fixing existing bugs / implementing what needs to implemented. I assume that I start looking here; https://github.com/hpi-swa/trufflesqueak/tree/main/src/de.hpi.swa.trufflesqueak/src/de/hpi/swa/trufflesqueak/nodes/plugins/network Do you have any more hints or ideas on how to go about this, obvious shortlist of what is not working or something like that? |
|
Yes, it is a known issue. It just never was pressing enough to fix because, well, you can do networking via interop with Ruby or other programming languages. 😉 I don't think you need to know much about Python. GraalPython is, just like TruffleSqueak, implemented in Java using Truffle. An example: One more tip: If you select the TruffleSqueak test image the first time you start TruffleSqueak, you can also look at the original Slang sources of the SocketPlugin, which may be helpful as well. I don't want to discourage you in any way but I think it's fair to warn you that this is not really a beginner project. Please feel free to give it a shot and asked questions! |
|
Yes, of course. But it would be more easy for first-time users to come to what otherwise works as a normal Squeak environment, and then learn a bit at a time from there. Thank you! This is much more info that I had hoped for. I am not a beginner, really. I will definitely ask. By chance my vacation start tomorrow also, so good winds ahead.. |
|
Just thought I could post my proceedings here, for reference. I have been bale to compile from command-line both GraalVM and truffleSqueak according to instructions, using the proper JVM downloaded by mx. It then did prove possible to connect eclipse as a remote debugger (using an already defined target for this - thanks) to connect to a truffleSqueak instance using according to the development instructions. I then set a lot of breakpoints in SqueakSocket.java and SqueakTCPSocket.java. It turns out that name resolution works well, I'm able to set breakpoints in SqueackTCPSocket::receiveDataFrom where an address is present. But there things are stuck and I'm trying to figure out why. It feels like receive never returns, and then eventually it times out and truffleSqueak gets back nil and throws an exception. The odd hting is that I can see that data is received, and follow it a couple of times. I'm still trying to get to grips with Eclipse deugging again, but I'm fairly sure that something comes across.
|
|
Hmm, I might juts as well have started properly inside truffleSqueak, though, as when I look more closely, it does actually receive data, the main problem seems to be that the connectiontimes out before data is fully read, or that it thinks so.
|
|
Great that you're able to dive into networking in TruffleSqueak! I remember that there are some issues with certain websites. I just had a quick look and the non-SSL version of the website seems to work fine ( In case you only need to send some simple http requests, you can also implement a subclass of |
|
Ahaa, good catch! yes, this seems to be just about https indeed. It might be SqueakSSL, but I also saw some comments in the SecureSocketStream, which says: "Note: The loop here is necessary to catch cases where a TLS packet is So I'll check that out first. It feels like that for SSL, something thinks there is data to be read when it really isn't any, so I hpoe to narrow down where soon. And yes, I could call out to node.js for example, but my goal is to see if it would be possible to lift truffleSqueak to a level where it's more or less as usable as the standard distribution, but with ultra-mega-superpowers. So I'll continue to weld here in the cellar for a while longer :) |
|
Silly question, but what is the easiest way to get some logging output from the Java/vm code? I tried something like this; And checked bout stdout and the console tab in Eclipse so of course nothing showed (but it does get called, I can set a breakpoint there, for example). Since data transport function get called many times, debugging doesn't give the whole picture I need to see where things go awry, so I do need a way to print things over time. In Squeak land Transcript works well as usual, of course. There are also logging options to set for the vm when starting it, but I don't understand what to write here; --log.[logger].level= Set language log level to OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST or ALL What are possible values of [logger] here? |
|
Not a silly question at all, looks like we haven't documented this yet. You've seen the list of loggers, and the way you use enable is like this: # enable only the socket logger on the FINER level
trufflesqueak --log.smalltalk.socket.level=FINER
# enable all loggers at the same time on the ALL level (this is very verbose)
trufflesqueak --log.smalltalk.level=ALL |
|
Thanks! Also, I tried to do --log.java.level=ALL but nothing happened with that either. It sounded like it should work and I am 100% for this, but more magic is needed, apparently |
|
For the ongoing research I'm now fairly convinced that the culprit is the SqueakSocket::isDataAvailable method in teh plugin, which I think does not flag that data is available when it is. |
|
Can you check that BTW, I recommend something like this: Bonus points for setting |
|
and this would let you run with the Java debugger: |
|
Aha! Wow, of course. I had all the time happily just started trufflesqueak without specifying which one, so of course I ran the earlier installation of trufflesqueak in my classpath from when I downloaded it while trying things out. Now, when I correctly start using the following line;
I do see a lot of logging, including my System.out.println. Thank you, the car is moving again :) |
|
However, now that am using the correct and locally built truffleSqueak, I am starting to see a lot of DNUs for getLoopAndCount . They appear after I've get the usual exception on not being able to go to the https site, and after that appear every time I try to do anything with a calltarget browser (like looking at any code, so I then need to restart again. Any ideas what it might be? |
|
It seems like you're using the If you don't need to use the CallTargetBrowser, I suggest you switch back to the standard Browser via, for example, |
|
Thanks a lot for the help |
|
I have now run into another problem which is very confusing. I merely made the suggested changes (and changed browser), saved the image and then restarted. Now, after I start truffleSqueak and then tries to attach the Eclipse debugger, I get this; I have tried to rebuild the truffleSqueak vm and then all of it, but to no avail. I then thought that the changes file of the image might have been damaged (maybe I terminated it too soon?) so I removed it and tried again, but got the same thing again. It is very strange, I have not had time to do anything in particular (what I can remember). Is this something you have come across before? As a last resort I'll download everything again and begin from scratch. |
|
It seems the image has been damanged somehow. I believe the first positional argument is the image and all following are considered image arguments. So to avoid confusion, I'd either move the log option before the image file or drop the image argument. If no image is provided, TruffleSqueak will automatically use the one in the resources directory. To get a fresh test image, simply empty "Installs/graalvm-ce-java17-22.1.0/languages/smalltalk/resources/" and run TruffleSqueak again. It should asked you again to select an image. |
|
Thank you, that did the trick. |
|
Some observations while spelunking the code;
This just in case anyone else stumbles on the same things in the future :) The Graal socket python code is not like the smalltalk one, so they don't compare directly. But it's very little code in both cases. The problem arises when in the SecureSocketStream tries to read further from the socket but is blocked until timeout. It reads initial data but then in a subsequent step just sits there. It has to be an assumption on what the underlying layer should do, or not, but it is not very obvious (yet) what it is. I'm reading up on java.nio and getting my bearings. Will be travelling for a couple of days now, so getting back to this later in the week. |
|
So, while waiting for packing to be done, I noticed something interesting. When using the WebClient to go directly to 'https://dn.se' it reads a bit then times out, but when going to the basic vanilla http url 'http://dn.se', I notice that it first reads from port 80 but then apparently gets redirected to https --- and that works. That is super-weird though, so it means that something special happens when connecting initially using TLS. Oh well, vacation it is, then. |
|
Good find, that is indeed interesting! And enjoy your vacation! I'm currently occupied with some other things anyway until the end of next week, but it'd be great to fix this at some point and happy to help as much as my time allows. |
|
Since the truffleSqueak SockePlugin obviously works in some cases, I have been trying to find the difference in handling from the SqueakSSL side between going directly to https and being redirected to https. I had several great theories, but none stuck unfortunately. I now tried other sites than the one I've been testing, and interesting things happens. vanilla http request On the other had, going directly to https to github; The only reason I can think of is that it is the WebClient which is wonky. Trying to update Squeak fails happily as usual, trying to access it might be that it is the WebClient which is not correctly hotted-up to dance the latest dances? However, that can't be true since it does actually work in a vanilla VM squeak 6.0. Very X-files right now :) |
|
TruffleSqueak's image and test image are based on a fixed Squeak trunk image that we haven't updated in a while. So maybe worth doing these experiments in a Squeak 6.0 release image since |
|
Yes, that is a good idea. Actually, I was wondering how to create new truffleSqueak images and a bit about how one would go about to write the Smalltalk parts (if needed) to get truffleSqueak to work with another Smalltalk (I was thinking Pharo, for example). |
|
Hmm, I have also been trying to find the 'gu' utility in the graalvm directory in the trufflesqueak-build folder I created, but I can't seem to find it. In the pre-built graalvm installation it can be found in bin/gu which is a link to lib/installer/bin/gu but nothing can be found in the mx built graalvm folder in the same places. I have then tried to understand if there are any specific mx build command that can be used to build and 'install' the gu command, but I don't see ny there either. To be fair, I've never used mx before so I'm mostly plodding along semi-randomly. If there is an easy way to to this, it would be great, because then I can start installing node.js and see if I can install packages (which was my next step anyway) |
|
Thank you, of course i was already documented, sry :) I'll give it a try and see if anything have changed in WebClient to the better. And I can't say I'm surprised that Pharo is a tougher nut to crack. Sometimes it feels like the whole point of something is to be incompatible, when possible. But it is much easier to work with out of the box (code completion, searching for anything, et.c.) Maybe I can find something done for Squeak too, I'll look around. |
|
The first parts of the setup image instructions went well, but when executing the last 'setupImage' part,
I get this exception More specifically it is this that fials;
|
You're just building TruffleSqueak plus the base GraalVM JDK, so which builds a GraalVM + compiler + TruffleSqueak + to build a GraalVM + compiler + TruffleSqueak + GraalVM's Node.js runtime + However, I'd suggest you do this instead:
This way, you can install any other language (e.g., |
There's also a
I'm not surprised, but this is not too much of a problem because it's setting up some workspace or something. Try running the TruffleSqueak tests, that'd be interesting. |
|
This is excellent stuff! Could you please add this to the existing documentation, I think it would be really helpful to have it all in one place (and so one can search for it also). OK, now I need to make a choice hmmm :) |
|
I hope it's not too much to ask, but it'd be nice if you could contribute some extensions for the docs via a PR. That way, I can also learn and comment on how you use things. Otherwise, maybe open an issue for tracking this so I don't forget and do it when I have some more time? |
|
NP, I'll make a small PR of an edited version of your text, to the development documentation |
|
So, hmm. I have had a bout of Covid so been under the ice for most of the week. But today I managed to fix the trufflesqueak setup utilities to work with the actual squeak 6.0 release. They have made several changes that are probably nice but completely ignores backward compatibility. It almost feels like a Pharo core dev have been working with this :) . This fix might not be ideal, but I crated a PR for the 'image' branch and we can crate a new issue if/when you want to have any changes done to it. Anyhow, after I got the setup working, I again tried to do the very first WebClient test that is the reason for this issue in the first place -- and it worked. Of course it now worked.
So the problem was indeed some weird bug in the WebClient handling of SqueakSSL which was fixed between the release used in the TruffleSqueak documentation and the official release - I don't even want to go back and find out what it was, but now it runs and purrs. |
Oh, I hope you are better now! Thanks for the PR!
That's great news, thanks for checking! According to the Squeak/Smalltalk 6.0 release notes, We'll upgrade to the 6.0 release image very soon (after the upcoming 22.2 release), but that will mean that the new 6.0-based image will ship with TruffleSqueak 22.3 in a couple of months (we'll need some time to make sure everything's working ok). |
|
I am, thanks! Yes, this has been a bit of a surprising ride, but it all ended well. |
|
Good! I'm keeping this open until we've adopted the 6.0 release. The most work is going to be updating the test map in |
|
OK, sounds good! I'll lick my wounds a bit until the next release then. I also had some problems using a recent dev graalVm , this dev truffleSqueak and adding node.js. Also I could not figure out a way to list the installed languages in truffleSqueak in a printable format. All I got was just remote objects. I suspect this is just a bit of missing docs (or me not finding it) but we can take that later then. Very glad we got this far. |
|
Sure, let me know in case you run into problems and don't know how to proceed. For language names, you can, for example, do this: |
|
OK, I will try this, thx |
|
So, this is embarrassing. I have had fevers on and off and it's sometimes easy to mix things up. The image I tested was indeed the latest 6.0 release - but it was running on a standard VM. I had just started it some days earlier trying to debug the trufflesqueak image setup process. So o course the (now also fixed) WebClient worked when I tried it. Now I saved the image and restarted it under my latest GraalVM/Trufflesqueak installation, and it sadly show the same problems as before - unable to read more than an initial amount from a https url, but works well when using http which redirects to https. |
I hope you are feeling better now!
Ok, so this issue is essential still a problem? |
|
A little bit, thank you. Yes, it seems so, unfortunately. I will try out Cuis as well (and Maybe Pharo) to see if I can get something out of that. Also, I'll start checking out how the open smalltalk VM implements things, in case something pops out. |
|
Cuis seems to have the same issue, but behaves a little bit differently. It doesn't time out after 45s but instead throws an exception when the remote side closes the connection.
I am trying to understand what needs to be installed in Pharo to make it work over truffleSqueak but I'm stuck trying to figure out how the Cuis image setup works and what it is doing; https://github.com/hpi-swa/trufflesqueak/blob/main/docs/images.md#trufflesqueak-image-creation I did look quickly at the squeak baseline for trufflesqueak and that of course does not work with Pharo, but perhaps it would be possible to modify it a little bit so it can detect Pharo and do something else instead. |
|
Interesting, maybe the As mentioned in #170 (comment), Pharo isn't as easy to support as Cuis and probably requires significantly more work than needed to fix the |
|
Yes, that could be the case. Hmm, yes. You know that much more than I do of course, but it seems to be the need for filesystem access in the editor and should be not too difficult if so. The reason Pharo is interesting is that is has its very own web client, as part of the Zinc Web server/client package by Sven von Caekenberg, so it would have provided an excellent triangulation vs. the Squeak and Cuis WebClient. But OK, I'll start comparing the open smalltalk vm SSL plugin and see what I can find. Will probably be some time. |
|
So I had some time to look into this myself. I logged a couple of things inside the image and found out that the connect primitive never returns With f44be51, |
|
Thank you! This is awesome. Is there anything else, maybe a bit higher-level that I could help out with? |
|
Loading the TruffleSqueak image into Cuis is still on the todo list for Cuis support. I'm not sure that the tools work (e.g., polyglot workspace), but the core package should. I also noticed some input issues (e.g., in Cuis or #160). That should be relatively easy to fix, you can find mouse and keyboard classes in here. Otherwise, feel free to build a tool or polyglot application in TruffleSqueak. I'm always curious to see what others build with it :) |
|
Alright, I'll do some exploring loading the TruffleSquek core stuff in Cuis and report back in that ticket. But actually building on top of TruffleSqueak is what I came for :) I have a quite promising idea for cloud deployment that should be applicable to any cloud provider, but where I'm constantly hampered by lack of library support (gRPC, for example), so my immediate goal is to see if I can leverage node.js to this end. The interesting part then is how to make sure that Baselines work with/maps automagically to (for example) npm, so that polyglot dependencies can be handled in a (hopefully) generic way. |
|
Dependency management for polyglot applications is definitely an interesting topic! Please note though that Node.js is usually not exposed as a polyglot language directly. You can call out to Smalltalk from within GraalVM's Node.js runtime, but I don't think you can use the programming system then. You may want to have a look at NodeJVM although it's a bit dated. |
|
OK, I'm not sure I understand. Wouldn't I be able to install node.js as a graalvm language, so it would be available symmetrically to smalltalk and (let's say) python? |
Yes, that's correct. The problem is that Node.js is a framework built on top of JavaScript. While you can interop with JavaScript, GraalVM doesn't allow you to interop with Node.js, e.g., using npm packages. I'm still hoping this will be supported at some point, but it's a bit tricky because apparently, Node.js has its own main loop. |
|
Cuis is quite very much different when it comes to code loading. I think I withdraw my offer for now :) Maybe better to focus on learning how to do polyglot stuff in a more narrow context first. When reading the graal node.js documentation I had hoped that it would be possible to use node.js full-out. And I assume that it is possible if you start out with node. Graal can run node.js and npm t.c. but when polyglot calling out to node.js this does not hold, have I understood it properly? In that case, the node.js polyglot support seems currently to be be not of very much use, since just evaluating JS will not go very far. Hmmm, but wouldn't the grallvm node.js module be that thread? The use-case I'm thinking of hear is that of calling out from languag a (Smalltalk, of course) to language b (js/node.js) and loading and then executin code in a npm package. Hmmmmm.. perhaps the problem resides in node.js wonky package loading, which was staunchly blocking for many years and which need to make use of its own thread. In recent years, the common js way of loading have started to be supported, so perhaps if one loaded packages in the right way it might work. I think I'll start to ask adjacent questions like this in the slack channel instead so as not pollute further this issue :) |
I've donlnoaded and started the latest version (TruffleSqueak-22.1.0.image) running under GraalVm for OpenJDk 17 on Linux Mint 20.2.
It starts fine and I can run som eexamples, but as soon as I try to do anything involving the network (like starting a monitcello installer), the image hangs and eventually produces a debugger where it's obvious that the result was nil.
The text was updated successfully, but these errors were encountered: