Language servers and IDEs

After my post last week that I would be implementing the Rust plugin the classic way (that is, manually parsing, deriving type information, figuring out completions, refactorings, etc.), I received quite a few replies about the language server protocol and RLS specifically. I'd like to address RLS in more detail, since some of the replies were rather strongly-worded and sounded a bit patronizing.

The Good: in defense of language servers

Given that I will be criticising the language server protocol for the remainder of this post, I want to note that I'm not against the concept. I quite like it, actually. Language servers work great with lightweight editors: implement the protocol once and the editor becomes reasonably powerful for any language which has a corresponding language server. Visual Studio Code is a great example of an editor that benefits from language servers, which I guess is quite obvious given that the language server protocol was practically designed for VSCode. Still, others like Atom, Sublime, Vim and Emacs would also greatly benefit from the language server protocol. (Now, whether I would personally use it in Vim is a whole different story.)

From what I've seen in the language server protocol docs, that's pretty much what they're pushing the protocol for, at least for the time being. I think the language server protocol needs a lot more additions in order for it to be useful for IDEs. Which brings me to my point.

The Bad

IDEs are not just editors

The thing is, IDEs have to support many things besides just the language itself; debugging and project management come to mind, but for those, the IDE doesn't necessarily need to know much about the language. For other things, such as static analysis, type hierarchy, dependencies, etc., the IDE needs to understand the language. And yes, you could argue that these functions can be implemented by the language servers, but the thing is, they're not just unavailable at the moment for some language servers, the protocol itself does not specify how they should be implemented.

LSP is incomplete

The protocol currently only specifies very basic functionality: autocompletions, quick fixes, semantic highlighting, finding uses, formatting and that just about covers it. This is nothing compared to things a good IDE can do. Specifically when it comes to code generation, I think that's where IDEs truly shine, but at the moment LSP does not specify how to do general code generation.

LSP is inextensible

Sure, individual language servers can extend the language server protocol with additional methods, such as rustDocument/diagnosticsBegin and rustDocument/diagnosticsEnd, but considering the point of the language server protocol is to reduce the amount of work necessary for tool developers, there is little incentive to keep track of every language server's extensions.

And The Ugly

Making an IDE jump over its own data structures is just plain ugly

Most large IDEs are built using a plugin-based architecture, in order to support many languages, and the core usually has some abstract language structures. Declarations and uses; the different kinds of language objects like functions, locals, classes, structures, etc. And just based on those, the IDE can implement a whole lot of functionality like go-to-declaration, rename, semantic highlighting, etc. All the language plugin needs to do is to populate these structures.

The problem with language servers is that they take all of that information away from the IDE. So in the end, it doesn't matter that you don't need to do parse and understand the language since the language server can do it for you, because you end up having to reinvent the wheel anyway by having to reimplement the core functionality to depend on the language server, rather than language support plugins.

How it can be improved

Just bashing on the protocol is not particularly useful for anyone, so I want to suggest a way many of these problems can be avoided: provide code structure information. Instead of trying to rip away the main component of an IDE from it, allow the IDE to work with the language server. There's no point in the IDE asking the language server what to highlight when the cursor is on a variable if the language server just tells the IDE where the declaration and where all the uses of that variable is in that document.

There's no need to remove any existing functionality either and the language server already computes all this information anyway! If it's exposed to the client, the client can decide if it wants/needs to use it or not.

And yet ignoring all of that, RLS is lacking in functionality

I decided to spend 72 hours last week implementing (a significant part of) the language server protocol in KDevelop in order to prove a point: RLS still doesn't even support many of the features that the language server protocol permits.

Completion support for locals works and not much else.

No completions for functions of something. (The ones shown are keyword completions from Kate.)

No completions could be triggered here. RLS returns an empty list.

No contextual completions: we are in a match block for foo, which is an Option type, which is an enum. It is reasonable to expect that we can autocomplete the enum values.

And this is just code completion. RLS returns empty lists for the documentSymbol and documentHighlight methods, hence why only the syntax highlighting from Kate is available. I haven't tried references, but since RLS computes that identically to documentHighlight, I expect it won't work either. The hover method doesn't work either. Even if the reason for those not working is that I missed setting up something (it happened initially with code completions requiring racer which needed an environment variable to be set), having the user set up all of this manually is not at all user-friendly.

Go-to-definition works. I haven't tried the formatting stuff, but that calls rustfmt, which the IDE can do by itself.

No other methods are supported by RLS at the time of writing. So things like searching for a struct or function in the project, getting information about a function signature, quick fixes, refactorings, etc. are not available.

My project proposal was to create a plugin providing good support for Rust in KDevelop. I intend to deliver on that project, but I do not think at its current state RLS provides good support. That may change in the future and I've shown a proof of concept that KDevelop can interact with a language server. Right now, I think I can do better.


...


Also, Rust's type system is inspired by the Hindley-Milner type inference algorithm. :D No way am I missing the opportunity to implement type inference. After all, Google Summer of Code is an opportunity for me to learn more about things I find interesting.

Comments

  1. Even if I wasn't among the people who replied to the previous post, I was a little concerned about how much avoiding the language server would make things more difficult for you. But you have made a good point showing RLS flaws.

    I have also faced situations when a library or tool makes work really easy in the early stages, but at the cost of limitations for advanced stuff, that are really hard of even impossible to work around.

    As a Rust enthusiast, I am glad that you as willing to do the heavy lifting to provide great support for the language in Kdevelop, my favourite IDE.

    You might already be aware of it, but there was an earlier attempt of unofficial Rust support for Kdevelop, not based on the language server. Maybe it could serve as a good reference: https://github.com/michalsrb/kdev-rust

    Thanks for your work and good luck !

    ReplyDelete
  2. Hi! I am the author of the IntelliJ plugin for Rust (which also does it's own thing), and I think you are right! I've left some more thoughts at Reddit: https://www.reddit.com/r/rust/comments/6fs5q9/language_servers_and_ides/dinhtiz/

    ReplyDelete
  3. Hi, Emma! Nice blog post. I can't convince you to work for RLS but here are my thoughts. A protocol defines almost nothing. They are created only so that systems can interoperate. LSP is no different. Take TCP, it does not define congestion control. One can make a very bad implementation of TCP but that is not a reason to blame TCP. What ever is not defined in the protocol it is upto the implementor to invent things. That said I'm not saying that LSP is totally right. One advantage of using LSP is that it is universal, that is you can use it with other text editors and IDEs. When you have a large number of people who can use it, then you can have more bug reports and many potential contributors (being optimistic). Think about the future of the project. Who will maintain it? When you draw a venn diagram of people using Rust and KDevelop, it is not very big. Compare that to all text editors and IDEs and Rust users, the number are going to be comparatively larger. These are just my thoughts. You must be super busy with implementation now but all the best with your GSoC.

    ReplyDelete
    Replies
    1. Hi, Salad111! Thanks for your comment. However, it is not entirely correct. Protocols do define the interaction between two parties, which is what I was referring to in the blog post. To use your example, TCP does not define how I can send an encrypted message, for instance. Hence why encryption tends to happen on layer 7 services. Similarly, LSP does not define how certain interactions should happen, which I've described above. With regards to using RLS, as you can see from the post, I did implement a language client in KDevelop and it does work. I am interested in doing something else, and I think I've shown sufficient evidence why it's a better approach, in my opinion, than using RLS, at least for the time being.

      Delete
  4. So, I've read both blog posts and all the comments on them. Also I use KDevelop for C++, Python and PHP, but not for Rust, and I don't plan on doing it in the short/medium term, for a lack of time mostly, so this really doesn't and probably will never affect me.

    Having said that, I just wanted to chim in to say that I don't really get why the "I can't have all, so I start from scratch" stance you are taking here. Why not use the protocol for everything it allows (even if it has its flaws, which you can help solve), and then build everything else you want/need on top of it?. I think this approach would have two main advantages:

    - The most obvious one is that you avoid writing what is already written (along with the language itself) and, most importantly, then having to maintain it or making someone else maintain it after you.
    - You can progressively switch from custom code to the language protocol as soon as new equivalent features are implemented, when you or the next maintainer(s) feel they are ready.

    ReplyDelete
    Replies
    1. I think Emma nicely grasped and explained the core issue with all this in her post. Using this kind of language server, you get information on a very high level of abstraction, for example "give me all the possible comlpletion items at position X". This is great if you have a text editor, say, vim and want to get some completion to show up. But it is not what KDevelop is designed to work like: here you want to have all the detailled information in a kind of database and query it for different purposes. You want a list of all declarations in a document, and you want their ranges, their type, and where they are used. From that, you can then infer all the things: the completion list, but also the semantic highlighting, the tooltips, quickopen, find uses, etc etc.

      A typical language server doesn't give you this information in a usable form. You also cannot build on top of it because it's simply not giving you the stuff you need.

      Delete
  5. Hi Emma, if you're interested in a sort of hybrid approach, you might be interested in Roslyn for C#, and the Omnisharp-roslyn project for providing a language server like out of process experience for VS Code, Sublime, etc.

    Kevin Pilch / @Pilchie

    ReplyDelete
  6. Hi, interesting post, love to read this, let's start from scratch.

    ReplyDelete

Post a Comment

Popular Posts