A vision how to combine MPS and GraalVM<\/p>\n
Way too long ago, I prototyped<\/a> a way to use GraalVM<\/a> and Truffle<\/a> inside JetBrains MPS<\/a>. I hope to pick up this work soon. In this article, I describe the grand picture<\/em> of what might be possible with this combination.<\/p>\n Truffle uses Java Annotation Processors<\/a> heavily. Unfortunately, MPS doesn\u2019t support them during its internal Java compilation. The feature request<\/a> doesn\u2019t show any activity.<\/p>\n So, we have to do it ourselves. A little less time ago, I started with an alternative Java Facet<\/a> to include Annotation Processors. I just pushed my work-in-progress state from 2018. As far as I remember, there were no fundamental problems with the approach.<\/p>\n For Truffle, all executed programs stem from a Source<\/tt><\/a>. However, this Source<\/tt> can only provide Byte<\/tt>s or Character<\/tt>s. In our case, we want to provide the input model. The prototype just put the Node id<\/em> of the input model as a String into the Source; later steps resolved the id against MPS API. This approach works and is acceptable; directly passing the input node as object would be much nicer.<\/p>\n We have to provide all additional hints as Annotations to Truffle. They are complex enough, so we want to leverage MPS’ language features to directly represent all Truffle concepts.<\/p>\n This might be a simple one-to-one representation of Java Annotations as MPS Concepts, but I\u2019d guess we can add some more semantics and checks. Such feedback within MPS should simplify the next steps: Annotation Processors (and thus, Truffle) have only limited options to report issues back to us.<\/p>\n We use this MPS language to implement the interpreter for our DSL. This results in a TruffleLanguage<\/tt><\/a> for our DSL.<\/p>\n At the time when I wrote the proof-of-concept, a TruffleLanguage<\/tt> had to be loaded at JVM startup. To my understanding, Truffle overcame this limitation. I haven\u2019t looked into the current possibilities in detail yet.<\/p>\n I can imagine two ways to provide our DSL interpreter to the Truffle runtime:<\/p>\n What are non-colliding interpreters? Assume we have a state machine DSL, an expression DSL, and a test DSL. The expression DSL is used within the state machines; we provide an interpreter for both of them. A lot of Truffle\u2019s magic stems from its AST representation. Thus, we need to translate our input model (a.k.a. DSL instance, a.k.a. program to execute) from MPS nodes into Truffle Node<\/tt>s<\/a>.<\/p>\n Ideally, the Truffle AST would dynamically adopt any changes of the input model\u2009\u2014\u2009like hot code replacement in a debugger, except we don\u2019t want to stop the running program. From Truffle\u2019s point of view this shouldn\u2019t be a problem: It rewrites the AST all the time anyway.<\/p>\n DclareForMPS<\/a> seems a fitting technology. We define mapping rules from MPS node to Truffle Node<\/tt>. Dclare makes sure they are in sync, and input changes are propagated optimally. These rules could either be generic, or be generated from the interpreter definition.<\/p>\n We need to take care that Dclare doesn\u2019t try to adapt the MPS nodes to Truffle\u2019s optimizing AST changes (no back-propagation).<\/p>\n We require special handling for edge cases of MPS \u2192 Truffle change propagation, e.g. the user deletes the currently executed part of the program.<\/p>\n For memory optimization, we might translate only the entry nodes of our input model immediately. Instead of the actual child Truffle Node<\/tt>s, we\u2019d add special nodes that translate the next part of the AST. The MPS interpreter framework<\/a> removes even more boilerplate from writing interpreters than Truffle. The same language concepts should be built again, as abstraction on top of the Truffle Annotation DSL. This would be a new language aspect.<\/p>\n Once we had the Truffle-based interpreter language, we want to use it! Also, we don\u2019t want to rewrite all our nice interpreters.<\/p>\n I think it\u2019s feasible to automatically migrate at least large parts of the existing MPS interpreter framework to the new language. I would expect some manual adjustment, though. That\u2019s the price we had to pay for two orders of magnitude performance improvement.<\/p>\n Using the interpreter should be as easy as possible. Thus, we have to provide the appropriate utilities:<\/p>\n The simple implementation of interpreter support accepts a language, parameters, and a program (a.k.a. input model), and blocks until the interpretation is complete.<\/p>\n This working mode is useful in various situations. However, we might want to run long-running interpretations in the background, and notify a callback once the computation is finished.<\/p>\n Example: An MPS editor uses an interpreter to color a rule red if it is not in accordance with a provided example. This interpretation result is very useful, even if it takes several seconds to calculate. However, we don\u2019t want to block the editor (or even whole MPS) for that long.<\/p>\n Extending the example, we might also want to show an error on such a rule. The typesystem runs asynchronously anyways, so blocking is not an issue. However, we now run the same expensive interpretation twice. The interpreter support should provide configurable caching mechanisms to avoid such waste.<\/p>\n Both asynchronous interpretation and caching benefit from proper language extensions<\/a>.<\/p>\n Truffle needs to know about our DSL\u2019s types, e.g. for resolving overloaded functions or type casting. We already provide this information to the MPS typesystem. I didn\u2019t look into the details yet; I\u2019d expect we could generate at least part of the Truffle input from MPS’ type aspect.<\/p>\n Truffle requires scoping knowledge to store variables in the right stack frame (and possibly other things I don\u2019t understand yet). I\u2019d expect we could use the resolved references in our model as input to Truffle. I\u2019m less optimistic to re-use MPS’ actual scoping system.<\/p>\n For both aspects, we can amend the missing information in the Interpreter Language, similar to the existing one.<\/p>\n As DSL developers, we want to make sure we implemented our interpreter correctly. Thus, we write tests; they are similar to other tests involving the interpreter<\/a>.<\/p>\n However, if they fail, we don\u2019t<\/em> want to debug the program expressed in our DSL<\/a>, but our interpreter. For example, we might implement the interpreter for a switch<\/tt>-like construct, and had forgotten to handle an implicit default<\/tt> case.<\/p>\n Using a regular Java debugger (attached to our running MPS instance) has only limited use, as we had to debug through the highly optimized Truffle code. We cannot use Truffle\u2019s debugging capabilities, as they work on the DSL. Simpler ways to support the developers might be providing traces through the interpreter, or ship test support where the DSL developer can assure specific evaluators were (not) executed.<\/p>\n Truffle provides a framework to describe any runtime in-memory data structure as Shape<\/tt><\/a>, and to convert them between languages. This should be a nice extension of MPS’ multi-language support into the runtime space, supported by an appropriate Meta-DSL (a.k.a. language aspect).<\/p>\n MPS contains the standard interactive debugger inherited from IntelliJ platform.<\/p>\n Truffle exposes a standard interface for interactive debuggers<\/a> of the interpreted input. It takes care of the heavy lifting from Truffle AST to MPS input node.<\/p>\n If we ran Truffle in a different thread than the MPS debugger, we should manage to connect both parts.<\/p>\n Truffle also exposes an instrumentation interface<\/a>. We could provide standard instrumentation applications like “code” coverage (in our case: DSL node coverage) and tracing out-of-the-box.<\/p>\n One might think of nice visualizations:<\/p>\n Other possible applications:<\/p>\n If we could serialize Truffle Node<\/tt>s (before any run-time optimization), we would have an MPS-independent representation of the executable DSL. Depending on the serialization format (implement Serializable<\/a>, custom binary format, JSON, etc.), we could optimize for use-case, size, loading time, or other priorities.<\/p>\n Assume an insurance calculation DSL. With serialized Truffle Node<\/tt>s, we need only one interpreter:<\/p>\n By combining Instrumentation<\/a> and debugger<\/a>, it might be feasible to provide step-back debugging.<\/p>\n In the interpreter, we know the complete global state of the program, and can store deltas (to reduce memory usage). For quite some DSLs, this might be sufficient to store every intermediate state and thus arbitrary debug movement.<\/p>\n By stepping back<\/a> through our execution and following different execution paths, we could explore alternate outcomes. The different execution path might stem from other input values, or hot code replacement<\/a>.<\/p>\n If we had a side step debugger<\/a>, nice support to project interpretation results<\/a>, and a really fast interpreter<\/a>, we could run explorative simulations on lots of different executions paths. This might enable legendary interactive development<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" A vision how to combine MPS and GraalVM<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[35,33,36,32,34],"class_list":["post-458","post","type-post","status-publish","format-standard","hentry","category-mps","tag-dsl","tag-graalvm","tag-interpreter","tag-jetbrains-mps","tag-truffle"],"_links":{"self":[{"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/posts\/458","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/comments?post=458"}],"version-history":[{"count":6,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/posts\/458\/revisions"}],"predecessor-version":[{"id":491,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/posts\/458\/revisions\/491"}],"wp:attachment":[{"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/media?parent=458"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/categories?post=458"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.nikostotz.de\/blog\/wp-json\/wp\/v2\/tags?post=458"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Part I: Get it Working<\/h2>\n
Step 0: Teach Annotation Processors to MPS<\/h3>\n
Optional Step 1: Teach Truffle Structured Sources<\/h3>\n
Step 2: Implement Truffle Annotations as MPS Language<\/h3>\n
Step 3: Start Truffle within MPS<\/h3>\n
\n
\nAll non-colliding<\/em> DSL interpreters would be MpsTruffleLanguage1<\/tt> from Truffle\u2019s point of view. This works, as we know the MPS language for each input model, and can make sure Truffle uses the right evaluation for the node at hand. We might suffer a performance loss, as Truffle had to manage more evaluations.<\/p>\n
\nWe provide two interpreters for the test DSL: One executes the test and checks the assertions, the other one only marks model nodes that are covered by the test.
\nThe state machine interpreter, the expression interpreter, and the first test interpreter are non-colliding, as they never want to execute on the same model node. All of them go to MpsTruffleLanguage1<\/tt>.
\nThe second test interpreter does<\/em> collide, as it wants to do something with a node also covered by the other interpreters. We put it to MpsTruffleLanguage2<\/tt>.<\/li>\nStep 4: Translate Input Model to Truffle Nodes<\/h3>\n
\nUnloading the not required parts might be an issue. Also, on-demand processing seems to conflict with Dclare\u2019s rule-based approach.<\/p>\nPart II: Adapt to MPS<\/h2>\n
Step 5: Re-create Interpreter Language<\/h3>\n
Step 6: Migrate MPS Interpreter Framework<\/h3>\n
Step 7: Provide Plumbing for BaseLanguage, Checking Rules, Editors, and Tests<\/h3>\n
\n
\nWe had to make sure we get language \/ model loading and dependencies right. This should be easier with Truffle than with the current interpreter, as most language dependencies are only required at interpreter build time.<\/li>\n
\nCreating warnings or errors based on the interpreter\u2019s results is a standard use-case, and should be supported by dedicated language constructs.<\/li>\n
\nAs another standard use-case, we might want to show the interpreter\u2019s results (or a derivative) inside an MPS editor. Especially for long-running or asynchronous calculations, getting this right is tricky. Dedicated editor extensions should take care of the details.<\/li>\n
\nYet another standard use-case: our DSL defines both calculation rules and examples. We want to assure they are in sync, meaning executing the rules in our DSL interpreter and comparing the results with the examples. This must work both inside MPS, and in a headless build \/ CI test environment.<\/li>\n<\/ul>\nStep 8: Support Asynchronous Interpretation and\/or Caching<\/h3>\n
Step 9: Integrate with MPS Typesystem and Scoping<\/h3>\n
Step 10: Support Interpreter Development<\/h3>\n
\nThere might be ways to attach a regular Java debugger running inside MPS in a different thread to its own JVM. Combining the direct debugger access with our knowledge of the interpreter\u2019s structure, we might be able to provide sensible stepping through the interpreter to the DSL developer.<\/p>\nStep 11: Create Language for Interop<\/h3>\n
Part III: Leverage Programming Language Tooling<\/h2>\n
Step 12: Connect Truffle to MPS’ Debugger<\/h3>\n
Step 13: Integrate Instrumentation<\/h3>\n
\n
\n
Part IV: Beyond MPS<\/h2>\n
Step 14: Serialize Truffle Nodes<\/h3>\n
Step 15: Execute DSL stand-alone without Generator<\/h3>\n
\nUsually, we would implement<\/p>\n\n
\n
Part V: Crazy Ideas<\/h2>\n
Step 16: Step Back Debugger<\/h3>\n
Step 17: Side Step Debugger<\/h3>\n
Step 18: Explorative Simulations<\/h3>\n