Crate syntastica

source ·
Expand description

syntastica

Modern and easy syntax highlighting using tree-sitter

Note

If viewing this file on GitHub or crates.io, some links might not be working. Go to the custom docs page or the docs.rs page instead, which additionally include the Features section.

Overview

To use syntastica, you probably want to depend on three crates:

  1. The main syntastica crate for all the logic.
  2. A parser collection to provide language support (see parser collections)
  3. The theme collection for some default themes (see theme collection)

So for example:

syntastica = "<version>"
syntastica-parsers = { version = "<version>", features = ["some"] }
syntastica-themes = "<version>"

Use cases

syntastica has three main ways of highlighting code, for three different use cases:

  1. Highlight one input exactly once: see highlight and this example
  2. Highlight one input multiple times (e.g. with different themes or renderers): see Processor::process_once, render, and this example
  3. Highlight multiple different inputs any number of times: see Processor, render, and this example

Subprojects

Besides the main syntastica crate, many other crates for different purposes were developed and are included in the repository. This section aims to provide a good overview.

Parser collections

The main syntastica crate provides no tree-sitter parsers and queries by itself. However, the project does provide three different parser collections with different advantages and drawbacks each. All three collections depend on syntastica-queries for the tree-sitter queries. Choose one, and add it as a dependency next to syntastica itself.

All three parser collections also provide the same public API and provide features for all supported languages, as well as the three feature groups some, most, and all. Take a look at the respective crate documentation for more information.

If you want to additionally use languages that are not in any of these parser collections, one approach is shown in the custom languages example.

  • syntastica-parsers is probably the easiest to start with. It uses parsers from crates.io. This has the main benefit of being well integrated in the cargo ecosystem. However, many tree-sitter parsers do not get published to crates.io, and those that are, are usually very outdated. Thus, this collection is relatively limited.
  • syntastica-parsers-git is probably the best choice overall. It contains all supported languages, and when WebAssembly compilation will be supported, this will be the collection to use. It pulls pinned revisions of parser git repositories in the build script and links to the C and C++ parser sources. As such, it does not depend on the upstream parsers to have up-to-date Rust bindings. However, this way of fetching the parsers requires the git command to be accessible and internet access during compilation, which may not be desirable. Additionally, compilation can take very long, because there is no clean way to cache the fetched repositories between builds.
  • syntastica-parsers-gitdep is a mix of both of the above. It uses cargo git dependencies to fetch the parser repositories and depends on a remote Rust binding (which is why not all parsers are included). The main disadvantages are that this collection cannot be published to crates.io, because it depends on crates that are not on crates.io (namely the parsers). This means, to use it you must also depend on it using a git dependency, which in turn forbids your crate to be published on crates.io. Unlike syntastica-parsers-git however, the parsers only need to be fetched once by cargo, and following builds will be much faster.

Theme collection

To render highlighted code to end users, a theme is needed, which specifies the colors to use for which theme key. The syntastica project comes with a separate crate containing a few default themes: syntastica-themes.

If you wish to create your own theme, have a look at the custom theme example and the documentation for the theme! macro.

Crates for internal use

The syntastica repository/workspace also includes some crates which are not meant for outside use, but are instead used internally. These are listed below.

Note: There are no guarantees about the public API of these crates! If, for any reason, you have to depend on one of them, then pin the exact version using <crate> = "=<version>".

  • syntastica-core defines types, traits, constants, etc. which are used in multiple of the other crates. The main syntastica crate re-exports all those items transparently, so that external projects only need a dependency on that. The items are defined in syntastica-core however, to avoid cyclic (dev-)dependencies inside this workspace.
  • syntastica-macros defines procedural macros for use exclusively inside this workspace. This crate allows the list of languages/parsers to be in one combined languages.toml file, and the different macros are used in the different places where this list needs to be referenced.
  • syntastica-highlight is a fork of tree-sitter-highlight, which is adjusted and trimmed down for the use in syntastica. It contains the main highlighting logic.
  • syntastica-queries is a collection of tree-sitter queries for all supported languages. It is marked as “for internal use”, because all three parser collections depend on this crate and expose the queries through their implementation of LanguageSet. Unlike the previous crates in this list however, you may actually want to depend on this crate yourself, if you only need the queries.

General side-products

This list includes crates which were developed for syntastica but have no direct association with the main project and can be used completely separately.

  • rsexpr is a generic S-expression parser with added support for square-brackets, strings, and comments. Additionally, the parsed S-expressions can be pretty-printed to provide a uniform formatting. See dprint-plugin-sexpr for more information on using this as a formatter. In syntastica this crate is used for parsing (and formatting) the tree-sitter queries in the queries directory. These are processed by cargo xtask codegen queries and result in the queries inside the generated_queries directory, which are the ones that are bundled with syntastica-queries.
  • lua-pattern is a parser for Lua patterns. These are similar to regular expressions, but generally more limited. The crate also provides a best-effort conversion to regular expression strings. In syntastica this is used, as many of the source queries are forked from nvim-treesitter which makes heavy use of #lua-match? predicates for matching with Lua patterns. The official tree-sitter Rust bindings do not support Lua pattern matching however (obviously), which is why during the processing of the queries (with cargo xtask codegen queries), all Lua patterns are replaced with regular expressions using this crate.
  • syntastica-query-preprocessor is a pre-processor for tree-sitter queries which allows usage of ; inherits <lang> comments, conditional skipping of nodes with comments, usage of additional predicates like lua-match?, contains? and any-of?, Neovim’s old injections syntax, and order reversing for priority flipping. The crate can be used to use queries designed for Neovim with the official tree-sitter Rust bindings with minimal manual changes. Despite having syntastica in the name, the crate can be used externally and does not depend on any of the other syntastica- crates. In syntastica it is used in the codegen queries xtask, because many of the queries are forked from nvim-treesitter, and to adjust the queries for older parser versions from crates.io.

WebAssembly support

syntastica can be used with WebAssembly, although the current support is a bit lacking. There are currently two primary ways to use syntastica in a WebAssembly context.

1. Using the tree-sitter-c2rust runtime

In order to make syntastica compile to wasm32-unknown-unknown targets, feature flags can be used to use the c2rust transpilation of tree-sitter instead of the official C implementation. This is only supported by the syntastica-parsers-git parser collection, and only parsers that don’t use an external C++ scanner are available.

To use this approach, simply set default-features = false and enable the runtime-c2rust feature for all syntastica dependencies. An example using this approach for use of syntastica in a Dioxus project can be found here.

2. Using Emscripten / the syntastica-js package

syntastica can also be compiled to wasm32-unknown-emscripten which has much better support for C and C++ interop. But annoyingly, basically the entire Rust Wasm ecosystem is built around the wasm32-unknown-unknown target (e.g., wasm-pack and wasm-bindgen can only be used with wasm32-unknown-unknown), which makes it very cumbersome to use Emscripten for Rust. In the attempt to make using syntastica on the web a bit easier, the syntastica-js crate and accompanying syntastica NPM package provide a JavaScript/TypeScript wrapper around an Emscripten build of syntastica.

There are three examples using syntastica-js:

Note

The syntastica NPM package is currently not being updated and uses an old version of syntastica, because the current implementation always includes all parsers in one big binary, which would be over 60 MB big with all currently supported parsers. The eventual plan is to find a way to split the package into multiple binaries that can be fetched from a server on-demand, and to provide multiple NPM packages for manual selection of the parsers.

Examples

This section contains some basic usage examples. More specific examples can be found in the documentation of some items such as the Processor type or the render function. Additionally, the examples directory contains a few complete examples.

This is the list of examples found here:

Example: highlight once

This example shows the easiest and quickest way to use syntastica. See the section about use cases for when it is appropriate to use syntastica this way.

use syntastica::renderer::TerminalRenderer;
use syntastica_parsers::{Lang, LanguageSetImpl};

let output = syntastica::highlight(
    // the code to highlight
    r#"fn main() { println!("42"); }"#,
    // the input's language
    Lang::Rust,
    // use `syntastica-parsers` language set
    &LanguageSetImpl::new(),
    // use the TerminalRenderer with no background color
    &mut TerminalRenderer::new(None),
    // use the gruvbox dark theme from `syntastica-themes`
    syntastica_themes::gruvbox::dark(),
)
.unwrap_or_else(|err| panic!("highlighting failed: {err}"));

println!("{output}");

Example: highlight the same input multiple times

This example shows how to render the same input with two different themes using two different renderers.

use syntastica::{Processor, style::Color, renderer::*};
use syntastica_parsers::{Lang, LanguageSetImpl};

// process the input once, but store the raw highlight information
let highlights = Processor::process_once(
    // the code to highlight
    r#"fn main() { println!("42"); }"#,
    // the input's language
    Lang::Rust,
    // use `syntastica-parsers` language set
    &LanguageSetImpl::new(),
)
.unwrap_or_else(|err| panic!("highlighting failed: {err}"));

// render the highlights to the terminal using the
// gruvbox dark theme on a dark gray background
println!("{}", syntastica::render(
    &highlights,
    &mut TerminalRenderer::new(Some(Color::new(40, 40, 40))),
    syntastica_themes::gruvbox::dark(),
));

// render the same input to HTML using the onelight theme
let html = syntastica::render(
    &highlights,
    &mut HtmlRenderer::new(),
    syntastica_themes::one::light(),
);
// you could for example write that to a file called `index.html`:
// std::fs::write("index.html", html).unwrap();

Example: highlight multiple different inputs

This example shows how a Processor can be reused if multiple different inputs should be highlighted.

use syntastica::{Processor, style::Color, renderer::*};
use syntastica_parsers::{Lang, LanguageSetImpl};

// create a language set and a `Processor`
let language_set = LanguageSetImpl::new();
let mut processor = Processor::new(&language_set);
// Note: `language_set` has to be stored in a variable, because the processor
// is bound to the lifetime of the reference passed to `new`

// process some input
let highlights_rust = processor.process(
    // the code to highlight
    r#"fn main() { println!("42"); }"#,
    // the input's language
    Lang::Rust,
)
.unwrap_or_else(|err| panic!("highlighting failed: {err}"));

// process some other input in another language
let highlights_js = processor.process(r"console.log('42')", Lang::Javascript)
    .unwrap_or_else(|err| panic!("highlighting failed: {err}"));

// render the rust code to the terminal using the
// gruvbox dark theme on a dark gray background
println!("{}", syntastica::render(
    &highlights_rust,
    &mut TerminalRenderer::new(Some(Color::new(40, 40, 40))),
    syntastica_themes::gruvbox::dark(),
));

// render the same rust code to HTML using the onelight theme
let html = syntastica::render(
    &highlights_rust,
    &mut HtmlRenderer::new(),
    syntastica_themes::one::light(),
);
// you could for example write that to a file called `index.html`:
// std::fs::write("index.html", html).unwrap();

// now render the javascript code to the terminal using the
// onedark theme and no background color
println!("{}", syntastica::render(
    &highlights_js,
    &mut TerminalRenderer::new(None),
    syntastica_themes::one::dark(),
));

Example: detect language from file type

This is an alteration of the first example showing how to detect the language to use based on a file type. See that first example for explanations of the rest of the code.

syntastica uses tft for file types which provides automatic detection.

use syntastica::{renderer::TerminalRenderer, language_set::{LanguageSet, SupportedLanguage}};
use syntastica_parsers::{Lang, LanguageSetImpl};

// detect the file type given a file's path and content.
// this requires a dependency on `tft`
let ft = tft::detect("main.rs", "");

let language_set = LanguageSetImpl::new();
let output = syntastica::highlight(
    r#"fn main() { println!("42"); }"#,
    // the `SupportedLanguage` trait provides a `for_file_type` function
    // which returns an `Option<Lang>`
    // make sure to have the trait in scope
    Lang::for_file_type(ft).unwrap(),
    &language_set,
    &mut TerminalRenderer::new(None),
    syntastica_themes::gruvbox::dark(),
)
.unwrap_or_else(|err| panic!("highlighting failed: {err}"));

println!("{output}");

Example: custom theme

This is an alteration of the first example showing how to create a simple custom theme. See that first example for explanations of the rest of the code, and see the documentation of the theme! macro for more information.

use syntastica::{renderer::TerminalRenderer, theme};
use syntastica_parsers::{Lang, LanguageSetImpl};

let theme = theme! {
    // specify colors using hex literals
    "purple": "#c678dd",
    "blue": "#61afef",
    "green": "#98c379",

    // link to other keys using a `$` sign
    "keyword": "$purple",
    "function": "$blue",

    // specify more styling options in curly braces
    // (note that currently this order is required by the macro)
    "string": {
        color: None,
        bg: None,
        underline: false,
        strikethrough: false,
        italic: true,
        bold: false,
        link: "green",
    },
};

let output = syntastica::highlight(
    r#"fn main() { println!("42"); }"#,
    Lang::Rust,
    &LanguageSetImpl::new(),
    &mut TerminalRenderer::new(None),
    theme,
)
.unwrap_or_else(|err| panic!("highlighting failed: {err}"));

println!("{output}");

Versioning

All crates in this workspace whose names start with syntastica share the same version. The typical semantic versioning rules are used across the public APIs of all of these, except for the ones listed as internal. The other crates in this workspace have their own separate versions.

Versions are specified as MAJOR.MINOR.PATCH. As long as the MAJOR version specifier is still at 0, changes to the MINOR version may also be breaking changes. The PATCH part is only incremented if the public API stays exactly the same.

Inspiration

The entire idea of this project started out as a way to use tree-sitter code highlighting in a LaTeX project. While working with @MikMuellerDev on our paper on rush I created a CLI app called lirstings. The initial sketch simply called out to the tree-sitter-cli and converted the output HTML to LaTeX code. However, not long after that I already implemented some of the logic myself and made a first public commit. This version of lirstings (called ts2tex at the time) already laid out some groundwork like query pre-processing and theming that is still present in syntastica today. Towards the end of our project we wanted to use the same highlighting on our rush playground, which would require lirstings to become more general and support WebAssembly. Work on that started in the generalize branch just enough to suffice for our needs at the time.

After the entire rush project was done and after taking a break for a while, I started syntastica with the intent to be a library from the ground up, and a possible replacement for syntect. The main difference from lirstings at the start was the parser collection(s), providing a rigid set of parsers and queries for users. Over time syntastica then grew to the big project it is today.

Features

  • runtime-c (enabled by default) — Use the standard tree-sitter C runtime. See WebAssembly support for more information.

  • runtime-c2rust — Use the pure Rust tree-sitter runtime. See WebAssembly support for more information.

  • serde — Provide implementations of serde’s Serialize and Deserialize traits for Theme and Style

  • docs — Enable this when building the docs

Re-exports

Modules

Macros

  • theme
    Convenience macro for constructing new Themes.

Structs

  • Processor
    A type for easy reuse of resources when highlighting multiple inputs.

Enums

  • Error
    The main error type.

Functions

Type Aliases