Static Lexer

This website is made using a custom static site generator written in Go. It generates the HTML files from a different file structure with a small custom lexer. The lexer is very simple and is used to include other files and to add some basic things such as metadata and content slots.

The syntax of this lexer is as followed:

• + is used to add metadata to the current file.

  The metadata can either be a local variable or a global variable by prefixing it with a dot. The global metadata can be accessed by files loaded in after the meta statement. This means global metadata can be used to pass arguments to other files such as this:

      
  +url = /;
  {code:
      +.lang=go;`package main
  
          func main() {
              println("hello world!")
          }`
  }
      
  

  The metadata can also be used to retrieve metadata. It can retrieve data from other files or from the global/local metadata. For global/local metadata, local metadata is always higher priority. Metadata can be retrieved as followed:

      
  +local_data;
  +global_data;
  +$/data.htm:file_data;
      
  

• {} is used to include other files.

  The file included can be referenced either by a relative path or a path from root by prefixing it with a dollar sign. When including a file, the lexer will either include the file as is or it will insert content into the file if any content has been given. Content can be specified by using a colon right after the path. Files can be included as followed:

      
  {$/components/without_content.htm}
  {$/components/with_content.htm:
      
  
          hello!
      
  }
  {./relative_path.htm}
      
  

• Strings can be made using ", ', or `

  Everything inside strings will be taken literal, except for the escaped characters. Though the lexer will not parse escaped characters as usual languages like \n for example. In order to make the lexer as simplistic as possible, escaped characters will just mean that the lexer adds the character to the buffer and continues without processing the token.

  One issue ive fixed with a workaround is indenting within strings. Code blocks for example are hard since you cant indent them within the code since those indentations will show up in the html view. To fix this, i made it so that the lexer will allow you to place a string indentation token right after theing string token. This token will be used to determine the indentation of the string. String indentations can be made as followed:

      
  {$component.htm:`this is where the indentation starts.
            marked by the horizontal line above right after the string opening on the same line.`
  }
      
  

Update

I've been working on updating the lexer, i've made an entirely new version since the previous one was very messy and hard to work with. The new lexer, "lexern2" is much more simple and easy to work with, but at the same time also more complex. The lexer is now able to parse more complex things such as statements and expressions. Before the lexer was only able to do static things and could not import data from files that werent loaded yet. The new lexer's process looks as followed:

1. A new lexer will be made.
2. The lexer will be given every file found.
3. The files loaded in will parse through them and save the statements as different nodes.
4. Once all files are parsed it will call the String method which will go through all the nodes saved per page and generate the html.
5. The generated html will be saved in the corresponding location of the url metadata.

My next 2 goals is to make the lexer be able to call small functions in order to find all the pages using a certain metadata and loop through it. Mainly to make the navigation bar easier to update. The second goal is to make the lexer logging more useful to find issues, i want to make it so that it logs the line and column of the error. So far syntactically the lexer hasnt changed much, the biggest change is that metadata uses the # token instead of + to save metadata. + is only used to insert metadata to the current file.

Some examples

    
#url=/projects/lexer;
#title=Lexer;
#tag=project;

{template:
    
Static Lexer
    
...
}
    

    
#alias = code;

    
        +content;
    

    

    
{code:
    #.lang=html;`this is a code block`
}
    

Github repository: [link]