A confusion of grammars: Need help integrating one grammar within another


#1

Hello,

I’m a mathematics instructor at a large two year college in the San Francisco Bay Area. I use an online homework system called “WebWork” for some of my classes. Within this system, I can author my own mathematics problems for my classes. This authoring system is based or PERL but allows the incorporation of LaTeX for displaying mathematics. The code for a problem is placed in a *.pg file.

I want to create a pg-grammar based on the PERL grammar but anything within the pg-file enclosed within delimiters:

[and] for inline display of mathematics, or
[and] for displayed mathematics on its own line

would be handled by the LaTeX grammar.

I created the pg.cson file below in ~/.atom/packages/language-pg/grammars

‘scopeName’: ‘source.perl6.pg
‘name’: ‘PG’
‘fileTypes’: [
‘pg’
]
patterns: [
{‘include’: ‘source.perl6’}
{
‘begin’: ‘[' 'end': ']’
‘contentName’: ‘source.embedded.text.tex.latex’
‘patterns’: [
{
‘include’: ‘source.text.tex.latex’
}
]
}
]

This does give PERL syntax highlighting to perl code in the .pg file but doesn’t do syntax highlighting for LaTeX code within delimiters [and]

Is there some way to modify this pg.cson file to get the desired behavior?

Thanks much for any help you can offer.


Importing Grammar javascript file
#2

When posting code on here, you should fence it with backticks to mark it as code, like so:

```
Formatting and
               indentations will be preserved.
This also lets you use ` and <tags> without the forum trying to parse them.
```

For more information, you can look up GitHub-flavored markdown.


#3

@DamnedScholar

Thanks for the tip. Is there a way I can edit my original post to fix it as you suggest?


#4

You’re such a new user that I don’t think you have permission to edit anything yet. Just repost your code.


#5

Sorry for the poorly formatted code in my pg.cson file. Here it is more properly formatted:

'scopeName': 'source.perl6.pg'
'name': 'PG'
'fileTypes': [
  'pg'
]
patterns: [
  {'include': 'source.perl6'}
  {
    'begin': '\\[`'
    'end': '`\\]'
    'contentName': 'source.embedded.text.tex.latex'
    'patterns': [
      {
        'include': 'source.text.tex.latex'
      }
    ]
  }
]

I also had trouble with the regex’s for the delimiters:

[` and `]

as the regex \[` for the delimiter [` and the regex `\] for the delimiter `] results
in "Uncaught Error: premature end of char-class". Double escaping the brackets
eliminates that error but I'm not sure it is still matching the delimiters.

Below is a sample *.pg file I want the new grammar to act upon:

## DESCRIPTION
## A sample PG file.
## ENDDESCRIPTION

## Tagged by XW

## DBsubject(Calculus)
## DBchapter(3)
## DBsection(2)
## Institution(CSM)
## MLT()
## Level(4)
## TitleText1('Calculus')
## AuthorText1('Stewart')
## EditionText1('6')

DOCUMENT();

loadMacros(
   "PGstandard.pl",     # Standard macros for PG language
   "MathObjects.pl",
   "PGML.pl",
   "PGcourse.pl",      # Customization file for the course

   #"source.pl",        # allows code to be displayed on certain sites.
);

Context("Numeric");
$showPartialCorrectAnswers = 1;

# Define variables and math objects section
# e.g:
$a = random(2,4,1);
$f = Formula("cos^2(x)+sin^2(x)");

TEXT(beginproblem());

BEGIN_PGML
# Problem description section.
# *   Inline math placed between [` `]
# *   Display math placed between [`` ``]
# *   Display the value of declared variables by placing them between [ ]
# e.g:
In this problem we will calculate the value of [` \int_{[$a]}^3 [$f]\,dx `].

[` \int_{[$a]}^3 [$f]\,dx `] is equal to [_______]{Real(3)}

END_PGML

# BEGIN_PGML_SOLUTION
# END_PGML_SOLUTION

ENDDOCUMENT();

#6

When working with grammars in Atom, it is necessary to double-escape everything. This is because there are two levels of processing before it reaches the tokenization engine - one from CSON/JSON to JavaScript, and another from JavaScript to regex. So what would normally be /\[`/ is instead /\\[`/.

I believe your main problem is that you are trying to include the source.text.tex.latex grammar, when in reality the grammar is text.tex.latex.


#7

Thanks for the reply.

You are correct; changing “source.text.tex.latex” to “text.tex.latex” did fix that problem. Would you comment on the rest of the code in my pg.cson file? I’m wondering about the contentName field. Can I name it anything and does it do anything?

Another question, if I may. These *.pg files allow for markdown syntax between “BEGIN_PGML” and “END_PGML” or “BEGIN_PGML_SOLUTION” and “END_PGML_SOLUTION” tags (see sample “temp.pg” file below) and within these tags you can have latex syntax for text contained between the bracket-backquote tags.

How would I modify my pg.cson file to allow for this behavior?

Below is the content of pg.cson and temp.pg files again, for your reference:

pg.cson

'scopeName': 'source.perl6.pg'
'name': 'PG'
'fileTypes': [
  'pg'
]
patterns: [
  {'include': 'source.perl6'}
  {
    'begin': '\\[`'
    'end': '`\\]'
    'contentName': 'source.embedded.text.tex.latex'
    'patterns': [
      {
        'include': 'text.tex.latex'
      }
    ]
  }
]

temp.pg

## DESCRIPTION
## A sample PG file.
## ENDDESCRIPTION

## Tagged by XW

## DBsubject(Calculus)
## DBchapter(3)
## DBsection(2)
## Institution(CSM)
## MLT()
## Level(4)
## TitleText1('Calculus')
## AuthorText1('Stewart')
## EditionText1('6')

DOCUMENT(); # PERL6 highlighting

loadMacros(
   "PGstandard.pl",     # Standard macros for PG language
   "MathObjects.pl",
   "PGML.pl",
   "PGcourse.pl",      # Customization file for the course

   #"source.pl",        # allows code to be displayed on certain sites.
);

Context("Numeric");
$showPartialCorrectAnswers = 1;

# Define variables and math objects section
# e.g:
$a = random(2,4,1);
$f = Formula("cos^2(x)+sin^2(x)");

TEXT(beginproblem());

# This block should use the "Github Markdown" grammar except for text
# between [` `] or [`` ``] tags; This should use LaTeX grammar.
BEGIN_PGML
# Problem description section.
# *   Inline math placed between [` `]
# *   Display math placed between [`` ``]
# *   Display the value of declared variables by placing them between [ ]
# e.g:
In this problem we will calculate the value of [` \int_{[$a]}^3 [$f]\,dx `].

[` \int_{[$a]}^3 [$f]\,dx `] is equal to [_______]{Real(3)}
END_PGML

# This block should use the "Github Markdown" grammar except for text
# between [` `] or [`` ``] tags; This should use LaTeX grammar.
BEGIN_PGML_SOLUTION
Since [`[$f] = 1`], it follows that
[` \int_{[$a]}^3 [$f]\,dx = \int_{[$a]}^3 1\;dx = 3`]
END_PGML_SOLUTION

ENDDOCUMENT();

#8

Let’s say your pattern is:

{
  'begin': 'BEGIN'
  'end': 'END'
  'name': 'name'
  'contentName': 'contentName'
}

as a very simple example. Then:

    BEGIN stuff more stuff END
#   ^^^^^^^^^^^^^^^^^^^^^^^^^^ name
#        ^^^^^^^^^^^^^^^^^^ contentName

contentName does not include the begin/end patterns, while name does.

Adding Markdown support is similar to how you added LaTeX support. The scope name is source.gfm.

Your latex-in-markdown question is more complicated and hence requires an advanced grammar technique called “injections”. Injections allow you to inject grammar patterns into other grammars that you do not have direct control of. I imagine that you would need something like the following:

# This should be on the same level as `patterns`
'injections':
  'source.gfm': # If this doesn't work, try L:source.gfm instead
    'patterns': [
      {
        'begin': '\\[(``?)'
        'end': '\\1\\]'
        'contentName': 'source.embedded.text.tex.latex'
        'patterns': [
          {
            'include': 'text.tex.latex'
          }
        ]
      }

You may want to consider adding scopes to your begin and end captures as well (using beginCaptures and endCaptures, respectively).


#9

Thanks again for all your help; You are very kind :slight_smile:

I made the changes you suggested (I think…) but the grammar won’t load. Below is what I have so far:

'scopeName': 'source.perl6.pg'
'name': 'PG'
'fileTypes': [
  'pg'
]
patterns: [
  {'include': 'source.perl6'}
  {
    'begin': '^BEGIN_PGML$'
    'end': '^END_PGML$'
    'name': 'markup.code.gfm'
    'contentName': 'source.embedded.gfm'
    'patterns': [
      {
        'include': 'source.gfm'
      }
      # This should be on the same level as `patterns`
    'injections':
      'source.gfm': # If this doesn't work, try L:source.gfm instead
        'patterns': [
          {
            'begin': '\\[(``?)'
            'end': '\\1\\]'
            'contentName': 'source.embedded.text.tex.latex'
            'patterns': [
              {
                'include': 'text.tex.latex'
              }
            ]
          }
        ]
  }
]


#10

When I said “on the same level as patterns”, I meant that its indentation level should be next to patterns. Sorry if that was confusing.

'scopeName': 'source.perl6.pg'
'name': 'PG'
'fileTypes': [
  'pg'
]
'injections':
  'source.gfm': # If this doesn't work, try L:source.gfm instead
    'patterns': [
      {
        'begin': '\\[(``?)'
        'end': '\\1\\]'
        'contentName': 'source.embedded.text.tex.latex'
        'patterns': [
          {
            'include': 'text.tex.latex'
          }
        ]
      }
    ]
patterns: [
  {'include': 'source.perl6'}
  {
    'begin': '^BEGIN_PGML$'
    'end': '^END_PGML$'
    'name': 'markup.code.gfm'
    'contentName': 'source.embedded.gfm'
    'patterns': [
      {
        'include': 'source.gfm'
      }
    ]
  }
]

#11

Ah, I see…

I tried this modified code and I get PERL6 highlighting throughout the whole pg file; No markup highlighting between the BEGIN_PGML and END_PGML tags and no LaTeX highlighting between its tags. I also tried changing ‘source.gfm’ to ‘L:source.gfm’ but no change then either. Very odd.