Here is a simple example defining C identifiers:

examples/regex/re1.flx

  regdef digit = charset "01234356789";
  regdef lower = charset "abcdefghijklmnopqrstuvwxyz"; 
  regdef upper = charset "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; 
  regdef letter = lower | upper;
  regdef underscore = "-";
  
  // C identifier
  regdef cidlead = underscore | letter;
  regdef cidtrail = underscore | letter | digit;
  regdef cid = cidlead cidtrail*;

The regdef binder defines an ordinary var containing a representation of the specified regular expression. But it does a lot more than that: it enables a regular definition grammar based on EBNF.

Regular definitions are vastly superior to regular expressions because they allow the expression to be factored into components; in other words, they provide modularity.

In addition, the use of program language syntax is a killer advantage over the usual regular expression string forms because the distinction between operators and data is not only clear .. the syntax is checked by the parser at compile time.

We'll now define something a bit more complicated:

examples/regex/re1.flx

  // C int
  regdef cint = digit+;
  
  // C decimal fixed point float
  regdef ffixed = digit* "." digit+ | digit+ "." digit*;
  
  // Scifloat
  regdef expon = ("E" | "e") ("+"|"-")? digit+;
  regdef sci = (cint | ffixed) expon;
  
  regdef ffloat = ffixed | sci;

These definitions should be quite clear as written: for fixed point float, a sole dot is not a floating point number, but leading or trailing digists can be left out provcided there is a dot, but not both.

For a escientific float, we need either a plain integer of fixed point float, followed by an exponent, which allows but does not require a sign.

A variable defined by a regdef has type regex. However we are using Google RE2 as our regular expression engine, and that requires a compiled regular expression or type RE2 to work. The RE2 compiler compiles the usual kinds of strings, so we need first to convert a regex to a string.

So here is our first test case:

examples/regex/re1.flx

  var float_s : string = ffloat.Regdef::render; // convert regex to string
  var float_r : RE2 = float_s.RE2; // compile string to RE2 regex
  println$ "123.4e-7" in float_r;  // if it is a float? 
  println$ float_r;

Now we printed out the actually rendered regular expression encoded as a string and here it is, with some newlines added so it fits on our page:

(?:[01234356789])*\x2E(?:[01234356789])+|(?:[01234356789])+
\x2E(?:[01234356789])*|(?:(?:[01234356789])+|(?:[01234356789])*
\x2E(?:[01234356789])+|(?:[01234356789])+\x2E(?:[01234356789])*)
(?:E|e)(?:\x2B|\x2D)?(?:[01234356789])+

The chance of getting this very simple regex correct as a string does not seem very high.

An anecdote: there was once a site that performance tested various languages, and one of the tests was a simple phone number regex. I think everyone copied the regex used by the first contribution, except Felix which used regular definitions. Every single language except Felix got it wrong.

Ofen we want to capture data from a regex. Although it made a mess and got the maths wrong, Perl had the ability to perform captures. In light of some mathematical analysis new techniques were developed, but in the end Google RE used a simple capture engine and threw out balancing captures to ensure it worked.

Here's s simple example

examples/regex/re1.flx

  regdef assignment = " "* group(cid) " "* "=" " "* (group(cint) | group(ffloat)) " "* ";";
  var a_s = assignment.Regdef::render;
  var a_r = a_s.RE2;
  var result = Match (a_r, "x = 123.45;");
  match result with
    | None => println$ "No match";
    | Some v =>
       print$ "Variable is " + v.1;                // group 1
       if v.2 == "" do                             // if group 2 didn't match
         println$ " Initialiser is float " + v.3;  // group 3 matched
       else
         println$ " Initialiser is int " + v.2;    // group 2 matched
      done
  endmatch;

The Match function returns an option type which is None is there is no match, otherwise Some v if there is. The v is a Felix varray[string] with slot 0 containing the whole matchingh pattern (which is always the whole input) and then one slot for each catpture group. The capture groups are numbered by a left to right depth first recursive descent of the implicit tree structure.

If a group doesn't match anything it is set to the empty string.

If a group is inside a repetition only the last matching substring is retained.

So far, we have shown what is effectively a Domain Specific Language or DSL. Yes, it is nested in Felix and interacts with it, but to truly be a sub language we need to more completely integrate it with Felix code.

The Match function always matches the whole string. To work around this we can define this:

examples/regex/re1.flx

  var anys = ".*";
  regdef anystring = perl(anys);

and stick it at the end. Alternatively we can do this:

examples/regex/re1.flx

  var prefix_assignment = a_s + anys;

The important point here is that we can lift a Felix string which is a traditional Perl regex string into the computation, and the string can be denoted by any Felix expression of type string. You can write any kind of functional rendering of repex string generating code you like.

There is a second way to generate a regex by using a combinator tree or type regex directly. In fact, the Regdef DSSL grammar just provides a convenient syntax for generating these trees. Here is the important part of the definition from the library:

  class Regdef {
    variant regex =
    | Alts of list[regex]
    | Seqs of list[regex]
    | Rpt of regex * int * int
    | Charset of string
    | String of string
    | Group of regex
    | Perl of string
    ;
  ...

Although the grammar provides a convenient way to construct some of these trees, it cannot construct all of them. For example suppose you want to syntax colour a programming language by highlighting keywords.

You would of course like to do this:

  regdef kw = "proc" | "fun" | "variant";

but alas, this grammar only works with a fixed list of known keywords which have to be cited literally as shown. What if you wanted to load the keyword list from a file?

You mean, like this?

examples/regex/re1.flx

  var data = "proc fun variant";
  var lines = split(data," ");
  var kws = unbox$ map (fun (x:string) => Regdef::String x) lines;
  var kws_r = kws.Regdef::Alts;
  println$ kws_r.str;

It is possible to build regex inline like this:

examples/regex/re1.flx

  var cid2 = regexp ( cidlead cidtrail*);

which in this case is precisely equivalent to saying

  regdef cide = cidlead | cidtrail*;

The parser switched to the regex syntax inside the parens. Remember, a regex is just a simple variant that builds a term tree. The grammar that does all this is defined in the library; that is, in user space.