Variation selector for roundhand capitals in NewCM

[antonis-tsolomitis]

Currently roundhand capitals (eg u1D49C.alt in NewCM) are selected with an ssXX feature. However Unicode has another mechanism for this using Variation Selector U+FE01 following the script capital letter. That is, U+1D49C followed by U+FE01 gives the roundhand (u1D49C.alt in NewCM).

This is a more clean way to select roundhand and from the version 7.0.4 and on, NewCM supports this for all of its Math fonts. Currently only NewCM and NotoSansMath support this method. If unicode-math considers this way, NewCM has already support for this.

[davidcarlisle]

Your thoughts would also be welcome at w3c/mathml#536

[dflipo]

There are three more Opentype Math fonts providing both "roundhand" and "chancery" capitals (regular and bold):

• kpfonts-otf (v 0.70),
• erewhon-math (v 0.70),
• xcharter-math (v 0.70),
  (new versions released yesterday on CTAN).

The default is "roundhand" , the "chancery" version is accessible through the +ss01 feature.
The fonts have U+FE00 and U+FE01 as zero-width characters, what else should be done to support the Variation Selector method?

[antonistsolomitis]

unicode-math defines \symscr (and \symcal). Its definition must probably be examined if it is to follow the Unicode "recipe". To make it work with NewCM, one needs to do something like
\setmathfont[range={\mathscr,\mathbfscr},StylisticSet=1]{NewCMMath-Regular.otf}
(this is how fontsetup.sty handles these commands). But following Unicode and variation selectors \symscr would probably need to expand to \symcal followed by U+FE01 and let the font handle the substitution.

According to the documentation of unicode-math (Section 5.4.3) the commands \symcal and \symscr are by default synonyms. However they should not be if variation selectors are used.

As for the fonts themselves they must have tables that map the pair (u1D49C + U+FE01) to u1D49C.alt NewCM will have this in coming releases.

[u-fischer]

I don't know if (and how) one can use properly use variant selectors in math. In xetex it probably doesn't work at all. With luatex they work in text mode but math mainly fails. With cambria one case works (but not in a superscript). Imho that is because cambria actually has U+FE00 as a char. The lua shows this in the table:

  [65024]={
   ["boundingbox"]=252,
   ["index"]=6985,
   ["unicode"]=65024,
   ["width"]=1489,
  },

The other font mentions the selector only in the variants:

--- stix two math
  ["variants"]={
   [65024]={
    [48]=984374,

--- newcm math
  ["variants"]={
   [65025]={
    [8459]=985837,
    [8464]=985838,

And when one try to use the selector in math one gets a missing char message and it is not used.

\documentclass{article}
\newfam\ncm
\font\ncmx = {name:newcomputer modernmath:mode=base;script=math;language=dflt;}
\textfont\ncm = \ncmx
\Umathcode "01D49C =  "0 \ncm "01D49C
\Umathcode "FE01  =  0 \ncm "FE01

\newfam\cambria
\font\cambriax = {name:cambria math:mode=base;script=math;}
\font\cambriavii = {name:cambria math:mode=base;script=math;+ssty=0;} at 7pt
\textfont\cambria = \cambriax
\scriptfont\cambria = \cambriavii
\Umathcode "222A =   0 \cambria "222A
\Umathcode "FE00  =  0 \cambria "FE00
\begin{document}
\ncmx
^^^^^^01d49c^^^^^^01d49c^^^^fe01

$^^^^^^01d49c ^^^^^^01d49c^^^^fe01$

\cambriax

^^^^222a^^^^222a^^^^fe00 

$^^^^222a ^^^^222a^^^^fe00 a^{^^^^222a^^^^fe00}$

\end{document}

[antonistsolomitis]

Hmmm, then maybe it does not worth the trouble... I guess I will postpone further editing of the NewCM fonts for this issue. It also seems that fontforge does not support cmap and one has to go to ttx which makes things more complicated.

[davidcarlisle]

@antonis-tsolomitis if it could be made to work reliably with luatex then I think it would be worth the trouble, VS-1 support is used not just for the script variants but also variant empty set and several other forms, it's very hard to find these things if they just exist in font specific font features, when switching between fonts or converting to MathML for accessibility or other reasons.

This may not be the best forum for deciding what exactly needs to happen, how the mapping is exposed in the font and how the luatex font loader should handle it, but if it could be made to work it would be good I think.

Not least a possible update to Unicode TR5 https://www.unicode.org/reports/tr25/ really needs to say how to access these characters and just saying "use VS-1 but it doesn't work in TeX" doesn't make for a good story....

I realise I'm asking for you to spend time on it, and not really offering to help other than testing, but I didn't want this issue to end on a comment implying that it wasn't an issue.

[u-fischer]

@antonis-tsolomitis I agree with David, as these selectors are in unicode we should try to make them work at least with luatex. And that there are fonts like the new computer modern math which actually have them (and so can be used for tests) is helpful.

[dflipo]

I have tried to set a "rlig" ligature to substitute f.i. \mscrA + U+FE00 to \mcalA (and the like) but it didn't work at all. Are ligatures supposed to work also for Math fonts in "mode=base"?

[antonis-tsolomitis]

I think that here the proper way is through calt and not rlig. I have added this just for the letter \symcal{A} in the font found in https://damo.math.aegean.gr/~atsol/tmp/vs1.zip which also includes a tex file. On each line the first appearance of u1D49C is followed by uniFE01. Watch out not to delete this char as it is not visible (al least on emacs).

The tex file sets the Math font both as text and math. Running xe|lua-latex shows that the font works and changes the first A of the first line to roundhand.

But you will see a strange message in the log (strange at least for me). Both engines complain that cmmi (!) does not contain uniFE01. Why did it fall back to cmmi? A math font containing uniFE01 is provided. Maybe one has to start from this issue.

In any case the font is correct and calt works properly outside math mode as expected.

I thought before to abandon because chatgpt :-) told me that the proper method here (that will cover mathml etc) is through cmap, and cmap is not supported by fontforge. Even if ttx manages to build-in such a table (I tried and failed, but assume that one succeeds) then each save of the font with fontforge will remove cmap and every time it has to be rebuild. This is a pain and easy to make mistakes. TeX though should be happy with calt (at least chatgpt agrees with that). So maybe one has first to try to make this work.

[antonistsolomitis]

It seems that calt or rlig or any lookup is not what is needed here. What is needed is just to define roundhand as alternate shape in the glyph info window. I tested and it works immediately in text mode with both engines but does not work in mathmode. (see comments below).

[mkorje]

Whilst I can't offer much insight into the XeTeX/LuaTeX issues with supporting variation sequences in math, I can give some info on what needs to be done in fontforge for the features to work in other engine (e.g. browsers, Typst). And thus when the issues are resolved on the TeX side things should (hopefully) just work.

I believe you shouldn't need to do any sort of ligature setting/extra table mapping/making the variation selectors zero-width chars in fontforge. It should just suffice to go to the Glyph Info page for the alternate glyph, and on the Unicode page it has the "Alternate Unicode Encodings / Variation Selectors" table, where you can add the Unicode code point of the original glyph and then the variation selector. The ssty feature will work automatically, if you have it setup for the glyph, so you don't need to add anything.

More fontforge details and examples

For example, in the New Computer Modern Math font the default scripts glyphs are the chancery ones. So only the alternate roundhand glyph needs to have alternate Unicode encoding done. Here's an image of the roundhand script capital A in fontforge with the encoding added (with FE01 as it is the roundhand style):

AFAIK there is no need to also do this for the default chancery glyphs, but there should also be no harm in doing so.

This is what Noto Sans Math does. Their script A is roundhand A. And so for the chancery A in the stylistic set they add the variation as script A and FE00. But they don't bother adding the variation script A and FE01 for roundhand A (which is just script A).

It is similar then for the U+2205 empty set glyph and its variation U+2205 U+FE00: zero with long diagonal stroke overlay form (see https://www.unicode.org/Public/15.1.0/ucd/StandardizedVariants.txt). You should just need to go to the variant glyph's Glyph Info page and add to the aforementioned "Alternate Unicode Encodings / Variation Selectors" table the row:

Unicode	Variation Selector (or 0)
U+2205	U+FE00

At some point during text layout a variation sequence like U+2205 U+FE00 should go into the shaper (which will probably be harfbuzz) which will apply it. The change above suffices to get this working.

You can test that this is all working as expected with the hb-shape command from harfbuzz.

Using the hb-shape command

You need to specify an input font, the script Zmth, and then a list of codepoints. You can also provide features to test ssty. It will then return the name of the glyph in the font selected:

$ hb-shape --script=Zmth ~/Documents/fonts/newcm/NewCMMath-Regular.otf -u 0x1D49E
[u1D49E=0+654]

$ hb-shape --script=Zmth ~/Documents/fonts/newcm/NewCMMath-Regular.otf -u 0x1D49E,0xFE00
[u1D49E=0+654]

$ hb-shape --script=Zmth ~/Documents/fonts/newcm/NewCMMath-Regular.otf -u 0x1D49E,0xFE01
[u1D49E.alt=0+797]

$ hb-shape --script=Zmth --features=ssty ~/Documents/fonts/newcm/NewCMMath-Regular.otf -u 0x1D49E,0xFE01
[u1D49E.alt.st=0+845]

$ hb-shape --script=Zmth --features=ssty ~/Documents/fonts/newcm/NewCMMath-Regular.otf -u 0x1D49E
[u1D49E.st=0+716]

I have no idea about what exactly XeTeX or LuaTeX are doing though. Given that it doesn't seem to be working seems to suggest some issue with variation sequence not staying grouped together or going through the shaper in math, maybe.

But it does work in browsers and Typst.
Attached is a HTML file showing the chancery/roundhand variation sequences working in both MathML Core and Typst.

playground-2.html

Typst source code to generate the above file

Note that it currently uses a WIP MathML Core export feature that is not yet released, so the below won't actually compile to the above document with the latest 0.14.0 binary.

#show: doc => [
  #import html: *
  #show: html
  #head[
    #meta(charset: "utf-8")
    #meta(name: "viewport", content: "width=device-width, initial-scale=1")
    #link(
      rel: "stylesheet",
      href: "https://mkorje.github.io/MathFonts/Typst/mathfonts.css",
    )
  ]
  #body(doc)
]

#let test = (
  [`scr` and `cal`],
  ```typ
  $ scr(A) cal(A) scr(a) cal(a) scr(bold(A)) cal(bold(A)) $
  ```,
)

#let process(test) = {
  let eval = eval(
    "#show: block.with(width: 35em)\n" + test.last().text,
    mode: "markup",
  )
  (
    [#test.at(0)],
    test.at(1),
    html.frame(eval),
    eval,
  )
}

#table(
  columns: (auto, 1fr, 35em, 35em),
  table.header(
    [*Test*],
    [*Typst code*],
    [*As rendered by Typst*],
    [*As rendered by your browser*],
  ),
  ..process(test),
)

[davidcarlisle]

@mkorje thanks that looks very helpful

[antonistsolomitis]

The setup suggested by @mkorje above is already available in all NewCM Math fonts available on CTAN (v7.0.4) for the swash and script capitals. So I do not need to prepare anything. Both Xe and Lua engines make use of this in text mode but not in math mode. So the fonts are ready to run tests in Math mode if someone can do it. As for NewCM the emptyset is not yet setup.

[dflipo]

@mkorje Thanks for the hints! I have updated Erewhon--Math, XCharter-Math and Kpfonts accordingly and uploaded them to CTAN (versions 0.71). The U+FE00 variation selector seems to work with Typst (v. 014) for the scr/cal conversion. It should work too with zero (slashed variant), emptyset and other inequalities (see the README for the list).