about https://dharmalekha.info/search

[arlogriffiths]

as a reminder: https://github.com/erc-dharma/project-documentation/issues/7
regarding fuzzy settings for Old Javanese, the following gives a useful list: https://chromewebstore.google.com/detail/sealang-plus/elnlcjojbmjmhheahimkikghiajeloha though it implies a slightly different transliteration system and so the list wouild need to be adapted

@michaelnmmeyer wrote:

The new search system is at: https://dharmalekha.info/search. I will eventually move it to https://dharmalekha.info/texts when done with the interface.

might I suggest the reverse, and possibly the use of "find" rather than search", i.e. make both texts as such and strings in texts findable via https://dharmalekha.info/find or https://dharmalekha.info/search? It might be desirable immediately to separate searching metadata about texts (i.e., searching in teiHeader) and searching within texts by presenting two different search boxes.

@michaelnmmeyer wrote:

I invite you to tell me what you want the matching behavior to be like. My plan is to define one or more matching modes. For instance, there could be a "default" mode which is case-insensitive and ignores hyphens, a "Tamil" mode that does the same and also treats 'k' and 'g' as equivalent, a "Sanskrit" mode that ignores spaces, etc.

MATCHING BEHAVIOR

• I approve of the idea of modes, though I would like us to try not to call them by the names of specific languages (as one of the fundamental ambitions of DHARMA was and is to make terminological affinities findable across language boundaries)
• we could have two basic modes, 'precise' and 'loose'
• there could be a generic (default) version of 'loose', with features such as
  -- 1. being case-insensitive,
  -- 2. ignoring hyphens,
  -- 3. ignoring any milestone elements and any <g> elements
  -- 4. ignoring any transliterated virāmas ·
  -- 5. ignoring differences ē/e and ō/o
• and then there could be custom settings for 'loose'
  -- 1. ignoring difference between voiced/unvoiced,
  -- 2. ignoring difference between aspirated/unaspirated,
  -- 3. ignoring difference between dental/retroflex plosives,
  -- 4. ignoring difference between sibilants
  -- 5. ignoring difference between vowel characters with or without macron (i.e., between vowels transliterated as long or short)
  -- 6. ignoring difference between characters with and without diacritics (like searching in google docs)
  -- 7. ignoring difference between sequences CC and CəC
• I would like us to offer a selection of regular expressions

PRESENTATION OF SEARCH RESULTS
I would like this to be more space-efficient and suggest that we might display only title and complete file name. E.g., compared to this

displaying only

Camundi
tfc-nusantara-epigraphy/DHARMA_INSIDENKCamundi
⟨01⟩ (0) [nama]ś cāmuṇḍyai

might be sufficient

[michaelnmmeyer]

might I suggest the reverse, and possibly the use of "find" rather than search", i.e. make both texts as such and strings in texts findable via https://dharmalekha.info/find or https://dharmalekha.info/search?

I do not see why we should have two different search interfaces, one at /texts and one at /find, when they would be exactly the same save for the fact that the /find interface would also allow you to query the contents of the edition.

we could have two basic modes, 'precise' and 'loose'

You don't explain what is the 'precise' mode.

and then there could be custom settings for 'loose'
-- 1. ignoring difference between voiced/unvoiced,
-- 2. ignoring difference between aspirated/unaspirated,
-- 3. ignoring difference between dental/retroflex plosives,
-- 4. ignoring difference between sibilants
-- 5. ignoring difference between vowel characters with or without macron (i.e., between vowels transliterated as long or short)
-- 6. ignoring difference between characters with and without diacritics (like searching in google docs)
-- 7. ignoring difference between sequences CC and CəC

Having "custom settings" like that forces me to generate massive regular expressions and to use them for searching instead of doing simple string matching operations, which will make search much slower. This is precisely why I was talking about "modes". If we can limit the number of modes to a handful (say, 4 max), I can preprocess each file (possibly with language-specific rules) before search time and produce the representations we will need for searching.

PRESENTATION OF SEARCH RESULTS
I would like this to be more space-efficient and suggest that we might display only title and complete file name.

The search system matches both the contents of the edition and metadata. If metadata fields are not displayed and highlighted when need be, the user has no way to know what exactely matches in the document, he will only see the title and nothing below it if there is no match in the body of the edition.

[arlogriffiths]

Thanks. Clarifications:

1. I mean that we could offer two search machines, one (A) that can do more or less the same things as what the current (old) page https://dharmalekha.info/texts can do (but with a more user-friendly interface, less text and more buttons), and another (B) that would deliver string search within our texts
2. Both should (I think) be made accesssible via a single page, to be named https://dharmalekha.info/search or https://dharmalekha.info/find (in other words, https://dharmalekha.info/texts could be removed or could lead to the contents that are currently shown at https://dharmalekha.info/display)
3. Machine B could offer two basic modes, perhaps to be called (1) strict and (2) loose.
4. Mode B1 would have the capabilities of the machine presently accessible via https://dharmalekha.info/search, i.e. only to match specific strings, ideally with some regex options to be added (wild cards, combinations in single lines of code as with regex xxx.*yyy)
5. Maybe I misunderstand what you are saying, but I believe my point 1 above explains why I request search results yielded by machine B to be accompanied by a very miminal set of metadata, precisely because machine B should only search in editions and not in metadata, so this machine would never generate any matches other than in the bodies of editions.
6. Mode B2 would ideally be customizable in the manners I have suggested in my previous message. I have not understood why those customization options could not be prepared in the manner you have in mind: wouldn't the only programming that you would then need to do be in the input interface, and in preparing of a greater number of modified datasets? E.g., if one would type mādhava in the search window and opt for "ignoring difference between dental/retroflex plosives", then the machine would search in the prepared dataset where the place of dh has been taken by symbol meaning "either dental or retroflex voiced aspirate" , say +, and yield all instances of mā+ava.

My last comment is way beyond my paygrade, so may be completely off the wall. Anyhow, to facilitate meaningful discussion, arguments like search becoming "much slower" need to be quantified for us to weigh pros and cons, and ideally we would have the opportunity to test and then make a reasoned choice between search that is slow but flexible, or fast but less flexible.

I look forward to Dan and possibly others stepping in.

[michaelnmmeyer]

Both should (I think) be made accesssible via a single page

So when you are talking about having 2 "machines", you in fact mean that you want 2 search fields on the same page, one for searching metadata, the other for searching the contents of editions, is that right?

I request search results yielded by machine B to be accompanied by a very miminal set of metadata

I don't see how this could work, because people will want to search both in the edition and in metadata (for filtering by language, etc.). Not allowing search to match both metadata and the edition would severely limit the usefulness of the tool.

Mode B2 would ideally be customizable in the manners I have suggested in my previous message. I have not understood why those customization options could not be prepared in the manner you have in mind: wouldn't the only programming that you would then need to do be in the input interface, and in preparing of a greater number of modified datasets?

To implement the 6 toggles you propose, we would need 2⁶ = 64 representations of each XML document. This is not viable. I think we can get away with 3 or 4, but not much more.

[arlogriffiths]

So when you are talking about having 2 "machines", you in fact mean that you want 2 search fields on the same page, one for searching metadata, the other for searching the contents of editions, is that right?

That is essentially what I was suggesting. But I guess I was starting from the assumption that there is a fundamental (technical) difference between the "machine" that we have had access so for a long time now, and the new one you have recently made available. It is hard to formulate my desires in the abstract, not knowing what the search interface(s) will look like and how results will be displayed.

I request search results yielded by machine B to be accompanied by a very miminal set of metadata

I don't see how this could work, because people will want to search both in the edition and in metadata (for filtering by language, etc.). Not allowing search to match both metadata and the edition would severely limit the usefulness of the tool.

I don't understand the language example, as this information is built into the editions themselves, not the teiHeader. But I understand that there may be a need to combine searches for information contained in teiHeader with information contain in editions.

Nevertheless, we have an interest in reducing as much as possible the need to spread the listing of search results over multiple pages. Yesterday I tested "sādhana", and found that I had to navigate between 4 pages of results, apparently organized in alphabetical order of the edition titles, to evaluate 65 hits. It would be convenient if results could be more densely packed, and perhaps different principles for their ordering could be considered, e.g. per repository and then in alphabetical order.

Mode B2 would ideally be customizable in the manners I have suggested in my previous message. I have not understood why those customization options could not be prepared in the manner you have in mind: wouldn't the only programming that you would then need to do be in the input interface, and in preparing of a greater number of modified datasets?

To implement the 6 toggles you propose, we would need 2⁶ = 64 representations of each XML document. This is not viable. I think we can get away with 3 or 4, but not much more.

I am adding a seventh toggle (CC/CəC) into the wish-list above, for the record. I cannot judge what is viable or not, nor do I know whether whether working with 64 prepared versions is the only way in which we could offer 6 toggles. In your previous response, you suggested that there is another approach, namely with more regex and with less preparation of search versions. I asked you to quantify the loss of speed that this might generate. I think we should experiment with it, because I think we need those toggles.

But if it's impossible then I'll shut up on toggles, and pull out a burning desire that is much more important to me: for the CritEd and DiplEd editions to be included in the searched corpus.

[michaelnmmeyer]

But I guess I was starting from the assumption that there is a fundamental (technical) difference between the "machine" that we have had access so for a long time now, and the new one you have recently made available.

The search implementation is indeed completely separate. The old system used a search function built into the database system we are using. The new system uses a custom search engine I wrote. But this has no bearing on the appearance of search pages, the number of search bars, etc. The latter are presentation details, they are but the tip of the iceberg. The immense majority of the code I write has nothing to do with presentation.

It is hard to formulate my desires in the abstract, not knowing what the search interface(s) will look like and how results will be displayed.

The new search interface is at https://dharmalekha.info/search. I don't see what you're trying to get at.

I don't understand the language example, as this information is built into the editions themselves, not the teiHeader.

It is not the TEI files themselves that are made searchable, but a transformation of them that pulls the necessary data and structures them in a sensible way. This is why you have search fields in the search engine: "title", "editor", "lang", etc. If I didn't do that, you wouldn't be able to filter by "title", etc.

Nevertheless, we have an interest in reducing as much as possible the need to spread the listing of search results over multiple pages. [...] It would be convenient if results could be more densely packed, and perhaps different principles for their ordering could be considered, e.g. per repository and then in alphabetical order.

If I hide most fields and only display the editions' text, the user will not see why a document is returned if his query matches, say, the summary, because the field's content will not be displayed. What I might try to do is to display only fields that match in a given document, to make things more compact.

In your previous response, you suggested that there is another approach, namely with more regex and with less preparation of search versions. I asked you to quantify the loss of speed that this might generate. I think we should experiment with it, because I think we need those toggles.

I don't think you realize how intricate writing a search system is. People spend years doing just that, and I only have a few months left, alongside all the extra programming tasks I have to do for the project. Now, the "toggles" feature might seem like nothing from your point of view, but you should know that this will take much, much longer to implement than the other option. I will do what I can in the remaining time.

burning desire that is much more important to me: for the CritEd and DiplEd editions to be included in the searched corpus.

Ditto.

[michaelnmmeyer]

Also, for the table at https://chromewebstore.google.com/detail/sealang-plus/elnlcjojbmjmhheahimkikghiajeloha, I need you to modify it according to our transliteration system, I can't do that on my own. Here is the list:

[ "a", "ā" ],
[ "b", "bh" ],
[ "c", "ch" ],
[ "d", "dh", "ḍ", "ḍh" ],
[ "ə", "ə̄", "ə:" ],<!-- I have added the equivalence with "ə:" — I hope the use of the colon here is unproblematic -->
[ "e", "ai" ],
[ "g", "gh" ],
[ "h" ],
[ "i", "ī" ], 
[ "j", "jh" ],
[ "k", "kh" ],
[ "l" ],
[ "m", "ḿ" ],
[ "n" ],
[ "ṅ", "ṁ" ],
[ "ñ" ],
[ "o" ],
[ "p", "ph" ],
[ "r", "r̥" ],<!--I don't really understand why the author that table included this equivalence; in practise, in diplomatically edited OJ texts, r̥ and l̥ are nrpmalized to what rə/ər and lə in critical editions -->
[ "s", "ś", "ṣ" ],
[ "t", "th", "ṭ", "ṭh" ],
[ "u", "ū" ],
[ "v", "w" ],
[ "y" ]
[ "h", "ḥ" ]

<!-- should the facts that capital letters A, I, U, E, Ai, O, Au in strict translteration are equivalent to their lower case versions in loos transliteration, and that R̥, L̥ are equivalent to r̥ or rə and to lə, respectively, also be expressed above?-->

[michaelnmmeyer]

Also, about the rule:

7. ignoring difference between sequences CC and CəC

• I do not know the set of consonants in Old Javanese.
• Does the rule CəC implies CəCəC = CCC? If so, I can assume that ə is only significant when it appears at the beginning or at the end of a block of text.

[arlogriffiths]

I have edited the table above and inserted three comments/questions. I wasn't sure of the range of editions (only diplomatic, or both diplomatic and normalized?) to which the above equivalences will be applied.

As for your questions about 7:

• since Old Javanese uses an Indic alphabet, what counts as consonants is exactly the same as in the case of Sanskrit
• I don't think so. (Although I confess I haven't grasped the meaning of your sentence "if so, I can assume that ə is only significant when it appears at the beginning or at the end of a block of text".) I am thinking of very common cases like kn = kən within words. I can't think of any case where a word could be spelt both with CəCəC and with CCC.