README

Slate™ Toolkit is a collection of free open source software to generate and use phrase-based statistical machine translation (SMT) models. It is the only MS Windows compatible collection of SMT utilities and serves as the foundation for our commercial Slate™ Desktop products.

Supported SMT Utilities

Support for open source utilities maintains cross-platform functionality of phrase-based “mode” of statistical machine translation. Factored phrase-based, hierarchical and other SMT modes might work, but are not tested or supported.

From MGIZA++:

  • mgiza(.exe)
  • mkcls(.exe)
  • snt2cooc(.exe)
  • merge_alignment.py

From Moses Toolkit:

  • build_binary(.exe)
  • consolidate(.exe)
  • evaluator(.exe)
  • extract(.exe)
  • extractor(.exe)
  • lexical-reordering-score(.exe)
  • lmplz(.exe)
  • mert(.exe)
  • moses(.exe)
  • processLexicalTable(.exe)
  • processPhraseTable(.exe)
  • query(.exe)
  • score(.exe)
  • symal(.exe)
  • extract-parallel.perl
  • filter-model-given-input.pl
  • filter-rule-table.py
  • flexibility_score.py
  • giza2bal.pl
  • LexicalTranslationModel.pm
  • mert-moses.pl
  • moses_sim_pe.py
  • reduce_combine.pl
  • score-parallel.perl
  • train-model.perl
Unsupported Moses Utilities

Slate™ Toolkit does not include all of the utilities in the original Moses and MGIZA++ projects. We do not support utilities that are not listed above. For example, these utilities are neither included nor supported:

  • BerkeleyAligner
  • IRSTLM
  • RandLM
  • SRILM

Some utilities that we package with in Slate™ Toolkit may or may not work. We have not tested them because they are not needed by our commercial Slate products and therefore, we do not support them. These utilities includes but are not limited to the following:

  • clean-corpus-n.perl
  • detokenizer.perl
  • lowercase.perl
  • snt2cooc.pl
  • tokenizer.perl

The above lists may change. Contact us if you have questions.

Supported SMT Features

Support for phrase dictionary, lexical reordering, language modeling and other advanced features is maintained through the supported utilities.

  • PhraseDictionaryMemory
  • PhraseDictionaryBinary
  • LexicalReordering (memory)
  • LexicalReordering (binary)
  • KenLM (all modes)
  • max-kenlm-order=12
  • with-xmlrpc-c (support for -xml-input)
  • cmph support
Unsupported Moses Features

We do not support feature that are not listed above. Those include, but are not limited to these features.

  • BerkeleyAligner
  • PhraseDictionaryOnDisk
  • PhraseDictionaryCompact
  • LexicalReordering (compact)
  • IRSTLM
  • RandLM
  • SRILM
  • hierarchical models
  • suffix arrays
  • bilingual language models

If you need cross-platform (e.g. on Windows) support for a particular utility or feature from one of the original open source packages that is not listed here, please let us know. We may be able to add it to Slate™ Toolkit.

Getting Started

Lengthy command lines from the various open source utilities can be error-prone. Therefore, this toolkit includes Windows .cmd and Bash .sh shell scripts, plus a very small sample training corpus. These do not constitute a production-ready environment. Rather, they demonstrate the open source command lines for essential steps that prepare corpora, train & tune models and translate text.

The demo-all script is the best place to start. Just run it in-place.

Outputs from upstream scripts become the inputs to downstream scripts. Therefore, the script names are numbered in the order to follow when you run them individually. The order is also referenced in the demo-all script.

File Types

SlateToolkit can use these file types:

  • Text files with UTF-8 character encoding, Linux or Windows new line separators
  • Tab-delimited files are specialized Text file (as above) with one tab per line. Text left of the tab is the source language. Text right of the tab is target language.
Caveats

These open source projects were written for academic use on Unix-like systems. Therefore, there are a few things you can do to protect yourself from problems.

Naming of files and folders

Unix and Windows deal with locations of files and folders in different ways:

  • Windows paths use drive letters and backslashes; Unix paths use slashes to indicate where a file is. Slate™ Toolkit generally supports each system’s native style, but if you run into glitches, please let us know.
  • In Windows, “a.txt” and “A.txt” are the same file; in Unix they are different. Avoid names that can be confused in this way, and make sure you capitalize all names consistently. The software may not always realize that the two are the same name on your system.
  • Unix software often uses whitespace to separate one filename from another. Files or folders are allowed to have whitespace in them, but it often brings out bugs in software. Avoid whitespace in file and folder path names.
  • Handling of non-ASCII characters can differ between individual computers depending on configuration, and likewise often triggers software bugs.
  • Many punctuation marks can have special meanings on different systems, such as colons, quotes and apostrophes, equals signs, dollar signs, percentage signs, asterisks, tildes, and so on. Avoid these marks. Keep it simple! When in doubt, use dashes and/or underscores.

For trouble-free use, we recommend that you use only files and folders with names consisting exclusively of ASCII letters (a-z), digits (0-9), dots, and dashes or underscores (-, _). With your help and patience we hope to improve the user experience over time.

Line endings

On Windows systems, a line of text ends in a fixed sequence of two characters: carriage return and line feed, also written as “”. Unix systems use just the line feed, or “”.

This may confuse some tools when dealing with files that were not written with your system’s native line endings. Windows Notepad may show all contents in a Unix text file as a single, long line; or instead of returning to the starting column for every new line, some software may just start the next line right below where the last one ended. Rare Unix tools may interpret the carriage returns as “jump back to the beginning of the line and erase all text that was previously displayed.”

Slate™ Toolkit accepts input files with either style of line endings. It generally creates output files with platform-specific line endings, but at times it creates Unix-style files on Windows systems. This may not be perfect and with your feedback we hope to improve it over time.

Contact Information
Slate Rocks LLC
Web: https://support.slate.rocks/
Email: sales@slate.rocks
  • sort(.exe)
  • split(.exe)
  • libiconv2(.dll) and libintl3(.dll)
  • gzip(.exe), also copied as gunzip(.exe) and bzcat(.exe)
  • The Windows package installs Perl from Strawberry Perl and updates the system %PATH%.
  • The Linux package works “out of the box” on Ubuntu 12.04 or newer systems. Other Linux systems may work. We welcome your feedback about your experiences.
  • The Windows package installs Python from python.org and updates the system %PATH%.
  • The Linux package works “out of the box” on a standard Ubuntu 12.04 or newer systems. Other Linux systems may work and we welcome your feedback about your experiences.