Forum Web>FstForum (2024-09-14, JakubKaszycki)

EditAttach

OpenFst Forum

*11/11/22: Spam-fighting code gone haywire was causing various failures/slowdowns on the forum. It has been addressed.

You need to be a registered user to participate in the discussions.
Log In or Register

OpenFst Forum

OpenFst Forum Archives

You can start a new discussion here:

You can use the formatting commands describes in TextFormattingRules in your comment.
If you want to post some code, surround it with <verbatim> and </verbatim> tags.
Auto-linking of WikiWords is now disabled in comments, so you can type VectorFst and it won't result in a broken link.
You now need to use   to force new lines in your comment (unless inside verbatim tags). However, a blank line will automatically create a new paragraph.

Subject
Comment

Bug of CyclicMinimizer

JakubKaszycki - 2024-09-14 - 05:19

I have been getting random segmentation faults in the CyclicMinimizer and this time instead of the ostrich algorithm I decided to solve it.

A comment in the code claims that moving out of the top of priority queue is safe. However, after investigating, it turned out that the heap indeed tried to compare the element being removed. I have swapped it out to manually calling push_heap, pop_heap, etc., so now we can move the element out after any heap operation, just before it's removed from the end of the heap.

Below is my patch that seems to fix the bug:

<verbatim> diff --git a/src/include/fst/minimize.h b/src/include/fst/minimize.h index 7e59f9c..69518c9 100644

a/src/include/fst/minimize.h +++ b/src/include/fst/minimize.h @@ -181,9 +181,7 @@ class CyclicMinimizer { } };

- using ArcIterQueue = - std::priority_queue<RevArcIterPtr, std::vector<RevArcIterPtr>, - ArcIterCompare>; + using ArcIterQueue = std::vector<RevArcIterPtr>;

private: // Prepartitions the space into equivalence classes. We ensure that final and @@ -241,30 +239,26 @@ class CyclicMinimizer { P_.Initialize(Tr_.NumStates() - 1); // Prepares initial partition. PrePartition(fst); - // Allocates arc iterator queue. - aiter_queue_ = std::make_unique<ArcIterQueue>(); } // Partitions all classes with destination C. void Split(ClassId C) { + // Prepares priority queue: opens arc iterator for each state in C, and // inserts into priority queue. for (PartitionIterator<StateId> siter(P_, C); siter.Done(); siter.Next()) { const auto s = siter.Value(); if (Tr_.NumArcs(s + 1)) { - aiter_queue_->push(std::make_unique<RevArcIter>(Tr_, s + 1)); + aiter_queue_.push_back(std::make_unique<RevArcIter>(Tr_, s + 1)); } } + std::make_heap(aiter_queue_.begin(), aiter_queue_.end(), ArcIterCompare{}); // Now pops arc iterator from queue, splits entering equivalence class, and // re-inserts updated iterator into queue. Label prev_label = -1; - while (aiter_queue_->empty()) { - // NB: There is no way to "move" out of a std::priority_queue given that - // the `top` accessor is a const ref. We const-cast to move out the - // unique_ptr out of the priority queue. This is fine and doesn't cause an - // issue with the invariants of the pqueue since we immediately pop after. - RevArcIterPtr aiter = - std::move(const_cast<RevArcIterPtr &>(aiter_queue_->top())); - aiter_queue_->pop(); + while (aiter_queue_.empty()) { + std::pop_heap(aiter_queue_.begin(), aiter_queue_.end(), ArcIterCompare{}); + RevArcIterPtr aiter = std::move(aiter_queue_.back()); + aiter_queue_.pop_back(); if (aiter->Done()) continue; const auto &arc = aiter->Value(); auto from_state = aiter->Value().nextstate - 1; @@ -274,7 +268,10 @@ class CyclicMinimizer { if (P_.ClassSize(from_class) > 1) P_.SplitOn(from_state); prev_label = from_label; aiter->Next(); - if (aiter->Done()) aiter_queue_->push(std::move(aiter)); + if (aiter->Done()) { + aiter_queue_.push_back(std::move(aiter)); + std::push_heap(aiter_queue_.begin(), aiter_queue_.end(), ArcIterCompare{}); + } } P_.FinalizeSplit(&L_); } @@ -298,7 +295,7 @@ class CyclicMinimizer { VectorFst<RevArc> Tr_; // Priority queue of open arc iterators for all states in the splitter // equivalence class. - std::unique_ptr<ArcIterQueue> aiter_queue_; + ArcIterQueue aiter_queue_; };

// Computes equivalence classes for acyclic FST. </verbatim>

JakubKaszycki - 2024-09-14 - 05:21

The patch got mangled by the forum, so I put in on Pastebin: https://pastebin.com/xi8KvdQH

'pywrapfst.Fst' object has no attribute '_class__'

AariciaHerygers - 2024-08-08 - 08:46

I have installed OpenFst v1.8.3 on Ubuntu 24.04 and enabled Python (v3.12). However, when running

<verbatim>WEIGHT_TYPE = pywrapfst.Fst().weight_type()</verbatim>

I get the following error:

<verbatim>File "pywrapfst.pyx", line 1681, in pywrapfst.Fst.__init__ AttributeError: 'pywrapfst.Fst' object has no attribute '_class__'. Did you mean: '__class__'?</verbatim>

Note the single underscore for class. How do I solve this? Thanks.

Checking to see if a word is accepted

AlexW - 2024-06-28 - 10:23

Using the shell, what's the easiest way to see if a word is accepted by an acceptor? I don't see a command for this.

Regular expressions

AlexW - 2024-06-28 - 10:13

I am new to this. Is there a function in the library to build an automaton from a regular expression?

windows support

JeremyBraun - 2024-01-18 - 07:53

Are you open to accepting patches to enable this project to be compiled/used on a windows system through MSYS2 (gcc or clang as compilers, mingw as the bulk of the adaptation layer)?

KyleGorman - 2024-01-22 - 14:55

We are not in theory opposed to anything to do with Windows support, but we don't have any means currently to test Windows support, except that OpenFst is now built for Windows on Conda-Forge CI.

As to how best to tackle the Windows compilation issue I plead ignorance.

JeremyBraun - 2024-01-23 - 11:00

Great, thank you. I'll look into this a little, as well as whether the Conda-Forge stuff will work for my use. Thanks!

build error in new debian

StoneLee - 2023-05-09 - 02:37

build command is ./configure --enable-python and make it throw error with logs pywrapfst.cpp:196:12: fatal error: longintrepr.h not found due to cython generate code pywrapfst.cpp is not compatible to python3.11 please update it

KyleGorman - 2024-01-22 - 14:54

Try again with the current release (1.8.3), please.

pywrapfst support in conda package?

AdamJanin - 2022-11-04 - 13:30

Are there any plans to support pywrapfst in the conda package?

KeWu - 2022-11-10 - 11:45

pywrapfst is currently packaged with pynini [https://anaconda.org/conda-forge/pynini].

Patch to make headers C++20 compatible

LenardSzolnoki - 2022-08-26 - 03:30

This patch is against 1.8.2 r1

Use allocator_traits::allocate in memory.h

The two argument overload of std::allocator<T>::allocate got deprecated in C++17 and removed in C++20. std::allocator_traits<A>::allocate supports the hint argument.

src/include/fst/memory.h +++ src/include/fst/memory.h @@ -241,7 +241,8 @@ class BlockAllocator { if (n * kAllocFit <= kAllocSize) { return static_cast<T *>(Arena()->Allocate(n)); } else { - return Allocator().allocate(n, hint); + auto alloc = Allocator(); + return std::allocator_traits<Allocator>::allocate(alloc, n, hint); } }

@@ -307,7 +308,8 @@ class PoolAllocator { } else if (n <= 64) { return static_cast<T *>(Pool<64>()->Allocate()); } else { - return Allocator().allocate(n, hint); + auto alloc = Allocator(); + return std::allocator_traits<Allocator>::allocate(alloc, n, hint); } }

</verbatim>

LenardSzolnoki - 2022-08-26 - 03:35

verbatim tags just don't seem to work, here is a pastebin URL for the patch: https://pastebin.com/HkUAwHcZ

KyleGorman - 2022-11-17 - 11:17

We'll move to the C++20-compatible style in the next release, thanks.

SymbolTable::Copy() const correctness

KarelVesely - 2022-06-30 - 06:26

Hello, i came across an issue with SymbolTable::Copy() const correctness recently.

I have noticed that SymbolTable::Copy() makes a shallow copy of the data present in SymbolTableImpl. The method signature is `virtual SymbolTable* SymbolTable::Copy() const`, so if i call (const SymbolTable*)->Copy() i get a SymbolTable*. And, accidentelly, I was changing the content of the original table.

I'd suggest to have there methods: `virtual SymbolTable* SymbolTable::Copy()` `virtual const SymbolTable* SymbolTable::Copy() const` Or just the 1st that just removes the 'const' modifier.

Would it make sense for you as well? Best regards, Karel

Error Compiling with --enable-python

MatanAbudy - 2022-06-09 - 06:24

I've tried compiling OpenFST 1.8.2 with the Python extension but got errors. I've ran: ./configure --enable-python && make -j

I've attached the compilation output here: https://pastebin.com/m6aV29m8

What am I missing?

BartDHoore - 2022-08-04 - 07:36

I got an error when confguring due to python version checks failing. I notived that the configure script checks python versions suboptimally. Here is a suggestion for a better check:

Instead of:

ac_supports_python_ver=`$PYTHON -c "import sys; ver = sys.version.split ()[0]; print (ver >= '3.6')"`

it is probably better to do:

ac_supports_python_ver=`$PYTHON -c "import sys; from packaging import version; ver = sys.version.split ()[0]; print (version.parse(ver) >= version.parse('3.6'))"`

KyleGorman - 2024-01-04 - 10:32

That looks like it would work though we don't actually maintain those macros; they're autogenerated by autoconf. (Yes, they're buggy.)

KyleGorman - 2024-01-22 - 14:56

I think the updated macro (grabbed with `autoupdate`) is handling this case properly---please check though.

Creating an executable without the libfst.so runtime dependency?

AlexMacLean - 2022-05-27 - 16:13

Based on the README I've been able to get a program that links against the libfst.so library running. I'd like to create an executable that doesn't depend on the .so file being in the expected place at runtime to work. Is this possible? is there a libfst.o file somewhere I can link against? What would the Makefile of such an executable look like?

fstcompile bug with long lines

RemiFrancis - 2022-04-29 - 07:16

I've recently moved from openfst 1.6.2 to version 1.7.2, and I have a test case falling: fstcompile produces an empty output when one of the word is too long. I've noticed it starts to happen when the text line has more than 8096 characters.

Code to reproduce the issue (set num=8080 to see the error):

#!/bin/bash

num=$1 word=$(for i in $(seq 1 $num); do echo -n 'a'; done) echo "<eps> 0" > words.txt echo "$word 24" >> words.txt

phone=jnk_S echo "<eps> 0" > phones.txt echo "$phone 186" >> phones.txt

echo -e "0\t0\t$phone\t$word\t4.265" > L.txt wc L.txt echo -e "0\t0" >> L.fst

fstcompile --isymbols=phones.txt --osymbols=words.txt L.txt L.fst

echo "Printing L.fst" fstprint L.fst

RemiFrancis - 2022-04-29 - 07:18

Sorry can't edit the text, let me try to format the code:

<verbatim> #!/bin/bash

num=$1 word=$(for i in $(seq 1 $num); do echo -n 'a'; done) echo "<eps> 0" > words.txt echo "$word 24" >> words.txt

phone=jnk_S echo "<eps> 0" > phones.txt echo "$phone 186" >> phones.txt

echo -e "0\t0\t$phone\t$word\t4.265" > L.txt wc L.txt echo -e "0\t0" >> L.fst

fstcompile --isymbols=phones.txt --osymbols=words.txt L.txt L.fst

echo "Printing L.fst" fstprint L.fst </verbatim>

RemiFrancis - 2022-04-29 - 07:19

Sorry can't edit the text, let me try to format the code:

#!/bin/bash

num=$1 word=$(for i in $(seq 1 $num); do echo -n 'a'; done) echo "<eps> 0" > words.txt echo "$word 24" >> words.txt

phone=jnk_S echo "<eps> 0" > phones.txt echo "$phone 186" >> phones.txt

echo -e "0\t0\t$phone\t$word\t4.265" > L.txt wc L.txt echo -e "0\t0" >> L.fst

fstcompile --isymbols=phones.txt --osymbols=words.txt L.txt L.fst

echo "Printing L.fst" fstprint L.fst

</verbatim>

RemiFrancis - 2022-04-29 - 07:20

Sorry can't edit the text, let me try to format the code:

#!/bin/bash

num=$1 word=$(for i in $(seq 1 $num); do echo -n 'a'; done) echo "<eps> 0" > words.txt echo "$word 24" >> words.txt

phone=jnk_S echo "<eps> 0" > phones.txt echo "$phone 186" >> phones.txt

echo -e "0\t0\t$phone\t$word\t4.265" > L.txt wc L.txt echo -e "0\t0" >> L.fst

fstcompile --isymbols=phones.txt --osymbols=words.txt L.txt L.fst

echo "Printing L.fst" fstprint L.fst

RemiFrancis - 2022-04-29 - 07:21

Ok I've got no clue how to use this forum...

fstcompile bug with long words

RemiFrancis - 2022-04-29 - 07:15

Code to reproduce the issue (set num=8080 to see the error):

#!/bin/bash

num=$1 word=$(for i in $(seq 1 $num); do echo -n 'a'; done) echo "<eps> 0" > words.txt echo "$word 24" >> words.txt

phone=jnk_S echo "<eps> 0" > phones.txt echo "$phone 186" >> phones.txt

echo -e "0\t0\t$phone\t$word\t4.265" > L.txt wc L.txt echo -e "0\t0" >> L.fst

fstcompile --isymbols=phones.txt --osymbols=words.txt L.txt L.fst

echo "Printing L.fst" fstprint L.fst

KyleGorman - 2024-01-22 - 14:57

Yes, there is a hard limit for text of this length, but why would you have such a long line on these inputs anyways?

disable the dynamic linking during the openfst configuration

MaiDaly - 2021-10-11 - 13:47

I need to disable the dynamic linking during the openfst configuration before library building.

chain fstspecial

AntonLeuski - 2021-07-27 - 11:25

Does fstspecial support chaining calls? For example, if my FST uses both sigma and rho symbols, I chain it like this:

fstspecial --sigma_fst_sigma_label=1 --fst_type=sigma model.fst | fstspecial --rho_fst_rho_label=2 --fst_type=rho > model1.fst

However, it seems like model1.fst ignores sigma labels and only handles arcs with the rho label. What am I missing? Thank you.

KyleGorman - 2021-09-25 - 09:00

It supports chaining in the narrowest sense but adding a rho matcher in the second step overwrites the sigma mature in the first step. I don't think there's any way to implement an FST with both rho and sigma arcs yet.

Assorted issues and possible fixes (1.8.0)

NimatashiSherpa - 2021-03-02 - 12:39

============================================================================ Summary of files with issues:

.../include/fst/ compose.h pair-weight.h prune.h randequivalent.h reweight.h rmfinalepsilon.h shortest-path.h string-weight.h

.../include/fst/test/ algo_test.h

============================================================================ Detailed descriptions:

The lines with issues are prefixed with "NS:"

****************************************************************************** compose.h: Line 660 (Approx): ... // Create compose implementation specifying one matcher type; requires that // input and matcher FST types be Fst<Arc>. template <class Matcher, class Filter, class StateTuple> static std::shared_ptr<Impl> CreateBase1( const Fst<Arc> &fst1, const Fst<Arc> &fst2, const ComposeFstOptions<Arc, Matcher, Filter, StateTuple> &opts) { ComposeFstImplOptions<Matcher, Matcher, Filter, StateTuple, CacheStore> nopts(opts, opts.matcher1, opts.matcher2, opts.filter, opts.state_table); // NS: Bug Fix. Allow non-commutative weights nopts.allow_noncommute = true; return CreateBase2(fst1, fst2, nopts); ... ****************************************************************************** pair-weight.h:

// NS: Rewrote entire class so that allow_zero can be set for sub-weight // generators, otherwise we will get <Zero, NonZero> or <NonZero, Zero> pairs template <class W1, class W2> class WeightGenerate<PairWeight<W1, W2>> { public: using Weight = PairWeight<W1, W2>; using Generate1 = WeightGenerate<W1>; using Generate2 = WeightGenerate<W2>;

explicit WeightGenerate(uint64 seed = std::random_device()(), bool allow_zero = true) // NS: Initialize additional fields : gen_(seed), allowZero_(allow_zero), maxValue_(kNumRandomWeights), generate1_(seed, false), generate2_(seed, false) {};

// NS: Fill out the generation algorithm Weight operator()() const { std::uniform_int_distribution<int> dieRoll(0, maxValue_);

// Special cases... while (true) { int roll = dieRoll(gen_); if (roll == 0) { if (allowZero_) return Weight::Zero(); } else if (roll == 1) { return Weight::One(); } else { return Weight(generate1_(), generate2_()); } } // End while() };

private: // NS: Add gen_, allowZero_, maxValue_ mutable std::mt19937 gen_; const bool allowZero_; const int maxValue_; const Generate1 generate1_; const Generate2 generate2_; }; // End class WeightGenerate

****************************************************************************** prune.h: // NS: There is an undesirable case when threshold == Weight::One(). Consider an // Fst with a single path. Floating point rounding means that the best path // may fail the comparison with "limit", which would lead to a surprising // return value. We can allow some wiggle room by defining the following function // which uses ApproxEqual(): template<class W, typename std::enable_if<(W::Properties() & kPath) == kPath>::type * = nullptr> bool reallyLess(const W& w1, const W& w2) { NaturalLess<W> natLess;

return natLess(w1, w2) && ApproxEqual(w1, w2, kDelta); } --- In Prune() routines replace calls to "less()" with calls to "reallyLess()" -- 7 instances.

****************************************************************************** randequivalent.h: // NS: If an Fst is cyclic, then the weight must be both idempotent and commutative. ... Line 88 (Approx): if ( (!(Weight::Properties() & kIdempotent) || !(Weight::Properties() & kCommutative)) && pfst1.Properties(kCyclic, true)) { continue; } ... Line 100 (Approx): if ( (!(Weight::Properties() & kIdempotent) || !(Weight::Properties() & kCommutative)) && pfst2.Properties(kCyclic, true)) { continue; } ****************************************************************************** reweight.h: // NS: Reweighting relies on Divide() which may not work. Watch for bad divisions // and revert changes. // Look for lines with "divideError" in the excerpted code. // The changes are illustrative rather than optimal. Reweight(....) { ... MutableFst<Arc>* cachedFst = fst->Copy(); bool divideError = false;

const uint64 input_props = fst->Properties(kFstProperties, false); StateIterator<MutableFst<Arc>> siter(*fst); for (; siter.Done(); siter.Next()) { const auto s = siter.Value(); if (s == potential.size()) break; const auto &weight = potential[s]; if (weight = Weight::Zero()) { for (MutableArcIterator<MutableFst<Arc>> aiter(fst, s); aiter.Done(); aiter.Next()) { auto arc = aiter.Value(); if (arc.nextstate >= potential.size()) continue; const auto &nextweight = potential[arc.nextstate]; if (nextweight == Weight::Zero()) continue; if (type == REWEIGHT_TO_INITIAL) { arc.weight = Divide(Times(arc.weight, nextweight), weight, DIVIDE_LEFT); } if (type == REWEIGHT_TO_FINAL) { arc.weight = Divide(Times(weight, arc.weight), nextweight, DIVIDE_RIGHT); } aiter.SetValue(arc); // NS: Bug Fix. divideError |= arc.weight.Member(); } if (type == REWEIGHT_TO_INITIAL) { fst->SetFinal(s, Divide(fst->Final(s), weight, DIVIDE_LEFT)); // NS: Bug Fix. divideError |= fst->Final(s).Member(); } } if (type == REWEIGHT_TO_FINAL) { fst->SetFinal(s, Times(weight, fst->Final(s))); } } // This handles elements past the end of the potentials array. for (; siter.Done(); siter.Next()) { const auto s = siter.Value(); if (type == REWEIGHT_TO_FINAL) { fst->SetFinal(s, Times(Weight::Zero(), fst->Final(s))); } } const auto startweight = fst->Start() < potential.size() ? potential[fst->Start()] : Weight::Zero(); bool added_start_epsilon = false; if ((startweight = Weight::One()) && (startweight = Weight::Zero())) { if (fst->Properties(kInitialAcyclic, true) & kInitialAcyclic) { const auto s = fst->Start(); for (MutableArcIterator<MutableFst<Arc>> aiter(fst, s); aiter.Done(); aiter.Next()) { auto arc = aiter.Value(); if (type == REWEIGHT_TO_INITIAL) { arc.weight = Times(startweight, arc.weight); } else { arc.weight = Times(Divide(Weight::One(), startweight, DIVIDE_RIGHT), arc.weight); } aiter.SetValue(arc); // NS: Bug Fix. divideError |= arc.weight.Member(); } if (type == REWEIGHT_TO_INITIAL) { fst->SetFinal(s, Times(startweight, fst->Final(s))); } else { fst->SetFinal(s, Times(Divide(Weight::One(), startweight, DIVIDE_RIGHT), fst->Final(s))); // NS: Bug Fix. divideError |= fst->Final(s).Member(); } } else { const auto s = fst->AddState(); const auto weight = (type == REWEIGHT_TO_INITIAL) ? startweight : Divide(Weight::One(), startweight, DIVIDE_RIGHT); // NS: Bug Fix. divideError |= weight.Member(); fst->AddArc(s, Arc(0, 0, weight, fst->Start())); fst->SetStart(s); added_start_epsilon = true; } } // NS: Bug Fix if (divideError) { fst->SetProperties(ReweightProperties(input_props, added_start_epsilon) | fst->Properties(kFstProperties, false), kFstProperties); } else { // NS: Bug Fix. *fst = *cachedFst; LOG(WARNING) << "Reweight: Divide() won't work, so not reweighting. " << "Weight type: " << Weight::Type() << std::endl; } delete cachedFst; } // End of Reweight() ****************************************************************************** rmfinalepsilon.h: Line 75 (Approx): if (arc.ilabel = 0 && arc.olabel = 0) { // NS: Correct the order arguments in call to Times() for non-commutative weights weight = Plus(Times(arc.weight, fst->Final(arc.nextstate)), weight); } else { ... ****************************************************************************** shortest-path: // NS: May need to reverse weight. Line 513 (Approx): distance->insert(distance->begin(), d); if (opts.unique) { internal::NShortestPath(rfst, ofst, *distance, opts.nshortest, opts.delta, opts.weight_threshold, opts.state_threshold); } else { std::vector<Weight> ddistance; const DeterminizeFstOptions<RevArc> dopts(opts.delta);

// NS: If RevWeight = Weight... using RevWeight = typename Weight::ReverseWeight;

std::vector<RevWeight> rddistance(ddistance.size()), rdistance(distance->size()); transform( distance->begin(), distance->end(), rdistance.begin(), [](Weight& w){return w.Reverse() transform(ddistance.begin(), ddistance.end(), rddistance.begin(), [](Weight& w){return w.Reverse();} const DeterminizeFst<RevArc> dfst(rfst, &rdistance, &rddistance, dopts); transform( rdistance.begin(), rdistance.end(), distance->begin(), [](RevWeight& w){return w.Revers transform(rddistance.begin(), rddistance.end(), ddistance.begin(), [](RevWeight& w){return w.Reverse // NS: NShortestPath() handles the weight reversal correctly internal::NShortestPath(dfst, ofst, ddistance, opts.nshortest, opts.delta, opts.weight_threshold, opts.state_threshold); } ****************************************************************************** string-weight.h: StringWeight methods: DivideLeft(): The code does not check that the divisor really is a prefix of the dividend. It should return NoWeight if this is not the case. DivideRight(): The code does not check that the divisor really is a suffix of the dividend. It should return NoWeight if this is not the case. ******************************************************************************

Missing dependency to pdt in some Bazel rules

YusukeOda - 2021-01-17 - 12:33

I have tried to build/test the library using Bazel (by invoking "bazel build/test ..." at the workspace root), and found following rules:

- :linearscript - :linear-fst

require a dependency to :pdt, but the rules do not have it. As far as I checked, all other rules can be built correctly.

It seems this is a trivial bug that can be fixed soon.

KyleGorman - 2021-09-25 - 09:58

Fixed in the next release, thanks for the find. (I see why this dependency exists now.)

OpenFST unique ShortestPath with epsilon

AmT - 2021-01-10 - 16:12

Given an FST and alphabet which includes a blank char (''), I want to find the N unique shortest paths:

- Unique in the sense that there are no two paths with equal labels concatenation - e.g., ['a', '', 'b'] and ['a', 'b', ''] are considered equal.

- The output should be a list of labels including blank chars. All the paths from start node to end node in the input FST are of the same length, and the shortest path output should maintain this property.

ShortestPath takes an unique argument. However, as the documentation suggests "epsilons treated as regular symbols", therefore I cannot define blank as epsilon and use ShortestPath on the FST directly.

One idea I had, is to define blank as the epsilon symbol, remove all the epsilon symbols (using RmEpsilon), find the N unique shortest paths, and try to search each of the shortest paths in the original FST to re-construct paths with blank symbols. However, I am not sure how I can do the last part.

KyleGorman - 2021-09-25 - 09:11

I don't see any way to do this, sorry. Yes, removing epsilons is necessary if you're calling the `unique` option of ShortestPath and you want unique strings rather than unique paths.

Speedup lookahead composition

NickolayShmyrev - 2021-01-07 - 11:48

Hi there. Me again

Current composition with ngram fst in openfst is pretty slow due to the branching/writeback issues and due to the irregular reachable interval search.

The following patch greatly speedups the composition in runtime:

https://github.com/alphacep/openfst/commit/65e9de91b35a7238ffa035ed8bb2594a50dbdc5a

with more intelligent binary search pattern and it also removes lazy field in the iterator since writes conflict with branching and slowdown cpu pipeline a lot.

Please consider

Wrong type for vcnt

NickolayShmyrev - 2021-01-05 - 19:21

vcnt_s8 arm64 command takes int argument, not uint, leading to error in nthbit.h about type mismatch.

Fix here:

https://github.com/alphacep/openfst/commit/914217171f7a7c5e13227a8e9e29877edd18272c

Overdeprecation of SymbolTableReadOptions

NickolayShmyrev - 2020-12-22 - 08:15

Compiling this simple code:

#include <fst/symbol-table.h>

Ends in many deprecation warnings like this:

<verbatim> In file included from test.cc:1: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In constructor ‘fst::SymbolTableReadOptions::SymbolTableReadOptions()’: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:68:28: warning: ‘fst::SymbolTableReadOptions::string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations]

| SymbolTableReadOptions() {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:68:28: warning: ‘fst::SymbolTableReadOptions::string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations]

| SymbolTableReadOptions() {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

| SymbolTableReadOptions() {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

| SymbolTableReadOptions() {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In constructor ‘fst::SymbolTableReadOptions::SymbolTableReadOptions(std::vector<std::pair<long int, long int> >, const string&)’: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:74:38: warning: ‘string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations]

| : string_hash_ranges(std::move(string_hash_ranges)), source(source) {} | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:72:44: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges,

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:74:73: warning: ‘fst::SymbolTableReadOptions::string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations]

| : string_hash_ranges(std::move(string_hash_ranges)), source(source) {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

| : string_hash_ranges(std::move(string_hash_ranges)), source(source) {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

| : string_hash_ranges(std::move(string_hash_ranges)), source(source) {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

| : string_hash_ranges(std::move(string_hash_ranges)), source(source) {} | ^

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: At global scope: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:247:66: warning: ‘SymbolTableReadOptions’ is deprecated [-Wdeprecated-declarations] 247 | const SymbolTableReadOptions &opts); | ^ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:421:62: warning: ‘SymbolTableReadOptions’ is deprecated [-Wdeprecated-declarations] 421 | const SymbolTableReadOptions &opts) { | ^ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In static member function ‘static fst::SymbolTable* fst::SymbolTable::Read(std::istream&, const string&)’: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:428:28: warning: ‘SymbolTableReadOptions’ is deprecated: Reading via SymbolTableReadOptions is deprecated. Please directly provide source string instead. [-Wdeprecated-declarations] 428 | SymbolTableReadOptions opts; | ^~~~ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:67:31: note: declared here

| "source string instead.") SymbolTableReadOptions { | ^~~~~~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In destructor ‘fst::SymbolTableReadOptions::~SymbolTableReadOptions()’: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:67:31: warning: ‘fst::SymbolTableReadOptions::string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations] /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:67:31: warning: ‘fst::SymbolTableReadOptions::string_hash_ranges’ is deprecated: Do not use `string_hash_ranges`; it is ignored. [-Wdeprecated-declarations]

| "source string instead.") SymbolTableReadOptions { | ^~~~~~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:77:40: note: declared here

| std::vector<std::pair<int64, int64>> string_hash_ranges; | ^~~~~~~~~~~~~~~~~~

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In static member function ‘static fst::SymbolTable* fst::SymbolTable::Read(std::istream&, const string&)’: /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:428:28: note: synthesized method ‘fst::SymbolTableReadOptions::~SymbolTableReadOptions()’ first required here 428 | SymbolTableReadOptions opts; | ^~~~ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:430:27: warning: ‘static fst::SymbolTable* fst::SymbolTable::Read(std::istream&, const fst::SymbolTableReadOptions&)’ is deprecated: Use `Read(std::istream&, const std::string&)` instead. [-Wdeprecated-declarations] 430 | return Read(strm, opts); | ^ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:420:23: note: declared here 420 | static SymbolTable *Read(std::istream &strm, | ^~~~ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:430:27: warning: ‘static fst::SymbolTable* fst::SymbolTable::Read(std::istream&, const fst::SymbolTableReadOptions&)’ is deprecated: Use `Read(std::istream&, const std::string&)` instead. [-Wdeprecated-declarations] 430 | return Read(strm, opts); | ^ /home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h:420:23: note: declared here 420 | static SymbolTable *Read(std::istream &strm, | ^~~~ </verbatim>

Looks like this part was deprecated too early since it is used in the header code actively, for example in:

/home/shmyrev/travis/kaldi/tools/openfst/include/fst/symbol-table.h: In static member function ‘static fst::SymbolTable* fst::SymbolTable::Read(std::istream&, const string&)’:

It would be nice to rework this part.

Pynini 2.1.3 with OpenFST 1.8.0: pynini_test.py Failures and Errors

MichaelHenderson - 2020-12-15 - 16:25

Hi, I successfully downloaded the source and installed Pynini 2.1.3 after successfully installing OpenFST 1.8.0 using gcc version 10.2.0 (Debian 10.2.0-9) with Python 3.6.9.

I compiled OpenFST with the following options:

/openfst-1.8.0$ sudo ./configure --enable-far=yes --enable-pdt=yes --enable-mpdt=yes --enable-static=no --cache-file=config.cache --enable-grm=yes --enable-bin=yes

I followed the instructions in the Pynini README and ran the pynini_test.py after completing the install of Pynini. It looks like 145 tests were run in total with the following 4 failures and 6 errors.

To confirm successful installation, run `python pynini_test.py`; if all tests succeed, the final line will read `OK`.

/pynini-2.1.3/tests$ python pynini_test.py ..ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor

....ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

...ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor

.ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

...EE...ERROR: Verify: FST destination state ID of arc at position 0 of state 0 is negative

.ERROR: State ID -1 not valid

.....E..E....ERROR: GenericRegister::GetEntry: nonexistent.so: cannot open shared object file: No such file or directory

ERROR: WeightClass: Unknown weight type: nonexistent

.ERROR: ComplementFst: Argument not an unweighted epsilon-free deterministic acceptor

ERROR: DifferenceFst: 1st argument not an acceptor

.ERROR: AddArc: FST and weight with non-matching weight types: tropical and log

.ERROR: Determinize: FST and weight with non-matching weight types: tropical and log

.ERROR: Disambiguate: FST and weight with non-matching weight types: tropical and log

.ERROR: Prune: FST and weight with non-matching weight types: tropical and log

.ERROR: RmEpsilon: FST and weight with non-matching weight types: tropical and log

.ERROR: SetFinal: FST and weight with non-matching weight types: tropical and log

.....ERROR: ReadFstClass: Can't open file: nonexistent

.....FFFF.....EE..............ERROR: GenericRegister::GetEntry: nonexistent-arc.so: cannot open shared object file: No such file or directory

ERROR: CreateFstClass: Unknown arc type: nonexistent

.ERROR: StrToWeight: Bad weight: nonexistent

.ERROR: CompileSymbolString: FST and weight with non-matching weight types: log and log64

.ERROR: CompileSymbolString: FST and weight with non-matching weight types: tropical and log

.ERROR: StringFstToOutputLabels: State 1 has multiple outgoing arcs

.ERROR: StringToLabels: Unmatched ]

ERROR: Failed to compile string `Red Leicester]`, with token_type: byte

.ERROR: StringToLabels: Unmatched [

ERROR: Failed to compile string `[I'm afraid we're fresh out of Red Leicester sir`, with token_type: byte

...........................................................

================================================================== ERROR: testDowncastTypesAreCorrect (main.DowncastTest)

Traceback (most recent call last): File "pynini_test.py", line 168, in testDowncastTypesAreCorrect f_downcast = Fst.from_pywrapfst(self.f) TypeError: Argument 'fst' has incorrect type (expected _pywrapfst.Fst, got pywrapfst.VectorFst)

================================================================== ERROR: testDowncastedMutationTriggersDeepCopy (main.DowncastTest)

Traceback (most recent call last): File "pynini_test.py", line 172, in testDowncastedMutationTriggersDeepCopy f_downcast = Fst.from_pywrapfst(self.f)

TypeError: Argument 'fst' has incorrect type (expected _pywrapfst.Fst, got pywrapfst.VectorFst)

================================================================== ERROR: testGarbageInputTokenTypeStringPathIteratorRaisesFstArgError (main.ExceptionsTest)

Traceback (most recent call last): File "pynini_test.py", line 284, in testGarbageInputTokenTypeStringPathIteratorRaisesFstArgError unused_sp = StringPathIterator(self.f, input_token_type="nonexistent")

NameError: name 'StringPathIterator' is not defined

================================================================== ERROR: testGarbageOutputTokenTypeStringPathIteratorRaisesFstArgError (main.ExceptionsTest)

Traceback (most recent call last): File "pynini_test.py", line 288, in testGarbageOutputTokenTypeStringPathIteratorRaisesFstArgError unused_sp = StringPathIterator(self.f, output_token_type="nonexistent")

NameError: name 'StringPathIterator' is not defined

================================================================== ERROR: testStringPathLabelsWithEpsilons (main.StringPathIteratorTest)

Traceback (most recent call last): File "pynini_test.py", line 720, in testStringPathLabelsWithEpsilons sp = StringPathIterator(f)

NameError: name 'StringPathIterator' is not defined

================================================================== ERROR: testStringPathsAfterFstDeletion (main.StringPathIteratorTest)

Traceback (most recent call last): File "pynini_test.py", line 710, in testStringPathsAfterFstDeletion sp = StringPathIterator(f)

NameError: name 'StringPathIterator' is not defined

================================================================== FAIL: testByteToByteStringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 593, in testByteToByteStringFile self.ContainsMapping("[Bel Paese]", mapper, "Sorry") File "pynini_test.py", line 589, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID)

AssertionError: -1 == -1

================================================================== FAIL: testByteToSymbolStringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 622, in testByteToSymbolStringFile self.ContainsMapping("[Bel Paese]", mapper, symc("Sorry")) File "pynini_test.py", line 589, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID)

AssertionError: -1 == -1

================================================================== FAIL: testByteToUtf8StringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 601, in testByteToUtf8StringFile self.ContainsMapping("[Bel Paese]", mapper, utf8("Sorry")) File "pynini_test.py", line 589, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID)

AssertionError: -1 == -1

================================================================== FAIL: testUtf8ToUtf8StringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 610, in testUtf8ToUtf8StringFile self.ContainsMapping(utf8("[Bel Paese]"), mapper, utf8("Sorry")) File "pynini_test.py", line 589, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID)

AssertionError: -1 == -1

Ran 145 tests in 0.058s

FAILED (failures=4, errors=6)

Thank you, Michael

MichaelHenderson - 2021-01-06 - 15:21

Update: I rebuilt the Pynini 2.1.3 from source code and confirmed all the extensions were compiled:

/pynini-2.1.3$ python setup.py build_ext --debug install

I eliminated the Pynini tests/pynini_test.py unit test pywrapfst errors by removing the pywrapfst.so that was built and deployed by the openfst installation process:

/usr/local/lib/python3.6/site-packages$ rm -rf pywrapfst.so

I assume I should not have used the --enable-python configure flag when I installed openfst 1.8.0.

The 'import pywrapfst' in pynini_test.py now appears to be resolved from the Pynini 2.1.3 egg:

/usr/local/lib/python3.6/site-packagespynini-2.1.3-py3.6-linux-x86_64.egg

I fixed the 'StringPathIterator not defined' errors by changing way the StringPathIterator constructor is called pynini_tests.py:

def testGarbageInputTokenTypeStringPathIteratorRaisesFstArgError(self):

with self.assertRaises(FstArgError):

#unused_sp = StringPathIterator(self.f, input_token_type="nonexistent")

unused_sp = Fst.paths(self.f, input_token_type="nonexistent")

Why doesn't 'import pynini' allow calling the StringPathIterator() constructor directly instead of through the Fst instance?

I am still getting the same 4 unit test failures from pynini_test.py that I reported earlier:

mhenderson2@solr-1-vm:/mnt/disks/data/pynini-2.1.3/tests$ sudo python pynini_test.py

..ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor

....ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

...ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor .

ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

........ERROR: Verify: FST destination state ID of arc at position 0 of state 0 is negative .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .

ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .............

ERROR: GenericRegister::GetEntry: nonexistent.so: cannot open shared object file: No such file or directory ERROR: WeightClass: Unknown weight type: nonexistent .ERROR: ComplementFst: Argument not an unweighted epsilon-free deterministic acceptor ERROR: DifferenceFst: 1st argument not an acceptor .

ERROR: AddArc: FST and weight with non-matching weight types: tropical and log .

ERROR: Determinize: FST and weight with non-matching weight types: tropical and log

.ERROR: Disambiguate: FST and weight with non-matching weight types: tropical and log

.ERROR: Prune: FST and weight with non-matching weight types: tropical and log

.ERROR: RmEpsilon: FST and weight with non-matching weight types: tropical and log

.ERROR: SetFinal: FST and weight with non-matching weight types: tropical and log

.....ERROR: ReadFstClass: Can't open file: nonexistent .....FFFF.....................

ERROR: GenericRegister::GetEntry: nonexistent-arc.so: cannot open shared object file: No such file or directory

ERROR: CreateFstClass: Unknown arc type: nonexistent .ERROR: StrToWeight: Bad weight: nonexistent .ERROR: CompileSymbolString: FST and weight with non-matching weight types: log and log64

.ERROR: CompileSymbolString: FST and weight with non-matching weight types: tropical and log

.ERROR: StringFstToOutputLabels: State 1 has multiple outgoing arcs

.ERROR: StringToLabels: Unmatched ] ERROR: Failed to compile string `Red Leicester]`, with token_type: byte

.ERROR: StringToLabels: Unmatched [

ERROR: Failed to compile string `[I'm afraid we're fresh out of Red Leicester sir`, with token_type: byte ...........................................................

==================================================================

FAIL: testByteToByteStringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 595, in testByteToByteStringFile self.ContainsMapping("[Bel Paese]", mapper, "Sorry") File "pynini_test.py", line 591, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID) AssertionError: -1 == -1

==================================================================

FAIL: testByteToSymbolStringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 624, in testByteToSymbolStringFile self.ContainsMapping("[Bel Paese]", mapper, symc("Sorry")) File "pynini_test.py", line 591, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID) AssertionError: -1 == -1

==================================================================

FAIL: testByteToUtf8StringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 603, in testByteToUtf8StringFile self.ContainsMapping("[Bel Paese]", mapper, utf8("Sorry")) File "pynini_test.py", line 591, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID) AssertionError: -1 == -1

==================================================================

FAIL: testUtf8ToUtf8StringFile (main.StringFileTest)

Traceback (most recent call last): File "pynini_test.py", line 612, in testUtf8ToUtf8StringFile self.ContainsMapping(utf8("[Bel Paese]"), mapper, utf8("Sorry")) File "pynini_test.py", line 591, in ContainsMapping self.assertNotEqual(lattice.start(), NO_STATE_ID) AssertionError: -1 == -1

Ran 145 tests in 0.058s

FAILED (failures=4)

Has anyone else tried building, installing and running pynini_test.py with any success??

Thank you, Michael

MichaelHenderson - 2021-01-08 - 12:56

MichaelHenderson - 2021-01-08 - 13:03

An additional follow up note:

The testdata directory was missing from the Pynini 2.1.3 distribution so I copied it from the Pynini 2.1.2 distribution:

/pynini-2.1.3/tests/testdata/str.map

More of the unit tests are passing now in pynini_test.py but there are still errors being reported from the CDRewriteRule tests:

/pynini-2.1.3/tests$ sudo python pynini_test.py

..ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor

....ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

...ERROR: CDRewriteRule::Compile: lambda must be a unweighted acceptor

.ERROR: CDRewriteRule::Compile: rho must be a unweighted acceptor

........ERROR: Verify: FST destination state ID of arc at position 0 of state 0 is negative

.ERROR: State ID -1 not valid

.............ERROR: GenericRegister::GetEntry: nonexistent.so: cannot open shared object file: No such file or directory

ERROR: WeightClass: Unknown weight type: nonexistent

.ERROR: ComplementFst: Argument not an unweighted epsilon-free deterministic acceptor

ERROR: DifferenceFst: 1st argument not an acceptor

.ERROR: AddArc: FST and weight with non-matching weight types: tropical and log

.ERROR: Determinize: FST and weight with non-matching weight types: tropical and log

.ERROR: Disambiguate: FST and weight with non-matching weight types: tropical and log

.ERROR: Prune: FST and weight with non-matching weight types: tropical and log

.ERROR: RmEpsilon: FST and weight with non-matching weight types: tropical and log

.ERROR: SetFinal: FST and weight with non-matching weight types: tropical and log

.....ERROR: ReadFstClass: Can't open file: nonexistent

..............................ERROR: GenericRegister::GetEntry: nonexistent-arc.so: cannot open shared object file: No such file or directory

ERROR: CreateFstClass: Unknown arc type: nonexistent

.ERROR: StrToWeight: Bad weight: nonexistent

.ERROR: CompileSymbolString: FST and weight with non-matching weight types: log and log64

.ERROR: CompileSymbolString: FST and weight with non-matching weight types: tropical and log

.ERROR: StringFstToOutputLabels: State 1 has multiple outgoing arcs

.ERROR: StringToLabels: Unmatched ]

ERROR: Failed to compile string `Red Leicester]`, with token_type: byte

.ERROR: StringToLabels: Unmatched [

ERROR: Failed to compile string `[I am afraid we are fresh out of Red Leicester sir`, with token_type: byte

...........................................................

Ran 145 tests in 0.060s

Has anyone seen these errors reported related to the CDRewriteRule unit tests in pynini_test.py?

Thank you, Michael

KyleGorman - 2021-09-25 - 09:05

Hi Michael, sorry but I can't replicate.

FYI: the "ERROR" logs are not failed tests: those are tests that (deliberately) raise error messages.

KyleGorman - 2021-09-25 - 09:07

Hi Michael, sorry but I can't replicate. Could you try the Conda or PyPI installation path instead? (Conda works for Mac OS X and Linux; a fat PyPI precompiled binary with all dependencies is available for Linux x86_64 too.)

FYI: the "ERROR" logs are not failed tests: those are tests that (deliberately) raise error messages.

pywrapfst, undefined symbol

KarelVesely - 2020-10-29 - 14:57

Hello, I am trying to build 'pywrapfst' library for python using openfst 1.7.9. After building it, when i 'import pywrapfst' from python 3.6, i get following error:

ImportError: /mnt/matylda5/iveselyk/SOFTWARE/openfst-1.7.9/lib/python3.6/site-packages/pywrapfst.so: undefined symbol: _ZTVN3fst8internal15SymbolTableImplE

Could it be that something important was not added into the *.so file?

I use this configure run: OPENFST_CONFIGURE="--enable-static --enable-shared --enable-far --enable-ngram-fsts --enable-lookahead-fsts --with-pic --enable-python" ./configure --prefix=$PWD ${OPENFST_CONFIGURE} CXXFLAGS="-g -O2" LDFLAGS="-L/usr/local/lib64 -L/usr/lib64" LIBS="-ldl" PYTHON="python3.6"

And I add the dir with pywrapfst.so to $PYTHONPATH.

Thank you. KV

KarelVesely - 2020-10-29 - 15:01

gcc version is 7.50

KarelVesely - 2020-10-30 - 06:59

I just realized the pywrapfst.so was loading outdated system-wide version of openfst libs. I do a local compilation in my user folder (i don't have root permission).

Adding "-Wl,-rpath=$FST_LIB" to $LDFLAGS in ./configure call solved the problem.

Thanks, KV

Does “fst::script::CompileFstInternal” have memory leakage?

VcZhou - 2020-09-01 - 02:16

gcc version 5.5.0 (Ubuntu 5.5.0-12ubuntu1) openfst-1.6.7

Valgrind memory check tool told me a definitely lost exists in:

void fst::script::CompileFstInternal<fst::ArcTpl<fst::TropicalWeightTpl<float> > >(fst::script::WithReturnValue<fst::script::FstClass*, fst::script::CompileFstInnerArgs>*)

VcZhou - 2020-09-01 - 02:20

I printed out some construction and deconstruction log and found FstClass pointer indeed doesn't been deconstructed.

KyleGorman - 2020-12-03 - 13:51

Does this hold in newer versions? OpenFst 1.6.7 is several years old now and it looks to me (just looked quickly) this has since been fixed, to me.

OpenFST 1.7.7, Pi4 Make fail

ChrisRaven - 2020-06-28 - 03:21

gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1) , Buster Raspbian,

Libraries have been installed in: /usr/local/lib

If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the '-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable during execution - add LIBDIR to the 'LD_RUN_PATH' environment variable during linking - use the '-Wl,-rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages.

make[3]: Nothing to be done for 'install-data-am'. make[3]: Leaving directory '/home/pi/jasper/openfst-1.7.7/src/script' make[2]: Leaving directory '/home/pi/jasper/openfst-1.7.7/src/script' Making install in bin make[2]: Entering directory '/home/pi/jasper/openfst-1.7.7/src/bin' /bin/bash ../../libtool --tag=CXX --mode=link g++ -std=c++11 -fno-exceptions -g -O2 -Wl,-no-as-needed -lm -ldl -o fstarcsort fstarcsort.o fstarcsort-main.o ../script/libfstscript.la ../lib/libfst.la -ldl libtool: link: g++ -std=c++11 -fno-exceptions -g -O2 -Wl,-no-as-needed -o .libs/fstarcsort fstarcsort.o fstarcsort-main.o -lm ../script/.libs/libfstscript.so ../lib/.libs/libfst.so -ldl /usr/bin/ld: ../script/.libs/libfstscript.so: undefined reference to `__atomic_fetch_or_8' /usr/bin/ld: ../script/.libs/libfstscript.so: undefined reference to `__atomic_load_8' /usr/bin/ld: ../script/.libs/libfstscript.so: undefined reference to `__atomic_store_8' collect2: error: ld returned 1 exit status make[2]: * [Makefile:806: fstarcsort] Error 1 make[2]: Leaving directory '/home/pi/jasper/openfst-1.7.7/src/bin' make[1]: * [Makefile:370: install-recursive] Error 1 make[1]: Leaving directory '/home/pi/jasper/openfst-1.7.7/src' make: * [Makefile:426: install-recursive] Error 1 </verbatim> I initially started trying to install a 1.3.4 version but after much poking, prodding and cussing, all my current attempts are with, I've tried the editing make.am files by removing the -lm marker after the LDADD and adding the line: AM_LDFLAGS = -Wl,-no-as-needed -lm -ldl, didn't help

KyleGorman - 2020-08-24 - 11:44

Haven't seen this before; can you replicate on 1.7.9? ALso could you include your GCC version?

Can't compile OpenFst 1.7.7 with Python wrapper: no matching function for call to RandEquivalent

JoanPuigcerver - 2020-06-18 - 20:21

I tried to build OpenFST with the following configure call: $ ./configure --enable-compact-fsts --enable-compress --enable-const-fsts --enable-far --enable-linear-fsts --enable-lookahead-fsts --enable-python --enable-special;

But when I try to compile I get the following error:

<verbatim> $ make [,,,] libtool: compile: g++ -std=c++11 -fno-exceptions -DHAVE_CONFIG_H -I./../../include -I/usr/include/python3.6m -fexceptions -g -O2 -MT pywrapfst_la-pywrapfst.lo -MD -MP -MF .deps/pywrapfst_la-pywrapfst.Tpo -c pywrapfst.cpp -fPIC -DPIC -o .libs/pywrapfst_la-pywrapfst.o pywrapfst.cpp: In function ‘bool __pyx_f_9pywrapfst_randequivalent(__pyx_obj_9pywrapfst_Fst*, __pyx_obj_9pywrapfst_Fst*, int, __pyx_opt_args_9pywrapfst_randequivalent*)’: pywrapfst.cpp:40595:148: error: no matching function for call to ‘RandEquivalent(std::__shared_ptr_access<fst::script::FstClass, (__gnu_cxx::_Lock_policy)2, false, false>::element_type&, std::__shared_ptr_access<fst::script::FstClass, (__gnu_cxx::_Lock_policy)2, false, false>::element_type&, int32&, float&, time_t&, fst::RandGenOptions<fst::script::RandArcSelection>&)’ __pyx_r = fst::script::RandEquivalent((*__pyx_v_ifst1->_fst), (*__pyx_v_ifst2->_fst), __pyx_v_npath, __pyx_v_delta, __pyx_v_seed, (*__pyx_v_opts)); ^ In file included from ./../../include/fst/script/fstscript.h:60:0, from pywrapfst.cpp:657: ./../../include/fst/script/randequivalent.h:25:6: note: candidate: template<class Arc> void fst::script::RandEquivalent(fst::script::RandEquivalentArgs*) void RandEquivalent(RandEquivalentArgs *args) { ^~~~~~~~~~~~~~ ./../../include/fst/script/randequivalent.h:25:6: note: template argument deduction/substitution failed: pywrapfst.cpp:40595:148: note: candidate expects 1 argument, 6 provided __pyx_r = fst::script::RandEquivalent((*__pyx_v_ifst1->_fst), (*__pyx_v_ifst2->_fst), __pyx_v_npath, __pyx_v_delta, __pyx_v_seed, (*__pyx_v_opts)); ^ In file included from ./../../include/fst/script/fstscript.h:60:0, from pywrapfst.cpp:657: ./../../include/fst/script/randequivalent.h:57:6: note: candidate: bool fst::script::RandEquivalent(const fst::script::FstClass&, const fst::script::FstClass&, int32, const fst::RandGenOptions<fst::script::RandArcSelection>&, float, uint64) bool RandEquivalent(const FstClass &fst1, const FstClass &fst2, int32 npath = 1, ^~~~~~~~~~~~~~ ./../../include/fst/script/randequivalent.h:57:6: note: no known conversion for argument 4 from ‘float’ to ‘const fst::RandGenOptions<fst::script::RandArcSelection>&’ pywrapfst.cpp: In function ‘__pyx_obj_9pywrapfst_MutableFst* __pyx_f_9pywrapfst_randgen(__pyx_obj_9pywrapfst_Fst*, int, __pyx_opt_args_9pywrapfst_randgen*)’: pywrapfst.cpp:40929:96: error: no matching function for call to ‘RandGen(std::__shared_ptr_access<fst::script::FstClass, (__gnu_cxx::_Lock_policy)2, false, false>::element_type&, std::unique_ptr<fst::script::VectorFstClass>::pointer, time_t&, fst::RandGenOptions<fst::script::RandArcSelection>&)’ fst::script::RandGen((*__pyx_v_ifst->_fst), __pyx_v_tfst.get(), __pyx_v_seed, (*__pyx_v_opts)); ^ In file included from ./../../include/fst/script/fstscript.h:61:0, from pywrapfst.cpp:657: ./../../include/fst/script/randgen.h:23:6: note: candidate: template<class Arc> void fst::script::RandGen(fst::script::RandGenArgs*) void RandGen(RandGenArgs *args) { ^~~~~~~ ./../../include/fst/script/randgen.h:23:6: note: template argument deduction/substitution failed: pywrapfst.cpp:40929:96: note: candidate expects 1 argument, 4 provided fst::script::RandGen((*__pyx_v_ifst->_fst), __pyx_v_tfst.get(), __pyx_v_seed, (*__pyx_v_opts)); ^ In file included from ./../../include/fst/script/fstscript.h:61:0, from pywrapfst.cpp:657: ./../../include/fst/script/randgen.h:56:6: note: candidate: void fst::script::RandGen(const fst::script::FstClass&, fst::script::MutableFstClass*, const fst::RandGenOptions<fst::script::RandArcSelection>&, uint64) void RandGen(const FstClass &ifst, MutableFstClass *ofst, ^~~~~~~ ./../../include/fst/script/randgen.h:56:6: note: no known conversion for argument 3 from ‘time_t {aka long int}’ to ‘const fst::RandGenOptions<fst::script::RandArcSelection>&’ Makefile:471: recipe for target 'pywrapfst_la-pywrapfst.lo' failed </verbatim>

KyleGorman - 2020-08-24 - 11:45

Does this persist in OpenFst 1.7.9?

Test in configure prevents cross-compiling

NickolayShmyrev - 2020-06-16 - 07:00

The chunk below in configure.ac fails on cross-compiling with

<verbatim> configure: error: in `/opt/kaldi/tools/openfst-1.7.7': configure: error: cannot run test program while cross compiling See `config.log' for more details </verbatim>

it would be nice to check cross-compiling mode and skip it at least.

<verbatim> # Flags may be changed after configuring, so this is checked again by # weight_test.cc. The check here is to save time in the common case, # or when someone does not run `make check`. AC_RUN_IFELSE([AC_LANG_PROGRAM([ #include <cstdio>

template <typename T> bool FloatEqIsReflexive(T m) { volatile T x = 1.111; </verbatim>

NickolayShmyrev - 2020-06-16 - 08:48

A patch like this one should help:

- ]))]) + ]))], + [echo "Ignoring test for cross-compilation"])

https://github.com/alphacep/openfst/commit/256f83e52112a5cd37e37a34beff2c4f0eae4660.diff

How to get state table in ComposeFstImpl

StevenWang - 2020-04-20 - 23:47

Hi, all. I want to get the state table in ComposeFstImpl and I don't want to revise the code of openfst directly. So is there an easy way to write some code outside to get it?

KyleGorman - 2020-08-24 - 11:48

If you want access to the state table, you construct it yourself, pass a pointer to it to construct a ComposeFstOptions struct, and then pass that ComposeFstOptions struct to ComposeFst(Impl). There's an example of this in OpenGrm-BaumWelch: http://opengrm.org/doxygen/baumwelch/html/cascade_8h_source.html

look up failed in shared object: olabel_lookahead-fst.so

HapZhang - 2020-03-25 - 05:50

Hi,all I config openfst-1.6.1 with option --enable-lookahead-fsts and build&install recently. I got a error when read a Hclr.fst by the code "hcl_fst = fst::StdFst::Read(hcl_fst_rxfilename)",the error as list: ERROR:GenericRegister::GetEntry:lookup failed in shared object:olabel_lookahead-fst.so. ERROR:Fst::Read:Unknown FST type olabel_lookahead(arc type = standard): <unspecified>.Could anybody tell me what can i do to solve this?

SymbolTable problems in openfst-1.7.6

AdamJanin - 2020-02-28 - 13:40

The following program dumps core in AddSymbol in openfst-1.7.6 but works correctly in openfst-1.7.5 (and 1.6.7).

<verbatim> #include <fst/symbol-table.h>

int main() { fst::SymbolTable *syms = fst::SymbolTable::ReadText("some.syms"); syms->WriteText(std::cout); int64 i0 = syms->AddSymbol("foo"); syms->WriteText(std::cout); } </verbatim>

<verbatim> > g++ --version g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 </verbatim>

KyleGorman - 2020-08-24 - 11:45

Try upgrading (say to 1.7.9) and let us know if this is not resolved by that.

error: 'SplitToVector' is not a member of 'fst'

GeSang - 2020-02-18 - 02:59

I use 'make -j 16' to compile thrax, appearing the following error.How to do? n file included from walker/loader.cc:47:0: ./../include/thrax/stringfile.h: In member function 'bool thrax::function::StringFile<Arc>::ConvertStringToLabels(const string&, std::vector<typename Arc::Label>*, int, const fst::SymbolTable*) const': ./../include/thrax/stringfile.h:190:7: error: 'SplitToVector' is not a member of 'fst' fst::SplitToVector(c_str, separator.c_str(), &vec, true);

KyleGorman - 2020-04-12 - 14:42

Use the newest versions of both OpenFst and Thrax...the function was renamed a few releases ago.

Error when using "pip3 install openfst" on macOs, need HELP

AnqiWang - 2020-01-21 - 09:22

My computer information: "macOS catalina version 10.15.2 xcode: 11.3.1 clang: 11.0.0 python: 3.7 (install by homebrew)"

Instruction I followed: "./configure --enable-python --enable-far make make install"

Everything is great but I cannot import pywrapfst Then I use "pip3 install openfst" and get following error

Installing collected packages: openfst Running setup.py install for openfst ... error ERROR: Command errored out with exit status 1: command: /usr/local/opt/python/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-install-3l5b46rq/openfst/setup.py'"'"'; file__='"'"'/private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-install-3l5b46rq/openfst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-record-8e3n_zsl/install-record.txt --single-version-externally-managed --compile

cwd: /private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-install-3l5b46rq/openfst/ Complete output (526 lines): running install running build running build_ext building 'pywrapfst' extension creating build creating build/temp.macosx-10.15-x86_64-3.7 clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c pywrapfst.cc -o build/temp.macosx-10.15-x86_64-3.7/pywrapfst.o -std=c++11 -Wno-unneeded-internal-declaration -Wno-unused-function In file included from pywrapfst.cc:595: In file included from /usr/local/include/fst/fstlib.h:28: In file included from /usr/local/include/fst/expanded-fst.h:19: In file included from /usr/local/include/fst/fst.h:30: /usr/local/include/fst/symbol-table.h:243:24: warning: comparison of integers of different signs: 'ssize_t' (aka 'long') and 'size_t' (aka 'unsigned long') [-Wsign-compare] if (pos < 0 || pos >= symbols_.Size()) return kNoSymbol; ~~~ ^ ~~~~~~~~~~~~~~~ /usr/local/include/fst/symbol-table.h:515:36: warning: comparison of integers of different signs: 'const ssize_t' (aka 'const long') and 'const size_t' (aka 'const unsigned long') [-Wsign-compare] bool Done() const { return (pos_ = nsymbols_); } ~~~~ ^ ~~~~~~~~~ /usr/local/include/fst/symbol-table.h:526:14: warning: comparison of integers of different signs: 'ssize_t' (aka 'long') and 'size_t' (aka 'unsigned long') [-Wsign-compare] if (pos_ < nsymbols_) iter_item_.SetPosition(pos_); ~~~~ ^ ~~~~~~~~~ In file included from pywrapfst.cc:595: In file included from /usr/local/include/fst/fstlib.h:34: /usr/local/include/fst/const-fst.h:406:25: warning: comparison of integers of different signs: 'int64' (aka 'long long') and 'size_t' (aka 'unsigned long') [-Wsign-compare] if (hdr.NumStates() num_states) { ~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~ /usr/local/include/fst/const-fst.h:410:23: warning: comparison of integers of different signs: 'int64' (aka 'long long') and 'size_t' (aka 'unsigned long') [-Wsign-compare] if (hdr.NumArcs() = num_arcs) { ~~~~~~~~~~~~~ ^ ~~~~~~~~ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:89:5: warning: unused variable 'arc_dispatched_operation_ArcSortArgsArcSortArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(ArcSort, Arc, ArcSortArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:156:1: note: expanded from here arc_dispatched_operation_ArcSortArgsArcSortArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:90:5: warning: unused variable 'arc_dispatched_operation_ClosureArgsClosureArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Closure, Arc, ClosureArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:161:1: note: expanded from here arc_dispatched_operation_ClosureArgsClosureArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:91:5: warning: unused variable 'arc_dispatched_operation_CompileFstArgsCompileFstInternalArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(CompileFstInternal, Arc, CompileFstArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:166:1: note: expanded from here arc_dispatched_operation_CompileFstArgsCompileFstInternalArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:92:5: warning: unused variable 'arc_dispatched_operation_ComposeArgsComposeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Compose, Arc, ComposeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:171:1: note: expanded from here arc_dispatched_operation_ComposeArgsComposeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:93:5: warning: unused variable 'arc_dispatched_operation_ConcatArgs1ConcatArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Concat, Arc, ConcatArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:176:1: note: expanded from here arc_dispatched_operation_ConcatArgs1ConcatArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:94:5: warning: unused variable 'arc_dispatched_operation_ConcatArgs2ConcatArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Concat, Arc, ConcatArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:181:1: note: expanded from here arc_dispatched_operation_ConcatArgs2ConcatArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:95:5: warning: unused variable 'arc_dispatched_operation_ConcatArgs3ConcatArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Concat, Arc, ConcatArgs3); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:186:1: note: expanded from here arc_dispatched_operation_ConcatArgs3ConcatArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:96:5: warning: unused variable 'arc_dispatched_operation_MutableFstClassConnectArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Connect, Arc, MutableFstClass); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:191:1: note: expanded from here arc_dispatched_operation_MutableFstClassConnectArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:97:5: warning: unused variable 'arc_dispatched_operation_ConvertArgsConvertArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Convert, Arc, ConvertArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:196:1: note: expanded from here arc_dispatched_operation_ConvertArgsConvertArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:98:5: warning: unused variable 'arc_dispatched_operation_DecodeArgsDecodeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Decode, Arc, DecodeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:201:1: note: expanded from here arc_dispatched_operation_DecodeArgsDecodeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:99:5: warning: unused variable 'arc_dispatched_operation_DeterminizeArgsDeterminizeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Determinize, Arc, DeterminizeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:206:1: note: expanded from here arc_dispatched_operation_DeterminizeArgsDeterminizeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:100:5: warning: unused variable 'arc_dispatched_operation_DifferenceArgsDifferenceArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Difference, Arc, DifferenceArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:211:1: note: expanded from here arc_dispatched_operation_DifferenceArgsDifferenceArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:101:5: warning: unused variable 'arc_dispatched_operation_DisambiguateArgsDisambiguateArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Disambiguate, Arc, DisambiguateArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:216:1: note: expanded from here arc_dispatched_operation_DisambiguateArgsDisambiguateArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:102:5: warning: unused variable 'arc_dispatched_operation_DrawArgsDrawArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Draw, Arc, DrawArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:221:1: note: expanded from here arc_dispatched_operation_DrawArgsDrawArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:103:5: warning: unused variable 'arc_dispatched_operation_EncodeArgsEncodeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Encode, Arc, EncodeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:226:1: note: expanded from here arc_dispatched_operation_EncodeArgsEncodeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:104:5: warning: unused variable 'arc_dispatched_operation_EpsNormalizeArgsEpsNormalizeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(EpsNormalize, Arc, EpsNormalizeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:4:1: note: expanded from here arc_dispatched_operation_EpsNormalizeArgsEpsNormalizeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:105:5: warning: unused variable 'arc_dispatched_operation_EqualArgsEqualArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Equal, Arc, EqualArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:9:1: note: expanded from here arc_dispatched_operation_EqualArgsEqualArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:106:5: warning: unused variable 'arc_dispatched_operation_EquivalentArgsEquivalentArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Equivalent, Arc, EquivalentArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:14:1: note: expanded from here arc_dispatched_operation_EquivalentArgsEquivalentArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:107:5: warning: unused variable 'arc_dispatched_operation_InitArcIteratorClassArgsInitArcIteratorClassArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(InitArcIteratorClass, Arc, ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:19:1: note: expanded from here arc_dispatched_operation_InitArcIteratorClassArgsInitArcIteratorClassArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:109:5: warning: unused variable 'arc_dispatched_operation_InitMutableArcIteratorClassArgsInitMutableArcIteratorClassArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(InitMutableArcIteratorClass, Arc, ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:24:1: note: expanded from here arc_dispatched_operation_InitMutableArcIteratorClassArgsInitMutableArcIteratorClassArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:111:5: warning: unused variable 'arc_dispatched_operation_InitStateIteratorClassArgsInitStateIteratorClassArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(InitStateIteratorClass, Arc, ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:29:1: note: expanded from here arc_dispatched_operation_InitStateIteratorClassArgsInitStateIteratorClassArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:113:5: warning: unused variable 'arc_dispatched_operation_InfoArgsInfoArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Info, Arc, InfoArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:34:1: note: expanded from here arc_dispatched_operation_InfoArgsInfoArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:114:5: warning: unused variable 'arc_dispatched_operation_IntersectArgsIntersectArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Intersect, Arc, IntersectArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:39:1: note: expanded from here arc_dispatched_operation_IntersectArgsIntersectArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:115:5: warning: unused variable 'arc_dispatched_operation_MutableFstClassInvertArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Invert, Arc, MutableFstClass); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:44:1: note: expanded from here arc_dispatched_operation_MutableFstClassInvertArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:116:5: warning: unused variable 'arc_dispatched_operation_IsomorphicArgsIsomorphicArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Isomorphic, Arc, IsomorphicArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:49:1: note: expanded from here arc_dispatched_operation_IsomorphicArgsIsomorphicArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:120:5: warning: unused variable 'arc_dispatched_operation_MapArgsMapArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Map, Arc, MapArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:54:1: note: expanded from here arc_dispatched_operation_MapArgsMapArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:121:5: warning: unused variable 'arc_dispatched_operation_MinimizeArgsMinimizeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Minimize, Arc, MinimizeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:59:1: note: expanded from here arc_dispatched_operation_MinimizeArgsMinimizeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:122:5: warning: unused variable 'arc_dispatched_operation_PrintArgsPrintArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Print, Arc, PrintArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:64:1: note: expanded from here arc_dispatched_operation_PrintArgsPrintArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:123:5: warning: unused variable 'arc_dispatched_operation_ProjectArgsProjectArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Project, Arc, ProjectArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:69:1: note: expanded from here arc_dispatched_operation_ProjectArgsProjectArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:124:5: warning: unused variable 'arc_dispatched_operation_PruneArgs1PruneArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Prune, Arc, PruneArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:74:1: note: expanded from here arc_dispatched_operation_PruneArgs1PruneArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:125:5: warning: unused variable 'arc_dispatched_operation_PruneArgs2PruneArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Prune, Arc, PruneArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:79:1: note: expanded from here arc_dispatched_operation_PruneArgs2PruneArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:126:5: warning: unused variable 'arc_dispatched_operation_PushArgs1PushArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Push, Arc, PushArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:84:1: note: expanded from here arc_dispatched_operation_PushArgs1PushArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:127:5: warning: unused variable 'arc_dispatched_operation_PushArgs2PushArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Push, Arc, PushArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:89:1: note: expanded from here arc_dispatched_operation_PushArgs2PushArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:128:5: warning: unused variable 'arc_dispatched_operation_RandEquivalentArgsRandEquivalentArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(RandEquivalent, Arc, RandEquivalentArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:3:1: note: expanded from here arc_dispatched_operation_RandEquivalentArgsRandEquivalentArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:129:5: warning: unused variable 'arc_dispatched_operation_RandGenArgsRandGenArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(RandGen, Arc, RandGenArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:8:1: note: expanded from here arc_dispatched_operation_RandGenArgsRandGenArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:130:5: warning: unused variable 'arc_dispatched_operation_RelabelArgs1RelabelArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Relabel, Arc, RelabelArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:13:1: note: expanded from here arc_dispatched_operation_RelabelArgs1RelabelArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:131:5: warning: unused variable 'arc_dispatched_operation_RelabelArgs2RelabelArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Relabel, Arc, RelabelArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:18:1: note: expanded from here arc_dispatched_operation_RelabelArgs2RelabelArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:132:5: warning: unused variable 'arc_dispatched_operation_ReplaceArgsReplaceArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Replace, Arc, ReplaceArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:23:1: note: expanded from here arc_dispatched_operation_ReplaceArgsReplaceArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:133:5: warning: unused variable 'arc_dispatched_operation_ReverseArgsReverseArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Reverse, Arc, ReverseArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:28:1: note: expanded from here arc_dispatched_operation_ReverseArgsReverseArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:134:5: warning: unused variable 'arc_dispatched_operation_ReweightArgsReweightArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Reweight, Arc, ReweightArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:33:1: note: expanded from here arc_dispatched_operation_ReweightArgsReweightArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:135:5: warning: unused variable 'arc_dispatched_operation_RmEpsilonArgsRmEpsilonArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(RmEpsilon, Arc, RmEpsilonArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:38:1: note: expanded from here arc_dispatched_operation_RmEpsilonArgsRmEpsilonArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:136:5: warning: unused variable 'arc_dispatched_operation_ShortestDistanceArgs1ShortestDistanceArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(ShortestDistance, Arc, ShortestDistanceArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:43:1: note: expanded from here arc_dispatched_operation_ShortestDistanceArgs1ShortestDistanceArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:137:5: warning: unused variable 'arc_dispatched_operation_ShortestDistanceArgs2ShortestDistanceArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(ShortestDistance, Arc, ShortestDistanceArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:48:1: note: expanded from here arc_dispatched_operation_ShortestDistanceArgs2ShortestDistanceArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:138:5: warning: unused variable 'arc_dispatched_operation_ShortestPathArgsShortestPathArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(ShortestPath, Arc, ShortestPathArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:53:1: note: expanded from here arc_dispatched_operation_ShortestPathArgsShortestPathArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:139:5: warning: unused variable 'arc_dispatched_operation_SynchronizeArgsSynchronizeArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Synchronize, Arc, SynchronizeArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:58:1: note: expanded from here arc_dispatched_operation_SynchronizeArgsSynchronizeArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:140:5: warning: unused variable 'arc_dispatched_operation_TopSortArgsTopSortArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(TopSort, Arc, TopSortArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:63:1: note: expanded from here arc_dispatched_operation_TopSortArgsTopSortArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:141:5: warning: unused variable 'arc_dispatched_operation_UnionArgs1UnionArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Union, Arc, UnionArgs1); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:68:1: note: expanded from here arc_dispatched_operation_UnionArgs1UnionArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:142:5: warning: unused variable 'arc_dispatched_operation_UnionArgs2UnionArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Union, Arc, UnionArgs2); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:73:1: note: expanded from here arc_dispatched_operation_UnionArgs2UnionArc_registerer ^ In file included from pywrapfst.cc:596: /usr/local/include/fst/script/fstscript.h:143:5: warning: unused variable 'arc_dispatched_operation_VerifyArgsVerifyArc_registerer' [-Wunused-variable] REGISTER_FST_OPERATION(Verify, Arc, VerifyArgs); ^ /usr/local/include/fst/script/script-impl.h:150:7: note: expanded from macro 'REGISTER_FST_OPERATION' arc_dispatched_operation_##ArgPack##Op##Arc##_registerer ^ <scratch space>:78:1: note: expanded from here arc_dispatched_operation_VerifyArgsVerifyArc_registerer ^ pywrapfst.cc:17194:16: error: no member named 'DrawFst' in namespace 'fst::script' fst::script::DrawFst((*__pyx_v_self->_fst), __pyx_v_self->_fst.get()->InputSymbols(), __pyx_v_self->_fst.get()->OutputSymbols(), NULL, (__pyx_v_self->_fst.get()->Properties(fst::kAcceptor, 1) == fst::kAcceptor), __pyx_k__24, 8.5, 11.0, 1, 0, 0.4, 0.25, 14, 5, __pyx_k_g, 0, (&__pyx_v_sstrm), __pyx_k_repr_svg); ~~~~~~~~~~~~~^ pywrapfst.cc:18796:16: error: no member named 'DrawFst' in namespace 'fst::script' fst::script::DrawFst((*__pyx_v_self->_fst), __pyx_t_21, __pyx_t_22, __pyx_v_ssymbols_ptr, __pyx_v_acceptor, __pyx_t_17, __pyx_v_width, __pyx_v_height, __pyx_v_portrait, __pyx_v_vertical, __pyx_v_ranksep, __pyx_v_nodesep, __pyx_v_fontsize, __pyx_v_precision, __pyx_t_23, __pyx_v_show_weight_one, __pyx_v_ostrm.get(), __pyx_v_filename_string); ~~~~~~~~~~~~~^ 54 warnings and 2 errors generated. error: command 'clang' failed with exit status 1 ----------------------------------------

ERROR: Command errored out with exit status 1: /usr/local/opt/python/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-install-3l5b46rq/openfst/setup.py'"'"'; file__='"'"'/private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-install-3l5b46rq/openfst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/5d/hq7yj4_x1wngp7zqmxs2ylw80000gn/T/pip-record-8e3n_zsl/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

KyleGorman - 2020-02-12 - 11:51

If you compile with `--enable-pywrapfst` and `make install` completes, then it has installed `pywrapfst.so` somewhere on your system. (It uses autoconf macros to guess where to install.) You have to track it down (though it'll be logged by the installation) and make sure it's in your Python import path.

Don't install from pip if you built Pywrapfst yourself.

OpenFST version number

JonWebb - 2020-01-07 - 10:17

I need to make some code that will compile with old and new versions of OpenFST. Is there an OpenFST version number defined somewhere in the header files so I can deail with non-backwards compatible changes (i.e., moving FstImpl to the internal namespace in 1.6.0)?

KyleGorman - 2020-02-12 - 11:49

Sorry, we don't have any mechanisms in place to support that sort of thing.

Please change FLAGS_v to something else

RudolfABraun - 2019-12-18 - 09:09

Using openfst while also using pytorch in a project is extremely annoying due to both using FLAGS_v. I've got around it by making changes but it would be nice if this problem could be fixed at the source.

No Program Flags when building in cygwin

JonasE - 2019-09-27 - 08:52

I compiled openfst version 1.7.2 with cygwin like this: export CFLAGS=-D_POSIX_SOURCE export CXXFLAGS="-O -D_POSIX_SOURCE"

./configure --enable-grm

make LDFLAGS=-no-undefined make install

However, when I try to run the command line tools, all the program flags are missing. Here I want to use the --symbols flag but it is not recognized and not found in the help too:

$farcompilestrings.exe --help Compiles a set of strings as FSTs and stores them in a finite-state archive.

Usage:farcompilestrings [in1.txt [[in2.txt ...] out.far]]

PROGRAM FLAGS:

LIBRARY FLAGS:

Flags from: flags.cc --help: type = bool, default = false show usage information --helpshort: type = bool, default = false show brief usage information --tmpdir: type = string, default = "/tmp" temporary directory .........................................................

I found that his was a known issue and was fixed in 1.6.8, however I have this behavior with that version too. I wasn't able to test 1.7.3 due to a compiling error.

Any idea what I am missing here?

fst file cannot be minimized

ZheWang - 2019-09-18 - 01:46

Hi,

I got a fst file that contains the following contents. When I tried to call fstminimize to minimize it, the program would not stop and never returned a result. Is there anything wrong with the fst file?

Thanks!

The fst file's content:

0 1 0 0 -1.94591022 0 2 5 1 -0.693147182 0 3 6 4 0 4 8 5 0 5 12 3 -1.09861231 1 6 5 1 -0.693147182 1 7 6 4 -1.38629436 1 8 8 5 -1.09861231 1 9 12 3 -1.09861231 1 10 13 2 -0.693147182 1 -1.9459101 2 11 4 0 3 12 11 0 4 13 4 0 5 14 5 0 6 15 4 0 7 16 11 0 8 17 4 0 9 18 5 0 10 19 9 0 11 20 7 0 12 16 0 0 12 21 8 5 13 17 0 0 13 14 22 6 0 15 23 7 0 16 1 0 0 -1.38629436 16 21 8 5 -0.693147182 16 24 13 2 16 17 1 0 0 -1.09861231 17 -1.09861231 18 25 6 0 19 26 10 0 20 27 7 0 21 28 4 0 22 29 11 0 23 30 7 0 24 31 9 0 25 32 11 0 26 33 7 0 27 34 9 0 28 17 0 0 -0.693147182 28 -0.693147182 29 32 0 0 -1.09861231 29 35 6 4 -1.09861231 30 36 9 0 31 37 10 0 32 1 0 0 -1.09861231 32 35 6 4 -1.09861231 33 38 3 0 34 36 0 0 -0.693147182 34 39 13 2 34 35 40 11 0 36 1 0 0 -0.693147182 36 39 13 2 36 37 41 7 0 38 1 0 0 -0.693147182 38 -0.693147182 39 42 9 0 40 16 0 0 -1.09861231 40 21 8 5 40 24 13 2 40 41 43 3 0 42 44 10 0 43 38 0 0 43 44 45 7 0 45 46 3 0 46 38 0 0 46

ZheWang - 2019-09-18 - 01:48

Sorry that the content might not be correctly shown. The full file can be found here: https://raw.githubusercontent.com/zhwa/notes/master/Miscellaneous/sample.fst.txt

KyleGorman - 2019-11-03 - 12:08

Can confirm that's the case for me too.

CyrilAllauzen - 2019-11-06 - 01:21

The first step of minimization is to normalize the distribution of the weights along the paths using weight pushing. Weight pushing itself calls ShortestDistance which does not terminate on this example because there are cycles with (strictly) negative weights.

Since there are no cycles with (strictly) positive weights in this Fst, one could negate all the weights, minimize and negate all the weights in the result, essentially minimizing in the (max, +) semiring:

fstmap --map_type=invert sample.fst  | fstminimize | fstmap --map_type=invert > sample.min.fst

Epsilon transition handling

AndreyAbramov - 2019-08-29 - 12:04

Hi,

As far as I can see not SortedMatcher nor HashMatcher don't handle epsilon transitions. Both don't see explicit transitions. Should I remove all epsilon transition before matching or write my own matcher? In case of latter why "default" matchers behave such a way?

Best, Andrey

KyleGorman - 2019-11-03 - 12:07

The default matcher is SortedMatcher when at least one side of the composition is arc-sorted, and it does support epsilons (if matching on the input side then epsilsons match an imlicit self loop in addition to any actually epsilon transitions, and respectively when matching on the output side).

Range handling

AndreyAbramov - 2019-08-22 - 15:51

Hi,

How can I build an acceptor to properly handle ranges, e.g. "a[b..c]d"? Of course the obvious solution would be to generate enough arcs to handle every value within a range, but it can be optimized, e.g. with the custom arc type. Any suggestions?

Best, Andrey

MichaelRiley - 2019-08-28 - 16:26

You could create a matcher (see doc) that treats certain labels as ranges. Labels need to be integers but you encode the label range in there (e.g. high bits for begin vs low bits for end). This could then be used with composition to effect the range interpretation. It'd be somewhat similar to how e.g. RhoMatcher is written where a new label represents the complement of the other labels on transitions at a state.

AndreyAbramov - 2019-08-29 - 06:23

Thanks for the confirmation, MichaelRiley, that's what I expected smile

Epsilon is hardcoded to 0

AndreyAbramov - 2019-08-22 - 15:46

Hi,

Why epsilon is hardcoded to 0? Given unicode code points are in range `[0x0 .. 0x10ffff]` it seems to be impossible to correctly process min code point? Am I right?

Best, Andrey

KyleGorman - 2019-11-03 - 12:02

A Unicode string with character(0), the null byte '\0', in the middle, is not really a valid Unicode string.

AndreyAbramov - 2020-08-10 - 07:18

Yes, but `\0` is a valid single-byte UTF-8 character, isn't it?

Pynini on PyPI - Function as a Service

BenoitThiery - 2019-07-02 - 06:43

Hi,

I have to migrate a Python software using pynini to a public cloud. I wish I could use Function as a Service, but that requires all dependencies to be available on PyPI.

Do you plan to make pynini available on PyPI?

Thanks a lot,

Benoit

KyleGorman - 2019-07-07 - 22:00

There are two ways to do this:

* I could put the source up there anyways but it won't work unless you have both current headers and libs installed in the normal location. * Alternatively I coudl set it up so that it first downloads and compiled a current OpenFst as part of the process. I looked into doing this and I couldn't find any useful documentation on how to go about this, but I would welcome help.

In the meantime some of my colleagues have had good look setting it up inside Docker.

KyleGorman - 2020-02-12 - 11:55

Pynini is now available on Conda-Forge for Linux and Mac OS X. This system builds the code and performs continuous integration testing, so that installation is as simple as downloading and moving the assets to the correct paths. (There's nothing similar for PyPI.) If you have Conda (Anaconda or Miniconda) installed, issue the following command:

conda install -c conda-forge pynini

Vagrant environment for a quick OpenFST /OpenGRM setup

WincentBalin - 2019-05-01 - 18:44

Recently I published a Vagrant environment for a quick OpenFST/OpenGRM setup at https://github.com/wincentbalin/opengrm-vagrant. The whole package compilation process was put into a separate shell script.

If you already use Vagrant and need a quick automated setup of OpenFST/OpenGRM libraries and tools, feel free to check it out!

Compilatoin problems CentOS

EstherJudd - 2019-02-05 - 18:19

CentOS gcc 4.8.5 export CXX="g++ -std=c++11"

openfst 1.7.1 ./configure --enable-compact-fsts --enable-const-fsts --enable-far --enable-lookahead-fsts --enable-pdt --enable-mpdt --enable-ngram-fsts --enable-python --prefix=/home/user/scripts

/bin/sh ../../libtool --tag=CXX --mode=compile g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT compat.lo -MD -MP -MF $depbase.Tpo -c -o compat.lo compat.cc &&mv -f $depbase.Tpo $depbase.Plo libtool: compile: g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT compat.lo -MD -MP -MF .deps/compat.Tpo -c compat.cc -fPIC -DPIC -o .libs/compat.o depbase=`echo flags.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;/bin/sh ../../libtool --tag=CXX --mode=compile g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT flags.lo -MD -MP -MF $depbase.Tpo -c -o flags.lo flags.cc &&mv -f $depbase.Tpo $depbase.Plo libtool: compile: g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT flags.lo -MD -MP -MF .deps/flags.Tpo -c flags.cc -fPIC -DPIC -o .libs/flags.o depbase=`echo fst.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;/bin/sh ../../libtool --tag=CXX --mode=compile g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT fst.lo -MD -MP -MF $depbase.Tpo -c -o fst.lo fst.cc &&mv -f $depbase.Tpo $depbase.Plo libtool: compile: g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT fst.lo -MD -MP -MF .deps/fst.Tpo -c fst.cc -fPIC -DPIC -o .libs/fst.o In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:389:53: error: redeclaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::TropicalWeightTpl<float>; size_t = long unsigned int]’ differs in ‘constexpr’ const TropicalWeightTpl<float> &weight, size_t n) { ^ In file included from ./../include/fst/pair-weight.h:18:0, from ./../include/fst/expectation-weight.h:24, from ./../include/fst/arc.h:15, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/weight.h:204:8: error: from previous declaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::TropicalWeightTpl<float>; size_t = long unsigned int]’ Weight Power(const Weight &weight, size_t n) { ^ In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:395:54: error: redeclaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::TropicalWeightTpl<double>; size_t = long unsigned int]’ differs in ‘constexpr’ const TropicalWeightTpl<double> &weight, size_t n) { ^ In file included from ./../include/fst/pair-weight.h:18:0, from ./../include/fst/expectation-weight.h:24, from ./../include/fst/arc.h:15, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/weight.h:204:8: error: from previous declaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::TropicalWeightTpl<double>; size_t = long unsigned int]’ Weight Power(const Weight &weight, size_t n) { ^ In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:573:48: error: redeclaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::LogWeightTpl<float>; size_t = long unsigned int]’ differs in ‘constexpr’ const LogWeightTpl<float> &weight, size_t n) { ^ In file included from ./../include/fst/pair-weight.h:18:0, from ./../include/fst/expectation-weight.h:24, from ./../include/fst/arc.h:15, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/weight.h:204:8: error: from previous declaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::LogWeightTpl<float>; size_t = long unsigned int]’ Weight Power(const Weight &weight, size_t n) { ^ In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:579:49: error: redeclaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::LogWeightTpl<double>; size_t = long unsigned int]’ differs in ‘constexpr’ const LogWeightTpl<double> &weight, size_t n) { ^ In file included from ./../include/fst/pair-weight.h:18:0, from ./../include/fst/expectation-weight.h:24, from ./../include/fst/arc.h:15, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/weight.h:204:8: error: from previous declaration ‘Weight fst::Power(const Weight&, size_t) [with Weight = fst::LogWeightTpl<double>; size_t = long unsigned int]’ Weight Power(const Weight &weight, size_t n) { ^ make[3]: * [fst.lo] Error 1 make[3]: Leaving directory `/home/user/openfst-1.7.1/src/lib' make[2]: * [all-recursive] Error 1 make[2]: Leaving directory `/home/user/openfst-1.7.1/src' make[1]: * [all-recursive] Error 1 make[1]: Leaving directory `/home/user/openfst-1.7.1' make: * [all] Error 2

openfst 1.7.0

openfst 1.6.9 compiles correctly, but then I run into the problem of compiling a compatible version of opengrm? Which version should work with it? I tried 1.3.3-5 and they all give this error:

/bin/sh ../../libtool --tag=CXX --mode=compile g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT ngram-context.lo -MD -MP -MF .deps/ngram-context.Tpo -c -o ngram-context.lo ngram-context.cc libtool: compile: g++ -std=c++11 -DHAVE_CONFIG_H -I./../include -fno-exceptions -funsigned-char -std=c++11 -MT ngram-context.lo -MD -MP -MF .deps/ngram-context.Tpo -c ngram-context.cc -fPIC -DPIC -o .libs/ngram-context.o ngram-context.cc: In static member function ‘static void ngram::NGramContext::ParseContextInterval(const string&, std::vector<int>*, std::vector<int>*)’: ngram-context.cc:128:3: error: ‘SplitString’ is not a member of ‘fst’ fst::SplitString(line, ":", &contexts, true); ^ ngram-context.cc:131:3: error: ‘SplitString’ is not a member of ‘fst’ fst::SplitString(contexts[0], " ", &labels1, true); ^ ngram-context.cc:132:3: error: ‘SplitString’ is not a member of ‘fst’ fst::SplitString(contexts[1], " ", &labels2, true); ^ ngram-context.cc: In static member function ‘static void ngram::NGramExtendedContext::ParseContextIntervals(const string&, int, std::vector<ngram::NGramContext>*)’: ngram-context.cc:234:3: error: ‘SplitString’ is not a member of ‘fst’ fst::SplitString(line.get(), ",", &context_patterns, true); ^ make[3]: * [ngram-context.lo] Error 1 make[3]: Leaving directory `/home/user/opengrm-ngram-1.3.5/src/lib' make[2]: * [all-recursive] Error 1 make[2]: Leaving directory `/home/user/opengrm-ngram-1.3.5/src' make[1]: * [all-recursive] Error 1 make[1]: Leaving directory `/home/user/opengrm-ngram-1.3.5' make: * [all] Error 2

NickolayShmyrev - 2020-06-23 - 02:53

Same problem

As far as I understand you simply need newer gcc, that old one doesn't work. the patch like this helps with older gcc but breaks newer ones:

a/src/include/fst/weight.h +++ b/src/include/fst/weight.h @@ -202,7 +202,7 @@ class NaturalLess<W, typename std::enable_if<!IsIdempotent<W>::value>::type> { // Power is the iterated product for arbitrary semirings such that Power(w, 0) // is One() for the semiring, and Power(w, n) = Times(Power(w, n - 1), w). template <class Weight> -Weight Power(const Weight &weight, size_t n) { +constexpr Weight Power(const Weight &weight, size_t n) { auto result = Weight::One(); for (size_t i = 0; i < n; ++i) result = Times(result, weight); return result;

Non-constant assertion for static assertion

AaronChantrill - 2018-12-28 - 17:45

I'm trying to build OpenFST 1.7.0 for use on a Raspberry Pi. I'm using the following configure:

./configure --enable-static --enable-shared --enable-far --enable-lookahead-fsts --enable-const-fsts --enable-pdt --enable-ngram-fsts --enable-linear-fsts --prefix=/usr

and getting the following error back:

In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:229:1: error: non-constant condition for static assertion static_assert(TropicalWeight::NoWeight().Member(), "NoWeight not member"); ^~~~~~~~~~~~~ ./../include/fst/float-weight.h:229:49: in constexpr expansion of ‘fst::TropicalWeightTpl<T>::NoWeight<float>().fst::TropicalWeightTpl<T>::Member<float>()’ ./../include/fst/float-weight.h:208:20: error: ‘(+QNaNf >= -3.40282347e+38f)’ is not a constant expression return Value() > Limits::NegInfinity(); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~ Makefile:460: recipe for target 'fst.lo' failed

At first I thought this was an architecture issue, but when I tried building on an x86_32 and my primary x86_64 system I got the exact same issue. I've searched the internet and read through the INSTALL, README and NEWS, but don't see anything specifically addressing this. Do I need to be passing an additional argument to the compiler?

Thanks!

AaronChantrill - 2018-12-28 - 17:53

Additional: I guess I can't edit a post after submitting it.

The error occurs during "make" not during configure.

<verbatim> In file included from ./../include/fst/arc.h:16:0, from ./../include/fst/fst.h:26, from fst.cc:6: ./../include/fst/float-weight.h:229:1: error: non-constant condition for static assertion static_assert(TropicalWeight::NoWeight().Member(), "NoWeight not member"); ^~~~~~~~~~~~~ ./../include/fst/float-weight.h:229:49: in constexpr expansion of ‘fst::TropicalWeightTpl<T>::NoWeight<float>().fst::TropicalWeightTpl<T>::Member<float>()’ ./../include/fst/float-weight.h:208:20: error: ‘(+QNaNf >= -3.40282347e+38f)’ is not a constant expression return Value() > Limits::NegInfinity(); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~ Makefile:460: recipe for target 'fst.lo' failed </verbatim>

KyleGorman - 2018-12-29 - 19:30

You can just comment out those lines in float-weight.h if they're missing. The C++11 spec says all the pieces ought to be available, so your compiler is probably non-conformant.

MartinJansche - 2019-01-02 - 21:22

Aaron, could you tell us what compiler and C++ library you are using? If $CXX refers to your c++ compiler, you could run

$CXX --version

echo "#include <cstddef>" | $CXX -E -x c++ -std=c++11 -v - 2>&1

to get the compiler version and information about the standard library.

DewaldDeJager - 2019-01-03 - 13:15

I am facing the same issue. My output for the above commands (From MartinJansche) is as follows:

[dearvolt@vostro openfst]$ g++ --version g++ (GCC) 8.2.1 20181127 Copyright (C) 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[dearvolt@vostro openfst]$ echo "#include <cstddef>" | g++ -E -x c++ -std=c++11 -v - 2>&1 Using built-in specs. COLLECT_GCC=g++ Target: x86_64-pc-linux-gnu Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --enable-libmpx --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --enable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --enable-cet=auto Thread model: posix gcc version 8.2.1 20181127 (GCC) COLLECT_GCC_OPTIONS='-E' '-std=c++11' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/cc1plus -E -quiet -v -D_GNU_SOURCE - -mtune=generic -march=x86-64 -std=c++11 ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../x86_64-pc-linux-gnu/include" #include "..." search starts here: #include <...> search starts here: /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1 /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/x86_64-pc-linux-gnu /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../include/c++/8.2.1/backward /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include /usr/local/include /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include-fixed /usr/include End of search list. # 1 "<stdin>" # 1 "<built-in>" # 1 "<command-line>" # 1 "/usr/include/stdc-predef.h" 1 3 4 # 1 "<command-line>" 2 # 1 "<stdin>" # 1 "/usr/include/c++/8.2.1/cstddef" 1 3 # 42 "/usr/include/c++/8.2.1/cstddef" 3

# 43 "/usr/include/c++/8.2.1/cstddef" 3

# 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 1 3 # 236 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3

# 236 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3 namespace std { typedef long unsigned int size_t; typedef long int ptrdiff_t;

typedef decltype(nullptr) nullptr_t;

} # 258 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3 namespace std { inline namespace cxx11 __attribute__((__abi_tag ("cxx11"))) { } } namespace __gnu_cxx { inline namespace cxx11 __attribute__((__abi_tag ("cxx11"))) { } } # 508 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 3 # 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 1 3 # 39 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 3 # 1 "/usr/include/features.h" 1 3 4 # 428 "/usr/include/features.h" 3 4 # 1 "/usr/include/sys/cdefs.h" 1 3 4 # 442 "/usr/include/sys/cdefs.h" 3 4 # 1 "/usr/include/bits/wordsize.h" 1 3 4 # 443 "/usr/include/sys/cdefs.h" 2 3 4 # 1 "/usr/include/bits/long-double.h" 1 3 4 # 444 "/usr/include/sys/cdefs.h" 2 3 4 # 429 "/usr/include/features.h" 2 3 4 # 452 "/usr/include/features.h" 3 4 # 1 "/usr/include/gnu/stubs.h" 1 3 4 # 10 "/usr/include/gnu/stubs.h" 3 4 # 1 "/usr/include/gnu/stubs-64.h" 1 3 4 # 11 "/usr/include/gnu/stubs.h" 2 3 4 # 453 "/usr/include/features.h" 2 3 4 # 40 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/os_defines.h" 2 3 # 509 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 2 3

# 1 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/cpu_defines.h" 1 3 # 512 "/usr/include/c++/8.2.1/x86_64-pc-linux-gnu/bits/c++config.h" 2 3 # 50 "/usr/include/c++/8.2.1/cstddef" 2 3 # 1 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 1 3 4 # 149 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4 typedef long int ptrdiff_t; # 216 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4 typedef long unsigned int size_t; # 426 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4 typedef struct { long long __max_align_ll __attribute__((__aligned__(__alignof__(long long)))); long double __max_align_ld __attribute__((__aligned__(__alignof__(long double)))); # 437 "/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/include/stddef.h" 3 4 } max_align_t;

typedef decltype(nullptr) nullptr_t; # 51 "/usr/include/c++/8.2.1/cstddef" 2 3

namespace std {

using ::max_align_t; } # 1 "<stdin>" 2 COMPILER_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../../lib/:/lib/../lib/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.1/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS='-E' '-std=c++11' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'

DewaldDeJager - 2019-01-03 - 13:19

The compiler error as well as the output for the above to commands on my system can be found here: https://gist.github.com/DewaldDeJager/96d6236d289fe906a2624c6a747ff8ea

The formatting is just much nicer than on the forum.

MartinJansche - 2019-01-03 - 17:20

Thanks for that link. Very helpful!

As Kyle mentioned above, you can simply comment out or remove the static_assert lines in float-weight.h. That should be entirely harmless, and we'll most likely (re)move those lines very soon.

With the static_assert's present, the best working hypothesis is that there is some misunderstanding or disagreement about which expressions are in fact constexpr in C++11. Your compilation fails for static_assert(TropicalWeight::NoWeight().Member(), "NoWeight not member"), which involves the expanded subexpression (+QNaNf >= -3.40282347e+38f) which the compiler considers to not be a constant expression. Either way, that's a problem. If the compiler is right, that's most definitely a problem; if the compiler is wrong, it still requires a workaround.

MartinJansche - 2019-01-03 - 22:17

After some further investigation, I've filed the following bug: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88683

DewaldDeJager - 2019-01-04 - 05:47

Thanks for getting back to me, Martin!

I have tried with different versions of GCC and can reproduce the issue on GCC 8.2.1, 7.4.0, 6.5.0 and 5.5.0. Basically, I have not been able to compile it with any of the versions I have tried. Which compiler are you using?

To make it easier to test different versions of the compiler, I have created a Dockerfile for the project. The file along with usage instructions can be found here: https://gist.github.com/DewaldDeJager/ef2a5845866f8008d6540b7dd72eb4cd

I am the maintainer of OpenFst on the Arch User Repository and the packaging script downloads OpenFst directly from the website to the user's computer. As such, it is impractical for me to remove the static asserts from the source code before the code gets compiled on the user's computer.

Thank you for the help =)

DewaldDeJager - 2019-01-07 - 06:30

@MartinJansche @KyleGorman Any update on the compilation errors?

KyleGorman - 2019-01-07 - 19:55

I suppose we could bump the version, but isn't there any way to patch "downstream" in the meantime?

DewaldDeJager - 2019-01-10 - 06:28

@KyleGorman Unfortunately not really. I could pull a patched version of float-weight.h but for security reasons it would be best if I could pull the patch from the OpenFst website and verify it with the checksums.

DewaldDeJager - 2019-01-14 - 11:59

@MartinJansche @KyleGorman Any news? Is there any public facing Git repo with the source? I'm even willing to make the patch myself. All I've found is an outdated mirror on Martin's GitHub profile.

On a side note, is there anything that can be done to stop these annoying spam forum posts? Perhaps a CAPTCHA step during registration?

Access control:

Set ALLOWTOPICCHANGE = AllAuthUsersGroup

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	lex-unk.png	r1	manage	242.8 K	2014-06-21 - 01:48	UnknownUser	Example lexicon with unknown handling
diff	openfst_auto_20071112.diff	r1	manage	8.7 K	2007-11-13 - 05:13	UnknownUser	Autoconfiscating patch
jpg	wfsa1.JPG	r1	manage	8.8 K	2014-01-16 - 10:22	UnknownUser	Tropical WFSA redundant paths example 1
jpg	wfsa2.JPG	r1	manage	6.4 K	2014-01-16 - 10:23	UnknownUser	Tropical WFSA without redundant paths example

Topic revision: r125 - 2024-09-14 - JakubKaszycki

Forum