OpenFst Forum 2010 Archive

KonstantinSelivanov - 11 Sep 2010 - 08:58

But it seems they have some distinctions. E.g. when I create language model using grmcount/grmake there are some arcs with weight "Infinity". I didn't see such weights in openfst library. Or maybe I'm doing something wrong?

MichaelRiley - 20 Oct 2010 - 18:00

FAST_LOG_PROB_ARC_SELECTOR

AMorenoDaniel - 08 Sep 2010 - 13:01

FAST_LOG_PROB_ARC_SELECTOR

My first compilation succeeded on Cygwin after I commented out that option. Cheers

BusyBeaver - 08 Sep 2010 - 13:27

CyrilAllauzen - 09 Sep 2010 - 11:40

MichaelRiley - 09 Sep 2010 - 17:11

Compiling 1.2.1 on Mac OSX

RichardSproat - 02 Sep 2010 - 17:09

g++ -DHAVE_CONFIG_H -I./../include -g -O2 -MT replace.lo -MD -MP -MF .deps/replace.Tpo -c replace.cc -fno-common -DPIC -o .libs/replace.o /usr/include/c++/4.0.0/tr1/hashtable: In member function 'typename std::tr1::hashtable<Key, Value, Allocator, ExtractKey, Equal, H1, H2, H, RehashPolicy, cache_hash_code, mutable_iterators, unique_keys>::const_iterator std::tr1::hashtable<Key, Value, Allocator, ExtractKey, Equal, H1, H2, H, RehashPolicy, cache_hash_code, mutable_iterators, unique_keys>::find(const Key&) const [with Key = int, Value = std::pair, Allocator = std::allocator<std::pair >, ExtractKey = Internal::extract1st<std::pair >, Equal = std::equal_to, H1 = std::tr1::hash, H2 = Internal::mod_range_hashing, H = Internal::default_ranged_hash, RehashPolicy = Internal::prime_rehash_policy, bool cache_hash_code = false, bool mutable_iterators = true, bool unique_keys = true]': ./../include/fst/replace.h:531: instantiated from 'bool fst::ReplaceFstImpl<A, T>::IsNonTerminal(typename A::Label) const [with A = fst::LogArc, T = fst::DefaultReplaceStateTable<fst::LogArc, ssize_t>]' ./../include/fst/replace.h:560: instantiated from 'size_t fst::ReplaceFstImpl<A, T>::NumInputEpsilons(typename A::StateId) [with A = fst::LogArc, T = fst::DefaultReplaceStateTable<fst::LogArc, ssize_t>]' ./../include/fst/fst.h:732: instantiated from 'size_t fst::ImplToFst<I, F>::NumInputEpsilons(typename I::Arc::StateId) const [with I = fst::ReplaceFstImpl<fst::LogArc, fst::DefaultReplaceStateTable<fst::LogArc, ssize_t> >, F = fst::Fst<fst::LogArc>]' replace.cc:43: instantiated from here /usr/include/c++/4.0.0/tr1/hashtable:1136: error: passing 'const std::tr1::hashtable<int, std::pair, std::allocator<std::pair >, Internal::extract1st<std::pair >, std::equal_to, std::tr1::hash, Internal::mod_range_hashing, Internal::default_ranged_hash, Internal::prime_rehash_policy, false, true, true>' as 'this' argument of 'typename std::tr1::hashtable<Key, Value, Allocator, ExtractKey, Equal, H1, H2, H, RehashPolicy, cache_hash_code, mutable_iterators, unique_keys>::node* std::tr1::hashtable<Key, Value, Allocator, ExtractKey, Equal, H1, H2, H, RehashPolicy, cache_hash_code, mutable_iterators, unique_keys>::find_node(Internal::hash_node<Value, cache_hash_code>*, const Key&, typename std::tr1::hashtable<Key, Value, Allocator, ExtractKey, Equal, H1, H2, H, RehashPolicy, cache_hash_code, mutable_iterators, unique_keys>::hash_code_t) [with Key = int, Value = std::pair, Allocator = std::allocator<std::pair >, ExtractKey = Internal::extract1st<std::pair >, Equal = std::equal_to, H1 = std::tr1::hash, H2 = Internal::mod_range_hashing, H = Internal::default_ranged_hash, RehashPolicy = Internal::prime_rehash_policy, bool cache_hash_code = false, bool mutable_iterators = true, bool unique_keys = true]' discards qualifiers make[3]: * [replace.lo] Error 1

RichardSproat - 02 Sep 2010 - 19:01

http://openfst.cs.nyu.edu/twiki/bin/view/FST/CompilingOnMacOSX

did the relevant updates, and was fine compiling 1.1

CyrilAllauzen - 03 Sep 2010 - 11:56

CyrilAllauzen - 03 Sep 2010 - 16:50

other bug in gcc 4.0 TR1 implementation

Application of RTNs?

KennethRBeesley - 01 Sep 2010 - 13:16

CyrilAllauzen - 01 Sep 2010 - 15:38

Mindtuning for PDT?

KennethRBeesley - 01 Sep 2010 - 13:05

CyrilAllauzen - 01 Sep 2010 - 15:41

pre-determinization

WeiwenLeung - 30 Aug 2010 - 06:19

CyrilAllauzen - 01 Sep 2010 - 15:42

fstepsnormalize

WeiwenLeung - 27 Aug 2010 - 01:56

is there anyone encounter the same problem?

fstcompose output

AliAzimizadeh - 20 Aug 2010 - 16:47

AMorenoDaniel - 21 Aug 2010 - 15:04

AliAzimizadeh - 21 Aug 2010 - 16:34

BlueSky - 29 Oct 2010 - 05:15

Compiling openfst-1.2 on cygwin

AMorenoDaniel - 18 Aug 2010 - 17:48

./configure CXX=g++-4.exe CC=gcc-4.exe --prefix=$HOME/local
make
#make install  # <--- I haven't made it here though.

g++-4.exe -g -O2 -o fstarcsort.exe fstarcsort.o  -ldl ../lib/.libs/libfst.a ../script/.libs/libfstscript.a /usr/lib/gcc/i686-pc-cygwin/4.3.4/libstdc++.dll.a   -L/usr/lib/gcc/i686-pc-cygwin/4.3.4 -L/usr/lib/gcc/i686-pc-cygwin/4.3.4
../script/.libs/libfstscript.a(fst-class.o):fst-class.cc:(.text$_ZN3fst16CompatPropertiesEyy[fst::CompatProperties(unsigned long long, unsigned long long)]+0x143): undefined reference to `fst::PropertyNames'
~~~----ETCETERA----~~~
collect2: ld returned 1 exit status
make[3]: *** [fstarcsort.exe] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

libtool: link: warning: undefined symbols not allowed in i686-pc-cygwin shared libraries

/bin/sh ../../libtool --tag=CXX   --mode=link g++-4.exe  -g -O2 -version-info 0:0:0  -o libfst.la -rpath /usr/local/lib compat.lo flags.lo fst.lo properties.lo symbol-table.lo util.lo

/bin/sh ../../libtool --tag=CXX   --mode=link g++-4.exe  -g -O2 -version-info 0:0:0  -o libfstscript.la -rpath /usr/local/lib arcsort.lo closure.lo compile.lo compose.lo concat.lo connect.lo convert.lo decode.lo determinize.lo difference.lo draw.lo encode.lo epsnormalize.lo equal.lo equivalent.lo fst-class.lo info.lo intersect.lo invert.lo map.lo minimize.lo print.lo project.lo prune.lo push.lo randequivalent.lo randgen.lo relabel.lo replace.lo reverse.lo reweight.lo rmepsilon.lo script-impl.lo shortest-distance.lo shortest-path.lo synchronize.lo text-io.lo topsort.lo union.lo weight-class.lo

DanielPovey - 19 Aug 2010 - 05:43

DanielPovey - 19 Aug 2010 - 06:06

AMorenoDaniel - 19 Aug 2010 - 11:37

DanielPovey - 19 Aug 2010 - 13:26

DanielPovey - 20 Aug 2010 - 05:09

PaulDixon - 21 Aug 2010 - 00:21

www.furui.cs.titech.ac.jp/~dixonp/openfstwin-1.2-210810.zip

It includes prebuilt 32-bit binaries that should also work under cygwin. It's all 1.2 code the 1.1 directory name still needs to be changed.

DanielPovey - 21 Aug 2010 - 05:10

ReplaceFst() and testing for CyclicDependencies

KennethRBeesley - 17 Aug 2010 - 14:21

if (ReplaceFst<StdArc>(pairlabelfsts, slabel).CyclicDependencies()) { ...}

returns true, apparently detecting the cyclic dependencies correctly.

Question: Have I used the proper idioms for testing for cyclic dependencies and doing the actual replacement?

#include <fst/fstlib.h>
#include <iostream>

using namespace fst ;

int main() {
   std::cout << "Hello, world!\n" ;

   int a = 97 ;   // code point values
   int b = 98 ;

   // example with cyclic dependencies

   // StdVectorFst is a typedef for VectorFst<StdArc>
   StdVectorFst fst1, fst2, fst3 ;

   // define fst1, a subnetwork (will be arbitrarily associated with -1)
   fst1.AddState() ;     // return 0
   fst1.SetStart(0) ;
   fst1.AddState() ;   // returns 1
   fst1.AddState() ;   // returns 2
   fst1.SetFinal(2, StdArc::Weight::One()) ;
   fst1.AddArc(0, StdArc(a, a, StdArc::Weight::One(), 1)) ;
   // fst1 refers to fst2 (associated below with -2)
   fst1.AddArc(1, StdArc(0, -2, StdArc::Weight::One(), 2)) ;
   //         src        i  o          w             dest

   // define fst2, a subnetwork (will be arbitrarily associated with -2)
   fst2.AddState() ;   // return 0
   fst2.SetStart(0) ;
   fst2.AddState() ;   // return 1
   fst2.AddState() ;   // return 2
   fst2.SetFinal(2, StdArc::Weight::One()) ;
   fst2.AddArc(0, StdArc(b, b, StdArc::Weight::One(), 1)) ;
   // fst2 refers to fst1 (associated below with -1)
   fst2.AddArc(1, StdArc(0, -1, StdArc::Weight::One(), 2)) ;

   // define fst3, the base network (will be arbitrarily associated with -3)
   fst3.AddState() ;   // return 0
   fst3.SetStart(0) ;
   fst3.AddState() ;   // return 1
   fst3.AddState() ;   // return 2
   fst3.SetFinal(2, StdArc::Weight::One()) ;
   // add arcs with "references" to subnetworks
   // output-side -1 will be associated with fst1
   // output-side -2 will be associated with fst2
   // The label references to subnetworks are on the Output Side.
   // The corresponding Input-Side labels can be anything (here 0 for epsilon).
   fst3.AddArc(0, StdArc(0, -1, StdArc::Weight::One(), 1)) ;
   fst3.AddArc(1, StdArc(0, -2, StdArc::Weight::One(), 2)) ;
   //         src        i   o           w            dest

   // set up the vector associating arbitrary numbers with the three networks
   // -1 refers to fst1
   vector< pair< StdArc::Label, const Fst<StdArc> * > > pairlabelfsts ;
   pairlabelfsts.push_back(pair< StdArc::Label, const Fst<StdArc> *>(-1, &fst1) ) ;
   // -2 refers to fst2
   pairlabelfsts.push_back(pair< StdArc::Label, const Fst<StdArc> *>(-2, &fst2) ) ;
   // -3 is the base network
   pairlabelfsts.push_back(pair< StdArc::Label, const Fst<StdArc> *>(-3, &fst3) ) ;
   
   StdArc::Label slabel = -3 ;      // specifies the "base" Fst, which is fst3

   fst3.Write("before.fst") ;      // the network with references

   // Using delayed ReplaceFst
   if (ReplaceFst<StdArc>(pairlabelfsts, slabel).CyclicDependencies()) {
      std::cout << "Has cyclic dependencies.\n" ;
   } else {
      std::cout << "No cyclic dependencies.\n" ; 
      StdVectorFst expandedFst ;   // the result
      // slabel (here -3) is used to find the base Fst supplied in pairlabelfsts
      expandedFst = ReplaceFst<StdArc>(pairlabelfsts, slabel) ;
      expandedFst.Write("intermediate.fst") ;   // the network just after ReplaceFst()
      RmEpsilon(&expandedFst) ;
      expandedFst.Write("after.fst") ;   // the network after RmEpsilon()
   }

   std::cout << "Good-bye, world!\n" ;
}

CyrilAllauzen - 17 Aug 2010 - 15:52

Best, Cyril

Multithreading in OpenFST

AlbertPark - 09 Jul 2010 - 18:30

I am currently using the OpenFST library, and I was thinking of multithreading my program, and I was curious as to whether the OpenFST library currently supports multithreading. Thank you!

Albert

AlbertPark - 21 Jul 2010 - 04:13

I've gone through looking at the implementation and from what I can tell the OpenFST library currently is not multithreading safe. I've resorted to doing a deep copy of FSTs for each thread. Unfortunately, doing a deep copy was not as straightforward as I thought it would be, or at least I couldn't figure out how to do it elegantly and quickly (The FST I am reading in is very large, and trying to read in the FST multiple times takes much longer than I want). I tried doing a deep copy using Map() with the IdentityMapper as mentioned in a previous post, but found that this only does deep copies of the arcs (given the copy constructor of the Arc and Weight do deep copies), and does a shallow copy of the symbol table, which creates problems for multithreading (when multiple threads try to access the symbol table at once).

I'm not sure if there is an easier way to do this, but here is what I did to change the OpenFST code to support deep copying of the symbol table. I added a new enum code in map.h to MAP_SYMBOLS_ACTION as well as a couple of lines of code for supporting symbol table deep copying, added a function to do a deep copy of the symbol table to symbol-table.h. Finally, I added a Mapper almost identical to the identity mapper, but returns the new enum code for InputSymbolsAction() and OutputSymbolsAction().

Hope this helps anybody trying to multithread!

MichaelRiley - 09 Aug 2010 - 20:42

MichaelRiley - 09 Aug 2010 - 20:44

memory problem in fstcompose

AliAzimizadeh - 07 Jul 2010 - 09:18

0 #_f0 1 #_m0_s2 2 #_m0_s3 3 #_m0_s4 4 #_m0_s5 5 #_m0_s6 6 e$_f0 7 e$_m1_s2 8 e$_m1_s3 9 e$_m1_s4 10 e$_m1_s5 11 e$_m1_s6 12 . . . ow_f6037 170654 s$_f6037 170655 k_f6037 170656 u_f6037 170657 i_f6037 170658 s_f6037 170659 v_f6037 170660 h_f6037 170661 c$_f6037 170662 j_f6037 170663 z$_f6037 170664

its size is about 2.5Mb. then i run the below command successfully:

fstcompile --isymbols=a.sym --osymbols=a.sym tmp1.fst fstcompile --isymbols=a.sym --osymbols=a.sym tmp2.fst

then i want to compose them and run this:

fstcompose tmp2.fst tmp3.fst tmp4.fst

when the symbol number is little the composition is done but in some symbol files the symbol number is very large, for example 170655. in this case, the memory is fulled rapidly and the CPU doesn't work any more. this is my system properties: CPU: Pentium 4 Dual core 2.26GHz Ram: 4G Swap: 15G OS: CentOS 5.3

could you please give me suggestions about the case? Thanks a lot -Ali

PaulDixon - 08 Jul 2010 - 21:52

fstarcsort -sort_type=olabel tmp2.fst > sort2.fst

fstarcsort -sort_type=ilabel tmp3.fst > sort3.fst

fstcompose sort2.fst sort3.fst > tmp4.fst

AliAzimizadeh - 09 Jul 2010 - 06:37

WeiwenLeung - 27 Aug 2010 - 01:33

did you figure out what the problem was ?

Composition and ShortestPath

ChristopherKermorvant - 22 Apr 2010 - 12:20

I make a composition of 3 transducers F o G o H and I extract the ShortedPath on the resulting transducer. Is there a way to have access to the arcs in F, G and H corresponding to the arcs in the ShortestPath transducer ? I would like to have access to the output symbols of the arcs in F for example.

thank you,

-- Chris

DoganCan - 27 Apr 2010 - 09:59

There may be better solutions to your problem provided by the library but a quick solution is:

1. remove the weights of the shortestpath fst, S = shortest(F o G o H), and project it onto the input and output labels

fstmap -map_type=rmweight s.fst | fstproject > s.in.fst
fstmap -map_type=rmweight s.fst | fstproject -project_output > s.out.fst

2. compose shortest input with F and project the result onto the output labels to get the candidate paths in F

compose shortest output with H and project the result onto the input labels to get the candidate paths in H

fstcompose s.in.fst f.fst | fstproject -project_output > f.out.fst
fstcompose h.fst s.out.fst | fstproject > h.in.fst

3. compose the candidate paths with G (F_out o G o H_in) to obtain the paths in G that survive during the composition

fstcompose f.out.fst g.fst | fstcompose - h.in.fst > internal.fst

4. finally apply shortestpath to this fst to obtain the path in G internal to the shortest path in F o G o H

fstshortestpath internal.fst > shortest.internal.fst

This final transducer should provide the internal path you need. Note that the weights of this transducer are those obtained during the composition not the original weights of G.

Best, Dogan

RolandSchwarz - 13 May 2010 - 09:47

DoganCan - 14 May 2010 - 18:05

I am not entirely sure that I got your problem. I mean the internal path in G should be as complex as S.

Can you post a simple example where the above procedure fails?

Best, Dogan

RolandSchwarz - 15 May 2010 - 04:49

Roland

transducing phoneme lattices using WFSTs

BusyBeaver - 03 Apr 2010 - 11:38

I'd like to convert phoneme lattices into WFSTs and use a grammar transducer to obtain a word lattice out of it. The thing is, that I want to keep additional information e.g. time stamps, acoustic scores, etc. while doing so. Note, that I do not want to obtain any likelihoods on paths in the first place. Instead, I would traverse the final lattice using those accumulated values to determine a likelihood for each transition and user-defined scaling factors myself.

I see two possibilities to keep track on times/scores - by storing them on arcs or using the string semiring to push just arbitrary information in the weights. When I use the string semiring, I could simply push there as much as I want to,and do a composition with my grammar. But then I am somewhat restricted: composition is not supported, because the string semiring is not commutative. I was thinking to rewrite the weighted composition algorithm from the paper, just change the weighting part, since I do not care about the order of elements within my strings anyways. But maybe I could keep my input fst as log/tropical semiring type and attach my scores/time stamps in the arcs. Would it then be possible to use the existing composition algorithm and still copy my attached information into the resulting FST? Maybe it is all much easier than what I thought of here right now, then I'd be glad if you give me some hints Thanks a lot in advance, Stefan

Probabilities as weights

NoChips - 17 Mar 2010 - 16:35

As far as I know the default arcs StdArc is using Tropical weight. How can I change that into Probabilty weight, so that transition weights are multiplied and not summed up?

I was looking for something predefined, but couldn't find anything...

Thank you in advance!

PaulDixon - 18 Mar 2010 - 04:14

There is a counting utility that includes a real semiring implementation and arc registration with the standard command line tools [here][1]

[1]: http://www.furui.cs.titech.ac.jp/~dixonp/wfstcount.zip]

Deleting arcs in an Fst

KennethRBeesley - 05 Mar 2010 - 19:37

KennethRBeesley - 12 Mar 2010 - 16:46

CyrilAllauzen - 17 Aug 2010 - 15:56

JoachimChau - 01 Jun 2011 - 14:58

arcsum.h and arcmerge.h minor bug?

PhilSours - 05 Mar 2010 - 17:29

Memory footprint of an Fst

KennethRBeesley - 22 Feb 2010 - 15:16

hbka reference paper

NewestUser - 26 Jan 2010 - 05:05

http://www.cs.nyu.edu/~mohri/postscript/hbka.pdf

and the figure for the minimization example, pg. 8, fig. 5(c) seems quite wrong. particularly the 'minimized' version is not equivalent to the original transducer. in the other reference

http://www.cs.nyu.edu/~mohri/pub/csl01.pdf

an identical transducer, albeit with different weights on a couple of the arcs, appears very differently in its minimized form.

i verified that the second version referenced in the latter link produces the expected output but it would be nice to confirm this, or alternatively to understand why they are different.

CyrilAllauzen - 26 Jan 2010 - 18:44

NewestUser - 26 Jan 2010 - 22:28

Problems with fstshortestpath

JoeWang - 15 Jan 2010 - 03:56

ERROR: FstMainRegister::GetMain: log-arc.so: cannot open shared object file: No such file or directory ERROR: fstshortestpath: Bad or unknown arc type "log" for this operation (ShortestPathMain)

I have checked my $LD_LIBRARY_PATH, it already includes /usr/local/lib, how come it still cannot find the shared object? Thanks in advance

CyrilAllauzen - 15 Jan 2010 - 13:41

The issue is that the shortest path algorithm does not work over the log semiring.

Hence support for LogArc is not compiled into the fstshortestpath utility and the code tries to locate a shared object containg the definition of that arc type.

Best, Cyril

Is it possible to use fstcompile to construct an acceptor instead of a transducer?

JoeWang - 12 Jan 2010 - 22:09

PaulDixon - 13 Jan 2010 - 00:46

JoeWang - 13 Jan 2010 - 02:43

Missing installation files for creating a user-defined arc type?

PaulDixon - 12 Jan 2010 - 01:11

For the shared object to be detected and loaded a name of the form arc-type.so worked but arc_type.so did not.

I put sample code for a real semiring arc type here

KeithPonting - 19 Nov 2010 - 07:41

CyrilAllauzen - 19 Nov 2010 - 11:22

The contain of my-arc.cc (the source for my-arc.so) should look like:

#include "nlp/fst/lib/fst-inl.h"
#include "nlp/fst/lib/const-fst-inl.h"
#include "nlp/fst/script/register.h"
#include "nlp/fst/script/fstscript.h"
#include "my-arc.h"
namespace fst {
namespace script {

REGISTER_FST(VectorFst, MyArc);
REGISTER_FST(ConstFst, MyArc);
REGISTER_FST_CLASSES(MyArc);

REGISTER_FST_OPERATIONS(MyArc);

}
}

Hope this will work for you.

Cyril

KeithPonting - 30 Nov 2010 - 05:17

CyrilAllauzen - 30 Nov 2010 - 15:36

#include <fst/fst.h>
#include <fst/const-fst.h>
#include <fst/script/register.h>
#include <fst/script/fstscript.h>

KeithPonting - 01 Dec 2010 - 05:39

Problems on Ubuntu Linux 9.10

JoeWang - 11 Jan 2010 - 04:40

CyrilAllauzen - 11 Jan 2010 - 12:05

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

JoeWang - 12 Jan 2010 - 02:28

Relabel() and epsilon Properties

KennethRBeesley - 04 Jan 2010 - 15:50

vector < pair<StdArc::Label, StdArc::Label> > vect ; vect.push_back(pair<StdArc::Label, StdArc::Label> ( (StdArc::Label) 5, (StdArc::Label) 0 )) ; Relabel(fstp, vect, vect) ;

where fstp is a pointer to a StdVectorFst. ilabels and olabels originally labeled 5 are relabeled correctly as 0, possibly introducing input-side, output-side and double-sided epsilons; but (here's the problem) the Properties kIEpsilons, kNoIEpsilons, kOEpsilons, kNoOEpsilons, kEpsilons and kNoEpsilons are not updated.

At least that's what seems to be happening.

KennethRBeesley - 04 Jan 2010 - 18:03

Access control:

• Set ALLOWTOPICCHANGE = AllAuthUsersGroup