Text generation with RNN - how to encode with perceptilabs

I thought it would be informative to build a model from scratch, and this sort of thing appeals: text generation by RNN

However, I don’t think TF1.15 includes the required preprocessing, as described there

The preprocessing.StringLookup layer can convert each character into a numeric ID. It just needs the text to be split into tokens first.

In the absence of preprocessing.StringLookup what’s the best way of doing generating the character IDs in perceptilabs?

Hi Julian,

Indeed, preprocessing.StringLookup doesn’t seem to be available in TF1.15. The contrib library (deprecated in TF2) does seem to have a similar operation:

tf.contrib.lookup.string_to_index

Would you mind trying that to see if it works for you?

I see it here: https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/contrib/lookup, as you say, tf.contrib.lookup.string_to_index and a few other useful things. [Aside: wow. int64. plenty of space left after the alphabet and a bit of punctuation!]

And I have even found it in loaded modules… listing results of homebrew search for future ref:

Thanks - I will try it, but maybe nothing to report until the weekend.

modulesAndMembers(inModuleName ='tensorflow.contrib', inMemberName='index', caseSensitive = False, predicate = inspect.isfunction)

tensorflow.contrib.constrained_optimization
	find_best_candidate_index
tensorflow.contrib.constrained_optimization.python.candidates
	find_best_candidate_index
tensorflow.contrib.graph_editor.subgraph
	_finalize_index
tensorflow.contrib.keras.api.keras.datasets.imdb
	get_word_index
tensorflow.contrib.keras.api.keras.datasets.reuters
	get_word_index
tensorflow.contrib.learn.python.learn.learn_io.dask_io
	_add_to_index
tensorflow.contrib.lookup
	index_table_from_file
	index_table_from_tensor
	index_to_string
	index_to_string_table_from_file
	index_to_string_table_from_tensor
	string_to_index
	string_to_index_table_from_file
	string_to_index_table_from_tensor
tensorflow.contrib.lookup.lookup_ops
	index_table_from_file
	index_table_from_tensor
	index_to_string
	index_to_string_table_from_file
	index_to_string_table_from_tensor
	string_to_index
	string_to_index_table_from_file
	string_to_index_table_from_tensor
tensorflow.contrib.recurrent.python.ops.recurrent
	_Index