Autoreplace Upgrade

[dimfeld]

EDIT: There are a few autoreplace list links in this post, but this is the one you want: http://drop.io/dimfeld/asset/autoreplace-common-zip

I spent the last few hours working on an upgrade to the autoreplace system. I found that when typing on the Pre, most of my typing mistakes would come from pressing the key either to the left or the right of the correct key. Personally, I never have a problem with hitting the wrong row, so that makes things somewhat simpler.

I wrote a Python script to read in a word list file (I used a standard Mac OSX word list with contractions added) and generate all the possible single-letter mistakes from that file that could result from pressing the button to the left or right of the intended button. This script also reads in the Pre's autoreplace file, and it gives precedence to the corrections in there. To keep the output file to a reasonable size, I didn't do corrections for words more than 7 letters long. With this restriction, the file is 9MB, and with a max length of 9 letters, it's 27MB.

I've uploaded the Python script, my modified word list, and the resulting output file at [removed because it's outdated]
I haven't included the Pre's autoreplace file here to avoid any IP concerns, but if you can get the replacement file in to the Pre, I'm sure you can get the original out

EDIT: The original file here causes Webkit to use a lot of memory and load slowly. I've uploaded a new file with words only up to 5 characters long at http://drop.io/dimfeld/asset/autoreplace5-zip which seems to work much better.

EDIT2: And here's another file with a word list generated from a list of the 15000 most common English words. I recommend using this one as it also has other improvements. See posts #21 through #24 for more details. http://drop.io/dimfeld/asset/autoreplace-common-zip

To run the script:
python generate_autoreplace.py original-autoreplace wordlist new-autoreplace

Upload the output file to
/etc/palm/autoreplace/en_us/text-edit-autoreplace

Then reboot your Pre and you're all set.

I haven't done extensive testing on this, so do this at your own risk, but it's working well for me so far.

[jhoff80]

From my understanding, it seems to be the latter file that webOS uses.

[FuManChuu]

sweet mother of lord this is exactly what I was planning on doing, but had no idea in hell how to do it

Edit: what an odd collection of words :P However, a great improvement none the less

[nimer55]

Holly (can't swear, but imagine the 10 worse words you have ever heard).


Have you considered sending this to palm? Just make sure they make it optional...

Can't wait for the pre to come out in canada. one of the first things i will try.

[posguy99]

I imagine it's more likely that the file in /etc is either placed there by the language selection app, or it's a symlink/hardlink to the other file.

[knightflores]

crap u beat me to it lol... nonetheless great job!!!

[JcLusso]

Nice job. Works great. Also I was really hoping for someone to make the autoreplace better. Thanks

[nimer55]

Could anyone add spell check, and grammar like the one microsoft word 2007 (the old words spell check is worse then nothing).

Or even firefox's spell would be great, dosen't do grammar, but who cares, spell check is good enough.

[dimfeld]

Glad everyone likes it so far. Unfortunately a large autoreplace list like this seems to cause Webkit to both use a lot of memory and take forever to start up. So email and the web browser are not too useful with this list. It may work better with a small max word length like 5 or 6, so I'll try that and report back.

[dimfeld]

Originally Posted by posguy99

I imagine it's more likely that the file in /etc is either placed there by the language selection app, or it's a symlink/hardlink to the other file.

Well, that made me curious so I did a couple of tests. It's not a hard or symbolic link, and switching languages doesn't seem it update /etc/palm/text-edit-autoreplace either.

If that file and the other one are different, it uses /etc/palm/autoreplace/en_us/text-edit-autoreplace, which is in line with what everyone was saying. So it looks like /etc/palm/text-edit-autoreplace is simply unused.

On another note, performance with a file that has a max word length of 5 seems to be much better. The file is about 1.5 MB instead of 9 MB which I'm sure helps a lot. I've uploaded this to drop.io/dimfeld/asset/autoreplace5-zip

[sur4die]

Originally Posted by dimfeld

Then just upload the output file to
/etc/palm/text-edit-autoreplace
and
/etc/palm/autoreplace/en-us/text-edit-autoreplace

you have a typo, it should be an underscore
/etc/palm/autoreplace/en_us/text-edit-autoreplace

[dimfeld]

Fixed. Thanks for letting me know.

[sacherjj]

Need to get a good sample of emails, SMSs, etc and do a word frequency count. Should be able to reduce the base list quite a bit without reducing functionality much. i.e. Better performance with nearly the same functionality.

[jngai]

Instead of shortening the word length to improve performance, is there a way to shortened the list to words that are commonly used? I don't think I will ever use "Zyzomys" or "xysti" (from the original 7 letter Autoreplace file).

I'm thinking of looking up something like "most commonly used words" or something like that. But thanks for getting the ball rolling on this. This is a really useful mod!

[dimfeld]

Yeah, a max word length was quick to implement, so I did that first. Changing the list to include only commonly-used words would definitely be more useful though.

You can use whatever word list you want with the python script I included in the ZIP file, so if you get a list of commonly used words, you can just feed that and the original autoreplace file into the script and you'll get a much better autoreplace file.

[sacherjj]

The hardest part is getting the list. It needs to not reformat common "slang" terms used often in texting and posting. It is hard to take just a wild feed from somewhere without filtering for all the misspellings people have done in the feed.

[nimer55]

Originally Posted by sacherjj

The hardest part is getting the list. It needs to not reformat common "slang" terms used often in texting and posting. It is hard to take just a wild feed from somewhere without filtering for all the misspellings people have done in the feed.

Yea, it's hard..

[FuManChuu]

hrm, i'm all for this idea, so ill dig around and see if i can come up with a word list... I'm a tard though, so no promises <3

[dimfeld]

There's a list of 15000 most commonly used English words at Sustainable English

It's based on a project from England though, so some British->American conversion will be needed before it's useful for most of us. Also need to add contractions and I'm not sure if it included plurals or not. Might be a good starting point though. I'm at work right now so I can't do more than post quick replies, but I'll be looking into this more later tonight.

[FuManChuu]

I found this while googling...

Using Python to detect the most frequent words in a file | Zen and the Art of Programming

Its a python script, and many variants, for collecting commonly used words from a text file. It also has links to perl and command line alternatives.

Either way, what I'm looking for now, and it might be helpful if someone else has theirs and could try one of these scripts, are AIM logs / text message logs. Unfortunately I hard reset my phone all the time, so my treo logs are worthless, but I've got a buddy with about 6 months of iphone logs, and I'm sure I've got my AIM logs somewhere... I think those would be a pretty good starting point to run one of these scripts on.

Anyway, I'll post when I start filtering out muh logs <3