Word Filtering - The "Interceptor"

I've started looking into modifying my older KWIC keyword Extracter / Analyzer to be able to better trap spam, bad words, etc. Basically we ourselves have a need to for this for outr own elgg-based web sites. Most word filtering alogirthms, including those based on the time-honored Bayesian techniques would seem to fall off the cliff with some simple tricks that will bypass such filtering.
Let us say we block for a bad word "badword"
users, even kids.. can figure out they will be blocked,
so.. they do not use "badword" anymore
they switch to variations

A human reading this can still read the text and cognize, but our funky word filtering algorithms get a failing grade..!!! ;-)

I did those earlier KWIC routines to extract and classify Elgg's Google group posts so that I could create a nicely indexed archive which I could then search with ease. For that I had used a combination of StopWords + KeyWords -- so that it would ignore the Stopwords, while paying particular attention to the Keywords ( which had special meaning to me, even if these were on the Stopwords list )

So, anyway, the first problem to solve will be :=
How to match the dictionary "badword" with a misspelt "b.a.d.w.o.r.d" and similar...

The end result I hope to achieve is - word filtering for UserID, UserName, Messages content ( Carlos' "Messages Interceptor' ), any other *content posting where there might be a need for *filtering and *safety.

I'm not particularly looking for coding help here, however if you're reading this and have experience with such techniques, feel welcome to ppost and discuss.



  • The easiest and most naiive algorithm to do this:

    • Create a copy of the string without any sentence critical characters that are not in your current locale. (Filter out all punctuations etc.. something similar to php's ctype_alpha() )
    • Use php's stristr or stripos and look for bad words.
    • If you've got a match, error
    • If you don't, pass though to method

    It'll be pretty easy to generate bad words in context

  • BTW - You have me stumped right now...

    BUT that is ***IS the kind of "harmless" language that needs to be detected
    because we all know what it means..
    but a word based filter ain't got no hope...

    Carlos and I have already spome had discussions re: "harmless" language which might actually be incriminating.... and suddenly makes one wish that we coded some hard fancy-ass A.I. logic with PHP to make the web a "safer place"... for all...

    I get a funny feeling that we are opening a very special, very security / safety sensitive aspect of web- presence here... we  might just not quiet know it yet...


  • lololol  hey it just came to mind.... just got done watching a cop show....  lololol  :-)

  • yeah thats a difficult task

    do you got bud's cell phone number?

    trying to remember all my covert lingo. although i am thinking goof has a few good stories to tell.

  • @MM

    no hurry with the php code

    but... how does a LLR based lex analysis/ tokenizer parse language context variants such as


    at some level -- we see that there is a *little *big problem because the task of identifying alternatives for the syntax parser's directed graphs --suddenly becomes endless ;-(

    ( I'm guessing.... here...)
    picture Niklaus Wirth'S really smarty table driven parser trying to parse all those variants... !!!!

    We seem to have a problem... with filtering content...
    Let's see what tomorrow brings...


  • I doubt that you can lexically chain a directed graph to detect euphamisms in speech. I agree that it would be endless, and almost non-realistically performant. I think that you should first start out as lightweight.

    Tackling euphamisms is not an easy task; but, that being said, one could generate some kind of contextual probability detector that can use tokens. The easiest dictionary to parse would be urbandictionary.com. Generating some kind of token set, where by some type of fuzzy logic/searching, you can statistically predict when a euphamism is being used and invoke the proper error handlers.

    I think you would have some re-visiting to do after this would be implemented. For example, I can say something like, "Hey Dhrup, want to mow my grass for 50 bucks?" it would be pretty difficult to narrow tokens down to JUST the substantial part.

  • @MM

    It is refreshing to see comments from one who  "comprend mon style de langue ..."

  • You know.. I have something to say about all this.. and I really get each and every point, with every tech term on show here, but honestly, I am so busy doing work to make a detailed comment.. but I will say this:

    You can make a code do an inventory on how many farts were done on you site.. the question is: is it necessary?? I think the entire thread here is heading way in the wrong direction.. this shouldn't be a show off engineering skills, this should be about user safety.. now, since you are dealing with a social network, you have to think social network... the code, and I can tell you all, can be accomplished the way you want it with relevant ease... but, as I said before, do we need to? in this case, I would use a linguist and a social engineer, preferrably with hacking experience, to build an expandible data file containing language which I would use as reference.

    You can sit there and argue algorithm, contextual detectors, A.I Logic, euphamisms and passing tokens... but, I shit you not, a 6 year old can throw all that IT college jibjab in tatters using simple SMS teen lingo.... then what.. lets see how long it will take to modify your code to meet little Johnny's intelligence.

    The lesson is: want to monitor people?????? understand people... after all, this is a social network, not a bank's website.

    My 2 cents.

  • Oh, and one more thing...


    I'd still filter the water in the bong!