Natural language processing, usually shortened to NLP, covers the techniques that allow a computer to read, understand, analyse or produce human language. Translating a text, correcting spelling, classifying an e-mail, summarizing a document or answering a question all fall within it.
An old discipline
Language processing is almost as old as computing. In 1950, Alan Turing proposed in the paper "Computing Machinery and Intelligence" a conversation-based test to judge whether a machine can think. In 1954, the Georgetown-IBM experiment automatically translated some sixty sentences from Russian into English. In 1966, ELIZA, a program by Joseph Weizenbaum at MIT, simulated a conversation using keywords.
For decades, the discipline advanced in stages, each with its own methods:
- Hand-written rules (1950s to 1980s): linguists describe grammar and vocabulary so the program can parse a sentence
- Statistical methods (1990s and 2000s): the program learns patterns from large amounts of text, which notably improves machine translation
- Deep learning (2010s): neural networks and embeddings represent the meaning of words and sentences with numbers
- Large language models (since the late 2010s): a single language model, trained on huge corpora, performs most of the tasks that used to require a specialized tool
The main NLP tasks
- Classification: assigning a category to a text, such as "complaint", "quote request" or "practical question"
- Entity recognition: spotting names, dates, places, amounts or order numbers in a text
- Intent detection: understanding what the author of a message wants, beyond the words used
- Sentiment analysis: estimating whether a message is positive, neutral or negative
- Translation from one language to another
- Summarization of a document or a conversation
- Information retrieval, including semantic search, which finds a passage by its meaning
- Text generation: writing an answer, an e-mail or a description
NLP and chatbots
The first business chatbots used NLP for one precise task: recognizing a visitor's intent from a planned list, then displaying the answer written in advance for that intent. The program understood the question better than a simple keyword search, but it could say nothing outside the prepared answers.
Today's AI assistants use a language model that performs several NLP tasks at once: understanding the question, spotting useful information, searching the documents, writing an answer in the visitor's language. This versatility explains their flexibility, and also their main risk: a model that writes freely can produce a hallucination if it is not kept within limits.
Not all languages are equal
NLP tools work better in languages with a lot of digital text. English is the best served. French, Dutch and German are very well covered by current models. Regional variants need more attention: Belgian words such as "septante", "GSM" or "kot" are understood, but a model does not use them spontaneously in its answers. Belgian administrative terms, which differ from those of France or the Netherlands, are worth spelling out in the documents.
What NLP makes possible in an SME
Even without a conversational assistant, several uses of NLP are within reach of a small organization:
- Automatically sorting incoming e-mails by type of request
- Extracting information from a free-text form or message
- Summarizing a long conversation before passing it to a colleague
- Spotting recurring questions in customer messages, to complete an FAQ
In an assistant, these tasks combine: the question is understood, the intent identified, the documents consulted, the answer written, and the conversation summarized if it goes to the team.
NLP in KernChat
KernChat relies on language models available on the market, which perform all these tasks. They understand questions in French, Dutch and English, answer from the organization's documents and produce, when the conversation closes, a summary, the visitor's intent, the tone of the exchange and the topics discussed. This information helps the team take over a request without rereading the whole exchange.