Core API

The tokenize function is framework-agnostic and can be used without React.

tokenize

@emoji-mart/data

import emojiMartData from '@emoji-mart/data';
import { fromEmojiMart } from 'react-emoji-text/adapters/emoji-mart';
import { tokenize } from 'react-emoji-text/core';

const data = fromEmojiMart(emojiMartData);
const tokens = tokenize('hello :wave: world', { data });
// [
//   { type: 'text', value: 'hello ' },
//   { type: 'emoji', emoji: {...}, native: '👋', source: 'shortcode', ... },
//   { type: 'text', value: ' world' },
// ]

emojibase

import emojis from 'emojibase-data/en/data.json';
import shortcodes from 'emojibase-data/en/shortcodes/emojibase.json';
import { fromEmojibase } from 'react-emoji-text/adapters/emojibase';
import { tokenize } from 'react-emoji-text/core';

const data = fromEmojibase(emojis, { shortcodes });
const tokens = tokenize('hello :wave: world', { data });
// [
//   { type: 'text', value: 'hello ' },
//   { type: 'emoji', emoji: {...}, native: '👋', source: 'shortcode', ... },
//   { type: 'text', value: ' world' },
// ]

Try It

@emoji-mart/dataemojibase

ASCII

[
  {
    "type": "text",
    "value": "hello "
  },
  {
    "type": "emoji",
    "native": "👋",
    "shortcode": "wave",
    "match": ":wave:",
    "source": "shortcode"
  },
  {
    "type": "text",
    "value": " world "
  },
  {
    "type": "emoji",
    "native": "🙂",
    "match": ":)",
    "source": "ascii"
  },
  {
    "type": "text",
    "value": " "
  },
  {
    "type": "unknown",
    "shortcode": "missing",
    "match": ":missing:"
  }
]

TokenizeOptions

Option	Type	Default	Description
data	EmojiData	required	Emoji dataset
customEmojis	CustomEmojiCategory[]	—	Custom emoji categories
ascii	boolean	true	Match ASCII emoticons
defaultSkin	SkinTone	—	Default skin tone
extraAliases	Record<string, string>	—	Additional shortcode mappings

Token Types

type Token = TextToken | EmojiToken | UnknownToken;

TextToken

type TextToken = {
  type: 'text';
  value: string;
};

Plain text between emoji matches.

EmojiToken

type EmojiToken = {
  type: 'emoji';
  emoji: EmojiEntry;
  native: string;
  skin?: SkinTone;
  shortcode?: string;
  match: string;
  source: 'unicode' | 'shortcode' | 'ascii' | 'custom';
};

Field	Description
emoji	Full emoji entry from the dataset
native	Resolved native Unicode string
skin	Applied skin tone (if any)
shortcode	The matched shortcode (for shortcode/custom sources)
match	The original matched text in the input
source	How the emoji was matched

UnknownToken

type UnknownToken = {
  type: 'unknown';
  shortcode: string;
  match: string;
};

Shortcodes that look like :name: but don’t match any known emoji.