Gesture-Driven Annotations in Adobe Illustrator

By Michael Peters · Published April 8, 2026 · Last updated May 8, 2026 · 11 min read

The Client

SMART Technologies makes the interactive whiteboards used in classrooms, meeting rooms, and design studios worldwide. Their hardware supports multi-touch, pen, eraser, and gesture input through a vendor SDK (the SMART Board SDK, or SBSDK), and the boards are designed for natural collaboration around a shared display.

The Problem

SMART wanted to extend the gesture-driven experience of their whiteboards into professional design tools. Specifically, designers and educators using SMART Boards needed to be able to:

• Add freeform annotations and markup directly onto Illustrator artwork using SMART Board pens, with the same line weights and colours the board exposed
• Use multi-touch gestures — rotate, scale, pan, duplicate — to manipulate selected Illustrator objects with the same tactile feel users were already accustomed to on the SMART Board
• Switch between Illustrator tools by picking up the physical pen, eraser, or marker from the board’s pen tray, without ever touching the keyboard or mouse
• Support simultaneous multi-pointer input from multiple users at the same board, since SMART Boards are explicitly designed for collaborative use

None of this was supported out of the box by Adobe Illustrator. The Illustrator SDK delivers only events sourced inside the application’s own event loop — mouse clicks, key presses, menu choices — and exposes no extension point for third-party hardware to inject events. Bridging the SMART Board into Illustrator required a C++ plug-in that ran the vendor SDK on its own thread, classified the resulting raw touches into meaningful gestures, and then called Illustrator’s scripting suites on the main thread to do the actual work.

The architecture: two threads, one application context

The plug-in runs two threads concurrently inside the Illustrator process:

1. The Illustrator main thread, which Illustrator itself owns. The plug-in subclasses Adobe’s Plugin base via a MarkedObjectsPlugin class that handles StartupPlugin/ShutdownPlugin, menu items, and notifiers (selection changed, document opened, save, colour-model change, and so on). All Illustrator API calls have to happen here, wrapped in a stack-allocated IllustratorContext that pushes the plug-in’s SPPluginRef so the suite calls land in the right plug-in’s context.
2. A SMART SDK polling thread, started by the plug-in at StartupPlugin time, which spins through the vendor SDK’s pump-events loop. This thread sleeps for a configurable interval, then either drains queued annotation delays or calls into SBSDK2 / SBSDK2Advanced to pump pending board events. A static SmartSDKHelper singleton owns the SDK instances, a Mutex, the list of attached board windows, and a global configuration object that holds the gesture state machine’s tuning constants.

Cross-thread shared state is guarded by a portable Lock/Mutex abstraction — Win32 CRITICAL_SECTION on Windows, pthreads on Mac. The mutex protects the queue of pending board events, the active-window list, and any state that the gesture state machine reads while the polling thread is also writing it. Holding the lock across an Illustrator API call is forbidden — suite calls re-enter Illustrator’s event loop, which can deadlock against another thread that also wants the SDK lock.

The event flow from a touch on the board

The pipeline a single touch travels through, from the board glass to a transformation on a selected Illustrator object:

SMART Board hardware
  → SBSDK2 / SBSDK2Advanced  (vendor SDK; pumped from polling thread)
  → CSmartEventHandler       (implements CSBSDK2EventHandler +
                                  CSBSDK2AdvancedEventHandler)
       OnXYDown / OnXYMove / OnXYUp
       OnPenTrayButton
       OnGestureDown / OnGestureMove / OnGestureUp
       OnMultiPointContact
       OnXMLToolChangeA
       OnXMLAnnot
  → GestureSM                 (gesture state machine)
       Click / Drag / Rotation / Scale / RotationScale
       RightClick / Waggle / Pan / Duplicate
  → CMyCustomGestureEventHandler::GestureSMGesture{Start,Move,End}
  → IllustratorFunctions
       SetIllustratorToolByName, SetActiveSwatchColour,
       ScaleSelection, RotateSelection, PanSelection,
       MoveSelection, SetupDuplicate*, DrawXMLAnnotation
  → Illustrator AI suites    (sAITool, sAIArt, sAIDocument, sAIMatchingArt)

Each layer in this chain solves one problem. The vendor SDK delivers timestamped raw events. The event handler de-duplicates them and routes them by pointer ID. The gesture state machine accumulates a sequence of touches into a recognisable gesture — classifying a brief stationary touch as a Click, a longer drag as a Drag, two simultaneous touches diverging in distance as a Scale, the same two touches rotating around their midpoint as a Rotation, both at once as a RotationScale. The Illustrator function layer translates each gesture into the right scripting-suite calls.

The gesture state machine

Ten gesture states, declared in GestureSM.h:

enum GestureSMEventType
{
    Click,
    Drag,
    Rotation,
    Scale,
    RotationScale,
    RightClick,
    Waggle,
    Pan,
    Duplicate,
    DuplicateInit
};

Each state has its own classifier in the state machine. Click is a single touch that goes down and up within a configurable time window, on a single point. Drag is a single touch that moves more than a tolerance distance before lifting. Rotation and Scale are two simultaneous touches whose midpoint stays put while their angle (Rotation) or distance (Scale) changes; RotationScale is both at once. RightClick is a long press — a single touch held still for longer than the click window. Waggle is a fast left-right-left-right gesture used to deselect; the state machine watches for direction reversals within a tight time window. Pan is a translation of the whole canvas viewport rather than the selection. Duplicate is a Drag with a held second pointer, treated as “copy then drag the copy”.

Every threshold — click time, waggle distance, rotation and scale sensitivity, pan vector limits, the timer interval the polling thread spins on — lives in a SmartIllustratorConfiguration class that loads from disk and is consulted at every state-machine transition. Tuning gesture feel never required touching the state-machine code itself, only the configuration. That separation paid back constantly during user testing: the SMART team would ask for the click window to come down by 50ms or the waggle threshold to be more forgiving, and we shipped a configuration tweak rather than a code change.

The XML tool-change quirk

One of the more interesting bugs we found in the vendor SDK is that SBSDKGetXMLTool() — the API that reports which physical implement (pen, eraser, marker) is currently associated with a given pointer — was unreliable. The SDK’s own source comments document that the call can return stale values or miss tool changes entirely, particularly when multiple pointers are active. Polling it on every touch produced inconsistent tool selection in Illustrator: a user would pick up the eraser, the call would report “pen”, and they’d find themselves drawing instead of erasing.

The fix in the Fix/MySmartEventHandler.cpp branch was to maintain a per-pointer XML tool-change map, populated from the OnXMLToolChangeA notification stream as the SDK fired it. The handler trusts the notifications (which are individually delivered events) rather than the polling API (which is a state query):

// Per-pointer cache of the most recent tool-change XML
// keyed by pointer_id, parsed from OnXMLToolChangeA events
std::map<int, std::string> m_XMLToolChangeEvents;

void CSmartEventHandler::OnXMLToolChangeA(
    const CSBSDKWnd& wnd,
    const char* xml)
{
    // Extract pointer_id with TinyXML
    TiXmlDocument doc;
    doc.Parse(xml);
    int pointerId = ExtractPointerId(doc);

    // Cache the full XML payload for that pointer
    Lock guard(m_mutex);
    m_XMLToolChangeEvents[pointerId] = std::string(xml);
}

When a subsequent touch event fires for the same pointer, the handler consults m_XMLToolChangeEvents rather than calling back into the SDK. The cost is a small map keyed by pointer ID; the benefit is that tool identification became consistent even with multiple pens in simultaneous use. This is the kind of fix that doesn’t exist anywhere in the SDK documentation — you find it by watching the bug happen in production and reading the SDK source carefully enough to understand why.

Cross-platform: Windows and Mac diverge

The code base targets both Windows and macOS through the standard Illustrator SDK WIN_ENV / MAC_ENV macros, but the two paths diverged considerably during the project. Three categories of difference:

Window types

Windows uses Win32 HWND; Mac (in the era this plug-in was first written) used Carbon WindowPtr. The portable TouchEvent struct papers over the difference with a #define HWND WindowPtr on Mac, but the actual event-routing code that walks the Illustrator window hierarchy (to find the right document window to deliver the synthetic mouse event to) had to be written twice — EnumChildWindows on Windows, the Carbon HIToolbox window iterator on Mac.

Threading primitives

The polling thread on Windows starts with _beginthreadex and uses CRITICAL_SECTION for the mutex. On Mac it’s pthreads (pthread_create + pthread_mutex_t). The Mutex wrapper class hides the implementation difference; the threading lifecycle code that starts and stops the thread does not, and is conditionally compiled per platform.

Touch coordinate origin handling

This one bit us repeatedly. Windows touch events came in with screen-relative coordinates; Mac touch events came in with window-relative coordinates already transformed by the OS. The TouchEvent::GetPoint method has explicit #ifndef WIN_ENV branches handling the difference. Code that worked perfectly on Win32 produced touches landing in the wrong place on Mac until we traced the coordinate-space mismatch, which took longer than it should have because the symptom looked like a gesture-classification bug rather than an input-handling bug.

The Mac branch additionally hooks IOKit’s IOHIDLib.h for synthesising HID events that Carbon couldn’t express directly — specifically, the right-click event Illustrator expects when the gesture state machine recognises a long press. OpenHIDService / CloseHIDService wrap the lifecycle.

Lessons learned

Fork the sample, don’t link to it

The plug-in started as a fork of Adobe’s MarkedObjects SDK sample. Many file names, IDs, and copyright headers still carry the legacy: MarkedObjectsPlugin, MarkedObjectID.h, the MarkedObjectManager. The sample’s scaffolding for plug-in lifecycle, ADM dialogs, and notifiers was the right starting point; the rest is custom. Forking the sample meant we never had to fight the build to keep two trees in sync — though it did mean inheriting some commented-out fields and disabled code paths that we left alone rather than “cleaning up” without understanding why they were originally disabled.

Don’t throw from StartupPlugin

The original sample threw SDKErrors from StartupPlugin for non-fatal initialisation problems. In the field this caused crashes in the Path Construction interface during startup. The fix was to replace throw(error) with return error at the relevant sites — the comment //changed by MP to stop the crash from the Path Construction interface still flags the line in the source.

Always wrap suite calls in IllustratorContext

Any code that calls Illustrator suites from outside the main Plugin callbacks must instantiate an IllustratorContext on the stack first. Without it, suite calls hit the wrong plug-in’s context and either fail silently or crash. This caught us early in development — the polling thread’s callbacks into IllustratorFunctions mostly worked, until a particular gesture combined with a particular tool fired a suite call that needed the plug-in’s context to resolve a resource lookup, and the call returned garbage. The discipline now is universal: any cross-thread entry into Illustrator code starts with an IllustratorContext on the stack.

The two-folder convention

The repository ended up with a Source/ tree (the canonical structure inherited from the Adobe sample) and a Fix/ folder containing newer, patched versions of the most-edited files (notably MySmartEventHandler.cpp, where the per-pointer tool-change map lives). The build pulls the Fix/ versions, overriding the equivalents in Source/Source/. Anyone editing the event handler edits the Fix/ copy. We tried to reconcile the two folders periodically and stopped — the moving target made it not worth the merge cost. The convention is now part of the project’s onboarding documentation.

Watch the demo

Technology used

• Adobe Illustrator C++ SDK for native plug-in development — AITool, BaseADMDialog, the sAI* suite globals, ai::UnicodeString, ai::FilePath, the notifier system
• SMART Board SDK 2 — SBSDK2.h, SBSDK2Advanced.h, providing CSBSDK2, CSBSDK2Advanced, CSBSDKWnd, the contact list, and the event handler interfaces
• TinyXML for parsing the tool_meta_data XML the board emits when a pen, eraser, or marker is picked up from the tray
• Win32 + ATL on Windows, with a MIDL-generated COM wrapper around the SMART SDK so the plug-in can talk to the SDK across thread boundaries cleanly
• Carbon HIToolbox + IOKit HID on Mac, for window enumeration and synthesised input events
• Custom gesture state machine implemented in C++, configurable through a SmartIllustratorConfiguration file

Need a similar solution?

Hardware-to-Illustrator integration is one of the harder corners of the Adobe SDK landscape. The application is single-threaded by assumption, the suite-context discipline is unforgiving, and vendor SDKs inevitably have their own quirks. We’ve built plug-ins for Illustrator, Acrobat, InDesign, and Photoshop for over thirty years — including custom hardware bridges, gesture systems, server-side automation, and accessibility integrations. If you have a third-party system you need brought inside an Adobe application, we know how to do it cleanly.