Medical image annotation has a different center of gravity than any other domain: the boundary is the diagnosis. A bone-fracture detector, a joint-space measurement model, or an implant-placement assistant lives or dies on whether its training masks follow true anatomical boundaries - cortical edges, joint margins, fracture lines - rather than approximate blobs. Payana treats medical work as its own engine, not a re-skinned general pipeline, and a bone X-ray dataset has already been produced on the platform.
The stakes are also different. A mislabeled car costs a perception model some accuracy; a mislabeled fracture teaches a clinical model to miss one. That is why the medical workflow on Payana is built around a simple rule: the AI proposes, a human disposes. Model-generated masks are treated as fast drafts for expert correction, never as final answers.
A dedicated medical segmentation engine
Payana projects can be created in Medical mode. Medical projects route to a segmentation model fine-tuned specifically on medical imagery - X-ray, dermoscopy, endoscopy, microscopy, ultrasound and more - built as an adapter on top of a state-of-the-art foundation segmentation model. The key difference from general segmentation is that the medical engine is concept-prompted: the label's clinical name - "femur", "fracture line", "skin lesion" - is part of the prompt, so the model searches the image for that concept, and the annotator's click simply disambiguates which instance is meant (the left femur, not the right).
This design exists because general-purpose segmentation models notoriously underperform on radiographs. They are trained on natural photographs, where objects announce themselves with texture and contrast; bone edges on an X-ray are gradual intensity gradients, overlapping structures superimpose, and the "object" is often defined by anatomy rather than appearance. A medically fine-tuned engine is the difference between a usable pre-label and a from-scratch hand-tracing job.
What we annotate in medical projects
- Bone and anatomy segmentation - per-bone instance masks on X-rays (femur, tibia, radius, vertebrae, phalanges), joint regions, and implant hardware.
- Pathology localization - bounding boxes or masks on fractures, lesions, opacities, and foreign objects.
- Keypoint annotation - anatomical landmarks for measurement models: joint centers, cortical reference points, and the angle landmarks used in orthopedic planning.
- Study-level classification - normal/abnormal tags, view labels (AP, lateral, oblique), and quality flags for unreadable films.
All standard image formats are accepted (PNG, JPEG, and frames exported from DICOM). For research datasets we annotate at native resolution, so subtle fracture lines are not lost to downscaling before a human ever sees them.
Workflow: clinical accuracy at annotation speed
The annotator selects the label - say, fracture - and clicks the suspicious region. The medical engine, prompted with that concept, returns every candidate region it finds in the image; the click picks the one the annotator meant, and the winning mask lands in the editor as an editable polygon. From there the reviewer adjusts boundaries vertex-by-vertex wherever clinical judgment disagrees with the model. For fluoroscopy, endoscopy, or ultrasound clips, the same whole-video tracking used in other domains propagates a structure through the sequence frame-by-frame, with the same occlusion-aware behavior when instruments cross the field.
Data handling expectations
Medical projects are expected to arrive de-identified - no patient names, IDs, or burned-in PHI in the pixel data - and we advise clients on preparing exports that way. Data is stored in access-controlled cloud storage, processing happens on infrastructure we control end-to-end (no third-party labeling marketplaces), and datasets are only ever delivered back to the originating client. For regulated studies we align handling with the client's own compliance framework rather than making blanket certification claims.
Bone segmentation in practice: what actually makes it hard
Bone work sounds simple - bones are bright on X-rays - until you annotate your first hundred films. Structures superimpose: on a lateral knee, the femoral condyles overlap and the "boundary" of each is partly a clinical convention. Growth plates in pediatric films look like fractures to the untrained eye and to most models; sesamoid bones appear and disappear across patients; orthopedic hardware (plates, screws, wires) both occludes anatomy and creates scatter artifacts that fool naive edge detection. A usable bone dataset encodes decisions about all of these, made once, in writing, and applied uniformly: how overlapping regions are assigned, whether the growth plate splits the bone into two masks or one, how hardware is labeled (its own class, in our default schema) and whether the bone mask continues behind it. This is the unglamorous work that separates a dataset a model can learn from and a folder of pretty masks - and it is where a service with a written protocol beats ad-hoc annotation every time.
From pilot to publication-grade dataset
Medical engagements follow a stricter ramp than other domains. We start with a pilot set - typically a few dozen studies - annotated against a draft labeling guide, then review the pilot with the client's clinical lead. Every disagreement becomes a written rule in the guide. Only then does bulk annotation start, with the pilot set doubling as a gold standard: annotators are periodically measured against it, and drift is caught in days rather than discovered at delivery. For datasets intended for publication or regulatory submission, we add double-annotation on a sampled subset and report inter-annotator agreement per class - the number reviewers will ask for - plus a dataset card describing capture sources, demographics where known, class balance, and the exact labeling rules applied.
Why annotation is the bottleneck in medical AI
The scarce resource in medical AI is not images - hospitals generate them by the million - but expert time. Every hour a radiologist spends tracing outlines is an hour of the most expensive labor in the pipeline. The economic logic of Payana's medical workflow is to move expert time up the value chain: the concept-prompted engine and trained annotators produce the geometry, and clinical experts spend their hours on judgment - reviewing boundaries, adjudicating ambiguous pathology, signing off the gold set. In practice this means a clinician reviews in minutes what would have taken hours to draw, and the dataset gets more expert eyes per rupee, not fewer.
Beyond X-ray: building multi-modal medical datasets
The same project structure extends across modalities, and mixed-modality datasets are increasingly the point - a dermatology model trained on both clinical photos and dermoscopy, an orthopedic tool that sees X-ray and intra-operative fluoroscopy. Video modalities (endoscopy, surgical recordings, ultrasound sweeps) use the platform's tracking: an instrument or anatomical structure is labeled once and followed frame-by-frame through the sequence, with occlusion handling covering the constant instrument crossings of surgical video. Frame tags mark clinically meaningful events - insufflation, instrument changes, bleeding - turning raw procedure video into structured surgical-phase datasets for workflow-recognition models.
Choosing the right supervision: detection, segmentation, or measurement
Not every medical model needs masks, and over-annotating wastes clinical budget. A triage model that flags studies for radiologist priority may need only study-level classification - cheap, fast, and collectible at scale. A localization aid that points at the suspicious region needs bounding boxes. A surgical-planning or measurement tool needs pixel-accurate segmentation and landmarks - the expensive kind. Part of every medical engagement is mapping the clinical claim the product will make to the cheapest supervision that supports it, then designing the schema so the dataset can be upgraded later: boxes drawn today can be refined into masks tomorrow within the same project, without re-annotating from zero. This staged approach is how research groups stretch a fixed annotation budget across a publication and the follow-up study.
Common medical dataset requests
The patterns we see most: fracture detection datasets (extremity X-rays, boxes or masks on fracture lines, view and quality tags); bone segmentation datasets for measurement and surgical-planning models (per-bone masks, landmark keypoints, hardware as its own class); dermatology datasets (lesion masks with diagnosis-level classification); surgical-video datasets (instrument and anatomy tracks plus phase tags for workflow recognition); and microscopy datasets (cell and structure instance segmentation, where per-image object counts run into the hundreds and click-to-segment assistance saves the project). Each of these has a reference schema we adapt rather than reinvent, which is a large part of why medical pilots on Payana converge in one review round instead of three. Teams searching for a medical image annotation service, X-ray bone segmentation labeling, or a radiology dataset partner that pairs a medical segmentation engine with disciplined clinical QA will find that this is the exact stack described above - engine, protocol, gold sets, agreement metrics - not a general pipeline with a medical label on it.
Landmarks and measurements: the geometry of orthopedics
A distinct and growing category of medical request is not "find the pathology" but "measure the anatomy": leg-length discrepancy, joint-space width, Cobb angle for scoliosis, implant alignment angles. These models train on landmark datasets - precise anatomical keypoints placed to sub-millimeter tolerance at image resolution - and their annotation discipline differs from mask work. Landmark definitions must be operationalized ("the most inferior point of the medial femoral condyle" - with a reference image, because two annotators will otherwise pick points millimeters apart), placement happens at full zoom on native-resolution images, and QA is statistical: repeat placements on a sampled subset, with per-landmark standard deviation reported so the client knows the measurement noise floor their model inherits from the labels. Payana's keypoint project type carries configurable landmark schemas per anatomy, and landmark, mask, and classification labels coexist in one project - because a real orthopedic dataset usually needs all three from the same films.
General models are trained on natural images, where objects have texture and contrast boundaries. Bone edges on radiographs are gradual intensity gradients - general models under-segment or bleed across joints. Payana's medical engine is fine-tuned on medical imagery specifically to follow those boundaries.
Yes. Many clients have their own radiologist or orthopedist do final sign-off; we deliver review-ready projects where every mask remains editable, and incorporate clinical corrections into the final export.
X-ray and any modality exportable as standard images or video: dermoscopy, endoscopy and surgical video, ultrasound clips, microscopy, and fundus photography. CT and MRI are annotated slice-by-slice as image series.
By agreement. We can run double-annotation on a sampled subset and report inter-annotator agreement (IoU) so you can quantify label reliability before training a clinical model on it.