
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# beit-sketch-classifier

This model is a version of [microsoft/beit-base-patch16-224-pt22k-ft22k](https://huggingface.co/microsoft/beit-base-patch16-224-pt22k-ft22k) fine-tuned on a dataset of Quick!Draw! sketches (~10% of [QuickDraw's 50M sketches](https://huggingface.co/datasets/kmewhort/quickdraw-bins-50M)).
It achieves the following results on the evaluation set:
- Loss: 0.7372
- Accuracy: 0.8098

## Intended uses & limitations

It's intended to be used to classifier sketches with a line-segment input format (there's no data augmentation in the fine-tuning; the input raster images ideally need to be generated from line-vector format very similarly to the training images).

You can generate the requisite PIL images from Quickdraw `bin` format with the following:

```
# packed bytes -> dict (fro mhttps://github.com/googlecreativelab/quickdraw-dataset/blob/master/examples/binary_file_parser.py)
def unpack_drawing(file_handle):
    key_id, = unpack('Q', file_handle.read(8))
    country_code, = unpack('2s', file_handle.read(2))
    recognized, = unpack('b', file_handle.read(1))
    timestamp, = unpack('I', file_handle.read(4))
    n_strokes, = unpack('H', file_handle.read(2))
    image = []
    n_bytes = 17
    for i in range(n_strokes):
        n_points, = unpack('H', file_handle.read(2))
        fmt = str(n_points) + 'B'
        x = unpack(fmt, file_handle.read(n_points))
        y = unpack(fmt, file_handle.read(n_points))
        image.append((x, y))
        n_bytes += 2 + 2*n_points
    result = {
        'key_id': key_id,
        'country_code': country_code,
        'recognized': recognized,
        'timestamp': timestamp,
        'image': image,
    }
    return result

# packed bin -> RGB PIL
def binToPIL(packed_drawing):
    padding = 8
    radius = 7
    scale = (224.0-(2*padding)) / 256
    
    unpacked = unpack_drawing(io.BytesIO(packed_drawing))
    unpacked_image = unpacked['image']
    image = np.full((224,224), 255, np.uint8)
    for stroke in unpacked['image']:
        prevX = round(stroke[0][0]*scale)
        prevY = round(stroke[1][0]*scale)
        for i in range(1, len(stroke[0])):
            x = round(stroke[0][i]*scale)
            y = round(stroke[1][i]*scale)
            cv2.line(image, (padding+prevX, padding+prevY), (padding+x, padding+y), 0, radius, -1)
            prevX = x
            prevY = y
    pilImage = Image.fromarray(image).convert("RGB")     
    return pilImage
```

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step  | Accuracy | Validation Loss |
|:-------------:|:-----:|:-----:|:--------:|:---------------:|
| 0.939         | 1.0   | 12606 | 0.7853   | 0.8275          |
| 0.7312        | 2.0   | 25212 | 0.7587   | 0.8027          |
| 0.6174        | 3.0   | 37818 | 0.7372   | 0.8098          |


### Framework versions

- Transformers 4.25.1
- Pytorch 1.13.1+cu117
- Datasets 2.7.1
- Tokenizers 0.13.2

the eiffel tower the great wall of china the mona lisa aircraft carrier airplane alarm clock ambulance angel animal migration ant anvil apple arm asparagus axe backpack banana bandage barn baseball baseball bat basket basketball bat bathtub beach bear beard bed bee belt bench bicycle binoculars bird birthday cake blackberry blueberry book boomerang bottlecap bowtie bracelet brain bread bridge broccoli broom bucket bulldozer bus bush butterfly cactus cake calculator calendar camel camera camouflage campfire candle cannon canoe car carrot castle cat ceiling fan cell phone cello chair chandelier church circle clarinet clock cloud coffee cup compass computer cookie cooler couch cow crab crayon crocodile crown cruise ship cup diamond dishwasher diving board dog dolphin donut door dragon dresser drill drums duck dumbbell ear elbow elephant envelope eraser eye eyeglasses face fan feather fence finger fire hydrant fireplace firetruck fish flamingo flashlight flip flops floor lamp flower flying saucer foot fork frog frying pan garden garden hose giraffe goatee golf club grapes grass guitar hamburger hammer hand harp hat headphones hedgehog helicopter helmet hexagon hockey puck hockey stick horse hospital hot air balloon hot dog hot tub hourglass house house plant hurricane ice cream jacket jail kangaroo key keyboard knee knife ladder lantern laptop leaf leg light bulb lighter lighthouse lightning line lion lipstick lobster lollipop mailbox map marker matches megaphone mermaid microphone microwave monkey moon mosquito motorbike mountain mouse moustache mouth mug mushroom nail necklace nose ocean octagon octopus onion oven owl paint can paintbrush palm tree panda pants paper clip parachute parrot passport peanut pear peas pencil penguin piano pickup truck picture frame pig pillow pineapple pizza pliers police car pond pool popsicle postcard potato power outlet purse rabbit raccoon radio rain rainbow rake remote control rhinoceros rifle river roller coaster rollerskates sailboat sandwich saw saxophone school bus scissors scorpion screwdriver sea turtle see saw shark sheep shoe shorts shovel sink skateboard skull skyscraper sleeping bag smiley face snail snake snorkel snowflake snowman soccer ball sock speedboat spider spoon spreadsheet square squiggle squirrel stairs star steak stereo stethoscope stitches stop sign stove strawberry streetlight string bean submarine suitcase sun swan sweater swing set sword syringe t-shirt table teapot teddy-bear telephone television tennis racquet tent tiger toaster toe toilet tooth toothbrush toothpaste tornado tractor traffic light train tree triangle trombone truck trumpet umbrella underwear van vase violin washing machine watermelon waterslide whale wheel windmill wine bottle wine glass wristwatch yoga zebra zigzag