Layout Collection and Detection

Layout Collection

To emulate Bill Cunningham’s distinct layout style, the first task is to gather layout data. In 2016, I archived all of Bill Cunningham’s “On the Street” posters from The New York Times, aiming to reuse the layouts from these iconic posters.

Therefore, following the collection of the posters, the subsequent task involved detecting the layouts to determine the placement of images and text. In total, 243 posters were collected, and below are four exemplars:

2011-04-03 2011-06-05
2011-09-25 2011-10-16

At first glance, the posters’ design seems straightforward—a collection of rectangles in various sizes. Yet, these rectangles are arranged in a harmonious composition. The images inside each rectangle are artfully cropped to emphasize the most captivating elements. It’s conceivable that the prominence of the featured item influences the rectangle’s size.

While Bill’s process was manual, we aspire to replicate his distinctive style through computer vision technology.

Layout Detection

To adopt Bill Cunningham’s poster layout, an algorithm is crafted to identify the rectangular areas within the original poster. By pinpointing every rectangle’s position, the placement of images can be effectively determined.

Below is an outline of the process:

  • Image preprocessing:
    • Convert the poster image to grayscale.
    • Apply a threshold to transform the grayscale image into a black-and-white one.
    • Denoise the grayscale image with morphological transformations, aiming to remove noise between the rectangles.


    Original -> Grayscale


    Black and White -> Denoise

  • Implement two iterations of rectangle detection on the denoised black-and-white image:
    • Detect connected components within the image.
    • Identify the minimum area rectangle for each component.
    • Record the corner points for every identified rectangle.
    • Overlay the detected rectangles onto the image.
    • Repeat steps 4.1 to 4.4 on the image now containing the initial rectangles.
    • The final result will be the rectangles marked on an image with a black background and white rectangles.


    Denoise -> Rectangle Detection

  • Identify the text rectangle:
    • Map the image rectangles on a black background, outlined in white.
    • Trim black edges by placing the rectangle area on a new white background image.
    • Enlarge the rectangles to cover any black gaps, so only the text block remains.
    • Identify the largest connected black area as the text block.


    Crop Outer -> Enlarge and Cover -> Detect Text Rectangle

  • Fine-tune the spacing between rectangles.


    Before adjustment -> After adjustment

  • Illustrate the text block in red to differentiate it from image rectangles. Output the accurately detected rectangles and text block for poster layout assembly.


    Detected and adjusted image rectangles and text rectangle.

Text Extraction

The textual content within the poster is also extracted, utilizing the Tesseract Open Source OCR Engine facilitated by pyocr.

For example, the text content extracted from the poster above is:

ON THE STREET Bill Cunningham Fox Hunt Out of the past comes the springtime fox-fur scarf that glamorized fashion from 1910 to 1950, when the show was stolen by the mink stole. In Paris, while $5,000 real fox scarves are still being worn, they have been outfoxed by a fabu- lous $500 fake in stripes and colors from Prada.

The text content extracted from the posters, while not flawless, is predominantly accurate. Given that text extraction is not the central aspect of this project, the level of precision achieved is sufficient for our needs.

The extracted text from the poster is captured and paired with its corresponding image to form an (image, text) pair dataset. This dataset serves as reference material for GPTs to name and generate descriptions for newly created posters in a similar style.

Layout Detection Results

Out of 243 posters processed, text rectangles are successfully detected in 220 of them. The distribution of image rectangles identified in each poster is depicted in the graph below, showing a median number of 34 rectangles per poster.


Text Detection Failures

In the 23 instances where text rectangles failed to be detected, here are some examples illustrating these failures:

Failure Case 1


In this case, the layout deviates from the norm, and the text isn’t located within a standard rectangular area.

Failure Case 2



This particular case presents an intriguing scenario. The text is situated within a rectangular area, yet it remains undetected. The text block appears to be missed after the second iteration of rectangle detection, likely due to a misstep in detecting the image rectangles.

Image Rectangle Detection Failures

Posters featuring a relatively small number of image rectangles, particularly those with fewer than 10, are examined. It is probable that the detection algorithm encountered difficulties in these instances, mostly due to the unconventional layouts of these posters.

Below are two examples demonstrating such cases:

Failure Case 1


Failure Case 2


Success Cases

Apart from these instances of failure, the detection algorithm generally performs effectively. Below are some examples showcasing its success:

Good Case 1


‘ORAY MATTERS A soman.on ith Avenue wore a shirt proclaiming “GREY IS THE NEW BLACK” Gray isthe summer shade inthe city. There are gray handbags; note thebriefease ofa manina, ‘ray suit and shirt (botom right) Silver jewelry and gray pearls add tothe effec. And women with gray hae are having afashion moment.

Good Case 2


ON THE STREET Bill Cunningham Upsize Last week, cherry blossoms were falling all over the city, and children chased them in the same way they chase snowflakes. The most spectac- Uular drifts of this pink snow were to be seen along Park Avenue, where cherry tees line the centersland, and speeding cars cause the petals to swirl and dance. On the fashion scene, there’s anew interest in the carryall bag. Various versions have been around for a decade, bottom left, but now some of the most sought-af- ter are from the Paris house Céline, right. Devoid of hardware, they rely on the simplicity of design and crafts- manship. Another status carryall, Jef is marked witha for Yves Saint Laurent. The large bag dripping in fFinge, below, is back; this time, it has nothing to do with the hippie era.

Good Case 3


‘THE WINNING NUMBER 15..-Inayear when fashion accessories are the stars, the see- ‘through bag shaped like a Chanel No Sbot- Uleisthe most whims cal. It seemed tosellout immediately andis now collector’s item, Shown here, itis carried in daytime by all but two of the women. And for ‘those who cant get enough of the perfume bag, there’sa smartphone cover (far right :olumn and top right) We haven’t seena ‘ovelty ike this since Sciapareli’s Surreal- stacessories in the late 1930s. There are ven Chane! sunglasses, with mademoislle Derched atop the frame (inset.

Implementation Details

Input

Name Description
bill_template A folder containing the original posters

Folder Structure:

bill_template
├── 2011-03-06.jpg
├── 2011-03-13.jpg
├── ...
└── 2016-06-12.jpg

Process

Code Description
codes/poster_generation/extract_layout.ipynb Detect the layout of the posters

Output

Name Description
bill_template_processed_results.csv The layout detection results for each poster
bill_template_processed A folder containing the processed posters

Data Sample:

{
    "img_name": "2014-11-09.jpg",
    "shape": "{'width': 970, 'height': 772}",
    "text": "MARATHON MEDLEY It seemed asifa kaleidoscope of colors had exploded on the runners participating in thefve- borough New York City Marathon last \u2018Sunday. Ending in Central Park, the run was set against a backdrop of au \u2018umn eaves, Some ofthe international runners sported traditional garb from theirhome countries. Curiously, a \u2018numberof runners wore tutus",
    "img_path": "bill_template/2014-11-09.jpg",
    "res_path": "bill_template_processed/2014-11-09/2014-11-09.json",
    "has_text": true,
    "num_img_rects": 57
}

Result Sample:

{
    "img_width": 970,
    "img_height": 765,
    "text_content": "\u2018FINALLY, SOMETHING TO WEAR TO THE OFFICE\u2019 piversty in alts facets was vividly displayed on the guests attending last week's near-300 fashion shows and presentation. \u2018At times, It seemed as if we were ack in the 1980s, with the tattooed, pierced and embellished \u2018downtown kids (spotted at Jeremy Scot, Gypsy Spor, Hood by Air and Xuly:Bet) presenting a \u2018radical contrast to uptown\u2019 simplicity of style (seen at Ralph Lauren, Michael Kors and Proenza Schouler). Each sroup interprets taste or its este",
    "img_rects": [
        [314, 33, 418, 197],
        [775, 582, 860, 749]
    ],
    "img_rects_adjusted": [
        [310, 6, 424, 205],
        [775, 582, 860, 759]
    ],
    "text_rect": [232, 327, 665, 402],
    "text_rect_adjusted": [232, 327, 669, 406]
}