AI Meets Beauty

Perfect Half Million Beauty Product Image Recognition Challenge

Artificial intelligence (AI) is currently driving a global revolution, reshaping industries, including the multibillion-dollar beauty and personal care industry. In order to promote impactful research and promote problem solving in this disruptive AI beauty space, leading beauty app developer, Perfect Corp.*, in collaboration with CyberLink Corp., National Chiao Tung University, Tsinghua University, Peking University, Beihang University, Microsoft Research, and Institut Teknologi Sepuluh Nopember, are providing a large-scale image dataset of over half million images of beauty and personal care products, namely the Perfect-500K dataset. The Perfect-500K dataset is vast in scale, rich, and diverse in content collecting as many beauty and personal care items as possible from major e-commerce sites.

*Reaching over 800 million app downloads for its YouCam Apps suite, installs worldwide as of November 2019.

AI Meets Beauty

Perfect Half Million Beauty Product Image Recognition Challenge

As the revolution of artificial intelligence (AI) is underway around the globe, AI has already begun to reshape industries including the multibillion dollar beauty and personal care industry. In order to promote impactful research and problem solving in this disruptive AI beauty space, the leading beauty app developer, Perfect Corp. in collaborating with CyberLink Corp. and National Chiao Tung University to provide a large-scale image dataset of over half million images of beauty and personal care products, namely the Perfect-500K dataset. Perfect-500K is vast in scale, rich and diverse in content in order to collect as many as possible beauty and personal care items from major e-commerce sites.

Photos of the top-3 winners of the 2nd “AI Meets Beauty” Challenge in ACM Multimedia 2019

The first edition of “AI Meets Beauty” Challenge, exploring the Perfect-500K dataset was held at ACM Multimedia 2018 ( It was a very successful event and received excellent media coverage ( As an inaugural event, the challenge attracted a strong number of participants, with a total of 97 teams from 13 countries around the world. The second edition was again held in conjunction with ACM Multimedia 2019 (, with a total of 55 teams from 7 countries. Figure 1 shows the breakdown of the location of participants. Table 1 and Table 2 show a ranking list of the top-5 teams in terms of evaluation metrics in the first and the second “AI Meets Beauty” Challenge at ACM Multimedia, respectively. Based on the reported performance results that generally increase from 2018 to 2019, the Challenge rounds do appear to be effective in overcoming barriers to beauty and personal care product recognition, and thus did help improve the fashion research.

Figure 1. The breakdown of the location of participants in the second “AI Meets Beauty” Challenge at ACM Multimedia 2019.

Rank Team Name Score(mAP)
1 VCA 0.577567
2 GAN 0.291350
3 WISLAB 0.270676
4 CISIP 0.251895
5 ivlab2017 0.229133

Table 1. A ranking list of the top-5 teams in terms of evaluation metrics in the first “AI Meets Beauty” Challenge at ACM Multimedia 2018.

Rank Team Name Score(mAP)
1 USTC-NELSLIP 0.408614
2 007 0.407233
3 CVTE-Research-DM 0.364021
4 the makeup blogger 0.327452
5 GoGo 0.31181

Table 2. A ranking list of the top-5 teams in terms of evaluation metrics in the second “AI Meets Beauty” Challenge at ACM Multimedia 2019.

Following the success of our first and second “AI Meets Beauty” Challenge, this year we will organize the third “AI Meets Beauty” Challenge at ACM Multimedia 2020. A new test dataset for evaluations will be released for this challenge. Furthermore, we will publish all information related to this challenge on a website, and commit to maintaining it for at least the next three years.


The “AI Meets Beauty” Challenge 2020 is a team-based competition. Each team can have one or more members, and each individual can only be part of one team. This challenge is open to the public. Interested individuals, researchers/developers of tertiary education, research institutes or organizations from different sectors and fields are welcome to take part independently or as a team.
At the end of the challenge, all teams will be ranked based on both objective evaluation and subjective human evaluation criteria described above. The top three performing teams will receive award certificates and cash prizes:

At the same time, all accepted submissions will qualify for the conference’s grand challenge award competition. Each winning team for the cash prizes will be required to open source their proposed solution on Github before qualifying to receive the cash prizes.


Unless otherwise stated, all deadlines will be set at 23:59 Anywhere on Earth (AoE), UTC-12.
please follow the instructions on the main conference website

Dataset available for download April 1, 2020
Test set available for download and start to submit result June 1, 2020
Results submission deadline June 4, 2020
Objective evaluation and human evaluation starts June 8, 2020
Objective evaluation and human evaluation ends June 12, 2020
Evaluation results announced June 15, 2020
Paper submission deadline June 29, 2020


If you are reporting results of the challenge or using the dataset, please cite with:

					  	@misc{perfecthalfmillionchallenge2020, title={Perfect Corp. Challenge 2020: 
Half Million Beauty Product Image Recognition}, author={Wen-Huang Cheng,
Jia Jia, Si Liu, Jianlong Fu, Jiaying Liu, Shintami Chusnul Hidayati,
Johnny Tseng, and Jau Huang}, howpublished=
{\url{}}, year={2020}}

Paper Submission

We will follow the guideline of ACM Multimedia 2020 Grand Challenge for the paper submission.

Main Organizers

  • Jau Huang

    Jau Huang

    CyberLink Corp.

  • Johnny Tseng

    Johnny Tseng

    Perfect Corp.

  • Wen-Huang Cheng

    Hong-Han Shuai

    National Chiao Tung University

  • Wen-Huang Cheng

    Wen-Huang Cheng

    National Chiao Tung University

  • Jia Jia

    Jiaying Liu

    Peking University

  • Jia Jia

    Jia Jia

    Tsinghua University

  • Si Liu

    Si Liu

    Beihang University

  • Jianlong Fu

    Jianlong Fu

    Microsoft Research

  • Jia Jia

    Shintami Chusnul Hidayati

    Institut Teknologi Sepuluh Nopember


PerfectCorp is dedicated to transforming how consumers, content creators and beauty brands interact together. Our unique platform is the premier online destination for all beauty lovers. Our PERFECTTM apps, YouCam Makeup, recently voted Best Fashion & Apparels App* of 2015, and market leading YouCam Perfect redefine virtual beauty for tens of millions of users.
*YouCam Makeup is the winner of 2015 App Award of Best Fashion & Apparels App.

Founded in 1996, CyberLink Corp. is the world’s leading multimedia software company and pioneer in video and audio technologies on PCs and portable devices including tablets and smartphones. From what began as a mission to create superb digital multimedia products for consumers by a group of National Taiwan University students led by Professor Huang, CyberLink has since grown into a global and award-winning brand with nearly 30 products and a solid reputation for efficiently delivering innovative, interoperable solutions.

Task Chairs & Main Contact


Perfect-500K is a collection of beauty and personal care items from 14 popular e-commerce sites, including Amazon (USA, India), Cult Beauty, Flipkart, Galleria, Gmarket, JD.COM, Nordstrom, Sephora, Strawberrynet, Target, Walgreens, Walmart, and Yahoo Shopping Mall. Table 3 shows the detailed statistics of the dataset.

Source # of images
Amazon (India) 208,936
Amazon (USA) 129,192
Flipkart 38,726
Walmart 34,428
Yahoo Shopping Mall 22,407
Gmarket 19,579
Strawberrynet 16,040
Target 11,896
Nordstrom 8,786
Galleria 8,142
Walgreens 7,273
Sephora 7,099
JD.COM 5,084
Cult Beauty 3,139
TOTAL 520,727

Download Link

Note that the datasets will ONLY be released to participants who have registered for the challenge during the competition period. After the challenge is completed, we will make the data publically available to the whole research community.

Task Description

This year we will focus on one particular task: beauty and personal care product recognition. Meanwhile, we are open to innovative self-proposed topics.

Main Task: Beauty and personal care product recognition

Given a real-world image containing one beauty or personal care item (some samples are given in Figure 2), the task is to match the real-world example of this item to the same item in the Perfect-500K data set. This is a practical but extremely challenging task, given the limitation that only images from e-commerce sites are available in Perfect-500K and no real-world examples will be provided in advance.


Figure 2. The real-world examples of beauty and personal care items.

Additional Task: Open topics

To encourage the exploration of the Perfect-500k application scope, we also accept innovative topics proposed by the participants themselves, e.g., object localization, data labeling, etc. For the open topics, the participants need to clearly define the topic, demonstrate the technical advancement of their proposed solutions, specify the evaluation protocols, and provide Perfect-500k based experimental results.

Submission Format & Evaluation Metrics


We require every participant to submit the Docker images for our evaluation. This method enables us to keep the test set secret and to easily reproduce the result. Each team is allowed to submit the results of at most 3 runs, selecting one run as the primary run of the submission. When submitting, they should specify 3 URLs at most for us to download their images.

The format of the output CSV file as a result of the execution is the same as that for last year’s challenge: each line contains an image id followed by 7 predicted labels sorted by confidence in descending order. It looks as follows:

<Test Image ID id#1>, <Training Image ID id#1_1>, <Training Image ID id#1_2>, ..., <Training Image ID id#1_6>, <Training Image ID id#1_7>
<Test Image ID id#2>, <Training Image ID id#2_1>, <Training Image ID id#2_2>, ..., <Training Image ID id#2_6>, <Training Image ID id#2_7>

<Test Image ID id#N>, <Training Image ID id#N_1>, <Training Image ID id#N_2>, ..., <Training Image ID id#N_6>, <Training Image ID id#N_7>

Preparation of the Image

We suggest each team base their Docker image on Ubuntu:16.04 or above and test their solution on it. Once they are ready to submit the image, they need to perform the following steps:

1. docker commit --change 'ENTRYPOINT ["<main_executable>"]' <container_id> <their_image>
2. docker save <their_image> | gzip -c > <saved_image_file>
3. upload the file to Google Drive and submit the downloadable URL to us

In the first command , the <main_executable> should be an executable, e.g., bash script, which accepts 2 arguments described below.

We provide a sample image at

The ENTRYPOINT in the sample is set to ["/challenge/predict"]. The /challenge/predict script calls to actually perform the prediction and write the output to the given path.

How We Run the Image

1. curl -L -o <downloaded_image> <url>
2. docker load < <downloaded_image>
3. docker run –-network none -ti -v <testset_path>:/testset:ro -v /result --name <container_name> <their_image> /testset /result/predictions.csv
4. docker cp <container_name>:/result/predictions.csv <all_results_path>

Note that:

  • The third command accepts 2 arguments:
    a. /testset is the path to the folder of the test images:

    b. /result/predictions.csv is the path to the output CSV file
  • The image is run under CPU mode
  • There is no internet access in the container
  • The final command basically copies the produced CSV file to our local path so that we can evaluate your result.
  • The total execution time must not be over 1 hour. The score will be 0 if the execution can not produce the expected output or not finish within 1 hour.

Please test your image with these commands before submitting.

Hardware Used to Run the Image

AWS EC2 instance of type r4.4xlarge:

  • Dual socket Intel Xeon E5 Broadwell Processors (2.3 GHz), 16 vCPUs
  • DDR4 memory, 122GB

Evaluation Metrics

For each test image, the evaluation metric is Mean Average Precision@7 (MAP@7). The performance of each individual test image is then averaged to evaluate the submissions to this challenge.

Leader Board

Rank Team Name Score(mAP)
1 USTC_NELSLIP 0.534848
2 GDUT611 0.43729
3 NTU-Beauty 0.405302
4 bladefashion 0.402402
5 toan 0.375517
6 TSST 0.362986
7 WHO_Knows 0.323124