Submission Format & Evaluation Metrics
We require every participant to submit the Docker images for our evaluation. This method enables us to keep the test set secret and to easily reproduce the result. Each team is allowed to submit the results of at most 3 runs, selecting one run as the primary run of the submission. When submitting, they should specify 3 URLs at most for us to download their images.
The format of the output CSV file as a result of the execution is the same as that for last year’s challenge: each line contains an image id followed by 7 predicted labels sorted by confidence in descending order. It looks as follows:
<Test Image ID id#1>, <Training Image ID id#1_1>, <Training Image ID id#1_2>, ..., <Training Image ID id#1_6>, <Training Image ID id#1_7>
<Test Image ID id#2>, <Training Image ID id#2_1>, <Training Image ID id#2_2>, ..., <Training Image ID id#2_6>, <Training Image ID id#2_7>
<Test Image ID id#N>, <Training Image ID id#N_1>, <Training Image ID id#N_2>, ..., <Training Image ID id#N_6>, <Training Image ID id#N_7>
Preparation of the Image
We suggest each team base their Docker image on Ubuntu:16.04 or above and test their solution on it. Once they are ready to submit the image, they need to perform the following steps:
1. docker commit --change 'ENTRYPOINT ["<main_executable>"]' <container_id> <their_image>
2. docker save <their_image> | gzip -c > <saved_image_file>
3. upload the file to Google Drive and submit the downloadable URL to us
In the first command , the <main_executable> should be an executable, e.g., bash script, which accepts 2 arguments described below.
We provide a sample image at https://hub.docker.com/r/aimeetsbeauty/sample-submission.
The ENTRYPOINT in the sample is set to ["/challenge/predict"]. The /challenge/predict script calls random_predict.py to actually perform the prediction and write the output to the given path.
How We Run the Image
1. curl -L -o <downloaded_image> <url>
2. docker load < <downloaded_image>
3. docker run –-network none -ti -v <testset_path>:/testset:ro -v /result --name <container_name> <their_image> /testset /result/predictions.csv
4. docker cp <container_name>:/result/predictions.csv <all_results_path>
Please test your image with these commands before submitting.
Hardware Used to Run the Image
AWS EC2 instance of type r4.4xlarge:
- Dual socket Intel Xeon E5 Broadwell Processors (2.3 GHz), 16 vCPUs
- DDR4 memory, 122GB
For each test image, the evaluation metric is Mean Average Precision@7 (MAP@7). The performance of each individual test image is then averaged to evaluate the submissions to this challenge.