Results of an Analysis of 2D Log Files of Recent Tournaments
Dear 2D Community! [Same text will be shared in Discord as well!]
TL&DR;
We analyzed the log files of major competitions during past years and found a degradation of the reliability of the hardware setup and its computational power provided to the teams. In particular, we observed that - starting in 2022 - many teams' players did frequently miss to send a command within 100ms.
If you have any questions or remarks on this, please share them. Best regards, Thomas Gabel (on behalf of the FRA-UNIted team).
=====FULL EXPLANATION=====
~~Assumptions~~
We assume that all teams do, in general, send at least one command to the soccer server each time step. By commands we refer to their (A) body-related commands (dash, kick, turn, catch, tackle) and their (B) other-parts-related commands (attentionto, turn_neck, say, change_view, pointto). It is standing to reason to assume that each player sends at least one of those commands to the server each time step. Moreover, most teams do indeed send one of the body-related commands (dash, kick, turn, catch, tackle) to the server each time step.
~~Definitions~~
If some agent does not send a command in one time step at all (neither from category (A) nor from category (B)), then usually something is wrong. In accordance to Ralf Berger from the first Soccer Simulation 2D Champion team ever (AT-Humboldt'97) we call this situation a HOLE.
If some agent sends in one time step t more than one command, then usually something is very wrong (e.g. two dashes, or a dash and a turn). It usually means that this agent did not manage to send the first of the two commands within the 100ms of the previous time step t-1, i.e. it is being sent one step too late. Simultaneously, the agent sends the command of the current time step t. Thus, the server receives the command from the previous *and* the current time step in the same cycle. In accordance to Ralf Berger from the first Soccer Simulation 2D Champion team ever (AT-Humboldt'97) we call this situation a BLACK HOLE.
~~History~~
Holes and black holes are bad things for any agent. They used to be a big issue in the early days of soccer simulation 2D, when computational resources were scarce (e.g. the 12 processes of one team had to be run on just one or two single-core PC) as well as when the network connections needed for exchanging messages via UDP were not as reliable as they are today. While we believed that holes and black holes are a problem of another decade and they occur only very rarely or due to programming errors by some team nowadays, we observed their systematic return in competitions of the last and current year where, as far as we can tell, for the first times tournaments were run on 12 docker instances for each team and one for the rcssserver (in total 25 docker instances) entirely on one single machine (which, of course, is equipped with a sufficient amount of cores, e.g. 64 cores).
~~Methodology~~
We took the log files from the 2019, 2021 and 2022 world championships plus the very recent IranOpen 2023 tournament and analyzed the rcl files for the occurrence of holes and black holes. Our analysis tool and script are available on request.
~~Findings~~
In RoboCup 2019, there were no problems across all teams (with the exception of Ri-One which we think to be due to a team-internal problem). And a minor issue for YuShan2019 (with, however, less than 1 black hole in the course of two matches).
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch --------------------------------------------------------------------------- Receptivity, 22, 0, 0, 0.0, 0.0 MT2019, 20, 0, 0, 0.0, 0.0 ITAndroids, 21, 0, 0, 0.0, 0.0 Fractals2019, 22, 0, 0, 0.0, 0.0 Ri-one, 21, 18938, 572, 901.8095, 27.238094 YuShan2019, 21, 9, 9, 0.42857143, 0.42857143 CYRUS2019, 22, 0, 0, 0.0, 0.0 HillStone, 21, 0, 0, 0.0, 0.0 Razi2018, 20, 0, 0, 0.0, 0.0 FRA-UNIted, 20, 0, 0, 0.0, 0.0 FCP, 20, 0, 0, 0.0, 0.0 RoboCIn, 22, 0, 0, 0.0, 0.0 HfutEngine2019, 20, 0, 0, 0.0, 0.0 HELIOS2019, 20, 0, 0, 0.0, 0.0 Titas2019, 20, 0, 0, 0.0, 0.0
Likewise, there were no issues for RoboCup 2021:
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch --------------------------------------------------------------------------- Austras2D, 35, 812029, 0, 23200.828, 0.0 Persepolis, 35, 7916, 0, 226.17143, 0.0 Oxsy, 35, 0, 0, 0.0, 0.0 YuShan2021, 36, 0, 0, 0.0, 0.0 CYRUS, 36, 0, 0, 0.0, 0.0 HfutEngine2021, 38, 11, 0, 0.28947368, 0.0 ITAndroids, 38, 0, 0, 0.0, 0.0 Hades2D, 36, 1034, 0, 28.722221, 0.0 RoboCIn, 36, 16, 0, 0.44444445, 0.0 MT2021, 36, 0, 0, 0.0, 0.0 ARAS, 35, 6, 0, 0.17142858, 0.0 Jyo, 35, 0, 0, 0.0, 0.0 HELIOS2021, 35, 8, 1, 0.22857143, 0.028571429 FRA-UNIted, 37, 0, 0, 0.0, 0.0 Alice2021, 35, 0, 0, 0.0, 0.0 ThunderLeague, 36, 0, 0, 0.0, 0.0
Some teams (e.g. Austras or Persepolis) seem to refrain from sending a command in each time step, producing what we call a hole. Each team must decide for itself whether this is intended, or acceptable, or bad behavior (so we assume that this large amount of holes is fine for the two team mentioned).
Anyways, black holes are always most likely a bad thing for any team. If we count the total number of black holes for RoboCup 2021, we see that there was just a single black hole in the entire tournament (by HELIOS2021).
Unfortunately, things changed in RoboCup 2022 where, for the first time, a docker-based environment was used. Unfortunately, the entirety of logs for that year can no longer be found on archive.robocup.info which is why our analysis focuses only on the fraction of logs that we downloaded during the tournament:
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch --------------------------------------------------------------------------- MT2022, 32, 52, 12, 1.625, 0.375 YuShan2022, 36, 416, 248, 11.555555, 6.888889 Alice2D, 32, 73671, 7, 2302.2188, 0.21875 ITAndroids, 34, 53, 6, 1.5588236, 0.1764706 Oxsy, 32, 70, 19, 2.1875, 0.59375 HELIOS2022, 31, 21, 12, 0.67741936, 0.38709676 RoboCIn, 34, 51, 16, 1.5, 0.47058824 Hfutengine2022, 38, 129, 56, 3.3947368, 1.4736842 Tehran2D, 36, 72, 18, 2.0, 0.5 CYRUS, 33, 51, 17, 1.5454545, 0.5151515 FRA-UNIted, 32, 161, 139, 5.03125, 4.34375 Hades2D, 32, 59, 7, 1.84375, 0.21875
The summed number of black wholes has been increased to 557 in the entire tournament - a number which is certainly far beyond anything that is acceptable. Apparently, all teams were affected where YuShan2022 was affected most with almost 7 black holes per game on average.
As a safety check, we ran our analysis also on a big part of the log files produced by the recent IranOpen 2023 tournament which was also held in a docker-based environment with each agent/coach running in its own docker instance, all on the same single machine.
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch --------------------------------------------------------------------------- YuShan2023, 15, 63, 32, 4.2, 2.1333334 FRA-UNIted, 22, 113, 100, 5.1363635, 4.5454545 Mcafee, 8, 26, 4, 3.25, 0.5 Mrl, 14, 361, 289, 25.785715, 20.642857 The8, 20, 54, 13, 2.7, 0.65 Robo2D, 7, 242, 131, 34.57143, 18.714285 Pyrus, 14, 50, 23, 3.5714285, 1.6428572 Apollo2d, 9, 0, 0, 0.0, 0.0 RoboCIn, 15, 11, 0, 0.73333335, 0.0 ThunderLeague, 12, 11, 0, 0.9166667, 0.0 Zina-AI, 17, 44, 21, 2.5882354, 1.2352941 R3CESBU, 4, 22, 0, 5.5, 0.0 football-iauk, 9, 0, 0, 0.0, 0.0 EMPEROR, 7, 46, 14, 6.571429, 2.0
This time, some teams where affected even more severely, e.g. Mrl or Robo2D each with nearly 20 black holes per match on average. The negative impact on our own team was nearly the same in RoboCup 2022 and IranOpen 2023 with ca. 4.5 black holes per match.
Of course, one might argue that the prevention of black holes is in the responsibility of each team itself. However, we strongly belief that the wide re-occurrence of black holes from 2022 onward points to some inherent problem with the way the simulation environment has been set up recently.
As an additional safety check (for our team, FRA-UNIted, only), we also ran a large amount of matches with our team against various opponents on standard PCs (without docker or other virtualizations) using exactly the same version as we employed recently in the IranOpen 2023 tournament. We observed no black hole at all.
~~Details~~
Our analysis reveals that often the entire simulation seems to be "frozen" for one time step or for time steps that are near to one another. We can only speculate about the reasons; one such speculation would be that other docker environments (of another match that is running in parallel on the same machine) are initialized at match start-up/clean-up and this increases the overall system load for a few seconds.
Please, see the detail analysis of the exemplary match between RoboC2 and YuShan2023 during the recent IranOpen 2023 tournament in which these teams were affected by 51 and 32 black holes, respectively. All these black holes occur between time steps 600 and 800. In some time steps many players are affected simultaneously (see below: Appendix).
~~Conclusion~~
The hardware, virtualization, and software setup for upcoming tournaments must be very critically questioned in order to allow for fair competitions.
=====Appendix=====
Detail analysis of one exemplary match from IranOpen 2023:
Program arguments: ./GGHv1WP_server1_Robo2D_1-vs-YuShan2023_4.rcl I extracted 2 team names: Robo2D, YuShan2023 I extracted 22 agent names: Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11. t=623: Holes for Robo2D_1, Robo2D_3, Robo2D_8, Robo2D_10 -> {Robo2D=4} t=624: BLACK HOLE FOR Robo2D_1 t=624: BLACK HOLE FOR Robo2D_3 t=624: BLACK HOLE FOR Robo2D_8 t=624: BLACK HOLE FOR Robo2D_10 t=680: Holes for Robo2D_1, Robo2D_3, Robo2D_4, Robo2D_7, YuShan2023_1, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_7, YuShan2023_8, YuShan2023_10 -> {Robo2D=4, YuShan2023=7} t=681: BLACK HOLE FOR Robo2D_4 t=681: BLACK HOLE FOR YuShan2023_4 t=681: BLACK HOLE FOR YuShan2023_7 t=681: BLACK HOLE FOR YuShan2023_8 t=748: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_5, YuShan2023_6, YuShan2023_8, YuShan2023_10 -> {Robo2D=10, YuShan2023=7} t=749: BLACK HOLE FOR Robo2D_6 t=749: BLACK HOLE FOR Robo2D_7 t=749: BLACK HOLE FOR Robo2D_9 t=749: BLACK HOLE FOR Robo2D_10 t=749: BLACK HOLE FOR Robo2D_11 t=749: BLACK HOLE FOR YuShan2023_2 t=749: BLACK HOLE FOR YuShan2023_3 t=749: BLACK HOLE FOR YuShan2023_6 t=749: BLACK HOLE FOR YuShan2023_10 t=751: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=11, YuShan2023=9} t=752: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=752: BLACK HOLE FOR YuShan2023_2 t=752: BLACK HOLE FOR YuShan2023_3 t=752: BLACK HOLE FOR YuShan2023_5 t=752: BLACK HOLE FOR YuShan2023_6 t=752: BLACK HOLE FOR YuShan2023_7 t=752: BLACK HOLE FOR YuShan2023_9 t=752: BLACK HOLE FOR YuShan2023_10 t=753: BLACK HOLE FOR Robo2D_1 t=753: BLACK HOLE FOR Robo2D_2 t=753: BLACK HOLE FOR Robo2D_3 t=753: BLACK HOLE FOR Robo2D_4 t=753: BLACK HOLE FOR Robo2D_5 t=753: BLACK HOLE FOR Robo2D_6 t=753: BLACK HOLE FOR Robo2D_7 t=753: BLACK HOLE FOR Robo2D_8 t=753: BLACK HOLE FOR Robo2D_9 t=753: BLACK HOLE FOR Robo2D_10 t=753: BLACK HOLE FOR Robo2D_11 t=758: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=11, YuShan2023=11} t=759: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=759: BLACK HOLE FOR YuShan2023_1 t=759: BLACK HOLE FOR YuShan2023_3 t=759: BLACK HOLE FOR YuShan2023_7 t=759: BLACK HOLE FOR YuShan2023_11 t=760: BLACK HOLE FOR Robo2D_1 t=760: BLACK HOLE FOR Robo2D_2 t=760: BLACK HOLE FOR Robo2D_3 t=760: BLACK HOLE FOR Robo2D_4 t=760: BLACK HOLE FOR Robo2D_5 t=760: BLACK HOLE FOR Robo2D_6 t=760: BLACK HOLE FOR Robo2D_7 t=760: BLACK HOLE FOR Robo2D_8 t=760: BLACK HOLE FOR Robo2D_9 t=760: BLACK HOLE FOR Robo2D_10 t=760: BLACK HOLE FOR Robo2D_11 t=762: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=763: BLACK HOLE FOR Robo2D_1 t=763: BLACK HOLE FOR Robo2D_2 t=763: BLACK HOLE FOR Robo2D_3 t=763: BLACK HOLE FOR Robo2D_4 t=763: BLACK HOLE FOR Robo2D_5 t=763: BLACK HOLE FOR Robo2D_6 t=763: BLACK HOLE FOR Robo2D_7 t=763: BLACK HOLE FOR Robo2D_8 t=763: BLACK HOLE FOR Robo2D_9 t=763: BLACK HOLE FOR Robo2D_10 t=763: BLACK HOLE FOR Robo2D_11 t=779: Holes for Robo2D_1, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=10, YuShan2023=8} t=780: BLACK HOLE FOR Robo2D_1 t=780: BLACK HOLE FOR Robo2D_3 t=780: BLACK HOLE FOR Robo2D_5 t=780: BLACK HOLE FOR Robo2D_6 t=780: BLACK HOLE FOR Robo2D_8 t=780: BLACK HOLE FOR Robo2D_10 t=780: BLACK HOLE FOR Robo2D_11 t=780: BLACK HOLE FOR YuShan2023_4 t=780: BLACK HOLE FOR YuShan2023_5 t=780: BLACK HOLE FOR YuShan2023_6 t=780: BLACK HOLE FOR YuShan2023_7 t=780: BLACK HOLE FOR YuShan2023_8 t=780: BLACK HOLE FOR YuShan2023_9 t=799: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_10, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=9, YuShan2023=11} t=800: BLACK HOLE FOR Robo2D_5 t=800: BLACK HOLE FOR YuShan2023_1 t=800: BLACK HOLE FOR YuShan2023_2 t=800: BLACK HOLE FOR YuShan2023_3 t=800: BLACK HOLE FOR YuShan2023_4 t=800: BLACK HOLE FOR YuShan2023_5 t=800: BLACK HOLE FOR YuShan2023_6 t=800: BLACK HOLE FOR YuShan2023_7 t=800: BLACK HOLE FOR YuShan2023_8 TOTALHBH, Robo2D, 92, 51, YuShan2023, 53, 32,
Second example in full detail (with less black holes):
I extracted 2 team names: Tehran2D, FRA-UNIted I extracted 22 agent names: Tehran2D_1, Tehran2D_2, Tehran2D_3, Tehran2D_4, Tehran2D_5, Tehran2D_6, Tehran2D_7, Tehran2D_8, Tehran2D_9, Tehran2D_10, Tehran2D_11, FRA-UNIted_1, FRA-UNIted_2, FRA-UNIted_3, FRA-UNIted_4, FRA-UNIted_5, FRA-UNIted_6, FRA-UNIted_7, FRA-UNIted_8, FRA-UNIted_9, FRA-UNIted_10, FRA-UNIted_11. t=2: Holes for Tehran2D_2, Tehran2D_10 -> {Tehran2D=2} t=3: BLACK HOLE FOR Tehran2D_2 t=3: BLACK HOLE FOR Tehran2D_10 t=1014: Holes for FRA-UNIted_5 -> {FRA-UNIted=1} t=1015: BLACK HOLE FOR FRA-UNIted_5 t=1019: Holes for FRA-UNIted_8 -> {FRA-UNIted=1} t=1020: BLACK HOLE FOR FRA-UNIted_8 t=2753: Holes for FRA-UNIted_10 -> {FRA-UNIted=1} t=2754: BLACK HOLE FOR FRA-UNIted_10 t=4791: Holes for FRA-UNIted_2 -> {FRA-UNIted=1} t=4792: BLACK HOLE FOR FRA-UNIted_2 t=5039: Holes for FRA-UNIted_5, FRA-UNIted_6, FRA-UNIted_7, FRA-UNIted_9 -> {FRA-UNIted=4} t=5040: BLACK HOLE FOR FRA-UNIted_5 t=5040: BLACK HOLE FOR FRA-UNIted_6 t=5040: BLACK HOLE FOR FRA-UNIted_7 t=5040: BLACK HOLE FOR FRA-UNIted_9 TOTALHBH, Tehran2D, 2, 2, FRA-UNIted, 8, 8,
~~The End~~
Thanks for reading!
Hi there,
is it really beneficial to have 25 Docker instances per match instead of just one? At least this might influence the TCP communication Thomas mentioned. In my experiments I didn't see any problems in this setup (one Docker instance). But further testing is definitively crucial and probably depends on the next championship hardware setup, too.
Just my 2¢, Eicke
-- Don't take yourself too seriously. No one else does.
Prof. Dr. Eicke Godehardt Software Engineering Leiter der Lehreinheit Informatik Studiengangsleiter Inclusive Design – Intelligente Systeme M.Sc.
Frankfurt University of Applied Sciences „Wissen durch Praxis stärkt“ FB2 Computer Science and Engineering Nibelungenplatz 1, Raum 1-202 60318 Frankfurt am Main Tel.: +49(0)69/1533-2790 g*******t@fb2.fra-uas.de http://www.frankfurt-university.de
On 5/14/23 7:25 PM, Prof. Dr. Thomas Gabel via RoboCup-sim wrote:
Dear 2D Community! [Same text will be shared in Discord as well!]
TL&DR;
We analyzed the log files of major competitions during past years and found a degradation of the reliability of the hardware setup and its computational power provided to the teams. In particular, we observed that - starting in 2022 - many teams' players did frequently miss to send a command within 100ms.
If you have any questions or remarks on this, please share them. Best regards, Thomas Gabel (on behalf of the FRA-UNIted team).
=====FULL EXPLANATION=====
~~Assumptions~~
We assume that all teams do, in general, send at least one command to the soccer server each time step. By commands we refer to their (A) body-related commands (dash, kick, turn, catch, tackle) and their (B) other-parts-related commands (attentionto, turn_neck, say, change_view, pointto). It is standing to reason to assume that each player sends at least one of those commands to the server each time step. Moreover, most teams do indeed send one of the body-related commands (dash, kick, turn, catch, tackle) to the server each time step.
~~Definitions~~
If some agent does not send a command in one time step at all (neither from category (A) nor from category (B)), then usually something is wrong. In accordance to Ralf Berger from the first Soccer Simulation 2D Champion team ever (AT-Humboldt'97) we call this situation a HOLE.
If some agent sends in one time step t more than one command, then usually something is very wrong (e.g. two dashes, or a dash and a turn). It usually means that this agent did not manage to send the first of the two commands within the 100ms of the previous time step t-1, i.e. it is being sent one step too late. Simultaneously, the agent sends the command of the current time step t. Thus, the server receives the command from the previous *and* the current time step in the same cycle. In accordance to Ralf Berger from the first Soccer Simulation 2D Champion team ever (AT-Humboldt'97) we call this situation a BLACK HOLE.
~~History~~
Holes and black holes are bad things for any agent. They used to be a big issue in the early days of soccer simulation 2D, when computational resources were scarce (e.g. the 12 processes of one team had to be run on just one or two single-core PC) as well as when the network connections needed for exchanging messages via UDP were not as reliable as they are today. While we believed that holes and black holes are a problem of another decade and they occur only very rarely or due to programming errors by some team nowadays, we observed their systematic return in competitions of the last and current year where, as far as we can tell, for the first times tournaments were run on 12 docker instances for each team and one for the rcssserver (in total 25 docker instances) entirely on one single machine (which, of course, is equipped with a sufficient amount of cores, e.g. 64 cores).
~~Methodology~~
We took the log files from the 2019, 2021 and 2022 world championships plus the very recent IranOpen 2023 tournament and analyzed the rcl files for the occurrence of holes and black holes. Our analysis tool and script are available on request.
~~Findings~~
In RoboCup 2019, there were no problems across all teams (with the exception of Ri-One which we think to be due to a team-internal problem). And a minor issue for YuShan2019 (with, however, less than 1 black hole in the course of two matches).
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch
Receptivity, 22, 0, 0, 0.0, 0.0 MT2019, 20, 0, 0, 0.0, 0.0 ITAndroids, 21, 0, 0, 0.0, 0.0 Fractals2019, 22, 0, 0, 0.0, 0.0 Ri-one, 21, 18938, 572, 901.8095, 27.238094 YuShan2019, 21, 9, 9, 0.42857143, 0.42857143 CYRUS2019, 22, 0, 0, 0.0, 0.0 HillStone, 21, 0, 0, 0.0, 0.0 Razi2018, 20, 0, 0, 0.0, 0.0 FRA-UNIted, 20, 0, 0, 0.0, 0.0 FCP, 20, 0, 0, 0.0, 0.0 RoboCIn, 22, 0, 0, 0.0, 0.0 HfutEngine2019, 20, 0, 0, 0.0, 0.0 HELIOS2019, 20, 0, 0, 0.0, 0.0 Titas2019, 20, 0, 0, 0.0, 0.0
Likewise, there were no issues for RoboCup 2021:
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch
Austras2D, 35, 812029, 0, 23200.828, 0.0 Persepolis, 35, 7916, 0, 226.17143, 0.0 Oxsy, 35, 0, 0, 0.0, 0.0 YuShan2021, 36, 0, 0, 0.0, 0.0 CYRUS, 36, 0, 0, 0.0, 0.0 HfutEngine2021, 38, 11, 0, 0.28947368, 0.0 ITAndroids, 38, 0, 0, 0.0, 0.0 Hades2D, 36, 1034, 0, 28.722221, 0.0 RoboCIn, 36, 16, 0, 0.44444445, 0.0 MT2021, 36, 0, 0, 0.0, 0.0 ARAS, 35, 6, 0, 0.17142858, 0.0 Jyo, 35, 0, 0, 0.0, 0.0 HELIOS2021, 35, 8, 1, 0.22857143, 0.028571429 FRA-UNIted, 37, 0, 0, 0.0, 0.0 Alice2021, 35, 0, 0, 0.0, 0.0 ThunderLeague, 36, 0, 0, 0.0, 0.0
Some teams (e.g. Austras or Persepolis) seem to refrain from sending a command in each time step, producing what we call a hole. Each team must decide for itself whether this is intended, or acceptable, or bad behavior (so we assume that this large amount of holes is fine for the two team mentioned).
Anyways, black holes are always most likely a bad thing for any team. If we count the total number of black holes for RoboCup 2021, we see that there was just a single black hole in the entire tournament (by HELIOS2021).
Unfortunately, things changed in RoboCup 2022 where, for the first time, a docker-based environment was used. Unfortunately, the entirety of logs for that year can no longer be found on http://sea-02.cit.frankfurt-university.de:32224/?dmVyPTEuMDAxJiY3ZGZkNzk5YzI... which is why our analysis focuses only on the fraction of logs that we downloaded during the tournament:
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch
MT2022, 32, 52, 12, 1.625, 0.375 YuShan2022, 36, 416, 248, 11.555555, 6.888889 Alice2D, 32, 73671, 7, 2302.2188, 0.21875 ITAndroids, 34, 53, 6, 1.5588236, 0.1764706 Oxsy, 32, 70, 19, 2.1875, 0.59375 HELIOS2022, 31, 21, 12, 0.67741936, 0.38709676 RoboCIn, 34, 51, 16, 1.5, 0.47058824 Hfutengine2022, 38, 129, 56, 3.3947368, 1.4736842 Tehran2D, 36, 72, 18, 2.0, 0.5 CYRUS, 33, 51, 17, 1.5454545, 0.5151515 FRA-UNIted, 32, 161, 139, 5.03125, 4.34375 Hades2D, 32, 59, 7, 1.84375, 0.21875
The summed number of black wholes has been increased to 557 in the entire tournament - a number which is certainly far beyond anything that is acceptable. Apparently, all teams were affected where YuShan2022 was affected most with almost 7 black holes per game on average.
As a safety check, we ran our analysis also on a big part of the log files produced by the recent IranOpen 2023 tournament which was also held in a docker-based environment with each agent/coach running in its own docker instance, all on the same single machine.
Team Name, #matches, #holesTotal, #blackholesTotal, holesPerMatch, blackHolesPerMatch
YuShan2023, 15, 63, 32, 4.2, 2.1333334 FRA-UNIted, 22, 113, 100, 5.1363635, 4.5454545 Mcafee, 8, 26, 4, 3.25, 0.5 Mrl, 14, 361, 289, 25.785715, 20.642857 The8, 20, 54, 13, 2.7, 0.65 Robo2D, 7, 242, 131, 34.57143, 18.714285 Pyrus, 14, 50, 23, 3.5714285, 1.6428572 Apollo2d, 9, 0, 0, 0.0, 0.0 RoboCIn, 15, 11, 0, 0.73333335, 0.0 ThunderLeague, 12, 11, 0, 0.9166667, 0.0 Zina-AI, 17, 44, 21, 2.5882354, 1.2352941 R3CESBU, 4, 22, 0, 5.5, 0.0 football-iauk, 9, 0, 0, 0.0, 0.0 EMPEROR, 7, 46, 14, 6.571429, 2.0
This time, some teams where affected even more severely, e.g. Mrl or Robo2D each with nearly 20 black holes per match on average. The negative impact on our own team was nearly the same in RoboCup 2022 and IranOpen 2023 with ca. 4.5 black holes per match.
Of course, one might argue that the prevention of black holes is in the responsibility of each team itself. However, we strongly belief that the wide re-occurrence of black holes from 2022 onward points to some inherent problem with the way the simulation environment has been set up recently.
As an additional safety check (for our team, FRA-UNIted, only), we also ran a large amount of matches with our team against various opponents on standard PCs (without docker or other virtualizations) using exactly the same version as we employed recently in the IranOpen 2023 tournament. We observed no black hole at all.
~~Details~~
Our analysis reveals that often the entire simulation seems to be "frozen" for one time step or for time steps that are near to one another. We can only speculate about the reasons; one such speculation would be that other docker environments (of another match that is running in parallel on the same machine) are initialized at match start-up/clean-up and this increases the overall system load for a few seconds.
Please, see the detail analysis of the exemplary match between RoboC2 and YuShan2023 during the recent IranOpen 2023 tournament in which these teams were affected by 51 and 32 black holes, respectively. All these black holes occur between time steps 600 and 800. In some time steps many players are affected simultaneously (see below: Appendix).
~~Conclusion~~
The hardware, virtualization, and software setup for upcoming tournaments must be very critically questioned in order to allow for fair competitions.
=====Appendix=====
Detail analysis of one exemplary match from IranOpen 2023:
Program arguments: ./GGHv1WP_server1_Robo2D_1-vs-YuShan2023_4.rcl I extracted 2 team names: Robo2D, YuShan2023 I extracted 22 agent names: Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11. t=623: Holes for Robo2D_1, Robo2D_3, Robo2D_8, Robo2D_10 -> {Robo2D=4} t=624: BLACK HOLE FOR Robo2D_1 t=624: BLACK HOLE FOR Robo2D_3 t=624: BLACK HOLE FOR Robo2D_8 t=624: BLACK HOLE FOR Robo2D_10 t=680: Holes for Robo2D_1, Robo2D_3, Robo2D_4, Robo2D_7, YuShan2023_1, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_7, YuShan2023_8, YuShan2023_10 -> {Robo2D=4, YuShan2023=7} t=681: BLACK HOLE FOR Robo2D_4 t=681: BLACK HOLE FOR YuShan2023_4 t=681: BLACK HOLE FOR YuShan2023_7 t=681: BLACK HOLE FOR YuShan2023_8 t=748: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_5, YuShan2023_6, YuShan2023_8, YuShan2023_10 -> {Robo2D=10, YuShan2023=7} t=749: BLACK HOLE FOR Robo2D_6 t=749: BLACK HOLE FOR Robo2D_7 t=749: BLACK HOLE FOR Robo2D_9 t=749: BLACK HOLE FOR Robo2D_10 t=749: BLACK HOLE FOR Robo2D_11 t=749: BLACK HOLE FOR YuShan2023_2 t=749: BLACK HOLE FOR YuShan2023_3 t=749: BLACK HOLE FOR YuShan2023_6 t=749: BLACK HOLE FOR YuShan2023_10 t=751: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=11, YuShan2023=9} t=752: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=752: BLACK HOLE FOR YuShan2023_2 t=752: BLACK HOLE FOR YuShan2023_3 t=752: BLACK HOLE FOR YuShan2023_5 t=752: BLACK HOLE FOR YuShan2023_6 t=752: BLACK HOLE FOR YuShan2023_7 t=752: BLACK HOLE FOR YuShan2023_9 t=752: BLACK HOLE FOR YuShan2023_10 t=753: BLACK HOLE FOR Robo2D_1 t=753: BLACK HOLE FOR Robo2D_2 t=753: BLACK HOLE FOR Robo2D_3 t=753: BLACK HOLE FOR Robo2D_4 t=753: BLACK HOLE FOR Robo2D_5 t=753: BLACK HOLE FOR Robo2D_6 t=753: BLACK HOLE FOR Robo2D_7 t=753: BLACK HOLE FOR Robo2D_8 t=753: BLACK HOLE FOR Robo2D_9 t=753: BLACK HOLE FOR Robo2D_10 t=753: BLACK HOLE FOR Robo2D_11 t=758: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=11, YuShan2023=11} t=759: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=759: BLACK HOLE FOR YuShan2023_1 t=759: BLACK HOLE FOR YuShan2023_3 t=759: BLACK HOLE FOR YuShan2023_7 t=759: BLACK HOLE FOR YuShan2023_11 t=760: BLACK HOLE FOR Robo2D_1 t=760: BLACK HOLE FOR Robo2D_2 t=760: BLACK HOLE FOR Robo2D_3 t=760: BLACK HOLE FOR Robo2D_4 t=760: BLACK HOLE FOR Robo2D_5 t=760: BLACK HOLE FOR Robo2D_6 t=760: BLACK HOLE FOR Robo2D_7 t=760: BLACK HOLE FOR Robo2D_8 t=760: BLACK HOLE FOR Robo2D_9 t=760: BLACK HOLE FOR Robo2D_10 t=760: BLACK HOLE FOR Robo2D_11 t=762: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11 -> {Robo2D=11} t=763: BLACK HOLE FOR Robo2D_1 t=763: BLACK HOLE FOR Robo2D_2 t=763: BLACK HOLE FOR Robo2D_3 t=763: BLACK HOLE FOR Robo2D_4 t=763: BLACK HOLE FOR Robo2D_5 t=763: BLACK HOLE FOR Robo2D_6 t=763: BLACK HOLE FOR Robo2D_7 t=763: BLACK HOLE FOR Robo2D_8 t=763: BLACK HOLE FOR Robo2D_9 t=763: BLACK HOLE FOR Robo2D_10 t=763: BLACK HOLE FOR Robo2D_11 t=779: Holes for Robo2D_1, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_9, Robo2D_10, Robo2D_11, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=10, YuShan2023=8} t=780: BLACK HOLE FOR Robo2D_1 t=780: BLACK HOLE FOR Robo2D_3 t=780: BLACK HOLE FOR Robo2D_5 t=780: BLACK HOLE FOR Robo2D_6 t=780: BLACK HOLE FOR Robo2D_8 t=780: BLACK HOLE FOR Robo2D_10 t=780: BLACK HOLE FOR Robo2D_11 t=780: BLACK HOLE FOR YuShan2023_4 t=780: BLACK HOLE FOR YuShan2023_5 t=780: BLACK HOLE FOR YuShan2023_6 t=780: BLACK HOLE FOR YuShan2023_7 t=780: BLACK HOLE FOR YuShan2023_8 t=780: BLACK HOLE FOR YuShan2023_9 t=799: Holes for Robo2D_1, Robo2D_2, Robo2D_3, Robo2D_4, Robo2D_5, Robo2D_6, Robo2D_7, Robo2D_8, Robo2D_10, YuShan2023_1, YuShan2023_2, YuShan2023_3, YuShan2023_4, YuShan2023_5, YuShan2023_6, YuShan2023_7, YuShan2023_8, YuShan2023_9, YuShan2023_10, YuShan2023_11 -> {Robo2D=9, YuShan2023=11} t=800: BLACK HOLE FOR Robo2D_5 t=800: BLACK HOLE FOR YuShan2023_1 t=800: BLACK HOLE FOR YuShan2023_2 t=800: BLACK HOLE FOR YuShan2023_3 t=800: BLACK HOLE FOR YuShan2023_4 t=800: BLACK HOLE FOR YuShan2023_5 t=800: BLACK HOLE FOR YuShan2023_6 t=800: BLACK HOLE FOR YuShan2023_7 t=800: BLACK HOLE FOR YuShan2023_8 TOTALHBH, Robo2D, 92, 51, YuShan2023, 53, 32,
Second example in full detail (with less black holes):
I extracted 2 team names: Tehran2D, FRA-UNIted I extracted 22 agent names: Tehran2D_1, Tehran2D_2, Tehran2D_3, Tehran2D_4, Tehran2D_5, Tehran2D_6, Tehran2D_7, Tehran2D_8, Tehran2D_9, Tehran2D_10, Tehran2D_11, FRA-UNIted_1, FRA-UNIted_2, FRA-UNIted_3, FRA-UNIted_4, FRA-UNIted_5, FRA-UNIted_6, FRA-UNIted_7, FRA-UNIted_8, FRA-UNIted_9, FRA-UNIted_10, FRA-UNIted_11. t=2: Holes for Tehran2D_2, Tehran2D_10 -> {Tehran2D=2} t=3: BLACK HOLE FOR Tehran2D_2 t=3: BLACK HOLE FOR Tehran2D_10 t=1014: Holes for FRA-UNIted_5 -> {FRA-UNIted=1} t=1015: BLACK HOLE FOR FRA-UNIted_5 t=1019: Holes for FRA-UNIted_8 -> {FRA-UNIted=1} t=1020: BLACK HOLE FOR FRA-UNIted_8 t=2753: Holes for FRA-UNIted_10 -> {FRA-UNIted=1} t=2754: BLACK HOLE FOR FRA-UNIted_10 t=4791: Holes for FRA-UNIted_2 -> {FRA-UNIted=1} t=4792: BLACK HOLE FOR FRA-UNIted_2 t=5039: Holes for FRA-UNIted_5, FRA-UNIted_6, FRA-UNIted_7, FRA-UNIted_9 -> {FRA-UNIted=4} t=5040: BLACK HOLE FOR FRA-UNIted_5 t=5040: BLACK HOLE FOR FRA-UNIted_6 t=5040: BLACK HOLE FOR FRA-UNIted_7 t=5040: BLACK HOLE FOR FRA-UNIted_9 TOTALHBH, Tehran2D, 2, 2, FRA-UNIted, 8, 8,
~~The End~~
Thanks for reading!
participants (2)
-
Eicke Godehardt -
Prof. Dr. Thomas Gabel