There is a myth about the first Olympic baseball to have been played in St. Louis in the year 1904. Historians are skeptical about the accuracy as most sports events held at St. Louis were titled Olympic events.
The first recorded Olympic event dates to 1912. A baseball team from Vasteras competed against the US team at the Olympic Games in Stockholm in Sweden. The US team beat Sweden in two games, with Jim Thorpe as a right fielder. During the Olympics held in 1936, the German hosts invited the United States to demonstrate a match against Japan. For an amateur crowd of 125,000, the World Champions team from the US won by a lead of 6-5.
In the Olympics of 1940, the original plan to include baseball was discarded as Japan withdrew due to war in Manchuria.
Another baseball demonstration took place in 1954 at the Melbourne Olympics in Australia. A team consisting of the U.S. Far East Command servicemen played in Australia. During the match, the audience gathered for the other athletic events summed up to a striking crowd of 114,000 spectators. The US won the match by 11-5.
The Olympic events in Tokyo took place in 1964. Baseball was very popular in Japan and a team consisting of American college baseball players along with eight major league players fielded against the Japanese all-star team. They won the championship by 6-2. Again in 1981, the contest took place between teams from United States, Japan, South Korea, Dominican Republic, Canada, Taiwan, Italy, and Nicaragua. Japan and the USA reached the finals and the Americans won by 6-3.
Another demonstration tournament in Seoul, South Korea in 1988 and Cuba, the dominant team winning all major international championships since 1984, again boycotted the Games just like in 1981. The tournament was held between teams from United States, Japan, South Korea, Puerto Rico, Canada, Taiwan, Netherlands, and Australia, Japan and the US again reached the final. The United States won by 5-3.
The 1986 IOC Congress decided in a meeting that the first official Olympic baseball tournament be held in Barcelona in 1992. It was decided at the 117th IOC Session to remove baseball and softball from the 28 existing Olympic sports to be held in 2012 in London. A major fraction of the world does not play baseball, but the driving factor was also the reluctance of Major League Baseball to have breaks during the games.